Way back (circa 1988ish timeframe) I remember a digital logic professor giving a little aside on the 8087 and remarking at the time that it (the 8087) used some three value logic circuits (or maybe four value logic). That instead of it being all binary, some parts used base 3 (or 4) to squeeze more onto the chip.
From your microscopic investigations, have you seen any evidence that any part of the chip uses anything other than base 2 logic?
I wrote about this in detail a few years ago: https://www.righto.com/2018/09/two-bits-per-transistor-high-...
It is notable that ARM designed explicit co-processor instructions, allowing for 16 co-processors. They must have taken the 8086/8087 approach into account when doing that.
Datasheet: https://www.hartetechnologies.com/manuals/AMD/AMD%209511%20F...
Instead of simply reading the first word of a memory operand and otherwise ignoring ESC opcodes, the CPU had to be aware of several different groups of FPU opcodes to set up the transfer, with a special register inside its BIU to hold the direction (read or write), address, and segment limit for the operand.
It didn't do all protection checks "up front", since that would have required even more microcode, and also they likely wanted to keep the interface flexible enough to support new instructions. At that time I think Intel also had planned other types of coprocessor for things like cryptography or business data processing, those would have used the same interface but with completely different operand lengths.
So the CPU had to check the current address against the segment limit in the background whenever the coprocessor requested to transfer the next word. This is why there was a separate exception for "coprocessor segment overrun". Then of course the 486 integrated the FPU and made it all obsolete again.