Basically, a single hardware device like a network card pretends to be a whole bunch (say 16) virtual devices. Each device can be passed as a PCIe device to a guest VM, and will be handled inside as a hardware device. So your Windows VM will need the Broadcom driver or whatnot, rather than using the VirtIO one.
Why do this? Part because it turns out that putting your VM host's hardware interface into a Linux software bridge disables part of the hardware acceleration. This can actually make it so that you can't reach the full bandwidth of the device. On slower CPUs this may mean you can't get to 10 Gbps.
Part because there's overhead in the VM transition and this also greatly reduces this.
I also like that it doesn't need you to fiddle with the network configuration on the host.
It's well supported, including on some consumer motherboards but you have to do a bunch of fiddling in the BIOS config to enable it.
If both VMs are on the same host, is there any way to essentially achieve RDMA? VM1 says to VM2, "It's in memory at this location", and VM2 just reads directly from that memory location without a copy by the CPU?
I'm no expert, obviously, but I fail to see why VM to VM memory operations should be slower than RAM sans some latency increase due to setting up the operation.
Hyper-V has something akin to this that they call "Hyper-V sockets" [2]. But it seems it only works between guest and host.
[1] https://www.qemu.org/docs/master/system/devices/ivshmem.html
[2] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-...
macvlan: is VEPA mode still a thing that people use, or did it not take off (in terms of switches that support it--last time I looked I didn't find anything newer than about 10 years old that talked about VEPA but maybe I suck at searching?)
macvlan: 'bridge' sounds really convenient, if you try it you'll find that the host can't communicate with macvtap interfaces.
macvtap: does this suffer from the same limitation as macvlan in 'bridge' mode (host can't communicate with mactvap interfaces?
In particular I've learned from that doc that there's special handling for putting a vlan device on top of a bridge (br0.123) even if the bridge is vlan unaware.
DSA might also be relevant if you're working with hardware that supports it: https://docs.kernel.org/networking/dsa/dsa.html
[1]: https://blog.ipspace.net/2012/02/edge-virtual-bridging-8021q...
OpenWRT (what I use) also allows such interfacing using config files. It’s Linux anyway though.