> We wanted FoundationDB to survive failures of machines, networks, disks, clocks, racks, data centers, file systems, etc., so we created a simulation framework closely tied to Flow. By replacing physical interfaces with shims, replacing the main epoll-based run loop with a time-based simulation, and running multiple logical processes as concurrent Flow Actors, Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-thread! Even better, we are able to execute this simulation in a deterministic way, enabling us to reproduce problems and add instrumentation ex post facto. This incredible capability enabled us to build FoundationDB exclusively in simulation for the first 18 months and ensure exceptional fault tolerance long before it sent its first real network packet. For a database with as strong a contract as the FoundationDB, testing is crucial, and over the years we have run the equivalent of a trillion CPU-hours of simulated stress testing.
[1]https://pierrezemb.fr/posts/notes-about-foundationdb/#simula...
The best system for this I've ever used was Thrift, which properly abstracts data formats, transports and so on.
https://thrift.apache.org/docs/Languages.html
Unfortunately Thrift is a dead (AKA "Apache") project and it doesn't seem like anyone since has tried to do this. It probably didn't help that there are so many gaps in that support matrix. I think "Google have made a thing! Let's blindly use it!" also helped contribute to its downfall, despite Thrift being better than Protobuf (it even supports required fields!).
Actually I just took a look at the Thrift repo and there are a surprising number of commits from a couple of people consistently, so maybe it's not quite as dead as I thought. You never hear about people picking it for new projects though.
Tigris uses it: https://www.tigrisdata.com/blog/building-a-database-using-fo...
A good collection of papers, blog posts, talks, etc.: https://github.com/FoundationDB/awesome-foundationdb
1: https://apple.github.io/foundationdb/configuration.html#choo...
I've never spent less time thinking about a data store that I use daily.
But I wonder if this can be a better abstraction than async. (And whether I can build something like this in existing Rust.)
This is more node.js-like communication than erlang.