I built StreamHouse, an open-source streaming platform that replaces Kafka's broker-managed storage with direct S3 writes. The goal: same semantics, fraction of the cost.

How it works: Producers batch and compress records, a stateless server manages partition routing and metadata (SQLite for dev, PostgreSQL for prod), and segments land directly in S3. Consumers read from S3 with a local segment cache. No broker disks to manage, no replication factor to tune — S3 gives you 11 nines of durability out of the box.

What's there today: - Producer API with batching, LZ4 compression, and offset tracking (62K records/sec) - Consumer API with consumer groups, auto-commit, and multi-partition fanout (30K+ records/sec) - Kafka-compatible protocol (works with existing Kafka clients) - REST API, gRPC API, CLI, and a web UI - Docker Compose setup for trying it locally in 5 minutes

  What's not there yet:
  - Battle-tested production deployments (I'm the only user so far)
  - Connectors for consumers to immediately connect to (i.e clickhouse, elastic search etc)
  

Written in Rust, 15 crates thus far. Apache 2.0 licensed.

GitHub: https://github.com/gbram1/streamhouse How it works blog on my main website: https://streamhouse.app/how-it-works

Happy to answer questions about the architecture, tradeoffs, or what I learned building this.