With long Chain-of-Thought reasoning and multi-turn agents, prompts are getting much longer. According to OpenRouter’s State of AI 2025, average context length has grown about 4× in the past year.

This creates two problems in inference systems:

• Slow TTFT — long contexts make prefill expensive • High GPU memory cost — KV cache quickly exhausts HBM

Instead of recomputing long prompts or keeping all KV cache in GPU memory, we explored a different approach:

treat KV cache as a distributed memory system.

SiMM is an open-source distributed KV cache engine for LLM inference. It stores KV cache in a high-speed RDMA-backed memory pool and lets engines like SGLang and vLLM reuse cached states across requests.

This converts prefill from a compute-heavy step into a fast I/O lookup.

In our tests with long-context multi-turn workloads:

3.1× speedup vs no cache

2.1× vs local CPU cache

up to 9× lower KV I/O latency

SiMM scales horizontally across nodes and fully utilizes RDMA NIC bandwidth.

GitHub: https://github.com/scitix/SiMM