Supermicro has announced a new KV cache storage solution developed with IBM and NVIDIA, aimed at improving performance for large-scale AI inference. The joint offering integrates Supermicro Petascale all-flash storage servers using AMD EPYC 9005 Series processors (“Turin”), IBM Storage Scale software, NVIDIA Dynamo inference framework software and NVIDIA Spectrum-X Ethernet networking.

According to the company, the combined technologies can reduce the time it takes AI models to begin generating a response, an advantage it frames as critical as enterprises adopt agentic and long-context AI workflows. Extending KV cache capacity beyond local GPU memory is described as a way to improve GPU utilization and inference throughput, while enabling high availability of KV cache content and non-disruptive scaling of KV cache capacity.

How the KV cache approach works

Supermicro explains that inference workloads consist of two phases: prefill and decode. During prefill, input text is converted into tokens and processed by the LLM to generate the intermediate key-value (KV) states needed for inference. By storing these previously computed KV states in a persistent KV cache, the system can reuse them instead of recomputing the same context for subsequent requests.

Agentic workflows, coding assistants, long-context RAG, multimodal AI and multi-turn applications can generate substantial KV state as they process large codebases, documents, conversations, images and other context, quickly exceeding GPU and CPU memory and driving the need to offload KV cache to high-capacity, shared network storage. The solution from Supermicro, IBM and NVIDIA provides this persistent storage layer, helping improve inference efficiency, throughput and scalability while reducing redundant computation.

Testing results and customer benefits

Supermicro said the three companies conducted extensive cluster-level testing to measure performance, including high concurrency and noisy-neighbor network traffic scenarios. In joint testing by Supermicro and NVIDIA with a 120-billion-parameter LLM, persistent KV cache delivered up to 56x faster time-to-first-token (TTFT) at 130,000-token prompts. As input length increased from 30,000 to 130,000 tokens, TTFT increased by only 0.377 seconds with KV cache, compared with 29.55 seconds without it, according to the companies.

Supermicro lists the following customer benefits for the solution:

  • Joint solution validation across high-concurrency and noisy-neighbor scenarios
  • Dramatically faster and more consistent inference performance for long-context workloads
  • Higher GPU utilization and throughput by keeping frequently reused context available outside GPU memory, reducing GPU idle time and supporting greater concurrency without proportionally increasing GPU resources
  • Enterprise resilience through a persistent, highly available KV cache that protects cached inference state and provides shared access across GPU nodes and clusters
  • A validated, turnkey architecture combining NVIDIA Dynamo, IBM Storage Scale software, Supermicro Petascale servers and NVIDIA Spectrum-X networking
  • Accelerated deployment services, with Supermicro's factory rack integration and on-site deployment services able to reduce deployment time from weeks to days

Executive comments

"As AI workloads expand from training to inference, enterprises and neoclouds deploying high-throughput inference workloads characteristic of agentic AI and long-context, multi-turn generative AI need to remove bottlenecks to improve GPU efficiency,“ said Charles Liang, president and CEO of Supermicro. ”Our solution with IBM and NVIDIA clearly demonstrates these performance benefits."
"Enterprises need a fundamentally better way to scale AI inference. Together, IBM, Supermicro, and NVIDIA are delivering a high-performance, resilient KV cache solution that helps customers maximize their GPU investments, scale independently of compute, and significantly improve infrastructure efficiency,“ said Sam Werner, GM, IBM Storage. ”This collaboration represents an important step toward making enterprise AI inference more scalable, reliable, and cost-effective."
"Agentic AI and long-context applications depend on reusing context efficiently across many steps, and NVIDIA Dynamo helps orchestrate distributed inference so enterprises can get more usable throughput from every GPU,“ said Shruti Koparkar, NVIDIA AI Inference Product Lead. ”By pairing NVIDIA Dynamo and Spectrum-X Ethernet with Supermicro Petascale storage servers and IBM Storage Scale, this KV cache solution creates a high-performance shared storage tier that can reduce redundant computation, improve time-to-first-token and help customers scale resilient AI inference services."

Company background

Supermicro (NASDAQ: SMCI) describes itself as a global leader in Application-Optimized Total IT Solutions. Founded and operating in San Jose, California, the company says it is committed to delivering first-to-market innovation for Enterprise, Cloud, AI and 5G Telco/Edge IT Infrastructure. Its portfolio includes server, AI, storage, IoT, switch systems, software and support services, with products designed and manufactured in-house in the US, Asia and the Netherlands.