Agentic AI scaling requires new memory architecture

Agentic AI represents a distinct evolution from stateless chatbots toward complex workflows, and scaling it requires a new memory architecture.

As basic models expand toward trillions of parameters and context windows into millions of tokens, the computational cost of remembering history rises faster than the ability to process it.

Organizations deploying these systems are now facing a bottleneck as the sheer volume of “long-term memory” (technically known as key-value (KV) cache) overwhelms existing hardware architectures.

Current infrastructure forces a binary choice: store inference context in scarce high-bandwidth GPU memory (HBM) or move it to slow general-purpose storage. The former is too expensive for large contexts; The latter creates latency that makes real-time agent interactions unviable.

To address this growing disparity that hinders scaling of agentive AI, NVIDIA introduced the Inferential Context Memory Storage (ICMS) platform within its Rubin architecture, proposing a new storage layer specifically designed to handle the ephemeral, high-speed nature of AI memory.

“AI is revolutionizing the entire computing system, and now in the storage space,” Huang said. “AI is no longer about one-time chatbots, but about intelligent collaborators who understand the physical world, think across long horizons, stay grounded in facts, use tools to do real work, and retain short- and long-term memory.”

The operational challenge lies in the specific behavior of transformer-based models. To avoid recalculating the entire conversation history for each new word generated, the models store previous states in the KV cache. In a proxy workflow, this cache acts as persistent memory across tools and sessions, and grows linearly with sequence length.

This creates a distinct data class. Unlike financial or customer records, KV cache is derived data; It is essential for immediate performance but does not require high durability guarantees for enterprise file systems. General-purpose storage clusters run on standard CPUs, and consume energy in metadata management and replication that proxy workloads do not require.

The current hierarchy, extending from GPU HBM (G1) to shared storage (G4), has become ineffective:

As context leaks from the GPU (G1) to system RAM (G2) and eventually to shared storage (G4), efficiency decreases. Moving active context to the G4 layer results in millisecond latency and increases the energy cost per token, leaving expensive GPUs idle while waiting for data.

For organizations, this manifests itself in an inflated total cost of ownership (TCO), where energy is wasted on overhead infrastructure rather than active thought.

New memory layer for AI factory

The industry response includes introducing a purpose-built layer into this hierarchy. The ICMS platform creates a “G3.5” layer – an Ethernet-connected flash layer designed explicitly for Gigabit inference.

This approach integrates storage directly into the compute room. By using the NVIDIA BlueField-4 data processor, the platform offloads the management of this context data from the host CPU. The system provides petabytes of shared capacity per pod, promoting scaling of agent AI by allowing agents to retain massive amounts of history without occupying an expensive HBM.

The operational benefit is quantifiable in productivity and energy. By keeping the relevant context in this middle layer—which is faster than standard storage, but cheaper than HBM—the system can “replace” memory back to the GPU before it’s needed. This reduces GPU decoder idle time, enabling up to 5x more tokens per second (TPS) for long-context workloads.

From an energy perspective, the impacts are equally measurable. Because the architecture eliminates the overhead of general-purpose storage protocols, it provides 5x better energy efficiency than traditional methods.

Data plane merging

Implementing this architecture requires a change in how IT teams view storage networks. The ICMS platform relies on NVIDIA Spectrum-X Ethernet to provide the high bandwidth and low-jitter communication required to treat flash storage almost as if it were local memory.

For enterprise infrastructure teams, the point of integration is the orchestration layer. Frameworks such as NVIDIA Dynamo and the Inference Transfer Library (NIXL) manage the movement of KV blocks between layers.

These tools coordinate with the storage layer to ensure that the correct context is loaded into GPU (G1) or host memory (G2) exactly when the AI ​​model requires it. The NVIDIA DOCA framework also supports this by providing a KV communication layer that treats the context cache as a first-class resource.

Major storage vendors are already compatible with this architecture. Companies including AIC, Cloudian, DDN, Dell Technologies, HPE, Hitachi Vantara, IBM, Nutanix, Pure Storage, Supermicro, VAST Data and WEKA are building platforms with BlueField-4. These solutions are expected to be available in the second half of this year.

Redefining infrastructure to scale agentic AI

The adoption of a dedicated context memory layer impacts capacity planning and data center design.

• Data reclassification: IT managers should recognize KV cache as a unique data type. It is “ephemeral but latency sensitive,” different from compliance data that is “hard and cold.” The G3.5 layer handles the first layer, allowing durable G4 storage to focus on long-term records and artifacts.

• Organizational maturity: Success depends on software that can place workloads intelligently. The system uses a topology-aware format (via NVIDIA Grove) to place tasks close to their cached context, reducing data movement across the fabric.

• Energy Density: By installing more usable capacity in the same rack space, organizations can extend the life of existing facilities. However, this results in higher computing density per square metre, requiring proper cooling and power distribution planning.

The move to agentic AI forces a physical reconfiguration of the data center. The prevailing paradigm of completely decoupling computation from slow, persistent storage is incompatible with the real-time retrieval needs of agents with photographic memories.

By providing a specialized contextual layer, organizations can decouple typical memory growth from the cost of GPU HBM. This architecture for agent AI allows multiple agents to share a massive pool of low-power memory to reduce the cost of serving complex queries and promote scaling by enabling high-throughput reasoning.

As organizations plan their next cycle of infrastructure investment, evaluating the efficiency of the memory hierarchy will be as vital as choosing the GPU itself.

See also: The AI ​​chip wars of 2025: What enterprise leaders have learned about the reality of supply chain

Want to learn more about AI and Big Data from industry leaders? Check out the Artificial Intelligence and Big Data Expo taking place in Amsterdam, California and London. This comprehensive event is part of TechEx and is co-located with other leading technology events. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.