NVIDIA Dynamo 1.0 Ships With 7x Inference Boost for AI Data Centers
Luisa Crawford Mar 16, 2026 21:10
NVIDIA releases Dynamo 1.0, an open-source inference OS adopted by AWS, Azure, Google Cloud, and major AI companies. Claims 7x performance gains on Blackwell GPUs.
NVIDIA shipped Dynamo 1.0 on March 16, 2026, marking the production release of what the company calls the first operating system purpose-built for AI inference at data center scale. The open-source framework has already secured adoption from AWS, Microsoft Azure, Google Cloud, and Oracle Cloud Infrastructure, alongside production deployments at Perplexity, PayPal, Pinterest, and Cursor.
The headline number: a 7x increase in requests served on NVIDIA Blackwell GPUs, according to the SemiAnalysis InferenceX benchmark running DeepSeek R1-0528. That performance gain comes from Dynamo's disaggregated serving architecture combined with wide expert parallel processing across GB200 NVL72 systems.
What Dynamo Actually Does
Modern AI reasoning models have grown too large for single GPUs. Dynamo orchestrates inference workloads across multiple GPU nodes, handling the coordination that becomes nightmarish at scale. The framework splits work into three core components: a GPU Planner for dynamic resource management, a Smart Router that optimizes request distribution based on KV cache state, and a memory manager that shuttles data between GPU memory and cheaper storage tiers.
For enterprises running agentic AI workflows—where multiple models interact with external tools—Dynamo introduces "agent hints" that let applications signal latency sensitivity and expected output length. Running with NVIDIA's NeMo Agent Toolkit, this delivered 4x lower time-to-first-token and 1.5x higher throughput on Llama 3.1 using Hopper GPUs.
Production Adoption Accelerates
The adopter list reads like a who's who of cloud and AI infrastructure. AstraZeneca, ByteDance, CoreWeave, Tencent Cloud, and Together AI have deployed Dynamo in production. Storage vendors including Dell, IBM, NetApp, and WEKA have built integrations for KV cache offloading beyond GPU memory limits.
Open source integration runs deep. SGLang, vLLM, and TensorRT LLM all use Dynamo's NIXL library for KV cache transfers. LangChain built a direct integration for injecting routing hints. Microsoft contributed deployment guides and hardening patches after testing on Azure Kubernetes Service.
New Capabilities in 1.0
ModelExpress cuts replica startup time by 7x for large mixture-of-experts models like DeepSeek v3. Instead of each new worker downloading and initializing weights independently, Dynamo loads once and streams weights over NVLink to additional GPUs.
Multimodal workloads get dedicated optimizations. Disaggregated encode/prefill/decode separates image processing from text generation, with an embedding cache that skips GPU encoding for repeated images—yielding 30% faster time-to-first-token on the Qwen3-VL-30B model.
Video generation support arrived through integrations with FastVideo and SGLang Diffusion. NVIDIA demonstrated generating a 5-second video in roughly 40 seconds on a single Hopper GPU using Wan2.1.
The Infrastructure Play
Dynamo fits NVIDIA's broader strategy of owning the full AI stack beyond silicon. As inference costs become the dominant expense for AI deployments, software that squeezes more throughput from existing hardware becomes as valuable as the GPUs themselves. The open-source approach—unusual for NVIDIA—suggests the company views ecosystem lock-in as more valuable than licensing revenue.
For data center operators evaluating Blackwell purchases, Dynamo's performance claims change the ROI math. A 7x throughput improvement on the same hardware effectively slashes per-inference costs, though real-world results will vary based on model architecture and workload patterns. The framework's roadmap targets reinforcement learning and expanded multimodal capabilities—areas where inference demands are only growing.
Image source: Shutterstock.jpg)