NVIDIA has introduced NVIDIA Dynamo, an open-source software designed to enhance and scale AI reasoning models in AI factories. The new inference-serving software aims to optimize token revenue generation by efficiently managing AI inference requests across extensive GPU networks.
Jensen Huang, founder and CEO of NVIDIA, stated, “Industries around the world are training AI models to think and learn in different ways, making them more sophisticated over time. To enable a future of custom reasoning AI, NVIDIA Dynamo helps serve these models at scale, driving cost savings and efficiencies across AI factories.”
NVIDIA Dynamo succeeds the NVIDIA Triton Inference Server and is engineered to improve performance by orchestrating inference communication among thousands of GPUs. It employs disaggregated serving to separate processing and generation phases on different GPUs for large language models (LLMs), optimizing each phase independently.
The platform reportedly doubles the performance and revenue for AI factories using Llama models on the current NVIDIA Hopper platform. For instance, running the DeepSeek-R1 model on a large cluster of GB200 NVL72 racks enhances token generation by over 30 times per GPU due to intelligent inference optimizations.
Denis Yarats, chief technology officer of Perplexity AI, commented on its potential: “To handle hundreds of millions of requests monthly, we rely on NVIDIA GPUs and inference software to deliver the performance, reliability and scale our business and users demand. We look forward to leveraging Dynamo...to drive even more inference-serving efficiencies.”
Cohere plans to utilize NVIDIA Dynamo for agentic AI capabilities in its Command series. Saurabh Baji from Cohere noted that scaling advanced models requires sophisticated scheduling and coordination which they expect from this new platform.
Ce Zhang from Together AI highlighted the importance of advanced techniques like disaggregated serving for scaling reasoning models effectively: “Together AI provides industry-leading performance using our proprietary inference engine. The openness and modularity of NVIDIA Dynamo will allow us to seamlessly plug its components into our engine.”
NVIDIA Dynamo introduces four innovations aimed at reducing costs while improving user experience: a GPU Planner for dynamic resource allocation; a Smart Router minimizing recomputations; a Low-Latency Communication Library for efficient data transfer; and a Memory Manager optimizing memory usage without impacting user experience.
The platform will be integrated into NVIDIA NIM microservices with future support planned within the NVIDIA AI Enterprise software platform.
For further details about NVIDIA Dynamo's capabilities or upcoming sessions at GTC 2025, interested parties can watch related keynotes or read additional materials provided by NVIDIA.
