Performance Optimized

NVIDIA Dynamo Simply Explained

An open-source framework designed to maximize efficiency for Large Language Model (LLM) inference across multiple GPUs at data center scale.

Key Features for High Performance

Disaggregated Serving

Splits the inference process into phases:

  • Prefill: Reading and understanding the initial prompt.
  • Decode: Generating the output tokens one by one.

By separating these phases, Dynamo can assign them to different GPUs, optimizing throughput for each specific task.

KV-Aware Routing

Minimizes redundant computations by tracking the Key-Value (KV) cache, which acts as the model's memory for context.

Dynamo intelligently routes incoming requests to GPUs that already contain relevant cached context, speeding up response times.

Understanding Disaggregated Serving

Request In
→
GPU A
Prefill Phase
(Process Prompt)
→
GPU B
Decode Phase
(Generate Output)
→
Response Out

Visualizing KV-Aware Routing

Request with Context
(e.g., a followup question)
→
Dynamo Router
Checks: Which GPU has this context?
GPU C
Empty Cache
GPU D
Context Found!
(Processing Request)

The router skips GPU C and sends the request directly to GPU D, avoiding the slow process of rebuilding context.

Think of it as the intelligent orchestrator or traffic controller for a large fleet of GPUs, efficiently distributing work to avoid bottlenecks.