NVIDIA Dynamo Architecture

Data Plane

Control Plane

API Plane

API Server

KV-aware Router

SLO Planner

KV Block Manager

Inference Engine TRT-LLM

Inference Engine vLLM

NIXL Comm Library

A high-level view of the NVIDIA Dynamo inference framework components.

View Source (NVIDIA Developer)

NVIDIA Dynamo KVBM Components

KV Block Manager Stack

Memory & Storage Tiers

GPU HBM - G1

CPU DRAM - G2

Local SSD - G3

Remote Storage - G4

Inference Runtimes Layer

vLLM / SGLang

TensorRT-LLM

Runtime Connectors

KVBM Adapter

Core Memory Manager Logic

Allocation & Eviction Policies

Detailed architecture of the NVIDIA Dynamo KV Block Manager (KVBM) showing its layered design and multi-tier memory management.