graph TD
subgraph API_Plane [API Plane]
APIServer[API Server]
end
subgraph Control_Plane [Control Plane]
Router[KV-aware Router]
Planner[SLO Planner]
end
subgraph Data_Plane [Data Plane]
KVBM[KV Block Manager]
Engine1[Inference Engine TRT-LLM]
Engine2[Inference Engine vLLM]
NIXL[NIXL Comm Library]
end
APIServer --> Router
Router --> Planner
Router --> KVBM
Router --> Engine1
Router --> Engine2
Engine1 <--> NIXL
Engine2 <--> NIXLView Source (NVIDIA Developer)
graph TD
subgraph Runtimes [Inference Runtimes Layer]
vLLM[vLLM / SGLang]
TRTLLM[TensorRT-LLM]
end
subgraph KVBM_Stack [KV Block Manager Stack]
Connectors[Runtime Connectors]
Adapter[KVBM Adapter]
Core[Core Memory Manager Logic]
Policy[Allocation & Eviction Policies]
end
subgraph Memory_Tiers [Memory & Storage Tiers]
GPU_HBM[GPU HBM - G1]
CPU_DRAM[CPU DRAM - G2]
Local_SSD[Local SSD - G3]
Remote_Storage[Remote Storage - G4]
end
Runtimes --> Connectors
Connectors --> Adapter
Adapter --> Core
Core <--> Policy
Core <--> Memory_Tiers
style KVBM_Stack fill:#f9f,stroke:#333,stroke-width:2px;