NVIDIA's high-performance inference SDK. By 2026, TensorRT-LLM is a default optimization layer for serving large language models on NVIDIA GPUs, offering kernel
TensorRT and TensorRT-LLM are NVIDIA's inference maximizers: kernel-level optimization, quantization (FP8/INT4), and LLM-specific serving features (in-flight batching, paged KV cache) that extract peak tokens-per-second from GPUs, the performance ceiling for latency-critical and high-volume serving.
Pricing: Free SDKs with NVIDIA hardware; value realized through GPU efficiency.
vLLM for velocity and excellent default performance; TensorRT-LLM when the last 20–40% of throughput justifies optimization engineering: typically at serious scale or strict latency SLOs.
Commonly 2–5x over naive serving via quantization, fused kernels, and batching: directly reducing GPU count for a workload, which is the whole economic point.