LLM Serving and Infrastructure
Key Questions
What AMD offering is Vultr making available?
Vultr is offering AMD Helios MI455X GPUs with pre-orders planned for Q4 2026 to support next-gen AI inference.
How does VAST Data help with KV cache scaling?
Combined with AMD Instinct GPUs, it enables KV cache offloading that yields 9X TTFT speedup by bypassing GPU memory limits.
What does the vLLM MoE plugin do?
It supports disaggregating attention and FFN layers in Mixture-of-Experts models for more flexible and efficient serving.
Why is goodput emphasized over throughput?
Goodput better reflects actual useful output and user experience in production LLM serving environments.
What new observability focus is emerging for GenAI?
Observability is becoming a distinct discipline with dedicated metrics for latency, scaling, quality, and cost in LLM infrastructure.
Major infrastructure moves: Netflix details in-house LLM serving platform using Triton and vLLM. GPU memory ballooning for multi-LLM serving targets TTFT reduction. New tool: llm-d cluster orchestration with prefix-aware routing, KV cache offloading, PD disaggregation (3x throughput, 2x TTFT). MOREH MoAI on AMD MI300X. Batching primer covers static/dynamic/continuous batching and chunked prefill. OpenRouter provides framework for evaluating providers across latency, throughput, uptime, quantization. Co-optimization of compute/storage/networking yields 3X faster TTFT. Also: vLLM day-0 support for Kimi K3; AI Router (MegaRouter) as routing layer; batching APIs guide for 50% cost reduction; Vultr AMD Helios MI455X; VAST Data+AMD KV cache offloading; Microsoft/AMD Helios on Azure; NVLink 6; goodput metric; Xiaomi MiMo; network overhead (100ms cold connection tax); connection reuse in serverless; policy-based traffic control; AWS disaggregated prefill/decode; DuetServe; DDN/Nebul/NVIDIA KV cache; SGLang vs vLLM; OpenAI/Broadcom Jalapeño chip; GORGO; NVIDIA Dynamo; Cerebras disaggregation; AMD ZenDNN 6.0; GPU buyer's guide; China inference chips; NVIDIA BlueField-4; HeteroMosaic; Netflix vLLM shift; vLLM p99 fix; fractional GPUs; Kimi K3 serving challenges; observability for Gen AI as discipline. Recent additions: Kimi K3 serving details (KVV verifier, concurrency sweet spot, hardware choices), GenRec prefill-only serving for cost efficiency, long-context tradeoffs (KV cache vs batching), autoscaling endpoints for LLM inference (practical metrics, asymmetric scale-up/down), Together AI cold start solution (native metrics reveal hidden queue bottlenecks), Triton Inference Server on Kubernetes for multi-model serving. Newest: Top 5 open-source LLM runtimes guide for self-hosted AI with Kubernetes; ISJL scheduling algorithm for batched serving (throughput/latency improvements); RAG latency is a prefill problem (reframes optimization priorities); topology-aware data movement for disaggregated GPU inference (60-85% latency hiding); compiler-generated kernels beating vLLM on Gemma4-12B (1.6x speedup); AgentCompile LLM-guided CUDA compiler; practical guide on measuring inference performance (TTFT, TPOT, tail latency); cost-efficient LLMOps on Kubernetes (GPU sharing, batching, storage governance); AI Gateway for enterprise (failover, cost attribution, token-aware rate limiting); Sakana AI Namazu on Modal for production agentic serving. FLARE length-aware routing improves heterogeneous serving. Test-time scaling regimes clarified for production decisions. LLM gateway tools essential for cost governance (only 21% mature).