Spotlight: The InfoQ Podcast
The episode this page is built from. Summaries follow the episode page and transcript.
Projects Discussed
3 entriesvLLM makes one GPU fast, and llm-d makes a fleet of GPUs efficient. Open source projects covered in the episode, with one link each.
01llm-dKubernetes-native orchestration layer that runs vLLM across a cluster so replicas work as a team: cache-aware routing, a prefill/decode split onto separate pools, KV cache offloading from VRAM to RAM to storage, batch inference and flow control. The podcast host notes it was donated to the CNCF, and the guest says it had just reached 1.0.
02vLLMOpen-source inference server that gets the most out of each GPU through PagedAttention memory management and continuous batching: one server, one model, very fast. The guest likens it to the JVM of language models.
03TornadoVMPlug-in for the JVM that compiles Java code at run time into GPU kernels such as PTX for NVIDIA and SPIR-V; the podcast cites a GPU Llama 3 implementation written entirely in Java.