Podcast: Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM

In this episode, Edoardo Vacchi discusses the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and optimizing distributed AI inference using llm-d and vLLM on Kubernetes. Vacchi explains how llm-d manages cache locality (KV and prefix caches), request routing, and batch inference across Kubernetes clusters to maximize expensive GPU utilization. By Edoardo Vacchi

Podcast: Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM

In this episode, Edoardo Vacchi discusses the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and optimizing distributed AI inference using llm-d and vLLM on Kubernetes. Vacchi explains how llm-d manages cache locality (KV and prefix caches), request routing, and batch inference across Kubernetes clusters to maximize expensive GPU utilization.

By Edoardo Vacchi

What's Your Reaction?

like

dislike

love

funny

angry

sad

wow