LLM inference engineering logopeterindia.net
Knowledge Hub

Learn LLM Inference Engineering for Free: A Step-by-Step Learning Path

Free courses, papers, documentation, code and benchmarks for serving large language models efficiently — from transformer basics to vLLM, SGLang, KV caching, quantization and GPU kernels.

98 topics•10 categories•Updated November 2026

1. Foundations

8 entries

Start here: how transformers generate text, why decoding is memory-bound and how to reason about latency and throughput.

01Hugging Face LLM CourseFree course on transformers, fine-tuning and using LLMs. 02Let's build GPT (Andrej Karpathy)Build a GPT-style transformer from scratch in code. 03nanoGPTMinimal GPT training and inference code to read and modify. 04Transformer Inference Arithmetic (kipply)Back-of-envelope maths for latency, memory and FLOPs. 05Making Deep Learning go Brrrr (Horace He)Compute, memory bandwidth and overhead from first principles. 06Large Transformer Model Inference Optimization (Lilian Weng)Survey of distillation, pruning, quantization and decoding speedups. 07Optimizing your LLM in production (Hugging Face)Lower precision, Flash Attention and architectural tricks explained. 08Efficiently Serving LLMs (DeepLearning.AI)Short course on KV caching, batching and LoRA serving.

2. Core Serving Papers

10 entries

The research behind modern inference engines. Read these alongside the code.

09PagedAttention / vLLMEfficient KV cache memory management for LLM serving. 10FlashAttentionIO-aware exact attention that avoids materialising the attention matrix. 11FlashAttention-2Better parallelism and work partitioning for faster attention. 12Fast Inference via Speculative DecodingDraft-and-verify decoding that preserves the output distribution. 13Speculative Sampling (DeepMind)Independent formulation of speculative decoding. 14MedusaMultiple decoding heads for faster generation without a draft model. 15SARATHIChunked prefills piggybacked on decodes. 16Sarathi-ServeBalancing the throughput-latency trade-off in LLM serving. 17DistServeDisaggregating prefill and decoding for goodput-optimized serving. 18SGLang paperEfficient execution of structured language model programs.

3. Quantization & Compression

11 entries

Fit larger models on less hardware and speed up decoding.

19GPTQAccurate post-training quantization for generative transformers. 20AWQActivation-aware weight quantization for LLM compression. 21llm-awqReference implementation of AWQ from MIT HAN Lab. 22Quantization Fundamentals with Hugging FaceShort course on quantizing open models. 23Quantization in DepthShort course building quantization methods in PyTorch. 24vLLM Quantization docsSupported quantization formats and how to run them. 25LLM CompressorCompression library for deployment on vLLM. 26NVIDIA Model OptimizerQuantization, pruning and distillation for NVIDIA deployment. 27bitsandbytes8-bit and 4-bit quantization for PyTorch. 28torchaoPyTorch-native quantization and sparsity. 29Hugging Face OptimumHardware-specific optimization toolkit for Transformers.

4. Hands-On with Inference Engines

14 entries

Run, read and benchmark real serving stacks.

30vLLM on GitHubHigh-throughput, memory-efficient LLM serving engine. 31SGLang on GitHubFast serving framework for LLMs and multimodal models. 32TensorRT-LLM documentationNVIDIA's optimized LLM inference library docs. 33gpt-fastSimple PyTorch-native text generation in under 1000 lines. 34MLC LLMCompiler-based universal LLM deployment engine. 35MLXArray framework for machine learning on Apple silicon. 36OllamaRun open models locally with a simple CLI and API. 37ggmlTensor library behind llama.cpp. 38ExecuTorchOn-device inference for mobile, embedded and edge. 39Triton Inference ServerNVIDIA's multi-framework model server. 40Ray Serve LLMServing LLMs on Ray with vLLM integration. 41BentoML documentationPackage and deploy model-serving APIs. 42FastChatOpen platform for training, serving and evaluating chat models. 43NVIDIA NIM for LLMsContainerised inference microservices for LLMs.

5. KV Cache & Distributed Serving

10 entries

Scale beyond one GPU: cache reuse, offloading, expert parallelism and prefill/decode split.

44NVIDIA: KV cache reuse optimizationsHow early reuse and flexible block sizes cut time to first token. 45Hugging Face KV cache strategiesCache implementations in Transformers. 46LMCacheKV cache layer for reuse across requests and engines. 47MooncakeKV-cache-centric serving platform behind Kimi. 48StreamingLLMEfficient streaming with attention sinks. 49FlexLLMGenHigh-throughput generation on a single GPU with offloading. 50DeepSpeedDistributed training and inference optimization library. 51DeepEPExpert-parallel communication library for MoE models. 52FlashMLAAttention kernels for multi-head latent attention. 53DeepSeek Open Infra IndexProduction-tested AI infrastructure components shared openly.

6. GPU Programming & Kernels

9 entries

Understand what the engines do underneath.

54LeetGPUPractice GPU programming challenges in the browser. 55GPU PuzzlesSolve puzzles to learn CUDA. 56Triton PuzzlesPuzzles for learning the Triton language. 57CUDA courseFree course material for CUDA programming. 58Triton languagePython-like language and compiler for GPU kernels. 59CUTLASSCUDA templates and DSLs for high-performance linear algebra. 60TileLangDomain-specific language for writing GPU kernels. 61FlashAttention repositoryReference kernels for fast, memory-efficient attention. 62FlashInferKernel library for LLM serving.

7. Benchmarking & Evaluation

5 entries

Measure latency, throughput, cost and quality before and after each optimization.

63LLMPerfLibrary for validating and benchmarking LLM APIs. 64GuideLLMBenchmark LLM deployments under realistic load. 65LM Evaluation HarnessStandard framework for evaluating language models. 66Artificial AnalysisIndependent speed, price and quality comparisons of models and providers. 67Modal LLM Engineer's AlmanacBenchmarks of engines and models on common GPUs.

8. University Courses & Lectures

6 entries

Free course sites with slides, readings and assignments.

68CMU 15-442/15-642: Machine Learning SystemsML systems course covering serving and compilers. 69CMU 11-868: Large Language Model SystemsLLM systems course with inference and serving topics. 70CMU 15-779: Advanced ML SystemsAdvanced course on systems for ML. 71MIT Efficient ML (efficientml.ai)Course site for efficient deep learning computing. 72Hugging Face LearnFree courses on LLMs, agents and more. 73DeepLearning.AI: Efficient Inference with SGLangShort course on SGLang for text and image generation.

9. Engineering Blogs & Curated Lists

13 entries

Ongoing reading from teams that run inference at scale.

74vLLM BlogRelease notes and deep dives from the vLLM project. 75LMSYS BlogPosts from the SGLang and Chatbot Arena team. 76NVIDIA Technical Blog: Inference PerformanceNVIDIA posts on inference optimization. 77PyTorch BlogPosts on compilation, quantization and serving. 78Anyscale: Continuous BatchingWhy continuous batching raises throughput. 79Databricks: LLM Inference Performance EngineeringBest practices for measuring and tuning inference. 80Together AI BlogResearch and engineering posts on inference. 81Fireworks AI BlogPosts on serving and optimizing models. 82Baseten BlogInference engineering write-ups. 83Tri Dao's BlogPosts from the FlashAttention author. 84Modal ExamplesRunnable examples for serving models. 85SGLang Learning MaterialsSlides, talks and tutorials for SGLang. 86Awesome LLM InferenceCurated paper and code list for LLM inference.

Related Pages

12 entries

More on LLM inference on this site.

87Free Resources for Learning LLM Inference EngineeringCompanion list of free inference engineering resources. 88How vLLM WorksInside the vLLM engine. 89Inference EnginesInference engines compared. 90LLM Inference PlatformsHosted and self-managed inference platforms. 91Production AI Inference StackBuilding a production inference stack. 92AI Model Quantization TechniquesQuantization methods. 93LLM Training and Inferencing Optimization TechniquesOptimization techniques. 94ML Model Serving ToolsModel serving tools. 95Edge-Native Inference EnginesInference at the edge. 96Platforms for Local LLMsRun LLMs locally. 97LLMOpsLLM operations. 98LLM HubHub of LLM resources.

Frequently Asked Questions

What is LLM inference engineering?

It is the practice of serving large language models efficiently: reducing latency and cost per token, increasing throughput and fitting models onto available hardware using techniques such as batching, KV caching, quantization and optimized kernels.

Where should a beginner start?

Start with the Foundations section: the Hugging Face LLM Course, Andrej Karpathy's GPT build, and the inference arithmetic and Brrrr articles. Then read the PagedAttention and FlashAttention papers and run vLLM or SGLang.

Which inference engine should I learn first?

vLLM and SGLang are open-source, well documented and widely used, so they are good starting points. TensorRT-LLM suits NVIDIA-specific optimization, and llama.cpp or MLX suit local and Apple silicon use.

What is the difference between prefill and decode?

Prefill processes the whole prompt in parallel and is compute-bound. Decode generates one token at a time and is usually memory-bandwidth-bound, which is why KV cache management and batching matter.

Do I need to know GPU programming?

Not to start. Serving engines hide most of it. For deeper optimization, learning CUDA or Triton helps, and the GPU Programming section lists free practice resources.

Are all these resources free?

Links go to free-to-read papers, documentation, code and course material. Some platforms may ask you to sign in, and certificates or graded versions of courses may cost money. Links were checked in November 2026.