Inference2 from

Inference Infrastructure

This is a discussion of the basics of inference and the serving ecosystem from a systems design perspective. It largely focuses on techniques to scale serving, reduce latency, or drive better efficiency and utilization in modern inference stacks.

  1. LLMs and Transformers Introduction, embeddings, transformers and attention mechanisms
  2. Inference and the KV Cache Inference execution and the KV cache
  3. Sharding a Model Pipeline, tensor, and expert parallelism
  4. Batching, Scheduling, and Paging Continuous batching, Orca, and PagedAttention
  5. I/O-Aware Kernels FlashAttention, FlashInfer, and Blockwise Parallel Transformers
  6. Speculative Decoding Sequential and parallel drafting, EAGLE, Medusa, MTP, DFlash, and DSpark
  7. Prefill-Decode Scheduling and Disaggregation Chunk prefill and prefill-decode disaggregation
  8. KV Cache Management and Offload Prefix caching and KV offload
  9. Sharding the Input Sequence and context parallelism, Ring Attention, and DeepSpeed-Ulysses
  10. Appendix: Overview of Training Fine-tuning, RLHF, RLAIF, quantization, and alignment techniques
  11. Appendix: GPU Hardware Architecture, CUDA and ROCm, kernels and Triton, memory hierarchy
  12. Appendix: Inference Runtimes LLM Serving Stacks, TensorRT, Triton, vLLM, and SGLang