---
title: "Inference Infrastructure"
description: "A systems view of inference and serving runtimes"
url: "https://iop.systems/blog/llm-inference/"
---

# Inference Infrastructure

This is a discussion of the basics of inference and the serving ecosystem from a systems design perspective. It largely focuses on techniques to scale serving, reduce latency, or drive better efficiency and utilization in modern inference stacks.

- [LLMs and Transformers](https://iop.systems/blog/llm-inference/transformers/) — Introduction, embeddings, transformers and attention mechanisms
- [Inference and the KV Cache](https://iop.systems/blog/llm-inference/kvcache/) — Inference execution and the KV cache
- [Sharding a Model](https://iop.systems/blog/llm-inference/modelsharding/) — Pipeline, tensor, and expert parallelism
- [Batching, Scheduling, and Paging](https://iop.systems/blog/llm-inference/batching/) — Continuous batching, Orca, and PagedAttention
- [I/O-Aware Kernels](https://iop.systems/blog/llm-inference/flashattn/) — FlashAttention, FlashInfer, and Blockwise Parallel Transformers
- [Speculative Decoding](https://iop.systems/blog/llm-inference/speculative/) — Sequential and parallel drafting, EAGLE, Medusa, MTP, DFlash, and DSpark
- [Prefill-Decode Scheduling and Disaggregation](https://iop.systems/blog/llm-inference/disaggregation/) — Chunk prefill and prefill-decode disaggregation
- [KV Cache Management and Offload](https://iop.systems/blog/llm-inference/kvoffload/) — Prefix caching and KV offload
- [Sharding the Input](https://iop.systems/blog/llm-inference/shardinput/) — Sequence and context parallelism, Ring Attention, and DeepSpeed-Ulysses
- [Appendix: Overview of Training](https://iop.systems/blog/llm-inference/training/) — Fine-tuning, RLHF, RLAIF, quantization, and alignment techniques
- [Appendix: GPU Hardware](https://iop.systems/blog/llm-inference/hardware/) — Architecture, CUDA and ROCm, kernels and Triton, memory hierarchy
- [Appendix: Inference Runtimes](https://iop.systems/blog/llm-inference/software/) — LLM Serving Stacks, TensorRT, Triton, vLLM, and SGLang