karnstack
All courses

How LLM Inference Works

20 lessons · ~4 hours

0/20 done3h 53m left

  • What happens in one model callFree
  • The pause, the rush and the billFree
  • Tokens: what you pay forFree
  • Counting every tokenFree
  • Inside the forward passFree
  • Where the next token is chosenFree
  • Counting the work, forcing the shapeFree
  • Module quiz (Locked)
  • Prefill and decode (Locked)
  • Why output costs more (Locked)
  • The KV cache (Locked)
  • What the KV cache costs (Locked)
  • Module quiz (Locked)
  • Batching many users (Locked)
  • The limits of a batch (Locked)
  • Service tiers and temperature zero (Locked)
  • Paged memory (Locked)
  • Prompt caching (Locked)
  • Speculative decoding (Locked)
  • Fewer bytes, and fast tiers (Locked)
  • Module quiz (Locked)
  • Where the money goes (Locked)
  • Designing for cost and latency (Locked)
  • Module quiz (Locked)
Unlock every courseSee plans
  • Support
7d4f46a
  1. Courses
  3. How LLM Inference Works
  5. Batching many users
Sign in