How LLM Inference Works

What a model call does on the GPU, token by token, and why that decides your bill and your latency: tokens, prefill and decode, the KV cache, batching, caching and the tricks providers use to go faster.

Watch lesson 01

Free preview01 What happens in one model call

Lessons
9
Modules
3
Running time
~3.75 hours
Before you start
Called an LLM API once
Free to watch
3 lessons
Last updated
Nov 4, 2026

Part of

[01] Access

[02] Who it's for

  • Engineers who build on LLM APIs and want to know why a call costs what it costs and takes as long as it does.
  • Developers who see words like KV cache, prefill or prompt caching in pricing pages and docs and want them to mean something.
  • No machine learning background is needed. You should be able to read a little Python and a JSON response.

[03] What you'll learn

  • 01Follow one model call from your prompt to the last streamed token, and say where the time goes
  • 02Count tokens the way the provider does and read a bill or a usage report line by line
  • 03Explain why the first token is slow and the rest are fast, using prefill and decode
  • 04Size the KV cache for a context window and see why long prompts cost memory, not just money
  • 05Know how batching, paged memory, prompt caching and speculative decoding let a provider serve many users at once
  • 06Design a feature around cost and latency: prompt layout, caching, model choice and streaming

[04] Syllabus

Module 01

One call

5 lessons100 min
  1. 01

    What happens in one model call

    Free
    21m
  2. 02

    Tokens: what you pay for

    Free
    22m
  3. 03

    Inside the forward pass

    Free
    29m
  4. 04

    Prefill and decode (locked)

    25m
  5. 05

    The KV cache (locked)

    23m

  6. Module quiz

    5 questions, about 5 minutes

Module 02

Many calls

3 lessons60 min
  1. 06

    Batching many users (locked)

    29m
  2. 07

    Paged memory and prompt caching (locked)

    27m
  3. 08

    Making decode faster (locked)

    23m

  4. Module quiz

    5 questions, about 5 minutes

Module 03

Your product

1 lessons20 min
  1. 09

    Designing for cost and latency (locked)

    25m

  2. Module quiz

    5 questions, about 5 minutes

[05] How it's made

The curriculum is curated by karnstack and reviewed by senior engineers in the industry before it ships. Narration is an AI voice (ElevenLabs) reading human-written, human-reviewed scripts. Read how courses are made.