Why do we need KV caching?
A visual explanation of why LLM inference caches key and value vectors, and why the trick saves compute during token-by-token generation.
9 min read
Nguyen Le
@yasuonguyen01
build · break · understand
about
- I'm interested in LLM inference. So basically I will try to learn everything around it, quantization, kernel engineering, etc.
- Beside LLM inference, I'm also interested in mathematics and some theoretical computer science stuffs such as complexity theory or programming language theory.
-
I also love to spend time with
my family 
my friends
work
VinMotion
Robotics Engineer December 2025 – June 2026
VinBigdata
AI Engineer Trainee July 2025 – December 2025
Made some good friends along the way.
education
VNUHCM - University of Science
Bachelor of Science in Artificial Intelligence Oct 2021 – Oct 2025
Best investment in myself I ever made