GitHub - daneb/ai-system-design-primer: Learn how to design systems built on LLMs. Prep for the system design interview. Includes Anki flashcards.

GitHub

10 min read Original article ↗

CI Content: CC BY 4.0 Code: MIT

Learn how to design systems built on large language models: the trade-offs, the components, and the patterns that have converged across the industry, along with the ones that are still moving.

Animated: a request travels from the user through the gateway (3.1), input guardrails (3.2) and context engineering (3.3), then retrieval (3.4 to 3.6) and a workflow or agent (3.9 to 3.11), reaches the model layer (3.7, 3.8), and returns through output checks (3.12) to the user. Caching, asynchronism, evals, observability and security (3.13 to 3.17) support the whole path.

Start here

If you are… Read
New to building on models Orientation and the foundations 0.1–0.3, then the trade-offs in Part 2
Preparing for an interview or design review The method, two solved designs, then the practice questions; pace yourself with the study guide
Building a system now The component you need and its deep dive, the numbers and the cloud mapping
Reviewing risk or security 2.4 Capability vs security, 3.2, 3.12, 3.17 and the governance crosswalk

Motivation

Building on models is now a core engineering skill, and the field moves fast. Most of what's written about it is a product announcement, a framework tutorial or a single team's experience. This primer is an organised collection of what has converged across providers and researchers, with every trade-off named and every number dated and sourced.

It is modelled on Donne Martin's System Design Primer: the same arc of method, trade-offs, components, solved designs and reference numbers, rebuilt for systems whose core component is probabilistic, priced per token, and open to prompt injection.

Learn how to design AI systems. Each topic is a short page you can read in ten minutes, with a diagram, when to use it, when not to, and its disadvantages. A deep dive per topic holds the provider survey, numbers, risk controls and cloud mappings.

Prepare for the AI system design conversation, in an interview or a design review. Work through the method, the solved designs and the practice questions.

Anki flashcards

Every page ends in question-and-answer flashcards. Build the deck with either command (standard library only):

python3 scripts/primer.py anki         # writes build/ai-system-design-primer.apkg: double-click to import
python3 scripts/primer.py flashcards   # writes build/flashcards.tsv: plain text for Anki's File > Import

The package has one sub-deck per part. Re-importing a newer build updates answers in place and keeps your review history; a reworded question arrives as a new card. Cards are tagged by section and maturity, so you can study one part, or suspend maturity::frontier cards when they go stale.

Labs

Most topics come with a small, runnable lab: a simulation or calculation, numpy or the standard library only, with the prediction written down before the run and an honest note when it was wrong.

All 31 labs
Lab Question
0.1 What a token is, and what temperature does
0.2 Prefill, decode, the KV cache and reasoning tokens
0.3 The tool-calling loop, its token bill, and what checks catch
2.1 When does routing between models pay off?
2.2 What autonomy costs in reliability and predictability
2.3 When is the whole corpus in the prompt cheaper than retrieval?
2.4 Count to three, then size what a detector leaves behind
2.5 One agent or several?
2.7 Batching, per-user speed, and why you can't run at 100%
3.2 What a guardrail's error rates mean at volume
3.3 What each context strategy costs, and what it forgets
3.4 Keyword vs dense vs hybrid retrieval
3.5 Single-shot retrieval, query fan-out, or an agent loop?
3.6 Approximate search and filtering
3.7 Picking the cheapest model that "meets the bar", with a noisy eval
3.8 Cache arithmetic and when speculation pays
3.9 An evaluator loop is only as good as its evaluator
3.10 What does connecting more tools cost, and does tool search pay?
3.11 What should an agent remember, and how?
3.12 How far does a quote check get you?
3.13 When a prompt cache pays, and what a semantic cache gets wrong
3.14 Retries during an overload, and checkpoints for long runs
3.15 How many eval cases do you need?
3.16 Which agent traces should you keep?
3.17 Pin your tools, and see what pinning can't see
3.18 What LoRA trains, and when a fine-tune pays for itself
4.1 How many accelerators does a self-hosted inference API need?
4.2 Where the money goes in an enterprise RAG assistant
4.4 What 98% recall costs in lawyer time
4.5 Consistency with policy in the prompt or in a policy engine
4.6 Review capacity caps merged work

Contributing

Learn from the community.

Pull requests are welcome to fix errors, improve sections and add new ones. Read STYLE.md for how sections are researched and written (learner-first, several providers per topic, every claim sourced) and CONTRIBUTING.md for the mechanics. Decisions about the primer itself are recorded in adr/. Work still to do is in ROADMAP.md.

Index of AI system design topics

Summaries of AI system design topics, including pros and cons. Everything is a trade-off.

Each section links to its deep dive. Maturity is marked S (stable core), S/F (stable concept, volatile specifics) or F (frontier, expect rewrites).

Part 0: Orientation

Part 1: Method

Part 2: Trade-offs

Part 3: Components, in request order

Part 4: Solved designs

Part 5: Appendix

Study guide

Suggested topics to review based on your timeline.

Short (days) Medium (weeks) Long (months)
Read Orientation, the foundations (0.1–0.3) and the method ✓ ✓ ✓
Read the trade-offs (Part 2) Some All All
Read the components (Part 3) 3.1, 3.4, 3.9, 3.15, 3.17 Most All, plus deep dives
Read Numbers and the Glossary ✓ ✓ ✓
Work the solved designs (Part 4) Some Many All
Run the labs and their exercises Some Many All
Practise the additional questions Some Many Most
Read real-world architectures Some Many All
Map your platform with cloud and governance crosswalks — Some All

How to approach an AI system design question

How to tackle an AI system design question, in an interview or a design review.

The classic four steps, with two changes: the quality bar comes first, because a model-backed feature is right only some of the time; and hardening is its own step, because prompt injection, data leakage and runaway agents are design constraints, not add-ons.

  1. Use cases, constraints and the quality bar. Who, what, how many, how fast, what data, and how you'll know it works.
  2. Back-of-the-envelope. Tokens per request, requests per second, cost per day, latency.
  3. High-level design on the complexity ladder. Single call → workflow → single agent → multi-agent. Start at the lowest rung that meets the bar.
  4. Core components. Retrieval, context, tools, model choice, output checks.
  5. Harden. Which legs of the lethal trifecta are present, and which does the design break?
  6. Scale and optimise. Bottlenecks in order; caching, routing, batching.

Full walkthrough with a worked example: How to approach an AI system design question.

AI system design questions with solutions

Common AI system design questions with sample discussions, numbers and diagrams.

Question
Design an LLM inference API Solution
Design an enterprise RAG assistant Solution
Design an enterprise AI gateway Solution
Design a regulated document-review assistant Solution
Design a customer-support agent Solution
Design a coding agent platform Solution
Design an eval and release pipeline Solution
Add an AI system design question Contribute

More practice: additional design questions.

Appendix

How this primer differs from the original, and from a blog post

  • Trade-offs first. Every pattern says which trade-off it moves and in which direction.
  • Quality bar first. Evals are part of the method, not an afterthought.
  • Maturity is labelled and dated. Each page carries last_verified and review_by; the checker fails when a review is overdue. Frontier pages are reviewed every three months.
  • Several providers on every topic. Anthropic, OpenAI, Google, AWS and Microsoft, with the names each uses.
  • Vendor-neutral prose. Product names live in the deep dives and the cloud mapping.
  • Risk and controls on every topic, with the evidence an auditor would ask for.
  • Runnable labs with predictions recorded before the run.

Progress

Part Sections Published
Part 0: Orientation 4 4
Part 1: Method 1 1
Part 2: Trade-offs 7 7
Part 3: Components, in request order 18 18
Part 4: Solved designs 8 8
Part 5: Appendix 6 6

See ROADMAP.md for what's next and PLAN.md for the build plan and the verification debt (claims still waiting for a first-party source).

Credits

Contact info

Open an issue to discuss errors, missing topics or new solved designs. Report security problems, or guidance that would make a system less safe, privately as described in SECURITY.md. Participation is covered by the Code of Conduct.

License

  • Content (pages, diagrams, flashcards, lab READMEs) is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0), the same licence as the original System Design Primer.
  • All diagrams and charts are original to this repository and covered by the content licence.
  • Code (scripts/, the lab scripts and the CI workflow) is licensed under the MIT License.
  • Contributions are accepted under the same terms. The reasoning is in ADR 0008.

How this was written

The author researched and wrote this primer with an AI assistant, and reviewed every section. The method is in STYLE.md: primary sources from several providers for every claim, a dated last_verified on every page, runnable labs instead of asserted numbers, and a public list of what isn't yet verified. Mistakes are still possible; please report them with a source.