10-411: Compilers for Machine Learning

Carnegie Mellon University · Spring 2027

Draft syllabus. Items marked TBD must be finalized before the course begins.

Course

• Instructor: George Hotz — geohot@gmail.com; office and office hours TBD
• TAs: TBD — names, emails, office hours and locations
• Lectures: Monday, Wednesday, Friday, 10–10:50 AM · TEP 1403
• Materials: GitHub (course repository link TBD)
• Discussion: EdStem (course link TBD)
• Submissions: Private GitHub repositories, collected by a custom autograder (instructions TBD)

A hands-on, one-semester course where you build your own compiler from scratch, starting with elementwise programs and ending with training modern LLMs on GPUs. Each week builds aggressively on the previous week. Keep your code simple: mistakes and unnecessary complexity compound.

This course covers the design and implementation of a modern machine learning compiler, and the interaction between IR design, hardware capabilities, and the structure of machine learning programs. Topics include term rewriting, code generation, movement operators, kernel fusion, memory hierarchies, GPU architecture, automatic differentiation, and flash attention.

Format: In-person lectures and quizzes, a semester-long compiler project, and a final project. Work individually or in teams of two. Choose your own implementation language. There is no starter code; the specification uses a simple interchange format for the compiler IR.

Preparation: No formal prerequisites. You should be deeply familiar with 15-213 or equivalent material, highly skilled in your chosen implementation language, comfortable with C and some GPU programming, and able to manage a programming project that lasts a full semester.

Learning objectives

By the end of the course, you should be able to:

• Design an intermediate representation and implement graph rewrites and symbolic simplification.
• Generate and execute correct C and GPU code from tensor programs.
• Implement movement operations, reductions, kernel fusion, and automatic differentiation.
• Optimize kernels for memory hierarchies, parallelism, and accelerator hardware.
• Test and maintain a growing compiler codebase, and use it to train real models.

Grading

All percentages below are percentages of the final course grade.

• 55% — Compiler (weeks 1–11)
  • 5% — Ten tests due each Monday in weeks 1–10, judged on correctness.
  • Up to +3% bonus credit — if other students’ compilers (or the reference compiler!) fail your correct tests.
  • 25% — Correctness: each weekly submission is graded by the percentage of student and instructor tests it passes.
  • 15% — Style: is your code readable, and can it be simpler?
  • 10% — Performance in weeks 8–11, graded on a linear curve.
• 25% — In-person quizzes at the start of class; at least 20 quizzes, lowest two scores dropped.
• 20% — Final project (weeks 12–14)

Weekly deadlines: Tests are due Monday in weeks 1–10; compilers are due Wednesday following weeks 1–11. All are due at 10 AM, on the dates in the schedule. No submissions are due during spring break. All course times and deadlines are Pittsburgh local time (America/New_York).

No midterm or final exam. Letter grades: A ≥ 90%; B ≥ 80% and < 90%; C ≥ 70% and < 80%; F < 70%.

Grading environment: Weeks 1–7 focus on correctness; GPU work begins in week 8. Performance grading in weeks 8–11 will use a fixed benchmark suite on identical GPUs, tentatively NVIDIA GeForce RTX 3080s. Final hardware and access details are TBD. Performance scores are 0 for the lowest-performing submission, 100 for the highest, with scores interpolated linearly between them.

Autograder: A custom autograder will pull code from each student or team’s private GitHub repository at a designated branch/tag and run the tests. Repository setup, branch/tag naming, submission mechanics, test visibility, and resubmission rules are TBD.

Final project: The planned deadlines are Monday, April 12 at 10 AM for a short proposal, and Friday, April 30 at 10 AM for code, reproducible results, and a short report. Projects start on April 14; the detailed rubric will be published before proposals are due.

TBD: Detailed style rubric, reference hardware, performance benchmark aggregation and tie handling, and rounding policy.

Schedule

Tentative course plan, based on CMU’s official Spring 2027 calendar. Course weeks run Wednesday–Friday–Monday, with the compiler due the following Wednesday. All dates below are in 2027; all deadlines are 10 AM. See the technical syllabus for operation definitions.

Short UIR examples illustrate weeks 1–5. Each assert pairs a result with its expected value.

First class: January 20. Last class: April 30. No class January 18 (MLK Day, before the semester starts), March 8–12 (spring break), or April 16 (Carnival). The course has 40 class meetings and no final exam.

%sum = add 2 3
%test = assert %sum 5

%buf = buffer [] slot=0 dtype=i32
%write = store %buf 7
%ready = after %buf %write
%value = load %ready
%test = assert %value 7

%buf = buffer 3 slot=0 dtype=i32
%i = range 3 slot=0
%write = store (index %buf %i) %i
%done = end %write %i
%test = assert (after %buf %done) [0 1 2]

%values = add [1 2 3] (expand 1 3)
%sum = reduce %values op=add pop=1
%test = assert %sum 9

%matrix = reshape [1 2 3 4] [2 2]
%transposed = permute %matrix [1 0]
%test = assert (index %transposed 1 0) 2

Religious observance conflicts, including the April 30 project deadline on Orthodox Holy Friday, will be accommodated. Contact the instructor in advance; see CMU’s religious observances guide.

Policies

Attendance. Attend class; lectures are not posted online. Quizzes occur at the start of class. No separate attendance or participation grade.

Quizzes. Quizzes are individual and closed-book, with no devices except approved accommodations. The lowest two quiz scores are dropped, including zeros for missed quizzes. Contact the instructor about additional excused absences so that make-up or alternative arrangements can be made.

Partners. Work alone or with the same partner throughout the semester; partner changes require instructor approval. Both partners must understand all submitted code. Project grades are normally shared, but may be adjusted individually for substantially unequal contributions. Raise partner problems early.

Academic integrity. CMU’s academic integrity policy applies. Discuss concepts outside your team, not implementation code. Keep repositories private; share tests only through course submissions. Study existing compilers, but do not copy their implementations. Cite sources and ask before using other assistance.

Libraries and tools. General-purpose libraries, including graph and DAG utilities, are allowed, as are Clang, LLVM, and CUDA APIs or thin wrappers such as cuda-python. Libraries providing prewritten numerical kernels, such as BLAS or cuBLAS, are not allowed. ML frameworks and tensor/kernel compiler frameworks, such as MLIR, PyTorch, tinygrad, Triton, TileLang, cuTile, etc., are also not allowed.

AI tools. AI may help explain concepts, debug code you wrote, and write tests or tooling. It may not write or rewrite your compiler implementation. Disclose its use and understand everything you submit. Academic integrity rules still apply. No AI during quizzes.

This course aggressively builds on the previous week, and is an exercise in slop management. If you let any slop in early, it will compound and you will not finish the class.

Late work. Each student has three 24-hour late days, for weekly compiler submissions only, at most one per submission. Both partners must have and spend a late day. No late tests. Late work without an available late day or approved extension receives no credit. Contact the instructor promptly about illness or other extension requests; approved accommodations are handled separately.

Regrading. Submit a written request to the course staff explaining the suspected grading error within seven business days of the grade being released.

Devices and recording. Devices are allowed unless disruptive, except during quizzes. Recording classroom activities requires instructor permission or an approved accommodation.

Accessibility and support

Accommodations. Discuss approved disability accommodations with the instructor early so we can arrange appropriate support. To request accommodations, contact Disability Resources. Let us know in advance about conflicts with religious observances so we can arrange appropriate accommodations.

Respect. Treat everyone with respect. This course holds all students to high standards and evaluates the quality of their work, not their background. We aim to give students of all backgrounds the opportunity to meet these standards.

Well-being. Take care of yourself and ask for help early. Contact the instructor or your advisor with concerns. For confidential mental health support, CaPS is available 24/7 at 412-268-2922.

Mandatory reporting. The instructor, and TAs acting in their TA roles, must report information about discrimination or sexual misconduct to CMU’s Office for Institutional Equity and Title IX. The office offers support; responding is your choice. Confidential resources are available from people who are not required to report.

Resources

No required textbook. The technical syllabus and assignment specifications are required course materials. Materials on GitHub; discussion on EdStem (links TBD). Additional readings will be marked required or supplementary when assigned.

Required tools: Git, a toolchain for your implementation language, a C compiler, and access to CUDA-capable GPUs for the GPU assignments. Supported environments, installation instructions, and GPU access details are TBD.

• CMU academic calendar
• University policies
• Examination policies