Writing high-performance GEMM kernels for new GPU generations requires tensor cores.
There are many blog posts that describe the corresponding programming model. However, few resources describe the arithmetic inside tensor cores in detail.
In this blog post, we aim to give a short introduction to the topic and lay the groundwork for further analysis.
The model we describe here is taken from the excellent MMA-Sim paper, which we recommend reading. Future blog posts will describe how to verify it and draw further conclusions.
FDA is the algorithm used by many tensor core instructions, for example UTCHMMA on Blackwell. It computes each output as d = c + Σ a_i b_i, where c is the accumulator.
You can find a precise description in the paper. Here we aim to understand it better by following a simple example.
Each output takes a value c and L pairs (a_i, b_i), i = 0, ..., L-1, as input. L depends on the hardware and data type. For example, on Blackwell we have L ∈ {8, 16, 32} for the data types TF32, FP16/BF16 and FP8, respectively.
The first step handles special values (i.e., NaNs and infinities). We omit it here for simplicity.
Our example uses c = 2^23 and the four pairs a = (-2^13, 48, 0.375, 0.125), b = (2^10, 48, 1, 1), with F = 25, as for FP16/BF16 on Hopper and Blackwell. The exact result is 2304.5. We use four pairs to keep it simple and assume the other pairs are (0, 0).
The second step forms the exact floating-point products. To do this, we split each input into its significand and exponent and compute the exact product by multiplying the significands and adding the exponents.
Note that the product is not renormalized, i.e., the extra factor of 2 is not moved into the exponent.
The next step is to compute e_max as the maximum of all product exponents, including the exponent e_c of the accumulator c.
After that, we align the significands by shifting each one e_max - e_i bits to the right and keep only F fraction bits, where F depends on the hardware and instruction.
In the example, e_max is the exponent of both c and the first product x_0. Thus all bits from 2^-3 downward are cut off.
This truncation can introduce error. This is why FP8 MMAs on Hopper need promotion, as noted in the DeepSeek-V3 report. For FP8 MMAs on Hopper, F = 13, and the output is also cut to 13 fraction bits. On Blackwell, this is fixed, with FP8 MMAs using F = 25, the same as FP16/BF16, and the FP32 output also keeps the full 23 fraction bits.
Next, we compute the exact sum of the aligned and truncated terms from the previous step. All of them have F fraction bits, so the sum is exact and does not depend on the order of the terms within one instruction.
A dot product of length K needs a chain of K/L instructions, where each rounded result is fed as c into the next instruction.
Finally, we normalize the sum and round it to the floating-point format of the output. The rounding mode depends on the output data type. FP32 output rounds toward zero, and FP16 output rounds to nearest, ties to even. We call this final rounding ρ.
An FMA computes a·b + c exactly and rounds the result once. One may naively think that the tensor core computes d = c + Σ a_i b_i as a chain of FMAs, as shown below.
The first difference is that the FMA version is a chain of L FMAs, and each step rounds. That means the order of the products matters, which is not the case for FDA.
The second difference is that FMA cuts off no bits before the addition. Each step adds the exact product and rounds only the result.
Overall, FDA is one fused step. All L products and c are added at once, so their order does not matter, while the FMA chain is sequential.
In our example, each FMA step in the chain is exact, so the FMA and FDA results differ by precisely the part FDA cuts off. This does not mean FMA is always more precise, because the order of the products matters. For instance, reversing the chain gives 2304 because of rounding.
We hope this blog post serves as a good introduction to the FDA algorithm described in the MMA-Sim paper. Thanks to Peichen Xie for helping answer our questions.
Please let us know if you have specific questions in mind that we can try to answer in the next part of this series on tensor core arithmetic.