Why diffusion models do not need noise conditioning

Diffusion · Theory

The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning

A diffusion model doesn't need to be told how noisy its input is.

Standard diffusion models are handed the noise level at every step. Drop that input and the best ones keep working, because the geometry of high dimensions already encodes it. Whether generation survives depends on what the model is trained to predict.

Every diffusion model is built around a noise-level input. For the right kind of model, it is redundant.

Every diffusion or flow model is given the noise level as an input. You show the model a noisy version of the data, and you also tell it how much noise is on that input: a single number, usually written $t$ or $\sigma$ (you can read $t$ as how far the noising process has run). The model uses that number to decide how hard to clean. The architecture injects the noise level into every layer, on the assumption that the model needs it. What the model should do at high noise (guess the rough shape of an image) is nothing like what it should do at low noise (sharpen the last few details). How could one network do both without being told how noisy its input is?

Recent work showed you can remove that input entirely. Sun and collaborators (Is noise conditioning necessary?) trained a single network that sees only the noisy image, never the noise level, and it still generates clean samples. Equilibrium Matching trains the field to keep the data as its fixed points, with no time index at all, so it is noise-blind by design. The literature calls this the autonomous (or noise-blind) setting: one static vector field $f(\mathbf{u})$, the same function of the input no matter what noise produced it. A single static field somehow has to serve both pure noise and nearly-clean data, and it has to stay stable right at the data, where the gradient diverges.

This paper, from Sahraee-Ardakan, Delbracio, and Milanfar at Google, is the theory of why blind models work. It answers two questions. How does a model know the noise level it was never told? And why do some blind models generate beautifully while others collapse into static? Geometry resolves the first; a stability race resolves the second. The argument runs in a few steps: what the conditioned model is, what happens when you remove the noise input, how high-dimensional geometry supplies the level, what landscape the blind model descends, and why one parameterization survives it while another shatters.

Conditioning on the noise level

The forward process is the easy half of any diffusion model: it takes a clean datapoint $\mathbf{x}$ and a noise level indexed by $t$, and mixes them into a noisy observation.

The two schedule functions do the bookkeeping: $a(t)$ scales the signal and $b(t)$ scales the noise. Near $t \to 0$ you sit on clean data; crank $t$ up and the noise term swamps everything until the observation is featureless. Different model families pick different $a,b$. DDPM (the original denoising-diffusion model) shrinks the signal as it adds noise so the total variance stays pinned ($a^2+b^2=1$); EDM (the design from Karras et al.) keeps the signal at full scale while the noise term grows; flow matching (a straight line from data to noise) slides linearly with $a=1-t$, $b=t$. The ratio that summarizes how much signal survives is the signal-to-noise ratio (SNR):

Training is a plain regression. You show the model a noisy $\mathbf{u}_t$ and ask it to predict a linear target $r = c(t)\mathbf{x} + d(t)\boldsymbol{\epsilon}$, scored by squared error:

Squared error has one minimizer: the average of the thing you are predicting, conditioned on everything you can see. Here that is the conditional mean of the target, given the observation and the noise level.

This is a time-dependent vector field: a different arrow at every (position, noise level) pair. The four coefficients $(a,b,c,d)$ are the only difference between the famous models. DDPM predicts the noise $\boldsymbol{\epsilon}$ ($c{=}0, d{=}1$), EDM predicts the clean signal $\mathbf{x}$ ($c{=}1, d{=}0$), flow matching predicts a velocity, the straight-line direction from data toward noise ($c{=}{-}1, d{=}1$). The $t$ input is the switch that lets one network be a different function at each noise level.

A switch seems necessary. At high noise the field should pull any point toward the blurry average of all the data, since that is the best guess when you can barely see. At low noise it should pin a point to the nearest sharp datapoint. Those are opposite fields, and $t$ is what tells the network which one to be. Drop $t$ and you seem to lose the switch. You do not.

Removing the noise input

With the noise level removed, the network sees only $\mathbf{u}$ and must output a single vector $f(\mathbf{u})$, the same function no matter what noise produced the input. The optimal such field is not mysterious. It is still the least-squares answer, but you can no longer condition on $t$ because you do not have it, so the best you can do is average over your uncertainty about it:

The weight in that average is the posterior over the noise level, $p(t\mid\mathbf{u})$: your belief about which $t$ produced this observation, by Bayes' rule from the likelihood of seeing $\mathbf{u}$ at each level. So the optimal blind field is a posterior-weighted blend of all the conditioned fields. This is the law of iterated expectations: your best blind guess is the average of your informed guesses, each weighted by how likely its scenario is.

That blend is enough to generate. Below is the optimal blind field for a small five-point dataset, computed in closed form. The field never changes, and nothing tells it the noise level. Press play and a ring of pure-noise particles rides this one frozen field inward and settles exactly onto the data, where the field vanishes, then stops.

How can an average over noise levels be the right field anywhere? At a given $\mathbf{u}$, if the posterior $p(t\mid\mathbf{u})$ is spread across many levels, then (6) blends a high-noise field (points to the center) with a low-noise field (points to a specific mode), and the average of two contradictory arrows should be meaningless. But the posterior is usually not spread at all.

Drag the probe below and watch $p(t\mid\mathbf{u})$ directly. Out in the void the curve is broad, placing most of its weight on large noise levels. Slide the probe onto a datapoint and the curve collapses to a spike at $t\to 0$, because an observation sitting on the data is indistinguishable from clean data.

When the posterior is a spike at some $\hat{t}$, the average in (6) reduces to the single conditioned field at $\hat{t}$, so the blind field equals the model that knew the noise level all along. To make that concrete it helps to rewrite (6) in terms of the denoiser $D^*_t(\mathbf{u}) = \mathbb{E}[\mathbf{x}\mid\mathbf{u},t]$, the best guess of the clean data at level $t$:

Every parameterization is some affine mix of "where you are" ($\mathbf{u}$) and "where the clean data probably is" ($D^*_t$). So the blind model is Bayes-optimal averaging, and it matches the informed model exactly when the posterior over the noise level is sharp. The real question is no longer whether the average is reasonable; it is when the average is sharp. That is a question about geometry.

Reading the noise level off the geometry

In high dimensions, noise has a very predictable size, and a blind model relies on exactly that. Add Gaussian noise of level $\sigma$ to a point in $D$ dimensions and the noise vector's length is almost exactly $\sigma\sqrt{D}$, with vanishing relative wiggle. (This is concentration of measure: a high-dimensional Gaussian puts essentially all its mass on a thin spherical shell, not near its center.)

So suppose the data does not fill the space but sits on a thin $d$-dimensional manifold (a curved lower-dimensional sheet) inside a large $D$-dimensional ambient space. The part of a noisy observation that sticks out off the manifold is pure noise in the remaining $D-d$ directions, so its length is about $\sigma\sqrt{D-d}$. That length is essentially the noise level. Measuring the distance $r$ from the observation to the manifold gives an estimate

The bigger the gap between the ambient dimension and the manifold's intrinsic dimension, the sharper the estimate. Two different noise levels live on two shells of different radius; in low dimensions the shells are fat and overlap, so the level is ambiguous, and in high dimensions they are razor-thin and disjoint, so the level is read straight off the radius. Drag the dimension below and watch the two shells separate.

The 1D plot above superimposes the two bells so you can watch their overlap shrink as $D$ grows. The shells themselves are a 3D-shaped picture: two concentric spheres of points around the single data point, fatter for high noise, thinner for low. Figure 4 draws the drawable case, $D=3$ with a single data point at the origin ($d=0$), and lets you rotate in any direction to see both spheres at once. The real model lives in hundreds of dimensions where the same spheres are paper-thin and far apart, exactly the regime the 1D bells reach as you crank $D$ up.

Two caveats. The geometry only makes the noise level recoverable from the observation; whether a trained network actually exploits the distance $r$ to recover it is an empirical fact, and the answer (from Sun et al.) is that it does. And there is a second, stronger reason the posterior is sharp, one that needs no high dimension at all. As an observation approaches the data manifold it becomes indistinguishable from clean data, so the smallest noise levels dominate and $p(t\mid\mathbf{u})$ concentrates at $t\to 0$ by sheer proximity. That near-manifold collapse keeps generation stable at the end, regardless of dimension.

So the noise level is not lost when you stop feeding it in. It is encoded in the observation's geometry, and the model recovers it from there. Wherever the posterior is sharp (which is almost everywhere) the blind field equals the informed field. But "equals a good field" is a statement about the target. Whether you can safely follow that field all the way down to the data is a separate question, and it has a sharp geometric obstruction.

An infinitely deep well

The paper's reframing is that a blind model is not chasing a moving target at all. It is descending a single fixed landscape: the marginal energy. Define it as the negative log-likelihood of a noisy datapoint at some unknown noise level,

where the integral averages the noisy-data density over a prior on the noise level. Generation is then rolling downhill on this one static potential toward where noisy data is most plausible, which is the clean data itself. A static field can generate because it is the gradient field of a static potential, so it needs no noise-level input, only a slope to follow downhill.

The slope is the posterior-averaged score. To write it in terms of the denoiser we use Tweedie's formula, the empirical-Bayes identity (Robbins 1956, Efron 2011) that the conditional score points from your noisy point toward your best guess of the clean one:

Averaging that over the posterior on $t$ gives the gradient of the marginal energy:

That gradient is where the problem sits. It carries a $1/b(t)^2$, and $b(t)\to 0$ as you near the data, so the marginal energy has an infinitely deep, infinitely steep well at every datapoint:

A neural network outputs finite vectors. A finite field cannot equal an infinite gradient. So either a blind model cannot represent the landscape near the data, exactly where generation finishes and matters most, or something is rescuing it. The 3D view makes this concrete: lift the data plane and the marginal energy becomes a surface with pits at every data point. Toggle between the energy itself, the gradient magnitude, and the preconditioned field, and rotate to see how each surface treats the data.

The same math in one dimension makes the cancellation exact. Three data points sit on a line, and the three quantities involved (energy, raw gradient, and the field the model actually follows) are plotted as three curves. The raw gradient runs off the top of the chart while the bold teal product, the field f*, stays finite and crosses zero at every datum.

The singularity cancels. The blind field decomposes into three pieces: a scaled copy of the energy gradient (the natural-gradient term, a gradient rescaled by a local metric), a transport correction, and a linear drift.

Take the three terms in order. The first is the energy gradient scaled by an effective gain $\overline{\lambda}(\mathbf{u}) = \mathbb{E}_{t\mid\mathbf{u}}[\lambda(t)]$, where $\lambda(t) = \tfrac{b}{a}(da - cb)$. The second, the transport correction, measures how far the per-noise-level gradients sit from their own average; it is nonzero only when the posterior is spread across several levels, and it dies once the posterior concentrates on one. The third is a linear drift the noise schedule contributes. The gain cancels the singularity. Near the data $b\to 0$, so the $1/b^2$ in the gradient sends it to infinity, while $\lambda$ carries factors of $b$ that vanish at the matching rate, leaving the product $\overline{\lambda}\,\nabla E_{\text{marg}}$ finite. It is like descending an infinitely steep slope while shortening your stride toward zero at the same rate, so every step stays finite. The paper calls this a Riemannian gradient flow: the gain acts as a local metric, a position-dependent rescaling of distance, that turns the singular landscape into a smooth, finite-speed descent with a stable resting point at the data.¹ Because the transport correction dies wherever the posterior concentrates (high dimension, or near the manifold), what is left is a clean preconditioned gradient flow.

The same decomposition holds at every point on the plane. The figure samples a grid of locations and, at each, draws the three terms of equation (14) head to tail: a teal natural-gradient arrow, then an amber transport-correction arrow, then a small drift arrow, with a dashed line showing the total. The data points sit inside amber rings; the chains shrink to zero as you approach them. Slide the posterior sharpness and watch the amber transport correction die away as the posterior concentrates; what is left is the teal natural-gradient flow plus the drift.

So the infinite well never appears in the field the model learns. That field is bounded, with the data as a stable attractor, which is what you saw the particles settle into in Figure 1. Whether a real numerical sampler can follow that bounded field down to the data is a separate question.

Which parameterization survives

Having a bounded field is not the same as being able to follow it; that is the question of the samplingdynamics. You integrate an ODE driven by the field, and a bounded field divided by a vanishing noise scale can still make that ODE stiff (one that forces a numerical solver into vanishingly small steps) and explosive. The sampler integrates

where $\mu(t)$ is the schedule's drift and $\nu(t)$ is the effective gain of the parameterization. One subtlety the paper is careful about: even though the network $f^*$ is never told the noise level, the sampler still uses $t$, because $\mu$ and $\nu$ depend on it. The architecture is autonomous; the integration schedule around it is not.

To judge stability, compare the blind sampler against an oracle that knows the true $t$ at every step. Subtracting the two cancels the shared drift term and leaves the error introduced purely by being blind:

Stability is a race as $t\to 0$. The estimation error falls (the posterior over the noise level concentrates), but the gain $\nu(t)$ may diverge. Which of the two wins decides the outcome, and the three standard targets land in three different places.

Noise prediction (DDPM, and DDIM, its deterministic-sampler variant) has gain $\nu\sim 1/b$, which diverges. Worse, its estimation error does not fall to zero. It floors at a positive "Jensen gap": because $1/b$ is convex, the posterior-average of $1/b$ exceeds $1/(\text{average } b)$, and that gap stays nonzero unless the posterior is a perfect spike. A diverging gain times a floored error goes to infinity. Blind noise prediction blows up.

Signal prediction (EDM) has an even worse gain, $\nu\sim 1/b^2$. But near a discrete data manifold the denoising error decays exponentially, like $e^{-C/b^2}$. Near a data point the denoiser is a softmax over the data points weighted by a Gaussian in distance, so once you sit close to one point every other point is suppressed by a Gaussian factor in its squared distance, and the error to the nearest point dies that fast. Exponential decay beats any polynomial blow-up, so the product goes to zero: a stronger gain singularity than noise prediction, yet still stable, because the error crashes faster than the gain climbs.

Velocity prediction (flow matching) has gain $\nu = 1$, flat. There is nothing to amplify, so the error stays bounded and the dynamics are inherently stable. Toggle the target below and watch the drift error $\Delta\mathbf{v}$ as the noise vanishes: only noise prediction runs to infinity.

This makes the title's claim precise. You can drop the noise-level input, but only if your parameterization has a bounded gain (velocity) or a self-correcting one (signal). Velocity-based models are inherently safe; noise-prediction models are structurally broken when blind. The difference is one coefficient, $\nu(t)$: bounded for velocity, divergent for noise.

What collapses, what survives

On the CIFAR-10 numbers Sun et al. report, run blind, a noise-prediction DDIM collapses to FID $40.90$, while blind flow matching and uEDM (both velocity) reach $2.61$ and $2.23$. FID is a distance between the real and generated image distributions where lower is better, and on CIFAR-10 anything around $2{-}3$ is near the best published, while $40$ means the samples are visibly broken. That roughly eighteen-fold gap is not a tuning artifact. It is the $1/b$ gain singularity turning ordinary estimation noise into garbage. The paper's own runs on CIFAR-10, SVHN, and Fashion-MNIST show the same pattern by eye: blind noise-prediction images are dominated by high-frequency artifacts, while blind velocity images are sharp and match the conditioned models.

The dimensionality experiment makes the geometry visible. Take a 2D ring of data, embed it in a $D$-dimensional space, and sweep $D$. At $D=2$ both blind models struggle: the shells overlap, the posterior is ambiguous, and the samples are diffuse. At $D=8$ and $32$ velocity is already clean while noise prediction is scattered. At $D=128$ the concentration is so sharp that even the unstable noise-prediction model converges, because its estimation error finally crashes faster than its gain diverges. The blessing of dimensionality, start to finish.

So do diffusion models need noise conditioning? Not always: a velocity- or signal-based model can drop the explicit noise input, because the geometry supplies the level and the dynamics stay bounded, while a noise-prediction model effectively still needs it, because run blind it is structurally unstable. The clean argument leans on data living on a low-dimensional manifold inside a high-dimensional space, and the exponential-stability case for signal prediction assumes a discrete manifold, so the result is sharpest exactly where real image data tends to live.

Wherever the geometry fixes the noise level, the blind model matches the informed one, and whether the sampler reaches the data comes down to the gain coefficient. For the stable parameterizations, then, the noise-level input that every diffusion model carries was information the observation already held. The model was never told the level; it recovers it from the geometry of its own input.