Building the AAA Lighting Pipeline in Three.js

Ben Houston's Website ·

19 min read Original article ↗

Part 2 picked a pipeline for indoor scenes: lightmaps, light probes and reflection probes, with SSR, SSGI and ambient occlusion on top. Here I measure Three.js against it one stage at a time: what the games require, what Three.js has, and what is missing.

The pipeline combines its sources of bounced light in a cascade: each source returns a color and a coverage from 0 to 1, and the next source fills in the rest. Part 1 explains where that comes from. Five more terms come up often:

  • BRDF: the material's shading function: how a surface turns incoming light into the light you see.
  • G-buffer: per-pixel images of the visible surface (depth, normal, roughness) that screen-space effects read.
  • TRAA: temporal anti-aliasing; each frame is blended with the reprojected frames before it.
  • TAAU: temporal anti-aliasing with upscaling; the scene is rendered below the output resolution and the history is kept at the output resolution.
  • PMREM: an environment map pre-blurred for every roughness level, which is how Three.js lights from a cubemap.

The pipeline#

The pipeline has two parts. The setup fills three caches: lightmaps for static meshes, a light probe grid for moving objects, and reflection probes for each room.

Pipeline setup, three caches filled on load or at authoring time. Lightmaps take four steps: UV2 unwrap, bake with ray radiosity, denoise, then stitch seams. The light probe grid is a single bake that captures each probe and projects it to L2 spherical harmonics. Reflection probes take three: capture HDR cubemaps, GGX prefilter, then pack into a cube array with influence boxes. Seam stitching and the cube array packing are in review; the rest is in Three.js.

The setup: three caches.

Each frame then reads those caches and adds the screen-space work on top:

One frame of the pipeline: a G-buffer prepass, a Hi-Z build feeding an SSR trace, an SSGI trace beside it that reads the G-buffer directly and also returns occlusion and bent normals, their denoise, then a forward beauty pass that reads both coverage cascades and three cached inputs, lightmaps, the light probe grid and reflection probes, then TRAA or TAAU, with the resolved output fed back into the next frame's screen-space traces. The Hi-Z build and the beauty pass that reads both cascades are still to build; the SSR and SSGI trace and their denoise are partly there; the rest is in Three.js.

One frame of the pipeline.

Where things stand#

StageIn Three.jsIn review or draftedStill to build
G-bufferMRT prepassOctahedral normals
Baked GIlightMap, unwrapper, baker, ray radiosity, denoiser, HDR JPEG XLSeam stitching, KHR_lightmap (draft)glTF JPEG XL and UltraHDR extensions, runtime BC6H
Dynamic GILightProbeGrid, 8× faster bakes, multi-bounce, updates spread across framesDDGI visibility (draft)—
OcclusionGTAO, builtinAOContext, AO from SSGINodeBent normals and GTSO (draft), joint bilateral upsample (draft)Bent normals from SSGINode
SSRStochastic GGX SSR, temporal accumulation, multi-bouncePrevious-frame tracing, Hi-Z tracing (draft), radiance and coverage output (draft), coverage in the denoiser (draft)Ray reuse, two-lookup history, skipping mirrors and matte pixels
SSGISSGINode, bounce light and AO from one horizon search, composited over the finished imagePrevious-frame tracing (draft)Coverage output, a diffuse source for the BRDF
Reflection probesSingle box-projected env map, PMREMProbe system (draft), cube arraysProbe normalization, 128² prefiltering
BRDF integrationAO context, environment fallback—Pluggable indirect specular and diffuse, coverage cascade
TRAA and TAAUTRAA node, TAAU nodeCatmull-Rom history (draft), wider clip at rest and test harness (draft)TAAU in the full pipeline, texture mip bias, the TRAA refinements ported to TAAU

The biggest remaining gaps are in the SSR, SSGI and BRDF rows: Hi-Z tracing and coverage for SSR, the same coverage for SSGI, and above all the pluggable indirect lighting that ties both cascades into the BRDF.

It has moved this fast because of the Three.js maintainers. Michael Herzog (@Mugen87) has reviewed most of these PRs, and more than once has reworked one into a better design that fits how Three.js as a whole is built, adjusting the renderer's internals to make room for it. mrdoob has expressed strong support for many of them. I am very lucky to have both.

So far, though, these pieces have been proposed and reviewed one at a time. Nobody has been asked to sign up to the whole. This series is my attempt to put them together into one picture, so that each PR can be judged for where it fits as well as for what it does.

The pipeline, stage by stage#

The G-buffer#

What the games require. SSR, GTAO and the denoisers run in screen space before lighting, so they need per-pixel depth, normals, roughness, metalness and motion vectors. A deferred engine also stores base color and emissive, because its lighting pass reads only the G-buffer. Three.js renders forward and its beauty pass reads those from the material, so the G-buffer fits in 16 bytes per pixel. Part 2 has the layout and the memory budget for the whole frame.

What Three.js has. WebGPURenderer already renders a G-buffer prepass with multiple render targets, and the SSR examples use it today:

const scenePass = pass( scene, camera );
scenePass.setMRT( mrt( {
	output: output,
	diffuseColor: vec4( diffuseColor.rgb, materialMetalness ),
	normal: vec4( packNormalToRGB( normalView ).rgb, materialRoughness ),
	velocity: velocity
} ) );

What is missing. Normal precision. SSR is unforgiving of normal error: a ray reflected off a slightly wrong normal lands somewhere visibly wrong at the end of a long trace. With RGB8 normals, smooth curved surfaces snap to a limited set of directions, and their reflections show banding and clumping.

AAA G-buffers use octahedral encoding: fold the sphere of directions onto an octahedron, unfold it into a square, and store two coordinates. It is the most accurate of the encodings surveyed by Cigolle et al. for its size. PR #35480 adds packNormalToOctahedral() and unpackOctahedralToNormal():

Encoding (4 bytes / pixel)Mean errorMax errorAvailability
RGB8 (today)0.17°0.38°Everywhere
Octahedral RG16F0.011°0.040°Everywhere in WebGPU
Octahedral RG16 snorm0.0013°0.0037°93% of WebGPU devices

RG16 snorm is the target, with RG16F as the fallback.

Reflections on a glossy surface with RGB8 normals on the left, showing banding, and octahedral RG16F normals on the right, smooth

Reflections from RGB8 normals (left) and octahedral RG16F normals (right).

Angular normal error at 0.25° full scale: RGB8 on the left shows error everywhere, octahedral RG16F on the right is near zero

Normal error: RGB8 (left) and octahedral RG16F (right).

Global illumination#

What the games require. Bounce light baked into lightmaps for everything that does not move, and a probe volume to light dynamic objects, ideally one that does not leak light through walls.

What Three.js has. Every standard material takes a lightMap, and there is now a complete baking pipeline in the browser to make one; that is Part 4. For dynamic objects, LightProbeGrid places a grid of probes through a scene, each storing incoming light as L2 spherical harmonics, and lights each object from the probes around it (example).

Sponza lit by the Three.js light probe grid

Sponza lit by LightProbeGrid.

The grid's limit was how long a capture took. Profiling showed the bake was CPU-bound on submitting render passes, so the batched capture renders eight probes per pass with one array camera, and a resolution-12 grid dropped from 5.6 s to 1.2 s. Culling each draw against its own sub-camera doubled that again. The grid now bakes about eight times faster than it did at the start of September, fast enough to re-capture a few probes per frame, and supports multi-bounce light. Integrating the spherical harmonics exactly also cut their error from 2.4% to 0.08%.

What is missing. Probes leak. A probe on one side of a wall can light an object on the other, so a dark room glows with the light of the bright room next door. DDGI solves this by storing, with each probe, the distance to the nearest surface in every direction, and ignoring probes that cannot see the point being lit. I have DDGI-style visibility weighting working as a draft PR. Testing visibility against all eight surrounding probes was costly, so the draft can use tetrahedral interpolation, which blends four:

On SponzaAdded render time
Eight probes, trilinear30%
Four probes, tetrahedral10%

Two adjacent rooms lit by the light probe grid with visibility weighting, one warm and one blue, with no light leaking through the wall between them

Two rooms with probe visibility weighting.

Ambient and specular occlusion#

What the games require. Lightmaps are too coarse for contact shadows, and they know nothing about dynamic objects. Games add screen-space occlusion on top, and Activision's GTAO is the standard. It also produces a bent normal, the average open direction, which gives specular occlusion (GTSO): a reflection pointing into a fold of the surface is occluded, one pointing out of the opening is not.

Horizon-based SSGI, which is what Three.js has, runs the same depth-buffer search as GTAO and gathers light along the way. With SSGI on, its pass supplies the occlusion and the bent normal, and GTAO runs only when SSGI is off.

Which light the occlusion applies to matters as much as how it is computed. Occlusion is a shared buffer, set with builtinAOContext, and each source decides whether to read it:

  • The lightmap, probe grid and environment apply AO to their indirect diffuse, combined with the material's AO map as min( materialAO, gtao ).
  • The reflection probes and environment apply GTSO to their indirect specular.
  • SSR and SSGI apply neither. A screen-space hit is something the ray found, so it is already correctly occluded.
  • Behind SSGI, the diffuse sources skip AO. SSGI's coverage is the share of the hemisphere that something on screen blocks, which is the occlusion itself. The cascade already scales the lightmap or probe grid by 1 − coverage.
  • Occlusion darkens a source's color and leaves its coverage alone. Lowering an occluded probe's coverage would let the next source's unoccluded light leak into the corner.

What Three.js has. GTAO already feeds the lighting through a context, which is the pattern the indirect lighting design in Part 5 extends:

const aoPass = ao( prePassDepth, prePassNormal, camera );
scenePass.contextNode = builtinAOContext( aoPass.getTextureNode().sample( screenUV ).r );

What is missing.

  • Specular occlusion. GTAO outputs a single visibility value, and the lighting model turns it into specular occlusion with a heuristic. It does not know which directions are blocked, so metal keeps reflecting the sky inside folds. Bent normals and GTSO are up as a draft PR. The bent normal comes out of the same horizon search; the cost is an AO target that grows from one byte per pixel to four.
  • Edge-aware upsampling. Half-resolution GTAO upsampled bilinearly leaves dark outlines around objects. jointBilateralUpsample() uses the full-resolution depth and normals as guides (Kopf et al. 2007), so the result stops at the edges. It is drafted and not yet submitted; half-resolution SSR and SSGI need the same filter.
  • Bent normals from SSGINode, so that specular occlusion does not need a second pass when SSGI is on.

A gold bust lit by an environment map: with GTAO and GTSO, folds and cavities stop reflecting the sky; with GTAO and a scalar specular occlusion, they still glow

GTSO (left) and scalar specular occlusion (right).

Screen-space reflections#

SSR has its own series, Fixing Three.js's Screen Space Reflections. This section is the summary.

What the games require. Shipped engines agree on a recipe, set out with its sources in How Games Do Screen-Space Reflections, and What Three.js Does Instead. In a cascade, SSR has to:

  1. Trace the previous frame's lit image, so the reflection exists before the frame is shaded.
  2. Trace exactly over any distance, using a Hi-Z depth pyramid.
  3. Trace at half resolution and reuse rays between pixels, which is what brings the trace under a millisecond.
  4. Output radiance and coverage, so the probes can fill what it misses.
  5. Denoise that radiance and coverage, and leave the BRDF to the material.

What Three.js has. SSRNode importance-samples the full GGX lobe with correct Fresnel, can validate and refine its hits, and its result is reprojected and accumulated over time. The trace is a fixed-step march with short rays. The three SSR examples find 98%, 82% and 46% of the hits within their own ray length, in 2 to 3 ms, and the denoiser fills the gaps from neighbouring pixels. The result is composited over the image, as in the SSR example:

const litColor = scenePassColor.rgb.add( denoisedReflections.rgb );

What is missing. All five requirements, each prototyped or in review except ray reuse.

Exact tracing. A Hi-Z pyramid is a mip chain where each texel stores the nearest depth of the four below it. The ray takes large steps through empty space at coarse levels and single-pixel steps near surfaces, so it finds every hit the depth buffer contains and has no maximum distance to tune. My implementation adds a HiZNode that builds the pyramid in a compute pass. Trace time at 1920×1080, against the fixed-step march raised to the same hit rate:

ExampleLinear march at full qualityHi-Z
ssr_mirror27.6 ms2.9 ms
ssr_stochastic13.7 ms9.2 ms
ssr_animation8.1 ms5.2 ms

That is still a ray for every pixel. Tracing at half resolution with a joint bilateral upsample is written, and ray reuse between pixels, as Frostbite does it, is still to build. Exact Screen-Space Reflections on a Budget has the full plan.

A 1D cross-section of Hi-Z ray marching: coarse mip levels let a reflected ray cross empty space in a few large steps, descending to finer levels near the surface it hits

Hi-Z ray marching.

Coverage. Today a ray that misses samples an environment map inside SSR: an equirectangular texture passed in as environmentNode, which cannot be the PMREM-filtered scene.environment, let alone a local probe. Adding SSR over the image also counts the environment reflection twice. In a cascade, SSR outputs one thing: radiance premultiplied by coverage, with coverage 1 for a hit, 0 for a miss, and fading toward the screen border. The environment input and the BRDF estimate leave the node. This is prototyped as part of Pluggable Indirect Lighting for Three.js.

Tracing the previous frame. SSR cannot be part of the lighting while it reads the lighting's output. DOOM's answer is to trace the current depth buffer but read colors from the previous frame: reproject each hit point into last frame's TRAA output and sample it there. A hit that fails the check against last frame's depth gets zero coverage. Last frame's image already contained last frame's reflections, so a mirror facing a mirror gains a bounce each frame. Previous-frame tracing is in review.

The loop needs two rules. The image SSR reads must be linear HDR radiance, taken after TRAA but before tone mapping and bloom, or reflections pick up tone-mapped colors and bloom halos. And the fed-back radiance must be clamped, because each frame's reflections become the next frame's input and a single firefly compounds frame after frame.

Reflections only, with SSR multi-bounce feedback off and then restored: the restored version shows reflections inside reflections

Reflections with previous-frame feedback.

Denoising. Three.js has two reflection denoisers. The pair the SSR example runs, TemporalReprojectNode and RecurrentDenoiseNode, reads every black pixel as missing data and fills it from its neighbours. That hides the tracer's misses and leaves nothing for the probes to answer. zalo's SVGF-based DenoiseNode replaced the pair in the SSR example and was taken back out five days later. I traced the flicker that caused it to the missing coverage: at the edge of a reflection a hit and a miss were averaged as two samples of one signal, and the history kept resetting.

My ssr-coverage branch carries coverage through every stage of DenoiseNode and takes its statistics from covered samples only. Still to do are a two-lookup history, a history cap that follows roughness, and skipping mirrors and matte pixels. The denoiser filters incoming light and the material applies its BRDF afterwards, so it needs no albedo in the G-buffer and no demodulation step. How Games Denoise Reflections, and What Three.js Does Instead has the comparison and the plan.

Screen-space global illumination#

What the games require. Engines that use SSGI as part of their lighting put it in front of their baked light as the first stage of the diffuse cascade. That means SSGI has to output coverage, trace the previous frame's lit image, and accumulate over time at half resolution.

What Three.js has. SSGINode is built on the visibility-bitmask approach of SSRT3, which extends GTAO's horizon search to gather light as well as occlusion. One search of 24 depth samples yields a GI texture and an AO texture, and it is much less noisy than a single random ray, because each sample accounts for a whole arc of directions. The SSGI example traces the current frame's beauty pass and composites over it.

What is missing.

  • Coverage output. The sectors the search marks as blocked are the directions in which it found a surface, so coverage is 1 − ao, and the node already computes it. What is missing is handing the GI and that coverage to the cascade as a pair, so SSGI can replace the baked bounce where it has an answer.
  • Previous-frame tracing. SSR reads the previous frame once per pixel, where its ray lands. SSGI reads it at every sample that finds a surface, so reprojecting at each read is paid many times per pixel. A draft measures the options on an Apple M3:
ReprojectionFrame rate cost
Every sample, by camera motion14–22%
Every sample, by per-pixel velocity32–44%
Once, by velocity, into a half-resolution buffer the search reads directlyNone measured

The third option also follows moving objects, and shows no visible artifacts in the test scenes.

Reflection probes#

What the games require. Indoors, one environment map is wrong: walk from a red room into a blue corridor and the floor should reflect red, then blue. Every game in the survey, and Unreal's reflection captures, use local reflection probes: cubemaps captured inside each room, each with a box describing the volume it influences. Inside its box a probe is box-projected, which removes most of a cubemap's parallax error (Lagarde's write-up is the classic explanation). Where boxes overlap, probes blend.

What Three.js has. CubeCamera captures a cubemap, PMREMGenerator prefilters it, and getParallaxCorrectNormal() box-projects it, as the box-projected environment map example shows. That is one probe per material, set by hand.

What is missing.

  • A probe system. Box-projected reflection probes for WebGPURenderer adds scene-level probes with influence boxes and blending, and is up as a draft PR.
  • Fast access to many probes. Cube array textures put every probe in one texture; sampling them is merged, and the texture class is in review.
  • Probe normalization. A probe is captured at one point in a room, so a dark corner or the inside of a cupboard reflects a brightly lit room. Call of Duty's fix, introduced in Black Ops II and carried into Infinite Warfare, is to scale the probe's reflection by the ratio of the diffuse light at the pixel to the diffuse light at the probe. The lightmaps and probe grid already provide both. Still to do.
  • Small probes. PMREMGenerator has a minimum size of 256². A 128² probe is enough for blurry room reflections, since SSR handles the sharp ones, and takes a quarter of the memory. Still to do.

Two connected rooms in Three.js lit by three box-projected reflection probes

Two rooms lit by three box-projected probes.

Indirect lighting in the BRDF#

What the games require. The bounced light has to go through the material's BRDF, so Fresnel, roughness and metalness apply to it as they do to direct light. In a forward renderer the beauty pass needs two inputs:

  • Indirect diffuse: SSGI where it has coverage, then the lightmap on static meshes or the probe grid on everything else, then the environment map.
  • Indirect specular: SSR, then the box-projected probes, then the environment map.

Three.js renders forward, so SSR, SSGI and AO are computed before the beauty pass and each material reads them while it shades. Tracing the previous frame is what makes that order possible.

What Three.js has. Indirect diffuse is mostly in place: lightMap, LightProbeGrid and scene.environment all feed it, and AO reaches it through builtinAOContext. Indirect specular comes only from scene.environment, or a material's own envMap.

What is missing. A way to plug a screen-space result, or a cascade of sources, into a material's indirect specular, the way AO is plugged into its diffuse. This is the biggest gap in the pipeline, and my proposed design is in Pluggable Indirect Lighting for Three.js.

TRAA and TAAU#

What the games require. Temporal anti-aliasing whose output can also be next frame's input to SSR and SSGI, so it has to stay sharp under motion and must not feed ghosts or fireflies back into the reflections. Every engine this series measures itself against also renders below the output resolution and reconstructs the difference over time: TSR in Unreal, FSR 2 in Godot, DLSS and FSR in Cyberpunk 2077.

What Three.js has. A TRAA node with Halton jitter, motion-vector reprojection and variance clipping of the history, and TAAUNode, which follows Karis's temporal supersampling and FSR 2. The TAAU example shows it at a quarter of the pixels:

scenePass.setResolutionScale( 0.5 );

const taauNode = taau( scenePassColor, scenePassDepth, scenePassVelocity, camera );

TAAU takes TRAA's place as the last stage of the frame. Everything before it runs at the render resolution, its linear HDR output is what the next frame's SSR and SSGI read, and bloom and tone mapping run after it. At a scale of 0.5, the screen buffers at 1080p drop from about 157 MB to 81 MB.

What is missing.

  • Wider clipping at rest. Variance clipping stops ghosting by rejecting history that no longer matches the current frame, and it also rejects good history. TRAA tightens its clip box to 0.5σ under fast motion, but at rest the box was 1σ, tight enough to clip converged sub-pixel detail, so a still image never fully resolved and longer histories scored worse. A drafted change widens the box to 2σ at rest and adds a historyLength control. Against a supersampled reference:

    ViewPSNR change
    Static+2.15 dB
    Still camera, moving objects+1.3 dB
    Fast camera motionWithin 0.06 dB
  • Sharper history. History sampled bilinearly blurs a little more every frame the camera moves. Catmull-Rom history sampling, a draft PR, keeps it sharp. Both refinements apply to TAAU as well.

  • TAAU in the full pipeline. The example upscales a single scene pass. The pipeline has a prepass, half-resolution effects and a beauty pass that all have to agree on resolution and jitter, and none of that has been tried together yet. With TAAU at 0.5, half-resolution SSR and SSGI would run at a quarter of the output resolution on each axis, so it may be better to run them at the full render resolution and skip the bilateral upsample.

  • Texture mip bias. Rendering at half resolution makes the GPU pick blurrier texture mips than the output deserves. Temporal upscalers pair with a negative mip bias of about log2 of the resolution scale, and Three.js does not apply one yet.

Future work#

  • Cheaper noise. Houston Stratified Noise costs about 400 operations per sample, on every pixel, every frame. I want to move some of that into a small map generated at startup, so each sample is a lookup and a few adds.
  • Auto-exposure. Walking from a dark room into a bright courtyard needs eye adaptation, and Three.js does not have it yet.
  • A physically based sun and sky. A room lit through its windows needs a sun and sky in real units that agree with each other and with the exposure. A prototype does single scattering; multiple scattering and clouds are still to do.
  • Reflection probes in glTF. There is no standard way to ship reflection probes. The minimum is each probe's position and influence box, so a loader can capture the probes on load.
  • Directional lightmaps. The lightmaps here store one irradiance value per texel, so normal maps cannot respond to baked light. The options are an ambient term plus one dominant direction as in The Last of Us and Call of Duty, L1 spherical harmonics as in Frostbite, or spherical Gaussians as in The Order: 1886.