HDR video is coming to Firefox for Windows users (and has been available for some time on macOS).  This blog post explains how we developed the feature and gives a retrospective on the technical choices we made.

A primer on video playback for the web:

  • Video file demux and decode: A video stream generally consists of parallel image and audio streams, along with captions, HDR scene metadata, and the like. “Container” formats like MP4 or MKV specify how these streams are combined, or multiplexed, into a single byte stream for transmission. On receipt, Firefox needs to divide that byte stream back into the individual media streams; this is de-multiplexing or “demuxing”. Then Firefox must uncompress the data to get images, audio samples, and so on. Firefox’s media team provides the demuxers, and pulls in appropriate codecs to decode them. We prefer using hardware video decoders if they work reasonably well. Video decompression usually produces roughly a YUV 4:2:0 image in NV12 for SDR or P010 for HDR. (If you visit about:support in Firefox, and search for Codec Support Information (or one of the codec names like AV1), you can see a whole feature matrix of support details for which codecs are hardware and software on your system.)
  • Gecko displaylist building: Given a demultiplexed, uncompressed frame of video, Gecko displaylist building incorporates it into a video element in the displaylist being sent to WebRender. If the frame was decoded in hardware, it is generally represented by a texture in GPU memory. Or, if it was decoded in software, then it is represented by a memory mapping holding some raw pixel data in system memory shared with Firefox’s media decoder process.
  • WebRender: Given the video element in the displaylist, WebRender decides whether to promote it to a desktop compositor overlay, or whether it must instead be rendered using a pathway more like an ordinary HTML element. A compositor overlay is faster and uses less power; on Windows this uses DWM with the DirectComposition API, which manages a graph of visuals. But if complex CSS is involved (rounded corners, blur filters, or similar features), Firefox must use WebRender’s ordinary rendering pathway. Currently the latter is not HDR capable, so Firefox favors the desktop compositor overlay for animated elements such as video and canvas.

As we began designing Firefox’s HDR support, we had to lay out some assumptions and found many complications:

  • Initially, we had hoped that on a modern system, BT2100 HDR videos could be displayed on Windows by simply sending them to DirectComposition.
    • In theory, the Desktop Window Manager (DWM) honors the DXGISwapChain3::SetColorSpace1 method which should let us request either DXGI_COLOR_SPACE_YCBCR_STUDIO_G2084_LEFT_P2020 or DXGI_COLOR_SPACE_YCBCR_STUDIO_GHLG_LEFT_P2020. The former refers to SMPTE 2084, more commonly called PQ, the Perceptual Quantizer function and the latter is ARIB-STD-B67 also known as HLG, the Hybrid Log Gamma function, most commonly used on HDR TV broadcasts.
    • Unfortunately, this was a dead end. In testing with a mocked up compositor test app, calling SetColorSpace1 with this value seems to be ignored on P010 (at least in testing on AMD), so it incorrectly displays BT2100 PQ video as if it were BT709, which makes the video dull and muddy, since BT709 is a narrower gamut than BT2020, and the BT1886 transfer function used by BT709 is very different from PQ defined by BT2100. SetColorSpace1 may work on other vendors with P010, so it may be a valid optimization, but we were looking for a universal solution.
    • For the future, Windows 11 23H2 has added a new interface called IDCompositionTexture which may serve our purposes better; from what we have been told, it is universally supported for all formats and color spaces. We haven’t used it for video so far, but it’s an interesting future direction.
  • As noted above, HDR videos must use a desktop compositor overlay. HDR video uses the BT2100 PQ colorspace with an RGB10A2 format, while WebRender can only work with images in the sRGB colorspace (appropriate for standard-dynamic-range BT709 video).
    • Until HDR came along, Gecko and WebRender only used desktop compositor overlays as a power/performance optimization. With HDR, overlays become a necessity as the pixel format and color space differ from classic sRGB.
    • Fortunately, HDR videos tend to be shown without particularly fancy CSS rendering such as clip masks and rounded corners, which would require WebRender to perform further copies. Technically, DirectComposition does support all of those features, but Firefox doesn’t use that functionality much.
    • In the future, we expect to upgrade WebRender for HDR rendering, allowing us to deal with complex cases like clip masks or blur filters on video elements.
  • We considered whether we could use VideoProcessorBlt, or whether we should write our own shader instead.
    • In favor of VideoProcessorBlt:
      • It uses less power on GPUs that have a video processor unit.
      • We discovered in testing (using CheckVideoProcessorFormatConversion) that while many modern GPUs support one of the needed conversions (P010 PQ -> RGB10 PQ), few support the ones we need for HLG videos (P010 HLG -> RGB10 PQ).
      • The ‘video-dynamic-range’ query used on the web is not fine-grained enough to be able to say “the web browser can display PQ video but not HLG video”, so if we went with VideoProcessorBlt as a required feature, only about 20% of HDR desktop users would be able to use the feature.
      • In the future, we could explore using VideoProcessorBlit to save power on hardware that supports the conversions we need. But other web browsers are not using this functionality, so there may be more issues we haven’t found yet.
    • In favor of writing our own shader with all of the features:
      • This would work consistently on all vendors – nothing special here.
      • This would look the same on all vendors, regardless of hardware capabilities. This is generally the aim of web standards.
      • This would support anything we want it to. HDR tonemapping can be implemented. Video orientation can be implemented (for videos recorded on phones which may be rotated 90, 180 or 270 degrees). We can support any kind of YUV->RGB conversion with a color matrix (even weird legacy formats like GBR 4:2:0).  We can support conversion between color primaries (e.g. BT2020->BT709).  We can convert to linear color (for scRGB using RGBA16F) or any EOTF we want (notably BT2100 PQ with RGB10A2, for our use-case).
    • In the end we went with the shader after a significant period of time experimenting with VideoProcessorBlt in our Nightly releases.
  • There is a very large amount of graphics code in Gecko and WebRender that needs to be upgraded for HDR.
    • We decided that the most important code paths to upgrade first are the ones for regular video playback and DRM-protected video playback, and later canvas video import (Canvas2D, WebGL, WebGPU) which will require upgrading canvas for HDR first – another big project.
    • We had to upgrade several dozen structs to carry the transfer function for video data, as previously all code assumed video used BT1886 EOTF.
  • We hope we can avoid tone mapping HDR content when viewed on HDR displays.
    • It’s reasonable to expect that most displays going forward will be HDR displays (partly because of marketing momentum, partly because displays are made by a very finite set of manufacturers who are all making HDR display panels), and eventually tone mapping may become unnecessary on the web.
    • For the short-term we will have to apply a tone mapping effect when HDR content is viewed on SDR displays, likely using  ‘Reinhard tonemapping’ which refers to the widely available paper Photographic Tone Reproduction for Digital Images by Erik Reinhard et al, and configuring it for a fixed brightness ratio of 400 cd/m^2 -> 100 cd/m^2 when used on SDR displays, and see if that fits all HDR content on the web well enough for a good user experience – and if it does not, we will iterate based on feedback from users on Firefox Nightly.
    • We are hoping that we will never have to apply tonemapping for HDR content on HDR displays, there are multiple factors in this decision:
      • Varying the brightness limit would make it a significant fingerprinting vector if not handled very carefully if the script can inspect pixels or parameters related to that.  There are ways to mitigate this but they are all awkward restrictions to impose, and queries would have to get a different answer than what the rendering is using.
      • Phones and laptops with light sensors may vary the reference brightness in real time, and this changes the maximum displayable ratio (aka HDR headroom) every refresh, which is also a major battery drain if we keep redrawing all of the time.
      • Documents composed of multiple images (a gallery or some form of art composition) would apply different tonemapping to each image if the brightest pixel in each image is different brightness.  We’d have to do something about that to make it controllable via CSS.
      • In general the detailed parts of an image are within a certain brightness band – see Debunking HDR for a detailed lecture on film grading and why you would not have significant difference in brightness between scene elements.
      • User feedback so far has indicated that not applying tonemapping has given them a better viewing experience on some videos.
  • WebRTC is implemented using a library, common to all web browsers, which has limited support for HDR.
    • While we didn’t prioritize this for an initial feature launch, we are looking at how to implement HDR support properly in libwebrtc. This is in the early assessment phase but we know this is wanted for a couple of use-cases, like video calls for meetings, or game streaming with friends watching.

In general, one of the biggest challenges in working on graphics code in a web browser is a lack of documentation for how to best use features like video playback and desktop compositing in the context of a web browser (e.g. multiple processes, sandboxing, shared memory, sharing external textures, etc). This parallels the rarity of graphics engineers with such experience. Building new features in this space requires a lot of research (and a lot of trial and error). The solution you end up with may not look at all like the one you initially imagined.

On behalf of the graphics team at Mozilla, I want to thank the people who use Firefox Nightly regularly and file bug reports when things aren’t working the way they want. Comments on Experimental High Dynamic Range video playback on Windows in Firefox Nightly 148, Mozilla Connect, and Bugzilla bug reports have guided us to focus on the use-cases that matter to people using Firefox. When we succeed, it’s a great feeling.

We’re working on extending HDR support to photos, apps/games and general web content.

Published