-
Alexander Mai
University of California, San Diego
-
Trevor Hedstrom
University of California, San Diego
-
George Kopanas
Runway ML & Google
-
Janne Kontkanen
Google
-
Falko Kuester
University of California, San Diego
-
Jonathan T. Barron
Google
Abstract
We introduce radiance meshes, a technique for representing radiance fields with constant density tetrahedral cells produced with a Delaunay tetrahedralization. Unlike a Voronoi diagram, a Delaunay tetrahedralization yields simple triangles that are natively supported by existing hardware. As such, our model is able to perform exact and fast volume rendering using both rasterization and ray-tracing. We introduce a new rasterization method that achieves faster rendering speeds than all prior radiance field representations (assuming an equivalent number of primitives and resolution) across a variety of platforms. Optimizing the positions of Delaunay vertices introduces topological discontinuities (edge flips). To solve this, we use a Zip-NeRF-style backbone which allows us to express a smoothly varying field even when the topology changes. Our rendering method exactly evaluates the volume rendering equation and enables high quality, real-time view synthesis on standard consumer hardware. Our tetrahedral meshes also lend themselves to a variety of exciting applications including fisheye lens distortion, physics-based simulation, editing, and mesh extraction.
Our work, the radiance mesh, fills a gap in the taxonomy of radiance field techniques by introducing rasterization for fields. Techniques for representing radiance fields can be split into two categories: fields and sets of primitives. Then, we can either ray trace or rasterize the result. Rasterization, which is often faster, works by projecting primitives into image space before rendering, while ray tracing works with the individual rays in world space.
Circumcenter Smoothing
We chose to optimize the Delaunay triangulation. If we naively attach color to the vertices, this flip causes a discontinuous jump in appearance. Our solution is to query a neural field at the tetrahedron's circumcenter. Now, at the exact moment of the flip, the circumcenters of the new tetrahedra are co-located with the old ones. This makes the transition continuous and allows for stable, gradient-based optimization.
Two different approaches for coloring the same Delaunay triangulation. Left: Naive approach using the average color of the vertices. Right: our approach, where triangles are colored according to their circumcenters (shown using the dots).
We store the attributes of the field within an instant-NGP, which we query to obtain the attributes of each tetrahedron. During optimization, the scene is represented by the following optimizable parameters: a set of 3D points \(\mathcal{X}\) and a neural network weights \(\theta\).
Our improvements stabilize the optimization, allowing our method to surpass Radiant Foam in quality. Not only is our optimization better behaved, but the resulting radiance mesh is more flexible and composed entirely of triangles.
Sorting
We start by sorting the tetrahedra by a quantity called the power of the circumsphere with respect to the camera position. Back in 1997, Karasick et al. proved that this gives the correct sort order for that camera position, and gives us the ability to sort all of the primitives using a single radix sort, which takes less than a millisecond on a fast GPU.
To sort two adjacent triangles relative to a viewpoint, we need only look at their circumcircles. The radical axis (the line along which the power of a spheres is equal) goes through the two intersection points between the circles, and divides the circumscribed triangles. Consider the centers of two circumcircles $A$ and $B$ and a viewpoint $C$ located along the radical axis. Since $|A-C|$ increases as the camera moves right and $|B-C|$ decreases, the power of B is less than the power of A to the right, and the reverse is true to the left. This extends to non-intersecting circles, and can be used for non adjacent triangles. Combining this observation with the the Delaunay empty sphere property allows us to sort all triangles.
Rasterization Pipeline
We take our sorted primitives and feed them into our rasterization pipeline, which emits a triangle strip for each tetrahedron. We then attach the tetrahedron attributes to the vertices, which the fragment shader then accesses to calculate the transparency and color of each pixel.
Results
Our novel rendering technique is extremely fast. At 1440p resolution, our method is 32% faster than 3D Gaussian Splatting. Across all MipNeRF 360 scenes, Radiance Meshes define the new state-of-the-art for performance per primitive.
Although our radiance mesh is not a surface mesh, it’s still compatible with many parts of the graphics ecosystem. Here, we're using Position Based Dynamics to interactively simulate forces on the mesh.
We can easily support complex camera models, like this fisheye lens.
We can extract a watertight surface mesh for use in any standard 3D modeling, animation, or game engine.