Lyra 2.0: Explorable Generative 3D Worlds
research.nvidia.comThis is the general idea of how I understood it works.
- It's based on WAN a video model, it generates a frame at a time in latent space that we later we decode into RGB
- We keep the latent space and its RGB of each frame in a database. Along with the RGB we compute depth so we get a cloud point (RGBD). This will be used for persistence.
- For each new frame we check which past frames have their point cloud contained in the camera frustum. We take the top 3 frames with more overlap and get their latent space.
- We feed these 3 frames to WAN via the cross attention layers for conditioning and that is how we achieve consistency
- The RGBD can be used to generate a gaussian splatting of the scene.