AI RESEARCH FROM META
Introducing
Meta SAM 3D
SAM 3D can bring any 2D image to life, accurately reconstructing objects and humans, including their shape and pose.
SAM 3D CAPABILITIES
Accurately reconstruct objects and bodies
Object reconstruction
SAM 3D enables precise 3D reconstruction of objects from real images, while accurately reconstructing their geometry and texture.
Body pose & shape estimation
SAM 3D allows for accurate 3D reconstruction of human body shape and position from a single image.
Scene reconstruction
SAM 3D works on real images in-the-wild, maintaining strong fidelity and quality.
Real world 3D perception
SAM 3D enables full scene reconstructions, placing objects and humans in a shared context together.
The SAM 3D models
SAM 3D contains two state-of-the-art models that enable 3D reconstruction of objects and humans from a single image.
SAM 3D Objects
Detailed 3D reconstruction of any masked objects, including geometry and texture
Independent, posed 3D models, suitable for manipulation & interaction
Reconstructions are robust to occlusion in the input image
Position multiple objects into a scene, jointly with SAM 3D Body reconstructions
SAM 3D Body
Reconstructs body shape and pose, including unique positions and partial visibility
Suitable for manipulation and interaction
Promptable with joint reconstructions
Position multiple people into a scene, jointly with SAM 3D Objects reconstructions
Designed for practical 3D applications
Enhancing Facebook Marketplace shopping
Transforming physical therapy
Empowering robotics applications
Place a 3D AR overlay of home decor, like a lamp or a table, from Marketplace in your room to visualize the style and fit within your space before purchasing.
OUR APPROACH
Model architecture
SAM 3D is a suite of two models: SAM 3D Body and SAM 3D Objects:
- The SAM 3D Body model architecture uses a transformer-based encoder-decoder architecture to predict 3D human pose and mesh parameters directly from images, enabling accurate and interactive pose regression.
- The SAM 3D Objects model employs two stages of DiTs—first generating 3D object shape and pose, then refining texture and details—to deliver high-fidelity, realistic 3D reconstructions.
BENCHMARKS
State-of-the-art performance
SAM 3D achieves beyond state-of-the-art performance across a series of benchmarks for both its models.
THE SAM 3D ARTIST OBJECT DATASET
A dataset of diverse and high-quality 3D meshes
A new first-of-its-kind evaluation set for visually grounded 3D reconstruction in real-world images, with diverse images and objects that are significantly more challenging than existing 3D benchmarks. This represents a new way to measure research progress in 3D, and pushes the field away from curated images/synthetic assets and towards real-world perception and common-sense 3D understanding.
