USDCraft — Articulated 3D Assets for Simulation

4 min read Original article ↗

Demo

Method

USDCraft method: image and geometry inputs, axial geometry encoding, mesh-native modeling toolkit, physical parameters, program authoring and iterative geometric rechecking, with articulated assets exported for simulation.

Generation

New articulated assets from text or images

Without any input mesh, USDCraft writes an asset program from a prompt or a reference image. Pretrained LLMs bring broad knowledge of everyday objects, so new categories need no task-specific training. Every result below is the compiled, articulated USD asset — drag to orbit; parts move through their joint limits.

Upright piano

Image → asset

Input image for Upright piano
Input image

Backyard playground set

Image → asset

Input image for Backyard playground set
Input image

Manual wheelchair

Text → asset

Prompt

Create a complete, functionally usable, simulator-ready manual wheelchair with two rotating drive wheels, swiveling front casters, fold-up footrests, push rims, brakes, seat, backrest, and armrests.

Parcel locker bank

Text + image → asset

Input image for Parcel locker bank
Input image

Prompt

Create a complete, functionally usable, simulator-ready parcel locker bank with eight differently sized independently hinged doors, handles or electronic latches, kiosk screen, stable enclosure, and clear retrieval access.

Deployable satellite antenna

Text → asset

Prompt

Create a complete, functionally articulated, simulator-ready deployable satellite high-gain antenna assembly with a spacecraft mounting pedestal, two-axis azimuth-and-elevation gimbal, a parabolic reflector divided into a rigid center dish and two foldable side petals, locking hinge housings, central feed horn, support struts, cable-routing brackets, hard stops, and launch-restraint latches. The antenna must fold into a compact stowed configuration and deploy into a coherent dish shape, while the deployed reflector can aim through both gimbal axes without colliding with the pedestal.

Manual pallet jack

Text → asset

Prompt

Create a complete, functionally usable, simulator-ready manual pallet jack with pivoting steering handle, rotating steer and load wheels, lifting fork mechanism, control lever, and clear pallet entry.

Explore the generated assets

16 categories · pick an asset, scrub its joints, download the USDZ

Drag to orbit · scroll to zoom

Input image credits

Reconstruction

Rebuilding real objects for simulation

For real-to-sim, USDCraft rebuilds each source mesh as a program, separating fused parts and completing missing ones. The input can be a static asset, a raw phone scan, or a single image lifted to a mesh.

Views are linked — orbit either side to compare the source with the rebuilt, articulated asset.

Comparison with prior methods

Same Hunyuan3D 2.1 input for every method · paper results (USDCraft and Mini Workflow: round 1)

Loading 3DInput image of the ergonomic office chair

Input · image + Hunyuan3D 2.1 mesh

No complete articulated outputpart geometry missing

ArtLLM

Particulate9 parts · 8 joints

Mini Workflow–Astra12 parts · 11 joints

USDCraft–Astra15 parts · 14 joints

All six views share one camera — part colors show how each method splits the object into rigid parts; joints sweep through their limits.

Simulation

Loads into Isaac Sim, ready to run

Every compiled asset carries collision geometry, masses and joint limits, so it runs in Isaac Sim (PhysX) without manual fixes. Each clip shows the static asset, its dynamic articulation, and its collision geometry.

Warehouse forklift
Compact excavator
Baby grand piano
Robotic workbench arm

Real2Sim2Real

Policies trained only in simulation, deployed on real objects

We scan three real objects, rebuild them with USDCraft, collect 200 demonstrations per task in simulation, and train a Diffusion Policy that is deployed on the physical robot with no real-world data.

Open drawer

Simulation · data collection
Real world · policy rollout (4× speed)

Press toaster lever

Simulation · data collection
Real world · policy rollout (4× speed)

Turn on toaster

Simulation · policy rollout
Real world · policy rollout (4× speed)

Success rate across asset methods

Paper Table · 20 trials per setting · ↓ sim-to-real drop · I/M: image/mesh input

Asset methodInputOpen drawerPress toaster leverTurn on toaster
SimRealSimRealSimReal
Articraft–SolI90%20%↓7090%10%↓8080%20%↓60
ParticulateM0%0%50%10%↓400%0%
Mini Workflow–AstraI+M100%0%↓10090%60%↓3085%70%↓15
USDCraft–AstraI+M100%90%↓1090%85%↓585%85%↓0

BibTeX

@article{zhang2026usdcraft,
  title={USDCraft: Geometrically Grounded Programmatic Modeling of Articulated 3D Assets for Simulation},
  author={Zhang, Chuanrui and Yang, Zaijia and Wang, Duomin and Shi, Lu and Zhou, Daquan and Zhang, Ruihua and Wang, Ziwei},
  journal={arXiv preprint arXiv:2610.11322},
  year={2026}
}