Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches like Kimodo offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations demonstrate strong motion quality and constraint adherence, and we present an interactive demo with dynamic text control, keyframe constraints, path following, and real-time locomotion control.
Key Capabilities of ARDY
Application: Interactive Humanoid Control
ARDY enables online motion synthesis for interactive applications, which can be valuable for game character control, downstream robotics, and simulation workflows. It supports real-time locomotion control via mouse waypoint editing and keyboard velocity commands.
Humanoid Robot Control
By combining ARDY’s real-time humanoid motion generation with the SONIC physical tracking policy, we enable interactive robot motion control with streaming constraints and user inputs. We demonstrate applications on the Unitree G1 robot.
Method
ARDY is an autoregressive diffusion model for interactive motion generation. It is built around two key ideas: (1) a hybrid motion representation that combines explicit global root motion with a compact latent embedding of body motion, and (2) an autoregressive two-stage transformer denoiser that generates motion in a streaming fashion while conditioning on online text prompts and flexible, spatiotemporally sparse kinematic constraints over long horizons.
Motion Tokenizer. The encoder first embeds the patchified body motion into a latent representation. This latent body motion is concatenated with the patchified global root motion to form our hybrid representation, which is decoded back to reconstruct the body motion.
This hybrid representation balances precise, interpretable root control (useful for global scene-space constraints) with efficient generative learning in a lower-dimensional latent space for body motion.
Autoregressive Two-Stage Transformer Denoiser. (Left) Conditioned on a variable-length history context and optional spatial goal constraints, the autoregressive denoiser predicts a sequence of C clean motion tokens within the current generation window. Spatial goal constraints can be arbitrarily sparse and may be located within or beyond the current motion generation window. (Right) The two-stage denoiser first predicts clean global root motion, which then conditions the second stage to predict clean latent body tokens, together forming the complete hybrid motion prediction.
The denoiser supports a variable-length history context to capture longer-term semantics, and conditions on masked kinematic constraints that can be sparse in both time and joints. These constraints can span beyond the current generation window to enable long-horizon goals (e.g., far-future waypoints).
Our interleaved two-stage design predicts root first and then body conditioned on root, helping maintain motion fidelity while satisfying online text prompts and diverse spatial controls such as root trajectories/waypoints, full-body keyframes, and sparse joint positions/rotations.
More details can be found in the paper.
Humanoid Motion at NVIDIA
ARDY is part of a broader effort to support interactive humanoid motion generation and downstream robotics/animation applications. Related projects include:
SOMA Body Model
A parametric body model used across NVIDIA humanoid motion projects.
ProtoMotions
A framework for training physics-based humanoid policies.
SOMA Retargeter
Tools for retargeting motion data across humanoid skeletons.
GEM
A motion diffusion model for reconstructing motion from monocular videos.
Kimodo
An offline controllable motion diffusion model for high-quality 3D motion authoring.
MotionBricks
Scalable real-time motions with modular latent generative model and smart primitives.
GEAR SONIC
A framework for humanoid whole-body tracking and control.