We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.
We set out with the goal of understanding what it would take to make human data useful for VLAs like π0.5. We record egocentric human data with wearable cameras, and then include it in a co-training recipe with hand poses serving as actions.
If we use our full pre-trained pi05 model, simply finetuning with human video data can double the performance on tasks that are depicted in the human videos!
We were surprised, and wanted to understand why. What about π0.5 enabled emergent human-robot transfer? We ran an experiment to test if it only appears above a certain scale. Turns out human transfer scales with the amount & diversity of robot data in VLA pre-training!
This also shows up in the representations learned by the model. We plot the model’s representations of human and robot images. As pre-training is scaled up, the representation of humans and robots become more aligned: to a scaled-up model, human videos "look" like robot demos.
Check out the blog post and full research paper for more details and experiments, including studies into high level vs low level transfer, comparisons to robot data, and quantifying the utility of wrist cameras.
Seeing the activations between the human and robot actions get closer and closer is incredible Hats off to the team for presenting their generalization evidence in a visually compelling way!