General purpose robotic foundation models are viable. The recipe that made LLMs work - massive noisy pre-training, curated expert post-training - transfers directly to robotics.
Source (Feb 2025): TWIML AI Podcast - Sergey Levine (Physical Intelligence)
The pre-train/post-train split works for robots, not just text
- Pre-training: ~10,000 hours of teleoperated data across 20+ robot types. Heterogeneous, noisy, multi-operator.
- Post-training: 2-20 hours of curated expert demonstrations per task. Consistent strategy matters more than volume.
- High-quality-only pre-training is actively harmful - low-quality data contains recovery behaviors. Without them the model breaks on its first mistake.
The architecture is a VLM with a diffusion motor cortex bolted on
- Gemma VLM (3.3B) handles vision and language. A smaller diffusion action expert attends to VLM activations and outputs joint angles in real time.
- Inspired by Meta's transfusion pattern - mixture-of-experts where one expert produces text, another produces actions.
- Levine: the recipe matters more than the architecture. A different architecture with the same data recipe would likely work similarly.
JPEG-style action tokenization made training 4x faster
- Naive tokenization (joint angles as digit strings) creates wildly uneven information per token. Internet frequency of "132" has no bearing on action frequency.
- Frequency-domain compression equalizes information per token - same principle as BPE for text, applied to continuous actions.
- 4x training speedup, nothing else changed. Enabled zero-shot transfer to unseen robots at other universities.
There is no internet of robot data
- Every hour of training data requires humans physically teleoperating robots via leader-follower rigs.
- Realism matters - PI collects in real environments (their office kitchen, actual lunch messes). Training must match test.
- Synthetic data is promising but unproven. Real-world foundation + synthetic augmentation may work, but no results yet.
pi0 is at the GPT-2 stage - RL and test-time compute are next
- Levine maps the trajectory: architecture, pre-training, instruction tuning, RLHF, chain-of-thought. pi0 is early in this arc.
- Embodied Chain of Thought already shows 50% improvement - robot reasons through tasks step by step. Currently too slow for real-time.
- Next goal: complex instruction following. Not "fold the shirt" but "clean up the trash, leave the plates, put fork and knife next to each plate."