Source (Jul 2025): Shrea (Meta Reality Labs) - Exploring VLA Models
Pre-LLM robotics required custom models, simulators, and massive data collection for every task
- 2022 approach: train a neural network (goal net) on thousands of human demonstrations in simulation, pair it with a classical planner, predict one sub-goal at a time autoregressively.
- Even simple tasks like "place all utensils in the sink" required huge state spaces - 20 objects x 20 states = combinatorial explosion.
- Every new task meant building a new simulator, collecting new demonstrations, and retraining. Sim-to-real transfer added another layer of engineering.
Pretrained LLMs replaced most of that pipeline with few-shot prompting
- The same goal prediction problem (language instruction to grounded robot actions) now solved by prompting a frozen LLM with a few examples of the desired output format.
- Common sense reasoning comes free from internet-scale pretraining - the model already knows apples and oranges are both food, no domain-specific training needed.
- Early word encoders (BERT) gave limited generalization to unseen objects. Current VLMs give full scene understanding and instruction decomposition out of the box.
VLAs close the loop: visual perception + language reasoning + action output in one model
- Vision encoder perceives the environment, language model encodes instructions, action decoder outputs motor commands. All three integrated.
- The action decoder can be RL-based or imitation learning - the key shift is that perception and reasoning are handled by pretrained foundation models, not trained from scratch.
- Suited to both structured environments (factory assembly lines) and unstructured ones (household tasks) because the common sense reasoning transfers.
LLMs are also useful indirectly - reward modeling, task decomposition, planning
- Off-road autonomous navigation: fed simulator logs into an LLM (Eureka, Nvidia) to auto-generate reward functions for RL policy training. No hand-designed rewards.
- High-level task planners: LLMs decompose complex instructions into executable subtasks at the right granularity for the robot's action space.
- Open research: what level of fidelity should the LLM plan at vs. leaving to low-level control? Better VLMs, better policy learning, better planners are all active fronts.