How Physical AI Learns Across Language, Video and Action — Ming-Yu Liu

Summary

Ming-Yu Liu describes Cosmos 3 as a multimodal physical-AI system that starts with a vision-language model and then uses a diffusion-based generator tower to produce video, audio, and robot actions. He argues that world models should be treated as a collection of useful tools—forward dynamics, inverse dynamics, and policy—trained with shared constraints so they help one another. A major theme is temporal and embodiment alignment: Cosmos normalizes different signal rates across modalities and leverages human egocentric video as a source of transferable action-visual structure for robots. On the evaluation side, Liu says the near-term value of neural simulators is ranking preservation, not perfect realism, because they can help teams eliminate weaker checkpoints before costly real-world trials. The episode closes with deployment and openness details, including Super, Nano, and Edge variants, edge targets like Jetson Thor and Orin, and public models, code, and some data on Hugging Face and GitHub.

Notes

• Cosmos 3 is presented as a multimodal physical-AI platform spanning text, video, audio, and actions.
• The episode opens with a generated road scene that was never filmed, used as a concrete example of model-driven simulation.
• Cosmos is framed as one model that can describe video, generate video, and produce robot actions.
• The architecture starts with a language model, optionally initialized from an open model.
• A vision encoder is attached to the language model to build a vision-language model.
• The vision-language model reasons autoregressively, one token at a time.
• Cosmos then reuses those weights to initialize a separate generator tower.
• The generator tower is diffusion-based rather than autoregressive.
• That bidirectional generator learns to produce video, action, and audio.
• The model uses a shared temporal-position scheme to align modalities with different sampling rates.
• Temporal embeddings let the system identify which tokens belong to the same moment in time.
• Liu treats world models as a practical toolbox rather than a single agreed definition.
• He identifies three core robotics modeling objectives: forward dynamics, inverse dynamics, and policy.
• Forward dynamics predicts the future from the starting state and action.
• Inverse dynamics infers the action from the observed visual transition.
• Policy modeling asks what the robot should do to complete the task.
• The training objective is described as learning shared correlations under an information bottleneck.
• Cosmos is trained to capture correlations between observations and actions.
• Human egocentric video is positioned as a rich source of transferable embodiment knowledge.
• The key transfer claim is shared visual-action patterns, even when action spaces do not match exactly.
• This embodiment argument is used to justify learning from human video and transferring to robots.
• Ambiguous tasks are a known failure mode for lower-level execution.
• He suggests a higher-level system-two planner should concretize tasks for a system-one executor.
• Liu says neural simulators are first useful for passive verification rather than full deployment replacement.
• The practical use case is comparing many checkpoints before real-world trials.
• A simulator does not need exact success rates to be useful.
• Ranking preservation is the key criterion for simulator usefulness.
• If rankings are preserved, teams can reduce the number of real-world trials.
• Cosmos is positioned as a starting point for robot policies, not just a theory project.
• The team says it post-trained Cosmos on the DROID dataset.
• They claim state-of-the-art pick-and-place results after post-training.
• Cosmos Dreams extends the closed-loop simulator idea to driving and robotics.
• Safety matters especially for humanoids around children and pets.
• Cosmos comes in three sizes: Super, Nano, and Edge.
• Super is described as the highest-fidelity frontier model.
• Nano is the intermediate option for easier adaptation across available GPU resources.
• Edge is designed to run on-device rather than requiring a data-center round trip.
• Target edge hardware includes Jetson Thor and Orin.
• The models, code, and some training data are said to be open.
• Models and data are hosted on Hugging Face, with code on GitHub.
• The landing repository is called media cosmo.
• The repository links out to models, datasets, and post-training scripts.
• The episode emphasizes open weights and open tooling as part of the Cosmos release strategy.
• The broader claim is that more abundant human first-person video can help bootstrap robot learning because the visual structure overlaps.
• The main evaluation insight is that simulator fidelity can be traded for better policy ranking.
• The episode positions Cosmos 3 as a practical system for both generative and control tasks.