Too school for cool: Runway AI’s entertaining video models accidentally learned physics, now train robots

Runway’s AI video models have evolved beyond creating special effects to accidentally learning the laws of physics through video training data. The company’s technology now attracts robotics firms seeking realistic simulations for machine training, marking an unexpected shift from entertainment applications to real-world physics modeling.

What you should know: Runway’s Gen-4 model demonstrates an emerging ability to simulate complex physical phenomena without being explicitly programmed for physics.

• The models predict realistic movement of light, water, and other physical elements by analyzing patterns in massive video datasets.
• “The more we put compute and data behind scaling those models, the more capable they become at simulating reality,” said Anastasis Germanidis, Runway’s co-founder and chief technology officer.
• Robotics companies now use the technology to create three-dimensional world simulations for training autonomous systems.

Why this matters: AI video models could revolutionize how we approach physics simulation and scientific modeling across industries.

• Traditional physics models require enormous computational resources for narrow predictions, while neural networks offer a potential shortcut.
• “Simulation is the big bottleneck that’s going to be required to solve challenges that are in robotics and self-driving, or any real-world problems that one might care about,” Germanidis explained.
• The technology addresses a critical need in autonomous vehicle development and robotics training.

How it works: The models learn physics as an unintended consequence of processing vast amounts of video content.

• Runway tests its models by generating simple scenarios like a ball hanging from a ceiling, then comparing the trajectory to real-world physics.
• The AI systems act as “black boxes” whose internal mechanisms remain mysterious, even to their creators.
• Current models still make mistakes and cannot replace traditional physics modeling, but show consistent improvement.

The Hollywood transformation: Runway’s newest product, Aleph, is reshaping acting by separating performance skills from physical appearance.

• The tool allows real-time editing of actors’ appearances using AI-generated special effects, functioning like a digital makeup and costume department.
• “One way of putting it is that you have a bit of unbundling of those skills. So if you’re amazing at the performance itself, you can just focus on that,” Germanidis noted.
• This technology could create opportunities for talented performers who previously lacked the “right look” for certain roles.

Current adoption: Major entertainment industry players are already integrating Runway’s technology into their workflows.

• Most top film studios and advertising agencies use Runway’s AI models for video production.
• The company contributed to synthetic video generation for films like “Everything Everywhere All at Once.”
• Recent models like Runway’s Gen-4 and Google’s Veo 3 produce videos nearly indistinguishable from reality, lasting longer than early 15-second clips.

Future potential: Runway sees significant room for improvement as the technology incorporates more diverse data sources.

• “We still haven’t trained models on the vast majority of the observations of the world, like the outcomes of scientific experiments and the sensor data we have available to us,” Germanidis said.
• The company believes these additions will create “more powerful simulators of the world.”
• Current capabilities haven’t reached a plateau, suggesting continued rapid advancement.