Waymo is using Google’s Gemini to train its robotaxis

Waymo’s AI innovation in autonomous driving: Waymo, the Alphabet-owned autonomous vehicle company, is developing a new training model for its robotaxis built on Google’s multimodal large language model (MLLM) Gemini, signaling a potential breakthrough in the application of AI to self-driving technology.

• Waymo has introduced EMMA (End-to-End Multimodal Model for Autonomous Driving), a new end-to-end training model that processes sensor data to generate future trajectories for autonomous vehicles.
• This development represents one of the first indications that a leader in autonomous driving is exploring the use of MLLMs in its operations, potentially expanding the application of large language models beyond chatbots and image generators.

The technology behind EMMA: Waymo’s research paper proposes developing an autonomous driving system with the MLLM as a “first class citizen,” aiming to overcome limitations of traditional modular approaches in self-driving technology.

• Traditional autonomous driving systems use specific modules for functions like perception, mapping, prediction, and planning, which can face scaling issues due to accumulated errors and limited inter-module communication.
• MLLMs like Gemini offer potential solutions by providing rich “world knowledge” beyond typical driving logs and demonstrating superior reasoning capabilities through techniques like chain-of-thought reasoning.

Advantages of the EMMA model: Waymo’s new approach aims to enhance the capabilities of its robotaxis in navigating complex environments and handling unexpected situations.

• The company has identified several scenarios where EMMA helped driverless cars find optimal routes, including encounters with various animals or construction on the road.
• EMMA has shown excellence in trajectory prediction, object detection, and road graph understanding, suggesting potential for further integration of core autonomous driving tasks.

Comparison to competitors: Waymo’s pursuit of an end-to-end AI system for autonomous driving aligns with similar efforts by other companies in the industry, though with notable differences.

• Tesla, for instance, claims its latest Full Self-Driving system (version 12.5.5) uses “end-to-end neural nets” to translate camera images into driving decisions.
• Waymo’s approach, however, leverages its existing lead in deploying real driverless vehicles on the road, potentially giving it an edge in real-world application and testing.

Limitations and challenges: Despite the promising aspects of EMMA, Waymo acknowledges several limitations that require further research before practical implementation.

• The model currently cannot incorporate 3D sensor inputs from lidar or radar due to computational expense.
• EMMA can only process a small number of image frames at a time, limiting its real-time processing capabilities.
• The use of MLLMs in autonomous driving raises concerns about potential errors, given the known issues with hallucinations and simple task failures in language models like Gemini.

Future implications and research directions: Waymo’s research into EMMA opens up new avenues for advancing autonomous driving technology while also highlighting areas that require further investigation.

• The company emphasizes the need for more research to address current limitations and evolve the state of the art in autonomous driving model architectures.
• This development could potentially inspire other companies and researchers to explore similar approaches, potentially accelerating progress in the field of self-driving technology.

Balancing innovation and safety: As Waymo pushes the boundaries of AI in autonomous driving, the company must carefully navigate the challenges of implementing advanced models like EMMA while ensuring the safety and reliability of its vehicles.

• The high stakes of autonomous driving, where vehicles operate at speed in complex environments, necessitate a cautious approach to implementing new AI models.
• Waymo’s acknowledgment of the need for further research before deployment demonstrates a commitment to responsible innovation in this critical field.