The performance gap between open and closed AI models is shrinking

The AI landscape evolves: Open AI models are catching up to closed models in performance, with only a one-year lag, according to a new report by Epoch AI.

• Meta’s Llama 3.1 405B, released in July, took about 16 months to match the capabilities of GPT-4’s first version.
• The gap between open and closed models could shrink further if Meta releases its next-generation AI, Llama 4, as an open model.
• Researchers analyzed hundreds of notable models released since 2018, measuring performance on technical benchmarks and computing power used for training.

Implications for policymakers: The narrowing gap between open and closed AI models presents both opportunities and challenges for regulators and AI labs.

• The lag provides a window for assessing frontier capabilities before they become widely available in open models.
• Policymakers are grappling with how to deal with increasingly powerful AI systems that can reshape information environments and potentially cause harm.
• The distinction between ‘open’ and ‘closed’ AI models is not straightforward, with varying definitions and licensing agreements.

Benefits of open AI models: Making AI models open can democratize access to technology and drive innovation and competition.

• Open communities involve a wider, more diverse group in AI development.
• They drive innovation through collaboration, particularly in making technical processes more efficient.
• Open models enable greater transparency and accountability, allowing researchers to examine training data and address potential biases.

Risks and challenges: The accessibility of open models creates inherent risks and governance challenges.

• Malicious actors can use open models for harmful purposes, such as producing child sexual abuse material or developing military applications.
• Chinese research institutions have reportedly used Meta’s Llama model for military purposes, highlighting the inability to recall publicly released models.
• Governance of open models is challenging due to the lack of centralized control, particularly concerning extreme risks posed by future AI systems.

Closed model considerations: While more secure, closed proprietary models present their own set of challenges.

• Access is controlled by developers, but they are more opaque and difficult for third parties to inspect.
• Organizations processing sensitive data may avoid closed models due to privacy concerns.
• Despite stronger built-in guardrails, many people have found ways to ‘jailbreak’ closed models.

Governance and regulation: The safety of AI models is increasingly becoming a focus for both private companies and government institutions.

• The U.S. AI Safety Institute (AISI) is playing a growing role in safety-testing models before release.
• The shrinking gap between open and closed models may necessitate new approaches to regulating open model development.
• Measuring AI capabilities remains challenging due to the lack of standardized definitions and the varying strengths of different model types.

Future implications: As AI capabilities continue to advance, the complexity of governing this technology will increase.

• Even without further progress, it could take years to fully integrate existing AI systems into our world.
• New capabilities, such as Anthropic’s model’s ability to directly control a computer, are continuously being added.
• Experts emphasize the need to establish clear threat models and identify the best points for intervention in addressing potential harms.