Systems, not models: How a holistic view of AI leads to better implementation and alignment

AI systems are increasingly being deployed in real-world applications, yet there remains a tendency to focus narrowly on AI models while overlooking the broader systems in which they operate.

The systems perspective: Understanding AI as complete systems rather than isolated models provides a more comprehensive framework for evaluating capabilities, safety, and regulatory approaches.

• AI systems comprise multiple interconnected components including the core predictive model, sampling strategies that convert outputs to text, prompting strategies that guide behavior, and optional tool integrations
• This holistic view better reflects how AI actually functions in real-world applications
• Capabilities often attributed solely to models, such as mathematical ability, are actually properties of the entire system working in concert

Technical implementation details: The effectiveness of an AI solution depends heavily on how its various components are integrated and optimized to work together.

• Sampling strategies determine how model probability distributions are converted into concrete outputs
• Prompting strategies shape both input formatting and guide the model’s behavioral patterns
• External tool integration can significantly expand a system’s capabilities beyond what the core model alone could achieve

Evaluation considerations: Assessment of AI capabilities and safety measures must account for the complete system rather than focusing exclusively on model performance.

• Current model evaluations are actually testing specific implementations of broader systems
• Safety evaluations need to consider potential risks and vulnerabilities at both the model and system levels
• Chain-of-thought analysis becomes particularly relevant for understanding advanced system behavior

Regulatory implications: While current regulatory frameworks primarily target models, a systems-based approach to oversight may prove more effective.

• System-level regulation could better address real-world implementation concerns
• Practical challenges exist in defining and enforcing system-level regulations
• A balanced approach considering both model and system-level factors may be necessary

Future developments and considerations: As AI technology continues to evolve, the distinction between models and systems becomes increasingly critical for successful deployment and governance.

• The effectiveness of safety measures may be enhanced by focusing on system-level controls
• Understanding interpretability at both model and system levels will be crucial for transparent AI development
• Organizations implementing AI solutions need to consider the full system architecture rather than just model selection