The Brains and Brawn of AI Models and How to Understand Their Output

Recent insights from a talk by Devavrat Shah shed light on conceptual frameworks for understanding and regulating artificial intelligence systems.

The mind and muscle of AI: Cognitive output, whether from humans or AI, can be viewed as a combination of learning capability (mind) and mechanistic automation (muscle).

• The ‘mind’ component represents the learning aspect, involving data interpretation and logical reasoning.
• The ‘muscle’ refers to the brute-force application of assessment to data, or what Shah terms ‘mechanistic automation’.
• This conceptual framework helps in distinguishing between AI systems that simply process large amounts of data and those that demonstrate more sophisticated learning and reasoning capabilities.

Evaluating AI outcomes: Shah introduced the concepts of “probability distribution” and “counterfactual distribution” as tools for assessing AI system results.

• These statistical concepts can be applied to analyze the range of possible outcomes produced by an AI system.
• By examining both the actual and potential outcomes, researchers and regulators can gain a more comprehensive understanding of an AI system’s performance and potential biases.

Regulatory challenges: The regulation of AI systems presents unique challenges due to their complex nature and rapid evolution.

• Regulations are defined as norms or principles that should be complied with, with enforcement involving the evaluation of whether a law is observed.
• The complexity of AI systems makes it difficult to establish a global definition of consistency, complicating the creation of universal regulatory frameworks.
• Shah argues that interdisciplinary collaboration is crucial for developing effective AI regulations and audit processes.

Information source evaluation: Shah presented two contrasting approaches to evaluating information sources in AI systems.

• The first approach restricts information to certain accepted sources, potentially limiting the scope of data but ensuring credibility.
• The second approach allows information from all sources but requires it to be similar to information from accepted sources, promoting inclusivity while maintaining a standard of quality.
• This dichotomy is reminiscent of the consensus-based systems used in blockchain technology, where verification relies on community observation rather than centralized authority.

AI audit as a business: The concept of AI auditing is emerging as a potential industry, with organizations like the Ikigai AI Ethics Council forming to address ethical concerns and best practices.

• The council recommends that AI companies define clear data strategies, test for causation in their models, and use simulations to experiment with potential outcomes.
• These practices aim to enhance transparency and accountability in AI development and deployment.

Sophisticated AI architectures: The talk touched on the evolution of AI architectures, from simple neural networks to more advanced systems.

• Advanced AI models, such as transformers and those using liquid neurons, represent a shift towards more sophisticated input functions and data interpretation capabilities.
• These advancements blur the line between simple data processing and more complex reasoning, further complicating the regulatory landscape.

Broader implications: As AI continues to permeate various industries and aspects of society, the need for robust frameworks to understand and regulate these technologies becomes increasingly critical.

• The concepts presented by Shah offer a starting point for developing more nuanced approaches to AI governance.
• However, the rapid pace of AI advancement means that these frameworks will need to be continuously updated and refined to remain relevant and effective.
• Balancing innovation with responsible development and deployment of AI systems will likely remain a key challenge for policymakers, industry leaders, and researchers in the coming years.