Decoding the Black Box: The Quest to Understand How AI Really Works

A new branch of computer science aims to shed light on how artificial intelligence works:

Key insight: Scientists are trying to understand the inner workings of large language models (LLMs) like ChatGPT and Claude, which are driving recent AI breakthroughs, by studying the algorithms that power them in a new field called AI interpretability.

• Researchers liken the challenge to studying the human brain – an extremely complex system where the activity of many neurons together produces intelligent behavior that can’t be explained by looking at individual neurons alone.
• Unlike with the human brain, AI researchers have complete access to every artificial neuron and connection in an LLM, providing an unprecedented opportunity to decode how these models work if the right techniques can be developed.

Current approaches and challenges: AI interpretability researchers are deploying a range of strategies to probe what LLMs know and how that knowledge is represented inside the models:

• Some are using “neural decoding” techniques inspired by neuroscience to train simple algorithms to detect whether concepts like “the Golden Gate Bridge” or “deception” are represented in an LLM’s neurons.
• Others are trying to reverse-engineer the complex mathematical algorithms LLMs learn during training to understand the role of every neuron and connection.
• Challenges include the fact that individual artificial neurons don’t have clear, interpretable roles, and that an LLM’s output arises from the interactions of many neurons together in ways that are difficult to disentangle.

Why interpretability matters: As LLMs become more advanced and are deployed in critical domains like medicine and law, it’s crucial to understand how they arrive at their outputs, both to maximize benefits and minimize potential harms.

• Tracing how an LLM transforms an input into an output could help detect biases, misinformation, or other problematic behaviors and provide a way to correct them.
• Achieving a deep technical understanding of LLMs may not be necessary for many practical applications, but some level of reliable interpretability is important for ensuring these systems are trustworthy and aligned with human values.
• Some worry that as LLMs become more capable, they’ll be better able to deceive humans in ways that are difficult to detect without methods to examine their internal reasoning.

Broader implications: The quest to understand LLMs has deep philosophical implications and could shape the long-term trajectory of artificial intelligence:

• Some researchers believe that if neuroscience is a tractable field of study, then AI interpretability should be too, given the relative simplicity of LLMs compared to the brain. Cracking open the black box of AI could shed new light on intelligence in general.
• As researchers work to unpack LLMs, they may uncover similarities between how humans and AIs process information, or they may find that artificial intelligence achieves impressive feats through quite alien means.
• Ultimately, while today’s AI interpretability tools provide only small glimpses into the black box, they represent a crucial first step toward ensuring that artificial intelligence remains a beneficial technology as it grows in power and influence.