AI researchers at OpenAI have developed a new algorithm that helps large language models (LLMs) like GPT-4 better explain their reasoning, addressing the critical issue of AI trustworthiness and legibility.
The Prover-Verifier Game: A novel approach to improving AI explainability; The algorithm is based on the “Prover-Verifier Game,” which pairs two AI models together:
- A more powerful and intelligent “prover” model aims to convince the verifier of a certain answer, regardless of its correctness.
- A less powerful “verifier” model, unaware if the prover is being helpful or deceptive, attempts to select the correct answer based on its own training.
OpenAI’s implementation: Iterative training for enhanced legibility; OpenAI researchers used custom fine-tuned GPT-4 models in the Prover-Verifier Game, with several rounds of training:
- The prover model alternated between being “helpful” (trying to deliver the right answer) and “sneaky” (trying to convince the verifier of its view, regardless of correctness).
- Both models were retrained between each round using data from previous rounds, improving their ability to determine answer correctness and provide legible explanations.
- Human evaluators rated the prover model’s answers for understandability and legibility.
Key findings: Improved resistance to persuasion and enhanced explainability; After multiple rounds of training, the researchers observed:
- The verifier model became better at resisting the persuasion techniques of the sneaky prover model.
- The prover model became better at explaining its reasoning to human users in a legible manner.
- Solutions that were harder for smaller verifiers to detect were also harder for humans to detect, while solutions more legible to smaller verifiers were also more legible to humans.
Broader implications: Trustworthiness and safety in real-world AI applications; OpenAI’s research aims to develop AI systems that provide not only correct but also transparently verifiable outputs, enhancing trust and safety in real-world applications:
- The algorithm has the potential to align future models that surpass human intelligence, ensuring their outputs remain legible and trustworthy.
- As AI becomes more integrated into critical fields like healthcare, law, and defense, establishing trust in AI systems is crucial for their widespread adoption and safe implementation.
- By sharing their findings with the broader AI community, OpenAI hopes to encourage further research and contributions to solving the legibility problem in AI.
OpenAI used a game to help AI models explain themselves better
