AI still can’t explain its own output — we need more humans who can

AI’s knowledge conundrum: The limitations of large language models: Large language models (LLMs) like ChatGPT and Gemini are increasingly relied upon by millions for information on various topics, but their outputs lack true justification and reasoning, raising concerns about their reliability as knowledge sources.

• Over 500 million people monthly use AI systems like Gemini and ChatGPT for information on diverse subjects, from cooking to homework.
• OpenAI CEO Sam Altman has claimed that AI systems can explain their reasoning, allowing users to judge the validity of their outputs.
• However, experts argue that LLMs are not designed to reason or provide genuine justification for their responses.

The nature of knowledge and AI’s shortcomings: True knowledge requires justification, which LLMs are fundamentally incapable of providing due to their design and functioning.

• Knowledge is typically associated with well-supported beliefs backed by evidence, arguments, or trusted authorities.
• LLMs are trained to detect and extend patterns in language, not to reason or justify their outputs.
• The responses generated by AI systems mimic knowledgeable human responses but lack the underlying reasoning process.

AI outputs as “Gettier cases”: The information produced by LLMs can be likened to philosophical “Gettier cases,” where true beliefs are combined with a lack of proper justification.

• Gettier cases, named after philosopher Edmund Gettier, illustrate situations where true beliefs are held without genuine knowledge.
• AI-generated content, even when factually accurate, falls into this category because the underlying process doesn’t consider truth or justification.
• The outputs can be compared to a mirage that accidentally leads to a real discovery, as in the example from 8th-century philosopher Dharmottara.

The deception of AI justifications: When asked to explain their reasoning, AI systems produce convincing but ultimately false justifications, further complicating the issue of trust.

• AI-generated justifications are merely language patterns mimicking real explanations, not genuine reasoning.
• As AI systems improve, their false justifications may become more convincing, leading to two potential outcomes:
  1. Those aware of AI’s limitations will lose trust in the system’s credibility.
  2. Those unaware may be deceived, unable to distinguish fact from fiction.

Appropriate use of AI tools: Understanding the limitations of LLMs is crucial for their effective and responsible use across various fields.

• Experts in fields like programming and academia use AI-generated content as a starting point, applying their own knowledge to verify and refine the outputs.
• However, many people turn to AI for information in areas where they lack expertise, potentially leading to misinformation.

Broader implications and concerns: The widespread use of AI as an information source raises important questions about trust, knowledge acquisition, and the potential for misinformation.

• The inability of LLMs to provide true justification for their outputs is particularly concerning when they are used for crucial information like medical or financial advice.
• Users must be aware of the potential for “swallowing” misinformation from AI sources without proper verification.

Critical analysis: The need for AI literacy: As AI continues to play a significant role in information dissemination, developing AI literacy and critical thinking skills becomes increasingly important for society.

• Users must learn to approach AI-generated content with skepticism and seek additional verification for important information.
• The development of AI systems that can provide genuine justification for their outputs may be necessary for them to become truly reliable knowledge sources.
• In the meantime, fostering a better understanding of AI’s limitations and proper use is crucial for navigating the evolving landscape of artificial intelligence and information.