AI achievements have historically been linked to chess — not so with today’s LLMs

Historical context: Chess has played a pivotal role in artificial intelligence development, starting with the first chess engines in the 1950s and culminating in IBM’s Deep Blue victory over world champion Garry Kasparov in 1997.

• Early chess computers could only compete with amateur players due to limited computing power
• Deep Blue’s victory marked a turning point in public perception of AI capabilities
• Traditional chess engines like Deep Blue and Stockfish rely on hard-coded rules and analysis of historical games

Technical distinctions: Modern AI systems like ChatGPT operate fundamentally differently from traditional chess engines, explaining their contrasting performance levels.

• Chess engines like Stockfish use symbolic AI with pre-programmed rules and game analysis
• ChatGPT employs a “connectionist” approach, learning adaptively from training data
• Despite having vastly more computing power than Stockfish, ChatGPT struggles with basic chess rules

Performance analysis: International Chess Master Levy Rozman’s demonstrations reveal striking limitations in ChatGPT-3’s chess abilities.

• ChatGPT-3 frequently makes illegal moves, such as having pawns jump over pieces
• The AI accumulates impossible numbers of pieces during games
• When playing against Stockfish, ChatGPT-3’s performance deteriorates into nonsensical moves

Recent developments: ChatGPT-4 shows marked improvement in chess capabilities, though still falls short of specialized chess engines.

• ChatGPT-4 demonstrated sophisticated chess strategies in games against Rozman
• The AI maintained competent play for over 30 moves before deteriorating
• Rozman noted being nearly defeated by ChatGPT-4, marking significant progress from earlier versions

Future implications: The evolution of chess capabilities in language models presents an interesting paradox in AI development.

• Traditional chess engines will likely maintain superiority in chess-specific tasks
• The improvement in ChatGPT’s chess abilities suggests potential for broader learning capabilities
• This progression highlights the ongoing tension between specialized and generalist AI systems