“Smart scaling” is poised to outpace data in driving AI progress

Artificial intelligence is entering a new phase where brute force scaling has reached its limits, according to prominent AI researcher Yejin Choi. Speaking at Princeton’s Laboratory for Artificial Intelligence Distinguished Lecture Series, Choi argues that algorithmic innovations will be crucial to continue advancing large language models as existing data scaling becomes unsustainable. This shift from “brute force scaling” to “smart scaling” represents a fundamental reorientation in AI development, potentially establishing a new paradigm where algorithmic creativity replaces massive datasets as the primary driver of progress.

The big picture: AI researcher Yejin Choi believes the era of scaling language models through massive datasets is ending, necessitating a shift toward algorithmic innovation.

• “It might be that the era of brute force scaling is over, and the era of smart scaling begins,” Choi stated during her Princeton lecture, emphasizing that researchers must now focus on what “computer science is all about, which is algorithms.”
• This perspective aligns with other prominent researchers who believe retraining language models with new datasets will soon cease to be effective.

Why this matters: The limited growth of internet-based training data creates a fundamental constraint for traditional AI development approaches.

• Choi noted that “We humans are not writing internet data fast enough for [large language models] to train more,” highlighting a critical bottleneck in the current development pipeline.
• This limitation forces researchers to find ways to “bend” scaling laws rather than simply applying more computational power to larger datasets.

Key details: Choi’s research explores alternative approaches to continue advancing AI capabilities despite data limitations.

• Her work focuses on methods for enhancing synthetic data, developing symbolic search algorithms for reasoning, implementing test-time training techniques, and creating new tokenization algorithms for better inference.
• Choi summarized her vision stating, “In the end, I think the scaling of intelligence will continue, but we as a community, I hope, will do it in a more exciting, smart way.”

Background: Choi brings significant expertise to these discussions as an accomplished AI researcher with an impressive academic pedigree.

• She was recently named among Time’s 100 Most Influential People in AI and received the prestigious MacArthur Foundation “genius grant” in 2022.
• Princeton AI Lab director Sanjeev Arora described Choi as “one of the brightest stars of AI and language models” during her introduction at the Distinguished Lecture Series.
• Choi is an incoming professor of computer science at Stanford University, following her work as a senior director at Nvidia and as a professor at the University of Washington.