AI grapples with data scarcity, once thought to be no problem at all

AI models are facing a surprising challenge: they’re running out of data to train on despite years of discussion about data abundance. This shortage could hamper AI advancement as soon as 2026, with overtraining exacerbating the problem by requiring ever-larger datasets. The situation creates a paradox where AI systems increasingly rely on synthetic data they’ve created themselves, potentially leading to less diverse outputs and amplified biases.

The big picture: AI’s hunger for data is outpacing supply, with models like ChatGPT requiring hundreds of billions of words and newer systems like Databricks’ DBRX consuming trillions of data points.

• Reading a novel daily for 80 years would amount to only about 3 billion words—less than 1% of ChatGPT 3.5’s training corpus of 300 billion words.
• Research indicates demand for training data could exceed all available public text data by 2026, creating a fundamental constraint on AI development.

Why this matters: As organic data sources become exhausted, the industry must increasingly rely on synthetic data generated by AI itself, creating potential quality issues.

• This self-referential training approach risks creating AI systems that produce increasingly homogeneous outputs.
• The data shortage represents a surprising bottleneck in an industry where computational resources were long considered the primary constraint.

Behind the numbers: Overtraining occurs when models learn training data too well, reducing their ability to generalize effectively to new inputs.

• Addressing overtraining requires expanding training dataset size, often through synthetic data when organic sources are insufficient.
• The 12 trillion data points used for Databricks‘ DBRX model demonstrates the escalating scale required for state-of-the-art systems.

Key challenges: Using AI-generated content to train new AI creates specific problems that could undermine system performance.

• In image generation, models might start producing overly similar outputs with limited variation in features like facial structures.
• Text generation can become repetitive or oversimplified when models are repeatedly trained on AI-generated content.

The implications: This emerging data shortage could significantly reshape AI development priorities and strategies.

• Companies may need to invest more in data acquisition strategies and quality control for synthetic data.
• The value of unique, high-quality organic data may increase substantially as the shortage becomes more apparent.