AI progress needs experts, not white-collar sweatshop workers, says study

The era of “sweatshop data”—where low-skill contractors performed basic labeling tasks for AI training—is ending as artificial intelligence models require more sophisticated training approaches. A new analysis from AI researchers at Mechanize Inc. argues that advancing beyond current AI capabilities will demand high-skill specialists, interactive software environments, and deep subject-matter expertise rather than traditional dataset creation methods.

The big picture: Current AI models have mastered basic tasks but struggle with complex, long-horizon challenges like managing large-scale software projects or autonomous debugging of intricate systems.

• Early AI systems benefited from simple, mass-produced datasets created by contractors paid “just a few dollars per hour” for monotonous labeling tasks.
• Today’s models need to learn sophisticated capabilities that require sustained, expert-level attention rather than quick, isolated tasks.

What needs to change: Three fundamental shifts are necessary to advance AI capabilities beyond their current limitations.

• Software over datasets: Interactive environments that offer ongoing challenges as models improve, similar to how games engage players across skill levels, rather than static datasets.
• Full-time specialists over contractors: Dedicated experts who can design comprehensive training environments that teach end-to-end job performance, including strategic thinking and long-horizon problem-solving.
• Deep expertise integration: Subject-matter experts must become central to AI development, as their “tacit knowledge, skills, and experience are now the bottleneck to further AI progress.”

Why reinforcement learning environments matter: The researchers argue that quality training environments, not just computational power, will determine future AI progress.

• They point to the contrast between AlphaGo Zero, which despite more compute than GPT-3 could only play Go, while GPT-3’s diverse language training enabled multiple capabilities.
• Current reinforcement learning with verifiable rewards (RLVR) methods can teach AIs to “prove theorems and solve hard puzzles” but fall short of handling “the open-ended nature of reality.”

In plain English: Think of it like this: current AI training is like teaching someone to be a chef using only multiple-choice tests about cooking techniques. But to actually run a restaurant, they need hands-on experience in a real kitchen with all its chaos, timing pressures, and unexpected problems. The researchers are saying AI needs more “kitchen experience” and less “textbook learning.”

The infrastructure challenge: Training AI for complex roles like infrastructure engineering requires comprehensive testing environments that go far beyond basic functionality.

• AIs must learn to build systems that are “highly available, fault-tolerant, and easily scalable” while preventing single points of failure and maintaining security practices.
• Current AI coding tools, “rewarded mainly for producing code that satisfies simple test cases, routinely fall short of these standards, creating headaches and frustration for anyone who tries to use them to build or maintain complex software.”

What they’re saying: The researchers emphasize the need to elevate data generation from a low-status activity to sophisticated engineering.

• “This will require reframing how we think about the data generation process: from a low-status activity outsourced to workers in poor countries, to an elaborate process requiring the world’s finest talent and clever engineering.”
• They warn that “many have observed that pretraining is already saturating” with GPT-4.5 not feeling “like a major generational leap in the way GPT-4 did over GPT-3.5.”