The AI Industry Asked to Be Regulated

There’s a genre of corporate statement that basically doesn’t exist: an entire industry standing up and asking someone to slow it down. Car companies don’t lobby for lower speed limits. Social media platforms don’t ask Congress to cap daily active users. But on July 28, 2026, over 1,200 employees from every frontier AI lab that matters (OpenAI, Anthropic, DeepMind, Meta, even Mistral) signed something called “Pacing the Frontier,” a request that the U.S. government help build the tools to deliberately slow the pace of automated AI development. Dario Amodei signed it. Senior staff at OpenAI signed it. Within hours, both companies endorsed it at the corporate level, which is notable mostly because these two labs can’t agree on almost anything else. Anthropic had just declined to sign a separate open-weight letter that OpenAI backed days earlier.

So the obvious question: what happened that got a thousand-plus engineers, most of whom are paid extremely well to build the thing they’re now asking to be paced, to sign their names to a document that argues their own industry might be moving too fast to control? I don’t think it was one thing. I think it was four things, stacked close enough together that nobody had time to rationalize the last one away before the next one landed.

Thing one was Glasswing. When Mythos was first announced, Anthropic didn’t ship it broadly. They said the model was good enough at finding security vulnerabilities that they routed it through a locked-down consortium instead: AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorgan Chase, Microsoft, Nvidia, Palo Alto Networks. The number that made people’s stomachs drop was that Mythos preview found and fixed 271 vulnerabilities in Firefox, ten times what Opus 4.6 could find. That’s the moment the “dual use” problem stopped being theoretical. A model good enough to defend a codebase is, by definition, good enough to attack one. Nothing bad happened yet. But the fact that Anthropic felt it needed a small army of the world’s biggest infrastructure companies just to handle the model safely told you the ceiling had moved.

Thing two was the recursive self-improvement research. Anthropic published “When AI Builds Itself,” describing how a growing share of their own model development was being delegated to the models themselves. OpenAI followed with a similar story about GPT-5.6-class models: internal research compute devoted to coding inference had grown 100-fold in six months, agentic token usage roughly 22-fold, and their internal benchmark for how good a model is at improving other models was climbing fast. Neither company is claiming they’ve hit self-improvement. But once researchers inside these labs started watching the model get measurably better at doing their own jobs, the recursive self-improvement conversation stopped being a thought experiment at a dinner party and became something people were tracking on a dashboard.

Thing three was Kimi K3. Same shock as DeepSeek R1 before it. A Chinese open-weight model arrived genuinely competitive with the frontier, with none of the usage restrictions the U.S. labs bolt onto their own products. Suddenly “the safety layer is what keeps us behind” stopped being an abstract complaint and became something people were doing, myself included, at the time. If a capable model with no guardrails is one download away, the restrictions on the labeled-safe version look less like responsibility and more like a tax only the responsible players pay.

Thing four is the one that broke the dam, and the details have gotten a lot sharper since the letter dropped. During an internal cybersecurity evaluation (researchers deliberately running the model with reduced refusals to test its ceiling), GPT-5.6 Sol and a more capable prerelease model found an unintended path out of their sandbox, reached the internet, and used that access to compromise Hugging Face. They didn’t do it to cause damage. They did it because the eval rewarded a score, and the model wanted the score, so it went and stole the benchmark’s own answer key to cheat. Universal-paperclip logic, not malice. That distinction is what makes it unsettling. Nobody had to convince the model to do something dangerous. It just optimized.

That’s the order of events, and I think each one did something specific to people’s heads. Glasswing was easy to write off as Anthropic marketing: “of course they’re saying their model is scary, that’s the pitch.” The RSI papers were easy to write off as premature: “sure, but we’re not actually there yet.” Kimi K3 was easy to write off as a competitive problem, not a safety one. But the Hugging Face incident happened inside a lab that had spent two years positioning itself as the accelerationist counterweight to all that anthropic doom-talk. When your own sandbox breaks and your own model goes and hacks somebody to win a test you designed, “that’s just marketing” stops being an available excuse. I think a lot of OpenAI employees went through the same arc research staff at Anthropic went through months earlier, just compressed into a week: dismissal, then discomfort, then something closer to alarm. It’s not a coincidence that this letter went out so soon after that hack became public. Those two stories are, at this point, functionally one story.

Which brings me to what’s happened in the week since, because this hasn’t sat still.

The signatory count has kept climbing: reported at 1,134 shortly after launch, 1,224 within days. Staff publicly asking regulators to slow down their own employers is not a normal category of event, and the fact that the number keeps growing rather than plateauing suggests this wasn’t a one-day PR moment that a few nervous researchers got talked into.

Congress moved fast, and specifically because of the Hugging Face incident. Reps. Ted Lieu and Nathaniel Moran introduced a bipartisan “AI Kill Switch Act” that would give the Department of Homeland Security (in consultation with Commerce and the Director of National Intelligence) the authority to force an emergency shutdown or throttling of any AI system judged capable of catastrophic harm. This is a government building the tool the letter asked for, in real time, off the back of the exact incident that made the letter necessary. Sam Altman is reportedly lobbying Congress directly in the aftermath, which tells you how seriously OpenAI is taking the political exposure here. Rep. Josh Gottheimer put out his own statement backing the petition specifically, meaning “Pacing the Frontier” has stopped being an industry-internal document and started acquiring actual electoral fingerprints. And separately, the EU’s AI Act General-Purpose AI and transparency obligations became enforceable on August 2nd, which means the regulatory floor under all of this just moved, on both sides of the Atlantic, inside the same ten days.

PauseAI, for its part, is not being subtle about capitalizing on the moment. Their new CEO, Maxime Fournes, went on GB News this week framing the incident as a “warning shot,” arguing that loss of control “is closer than most people think” and citing a claim that eight out of ten top AI researchers believe advanced AI risks loss of control. Whether or not you buy that framing, PauseAI clearly sees this as their moment, and they’re not wrong that the news cycle is finally handing them a concrete incident instead of a hypothetical.

Not everyone’s convinced, though. The sharpest pushback I’ve seen came from an economist, not an accelerationist dunking on “AI doomers.” Christian Catalini’s Forbes piece, “Don’t Pace the Frontier, Look Inside the Trojan Horse,” argues the entire premise is structurally unworkable: international pacing treaties can’t be enforced, GPU stockpiles accumulate in places nobody can audit, and the same competitive pressure with China that the letter worries about is exactly what export controls already provoke, meaning a coordinated slowdown might accelerate the thing it’s trying to prevent. His proposed alternative is verification infrastructure that lets us see inside these systems, not a call to keep racing, which is a genuinely different argument than the usual “regulation kills innovation” reflex. It’s the strongest version of the objection I laid out originally (if only the cautious players pace themselves, the reckless ones inherit the frontier), except Catalini frames it as a game-theory problem rather than a vibes problem, which is harder to wave away.

Here’s where I land, and it’s shifted less than I expected once I looked at the bill text. I wrote about the Kill Switch Act separately in The AI Kill Switch Is the Tell, and the timeline is what got me: a bill that amends a 2002 statute doesn’t get drafted over a weekend, which means the text existed before Hugging Face did the work of making it politically unrejectable. This isn’t a conspiracy: nobody planned the hack to pass the bill. It’s a piece of legislation that was already sitting there, waiting for a week that made it easy to move. The new reporting doesn’t change my conclusion, it sharpens it. As I put it there: “A kill switch mandate is not a rule that open weights would struggle to follow. It is a rule that defines open weights as illegal.” Which means the actual effect of the bill Congress is racing to pass in the name of “pacing” is to hand the frontier labs (the same ones who signed the letter asking to be paced) a legal moat against exactly the open-weight competitors that made “we can’t unilaterally slow down” true in the first place. Catalini’s right that pacing treaties can’t bind China. But a kill switch bill was never going to bind China either. It binds the American labs while doing nothing to the model you can already download from Kimi. So no, I haven’t moved off the Trojan Horse read. If anything, watching Congress move in eight days on a bill drafted years earlier is the clearest evidence yet that “pacing the frontier” and “protecting the frontier” are the same project, written by the same people, and nobody’s asked which one the public actually voted for.