Garbage In, Gospel Out

THE NUMBER: 123. That is how many children were killed in February when two Tomahawk missiles hit the Shajarah Tayyebeh Elementary School in Minab, Iran, on the opening day of the Iran war. More than 150 people died in all. A Bloomberg investigation this week, built on an unreleased internal Pentagon review, describes a cascade of preventable failures — bad intelligence, stale satellite imagery, and an overreliance on a Palantir AI targeting tool that took work which used to take hours and “condensed it into minutes.” The team inside Central Command whose entire job was to catch that kind of error had been cut from ten people to one. Hold that number, because everyone this week is going to argue about a superintelligence that might hurt us someday, and the number already has a body count. It didn’t come from a genius machine. It came from a dumb one nobody had time to check.

The room where it almost happened

Start with the ship, because the ship is the whole issue in miniature, and unlike the school, everybody in the room got lucky.

Earlier this year, in the middle of the Iran war, an analyst at Special Operations Command Pacific did something that by January of this year the Pentagon was actively encouraging three million people to do: he asked an AI to help. He fed a chatbot intelligence about a Chinese cargo ship’s manifest. The system blended open-source information with classified signals intelligence and returned a confident conclusion — the vessel was carrying components for a nuclear weapons program, bound for Iran. The analyst then used AI a second time, to turn that conclusion into a formal intelligence report. Two AI passes. Zero verification in between.

The report moved through the system the way a true thing moves, because nothing about it looked false. It set off alarm bells across the U.S. military. Plans were drawn to intercept the vessel. According to the four sources who described the episode to CNN, armed members of the U.S. military were preparing to board a Chinese ship, and military planes were already in the air. Any operation against a Chinese-flagged vessel carried the live risk of spiraling into an armed conflict between the two largest militaries on earth.

Then someone checked the source. The report, in the flat words of one of those sources, was “entirely false.” It also, in the words of another, “almost started a war.”

I want you to sit in that room for a second, because the lesson is not the one the headlines will reach for. Nobody in that chain was fooled by a brilliant machine. They were rushed by a confident idiot. The most important sentence in the entire CNN account is a throwaway from one of the sources, eight words that belong on a placard in every office deploying this technology: “AI allows you to get to a bad idea faster.” Speed was supposed to be the benefit. Instead it was the thing that removed the last checkpoint where a human might have paused and asked whether the machine actually got it right.

The wrong noun

For two years, the entire public argument about AI danger has been about the wrong word.

The fashionable fear is superintelligence — the model wakes up, out-thinks every human alive, develops goals of its own, and does something we can’t take back. It is a genuine argument and I’m not here to mock it. Just last weekend Dario Amodei was still making it, publishing an essay begging the industry to pace the frontier before a self-improving intelligence gets away from us. Within the hour, Altman agreed. Musk backed it. The whole cast hit its marks.

And while everyone stood in that room debating a machine too smart to control, the actual catastrophe walked in wearing the opposite costume. Not a genius. An idiot with a clean shirt and no doubt in its voice, moving faster than any human process built to catch it.

Grant Janich put his finger on it this week, in an essay called “AI Is a Time Machine,” and he got there from a different door than we did, which is exactly why it’s worth citing. His line: “the fashionable word for the fear is superintelligence, and the trouble with the word is that it has the wrong noun in it.” His frame is John Boyd, the fighter pilot who spent twenty years working out why the American F-86 beat the technically superior Soviet MiG-15 over Korea. The Sabre didn’t win because it was the better plane — it wasn’t. It won because its pilot could see out of a bubble canopy and change the aircraft’s state faster than the other man could answer, until the other man was disoriented, then passive, then dead. Boyd called it getting inside the enemy’s decision loop, and his uncomfortable point was that the loser was never out-thought. He was out-paced, and from the cockpit it feels identical.

That is what we just did. We handed OODA-loop speed — observe, orient, decide, act, in seconds — to a system that cannot tell a real threat from a hallucinated one, and pointed it at institutions that were built, deliberately and for good reasons, to decide in weeks and quarters. The board meets monthly. The court grants its injunction in a hurry in a matter of weeks. The market opens at half past nine. Every one of them is the MiG-15: superior on paper, hauling on the stick, and utterly unable to make the question of whether the thing on the other side is a genius bear on the outcome.

Nobody read the source at the school, either

The ship got lucky. The school did not.

Back in February, on day one of the war, the U.S. hit what its databases had catalogued for years as an Islamic Revolutionary Guard Corps facility. It was an elementary school. Commercial satellite imagery showed what the classified databases didn’t: walls, separate entrances, a soccer pitch, playground markings, assembly rows — a school, plainly, by 2018. One analyst had actually spotted the change and logged remarks about it as early as 2019. Those notes were entered into a system that wasn’t connected to the primary military intelligence database used for targeting, so they never reached the people building the strike package. The school had a website. It was on Google Maps.

None of that mattered, because the machine was fast and the humans were faster to trust it. The Minab site was fed into Palantir’s Maven Smart System alongside other candidates and came out as a recommended day-one target. Target-list work that once took hours was condensed into minutes. The Trump administration had demanded an overwhelming opening assault — more than a thousand Iranian targets hit in the first twenty-four hours — and that pace compressed the time available for exactly the confirmations that would have caught it. And the human backstop that might have caught it anyway wasn’t there: staffing on the Defense Department’s civilian-harm mitigation teams had fallen by roughly 90% over recent years, to fewer than twenty people total. Centcom’s group had shrunk from ten people to one. No one on those teams reviewed the Minab site before the missiles were in the air.

The United Nations’ Independent International Fact-Finding Mission on Iran reached its conclusion this week: reasonable grounds to believe the United States committed the war crime of launching an indiscriminate attack, that it “failed in its obligation to do everything feasible to verify” the target, and that the failure “went beyond mere negligence.” Palantir’s position, for the record, is that it is “not responsible for the underlying data” — the government owns the quality of what gets fed in — and that after the strike it added features to re-review intelligence and flag inconsistencies. Which is its own tell: the fix they shipped afterward is a verification layer. They bolted the checker on after the school.

Every decision looks the same to the machine

Here is the part I can’t stop turning over, and it’s the reason this is a Signal/Noise and not a news bulletin.

To the model, the support ticket and the kill order are the same problem.

Think about what a frontier model actually does. It predicts the next token. It does that with the same architecture, the same confidence, the same half-second latency, whether you’re asking it to summarize a memo or to tell you what’s in the hold of a Chinese ship in a war zone. It has no organ that registers consequence. The dial that says this decision is a rounding error and that one ends lives does not exist inside the machine. It answers the reversible question and the irreversible one in the same voice, and the voice is always sure.

That gradient — the weighting of a decision by how much it costs to be wrong — has always lived somewhere. It lived in us. And we had encoded it, without ever writing it down, in the single thing we spent the last three years congratulating ourselves for eliminating: friction. The meeting that made you wait. The analyst who had to read the source. The review cycle that ate a week. We called all of it inefficiency and we killed it wherever we could. But the slowness wasn’t only cost. The slowness was the mechanism by which a human system spent more deliberation on the decisions that could kill you and less on the ones that couldn’t. Speed, applied uniformly, doesn’t just make good decisions faster. It strips the risk-weighting out of the whole apparatus and treats the war and the rounding error exactly alike — which is precisely how the machine already sees them.

Why the ruin case breaks the spreadsheet

And this is where my finance brain takes over, because there’s a principle here that the whole industry is running straight past, and it has been priced correctly for a very long time by people who managed real risk.

Almost every decision we make, we make on something like expected value. Weigh the outcomes, multiply by their odds, take the bet if the average is positive. It’s the right tool, and it works — right up until one of the possible outcomes is ruin: irreversible, absorbing, game-ending. Nassim Taleb’s cleanest illustration is Russian roulette with an obscene payout. The expected value is spectacular. It is also the stupidest bet on earth, because you only get to collect the average if you survive the one chamber that ends the game, and the branch where you don’t erases every branch where you do. Expected value quietly assumes you get to keep playing. Ruin is the state where you don’t.

The math underneath it is ergodicity, which Ole Peters has spent a career on. A gamble is non-ergodic when the average across many players at one instant diverges from the trajectory of one player over time. The casino, holding thousands of bets at once, lives in the comfortable ensemble average. You, playing your one life in sequence, live in the time average — and if your path can hit an absorbing barrier, the ensemble’s rosy average is a lie told about people who aren’t you. Buffett compressed the whole thing into Rule No. 1: never lose money. Munger inverted it: all I want to know is where I’m going to die, so I’ll never go there. When a decision carries a real chance of ruin, you stop maximizing return and start minimizing the chance of catastrophe, and you rationally pay a premium that looks insane on a spreadsheet, because the spreadsheet is averaging over a game you might not survive to finish.

Now marry that to a machine that optimizes expected value at the speed of light and cannot see the absorbing barrier. That’s not a productivity tool. In the wrong room, that’s a positive-EV Russian roulette dealer who never sleeps.

The one-way door is the whole ballgame

So the imperative isn’t “use less AI,” and it isn’t the vague “human in the loop” everyone nods at. It’s specific, and Jeff Bezos handed executives the vocabulary for it years ago: Type 1 and Type 2 decisions. Two-way doors and one-way doors. A two-way door you can walk back through if you’re wrong — the draft, the summary, the first-pass code, the reversible experiment. A one-way door you can’t — the money that leaves, the thing that ships to the customer, the missile that leaves the rail.

Let the AI rip through the two-way doors. That’s where its speed is pure upside and a mistake costs you an undo. Then put a human, hard, on the one-way doors, and make the verification proportional to the consequence — not a uniform checkbox, a real gate that gets heavier as the door gets more permanent. The catastrophe at Minab and on that ship wasn’t “the military used AI.” It was that they let a machine walk the institution through a one-way door at token speed, with the one person whose job was to check the hinges cut from a team of ten down to one.

And lest this sound like a defense-only problem, look at what the same failure costs in the boring commercial version, because the numbers are already in. McKinsey found that 30% of companies deploying agentic AI got less productive, not more. Gartner says 40% of agentic projects will be dead by the end of 2027. The cause, over and over, is not a weak model — it’s that nobody defined what “right” looked like, so the machine optimized nothing and shipped confident garbage fast. Glean clocked AI saving workers eleven hours a week, and six and a half of them going straight back into checking the output. The savings are real. The company that forgot to budget the checking is the one running over its head. Same disease, different stakes. In the enterprise it’s a canceled project and a blown quarter. In a war zone it’s a school.

Watch the hands, not the mouth

One more thing, because the timing this week is almost too on-the-nose. While the near-war story broke and the Minab review leaked, the industry kept doing the thing it does — arguing loudly about the smart machine while quietly shipping the fast, dumb one.

Anthropic published its R&D Automation Index and told the world that Claude now “leads” 26% of its own AI research, with a careful, load-bearing footnote that 0% of it runs fully autonomously. Read that as the transparency it partly is, and also as an IPO-era instrument: put a number on your own self-improvement, on your schedule, before a regulator does it for you — Anthropic is reportedly steering toward a November listing on a $100 billion revenue pace. Meanwhile the “slowdown” that three rivals announced with such feeling two weekends ago is now the subject of an antitrust suit, which is exactly the ugly old name we gave it when David Sacks called it a cartel on the record. And in the middle of all of it sits the one detail that should reframe the whole safety debate: Anthropic has been in a standoff with the Pentagon, refusing to strip the guardrails that keep its models out of autonomous-weapons and mass-surveillance use, on the argument that the models are not reliable enough for those jobs. The lab making the loudest noise about the superintelligence of the future is the one telling the military its intelligence of the present can’t be trusted to pull a trigger. That’s not a contradiction. That’s the whole thesis, said out loud by the people with the most to lose from saying it.

We have seen this movie before, and it isn’t the Terminator. It’s every platform shift where we mistook the removal of friction for pure progress and found out, a few casualties later, that some of the friction was load-bearing. The slowness we’re deleting was doing a job. It was reading the source. It was the second set of eyes on the one-way door.

The machine didn’t get too smart. It got too fast — and it can’t tell a two-way door from a one-way one. That’s your job now. Build the gate where ruin lives, or you’re just getting to a bad idea faster.

---

• CNN (near-war Chinese-vessel report; Warner/Reed/Coons letter to Hegseth and DNI Clayton), Sept 19, 2026
• Bloomberg / Gizmodo (Pentagon review of the Minab strike and Palantir’s Maven), Sept 19–20, 2026
• Security Affairs, Sept 20, 2026
• UN Independent International Fact-Finding Mission on Iran (war-crime finding)
• thatstartupguy, Grant Janich, “AI Is a Time Machine,” Sept 18, 2026
• Reuters / shattered.io (Anthropic R&D Automation Index), Sept 17, 2026
• McKinsey (agentic-AI productivity) and Gartner (40% by 2027)
• Glean Work AI Institute
• Reuters/Bloomberg (AI-slowdown antitrust suit; Anthropic November IPO)
• Nassim Taleb, Incerto and “The Precautionary Principle”; Ole Peters on ergodicity; Jeff Bezos on one-way vs. two-way doors.