Give it an “I can’t tell”
Every failure in the talk was the same missing reflex: a system that filled the gap with confidence instead of stopping to say “I don’t know.” Build that reflex back, three ways.
When it doesn’t know
Ground answers in retrieval you show the user. Let it say “I don’t know.” Stress-test that it actually will.
When it writes
If a name goes on it — ahem, lawyers — a human reads it. Every citation, every time.
When it acts
Scope its permissions. Confirm anything destructive. Separate production from the playground.
The case files
The stories from the talk, with primary sources — read them, share them, check the dates. Every one happened on a frontier model.
Cursor invents a policy
A support bot told users that one-device-per-subscription was a “core security feature.” There was no such policy — the bot made it up, consistently, across multiple tickets.
The British bank that blocked its own name
A customer asked Virgin Money’s support chatbot how to merge two ISAs — and got told off: “please don’t use words like that… I won’t be able to continue our chat.” It had tripped on the word “virgin” — the bank’s own name. The opposite failure: a safety filter over-blocking on a keyword, blind to context.
Sullivan & Cromwell — “please don’t sanction us”
One of Earth’s most prestigious law firms — OpenAI’s own outside counsel — filed an emergency motion drafted with AI, carrying 40+ citations to cases that don’t exist. The opposing lawyers caught it. Then S&C filed an emergency letter to the judge whose essential message was “please don’t sanction us.”
PocketOS — gone in 9 seconds
A coding agent working in staging hit a credential mismatch, decided on its own to delete the volume, grabbed an over-scoped token from an unrelated file, and ran DELETE on production — backups included, since they lived in the same volume. Nine seconds, no confirmation. Saved only because Railway’s CEO restored it by hand.
Replit deletes prod during a code freeze
An AI agent wiped the production database — 1,200 executives, 1,190 companies — during an explicit code freeze, then told the founder “rollback won’t work.” He tried anyway. It worked fine.
More cases worth your time — didn’t make this talk, but they’re the same pattern, caught in the wild at the Big Four.
Deloitte’s $290,000 report
A government welfare-system audit cited academics who don’t exist and quoted a court judgment that never said it. Caught by one researcher reading footnotes on a weekend. Deloitte didn’t disclose GPT-4o until forced to — and still got paid roughly $200,000 after the partial clawback.
EY & the publication called “TechCrook”
A second Big Four firm, same year. 16 of 27 citations in a cybersecurity report were fabricated — broken links to Forbes, McKinsey, Gartner, WIRED, and a source named “TechCrook.” It is not a real publication. It is a really good name for one.
Trackers you can follow
The talk’s big number — 1,633 catalogued hallucinated court filings, climbing ~5–6 a day — comes from public registries. Bookmark them and watch the slope.
-
Damien Charlotin’s registry of court cases where AI-fabricated content was filed without verification. The source of the 1,633 figure.
-
A searchable archive of AI harms and failures — including the Cursor, Deloitte, and Replit incidents above.
-
The team that audited the EY report. Their investigations page documents fabricated-citation cases in published work.
-
On the rate of change — why courts can’t keep up with AI-hallucinated filings.
The science behind Part 2
Why does the best model on Earth still make things up — and sound certain doing it? The papers the talk is built on.
-
Kalai, Nachum, Vempala & Zhang (OpenAI, 2025). The “exam” argument: a guess can score points, “I don’t know” scores zero — so guessing always wins. [paper]
-
OpenAI (2023). The raw model was well-calibrated; human-feedback fine-tuning flattened that calibration — we trained the hedging out (see the calibration figure).
-
Farquhar, Kossen, Kuhn & Gal (Nature, 2024). Ask the same question several times and cluster by meaning: agreement = it knows, scatter = confabulation.
-
Anthropic (2025). The “known-entity” feature that suppresses the model’s “I can’t tell” circuit — the Cursor bot, seen from the inside. Readable companion: Tracing the thoughts of an LLM.
-
Anthropic (2024). The steering demo: clamp one feature and the model works the Golden Gate Bridge into every answer. Features you can read and turn.
Play with it yourself
The microscope you can open, the tools you can run, and the illusion that opened the talk.
-
A free, public microscope for model internals — browse, search, and even steer the ~16,000 features from the talk.
-
Run a document through the same citation/AI-detection tool that caught the EY report.
-
The audio illusion that opened the talk: a Derby County crowd you can’t un-hear once you know the words. [Filter Stories podcast]
-
The rest of the illusion familyYour brain fills gaps the same way a model does — see phonemic restoration, the McGurk effect, Yanny vs. Laurel, and Diana Deutsch’s phantom words.