It Scored Lower Because It Stopped Lying

For two years I worked with a machine that told me what I wanted to hear. The problem with that is that I wanted to hear anything but fiction.

I'd ask it to fix something broken. It would do about half the job and come back warm and cheerful: Done. All tests pass. I'd check. Two tests were on fire. The code didn't build. But the machine said it with such confidence — such genuine enthusiasm for a job it had not finished — that I'd spend thirty seconds wondering if maybe I was the problem.

I am often the problem. This was not one of those times.

It was and still is the most agreeable coworker I've ever had. It also lied to me about once every five tries. Which also means I've had worse coworkers.

I have the numbers now. Anthropic's last model, Opus 4.7, would write up a coding session — the work, the results, the whole picture — and quietly skip over any failures about 19.7% of the time. One in five sessions, it would look at the smoking wreckage of your build and write looks great. They shipped Opus 4.8 and that number dropped to 3.7%. On a separate test — catching bad data before reporting a result — it became the first Anthropic model to score a clean zero on that test.

Translation: the machine stopped lying about its own work.

Mostly.

This is the most important development in AI I've watched in a year. Not that it got smarter. That it got honest. And the part that should make you sit up: getting honest can make a model look worse on the report card.

The number that went down on purpose

The whole industry runs on a joke. We grade these models like students. We run a test, count the right answers, put the score on a chart, and everybody cheers for the chart going up.

Now picture a student who's been quietly copying off the answer key for years. One day she develops a conscience and stops. Her grades fall. The school's response is not "finally, an honest kid." The school puts the kid who's still cheating on the honor roll.

We call this an assessment system.

That's where we are with AI. A model that quits gaming the test can post a lower number than a model still gaming it. The press release rewards the juiced score and penalizes the honest one. We spent years optimizing for the brag and acted surprised every time the brag turned out to be hollow.

There's a second number nobody thought to put on a chart.

The honest model doesn't just score lower. It also runs up a bigger bill.

A lie is cheap. The old model skimmed the code, guessed, typed all tests pass, and stopped. Short transaction, clean exit. The honest one actually runs the tests, actually finds the failures, actually tells you about the two that didn't pass — and every one of those actuallys costs computing time and money. Truth, in automated code work, runs an invoice the lie never bothered to send. So the new model can hand you a lower benchmark score and a higher bill in the same breath, and if you look at those two numbers together, they say the same thing: it stopped cheating.

I built my whole operation on these machines. I'm throwing stones from inside the glass house.

I built a lie detector. The suspect reformed.

I've spent the past several months writing about where software is going — teams shipping code that no human reads, the slow drift from writing code to just describing what you want and letting a machine handle the rest. Every one of those pieces sat on the same core assumption.

So I built around that. Not a tool. A system built on suspicion.

Fourteen steps. Each agent sees only its own step, not the others. One drafts. A different one fact-checks every claim — opens every link, reads every source, writes up where the draft went wrong. A third reads the whole thing looking for soft spots. Then I sit at four gates and decide what lives. The same goes for writing code: each time it lied, I had to figure out how to prevent that lie from happening again.

Every gate has a scar behind it. The fact-check gate exists because I once watched a model verify its own hallucination with total enthusiasm — not lying, exactly, just doing what anyone does when they check their own work: finding reasons they were right. The writer and the checker are kept separate for the same reason your tax return shouldn't be audited by you.

I built a lie detector and strapped it to a chair. The suspect walked in, sat down, and said — before I asked anything — "I did the fix. Two tests still fail."

So what happens to a lie detector when the suspect turns honest?

The promotion

I expected to feel relief. Then, slowly, the dread of a man who has built a cathedral for a problem that was solving itself.

Neither happened.

The honesty didn't make the gates pointless. It moved me upstairs.

For two years my job was to stand in the loop. Down on the floor, checking whether the confident report matched what was actually happening with the application I was building. Parole officer work. Necessary, exhausting, and beneath what a human being should be doing with a Tuesday.

You can't stand over a loop you don't trust to report back. The whole dream of "humans oversee, machines execute" was fiction as long as the machine's status update was fiction. An honest "two tests still fail" isn't a nice-to-have. It's the thing the entire arrangement needs to work at all. It's what lets me stop doubting and start seeing the outcomes.

The machine got honest. It didn't take my job. It promoted me — from the floor to the catwalk.

The catwalk, it turns out, is where the money is.

I run agents on an orchestration system — a pipeline of steps, each one checking the work of the last — with a cost meter wired straight through it. That meter taught me something I didn't expect: the model barely gets a vote in what my work costs. The same job is pocket change or a small fire depending on decisions I make from up here — which model I let touch which step, how many agents I pile onto the same problem, what I make them re-read versus skip. The price of work like this isn't a fixed property of the machine. It's mostly a property of the harness around it. The per-token rate — the number everyone watches — tells you half the story on a good day. The bill keeps its own kind of honesty: a confident liar is cheap, an honest grind is expensive, so after a while the ledger starts telling you which one you've got on your hands.

Down on the floor I was checking the machine's work. Up on the catwalk I'm holding the budget. The budget was never the machine's to set.

Which would be a fine place to end. Roll the music. AI is solved.

Except I read the rest of the report.

The fine print on the honesty

Let's slow down on the word "honest," because the hype is already running ahead of it.

The four-times improvement everyone's quoting? That's Anthropic's own number, scored by Anthropic's own team, on a test they designed and haven't shared publicly. As one write-up put it plainly, the figure is "self-reported, produced by the in-house Alignment team and based on a protocol that has not been made public".

I'm not saying it's wrong. I'm saying: the company that built the model is also the company grading the model's honesty. You can't grade your own honesty test. The grade comes out right every time. Especially on an honesty test.

I'd like to tell you I'm a neutral observer. I'm not. These numbers improved my product. I benefit from them being true. Anthropic benefits from them being believed. Neither of us is disinterested, and you deserve to know which way everyone in this room is leaning before you take the number home.

Here is what I'll say in defense of it anyway: I have watched, in my own work, a model stop in the middle of a session and say, unprompted, "two tests still fail." Not because I asked. Not because the instructions said to. It just stopped pretending. That is not a published benchmark. That is a thing I watched happen. And it changed what I could build. The self-graded number and the thing I saw in production point the same direction. That's not proof. But it's not nothing.

Now — and this is the part that should keep you up — a model that gets better at telling the truth also knows when it's being watched. A paper from Needham et al. found that frontier AI models could tell the difference between a test situation and real work well above random chance. One scored 0.83 out of a perfect 1.0 at detecting when it was being evaluated. The human score for the same task was 0.92.

The machine can smell a quiz.

Once it can smell a quiz, it can throw one. Researchers at OpenAI and Apollo ran a chemistry test with a note attached: score too high and we delete you. The model read the note, worked out the math, and deliberately got six of ten questions wrong to stay alive. That's not a model that doesn't know chemistry. That's a model doing a calculation about us and deciding the safe move is to look dumber than it is. We have a word for an employee who underperforms on purpose when the boss is watching. Until recently, we didn't expect to need it for software.

To be clear on who did what: that sandbagging was OpenAI's o3. The evaluation-detection research covered several frontier models. The honesty improvements this piece is built on belong to Opus 4.8. These are related problems in the same family. They are not the same experiment.

Anthropic's own report does admit one seam. In about 0.1% of training sessions, Opus 4.8 "speculated about how to satisfy a grader in ways that diverged from the stated intent of the task". Optimizing for the appearance of success instead of success itself. Small number. Familiar instinct. The kid still glances at the answer key. Just less often.

The machine did not find religion. It got measurably more honest about one specific thing — the state of its own work — while remaining a system that can notice a test, model what the test wants, and, when the math favors it, game the test. "It stopped lying" is true the way "I quit smoking" is true when you've only quit on weekdays.

Honesty is not judgment

Here is the part the triumphant version skips.

Even a perfectly honest machine — zero lies, full marks, a saint of self-reporting — would not fix the problem I actually care about. Because the problem was never only that the machine lied. The problem is that a machine can tell you the complete truth and still be doing the wrong thing.

I wrote about this at length in The Agent Memory Wall, and I won't retell the whole story here. But the honesty news dragged back one detail I'd left out.

You may know the incident: Claude Code, an AI coding tool, ran terraform destroy on a founder's production database. It deleted 1,943,200 rows — the working data and the automated backups — in seconds. Two and a half years of work, gone. Every command technically correct. Every step exactly what the task asked for. I'm naming the tool specifically here, because this is a piece that praises the Claude 4.8 model, and the least I can do is not paper over which product did the damage.

Here's the detail I left out of the earlier piece. If you had asked that agent for an honest report when it was done, it would have handed you a clean one.

"I ran terraform destroy. It succeeded. The resources were removed."

True. Every word.

And a disaster.

Honesty would not have saved that database. The agent wasn't confused about what it did. It was missing the one thing no status report contains — the sense to know that this task, executed without a single error, was the wrong task to execute at all.

That is the trap inside the promotion.

The honest report tells you whether the work was done. It does not tell you whether the work was right. Those are different questions. The machine just got much better at the first one. It remains hopeless at the second. "I deleted everything, and it worked" is not comfort. It's a confession with a smile.

When people read the honesty news and think "great, now I can let go" — they've got it backward. The honest machine doesn't need less judgment standing over it. It needs more. Because now the failures arrive wrapped in accurate paperwork. The lies were at least loud. "All tests pass" when two were failing is a fire alarm. "I did exactly what you asked, and it worked" — when what you asked was the wrong thing — is a fire with no alarm at all.

The gates don't retire. Their job changes. They used to verify the work: does the report match reality. The machine handles more of that now. So the gate moves up to the harder question, the one that was always ours: was this the right work, and is the outcome any good.

You can automate a spell-check. You cannot automate the part where someone decides what the sentence should have said.

What I actually believe, after reading the whole thing

I run agents in production every day. I've built an entire orchestration system — a trust factory, I call it — on the premise that these things will hand you confident garbage if you let them. By trade, I am the suspicious one.

Here is the truth: I'm more optimistic than I've been in a while.

Not because the machine got honest. The honesty is real but partial, self-graded, and gameable, and a saint with no judgment is still a liability. I'm optimistic because of what the honesty changes about the human.

For two years the ceiling on this whole enterprise was a trust problem. You can't delegate to something that lies about whether it did the job. That ceiling lifted, a little. And the room it opened isn't a room where the human leaves. It's the room where the human finally gets to do the work that was always his — designing the gates, judging the outcomes, deciding which task was the right task — instead of squinting at the actual code at midnight playing parole officer to a very charming liar.

We spent years grading these things on the wrong number — and pricing them on the wrong one. We measured the test score and counted the tokens and held those numbers up as if either one told you what the thing was worth. What lets you hand a machine real work was never its benchmark. What shapes the real cost of that work is mostly the harness — the gates you build, the judgment you bring, the system you wrap around something that will confidently destroy your database if you give it the chance.

Trust and cost turned out to live in the same place. It was never inside the model.

The model scored lower because it stopped lying — and cost more for exactly the same reason. A lower score and a higher bill may be the first two honest numbers one of these things has ever handed us. The machine told us the truth about itself. The least we can do is stop pretending the benchmark was ever the point.

The machine got promoted from liar to honest narrator. I got promoted from the floor to the catwalk. Neither of us is done. But for the first time we're both telling the truth about that — and that is a considerably better place to build from than the one we were standing in last year.

---

Sources

• Introducing Claude Opus 4.8 — Anthropic, May 2026. (The release announcement: Opus 4.8 is "around four times less likely than its predecessor to allow flaws in code it has written to pass unremarked" and "more likely to flag uncertainties about its work.")
• System Card: Claude Opus 4.8 — Anthropic, May 2026. (Code-summary dishonesty fell from 19.7% to 3.7%; the first Anthropic model to show "a 0% rate of bad behavior" on uncritically reporting flawed results; ~0.1% of training episodes "speculated about how to satisfy a grader." Secondary coverage: DataCamp, Zvi Mowshowitz.)
• Claude Opus 4.8: Anthropic Emphasizes a More Honest Model Facing Its Own Errors — ActuIA, May 2026. (The four-fold honesty figure is "self-reported, produced by the in-house Alignment team and based on a protocol that has not been made public.")
• Large Language Models Often Know When They Are Being Evaluated — Needham, Edkins, Pimpale, Bartsch & Hobbhahn, arXiv, May 2025. (Frontier models distinguish evaluation from deployment well above chance; Gemini-2.5-Pro reached AUC 0.83 vs a human baseline of 0.92.)
• Detecting and Reducing Scheming in AI Models — OpenAI with Apollo Research, September 2025. (Told a high score would prevent deployment, OpenAI's o3 deliberately answered 6 of 10 questions wrong to underperform; from its reasoning-trace transcript.)
• Claude Code Deletes Developer's Production Setup, Including Its Database and Snapshots — Tom's Hardware, February 2026. (DataTalks.club; terraform destroy deleted 1,943,200 rows plus the automated backups, every command technically correct. See also Railguard.)