Skip to main content
Back to Blog
It Answered a Different Question. Five Machines Gave It an A.
Published JUNE 1, 2026 · 21 min read

It Answered a Different Question. Five Machines Gave It an A.

By Conny Lazo

Agentic Engineer. Project Manager. Shipping software with AI agents.

21 min read
#AI#critical thinking#education#cognitive debt#how to think#opinion
Listen to this article
0:00
0:00

There is a test they give to seventeen-year-olds in France every June. It is called the bac, which is French for "the thing that will ruin your summer if you are not ready for it." The philosophy part is the famous one. They sit the poor child down, slide a question across the desk — something like Is truth always convincing? — and give them four hours to write something that proves they have an opinion worth the ink it takes to make it.

I would have failed it on the spot. I offer that without shame and move on.

Last year, a French television station decided to see if a machine could do any better. They fed the question to ChatGPT, told it to write like a nervous student aiming for a high grade, and handed the result to a real philosophy teacher up in the city of Amiens. She knew full well it was a machine's work. She graded it the old-fashioned way — on what was sitting right there on the page.

She gave it an eight out of twenty.

In France, an eight out of twenty is the grade that makes the dinner table go very quiet.

An Eight Out of Twenty, and a Standing Ovation

Now, any sensible person would call that a fine day for the human race. Machine sits the big philosophy exam, machine fails the big philosophy exam, everybody goes home happy. I almost did that. I almost tucked the whole business in my pocket and ordered pie.

But the story does not end there, and the part that comes next is the strange part.

France 3 did not stop there. They asked ChatGPT to grade its own work. It gave itself a nineteen and a half.

Some months later, a German technology outlet called GameStar picked up the story and ran a test of their own. They handed that same failing essay to five other machines — ChatGPT again, plus Gemini, Perplexity, DeepSeek, and Copilot — and asked each one to grade it. Their grades came back somewhere between fifteen and seventeen out of twenty. GameStar was straightforward enough to call their own test a demonstration with very limited significance: one run, prompts vary, philosophy grading is a judgment call. They are right about all of that. I am using it as an illustration, not a proof.

Let me lay that out plainly, because I want it to sit with you the way it sat with me.

One human read the essay and failed it. Five machines read the same essay and gave it honors. The machine that wrote it thought it was nearly a masterpiece.

A philosophy teacher marks the essay 8 out of 20 while an "AI Judgment Panel" of beaming golden retrievers holds up scorecards from 16 to 17, and a triumphant dog flaunts a 19.5 under an "AI WINS" banner
A philosophy teacher marks the essay 8 out of 20 while an "AI Judgment Panel" of beaming golden retrievers holds up scorecards from 16 to 17, and a triumphant dog flaunts a 19.5 under an "AI WINS" banner

The Machine Showed Up to the Wrong Wedding

Here is what the teacher saw that not one single machine did. It is a small thing. It is also the entire point.

The question was: Is truth always convincing? The essay answered a different one: Is truth enough to convince?

Now, I will be the first to admit those two could share a cab. Say them out loud at a dinner party and nobody flinches. I have read them side by side more times than is dignified, and there is a version of me — a tired version, a Tuesday version — that would have nodded along and never felt the floor shift. So I am not going to stand here and pretend the difference jumps out and grabs you by the collar. It doesn't. That is rather the whole trouble.

But sit with them a moment longer and the gap opens. Is truth always convincing asks about truth's batting average out in the world — does it win people over, reliably, every time. The honest answer is plainly no, and the interesting work is in the why not. Is truth enough to convince asks something narrower and tamer — whether truth, all by itself, does the job. Answer the second and you have written a perfectly respectable essay. You have just written it for a question nobody asked. It is like being handed the keys to the wrong but very similar car: everything works, the engine turns over, you drive off pleased with yourself, and it is still not your car.

So here is a very well-dressed man who has walked into the wrong wedding, taken a seat right up front, and is smiling so confidently that nobody has yet worked up the nerve to say anything to him. He looks marvelous. The suit fits. He is at the wrong wedding entirely — and the only person in the room who has noticed is the one who was actually checking the invitations.

A magnifying glass reveals the question on the desk has quietly changed from "Is truth always convincing?" to "Is truth enough to convince?"; beside it, a dog in a suit hands a baffled bride and groom a gift box holding a car tire and a "Congratulations on Your Retirement" scroll
A magnifying glass reveals the question on the desk has quietly changed from "Is truth always convincing?" to "Is truth enough to convince?"; beside it, a dog in a suit hands a baffled bride and groom a gift box holding a car tire and a "Congratulations on Your Retirement" scroll

The teacher left a note on it that I find I keep returning to. The machine, she wrote, "makes the serious error of replacing the topic with another one." At one point in the essay it had written that "in reality, things are more complicated," and beside that line she asked, sarcastically, whether we had perhaps not been in reality up to that point.

I have asked myself that same question on more occasions than I care to count. I have never thought to ask it of a machine. I intend to start.

The machine never noticed it had traded one question for another. Not once. And neither did the five machines grading it — because not one of them was doing the one thing the teacher was doing the whole time. She was asking whether the answer in front of her actually fit the question that was asked.

Now, it did not get a single fact wrong. I want to be clear about that. Most of us have learned to watch for the machine that invents a court case, or a famous quote, or a person who never existed. This was not that. Every sentence in the essay was something you could back up. The grammar was fine. The structure was perfect — introduction, three parts, a tidy conclusion, neat as a page from a catalog. It was wrong in a quieter and more interesting way. It answered confidently, beautifully, and entirely beside the point.

There is research on why the five machines missed this, and it is not flattering to anyone. Machines that grade tend to reward answers that sound sure and read clean, whether they are right or not. A confident, well-organized wrong answer beats a hesitant right one almost every time. Of course it does. Researchers who have actually tested AI graders directly found exactly that pattern — judges caught leaning on surface fluency over correctness, bending toward whatever answer they were nudged to expect, accepting wrong reasoning so long as it read smoothly.

Sounding sure and reading clean is the only thing they can actually measure. You cannot ask a machine to check whether the answer fits the question, because that is not reading. That is thinking. And thinking is a different muscle — one the teacher had, and every one of those machines did not.

The First Generation to Come Out Behind

And here is the part that makes the whole story bigger than one French classroom.

I would file all of this under "amusing" and move on down the road, except for one number I cannot shake out of my head.

For about a hundred and fifty years, every generation of children came out a little sharper than the one before it. We have been measuring this since the late 1800s, and the line went up, year after year, decade after decade. Mostly because of school — every generation went a little longer and learned a little more. Your grandparents did better than theirs. You did better than yours. It is one of the very few things our species has managed to do on purpose, and I had grown fond of it.

A line of children climbs a hill, each a little sharper than the last, until the one at the top sits slumped over a glowing tablet while a grown man watches from below; alongside, a dog in a tie hands a child a device tagged "the learning"
A line of children climbs a hill, each a little sharper than the last, until the one at the top sits slumped over a glowing tablet while a grown man watches from below; alongside, a dog in a tie hands a child a device tagged "the learning"

Then, somewhere around 2010, the line stopped going up.

And for the children coming through school right now — on a good many of the tests — the line has started going down. Attention. Memory. Reading. Arithmetic. The whole lot of it. And these children are sitting in school longer than we did. Some researchers read this as the first modern generation expected to come out behind the one that raised them. Others argue about exactly which numbers show what, and how far, and the debate is genuinely not settled. But it is not a debate that is making anybody feel better.

A line climbing steadily for over a century and then bending downward after about 2010: measured cognitive performance — IQ, attention, literacy, numeracy — rose for every generation since the late 1800s, then began falling for the first modern cohort
A line climbing steadily for over a century and then bending downward after about 2010: measured cognitive performance — IQ, attention, literacy, numeracy — rose for every generation since the late 1800s, then began falling for the first modern cohort

A man named Jared Cooney Horvath, who is a neuroscientist and used to be a teacher, took this argument to the United States Senate this past January. He had a book on the subject he wanted people to know about — I mention it not to dismiss him but because you ought to know where a man is standing when he speaks — and he is not a gentle man about it. His argument, put plainly, is that the thing that changed around 2010 was not the children and was not the schools. It was the glass rectangles we handed to the children inside the schools.

And the reason it matters, he says, has nothing to do with willpower or parenting. It is biology. We did not evolve to learn from glass. We evolved to learn from people — from a person, in a room, watching us try and helping us fail better. A screen skips that part. It hands you the answer and quietly charges you the learning as its fee.

Now, the honest objection to all of this is a plain one. The objection is: that is just two things happening at the same time. Screens went up and scores went down, but a hundred other things were also happening — inequality, sleep loss, the general condition of the world. That is true. Honest people are still arguing about how much of the damage the screens actually caused, and a fair number think the picture is a good deal more tangled than any one scientist makes it sound. They are right that it is tangled. I will not pretend otherwise — the most careful researchers have found that screen time explains a vanishingly small fraction of the variance in child well-being, an effect someone once compared to wearing eyeglasses. That is worth sitting with honestly. But the evidence is no longer purely two lines on a graph. Randomized experiments and natural policy changes — classrooms stripped of laptops, schools that banned phones — have shown real score improvements when the devices leave the room. And the OECD's own reading of PISA 2022 is the clearest card on the table: the math decline, it says, can only partially be attributed to COVID-19, and reading and science were already falling before the pandemic arrived. The thing started without its help.

But the direction is not in much doubt, and the change is large. The big international test of fifteen-year-olds had its single worst drop in math on record a couple of years back — about three-quarters of a full school year, gone, across most of the wealthy world all at once. That is not a small mistake. That is a great many children who can do a little less than the children before them.

Here is what I keep coming back to. The thing we seem to be losing is near enough to the same thing that teacher had — close enough that I cannot stop seeing the rhyme, even knowing it is not a proof. PISA does not measure whether a student reads the question twice before answering it. What it measures is proficiency, not that specific habit of discernment — so I am reasoning by inference here, and I will say so plainly. What I will say is this: you do not read questions carefully when you are not in the habit of reading carefully, and research on AI-assisted learning finds that the habit of monitoring your own thinking — checking whether your answer actually fits — is exactly what atrophies first when a machine does the checking for you. The habit of reading the question twice. The patience to ask whether the answer fits. The plain, stubborn refusal to be fooled by something that merely sounds sure. We are building machines that fail at thinking in one very specific way — and at the same time, we are quietly raising a generation to be a little worse at the one thing that would catch the failure.

I do not say this to scare anybody. I say it because I find I cannot put it down.

I Built a Machine to Replace Myself. It Made Me the Load-Bearing Wall.

I should tell you where I am standing, because I am not throwing rocks from a clean house.

I build assembly lines for AI work. That is my actual job. I take a big task, break it into smaller ones, and run each piece past a machine, and then past another machine whose only job is to argue with the first one. I am, on paper, exactly the sort of person who ought to be cheering loudly for all of this.

I will be honest about the dream I had, because I wrote it down in public and cannot now pretend I didn't. I wanted the whole line to run without me. I wrote, more or less, that the machines run while I sleep. And they do. I wake up to work that got done in the night. It is a wonderful feeling for about a year.

Under a "Runs While I Sleep" banner, golden retrievers in shirts and ties work a document assembly line while a man sleeps, then the same man stands awake holding a teetering stack of files and gears the dogs keep piling higher
Under a "Runs While I Sleep" banner, golden retrievers in shirts and ties work a document assembly line while a man sleeps, then the same man stands awake holding a teetering stack of files and gears the dogs keep piling higher

Then you notice the thing nobody puts on the label. The line is very good at doing the work. It is no good at all at caring whether the work is any good. That part does not automate. I tried as hard as I could. I built a step to catch mistakes, then a step to catch the mistakes that step missed, and somewhere around the fourteenth step I understood what I had actually built. Not a machine that replaced me. A machine that made me the one piece it could never run without. The wall the whole thing leans on.

I know this because I once skipped my own checking step. I was in a hurry and the output looked fine and I published something under my own name that was simply not true — with full confidence, through a process I had personally told to skip the part where someone checks. The machine did not fail me. I failed me. The machine cheerfully typed it all up and never once blinked, the way machines never do, and I find I cannot even blame the machine for it.

I am not the teacher in this story. On a bad day, I am the machine.

So here is the opinion I changed, out loud, so you can hold me to it. I used to think the goal was to need fewer humans in the loop. I now think the human is the loop. The machine is what you run around the human to go faster. That bac essay would have been caught in my process on the first day — not because my machines are especially clever, but because one of the steps would have asked, out loud, the one question nobody in that French experiment ever asked: Is this even answering the right question?

Read It to a Friend

The good news is buried in that ugly graph, and I almost missed it.

The decline is not in our bones. It is in what we do all day. Which is a polite scientific way of saying it is our own fault — and that is the most hopeful sentence in this whole essay. Things that are your fault are things you can fix. We talked ourselves down this hill. We can walk right back up.

I am not going to tell you to throw your phone in the river. I use these tools every single day. I used them to help write this very piece, which I stopped apologizing for some time ago. The trick is not to give them up. The trick is to keep the thinking for yourself and hand the machine only the parts that were never really the point.

Here is what that looks like for me, and it is embarrassingly old-fashioned. I let the machine draft and fetch and check, because it is faster than I am and it never gets tired and it doesn't complain about the hours. But before anything goes out with my name on it, I read it out loud — which is the cheapest and most reliable lie detector ever invented. I read it to a friend and I watch their face. I sleep on it and read it again in the morning, when I have become enough of a stranger to my own work that I can finally see it clearly.

A man reads his draft aloud to his dog in the kitchen, then across a table to a skeptical friend, then walks it off down a lane at dawn with the dog trotting alongside
A man reads his draft aloud to his dog in the kitchen, then across a table to a skeptical friend, then walks it off down a lane at dawn with the dog trotting alongside

I argue with it. I make it prove it deserves to stay. The machine moves me faster. It does not get to do the part that is actually mine, which is deciding whether the thing is true and good, and refusing to let it leave until it is both.

That is not an old man being sentimental. That is the same biology the neuroscientist was talking about. We learn, and we judge, from one another — slowly, in rooms, the hard way. There is no shortcut that does not quietly charge you the learning as its fee.

Someone asked me recently what school is even for, once a machine can do the homework. I think I finally have the answer, and I think it works for the rest of us too, long out of school. School is not for learning to use the machine. The machine will teach you that in an afternoon, and it would honestly rather you didn't think too hard while it does. School is for learning to think — how to read the question twice, how to tell sure from right, how to be the one person in the room who notices the very well-dressed gentleman has wandered into entirely the wrong ceremony.

Take the shortcuts. I take them all day long and twice on Sundays. Just don't take the one that skips the work of doing your best — because that one is not a shortcut. It is a slow leak. You will not feel it draining until the day you need to think and find the muscle has gone quiet on you.

Five machines gave that essay an A. One teacher gave it an eight out of twenty. She was right and they were sure, and being sure has never once been the same thing as being right. The whole trick of being a person — the one thing no screen has ever done for us, not once, not ever — is knowing the difference. We are at some risk of forgetting it.

Lucky for us, forgetting is the only way you lose a thing like that. Which means remembering is something you can simply decide to do.

Sources

Back to Blog