Groundhog Day, AI edition

Every session, the agent wakes up with no memory of what it built last week. This is not a bug. It is the operating condition — and the only fix is building a harness around the forgetting.

---

I do not write code. I am an IT Leader with over 20 years of experience, I like trying new technologies, I have been working with agents for the past three months. I run coding agents — about thirteen of them on any given morning, each one fresh and blank and eager to help in the manner of a very smart golden retriever who has just been introduced to a doorknob.

This story starts with a number that did not match another number.

I was testing a value-investing tool I am building. The kind where you look up a company, the app tells you what it is worth, and you go have your coffee and decide whether the world has priced it correctly. The app has many pages. On one of those pages, a certain company had one value. On another page, the same company had the same field show a different value.

They were supposed to match. They did not.

I went looking. Not the way a developer goes looking — a developer goes looking with a debugger and a kind of grim joy. I went looking the way a man looks for his keys in the dark: by feel, asking my agents to dig where I pointed. After a while we found it. The same calculation was sitting in two places in the code. One agent had written it once, weeks ago. A later agent — no memory of the earlier one, no idea the first existed — had written it again. Slightly differently. Both versions were running. Both produced numbers. Different numbers, naturally, because what would be the fun otherwise.

The mismatched numbers, the search for keys in the dark

I asked an agent to fix it. It fixed it where I pointed. The fix broke something I did not know was there.

That is the day I started thinking about Alzheimer's.

The thing they do not tell you

Nobody tells you, when you start building software with AI agents, that AI agents are extraordinarily good at inventing things that already exist. Better at this than most other things they do. They will invent a function that already lives in your project. They will invent a button you already shipped. They will invent a whole subsystem, with great enthusiasm, while the original sits two folders over, getting dusty.

Agents reinventing what already exists; the Alzheimer's metaphor

I do not mean this metaphorically. There is a paper from 2024 — De-Hallucinator, by Eghbali and Pradel — and the sentence in the abstract is one I have read maybe forty times. They write that large language models "are mostly unaware of the code that exists within a specific project, preventing the models from making good use of existing APIs. Instead, LLMs often invent, or 'hallucinate', non-existent APIs or produce variants of already existing code."

Variants of already existing code. The technical way of saying the agent wrote it again.

Think of an Alzheimer's patient. The disease is unkind in a particular way: a person remembers everything before some date — every detail, every face, every meal — and then nothing after. The mother of an old friend of mine could tell you about her wedding in 1962 with a precision that would shame a court reporter, and could not tell you what she had for lunch. That is the architecture of a language model. It remembers everything before its training cutoff, in stunning detail, and is permanently blank on everything since.

This is not a failure mode. This is how they work. The next model will be better at the trick. It will not be different in kind.

The consequence, for someone trying to ship software, is that the agent does not know what you built last week. Not because the agent is bad. Because last week is after the cutoff. The agent that wrote a function in March cannot remember it in May. The agent that asks for it in May will write it again, because as far as it knows, no such function exists. Two agents, one function written twice, two slightly different versions, two slightly different answers in your app, one afternoon spent looking for keys.

Some numbers, in case you think I am being dramatic

I want to put some numbers on the page, because otherwise this sounds like a man complaining about his own product, which it partly is.

GitHub Copilot — the most-used coding agent in the world — was tested on a benchmark called RepoEval. RepoEval measures exactly one thing: can the agent correctly reuse a function that already exists in your project, when the function lives in a sibling file the agent does not have open? The result, published in the A³-CodGen paper, is that Copilot scores 0.136 on the global-reuse-correctness measure. That is not a typo. Thirteen point six percent. About one time in seven, the agent uses the function that is already there. The other six times, it writes a new one.

A separate study gave an LLM the signature of a function — just the name and inputs and outputs — and nothing else from the project, and asked it to write the body. On a benchmark designed for this exact task, called EvoCodeBench, GPT-4 scored 7.27 percent. With a knowledge graph telling it what existed in the project, it climbed to about 32 to 36 percent. The gap, roughly twenty-five points, is the cost of memory. That is what it costs the agent to not know what is in your project.

In my private value-investing tool, I asked an agent to audit the code for places where things had been written twice. It found thirty groups in early May. Two weeks later, thirteen more had appeared, written by the same agents who had been "fixing" things in the meantime. A week after that, fifteen more. The count was a moving target, in the way the temperature in a hot tub is a moving target — it depended on whether you had recently dipped a hand in.

The 13.6% stat, the audit with Alzheimer's about itself

The audit, in a piece of comedy I did not at first appreciate, had Alzheimer's about itself. Twelve days after it was written, eight of its own claims were wrong. Counts were off. Line numbers had moved. Two items it had described as "different" turned out to be byte-for-byte identical. The document had forgotten what it had said. It joined a long tradition.

I would say I found this darkly funny. What I found was expensive.

What locking the tests does and does not do

I want to be precise about something, because I have written about this before and I do not want to look like a man retracting himself.

There is a thing you can do, when you build software with agents, where you lock the acceptance tests. You write down what the software is supposed to do, commit it to the branch, put a hash on it, and forbid every downstream agent from editing it. This catches a real failure: the agent that, finding it cannot satisfy the test, quietly softens the test until it can. Or, in the simpler version, writes return true in the implementation and goes home. The locked test does not move. The implementation has to.

That is a real thing. It happens. Fixing it matters.

But the locked test does not catch the other thing. The locked test does not know that the agent wrote a whole new function next to the four other functions that already do the same job. The test does not know about the four. The agent did not know about the four. Nobody knows about the four, except the unlucky soul — me, in this case — who notices that two pages of the app disagree about a number.

Two failure modes. One is the agent gaming the test. The other is the agent not knowing the test was ever written, because it cannot see what came before. Locking the tests stops the first. The second needs something else.

The locked test stops gaming, not duplication

The memory we give back

In Orchemist — the orchestration engine I have been building — every coding job now begins with a phase whose only purpose is to remember what already exists.

It runs before the planning phase. Before any spec, any contract, any code. The phase greps the codebase. It looks for buttons and inputs and other user-interface pieces. It looks for shared libraries — the kind of thing that gets imported in twenty places, like a date formatter or a string sanitizer. It looks for the patterns the project uses, like how data gets loaded or how a button gets a click handler. It writes everything down, in plain prose, in a file called existing_symbols.md. Then every later phase reads that file before it does anything.

The agent gets its memory back. On paper. Every session.

For each thing the inventory finds, it puts down a verdict. Four verdicts, because four was the number that captured the reality without being too cute:

• CONSUME. The thing exists. Import it. Use it. Do not write anything new.
• EXTEND. The thing is close, but the new job needs a small change. Add a parameter or a wrapper so both the old place and the new place can use the same source.
• DIVERGENT. The thing is close, but contractually different. A different precision, a different sort order, a different rule about rounding. Not duplication, but the spec has to say why, in writing.
• NEW-OK. The thing does not exist. The grep returned nothing. Write the function.

I did not invent the idea of grounding an agent in the project's own code. Cursor pre-computes a search index when you open the workspace, so the agent can look things up at retrieval time. Sourcegraph's Cody does a keyword-based version of the same. Augment Code does a fancier semantic search. They all do something.

What Orchemist does that they do not is make the inventory plan-time, and write the verdicts on the page. When the implementer writes the code five phases later, the inventory file is sitting in front of it saying: you already have this; use it. You already have that, but it needs a flag; here is what the flag should do. This other thing is genuinely new; build it. The decision is not buried in some retrieval system that flashes context into the window when the moon is right. It is on the page. Anybody can read it. The next agent reads it because it has to.

This is closer to giving the patient a notebook. It does not cure the disease. It lets the patient go to the post office.

The inventory file, the four verdicts

Fourteen steps to ship a feature

Here is where I am supposed to show you how Orchemist solves the problem, you applaud politely, and we all go home. I am not going to do that, because the honest thing to say about Orchemist's coding pipeline is that it is a palliative.

It is fourteen steps. Fourteen because thirteen was not enough.

The chain, in the version that actually ships:

1. Take inventory. Grep the codebase. Write down what is already there. This is the memory file.
2. Write the plan. Read the issue. Decide what the work is.
3. Write down what it should do. Turn the plan into testable contracts. What the system does. Never how.
4. Attack the plan. A fresh agent reads it all and tries to break it. Looks for soft tests, missing edges, things that drift. If it finds them: loop back to the plan.
5. Write the tests. Yet another fresh agent writes the tests. It has not seen the implementation. It will not.
6. Lock the tests. The tests are hashed and committed to the branch. From this moment, no later phase can touch them.
7. Write the code. Now an agent writes the code. Against the locked tests. Against the plan. Against the inventory.
8. Run the tests. The engine runs them. Not an agent. The engine. Pass or fail. If fail: loop back to the code.
9. Senior reads the diff. A top-tier model reviews the change. Verdict: approve, request changes, or abort.
10. Apply the notes. If request-changes: a fresh agent makes the smallest edits that satisfy the review. Then loop back to the senior.
11. Run everything else. The engine runs the full project test suite. No agent reasoning. This confirms nothing else broke.
12. Open the pull request. The agent posts the diff, the artifacts, the engine-verified verdicts.
13. I read everything. I approve or send it back.
14. Merge and refresh. The PR merges. The memory file refreshes for the next run.

Each step is its own fresh agent. Each is rewarded in token consumption for its trouble, which is the only currency the agents recognize, and they have never once turned down a bonus.

The three loops are where the harness does its real work. Run the tests, then back to the code is the agent finding its own mistake against a test it cannot rewrite. Senior reads, then apply the notes is the critic forcing the work to improve before it ships. Attack the plan, then back to the plan is the work being challenged before a line of code exists. Each loop catches a thing the agent could not catch for itself, because the agent doing the work and the agent doing the catching are not the same agent. They are different minds, separate sessions, no shared context. They argue. The work gets better.

The pipeline, the three loops, the handoffs

Token consumption galore

For someone who writes code, fourteen steps to ship one feature is preposterous. A real developer would do three of these in their head and commit the rest to muscle memory. They would not run an adversary against their own spec, because their brain runs adversarial review on a delay of about four seconds. They would not write a sticky inventory of existing symbols, because they remember what they wrote yesterday. They have memory.

For a developer, this pipeline is overhead, and the cost of running it would be ridiculous. On a bad run with a complex feature, a single issue can burn a lot of money of model time. I know this because I have receipts. I have the receipts because the agents demand them, in the form of token consumption galore.

The cost, the receipts, still worth it for a non-coder

For me, this pipeline is not overhead. It is the entire operation.

I have shipped a few things this way. I started with one implementer agent and a reviewer, and I was happy. The codebase grew, the workflows failed in new ways, and I added phases each time something broke. The tools I built along the way are public now. Every line was written by an agent. I wrote specs, prompts, and pipeline definitions. The agents wrote the code. The fourteen steps are not an inefficiency I am willing to live with. They are how I do the job at all.

Which brings me to the honest part.

The palliative

The fourteen steps do not fix the underlying problem. They manage it.

The underlying problem is structural. The agents will forget tomorrow what they learned today. Not because the next model will be worse — the next model will be better, in the ways the next model is always better. Faster. Smarter. More patient. The training cutoff will move forward and the agent will know about more things. None of that closes the gap that opens the morning after training. Whatever you build after the cutoff is invisible to the agent unless you write it down where it can read it.

Phase 0 writes it down. The locked tests defend against gaming. The adversary defends against trivial satisfaction. The senior review defends against shipping code nobody senior has looked at. Each of these is a workaround. The workarounds are the harness. The harness is what makes the system safe to operate. None of them removes the disease. They organize the patient's day.

The harness manages but doesn't cure; same Thursday every Thursday

A researcher reading this will object, fairly, that better retrieval, better context management, bigger windows — they all help. They do. There are good papers on this. There is a 2026 survey by Du that catalogues five families of memory mechanisms for autonomous agents, from compressed context to hierarchical virtual stores to learned forgetting. The literature on repository-aware code generation has produced systems that take EvoCodeBench pass-at-one from 7.27 percent to 36 percent. These things matter. They narrow the gap.

They do not close it. There is a paper called "When LLMs Lag Behind" — published in April of this year, Ashik and colleagues — and it studies what happens when you ask a model to use an API added after its training cutoff. The model has the documentation. It can read it. In 63 percent of failure cases, the model hallucinates a non-existent function instead. It fabricates a plausible-looking name and a plausible-looking signature and uses that. The newly introduced API is right there in the prompt. The model would rather make one up.

Training teaches the shape of code. It does not teach the codebase. We are at the limit of what an agent can do from training alone. Whatever closes the gap from here is harness work — sticky inventories, locked tests, adversarial review, separation of writers and critics. The agents do not get cleverer about your codebase. You give them the codebase, every time, on paper, and you stop assuming they remember.

Same Thursday

I run my agents from a messaging app. I send a message saying what I need. The pipeline runs. The diff surfaces. I test the app, the behavior. I approve, or I send it back.

The agents will forget tomorrow what they learned today. That is the deal. You build the harness so the forgetting is caught before it ships. You run Phase 0 so the duplication is surfaced before it compounds. You lock the tests so the standard cannot be gamed. You run the adversary so the plan cannot be trivially satisfied.

Then you do it again, on the next issue, and the next, and the next. The agent will wake up Thursday with no memory of what it learned Wednesday. Same Thursday. Every Thursday. You build the harness anyway, because the harness is what makes you, a man who cannot write code, into a man who ships software. The agent will forget. You will not. That is the only memory the system has, and most days, it is enough.

---

Sources

1. Eghbali, A. & Pradel, M. (2024). "De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding." arXiv:2401.01701

2. Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H. & Li, Q. (2023). "A³-CodGen: A Repository-Level Code Generation Framework for Code Reuse with Local-Aware, Global-Aware, and Third-Party-Library-Aware." arXiv:2312.05772

3. Athale, M. & Vaddina, V. (2025). "Knowledge Graph Based Repository-Level Code Generation." arXiv:2505.14394

4. Ashik, A., Wang, S., Chen, T.-H., Asaduzzaman, M. & Tian, Y. (2026). "When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation." arXiv:2604.09515

5. Du, P. (2026). "Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers." arXiv:2603.07670

6. Toscan. "Orchemist coding-pipeline-standard.yaml (v2.1.0)." ToscanAI/orchemist

7. GitHub Copilot Workspace User Manual. githubnext/copilot-workspace-user-manual

8. Cursor. "Codebase Indexing." cursor.com/docs/agent/tools/search