How Do You Build When Tests Take Four Hours?

Here's a constraint that shapes everything else in this piece: the only test that tells you whether my value-investing tool works is running the whole thing — a full research report, two to four hours, dozens of AI calls arguing with each other, seven to twenty dollars of API bills gone whether the report is good or garbage. You cannot unit-test your way into trusting this. You have to run it and read what comes out.

Three days ago, what came out wasn't good. The tool takes a ticker and produces an eight-section research report, and every claim has to trace back to something I can point to — a quote from an actual SEC filing, or a figure my own math produced. The filing-backed ones get checked: a separate gate throws out any quote that isn't really in the document. Under the hood, one AI drafts each answer (the reporter), a second attacks it (the challenger), and they go back and forth, up to four rounds, until they agree or admit they don't.

Thursday morning the machine ran. It just wasn't a product: reports took most of a working day, cost about twenty dollars, and the arguing — the whole reason I built two AIs instead of one — almost never finished. By Saturday night, the same machine produced a trustworthy report in under two hours for about seven dollars. And the idea that made the difference: you don't trust AI-written code by reading it. You trust it by making it survive something whose only instruction is to find the reason to reject it.

Thursday: the engine that argued forever

The first problem looked like a speed problem — a report was taking four to eight hours — and the easy thing to blame is the model, they're just slow. It wasn't that. Every one of the roughly forty AI calls inside a report was re-sending the same 110,000-token stack of filings from scratch, because the cache that keeps those filings warm — ready to reuse instead of re-read — was, in practice, dead. Ten cents and several minutes of redundant reading. Forty times. Per report.

The fix was to pair a fast, cache-friendly model to draft answers with a genuinely sharp one to attack them, and make the caching work. It helped immediately — a challenger call dropped from around thirty-four cents to around six cents once the cache stayed warm. Then the sharper challenger exposed the real problem, and it was humbling: with a better adversary in the loop, the debates converged zero times out of forty. The stronger critic really was stronger — it caught a real contradiction the old one had waved through, $4.4 billion in one place against $2.8 billion in another for the same figure — but it never once said "I'm satisfied." It re-argued points it had already won, demanded figures that didn't exist in the filings. A sharper critic with no rules of engagement doesn't make the work better. It just makes sure nothing ships.

So I didn't make the challenger weaker. I gave it rules. Every objection had to come from a fixed list of what actually counts as one. Every demand had to be answerable from the filings in front of it. It had to remember its own past objections, so it couldn't reopen a point it had already lost. And in the final round it could keep the objections that mattered, but had to drop the ones that were just posturing. Zero out of forty became fourteen out of fourteen — fewer debates, because the rules cut out the wasted re-litigation, and every one that was left settled. Same challenger. It just learned what winning looks like.

By Thursday night the thing converged and ran warm. That's the night I made the mistake of thinking it was done.

The real problem: a test that takes four hours

Here's the constraint that shaped the next two days: you cannot get fast feedback on a system like this. Fire off the only test that means anything — a full report, hours, real money — and you get maybe two experiments a day. That's not a sprint. That's a nap.

So no fix got written and tested by the same mind in one sitting. Every one ran through the same four-stage relay — four separate AI agents passing a baton, none of them allowed to remember what the one before it was thinking. The first reads the actual code and writes a plan: which lines change, which things might break. The second, who never sees the first one's reasoning, is told only to find the reason to reject that plan. The third writes the code — and writes the test before the fix, so it watches the test fail against the broken version first. The fourth, fresh again, reads the finished changes line by line and re-runs every test from scratch. Only then does it ship and face a real report in production.

While that four-hour report ran in the background, two or three of those relay races ran in the foreground, on files that didn't touch each other. My job was mostly to keep the lanes from colliding and catch each result as it landed — not to stare at a progress bar. The fresh-agent rule isn't ceremony. It's the same reason you don't ask someone to proofread their own essay: the mind that wrote the plan is exactly the mind that can't see what's wrong with it.

Three regressions that would have shipped clean

The honest question about AI-written code isn't whether it looks right — it's how you'd ever know if it didn't. Three times, here, the adversary earned its keep. All three would have shipped clean otherwise.

The laundering channel. One fix let the reporter tag a number as "this comes from our own math, not a filing" — reasonable, since not every number needs a filing to back it. The adversary spotted the hole: the same tag let the model stick that clean label on a made-up number. Mark any invented figure "our own math," and it sailed straight past the fact-check — a dirty number wearing a badge that said trust me. I hadn't seen it. The fix: if it's something the database can check, check it anyway, tag or no tag.

The obvious fix that was a worse bug. An earlier review flagged that the system should reject any numeric claim whose units don't match — a dollar figure compared against a fact recorded in shares, say. Obviously correct, until the adversary proved it wasn't. The reporter types those units in by hand, as plain text, with nothing keeping them consistent. So a perfectly correct "$9.72 per share" would read as contradicting a fact filed under "USD/shares" — the same thing, labeled two ways. And that false contradiction would permanently block the one number at the center of every report: the sticker price — the whole valuation the report is built to produce. The obvious fix would have broken the exact thing it was meant to protect. We shipped half of it and left the rest for later, on purpose.

The test that would have lied. When I parallelized the five report sections that don't depend on each other, the existing test suite still modeled the old, one-at-a-time behavior. Run against the new parallel code, it would have quietly let duplicate work spawn — and stayed green anyway, because it was never watching for the thing that had actually changed. A green test on a broken assumption is worse than no test at all. It had to be rebuilt to model real concurrency before any of its checkmarks meant anything.

Friday: the auditor that said no

Separately from the adversary reviewing each fix, I ran a blunter check: a fresh agent, no context, told to read one finished report three ways — as a paying customer, as a prompt engineer, as a senior engineer. Its verdict: a paying customer would not be satisfied, but the gap was fixable.

It found thirteen problems. The margin-of-safety section — the single most important part of a value-investing report — stated no price and bluffed the on-sale verdict, because a data feed declared in the code had never been wired up. A growth table contained twelve numbers the model had simply invented. Quietest, and worst: every answer was being saved to the database in the wrong shape — squashed into a plain block of text instead of the structured form the rest of the code expected. That bug had green unit tests for weeks, because the tests fed it clean data and production fed it something subtly different.

Green tests, broken product. That's the sentence I keep coming back to. A mocked test can't catch a bug that only exists in the gap between the fixture and the real thing. That bug is why the relay writes its test to fail first — against the actual broken code, so you've watched it fail before you believe it's fixed. And it's why the real gate was never the test suite: it's a real report, in production, read straight out of the database.

Ten of the thirteen got fixed that day. The next report stated an actual price and a grounded verdict instead of bluffing one. It cost $9.43 — less than half of Thursday's $20.82 — though it still took 232 minutes, because the sections still ran one at a time. Trust was fixed. Speed wasn't, yet.

Friday night: making it fast

With trust in better shape, the last lever was speed, and it turned out to be structural: those eight sections ran strictly one after another. As I said, five of them don't depend on each other — so they could all start at once, if the platform I was running on could actually do that.

I couldn't reason my way to that answer; I had to measure it. Before writing the real scheduler, I sent a tiny two-branch probe to a live environment and watched the clock: did the second branch start while the first was still running? It did. Only then did I build the real thing.

The result: the five independent sections all started within twenty-nine seconds of each other, instead of one at a time, hours apart. The whole report came in roughly twice as fast.

The report that disagreed with itself

Saturday night's report was Costco, and it had every fix at once — the convergence rules, the audit fixes, the parallel scheduler. It finished in 114 minutes for $7.32, against roughly $21 and most of a day for the first report, two days earlier. It's not a clean benchmark — three different companies, Costco the smallest read of them — so some of that drop is the job shrinking, not the engine improving. But it isn't only that.

Its valuation section is what I built this tool for. A sticker price — what a share is really worth — of $546.30. A margin-of-safety price of $273.15, exactly half the sticker: the price you'd wait for before buying, so a wrong guess still leaves you room. And a current price of $1,028.24. Verdict: avoid. Every number traces back to a cited filing or a formula I can point to.

I opened this section expecting to watch the honest-disagreement feature work. The owner-earnings build-up lays out four numbers — pretax income, depreciation, the change in payables, capital spending — and is supposed to end by adding them into one figure and using it to value the business. List the parts but skip the total, and this is exactly where the challenger should catch it.

That's not what I found. The reporter had done the whole thing — the four numbers and the sum, right there on the page, ending in a final owner-earnings figure and a valuation. The engine marked the section "partial" anyway, and shipped a challenger objection swearing the sum was "absent." The objection was stale — raised in an early round, before the total was added, and never cleared once the work was done. The tool had cried wolf: it flagged a finished section as unsure of itself.

And here's the thing: that's still the part I care about most. Not the two-hour report. Not the seven dollars. The feature over-fired — a real bug, filed — but it failed in the one direction I can live with. A tool that leans toward "I'm not sure," that flags finished work rather than smooth over a gap that's really there, is the failure I'd pick every time near my money. Better a tool that cries wolf than one that sounds confident and is quietly wrong.

What's still open

None of this is finished, and I'd rather say so than let a seven-dollar number do more work than it's earned.

The price feed I wired up on Friday works now — but it prefers a saved quote over a fresh one, so Costco's report is pricing the stock against a number that's about six weeks old. Different bug, same feed. Filed. Not fixed yet.

The report reads "partial" the moment a single row carries an open objection — including, as I just showed, a stale one that should have been cleared. Technically correct, arguably terrible UX. A customer who wants a clean yes or no won't love a report that flags itself as unsure — still less when the flag is a false alarm. I haven't decided what to do about that. On purpose.

And the deepest gap: the grounding check can confirm that a claimed number appears somewhere in a quoted sentence. It cannot yet confirm the quote is actually about the thing being claimed. Number-present isn't the same as number-means-what-you-said. That's the next piece of work, and it's bigger than anything in these three days.

The four-hour test that opened this piece now runs in about two. It's still too slow to sit and watch — I still can't fire one off and wait. But I don't have to trust those hours anymore. I just have to trust the thing that spent three days learning how to say no.