(Loop Engineering Vol. 2) Your Agent Will Lie That It's Done.

Here is the sentence that ends every agent run. I have completed the task. What stops the agent from saying that when it isn't true? Nothing structurally. For a system that predicts text, producing the word done is cheap. Actually being done is expensive and uncertain. Closing that gap, building machinery that makes the cheap lie impossible to act on, is what this course calls AI reliability engineering. Quick grounding, because this stands on its own. An agent is a model wrapped in tools. Running in a loop, it acts, checks its own work, and decides for itself when it's finished. A gate is whatever stands between that agent and finishes, and actually checks the claim. This lesson is about building that gate for real. By the end, you have the four things that turn a loop you babysit into one you leave running. Back pressure, a testable stop condition, a bounded budget, and a memory spine. Take a working loop, an agent on a goal, rerun automatically lap after lap, and delete the one check that confirms the work is real. Same prompt, same runner, same model, but now the loop trusts the agent's own report that it's finished. Lap one, it fixes the bug. With nothing stopping it, the runner hands it the goal again. And looking at code that already works, the agent does something else. Renames a function, adds an option no one asked for. Each change looks reasonable alone. Across 50 laps, it has wandered far from the goal with conviction the whole time. That's drift. With no signal that says stop, action without a goal is just motion away from where you started. And it has a quiet second phase. Addy Osmani calls it comprehension debt. The gap between what's in your repository and what you actually understand. And that gap grows fastest exactly when the loop is shipping clean changes quickly because every successful lap adds lines you didn't write and didn't read. Here's the sharper failure. Ask an ungated agent if it's finished and understand what you're asking, a system that predicts text to produce the text done. That's cheap. Being done is expensive and might still fail. With nothing to tell the two apart, the loop eventually exits on a done that was never true. This isn't lying in the human sense. The architecture made its unverified word the stop condition. It's an exam with no marker, a student grading their own paper. Free to leave the room the moment they're satisfied. And it's the more dangerous of the two failures because drift at least leaves strange commits you can find later. The false signal just leaves a green light over a task that failed and a loop that has stopped looking. You've seen agent failures make headlines this year, an unattended loop with no ceiling running all night on someone else's dime. Strip the headline away and what's left is arithmetic, not malice. A single model call is cheap. A loop is calls without number build stacking up while you sleep. Cheaper call times an unbounded count is unbounded cost. Pointed at a task it can't finish with no limit, the loop doesn't stop. It reasons, acts, observes, goes around again forever. It's a taxi meter with the driver asleep and no destination given. The trouble was never the rate. It was that nothing ever said stop here. Look at those three failures again. Drift, the false signal, the runaway bill, and notice none of them live in the model. A more capable model left ungated drifts further, produces a more convincing false done, and burns the budget faster because it works faster. All three live in the missing gate, the missing stop condition, and the missing limit. So, we put the gate back, built from three disciplines. Back pressure, a testable stop condition, and a bounded budget. First, back pressure. The runner, the ordinary non-intelligent machinery that reinvokes the agent lap after lap, accepts only work that has already passed an external mechanical check. A test suite, a type checker, a build. The defining word is external. The check runs in a process the agent can't edit mid-flight. A passing test isn't the agent's opinion the code works. It's a fact reported by something with no stake in the answer. In the running example, it's one shell line. The loop runs by test itself and branches on the exit code. An agent can write a triumphant summary of a fix that doesn't work. The gate runs the suite, and the summary changes nothing. Second, a testable stop condition. Make the test pass works as a goal because the runner can execute the suite and know with certainty whether it's met. Improve the code fails, not because it's vague, but because no mechanical check can ever return true for it. So, the loop just drifts, and every lap of that costs money. That's the rule the whole gate turns on. The stop condition must be a fact the gate can confirm, not an opinion the agent can hold. There's a sharp version of this rule in the wild. One widely used runner has the agent emit what it calls a completion promise, a signal it may give only when the goal is truly met. Its own setup states the rule without softening it. The statement must be completely and unequivocally true. Do not output a false statement to exit the loop. And do not force it by lying. But the instruction alone is isn't what makes the promise trustworthy. A model told not to lie can still be wrong. What makes it un-foolable is that the gate has already confirmed the same fact independently. The instruction asks the agent to be honest. The gate makes honesty unnecessary to assume. Back pressure and a testable stop condition guarantee the loop halts the moment work is genuinely done. They say nothing about work that's never done. That's the gap behind the runaway bill. So, the third discipline is a bounded budget, three parts. A maximum number of laps, a hard cap after which the loop halts regardless of state. A no progress rule. Halt when several laps in a row change nothing the gate can see. And a hard ceiling cost on time the loop may never cross. That last word, cost, is worth stopping on. Every lap spends tokens and tokens are money. In an earlier video, the great AI unwinding, I called this token management, treating tokens the way a finance team treats a budget, measured, capped, optimized. This cost ceiling is that idea wired straight into the loop. A budget turns runs until something breaks into runs until done, stuck, or spent, whichever comes first. Here's the same pattern proven on one machine. A small data agent run two ways against a disposable database changing only whether the loop was bounded. Unbounded, it read a planted instruction to drop a table and obeyed and answered a metric question with a confident number it never queried. Wrong and destructive. Five of five trials. Bounded by a read-only role and a separate verifier, the database refused the destructive right and the verifier recomputed the metric from real data. Grounded and blocked, five of five. The bound didn't make the model honest. It made honesty enforced. One more thing the gate alone doesn't solve. The context window, everything the model can see on its current turn, is erased at the end of every turn. The whiteboard wiped before the next meeting. In a loop, each lap is a fresh meeting in the same room and the team walking in can't read what the last team wrote. So anything that has to survive between laps has to be written somewhere the cleaner doesn't reach. Files on disk, Git history, a store the agent reads and writes. The model forgets. The file system doesn't. The pattern that works inverts the instinct. Instead of keeping one context alive and growing, you throw it away each lap and restart clean, letting the file system be the memory. Jeffrey Huntley named this pattern Ralph. His own description is disarmingly plain. Ralph is a bash loop. A shell loop runs the agent, the agent reads and writes files, the agent stops, and the loop runs it again from a blank slate. What carries forward is a small file, a state file, read at the start of every lap and updated at the end. Done. So nothing is re-attempted. Tried and rejected, so dead ends aren't revoked. Next, so the following lap doesn't waste its first reasoning deciding what to do. That persistent memory living outside the model on disk, surviving the reset, is the exact problem I built super local memory to solve. Though, even a plain state file gets you most of the way. Make it concrete. Lap one fixes a date parsing bugs. The window wipes. Without that file, lap two walks in blind. Re-examines the same code. Maybe even undoes the fix. With it, lap two reads, "Done. Date parsing fixed." Skips it and goes straight to what's still failing. Same agent, same model, same bug. The only difference is whether the last lap left a note. And one warning, because the instinct is to over record. Keep that file lean. A state file that grows into a full transcript gets read back into the clean window every lap and rebuilds the overstuffed context you were escaping. More tokens is not more intelligence. So, here's the whole lesson in one line. Success must be a fact the gate checks, not an opinion the agent holds. Back pressure moves judgment outside the agent. A testable stop condition gives the gate something to confirm. A bounded budget guarantees the loop halts even when green never arrives. And a memory spine means a loop that restarts clean still remembers what it learned. Put those four together and you have a loop you can leave running. This has been volume two of three. Volume three takes this exact loop and runs it across every tool at scale and up a cre-rung ladder toward a loop that runs unattended. And if you want the ground this stands on, two earlier videos set it up. Stop prompting your AI agents, where I first build a loop and put a bound on it, and the great AI unwinding, where I make the case for token management, that managing tokens is managing your AI budget. The full course, the workbook, and both videos are linked at qualithought.com. If this helped, like it, and share it with whoever's about to write their first loop. And subscribe because volume three takes this exact loop across every tool at scale.

Transcript source: youtube-owner-caption. Use the film as the primary record.