keeping agents accountable

So… I was telling you fine professional people of the internet about my agentic workflow problems, and I had landed myself back into the world of more experiments and more spending; a lot more experiments and a lot more spending actually (thank goodness for mid-week resets by Anthropic).

At the risk of actually sounding like I learned something - I found out that with a little creativity & a whole lot of structure that I didn’t have to choose between being involved (slow, controlling, and expensive) and stepping back completely (fast, blind, and cheap). I was able to create the Hannah Montana (best of both worlds, you get it) of agentic workflows.

I painstakingly audited every role & interaction of my workflow, and moved the ceremony that was peppered throughout to two places: up front into a proper technical scope of work, relentlessly interrogating an idea until there are almost no big unknowns, and then a lighter ad-hoc bit in the actual build. The build agent stops for big, tough to reverse decisions, and waits to discuss, but narrates smaller decisions as it goes (choice, reasoning, alternatives) - which are captured by the workflow telemetry - which means I can pull the important ones out afterwards instead of stopping it every five minutes. (there are many more details here, but I’ll spare you the jargon)

But now we had a bigger issue. Trust. See, I’m allowed to lie to my agent (kidding), my agent cannot lie to me.

An entire army of deterministic validator scripts, hooks, and gates later - the framework literally will not let the agent take shortcuts, drift from the process, or call something done when it’s not. I don’t have to sit in the driver’s seat to make sure things are done correctly, I don’t have to question whether or not something is true - deviating without express human intervention is a deterministic impossibility. I’m involved where I need & want to be, and I’m out of the way where I’m just a nuisance.

I should say though, changing the framework & workflow this much broke other things (so many things), like how I tracked bugs, how I added new features, how I changed my mind about something - so of course, I had to rebuild those too. (Story for another day.)

And at a second risk of being incredibly cliche, I realized that I was not even a human in the loop before, I was a human standing next to the loop nudging it along at each gate. Now (I think) I am in the loop… The agent works where it’s strong, the guardrails I set up hold the line I used to hold by hand, and I’m pulled in for the decisions I have specified I need to be included in.

So far, it’s holding up on smaller projects and one old project I backfilled to test it. Though I suppose it will only be properly tested on the next big swing… We shall see, either that or I’ll be back here with another story about how I’ve completely rebuilt everything again.