more experiments, more spending (in the name of science)

When I started learning to build things with AI, I’ll be honest - I didn’t really know what I was doing (some would argue that I still don’t). But I was happy just thumbing around in the dark, learning by doing, creating to create. I used my experience managing projects and the way I worked with my teams to define the vision, architect the solution, identify constraints, break it down into epic and features, define the acceptance criteria. I picked up features one by one, and iterated over them until they were extremely detailed packages of work, and then built them.

From a process standpoint, it was incredibly structured - the documentation was predictable and high quality. But the work often needed a bit of grooming; even though I had spent time defining the seams (how x ties into y) I still regularly had to iterate over it to get it right. There had to be a more reliable way to build.

So I experimented. What if I didn’t break things down into so many discrete tasks? What if I gave the agent more detail up front, and then let it build with more context? Did it help to have one agent manage a team of specific subagents? Was it better to have a single agent manage the whole build end to end? At the end of the day I rebuilt the same project from the same technical scope of work with eight different team configurations & strategies.

My problem was context. It turns out, one agent managing the build end to end, defining its own seams between activities not only produced dramatically more cohesive results, but did so significantly faster, using a fraction of the tokens. However, at what cost (to me)? I ended up back at my original starting point where I didn’t know what decisions the agent had made, how everything hung together, why we went in one direction instead of the other.

I honestly felt a bit frustrated. All my work to be part of the process wasn’t just slowing it down, it was spending unnecessarily too! What to do, what to do…

MORE EXPERIMENTS! MORE SPENDING! (in the name of science)