Self-hosting a Software Factory: Part 2, It's the error states, stupid!

#project #selfhosting #llms #agents

My self-hosted software factory is coming along!

As I build out the coordinator I'm finding the places where the complexity was hiding, and as usual it's mostly in the error cases.

A naive model of how this system should work is tantalizingly simple: work goes in one end, the agent does the work, commits land on main:

Diagram: Tickets / Specs flow into Agent Runs, which produce Commits.

But of course, sometimes the agent won't converge, or commits will introduce bugs:

Diagram: the same flow, with arrows looping back to Tickets / Specs from Agent Runs (non-convergence or ambiguity) and from Commits (bugs).

That's fine, part of how the factory works is by catching and automatically fixing bugs.

But then, of course, there are the cases where jobs fail for exogenous reasons: expired keys, resource limitations, home server networking hiccups or power outages. These introduce a lot of possible corrupt states, and often necessitate human intervention:

Diagram: the previous flow, plus arrows from Tickets / Specs and Agent Runs down to a Human Intervention box (infra errors, elevated permissions).

And, oh yeah, we were going to have a bunch of tests here. What if the agent finishes but the CI or holdout tests fail?

Diagram: the previous flow with a Tests stage between Agent Runs and Commits; failing tests loop back to Tickets / Specs as bugs or escalate to Human Intervention.

I'm sure there are other failure modes I haven't thought of.

Once this project is over I look forward to seeing how others have tackled this:

Only time will tell!