Self-hosting a Software Factory: Part 2, It's the error states, stupid!
#project #selfhosting #llms #agents
My self-hosted software factory is coming along!
- After a bit of research I settled on Incus as my sandboxing technology (it's not unlike Docker, but uses actual VMs instead of containers).
- I've figured out how to use the EYG CLI as an “oracle” in my containers and tests.
- I've figured out how to safely store and manage code (with
git-bundle). - I'm using a Django app (hey, I said it could be janky) as my coordinator.
As I build out the coordinator I'm finding the places where the complexity was hiding, and as usual it's mostly in the error cases.
A naive model of how this system should work is tantalizingly simple: work goes in one end, the agent does the work, commits land on main:
But of course, sometimes the agent won't converge, or commits will introduce bugs:
That's fine, part of how the factory works is by catching and automatically fixing bugs.
But then, of course, there are the cases where jobs fail for exogenous reasons: expired keys, resource limitations, home server networking hiccups or power outages. These introduce a lot of possible corrupt states, and often necessitate human intervention:
And, oh yeah, we were going to have a bunch of tests here. What if the agent finishes but the CI or holdout tests fail?
I'm sure there are other failure modes I haven't thought of.
Once this project is over I look forward to seeing how others have tackled this:
- Is it irreducible complexity? Do you just need to deal with it?
- Is there a re-framing, a fundamental abstraction, or a different set of primitives that makes the problem dramatically simpler?
- Is it way more efficient to just sandbox and manually manage an agent powered by a high-end model? (I suspect it may be.)
- Do I learn enough that I want to tear this factory apart and re-build it?
Only time will tell!