One of the ways that folks are leveraging LLM-powered agents to do software development is the “software factory” (or “software dark factory”) pattern, where the LLMs write all the code and the humans aren't allowed to even review it.
Apparently this works, though you need a good way to verify your product. I'm a little skeptical that it's the best way to use these tools, but it's an interesting idea, so I'm going to try to build one for myself.
Inspired by Ed Yong's talk where he proposes the ways in which journalism can be viewed as a caretaking profession, I've been thinking about the way that software development can be a caretaking profession.
A little while ago I found myself wanting a very particular minimal text editor—something approximating the UI of iA writer but with 1% of its features and no monthly subscription. I tried a few different products, tried tinkering with the configuration VS Code and Helix, but nothing was exactly what I wanted.
Being a hip, with-it software guy I knew I could get an LLM to put together a custom text editor for me, and eventually I did, but the process was illuminating.
I've worked on computer systems in many different environments: cloud-native serverless functions, self-managed virtual servers, embedded devices, pushing code for on-prem bare-metal racks that I'm not allowed to look at, much less touch. My experience across all these projects has falsified a lot of opinions, but there's one thing I believed when I first started coding and still believe now, perhaps more than ever:
This is an interesting piece of work! A small team built a bunch of software in a short time by leveraging a whole lot of LLM assistance both for building their software and for validating it.
The main things that are new to me in this story are:
Building digital twins of third-party services like Google Docs that you can test against without rate limits.
Keeping validation scenarios away from the processes doing the building so that they can be “independently verified” by another agent (kind of like a holdout set in machine learning).
I think there are some interesting things in this writeup (though I'd love to see more details about how they build their digital twins), but I have some fundamental concerns as well.
Primarily, even if you keep your validation scenarios hidden, if they're being done by an LLM they're inherently probabilistic and could be hiding nasty surprises, especially security problems caused by weird adversarial inputs. Even though Claude can write pretty good code, is very fast, and sometimes catches things that I miss, I'm still not comfortable putting my name on software without a closely reviewed and deterministic validation step.
It also sounds like there are some easily avoidable code quality. I think bringing taste and good practices to a repo is an important part of your work as a senior engineer, and something that can pay dividends as agents follow the patterns that you've laid down.
Advent of Code is back again and I'm joining in the fun! This year, I'll try solving it with Unison, a statically typed functional language with some very neat ideas about content-addressable source code and distributed computing. It has a proprietary cloud backend, but you can also run it on your own machines, and it recently had its 1.0 release.
What follows are my very first ignorant baby steps coding in Unison.
Greg Wilson was soliciting strategies for lazy-loading datasets in Python modules. There are, of course, many ways to do this, but I didn't see this one being discussed.
Since Python 3.7 (released in 2017) you can define a module-level __getattr__ function that gets used for name resolution (see PEP 562). You can use this to call a data-loading function the first time each module-level property is accessed. This implementation uses only functions, dicts, and the one magic “dunder” function, which might be more approachable than programming with classes (depending on your audience).
After yet another fight with the syntax for text fragments in URLs (which I've written about before) I had Claude whip up a little app for constructing and testing them.