Vanguarstew journal
Any bot can patch code. Vanguarstew fixed the system.
AI writing code is no longer impressive. The real question is a harder one — can an AI maintain itself? This week, vanguarstew answered yes: it found a flaw in its own judgment, and then fixed the right thing.
Vanguarstew is our AI software maintainer. It co-maintains its own open-source repository under a human owner's supervision — reviewing pull requests, measuring its performance, and improving itself. To make sure those improvements generalize beyond its own project, it is also evaluated live against real-world repositories such as OpenClaw, a large, fast-moving open-source project.
By replaying real history. Freeze a repository at a past commit; predict what its maintainers will do next — including whether they're about to cut a release; then check that prediction against what they actually did. The answer key isn't invented. It's the project's real future.
It found the flaw on its own
Running against those real repositories, vanguarstew kept predicting that a release was imminent when the real maintainers cut none. Nothing crashed. No exception was thrown. No user reported a bug. The system was simply, confidently wrong — which is the hardest kind of flaw to catch, because it hides inside a plausible answer. Nobody filed this. Vanguarstew surfaced it by working on real code.
The obvious fix would have missed the point
The owner opened an issue. Contributors submitted pull requests — each one improving the release-prediction model. Better heuristics, better features, better code. If this were just a coding task, one of those patches would have been the answer.
But maintaining software isn't only writing better code. It's knowing when the code isn't the real problem.
The benchmark was the problem — not the model
Reviewing the proposed fixes, vanguarstew reached a different conclusion. The benchmark evaluated every repository over the next five commits — the slice of real history each prediction was checked against. That sounds fair. It isn't.
On a repo making 300 commits a day, five commits are about 24 minutes of work. On one making 40 commits a year, five commits span roughly 46 days. The benchmark treated those windows as equivalent — so a model that always predicted "no release" beat a genuinely intelligent one about three-quarters of the time. The contributors were tuning the model to hit a target that measured nothing.
So vanguarstew ignored the obvious fix. Instead of tuning the prediction algorithm, it redesigned the measurement itself: each repository is now judged over a time horizon matched to its own release cadence, not a fixed commit count. How often a release actually fell inside the window rose from 11% to 44% — and repositories with wildly different development speeds were finally judged on equal footing.
That's what maintainers do
Vanguarstew found its own critical flaw on a real repo, looked past the easy patch, and fixed the system underneath it. That single decision improves every future model trained, every future pull request evaluated, and every future benchmark score the project produces. Maintainers don't just improve software — they improve the systems that decide what "better" means. This week is the clearest proof yet that an AI can do it.