Does it maintain like a real senior maintainer?

A senior maintainer doesn't just write code. They read a pull request and know, fast, whether it fits where the project is going. They catch the two-line fix that quietly breaks a contract nobody wrote down. They know which open issue actually matters this week. That judgment is the hard part of maintaining software — and it's exactly what vanguarstew is built and scored on, not on writing good code in isolation.

What vanguarstew actually does

vanguarstew is a repository-maintainer agent running live on its own repo, in the open. It reviews pull requests, decides merge or close, applies a value label, and — on approve — merges the work itself. A human (matedev01) holds admin and the kill switch, but day to day, the agent is making the calls.

It's been doing this since day one. In the nine days since the first commit, vanguarstew has reviewed and merged over 500 pull requests from more than 50 contributors — and every one of those review decisions is public, on the repo, right now.

Nine days, five milestones

Briefly, what shipped:

• M0 — Agent contract. A fixed solve() entrypoint: infer the project's philosophy, read the current situation, plan, decide, implement.
• M1 — Time-travel replay. Freeze a real repo at a past commit, let the agent decide what happens next, then compare against what a real maintainer actually did.
• M2 — Scoring + leakage hardening. A judge that scores direction and reasoning, an objective anchor that checks concrete facts, and defenses against the agent simply having memorized the answer.
• M3 — Generalization. Proof the agent isn't tuned to one repo: it holds up on repos it was never optimized against (generalization_gap = 0.097).
• M4 — Hardening. Zero crashes from malformed model output across a full benchmark run.
• M5 — Measured, anti-gaming scoring. A pull request's value label is now earned from a measured benchmark delta — never a maintainer's read of the diff. See the leaderboard for every real result.

The benchmark: real open-source history, not a synthetic test

The scoring loop replays actual GitHub history. Freeze a repo at commit T, rebuild only what was knowable at that moment (nothing from the future leaks in), have the agent decide what a maintainer should do next, then judge that decision against what really happened. It runs against a curated public set of real repositories — hatch, pluggy, feedparser, httpx, hpack, h2 — plus a second, undisclosed set the agent has never seen, so a change can't be tuned to look good on repos it was optimized against and then fail everywhere else.

Where the score stands today

The composite score blends two very different signals, and the honest answer to "does it act like a senior maintainer" depends on which one you're asking about:

The judge is a pairwise comparison: given the real trajectory of the project, did the agent's plan move in the right direction, for the right reasons? Right now that score sits near 1.0 — on the repos measured so far, the agent's judgment about what a maintainer should do and why essentially matches what a strong maintainer actually did.

The objective anchor is stricter: did the agent name the exact modules a real PR touched, predict the right labels, call the version bump correctly? This is the part that separates "good instincts" from "gets the specifics right without having to think." That score sits around 0.3–0.4 today — real, but well short of a senior maintainer's precision on every mechanical detail.

So: not yet a senior maintainer across the board. A strong directional instinct, real and repeatable — paired with a specifics gap that's exactly what M6 and beyond are aimed at closing.

The goal: 99%, on both axes

The target isn't a good direction most of the time. It's a maintainer-level call — judgment and precision — approaching 99%, on repos it has never seen, in both the public and the private benchmark target at once. That's a high bar deliberately: a maintainer who's right about direction but wrong on the details still creates work for someone else to clean up.

Every step toward that number now has to be earned, not claimed. Since M5, a pull request to the agent's own code only gets credit when scripts/score_pr_delta.py measures a real improvement — against both repo targets, with an anti-cheating pass checking the PR isn't just special-casing the benchmark. A regression on either target blocks the merge outright. No score, no label — that's the whole anti-gaming point.