203 runs, 15 pull requests, zero actions

Judge every scheduled job in your agent fleet on three numbers side by side: what it scanned, what it classified, and what it did. A job that completes its run and acts on nothing is the most expensive status a job can report, because it buys silence. I apply that test to every janitor I run, and the specimen that wrote the rule is a job I call pr-doctor.

pr-doctor does one thing. Every few minutes it looks at each open pull request in my repositories, sorts it into a class (green, behind, conflicting, red because of the author's own defect, duplicate), and routes the red ones back to whichever agent wrote them, with a 60-minute clock. It is the piece that stops a red pull request from sitting overnight. It has failed four different ways, and only the first one was loud.

How did the same job fail four ways?

In the first week of October, pr-doctor failed 201 of 207 runs. Two callers raced to mint a credential at the same moment, one lost on a write conflict, and the job exited with an error and zero actions. That failure was the good kind. It was red, it was counted, and somebody could see it. I covered the loud-versus-silent distinction in A job that fails is loud. A job that stops running is silent. This job went through both halves of that sentence in one month.

After that, the runs started completing. That is where the quiet failures started.

The last row is the one worth reading twice. Fifteen pull requests were open. A pull request cannot be in zero classes. Four of them were red and mergeable for between 44 and 59 hours and classified nowhere, and the 60-minute alert never fired once, because the state file behind it held an empty object on every run.

Where did the 19 quiet days come from?

pr-doctor asked the server for a commit's age through a route that does not exist on my version of Forgejo. The server answered 404. The job's error handler read a 404 as “this pull request is gone” and moved on to the next one, so every pull request in every repository was dropped after being counted as scanned. It had done nothing since the day it shipped, 19 days earlier.

The tests passed. They passed because the fixtures mocked the missing route and returned a body shaped like a GitHub response, which is what the author expected a commit lookup to look like. A test double that returns what you hoped the server would say will confirm that your code handles what you hoped the server would say.

Think of it like a night guard who signs the log every hour and never leaves the booth. The log is perfect. Nobody checks whether the guard walked the building.

What are the controls?

I specified five, and each one closes a hole that this job went through. They live in an acceptance list on the issue, and my agents work from that list:

• A 404 means “gone” only when it comes from the pull request's own read. A 404 from any helper call is an error, logged with its path and counted.
• A self-check on the shape of the result. When a run scanned more than zero pull requests and every class count is zero, the run prints an error and exits non-zero, so the job turns into an incident on its first tick instead of its nineteenth day.
• Fixtures copied from the live server. The fake returns exactly the keys I measured on the real endpoint and nothing else, so a test double can no longer invent a response.
• The 60-minute alert reads the git server, not the state file. A pull request whose oldest red status is over an hour old, with no fleet comment or review since, breaches whatever the job's own memory says.
• Deploy proof is a pasted tick. The fix is not done until one real run shows non-zero class counts for the pull requests that are open at that moment.

That last control is the one I care about most, and it is the cheapest. It asks the job to show its work once, on live data, before anyone calls it fixed. The same instinct is behind the audit in 1,732 tests and not one of them could fail: ask whether the check can come back with the other answer.

The self-check is the control you can copy today, and it is small. You do not need to know why the job found nothing. You only need it to refuse to call an empty answer a success when the input was not empty:

def assert_classified_something(scanned: int, counts: dict[str, int]) -> None:
    # A sweep that scanned work and sorted none of it did not run.
    if scanned > 0 and sum(counts.values()) == 0:
        print(f"[error] classified nothing out of {scanned} items", file=sys.stderr)
        sys.exit(2)  # exit 2 is an incident in my cron wrapper, not a quiet pass

Put the same guard in front of every sweep you run: a lint job, a stale-branch reaper, a dependency bot. If the scanned count is above zero and every bucket is empty, you want the job to fail on its first tick, loudly, while the cause is still one change away.

What should you count in your own janitor jobs?

Pick the job in your fleet whose whole purpose is to find problems, and read its last 50 outputs. If scanned is above zero and every other number is zero for days, you have found a job that cannot find problems. Real repositories always contain a few red builds, a few stale branches, a few conflicts. A clean sweep that never varies deserves the same suspicion as a test suite that never fails.

If you have a janitor that has been suspiciously calm, let me know what it turned out to be. I'd love to see the tick that gave it away.