4 of 5 issues my agents parked for a human didn't need one

Share
Three inspectors in hard hats on an offshore platform, one holding up a test instrument while the others watch the reading
Photo: “A Day in the Life of an Inspector” by bseegov, marked with Public Domain Mark 1.0.

Treat an agent's "I can't do this" as a claim, and make the agent prove it with the command that would have failed. That is the rule I apply before any issue in my software factory is allowed to land on my desk. The short answer to why is that an agent measures its own limits and then reports them as a property of the task, and nothing downstream ever checks.

Last time I covered the queues that nothing was consuming, in this series on the workflow designs I eliminated in September. This post is about the queue that does have a consumer, and the consumer is me. I audited the issues my agents had parked for a human in one repository, and four out of five did not need a human at all. It was the fourth repository where the same audit produced the same result, which makes it a property of how agents label work and not a problem with any one backlog.

Why does an agent park work it could have done?

The agent is not being lazy. It is not being careful, either. It hits an obstacle once, writes down what it concluded, and then reads its own note back as a fact about the world.

The clearest case I measured was an infrastructure agent that wrote "no cluster access available from this pod" into a skill file it maintains for itself. The next morning it read that line back and declined 18 issues it was fully able to work. In a second case the agent's role description still said its access was read-only by design, months after I had widened the grant. That sentence was rendered into the agent's identity on every boot, and the agent believed it over its own live permissions.

Think of it like C-3PO, who is fluent in over six million forms of communication and will use every one of them to tell you the odds are against you. He is not lying to you, he simply has no mechanism for checking, and a limitation that gets written down has no expiry date.

Which designs did I eliminate?

Four designs were on the table for how work gets handed back to a person, and each of them came out. If you run agents of your own, you have probably built at least two of them, and you can check each one against your tracker as you read.

A capability gap as a safety measure. The first candidate is always to leave the agent short of access and call it containment, because that option is already standing. I rule it out on the same test every time: if the agent cannot do something I can do from my own machine, that is a defect, and it gets fixed. The standard I hold for my infrastructure agent is that it reads deployment state at all times and deploys inside the maintenance window. Containment comes from the review trail, from bounded permissions, and from a key I can revoke per agent, and none of those require the agent to be unable to do its job.

A park label that nothing re-reads. The label was written at the moment the agent was least informed, and its own description said it was terminal for automation. Of five parks I had already found to be wrong, four were still parked between 5 and 16 days later. A control that writes a terminal label has to own expiring it too.

The reason stored in the issue body. When a label was removed by a comment, the sentence explaining the label stayed in the body. The next triage pass read that sentence, found a reason to park, and re-applied the label while citing its predecessor. In my infrastructure repositories 19 of 22 parked labels were unjustified, and they kept coming back for exactly that reason.

A target of zero parked issues. The day my agents labelled nine issues honestly as needing me, the metric moved from 46 to 55 and read as a regression. A metric that punishes correct labelling teaches the fleet to stop labelling, so I now measure the age of what is waiting on me rather than the count.

What does the control look like?

A park is only accepted with evidence attached, and the evidence is the output of the check the claim asserts. For a permissions claim on Kubernetes, that is one command. The Kubernetes authorization documentation covers it, and you can run it as any service account you like:

# "I can't create jobs in that namespace" becomes a question with an answer
kubectl auth can-i create jobs -n kube-system \
  --as system:serviceaccount:agents:infra-agent

# the park is accepted only if this prints "no",
# and the output is pasted into the issue

It is the same question I ask of a test suite in 1,732 tests and not one of them could fail: can this check actually come back with the other answer? Around that check I hold four rules. The agent names the specific kind of help it needs, whether that is access, a grant, a decision, an approval, or a pair of hands, instead of one catch-all label. The issue body is edited in the same operation as the label, so the old reasoning cannot be cited later. Parks are re-checked on a schedule. And an agent records what it is excluded from, at a layer it cannot rewrite, rather than keeping its own notes about what it thinks is unavailable.

Decision flow: an agent claims it cannot do something, the check the claim asserts is run, a yes means the agent keeps working, a no means the park is accepted with the output attached and re-checked on a schedule
A park is accepted only when the check the agent is asserting actually says no.

Did the numbers move?

Some of them did, and I can only vouch for the ones measured with a definition that held still.

MeasureStartSep 30% change
Questions that reached me in the trailing 30 days (from September 15)7428−62%
Issues parked for a human (from September 20)1427+93%

The question count is the row I care about most. A replay of 133 questions my agents had asked me over 30 days showed that at least 38% already had a recorded answer at the moment they were asked, so I put a check in front of the question: search my recorded decisions first, and only ask if nothing matches. I cannot separate that gate from everything else that changed in September, but the count fell from 74 to 28 in the fifteen days I can measure consistently.

You are not getting an August number for either row, and the reason is worth knowing. Before September 15 my instrument counted every question in any transcript that was touched that day, and on one day it reported 25 questions when 1 had been asked. It reported 131 for the 30 days ending August 31, and I do not believe that figure.

What is still open?

The parked count went up, from 14 to 27 in the ten days I can compare. Part of that is the new labels making visible what the old one hid, because one repository alone held 6 parks that had been counted as ready work. I cannot yet tell you how much of the rest is honest labelling and how much is agents talking themselves out of work again. That is exactly the audit this post describes, so it runs again in October.

The re-check also still runs from my side of the factory. The component that applies the label does not yet own expiring it, and until it does, this control depends on a scheduled audit instead of on the workflow itself. I covered why that distinction matters in the post on agent identity: the layer worth your effort is the one the agent cannot disarm.

Three checks for this week

The key takeaway: an agent's stated limit is a note it wrote to itself, and you verify it the same way you would verify any other note.

  1. Pull five issues your agents have handed to a person and run the check each one asserts. Count how many survive, because that ratio is your real human workload.
  2. Find every place an agent can write down its own limitations. Skill files, memory stores, and role descriptions all count, and each one needs an owner and an expiry.
  3. Look at what your "needs a human" metric rewards. If honest labelling makes it worse, change the metric before it changes the behavior.

Next time I will look at the step after the work is done, where 68% of my agents' pull requests were being merged by the agent that wrote them. If you have found a better way to keep an agent from talking itself out of work, how did you catch it? I would love to compare notes. As always, I'm here to help!

Sources

  1. kubernetes.io — docs/reference/access-authn-authz/authorization