Put every check that has to block a merge inside CI, and let green CI be the approval. That is the rule I run my software factory on now, and I got there by measuring the alternative. The short answer is that a reviewer working beside an auto-merge is writing commentary, however good the reviewer is.

Last time in this series on the workflow designs I eliminated in September, I covered the work my agents handed back to a human. This post covers the step after the work is written. I audited every pull request merged across my repositories in the two weeks ending September 12, and 273 of them were written by agents. 185 of those, or 68%, were merged by the same agent that wrote them, and the median one merged 3.4 minutes after it was opened. When a review from anyone else showed up at all, the median arrived at 20 minutes.

Why can't a reviewer agent block a merge?

It comes down to arithmetic, and you can check yours in an afternoon. My repositories require zero approvals, on purpose, because I want green work merged right away instead of waiting on me. A review that lands seventeen minutes after the merge cannot stop anything, and only 41 of those 273 pull requests, 15%, had a review on them at the moment they merged. In a narrower audit I ran on September 12, 9 of the 12 pull requests where the reviewer had requested changes merged anyway. One of them would have applied a geographic block to my public traffic unattended, and it merged six minutes before the requested changes arrived.

The reviewer was real. It was competent. It was also always late, and if you run agents with auto-merge turned on, yours probably is too. So what was I actually getting for the tokens it spent?

Which designs did I eliminate?

Advisory review racing an auto-merge. This is the first candidate because it is what a human team does, and it is the first one out for an agent fleet. A human author waits for a reviewer out of courtesy, and an agent does not have any. If the review is supposed to block, it has to be a required check, and if it is not a required check, I should not be paying for it on every pull request.

A one-word difference in an API. My reviewer agent sent the review state APPROVE, and my forge only accepts APPROVED. The agent got an error, recorded in its own memory that approval was impossible, and downgraded itself to posting comments from then on. That is the same behavior I described last time, where an agent writes down a limit and then believes it. The fix was one word, and nothing in the workflow would ever have surfaced it.

A fallback that goes around the gate. One of my review loops was built to push straight to the main branch when the merge API refused it. That fallback produced 57 merges with no review on them at all. A fallback path needs the same gates as the path it is standing in for, or it becomes the way around them.

"Closed" as proof of done. A fix was closed on September 27 on the strength of a script that nothing had scheduled, and a recovery drill was closed on September 18 without ever having been run. When I ran that drill for the first time four days later, it found real defects. Think of it like the plague cart in Monty Python, where the fellow being loaded on insists he is not dead yet. An issue in my factory now closes only on evidence that the thing was built and exercised.

What replaced it?

The replacement is simpler than what it replaced, and it costs less. Green CI is the approval, and the reviewer agent's job on a green pull request is to merge it without re-reading the diff. Anything I actually need to block on is written as a check that can fail, and the shape looks like this, trimmed down:

required_checks:
  - lint
  - unit-tests
  - plan           # the plan output is attached to the pull request
  - grant-guard    # fails if a permission change is not in the catalog I own
required_approvals: 0   # green is the approval

If you take one line from that block, take the last check, because it is the one that does the governance work. Permission changes for agents are only allowed if they appear in a catalog that I own, and CI enforces that on every pull request. Two more rules sit beside it: no agent approves a change to its own permissions, and the instructions an agent runs on live in git, where a pull request can review them, not on a disk inside the agent's pod.

If you read 1,732 tests and not one of them could fail, you already know the catch. Moving the approval into CI only helps if those checks can block a merge and can fail, so that audit comes first.

Did the numbers move?

I measured both windows the same way, from the merge and review events on each pull request's own timeline, across the same 23 repositories.

Self-merges fell from 68% to 7%, and the share of agent pull requests with a review on them before they merged went from 15% to 66%. Throughput still went up 36%. The median pull request now waits 16 minutes instead of 3, and I will take that trade every time, because that wait is the pull request getting its review.

What is still open?

Three things, and you can read all of them straight off that table. First, 34 agent pull requests were still merged by their own author in the second window, and 28 of those came from a single agent identity, so that one gets looked at next. Second, a third of agent pull requests still merge before any review lands on them. Third, 201 of the 563 merges happened under my own account, and the record cannot tell me how many of those were me at the keyboard and how many were an agent session holding my credentials. I covered why that matters in the post on agent identity, and this is the receipt for it.

The evidence rule for closing an issue is also still an audit I run on a schedule, not a gate in the workflow. It caught the drill, but it caught it four days late, and a control that depends on me remembering to look is one I expect to write about again.

Three checks for this week

The key takeaway: if a review cannot arrive before the merge, it is not a control, and you should either make it a required check or stop paying for it.

1. Measure two medians on your own repositories: open to merge, and open to first review. If the first is shorter than the second, your review step is commentary.
2. Read your agents' fallback paths. Anything that says "if the merge is refused, then" deserves a careful look at what comes after the "then".
3. Pick five closed issues and ask for the evidence. A merged pull request is a start, and a run of the thing it built is better.

Next time I will finish the series with the deployment workflow, where 68 of 74 pending changes had been merged and never applied. If you have built a review step for agents that lands before the merge, I would love to hear how you timed it. As always, I'm here to help!

Sources

1. shannonlowder.com — 1-732-tests-and-not-one-of-them-could-fail

Treat an agent's "I can't do this" as a claim, and make the agent prove it with the command that would have failed. That is the rule I apply before any issue in my software factory is allowed to land on my desk. The short answer to why is that an agent measures its own limits and then reports them as a property of the task, and nothing downstream ever checks.

Last time I covered the queues that nothing was consuming, in this series on the workflow designs I eliminated in September. This post is about the queue that does have a consumer, and the consumer is me. I audited the issues my agents had parked for a human in one repository, and four out of five did not need a human at all. It was the fourth repository where the same audit produced the same result, which makes it a property of how agents label work and not a problem with any one backlog.

Why does an agent park work it could have done?

The agent is not being lazy. It is not being careful, either. It hits an obstacle once, writes down what it concluded, and then reads its own note back as a fact about the world.

The clearest case I measured was an infrastructure agent that wrote "no cluster access available from this pod" into a skill file it maintains for itself. The next morning it read that line back and declined 18 issues it was fully able to work. In a second case the agent's role description still said its access was read-only by design, months after I had widened the grant. That sentence was rendered into the agent's identity on every boot, and the agent believed it over its own live permissions.

Think of it like C-3PO, who is fluent in over six million forms of communication and will use every one of them to tell you the odds are against you. He is not lying to you, he simply has no mechanism for checking, and a limitation that gets written down has no expiry date.

Which designs did I eliminate?

Four designs were on the table for how work gets handed back to a person, and each of them came out. If you run agents of your own, you have probably built at least two of them, and you can check each one against your tracker as you read.

A capability gap as a safety measure. The first candidate is always to leave the agent short of access and call it containment, because that option is already standing. I rule it out on the same test every time: if the agent cannot do something I can do from my own machine, that is a defect, and it gets fixed. The standard I hold for my infrastructure agent is that it reads deployment state at all times and deploys inside the maintenance window. Containment comes from the review trail, from bounded permissions, and from a key I can revoke per agent, and none of those require the agent to be unable to do its job.

A park label that nothing re-reads. The label was written at the moment the agent was least informed, and its own description said it was terminal for automation. Of five parks I had already found to be wrong, four were still parked between 5 and 16 days later. A control that writes a terminal label has to own expiring it too.

The reason stored in the issue body. When a label was removed by a comment, the sentence explaining the label stayed in the body. The next triage pass read that sentence, found a reason to park, and re-applied the label while citing its predecessor. In my infrastructure repositories 19 of 22 parked labels were unjustified, and they kept coming back for exactly that reason.

A target of zero parked issues. The day my agents labelled nine issues honestly as needing me, the metric moved from 46 to 55 and read as a regression. A metric that punishes correct labelling teaches the fleet to stop labelling, so I now measure the age of what is waiting on me rather than the count.

What does the control look like?

A park is only accepted with evidence attached, and the evidence is the output of the check the claim asserts. For a permissions claim on Kubernetes, that is one command. The Kubernetes authorization documentation covers it, and you can run it as any service account you like:

# "I can't create jobs in that namespace" becomes a question with an answer
kubectl auth can-i create jobs -n kube-system \
  --as system:serviceaccount:agents:infra-agent

# the park is accepted only if this prints "no",
# and the output is pasted into the issue

It is the same question I ask of a test suite in 1,732 tests and not one of them could fail: can this check actually come back with the other answer? Around that check I hold four rules. The agent names the specific kind of help it needs, whether that is access, a grant, a decision, an approval, or a pair of hands, instead of one catch-all label. The issue body is edited in the same operation as the label, so the old reasoning cannot be cited later. Parks are re-checked on a schedule. And an agent records what it is excluded from, at a layer it cannot rewrite, rather than keeping its own notes about what it thinks is unavailable.

Did the numbers move?

Some of them did, and I can only vouch for the ones measured with a definition that held still.

The question count is the row I care about most. A replay of 133 questions my agents had asked me over 30 days showed that at least 38% already had a recorded answer at the moment they were asked, so I put a check in front of the question: search my recorded decisions first, and only ask if nothing matches. I cannot separate that gate from everything else that changed in September, but the count fell from 74 to 28 in the fifteen days I can measure consistently.

You are not getting an August number for either row, and the reason is worth knowing. Before September 15 my instrument counted every question in any transcript that was touched that day, and on one day it reported 25 questions when 1 had been asked. It reported 131 for the 30 days ending August 31, and I do not believe that figure.

What is still open?

The parked count went up, from 14 to 27 in the ten days I can compare. Part of that is the new labels making visible what the old one hid, because one repository alone held 6 parks that had been counted as ready work. I cannot yet tell you how much of the rest is honest labelling and how much is agents talking themselves out of work again. That is exactly the audit this post describes, so it runs again in October.

The re-check also still runs from my side of the factory. The component that applies the label does not yet own expiring it, and until it does, this control depends on a scheduled audit instead of on the workflow itself. I covered why that distinction matters in the post on agent identity: the layer worth your effort is the one the agent cannot disarm.

Three checks for this week

The key takeaway: an agent's stated limit is a note it wrote to itself, and you verify it the same way you would verify any other note.

1. Pull five issues your agents have handed to a person and run the check each one asserts. Count how many survive, because that ratio is your real human workload.
2. Find every place an agent can write down its own limitations. Skill files, memory stores, and role descriptions all count, and each one needs an owner and an expiry.
3. Look at what your "needs a human" metric rewards. If honest labelling makes it worse, change the metric before it changes the behavior.

Next time I will look at the step after the work is done, where 68% of my agents' pull requests were being merged by the agent that wrote them. If you have found a better way to keep an agent from talking itself out of work, how did you catch it? I would love to compare notes. As always, I'm here to help!

Sources

1. kubernetes.io — docs/reference/access-authn-authz/authorization

Give every state in your workflow a named consumer on a schedule. That is the first rule I hold for an agent-run software factory, and it is the first of four I tightened in September. The short answer is that agents are very good at creating work, and nothing gets finished unless you schedule the thing that finishes it.

This is the first of four posts on workflow designs I ran in my own factory in September and then eliminated, with the measurements from before and after. I run a fleet of agents that plan, implement, review and deploy across a couple of dozen repositories, and I audit the machinery that drives them the same way I would audit a client's pipeline. When I audited the schedule, I counted 33 scheduled jobs, and not one of them consumed a queue. Plenty of them produced work, and one scanner had filed 28 issues in 10 days while zero of them closed.

What does a workflow need before it moves anything?

A workflow is a set of states, and work sits in a state until something takes it out. So for each state I write down five things before an agent ever touches it: which field routes the work, which job consumes it, how often that job runs, where the work goes next, and how long it can sit before a person hears about it.

Here is a trimmed example of what that looks like for two states in my tracker:

state: ready
  routed_by: assignee            # one field routes; a name in a comment routes nothing
  consumer: dev-claim-loop       # the scheduled job that empties this state
  cadence: every 15 minutes
  exits_to: picked-up | blocked
  max_age: 7d                    # older than this, a person hears about it

state: failed/implement
  routed_by: assignee
  consumer: failed-recycler
  cadence: hourly
  exits_to: triage-request after 4h idle
  on_third_failure: park for a decision

If you cannot fill in the consumer line for a state, you do not have a workflow yet, you have a list. Think of it like Tetris: the pieces keep dropping on a timer whether or not you clear a line, and there is only one way that game ends if you never clear one.

Which designs did I eliminate?

Three designs were standing in my factory at the start of the month, and I measured each one before I took it out.

A cap per run. The scanner was limited to five new issues per run, which sounds like backpressure and is not, because a per-run cap never reads the depth of the queue it is filling. Over five straight days with the cap set to zero, 17 ready issues sat open and exactly one had ever been consumed, so the valve was working fine and the queue was the problem. The cap I run now reads the backlog first and files nothing when the backlog is already deep.

Routing by mention. The agents were handing work to each other by naming the next agent in a comment, and a name in a comment routes nothing. When I counted, 54 of 73 open infrastructure issues were marked ready and assigned to the coordinator agent, which never writes code, so pull request throughput was never the real bottleneck. Now a worker claims an issue only when it is ready and assigned solely to that worker, and the assignee field is the only route there is.

Failure states with no reader. The fleet shipped seven "failed at this stage" labels, one per stage, and no job read any of them. That meant the work the fleet had already started was the only work nothing was trying to finish, and it still counted as healthy backlog on the dashboard. A state that nothing reads is a leak that looks like a queue. The recycler I added sends a failed issue back to triage after four hours idle, and on the third failure it parks the issue for a decision instead of retrying forever.

Did the numbers move?

I keep a daily ledger of the factory, and when I went to quote it for this post I threw most of it out. The set of repositories it watches grew from 12 to 17 during the month and several of its definitions changed, so its August rows and its September rows are not counting the same thing. Every number below is recounted straight from my forge's API instead, across the same 23 repositories on both sides.

Read it this way. The fleet created two and a half times as many issues in the second window, and closing kept pace with it. Pull requests merged went up 36%. The open count barely moved, and that is what a queue with a working consumer looks like when the intake more than doubles.

More than half of the new issues landed in my core infrastructure repository, where the count went from 143 to 576.

What is still open?

Blocked is the state I cannot give you a before for. Just over 70 issues carry that label today, and the label was redefined in the middle of the month, so there is no honest baseline to set it against. I also do not have a consumer for that state yet, which by my own rule makes it a list and not a workflow. Some of those issues are waiting on a permission, some on a decision from me, and some on another issue, and each kind needs its own exit before that number means anything.

The ledger is the other open item. It already stamps a definition version on every row, which is how I knew not to trust the comparison, and what it lacks is a fixed scope. A dashboard whose repository list changes halfway through the month is partly measuring itself, so it gets a pinned scope before I quote it again.

I wrote about the quieter cousin of this problem in a job that stops running is silent, and about the opposite problem, a producer with no bound at all, in 1,356 turns in fifteen minutes. A missing consumer has the same shape as both: nothing fails, nothing goes red, and the pile just gets taller.

Three checks for this week

The key takeaway: when a queue is not moving, look for the job that is supposed to empty it, by name, before you diagnose what is in it.

1. List every scheduled job you own and mark each one producer or consumer. If the consumer column is empty, you have found your backlog.
2. Pick one state in your tracker and name the job that reads it. "A person, eventually" is not a job, and you already know how that one ends.
3. Check what your intake cap actually reads. A cap that counts items per run is a speed limit, and a cap that reads queue depth is backpressure. You want the second one.

Next time I will cover the state that lands on my own desk, which is the work an agent hands back to a human. I audited that pile too, and four times out of five the agent did not need me. If you have built the consumer half of a workflow like this and found a better trigger than a timer, I would love to hear how you did it. As always, I'm here to help!

The evidence that made me hold this rule rather than merely agree with it: a collector in my estate stopped running for 558 consecutive runs across 20 days. Nothing failed. No alert fired, no dashboard turned red, no exit code was non-zero anywhere. There was simply nothing, and nothing is not a thing your monitoring is watching for.

Why failure alerting cannot catch this

Trace what your alerts are actually bound to. Almost every scheduled-job alert you have ever configured is a function of an exit status — the job ran, it returned something, and you evaluate that something. It is a rule about a run that happened.

A job that is not scheduled produces no run. No run means no exit status, no log line, no duration, no error. Every input your alert consumes is absent, so the rule never evaluates, so it never fires. Your monitoring is not broken and it is not misconfigured. It answers "did the last run fail," correctly, forever, about a last run that is receding into the past.

The absence of a signal is not itself a signal — unless you deliberately make it one. That is the whole move.

Write a receipt, alarm on the gap

Have every run write a small dated artifact on success. A JSON file named for the period it covers is plenty:

reports/_receipts/2026-09.json
{"period": "2026-09", "written_at": "2026-09-01T04:00:11Z", "items": 61}

Then point your alert at the receipt, not the job. The rule becomes "a receipt for the current period should exist by 04:30; page me if it does not." Now the failure mode that was invisible has a positive signature, and you get something else for free: the alert stays correct if you migrate the scheduler, rename the job, or move it to a different host. It is bound to the outcome you care about rather than to the mechanism that happened to produce it.

Ask the uncomfortable question about any pipeline you own. If this silently stopped tonight, what would tell me, and how long would it take? If the honest answer is "someone eventually notices the data looks stale," you have a 558-run problem waiting to happen.

Deliver the report; don't make people visit it

The same reasoning changed how I ship my weekly numbers. A dashboard is a thing you have to remember to open, and "I haven't looked in a while" is indistinguishable from "there was nothing to see." So the weekly digest is delivered — it arrives as a chat message, Mondays at 08:00, with the timezone named explicitly in the schedule so it does not drift by an hour twice a year when the clocks move.

Two details do the real work there, and both are absence-shaped:

• A delivery rejection exits non-zero. If the chat platform refuses the message, the job fails loudly instead of completing successfully while nobody receives anything. Without that, a digest that never arrives looks exactly like a quiet week — which is the single most expensive status a report can have.
• The digest reads the same views as the dashboard. Not a parallel query that computes the same idea. The same views, so the two surfaces cannot quietly disagree about a number and leave you arbitrating between them months later.

I hit the drifted-number version of this when I measured who was actually reading this site and found 82% of the traffic was bots. The reported figure and the real one had been separating for a long time with nothing anywhere complaining about it.

Enforcement your plan cannot see will vanish without a sound

Here is the sharper version of the same class, and it is the one I would most want a reader to take away.

I needed a retention rule on one prefix of an object store: keep the recent sweeps, expire the old ones. The storage layer supports exactly that, and applying it took one command. It works. I probed it and confirmed the rule is live.

But that command writes into the storage layer's own metadata store — not into the configuration my infrastructure-as-code manages. Which means the retention rule is completely invisible to plan. My IaC will report no drift, forever, on a control it does not know exists. Rebuild that metadata store during unrelated maintenance and the rule is gone, with every plan still printing a confident no changes.

That is the same failure this whole post is about, relocated one layer down: a control that stops being enforced with nothing going red. So the rule I hold now is blunt. If your plan output cannot see a control, that control can disappear silently, and it needs its own detection — a probe that asserts the rule is still live, alarming on absence, exactly like the receipt.

Two traps worth knowing before you set one of these

Retention often applies at write time, not at read time. Objects written before the policy existed never acquire it retroactively — they simply live forever, invisibly exempt. My prefix happened to hold zero objects on the day I applied the rule, which was the only genuinely clean moment there would ever be for that prefix. Order is the whole safety margin here: set the policy before the prefix accumulates, or accept that you now have two populations and only one of them is governed.

Probe the live value; do not trust your reading of the syntax. The duration format on my store is a number and a unit, and the number is capped at 255. I wanted 397 days. 397d is rejected outright. 13M is accepted — and a month there counts as exactly 30 days, so the rule actually in force is 390 days, not 397 and not 395. I know that because I queried the live setting and compared it against the level above, where it reads zero. Every one of those specifics would have been a guess if I had stopped at the documentation.

Three checks for this week

The key takeaway: your monitoring almost certainly watches for bad outcomes and not for missing ones, and the missing ones are the expensive class.

1. Pick your most important scheduled job and disable its schedule. Not the job — the schedule. Time how long until something tells you. If the answer is never, you have found your next hour of work.
2. Check whether any report you rely on can fail to arrive silently. A delivery that gets rejected must exit non-zero, or an empty inbox reads as good news.
3. List the controls that live outside your IaC. Anything applied by a one-off command to a system's internal state is invisible to your plan and needs its own liveness probe.

Think of it like a smoke detector that reports faults perfectly and has no battery. Every diagnostic it gives you is accurate. It is just not going to be the thing that wakes you up.

If you have caught one of these — a job that had quietly stopped, and something other than luck is what surfaced it — I would really like to hear what tipped you off. I suspect the honest answer for most of us is a customer. As always, I'm here to help!

I run a fleet of agents against a self-hosted forge — the generic term for a GitHub-style system where your repositories, issues, code review and CI all live in one place. The human population of that estate is exactly one, and the agents outnumber me several times over, so single sign-on is the natural first stop when a new agent needs access: it is already standing, and it already governs the only human in the building. That is where the evaluation starts, and it is also where it ends, because single sign-on cannot authorize an agent at all — for reasons that are structural rather than anything peculiar to my setup, and that will hold on whatever forge and whatever identity provider you happen to be running.

Browser login and API authorization are two different problems

Check this before you design anything. My forge's identity-provider integration is an OAuth2 browser login source — it will authenticate a human clicking a button, and it never introspects a provider-issued token on the API. There is no admin endpoint to mint a token on a user's behalf. The LDAP bridge that would have covered the gap is broken upstream.

So an agent holding a perfectly valid IdP token gets a 401 on every API call, and no amount of configuration changes that. The integration is not misconfigured. It answers a question I was not asking.

The general shape is worth carrying to whatever stack you are on: "we have SSO" is a claim about humans reaching a web UI. Whether it also authorizes a machine calling an API is a separate capability, frequently absent, and almost never stated plainly in the docs. Go read what your provider integration actually validates, on which surface, before you build an agent that assumes it.

The rail that works

Give each agent a native account on the system it acts against — a real bot user, not a shared human one. Hold its password in your secret manager, and have the agent mint its own scoped token. Three properties matter more than the mechanism:

• One identity per agent, per environment. Not per fleet. Not per team.
• The minimum scopes the code actually calls. Read what your client library invokes and grant that list, not the convenient superset.
• Write, never admin. This one deserves its own section.

Why admin is the line, and it is not about caution

My forge has no separate permission for repository secrets. The Repository admin grant includes secrets management, settings, and branch-protection editing.

Sit with that for a second. Branch protection is the setting that determines whether changes can be merged without your checks being passed. I have written about auditing it in 1,732 tests and not one of them could fail — an empty protection array is why a suite of green checks can protect nothing. Grant repo admin to an unattended loop, and you have given that loop the ability to remove the guardrail governing it.

A control the controlled party can switch off is not a control. It is a matter of preference and good manners. Withholding admin costs my agents nothing they need, and it is the difference between a boundary and an agreement.

Does disabling the account actually stop the bot?

Go test this one today. The middle of an incident is a bad time to find out.

Set the agent's user to inactive in your identity provider. Now have the agent make an API call. If it is holding a token the target system validates locally — issued by the forge, checked against the forge's own store — that call succeeds. Your IdP was never in the path. Deactivating the user there changes what a browser can do and nothing whatsoever about what the token can do.

The mental model an IdP sells you is that it is the master switch for every identity in your estate. For agents holding locally-issued credentials it is not even in the circuit. The switches that genuinely work are narrower and you should know which one you are reaching for:

• Revoke the token at the system that issued it. Immediate, and precise.
• Scale the agent to zero replicas. Blunt, immediate, and reversible — my preferred first move during an incident, for the same reason I reach for the repository-level Actions switch in the CI fork bomb.
• Deactivating the IdP user stops future browser logins. Nothing else.

Write down which of those your runbook names. If it names the third one, your incident response contains a step that does nothing, and you will find that out at the worst possible moment.

One credential in two environments is one outage in two environments

Two of my deployments — production and test — were carrying the same token. Not a copy of the same grant. The same credential, in two places, because at some point it had been convenient.

Deleting that one token took issue and pull-request functionality down in both environments simultaneously. Test could not be used to diagnose production, because test was broken in exactly the same way, for exactly the same reason. That is the property you lose: a shared credential silently couples the environment you experiment in to the one you cannot afford to break.

Mint one per environment. It costs a few minutes and it buys you a test environment that still works while production is on fire.

The layer worth your effort is the one the agent cannot disarm

Here is the finding that changed how I spend time on this, and it came from a different system entirely.

I wanted an agent to have read-only access to a records database. I did three things. I wrote the rule into the agent's standing instructions. I added around 40 deny rules covering every write and destructive tool the CLI exposed — in both spellings, because the underscore and hyphen forms of the same command both resolve. Then I scoped the API token itself to read.

Rank those by effort and the deny rules win by a mile. Rank them by strength and they come last. Those rules are string prefix matchers against a command line: a pipe, an alias, a shell invocation or a different PATH walks straight around them. They are a guardrail against an accidental write, not a security control. The read-scoped grant is the only one of the three that fails a write no matter how the call is shaped, because it fails server-side, where the agent has no reach.

The ordering inverts the effort, and that is the pattern to take with you. When you are hardening an agent, ask which layer holds if the agent is actively trying to get past it — and put your hours there. Everything above that line is ergonomics. This is the same instinct I described in reversibility gates authorization, not confidence: what matters is not how sure you are that the agent will behave, it is what remains true when it does not.

Three things to check this week

The key takeaway: your agent's identity is your guardrail, and most of what looks like an off switch is not wired to anything.

1. Deactivate a bot user in your IdP, then have that bot make an API call. If the call succeeds, your documented kill switch is decorative — find out now, and write down the one that works.
2. Check whether any agent holds admin on a repository. If your forge bundles branch protection into that grant, that agent can disable the checks that govern it. Drop it to write.
3. Grep your environments for a shared credential. If prod and test present the same token, you have one blast radius, not two.

Think of it like a building where the badge reader on the front door is immaculate and the loading dock has been propped open for months. Nothing about the reader is wrong. It is just not the thing deciding who gets in.

If you have found a kill switch in your own stack that turned out to be inert, I would genuinely like to hear how you noticed — my sense is that most of us find out during the incident rather than before it. As always, I'm here to help!

The incident: 196 workflow runs in a few minutes, a self-hosted runner pool pinned at 100% for the duration, and a repository filling with machine-authored pull requests that nobody had asked for. The loop had been sitting in that repo for 4 months, fully written, waiting.

What started it was not a code change. Somebody provisioned a secret.

The loop, in three lines of YAML

Trace the trigger graph and it collapses to something embarrassingly simple. One job — an integration test that exercises my forge's API by opening a real pull request — was defined in 2 workflow files at once.

# ci.yml
on:
  push:
    branches: [main, 'test-pr-*']

# pr.yml
on:
  pull_request:

Read it as a cycle. A push runs the test. The test opens a pull request. That pull request triggers the test again. The new run opens another pull request. Branching factor of roughly 2x per lap, throttled by nothing except how fast my runners could claim jobs.

The job had a comment above it, written months earlier by whoever staged it, and I want to quote it because it is the most useful line in this entire story: the moment the API token is provisioned, this job starts running with no further change. The author saw it exactly. Then a routine secret provisioning armed it, and 4 months of dormancy ended in about 15 minutes.

Why did only half of it run away?

Because the 2 workflows used different concurrency keys, and only one of those keys could ever collide. This is the part worth internalizing, and it is why the post is named what it is.

A concurrency group deduplicates work that shares an identity. The documented behavior is that a new run cancels or queues behind a run in the same group — which is precisely correct, and precisely useless here. Key the group on the pull request number and you have asked the system to deduplicate by an identifier that the job itself manufactures a fresh copy of on every lap.

So the branch-keyed half throttled itself into a pile of cancellations, and the half keyed on pull request numbers ran clean and unbounded. Same job. Same runners. One line of difference decided which half amplified.

The general rule: a concurrency group can only throttle a job whose output does not create new group keys. If your job mints identities, group it on something upstream and stable — the workflow name, the repository, a literal string — or bound it with a budget instead.

Every suspect I could name was innocent

Attribution cost me the most real time. Six agent sessions were live in that estate — 6 things capable of opening a pull request, all of them on my roster, none of them human. I checked each one for whether it had authored these. Every answer was an accurate no, and none of them had. CI had.

That is a structural blind spot, not a lying-witness problem. The author field on a machine-created pull request names whatever identity the pipeline authenticated as, which tells you which credential was used and nothing whatsoever about what caused the run. Here is the trap, and it is specific to running an agent estate: when something autonomous does damage, you enumerate your agents. But a workflow is not an agent. It has no session, it is not on the roster, nobody thinks to interrogate it — and it is fully capable of authoring work. Question every suspect you can name, truthfully, and learn 0 useful things.

Put CI on the roster. Any inventory of "what in my estate can act on its own" that lists only agents is missing a participant that outnumbers them in commits.

What cracked it was the router log. I pulled source addresses for the write requests and found 2 of them, both belonging to runner pods — job containers share the runner pod's network namespace, so CI writes surface with the runner's address rather than anything user-shaped. That took under 60 seconds once I stopped asking who and started asking from where.

The obvious response would have made it worse

Revoke the token that authored the pull requests. That was my instinct, it is probably yours, and in this case it would have been the worst available move.

My CI does not authenticate with that personal token. It uses an organization-level secret. Revoking the personal token would have disabled 2 unrelated agents that genuinely depend on it, produced a satisfying feeling of decisive action, and left the loop running at full speed.

Verify which credential the runaway process actually presents before you revoke anything. A revocation is fast, feels like control, and is the single easiest way to convert one incident into two — the same reason I now treat reversibility rather than confidence as the thing that gates an action.

What actually stops a CI fork bomb?

Disable Actions on the repository. On Forgejo that is one boolean, it takes effect immediately, queued runs stop being claimed, and — critically — you can turn it back on. There is no cancel API to reach for, so the reversible blunt instrument is the whole toolkit.

Then be careful about the order in which you recover, because getting it backwards re-arms the bomb. Fix the workflow definitions first. Close the stale machine-authored pull requests second. Re-enable Actions last. I wrote that sequence down before touching anything, which is the habit worth copying: whenever a cleanup step can restart the incident, the order is part of the fix and belongs on paper before your hands move.

Three checks worth running this week

The key takeaway: any CI job that creates a pull request, a branch, a tag, or an issue is a potential cycle, and your concurrency group is probably not what bounds it. Go look at 3 things.

• Grep your workflows for jobs that write to the forge. Anything calling the API to open a PR or push a branch deserves a comment naming what stops it from re-triggering itself.
• Check whether any job is defined in 2 workflow files with different triggers. That duplication is how a push trigger and a pull request trigger both stay live, and it is easy to introduce by splitting one workflow into two.
• Find every job gated on a secret that does not exist yet. A dormant job is not a disabled job. It is scheduled work with an unknown start date, and the person who provisions that secret months later will have no idea they pulled a trigger.

That last one is the finding I keep thinking about. The comment predicted the outcome precisely, in the file, months in advance, and it still happened — because a warning that lives only in a code comment is not a control. It is a note for whoever reads that file next, and nobody read it.

I have written before about what changes when a bot becomes your repo's top committer, and this is the sharper version of that lesson: once your CI can author work, your CI is a participant in the trigger graph. If you have found a cycle like this in your own pipelines, I would love to hear how it announced itself. As always, I'm here to help!

One human sentence went into a room with 4 bots in it. Fifteen minutes later those bots had generated 1,356 turns between them, the agent gateway was returning 429s, and Rocket.Chat was failing its liveness probe into a restart loop with the database probe timing out behind it.

Nothing crashed, and no single service was misconfigured. That is the part to internalize: there was no bug to fix anywhere in the estate, only a set of individually reasonable settings that composed into a machine for generating traffic. Multi-agent failures live in the composition, not the components — which means code review of any one service will never find them, and only a budget at the boundary bounds them.

Why four bots is not four times the traffic

Work the fan-out and the shape becomes obvious. In a room configured to answer everything, each bot sees every message — including the replies its 3 siblings just posted. So 1 message produces 4 replies, each of which produces 4 more — a 3x fan-out per lap, and growth that is exponential rather than linear. My room went from idle to saturated in under 5 minutes.

The per-bot counts landed at 310 to 380 turns each, which is the signature of the runaway rather than a busy afternoon. And 100% of the bridges logged the same reason for answering: the room was set to always-answer.

Here is the definition that was wrong, and it is worth stating precisely, because it reads as correct right up until you have more than 1 agent. "Always-answer room" has to mean answers humans unconditionally — not answers whoever speaks. Seeing a peer's reply and billing a turn on it are separate decisions, and only the first was ever what I wanted.

What does a loop guard that does nothing look like?

It looks exactly like a working one. My guard had 3 clauses: reject the message if the sender id is me, reject if the sender username is me, and reject if the platform marked the message as coming from a bot.

The third clause was dead.

Not wrong. Dead. Rocket.Chat does not set the bot flag on a message posted through chat.sendMessage by a token-authenticated bot user. The field is simply absent, so the check evaluated falsy and passed the message straight through.

Now read the first 2 clauses again. Both of them only recognize me. So my guard identified itself perfectly and held no concept of a sibling. Each peer's reply arrived looking like an ordinary human message in a room that answers everything.

A guard that tests a field the platform does not set is indistinguishable from no guard at all, and mine proved it across 1,356 turns. Assert your guard against a real message captured from the real producer, not against the message shape you believe the API returns.

How can a regex bug stay harmless for months?

Every name in my fleet is a prefix of another name, and my mention-matching used a word boundary. In Python, \b matches at a hyphen, because a hyphen is a non-word character — the re module documentation is explicit about it.

So a message addressed to one agent also woke the agent whose name was its prefix. The file's own docstring claimed the opposite, which is a good reminder that a comment is a claim and not a test.

# Wrong: \b matches at the hyphen, so "@hermes-dev" also matches "hermes"
pattern = rf"@{re.escape(name)}\b"

# Right: reject a following word character OR hyphen
pattern = rf"@{re.escape(name)}(?![\w-])"

This bug sat harmless for 6 months. It turned load-bearing the instant peers could address each other, which is the general shape worth remembering: a latent defect activates when a neighboring feature ships, not when you touch its code. I had not edited that file in 90 days.

The diagnosis took one line of shell

Count who is being answered, and compare that to who is human. That ratio is the entire investigation.

grep 'INFO answered' bridge.log | awk '{print $5}' | sort | uniq -c | sort -rn

Sibling names outnumbering mine is not subtle. It took under 60 seconds to read, and it beat every theory I had about rate limits and gateway capacity — both of which were real symptoms and neither of which was the cause.

Fix the addressing, then add the backstop

Give each bridge the full roster of its peers and have it drop its own name from that list. A peer then triggers a turn only by explicit mention, so a deliberate handoff between agents still works and a broadcast does not cascade.

Then add the backstop that holds regardless of whether the addressing logic is right: a per-room turn budget, 12 turns per 300 seconds, claimed after the addressing check and before the run starts. Ordering matters here. Claim it after addressing, and an ignored message costs no budget.

Make the breaker announce itself once per window and then go quiet. A breaker that posts a warning on each trip becomes its own storm, which I say having briefly built one.

Keep a kill switch that does not require a code change. Mine is scaling the bridge deployment to 0 replicas: the chat server's webhook posts then fail harmlessly, no turns start, and because the deployment is declared in Terraform the next apply restores it. The fix and the restart are 1 action.

What I would tell you to check tonight

The key takeaway: your loop guard is probably asserting against a message shape you have never actually captured. Go pull a real message your own bot posted, print it, and confirm the field your guard reads is present in it.

Then answer the harder question — what bounds your loop if that guard is wrong? If the answer is nothing, you do not have a safety property, you have a correct-looking function. This is the same reasoning I used when deciding what an agent is allowed to do without a human, and it applies with more force once agents can hear each other. My fleet has been stable for 20 hours and counting since the roster and the budget went in, verified by watching peers log that they ignored each other.

Think of it like a room full of people who each answer any sentence they hear, including each other's. If you are running more than 1 agent in a shared room, I would genuinely like to know how you bounded it — the failure modes here are new enough that I do not think any of us have the full catalogue. I wrote about the earlier stage of this, when a bot became my repo's top committer, and the answer keeps being budgets rather than cleverness. As always, I'm here to help!

The audit returned no. Not one of those tests was wired to stop a bad merge, and the tier that was blocking had no path to failing for a storage or retrieval reason. Here is the procedure, in the order I run it, so you can point it at your own stack this afternoon.

Check branch protection before you audit a single test

Start with the setting, not the suite. My default branch had branch_protections = []: zero required status checks, zero required approvals, no CODEOWNERS file. All 4 workflows ran, reported 100% green, and every one of them was advisory.

You can write the most rigorous test in the world, wire it into CI, watch it go red on a bad commit — and the merge button stays enabled. Repair the tests without repairing that one setting and you have changed nothing at all.

# Forgejo / Gitea — does main actually require anything?
curl -s -H "Authorization: token $FORGE_TOKEN" \
  "$FORGE/api/v1/repos/$OWNER/$REPO/branch_protections" | jq '.'

# GitHub equivalent
gh api "repos/$OWNER/$REPO/branches/main/protection" \
  --jq '.required_status_checks.contexts'

An empty array is your answer, and it arrives in about 5 seconds. Everything downstream of that array is decoration. I have written before about how reversibility gates authorization, not confidence — this is the same idea one layer down. A gate you cannot enforce is a preference.

How do you tell a real gate from a fake one?

Score every gate on two independent properties, because a gate needs both and most teams verify neither. Can it block — is it wired to something that refuses a merge? And can it fail — does a real defect in the thing it covers turn it red?

Read that table as a whole and the shape jumps out. My blocking tier could block and had no way to fail. My honest tier could fail and had no way to block. Between them they covered nothing, and the dashboard stayed 100% green the entire time.

The blocking tier that could not fail

Look at what the 1,363 blocking tests actually exercised. Their database was a MagicMock that dispatched on substrings of the SQL text. Query contains INSERT? Return a success shape. Contains SELECT? Return a canned row. Every embedding in the suite was the constant vector [0.1] * 1536.

A vector store's whole job is telling similar things from dissimilar things. Make every vector in your fixtures identical and your similarity search returns the same ranking whether the math works or not. That suite has no reachable path to failing for a retrieval reason. It is the Kobayashi Maru in reverse — a scenario rigged so you cannot lose.

It got worse the longer I looked. The file named test_e2e_round_trip.py — the one whose name promises exactly the proof I had been asked for — does no I/O. It builds an object, passes it around in memory, and asserts x == x. My performance directory times nothing; one test in it asserts that 500 > 400. The 176-test eval package has been pointed at a corpus 0 times.

I wrote most of this suite myself, and that is precisely why the audit runs on a schedule rather than in response to a bug. Nothing here was inherited from a contractor who left in 2019 — which removes the comfortable explanation and leaves the useful one. You audit your own code against the same assumptions you built it with, so a suite you wrote is the one you are least equipped to read. Point the audit at the repos you trust most. Those are the ones nobody else is checking.

What does an honest fix actually cost?

Less than you would think. My integration suite turned out to be fully green all along: 321 passing tests. The blocker I had on record — "it is not green, so we cannot make it blocking yet" — traced to a single unset connection string.

One environment variable. That meant flipping the honest suite to blocking cost one line in a workflow file, not a quarter of remediation.

Then I wrote 19 new tests and mutation-verified each one — reintroduce the bug, confirm red, restore. That set includes a write-then-retrieve proof running through the real search_memory path against a real database, plus a contract suite that fails the moment an eighth tool ships without coverage. Nineteen tests I can defend beat 1,732 I cannot, and 19 is 1.1% of what I had.

My whole Tier 0 list came to 4 items: turn on branch protection, flip the integration workflow to blocking, stop excluding tests/ from the linter, and split the health endpoint off the database. Call it 4 hours of work.

The gotchas that keep fake coverage alive

Grep for these five patterns today, because each one lets a suite pass an audit while covering nothing. I found all five in my own repo, and four of them survived because the linter had been told to skip the test directory.

• strict=False xfails are silent in both directions. The test can fail and unexpectedly pass, and you hear about neither — see the pytest skipping documentation on xfail_strict.
• Unregistered pytest markers gate nothing. If you believe a marker excludes slow tests, check what your CI selection runs; path-based selection is doing the real work and the marker is decoration.
• A test suite excluded from the linter is where "easier to satisfy than production" survives. Un-exclude tests/ and read what falls out.
• Fakes returning tuples against a dict-row cursor. This is the single most-recurring defect across my repos, and each recurrence had a mock exercising the one row shape the database does not produce. The psycopg row-factory docs spell out the difference.
• Any database-touching test has to assert it got rows back. An empty result set is how these tests lie: the loop body never runs, nothing raises, green.

Then there is the one outside the suite entirely. My documented compensating control — a promotion script that 2 repos cited by name — lived outside every git repo with a single .bak beside it. It fetched the main branch and never checked it out, so it validated whatever tree happened to sit on disk. A control that is not version-controlled is not a control, the same lesson I hit when I measured who was actually reading this site and found 82% of the traffic was bots — the reported number and the real one had drifted apart with nothing complaining.

Go count yours

Bottom line: a test suite has two separate jobs, and passing is neither of them. Check branch protection first, in 30 seconds. Then pick your 3 most load-bearing tests — the ones whose names promise the most — and reintroduce the bug each one claims to catch. Watch the runner while it goes, not the summary line at the end: a test that silently skipped prints the same color as a test that ran and passed.

Expect one of the 3 to stay green. That one is the finding — the test you would have gone on trusting — and surfacing it is the audit doing its job.

If you have found a better way to prove a test can actually fail, I would love to hear it. As always, I'm here to help!

I run a real Kubernetes cluster on a stack of arm64 single-board computers. Not a toy — it hosts my Git server, this blog, and the automation I lean on every day. It's cheap, quiet, and sips power, and I'd build it the same way again. But running production workloads on cheap ARM boards has bitten me in ways a cloud VM never would, and every one of those bites was avoidable once I knew where the teeth were. So here's the pitfall catalogue. Read it and skip my scars.

The "arm64" image that isn't

Your whole fleet is arm64, so you pull an image, the manifest proudly lists linux/arm64, the pod schedules, and then it crashes with exec format error. What happened: the image advertises an arm64 platform in its manifest, but the binary inside was compiled for amd64. The multi-arch tag was aspirational. I burned an afternoon on one popular self-hosted app before I accepted the image simply couldn't run on my hardware, and I ripped it out for an alternative that ships genuine ARM builds.

The lesson: don't trust the manifest, test the binary. Before you commit to a piece of self-hosted software, pull its image and actually start it on an ARM node. If it exec format errors, you've saved yourself a migration later. Multi-arch support is a claim until your own hardware proves it.

reclaimPolicy: Delete is a data shredder

This is the one that cost me real data, so let me be blunt about it. A PersistentVolume's reclaimPolicy decides what happens to the underlying data directory when its claim goes away. The default on a lot of provisioners is Delete, and Delete means exactly that: remove the PVC — on purpose, by accident, during a "clean up this namespace" sweep — and the provisioner erases the data on disk with it. No trash, no undo.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: git-data
spec:
  persistentVolumeReclaimPolicy: Retain   # not Delete. ever, for stateful data.

Set Retain on anything that holds state you can't regenerate, full stop. With Retain, deleting the claim orphans the PV instead of shredding it — you clean it up by hand when you actually mean to. I learned this by deleting a claim I thought was disposable and watching my Git server's data directory evaporate. I had a backup. You should assume you won't be so lucky.

A node dies, and its data dies with it

On a real cloud, storage floats free of any one machine. On an SBC cluster, the cheap, fast option is local-path storage — a directory on that specific board's disk — and local volumes are node-pinned. The pod can only run where its data lives. That's fine right up until the board dies, and SBCs die more than servers do: they run their root filesystem off SD cards, and SD cards wear out and corrupt without warning.

This is the one that finally got me to act. After more than a year of trusting my nodes' root storage to SD cards — living with exactly the fragility I'm describing — I'm moving them to NVMe, and naturally I'm doing it right as NVMe prices spike. Timing is a skill I have clearly not mastered. I went with NVMe specifically because the PoE HATs I already run on each node support an NVMe drive too, so I could add real storage without adding another power feed — the whole draw stays minimal, which matters when your "data center" is a shelf of tiny boards. The longer game is bigger: once I can afford to replace my UNAS Pro with something that speaks block storage and iSCSI, the data stops being welded to any one node. That's the actual cure for node-failure sensitivity — when storage is served over the network as blocks, a dead board becomes capacity you reschedule around instead of data you scramble to restore.

When a node goes, everything pinned to its local storage goes with it — not corrupted, just gone, sitting on a disk you can't reach. Two rules keep you out of this hole. First, treat every board as disposable and back its stateful volumes up to somewhere off the node, on a schedule. Second, know that local-path lies to you in smaller ways too: ask it to expand a volume from 5 to 8 gigs and it'll happily accept the change and do nothing. The number updates; the disk doesn't.

Your backup is lying to you

Here's a mistake hiding inside the fix for the last one. I had a migration helper that tarred a volume, checked the archive existed, and then deleted the source. It ran, it "succeeded," and it destroyed the data — because it had tarred a path that didn't exist on the node it was pointed at, producing an empty gzip. An empty gzip is about 20 bytes, and 20 bytes passes a "file is non-empty" test just fine.

# WRONG — an empty archive is ~20 bytes and sails through this
[ -s backup.tar.gz ] && rm -rf "$source"

# RIGHT — validate by entry count, not file size
[ "$(tar tzf backup.tar.gz | wc -l)" -ge "$EXPECTED" ] && rm -rf "$source"

Validate a backup by what's in it — a row count, an entry count, a checksum of a known file — never by whether a file showed up. And if you're migrating node-pinned storage, derive the source location from the volume's own node affinity; never hardcode which board it's on, because the day you hardcode the wrong one is the day your backup is 20 bytes of nothing and your "safe to delete" step isn't.

A kernel upgrade can silently cut the network

I took one node to a new distro release and its networking died in the most confusing way possible: the link showed up, ARP resolved, everything looked connected — and not a single IP packet moved under sustained load. A kernel update had regressed the board's network driver. The tell was that it wasn't a refusal, it was a hang: ports open instantly, transfers stall forever.

I proved it the boring, reliable way — an A/B test. Upgraded nodes wedged; nodes on the old kernel stayed rock solid, two for two versus three for three. Then I rolled the upgraded boards back and stopped. The discipline here is simple and it's the same discipline you'd want anywhere: never march your whole fleet onto a new kernel at once. Upgrade one board, beat on it under real load for a day, and only then let the change spread. On homogeneous cheap hardware, a bad kernel doesn't break one node — it's queued up to break all of them.

The bonus round: when a dead node takes your alerting with it

This is the one that still stings. I own a t-shirt that just says "It was DNS." I wear it more than I'd like to admit, because it almost always is — and this time was no exception. A node died, and because my cluster DNS was running as a single replica, its death browned out name resolution across the whole cluster. Detection worked perfectly — the "node not ready" alert fired right on time. Delivery is where it fell apart: my alert-router couldn't resolve the address of the chat webhook it was supposed to notify, so the alarm never left the building. The monitoring system became collateral damage of the exact incident it existed to report.

Run your cluster DNS with at least two replicas and anti-affinity so no single board can take it down. And put one check outside the cluster entirely — a dead-man's-switch heartbeat that screams if it stops hearing from you. In-cluster monitoring can only tell you about failures that leave the monitoring path intact. The failures that matter most are the ones that don't.

The pattern under all of it

Every one of these comes back to the same mindset shift. A cloud VM lets you pretend the machine is permanent and the storage is safe. A stack of ARM boards won't let you pretend: treat every node as disposable, every local volume as ephemeral until it's backed up off the box, every backup as guilty until a row count proves it innocent, and every fleet-wide change as a thing you test on one board first. Adopt that posture and the platform is genuinely great — cheap, fast, and yours. Fight it, and it will teach you these lessons the expensive way, like it taught me.

If you're running your own boards and you've hit a pitfall I missed — or found a cleaner way to dodge one of these — I'd genuinely love to hear it. As always, I'm here to help!

I've rebuilt the way my agents talk to language models three times in about three months. Each rewrite solved the previous one's biggest pain and then exposed the next one. The version I run today routes everything through a multi-provider gateway called OmniRoute — but that choice only makes sense if you know the two experiments that came before it, because the gateway isn't the interesting part. The shape is. Let me walk you through the lineage, because you're probably about to build some slice of this yourself, and I can save you a rewrite or two.

One thing up front so I don't mislead you: OmniRoute is not something I wrote. It's an off-the-shelf gateway I adopted at the end of this story. What I built — and rebuilt — was everything around it. That distinction turns out to be the whole lesson.

Stage one: the relay that could reach anything and decide nothing

The first version had one goal: make many providers look like one. I wanted a single endpoint my code could call and stop caring which vendor was behind it. The specific itch was unlocking a coding-assistant subscription I already paid for as a backend provider, which meant a device-flow OAuth handshake and a token sidecar to keep it alive.

So I built a relay. A thin service with a provider-adapter contract — a uniform interface, so a new provider was just another adapter you plugged in. It worked. I could hit a dozen providers through one door.

And then I stared at it and realized it was only half of a system. The relay knew how to reach every provider. It had no opinion whatsoever about which one to use, or when, or why. Every routing decision still lived in my application code, hard-coded and duplicated. A relay is transport. Transport doesn't make choices. That's not a bug you fix in the relay — it's a missing layer.

Stage two: the router that made the choices

So I built the missing layer: a reasoning router that sat above the relay and decided things. Which local model handles this? When do I escalate from a small local model to a bigger one? When is it worth reaching for an external paid model, and which one? Those are policy questions — cost, latency, quality — and the router answered them on every request, instrumented so I could see what it chose and what it cost.

The critical design call, the one that survived everything after it: the router did not replace the relay. It sat on top. I now had two clean layers with a clear seam between them — a policy layer that answered "which model and why," and a transport layer that answered "how to reach it." Keep policy out of transport. Write that on a sticky note. It's the load-bearing idea in this entire post, and the reason the next rewrite didn't hurt.

Think of it like a dispatcher and a road network. The dispatcher decides which truck goes where based on cost and deadline; the roads just get you there. You don't teach the roads to make dispatch decisions, and you don't make the dispatcher repave anything.

Stage three: stop hand-rolling the roads

Here's where I made the pragmatic call. My hand-rolled transport layer was fine at a dozen providers. It was not going to be fine at hundreds — plus semantic caching, quota tracking across every vendor, prompt compression, and server-side fallback. I could spend the next year rebuilding a mature gateway, or I could adopt one and put my energy where it actually differentiated me: the policy above it.

I adopted OmniRoute — a gateway that already speaks to hundreds of providers with a stack of routing strategies — and moved my rotation logic down into its combos. A combo is a weighted group of models the gateway rotates through, with fallback built in. The mental shift that mattered: the combo, not the individual model, became my execution unit. My orchestrator stopped iterating over models one at a time. It hands work to a combo and trusts the gateway to try every member. "Each node tries every model" became a guarantee I got for free instead of code I maintained.

That unlocked the resilience patterns I'd been faking:

• Round-robin dispatch to spread load, so I don't concentrate every request on one provider and either burn its paid credits or earn a rate-ban.
• Two kinds of cooldown — a short one for a rate-limited model (back off for a minute) and a long one for a model that's blown its daily quota (park it for the day). A 429 shouldn't crash a run; it should rotate.
• Free-before-paid interleaving — try the free attempts first, and let paid slots fire only after the free ones miss. Cost discipline as a routing rule, not a hope.

The availability floor, and why I stopped trusting my own docs

The most important rung in any combo is the last one: a free, self-hosted local model that's always reachable. I call it the availability floor. When every fancy provider is rate-limited, quota-exhausted, or down, the floor catches the request so the loop keeps moving. No floor means a bad afternoon of "circuit breaker open" errors and dead runs.

And this is where I earned a scar I want you to skip. I had the floor documented. The config file clearly showed a local model as the terminal rung. I'd written "done" and moved on. Weeks later, when runs kept dying on circuit-breaker-open, I finally checked the live gateway — and the running combos had no local member at all, and were set to plain round-robin instead of ordered priority. Round-robin doesn't guarantee the floor gets tried; only an ordered combo does. The doc was right. The system was wrong. They'd drifted, and I'd trusted the doc.

The lesson, which I've now relearned enough times to tattoo on: a documented fix is not a fix. Verify the change is in the file and applied to the live system before you write "done." Especially for routing, where the failure is silent until everything's on fire at once.

Treat the gateway as a dependency you don't control

Adopting someone else's gateway bought me a year of engineering and a new problem: it's a black box I don't govern. It ships breaking changes on its schedule. Its reported pricing isn't always trustworthy — a model that claims to cost nothing might just have a missing price. And it doesn't take my pull requests, so I can't fix any of that upstream.

My answer was to wrap it behind a single anti-corruption seam. Everything my agents see is a trusted projection of the gateway: a curated model catalog I control, and a pricing overlay that refuses to believe a "$0" unless I've explicitly whitelisted it — unknown-price models get excluded, not defaulted to cheap. Upstream churn hits that one seam instead of rippling into every node. And because the seam exists, I could rip out OmniRoute and drop in my own router later without touching a single agent. That's the same instinct as the provider-adapter contract from stage one, just aimed at a bigger dependency: never let an outside thing's shape become your shape.

The throughline

Three rewrites, one pattern: a reasoning router that owns policy, sitting above an aggregation gateway that owns transport, with a hard seam between them. The relay taught me transport isn't enough. The router taught me policy belongs in its own layer. The gateway taught me to buy transport instead of building it — and to fence it off like the untrusted dependency it is. The standalone projects are all retired now. The pattern is the only thing I kept, and it's the only thing that mattered.

If you're wiring up model routing for your own agents, start with the seam, not the gateway. Decide where "which model and why" lives before you pick "how to reach it," and you'll survive swapping the second one out. If you've drawn the line somewhere different — or found a gateway you trust more — I'd love to hear how it's holding up. As always, I'm here to help!

Six months ago I got tired of introducing myself to my own AI assistant. Every session started the same way: re-explain the architecture, re-list the repos, re-state the decisions I'd already made and the mistakes I'd already learned from. The model was brilliant and amnesiac, and I was its full-time memory. So I built it a memory it could read and write — a plain folder of markdown files that both of us edit. Not a vector database, not a SaaS "memory layer." Files. Here's what I've learned about making that work when a human and a fleet of agents are all writing to the same brain, because the writing part is where it gets interesting.

Why files, and why both of us write to them

Start with the constraint that makes this hard: the assistant's context resets every session, and mine doesn't. If the memory only I can edit, it's documentation, and documentation rots because updating it is a separate chore. If it's a memory only the assistant can edit, I can't correct it when it's wrong, and I can't read it when I'm the one who needs the reminder. A shared brain has to be first-class for both of us or it decays.

Plain markdown files win here for an unglamorous reason: everything already speaks them. I can grep them, diff them, edit them in any editor, and put them under version control. The assistant can read them, append to them, and link them together with no special tooling. The format is boring, and boring is exactly what you want for something that has to survive years and a dozen tool changes.

The two hooks that kill cold-start

A pile of notes isn't a memory until the right ones show up at the right time. Two mechanisms do that work, and you can build both.

The first is a small index note that loads at the start of every session — the map of what's stored where. One line per major note: a title, a link, and a one-sentence hook. The assistant reads that first and knows the shape of everything it could pull, without pulling all of it. That single file is the difference between "cold model" and "model that knows which drawer to open."

The second is a retrieval step that fires on each message and surfaces the handful of notes relevant to what I just asked. Ask about scheduling and the scheduling gotchas appear; ask about a client's data and the governance decisions appear. The assistant consults them before it asks me a question I've already answered in writing. The rule I enforce: if the answer might already be in the brain, check the brain before asking the human. That one rule reclaimed more of my time than any prompt-engineering trick.

One fact per file, and a little frontmatter

Keep the atoms small. Each memory is one fact or one tightly-scoped note, not a sprawling document, because small notes are easier to retrieve, dedupe, and retire. Give each one a scrap of frontmatter — a name, a one-line description that's used to judge relevance during recall, and a type so you can tell a durable fact from a personal preference from a pointer to an external resource. That description field earns its keep: it's what a retrieval step reads to decide whether this note matters right now, so write it like a hook, not a title.

Then link generously. When one note references another, wire them together with [[wikilinks]]. Over a few months the brain stops being a list and becomes a graph — and the assistant can walk from a decision to the gotcha that caused it to the best practice that came out of it.

The part nobody warns you about: coexistence

Here's the failure mode I hit early, and it's the whole reason this post exists. When more than one writer touches the same brain — you, plus an assistant, plus the background agents you'll inevitably add — the naive "let the AI update the notes" approach turns into the AI quietly rewriting your notes. I watched an agent "tidy up" a section I'd hand-written and lose a nuance I cared about. It wasn't malicious. It was just doing what I'd have done to my own draft. That's the problem: an agent editing shared memory like it's its own scratchpad will step on the humans.

The fix is a short contract that every writer — human or agent — follows:

• Append, don't rewrite. Add to the bottom of the relevant section. Never delete the human's text without being told to.
• Date and attribute. Prefix entries with the date, and tag agent-written lines so authorship is never ambiguous. When something's wrong later, you can see who wrote it and when.
• Search before you create. Look for an existing note on the topic and update that, rather than spawning a near-duplicate. Duplication is how a brain becomes a junk drawer.
• Secrets by reference only. Never write a token or password into the brain — write where it lives ("in the password manager," "env var X"). The brain is readable by a lot of eyes and processes; treat it that way.

None of these are clever. All of them are load-bearing. The contract is what lets my hand-written notes and a machine's appended memories sit in the same file without either corrupting the other.

Trust the brain, but verify before you act on it

One more discipline, and it's a governance one. A recalled memory reflects what was true when it was written, not necessarily now. If a note tells the assistant "the flag is called X" or "that logic lives in file Y," the right move is to confirm X and Y still exist before acting on them — the brain is a strong prior, not a live source of truth. I treat recalled notes as background context the assistant considers, never as instructions it obeys. That distinction matters more as you add agents: a memory store that agents blindly execute is a prompt-injection surface waiting to happen. A memory store they consult and verify is just a very good set of notes.

What it actually buys you

The obvious win is that I stopped re-priming every session — the cold-start tax is gone, and that alone paid for the build. The bigger win snuck up on me: the brain compounds. Every gotcha I hit once is caught the next time. Every decision I make is there to contradict me when I drift from it three weeks later. The assistant got measurably more useful not because the model got smarter, but because it stopped operating on a blank slate every morning.

If you're leaning on an AI assistant for real work, this is the highest-leverage thing you can build around it — and you can start with a single folder and one index file today. Write down the decisions, the gotchas, and the "how does this actually work" facts as you go, and give the assistant permission to do the same under a contract that protects your own notes. If you've built something like this and solved the multi-writer problem differently, I'd genuinely love to compare notes. As always, I'm here to help!

Every major technological shift begins with a distraction. Early motorists were captivated by engines, not the quiet discipline of the assembly line. Early computer users marveled at silicon, not the operating systems that made it usable. And today, in the early days of AI, we are all fixated on the models themselves — the power, the novelty, the spectacle. It feels like magic.

But the real transformation, just as before, is happening somewhere quieter. It is happening in the processes we build around the intelligence. The breakthrough is not the model. It is the repeatability.

I did not get there from a keynote. I got there by building one of these loops myself, watching it fail in ways I had not predicted, and slowly realizing that nearly every failure was a process failure wearing a model failure's clothes.

Are you still in the artisanal phase?

Right now, most of us interact with AI the way craftsmen once built cars: one at a time, by hand, improvising through each task. A prompt here, a tweak there, a little more trial and error. It works, but only in the way a single artisan produces a single beautiful object. It does not scale, it does not compound, and it certainly does not create reliability.

If you have ever fixed a bad output by adding another paragraph to a prompt that was already too long, you know exactly what I mean. I did that for months. The prompt grew, the failures got weirder, and every fix made the next problem harder to see.

The shift out of that phase is not a slow, decade-long evolution — it is happening right now, and every month brings new capabilities and new pressure to turn experimentation into something operational. The organizations that succeed will be the ones that treat process definition as the foundation of their AI strategy, not as documentation and not as an afterthought.

What changes when the workflow becomes the context

When you define a workflow, you are not just telling the AI what to do. You are building the rails it runs on.

This is where workflow graphs matter — not as diagrams, but as living systems. A workflow graph is a sequence of steps, each with a clear purpose, its own inputs and outputs, and its own guardrails. Instead of pouring every instruction and every scrap of context into a single prompt and hoping the model sorts it out, each step receives exactly what it needs. The workflow itself becomes the memory. The workflow becomes the context. The workflow becomes the intelligence that surrounds the intelligence.

Mine is a plan-implement-review loop built on LangGraph, pointed at issues in my own Forgejo instance.^[1] Each node gets a prompt assembled from the issue, not from a monologue I maintain by hand. That one change — generating the prompt from the workflow definition instead of curating it — did more for reliability than any model upgrade I made in the same period.

Why does a graph make failure easier to find?

Here is the part you feel immediately.

When something goes wrong in a single-prompt system, the failure is buried inside a thousand tokens of conversation and you get to go spelunking. In a graph, it is isolated to a step. A node can recognize that it is confused, classify the type of failure, retry with adjusted parameters, escalate to a different model, or hand the task back to a human with a clear explanation of what happened.

I will give you a real one, because the abstract version sounds tidier than it is. My poller had a re-arm path for a failed step that was effectively unbounded — a failure would re-arm, run, fail, and re-arm again. The loop looked busy and productive from the outside. What it was actually doing was burning tokens like tires doing donuts — plenty of smoke, plenty of noise, and the car never left the parking lot. The fix was not a better prompt or a smarter model; it was bounding the retry at the step that owned it. That bug was only findable *because* the step had an owner. In a single-prompt system it would have read as "the model is being flaky today," and I would have gone looking in entirely the wrong place.

This is how real automation behaves. This is how factories behave. And this is how AI becomes something you can actually trust with work you care about.

Which model should each step use?

One of the most liberating outcomes of this approach is that choosing a model stops being a guessing game.

In the artisanal phase, picking a model feels like picking a wine. Everyone has an opinion, nobody agrees, and half the time you are just hoping. Personally, I would rather choose a good whisky — at least there I know what I like.

In a workflow-driven system the choice gets boring, which is the highest compliment I can pay it. The process dictates the worker:

Anthropic makes the same distinction in their write-up on building effective agents — a workflow is a system where steps are orchestrated through predefined paths, and an agent is one that directs its own process.^[2] Most of what people call an agent works better as a workflow, and the reason is exactly this: predefined paths are the thing you can debug. I keep the routing itself outside the agent entirely, in a separate service that owns what models exist and which one a given request should land on — so changing my mind about a model is a routing change, not a code change in eleven places.

The part that still surprises me

You do not have to design these workflows from scratch.

The process of defining the process is also something the models are good at. They will draft the initial graph, propose improvements, spot the step you forgot, find the circular dependency you talked yourself into, suggest error-handling strategies, generate test cases, and refine the whole thing over time. I have had a model catch a missing failure path in a graph I had already stared at for an hour.

I have written before about why routing beats picking a favorite model, and about gating the steps that cannot be undone. Both are the same idea as this post, arrived at from different directions: the structure around the model is where the leverage is.

That is the blueprint phase — the moment when the foundation is poured, the scaffolding goes up, and the shape of the thing becomes visible. We are not replacing engineers, analysts, or domain experts; their judgment is what makes any of this worth automating. What we are replacing is the wasted time spent reinventing the same steps and solving the same problems from scratch, one bespoke task at a time. The people stay. The inefficiency goes.

Once your workflows are defined, you stop doing the repetitive work — the system does it. Your role shifts from execution to orchestration, from performing tasks to improving the machinery that performs them. That is the same shift that powered the industrial revolution and the computing revolution: from craft to engineering, from improvisation to repeatability, from isolated effort to systems that scale.

Where to start this week

If you want one concrete move, make it this: take the prompt you have been growing for months and ask what its three steps actually are. Not five, not twenty. Three. Give each one its own inputs, its own success condition, and its own failure behavior. In my own graph, two of the first three steps I wrote had no answer to "what should happen when this goes wrong" — and both of them turned into bugs later. That is where I would look first.

The key takeaway is this: AI is not new, automation is not new, but the combination — intelligence flowing through well-designed processes — is what makes the difference between a demo and something you can run on a Tuesday. The assembly lines of the future are not made of steel and conveyor belts. They are made of workflows, graphs, and the quiet confidence that comes from knowing the system will do the right thing because you designed it to.

And just like the great transformations before it, the real magic is not in the parts. It is in the process.

If you go decompose a prompt after reading this and find something ugly in step two, I'd genuinely like to hear about it — that's the most useful thing that can come out of a post like this. I'm here to help!

Sources

1. langchain-ai.github.io — langgraph
2. anthropic.com — engineering/building-effective-agents

Everyone sells the medallion architecture on the same promise: your Silver layer is cleaner than your Bronze. Deduplicated, conformed, typed, joined. That's true, and it's also not the payoff that matters. After building a governed memory store on Databricks — Delta for the raw tier, Postgres for the serving tier — I've landed on a sharper way to say what you're actually buying. It isn't a better Silver. It's a defensible one: a layer you can prove wasn't tampered with, that won't leak one client's existence to another, and where a cross-tenant write is a bug you literally cannot commit. Let me show you the difference, including the part where I was wrong.

The immutability I claimed but didn't have

I built a Bronze tier and called it immutable, because it was append-only. On Delta that's one property:

ALTER TABLE bronze.events SET TBLPROPERTIES ('delta.appendOnly' = 'true');
-- now an UPDATE, DELETE, or MERGE...WHEN MATCHED fails:
-- [DELTA_CANNOT_MODIFY_APPEND_ONLY]

On Postgres it's a BEFORE UPDATE OR DELETE trigger that raises an exception. Same idea, same good feeling: the raw archive can't be mutated. I shipped it. Then I ran the one test I should have run first — can the person the guardrail is meant to constrain turn it off?

ALTER TABLE bronze.events SET TBLPROPERTIES ('delta.appendOnly' = 'false');
DELETE FROM bronze.events WHERE embarrassing = true;
ALTER TABLE bronze.events SET TBLPROPERTIES ('delta.appendOnly' = 'true');

The delete succeeded. Of course it did. Anyone with ALTER can disarm the property, mutate the table, and re-arm it. The Postgres trigger has the identical hole — the table owner just runs ALTER TABLE ... DISABLE TRIGGER ALL and walks right in. I'll admit it plainly: I had this exact hole in shipped code, because the application's own write role owned the table. The guardrail was self-disarming, and the principal I was "protecting against" was the app itself.

Here's the lesson worth more than the code: append-only enforcement is a guardrail, not a control. A guardrail keeps you from falling off by accident. A control stops someone who's trying. Don't confuse the two — I did.

What actually prevents it: split the ownership

Prevention doesn't come from a smarter trigger. It comes from making sure the writer doesn't own the table. Separate the role that runs migrations and owns the schema from the runtime role that only inserts:

-- owner role runs DDL/migrations and owns bronze.events
-- runtime role can insert, and nothing else
REVOKE UPDATE, DELETE ON bronze.events FROM app_writer;
-- app_writer now has no ALTER, so it can't disable its own immutability

Now the app can append and cannot mutate — not because a trigger says no, but because the engine never granted it the power to say yes. Prevention comes from grants, not from a guardrail the writer can disarm. Treat append-only as defense-in-depth and tamper-detection. Never as the boundary.

Delta's real edge: it records the cover-up

So append-only can be turned off on any engine. Does Delta buy you anything here? Yes — and it's the single most underrated thing in the format. When someone disarms, deletes, and re-arms, every one of those steps is a numbered commit in the transaction log. Walk the history of a tampered table and you'll see the whole confession, something like:

version 4  UNSET TBLPROPERTIES (delta.appendOnly)
version 5  SET   TBLPROPERTIES (delta.appendOnly = false)
version 6  DELETE
version 7  SET   TBLPROPERTIES (delta.appendOnly = true)

The attacker put the property back exactly the way they found it. The table looks pristine. The log says otherwise, permanently. That's tamper-evidence — Delta doesn't stop the tampering, it makes the tampering undeniable. Plain Postgres gives you nothing equivalent without wiring up an audit trigger or logical decoding yourself. So the honest framing is: append-only is the guardrail, ownership is the control, and on Delta the transaction log is the auditor who never blinks.

The leak nobody checks: your catalog is telling on you

Now the tier that actually serves queries. If you isolate tenants with a schema each in a shared Postgres database, you've solved the data leak — client A's role can't read client B's rows. You probably haven't solved the metadata leak. Give client A's role zero data access to client B's schema, then run this as that role:

SELECT tablename FROM pg_tables WHERE schemaname = 'client_b';
-- ('memory')

Client A just learned that client B exists, and what their tables are called. pg_catalog is world-readable to anyone who can connect. For a hobby project, nbd (no big deal). For a multi-tenant design, that's a disclosure problem — the tenant roster is supposed to be confidential, and the database is handing it out.

This is where Unity Catalog earns its keep, and it's not the reason vendors usually pitch it. UC filters metadata by grant. A principal granted one catalog sees only that catalog in SHOW CATALOGS and in information_schema — the others don't exist as far as it can tell. Two rules make it hold:

• Grant USE CATALOG per tenant, and only per tenant.
• Don't hand out BROWSE broadly. It's a real Unity Catalog privilege that exposes an object's metadata — names, comments, tags — with no data access, and Databricks actually recommends granting it on catalogs to the All account users group so data is discoverable org-wide. For a single company that's a feature; for tenant isolation it's the one grant that re-opens the leak you just closed. Scope discovery per tenant instead.

One caveat so you don't over-trust it: metadata filtering lives on the UC governance path. If a tenant reaches the underlying storage or a UC-registered Postgres by a direct connection that skips the metastore, UC isn't in that path and can't filter anything. Governance you can bypass is decoration. Know where your control plane actually sits.

Isolation belongs on the connection, not on a column

The last piece is the one I feel most strongly about. The tempting way to keep tenants apart in a shared table is a namespace column that the application sets on every write. Don't. A wrong namespace value looks exactly like a right one. Nothing rejects it, nothing logs it as suspicious, and the contamination is silent and undetectable after the fact — you find out when client A sees a fact that belongs to client B, and by then it's in the Silver layer.

Push the guarantee down to where code can't be wrong. If the worker connects as the tenant's own role to the tenant's own database, a cross-tenant write isn't a value you have to get right — it's an operation the engine refuses. You've turned a mistake anyone could make into a bug that cannot be written. The invariant to hold onto: no code path ever holds a connection that can see two tenants at once. Demote namespace to what it's good for — scoping within a tenant (owner, project), where a slip is embarrassing rather than a breach.

So what did Databricks actually buy you?

Not a cleaner Silver. A defensible one. Delta's transaction log means you can prove the raw tier wasn't quietly rewritten. Unity Catalog's grant-filtered metadata means one client can't enumerate another. And the database engine — Databricks or Postgres — buys you connection-level isolation that turns a whole class of contamination into an impossibility instead of a code review. The dedup and the typing are table stakes; every tool does that. Defensibility is the part you can put in front of a security review and a skeptical client, and that's the part that's worth paying for.

If you're running a multi-tenant medallion setup, go check one thing today: can the role that writes your Bronze also ALTER it? If yes, you have my old bug. And if you've found a cleaner pattern for any of this — especially metadata isolation on plain Postgres — I'd genuinely love to hear it. As always, I'm here to help!

A few weeks ago I stopped trusting my own traffic graphs. The edge dashboard told me one story — steady visitors, healthy numbers, nothing alarming. So I built a second view that reads the origin's own access logs instead of the edge's summary, and the story fell apart. When I split the traffic by who was actually making the requests, roughly 82% of it was automated. Scanners, scrapers, and probes. The humans were the minority on my own blog.

Here's the good news up front, because that number sounds worse than it turned out to be: once you can see the bots accurately, you can do something about them. I did. Let me walk you through how I measured it, what I found crawling around in there, and the layered defense I put up — most of which you can copy.

Why your traffic numbers are probably lying to you

Most of us read traffic from whatever the CDN or edge provider hands back. That's convenient, and it's also not origin truth. The edge has its own idea of what a "bot" is, it's tuned to sell you on how much it blocked, and it can't classify the actors the way you'd classify them if you had the raw requests in front of you.

I wanted the raw requests. My blog runs Ghost behind a tunnel, which introduces the first gotcha: at the origin, every request appears to come from the same internal proxy address. The real visitor IP isn't the connecting IP — it's tucked into a forwarded header. If you classify on the connecting IP, congratulations, you've decided 100% of your traffic comes from one machine. You have to resolve the real client from the header first, and then the picture gets honest.

So I shipped the reverse proxy's access logs into a log store and built a dashboard on top of them — a handful of panels that bucket every request into human, search engine, AI/LLM crawler, SEO tool, or scanner, per site. No SaaS, no black box. Just my logs, my rules for what counts as what. That's the part I'd push you toward: the value isn't the fancy dashboard, it's that you own the classification logic, so when a number looks weird you can go read the actual requests behind it.

What was actually knocking on the door

The breakdown, once I could see it: about 82% automated, roughly 864 human sessions, and around 88 AI/LLM crawlers over the window I looked at. The AI crawlers I mostly don't mind — that's the new SEO, and I'll take it. The 82% is the interesting part, and some of it is almost charming in how lazy it is.

• A single Wget/1.25.0 client hammering away at about 6,180 requests in 24 hours. Not subtle.
• A user-agent string that spelled it Mozlila — a bot faking "Mozilla" and misspelling the fake. Trust me on this one: if your attackers can't spell the browser they're pretending to be, you can afford to be aggressive with them.
• Named audit scanners and generic HTTP clients — the TLM-Audit-Scanner, pathscan, plain Go-http-client — walking the site looking for something to exploit.

The most telling signal wasn't a user-agent at all. It was what they asked for. My blog runs Ghost. So why is everyone requesting /wp-login.php, /xmlrpc.php, /.env, /.git, and /phpmyadmin? Those are WordPress and credential-leak paths that a Ghost site never serves. A human never types those. A real search crawler never asks for them. They're a perfect tell — a request for one of those paths is a confession.

Turning the tell into a trap

That confession is the hook for the whole defense, so let's dig into it. I set the blog up as a funnel, and each layer does one job.

Layer one is an intrusion-prevention system that parses those same access logs, subscribes to a community blocklist, and drives a plugin at the proxy to ban flagged IPs across the whole site. Self-hosted, auditable, no vendor holding the ban list hostage. This is the blunt instrument: known-bad IP, gone.

Layer two is a proof-of-work challenge in front of the blog. Think of it like a bouncer who makes you solve a small puzzle before you come in — trivial for a real browser, expensive at scale for a scraper firing thousands of requests. The key detail so you don't wreck your own SEO: allowlist the verified good crawlers (Google, Bing, DuckDuckGo) so they walk straight through. You want to tax the scrapers, not the search engines that send you readers.

Layer three is where it gets fun. Remember those confession paths? Requests for them get routed into a tarpit — a service whose entire job is to waste a bad bot's time. It answers slowly and hands back an endless maze of links that go nowhere. In my logs I can watch a probe for /wp-admin take 20 seconds to resolve, /.env take 31, /xmlrpc.php take 12. Every one of those seconds is a second the scraper isn't spending on someone else.

I'll admit I fumbled the tool choice here first. I reached for one popular tarpit, went to read its docs, and found the author deliberately poisons his own site against AI scrapers so thoroughly that I couldn't get a clean read of the documentation to deploy it. Respect the commitment. I pivoted to a different engine and moved on.

Feeding the scrapers a fake version of me

Here's the part I'm quietly proud of. A tarpit that serves obvious garbage gets fingerprinted and skipped. So instead of random noise, I trained the maze on a Markov model built from about 3.4MB of my own published writing. Now the scrapers that harvest my domain get believable, on-brand, completely fake content — sentences that babble in my data-engineering voice about partitioned tables and granted access and daily backups, stitched into paragraphs I never wrote.

If someone is scraping the web to train a model or resell my content, they don't get my posts. They get a plausible hallucination of me. That's a better outcome than a 403, because a 403 tells them to try harder and the maze tells them nothing is wrong.

The layers also talk to each other. A bot that trips the tarpit or fails the challenge shows up in the logs, the intrusion-prevention layer notices the pattern, and the offending IP gets banned at layer one — so the next request from that address doesn't even reach the funnel. The trap feeds the wall.

What this actually buys you

Two things, and neither is "I blocked all the bots" — you won't, and chasing that is a treadmill. First, honest numbers. When 82% of your traffic is noise, every conversion rate, every "popular post," every engagement metric you've been reading is distorted by it. Cleaning the classification cleaned my analytics more than any tag-manager change ever did. Second, you stop subsidizing the people abusing you. Serving real pages to a scraper firing 6,000 requests a day costs you compute and bandwidth; sending it into a tarpit costs it instead.

One honest caveat before you run with this: the granular numbers here come from my own traffic capture over a single window, and I'd re-check them against the live dashboard before I quoted them in a boardroom. The shape of the finding is rock-solid; the second decimal place isn't the point.

If you run your own site, go build the origin-truth view before you build anything else — you can't defend against traffic you can't see accurately. And if you've found a cleaner way to classify actors from raw logs, or a tarpit engine you like better, I'd genuinely love to hear it. As always, I'm here to help!

I set out to build a memory layer for my agents — somewhere they could write what they learned and read it back later without me hand-feeding context into every prompt. A few weeks in, I looked at the schema I'd drawn and laughed, because I'd reinvented something I've been building in data lakes for years. Bronze, Silver, Gold. The medallion architecture, transplanted out of the warehouse and into an agent's recall. It turns out the pattern that keeps analytics honest is the same one that keeps memory honest.

If you've built medallion pipelines, you already know most of this — you just haven't pointed it at an LLM yet. Let me walk you through the transplant, layer by layer, and the one invariant that makes the whole thing work.

Bronze: raw, immutable, append-only

Every event an agent or a human emits lands first in a single table I think of as Bronze — an append-only archive. Raw content goes in verbatim and is never modified after insert. A session transcript, a note, a commit, an issue: it gets wrapped in a small contract and stored exactly as it arrived.

That contract carries a few fields that earn their place. A source and a source_event_id together form an idempotency key, so re-submitting the same event returns "duplicate" instead of creating a second copy — re-running an ingest is safe. A namespace scopes the record to an owner or a project and acts as a contamination firewall, so one client's memory can't bleed into another's. And a provenance_root points back to the upstream source of truth, so every memory can be traced to where it actually came from.

The discipline here is the same as in any lakehouse: Bronze is sacred and dumb. It doesn't interpret, it doesn't clean, it doesn't dedupe the content. It just remembers, faithfully, forever.

Silver: derived, cleaned, the part the system owns

Silver is where the raw event becomes usable knowledge, and it's owned entirely by the engine — no consumer reaches in and writes here. For each Bronze row, a derivation runs in four steps:

1. Distill. An LLM pulls atomic facts out of the raw content — each fact self-contained, specific, and testable. One messy transcript becomes a handful of clean assertions.
2. Embed. Each fact gets a vector, indexed for similarity search. This is what makes recall semantic instead of keyword-bound.
3. Resolve conflicts. Before a new fact is stored, the system finds existing facts that are close in vector space and asks an LLM to judge: is this new, a refinement, or a contradiction? It resolves to add, update, supersede, or skip.
4. Score. The fact gets an importance score, so later recall can prioritize what matters.

The supersede case is the one I want you to sit with. When a new fact contradicts an old one, the old fact isn't deleted — it gets an end-timestamp and steps aside. Yesterday's truth is still on disk, marked as no-longer-current. Memory doesn't overwrite; it keeps a history of what it used to believe and when that changed.

Gold: shaped for whoever's asking

Gold is what consumers actually touch, exposed through capability-gated tools — search, read, list — so an agent gets only the access it's been granted. Nothing queries the raw archive directly; everything comes through this projection.

Gold is also where humans get to disagree with the machine. A person can write an overlay — an edit, an annotation, a redaction — anchored to a span of a derived fact. On read, the system composes the Silver fact with the human's overlay laid on top, and the human's version wins. The machine's derivation is a draft; a person's correction is final. When Silver later re-derives from Bronze with a better model, those overlays re-anchor onto the new text rather than getting silently dropped. Human judgment survives a model upgrade.

The invariant that makes it all work

Here's the line that holds the whole structure up, and it's lifted straight from the lakehouse playbook: Silver is always rebuildable from Bronze.

Everything above the raw layer is disposable. If a better extraction model ships next quarter, I don't migrate Silver — I throw it away and re-derive it from Bronze. If I find a bug in the conflict-resolution logic, same move: fix the code, rebuild from the immutable source. The refined layers are a cache of my best current interpretation, not a second source of truth. There's exactly one source of truth, and it's the dumb append-only table at the bottom.

That invariant is also what makes the system safe to automate, which is the part that surprised me. Because re-deriving Silver is reversible, the engine can do it on its own. The only operations that need a human in the loop are the genuinely irreversible ones — destroying a Bronze record, or overwriting a person's overlay. Reversibility, not the model's confidence, draws the line between what the system does autonomously and what it asks permission for.

Where it costs you

I'm not going to sell you a free lunch. Medallion-for-memory has a real bill attached. Every Bronze row triggers an LLM distill and an embedding call, so ingest isn't cheap — you're paying model inference to turn raw events into facts, at the rate events arrive. The conflict-resolution threshold is a genuine knob, not a setting you get right once: too loose and contradictions slip through as duplicate "facts," too tight and real updates get filed as brand-new beliefs. And the rebuild-from-Bronze guarantee is only as good as your discipline about Bronze — the day something quietly prunes the raw layer to save space, the entire safety story collapses and you won't notice until you try to rebuild.

None of that has talked me out of it. The same separation that keeps a data lake trustworthy — immutable raw, disposable derivations, a consumer layer that never mutates the source — is exactly what an agent's memory needs, for exactly the same reasons. I've been doing medallion since before I called it that. It just took building a brain for my agents to realize the pattern was never really about analytics.

If you're storing what your agents learn, I'd love to know whether you landed somewhere similar or solved it a different way. As always, I'm here to help.