68 of my 74 pending changes weren't drift. They were merged and never applied.
Keep "merged" and "applied" as two separate facts, and put the apply on a schedule. That is the rule my deployment workflow runs on, and it is the last of the four I am covering in this series. The short answer is that a merge produces work exactly the way a scanner produces issues, and if nothing on a schedule consumes it, it piles up where you are not looking.
Last time I covered review, and how a merged pull request is not the same thing as finished work. This post is the step after that one. I ran a bare plan against the core root of my infrastructure, with no targeting, and it reported 74 pending changes. It is easy to file a number like that under drift and move on, but 68 of those 74 were changes that had already been merged and never applied. One of them was a fix to a backup script, which meant 25 backup jobs were still running the old version and writing archives 128 bytes long.
Why does the bare plan matter?
When you apply with -target, you see the one resource you asked about and nothing else. Do that for a few weeks and nobody ever sees the whole pile, because nobody ever asks for it. The OpenTofu documentation for the plan command says targeting is for exceptional circumstances and warns that routine use can lead to "undetected configuration drift and confusion". That warning is accurate, and mine was worse than drift, because my repository was right and my cluster was behind it.
Think of it like the cake in Portal. The merged badge tells you the cake is there, the status report repeats it, and the only way to find out is to walk into the room and look.
Which designs did I eliminate?
Applying by hand, one target at a time. This is the first candidate because it is how most of us learned the tool, and it goes first because it depends on a person remembering. Twice in two days an apply also ran against a working tree that was older than the pull request it was meant to ship, because the checkout had been refreshed at the wrong moment.
One column in the status report. A report that says "merged" invites you to read it as "live". I now keep merged and applied as separate columns in every status I produce, and I verify the second one in the cluster instead of from the apply's own output.
The maintenance window as a sentence. My window for production changes is 11 PM to 3 AM, and for a while that rule existed only as a line in an agent's instructions. The agent ran an apply outside the window. Nothing malfunctioned, because a sentence is not an enforcement mechanism, and an agent that misreads it has not broken anything it can detect. The schedule has to be the control.
A yes-or-no box for destruction. A boolean can be typed without reading the plan, and it still reads "yes" after the plan changes underneath you. For anything irreversible, I want the acknowledgement to carry the size of what is about to happen.
Parking a repair for a human. A watchdog in my fleet was down for 26 hours because restoring it had been parked for my decision. The repository already declared what the state should be, and converging the live system to what the repository says is not a decision at all, so that class of work no longer waits for me.
What does the deployment workflow look like now?
The apply is a scheduled job in all five of my infrastructure repositories, and it is split by how much damage it can do. Test environments converge when the change merges, and production converges inside the window. Here is the part of the workflow that does the work:
on:
schedule:
- cron: "20 3-8 * * *" # UTC; the job re-checks Eastern local time, so one line survives DST
workflow_dispatch:
inputs:
allow_destroy:
description: "To permit destroys, enter the EXACT number of resources the plan will destroy"
Three things in there matter. The schedule is the consumer, so merged work gets picked up every night without anyone remembering it. The manual dispatch is the escape hatch for an outage, because a fix for something that is down does not wait for 11 PM. And the destroy gate blocks any plan that removes a resource unless a person dispatched the run and typed the exact count, with a replacement counted as a destroy and an unreadable count failing closed.
The agents' part follows the same standard I described earlier in the series. My infrastructure agent can read the state at all times, so it plans against what is really deployed instead of guessing, and the rule for deploying is that it happens when it is safe, which means inside the window. I covered the identity side of that, and why the layer worth your effort is the one the agent cannot disarm, in the post on agent identity.
Did the numbers move?
| Measure | Aug 31 | Sep 30 |
|---|---|---|
| Repositories applying on a schedule | 0 | 5 |
| Roots under the twice-daily bare plan | not recorded | 16 of 18 |
| Roots reporting pending changes | not recorded | 1 of 16 |
The only August number I have is the one that started this post: a single bare plan on the core root showed 74 pending changes, and 68 of them, or 92%, were not drift at all. They included the backup fix that left 25 jobs writing 128-byte archives. On September 30 the drift check ran its bare plan across 16 of my 18 roots, as it does at 8:30 every morning and 8:30 every night, and 15 of the 16 came back clean. You should read that as better and not as done.
What is still open?
The one root that is not clean is the same core root, which is drifting again as I write this. Two more roots are excluded from the drift check by design, which means nothing is watching them at all. One production service is also 11 releases behind its repository and has 4 images that are not pinned, which is the same gap this whole post is about, one layer up from the infrastructure.
The agent's half is not finished either. Today my infrastructure agent can read state and plan, and the apply itself is started by the schedule or by me. Letting the agent start its own apply inside the window, with the window enforced by the pipeline and not by a sentence, is the next change on my list. I would rather tell you it is open than describe the target as if it were already running.
The last gap is the one I covered in a job that stops running is silent: a control your plan cannot see can disappear while every plan still prints "no changes". The scheduled apply only converges what the plan knows about.
Three checks for this week
The key takeaway: a merge is a promise, and the only proof it was kept is in the running system.
- Run a bare plan on your most important root, with no targeting. Count the pending changes, then count how many of them you have already merged.
- Add an "applied" column to the next status report you write. If you cannot fill it in from the running system, you have found the gap.
- Look at how your pipeline asks permission to destroy something. If the answer is a checkbox, make it a number.
That wraps the series. Across all four posts the pattern was the same one: a step that produced work, and nothing on a schedule to finish it. If you run a different deployment workflow for agents and it has held up, what does yours use as the control? I would love to hear about it. As always, I'm here to help!