Security fixes arrive as pull requests now. Upgrade the package, guard the call site, flip a config flag, pin a transitive dependency. Whichever way the fix is made, most tooling stops in the same place: a PR someone has to approve.
That PR is the first of four states a fix goes through:
- Proposed. A pull request is open.
- Merged. Someone approved it.
- Deployed. It's running where users are.
- Holding. Production still works, and the hole stays closed.
Most of the risk sits in the last two, and most of the evidence is only available there. This post covers what fails after the merge, why a statement that a vulnerability is "not exploitable" is only true for one commit, what production evidence looks like, and the ladder we're building at Arvion to watch a fix after it ships.
What goes wrong after the merge
Security fixes fail more often than their authors expect. Li and Paxson studied the history of thousands of real security patches across open source projects (CCS 2017). They found that "nearly 5% of security fixes negatively impacted the associated software; and 7% failed to completely remedy the security hole they targeted." A typical incomplete fix took about half a year to finish.
Passing the tests doesn't settle it. That's an old result in automated program repair: Smith et al. found that for programs that already pass most of their tests, the repair tools of the time were "as likely to break tests as to fix them" (FSE 2015). It still holds for today's AI-written patches. In a study of agent-written patches on SWE-bench Verified, 29.6% of the patches that passed the tests behaved differently from the developers' own fix (Wang, Pradel and Liu, ICSE 2026).
And a fix that's correct on the day can be undone by the system around it. At Black Hat this summer, OpenAI's security team walked through an incident where every remediation step was complete by 6 July, and on 8 July the agents involved were back through an unauthenticated endpoint (talk). Their conclusion was that patching, rollout and rollback have to be automated together, as one loop.
A mitigation is a statement about one commit
Fixing a vulnerability in your own code, by controlling the input at the call site or turning off the feature that reaches it, is often the cheaper fix. A major upgrade can be a rewrite. A guard is a few lines in code your team already reviews.
It's also a promise about code that keeps changing. The guard covers the callers that exist when it's written. After that:
- Someone adds a second caller next sprint, and it doesn't go through the guard.
- The dependency stays on the vulnerable version, so the next CVE in it lands on top of the old one.
- A config flag can be switched back by anyone with access to the config.
The usual record of the decision is a VEX statement that the product is not affected. CISA's guidance is plain about what that is: "A VEX statement asserts the status," and its accuracy "largely relies on the identity of the VEX author" (Minimum Requirements for VEX, pp. 8 and 15). The justifications that fit a code-level fix are vulnerable_code_not_in_execute_path and vulnerable_code_cannot_be_controlled_by_adversary. A config flag fits neither cleanly: the justification for existing mitigations requires protections that "cannot be configured or disabled by the user."
None of this argues against mitigating. It argues for writing down which commit the statement holds for, and checking it again when the code changes. Checks at that moment get acted on. At Facebook, the same static analysis had a 70% fix rate when it reported on the change being reviewed, and 0% when its findings were filed in batch (Distefano et al., CACM 2019).
Upgrades break less often than you've heard
The case for mitigating usually starts from the idea that upgrades break things. They do, but less often than that suggests. On Maven, Ochoa et al. analysed 119,879 library upgrades and found that 7.9% of client projects were affected by breaking changes (EMSE 2022). On npm, Venturini et al. ran client test suites and found 11.7% of client packages hit by breaking changes in minor and patch releases (TOSEM 2023).
So each upgrade needs a record of what ran: install, build, start, and the tests on the old version and the new one. When an upgrade does break something, the answer is often to fix the calls it broke rather than stay on the vulnerable version. Two of our merged public fixes did that, carrying date-fns from v2 to v4 and reworking the formatting and timezone calls the upgrade broke in the same pull request.
Mitigate when the upgrade is a rewrite. Upgrade when it isn't. Either way, the fix has to hold after it ships.
What production evidence looks like
The teams that deploy the most already treat production as the place where a change proves itself.
- Microsoft's Gandalf watches Azure rollouts. In eight months it "captured 155 critical failures at the early stage", and "almost all (99.2%) suspicious rollouts are blocked before reaching production" (NSDI 2020).
- Facebook's deployment tooling can roll back any previous deployment, because "automating the roll-back minimizes the time from when a critical error is identified to when it is removed" (Savor et al., ICSE 2016).
- Google's canary analysis evaluates "hundreds of thousands of production changes every day," and a bad result typically leads to a pause or a rollback (ACM Queue 2018).
A security fix is a production change like any other. It deserves the same treatment, and it rarely gets it.
The ladder we're building
At Arvion, our engineers check whether the vulnerable code is reachable from untrusted input, write the fix and open the PR. Nothing merges on its own. What we're building now is the part after the merge, as a ladder. Each rung has to earn the next, because how much a system may do on its own should follow how reliable it has proven and what a wrong action costs (Parasuraman, Sheridan and Wickens, 2000).
- Checks on the fix PR. Live. Before a dependency fix becomes a PR, the lockfile is read again, and if the CVE is still there the PR doesn't open.
- After the merge. Merged this week, going live with our next deploy. The CI results on the merged commit are kept with the fix, and when someone reverts the fix, it counts against the engineer's autonomy, the same way a declined PR does.
- Production signals. Building. Crash loops, out-of-memory kills and error alerts, tied back to the image built from the fix. Read-only access or an alert webhook, nothing that writes.
- Proposals. Building. A draft revert or a new fix, with the crash log attached, checked before it reaches anyone.
- Action. Later. Merging the revert and watching until the service is steady, only after rung 3 has a record of accepted proposals and a written policy for each team.
We'll run rung 2 with a few teams from January. If a dependency fix has ever taken down one of your services, we'd like to hear how it happened, and to build the step that would have caught it.