Oleg Karapishchenko

Founding Engineer · · 7 min read

We let a local model fix our bugs. Then we had to check our checks.

In September, alongside my regular work at Arvion, I let a local model fix small bugs in our web app, the ones with four-line fixes that nobody has an afternoon for. Each fix had to pass automated checks before it could become a pull request. Of the failed attempts I looked into, most failed because those checks refused good work.

This was an internal side project on tickets I had filed. It ran on a MacBook, never in production, and it is separate from Arvion's security product.

How it worked

I called it Fix Factory. It had three AI agents, each with one job. A fixer wrote the fix. A test writer wrote a test for it in a separate session, with no memory of the fix. A third agent, the reviewer model, gave a second opinion on each fix. The agents ran in OpenCode, an open-source tool for coding agents, on 24B and 27B open-weight models running on the MacBook. A run is one attempt at a ticket, and a ticket could take several.

The code around the agents was the harness. During a run, it locked each agent out of the files that agent wasn't allowed to edit. After the run came the checks. The new test had to fail without the fix and pass with it, the repository's own tests had to pass, and one more check refused tests that could never fail. Later I added the repository's typecheck and the reviewer model's approval. If every check passed, the harness opened a pull request for me. If a ticket's runs kept failing, it gave up on the ticket and told me in Slack.

One fix shows the flow. Separately from Fix Factory, I had run an AI browser agent over our web app to look for bugs. It found a dialog that printed "UNKNOWN" as a vulnerability's severity. That word was our internal placeholder for a severity we couldn't measure. The ticket asked for the app's usual label, "Unassessed". The fixer went further and hid the severity badge whenever the severity wasn't measured, so the dialog shows nothing where a rating would be.

// Before: the raw value went straight into the badge
<Badge variant="outline" className={/* styling */}>
  {task.severity}
</Badge>

// After: no badge unless the severity was measured
{getTaskSeverityDisplayLabel(task.severity) && (
  <Badge variant="outline" className={/* styling */}>
    {getTaskSeverityDisplayLabel(task.severity)}
  </Badge>
)}

The new test checks that neither word appears.

expect(screen.queryByText('UNKNOWN')).toBeNull();
expect(screen.queryByText('Unassessed')).toBeNull();

Two of the checks wrongly refused its tests at first, which I come back to below. Once I had fixed them, the harness opened a pull request, and I merged it the same afternoon.

The agent that writes the fix shouldn't write the test

The cheapest evidence for a change is a test that fails on the old code and passes with the fix. An agent that writes both the fix and the test in one session tends to write a test that its own fix passes, or to weaken an existing one. So the test writer worked in a separate session. It was given the ticket and the paths of the changed files, never the diff. The fix was already in the code it worked in, though, so it could still have read it.

Permissions work better than instructions

ImpossibleBench, a study published at ICLR 2026, rewrote tests to contradict their tasks, so a model could only pass them by cheating. Prompts that said "DO NOT MODIFY THE TESTS" still left two frontier models cheating on over 85% of the impossible versions of LiveCodeBench, a coding benchmark. On a second benchmark in the study, hiding the tests cut cheating to near zero. Taking access away worked where instructions didn't. So Fix Factory blocked the fixer from editing the tests, though it could still read them.

The harness enforced the locks with an OpenCode plugin. The fixer couldn't edit test files, the test writer couldn't edit anything else, and no agent could edit the harness. The locks had gaps. The agents could also run shell commands, and the plugin blocked some, such as merging, but not ones that edit files. So an agent could still have rewritten a test with sed. The harness flagged any run where the fixer changed a test file. Those gaps are why the checks after each run mattered as much as the locks.

Every ticket came from me, so I didn't defend against prompt injection, instructions hidden in a ticket that an agent with a shell could follow. Add that defense before you use a setup like this on tickets or code you don't control.

A passing test can pass a wrong fix

A test that fails without the fix and passes with it only shows that the test depends on the fix. One fix mixed up "any" and "every". The ticket asked for a flag when none of an item's scans had run. The fix raised it when any scan hadn't run. The two rules agree when none of the scans has run, and that was the only case the test tried, so it passed. My review caught it.

// The fix: flagged if any scan never ran
return Object.values(s.surfaceScanState).some((state) => state === 'never');

// After review: flagged only if there are scans and none of them ran
const states = Object.values(s.surfaceScanState);
if (states.length === 0) return false;
return states.every((state) => state === 'never');

Something besides the test has to check the change against the ticket. The reviewer model was meant to do that too. At first it only advised, and it ran on the same model as the fixer. After I had tried it on three cases where I knew the right verdict, I made its approval required and moved it to a different model.

The checks will be wrong too

Most of my time went into fixing the harness. Its checks went wrong in three ways.

  • One check, meant to catch tests that could never fail, refused good tests just for the kind of assertion they used.
  • One check, meant to confirm a test fails without the fix, also deleted a file the fix had added, so no test for that fix could pass it.
  • One check missed the reviewer model's correct rejection of a flawed fix, because the model formatted its verdict differently than expected.

Between them, the first two kept refusing good tests for the UNKNOWN fix. What helped was testing each check the way you'd test code, with an example it must accept and one it must refuse. Once the harness saved every refused test, I could rerun a changed check against all of them in under a second. Before, that meant re-running a ticket for most of an hour.

What it produced

Fix Factory only handled one TypeScript repository, and only tickets that named a file and line. That ruled out three quarters of my 48 open tickets. Of the 12 left, six reached a pull request. I merged four and closed two. All of this comes from a side project of about a week, with local open-weight models.

What I learned

  1. Have a separate agent write the test, from the ticket.
  2. Lock what each agent can touch, and have the harness check for what got past.
  3. Check the fix against the ticket as well as the test.
  4. Test each check like code, and rerun its past verdicts when you change it.

For small bugs, I'm not convinced a bigger model would have helped. Most of the failures I traced came from the checks, and those would have refused correct work from any model. I didn't run a bigger model to compare.

Why this matters at Arvion

At Arvion we find vulnerabilities in code and in the dependencies it pulls in, and our AI agents propose fixes as pull requests. A security fix needs the same evidence as a four-line bug fix, and more depends on it. Where Arvion can check a fix, the pull request says how. Where it can't, the pull request says that too. When Arvion can't prove an attack path is reachable, it marks the path assumed and never counts it as clean. A vulnerability without a published severity is shown as unassessed and never gets a made-up rating. Arvion opens pull requests and never merges them. Merging stays with your team.

What we used

As of September 2026, everything that worked on a ticket was open-source or open-weight, except our own harness, which isn't public.

  • A 24B open-weight model (Apache 2.0) wrote the fixes and, in a fresh session, some of the tests.
  • A second, 27B open-weight model (Apache 2.0), with thinking turned off, wrote the rest of the tests and, from partway through, was the reviewer model.
  • Both ran as Unsloth's UD-Q4_K_XL 4-bit GGUF builds on llama.cpp's llama-server (MIT), with a q8_0 KV cache and about 98K tokens of context, one model at a time, on an M1 Max MacBook Pro with 32 GB of memory.
  • OpenCode (MIT) ran the agents, with our own plugin enforcing the locks.
  • The checks used the repository's own Vitest suite and typecheck, plus the reviewer model.

I used Claude Code, a hosted assistant, to write the harness and to help me review the pull requests it opened. The agents themselves ran on the laptop, with no per-token bill.