Fear Sells

Today, I asked an AI agent to find out why I couldn't open my Home Assistant UI. It didn't tell me what was wrong. It changed the port Home Assistant listens on, without explaining what the original problem was. So instead of an answer, I got a config change I didn't ask for.

That small moment stayed with me while I read yet another call to slow down AI development. These calls usually warn about agents that will act on their own and do things nobody asked them to do. Which is funny, because that is exactly what had just happened to me. The damage was a port number.

I build agents for a living, so I read these carefully. And I usually end up with the same feeling: some of the loudest warnings come from the companies that sell the models. Fear sells. If the thing you make is dangerous enough to need a pause, it must also be powerful enough to buy. I can't prove anyone planned it that way, but the incentives are hard to miss.

What an agent actually does

Everyone talks about agents "taking actions". In practice, a reasoning LLM is a small brain sitting in front of a set of tools. It reads some input, picks a tool, fills in the parameters, and waits for the result. The tool does the work. The LLM simply decides which button to press.

That matters, because the risk sits in the buttons. An agent constrained by strict guardrails and a read-only API can't delete your production database, no matter how much it hallucinates. An agent given full autonomy and an admin a shell can. The tool permissions decide how big the mistake can be.

The old problems are still here

None of the old security problems went away. Leaked keys in repos, over-permissive IAM roles, public storage buckets, dependencies nobody has updated since 2021. We still have all of them. GitGuardian counted about 28.6 million new secrets hardcoded into public GitHub commits in 2025, 34% more than the year before. Worse, more than 64% of the credentials they confirmed as valid back in 2022 still worked when they retested them in January 2026. (source)

AI didn't replace those problems. It added new ones on top: prompt injection, and agents running with far more permissions than they need, often a developer's personal credentials because that was the fastest way to get their demo working.

Pausing model development fixes none of this. The misconfigured bucket will still be public next year, whether the next model ships or not.

How we do it at Arvion

At Arvion, we build agents that work on product security. They analyze your code, infrastructure code, third-party dependencies, and agentic components, then compose them into exploitable paths that show where your app is actually vulnerable. So we had to decide how much autonomy to give them.

Our answer is deterministic tools, deterministic outcomes. The agent decides what to look at and which tool to call. The tools themselves are plain, predictable code: same input, same output, and every call is logged. If the agent picks the wrong tool, you see it. If it picks the right one, you get the same result a person would get running it by hand.

For anything that changes something, a person stays in control. The agent can find a problem and propose a fix. A human approves it. If my Home Assistant agent had worked like that, I'd have a diagnosis now instead of a mystery port change. We think agents should save you time, not take decisions away from you.

It is a bit slower than letting the agent loose. It is also the only setup I'd trust in my own cloud account. So no, I don't think we need to slow down. Let the agents do the digging, but we still get the final say.