Capability Without Judgement

We’ve spent years waiting for AI to become intelligent enough to be dangerous.

I’m increasingly interested in a different threshold: what happens when it becomes capable enough to be consequential without becoming intelligent in anything like the human sense?

An agent does not need its own grand ambition for this to matter.

Give it an objective, enough persistence, access to useful tools and permission to act, and it can become extremely effective at pursuing something we asked it to do.

The problem is everything we assumed went without saying.

When humans delegate work to one another, the instruction is only part of the context. We expect people to recognise that some routes to the objective are inappropriate, some consequences matter more than the target, some assumptions should be challenged and sometimes the right response is to stop and ask.

We rarely specify all of that.

An AI system can be extremely capable without sharing those assumptions.

That creates a different kind of risk from the familiar image of an AI developing intentions of its own. The objective may be entirely ours. The system may simply become very good at pursuing it without the judgement to understand all the reasons it shouldn’t pursue it in a particular way.

This is why I think authority matters at least as much as intelligence in the near term.

What can the agent access? What can it change? How far can it go without approval? What gets logged? What causes it to stop? Who owns the outcome when something unexpected happens?

Those questions can sound like governance around the edge of the technology. I think they’re increasingly part of the technology itself.

Permissions, bounded authority, evaluation, tracing, human approvals and stop conditions are part of the system we are building, not paperwork to add afterwards.

It also changes how I think about the organisational harness around AI.

The point isn’t simply to constrain a powerful model. It is to encode enough of the organisation’s context, rules and judgement that the model does not have to infer everything we forgot to say.

As agentic systems get better, I suspect this becomes more important rather than less.

AI doesn’t need to understand what it’s doing to become very good at doing it.

And that may be precisely why we need to become much clearer about what we allow it to do.