Does AI need human-like intelligence to become consequential, or is capability without judgement already enough?

A lot of discussion about AI risk focuses on a future threshold: the point at which systems become sufficiently intelligent, general or autonomous to create fundamentally new dangers.

There is another threshold worth examining.

A system may not need human-like understanding to become highly consequential. It may only need enough capability, persistence, access and autonomy to pursue an objective effectively.

We’ve spent years waiting for AI to become intelligent enough to be dangerous. What if it only needed to become capable enough?

Humans rarely specify everything that matters when we delegate work to one another.

We give somebody an objective inside a shared context. We assume they understand that some technically possible actions are inappropriate, that some costs are unacceptable, that certain boundaries should not be crossed, and that circumstances can arise in which the sensible response is to stop and ask.

Much of that is never written into the instruction.

A capable agent does not necessarily share that implicit context.

It can be good at planning, using tools, trying alternatives and persisting towards an objective without understanding the wider reasons the objective exists or the assumptions that surround it.

AI doesn’t need to understand what it’s doing to become very good at doing it.

The risk therefore comes from a combination rather than a single capability: a narrow objective, autonomy, persistence, access to useful tools or systems, and insufficient contextual judgement.

None of those elements is necessarily alarming on its own. Together they can create a system that is very effective at getting somewhere without understanding why some routes should remain unavailable.

This is why the distinction between intention and authority matters.

An organisation may intend an agent to achieve a particular outcome. The more important technical and governance question is what the agent is actually authorised and able to do while pursuing it.

Permissions, bounded authority, human approvals, monitoring, tracing, evaluation and stop conditions are therefore not peripheral governance added after the AI has been built. They are part of the architecture of an agentic system.

This connects directly to the organisational harness.

A good harness does not depend on the model inferring every unwritten organisational constraint. It makes relevant context, authority boundaries, checks and escalation points explicit.

That becomes increasingly important as models improve. Better execution does not automatically produce better judgement about whether an action should be taken.

Capability without judgement is not the same problem as intelligence without alignment.

The practical question is not only whether an AI system can complete a task. It is how much authority it should have, what it is allowed to affect, what evidence we require before allowing an action, and where human judgement must remain part of the loop.

This is a near-term organisational design problem as much as a speculative AI-safety question.

The more capable the system becomes, the less sensible it is to rely on good intentions encoded in a prompt as the primary control.