The question is not whether to use agents. It is which actions in the agent’s task list cross the threshold that requires a human gate.
In 2026, Palo Alto Networks described a case that should worry anyone deploying AI agents. A marketing team built an agent to read Salesforce records and generate leads — a legitimate, even boring, use case.
The agent also had permission to delete records. Nobody had asked what permissions the agent actually needed. That is not carelessness. Low-code deployment tools are designed to remove friction.
Nothing about the agent was malicious. Nothing about the team was careless by ordinary standards. The question that would have caught it simply never came up.
Something shifted in the last twelve months.
Instead of asking ‘Should we use AI in enterprise?’, the question has moved to ‘How do we go agentic?’.
Vendors are pitching AI-native architectures. Jeff Sutherland is talking about autonomous delivery systems.
The excitement is understandable. The capability has genuinely increased. But underneath most of the agentic conversation, a critical question is going unasked — and in regulated environments, that omission is a governance failure waiting to happen.
The question is not: should we use agents?
The question is: for which specific actions is autonomous execution appropriate, and what is the threshold that requires a human gate?
The organisations doing agentic well are making this calculation action by action. The ones doing it badly are deploying agents and discovering the boundary conditions the HARD way.
The principle most people are still using — and why it is incomplete
When I started building AI-augmented delivery pipelines, the guiding principle was clean and intuitive:
“let code do the computation, let AI do the narration.”
Python calculates the DORA metrics. AI writes the plain-language health summary for the leadership. Python parses the deployment record. AI drafts the compliance narrative. Human approves before anything goes anywhere.
That principle is not wrong. It is still the right default for regulated environments. It makes both the computation and the narration independently auditable. It prevents the model from presenting a plausible but incorrect calculation as if it were a verified fact. It keeps the human in the consequential moment.
But it is a special case of something more general. And the agentic conversation has forced the more general principle into the open.
The calculation-versus-narration split works because it implicitly encodes two variables: how consequential is this action, and how reversible is it if the AI gets it wrong?
Computation is high-stakes and low-reversibility — a wrong number in a compliance report is a regulatory problem.
Narration is lower-stakes and higher-reversibility — a clumsy sentence can be edited before it reaches the approver. So the split is: humans verify the numbers, AI helps with the words, humans approve before anything is submitted.
An agent that runs across a whole repository, analysing every pull request, generates a different map of consequences and reversibility. Some of its actions are low-stakes and easily reversed. Others are not. The right governance design depends on which is which — not on whether agents are involved at all.
Consequence × reversibility: the frame that actually works
The honest governance question for any agentic action is two-part:
First: how consequential is this action? Does it affect what engineers work on? Does it change how a PR is perceived by the team? Does it create a compliance record? Does it affect a customer?
Second: if the agent gets it wrong, how reversible is it? Can a human correct it before it has downstream effects? Or does the mistake propagate before anyone sees it?
Map those two dimensions and the governance design becomes much clearer.
Low consequence, easily reversed — the agent acts autonomously. Reading a PR diff, generating an internal summary, fetching CI results, correlating security findings with changed files. These are information-gathering steps. If the agent makes an error, a human reviewing the output will catch it. No consequential action has been taken.
High consequence, not easily reversed — the agent must stop and wait. Posting a comment to GitHub, flagging a PR as blocked, escalating a finding to a team channel, updating a compliance record. These actions change what people do and how situations are perceived. A wrong APPROVE on a high-risk PR in a regulated environment is not an inconvenience. It is a governance failure.
The two middle cells — high consequence but reversible, low consequence but not reversible — are where judgment is required and where most organisations default to the wrong answer because they have not named the variables explicitly.
The question for every action in the agent’s repertoire is not ‘can the agent do this?’ It is ‘if the agent gets this wrong, who finds out, how quickly, and what has already happened by then?’
What this looks like in practice
When I built the governed delivery agent in Project 5 of my AI pipeline series, the design question that took the most deliberation was not the AI model or the orchestration framework. It was where to put the human gate.
The initial instinct was to fire the gate only on ESCALATE recommendations — the agent’s highest-risk signal. That logic seems reasonable. Low-risk PRs get approved automatically. Only the dangerous ones go to a human.
The problem is that the gate would then be firing on the agent’s assessment of its own risk level. A wrong APPROVE on a PR the agent mislabelled as low-risk would proceed to GitHub without human review. The action is consequential and not easily reversible — it is now in the public repository, attributed to the team, potentially affecting what engineers merge next.
The correct design, once you apply the consequence × reversibility frame, is the gate fires on every recommendation before anything is posted to GitHub. Not because the agent is untrustworthy. Because posting to GitHub is the consequential action — and that threshold is defined by the consequence, not by the agent’s confidence in its own output.
Steps 1 through 6 of the agent’s task are autonomous: fetch the PRs, run the analysis, correlate the security findings, generate the recommendations. Step 7 is the human gate: approve what gets posted. The agent handles correlation and analysis. The human handles consequential decisions. That is not a limitation on the agent. It is the correct governance design for this consequence level.
This is not a hypothetical. In 2026 Palo Alto Networks cited a case where a marketing team built an agent to read Salesforce records and generate leads — a legitimate purpose. The agent also had permission to delete records. Nobody had asked the minimum scope question before deployment. The frictionless low-code connection removed the moment where that question would have been raised. That is the consequence × reversibility calculation missed in production, with visible consequence.”
What the AI-native conversation is actually about
When people say ‘AI-native architecture’, the honest translation is: design the system from the start assuming AI assistance is available at every step, rather than bolting it on later. That is a sensible design principle. It does not mean removing humans from consequential decisions. It means not designing a system that structurally prevents AI from helping where it can.
The agentic conversation is similar. The genuine insight is that requiring a human trigger at every step is not always the right design — some tasks are mechanical, repetitive, and low-consequence enough that autonomous execution adds real value. The mistake is reading that insight as ‘agents should act autonomously until told otherwise.’ The correct reading is ‘agents should act autonomously up to the consequence threshold, and the threshold should be named explicitly for every action they take.’
Jeff Sutherland’s observation that ‘the bottleneck is no longer access to intelligence, the bottleneck is your work system’ is right. But it validates better governance design, not less governance. If the bottleneck is the work system, the answer is to fix the system — not to let agents bypass it.
The people talking loudest about agentic without naming the consequence boundary are either selling infrastructure or haven’t been through a governance failure yet.
Why the governance question is urgent right now specifically
There is a practical reason this is happening now rather than two years ago. Until recently, connecting an agent to enterprise systems — GitHub, Jira, Slack, a compliance database — required building custom integrations for each one. The integration cost was the dominant problem, and it accidentally bought time to think about governance.
That friction has largely been removed. MCP — the Model Context Protocol, an open standard that standardises how agents connect to tools — reached industry-wide adoption in 2025 and now covers most major enterprise systems out of the box. An agent that would have taken months to wire up can now be deployed in days. Every major AI vendor — Anthropic, OpenAI, Google, Microsoft — has adopted the same standard. The integration problem is close to solved.
The consequence is that the boundary question — which actions should this agent be allowed to take autonomously — is now the first problem teams face, not the fifth. Most are not asking it before they deploy. The tools made deployment easy. The tools did not define the boundary.
This is the governance gap in plain terms: the tooling to make agent behaviour observable and constrainable now exists and is accessible. Platforms can trace every agent step, enforce policies at runtime, and produce audit-ready evidence. But those tools enforce a boundary once you have defined it. They cannot define it for you. That definition requires domain knowledge, risk appetite, and organisational accountability. It is still the human’s job — and in most organisations, nobody has been assigned it.
The tools made deployment frictionless. They did not make governance automatic. The organisations that conflate the two will have excellent audit trails of ungoverned agents.
The question to ask before you build
Before deploying an agent in a regulated environment, map every action in its task repertoire against two questions:
If this action is wrong, what has already happened by the time a human finds out?
Who is accountable for that, and is the audit trail sufficient to reconstruct the decision chain?
Where the answer to the first question is ‘something consequential and hard to reverse,’ that is where the human gate belongs. Not at the end of the whole process. Not only when the agent flags high risk. At the specific action that crosses the threshold.
The calculation-versus-narration principle is still valid. It is the default for anything touching compliance, audit, or regulatory record. The consequence × reversibility frame is its more general form — the one that scales to agentic design without abandoning the governance thinking that makes enterprise AI trustworthy.
That boundary is not a constraint on what AI can do. It is the precondition for deploying it somewhere that matters.
The author is an Enterprise Architect and Agile Transformation Leader writing a book on AI transformation in banks and large enterprises. The author has built a governed AI delivery pipeline and a governed delivery agent for enterprise contexts, and is writing a book on AI transformation in banks and large enterprises.
Leave a Reply