When Governance Depends On Systems Nobody Fully Understands

Most governance works on a fairly comfortable assumption. Somewhere inside the organisation, somebody can eventually work out what happened. They may need logs, documentation, technical specialists or a vendor on the phone, but there’s an answer to be found. AI makes that assumption harder to hold. 

An organisation can understand why it bought an AI system, what data it can access and which business process it supports without being able to explain exactly why the model produced a particular response. It may know how an application is configured without knowing everything about the foundation model underneath it. 

That doesn’t make governance impossible. Organisations have always managed uncertainty. But AI governance increasingly has to work in environments where some uncertainty may remain even after the organisation has done everything reasonably expected of it. So the challenge is beginning to change. 

Governance can’t always wait for complete understanding before deciding whether a system should be trusted with a business decision, workflow or action. It needs another way to decide when understanding is sufficient, where uncertainty becomes unacceptable and who takes responsibility for the difference.

Governance Has Always Assumed Someone Can Ultimately Know

Governance usually depends on being able to establish a chain of responsibility:

• What does the system do?
• Who approved it?
• Which controls apply?
• What information does it use?
• Who owns the outcome if something goes wrong?

Those questions still make sense. The difficulty is that the answers can now sit across very different places. Some limits come from the technology itself. The International AI Safety Report 2026 says researchers still can’t reliably explain why general-purpose AI systems produce particular outputs, while developers can’t always predict which behaviours will emerge during training. 

Current interpretability techniques can provide useful information, but they also have limitations. Other limits come from outside the organisation. A company using a commercial foundation model may know considerably less about its training data, internal evaluations and development process than the provider does. 

The same report identifies these information asymmetries as a separate challenge because important evidence may exist without being available to the people making governance decisions. Transparency isn’t necessarily moving in the direction enterprises would prefer either. 

Stanford University’s 2026 AI Index found that the average Foundation Model Transparency Index score fell from 58 in 2024 to 40 in 2025, with continuing gaps around training data, computing resources and post-deployment impacts. Then there’s the organisation itself. One team may understand the model, another the application, another the data and another the business process. 

Each may have a perfectly reasonable view of its own piece while nobody holds a complete picture. Governance therefore has to begin by asking something more precise than whether an AI system is transparent. It needs to know what isn’t understood, why that knowledge is missing and whether the gap changes the decision being made.

Not Every Unknown Means The Same Thing

It’s easy to collapse all of this into one familiar phrase: AI is a black box. That description can be useful, but it hides several very different problems. There are things organisations don’t know because current technical methods can’t reliably explain them. There are things the vendor knows but doesn’t disclose. 

And there are things that could be known internally if information currently split across teams, systems and governance processes were brought together. Those gaps shouldn’t receive the same response. A missing internal owner can be fixed. So can poor documentation. A supplier may be asked for stronger evidence, although there will be limits to what it’s willing or able to provide. 

A genuine limit in model interpretability is different because more meetings and better paperwork won’t make the underlying science suddenly complete. The International AI Safety Report makes a similar distinction. Its governance challenges include gaps in scientific understanding, information asymmetries, market failures and problems with institutional design and coordination. 

In other words, imperfect knowledge isn’t one problem with one cause. That changes the first job of an AI risk assessment. Before deciding what to do about uncertainty, leaders need to understand which kind they’re dealing with. Some gaps can be closed. Some can be reduced. 

Others may simply need to be recognised and governed as uncertainty rather than quietly treated as information the organisation assumes somebody else must have. Once that distinction is clear, testing becomes easier to place in the picture.

Testing Can Reduce Uncertainty Without Eliminating It

The obvious response to a system we don’t completely understand is to test it. And we should. Evaluation can show how a model performs against particular tasks, how it responds under certain conditions and where known weaknesses appear. It gives governance something far more useful than confidence based on expectation alone. 

But testing has its own limits. The International AI Safety Report describes an emerging evaluation gap between controlled tests and real-world performance. Benchmarks can be narrow, outdated or affected by training data contamination, while systems can behave differently once they encounter the variety and unpredictability of real environments. 

That doesn’t make evaluation unreliable or pointless. It changes what the evidence proves. Passing a test can show that an AI system behaved acceptably under the conditions that were tested. It can’t automatically prove that the same system will behave as expected across every situation it may encounter after deployment. 

This is where a familiar governance model starts to strain. Approval often implies that uncertainty has been resolved: the requirements were set, the system was tested and somebody signed it off. With AI, the more accurate conclusion may be narrower. 

The organisation has gathered enough evidence to justify using the system under particular conditions and within particular limits. That brings governance to a decision it can’t avoid.

Governance Needs To Decide How Much Understanding Is Enough

Demanding complete understanding sounds responsible until complete understanding becomes impossible to demonstrate. The opposite response is just as weak. Organisations can’t allow technical complexity to become an excuse for deploying systems nobody can reasonably assess because “AI is unpredictable anyway”. 

There has to be something in between. A more useful approach is to connect the amount of evidence required to the authority the system receives and the consequence of getting the decision wrong. 

A low-impact assistant drafting internal meeting notes doesn’t need the same standard of assurance as a system influencing financial approvals, clinical decisions or critical infrastructure. This gives risk-based AI governance a clearer purpose. The goal isn’t to decide how much opacity the organisation is comfortable ignoring. 

It’s to decide whether the evidence available is strong enough for what the system is being allowed to do.

Authority should follow evidence

Every AI deployment involves some form of permission, even when nobody calls it that. A system may be permitted to generate a suggestion, recommend an action, change a record, initiate a process or act without waiting for a person. Each step increases what the organisation is trusting it to do. 

As that authority grows, so should the evidence supporting it. Where evidence remains limited, governance doesn’t have to pretend the uncertainty has disappeared. It can reduce the scope of the decision, keep consequential actions outside the system’s control, require additional validation or restrict where autonomy is appropriate. 

The principle is simple: the authority granted to an AI system should reflect how much confidence the organisation can reasonably justify in its behaviour. That also gives human oversight a clearer role. A person shouldn’t be added simply because the workflow needs a reassuring human-shaped control. 

Oversight needs to sit where human judgement can meaningfully reduce the uncertainty or consequence involved.

Some uncertainty should remain unacceptable

Risk-based governance only works if organisations are willing to say no. There will be situations where the consequences are too significant, the evidence too weak or the system’s behaviour too difficult to assess for a particular use. More monitoring after deployment doesn’t automatically make that acceptable. 

The threshold will differ between organisations and use cases. What shouldn’t differ is the need to define one. That means AI risk tolerance needs to account for what the organisation doesn’t know alongside the risks it already understands. A system can meet its technical requirements while still leaving unanswered questions that are too important for the authority being requested. 

Once an organisation accepts uncertainty deliberately rather than accidentally, another question follows very quickly. Who owns that decision?

Accountability Changes When Explanation Has Limits

Accountability is relatively easy to describe when responsibility follows technical control. A team builds a system, somebody approves it and there’s a clear path back to the people who made the important decisions. Modern enterprise AI can break that neat connection. The person responsible for an AI-enabled business process may not have trained the model. 

The organisation may not control the foundation model at all. Even the provider may be unable to explain every individual output in the way we’d expect from conventional software. None of that removes responsibility for using it. Instead, AI accountability begins to include the decisions surrounding the system:

• Why was it trusted?
• What could it do?
• Which uncertainty was accepted?
• What evidence supported that choice?
• Who had the authority to make it?

McKinsey’s 2026 AI trust research offers a useful signal here. Organisations with explicit ownership for responsible AI recorded higher average maturity scores than organisations without a clearly accountable function, 2.6 compared with 1.8. McKinsey connects that difference to clearer ownership and decision rights. 

Yet naming an owner only solves part of the problem. Accountability becomes meaningful when the owner has enough authority, information and organisational support to make the decision they’re supposedly responsible for. That distinction is becoming particularly important as AI stops simply producing outputs and starts taking action.

Agentic AI Makes The Governance Gap Harder To Ignore

A conventional AI assistant might generate an answer that a person decides whether to use. An AI agent can move further through the workflow. It can interpret a request, decide what needs to happen, call tools, retrieve information, take an action and then use the result to decide what comes next. 

The governance problem now stretches across a sequence rather than a single output. Enterprise adoption is moving faster than many governance models around it. Deloitte’s April 2026 research, based on 3,235 business and IT leaders across 24 countries, found that only 21 per cent said their organisations had mature governance in place for agentic AI. 

SAP LeanIX found a similar gap from another direction. Its 2026 Agentic AI Survey reported that 98 per cent of surveyed companies had deployed or planned to deploy AI agents, yet only 17 per cent had visibility into agent performance or conformance. Almost half, 48 per cent, lacked clear roles or responsibilities for managing those agents. 

The significance isn’t simply that agent governance is immature. Agents make the relationship between understanding and authority much harder to avoid. A system that generates an imperfect recommendation creates one kind of risk. A system that can turn its own interpretation into a series of actions creates another because every step can change what happens next. 

Governance therefore needs to consider what the agent is allowed to accomplish across the complete sequence, what evidence exists about its behaviour and how responsibility follows the actions it takes. This pushes the discussion beyond explanation alone.

Control May Depend More On Assurance Than Complete Explanation

For years, explainability has carried a lot of weight in responsible AI discussions. Understandably so. If a system influences an important decision, organisations want to know why. But explanation is only one form of evidence. An organisation may not be able to reconstruct everything happening inside a model and still be able to establish useful facts around it. 

It can test behaviour, define operating boundaries, record actions, trace which systems were involved and determine whether performance remains within accepted conditions. This is the territory of AI assurance. Assurance doesn’t mean declaring an AI system safe because enough controls have been added around it. 

It’s closer to building a body of evidence that supports the organisation’s decision to keep trusting the system with a particular role. That evidence also needs to remain useful after deployment because the conditions surrounding AI can change. Models get updated. Workflows evolve. New data appears. Agents gain new capabilities. 

A decision that was reasonable six months ago may no longer rest on the same assumptions. Current governance work is already moving in this direction. NIST’s AI Risk Management Framework uses a lifecycle approach built around governing, mapping, measuring and managing AI risk rather than treating assessment as a single approval point. 

NIST is also revising the framework and developing a dedicated profile for trustworthy AI in critical infrastructure. Its April 2026 concept note is especially revealing. Proposed examples include validated guardrails for autonomous cybersecurity agents, monitoring systems for changes outside verified operating conditions and traceable, auditable rationales for AI-assisted recommendations. 

None of those approaches requires an organisation to claim that every part of the AI is completely understood. They give governance something else to work with: evidence about where the system has been tested, what it’s allowed to do and whether the conditions supporting that trust still hold. 

That evidence becomes especially important when somebody later asks why the organisation allowed the system to operate in the first place.

Governance Has To Be Defensible Even When Understanding Is Incomplete

Good governance has never been about proving that nothing can go wrong. Organisations approve investments, suppliers, technologies and business decisions knowing there will always be uncertainty somewhere. What they need to demonstrate is that the decision was reasonable. 

AI adds a difficult variation because some of the uncertainty may sit inside the system being governed. The organisation can’t simply assume future technical progress will remove it before difficult decisions need to be made. 

A defensible governance decision therefore needs a record of what was known at the time, what remained uncertain and why the evidence available was considered sufficient for the authority granted. That shifts the conversation with boards, auditors and regulators. Instead of claiming, “We understand the system”, leaders may need to explain something more useful: 

“We understand enough about this system, its limits and the consequences of its use to justify the role we’ve given it.” The distinction is small on paper. Operationally, it’s significant. It requires organisations to make uncertainty visible in the decision itself rather than allowing it to disappear somewhere between technical teams, vendors and governance committees. 

It also creates a basis for revisiting the decision when the evidence changes. As AI becomes more embedded in consequential environments, regulators and standards bodies are also asking organisations to demonstrate stronger forms of transparency, accountability and risk management. 

NIST’s current critical-infrastructure work is one example of that direction, particularly because it connects trustworthy AI to repeatable lifecycle practices rather than a one-off claim of safety. The organisations best positioned for that environment won’t necessarily be the ones that can explain every parameter inside every model. They’ll be the ones that can explain their own decisions.

Final Thoughts: Good Governance Doesn't Require Perfect Understanding

We started with a comfortable assumption: if something important happens inside an enterprise system, somebody can eventually work out exactly why. AI is making that harder to promise. Some gaps in understanding can and should be closed. Better documentation, clearer ownership, stronger vendor information and more joined-up governance can remove uncertainty that never needed to exist in the first place. 

Other limits run deeper. Current AI systems can behave in ways that aren’t completely predictable or explainable, even to the organisations building them. Waiting for perfect understanding before making any governance decision may therefore become as unrealistic as ignoring uncertainty altogether. 

The more useful standard sits between those extremes. Good governance needs enough evidence to justify the authority a system has been given, clear responsibility for the uncertainty that remains and defined boundaries for the point where that uncertainty becomes unacceptable. That changes the meaning of control. 

It becomes less about proving that everything is known and more about being able to show why the organisation acted responsibly given what could reasonably be known. As AI takes on more operational responsibility, that distinction will become harder for technology leaders to avoid. 

EM360Tech will continue following how governance, assurance and enterprise decision-making evolve as organisations learn to manage systems whose capabilities may increasingly exceed their ability to completely explain them.