The Explainability Gap: When Enterprises Can No Longer Audit Their Own Systems

Enterprise systems have always left clues behind. An invoice has a transaction record. A login has a timestamp. A database change can usually be traced back to a user, process or application. If something important goes wrong three months later, an investigator can work backwards through those records and piece together what happened.

We’ve built decades of governance, compliance and audit practice around that basic expectation. AI makes it more complicated. Imagine an AI system rejects a customer application today. The customer challenges the decision three months from now, and the business needs to explain why it happened.

The system may be perfectly capable of producing an answer. Perhaps it says the application was rejected because the customer didn't meet a certain threshold. But can the business prove that?

Can it show which model made the recommendation, which version was running, what information it received, which records it retrieved, what instructions it followed and which controls were active at the time? Can it show whether a person reviewed the result, which permissions were used and what happened after the AI made its decision?

Those are very different questions. And this is where the AI explainability problem is starting to become something bigger. Enterprises don't only need systems that can explain their outputs. They increasingly need enough trustworthy evidence to reconstruct what those systems actually did.

Explainability And Auditability Aren't The Same Thing

Explainability tends to get used as a catch-all term for almost anything involving transparency around AI. But several different capabilities are bundled together when we talk about understanding these systems.

• Explainability helps someone understand why an AI system produced a particular result.
• Observability gives teams visibility into how the system behaves while it's operating.
• Accountability establishes who remains responsible for its decisions and consequences.

AI auditability is the ability to preserve enough reliable evidence to reconstruct and independently examine what an AI system did, what influenced it and which controls applied at the time.

There’s obviously some overlap. An explanation can be useful evidence during an audit. But it isn't the audit trail itself. This distinction becomes especially important with generative and reasoning models because an explanation produced by the model shouldn't automatically be treated as a literal transcript of the computation that produced its answer.

Anthropic has been researching this problem directly. In 2025, researchers found that reasoning models don't always faithfully report the information influencing their answers. More recent work published in July 2026 identified internal model activity that doesn't necessarily appear in the model's output at all.

OpenAI is investigating a related question through research into whether reasoning traces can be monitored reliably as models become more capable. None of this means model explanations are useless. Far from it. It means enterprises need to be careful about what they're asking an explanation to prove.

If a system says, "I rejected the application because X", that gives an investigator somewhere to start. It doesn't independently prove that X was the only factor involved. And once AI becomes part of a larger enterprise workflow, the model itself is only one piece of the story.

Enterprise AI Has An Audit Readiness Problem

For a long time, most enterprise AI systems didn't have much authority. They generated content. Summarised documents. Answered questions. Helped employees analyse information. If something went wrong, the consequences were often limited because a person still decided what happened next. 

That boundary is changing though. AI is moving deeper into workflows where systems recommend, approve, reject, classify, route, modify and increasingly execute actions on behalf of people. Agentic AI pushes this further by giving systems access to tools and applications so they can act rather than simply suggest what somebody else should do.

The governance structures around those systems aren't always developing at the same speed. Grant Thornton's 2026 AI Impact Survey of 950 senior US executives found that 78 per cent of organisations lacked strong confidence that they could pass an independent AI governance audit within 90 days. 

The same research found a considerable difference between AI maturity levels. Only seven per cent of organisations still piloting AI were very confident they could pass an audit, compared with 74 per cent of those reporting fully integrated AI.

That comparison is useful because it complicates the idea that more AI automatically creates less control. Because it doesn't have to. Organisations that mature their AI operations can mature their governance at the same time. The problem appears when one progresses without the other.

Agentic AI makes that mismatch easier to see. Deloitte's 2026 State of AI in the Enterprise research found that only 21 per cent of surveyed organisations had a mature governance model for autonomous agents. Governance capabilities and oversight were a concern for 46 per cent, while the same proportion cited model quality, consistency and explainability among their AI risks.

A chatbot drafting an internal email doesn't create the same audit burden as an agent changing production infrastructure. Nor does an AI recommendation automatically carry the same consequence as one used to make a customer-facing financial decision. But as authority increases, so does the need to know exactly what happened.

And that's where a conventional log of the final outcome starts looking rather thin.

An AI Audit Trail Needs More Than A Final Answer

Say an enterprise system contains the record: Application rejected.

Technically, the outcome has been logged. But imagine you're the person investigating that decision 90 days later. You know what happened. You don't yet know why. Perhaps the AI analysed information submitted by the applicant, retrieved several records from internal systems, applied a risk policy, called an external service to verify something and generated a recommendation. 

Another automated process acted on that recommendation, while a human only became involved if the case crossed a particular threshold. Suddenly, the line saying "application rejected" isn't really an audit trail. It's the last page of a story with most of the chapters missing.

For more consequential use cases, enterprises may need what we can call an AI evidence chain: connected records that allow an independent person to reconstruct a material AI-influenced decision or action from beginning to end.

What did the system know?

The first question is deceptively simple. What information did the AI actually have when it produced the result? That isn't necessarily the same as asking what information existed. 

A customer record may have contained 40 fields, but perhaps only 15 were passed to the model. A retrieval system may have searched 10,000 documents but selected four for the AI's context. An external database may have contained one value at the time and a completely different one three months later. So the evidence may need to show:

• What input data the system received.
• Where that data came from.
• Which records or documents were retrieved.
• What contextual information was supplied.
• Whether any of those sources subsequently changed.

This is where data provenance, meaning evidence of where information came from and how it moved through a system, starts becoming part of AI governance rather than something only data teams worry about. Without it, an enterprise may know what the model could theoretically access without being able to establish what influenced this particular event.

What system made the decision?

"AI system" can sound like a single thing. In practice, the answer might involve a foundation model from one provider, a retrieval layer maintained internally, system instructions written by another team, a policy engine, several APIs and configuration settings changed at different times.

Any one of those components could have changed since the original decision. That makes version information historical evidence. The organisation may need to know which model and configuration were active, which instructions applied, what retrieval setup was being used and whether a third-party provider had changed anything about the underlying service.

This isn't version control for the sake of neat engineering records. It's about being able to place the event back into the technological environment where it actually happened. Otherwise, an investigator may be examining today's system while trying to explain yesterday's decision.

What did the system do?

The audit question changes again once AI can act. A recommendation engine produces an output. An agent may open a file, query a database, send information to an API, alter a configuration, update a customer account or trigger another automated process. Now the enterprise needs action provenance as well as decision provenance.

That can mean recording which tools were called, which actions were attempted, what actually completed, which identity the agent operated under and what permissions were available at each point. Sequence becomes important too. If an agent performs six actions across four systems, knowing that all six happened isn't necessarily enough. 

The order may explain why the final outcome occurred. Traditional identity and access management already asks who did what and under which permissions. Agentic AI adds another question: when software is acting on someone's behalf, whose authority is really being exercised?

Which controls actually operated?

Most mature organisations already have controls written into policies. Certain decisions need approval. Sensitive information can't be sent to particular services. High-risk actions require additional validation. Privileged changes may have to pass security checks. The useful audit question isn't whether those controls exist on paper.

It's whether they ran when this event happened. That may require records showing which policy checks were applied, whether validation passed, what security restrictions were triggered and whether the system escalated the event when it was supposed to. Human oversight belongs in the same evidence chain. 

If a person approved, rejected, corrected or overrode an AI output, the organisation may need to know who intervened and what changed as a result. Put together, these records create something much more useful than a collection of disconnected logs. They create a defensible account of the event.

Reconstructing An AI Decision May Be Harder Than Reproducing It

When normal software behaves unexpectedly, one of the first troubleshooting steps is often to recreate the problem. Use the same version. Enter the same data. Follow the same steps. See whether it happens again. With AI, that approach gets less reliable. Part of the problem is the probabilistic nature of many AI systems, which means the same input doesn't always guarantee exactly the same output. 

But even that is only part of the story. The environment around the model keeps moving too. The documents available to a retrieval system can change. External data gets updated. Models are replaced. Providers alter services. System instructions get rewritten. Permissions change. Agent memory evolves. APIs start returning different information.

So an organisation trying to investigate a decision from three months ago may no longer be able to recreate the exact environment that produced it. That doesn't make the original event impossible to audit. It changes the objective.

Instead of depending on the system to reproduce the same answer, the organisation needs enough AI traceability to reconstruct the conditions under which the original answer was produced. Think of the difference between replaying an event and rebuilding its history. One requires the system to behave exactly as it did before.

The other requires the enterprise to have kept reliable evidence of what was true at the time. The second approach becomes much more dependable once the systems themselves are constantly changing. Which also means the evidence can't be an afterthought collected only when somebody asks for it.

By then, the parts you needed may already be gone.

Third-Party AI Creates An Evidence Problem Enterprises Still Own

Few enterprises control every component of their AI stack. They use foundation models from external providers. AI features arrive inside SaaS platforms. Agents call third-party APIs. Managed services store logs and handle parts of the processing. Operationally, that makes sense. From an audit perspective, it creates an awkward question.

What happens when your organisation is accountable for a decision, but some of the evidence required to explain it belongs to somebody else? NIST's AI Risk Management Framework Playbook treats pretrained and third-party models as ongoing risk dependencies. 

Its guidance calls for organisations to identify those models, track their risks and independently monitor their performance and trustworthiness alongside the surrounding components. The principle is simple enough. Outsourcing the technology doesn't automatically outsource accountability.

A provider might update a foundation model without giving customers access to the previous version. A SaaS platform may keep operational records for 30 days while your organisation needs them for considerably longer. Some proprietary systems may expose useful output logs while revealing very little about model changes or internal configuration.

If those limits only become visible after an incident, there may be nothing the enterprise can do to reconstruct the missing evidence. That turns third-party AI risk into a procurement and architecture problem as much as a compliance one.

Questions to ask AI vendors before the audit

Most vendor assessments already cover security, availability, data protection and service levels. AI auditability adds another set of questions:

• Can we identify which model and version processed a historical event?
• Will you tell us when models or important configurations change?
• Which logs and records can we export?
• How long do you retain them?
• Can retention periods align with our legal and organisational requirements?
• What records remain available if we stop using the service?
• Can we trace tool calls and actions performed through the platform?
• How is sensitive information protected inside audit records?
• What evidence can you provide about model evaluation and change management?

Not every product needs to answer every question to the same depth. But enterprises do need to understand where the boundary sits before they place a consequential workflow on the other side of it. Finding out that a supplier never retained the evidence you need isn't especially helpful when the auditor is already in the room.

Regulation Is Turning AI Governance Into An Evidence Requirement

AI governance has spent years being discussed in principles. Fairness. Transparency. Accountability. Human oversight. Risk management. Those principles are still important, but regulatory and standards frameworks are increasingly getting into the less glamorous question of how organisations prove they're actually following them.

NIST's AI Risk Management Framework is voluntary, but its Playbook provides a useful picture of this shift. It connects governance with documentation, risk tracking, monitoring and records that can support independent assessment rather than treating governance as a standalone policy exercise.

The European Union's AI Act takes that direction into regulation. For covered high-risk AI systems, the Act includes requirements for systems to support automatic logging and for providers to retain automatically generated logs where those logs are under their control. The implementation timetable has changed, however. 

Following changes associated with the AI Omnibus, rules for Annex III high-risk systems are now due to apply from 2 December 2027, while high-risk AI embedded in regulated products moves to 2 August 2028. Separate AI Act transparency requirements began applying in August 2026.

The standards environment is heading in a similar direction. ISO/IEC 42005:2025 provides guidance for AI system impact assessments, including identifying, evaluating and documenting potential impacts throughout an AI system's lifecycle. 

ISO/IEC 42006:2025 goes one step further by defining requirements for organisations that audit and certify AI management systems against ISO/IEC 42001. Financial services offers an early view of what this looks like when AI meets an industry that already lives with substantial regulatory scrutiny. 

In February 2026, the US Department of the Treasury announced new financial-sector AI resources developed through public-private collaboration, with work covering areas including explainability, identity, fraud and data practices. There’s a pattern forming across these different efforts.

Governance defines what the organisation expects to happen. An audit asks for evidence of what actually did. And once that evidence becomes part of the requirement, simply having the right policy isn't enough.

More Logging Won't Fix The Explainability Gap

The obvious response to all of this is to log everything. Every prompt. Every output. Every model call. Every retrieved document. Every tool action. Every piece of context. Keep it forever, just in case somebody needs it. Problem solved. Except now you've created another one.

Enterprise AI systems can process customer information, employee records, commercial secrets, internal documents, credentials and all kinds of data you probably don't want sitting indefinitely inside an enormous archive. The audit trail itself can become sensitive. 

Keeping unnecessary information also increases storage requirements, expands the amount of data that needs securing and can clash with privacy or records-management obligations. And there’s another issue that's easier to miss. Audit evidence only helps if you can trust it. 

If records can be altered, deleted or disconnected from the event they're supposed to document, collecting more of them doesn't suddenly make the system auditable. So AI auditability isn't a data-hoarding exercise. It's an evidence design problem.

The amount and type of evidence an organisation preserves should reflect the authority of the AI system, the consequence of its decisions and any regulatory requirements surrounding the use case. 

• An internal tool summarising meeting notes might need fairly basic operational records.
• An AI system recommending whether somebody receives a financial product sits somewhere very different.
• An autonomous agent capable of changing production infrastructure needs stronger records again, because the system isn't merely producing information. It's exercising authority inside a live enterprise environment.

Treating all three the same either creates unnecessary governance overhead or leaves the most consequential systems under-controlled. A better approach is to scale auditability with risk.

Auditability Has To Be Designed Around Risk

The simplest way to begin is to ask two questions about every AI system.

• How much authority does it have?
• And what happens if it's wrong?

The answers create a basic AI Auditability Matrix.

AI authority describes how much the system can decide or do without somebody stepping in. Business consequence describes what happens if the result is wrong, challenged or investigated. Regulatory exposure can move a use case into a higher evidence category regardless of where it would otherwise sit.

Take an internal summarisation assistant. It has little authority and relatively limited consequences if it occasionally gets something wrong. Basic records around system use and configuration may be enough. Now consider an AI recommendation shown to an employee making an important customer decision. 

The person still holds the final authority, but the recommendation may influence an outcome with much greater consequences. Strong decision provenance becomes more useful because the organisation may eventually need to establish what information shaped that recommendation.

An AI system making or heavily influencing the customer-facing decision increases the requirement again. Then there are agents. An autonomous system modifying infrastructure, moving money or changing access rights combines high authority with potentially high consequences. 

Here, the enterprise may need the complete evidence chain: data, system versions, instructions, actions, permissions, controls, human interventions and downstream outcomes, all preserved in a form suitable for independent review. The matrix isn't meant to create another compliance score.

It's a way for technology, security, risk, compliance and procurement teams to have the same conversation before deployment. Because once everyone agrees on the level of authority being delegated and the consequences attached to it, the next question becomes much easier: What would we need to prove if this decision were challenged six months from now?

Designing the answer into the architecture is far easier than trying to recover it afterwards.

The 90-Day AI Reconstruction Test

There’s a fairly simple way to find out how auditable your current AI environment really is. Pick one material AI-influenced decision or action that happened around 90 days ago. Not a demonstration. Not something that happened this morning while the logs are still conveniently sitting in front of everyone.

Choose something old enough that normal system changes, retention policies and staff memory have had time to intervene. Then try to establish:

1. Which AI system and version were involved?
2. What information did the system actually receive?
3. Which instructions and policies applied at the time?
4. Which external systems, tools or data sources were involved?
5. What actions did the AI attempt or complete?
6. Which identity and permissions did it operate under?
7. Which controls ran?
8. Did a human review, approve, reject or change the result?
9. What happened downstream?
10. Are the records complete and trustworthy enough for somebody independent to examine?

The interesting part isn't how many boxes you can tick. Pay attention to how you find the answers. If somebody has to remember what the configuration probably looked like, that's useful operational knowledge but weak audit evidence. 

If three teams have to manually piece together logs from separate platforms, you technically have records but may not have a dependable evidence chain. And if the only explanation available comes from asking the AI why it made the decision, you're relying on the system under investigation to provide the evidence for its own defence.

The goal isn't a perfect score. It's finding the point where evidence disappears. Because that point tells you where your organisation's explainability gap actually begins.

Final Thoughts: Accountability Depends On Evidence

Enterprise systems leave records behind for a reason. Decisions get challenged, incidents get investigated and regulators, customers and boards may eventually want to know what happened and why. AI doesn't remove those responsibilities. It changes what organisations need to preserve to meet them. 

An explainable AI system can help someone understand an outcome. An auditable AI system provides enough evidence to establish what happened, what influenced it, which controls operated and where responsibility sat. 

As AI becomes more connected and autonomous, that evidence reaches beyond the model into data, instructions, identities, permissions, tools, policies and human decisions. That's why auditability can't simply be added afterwards. By the time somebody asks for evidence, models may have changed, records may have expired and suppliers may have moved on.

Enterprises don't need complete knowledge of every calculation inside every AI model. They need enough evidence to defend consequential decisions without relying on the AI to explain itself. Today, enterprises are still asking whether AI can be explained. 

As these systems gain more authority, regulators, auditors, customers and boards may start asking a harder question: can you prove it? EM360Tech will continue following how that question reshapes AI governance and enterprise infrastructure as organisations give increasingly autonomous systems more responsibility.