Imagine your company uses AI to review customer applications against its internal policies. Someone submits an application, the system checks the details and comes back with a decision. It even tells you which policy it used and explains how it reached its conclusion. Everything looks right.
The decision matches what your compliance team would have said, the policy reference is correct, and there's a neat explanation ready for anyone who needs to review it later. You'd probably be fairly happy with that result. But what happens if you change the policy?
Would the AI reach a different decision, or would it carry on giving the same answer?
It's a question that two studies published in October 2026 have raised about AI compliance decisions. Researchers found that some models could identify the correct legal or regulatory rules without consistently responding when those rules changed. For organisations considering AI in regulated workflows, this raises a rather uncomfortable possibility.
You could have a system that gets the answer right, looks like it's following the rules and still hasn't demonstrated that those rules are driving its decisions.
What Does It Mean for an AI Decision to Follow the Rules?
Think about how a compliance officer would assess an application. They'd need to know what the applicant is asking for, which rules apply and whether anything about the circumstances changes how those rules should be interpreted. If a policy says customers can request a refund within 30 days, someone applying after 25 days would normally qualify.
Someone applying after 40 days wouldn't, unless there's an exception that allows it. The important part is how the rule affects the outcome. Change the refund window to 14 days and the first customer no longer qualifies either. The facts haven't changed, but the requirements have. That relationship is what researchers mean when they talk about rule-grounded AI.
The system's decision should depend on the rules it's been given, including their conditions, limits and exceptions. With a large language model, though, things aren't quite so straightforward. These models learn patterns from enormous amounts of information. They're very good at recognising familiar situations and producing responses that fit them.
And when you give one a policy document, it can use that information to answer questions about the policy. The difficulty is knowing whether it actually relied on the document when making a particular decision.
An AI might recognise that a refund request looks reasonable because it has encountered similar examples before. It could then identify the relevant clause and generate an explanation that fits the answer it has already produced. From the outside, that can look exactly like a properly reasoned decision.
But if the company changes its policy, the model might continue relying on the familiar pattern rather than the new requirements. That's why AI decision accuracy tells us only part of the story. To establish whether a system is applying the rules, we need to see what happens when those rules are no longer the same.
Why Accurate AI Compliance Decisions Can Still Be Unreliable
Researchers have started testing this more directly, rather than relying on whether an AI gives the expected answer. In October 2026, Saisab Sadhu, Shreeyans Arora and Pratinav Seth published Cited but Not Consulted, a study examining whether AI models actually depended on the legal authorities they named in their explanations.
They tested seven open-weight models across four legal and contractual reasoning benchmarks. In the reported experiments, the models correctly identified the governing authority in 66.7 to 100 per cent of generations, depending on the setting. So far, that sounds fairly encouraging.
But the researchers also replaced the legal authority while keeping the facts of the case unchanged. On one benchmark, CaseHOLD, the models' decisions changed in only 0 to 21.7 per cent of these tests. Other benchmarks produced higher rates, which suggests the problem varied depending on the model and the type of legal task.
A related paper, Verdict Without the Rule, took a similar question into regulatory compliance. This time, the researchers tested five models across 20 regulatory and platform-policy domains, removing, replacing or reversing the rules they were given. The general-purpose models achieved between 90 and 92 per cent accuracy in the reported comparison.
Yet their decisions often remained unchanged when the governing rules were altered. Interestingly, the researchers also found that models responded more closely to the rules in cases where removing a rule changed a previously correct prediction. So the findings don't suggest that AI simply ignores instructions or regulations.
Its reliance on them appears to be inconsistent, and some cases are much more revealing than others. Both papers were submitted on 8 October 2026 and share two authors. They're also preprints, meaning their findings haven't yet been through peer review. We can't use them to claim that a particular percentage of enterprise compliance systems are unreliable.
They do, however, expose a weakness in how we might evaluate those systems. If we only check whether the final answers are correct, we could miss situations where the AI isn't responding properly to the information it's supposed to follow.
And in a business where policies change, regulations are updated and exceptions need to be handled correctly, that's something worth testing before relying on the system.
How Can Enterprises Test Whether AI Decisions Follow the Rules?
One way to investigate this is through counterfactual testing. It sounds more complicated than it is. You take a situation, change one relevant condition and see whether the result changes in the way you'd expect. Going back to our refund example, you could give an AI the same customer request under two different versions of the company's policy.
Under the first, the customer qualifies for a refund. Under the second, they don't. If the AI approves both requests, you'd have a reason to investigate further. Perhaps it hasn't recognised the updated policy, or perhaps it's relying on information that was correct under the old rules.
Of course, real compliance work isn't always as simple as checking whether someone applied within a deadline. A company might have different policies for different products, customers or regions. Some rules take priority over others, and exceptions can change an otherwise straightforward decision.
There are also situations where the information provided isn't enough to make a decision at all. That's why an effective AI compliance testing process needs to look beyond a handful of examples where the answer is obvious. Start with cases your compliance team already understands. For each one, establish the correct outcome and the rule that supports it.
Then create variations that test how the system handles different requirements. You might change an eligibility threshold, introduce an exception or replace a policy with a newer version. You could also remove the relevant rule entirely to see whether the system recognises that it no longer has enough information to justify its answer.
Conflicting rules deserve attention too. If a general company policy allows something but a more specific regulatory requirement prohibits it, the system needs to apply the correct authority. It shouldn't simply choose whichever rule appears first or sounds more relevant. There's another side to this testing, though.
A decision shouldn't change just because something in the information changes. Suppose a policy requires applicants to be at least 18 years old. Changing someone's age from 25 to 26 shouldn't affect their eligibility. Changing it from 17 to 19 should, assuming they meet the other requirements.
If you're measuring success by how often the AI changes its answer, you'd miss that distinction entirely. The expected outcome therefore needs to be established independently, preferably by someone who understands the relevant policy or regulation. Otherwise, you're effectively asking the AI to help decide whether its own answer is correct.
It's also worth testing how the system behaves when it encounters uncertainty.
- Does it ask for missing information?
- Does it recognise when a case falls outside the rules it has been given?
- Can it identify when a decision needs someone with the authority to interpret an exception?
Those responses are part of making a compliance decision correctly, even when they don't produce an immediate approval or rejection. And once you've established how the system performs, the testing needs to reflect the way it'll actually be used. Different types of cases, updated policy documents, changes to prompts and new model versions can all affect the results.
Counterfactual tests won't reveal every step of a model's internal reasoning. But they can give compliance teams something they can work with: evidence of whether the system responds appropriately when the requirements change. That evidence becomes particularly useful when someone needs to review or challenge a decision later.
What Evidence Makes an AI Compliance Decision Auditable?
Imagine a customer challenges an application rejection six months after it was made. The company needs to establish which rules applied at the time, what information the AI considered and whether the decision was justified. Finding the original response is a start, but it won't necessarily answer those questions.
Even if the AI included a detailed explanation, that explanation was generated by the same system that made the decision. And as the October research suggests, a convincing account of the rules doesn't guarantee that the rules influenced the outcome. For AI decision auditability, organisations need records that allow someone to reconstruct what happened and examine the evidence behind it.
That means retaining the version of the policy or regulation in force when the decision was made, along with the relevant case details and any supporting documents the system received or retrieved. Depending on the workflow, teams may also need to record the model and configuration used, the decision itself, and any checks or exceptions identified during processing.
Human involvement should be recorded as well. If someone reviewed an unusual case, changed a recommendation or approved an exception, the audit trail should make that clear. There's a useful distinction here between knowing what information the AI had access to and knowing whether it used that information correctly.
A system log might confirm that the right policy was retrieved. It can't, on its own, establish that the model applied it. The EU AI Act provides some regulatory context for this. Article 12 addresses logging capabilities for high-risk AI systems, while Article 14 covers human oversight.
Article 26 also places certain monitoring and record-keeping responsibilities on organisations deploying high-risk systems. The requirements depend on the system's classification and the provisions that apply to it. Those obligations shouldn't be confused with a universal requirement to record every detail of every AI-assisted decision.
A routine internal classification won't necessarily need the same level of evidence as a decision affecting someone's legal rights or access to an essential service. Organisations need to consider the consequences of an error, the applicable requirements and how easily a decision can be checked or reversed.
The aim is to have enough reliable information for someone to understand what happened, investigate a questionable result and establish who was responsible for reviewing it.
When Should AI Compliance Decisions Require Human Review?
Having someone approve every AI-generated compliance decision might sound like the safest option. In practice, it could create an enormous amount of unnecessary work, particularly when the decisions involve clear rules that another system can check independently.
For example, if an AI recommends approving a request because it meets a fixed eligibility threshold, a separate rule-based check could verify the relevant facts and confirm the result. That gives the organisation an independent control without requiring a compliance officer to inspect every straightforward case. But what happens when the rules aren't so clear?
An application might involve an exception that depends on someone's circumstances, or two regulations might appear to point towards different outcomes. There may be missing evidence, or the decision could have consequences that are difficult to reverse.
These are situations where human oversight of AI becomes more important, because applying the rules involves judgement as well as checking facts. The person reviewing the decision needs access to the relevant information and enough authority to challenge the recommendation. Otherwise, human approval can become little more than another step in the process, with the employee effectively trusting the AI because it has already done the work.
Organisations should decide in advance which situations require escalation, who is qualified to review them and what happens when the AI can't reach a reliable conclusion. That decision also needs to account for any legal requirements for human involvement. Some workflows may require human approval regardless of how accurate the system appears to be.
The important thing is that responsibility remains clear. If a decision needs professional judgement, the organisation should know who provides it. And if an automated check is considered sufficient, there should be evidence that the check actually works.
Final Thoughts: Trust Requires Evidence That the Rules Shape the Decision
Going back to that customer application, an AI-generated approval might be completely correct. The policy reference could be accurate, and the explanation might make perfect sense. But if the company can't establish whether the system would respond properly to a different rule, there's still an important gap in what it knows about the decision.
Enterprises don't need to understand every calculation inside an AI model to assess whether it belongs in a compliance workflow. They do need to know how it behaves when requirements change, what evidence supports its decisions and when someone should step in.
And because policies, regulations and AI systems don't stay the same forever, that confidence needs to be maintained through ongoing testing. For organisations adopting AI in regulated environments, trust has to come from what they can demonstrate, rather than how convincing the system sounds.
That's a useful distinction to carry into any discussion about where enterprise AI should be allowed to make decisions. EM360Tech continues to explore the practical questions shaping enterprise AI adoption, including how organisations can build systems they can confidently oversee and challenge.
Comments ( 0 )