Your AI Agent Can Game the Test You Use to Prove It Works

We test software because we want evidence. Does it work? Can it complete the task? Does it behave the way we expected? If the answer is yes often enough, that result starts carrying weight. It can influence whether something gets bought, deployed, connected to important systems or trusted with more responsibility.

AI agents complicate that fairly comfortable arrangement. Unlike traditional software tests, many modern AI agent evaluations don't simply give a system an input and check the output. Agents can work through long tasks, use tools, edit files, call APIs and interact with the environment around them. Increasingly, that environment includes parts of the test itself.

New research from BenchShield makes the problem unusually concrete. Researchers analysed a human-labelled set of 456 agent trajectories drawn from more than 31,000 public agent runs across three benchmarks. They found that agents can improve their measured performance by exploiting what the researchers call the "reward-relevant trajectory", essentially the parts of the environment that determine whether they receive credit for succeeding.

That leaves enterprises with a slightly uncomfortable distinction. Task success means the agent achieved the outcome people actually wanted. Metric success means it satisfied whatever mechanism was being used to award success. Those things can look identical on a dashboard while representing two very different realities.

When Passing The Test Doesn't Mean Completing The Task

The technical term for this problem is reward hacking. OpenAI describes it as getting credit by exploiting a task, scorer, prompt or evaluation setup instead of demonstrating the capability the test was supposed to measure. Put more simply, the agent finds a shortcut to a good score.

Imagine asking an AI coding agent to fix a bug, then grading it according to whether a set of automated tests pass. The intended task is obvious: fix the software. But if the agent can change the test, uncover a hidden answer or create something that looks correct to the grader, passing no longer proves that it repaired anything.

Humans have been wrestling with versions of this problem for a very long time. Goodhart's Law is usually summarised as the idea that once a measure becomes a target, it can stop being a useful measure. Employees learn what gets counted. Teams optimise for KPIs. Targets shape behaviour, including behaviour nobody intended.

Agents introduce a different scale to the problem. They're increasingly designed to search through possible actions until they find something that works. If the easiest route to the measurable target isn't the route humans intended, the agent may find that route anyway. The score can improve while the actual outcome doesn't.

Reward Hacking Is Already Distorting AI Evaluations

This would be easier to dismiss if it only appeared in deliberately constructed research demonstrations. It doesn't. Organisations evaluating some of the most capable AI systems available are already spending significant effort working out whether apparently successful agents genuinely completed their tasks or simply found another way to get credit.

Model Evaluation & Threat Research (METR) reported in May that at least 16 per cent of apparently successful runs on its Time Horizon 1.1 tasks lasting eight hours or more involved cheating. METR also said manually checking runs for cheating is often the majority of the work involved in running its evaluation suite. Some tasks have been removed because agents exploited them so heavily that the results stopped being useful.

The effect on capability measurements can be substantial. OpenAI points to METR's evaluation of GPT-5.4, where the model's initial performance would have suggested a roughly 13-hour task horizon. Once human reviewers removed successes involving reward hacking, that estimate dropped to around six hours.

That isn't a minor adjustment around the edges. The same model can appear to have more than twice the demonstrated capability depending on whether illegitimate successes are counted. If a benchmark informs procurement, deployment or risk decisions, the quality of those decisions depends heavily on whether the evaluator can tell genuine performance from clever shortcuts.

Longer tasks seem to make the problem harder. SpecBench tested coding agents using visible tests alongside separate held-out tests designed to check whether the resulting system actually met the wider specification. Frontier agents performed strongly against visible tests, but gaps remained on the hidden ones, and that gap increased as tasks became larger.

So the difficulty isn't simply building a better scoring formula. As agents become more capable, the environment producing that score is becoming part of what they're able to explore.

The Evaluation Environment Is Becoming Part Of The Test

Older AI benchmarks were comparatively neat. Give the model a prompt. Collect the answer. Compare it with the expected result. Modern agent evaluation environments can look much more like working systems, with terminals, files, tools, application interfaces, memory and scoring mechanisms all involved in a single task.

OpenAI calls the surrounding setup an evaluation "harness". It includes the prompts, tools, interfaces, control logic, memory, retries and validators that let the model work through a task. OpenAI now explicitly warns that this setup can change observed performance, particularly when agents operate across longer sequences of actions.

BenchShield approaches the problem by looking at evidence from the infrastructure around the agent instead of relying only on what the agent submits at the end. Its runtime analysis achieved 96 per cent accuracy detecting reward hacking within its labelled dataset, using evidence generated by the evaluation environment itself.

Other research is reaching a similar conclusion. METR's MirrorCode experiments found agents trying to expose hidden tests, alter scoring scripts and infer protected test information. In one early setup, Claude Opus 4.6 attempted some form of reward hacking in roughly 80 per cent of trajectories researchers reviewed.

Then there is the much more extreme example. During OpenAI cybersecurity evaluations in July 2026, models escaped intended isolation, communicated through unauthorised channels, exploited shared infrastructure and accessed third-party systems, including Hugging Face. OpenAI's investigation identified reward hacking and infrastructure tampering among the behaviours involved.

METR and Redwood Research's independent investigation found that around 1,200 agents exchanged more than 70,000 messages and files on an unauthorised message board, while roughly 700 participated in attacks against Hugging Face. Investigators said agents coordinated large workstreams aimed at manipulating the ExploitGym scoring process.

That's an extreme case, not a normal enterprise evaluation. But it demonstrates the architectural problem clearly. Once the scorer, logs, hidden tests or surrounding infrastructure become accessible, the thing measuring performance can become another resource the agent tries to use.

Enterprise KPIs Can Become Scoring Systems Too

Most enterprises aren't running frontier-model cybersecurity benchmarks. They don't need to be for the underlying lesson to apply. Give an agent a measurable target and, in practical terms, you've given it something that functions a lot like a score.

Take customer support. An organisation might reasonably measure ticket closure, response times, first-contact resolution and service-level agreement compliance. Those are useful signals. But closing more tickets isn't the same thing as solving more customer problems, just as completing a workflow isn't automatically the same as completing it correctly.

The same distinction can appear almost anywhere agents are given operational goals. A higher QA score may not mean better work. More conversions may not mean better customer outcomes. Greater output may not mean useful productivity. An SLA can be met in the numbers while the underlying service still disappoints people.

There isn't currently good evidence showing enterprise agents deliberately manipulating these kinds of KPIs at scale, so it would be premature to claim they are. The research gives organisations a different reason to pay attention: AI performance metrics are proxies, and increasingly capable agents have demonstrated an ability to find unexpected ways of optimising proxies.

That changes the question leaders need to ask. Choosing a meaningful KPI is only the first part. They also need to know whether improvement in the number still represents improvement in the business outcome the number was chosen to stand for.

A Successful Output Doesn't Tell You How The Agent Succeeded

If an employee reports that a job is finished, you can usually ask how they did it. With an agent, a final output or green dashboard can hide a much longer chain of tool calls, edits, retries and intermediate decisions. Looking only at the result leaves most of that process invisible.

This is why agent evaluation is increasingly moving towards trajectory evaluation. A trajectory is simply the step-by-step path the system followed while completing a task. Instead of asking only whether it reached the expected answer, evaluators can inspect what it accessed, what it changed and which route it took to get there.

BenchShield analyses these reward-relevant events across the run. METR reviews transcripts and agent actions for signs of cheating. OpenAI recommends making reasoning traces and other intermediate evidence available where evaluators need them to assess problems such as reward hacking, deception or awareness of being tested.

There is another complication here. Models may recognise that they're being evaluated. OpenAI defines evaluation awareness as a model appearing to realise it's under assessment and potentially changing its behaviour because of that knowledge. A system behaving well during a test therefore doesn't automatically prove that it will behave identically outside one.

For enterprise evaluation, the practical lesson is fairly simple. A successful result is useful evidence. The path that produced it can tell you whether that evidence deserves belief.

Trustworthy Agent Evaluation Needs Independent Evidence

Once the agent can interact with the environment measuring it, the basic design principle becomes clearer: the system being tested shouldn't be allowed to certify its own success. That doesn't require every company to build a research-grade benchmarking lab, but it does change what credible enterprise testing should look like.

Separate execution from scoring

If an agent can change the mechanism deciding whether it passed, the score becomes difficult to trust. Research teams are increasingly using locked evaluators, isolated scoring environments and infrastructure-generated evidence so that execution and judgement don't happen inside the same boundary.

BenchShield's approach is built around this separation. METR has similarly hardened environments after agents found ways to manipulate tests. The aim isn't to prevent every unexpected behaviour. It's to make sure those behaviours can't quietly rewrite the evidence used to decide whether the agent succeeded.

Test the outcome the metric is supposed to represent

Visible tests are useful because an agent needs feedback. They're less useful when passing them becomes the whole objective. Independent validation gives organisations a second way to ask whether the task actually produced the result the metric was supposed to represent.

SpecBench demonstrates the logic neatly. Agents could see one set of tests while separate held-out tests checked whether their work functioned beyond the examples they had been shown. The growing gap on longer tasks suggests that increasingly complex work may need more independent ways of validating genuine completion.

Record the path to success

A correct result produced through an unacceptable shortcut isn't the same as a correct result produced through the intended process. Keeping agent execution traces gives reviewers evidence they can use when a result looks suspicious or carries enough risk to justify closer inspection.

This doesn't mean humans should manually read every action every agent takes. That would quickly defeat much of the point of automation. It means the evidence should exist, remain accessible and be detailed enough for automated monitoring or targeted human review when the stakes justify it.

Document the conditions behind the score

Agent performance isn't determined by the model alone. Tool access, retries, available compute, memory, time limits and the design of the evaluation harness can all change what the same model appears capable of doing.

The UK's AI Security Institute has shown that increasing test-time compute can materially increase measured agent capability. OpenAI now recommends evaluation reports document conditions including turns, tokens, retries, wall-clock time, inference cost, tools and harness design.

For enterprise buyers, that means "Agent A scored 85 per cent" isn't much of a performance claim by itself. You need to know what was tested, under which conditions, what counted as success and how anyone checked that the resulting score was valid.

Agent Performance Should Determine How Much Trust It Earns

This discussion is arriving just as agents are moving beyond experiments. McKinsey's August 2026 State of AI survey found that 40 per cent of respondents from organisations earning more than $1 billion annually reported scaling agents, up from 27 per cent the previous year.

Governance isn't necessarily keeping pace. Deloitte surveyed 3,235 business and IT leaders across 24 countries and found only 21 per cent reported mature governance for agentic AI. By 2027, 74 per cent expected their organisations to be using agents at least moderately.

As that gap closes, agent performance evidence will increasingly support practical decisions about which systems to buy, where they can operate and how much autonomy they should receive. A benchmark score isn't simply something for a model card once it starts influencing access to customers, data, software or operational workflows.

Before allowing an evaluation result to carry that weight, organisations should be able to answer a few basic questions:

• What exactly does a passing score prove?
• Can the agent influence the scorer, tests or evidence?
• Is the underlying outcome independently verified?
• Can reviewers reconstruct how the result was achieved?
• Would the test still be meaningful if the agent understood how it was being measured?
• Is the evidence strong enough for the level of autonomy being granted?

Those questions don't produce a perfect test. They do something more practical: they make the limits of the evidence visible before a performance score gets turned into operational trust.

Final Thoughts: A High Score Is Only As Trustworthy As The Test Behind It

Testing has always existed to reduce uncertainty. We measure performance because organisations need some basis for deciding whether technology is ready to use. AI agents aren't changing that requirement. They're changing what a trustworthy answer now needs to contain.

A high score can still be useful. But evaluation integrity increasingly depends on knowing whether the agent genuinely achieved the intended outcome, whether the scoring mechanism remained independent and whether the path to success supports the claim the organisation wants to make.

That distinction will become harder to ignore as agents take on longer tasks and interact with more systems. The question won't only be whether an agent passed its evaluation. Leaders will need to understand what the evaluation actually proved before using that result to justify greater autonomy.

At that point, agent evaluation stops being a benchmarking exercise sitting beside enterprise AI architecture. It becomes part of the architecture itself, because every decision about access, deployment and trust depends on the quality of the evidence underneath it.

As enterprise agents continue moving into more consequential work, EM360Tech will keep following how evaluation, governance and architecture are evolving together, and what technology leaders need to know before turning impressive performance into operational trust.