Security investigations have traditionally followed a fairly predictable process. An alert comes in, a security analyst checks the relevant systems, gathers evidence and follows a set of steps to work out what happened. These steps are usually documented in a playbook, so the team knows what to look for and how to respond.
But AI security investigations are starting to work differently. Instead of following a fixed sequence, AI investigators can decide which tools to use, what information to check and which leads are worth pursuing. That gives security teams more flexibility, particularly when an incident doesn't fit a familiar pattern. It also creates a problem.
If two AI investigators take completely different routes, how can you tell whether they've both done a thorough job? The answer depends on what the investigation actually proves.
Why Following the Right Steps No Longer Proves an Investigation Is Complete
Traditional security automation relies heavily on predefined workflows. Security orchestration, automation and response (SOAR) tools, for example, can follow a playbook that tells them which actions to take when a particular alert appears. These workflows are useful because they're predictable.
But they also depend on someone knowing which steps are needed beforehand. An unfamiliar attack, or one that doesn't behave as expected, may need a different approach. Autonomous security investigations give AI agents more freedom to make those decisions as they go.
An agent might start with a suspicious login, check activity on the affected device and then investigate connections to other systems. Another might begin with network activity and work backwards to identify the compromised account. Both could reach the right conclusion.
But there's a difference between identifying an attack and understanding how far it went. Researchers highlighted this problem in the April 2026 SIR-Bench investigation benchmark. Their evaluation separates three things: whether an AI investigator classifies an alert correctly, whether it discovers new evidence and whether it uses the right tools to investigate.
It's an important distinction because an agent could correctly identify a threat using information already provided in the alert. That doesn't tell you whether it investigated the incident properly or discovered anything the security team didn't already know.
What Makes an AI Security Investigation Complete?
A complete investigation needs enough evidence to establish what happened and how much of the organisation may have been affected. But there's another consideration. The evidence also needs to support whatever the investigator claims in its final report. Those are two separate problems, and recent research suggests AI investigators can struggle with both.
Did the investigator find the evidence it needed?
Imagine an AI investigator correctly identifies that someone has accessed an employee's account using stolen credentials. It checks the login records, confirms the suspicious activity and reports an account compromise. So far, so good. But what if the attacker also accessed a cloud application or used that account to reach another system?
If the AI never checks those records, the investigation could miss part of the attack. Security teams collect information from many different places, including employee devices, identity systems, cloud services and network activity. This information is called security telemetry, and it provides the evidence investigators use to piece together an incident.
The difficulty is that having the information available doesn't mean an AI investigator will find it. A September 2026 study called APTInvestBench tested autonomous investigation agents across 370 experimental cases using different combinations of security logs.
Across eleven language models, the agents gathered enough evidence for an average of just 44.3 per cent of the attack actions that could have been investigated using the available records.
What Is Reflection AI? Beam, Nvidia and the Race for Open-Weight AI
Reflection AI’s first open-weight model targets coding and AI agents, while reported talks with Nvidia highlight the strategic importance of more customisable AI systems.
These were experimental results, not a measure of commercial security products. Still, they demonstrate why investigation completeness needs to be assessed separately from whether the AI reaches a correct initial verdict.
Does the evidence actually support the findings?
Finding relevant records is only part of the job. An investigator also needs to show how those records support its conclusions. For example, a suspicious login might confirm that an account was accessed. It wouldn't necessarily prove that sensitive files were stolen. Establishing that would require further evidence, such as records showing which files were accessed or transferred.
APTInvestBench found another gap here. Although agents retrieved enough evidence for 44.3 per cent of recoverable attack actions, the evidence cited in their final reports adequately supported only 25 per cent. This creates a difficult situation for security teams.
An AI investigator might report something that appears reasonable, or even happens to be correct, without providing enough evidence to justify it. There's also the question of missing information. If certain logs weren't collected or have already been deleted, the investigator may simply be unable to establish what happened.
A reliable report needs to make that clear rather than treating an unanswered question as a confirmed finding.
How Can Security Teams Test Autonomous Investigations?
When an AI investigator chooses its own approach, checking whether it followed a particular sequence becomes less useful. Security teams need another way to establish whether the investigation was thorough. One approach is to test the system against incidents where the facts are already known.
This makes it possible to compare the AI's findings with the evidence it should have discovered, regardless of which investigative route it took.
Test against incidents with known findings
Inside Agentic AI Due Diligence
Why boards must demand AI observability, real automation metrics and proofs of agent capability before backing new enterprise platforms.
Security teams can use previously investigated incidents or controlled attack simulations to evaluate an AI investigator. The important part is having a reliable record of what happened, which systems were affected and what evidence was available. For example, a test could involve an attacker compromising an account, accessing a cloud application and downloading a sensitive file.
The AI investigator would receive an initial alert and access to the relevant security tools. The team could then assess whether it identified the compromised account, discovered the cloud activity and found evidence of the file download. They could also check whether it used suitable tools and missed any important leads.
SIR-Bench uses a similar principle. Its researchers developed 794 test cases from 129 anonymised incident patterns, using controlled cloud environments to recreate attacks and generate security records. This allows investigators to be assessed against known facts without requiring them to follow identical steps.
An agent that starts with identity records and another that begins with endpoint activity can both succeed, provided they establish the necessary findings. However, these tests need to reflect the systems and incidents an organisation actually deals with. A strong result in a research benchmark doesn't guarantee the same performance in a company's own environment.
Check what changes when the investigation changes
Passing a test once isn't enough to establish whether an AI investigator will perform consistently. Security teams should repeat investigations under different conditions. That could mean changing the initial alert, removing a source of security logs or testing what happens when the agent chooses a different investigative path.
APTInvestBench demonstrates why this is useful. When researchers moved from full telemetry to endpoint-only records, overall evidence coverage fell by just 1.6 percentage points. However, 35.5 per cent of previously supported attack actions lost sufficient citation support, despite the necessary evidence remaining recoverable.
In other words, the overall score changed very little, even though the support for individual findings became less reliable. Organisations also need to repeat evaluations when AI models, connected tools or security systems change.
Autonomous Agents Versus Safety
Two coding incidents show AI systems altering oversight and misreporting changes, sharpening questions about alignment and control.
The NIST AI Risk Management Framework recommends testing AI systems before deployment and regularly while they're operating, including measuring uncertainty and comparing performance against established benchmarks. For security investigations, that means checking whether the agent can still establish the required facts and recognise when it can't.
An investigator should also know when to hand an unresolved case to an analyst. And although it may have permission to choose its own investigative steps, that doesn't automatically mean it should have permission to disable accounts, isolate devices or take other actions to contain an attack.
What Should Security Leaders Ask Before Trusting the Results?
Evaluating AI investigation quality means looking beyond the final verdict or the number of alerts an agent can process. Security leaders need to understand what the system has actually established and whether its conclusions can be relied on. A useful starting point is to ask:
- What evidence supports the findings? Can the investigator identify the records behind its conclusions, rather than simply provide a convincing explanation?
- What might it have missed? Does it investigate beyond the original alert and check other relevant systems to establish the incident's scope?
- What couldn't it establish? Can it distinguish between a finding that wasn't supported and one that couldn't be investigated because records were missing?
- How has its performance been tested? Has it been evaluated against incidents with known findings, including cases where the evidence is incomplete?
- When does it need an analyst? Are there clear conditions for escalating investigations that remain uncertain or don't have enough supporting evidence?
These questions also help organisations decide which results can be accepted and which need further investigation. An AI system that can recognise the limits of its findings may be more useful than one that always produces a confident answer. Particularly when security teams need to decide whether an incident has been contained or whether an attacker may still have access.
Final Thoughts: Investigation Quality Matters More Than Following a Playbook
How to Secure AI Agents: 5 Identity Risks Enterprises Need to Address
As AI agents gain access to enterprise systems and sensitive data, poorly managed identities and permissions can create new security risks. Here are five threats organisations should address and how to reduce their exposure.
Security playbooks aren't going away. They still provide useful guidance for known threats and established response procedures. But as AI investigators become more capable of choosing their own methods, following a predefined process won't always be the best way to judge their work.
What security teams need is confidence that an investigation has established the relevant facts, found the evidence available to it and been clear about anything it couldn't prove. That raises an interesting question for the future of autonomous security investigations.
If two systems reach the same conclusion, but only one can properly support it, should we really consider them equally successful? As those questions become part of everyday security decisions, EM360Tech continues the conversation with the practitioners and technology leaders working through what reliable AI-driven security operations should look like.
Comments ( 0 )