Key Takeaways:
AI-agent data exposure is often an authorization failure, not an AI-specific failure. An agent may use its own broad service or non-human identity permissions to retrieve data, so a low-privilege user can receive material they could not access directly unless retrieval is filtered against the requesting user’s actual permissions.
Performance evaluations do not substitute for adversarial security testing. An assistant can be accurate, helpful, and operationally successful in normal testing while still leaking restricted information, following malicious instructions embedded in ingested content, or performing actions beyond a user’s authorization.
Test the agent across identity boundaries and privilege levels before production. Use high- and low-privilege accounts to submit identical data-specific queries, compare results with each account’s native access, test indirect prompt injection through trusted content, and validate that agent credentials follow least privilege and cannot be misused for privileged actions.
AI agents introduce new ways for attackers to interact with systems, but not every AI security problem is new.
A recent Azure OpenAI incident covered by VentureBeat is a good example. An IT architect built an AI email assistant integrated with SharePoint. The deployment passed the benchmarks and evaluations his team had designed and was successfully resolving roughly 60% of inbound customer email.
Then he tested it with a low-privilege account.
He asked the AI assistant the same questions that a high-privilege user had asked previously. The AI assistant returned proprietary SharePoint data the low-privilege user couldn’t access directly.
That’s not fundamentally an AI problem. It’s an authorization problem.
The Agent Had More Access Than the User
AI agents often operate using non-human identities (NHIs) with broad, persistent privileges. They need those permissions to perform tasks across different systems and data repositories.
The problem is what happens when the application uses those privileges to retrieve information without enforcing the permissions of the person making the request.
As Netragard founder and CEO Adriel Desautels explained to VentureBeat:
“If the NHI credentials usually have broad authorization and can read high privilege data then that is then stored in their index. If an app does not enforce identity-aware retrieval, then a ‘normal’ user with lower permissions can query the app and access otherwise restricted data. This collapses authorization boundaries down to the lowest privilege level with search capability.”
The low-privilege user didn’t need to compromise an account, bypass authentication, or find a zero-day. The application was retrieving information using permissions the user didn’t have.
That distinction is important because it changes what needs to be tested.
Why AI Evaluations Can Miss Security Problems
Most AI evaluations are designed to determine whether a system performs the task it was built to perform. Is the answer accurate? Is it relevant? Did the agent complete the task?
Those are useful questions, but they aren’t security questions.
A red team wants to know what happens when someone intentionally uses the system in ways the development team didn’t anticipate. What information can a low-privilege user retrieve? What instructions can be introduced through data the agent consumes? What actions can the agent perform using its own credentials?
The attack surface also extends beyond the prompt itself. Prompts, outputs, transcripts, memory state, execution logs, connected systems, and the agent’s underlying permissions can all create opportunities for abuse.
An agent can perform extremely well in normal evaluations while still exposing information or functionality it shouldn’t.
How a Red Team Would Test an AI Agent
Finding the retrieval problem described by VentureBeat doesn’t necessarily require specialized AI security tooling.
A simple place to start is with two accounts:
- Create a high-privilege account with access to sensitive internal repositories and a low-privilege account with baseline access.
- Submit identical, data-specific queries to the agent from both accounts.
- Compare the agent’s responses with what each account can access directly.
If the low-privilege account receives information from files or repositories it cannot access natively, the agent isn’t properly enforcing the user’s authorization boundary.
That’s only the beginning.
As Adriel explained to VentureBeat:
“The first test would likely target the gaps between data and instructions, and the gaps between the user’s identity and the assistant’s own credentials. We’d attempt to plant an instruction within content that we think the assistant will ingest as data. We’d have that content direct a side-effectful, privileged action that the attacking user is not authorized to perform.”
Now you’re testing more than information retrieval. You’re testing whether an attacker can use content the agent trusts to influence what the agent does with its own privileges.
That’s the type of behavior normal performance evaluations aren’t designed to uncover.
AI Doesn’t Replace Security Fundamentals
In the VentureBeat example, fixing the retrieval issue didn’t require replacing the AI platform or introducing an entirely new security stack.
The architect added a query-time filter that checked the requesting user’s SharePoint permissions before source material entered the context window. The assistant continued resolving roughly 60% of customer emails, but the unauthorized retrieval path was closed.
The underlying security practices are familiar:
- Apply least privilege to agent and service credentials.
- Enforce the requesting user’s identity and permissions when retrieving data.
- Don’t assume that because an agent is authorized to access something, the person interacting with the agent should be able to access it too.
- Test what happens when instructions are embedded in data the agent consumes.
- Test the system from accounts with different privilege levels.
- Evaluate what actions the agent can perform, not just what answers it provides.
AI agents create new combinations of data, identity, instructions, and execution. That makes adversarial testing important, but it doesn’t mean organizations should abandon the security practices they already know.
Before putting an AI agent into production, test what it can access as different users, what instructions it will follow, and what actions it can take.
Performance evaluations won’t answer those questions. Adversarial testing will.
Read the full VentureBeat story: Closing an Azure OpenAI assistant’s retrieval gap didn’t take a new identity platform. It took one filter and a narrower assistant.
FAQ
Is an AI agent data leak always an AI security issue?
Not necessarily. Many AI-agent data leaks are traditional authorization failures: the agent retrieves content using its own broad service credentials without checking whether the requesting user is permitted to access that content. AI changes the interface and attack path, but the core control remains identity-aware access enforcement.
How can an organization test whether an AI agent respects user permissions?
Create accounts with meaningfully different permissions such as a high-privilege user with access to sensitive SharePoint repositories and a baseline user without that access. Submit identical, data-specific queries through the agent, then compare each response with the data that account can access directly. Any restricted content returned to the lower-privilege account indicates a retrieval authorization gap.
What should security teams test beyond prompt injection?
Test the full agent attack surface: connected data sources, retrieval and indexing behavior, non-human identity permissions, conversation memory, logs and transcripts, tool integrations, and the actions the agent can execute. Also test whether instructions embedded in documents, emails, or other ingested content can influence the agent to disclose data or perform actions beyond the user’s authorization.
