Enterprise AI agents fail quietly when secrets and personal data slip into the prompt. The durable fix is a deterministic guard that runs before the model call, not a smarter model.
An ordinary leak
A developer pastes a stack trace into an internal AI assistant. The trace contains the details needed to diagnose a failing service. It also contains a database connection string with a live password.
The assistant responds correctly. No application crashes. No security alert appears. Yet the credential has crossed a boundary it should never have crossed.
That quiet sequence exposes a basic weakness in many AI agent architectures. Teams often concentrate on what a model might generate while paying less attention to the information assembled before the model is called.
For agents that read documents, process tickets, query systems, and invoke tools, the prompt is more than a user’s message. It is a package assembled from several sources, each with its own risks. Protecting that package requires a control outside the model.
An agent’s prompt has many authors
A conventional chatbot usually begins with text entered by a person. An agent may also receive retrieved documents, database records, webpage content, tool responses, conversation history, and instructions generated during earlier steps.
The model sees this material as one combined context. That creates two related security problems.
The first is sensitive data exposure. A user or connected system may supply an API key, password, personal identifier, financial record, or other information that should not be sent to a model. Current application-security guidance on sensitive information disclosure recommends sanitizing inputs and restricting access to sensitive sources rather than relying only on instructions inside the prompt.
The second is prompt injection. Untrusted content can contain instructions intended to alter the agent’s behavior. Those instructions may appear in a user message, a retrieved document, a webpage, or the output of a tool. Prompt-injection guidance treats external content as a security boundary because models do not reliably distinguish instructions from data.
These problems are not identical. One concerns information that should not enter the model. The other concerns information that may manipulate what the model does. In an agent, however, they often meet in the same place: the context assembled immediately before inference.
That is where enforcement should occur.
Why instructions inside the prompt are insufficient
A system prompt can tell a model not to reveal credentials or follow instructions found in retrieved content. Such guidance can improve behavior, but it should not carry the full burden of enforcement.
A model produces probabilistic outputs. Security controls need predictable results.
Once a secret has been included in a request, the application has already surrendered control over where that value may be processed or retained. A warning generated afterward does not reverse the transmission.
The agent loop makes the problem harder. Tool output from one step becomes input to the next. A credential returned by a connector, for example, can be copied into later prompts, traces, evaluation records, or debugging data unless each transition is inspected.
This is why security guidance cautions against treating the system prompt as a security control. Sensitive information and authorization decisions should remain in systems whose behavior can be tested directly.
The model should reason within the boundary. It should not define the boundary.
Put a deterministic guard before inference
A practical response is a deterministic inspection layer between the agent runtime and the model endpoint.
Before every model call, the layer examines the complete context that the agent has assembled. It then makes one of three decisions: allow the request, redact selected values, or block the request.
This does not require the guard to understand every possible security threat. Its first purpose is narrower: stop recognizable high risk material from crossing the boundary.
Block values that cannot safely be transformed
Some information should cause the model call to stop.
Examples include private key blocks, bearer credentials, access tokens with recognizable formats, database connection strings containing passwords, and explicit credential assignments such as password= or api_key= followed by a value.
Pattern matching will not identify every secret. Organizations also use internal token formats, encoded credentials, and values that resemble ordinary text. The rules therefore need to reflect the environment in which the agent operates.
When a high risk pattern is detected, the application should fail closed. The user should receive a specific explanation and an opportunity to remove the sensitive value.
A blocked request is inconvenient. An unnoticed credential disclosure is worse.
Redact information the model needs only in abstract form
Other values can be replaced without destroying the meaning of the request.
An email address can become [REDACTED_EMAIL]. A phone number can become [REDACTED_PHONE]. A payment card value can become [REDACTED_CARD].
Typed placeholders preserve context. The model can still understand that a message contains contact information or that a record refers to a payment method. It does not need the underlying identity or account number to perform many common tasks.
Redaction rules should be chosen carefully. Removing too much information can make the agent ineffective, while removing too little leaves the original risk unchanged. The correct policy depends on the task, the data involved, and the organization’s obligations.
This is consistent with the broader risk management approach in the generative AI profile of the AI Risk Management Framework, which emphasizes controls tied to the system’s context, intended use, and potential harm.
Inspect tool output, not only user input
A guard placed only at the chat interface leaves a large gap.
Tools can return environment variables, internal records, authentication headers, document metadata, or malicious instructions embedded in external content. An agent may then insert that output into the next model request without displaying it to the user.
Every route into the model should therefore pass through the same inspection point:
- the initial user message;
- retrieved documents and search results;
- tool responses;
- stored memory and conversation history;
- generated intermediate context.
This also helps contain indirect prompt injection. Secure implementation guidance for agent-based systems recommends sanitizing remote content, validating tool parameters, limiting permissions, and monitoring interactions rather than trusting retrieved material by default.
Input inspection alone cannot solve prompt injection. It is one layer in a larger design that should also include least-privilege tool access, explicit authorization checks, constrained tool schemas, and human approval for high risk actions.
The important architectural point is that untrusted tool output should never receive an uninspected path back into the model.
Keep policy outside the prompt
Security rules should be visible, versioned, and testable.
A policy might define which secret types always cause a block, which personal-data fields may be redacted, which tools can return sensitive information, and which workflows require additional approval.
These decisions belong in configuration or code controlled through the normal security-review process. Burying them inside a system prompt makes them harder to test and easier to alter accidentally.
External policy also makes the control portable. The same rules can sit in front of different models or agent frameworks without depending on how a particular model interprets a sentence.
Tests should include known secret formats, organization-specific credentials, encoded or spaced variants, false positives, and sensitive values returned by tools. They should verify the outcome of the control, not merely whether a warning appeared.
Measure the boundary directly
A useful control produces evidence.
Teams can track how many model calls were blocked, what categories caused the blocks, how frequently personal information was redacted, and whether particular tools repeatedly returned unsafe content.
The most important measure is the number of confirmed high risk secrets that pass through the guard and reach the model request. The desired result is zero, but that target should be validated through testing and telemetry rather than assumed from the presence of a filter.
Offline scanning can provide a second layer. Stored traces, evaluation records, and debugging data can be checked for residual sensitive values. Findings can then improve the rules at the model boundary.
Prevention and detection serve different purposes. The first stops known unsafe material from leaving the application. The second reveals gaps in the first control and shows where remediation is needed.
The last controllable moment
Teams building AI agents face risks in model behavior, tool permissions, authorization, memory, and external content. A pre-inference guard does not resolve all of them.
It does address a narrower question with unusual clarity: should this exact context be allowed to leave the application?
That decision should not be delegated to the model.
Inspect the assembled prompt, block recognized secrets, redact information the task does not require, and apply the same policy to every tool response that re-enters the agent loop.
The final moment before inference is the last point at which the application has complete control over what the model receives. Treating that moment as a security boundary turns a quiet data leak into a visible, testable decision.
