Why a system prompt is not a security boundary

By Yuvi, Founder & Technical Lead

Days before deployment, sandbox testing caught a permissions bug in a chatbot I was building. I had copied an internal service to create a public-facing one and carried over access the new service should never have had. I caught it before launch, while testing how the bot behaved under adversarial prompts.

Two audiences, different permissions #Link to Two audiences, different permissions

The project needed two chatbots. One would answer general questions on a public website. The other would help employees query internal documents during training. They shared a response style, but their access to information needed to be different.

I built the internal bot first. It retrieved documents from a vector store so the source material could be updated without retraining the model. I used fine-tuning for response style. Retrieval and tone were separate parts of the design.

For the public-facing bot, I duplicated the service and changed its retrieval configuration. The copied configuration still included a service account with read access to the private document store. Changing which data a service normally requests does not change what its credentials permit it to read.

What sandbox testing exposed #Link to What sandbox testing exposed

During testing, I could push the public-facing bot out of its intended role. It began answering as a general assistant. That behavior led me to examine what the service could still access when the conversation went off course.

The inherited credentials were the more serious problem. The bot's instructions described a narrow job, while the service running it had broader permissions. That mismatch existed regardless of how well the bot followed its prompt on ordinary questions.

Why I changed the credentials #Link to Why I changed the credentials

My first instinct was to strengthen the system prompt and add more refusal examples. Those changes might improve how the bot responds, but they would leave the service account's permissions untouched. A refusal in a conversation does not establish that the underlying read would be denied.

System prompts can guide behavior. They cannot replace authorization enforced by the application and the data store. That is also the distinction behind OWASP's least-privilege guidance for LLM applications: limit the access available to the application, including when the model behaves unexpectedly.

I removed the inherited credentials. The public-facing bot's service account has no read permission on the private collection. Access to that collection is decided outside the model, so a change in its conversational behavior does not grant that permission.

Checking the boundary #Link to Checking the boundary

For this kind of separation, the permission check needs to stand on its own. Using the public service's credentials, a request to the private collection should be denied even without a model in the loop. The internal service should retain the access it needs. Prompt tests then exercise a different question: how the bot behaves within the permissions it has.

These are separate checks. A bot refusing to answer a question is not evidence that its credentials cannot retrieve the underlying document.

What I changed in my workflow #Link to What I changed in my workflow

I now provision credentials for each new service instead of inheriting the permissions of the service I copied. Reusing application code can be useful; the new service still needs its own decision about access.

The code #Link to The code

Client work stays confidential, so I rebuilt this separation as a standalone project. The replication keeps the two service accounts and the denied read, simplified and carrying nothing that identifies a client.

To run the reconstruction, you need Python 3.12 or newer:

git clone https://github.com/Zetesis-dev/rag-credential-boundary
cd rag-credential-boundary
python run.py

On macOS and Linux, use python3 for the last command. The script creates a local environment, installs the dependencies, and opens the demo in your browser. It uses synthetic documents and a scripted agent, with no API keys or external model calls. The repository's setup instructions also explain how to run the tests.