SecureLLM.ai

Home › Could it leak information between departments?

Could it leak?

How an answer can give away a document you weren't allowed to open

Updated November 2026 · About 5 minutes to read

The short answer

Blocking someone from opening a file isn't enough. If the assistant has read that file, what it says can still give the contents away.

The leak can come through a summary, a source link, an answer saved for someone else, a shared chat, or even the coded copies the assistant keeps for searching. Each one needs the same check as the file itself.

Think of a colleague who has read the board papers

They would never hand you the folder. But ask them whether the Denver office is safe this year, and their pause, their hint, or a well-meant "I'd hold off on that lease" tells you what's in it.

An assistant can do the same thing, without the discretion.

Where the picture stops working: a colleague knows what's confidential. An assistant only knows what its software lets it use, so the protection has to be built into the system, not into its judgment.

Five ways an answer can leak

Step through it

One restricted file, five ways out

"Layoff plan, Denver"only HR may open it
The answer
The source list
Saved answers
Chat histories
Coded copies
Someone outside HR
  1. HR keeps a file called "Layoff plan, Denver". Only HR may open it.
  2. Even so, the assistant can leave traces of it in five places that someone outside HR might see.
  3. The answer itself can summarize it, or hint at it.
  4. The source list can show its title, which says a lot on its own.
  5. An answer saved for HR can be reused for someone else's similar question.
  6. A chat history, if it's shared or exposed, can carry what the answer said and where it came from.
  7. And the coded copies of its text can be turned back into words.
  8. A well-built system checks permission at all five, not just at the file.
  • The answer itself. A summary or a hint that draws on a file the person couldn't open. The OWASP security foundation lists sensitive information disclosure as one of the top risks for AI applications (OWASP, LLM02:2025).
  • The source list. Answers often name the documents they came from. A title such as "Layoff plan, Denver" says a lot even if the file won't open.
  • Saved answers. To save time and money, some systems reuse an earlier answer for a similar question. Researchers have shown that this kind of shared store of answers can be used to work out what other users asked (Song et al., 2024).
  • Chat histories. Past conversations are stored, sometimes with the sources they used. Microsoft says Copilot's stored history includes "the user's prompt and Copilot's response, including citations" (Microsoft Learn).
  • The coded copies. To search by meaning, the assistant turns passages into lists of numbers, called embeddings. They look like gibberish, but researchers rebuilt 92% of short passages word for word from them, and recovered full names from clinical notes (Morris et al., 2023).

How a well-built system does it

  • It only reads what the person may see. If restricted text is never read for this person, it can't shape the answer. Why the check must happen before the assistant reads anything.
  • Sources are checked too. A source is only listed, or linked, if the person may open it.
  • Saved answers are kept per person, or not kept. An answer saved for the finance director isn't reused for anyone else.
  • Histories and sharing have rules. Who can see a chat, how long it's kept, and whether it can be shared outside the company are all decided up front.
  • The coded copies are protected like the documents. OWASP recommends "fine-grained access controls and permission-aware vector and embedding stores" (OWASP, LLM08:2025).

It has happened

  1. A ChatGPT bug lets some users "view brief descriptions of other users' conversations" in their history. It may also have exposed payment details of 1.2% of paying subscribers for a few hours (The Hacker News).

  2. OpenAI removes an opt-in setting that let shared ChatGPT conversations be found through search engines. Its security chief said it "introduced too many opportunities for folks to accidentally share things they didn't intend to" (Malwarebytes).

Why it matters

Both were consumer services, not company assistants. But the same parts, histories, sharing and saved answers, exist in the tools you're being asked to approve, and each one is a way out for information.

Questions to ask your IT team or vendor

  1. Does the assistant only read passages this person may see, or does it read more and filter the answer?
  2. Are source titles and links checked against the person's access before they're shown?
  3. Does it reuse saved answers between people? If so, how does it make sure they're allowed to see them?
  4. Who can see a person's chat history, how long is it kept, and can chats be shared outside the company?
  5. Are the coded copies (embeddings) protected with the same access rules as the documents?

Other names you may hear

Sensitive information disclosure, data leakage
Information reaching someone who shouldn't have it.
Citations, grounding
The sources an answer is based on.
Semantic cache
A store of earlier answers reused for similar questions.
Embeddings, vectors
The number codes the assistant uses to search by meaning.
Embedding inversion
Turning those codes back into the original text.
Vector database, vector store
Where the embeddings are kept.