Home › Can we prove who saw what?
Can we prove who saw what?
Keeping a record: who asked what, and what they were shown
The short answer
If something goes wrong, you'll need to say exactly who asked what, when, and what they were shown. That's only possible if the assistant keeps a record from day one.
A good record notes who, when, where and which documents were used for each answer. It's kept long enough to be useful, only a few people can read it, and it doesn't become a second copy of everything sensitive.
Think of a hotel's key-card system
The hotel knows which card opened which door, and when. It doesn't film what happened inside the room.
If a guest complains that something went missing, the manager can check who entered. And the door log itself is kept locked away, not pinned up in the lobby.
What a useful record contains
The OWASP security foundation's rule for any software: logs "must record "when, where, who and what" for each event" (OWASP Logging Cheat Sheet). For an AI assistant, that means:
- Who asked, and when. The signed-in person, not just "the assistant".
- Which documents the answer used. Microsoft's records for Copilot, for example, include "references to files, sites, or other resources" it used to answer (Microsoft Learn; Microsoft sells this auditing tool).
- How sensitive those documents were. Microsoft's record notes each file's sensitivity label, which it says helps in "identifying whether Copilot accessed any sensitive information".
- What was blocked. The same records note when access to a file "was blocked or restricted based on some policy". Refusals matter: repeated attempts to reach HR files are a warning sign.
- Any action it took. If the assistant can send or change things, what it did and for whom. When the assistant can do things, not just answer.
An example record
Made up for illustration. Real records vary by product.
- Who
- A sales manager, signed in
- When
- Tuesday, 9:42 a.m.
- Where
- The assistant in the company's chat app
- Question
- Kept with the conversation, not copied here
- Documents used
- Price list 2026 (everyone)
Sales playbook (Sales) - Blocked
- Salary bands 2026 (HR only): not available to this person
- Actions taken
- None
With that, "did anyone outside HR see the layoff plan?" becomes a search, not a guess.
How a well-run record is kept
- It's switched on before launch. A record can't be created after the fact.
- It's kept long enough. Problems are often found months later. In Microsoft 365, audit records are kept "for 180 days by default", and keeping them longer, up to 10 years, depends on the license (Microsoft Learn). Check what your plan includes.
- Few people can read it, and reading it is recorded too. OWASP: "All access to the logs must be recorded and monitored", and the right to read them "should be restricted and reviewed periodically" (OWASP).
- It points to the conversation rather than copying it. A record that lists the question, the person and the documents used can answer most questions without storing every word of every answer in yet another place.
- It leaves out what it shouldn't hold. OWASP lists things that usually shouldn't be written into logs directly, including "Authentication passwords", "Access tokens", "Sensitive personal data" and "Commercially-sensitive information".
How long you must keep records, and what privacy law allows, are legal questions covered on our sister site EnterpriseLLM.ai.
When the record becomes the leak
DeepSeek, January 2025
Security researchers found databases belonging to the Chinese AI company DeepSeek open on the internet. They "held over a million log entries containing user chat history in plaintext form", and anyone could search them without a password. DeepSeek closed the exposure quickly after being told (BleepingComputer; found by Wiz, which sells cloud security tools).
- Logs are a target. OWASP warns that "logs may be a target of attacks" and "may contain personal and other sensitive information" (OWASP).
- No record, no answer. Without a record, after a suspected leak you can't say who saw what, so you may have to assume the worst.
- Records nobody looks at. A record helps after the fact. Reviewing blocked attempts and unusual searches helps you catch problems sooner.
- Records are another copy. When something is deleted, its traces in the record may need clearing too. Deleting something everywhere.
Questions to ask your IT team or vendor
- For each answer, does the record show who asked, when, and which documents were used?
- Does it record blocked attempts, and does anyone review them?
- How long are records kept, and does that depend on our license?
- Who can read the records, and is their reading recorded too?
- Do the records copy full questions and answers, or point to them? Where are those kept, and how are they protected?
- If we suspected a leak tomorrow, how quickly could you tell us who saw a given document?
Other names you may hear
- Audit log, audit trail
- The record of who did what, and when.
- Prompt and response logging
- Keeping the questions asked and the answers given.
- Citations, sources, grounding data
- The documents an answer drew on.
- Retention period
- How long records are kept before they're deleted.
- SIEM
- Software that gathers logs from many systems so security staff can search and monitor them.
- eDiscovery
- Searching company records for a legal case or investigation.
- Traceability
- Being able to trace an answer back to who asked and what it used.