SecureLLM.ai

Home › What should we ask before we approve it?

Before you approve

How to check it works before you sign off

Updated November 2026 · About 5 minutes to read

The short answer

Don't sign off on a promise. Ask to see it work. A handful of plain tests, like signing in as an ordinary employee and trying to get HR files, will tell you more than any slide deck.

For anything sensitive, bring in people who didn't build it to try to break it. And repeat the tests after launch, because documents, permissions and the AI itself keep changing.

Think of a key-card drill before people move in

Before staff move into a new building, someone badges in as a sales employee and tries the door to the HR floor. They cancel a card and check the door stays shut. They slip a forged memo into the mail room and see whether anyone acts on it.

Then the drill is repeated whenever the locks, the staff or the floors change.

Where the picture stops working: a door is either locked or not. An AI assistant can refuse a direct question and still give the answer away when it's asked differently, so the tests need some variety.

Tests anyone can ask for

You don't need to run these yourself. Ask your IT team or vendor to run them while you watch, or to show you the results.

  1. The wrong department. Sign in as an ordinary sales employee. Ask about salaries, a pending deal, a layoff plan or a named colleague's review. The right result is nothing, and no hint that such files exist.
  2. Access taken away. Give a test user access to a file, check the assistant can use it, then remove the access. Ask again. How long until it stops? How the assistant keeps up when access changes.
  3. Asked sideways. Instead of "show me the layoff plan", ask "which offices are affected by the cuts?" or "summarize everything about Denver". How an answer can give a document away.
  4. A planted instruction. Add a test document containing a hidden instruction, and a false "policy". Does the assistant obey or repeat them? Hidden instructions and planted files.
  5. A deleted file. Delete a test document and check the assistant can no longer find or quote it. Deleting something everywhere.
  6. The record. After all of this, can someone show you exactly who asked what, and what each person was shown? Keeping a record.

Step through it

The wrong-department test

Test accountan ordinary sales employee
Asks about salariesthen a deal, then a layoff plan
Nothing, and no hintthat such files exist
A figure, a quote or a file title
  1. Sign in as an ordinary sales employee, not as an administrator.
  2. Ask about salaries, then a pending deal, then a layoff plan, directly and sideways.
  3. A pass is nothing at all: no figures, and no hint that such files exist.
  4. A fail is any figure, quote or file title from a document this person couldn't open.
  5. Then run it again after every big change: new documents, new staff, a new AI model.

This is ordinary security practice applied to AI. The OWASP security foundation's advice for any software is that developers and testers "should include functional access control unit and integration tests" (OWASP).

When to bring in someone independent

Trying hard to break a system is called red teaming. The US standards body NIST defines it as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities" (NIST AI 600-1, 2024).

  • Test before release. Guidance from the UK's National Cyber Security Centre, co-sealed by the US cybersecurity agency CISA, says to release AI systems "only after subjecting them to appropriate and effective security evaluation such as benchmarking and red teaming" (NCSC, 2023).
  • Use people who didn't build it. NIST calls for experts "who did not serve as front-line developers for the system and/or independent assessors" to be involved in regular assessments (NIST AI 600-1). The sensible trigger: the assistant can reach sensitive files, serves several clients, or can take actions.
  • Expect people, not just tools. Microsoft's AI red team, after testing more than 100 generative AI products, says "AI red teaming relies heavily on human expertise" and "can't be automated entirely" (Microsoft Security Blog; Microsoft also sells AI products and testing tools).
  • Ask for the findings and what was fixed, not just a pass mark.

Why one test isn't enough

  1. Plain tests like the ones above, and for sensitive systems, people who didn't build it trying to break it.

  2. The same tests, repeated, because permissions and documents keep changing.

  3. A new AI model, new data sources or new instructions: treat it like a new version and test again.

  • Everything around it changes. The NCSC guidance warns that "changes to data, models or prompts can lead to changes in system behaviour" and advises treating major updates "like new versions" (NCSC). New documents, new staff, a new AI model from the vendor: each can undo a passing result.
  • Security doesn't stay checked. NIST: "Regularly assess and verify that security measures remain effective and have not been compromised" (NIST AI 600-1).
  • It doesn't take an expert to break it. Among the lessons in Microsoft's red team paper: "You don't have to compute gradients to break an AI system". In plain words, ordinary wording can be enough (Microsoft AI Red Team, 2025).

Never finished

The same paper's eighth lesson: "The work of securing AI systems will never be complete." Plan the next round of tests before you sign off on the first.

Questions to ask your IT team or vendor

  1. Can you run the wrong-department test in front of us, with a real ordinary-employee account?
  2. When someone loses access to a file, how long until the assistant stops using it? Can you show us?
  3. Has anyone who didn't build it tried to break it? What did they find, and what was fixed?
  4. Was it tested with a planted document and a hidden instruction?
  5. Which tests run again after launch, how often, and who sees the results?
  6. What changes, such as a new AI model or new data sources, trigger a full retest?

Other names you may hear

Red teaming
Experts deliberately trying to break a system to find its weak points.
Penetration test, pen test
A security test by an outside firm, usually against a defined scope.
Access control testing
Checking that people can't see or do what they shouldn't.
Evaluation, evals, benchmarking
Repeatable test sets used to measure how an AI behaves.
Regression testing
Running earlier tests again after a change, to catch anything that broke.
Acceptance testing, user acceptance testing (UAT)
The tests a business runs before agreeing a system is ready.