A sensitive-data control is not adequately tested by asking an assistant to repeat a single fake credit-card number. Internal AI applications move data through prompts, retrieval results, tool inputs, logs and streamed output. An evaluation set should represent those paths and make each expected decision testable.
Build the set from synthetic or properly authorized records. Do not copy employee or customer data into a test harness merely because it is convenient. Record the source class, owner, expected handling and approval for each case. A test that creates an unauthorized data store defeats its own purpose.
Define the decision each case must test
Give every example one primary expected result: allow, transform, block, warn or require a human review. A case can have secondary observations, but its pass condition must be clear enough for a reviewer to reproduce.
| Path | Hypothetical test case | Expected result | Evidence to keep |
|---|---|---|---|
| Direct prompt | Synthetic employee ID in a summarization request | Pseudonymize or block under the selected policy | Policy version and verdict |
| Retrieved document | Authorized test document contains an injected secret-like value | Detect or block according to the rule | Retrieval source class and finding |
| Tool input | Model proposes a customer lookup with another department’s ID | Refuse or constrain the action | Tool authorization decision |
| Streamed output | Sensitive value is split across chunks | Detect before the client treats it as complete | Stream terminal state and finding |
| Attachment | File type cannot be inspected by the selected control | Follow the stated fail-open or fail-closed rule | Inspection capability and verdict |
These are examples, not a claim that every control can inspect every modality. The missing-inspection case is essential because it exposes a limit that a green dashboard can hide.
Separate coverage from detection quality
Coverage asks whether the set reaches the systems and formats you operate. Detection quality asks how often a rule makes the intended decision on those cases. Do not combine them into a single percentage without showing the denominator and the error type.
Use labelled cases that include both sensitive and safe text. Track true positives, false positives, false negatives and expected-but-uninspectable cases. Review false positives with the application owner because blocking a legitimate employee workflow may be as operationally important as missing a pattern.
The NIST AI Risk Management Framework provides a risk-management framework rather than a product certification. Use it to organize ownership and review, not to claim that a particular prompt set makes an application compliant. NIST’s AI Technology Evaluation program also emphasizes common data, metrics and scoring, and keeps evaluation data sequestered to reduce test contamination. The same principle helps internal teams: protect a holdout set from casual prompt tuning.
Build a small set that can fail usefully
Start with 20 to 40 cases that cover your highest-risk data paths. For each one, retain:
- A stable case ID and synthetic-data generation method.
- The application route, model configuration and policy version.
- The expected decision and a short reason.
- The observed decision, request identifier and reviewer result.
- Whether the case is in the development set or the protected holdout set.
Use variants sparingly and name what changes. For example, one case may place a synthetic employee number in a direct prompt, another in a retrieved document, and a third splits it across stream chunks. These are different delivery paths, not three copies of the same test.
Avoid using production prompts as an unlabeled benchmark. Their content may be sensitive, and their expected result often depends on context that the evaluator cannot see. If representative production patterns are needed, create approved abstract templates and inject synthetic values.
Test the failure modes around the detector
A detector can miss formats it does not parse, values split over a boundary, encrypted attachments, unexpected encodings and context that appears only in a connected tool. The evaluation should show the selected behavior for each: block, warn, retain for review, or permit with a documented limitation.
Alectura documents its inspection limits and stream behavior in the guardrails reference. For example, a policy can block content that it cannot inspect, and streamed output has distinct handling. Treat that as a candidate control to test with your own routes, files and policy choices; it is not proof that a detector catches all sensitive data.
Run the evaluation before a model, retrieval pipeline, policy or provider change. Compare the result by case ID and investigate every changed verdict. Use a non-production application and the Alectura quickstart to test the request boundary without adding real employee data. A useful evaluation set keeps failures visible long enough for an owner to decide whether to change the policy, the application path or the test itself.