You can diagnose most AI service failures without copying employee prompts and answers into routine telemetry. Start from the operational question, then record the smallest data set that answers it. Content capture should be an explicit, short-lived exception with an access path and expiry.
The OpenTelemetry GenAI conventions warn that input messages, output messages, system instructions and tool arguments can contain sensitive information. The conventions provide shared names for operation, model, provider and request attributes; they do not require applications to retain message content.
Split operations evidence from content
| Need | Default record | Escalation record |
|---|---|---|
| Reliability | request ID, duration, status, retry count, route | bounded trace for a sampled failing request |
| Cost | model, token counts, price version, tenant or cost centre | never the prompt merely to explain cost |
| Safety | policy version, finding type, action and coverage | controlled evidence approved for investigation |
| Product quality | evaluation ID, score, prompt template version | de-identified test set, not employee traffic |
Use stable identifiers to join these records instead of duplicating content. Hashes can help correlate exact repeats, but a hash of sensitive material is still sensitive operational data. Define who can search it, how long it remains and whether an attacker could use it for guessing.
Set a collection decision for each field
NIST’s Privacy Framework is a voluntary tool for managing privacy risk. Apply its risk mindset to telemetry: name the processing purpose, affected people, recipients, retention period and safeguards before switching on a new attribute.
For example, an internal drafting assistant can emit request_id, application name, approved model, response status, latency, token totals and policy verdict by default. An incident workflow may let a designated team attach a redacted excerpt for a fixed investigation period. The workflow should state who approves the capture, where it is stored and how deletion is verified.
Test for accidental content capture
Run a canary request containing unique harmless markers through every log, trace, metric label, error reporter and support export. Search each destination for the markers. Repeat for tool arguments and streamed output. Do not assume a trace sampler or a provider SDK has the same content defaults as your application.
The common failure is a helpful debug flag that captures complete request bodies during an outage and is never removed. Other failures include high-cardinality prompt labels, copied traces in tickets, and a data-retention job that deletes the primary log but not the analytics export.
Alectura separates content retention from request records: request evidence can retain policy versions, verdicts and counts when a policy retains no content. Its platform documentation describes that boundary. Review technical coverage and test one internal application with a known-safe canary before you rely on the design in production.
Define access and deletion before the incident
Limit investigation records to named roles, a stated case identifier and a short expiry. Record the reviewer, purpose and deletion confirmation when an incident team opens content that is normally unavailable. Test the path with an expired case: the investigator should lose access, and every secondary system should receive the deletion request. This is more reliable than a policy that says “temporary” but has no owner, timer or evidence of removal.
Publish the result as an operational contract. It should name the records that are always collected, the specific event that permits content capture, and the deletion path for every copy. This gives employees and incident responders a shared answer before an urgent debugging request turns into a permanent surveillance mechanism.
Review vendors and downstream processors in the same pass. A trace exporter, session-replay product or hosted error collector can receive attributes that the application team thought were internal only. Confirm their destination, access model and retention separately. Do not infer that an API provider's content setting controls telemetry that your own systems export.
