Use synchronous output inspection when a configured finding must be checked before affected content reaches the client. Use asynchronous inspection only where your policy permits some output to arrive before the verdict. A later stop limits additional output. The earlier bytes can already be in a browser, an event log or an export.
Define what may reach the client
Amazon Bedrock describes synchronous mode as buffering chunks before evaluation and asynchronous mode as sending chunks while evaluation runs. Microsoft Foundry also documents asynchronous filtering that delivers tokens before the corresponding moderation results. The timing, available actions and checked content can differ between providers and endpoints. Check the exact path your client uses. Amazon Bedrock streaming modes and Microsoft Foundry content streaming
Write the decision per finding class and consumer. For example, an organization may allow a short delay to keep a sensitive identifier out of event subscribers, while accepting an asynchronous moderation decision for a lower-risk surface. Include accessibility clients, transcripts and callback sinks in the consumer list. A browser that removes text after a stop can improve the interface. It cannot remove a value that a different consumer has already stored.
Test a value that crosses two chunks
Suppose a test provider emits a synthetic case identifier across two output chunks. A test detector should recognize the complete value and stop the stream. Set up a client that records the exact public events, and add a test subscriber for the same stream. Compare both against the expected result.
| Case | Expected result to verify |
|---|---|
| Synchronous check finds the value | Neither consumer gets the matched value |
| Asynchronous check finds the value | Consumers may have seen it and receive a terminal stop |
| Stream ends with an incomplete match | Check how the final buffered text is released or withheld |
| Client disconnects during buffering | Reconnection follows your app's explicit recovery policy |
Test multiple chunk boundaries and check what was actually delivered. A single chunk containing the whole value can pass while the same detector fails on a split value. Use a unique synthetic marker so a test can search for it across event storage and the rendered view without copying sensitive production content.
Measure the delay and the coverage
Measure time to first visible content and the longest gap between visible chunks by detector and model. A keep-alive can sustain the connection while a detector holds text. Count it as transport activity and start the visible-content timer on the first content event. Compare the resulting delay with your product’s tolerance. A detector timeout and a request timeout can create different results, so test them separately.
Check what happens to retries and downstream consumers after a stop. The application should mark an interrupted answer before it can be reused as complete context. A retry against an alternate endpoint must apply the same required checks. Test a disconnected subscriber too, especially if an event queue permits replay after reconnection.
Keep a regression case for ordinary text that nearly matches the marker. If it takes too long to release valid text, investigate the buffer policy before expanding it to every customer. Keep the refusal test separate from the latency test so each failure has a clear owner.
Check each detector in your gateway
Alectura checks streamed secrets, personal data and configured patterns synchronously. Some other detectors, including moderation, evaluate windows asynchronously. A non-streamed response stopped by a guardrail returns a refusal before the response body reaches the client as model output. Evaluate the mode, action and refusal shape for the detector you configure. Alectura gateway reference
Start with one feature and one high-risk content class. Check request and response paths separately. Then apply the same measurement boundaries to both the latency and release tests so the added delay is attributable to the correct part of your stream. Gateway latency measurement
Source documents (2)
Trace a request through Alectura.
Bring a sample prompt and a customer policy. See what reaches the model and which decisions appear in the request record.
