An LLM stream that stops after some text is not one failure class. A gateway can refuse a request before any response starts, a guardrail can stop output after text has reached the client, or the connection can fail while the provider continues or has already completed work. Treating all three as “retry” creates duplicate actions and confusing user interfaces.
Your client should persist a logical request record before opening a stream. The record needs a request identifier, the intended action, the policy version, and a state that can become refused, streaming, complete, stopped, or unknown. This gives a user a truthful status even when an HTTP connection cannot.
Classify the event before you retry
Use the protocol event and whether the application has observed side effects to choose a response. An HTTP refusal before the first event is a new admission decision. A terminal error after partial text is a failed delivery or an intentional stop. A disconnect with no terminal event is unknown until the system checks durable state.
| What the client observed | What it means | Safe default |
|---|---|---|
| HTTP policy refusal before a stream | The request was not admitted under the current policy | Show the reason and offer a permitted path; do not loop retries |
| Terminal guardrail error after text | Output inspection stopped the response | Preserve the partial transcript as stopped; do not present it as complete |
| Connection closes without a terminal event | The request outcome is not proven | Query durable request state before sending another model request |
| Retryable upstream refusal before first byte | A provider call did not start a response | Retry within a deadline and attempt budget |
This classification is deliberately conservative. The OpenAI recovery guidance says that an error event or disconnected stream does not confirm a final outcome; it directs clients to check saved work before resubmitting. That rule is especially important when a model can call tools.
Make an interrupted stream a visible state
Consider a support assistant that streams a draft refund answer and then loses its connection. The browser has 220 words, the application has not yet executed any tool call, and the gateway has no terminal event. Mark the draft interrupted, store the received text, and offer “continue” only after the application checks the logical request record.
The continuation is a new model request, not evidence that the original request failed. For plain text, it can include the saved partial answer and ask for the remaining response. Anthropic documents a capture-and-resume pattern for interrupted streams, while noting that partially received tool-use and thinking blocks cannot be recovered in the same way. See its streaming error recovery guidance.
Do not concatenate a resumed answer with partial JSON or an unfinished tool argument. Keep structured output behind a parser that accepts a result only after its required terminal condition. For a tool call, use a separate durable action identifier and reconcile the tool’s state before asking the model to continue.
Keep policy stops different from transport failures
A policy refusal explains an authorization or safety decision. A transport failure explains that the client has insufficient evidence about delivery. Those messages should remain separate in telemetry and in the user interface.
For example, Alectura can stop a non-streamed response before it is sent, or end a streamed response with the provider format’s terminal error when a guardrail detects prohibited output. Its gateway reference documents these protocol-specific errors. A client should display the request ID and a short action such as “edit the request” or “contact an administrator”; it should not claim that the provider is unavailable.
Conversely, a timeout is not a reason to say a policy blocked the request. Record the provider, model, attempt number, first-byte status and final event type. Avoid recording prompt or answer content unless your retention policy permits it.
Put limits around automatic recovery
Automatic recovery has a cost and can repeat side effects. Set all of these boundaries before production:
- A deadline for the logical user action, rather than an unlimited chain of retries.
- A small attempt budget for failures that occur before the first response byte.
- A stable logical request identifier shared by the original request and each retry.
- A tool-action identifier that prevents a continuation from performing the same mutation twice.
- A clear terminal state when the deadline or attempt budget is reached.
The subtle failure case is a stream that has sent persuasive natural language before a policy stop. Users may act on it even though the response is incomplete. Render a visible stopped state, disable “copy final answer” affordances, and keep the original policy version in the request record for investigation.
Test the client with a refusal before first byte, a guardrail stop after several chunks, an upstream timeout, and a browser reconnect. Assert that each case has one user-visible state, that only retryable pre-byte attempts repeat automatically, and that no test executes a tool twice. You can evaluate the same boundary with the Alectura quickstart before routing production traffic through it.