An LLM timeout does not tell you whether the provider performed work. Retrying with a fresh request record can create two model calls, two usage records and sometimes two tool actions for one customer click. A logical request identifier makes the ambiguity visible and gives billing reconciliation a stable join key.
Give each user action one identifier before calling the model. Reuse it for retries that continue the same intended action. Generate a different identifier when the user deliberately starts another generation, even if the prompt text is identical. Identical inputs do not reliably express identical intent.
Separate the logical request from an attempt
Store one logical request and one or more provider attempts beneath it. The logical request describes what the customer asked for. Each attempt describes a specific network call, provider response or failure.
| Record | Example fields | Why it exists |
|---|---|---|
| Logical request | tenant, user action, request ID, input version, policy version, state | Joins billing and user-visible status |
| Provider attempt | attempt ID, provider request ID, model, started time, first-byte state, error | Explains one transmission and retry decision |
| Usage settlement | logical request ID, estimated and observed usage, status | Prevents a retry from becoming an unlabelled second charge |
| Tool action | action ID, mutation target, result state | Prevents repeated external side effects |
The AWS Builders’ Library guidance recommends a unique caller-provided request identifier rather than inferring duplication from parameters. It explains why two requests that look the same can still mean that a caller wants two outcomes. That distinction applies to “generate again” buttons as much as it does to infrastructure APIs.
Define the idempotency contract
Write down what a repeat with the same identifier returns. A useful contract has three cases:
- If the first attempt completed, return the recorded result or a semantically equivalent result.
- If the first attempt is still running, return an in-progress state and let the caller poll or reconnect.
- If the outcome is unknown, reconcile provider and tool records before allowing another attempt.
The identifier must be scoped to a tenant or authenticated caller. Persist an input fingerprint with it and reject reuse with materially different input. Otherwise a buggy client can attach a later user request to an old result.
This is architectural pseudocode, not an Alectura API:
start(logical_id, tenant, input_fingerprint):
atomically create logical request if absent
reject if existing fingerprint differs
return completed result, in-progress status, or permission to create attempt
settle(logical_id, attempt_id, observed_usage):
atomically record this attempt once
reconcile its usage with the logical request's billing stateThe atomic boundary matters. An application that reads “no record” and inserts later can still launch parallel attempts for the same request ID.
Do not retry an unknown outcome blindly
For a failed HTTP request before the provider receives it, a bounded retry may be suitable. After a streaming connection disconnects or an asynchronous operation fails, inspect the durable state first. OpenAI’s error recovery guidance makes the same point for disconnected streams: an error event does not prove the final state, and clients should check saved work before resubmitting input.
The difficult case is usage without a complete response. Keep the attempt as unknown until reconciliation can establish more. Your billing policy must say whether the customer is charged for observed provider usage, an estimate, or neither. Do not silently treat missing usage as zero if a provider may have completed billable work.
Include policy and model changes in the record
A retry can occur after a tenant loses access to a model or changes region. Decide whether it replays under the original admitted policy or starts a new admission decision. The choice affects fairness, data boundaries and the customer’s expectation.
Alectura request records include the applied policies and their versions, region, connection, findings and verdict. Its platform documentation describes record retention separately from content retention. Use that distinction when you design your own reconciliation: identifiers and state are often necessary for evidence, while prompts and responses may require stricter controls.
Test five paths: response lost after provider completion, duplicate browser submit, retry while the first attempt runs, policy change before retry, and a tool action that succeeds while the model response fails. A request-ID design is complete only when operations can explain each path without guessing which customer action a usage record belongs to.