Track batch inference cost at the provider job and input-record levels. Assign the job to the team that requested it and keep a stable internal ID for each record. When you retry, use a new attempt ID tied to the original record. Report cost per accepted record after the application validates the result. This catches cases where the batch processed thousands of documents but delivered few results your team can use.
Keep a submission ledger and a result ledger
At submission, record the provider job identifier, account, owner, model, service tier, input count, submission time and version of the input manifest. For shared jobs, also record an owner for each input item. Choose identifiers suitable for provider metadata and keep confidential department or employee details in your own mapping.
For example, OpenAI Batch uses a client-supplied custom ID on each request so you can match results to input records regardless of output order. Its guide describes both output and error artifacts. Amazon Bedrock’s batch results include output rows keyed by record ID and a job manifest with processed, successful and error counts plus input and output token totals. Inspect your chosen endpoint to establish which fields report usage per record and which are job totals. OpenAI Batch guide and Amazon Bedrock batch results
Reconcile attempts before computing cost per record
Use the provider billing export at the dimensions it provides. If usage is available only as a job total for a mixed-team job, choose a documented allocation method or show the job as shared. Team shares based on identical record counts can be misleading if one team submits long documents and another submits short ones. Consider separate jobs if a cost owner needs direct attribution.
In a hypothetical document extraction job, the provider charges $100 for a first attempt that yields 8,000 accepted records out of 10,000 inputs. A targeted retry costs $35 and produces another 1,000 accepted records. It leaves 1,000 inputs unresolved.
| Measure | Result |
|---|---|
| Total provider charges for both attempts | $135 |
| Distinct accepted records | 9,000 |
| Total cost divided by accepted records | $0.015 |
| Unresolved input records | 1,000 |
If you divide by submissions, you obtain $0.0135 per original input and understate the cost of usable output. If the retry resubmits already accepted records, count the additional cost and deduplicate the accepted record count.
Handle partially completed jobs
Record an application result for each input ID even if the provider job terminates early. Distinguish missing output, a provider error and a provider success your application rejects. A malformed extraction can consume paid tokens and still fail a schema check. Record that charge in the job cost. Reserve a separate budget for retry attempts and require a disposition for records that fail again.
Compare the billing close window with the last activity time for long-running jobs. One provider job can appear across reports or arrive after the initial usage window. Keep estimated cost and reconciled cost as separate fields so you can revise the accounting view without changing job history. Check that invoice reconciliation includes the correct account and service tier.
Close one job before expanding the design
Run a small batch with two record sizes, a rejected result and a targeted retry. Compare your count of accepted results with the downstream dataset and your estimate with provider reporting. Keep raw outputs in the approved data boundary. Your cost ledger can use identifiers, usage fields and prices without storing document content.
Use Alectura’s cost and limits reference to compare the usage dimensions available for routed request traffic. Track provider-native batch jobs in the job ledger and combine both sources at the department report. Internal AI cost showback
Source documents (2)
- OpenAI Batch API guideChecked
- Amazon Bedrock batch resultsChecked
Trace a request through Alectura.
Bring a sample prompt and a customer policy. See what reaches the model and which decisions appear in the request record.
