How to Track Batch Inference Cost per Job

Attribute internal AI batch costs by job and team, including partial results, retries and the cost per accepted record.

Track batch inference cost at the provider job and input-record levels. Assign the job to the team that requested it and keep a stable internal ID for each record. When you retry, use a new attempt ID tied to the original record. Report cost per accepted record after the application validates the result. This catches cases where the batch processed thousands of documents but delivered few results your team can use.

Keep a submission ledger and a result ledger

At submission, record the provider job identifier, account, owner, model, service tier, input count, submission time and version of the input manifest. For shared jobs, also record an owner for each input item. Choose identifiers suitable for provider metadata and keep confidential department or employee details in your own mapping.

For example, OpenAI Batch uses a client-supplied custom ID on each request so you can match results to input records regardless of output order. Its guide describes both output and error artifacts. Amazon Bedrock’s batch results include output rows keyed by record ID and a job manifest with processed, successful and error counts plus input and output token totals. Inspect your chosen endpoint to establish which fields report usage per record and which are job totals. OpenAI Batch guide and Amazon Bedrock batch results

Reconcile attempts before computing cost per record

Use the provider billing export at the dimensions it provides. If usage is available only as a job total for a mixed-team job, choose a documented allocation method or show the job as shared. Team shares based on identical record counts can be misleading if one team submits long documents and another submits short ones. Consider separate jobs if a cost owner needs direct attribution.

In a hypothetical document extraction job, the provider charges $100 for a first attempt that yields 8,000 accepted records out of 10,000 inputs. A targeted retry costs $35 and produces another 1,000 accepted records. It leaves 1,000 inputs unresolved.

Measure Result
Total provider charges for both attempts $135
Distinct accepted records 9,000
Total cost divided by accepted records $0.015
Unresolved input records 1,000

If you divide by submissions, you obtain $0.0135 per original input and understate the cost of usable output. If the retry resubmits already accepted records, count the additional cost and deduplicate the accepted record count.

Handle partially completed jobs

Record an application result for each input ID even if the provider job terminates early. Distinguish missing output, a provider error and a provider success your application rejects. A malformed extraction can consume paid tokens and still fail a schema check. Record that charge in the job cost. Reserve a separate budget for retry attempts and require a disposition for records that fail again.

Compare the billing close window with the last activity time for long-running jobs. One provider job can appear across reports or arrive after the initial usage window. Keep estimated cost and reconciled cost as separate fields so you can revise the accounting view without changing job history. Check that invoice reconciliation includes the correct account and service tier.

Close one job before expanding the design

Run a small batch with two record sizes, a rejected result and a targeted retry. Compare your count of accepted results with the downstream dataset and your estimate with provider reporting. Keep raw outputs in the approved data boundary. Your cost ledger can use identifiers, usage fields and prices without storing document content.

Use Alectura’s cost and limits reference to compare the usage dimensions available for routed request traffic. Track provider-native batch jobs in the job ledger and combine both sources at the department report. Internal AI cost showback

Source documents (2)

Trace a request through Alectura.

Bring a sample prompt and a customer policy. See what reaches the model and which decisions appear in the request record.

Try the quickstart

Keep reading

Build an Approved LLM Model Inventory for Internal AI ↗Enforce Document Access Before RAG Guardrails ↗How to Set Budget Alerts for Internal AI Applications ↗