OpenAI Batch API: When Slower on Purpose Is the Right Call
The Batch API cuts your OpenAI costs in half and I've been sleeping on it. Here's where it actually earns its keep.
The Batch API cuts your OpenAI bill in half, and most people building with GPT-4o have never touched it. I didn't for a long time either, because "up to 24 hours" sounds like a dealbreaker until you actually audit what your jobs need.
Spoiler: a lot of them don't need to be real-time.
What the Batch API Actually Is
OpenAI's real-time completions API is built for interactive latency - you send a request, you get a response in a few seconds, the user sees it. You pay full price for that speed, and you're burning quota against rate limits in real time.
The Batch API is different. You upload a .jsonl file of requests, submit a batch job, and OpenAI processes it asynchronously with a 24-hour SLA. Results come back as another .jsonl file you download when it's done. The tradeoff: 50% off the token price, higher throughput limits, and the whole thing runs outside your normal rate limit pool.
That's not a niche use case. For anything that isn't user-facing and doesn't have a hard latency requirement, it's the right default.
The Jobs That Don't Need to Be Fast
I've been running a background enrichment pipeline for a local e-commerce client - they sell specialty industrial parts, and their product catalog has a few thousand SKUs with inconsistent descriptions scraped from manufacturer feeds. Some are three words. Some are 400-word technical specs in German that someone ran through Google Translate in 2019.
Every night, new SKUs land from supplier EDI feeds. The job needs to:
- Normalize the product description to a consistent format
- Extract key specs (dimensions, material, compatibility) into structured JSON
- Generate a short SEO-friendly summary for the product page
None of that needs to happen in under a second. The product won't go live until a human reviews it anyway. I was running this through the real-time API in a Laravel queue worker, one request per SKU, and paying full price for it. Moving it to the Batch API cut that line item by half overnight.
A Working Implementation in Laravel
Here's roughly how I wired this up. It's three phases: build the batch file, submit it, poll for results and process them.
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Storage;
use App\Models\Product;
class SubmitProductEnrichmentBatch implements ShouldQueue
{
use Dispatchable, Queueable;
public function handle(): void
{
$products = Product::whereNull('enriched_at')
->whereNotNull('raw_description')
->limit(1000)
->get();
if ($products->isEmpty()) {
return;
}
// Build the .jsonl file - one JSON object per line
$lines = $products->map(function (Product $product) {
return json_encode([
'custom_id' => 'product-' . $product->id,
'method' => 'POST',
'url' => '/v1/chat/completions',
'body' => [
'model' => 'gpt-4o-mini',
'messages' => [
[
'role' => 'system',
'content' => 'You are a product data specialist. Return only valid JSON.'
],
[
'role' => 'user',
'content' => $this->buildPrompt($product->raw_description)
]
],
'max_tokens' => 512,
'response_format' => ['type' => 'json_object'],
]
]);
})->implode("\n");
$tmpPath = 'batches/enrichment-' . now()->format('Y-m-d-His') . '.jsonl';
Storage::put($tmpPath, $lines);
// Upload the file to OpenAI
$uploadResponse = Http::withToken(config('services.openai.key'))
->attach('file', Storage::get($tmpPath), basename($tmpPath), ['Content-Type' => 'application/jsonl'])
->post('https://api.openai.com/v1/files', [
'purpose' => 'batch',
]);
$fileId = $uploadResponse->json('id');
// Submit the batch
$batchResponse = Http::withToken(config('services.openai.key'))
->post('https://api.openai.com/v1/batches', [
'input_file_id' => $fileId,
'endpoint' => '/v1/chat/completions',
'completion_window' => '24h',
'metadata' => [
'job' => 'product-enrichment',
'date' => now()->toDateString(),
]
]);
$batchId = $batchResponse->json('id');
// Store the batch ID so we can poll for it later
\App\Models\BatchJob::create([
'batch_id' => $batchId,
'type' => 'product-enrichment',
'status' => 'submitted',
'product_ids' => $products->pluck('id')->toArray(),
]);
}
private function buildPrompt(string $rawDescription): string
{
return <<<PROMPT
Given this raw product description, return a JSON object with these keys:
- "normalized_description": a clean 2-3 sentence description in plain English
- "specs": an object of key technical specs you can extract (dimensions, material, etc.)
- "seo_summary": a 1-sentence SEO-friendly summary under 160 characters
Raw description:
{$rawDescription}
PROMPT;
}
}
Then a separate job that runs every few hours to poll and ingest results:
<?php
namespace App\Jobs;
use Illuminate\Support\Facades\Http;
use App\Models\BatchJob;
use App\Models\Product;
class PollEnrichmentBatches
{
public function handle(): void
{
$pendingBatches = BatchJob::where('status', 'submitted')
->where('type', 'product-enrichment')
->get();
foreach ($pendingBatches as $batchJob) {
$status = Http::withToken(config('services.openai.key'))
->get("https://api.openai.com/v1/batches/{$batchJob->batch_id}")
->json();
if ($status['status'] !== 'completed') {
continue;
}
$outputFileId = $status['output_file_id'];
$resultContent = Http::withToken(config('services.openai.key'))
->get("https://api.openai.com/v1/files/{$outputFileId}/content")
->body();
foreach (explode("\n", trim($resultContent)) as $line) {
if (empty($line)) {
continue;
}
$result = json_decode($line, true);
// custom_id was 'product-{id}'
$productId = (int) str_replace('product-', '', $result['custom_id']);
$content = $result['response']['body']['choices'][0]['message']['content'] ?? null;
if (!$content) {
continue;
}
$data = json_decode($content, true);
if (!$data) {
continue;
}
Product::where('id', $productId)->update([
'normalized_description' => $data['normalized_description'] ?? null,
'specs' => $data['specs'] ?? null,
'seo_summary' => $data['seo_summary'] ?? null,
'enriched_at' => now(),
]);
}
$batchJob->update(['status' => 'completed']);
}
}
}
Schedule both in routes/console.php or your Kernel.php - submit at midnight, poll every two hours.
The Gotchas That Will Bite You
The 24-hour window is a ceiling, not a promise. In practice I've seen batches complete in 15-30 minutes for jobs under a few hundred requests. But you can't depend on it, and OpenAI is explicit that they can take the full 24 hours during high load. Don't use this for anything that feeds a user-facing step that needs to happen today.
Partial failures are silent by default. If 50 of your 1000 requests fail, the batch still completes. You get an error_file_id alongside output_file_id. I missed this the first time, skipped checking the error file, and wondered why some products never got enriched. Always fetch and log the error file.
Files don't clean themselves up. Both your input file and output file sit in your OpenAI file storage and count against the 100 GB limit. Add a cleanup step that calls DELETE /v1/files/{file_id} after you've ingested the results.
The custom_id is your only join key. There's no way to pass arbitrary metadata through to the response - just that one string. Keep it meaningful and parseable. I use {type}-{model_id} conventions so the ingestion job doesn't need to look anything up.
gpt-4o-mini is almost always enough here. You're not doing reasoning tasks. You're doing structured extraction on known data. The price gap between mini and full 4o is big, and for batch work at 50% off on top of that, mini becomes genuinely cheap.
When I'd Reach for This (and When I Wouldn't)
Good fit:
- Nightly or weekly enrichment runs (descriptions, tagging, classification)
- Generating embeddings or summaries for a knowledge base rebuild
- Scoring or categorizing form submissions, support tickets, or CRM notes
- Pre-generating AI content for a CMS that an editor will review before publish
- Any job where "done by morning" is good enough
Wrong tool:
- Anything triggered by user interaction and displayed in the same session
- Webhook handlers where you need to respond or update state within seconds
- Anything with a hard SLA shorter than a few hours
- Real-time document review in a clinical or legal workflow - latency matters there
I work with a healthcare client where a clinician submits a document and waits for an AI-assisted summary before they proceed. That's a real-time call, full stop. Same codebase, different job type. The batch API lives in the background worker tier; it never touches the synchronous request path.
The Bottom Line
The instinct to reach for the real-time API for every LLM call makes sense at first - it's the simpler mental model. But when you've got a backlog of enrichment work that runs on a schedule anyway, paying double for millisecond latency you don't need is just waste. If you're running background AI jobs in Laravel and haven't looked at the Batch API yet, audit one pipeline. You'll probably find the latency tradeoff is a non-issue, and your OpenAI invoice will get smaller.
Need help shipping something like this? Get in touch.