The OpenAI context window trap that's quietly inflating your bill

Multi-turn conversations look trivial in the OpenAI docs. You append messages to an array, send the whole thing, get a response, append that, repeat. What the docs don't dwell on is that every single turn re-sends every single prior message — and the token meter runs on total tokens in the request, not just the new ones. I didn't fully feel this until I pulled a billing report on a healthcare client's internal triage assistant and found we were spending roughly 4x what a naive estimate would suggest.

What's actually happening under the hood

The Chat Completions API is stateless. OpenAI doesn't remember your previous call. That means if you want a 10-turn conversation, turn 10 has to include all 9 prior turns as context. By turn 20 or 30, you're shipping a small novel on every request.

Here's the compounding math. Say each turn averages 150 tokens of user input and 300 tokens of assistant response — modest numbers. By turn 10, your context payload alone is already around 4,500 tokens before you even add the new message. By turn 20, you're over 9,000. With gpt-4o at roughly $2.50 per million input tokens (as of mid-2025), that's not catastrophic per conversation — but multiply by hundreds of concurrent users and thousands of sessions a day and it adds up fast. More importantly: once you're deep in a long conversation, you start approaching the model's context limit, and response quality degrades because the model has to attend over a huge noisy window.

The two failure modes I see most often:

1. Unbounded growth — every message appended forever, no truncation, bill grows with every turn, and eventually you hit a context limit error at the worst possible moment.
2. Naive truncation — you just slice the oldest messages off the front. Fast to implement, but you lose the system prompt and early context that gives the model its instructions and the user's initial intent.

A working approach: sliding window with system prompt pinning

The pattern I've landed on in production is a sliding window that always pins the system prompt at index 0, optionally pins a summary of early context, and then keeps only the most recent N tokens of actual conversation.

Here's the core of what I ship in Laravel. I inject an OpenAIContextManager into whatever service handles the chat loop.

<?php

namespace App\Services;

use Tiktoken\Tiktoken; // use yethee/tiktoken or similar

class OpenAIContextManager
{
    // Hard ceiling — leave headroom for the response
    private int $maxContextTokens;
    private string $model;

    public function __construct(string $model = 'gpt-4o', int $maxContextTokens = 6000)
    {
        $this->model = $model;
        $this->maxContextTokens = $maxContextTokens;
    }

    /**
     * Given a system prompt, a "summary" message (nullable), and the full
     * conversation history, return a trimmed message array that fits within
     * the token budget.
     *
     * @param  array<int, array{role: string, content: string}>  $history
     * @return array<int, array{role: string, content: string}>
     */
    public function buildMessages(
        string $systemPrompt,
        ?string $contextSummary,
        array $history,
        string $newUserMessage
    ): array {
        $enc = Tiktoken::getEncoding('cl100k_base');

        // Always pin these
        $pinned = [
            ['role' => 'system', 'content' => $systemPrompt],
        ];

        if ($contextSummary !== null) {
            $pinned[] = [
                'role' => 'system',
                'content' => 'Summary of earlier conversation: ' . $contextSummary,
            ];
        }

        $pinnedTokens = $this->countTokens($enc, $pinned);
        $newMessageTokens = count($enc->encode($newUserMessage)) + 4; // +4 for framing

        $budget = $this->maxContextTokens - $pinnedTokens - $newMessageTokens;

        // Walk history newest-first, accumulate until budget exhausted
        $kept = [];
        $used = 0;
        foreach (array_reverse($history) as $message) {
            $t = count($enc->encode($message['content'])) + 4;
            if ($used + $t > $budget) {
                break;
            }
            $kept[] = $message;
            $used += $t;
        }

        $messages = array_merge(
            $pinned,
            array_reverse($kept),
            [['role' => 'user', 'content' => $newUserMessage]]
        );

        return $messages;
    }

    private function countTokens($enc, array $messages): int
    {
        $total = 0;
        foreach ($messages as $m) {
            $total += count($enc->encode($m['content'])) + 4;
        }
        return $total + 2; // reply priming
    }
}

The caller looks like this:

$messages = $this->contextManager->buildMessages(
    systemPrompt: $session->system_prompt,
    contextSummary: $session->context_summary, // nullable string, explained below
    history: $session->message_history,        // full array from DB
    newUserMessage: $request->input('message')
);

$response = OpenAI::chat()->create([
    'model' => 'gpt-4o',
    'messages' => $messages,
    'max_tokens' => 1024,
]);

// Append both sides to the stored history
$session->message_history[] = ['role' => 'user', 'content' => $request->input('message')];
$session->message_history[] = ['role' => 'assistant', 'content' => $response->choices[0]->message->content];
$session->save();

I store the full history in the database (Postgres, jsonb column). The context manager only controls what gets sent to the API — not what I persist. That's a deliberate separation. I want the full record for debugging, auditing (critical in healthcare), and re-summarization.

The summarization trick

Once a conversation goes past, say, 15 turns, I kick off a background job that asks the model to produce a 200-token summary of the conversation so far. That summary gets stored as context_summary on the session and gets pinned as a second system message on every subsequent call. The raw old history still lives in the DB, but I stop sending it.

// Queued job: App\Jobs\SummarizeConversation
public function handle(): void
{
    $summaryMessages = [
        ['role' => 'system', 'content' => 'You are a summarizer. Produce a concise 150-200 token summary of the following conversation, preserving key facts and user intent.'],
        ['role' => 'user', 'content' => $this->formatHistoryAsText($this->session->message_history)],
    ];

    $response = OpenAI::chat()->create([
        'model' => 'gpt-4o-mini', // cheaper for summarization
        'messages' => $summaryMessages,
        'max_tokens' => 256,
    ]);

    $this->session->context_summary = $response->choices[0]->message->content;
    $this->session->summarized_at_turn = count($this->session->message_history) / 2;
    $this->session->save();
}

Note I'm using gpt-4o-mini for the summarization pass. It's a fraction of the cost and perfectly capable of this task. Save the expensive model for actual reasoning.

The gotchas that bit me

Token counting library mismatch. The cl100k_base encoding is correct for gpt-4o and gpt-4-turbo. If you're on an older model or a fine-tune, double-check. I've had off-by-a-few-hundred-tokens bugs from using the wrong encoding that pushed requests right over the limit.

The +4 per message is load-bearing. OpenAI adds framing tokens per message (role label, separators). The official cookbook says 4 tokens per message plus 2 for reply priming. Skip this and your budget math will be slightly off — not catastrophic, but you'll creep over limits on longer messages.

Pinning too much in the system prompt. I had a client where the system prompt ballooned to 1,800 tokens because someone kept appending instructions over time. That's 1,800 tokens of overhead on every single turn. Audit your system prompts. They're not free.

Re-summarizing on every turn. Early version of this code triggered a summarization check synchronously on each request. Added ~800ms of latency. Obvious in hindsight — queue it.

Context summary staleness. If the user's intent shifts significantly mid-conversation, an old summary can actively mislead the model. I now re-summarize every 10 turns rather than once, and I include the most recent 3 turns verbatim alongside the summary so the model always has fresh immediate context.

When I'd reach for this (and when I wouldn't)

Any production application with multi-turn conversations needs some version of this. Customer support bots, internal assistants, intake workflows, research tools — anything where a user might send 10+ messages in a session. If you're building a quick internal tool for 5 people, you might get away with naive appending for a while. But the moment you have real usage, you need a budget strategy.

I wouldn't bother with this complexity for single-turn completions or RAG pipelines where you're constructing a fresh context per query anyway. The sliding window pattern is specifically for conversational continuity.

If you're using the Assistants API with threads, OpenAI handles some of this server-side — but you give up visibility into exactly what's in the context window, and the pricing story there is its own rabbit hole. For the level of control I need on client projects, I prefer managing it myself with Chat Completions.

The bill tells the whole story

After shipping the context manager and summarization job for that healthcare client, input token consumption dropped by about 60% within the first billing cycle. Response quality actually improved slightly because the model wasn't wading through 30 turns of irrelevant early chat. The fix is maybe 150 lines of code. The cost of not doing it compounds every day you're in production.