Algolia's index sync problem when Laravel events fail mid-write

Algolia's Scout integration looks like magic until the day you realize your search results are returning records that were deleted six weeks ago, or missing products that have been live for a month. The integration is genuinely good — but it has a consistency problem that Scout's documentation barely acknowledges and that will absolutely bite you on any high-write production system.

What Algolia Actually Solves

If you've ever tried to build decent full-text search on top of Postgres, you know the pain. ILIKE queries with leading wildcards kill indexes, tsvector gets you partway there but typo tolerance is basically nonexistent, and relevance tuning becomes a science project. Algolia sidesteps all of that. You push records to their hosted index, and you get typo-tolerant, faceted, relevance-ranked search in milliseconds. For product catalogs, provider directories, document libraries — it genuinely earns its price tag.

Laravel Scout wraps Algolia's PHP client and makes the basic case trivial. Add the Searchable trait, configure your toSearchableArray() method, and Scout automatically syncs records when Eloquent models are saved or deleted. It even queues those sync operations so they don't block your web requests. Looks clean. Feels great. Ships fast.

Then your queue worker crashes mid-flight, or Algolia returns a 503, or a deployment restarts workers with jobs in-flight, and you discover that Scout has no retry-with-reconciliation strategy. It just... doesn't sync that record. No alert. No dead-letter queue handling by default. The database and the index drift apart and nothing tells you.

The Failure Mode in Detail

Here's what Scout does under the hood when you save a model. It dispatches a MakeSearchable job (or RemoveFromSearch on delete). That job calls $model->searchable(), which calls $engine->update($models), which calls Algolia's saveObjects. If that job fails and exhausts its retries, Laravel moves it to the failed jobs table and moves on. Your model is in the database. It is not in Algolia.

The reverse is worse: a delete event fails, the record stays in Algolia, and now users are clicking search results that 404 because the underlying record is gone.

I ran into this hard on an e-commerce project a couple of years back — a mid-sized retailer with a catalog of around 40,000 SKUs. They were doing bulk imports via a CSV pipeline that fired thousands of model save events in sequence. During a particularly heavy import, Algolia started rate-limiting responses (their free and growth tiers have indexing rate limits that are easy to hit in bulk). Jobs started failing. The import pipeline had no idea. Two days later the client noticed search returning stale prices. We had about 800 records out of sync. Not catastrophic, but not acceptable for a storefront.

A Sync Strategy That Actually Holds

My current approach has three parts: a reliable queue setup for Scout, a reconciliation command I can run on a schedule or on demand, and a dead-letter handler that pages me instead of silently failing.

Part 1: Configure Scout Jobs for Real Retry Behavior

First, Scout's queued jobs don't set a $tries or backoff by default in older versions. Extend them.

<?php

namespace App\Jobs;

use Laravel\Scout\Jobs\MakeSearchable as BaseMakeSearchable;

class MakeSearchable extends BaseMakeSearchable
{
    public int $tries = 5;
    public int $maxExceptions = 3;

    public function backoff(): array
    {
        return [10, 30, 60, 120, 300];
    }

    public function failed(\Throwable $exception): void
    {
        // Page me. Don't let this die silently.
        \Log::critical('MakeSearchable job exhausted retries', [
            'model' => $this->models->first()?->getMorphClass(),
            'ids'   => $this->models->pluck('id'),
            'error' => $exception->getMessage(),
        ]);

        // Fire to your alerting system here — PagerDuty, Slack, whatever
    }
}

Then tell Scout to use your job instead of its own. In AppServiceProvider::boot():

use Laravel\Scout\EngineManager;
use Algolia\AlgoliaSearch\SearchClient;

// Override the job class Scout dispatches
\Scout\Scout::makeSearchableUsing(\App\Jobs\MakeSearchable::class);

Also worth putting Scout jobs on a dedicated queue so a backed-up email queue doesn't delay index updates:

// config/scout.php
'queue' => [
    'connection' => 'redis',
    'queue'      => 'search',
],

Part 2: A Reconciliation Command

Retries help with transient failures, but they don't help if Algolia was down for 10 minutes and all jobs exhausted before it came back, or if a deployment killed workers mid-job. For that I write a reconciliation command that I run nightly (or after any bulk operation).

The idea is simple: pull the current Algolia index state, diff it against the database, and fix the delta.

<?php

namespace App\Console\Commands;

use App\Models\Product;
use Illuminate\Console\Command;
use Algolia\AlgoliaSearch\SearchClient;

class ReconcileAlgoliaIndex extends Command
{
    protected $signature   = 'algolia:reconcile {model=Product} {--dry-run}';
    protected $description = 'Diff the DB against the Algolia index and fix gaps';

    public function handle(SearchClient $algolia): int
    {
        $modelClass = 'App\\Models\\' . $this->argument('model');
        $instance   = new $modelClass;
        $indexName  = $instance->searchableAs();
        $index      = $algolia->initIndex($indexName);

        $this->info("Reconciling {$modelClass} against index {$indexName}");

        // Pull all objectIDs from Algolia via browse (not search — no 1000 hit limit)
        $algoliaIds = [];
        foreach ($index->browseObjects(['attributesToRetrieve' => ['objectID']]) as $hit) {
            $algoliaIds[] = $hit['objectID'];
        }

        $this->info('Algolia record count: ' . count($algoliaIds));

        // Pull all DB primary keys
        $dbIds = $modelClass::pluck('id')->map(fn ($id) => (string) $id)->toArray();

        $this->info('DB record count: ' . count($dbIds));

        // Records in DB but missing from Algolia — need to be pushed
        $missingFromAlgolia = array_diff($dbIds, $algoliaIds);
        // Records in Algolia but deleted from DB — need to be removed
        $staleInAlgolia = array_diff($algoliaIds, $dbIds);

        $this->warn('Missing from Algolia: ' . count($missingFromAlgolia));
        $this->warn('Stale in Algolia: '    . count($staleInAlgolia));

        if ($this->option('dry-run')) {
            return self::SUCCESS;
        }

        if (count($missingFromAlgolia) > 0) {
            $modelClass::whereIn('id', $missingFromAlgolia)
                ->searchable(); // Scout chunk-pushes in batches of 500
        }

        if (count($staleInAlgolia) > 0) {
            $index->deleteObjects($staleInAlgolia);
        }

        $this->info('Reconciliation complete.');
        return self::SUCCESS;
    }
}

Key thing here: use browseObjects, not search. Algolia's search API caps results at 1000 hits. browse uses a cursor and returns the full index. If you use search for reconciliation you will have a very bad time on any real catalog.

I schedule this in app/Console/Kernel.php:

$schedule->command('algolia:reconcile Product')->dailyAt('03:00');

After any bulk import I also call it directly:

php artisan algolia:reconcile Product --dry-run
php artisan algolia:reconcile Product

The Gotchas That Actually Burned Me

The browseObjects rate. Algolia's browse endpoint is not throttled the same as search, but it's not instant on large indexes either. On a 200k-record index, a full browse takes a few minutes. Plan for that in your reconciliation window.

Soft deletes. If your model uses SoftDeletes, Scout's default behavior is to keep the record in Algolia when you soft-delete it. You need to override shouldBeSearchable() to return false for soft-deleted records, or you'll have ghost results forever.

public function shouldBeSearchable(): bool
{
    return $this->isPublished() && !$this->trashed();
}

Bulk imports bypass events. If you're using Model::insert() or bulk upserts via something like upsert(), Eloquent model events don't fire, which means Scout never hears about it. I've seen developers do a big import and wonder why search is empty. You have to call $modelClass::whereIn('id', $newIds)->searchable() explicitly after any bulk write.

Index settings drift. Scout will push records but it won't manage your index configuration — facet attributes, ranking, synonyms. I keep index settings in a separate artisan command that pushes config from a config/algolia_indexes.php file and run it in my deploy pipeline. Don't configure indexes by hand in the Algolia dashboard and then wonder why staging and production behave differently.

The 10MB payload limit. Algolia rejects batches where any single record exceeds 10KB or the batch exceeds 10MB. If your toSearchableArray() is pulling in eager-loaded relationships and returning large text blobs, you'll hit this in production but not in dev. Keep your searchable arrays lean — IDs, names, short descriptions, facet values. Store the full content elsewhere.

When I'd Reach for Algolia

I use Algolia when the client needs user-facing search that has to be fast, typo-tolerant, and faceted — product catalogs, provider/specialist directories, document libraries, real estate listings. The developer experience is genuinely good and the results quality is high. For a healthcare client's provider directory I had decent search running in an afternoon. That matters when you're billing a real business.

I would not reach for Algolia for internal admin search where Postgres tsvector or even a simple LIKE query is fine. The cost scales with record count and operations, and it adds an external dependency that can drift. If search doesn't need to be consumer-grade, don't pay for consumer-grade search infrastructure.

I also wouldn't reach for it on anything where data consistency is so critical that even a few hours of drift is unacceptable — financial records, medical data with search-driven workflows. In those cases I'd look at keeping search in the primary database or building a more robust event-sourced sync with explicit idempotency guarantees.

Bottom Line

Scout makes Algolia feel effortless, and that's exactly the problem — it hides the fact that you now have two sources of truth that can diverge. Add real retry logic, build a reconciliation job, and run it on a schedule before your client calls you about ghost results. The integration is worth using; it just isn't worth trusting blindly.