S3 Event Notifications vs. Polling: What Actually Makes Sense at Small Scale

S3 Event Notifications feel like the obvious answer the moment someone says "we need to process files when they land in S3." Serverless, event-driven, near-real-time — it hits all the right notes. But I've built enough of these pipelines to know the obvious answer isn't always the right one, especially when your volume is modest and your infra budget matters.

Here's where I actually land after running both approaches in production: at small scale, polling is simpler, cheaper to operate, and easier to debug. Event notifications earn their complexity at higher volumes or when latency requirements are genuinely tight. Let me walk through the tradeoffs as I've actually experienced them.

What Problem Are We Solving

The pattern comes up constantly. A client uploads a CSV export from their ERP, a lab instrument drops a results file, an e-commerce partner dumps an inventory feed — someone puts a file in S3 and something needs to happen next. Transform it, validate it, import it into Postgres, kick off a downstream job.

The naive answer is a cron that lists the bucket every N minutes. The "modern" answer is S3 Event Notifications routed to SQS, consumed by Lambda or a worker. Both work. The question is which one makes sense for your actual situation.

The Event Notification Setup

With S3 → SQS → Worker, the flow looks like this: file lands, S3 fires a notification within seconds, it hits an SQS queue, your consumer picks it up. Latency from upload to processing start is typically under 15 seconds in my experience, often under 5.

Here's how I wire this up in a Laravel app. First, the SQS consumer job:

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;

class ProcessS3UploadJob implements ShouldQueue
{
    use InteractsWithQueue, Queueable;

    public int $tries = 3;
    public int $backoff = 60;

    public function __construct(
        public readonly string $bucket,
        public readonly string $key,
    ) {}

    public function handle(): void
    {
        Log::info('Processing S3 file', ['bucket' => $this->bucket, 'key' => $this->key]);

        $contents = Storage::disk('s3')->get($this->key);

        if ($contents === null) {
            Log::warning('S3 object not found, may have been deleted', ['key' => $this->key]);
            return;
        }

        // Your processing logic here
        $this->processFileContents($contents, $this->key);
    }

    private function processFileContents(string $contents, string $key): void
    {
        // Transform, validate, import — whatever your pipeline needs
    }
}

Then a controller or artisan command that receives the raw SQS payload and dispatches the job:

<?php

namespace App\Console\Commands;

use App\Jobs\ProcessS3UploadJob;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Log;

class ConsumeS3EventsCommand extends Command
{
    protected $signature = 'pipeline:consume-s3-events';
    protected $description = 'Pull S3 event notifications from SQS and dispatch processing jobs';

    public function handle(): void
    {
        // Laravel's queue worker handles SQS polling automatically
        // This command is here for documentation clarity —
        // in practice you'd run: php artisan queue:work sqs --queue=s3-events
        $this->info('Use: php artisan queue:work sqs --queue=s3-events');
    }
}

For the SQS message itself, S3 wraps the event in an envelope. When Laravel's SQS driver receives it, you need to unwrap the S3 event records. I usually handle this with a custom SQS job payload transformer, or I route through a thin Lambda that normalizes the payload before pushing to a second queue my Laravel worker actually consumes. Either way, the S3 event record gives you s3.bucket.name and s3.object.key — that's all you need.

The Polling Setup

Polling is almost embarrassingly simple:

<?php

namespace App\Console\Commands;

use App\Jobs\ProcessS3UploadJob;
use Carbon\Carbon;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Storage;

class PollS3BucketCommand extends Command
{
    protected $signature = 'pipeline:poll-s3 {--prefix=incoming/} {--minutes=5}';
    protected $description = 'Poll S3 for new files and dispatch processing jobs';

    public function handle(): void
    {
        $prefix = $this->option('prefix');
        $lookbackMinutes = (int) $this->option('minutes');
        $cutoff = Carbon::now()->subMinutes($lookbackMinutes + 1); // small overlap to avoid gaps

        $disk = Storage::disk('s3');
        $files = $disk->listContents($prefix, recursive: false);

        $dispatched = 0;

        foreach ($files as $file) {
            if ($file['type'] !== 'file') {
                continue;
            }

            $lastModified = Carbon::createFromTimestamp($file['last_modified']);

            if ($lastModified->isAfter($cutoff)) {
                ProcessS3UploadJob::dispatch(
                    config('filesystems.disks.s3.bucket'),
                    $file['path'],
                );
                $dispatched++;
            }
        }

        $this->info("Dispatched {$dispatched} file(s) for processing.");
    }
}

Schedule it in app/Console/Kernel.php:

$schedule->command('pipeline:poll-s3 --prefix=incoming/ --minutes=5')
         ->everyFiveMinutes()
         ->withoutOverlapping();

That's the whole thing. No SQS queue, no Lambda, no IAM roles for notification delivery, no S3 bucket notification configuration to manage. It runs inside your existing Laravel scheduler.

The Gotchas That Will Bite You

With event notifications: The biggest one is duplicate delivery. SQS delivers at-least-once, and S3 will occasionally fire duplicate notifications for the same upload. I've seen this happen on multipart uploads especially. Your processing job needs to be idempotent. I track processed keys in a database table and bail early if I've already handled that object.

Also: S3 Event Notifications don't guarantee ordering. If someone uploads a file and immediately replaces it, you might process the second version first. Unlikely, but it happens.

The IAM setup is annoying. S3 needs permission to publish to your SQS queue (resource policy on the queue), your worker needs permission to read from SQS and from the bucket, and if you use Lambda as a relay you're adding another permission surface. It's not hard but it's tedious, and it's invisible failure territory — if the resource policy is wrong, files just silently don't get processed.

With polling: The main gotcha is the last_modified timestamp on S3 objects. It reflects when the upload completed, but for large multipart uploads that might be significantly after the upload started. For most use cases this is fine. For a pipeline where someone uploads a 2GB file and expects processing to start the instant it lands, polling lag plus multipart upload time stacks up.

The other gotcha: listing a large prefix is not free. ListObjectsV2 costs $0.005 per 1,000 requests. If your bucket has 500,000 files under that prefix and you're paginating through all of them every 5 minutes, you're doing real work and paying for it. The fix is to use a dedicated incoming/ prefix that you move or delete files out of after processing, keeping the list small and cheap.

The Actual Cost Math at Small Scale

Let's say you're processing 500 files a day. Event-driven path: 500 SQS messages, negligible cost. But you're also paying for the SQS queue, the Lambda (if you use one), and the operational overhead of maintaining that infrastructure.

Polling path at 5-minute intervals: 288 ListObjectsV2 calls per day. If your prefix stays under 1,000 objects (which it will if you clean up after processing), that's 288 API calls at $0.005/1,000 — about $0.0014/day. Essentially zero.

At 50,000 files a day, the math shifts. Event notifications scale linearly without you thinking about it. Polling with a large bucket starts requiring more careful prefix management and you're burning more API calls per useful file found.

When I'd Reach for Each

Polling makes sense when:

• Volume is under a few thousand files per day
• Latency tolerance is 5-15 minutes (which is most batch import pipelines)
• Your team already runs a Laravel scheduler and doesn't want to own SQS infrastructure
• You want straightforward debugging — a cron log is easier to reason about than tracing SQS message flow

I used this pattern for a print management client who receives supplier inventory files nightly. A cron every 10 minutes, processing files in the incoming/ prefix, archiving them to processed/ when done. It's been running without incident for two years.

Event notifications make sense when:

• You need sub-minute latency (document processing for users waiting on a result, for example)
• Volume is high enough that listing the bucket gets expensive or slow
• You're already in Lambda/SQS infrastructure and the additional complexity is marginal
• You need fan-out — one upload triggering multiple downstream consumers simultaneously

I built an event-driven pipeline for a biotech client processing instrument output files. The lab staff were waiting at their desks for results to appear in the LIMS. Five-minute polling latency was unacceptable. That's the right use case for event notifications.

The Closing Take

Event-driven architecture gets treated like the obviously mature, obviously correct answer, and polling gets treated like a workaround for people who don't know better. That's backwards thinking. Polling is a completely legitimate, operationally simple approach for the file volumes most small-to-mid businesses actually deal with. Reach for event notifications when you have a specific reason — latency or volume — not just because it sounds more sophisticated.