S3 Multipart Upload: The Hidden Cost Nobody Checks

S3 multipart upload is one of those features that feels optional until it isn't, and then when things go sideways you discover a billing surprise that's been quietly compounding for months. I've hit this twice with real clients - once with a biotech lab uploading genomic datasets, once with a print management shop pushing large press-ready PDFs - and both times the cleanup conversation was awkward.

Let me tell you exactly what to watch for.

What Multipart Upload Actually Does

The short version: instead of sending a file in one HTTP PUT, you split it into chunks (5 MB minimum per part, except the last), upload them in parallel or sequentially, then tell S3 to assemble the final object. AWS handles the stitching server-side.

The practical reason to care: S3 will flat-out reject a single PUT larger than 5 GB. So above that threshold, multipart is not optional. Below 5 GB it's still often the right call - anything over 100 MB benefits from parallel chunking and retry resilience, especially on flaky connections or from client browsers.

For the genomic work I mentioned, we were routinely pushing 2-8 GB FASTQ files from sequencing instruments. Single PUT was never going to work. For the print shop, 400-600 MB PDF files were borderline - we could have done single PUT, but multipart gave us per-chunk retry without restarting the whole upload, which mattered on their studio WiFi.

When It's Effectively Mandatory

• Object > 5 GB: required, full stop.
• Anything you're uploading from a browser via presigned URLs at meaningful file sizes. You want chunk-level retry, not "start over."
• Long-running uploads where network interruption is likely.
• When you want parallelism. Five 20 MB parts uploading simultaneously beats one 100 MB sequential PUT on most connections.

When It Backfires

Here's the thing nobody puts in the quickstart guide: incomplete multipart uploads are billed as stored data, indefinitely, until you explicitly clean them up.

If your code initiates a multipart upload, uploads three of ten parts, then crashes or times out or the user closes the tab - those three parts sit in S3. They are not visible as objects in your bucket. They don't show up in a normal ListObjects call. But you are paying for them. Every gigabyte, every month, until something removes them.

I inherited a codebase for a client that had been running for about 18 months. We were auditing their AWS bill and noticed their S3 storage costs were about 40% higher than what their actual object inventory could explain. Ran an aws s3api list-multipart-uploads against their primary bucket and found over 800 incomplete uploads totaling roughly 90 GB of orphaned parts. At us-west-2 standard pricing that's around $2 a month per terabyte - not catastrophic at that scale, but it had been there for a year and a half, and it was going to keep growing.

The root cause was a background job that initiated uploads, occasionally failed mid-stream, and had no cleanup path.

A Working Example in PHP/Laravel

Here's how I handle this now. I use the AWS SDK directly rather than Laravel's Flysystem abstraction, because Flysystem's S3 driver does handle multipart under the hood for large files, but it doesn't expose the lifecycle controls I want.

use Aws\S3\S3Client;
use Aws\Exception\MultipartUploadException;
use Aws\S3\MultipartUploader;

class S3MultipartService
{
    private S3Client $client;
    private string $bucket;

    public function __construct()
    {
        $this->client = new S3Client([
            'version' => 'latest',
            'region'  => config('filesystems.disks.s3.region'),
        ]);
        $this->bucket = config('filesystems.disks.s3.bucket');
    }

    public function upload(string $localPath, string $s3Key): string
    {
        $uploader = new MultipartUploader($this->client, $localPath, [
            'bucket'          => $this->bucket,
            'key'             => $s3Key,
            'part_size'       => 10 * 1024 * 1024, // 10 MB parts
            'concurrency'     => 3,                 // 3 parallel part uploads
            'before_initiate' => function (\Aws\Command $cmd) {
                // tag the upload so lifecycle rules can target it
                $cmd['Tagging'] = 'source=app&type=multipart';
            },
        ]);

        try {
            $result = $uploader->upload();
            return $result['ObjectURL'];
        } catch (MultipartUploadException $e) {
            // Log the upload ID so we can abort it explicitly
            $uploadId = $e->getState()->getUploadId();
            Log::error('Multipart upload failed', [
                'key'       => $s3Key,
                'upload_id' => $uploadId,
                'message'   => $e->getMessage(),
            ]);

            // Abort to clean up parts immediately rather than waiting on lifecycle
            $this->abort($s3Key, $uploadId);

            throw $e;
        }
    }

    public function abort(string $s3Key, string $uploadId): void
    {
        $this->client->abortMultipartUpload([
            'Bucket'   => $this->bucket,
            'Key'      => $s3Key,
            'UploadId' => $uploadId,
        ]);
    }
}

That abortMultipartUpload call in the catch block is what most example code omits. Without it you're leaking parts on every failure.

The Lifecycle Rule You Should Add to Every Bucket

Explicit abort-on-failure is good. A lifecycle rule as a safety net is better. This catches anything your application-level cleanup misses - crashes before the catch block, deploys that interrupt jobs, external clients you don't control.

In the AWS console: S3 bucket - Management tab - Create lifecycle rule. Or via Terraform:

resource "aws_s3_bucket_lifecycle_configuration" "main" {
  bucket = aws_s3_bucket.main.id

  rule {
    id     = "abort-incomplete-multipart"
    status = "Enabled"

    abort_incomplete_multipart_upload {
      days_after_initiation = 3
    }
  }
}

Three days is my default. It's long enough that a legitimately slow upload from a client machine won't get killed mid-stream, and short enough that orphaned parts don't accumulate for long. If you're dealing with uploads that could legitimately take more than a day, bump it to 7. But pick a number and set it. The default - no rule - means parts live forever.

You can also do this via the AWS CLI if you want to fix an existing bucket right now:

aws s3api put-bucket-lifecycle-configuration \
  --bucket your-bucket-name \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "abort-incomplete-multipart",
      "Status": "Enabled",
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 3
      }
    }]
  }'

Checking What's Already Orphaned

If you've never done this, run this now against any bucket you care about:

aws s3api list-multipart-uploads --bucket your-bucket-name

If the output is empty, great. If it's not, you'll see a list of in-progress upload IDs with their keys and initiation timestamps. Anything initiated more than a few days ago that hasn't completed is almost certainly orphaned.

You can abort them in bulk:

aws s3api list-multipart-uploads --bucket your-bucket-name \
  --query 'Uploads[].{Key:Key,UploadId:UploadId}' \
  --output text | while read key upload_id; do
    aws s3api abort-multipart-upload \
      --bucket your-bucket-name \
      --key "$key" \
      --upload-id "$upload_id"
    echo "Aborted: $key ($upload_id)"
done

Run that on your production bucket before setting the lifecycle rule. The rule will handle future orphans; this handles the ones already sitting there.

The Gotchas That Bit Me

Part size minimums. Every part except the last must be at least 5 MB. If you're building a custom chunked uploader and send a 1 MB part in the middle of the sequence, S3 will reject the CompleteMultipartUpload call with an EntityTooSmall error. The failure doesn't happen when you upload the part - it happens at assembly time. That delay makes it confusing to debug.

Part count ceiling. Maximum 10,000 parts per upload. If you're uploading a 100 GB file and set your part size to 5 MB, you'll hit this limit. Do the math upfront: file size / max parts = minimum viable part size.

Presigned URLs for parts. When you're coordinating browser-side uploads, each part needs its own presigned URL. You can't just generate one URL for the whole multipart session. I've seen people generate the initiate URL correctly and then get confused why part uploads are failing auth.

ListMultipartUploads is paginated. If you're checking programmatically for orphans, don't assume the first response is complete. Use the continuation tokens like any other S3 list operation.

When I'd Reach for This vs. When I Wouldn't

Reach for multipart upload when:

• Files are routinely over 100 MB
• You need per-chunk retry without full restart
• Files can exceed 5 GB (required)
• Uploads are coordinated from the browser and you want real progress

Skip the complexity when:

• You're moving files under 50 MB server-side with a reliable connection
• You're using a managed tool like the AWS CLI or the SDK's upload() helper - those handle multipart automatically above a configurable threshold, so you don't need to think about it
• The extra failure surface isn't worth it for your use case

For most Laravel apps using Storage::put() against S3, the Flysystem adapter calls the SDK's managed uploader, which will use multipart automatically above 100 MB. You get the parallelism for free. But you still need that lifecycle rule, because Flysystem doesn't set one for you, and a failed job mid-upload still orphans parts.

Closing

Multipart upload is one of those S3 features that works great when the happy path holds and silently costs you money when it doesn't. Add the lifecycle rule to every bucket you create - it takes two minutes and it's the kind of thing you'll forget to do until you're explaining a confusing S3 bill to a client. I've had that conversation twice. Once is educational. Twice is embarrassing.