log in
consulting hosting industries the daily tools about contact

Docker Compose Resource Limits: Why Your OOM Kills Are Silent

Without memory limits in your Compose file, a runaway container will silently eat your host and take everything else down with it. Here's how to fix that.

I lost a production server to a PHP worker that decided to cache an entire dataset in memory. No alert fired. No clear error in the logs. The container just disappeared and took three other services with it. That was the day I stopped treating deploy.resources in Compose files as optional.

What Actually Happens Without Limits

When you run a container without memory limits, the Linux kernel will let it consume every byte of available RAM on the host. When it finally runs out, the OOM killer picks a process to terminate. Sometimes that's your runaway container. Sometimes it's something else entirely - a database, a queue worker, the Docker daemon itself.

The problem is not just that things die. It's that they die silently, and the kill happens at the kernel level, not the application level. Your app never gets a signal it can handle. There's no graceful shutdown, no stack trace, no log line saying "killed due to memory pressure." The container just stops. docker ps shows it as exited with code 137, if you know to look.

Code 137 is SIGKILL. It means the kernel killed the process without asking permission. If you're not specifically watching for exit code 137 in your monitoring, you will miss it.

The Compose File Fix

Here's a realistic docker-compose.yml for a Laravel app with a queue worker - the kind of setup I deploy constantly for clients:

services:
  app:
    image: myapp:latest
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: "1.0"
        reservations:
          memory: 128M
          cpus: "0.25"

  queue:
    image: myapp:latest
    command: php artisan queue:work --max-jobs=100 --max-time=3600
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 256M
          cpus: "0.5"
        reservations:
          memory: 64M
          cpus: "0.1"

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 128M
          cpus: "0.25"
        reservations:
          memory: 32M

  postgres:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: myapp
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    deploy:
      resources:
        limits:
          memory: 1G
          cpus: "2.0"
        reservations:
          memory: 256M
          cpus: "0.5"

The limits block is what actually caps consumption. The reservations block tells the scheduler how much to expect at minimum - it's less critical on a single host but matters if you're using Swarm or thinking about eventually moving to one.

Notice --max-jobs=100 --max-time=3600 on the queue worker. That's not just housekeeping. When you have a memory limit, a queue worker that runs forever and slowly leaks will eventually hit the ceiling and get killed mid-job. Recycling the worker every 100 jobs or every hour means you flush accumulated memory before it becomes a problem. It's belt-and-suspenders.

The Gotcha That Will Bite You: Compose v2 vs. Swarm Mode

Here's the part that trips people up. The deploy key in a Compose file was originally a Swarm-only concept. In older versions of Docker Compose (v1, the Python-based one), deploy.resources was silently ignored when you ran docker-compose up. Not warned about. Ignored.

Docker Compose v2 (the Go rewrite, now the default as docker compose with a space) respects deploy.resources for local deployments. But if you're on a team with mixed environments, or running on a server with an older Docker install, you need to verify this is actually working.

Check it with:

docker compose version
# Should be v2.x.x

# After starting your stack, verify the limit is applied:
docker inspect myapp-app-1 | grep -A 5 'Memory'

You want to see "Memory": 536870912 (that's 512M in bytes) in the output. If you see "Memory": 0, the limit is not applied and you are flying blind.

You can also confirm with:

cat /sys/fs/cgroup/memory/docker/<container-id>/memory.limit_in_bytes

If it returns a huge number (like 9223372036854771712), the kernel has no limit set on that container.

Actually Catching OOM Kills

Once limits are in place, containers will get OOM-killed at their own ceiling instead of taking the host down. That's good. But you still need to know when it happens.

In Laravel, I add a simple health check script that runs via cron on the host (not inside Docker - the whole point is monitoring from outside):

#!/bin/bash
# /usr/local/bin/check-container-health.sh
# Run this every 5 minutes from crontab

CONTAINERS=$(docker ps -a --format '{{.Names}} {{.Status}} {{.ExitCode}}')

while IFS= read -r line; do
    name=$(echo "$line" | awk '{print $1}')
    status=$(echo "$line" | awk '{print $2}')
    code=$(echo "$line" | awk '{print $NF}')

    if [[ "$status" == "Exited" && "$code" == "137" ]]; then
        echo "OOM kill detected: $name" | mail -s "Container OOM Kill: $name" ops@example.com
    fi
done <<< "$CONTAINERS"

It's basic, but it works. For clients with more infrastructure, I wire this into their existing alerting (Datadog, Better Uptime, whatever they're running). The point is you need something checking for exit code 137 specifically, because general "container is down" alerts often don't tell you why.

You can also pull OOM events directly from the kernel:

dmesg | grep -i 'oom\|killed process'

That'll show you what the kernel actually killed and when, with more detail than Docker logs.

Sizing the Limits

This is the part nobody has a clean answer for. My approach:

Step 1: Baseline under real load. Run your app for a week with no limits and use docker stats to watch actual consumption. Note the typical range and the spikes.

docker stats --no-stream --format 'table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'

Step 2: Set the limit at roughly 2x the typical peak. Enough headroom that normal traffic spikes don't trigger kills, but tight enough that a runaway job hits the wall before it destabilizes the host.

Step 3: Watch for 137 exits in the first two weeks. If a container keeps getting killed, either the limit is too tight or you have a real memory problem worth investigating. Both are useful information.

For a typical Laravel app doing web requests, 256-512M is usually right. Queue workers doing heavy processing - image manipulation, PDF generation, hitting external APIs in loops - often need more and I'll go to 512M-1G. Databases I leave generous because Postgres will use whatever you give it for caching and that's usually a good thing, but I still cap it so a runaway query can't eat the whole host.

When I'd Skip This (Honestly, Never)

I cannot think of a production situation where you'd deliberately leave limits off. Even on a single-service host with nothing else running, an unlimited container can consume enough memory to destabilize the kernel itself. I've seen it happen with a biotech client's data processing job - single container, dedicated VM, no limits, and it still managed to crash the host by exhausting swap as well as RAM and corrupting a write in progress.

Local development is the one place I'm looser about it. I don't want artificial limits slowing down a dev environment or causing confusion when someone runs a one-off migration that uses more memory than the prod limit allows. My docker-compose.override.yml for dev omits the resource limits entirely, which Docker Compose merges cleanly.

For production, staging, and anything customer-facing: limits are not optional.

The Underlying Principle

Containers give you isolation, but they don't give you protection for free. The namespace and cgroup isolation that makes containers useful is also what makes failures quiet - the container's problem doesn't automatically become a visible error in your monitoring. You have to wire that up deliberately.

Setting memory and CPU limits is the minimum. It's one config block that takes ten minutes to think through and prevents the kind of 2am incident where you're staring at a dead server trying to reconstruct what happened from kernel logs.

I've shipped enough systems that failed in boring, preventable ways to have zero patience for "we'll add observability later." Later is when the server is on fire. Set the limits now.

Related

Need help shipping something like this? Get in touch.