The Rollback Conditions Nobody Writes Until 3am
Automated rollback sounds easy until production is on fire. Here's the actual condition logic I use to catch bad Laravel deploys before users do.
Every team I've talked to has automated deploys. Almost none of them have automated rollbacks. They have a rollback command — php artisan down, swap the symlink, php artisan up — but the part that decides to run it is a tired engineer squinting at Slack at 3am. That's not automation. That's a pager.
I've been burned enough times that I finally sat down and codified the conditions. This post is that list, with real code and the reasoning behind each check.
The Actual Problem
Zero-downtime deploys in Laravel — whether you're using Envoyer, a custom Deployer script, or rolling your own with rsync and symlinks — are great at the mechanics. New release directory, build assets, run migrations, flip the symlink, reload PHP-FPM. Clean.
But "zero downtime" refers to the process, not the outcome. You can swap a symlink perfectly and still serve a broken application for 20 minutes while your on-call engineer sleeps through the first three PagerDuty alerts.
The gap is health validation with teeth. Not just "did the deploy script exit 0" but "is this application actually working five minutes after we flipped the switch, and if not, did we automatically go back?"
The Deployment Structure I'm Working With
I'll assume a standard symlink-based setup. Current release lives at /var/www/releases/20240912143000, with /var/www/current pointing to it. A rollback is:
ln -sfn /var/www/releases/20240912131500 /var/www/current
sudo systemctl reload php8.3-fpm
Simple. The hard part is knowing when to run it.
The Health Check Script
After every deploy, I run a validation script. If it fails, we roll back automatically. Here's the core of it — a Laravel Artisan command I run post-deploy from the deployment pipeline:
<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Http;
class DeployHealthCheck extends Command
{
protected $signature = 'deploy:health-check
{--timeout=120 : Seconds to wait for health before failing}
{--error-threshold=5 : Max allowed 5xx responses per minute}
{--queue-threshold=500 : Max failed jobs before flagging}';
protected $description = 'Post-deploy health validation with rollback signal';
public function handle(): int
{
$timeout = (int) $this->option('timeout');
$deadline = now()->addSeconds($timeout);
$checks = [];
$this->info('Starting post-deploy health checks...');
// Check 1: Database connectivity
$checks['database'] = $this->checkDatabase();
// Check 2: Cache connectivity
$checks['cache'] = $this->checkCache();
// Check 3: HTTP smoke test
$checks['http'] = $this->checkHttp();
// Check 4: Queue worker health
$checks['queue'] = $this->checkQueueWorkers();
// Check 5: Failed job spike
$checks['failed_jobs'] = $this->checkFailedJobs(
(int) $this->option('queue-threshold')
);
// Check 6: Error rate (polling until timeout)
$checks['error_rate'] = $this->checkErrorRate(
(int) $this->option('error-threshold'),
$deadline
);
$failed = array_filter($checks, fn($result) => $result === false);
if (! empty($failed)) {
$this->error('Health checks FAILED: ' . implode(', ', array_keys($failed)));
return Command::FAILURE; // deployment script reads exit code 1
}
$this->info('All health checks passed.');
return Command::SUCCESS;
}
private function checkDatabase(): bool
{
try {
DB::select('SELECT 1');
$this->line(' [OK] Database');
return true;
} catch (\Throwable $e) {
$this->error(' [FAIL] Database: ' . $e->getMessage());
return false;
}
}
private function checkCache(): bool
{
try {
$key = 'deploy_health_' . time();
Cache::put($key, true, 10);
$result = Cache::get($key) === true;
Cache::forget($key);
$this->line($result ? ' [OK] Cache' : ' [FAIL] Cache: read-back failed');
return $result;
} catch (\Throwable $e) {
$this->error(' [FAIL] Cache: ' . $e->getMessage());
return false;
}
}
private function checkHttp(): bool
{
// Hit your internal health endpoint, not the public URL
// This bypasses load balancer routing during deploy
$url = 'http://127.0.0.1/health';
try {
$response = Http::timeout(5)->get($url);
$ok = $response->successful();
$this->line($ok ? ' [OK] HTTP' : ' [FAIL] HTTP: status ' . $response->status());
return $ok;
} catch (\Throwable $e) {
$this->error(' [FAIL] HTTP: ' . $e->getMessage());
return false;
}
}
private function checkQueueWorkers(): bool
{
// Check that at least one queue worker process is running
// We write a heartbeat from the worker via a scheduled task
$lastBeat = Cache::get('queue_worker_heartbeat');
if (! $lastBeat) {
// Give workers 30s to restart before flagging
sleep(30);
$lastBeat = Cache::get('queue_worker_heartbeat');
}
$ok = $lastBeat && now()->diffInSeconds($lastBeat) < 60;
$this->line($ok ? ' [OK] Queue workers' : ' [FAIL] Queue workers: no heartbeat');
return $ok;
}
private function checkFailedJobs(int $threshold): bool
{
$count = DB::table('failed_jobs')
->where('failed_at', '>=', now()->subMinutes(5))
->count();
$ok = $count < $threshold;
$this->line($ok
? " [OK] Failed jobs ({$count} in last 5m)"
: " [FAIL] Failed jobs: {$count} failures in last 5m (threshold: {$threshold})"
);
return $ok;
}
private function checkErrorRate(int $threshold, \Carbon\Carbon $deadline): bool
{
// Poll the error rate until we're confident or time out
// I pull this from a small stats table my middleware writes to
$this->line(' Watching error rate...');
while (now()->lt($deadline)) {
$rate = $this->getErrorRate();
if ($rate >= $threshold) {
$this->error(" [FAIL] Error rate: {$rate} 5xx/min (threshold: {$threshold})");
return false;
}
sleep(15);
}
$this->line(' [OK] Error rate');
return true;
}
private function getErrorRate(): int
{
return (int) DB::table('request_logs')
->where('status_code', '>=', 500)
->where('logged_at', '>=', now()->subMinute())
->count();
}
}
The deploy script — Bash, Deployer, whatever you're using — checks the exit code:
php artisan deploy:health-check --timeout=120 --error-threshold=5
if [ $? -ne 0 ]; then
echo "Health check failed. Rolling back."
ln -sfn "$PREVIOUS_RELEASE" /var/www/current
sudo systemctl reload php8.3-fpm
php artisan queue:restart
exit 1
fi
The Gotchas That Will Bite You
Migrations are the sneaky one. If your deploy runs migrations before flipping the symlink (correct order), the old code is running against a schema it doesn't fully understand during the window. If your migrations are additive-only, this is usually fine. If you renamed a column, you've just broken prod. I enforce additive-only migrations as policy and do destructive cleanup in a separate deploy after the smoke has cleared.
Queue workers don't restart themselves. After flipping the symlink, workers are still running the old code from memory. You need php artisan queue:restart in your deploy script, and the health check needs to wait for them to come back up. My 30-second sleep in checkQueueWorkers is not elegant but it works. I've seen teams check for workers immediately after restart and wonder why the heartbeat is stale.
The error rate window is almost always too short. I used to use a 30-second window. Then I deployed a bug that only triggered on a specific checkout flow — low traffic at 11pm — and it took 12 minutes before error rate climbed above my threshold. Now I use 120 seconds minimum for anything customer-facing. Yes, that makes deployments slower. That's the tradeoff.
Localhost vs. public URL matters for the HTTP check. If you hit the public domain, you might be hitting the load balancer, which may still be routing to the old release on another server. Hit 127.0.0.1 directly with a Host header if you need to, but make sure you're testing this server's new code.
Cache invalidation during rollback. If the new release wrote cache keys with a different structure, rolling back leaves the old code reading malformed cache entries. I append the release timestamp to cache key prefixes via a config value (CACHE_PREFIX=20240912143000) and flush on rollback. Painful to set up, worth it.
queue_worker_heartbeat has to actually exist. I write it from a scheduled command that runs every 30 seconds inside the worker process. If you skip building this heartbeat mechanism, the queue check is useless. Don't skip it.
When I'd Reach For This
Any application where a bad deploy is worse than a slow deploy. That's basically everything I ship for paying clients — e-commerce checkouts, healthcare portals, biotech LIMS integrations. I had a client with a print management platform where a bad release once corrupted job pricing calculations silently for 40 minutes. Nobody's rollback script caught it because nobody was checking business logic, only process health. Now I add domain-specific checks to DeployHealthCheck for anything with financial calculations — a quick sanity query that validates a known-good aggregate.
I wouldn't bother for internal tools with three users, or for projects where the deploy window is already a maintenance window with users locked out. This complexity has a cost. Pay it when downtime has a dollar sign attached.
When I Wouldn't
If you're running Kubernetes with proper readiness probes and a real rollout strategy, some of this is handled at the platform level. But most of my clients aren't running Kubernetes — they're on VPS clusters managed by me, and they don't need that operational overhead. For that world, a careful deploy script and this health check command is more maintainable than a full container orchestration stack.
The 3am version of this post is a lot shorter: it's just me typing ln -sfn frantically and hoping I got the path right. The version above took a few painful incidents to arrive at, and it's still not perfect — no automated system catches everything. But it catches most things, automatically, before I'm even awake to see the Slack notification. That's the whole point.
Need help shipping something like this? Get in touch.