Galera Cluster's Write Penalty Will Quietly Ruin Your Week

I spent three weeks diagnosing a latency problem for a client that turned out to be a feature, not a bug. That's the most frustrating kind of bug to find.

The setup was a three-node Galera cluster running MariaDB 10.6, hosted across two availability zones on AWS. The goal was high availability with no single point of failure and the ability to write to any node. Sounds great on paper. In practice, their checkout flow went from 180ms average response time on a single primary to 420ms on the cluster. Nothing in the application had changed. No slow queries. No lock contention. Just Galera doing exactly what Galera does.

What Galera Actually Is

Galera is synchronous multi-master replication. Every node in the cluster can accept writes. When you write to node A, that write doesn't commit until every other node in the cluster has certified it. "Certified" means each node has checked the write against its local certification database to confirm there are no conflicts with in-flight transactions on that node.

This is fundamentally different from standard MySQL/MariaDB async replication, where the primary commits locally and ships the binlog to replicas in the background. In async replication, the replica might be 50ms behind, but your write returns immediately. In Galera, your write waits for the network round-trip to every peer, plus the certification check on each one.

The marketing copy calls this "virtually synchronous" replication, and it's technically accurate — but the word "virtually" is doing a lot of work there. For reads and for workloads that are mostly read-heavy, Galera is excellent. For write-heavy or write-latency-sensitive workloads on geographically distributed nodes, you're paying a tax on every single commit.

The Mechanics of the Penalty

Here's what happens when you call $pdo->commit():

1. The local node prepares the write-set (all row changes in the transaction)
2. It broadcasts the write-set to all other nodes via Group Communication System (GCS)
3. Every node runs certification: does this write-set conflict with any other pending write-set?
4. Every node acknowledges back
5. Only after unanimous acknowledgment does the originating node actually apply and return success

That round-trip is the penalty. On a single datacenter with nodes 1ms apart, you might not notice it. Add a second AZ that's 12ms away and every commit takes at least 24ms in network time alone, before anything else happens. Now stack 10 small writes inside what used to be a fast request and you've added 240ms of pure waiting.

You can see this clearly if you instrument it:

<?php

use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;

function timedCommitExample(array $orderData): void
{
    $start = hrtime(true);

    DB::transaction(function () use ($orderData) {
        DB::table('orders')->insert([
            'user_id'    => $orderData['user_id'],
            'total'      => $orderData['total'],
            'status'     => 'pending',
            'created_at' => now(),
            'updated_at' => now(),
        ]);

        DB::table('inventory')->where('sku', $orderData['sku'])
            ->decrement('quantity', $orderData['qty']);

        DB::table('order_events')->insert([
            'order_id'   => DB::getPdo()->lastInsertId(),
            'event'      => 'created',
            'created_at' => now(),
        ]);
    });

    $elapsed = (hrtime(true) - $start) / 1_000_000; // ms

    Log::info('transaction committed', [
        'elapsed_ms' => round($elapsed, 2),
        'node'       => DB::select('SELECT @@hostname as h')[0]->h,
    ]);
}

Run that against a single MariaDB instance and you'll see 3–6ms. Run it against a two-AZ Galera cluster and you'll see 28–50ms consistently. Same query plan, same indexes, same data volume. The difference is entirely network certification overhead.

The Gotchas That Will Bite You

Write conflicts cause silent retries or hard errors. If two nodes receive conflicting writes to the same row at nearly the same time, one of them loses certification and gets a deadlock error — Error 1213: Deadlock found when trying to get lock. In standard MySQL this means two transactions genuinely deadlocked each other. In Galera it might mean two nodes both tried to update the same inventory row within the same certification window. Your app needs to handle this explicitly. Laravel's DB::transaction() doesn't automatically retry on certification failures.

<?php

function withGaleraRetry(callable $callback, int $maxAttempts = 3): mixed
{
    $attempt = 0;

    while (true) {
        try {
            return DB::transaction($callback);
        } catch (\Illuminate\Database\QueryException $e) {
            // 1213 = deadlock (includes Galera certification failure)
            // 1205 = lock wait timeout
            $attempt++;
            $shouldRetry = in_array($e->getCode(), ['1213', '1205'])
                && $attempt < $maxAttempts;

            if (! $shouldRetry) {
                throw $e;
            }

            usleep(random_int(10_000, 50_000)); // back off 10–50ms
        }
    }
}

I added something like this to the checkout service for that client and immediately stopped seeing 500 errors that had been chalked up to "occasional weirdness."

Flow control will pause your entire application. Galera has a backpressure mechanism called flow control. If a node falls behind applying write-sets (maybe it's under load, maybe there was a brief network hiccup), the GCS will pause the entire cluster to let it catch up. From the application side this looks like all your database queries just... stopped. For 200ms. Or 2 seconds. You'll see it in your slow query log as queries with normal execution times but huge wait times before they even start.

You can monitor this:

SHOW STATUS LIKE 'wsrep_flow_control_paused';
SHOW STATUS LIKE 'wsrep_flow_control_sent';
SHOW STATUS LIKE 'wsrep_local_recv_queue_avg';

If wsrep_flow_control_paused is anything other than zero, you have a lagging node. If it's consistently above 0.1, you have a real problem.

DDL is a cluster-wide lock. Running ALTER TABLE on a Galera cluster with the default TOI (Total Order Isolation) method locks the entire cluster for the duration of the migration. Every node, every write, blocked. For a busy table this is catastrophic in production. You need pt-online-schema-change or MariaDB's ALGORITHM=INPLACE where supported, and you need to know which DDL operations support it before you run a deploy.

Your "write to any node" assumption is probably wrong anyway. Most apps using Galera still pick a preferred write node through their load balancer (HAProxy or ProxySQL) and only fail over to other nodes. If you're truly round-robining writes across nodes, you're increasing your certification conflict rate. The multi-master capability is mostly useful for failover, not for simultaneous multi-node write throughput.

When I'd Reach for Galera

Galera is a good answer for a specific situation: you need automatic failover with no data loss, your write volume is moderate, and all your nodes are in the same datacenter or same low-latency AZ. Healthcare and biotech clients come to mind — they need the HA story, they need the no-data-loss guarantee, and their write patterns are usually transactional but not high-frequency. I run it in exactly that configuration for a couple of clients and it works well.

It's also reasonable if your application is read-heavy (80%+ reads) and you're scaling reads across nodes. The latency penalty only applies to writes. If you're doing one write per 20 reads, the math changes significantly.

When I Wouldn't Touch It

E-commerce checkout. Any real-time inventory system. Anything with a tight response-time SLA on write paths. Anything geographically distributed across regions where your inter-node latency is measured in tens of milliseconds.

For that Seattle biotech client's original problem, the answer ended up being boring and correct: single primary with one async read replica, automatic failover via Route 53 health checks, and an accepted RTO of about 60 seconds. They lost the "zero data loss" guarantee in theory, but async replication lag was under 100ms in practice, and their write latency went back to 8ms. The checkout conversion rate recovered the same week we made the switch.

For workloads that genuinely need distributed writes with strong consistency, I'd look hard at PlanetScale (Vitess under the hood) or just accept that CockroachDB or Postgres with Patroni might be a better architectural fit than trying to make Galera do something it wasn't designed for.

The Bottom Line

Galera Cluster is not a free upgrade from a single primary. It's a trade: you get synchronous replication and multi-master failover, and you pay for it with write latency proportional to your inter-node round-trip time. Small teams almost always underestimate that cost because they benchmark on a single machine before they deploy to production.

Read the wsrep_* status variables before you go live. Know your inter-AZ latency. And if your write path is performance-sensitive, add up how many commits you make per request before you commit to the architecture.