log in
consulting hosting industries the daily tools about contact

Move MariaDB Off Your App Server Before It Bites You

Running your database on the same VM as your app feels fine until it isn't. Here's the 24-hour migration plan I've used more than once.

Running your database on the same VM as your application feels completely reasonable when you're starting out. One server, one bill, one thing to SSH into. I've set up dozens of stacks this way for small clients. The problem is that "starting out" has a way of lasting five years, and by then you're staring at a 2 AM alert because the app server ran out of memory and took the database down with it.

This is the migration plan I've executed several times now - most recently for a Seattle-area print management company that had been running a single Linode for four years. No downtime, no data loss, done in an afternoon.

Why This Gets Urgent

The single-VM setup has a few specific failure modes that I've watched play out in production:

  • Your app starts leaking memory or spawning too many PHP-FPM workers under load. The OOM killer picks a victim. Sometimes it picks mysqld.
  • You need to scale the app horizontally - add a second web server, put something behind a load balancer. You can't do that if the database only lives on server one.
  • You want to snapshot or resize the app server. A live database makes that messy.
  • Log files or upload storage fills the disk. MariaDB starts refusing writes.

None of these are edge cases. I've hit all four with real clients.

The Plan Overview

Here's what the 24-hour window looks like, roughly:

  1. Provision the new DB server (1-2 hours)
  2. Install and harden MariaDB on it (30 minutes)
  3. Dump and restore the data (15 minutes to several hours depending on size)
  4. Set up replication from old to new (live sync while you prep)
  5. Update your app config and cut over (5 minutes of downtime, maybe)
  6. Verify, then decommission the old DB install

Step 4 is the part most guides skip, and it's what lets you do this with minimal downtime.

Provision and Harden the DB Server

I spin up a separate VPS on the same provider, same datacenter as the app server. Private networking between them is non-negotiable - you do not want your database port anywhere near the public internet.

On the new DB server:

apt update && apt install -y mariadb-server
mysql_secure_installation

Then edit /etc/mysql/mariadb.conf.d/50-server.cnf. The critical line:

bind-address = 10.x.x.x  # your private IP, not 0.0.0.0

Also tune for the hardware you have. A DB-only VM can be more aggressive than a shared one:

innodb_buffer_pool_size = 1G        # ~50-70% of RAM on a DB-only box
innodb_log_file_size = 256M
max_connections = 150
query_cache_type = 0                # query cache is a footgun in MariaDB 10.x

Restart MariaDB, verify it's listening on the private IP only:

ss -tlnp | grep 3306

Set Up the Replication User

On the old server (the one running both app and DB), log into MySQL as root and create a replication user:

CREATE USER 'replicator'@'10.x.x.y' IDENTIFIED BY 'a-strong-passphrase-here';
GRANT REPLICATION SLAVE ON *.* TO 'replicator'@'10.x.x.y';
FLUSH PRIVILEGES;

10.x.x.y is the new DB server's private IP.

Also enable binary logging on the old server if it isn't already. In /etc/mysql/mariadb.conf.d/50-server.cnf:

server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_format = ROW

Restart MariaDB on the old server. This is your one required restart on the source - do it during off-peak hours.

Take the Initial Dump

With binary logging on, take a consistent dump using --master-data. This embeds the binlog position in the dump file so replication can pick up exactly where the snapshot left off:

mysqldump \
  --single-transaction \
  --master-data=2 \
  --all-databases \
  --routines \
  --triggers \
  -u root -p \
  > /tmp/full_dump.sql

--single-transaction avoids table locks on InnoDB. If you have MyISAM tables (you shouldn't, but here we are), add --lock-all-tables instead and accept a brief write pause.

SCP the dump to the new server:

scp /tmp/full_dump.sql newdbserver:/tmp/

Restore it on the new server:

mysql -u root -p < /tmp/full_dump.sql

For a 5GB database this takes 10-15 minutes. For a 50GB database, go get lunch.

Wire Up Replication

The dump file has the binlog position commented near the top:

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=12345;

On the new server, run:

CHANGE MASTER TO
  MASTER_HOST='10.x.x.x',
  MASTER_USER='replicator',
  MASTER_PASSWORD='a-strong-passphrase-here',
  MASTER_LOG_FILE='mysql-bin.000003',
  MASTER_LOG_POS=12345;

START SLAVE;
SHOW SLAVE STATUS\G

You're looking for Seconds_Behind_Master to count down to zero and then hold there. Once it's at zero, the new server is fully caught up and staying current with every write happening on the old one.

Let it run for an hour. Watch it. Make sure it stays at zero.

Update Your App and Cut Over

In Laravel, your database config lives in .env:

DB_HOST=10.x.x.x   # old value, app server localhost or 127.0.0.1

Before you flip it, create the application's DB user on the new server:

CREATE USER 'appuser'@'10.x.x.z' IDENTIFIED BY 'app-db-password';
-- 10.x.x.z is the app server's private IP
GRANT ALL PRIVILEGES ON yourapp.* TO 'appuser'@'10.x.x.z';
FLUSH PRIVILEGES;

Now the actual cutover. I do this in a maintenance window, but it's usually under five minutes:

  1. Put the app in maintenance mode: php artisan down
  2. Stop the slave on the new server: STOP SLAVE;
  3. On the new server, make it writable (no longer a replica): RESET SLAVE ALL;
  4. Update .env on the app server: DB_HOST=10.x.x.y (new DB server's private IP)
  5. Run any pending migrations: php artisan migrate
  6. Bring the app back up: php artisan up

Test it. Hit the app. Watch your logs. If something is broken, you still have the old database on the app server sitting untouched - flip DB_HOST back and you're recovered.

The Gotchas That Get People

Firewall rules. Port 3306 needs to be open between the two private IPs. Just those two. I use UFW:

# On the new DB server
ufw allow from 10.x.x.z to any port 3306

If you forget this, Laravel throws SQLSTATE[HY000] [2002] Connection refused and you'll spend 20 minutes questioning your life choices before you check the firewall.

skip-name-resolve. Add this to the new server's config. Without it, MariaDB tries to do a reverse DNS lookup on every connection, and that adds latency:

skip-name-resolve

Different MariaDB versions. If you're migrating from, say, 10.3 on the old server to 10.11 on the new one, test your dump/restore in a staging environment first. Schema changes between minor versions are usually fine, but I've hit stored procedure syntax differences that required manual cleanup.

Replication errors on system tables. If SHOW SLAVE STATUS shows an error on mysql.* tables, that's usually permissions on system-level things that don't need to replicate anyway. You can skip specific errors with SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE; - but do this carefully and understand what you're skipping.

Laravel's DB_SOCKET setting. Some older setups connect via Unix socket instead of TCP. If your .env has a DB_SOCKET line, remove it or it'll ignore DB_HOST entirely.

When I'd Use This Approach

Any production app on a single VM that's been running more than 6 months and has real users. That's it. If you're building a demo or a staging environment, single-VM is fine. If someone is depending on it, they deserve separated concerns.

I'd also do this before any planned traffic increase - before a product launch, before onboarding a large client, before running a promotion. Much better to do it on your schedule than on the database's schedule.

When would I skip it? Very low traffic internal tools. Apps where the client has explicitly decided the cost of a second server isn't worth it and understands the tradeoff. Tools I'm sunsetting in the next 90 days.

What I Skip and What I Don't

I don't bother with a fancy orchestration tool or Ansible playbook for this. It's a one-time operation on a known server. SSH, a text editor, and MariaDB's own replication do the job.

What I don't skip: testing the application user's credentials on the new server before the cutover window. I once cut over, flipped the .env, and discovered I'd created the grant for the wrong IP. That was an extra three minutes of downtime at 11 PM that didn't need to happen.

Also don't skip: verifying the new server's time is synced with NTP. Replication can get weird with clock drift.

The whole thing sounds more complicated than it is. Once you've done it twice, it's a relaxed afternoon of mostly waiting for mysqldump to finish. The architecture on the other side - where your app can be restarted, resized, or scaled without touching your data - is worth every minute of it.

Related

Need help shipping something like this? Get in touch.