Five Observability Signals You Already Have (No Datadog Required)

I've never had a client who wanted to pay for Datadog. I've had plenty who needed what Datadog sells. Most of the time I can give them 80% of it for free, using log data that their server is already producing and throwing away.

This isn't a "Datadog is bad" post. Datadog is genuinely good. But for the kind of work I do — managed hosting for small-to-mid-size businesses, apps running on a handful of VMs, clients who don't have a DevOps budget — standing up a full SaaS observability stack is overkill. The signal you need is already sitting in three places: nginx's access log, MariaDB's slow query log, and the systemd journal. You just have to read them.

Here are five concrete signals I actually track, and how I pull them.

1. Request rate and traffic shape (nginx access log)

The first thing I want to know about a running app is: how many requests per minute, and is that number weird right now?

nginx writes a combined log entry for every request. The default format includes timestamp, status code, upstream response time, bytes sent. That's enough.

For a quick pulse, I reach for awk and sort:

# Requests per minute for the last 1000 lines
awk '{print $4}' /var/log/nginx/access.log \
  | cut -d: -f1-3 \
  | sort \
  | uniq -c \
  | tail -20

The $4 field in combined log format is [day/month/year:hour:minute:second. Cutting to hour:minute gives you per-minute bucketing.

For something I can actually alert on, I have a small shell script that runs from cron every five minutes, writes the RPM to a flat file, and emails me if it's more than 3x the rolling average. Not elegant. Works fine.

If you want to go further without SaaS, goaccess parses nginx logs in real time and renders a terminal dashboard. I've left it running in a tmux pane during deploys and it's saved me twice.

2. Error rate by status code (nginx access log)

Request volume tells you if traffic is normal. Status code distribution tells you if the app is healthy.

# Status code breakdown for the last hour
# Assumes combined log format, $9 is status code
awk -v cutoff="$(date -d '1 hour ago' '+%d/%b/%Y:%H:%M')" '
  $4 > "["cutoff { print $9 }
' /var/log/nginx/access.log \
  | sort \
  | uniq -c \
  | sort -rn

What I actually care about: the ratio of 5xx to total requests. If it climbs above 1% I want to know. If it's above 5% something is on fire.

I also watch for 499s specifically. That's nginx's code for "client closed connection before response." A spike in 499s usually means your upstream is slow — PHP-FPM is backed up, a query is running long, something is hanging. It's often the first signal that a problem is developing before you start seeing 504s.

I added $upstream_response_time to my nginx log format a few years ago and never looked back:

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
               '$status $body_bytes_sent "$http_referer" '
               '"$http_user_agent" rt=$request_time '
               'uct=$upstream_connect_time uht=$upstream_header_time '
               'urt=$upstream_response_time';

Now I can grep for slow requests directly from the access log without touching the app:

# Requests where upstream took more than 2 seconds
grep 'urt=' /var/log/nginx/access.log \
  | awk -F'urt=' '{print $2, $0}' \
  | awk '$1 > 2.0' \
  | tail -50

3. Slow queries (MariaDB slow query log)

This one is criminally underused. The MariaDB slow query log has been around forever, it's free, and it will tell you exactly which queries are hurting you.

Enable it in /etc/mysql/mariadb.conf.d/50-server.cnf:

slow_query_log = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time = 1
log_queries_not_using_indexes = 1
min_examined_row_limit = 100

The log_queries_not_using_indexes flag is the one I always turn on. It catches the full-table scans you forgot about.

Once you have a slow log, mysqldumpslow summarizes it:

# Top 10 slowest query patterns, last 24 hours
mysqldumpslow -s t -t 10 /var/log/mysql/mysql-slow.log

The -s t sorts by total time, so you see the queries that are costing you the most aggregate time across all executions — not just the single slowest one-off.

I integrated this for a Seattle biotech client last year. They had a LIMS-adjacent app that was getting sluggish around month-end when reports ran. Nobody knew why. Turned on the slow log, ran mysqldumpslow after the next report cycle, and the top entry was a COUNT query doing a full scan on a 4-million-row table because someone had filtered on an unindexed status column. One index, problem gone. Took twenty minutes once I had the data.

For something more automated, I use a small Laravel command that tails the slow log and ships entries to a local SQLite file, then exposes them through an internal admin route. Nothing fancy, but it means I can check slow queries from a browser instead of SSHing into the box:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\DB;

class IngestSlowQueryLog extends Command
{
    protected $signature = 'db:ingest-slow-log';
    protected $description = 'Parse MariaDB slow query log into SQLite for internal review';

    public function handle(): int
    {
        $logPath = config('monitoring.slow_query_log', '/var/log/mysql/mysql-slow.log');

        if (! file_exists($logPath)) {
            $this->error("Log not found: {$logPath}");
            return 1;
        }

        $content = file_get_contents($logPath);
        $entries = $this->parse($content);

        foreach ($entries as $entry) {
            DB::connection('sqlite_monitoring')->table('slow_queries')->insertOrIgnore([
                'fingerprint' => md5($entry['query']),
                'query'       => $entry['query'],
                'duration'    => $entry['query_time'],
                'rows_examined' => $entry['rows_examined'],
                'captured_at' => $entry['timestamp'],
            ]);
        }

        $this->info('Ingested ' . count($entries) . ' entries.');
        return 0;
    }

    private function parse(string $content): array
    {
        $entries = [];
        // Split on the # Time / # User markers that precede each entry
        $blocks = preg_split('/(?=# Time:)/m', $content, -1, PREG_SPLIT_NO_EMPTY);

        foreach ($blocks as $block) {
            if (! preg_match('/# Query_time: (\S+)/', $block, $qt)) continue;
            if (! preg_match('/# Rows_examined: (\d+)/', $block, $re)) continue;

            $timestamp = null;
            if (preg_match('/# Time: ([\d\s:-]+)/', $block, $ts)) {
                $timestamp = trim($ts[1]);
            }

            // The actual SQL is the last non-comment, non-SET line
            $lines = array_filter(explode("\n", $block), fn($l) => !
                str_starts_with(trim($l), '#') && trim($l) !== '' &&
                ! str_starts_with(trim($l), 'SET timestamp')
            );

            $query = trim(implode(' ', $lines));
            if (empty($query)) continue;

            $entries[] = [
                'query_time'    => (float) $qt[1],
                'rows_examined' => (int) $re[1],
                'timestamp'     => $timestamp,
                'query'         => $query,
            ];
        }

        return $entries;
    }
}

This runs from the scheduler every 15 minutes. Good enough.

4. Service restarts and crash loops (systemd journal)

If PHP-FPM or MariaDB or your queue worker is crashing and restarting, I want to know before a client calls me. systemd knows exactly when this happens.

# Services that have restarted more than once in the last hour
journalctl --since "1 hour ago" \
  | grep -E 'Started|start request repeated' \
  | awk '{print $5}' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -20

More precisely, for a specific service:

# How many times has php-fpm restarted today?
journalctl -u php8.3-fpm --since today \
  | grep -c 'Started'

I set up a cron that runs this check for a list of critical services and posts to a Slack webhook if any of them has restarted more than twice in an hour. The webhook call is a single curl. No agent, no SDK, no SaaS account required.

#!/bin/bash
SERVICES=("php8.3-fpm" "mariadb" "nginx" "redis")
WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
THRESHOLD=2

for svc in "${SERVICES[@]}"; do
    count=$(journalctl -u "$svc" --since "1 hour ago" 2>/dev/null | grep -c 'Started')
    if [ "$count" -gt "$THRESHOLD" ]; then
        curl -s -X POST -H 'Content-type: application/json' \
          --data "{\"text\":\"ALERT: $svc has restarted $count times in the last hour on $(hostname)\"}" \
          "$WEBHOOK_URL"
    fi
done

5. OOM kills and kernel-level crashes (systemd journal)

This one saves me from the most confusing incidents. When a process gets OOM-killed, nothing in your application logs will tell you why it died. It just stops. The kernel journal entry is the only record.

# OOM kills in the last 7 days
journalctl -k --since "7 days ago" \
  | grep -i 'killed process\|out of memory'

The output looks like:

Jun 15 03:42:17 prod-app kernel: Out of memory: Killed process 18842 (php-fpm) score 312 or sacrifice child
Jun 15 03:42:17 prod-app kernel: Killed process 18842 (php-fpm) total-vm:512844kB, anon-rss:388216kB

That tells me which process was killed, how much memory it was using, and exactly when. Combined with the nginx 502s that followed at the same timestamp, the incident reconstructs itself.

I've added this check to the same cron script as the service restart checker. It runs at 7am daily and emails me a summary. Most days the summary is empty. That's the goal.

When this is enough and when it isn't

This setup covers the failure modes I actually see in production: traffic anomalies, slow queries, crashing services, and OOM events. For most of the apps I run, those four categories account for 90% of incidents.

What it doesn't give you: distributed tracing, per-deployment comparison, anomaly detection that actually learns baselines, or any of the dashboard features that make Datadog worth paying for at scale. If you're running microservices across a fleet of 50 nodes, do not try to replicate this with shell scripts. Get a real observability platform.

But if you're running a Laravel app on two or three VMs, managing your own nginx and MariaDB, and your clients are small enough that a $500/month observability bill would be a significant line item — the data is already there. You just have to look at it.

The best monitoring system is the one you'll actually check. For me, that's a daily email digest and a Slack alert that fires maybe twice a month. Everything I need to diagnose those alerts is in logs that nginx, MariaDB, and systemd are already writing. No agents, no dashboards, no SaaS login to remember.

Start with the slow query log. Turn it on today. You'll find something within a week.