The Path to Logging Pro: Part 3. Structured logs, for free (you're welcome)

You’ve just set up a VPS. A few days in, out of curiosity, you check what SSH has been up to:

$ systemctl status sshd
● ssh.service - OpenBSD Secure Shell server
     Active: active (running) since Sat 2026-04-12 09:14:03 UTC
    ...
Apr 12 17:03:11 lpsz sshd[1432]: Failed password for root from 218.92.0.190 port 53412 ssh2
Apr 12 17:03:15 lpsz sshd[1433]: Failed password for root from 218.92.0.190 port 53413 ssh2
Apr 12 17:03:18 lpsz sshd[1434]: Failed password for root from 218.92.0.190 port 53414 ssh2

Welcome to the internet. Someone has been hammering your server since the moment you exposed it. This is completely normal — automated scanners hit every public IP constantly, probing for weak passwords and default credentials. But that’s a story for another day.

What I want to focus on is something less alarming: where are those log lines coming from? It must be reading from some log file, right? Maybe /var/log/sshd.log or something like that? Not quite. systemctl isn’t reading any service-specific log file. There isn’t one.

When a service runs on a Linux machine, its process has a file descriptor table — the same thing we talked about in the last two posts. Whatever the process writes to stdout or stderr goes straight into a pipe. And on the other end of that pipe sits a process called journald, part of the systemd init system. Something like this:

[sshd]  FD1 (stdout) ──┐
        FD2 (stderr) ──┤  pipe (in memory)  ──►  [journald]  ──►  /var/log/journal/

What’s happening under the hood is that systemd launches the process using fork(), exactly like the shell does — and as we saw in the previous post, the child process inherits the parent’s file descriptors. systemd just swaps stdout and stderr for the ends of a pipe before the process starts. The process has no idea. It just writes to its descriptors, same as always.

But journald doesn’t stop there. It doesn’t just save everything as plain text — it classifies each entry, tagging it with metadata so you can filter, search and cross-reference later.

# View logs for a specific service
$ journalctl -u ssh.service

# Same logs, with structure
$ journalctl -u ssh.service --output=json | head -1 | jq .

Here’s a real entry from my VPS — one of those failed login attempts:

{
  "_HOSTNAME": "lpsz",
  "_COMM": "sshd",
  "_EXE": "/usr/sbin/sshd",
  "_PID": "1432",
  "PRIORITY": "5",
  "MESSAGE": "Failed password for root from 218.92.0.190 port 53412 ssh2",
  "_TRANSPORT": "journal",
  "_BOOT_ID": "7a3f1c2d45b64ea8c8f1d2eb309e7e70",
  "_MACHINE_ID": "4199e14163be5cafc77fc656ee0def4e",
  ...
}

As plain text this would be one unreadable line — date, process name, and message all mashed together. Here, everything is separate and named. MESSAGE has the human-readable text. PRIORITY is a number from 0 to 7 indicating severity (5 is notice). _HOSTNAME tells you which machine it came from. _PID tells you which process. None of this was configured — journald inferred and added it automatically. Which means you can already filter by machine, by service, by severity level, without parsing anything:

# Only errors and above (priority 0-3)
$ journalctl -u ssh.service -p err

# Failed login attempts in the last hour
$ journalctl -u ssh.service --since "1 hour ago" | grep "Failed"

# Follow in real time, like tail -f
$ journalctl -u ssh.service -f

It’s the difference between rummaging through a junk drawer and searching an indexed archive.

That structure is journald’s superpower. And also its limit: it’s trapped on a single machine.

On a single server that’s fine. But real production systems have tens or hundreds of machines, each running multiple services, each with its own journald and its own logs. You can SSH into each one and filter away, but you can’t cross-reference. You’d have to copy request IDs by hand and hunt for them across machines one by one. So how do you investigate an incident when the error is spread across three services on three different servers? That’s what we’ll look at next post.