Checking logs using journalctl

Last updated: 10 May 2021

Systems that use systemd write log entries for the kernel and various services to a journal. These log entries are stored in a binary format and can be inspected via the journalctl utility. This article looks at the utility. We also got an article about logging on Linux, which covers how to configure systemd-journald.

journalctl doesn’t replace all log files. Some services, such as Apache, have their own logs. Also, rsyslog is still a thing on servers that run systemd. It works differently – rsyslog nowadays gets its log entries from systemd-journald. However, the end result is largely the same. You still got log files such as /var/log/messages and /var/log/secure.

Filtering options

As said, journal entries are stored in a binary format. Each entry has a large number of fields that can be displayed in several output formats. By default, log entries look much like entries you find in traditional syslog log files, such as /var/log/messages. Here is an example of a journalctl entry:

Sep 24 00:59:37 server.example.net sshd[17255]: Invalid user admin from 59.148.43.97 port 54128

And here is the same entry displayed in JSON format:

{
        "__CURSOR" : "s=a285c16fef31c7;i=b82;b=8d04cfd2dd066c;m=c5e8f;t=59341f0;x=11a8f0",
        "__REALTIME_TIMESTAMP" : "1569283177914864",
        "__MONOTONIC_TIMESTAMP" : "53113740943",
        "_BOOT_ID" : "8d04cfd2dd9744b888u8d3f9cb52066c",
        "PRIORITY" : "6",
        "_UID" : "0",
        "_GID" : "0",
        "_SYSTEMD_SLICE" : "system.slice",
        "_MACHINE_ID" : "21a223f3763983e62f89dc6a5b90c0d1",
        "_HOSTNAME" : "server.example.net",
        "_CAP_EFFECTIVE" : "1fffffffff",
        "_TRANSPORT" : "syslog",
        "SYSLOG_FACILITY" : "10",
        "SYSLOG_IDENTIFIER" : "sshd",
        "_COMM" : "sshd",
        "_EXE" : "/usr/sbin/sshd",
        "_SYSTEMD_CGROUP" : "/system.slice/sshd.service",
        "_SYSTEMD_UNIT" : "sshd.service",
        "_SELINUX_CONTEXT" : "system_u:system_r:sshd_t:s0-s0:c0.c1023",
        "_CMDLINE" : "sshd: unknown [priv]",
        "SYSLOG_PID" : "17255",
        "MESSAGE" : "Invalid user admin from 59.148.43.97 port 54128",
        "_PID" : "17255",
        "_SOURCE_REALTIME_TIMESTAMP" : "1569283177913968"
}

As you can see, the entry contains quite a few fields, including a boot ID (_BOOT_ID) and priority (PRIORITY). You can use these fields to filter the data, which is what we will look at now.

Filtering by time

The --since and --until options let you specify a timestamp in the format YYYY-MM-DD HH:MM:SS. You don’t have to include the time, and if you do include the time you may leave out the seconds (i.e. you can use HH:MM):

# journalctl --since="2019-09-26" --until="2019-09-26 11:00"

You can also use strings such as “yesterday”, “today” and “10m ago”:

# journalctl --since "10m ago"

Filtering by unit

The -u (--unit option is used to show entries related to a systemd unit. So, to return log entries for sshd you can use the following:

# journalctl -u sshd.service
-- Logs begin at Mon 2021-02-08 14:59:39 GMT, end at Mon 2021-03-29 22:20:00 BST. --
Feb 08 20:26:43 centos8 systemd[1]: Stopping OpenSSH server daemon...
Feb 08 20:26:46 centos8 sshd[1053]: Received signal 15; terminating.
Feb 08 20:26:49 centos8 systemd[1]: sshd.service: Succeeded.
Feb 08 20:26:49 centos8 systemd[1]: Stopped OpenSSH server daemon.
...

Alternatively, you can also use the -t (--identifier) option to filter by the SYSLOG_IDENTIFIER field. You can use this to retrieve entries that are not related to a systemd unit. For instance, you can filter out kernel messages or entries about the use of the su and sudo commands:

# journalctl -t su
-- Logs begin at Mon 2021-02-08 14:59:39 GMT, end at Mon 2021-03-29 22:30:16 BST. --
...
Mar 29 19:30:39 centos8 su[2339]: (to root) example on pts/0
Mar 29 19:30:39 centos8 su[2339]: pam_unix(su:session): session opened for user root by example(uid=1000)

Filtering by priority

The JSON output I showed includes a priority field. The priority number indicates the importance of the message. A priority of 0 means we got an emergency, while 7 is a humble debug message. In other words, the lower the number, the higher the priority:

• 0: emerg
• 1: alert
• 2: crit
• 3: err
• 4: warning
• 5: notice
• 6: info
• 7: debug

The -p option lets you filter messages by priority. If, say, you want to see all errors you can use either of these commands:

# journalctl -p 3
# journalctl -p err

Either command prints entries with a priority of 3 or lower (i.e. 3, 2, 1 and 0). You can also specify a range. For instance, to limit the output to levels 2 and 3 you can use journalctl -p 2..3.

Filtering by process

The output of journalctl includes the name of the service followed by the process ID (in square brackets). You can use the _PID option to get all entries for a specific process ID:

# journalctl _PID=17255
-- Logs begin at Mon 2021-02-08 14:59:39 GMT, end at Mon 2021-03-29 22:30:16 BST. --
Mar 24 00:59:37 server.example.net sshd[17255]: Invalid user admin from 59.148.43.97 port 54128
Mar 24 00:59:37 server.example.net sshd[17255]: input_userauth_request: invalid user admin [preauth]
Mar 24 00:59:39 server.example.net sshd[17255]: error: maximum authentication attempts exceeded for invalid user admin from 59.148.43.97 port 54128 ssh2 [preauth]
Mar 24 00:59:39 server.example.net sshd[17255]: Disconnecting: Too many authentication failures [preauth]

Tailing logs

You can monitor logs as entries are being written using the -f (--follow) option. This works exactly like tail -f. Of course, you can filter a specific unit at the same time. For example, here I’m following entries for Dovecot:

# journalctl -f -u dovecot.service

And now that I’ve mentioned tail I should also highlight the -n option. As you might have guessed, this option works exactly like tail -n – it displays the last x entries. For instance, to view the last three entries for the Dovecot service you can use:

# journalctl -u dovecot.service -n 3

Manipulating output

Although the filtering options covered so far are quite flexible they don’t necessarily return the data you want. For instance, when you are looking into SSH login attempts you might want to filter entries based on the IP address shown in the MESSAGE field. On Centos 7 and CloudLinux 7 servers you can do that by piping the output of journalctl to grep:

# journalctl -u sshd.service | grep 59.148.43.97
Mar 24 00:59:37 server.example.net sshd[17255]: Invalid user admin from 59.148.43.97 port 54128
Mar 24 00:59:39 server.example.net sshd[17255]: error: maximum authentication attempts exceeded for invalid user admin from 59.148.43.97 port 54128 ssh2 [preauth]

RHEL8-based systems ship with a version of journalctl that includes the -g (--grep) option, which lets you grep the message field. That means that you can instead use this command:

# journalctl -u sshd.service -g 59.148.43.97

More information

This article aimed to give a brief overview of journalctl. There are many topics I didn’t cover. If you want to learn more, the official documentation can be found at freedesktop.org.