1. Debugging the Kernel With Drgn and Corelens
  2. corelens Command Reference

9 corelens Command Reference

This table provides information about the corelens command.

Action Command Description

Review a summary of the system state for a running kernel or vmcore crash dump.

sudo corelens path/to/dumpfile

Provides information for debugging the running live kernel or generated crash dump.

Run a corelens module to analyze the contents of a running kernel or vmcore crash dump, and then review the results.

sudo corelens path/to/dumpfile -M module

Provides information for debugging a running kernel or generated crash dump filtered by module.

Review a list of modules that can be specified.

 corelens -L

Provides a listing of module filters for use with the -M option.

Create a report based on the output from the corelens command.

sudo corelens path/to/dumpfile -a -o report

Generates a diagnostic report containing all the debugging information captured by the corelens command.

Review further options provided with the corelens command.

corelens -h

Provides a listing of command line options for the drgn command.

For example, to debug /proc/kcore for a live kernel, run the following command: 

sudo corelens /proc/kcore

To perform the same operation on a vmcore crash dump file: 

sudo corelens /var/crash/127.0.0.1-2024-06-28-09:33:07/vmcore

Selecting Modules for corelens Command Output

Use the -M option to filter the output from corelens commands.

The corelens command can also filter output based on the parts of the system that require diagnosis by using the -M option. For example, to reproduce the full output for a live kernel, activate the sys module: 

sudo corelens /proc/kcore -M sys

Similarly, to display a list of I/O requests that are still in progress, activate the inflight-io module: 

sudo corelens /proc/kcore -M inflight-io

More than one module can be specified by reusing the -M option for each module. For example, to reproduce the full output for a live kernel and all the mounted directories that are now present, activate the sys and mounts modules: 

sudo corelens /proc/kcore -M sys -M mounts

Example output follows:

warning: Running corelens against a live system.
         Data may be inconsistent, or corelens may crash.

 ====== MODULE sys ======
MODE        : Live kernel
DATE        : Fri Jul 12 18:21:34 2024
NODENAME    : oracle-example-ol9
RELEASE     : 5.15.0-206.153.7.el9uek.x86_64
VERSION     : #2 SMP Thu May 9 15:59:05 PDT 2024
MACHINE     : x86_64
UPTIME      : 1 day, 19:47:22
LOAD AVERAGE: 0.08  ,  0.02 ,  0.01
JIFFIES     : 4452309417
MEMORY      : 7.49 GiB
TASKS       : 251 R:1 D:0 S:167 I:83
PLATFORM    : QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.6.6 08/22/2023 X86_HYPER_KVM
CPU VENDOR: AuthenticAMD
MODEL NAME: AMD EPYC 7763 64-Core Processor
CPU FAMILY: 25
CPUS      : 2
CPUS NUMA0: 0-1
MICROCODE : 0x1000065
CSTATES   : 9

====== MODULE mounts ======
DEVNAME                     TYPE        DIRNAME
-------                     ------      -------
none                        rootfs      /
proc                        proc        /proc
sysfs                       sysfs       /sys
devtmpfs                    devtmpfs    /dev
securityfs                  securityfs  /sys/kernel/security
tmpfs                       tmpfs       /dev/shm
devpts                      devpts      /dev/pts
tmpfs                       tmpfs       /run
cgroup2                     cgroup2     /sys/fs/cgroup
pstore                      pstore      /sys/fs/pstore
efivarfs                    efivarfs    /sys/firmware/efi/efivars
bpf                         bpf         /sys/fs/bpf
configfs                    configfs    /sys/kernel/config
/dev/mapper/ocivolume-root  xfs         /
rpc_pipefs                  rpc_pipefs  /var/lib/nfs/rpc_pipefs
selinuxfs                   selinuxfs   /sys/fs/selinux
systemd-1                   autofs      /proc/sys/fs/binfmt_misc
debugfs                     debugfs     /sys/kernel/debug
hugetlbfs                   hugetlbfs   /dev/hugepages
mqueue                      mqueue      /dev/mqueue
tracefs                     tracefs     /sys/kernel/tracing
none                        ramfs       /run/credentials/systemd-sysctl.service
none                        ramfs       /run/credentials/systemd-tmpfiles-setup-dev.service
fusectl                     fusectl     /sys/fs/fuse/connections
/dev/mapper/ocivolume-oled  xfs         /var/oled
/dev/sdb2                   xfs         /boot
/dev/sdb1                   vfat        /boot/efi
none                        ramfs       /run/credentials/systemd-tmpfiles-setup.service
tmpfs                       tmpfs       /run/user/983
tmpfs                       tmpfs       /run/user/1000

To see a full list of all the modules that can be specified, run the corelens command with the -L option: 

corelens -L

For more information about what each corelens module does, use the -h option after specifying each of them with the -M option: 

corelens -M module -h

For example, to learn more about the dentrycache module that outputs the kernel directory entry cache, use the following command: 

corelens -M dentrycache -h

The following output might be displayed:

usage: dentrycache [-h] [--limit LIMIT] [--negative] [--detailed]

List dentries from the dentry hash table

optional arguments:
  -h, --help show this help message and exit
  --limit LIMIT, -l LIMIT
                        list at most <number> dentries, 50 by default
  --negative, -n list negative dentries only, disabled by default
  --detailed, -d include inode, super, file type, refcount

Generating Reports With corelens

Use the provided corelens command options to generate reports for later review.

To generate a report from the corelens command, use the -o option and specify the output directory for that report. For example, to generate a report for the live kernel and output that report into a folder called report in the current working directory, use the following command: 

sudo corelens /proc/kcore -a -o report

If you don't explicitly specify modules by using the -M option, use the -a option to generate a report using standard modules, or the -A option to generate the report using detailed modules.

Note:

If you generate a report using every module, the final report might contain warnings that some modules couldn't be run. This is expected behavior, because some corelens modules require a core dump or can only function when specific kernel modules are loaded.

Diagnostic information is stored in a plain-text file for each module that was active when the corelens command was run. For example, to review the mounted directories that were output from the mounts module, view the contents of the report/mounts file: 

cat report/mounts

Example output follows:

DEVNAME                     TYPE        DIRNAME
-------                     ------      -------
none                        rootfs      /
proc                        proc        /proc
sysfs                       sysfs       /sys
devtmpfs                    devtmpfs    /dev
securityfs                  securityfs  /sys/kernel/security
tmpfs                       tmpfs       /dev/shm
devpts                      devpts      /dev/pts
tmpfs                       tmpfs       /run
cgroup2                     cgroup2     /sys/fs/cgroup
pstore                      pstore      /sys/fs/pstore
efivarfs                    efivarfs    /sys/firmware/efi/efivars
bpf                         bpf         /sys/fs/bpf
configfs                    configfs    /sys/kernel/config
/dev/mapper/ocivolume-root  xfs         /
rpc_pipefs                  rpc_pipefs  /var/lib/nfs/rpc_pipefs
selinuxfs                   selinuxfs   /sys/fs/selinux
systemd-1                   autofs      /proc/sys/fs/binfmt_misc
hugetlbfs                   hugetlbfs   /dev/hugepages
mqueue                      mqueue      /dev/mqueue
debugfs                     debugfs     /sys/kernel/debug
tracefs                     tracefs     /sys/kernel/tracing
fusectl                     fusectl     /sys/fs/fuse/connections
none                        ramfs       /run/credentials/systemd-sysctl.service
none                        ramfs       /run/credentials/systemd-tmpfiles-setup-dev.service
/dev/mapper/ocivolume-oled  xfs         /var/oled
/dev/sda2                   xfs         /boot
/dev/sda1                   vfat        /boot/efi
none                        ramfs       /run/credentials/systemd-tmpfiles-setup.service
tmpfs                       tmpfs       /run/user/0
tmpfs                       tmpfs       /run/user/982
tmpfs                       tmpfs       /run/user/1000