HOWTO: install kernel debuginfo packages on SUSE Linux Enterprise Server 11

http://blog.melski.net/2010/11/30/howto-install-kernel-debuginfo-packages-on-suse-linux-enterprise-server-11/

I needed to debug a kernel crash on SUSE Linux Enterprise Server 11 today. If you’re not familiar with debugging Linux kernel crashes, you need the kernel debug symbols in order to analyze the crash dump. These are typically not part of the kernel image
itself, but instead are bundled into a kernel debuginfo package corresponding to the kernel that produced the crash dump.

Although I’ve done this on RedHat Enterprise Linux many times, I had never debugged a kernel crash on SUSE before, so I was not familiar with the process for acquiring the debuginfo packages with that distro. I couldn’t find any single set of instructions
explaining how to get the packages, and although it wasn’t hard, I figured I’d try to save somebody else a little time by writing down the steps I followed.

SUSE uses a package manager called
ZYpp. I used zypper, the command-line interface to ZYpp, to install the packages.

Step 1: enable the debuginfo repositories

Before zypper can install the debuginfo packages, it must be able to find them. The packages reside in specialized debuginfo repositories, which are normally not enabled,
although the system is aware of them. Use zypper repos to get a list of the repositories:

lin4-ea6:~ # zypper repos
# | Alias                                                    | Name                                                   | Enabled | Refresh
--+----------------------------------------------------------+--------------------------------------------------------+---------+--------
1 | SUSE-Linux-Enterprise-Server-11 11-0                     | SUSE-Linux-Enterprise-Server-11 11-0                   | Yes     | No
2 | SUSE-Linux-Enterprise-Software-Development-Kit-11_11-0   | SUSE-Linux-Enterprise-Software-Development-Kit-11 11-0 | Yes     | No
3 | SUSE-Linux-Enterprise-Software-Development-Kit-11_11-0_1 | SUSE-Linux-Enterprise-Software-Development-Kit-11 11-0 | Yes     | No
4 | nu_novell_com:SLE11-Debuginfo-Pool                       | SLE11-Debuginfo-Pool                                   | No      | Yes
5 | nu_novell_com:SLE11-Debuginfo-Updates                    | SLE11-Debuginfo-Updates                                | No      | Yes
6 | nu_novell_com:SLES11-Extras                              | SLES11-Extras                                          | No      | Yes
7 | nu_novell_com:SLES11-Pool                                | SLES11-Pool                                            | No      | Yes
8 | nu_novell_com:SLES11-Updates                             | SLES11-Updates                                         | Yes     | Yes

You want the two Debuginfo repos. To enable them, use
zypper modifyrepo with the
alias of the repo:

lin4-ea6:~ # zypper modifyrepo --enable nu_novell_com:SLE11-Debuginfo-Pool
Repository 'nu_novell_com:SLE11-Debuginfo-Pool' has been sucessfully enabled.
lin4-ea6:~ # zypper modifyrepo --enable nu_novell_com:SLE11-Debuginfo-Updates
Repository 'nu_novell_com:SLE11-Debuginfo-Updates' has been sucessfully enabled.

Step 2: find the debuginfo package for your crash

It’s critical to get the debuginfo package that matches the kernel that created your crash dump. It’s easy to determine the version you need: check the README.txt alongside the vmcore in the crash directory:

lin4-ea6:~ # cat /var/crash/2010-11-30-00:43/README.txt
Kernel crashdump
----------------

Crash time     : 2010-11-30 00:43 (+0000)
Kernel version : 2.6.27.45-0.1-pae
Host           : SLES-11-32
Dump level     : 0
Dump format    : compressed

In this case, I need the debuginfo for the pae variant of kernel version
2.6.27.45-0.1. Now, search the package repository for kernel debuginfo packages with
zypper search:

lin4-ea6:~ # zypper search -s kernel-*-debuginfo*

Loading repository data...
Reading installed packages...

S | Name                     | Type    | Version          | Arch | Repository
--+--------------------------+---------+------------------+------+------------------------
| kernel-default-debuginfo | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.21-0.1.2  | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool
| kernel-pae-debuginfo     | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.21-0.1.2  | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo     | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool
| kernel-source-debuginfo  | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.21-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo  | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool
| kernel-vmi-debuginfo     | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.21-0.1.2  | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo     | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool
| kernel-xen-debuginfo     | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.21-0.1.2  | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo     | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool

You can see there are several versions of each variant available. One tricky thing is that there isn’t an
exact match for the kernel version I need. I’m looking for
2.6.27.45-0.1; the closest thing to it is
2.6.27.45-0.1.1. This seems to be nothing more than a minor inconsistency in labeling: the
2.6.27.45-0.1.1 is the correct package.

Step 3: install the kernel debuginfo package

Having identified the package, you are ready to install it with
zypper install:

lin4-ea6:~ # zypper install kernel-pae-debuginfo=2.6.27.45-0.1.1
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW package is going to be installed:
kernel-pae-debuginfo

The following package is not supported by its vendor:
kernel-pae-debuginfo

Overall download size: 153.1 M. After the operation, additional 673.8 M will be used.
Continue? [YES/no]:

At this prompt, you should type YES and hit Enter.
zypper will download the package (which may take a while depending on the speed of your internet connection), showing a progress bar as it does:

Retrieving package kernel-pae-debuginfo-2.6.27.45-0.1.1.i586 (1/1), 153.1 M (673.8 M unpacked)
Retrieving: kernel-pae-debuginfo-2.6.27.45-0.1.1.i586.rpm [90% (1.1 M/s)]

Then zypper will install the package, again with a progress bar:

Retrieving package kernel-pae-debuginfo-2.6.27.45-0.1.1.i586 (1/1), 153.1 M (673.8 M unpacked)
Retrieving: kernel-pae-debuginfo-2.6.27.45-0.1.1.i586.rpm [done (244.7 K/s)]
Installing: kernel-pae-debuginfo-2.6.27.45-0.1.1 [84%]

When installation is complete, there is no more notification than that the progress bar reads “done”:

Retrieving package kernel-pae-debuginfo-2.6.27.45-0.1.1.i586 (1/1), 153.1 M (673.8 M unpacked)
Retrieving: kernel-pae-debuginfo-2.6.27.45-0.1.1.i586.rpm [done (244.7 K/s)]
Installing: kernel-pae-debuginfo-2.6.27.45-0.1.1 [done]

Step 4: symlink the debuginfo into the crash directory

The debuginfo package, for some reason, does not install the debuginfo files to a location known to the
crash utility, so the final step is to make the debuginfo file available to
crash before we invoke it. We do so by creating a symlink to the debuginfo file alongside the kernel image in the crash directory:

lin4-ea6:~ # ln -s /usr/lib/debug/boot/vmlinux-2.6.27.45-0.1-pae.debug /var/crash/2010-11-30-00:43

Decompress the kernel image and you’re in business:

lin4-ea6:~ # gzip -d /var/crash/2010-11-30-00:43/vmlinux-2.6.27.45-0.1-pae.gz
lin4-ea6:~ # cd /var/crash/2010-11-30-00:43
lin4-ea6:/var/crash/2010-11-30-00:43 # crash vmlinux-2.6.27.45-0.1-pae vmcore

Share this:

Reddit
Twitter
Google