How to Map Kernel Stack Trace Offsets to Exact Source Code Lines in Linux


1 views

When debugging kernel issues, stack traces provide crucial information, but the real challenge lies in translating those hexadecimal offsets into actual source code lines. Let's examine this sample trace:

kernel: [<ffffffff80009a14>] __link_path_walk+0x173/0xfb9
kernel: [<ffffffff8002cbec>] mntput_no_expire+0x19/0x89

The key components we need to interpret are:

  1. The function name (e.g., __link_path_walk)
  2. The offset from function start (0x173 in the first line)
  3. The function size (0xfb9 in the first line)

You'll need these tools for accurate mapping:

# Install debugging tools
sudo apt-get install linux-image-$(uname -r)-dbgsym
sudo apt-get install dwarfdump
sudo apt-get install addr2line

1. Locate the Exact Kernel Source

First ensure you have matching source code for your running kernel:

uname -r
# Example output: 5.4.0-42-generic

Download the exact source either from your distribution's repositories or kernel.org.

2. Extract Debug Information

For Ubuntu/Debian systems with debug symbols installed:

# Find the debug file
find /usr/lib/debug -name "*.ko" -o -name "vmlinux" | grep $(uname -r)

# Example output:
/usr/lib/debug/boot/vmlinux-5.4.0-42-generic

3. Use addr2line for Precise Mapping

This converts addresses to file names and line numbers:

addr2line -e /usr/lib/debug/boot/vmlinux-5.4.0-42-generic -f -i 0xffffffff80009a14+0x173

# Sample output:
__link_path_walk
/usr/src/linux/fs/namei.c:1234

Automating Multiple Stack Frames

Create a script to process entire stack traces:

#!/bin/bash
DEBUG_FILE="/usr/lib/debug/boot/vmlinux-$(uname -r)"
STACK_TRACE="trace.txt" # Your stack trace file

while read -r line; do
    if [[ $line =~ \[\<(.*)\>\]\ (.*)\+(0x[0-9a-f]+) ]]; then
        addr="${BASH_REMATCH[1]}"
        func="${BASH_REMATCH[2]}"
        offset="${BASH_REMATCH[3]}"
        
        echo "Processing: $func + $offset"
        addr2line -e $DEBUG_FILE -f -i $(printf "%#x" $((16#$addr + 16#$offset)))
        echo "----------------"
    fi
done < "$STACK_TRACE"

Working Without Debug Symbols

If debug symbols aren't available, you can use the kernel map file:

grep __link_path_walk /boot/System.map-$(uname -r)

# Then calculate:
BASE_ADDR=0x$(grep __link_path_walk /boot/System.map-$(uname -r) | cut -d' ' -f1)
OFFSET=0x173
TARGET_ADDR=$((BASE_ADDR + OFFSET))

objdump -d --start-address=$TARGET_ADDR --stop-address=$((TARGET_ADDR+20)) \
    /usr/lib/debug/boot/vmlinux-$(uname -r) | less

Let's walk through a real-world example with our sample trace:

# For __link_path_walk+0x173
BASE=0x$(grep __link_path_walk /boot/System.map-5.4.0-42-generic | cut -d' ' -f1)
OFFSET=0x173
TARGET=$(printf "%#x" $((BASE + OFFSET)))

addr2line -e /usr/lib/debug/boot/vmlinux-5.4.0-42-generic $TARGET

# Output would point to the exact line in namei.c where the issue occurred
  • Mismatched kernel versions between running system and debug symbols
  • Optimized-out functions in production kernels
  • Inlined functions that don't appear in stack traces
  • KASLR (Kernel Address Space Layout Randomization) changing addresses

For production systems where installing debug symbols isn't possible:

# Use crash utility
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /proc/kcore

# Then in crash shell:
extend -s /usr/src/linux-source-$(uname -r | cut -d- -f1)
list -o __link_path_walk+0x173

When debugging kernel issues, stack traces provide function names and instruction pointers in this format:

[<ffffffff80009a14>] __link_path_walk+0x173/0xfb9

The key elements are:

  • Function name (__link_path_walk)
  • Offset from function start (0x173)
  • Total function size (0xfb9)

To map these to source lines, you'll need:

sudo apt-get install dwarfdump
sudo apt-get install linux-image-$(uname -r)-dbgsym

Essential files:

  • Original kernel source matching your running kernel
  • Debug symbols package (dbgsym)
  • vmlinux with debug information

For our example from namei.c:

1. Locate the function in DWARF info

dwarfdump -l vmlinux | grep __link_path_walk
0xffffffff80009a14: /usr/src/linux/fs/namei.c line 789

2. Calculate the exact instruction location

addr2line -e vmlinux -i ffffffff80009a14+0x173
/usr/src/linux/fs/namei.c:842

3. Verify with disassembly

objdump -dS --start-address=0xffffffff80009a14 \
--stop-address=$(printf "%x" $((0xffffffff80009a14+0x173))) \
vmlinux | less

Let's examine our sample trace in detail:

kernel:  [<ffffffff80009a14>] __link_path_walk+0x173/0xfb9

Using GDB with vmlinux:

gdb vmlinux
(gdb) list *(__link_path_walk+0x173)
789     static int __link_path_walk(const char *name, struct nameidata *nd)
790     {
...
842             if (this.name[0] == '.') switch (this.len) {

Create a helper script trace2source.sh:

#!/bin/bash
if [ $# -ne 1 ]; then
    echo "Usage: $0 <offset>"
    exit 1
fi

VMLINUX=/usr/lib/debug/boot/vmlinux-$(uname -r)
SYMBOL=$(echo $1 | cut -d+ -f1)
OFFSET=$(echo $1 | cut -d+ -f2 | cut -d/ -f1)

addr2line -e $VMLINUX -i $(nm $VMLINUX | grep " $SYMBOL$" | cut -d' ' -f1)+$OFFSET

When things don't match:

  • Verify kernel version matches debug symbols exactly
  • Check CONFIG_DEBUG_INFO was enabled during build
  • Confirm no address space randomization (disable KASLR temporarily)
  • Try alternative tools like eu-addr2line from elfutils

For optimized code where line numbers jump:

gcc -g -O2 -fno-inline -fno-omit-frame-pointer ...

For inline functions or tail calls:

dwarfdump -k vmlinux --debug-info | less

To view full context around the problem location:

addr2line -e vmlinux -f -i ffffffff80009a14+0x173 -C