How to Map Kernel Stack Trace Offsets to Exact Source Code Lines in Linux

When debugging kernel issues, stack traces provide crucial information, but the real challenge lies in translating those hexadecimal offsets into actual source code lines. Let's examine this sample trace:

kernel: [<ffffffff80009a14>] __link_path_walk+0x173/0xfb9
kernel: [<ffffffff8002cbec>] mntput_no_expire+0x19/0x89

The key components we need to interpret are:

The function name (e.g., __link_path_walk)
The offset from function start (0x173 in the first line)
The function size (0xfb9 in the first line)

You'll need these tools for accurate mapping:

# Install debugging tools
sudo apt-get install linux-image-$(uname -r)-dbgsym
sudo apt-get install dwarfdump
sudo apt-get install addr2line

1. Locate the Exact Kernel Source

First ensure you have matching source code for your running kernel:

uname -r
# Example output: 5.4.0-42-generic

Download the exact source either from your distribution's repositories or kernel.org.

2. Extract Debug Information

For Ubuntu/Debian systems with debug symbols installed:

# Find the debug file
find /usr/lib/debug -name "*.ko" -o -name "vmlinux" | grep $(uname -r)

# Example output:
/usr/lib/debug/boot/vmlinux-5.4.0-42-generic

3. Use addr2line for Precise Mapping

This converts addresses to file names and line numbers:

addr2line -e /usr/lib/debug/boot/vmlinux-5.4.0-42-generic -f -i 0xffffffff80009a14+0x173

# Sample output:
__link_path_walk
/usr/src/linux/fs/namei.c:1234

Automating Multiple Stack Frames

Create a script to process entire stack traces:

#!/bin/bash
DEBUG_FILE="/usr/lib/debug/boot/vmlinux-$(uname -r)"
STACK_TRACE="trace.txt" # Your stack trace file

while read -r line; do
    if [[ $line =~ \[\<(.*)\>\]\ (.*)\+(0x[0-9a-f]+) ]]; then
        addr="${BASH_REMATCH[1]}"
        func="${BASH_REMATCH[2]}"
        offset="${BASH_REMATCH[3]}"
        
        echo "Processing: $func + $offset"
        addr2line -e $DEBUG_FILE -f -i $(printf "%#x" $((16#$addr + 16#$offset)))
        echo "----------------"
    fi
done < "$STACK_TRACE"

Working Without Debug Symbols

If debug symbols aren't available, you can use the kernel map file:

grep __link_path_walk /boot/System.map-$(uname -r)

# Then calculate:
BASE_ADDR=0x$(grep __link_path_walk /boot/System.map-$(uname -r) | cut -d' ' -f1)
OFFSET=0x173
TARGET_ADDR=$((BASE_ADDR + OFFSET))

objdump -d --start-address=$TARGET_ADDR --stop-address=$((TARGET_ADDR+20)) \
    /usr/lib/debug/boot/vmlinux-$(uname -r) | less

Let's walk through a real-world example with our sample trace:

# For __link_path_walk+0x173
BASE=0x$(grep __link_path_walk /boot/System.map-5.4.0-42-generic | cut -d' ' -f1)
OFFSET=0x173
TARGET=$(printf "%#x" $((BASE + OFFSET)))

addr2line -e /usr/lib/debug/boot/vmlinux-5.4.0-42-generic $TARGET

# Output would point to the exact line in namei.c where the issue occurred

Mismatched kernel versions between running system and debug symbols
Optimized-out functions in production kernels
Inlined functions that don't appear in stack traces
KASLR (Kernel Address Space Layout Randomization) changing addresses

For production systems where installing debug symbols isn't possible:

# Use crash utility
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /proc/kcore

# Then in crash shell:
extend -s /usr/src/linux-source-$(uname -r | cut -d- -f1)
list -o __link_path_walk+0x173

When debugging kernel issues, stack traces provide function names and instruction pointers in this format:

[<ffffffff80009a14>] __link_path_walk+0x173/0xfb9

The key elements are:

Function name (__link_path_walk)
Offset from function start (0x173)
Total function size (0xfb9)

To map these to source lines, you'll need:

sudo apt-get install dwarfdump
sudo apt-get install linux-image-$(uname -r)-dbgsym

Essential files:

Original kernel source matching your running kernel
Debug symbols package (dbgsym)
vmlinux with debug information

For our example from namei.c:

1. Locate the function in DWARF info

dwarfdump -l vmlinux | grep __link_path_walk
0xffffffff80009a14: /usr/src/linux/fs/namei.c line 789

2. Calculate the exact instruction location

addr2line -e vmlinux -i ffffffff80009a14+0x173
/usr/src/linux/fs/namei.c:842

3. Verify with disassembly

objdump -dS --start-address=0xffffffff80009a14 \
--stop-address=$(printf "%x" $((0xffffffff80009a14+0x173))) \
vmlinux | less

Let's examine our sample trace in detail:

kernel:  [<ffffffff80009a14>] __link_path_walk+0x173/0xfb9

Using GDB with vmlinux:

gdb vmlinux
(gdb) list *(__link_path_walk+0x173)
789     static int __link_path_walk(const char *name, struct nameidata *nd)
790     {
...
842             if (this.name[0] == '.') switch (this.len) {

Create a helper script trace2source.sh:

#!/bin/bash
if [ $# -ne 1 ]; then
    echo "Usage: $0 <offset>"
    exit 1
fi

VMLINUX=/usr/lib/debug/boot/vmlinux-$(uname -r)
SYMBOL=$(echo $1 | cut -d+ -f1)
OFFSET=$(echo $1 | cut -d+ -f2 | cut -d/ -f1)

addr2line -e $VMLINUX -i $(nm $VMLINUX | grep " $SYMBOL$" | cut -d' ' -f1)+$OFFSET

When things don't match:

Verify kernel version matches debug symbols exactly
Check CONFIG_DEBUG_INFO was enabled during build
Confirm no address space randomization (disable KASLR temporarily)
Try alternative tools like eu-addr2line from elfutils

For optimized code where line numbers jump:

gcc -g -O2 -fno-inline -fno-omit-frame-pointer ...

For inline functions or tail calls:

dwarfdump -k vmlinux --debug-info | less

To view full context around the problem location:

addr2line -e vmlinux -f -i ffffffff80009a14+0x173 -C

ServerDevWorker