Scribd
Upload
38 views

Crash Dump Analysis: System Debugging in Linux

This document discusses various tools for debugging the Linux kernel, including user-space debugging tools like gdb, historically less supported kernel debugging tools, and the LKCD toolchain for generating crash dumps. It also covers standard debugging means using SysRq keys, kprobes for kernel instrumentation, kexec/kdump for rebooting without a physical reboot to analyze a crash dump, User-Mode Linux for running the kernel as a user process, the KDB kernel debugger, and KGDB which integrates remote kernel debugging into the Linux kernel since version 2.6.26.

Uploaded by

achilles7
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
38 views

Crash Dump Analysis: System Debugging in Linux

This document discusses various tools for debugging the Linux kernel, including user-space debugging tools like gdb, historically less supported kernel debugging tools, and the LKCD toolchain for generating crash dumps. It also covers standard debugging means using SysRq keys, kprobes for kernel instrumentation, kexec/kdump for rebooting without a physical reboot to analyze a crash dump, User-Mode Linux for running the kernel as a user process, the KDB kernel debugger, and KGDB which integrates remote kernel debugging into the Linux kernel since version 2.6.26.

Uploaded by

achilles7
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 12

Crash Dump Analysis

System Debugging in Linux


Jakub Jerm
Martin Dck
Crash Dump Analysis MFF UK System Debugging in Linux 2
Overview

User space debugging tools

Mostly already mentioned

gdb and friends

Kernel debugging tools

Historically less officialy supported tools


(in vanilla kernel)

Different developer's attitude

printk() as the ultimate debugging tool


Crash Dump Analysis MFF UK System Debugging in Linux 3
Linux Kernel Crash Dump

LKCD toolchain (http://lkcd.sourceforge.net/)

Kernel patches for simple crash dump generation


on kernel panic

Supported only on SCSI devices

In-kernel debugger and post-mortem debugger

Similar features as mdb -K

Heuristical analysis

Active development during 2.5.x timeframe

Last supported release from 2006


Crash Dump Analysis MFF UK System Debugging in Linux 4
Standard debugging means

Magic SysRq hot keys

For dealing with hangs and security issues

Operator's intervention to the running system

Can be enabled/disabled by
/proc/sys/kernel/sysrq

Alt + SysRq + 0 .. 9 set console logging level

Alt + SysRq + C activate kexec and generate a crash dump

Alt + SysRq + B immediate reboot

Alt + SysRq + O immediate shutdown

Alt + SysRq + S sync all mounted filesystems

Alt + SysRq + U remount all filesystems read-only


Crash Dump Analysis MFF UK System Debugging in Linux 5
Standard debugging means (2)

Alt + SysRq + P dump registers to console

Alt + SysRq + T dump process information to console

Alt + SysRq + L dump stack traces of running threads

Alt + SysRq + M dump memory statistics to console

Alt + SysRq + D dump locked locks to console

Alt + SysRq + K kill all processes on the current console

Alt + SysRq + E terminate all processes except init

Alt + SysRq + I kill all processes except init

Alt + SysRq + F execute the OOM killer

Alt + SysRq + N reset nice level of all real-time processes

Alt + SysRq + X switch off raw keyboard mode


Crash Dump Analysis MFF UK System Debugging in Linux 6
Standard debugging means (3)

Activate from command line by writing into


/proc/sysrq-trigger

Activate over network by a special sysrqd server

Raising Elephants Is So Utterly Boring


Reboot Even If System Utterly Broken

Raw keyboard

Send SIGTERM to all processes

Send SIGKILL to all processes

Sync data to disk

Remount all filesystems read-only

Reboot
Crash Dump Analysis MFF UK System Debugging in Linux 7
Standard debugging means (4)

kprobes

Basic kernel instrumentation

An execution of any kernel function can trigger a callback


function

kexec (kdump)

Booting a new (Linux) kernel without physical


reboot

Physical memory can be (more or less) retained and


analyzed by the new kernel (as a crash dump)
Crash Dump Analysis MFF UK System Debugging in Linux 8
User-Mode Linux

UML

Special pseudo-hardware architecture

Otherwise compatible with the target architecture

Running Linux kernel as a user space process

Originally a virtualization effort

Great for debugging and kernel development

A plain standard gdb can be used to attach to the running kernel

Guest threads are threads of the UML process

Slightly more complicated to follow processes


Crash Dump Analysis MFF UK System Debugging in Linux 9
KDB

Linux kernel debugger

http://oss.sgi.com/projects/kdb

A set of patches to the vanilla kernel


(very unprobably to be integrated soon)

For local debugging

Possibilities and features very similar to mdb -K

Activation in /proc/sys/kernel/kdb

Break-in by the Pause key (Ctrl+A on serial console)

md memory display

mm memory modify
Crash Dump Analysis MFF UK System Debugging in Linux 10
KDB (2)

rd register dump

rm register modify

bt stack trace

bp set breakpoint
...
Crash Dump Analysis MFF UK System Debugging in Linux 11
KGDB

Kernel GDB

http://kgdb.sourceforge.net/

Since 2.6.26 integrated in the vanilla kernel (finally!)

Client/server debugger for remote kernel debugging

Over serial line or ethernet

Client: Plain gdb

Assembly and source line debugging

Symbol information from vmlinux binary on the client

Activation

After loading the I/O module and connecting to the server

Just after boot when kgdbwait kernel option is used


Crash Dump Analysis MFF UK System Debugging in Linux 12
KGDB (2)

Features

Memory and register analysis on kernel oops or panic

Limited single-stepping of the running kernel


% gdb ./vmlinux
(gdb) set remotebaud 11!""
(gdb) target remote /dev/tty#"
% gdb ./vmlinux
(gdb) target remote
udp$1%!.1&'.1.1.$&(()

You might also like