转载_Linux内核OOPS调试

以前在写驱动的时候 ，遇到比较多的kernel panic oops 问题，然后

问一些 同事 ，比较多的回答都是加 printk,其实用 GDB 的中的一个功能是能很方便地调试这些问题的。

整理了一下，把自己给一家培训学校写的课件 贴上来。

•第一章 调试

•

•1.1. 工作环境配置

•

•1 ）安装好编译用的 kernel-source :RedHat :kernel-devel-xxx.rpm,

• suse:kernel-source-xxx.rpm, 自己编译的 kernel source ；

•2 ） GCC 包， gcc,g++,cpp,

•3) as,ld,objdump,etc

•4) glibc/uclibc

•5) make

•6) gdb

•7 ） SSH 工具 : SSH secure Shell 用于 windows 系统与 Linux 系统之间的文件传输

•8 ）串口工具，用于调试拿 log 信息， windos 下用超级终端或者 secureCRT ， Linux

•下用 minicom ,C-Kermit

•

•1.2 printk

•在内核中 printk () 的级别定义：

•#define KERN_EMERG "<0>" /* system is unusable */

•#define KERN_ALERT "<1>" /* action must be taken immediately */

•#define KERN_CRIT "<2>" /* critical conditions */

•#define KERN_ERR "<3>" /* error conditions */

•#define KERN_WARNING "<4>" /* warning conditions */

•#define KERN_NOTICE "<5>" /* normal but significant condition */

•#define KERN_INFO "<6>" /* informational */

•#define KERN_DEBUG "<7>" /* debug-level messages */

•通过 /proc/sys/kernel/printk 文件可以调节 printk 的输出级别，

•通过如下命令可以使得 Linux 内核的任何 printk 都被输出：

•#echo 8 > /proc/sys/kernel/printk

•同时设置 grub.conf :

•在 Kernel 这一行加上 ： console=tty0,console=ttyS0,115200

•

•1.3 oops 和 panic

•1.3.1 API oops DEBUG

•

•1.3.1.1. 定位 OOPS

•示例： apioops.c:

•

•#include <stdio.h>

•#include <stdlib.h>

•Const char array[]="/x6b/xc0 ";

•int main(int argc, char *argv[])

•{

• printf("%p/n", array);

• *(int *)0 = 0;

•}

•

•1. ）编译时打开 complie with debug info 选项 (-g) , 选项

•[root@localhost ~]# gcc -g -o apioops apioops.c

•

•

•2 ）执行 api_oops:

•[root@localhost ~]# ./apioops

•

•显示屏输出信息

•0x4005e0

•Segmentation fault

•[root@localhost ~]#

•

•串口输出信息：

•apioops[28910]: segfault at 0000000000000000 rip 00000000004004c0 rsp 00007fff22e15760 error 6

•rip 00000000004004c0 ： 表示执行到这个位置是 出错

•

•EIP RIP 值 一般表示代码运行时 ，出错的位置

•

3 ）调试

•[root@localhost ~]# gdb apioops

•GNU gdb Fedora (6.8-27.el5)

•Copyright (C) 2008 Free Software Foundation, Inc.

•License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

•This is free software: you are free to change and redistribute it.

•There is NO WARRANTY, to the extent permitted by law. Type "show copying"

•and "show warranty" for details.

•This GDB was configured as "x86_64-redhat-linux-gnu"...

•（ gdb ）

•

•1.list 调试 RIP 地址：很明显 出错在第 11 行 ，访问空指针

•(gdb) l*0x4004c0

•0x4004c0 is in main (apioops.c:11).

•6

•7 const char array[] = "/x63/x2e";

•8 int main(int argc, char *argv[])

•9 {

•10 printf("%p/n", array);

•11 *(int *)0 = 0;

•12 }

•(gdb)

•

•2. run apioops ，很明显 11 行 出错

•(gdb) r

•Starting program: /root/apioops

•0x4005c8

•Program received signal SIGSEGV, Segmentation fault.

•0x00000000004004c0 in main (argc=1, argv=0x7fffa7020a28) at oops.c:11

•11 *(int *)0 = 0;

•(gdb)

•

•3) 编译时没打开 complie with debug info 选项 (-g) , 选项的调试，或者只有 error 信息没有 代码的调试 。

•1. 运行 run

•(gdb) r

•Starting program: /root/apioops

•(no debugging symbols found)

•(no debugging symbols found)

•0x4005c8

•Program received signal SIGSEGV, Segmentation fault.

•0x00000000004004c0 in main ()

•(gdb)

•

•

•2. 反汇编

•( gdb ) disassemble

•Dump of assembler code for function main:

•0x0000000000400498 <main+0>: push % rbp

•0x0000000000400499 <main+1>: mov % rsp,%rbp

•0x000000000040049c <main+4>: sub $0x10,%rsp

•0x00000000004004a0 <main+8>: mov %edi,-0x4(% rbp )

•0x00000000004004a3 <main+11>: mov %rsi,-0x10(% rbp )

•0x00000000004004a7 <main+15>: mov $0x4005c8,%esi

•0x00000000004004ac <main+20>: mov $0x4005cb,%edi

•0x00000000004004b1 <main+25>: mov $0x0,%eax

•0x00000000004004b6 <main+30>: callq 0x400398 < printf@plt >

•0x00000000004004bb <main+35>: mov $0x0,%eax

•0x00000000004004c0 <main+40>: movl $0x0,(% rax )

•0x00000000004004c6 <main+46>: leaveq

•0x00000000004004c7 <main+47>: retq

•End of assembler dump.

•( gdb )

•可以看到出错地址

•0x00000000004004c0 <main+40>: movl $0x0,(% rax )

•表明是这个地址的代码访问了空指针

•

•3. 使用 objdump 反汇编出所有的信息 查看：

•[root@localhost ~]# objdump -d apioops > log

•………………………………….

•0000000000400498 <main>:

• 400498: 55 push %rbp

• 400499: 48 89 e5 mov %rsp,%rbp

• 40049c: 48 83 ec 10 sub $0x10,%rsp

• 4004a0: 89 7d fc mov %edi,0xfffffffffffffffc(%rbp)

• 4004a3: 48 89 75 f0 mov %rsi,0xfffffffffffffff0(%rbp)

• 4004a7: be c8 05 40 00 mov $0x4005c8,%esi

• 4004ac: bf cb 05 40 00 mov $0x4005cb,%edi

• 4004b1: b8 00 00 00 00 mov $0x0,%eax

• 4004b6: e8 dd fe ff ff callq 400398 <printf@plt>

• 4004bb: b8 00 00 00 00 mov $0x0,%eax

• 4004c0: c7 00 00 00 00 00 movl $0x0,(%rax)

• 4004c6: c9 leaveq

• 4004c7: c3 retq

• 4004c8: 90 nop

•…………………………………………

•可以看到出错地址

•0x00000000004004c0 <main+40>: movl $0x0,(%rax)

•

•1.3.2.kernel oops debug （以 x86 下为例）

•示例代码： oopsexam.c

•注意编译的时候代开 compile with debug info(-g)

•编译 ： make

•创建设备节点： mknod /dev/oopsexam c 251 0

•写操作：

•[root@localhost test]# echo 1 > /dev/oopsexam

•从串口拿到的 Log:

•

•<6>Enter oopsexam_write

•Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:

• [<ffffffff8848401c>] :oops:oopsexam_write+0x1c/0x29

•PGD 50ccd067 PUD 50ccc067 PMD 0

•Oops: 0002 [1] SMP

•last sysfs file: /block/sda/dev

•CPU 0

•Modules linked in: oops(FU) ipv6 xfrm_nalgo crypto_api autofs4 hidp rfcomm l2cap bluetooth sunrpc dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec button battery
asus_acpi acpi_memhotplug ac parport_pc lp parport floppy sg pcspkr i3000_edac edac_mc i2c_i801 i2c_core e1000 serio_raw e1000e dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache ata_piix libata shpchp mptsas mptscsih mptbase scsi_transport_sas
sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd

•Pid: 28853, comm: bash Tainted: GF 2.6.18-128.el5 #1

•

•

•Pid: 28853, comm: bash Tainted: GF 2.6.18-128.el5 #1

•RIP: 0010:[<ffffffff8848401c>] [<ffffffff8848401c>] :oops:oopsexam_write+0x1c/0x29

•RSP: 0018:ffff8100514dbf08 EFLAGS: 00010286

•RAX: 0000000000000002 RBX: 0000000000000002 RCX: ffffffff802f7aa8

•RDX: ffffffff802f7aa8 RSI: 0000000000000000 RDI: ffffffff802f7aa0

•RBP: 0000000000000002 R08: ffffffff802f7aa8 R09: 0000000000000046

•R10: ffff8100514dbc98 R11: ffffffff80161742 R12: 00002ba383367000

•R13: ffff8100514dbf50 R14: 0000000000000000 R15: 0000000000000000

•FS: 00002ba37fb84dc0(0000) GS:ffffffff803ac000(0000) knlGS:0000000000000000

•CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b

•CR2: 0000000000000000 CR3: 00000000508c4000 CR4: 00000000000006e0

•Process bash (pid: 28853, threadinfo ffff8100514da000, task ffff810051eb97a0)

•Stack: ffff810068aebd80 ffffffff8001659e c000003e00000001 ffff810068aebd80

• 0000000000000002 fffffffffffffff7 00002ba383367000 ffffffff80016e6b

• 0000000000000000 0000000000000000 0000000000000000 0000000000000002

•Call Trace:

• [<ffffffff8001659e>] vfs_write+0xce/0x174

• [<ffffffff80016e6b>] sys_write+0x45/0x6e

• [<ffffffff8005d28d>] tracesys+0xd5/0xe0

•Code: c7 04 25 00 00 00 00 01 00 00 00 5b c3 53 48 c7 c6 a7 40 48

•RIP [<ffffffff8848401c>] :oops:oopsexam_write+0x1c/0x29

• RSP <ffff8100514dbf08>

•CR2: 0000000000000000

• <0>Kernel panic - not syncing: Fatal exception

•

•很关键的地方 ：

•RIP [<ffffffff8848401c>] :oops:oopsexam_write+0x1c/0x29

调试：

•[root@localhost test]# gdb oops.ko

•GNU gdb Fedora (6.8-27.el5)

•Copyright (C) 2008 Free Software Foundation, Inc.

•License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

•This is free software: you are free to change and redistribute it.

•There is NO WARRANTY, to the extent permitted by law. Type "show copying"

•and "show warranty" for details.

•This GDB was configured as "x86_64-redhat-linux-gnu"...

•(gdb)

•1. list 调试：

•(gdb) l*oopsexam_write+0x1c

•0x1c is in oopsexam_write (/home/test/examoops.c:73).

•warning: Source file is more recent than executable.

•68 *off)

•69 {

•70 int *p=0;

•71

•72 printk(KERN_INFO "Enter %s/n",__func__);

•73 *p = 1; //create oops

•74 return len;

•75 }

•76

•77 module_init(oopsexam_init);

•(gdb)

•

•

•

•开发板上 NFS 环境下的 log 类似于以下信息，调试方法同上

•

•[root@utu-linux /test]# echo 1 >/dev/oopsexam

•<6>Enter oopsexam_write

•Unable to handle kernel NULL pointer dereference at virtual address 00000000

•pgd = c3ca4000

•[00000000] *pgd=33c96031, *pte=00000000, *ppte=00000000

•Internal error: Oops: 817 [#1]

•Modules linked in: oopsexam utuled button

•CPU: 0

•PC is at oopsexam_write+0x28/0x38 [oopsexam]

•LR is at 0x1

•pc : [<bf004060>] lr : [<00000001>] Not tainted

•sp : c3c29f3c ip : 60000093 fp : c3c29f4c

•r10: c3c29f78 r9 : c3c28000 r8 : c3c28000

•r7 : 40177000 r6 : c0628300 r5 : 00000000 r4 : 00000002

•r3 : 00000000 r2 : 00000001 r1 : 00004830 r0 : 00000002

•Flags: nZCv IRQs on FIQs on Mode SVC_32 Segment user

•Control: C000717F Table: 33CA4000 DAC: 00000015

•Process echo (pid: 829, stack limit = 0xc3c28194)

•Stack: (0xc3c29f3c to 0xc3c2a000)

•

•Stack: (0xc3c29f3c to 0xc3c2a000)

•9f20: 00000002

•9f40: c3c29f74 c3c29f50 c0070d84 bf004048 c0628324 c0628300 c3c29f78 00000000

•9f60: 00000000 401736bc c3c29fa4 c3c29f78 c0070ebc c0070cd4 00000000 00000000

•9f80: 00000000 00000002 40177000 40171c8c 00000004 c0022024 00000000 c3c29fa8

•9fa0: c0021ea0 c0070e80 00000002 c002877c 00000001 40177000 00000002 00000000

•9fc0: 00000002 40177000 40171c8c 40177000 00000002 0000c06c 401736bc bec36e68

•9fe0: 00000000 bec36dc4 00001920 40113490 60000010 00000001 00646f6d 00000000

•Backtrace:

•[<bf004038>] (oopsexam_write+0x0/0x38 [oopsexam]) from [<c0070d84>] (vfs_write+0

•xc0/0x138)

• r4 = 00000002

•[<c0070cc4>] (vfs_write+0x0/0x138) from [<c0070ebc>] (sys_write+0x4c/0x74)

•[<c0070e70>] (sys_write+0x0/0x74) from [<c0021ea0>] (ret_fast_syscall+0x0/0x2c)

• r8 = C0022024 r7 = 00000004 r6 = 40171C8C r5 = 40177000

• r4 = 00000002

•Code: eb40d1be e3a02001 e3a03000 e1a00004 (e5832000)

• Segmentation fault

•

•2. objdump

•Objdump –d oopsexam.ko > log

•Vim log : 查看地址： oopsexam_write+0x1c

•0000000000000000 <oopsexam_write>:

• 0: 53 push %rbx

• 1: 48 c7 c6 00 00 00 00 mov $0x0,%rsi

• 8: 48 89 d3 mov %rdx,%rbx

• b: 48 c7 c7 00 00 00 00 mov $0x0,%rdi

• 12: 31 c0 xor %eax,%eax

• 14: e8 00 00 00 00 callq 19 <oopsexam_write+0x19>

• 19: 48 89 d8 mov %rbx,%rax

• 1c: c7 04 25 00 00 00 00 movl $0x1,0x0

• 23: 01 00 00 00

• 27: 5b pop %rbx

• 28: c3 retq

•很明显 ，往一个空指针 赋值 1

•

•如何找到对应的代码，那就的对应着代码看汇编。

/********************************************************************************************************************************************/

什么是Oops？从语言学的角度说，Oops应该是一个拟声词。当出了点小事故，或者做了比较尴尬的事之后，你可以说"Oops"，翻译成中国话就叫做“哎呦”。“哎呦，对不起，对不起，我真不是故意打碎您的杯子的”。看，Oops就是这个意思。

在Linux内核开发中的Oops是什么呢？其实，它和上面的解释也没什么本质的差别，只不过说话的主角变成了Linux。当某些比较致命的问题出现时，我们的Linux内核也会抱歉的对我们说：“哎呦（Oops），对不起，我把事情搞砸了”。Linux内核在发生kernel panic时会打印出Oops信息，把目前的寄存器状态、堆栈内容、以及完整的Call trace都show给我们看，这样就可以帮助我们定位错误。

下面，我们来看一个实例。为了突出本文的主角--Oops，这个例子唯一的作用就是造一个空指针引用错误。

?
很明显，错误的地方就是第8行。

接下来，我们把这个模块编译出来，再用insmod来插入到内核空间，正如我们预期的那样，Oops出现了。

[ 100.243737] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 100.244985] IP: [<f82d2005>] hello_init+0x5/0x11 [hello]
[ 100.262266] *pde = 00000000
[ 100.288395] Oops: 0002 [#1] SMP
[ 100.305468] last sysfs file: /sys/devices/virtual/sound/timer/uevent
[ 100.325955] Modules linked in: hello(+) vmblock vsock vmmemctl vmhgfs acpiphp snd_ens1371 gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event
snd_seq snd_timer snd_seq_device ppdev psmouse serio_raw fbcon tileblit font bitblit softcursor snd parport_pc soundcore snd_page_alloc vmci i2c_piix4 vga16fb vgastate intel_agp agpgart shpchp lp parport floppy pcnet32 mii mptspi mptscsih mptbase scsi_transport_spi
vmxnet
[ 100.472178] [ 100.494931] Pid: 1586, comm: insmod Not tainted (2.6.32-21-generic #32-Ubuntu) VMware Virtual Platform
[ 100.540018] EIP: 0060:[<f82d2005>] EFLAGS: 00010246 CPU: 0
[ 100.562844] EIP is at hello_init+0x5/0x11 [hello]
[ 100.584351] EAX: 00000000 EBX: fffffffc ECX: f82cf040 EDX: 00000001
[ 100.609358] ESI: f82cf040 EDI: 00000000 EBP: f1b9ff5c ESP: f1b9ff5c
[ 100.631467] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 100.657664] Process insmod (pid: 1586, ti=f1b9e000 task=f137b340 task.ti=f1b9e000)
[ 100.706083] Stack:
[ 100.731783] f1b9ff88 c0101131 f82cf040 c076d240 fffffffc f82cf040 0072cff4 f82d2000
[ 100.759324] <0> fffffffc f82cf040 0072cff4 f1b9ffac c0182340 f19638f8 f137b340 f19638c0
[ 100.811396] <0> 00000004 09cc9018 09cc9018 00020000 f1b9e000 c01033ec 09cc9018 00015324
[ 100.891922] Call Trace:
[ 100.916257] [<c0101131>] ? do_one_initcall+0x31/0x190
[ 100.943670] [<f82d2000>] ? hello_init+0x0/0x11 [hello]
[ 100.970905] [<c0182340>] ? sys_init_module+0xb0/0x210
[ 100.995542] [<c01033ec>] ? syscall_call+0x7/0xb
[ 101.024087] Code: <c7> 05 00 00 00 00 01 00 00 00 5d c3 00 00 00 00 00 00 00 00 00 00
[ 101.079592] EIP: [<f82d2005>] hello_init+0x5/0x11 [hello] SS:ESP 0068:f1b9ff5c
[ 101.134682] CR2: 0000000000000000
[ 101.158929] ---[ end trace e294b69a66d752cb ]---

Oops首先描述了这是一个什么样的bug，然后指出了发生bug的位置，即“IP: [<f82d2005>] hello_init+0x5/0x11 [hello]”。

在这里，我们需要用到一个辅助工具objdump来帮助分析问题。objdump可以用来反汇编，命令格式如下：

objdump -S hello.o

下面是hello.o反汇编的结果，而且是和C代码混排的，非常的直观。

?
对照Oops的提示，我们可以很清楚的看到，出错的位置hello_init+0x5的汇编代码是：

?
这句代码的作用是把数值1存入0这个地址，这个操作当然是非法的。

我们还能看到它对应的c代码是：

?
Bingo！在Oops的帮助下我们很快就解决了问题。



我们再回过头来检查一下上面的Oops，看看Linux内核还有没有给我们留下其他的有用信息。

Oops: 0002 [#1]

这里面，0002表示Oops的错误代码（写错误，发生在内核空间），#1表示这个错误发生一次。

Oops的错误代码根据错误的原因会有不同的定义，本文中的例子可以参考下面的定义（如果发现自己遇到的Oops和下面无法对应的话，最好去内核代码里查找）：

* error_code:

* bit 0 == 0 means no page found, 1 means protection fault

* bit 1 == 0 means read, 1 means write

* bit 2 == 0 means kernel, 1 means user-mode

* bit 3 == 0 means data, 1 means instruction

有时候，Oops还会打印出Tainted信息。这个信息用来指出内核是因何种原因被tainted（直译为“玷污”）。具体的定义如下：

1: 'G' if all modules loaded have a GPL or compatible license, 'P' if any proprietary module has been loaded. Modules without a MODULE_LICENSE or with a MODULE_LICENSE that is not recognised by insmod as GPL compatible
are assumed to be proprietary.

2: 'F' if any module was force loaded by "insmod -f", ' ' if all modules were loaded normally.

3: 'S' if the oops occurred on an SMP kernel running on hardware that hasn't been certified as safe to run multiprocessor. Currently this occurs only on various Athlons that are not SMP capable.

4: 'R' if a module was force unloaded by "rmmod -f", ' ' if all modules were unloaded normally.

5: 'M' if any processor has reported a Machine Check Exception, ' ' if no Machine Check Exceptions have occurred.

6: 'B' if a page-release function has found a bad page reference or some unexpected page flags.

7: 'U' if a user or user application specifically requested that the Tainted flag be set, ' ' otherwise.

8: 'D' if the kernel has died recently, i.e. there was an OOPS or BUG.

9: 'A' if the ACPI table has been overridden.

10: 'W' if a warning has previously been issued by the kernel. (Though some warnings may set more specific taint flags.)

11: 'C' if a staging driver has been loaded.

12: 'I' if the kernel is working around a severe bug in the platform firmware (BIOS or similar).

基本上，这个Tainted信息是留给内核开发者看的。用户在使用Linux的过程中如果遇到Oops，可以把Oops的内容发送给内核开发者去debug，内核开发者根据这个Tainted信息大概可以判断出kernel panic时内核运行的环境。如果我们只是debug自己的驱动，这个信息就没什么意义了。



本文的这个例子非常简单，Oops发生以后没有造成宕机，这样我们就可以从dmesg中查看到完整的信息。但更多的情况是Oops发生的同时系统也会宕机，此时这些出错信息是来不及存入文件中的，关掉电源后就无法再看到了。我们只能通过其他的方式来记录：手抄或者拍照。

还有更坏的情况，如果Oops信息过多的话，一页屏幕显示不全，我们怎么来查看完整的内容呢？第一种方法，在grub里用vga参数指定更高的分辨率以使屏幕可以显示更多的内容。很明显，这个方法其实解决不了太多的问题；第二种方法，使用两台机器，把调试机的Oops信息通过串口打印到宿主机的屏幕上。但现在大部分的笔记本电脑是没有串口的，这个解决方法也有很大的局限性；第三种方法，使用内核转储工具kdump把发生Oops时的内存和CPU寄存器的内容dump到一个文件里，之后我们再用gdb来分析问题。



开发内核驱动的过程中可能遇到的问题是千奇百怪的，调试的方法也是多种多样，Oops是Linux内核给我们的提示，我们要用好它。

/************************************************************************************************************************************************/

这几天一直在调试atmel at91sam9x25的串口，用着用着总会导致Oops，Oops内容如下： [ 1023.510000] Unable to handle kernel NULL pointer dereference at virtual address 00000000 [ 1023.520000] pgd = c0004000 [ 1023.520000] [00000000] *pgd=00000000 [ 1023.520000] Internal error: Oops: 17 [#1] [ 1023.520000] last sysfs file: /sys/devices/virtual/misc/at91flash/dev [ 1023.520000] Modules linked in: at91flash at91gpio at91mc323 ds18b20 at91adc [ 1023.520000] CPU: 0 Tainted: G W (2.6.39 #35) [ 1023.520000] PC is at atmel_tasklet_func+0x104/0x690 [ 1023.520000] LR is at atmel_tasklet_func+0x10/0x690 [ 1023.520000] pc : [<c01a33ac>] lr : [<c01a32b8>] psr: 20000013 [ 1023.520000] sp : c7825f58 ip : 60000093 fp : 00000000 [ 1023.520000] r10: 00000006 r9 : 00000000 r8 : 0000000a [ 1023.520000] r7 : 00000000 r6 : c7824000 r5 : c78a2484 r4 : c03c0cb8 [ 1023.520000] r3 : 0000004c r2 : 0000004c r1 : 60000013 r0 : 00000001 [ 1023.520000] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel [ 1023.520000] Control: 0005317f Table: 27b40000 DAC: 00000017 [ 1023.520000] Process ksoftirqd/0 (pid: 3, stack limit = 0xc7824270) [ 1023.520000] Stack: (0xc7825f58 to 0xc7826000) [ 1023.520000] 5f40: 00000001 c7824000 [ 1023.520000] 5f60: 00000100 0000000a 00000000 00000006 c7825f8c 00000000 00000001 c7824000 [ 1023.520000] 5f80: 00000100 0000000a 00000006 c0045cf8 c03b995c c00461d8 c7aa6ae0 00000000 [ 1023.520000] 5fa0: 60000093 00000000 c7824000 c0046274 00000013 00000000 00000000 c00462e0 [ 1023.520000] 5fc0: 00000000 c7819f70 00000000 c00570e0 00000000 00000000 00000000 00000000 [ 1023.520000] 5fe0: c7825fe0 c7825fe0 c7819f70 c0057060 c0030b14 c0030b14 ffffffff ffffffff [ 1023.520000] [<c01a33ac>] (atmel_tasklet_func+0x104/0x690) from [<c0045cf8>] (tasklet_action+0x84/0xe [ 1023.520000] [<c0045cf8>] (tasklet_action+0x84/0xe from [<c00461d8>] (__do_softirq+0x88/0x124) [ 1023.520000] [<c00461d8>] (__do_softirq+0x88/0x124) from [<c00462e0>] (run_ksoftirqd+0x6c/0x12 [ 1023.520000] [<c00462e0>] (run_ksoftirqd+0x6c/0x12 from [<c00570e0>] (kthread+0x80/0x8 [ 1023.520000] [<c00570e0>] (kthread+0x80/0x8 from [<c0030b14>] (kernel_thread_exit+0x0/0x [ 1023.520000] Code: 1a000002 e59f057c e59f157c ebfa3d49 (e5973000) [ 1023.710000] ---[ end trace 786b41cd25d3b661 ]--- [ 1023.710000] Kernel panic - not syncing: Fatal exception in interrupt [ 1023.720000] [<c0034b10>] (unwind_backtrace+0x0/0xe0) from [<c02a8af8>] (panic+0x50/0x170) [ 1023.720000] [<c02a8af8>] (panic+0x50/0x170) from [<c0032e00>] (die+0x184/0x1c4) [ 1023.730000] [<c0032e00>] (die+0x184/0x1c4) from [<c0035aa8>] (__do_kernel_fault+0x64/0x84) [ 1023.740000] [<c0035aa8>] (__do_kernel_fault+0x64/0x84) from [<c0035c7c>] (do_page_fault+0x1b4/0x1c [ 1023.750000] [<c0035c7c>] (do_page_fault+0x1b4/0x1c from [<c002a240>] (do_DataAbort+0x30/0x9 [ 1023.760000] [<c002a240>] (do_DataAbort+0x30/0x98) from [<c002f86c>] (__dabt_svc+0x4c/0x60) [ 1023.770000] Exception stack(0xc7825f10 to 0xc7825f58) [ 1023.770000] 5f00: 00000001 60000013 0000004c 0000004c [ 1023.780000] 5f20: c03c0cb8 c78a2484 c7824000 00000000 0000000a 00000000 00000006 00000000 [ 1023.790000] 5f40: 60000093 c7825f58 c01a32b8 c01a33ac 20000013 ffffffff [ 1023.790000] [<c002f86c>] (__dabt_svc+0x4c/0x60) from [<c01a33ac>] (atmel_tasklet_func+0x104/0x690) [ 1023.800000] [<c01a33ac>] (atmel_tasklet_func+0x104/0x690) from [<c0045cf8>] (tasklet_action+0x84/0xe8) [ 1023.810000] [<c0045cf8>] (tasklet_action+0x84/0xe8) from [<c00461d8>] (__do_softirq+0x88/0x124) [ 1023.820000] [<c00461d8>] (__do_softirq+0x88/0x124) from [<c00462e0>] (run_ksoftirqd+0x6c/0x128) [ 1023.830000] [<c00462e0>] (run_ksoftirqd+0x6c/0x128) from [<c00570e0>] (kthread+0x80/0x88) [ 1023.840000] [<c00570e0>] (kthread+0x80/0x88) from [<c0030b14>] (kernel_thread_exit+0x0/0x8) 注意上述红色的地方。 下面就来显示如何定位出出错代码行: 1.首先，编译时打开complie with debug info选项，步则如下 make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- menuconfig 进入 Kernel hacking 选择 Compile the kernel with debug info 然后，保存，退出。 接着 make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- 编译， 等编译完成。 2.利用arm-none-linux-gnueabi-gdb 调试，如下： arm-none-linux-gnueabi-gdb vmlinux 对应着Oops 消息里面的这一行 [ 1023.520000] LR is at atmel_tasklet_func+0x10/0x690 在gdb下键入命令 ： l *at atmel_tasklet_func+0x10 这样就找到了出错的代码行。在这里鄙视一下atmel提供的内核，竟然还有bug，fuck it！ 从这里可以看出是由于串口的dma导致Oops的，于是我去掉了串口的dma传输。方法如下： 去掉之后还没有发现上述的Oops出现。