Linux driver oops异常的处理

如果Linux驱动里面出现了oops异常，这种情况如何快速定位到底什么原因引起的呢，需要对驱动debug。

例如：

#include <linux/init.h>
#include <linux/module.h>

static int __init hello_init(void)
{
int *p = NULL;
*p = 0;
return 0;
}

static void __exit hello_exit(void)
{
}

module_init(hello_init);
module_exit(hello_exit);

MODULE_LICENSE("GPL");

很多时候，kernel出现异常都是空指针引起的，这里也不例外，log如下：

[    1.848365]<0>.(0)[1:swapper/0]Unable to handle kernel NULL pointer dereference at virtual address 00000000
[    1.849578]<0>.(0)[1:swapper/0]pgd = ffffffc00007d000
[    1.850704][00000000] *pgd=00000000bfe14003, *pmd=00000000bfe15003, *pte=00e0000010221407
[    1.851722]<0>.(0)[1:swapper/0][KERN Warning] ERROR/WARN forces debug_lock off!
[    1.852629]<0>.(0)[1:swapper/0][KERN Warning] check backtrace:
[    1.853363]<0>.(0)[1:swapper/0]CPU: 0 PID: 1 Comm: swapper/0 Tainted: G        W    3.10.65 #3
[    1.854432]<0>.(0)[1:swapper/0]Call trace:
[    1.854955]<0>.(0)[1:swapper/0][<ffffffc00008902c>] dump_backtrace+0x0/0x16c
[    1.855838]<0>.(0)[1:swapper/0][<ffffffc0000891a8>] show_stack+0x10/0x1c
[    1.856678]<0>.(0)[1:swapper/0][<ffffffc000a13848>] dump_stack+0x1c/0x28
[    1.857519]<0>.(0)[1:swapper/0][<ffffffc000323e4c>] debug_locks_off+0x40/0x5c
[    1.858411]<0>.(0)[1:swapper/0][<ffffffc00009aa10>] oops_enter+0xc/0x28
[    1.859239]<0>.(0)[1:swapper/0][<ffffffc0000891d8>] die+0x24/0x1d8
[    1.860054]<0>.(0)[1:swapper/0][<ffffffc000a10e8c>] __do_kernel_fault.part.5+0x70/0x84
[    1.861048]<0>.(0)[1:swapper/0][<ffffffc00009481c>] do_page_fault+0x364/0x368
[    1.861942]<0>.(0)[1:swapper/0][<ffffffc00009490c>] do_translation_fault+0x40/0x4c
[    1.862887]<0>.(0)[1:swapper/0][<ffffffc000081410>] do_mem_abort+0x38/0x98
[    1.863743]<0>.(0)[1:swapper/0]Exception stack(0xffffffc07f85bc10 to 0xffffffc07f85bde4)
[    1.864754]<0>.(0)[1:swapper/0]bc00:                                     00fdf000 ffffffc0 7f858000 ffffffc0
[    1.865983]<0>.(0)[1:swapper/0]bc20: 7f85bdd0 ffffffc0 00e5d738 ffffffc0 00000003 00000000 7f858000 ffffffc0
[    1.867211]<0>.(0)[1:swapper/0]bc40: 00000000 00000000 00000006 00000000 00000000 00000000 00000100 00000000
[    1.868440]<0>.(0)[1:swapper/0]bc60: 7f85bc90 ffffffc0 00116884 ffffffc0 00ebb7f0 ffffffc0 00fdf000 ffffffc0
[    1.869668]<0>.(0)[1:swapper/0]bc80: 00000023 00000000 0000000f 00000000 7f85bca0 ffffffc0 0009d118 ffffffc0
[    1.871194]<0>.(0)[1:swapper/0]bca0: 7f85bd40 ffffffc0 00a11198 ffffffc0 00000000 00000000 00000000 00000000
[    1.872424]<0>.(0)[1:swapper/0]bcc0: 99d89d8a 00000000 00000000 00000000 00000002 00000000 00000000 00000000
[    1.873652]<0>.(0)[1:swapper/0]bce0: 0009cc68 ffffffc0 00ee6000 ffffffc0 00000001 00000000 00009425 00000000
[    1.874881]<0>.(0)[1:swapper/0]bd00: 00c742f0 ffffffc0 00000058 00000000 00000001 00000000 0001d238 00000000
[    1.876109]<0>.(0)[1:swapper/0]bd20: 00020000 00000000 0000001a 00000000 00000014 00000000 0000000e 00000000
[    1.877337]<0>.(0)[1:swapper/0]bd40: 00000007 00000000 00fdf000 ffffffc0 00e5d730 ffffffc0 00e89248 ffffffc0
[    1.878566]<0>.(0)[1:swapper/0]bd60: 6e2ba4d0 00000000 00fe0000 ffffffc0 7f858000 ffffffc0 00000000 00000000
[    1.879825]<0>.(0)[1:swapper/0]bd80: 00e3a1ac ffffffc0 00e94c90 ffffffc0 00000000 00000000 7f85bdd0 ffffffc0
[    1.881055]<0>.(0)[1:swapper/0]bda0: 00e3a8a8 ffffffc0 7f85bdd0 ffffffc0 00e5d738 ffffffc0 60000105 00000000
[    1.882284]<0>.(0)[1:swapper/0]bdc0: 00e89248 ffffffc0 00ee6000 ffffffc0 7f85be40 ffffffc0 00e3ab0c ffffffc0
[    1.883506]<0>.(0)[1:swapper/0]bde0: 00000127
[    1.884055]<0>.(0)[1:swapper/0][<ffffffc000083c58>] el1_da+0x1c/0x88
[    1.884854]<0>.(0)[1:swapper/0][<ffffffc000e3ab08>] kernel_init_freeable+0x138/0x1e0
[    1.885823]<0>.(0)[1:swapper/0][<ffffffc000a0a078>] kernel_init+0x18/0x154
[    1.886679]<0>-(0)[1:swapper/0]Internal error: Oops: 96000045 [#1] PREEMPT SMP
[    1.887583]<0>-(0)[1:swapper/0]CPU: 0 PID: 1 Comm: swapper/0 Tainted: G        W    3.10.65 #3
[    1.888656]<0>-(0)[1:swapper/0]task: ffffffc07f856000 ti: ffffffc07f858000 task.ti: ffffffc07f858000
[    1.889795]<0>-(0)[1:swapper/0]PC is at hello_init+0x8/0x10
[    1.890489]<0>-(0)[1:swapper/0]LR is at do_one_initcall+0xd4/0x1fc
[    1.891259]<0>-(0)[1:swapper/0]pc : [<ffffffc000e5d738>] lr : [<ffffffc000e3a8a8>] pstate: 60000105
[    1.892384]<0>-(0)[1:swapper/0]sp : ffffffc07f85bdd0
[    1.893001]x29: ffffffc07f85bdd0 x28: 0000000000000000
[    1.893651]x27: ffffffc000e94c90 x26: ffffffc000e3a1ac
[    1.894301]x25: 0000000000000000 x24: ffffffc07f858000
[    1.894952]x23: ffffffc000fe0000 x22: 000000006e2ba4d0
[    1.895603]x21: ffffffc000e89248 x20: ffffffc000e5d730
[    1.896253]x19: ffffffc000fdf000 x18: 0000000000000007
[    1.896904]x17: 000000000000000e x16: 0000000000000014
[    1.897555]x15: 000000000000001a x14: 0000000000020000
[    1.898205]x13: 000000000001d238 x12: 0000000000000001
[    1.898856]x11: 0000000000000058 x10: ffffffc000c742f0
[    1.899507]x9 : 0000000000009425 x8 : 0000000000000001
[    1.900158]x7 : ffffffc000ee6000 x6 : ffffffc00009cc68
[    1.900809]x5 : 0000000000000000 x4 : 0000000000000002
[    1.901459]x3 : 0000000000000000 x2 : 0000000099d89d8a
[    1.902110]x1 : 0000000000000000 x0 : 0000000000000000
[    1.902764]<0>-(0)[1:swapper/0]
[    1.902764]<0>PC: 0xffffffc000e5d6b8:
[    1.903607]d6b8  54000069 2a0003f3 14000014 52800140 52800001 52802002 aa1603e3 9103e264
[    1.904615]d6d8  97cd67e4 350000c0 f9400682 d0ffa841 91025021 f9001c41 1400000c f0fff2c0
[    1.905623]d6f8  52800141 9102e000 12800093 97eece7d f9400680 97d4d9cd 913802a0 d2800001
[    1.906632]d718  97cee667 2a1303e0 a94153f3 a9425bf5 a8c37bfd d65f03c0 d2800001 52800000
[    1.907641]d738  b900003f d65f03c0 52800000 d65f03c0 a9ba7bfd 910003fd a90363f7 90000478
[    1.908649]d758  a90153f3 91182313 a9025bf5 a9046bf9 a90573fb b9400261 7100803f 540000c9
[    1.909658]d778  f0fff2c0 910c6000 97eece5e 128002a0 140000bc 90005314 f0fff11a 912d6297
[    1.910667]d798  52800000 912ec341 97d2afcd 6b1f001f b9001ae0 540000cc f0fff2c0 910d2000
[    1.911679]<0>-(0)[1:swapper/0]
[    1.911679]<0>LR: 0xffffffc000e3a828:
[    1.912522]a828  340003e0 f0000be5 913640a0 aa0503fa 79404401 361000e1 f9400b03 b0fff1a1
[    1.913530]a848  913aa021 aa1403e2 b9430063 97d3f766 97cb3cd9 aa0003fb d63f0280 2a0003f5
[    1.914539]a868  97cb3cd5 aa0003e1 91364345 9100a0a5 794044a0 36100180 cb1b0024 b0fff1a1
[    1.915547]a888  aa0503e0 913b0021 aa1403e2 2a1503e3 d34afc84 97d3f754 14000003 d63f0280
[    1.916556]a8a8  2a0003f5 91000260 b9405302 91012001 6b19005f 3901201f 54000140 b0fff1a2
[    1.917565]a8c8  b9005319 913bc042 a9401444 b9401043 79402840 a9001424 b9001023 79002820
[    1.918573]a8e8  d53b4220 363801e0 91000260 b0fff1a1 913c2021 d2800802 91012000 97d390f2
[    1.919582]a908  d53b4220 363800a0 97d80fa1 97cb6fd8 d50342ff 14000003 97cb6fd5 d50342ff
[    1.920594]<0>-(0)[1:swapper/0]
[    1.920594]<0>SP: 0xffffffc07f85bd50:
[    1.921438]bd50  00e5d730 ffffffc0 00e89248 ffffffc0 6e2ba4d0 00000000 00fe0000 ffffffc0
[    1.922446]bd70  7f858000 ffffffc0 00000000 00000000 00e3a1ac ffffffc0 00e94c90 ffffffc0
[    1.923454]bd90  00000000 00000000 7f85bdd0 ffffffc0 00e3a8a8 ffffffc0 7f85bdd0 ffffffc0
[    1.924463]bdb0  00e5d738 ffffffc0 60000105 00000000 00e89248 ffffffc0 00ee6000 ffffffc0
[    1.925472]bdd0  7f85be40 ffffffc0 00e3ab0c ffffffc0 00000127 00000000 00e37a18 ffffffc0
[    1.926480]bdf0  00e89248 ffffffc0 00000006 00000000 00fdf000 ffffffc0 00fdf000 ffffffc0
[    1.927489]be10  00e89230 ffffffc0 00e3a1ac ffffffc0 00e94c90 ffffffc0 00ebfea8 ffffffc0
[    1.928497]be30  00000006 00000006 6e15aae9 00000000 7f85bea0 ffffffc0 00a0a07c ffffffc0
[    1.929513]<0>-(0)[1:swapper/0]
[    1.929513]<0>X6: 0xffffffc00009cbe8:
[    1.930355]cbe8  52800141 38336801 91000673 9124e280 f902101f 911fc2a0 91014000 942638da
[    1.931363]cc08  9124e280 aa1303e1 39510000 97fffdb3 36380f19 d51b4239 9401e7eb 17ffff9e
[    1.932371]cc28  b9096363 f904a768 b9095364 f9403fa0 52800021 aa1803e2 b909437f 9401f94a
[    1.933380]cc48  f9449b60 b5000ba0 aa1303e0 942638c6 910262a0 9400aaae aa1303e0 9426385c
[    1.934389]cc68  f944a77a f940037c aa1303e0 aa1403e1 942638d4 eb1c035f 54000060 97ffff02
[    1.935398]cc88  35fff220 35000bf7 a94153f3 a9425bf5 a94363f7 a9446bf9 a94573fb a8c67bfd
[    1.936406]cca8  910883ff d65f03c0 b9096363 aa0503fa f904a76a b9095369 f9403ba3 52800002
[    1.937415]ccc8  b9495361 d2808004 aa1a03e0 39403f5c 97fffb0c aa0003f9 b9496362 d3451f9c
[    1.938427]<0>-(0)[1:swapper/0]
[    1.938427]<0>X7: 0xffffffc000ee5f80:
[    1.939270]5f80  003102dc ffffffc0 0030ffc8 ffffffc0 00cb0a30 ffffffc0 000001a4 00000000
[    1.940279]5fa0  00000000 00000000 00000000 00000000 003102b0 ffffffc0 0030ff7c ffffffc0
[    1.941287]5fc0  00cb0a40 ffffffc0 000001a4 00000000 00000000 00000000 00000000 00000000
[    1.942296]5fe0  00310274 ffffffc0 0030ff38 ffffffc0 00cb0a50 ffffffc0 000001a4 00000000
[    1.943304]6000  00000000 00000000 00000000 00000000 00310238 ffffffc0 0030fef4 ffffffc0
[    1.944313]6020  00cb0a60 ffffffc0 000001a4 00000000 00000000 00000000 00000000 00000000
[    1.945322]6040  0031020c ffffffc0 0030fea8 ffffffc0 00cb0a70 ffffffc0 000001a4 00000000
[    1.946330]6060  00000000 00000000 00000000 00000000 003101d0 ffffffc0 0030fe5c ffffffc0
[    1.947344]<0>-(0)[1:swapper/0]
[    1.947344]<0>X10: 0xffffffc000c74270:
[    1.948197]4270  666f7270 20656c69 66667562 6e207265 6c20746f 65677261 6f6e6520 00686775
[    1.949204]4290  252c7525 2c756c6c 756c6c25 3b63252c 00000000 00000000 3025785c 00007832
[    1.950213]42b0  25286325 255b2978 73253a64 0000005d 78256325 00000029 3e75253c 00000000
[    1.951222]42d0  6c35255b 30302e75 30303030 0000005d 6c35255b 30252e75 5d756c36 00000000
[    1.952230]42f0  3e64253c 00000000 676f6c26 5f667562 6b636f6c 00000000 65732a28 6c2e296d
[    1.953239]4310  006b636f 00000000 616d6573 726f6870 6c3e2d65 006b636f 53797474 00000000
[    1.954248]4330  70203c3c 746e6972 6f63206b 6c6f736e 69642065 6c626173 3e3e2065 0000000a
[    1.955257]4350  70203c3c 746e6972 6f63206b 6c6f736e 61632065 2074276e 64206562 62617369
[    1.956274]<0>-(0)[1:swapper/0]
[    1.956274]<0>X19: 0xffffffc000fdef80:
[    1.957126]ef80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.958134]efa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.959143]efc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.960151]efe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.961160]f000  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.962169]f020  00000000 00000000 00000000 00000000 7fe9ccc0 ffffffc0 7fe9cb00 ffffffc0
[    1.963178]f040  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.964186]f060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.965199]<0>-(0)[1:swapper/0]
[    1.965199]<0>X20: 0xffffffc000e5d6b0:
[    1.966052]d6b0  f9000680 b140041f 54000069 2a0003f3 14000014 52800140 52800001 52802002
[    1.967060]d6d0  aa1603e3 9103e264 97cd67e4 350000c0 f9400682 d0ffa841 91025021 f9001c41
[    1.968069]d6f0  1400000c f0fff2c0 52800141 9102e000 12800093 97eece7d f9400680 97d4d9cd
[    1.969078]d710  913802a0 d2800001 97cee667 2a1303e0 a94153f3 a9425bf5 a8c37bfd d65f03c0
[    1.970086]d730  d2800001 52800000 b900003f d65f03c0 52800000 d65f03c0 a9ba7bfd 910003fd
[    1.971095]d750  a90363f7 90000478 a90153f3 91182313 a9025bf5 a9046bf9 a90573fb b9400261
[    1.972104]d770  7100803f 540000c9 f0fff2c0 910c6000 97eece5e 128002a0 140000bc 90005314
[    1.973113]d790  f0fff11a 912d6297 52800000 912ec341 97d2afcd 6b1f001f b9001ae0 540000cc
[    1.974125]<0>-(0)[1:swapper/0]
[    1.974125]<0>X21: 0xffffffc000e891c8:
[    1.974979]91c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.975987]91e8  00000000 00000000 00000000 00000000 00000000 00000000 00e94450 ffffffc0
[    1.976995]9208  00e94470 ffffffc0 00e94558 ffffffc0 00e945c8 ffffffc0 00e94658 ffffffc0
[    1.978004]9228  00e947c8 ffffffc0 00e94890 ffffffc0 00e95870 ffffffc0 00e95a58 ffffffc0
[    1.979013]9248  00c700e0 ffffffc0 00c8b280 ffffffc0 00c700e8 ffffffc0 00c700f8 ffffffc0
[    1.980021]9268  00c70100 ffffffc0 00c977b0 ffffffc0 00ca0240 ffffffc0 00c70108 ffffffc0
[    1.981030]9288  00000000 00000000 00000000 00000000 00000000 00000000 7665642f 6d61722f
[    1.982039]92a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.983052]<0>-(0)[1:swapper/0]
[    1.983052]<0>X23: 0xffffffc000fdff80:
[    1.983905]ff80  6f6f6220 6f727074 6c702e66 313d745f 20303838 746f6f62 666f7270 5f6b6c2e
[    1.984913]ffa0  31333d74 62203936 5f746f6f 73616572 303d6e6f 646e6120 64696f72 746f6f62
[    1.985921]ffc0  7265732e 6e6c6169 31303d6f 35343332 39383736 44434241 61204645 6f72646e
[    1.986930]ffe0  6f626469 622e746f 72746f6f 6f736165 6f703d6e 5f726577 2079656b 3d747067
[    1.987939]0000  66780a31 66666666 30306366 61336530 20303030 7830202d 66666666 30636666
[    1.988948]0020  39653030 30306163 28202020 33202020 6b203539 5b0a2942 20202020 30302e30
[    1.989957]0040  30303030 3e303c5d 20202020 742e2020 20747865 7830203a 66666666 30636666
[    1.990965]0060  38303030 30303030 30202d20 66666678 63666666 65303030 30306133 20202030
[    1.991978]<0>-(0)[1:swapper/0]
[    1.991978]<0>X24: 0xffffffc07f857f80:
[    1.992832]7f80  5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a
[    1.993840]7fa0  5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a
[    1.994848]7fc0  5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a
[    1.995857]7fe0  5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a
[    1.996865]8000  00000000 00000000 ffffffff ffffffff 7f856000 ffffffc0 00ebb5d8 ffffffc0
[    1.997874]8020  000b14ec ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
[    1.998883]8040  00000000 00000000 00000000 00000000 00000002 00000000 7f85bcb0 ffffffc0
[    1.999892]8060  00000001 00000000 57ac6e9d 00000000 16025142 210a12e0 660002ea 01420a0c
[    2.000905]<0>-(0)[1:swapper/0]
[    2.000905]<0>X26: 0xffffffc000e3a12c:
[    2.001758]a12c  350000c0 aa1803e0 aa1403e1 97d3930e 340001e0 14000012 f9400261 aa1803e0
[    2.002766]a14c  97ca1687 12001c00 34fffee0 f9400661 aa1903e0 d63f0020 34000120 aa1503e0
[    2.003774]a16c  aa1803e1 97ef5be2 14000005 f9400260 913402c1 97d392fc 34fffea0 91006273
[    2.004783]a18c  17ffffe5 52800000 f94023f9 a94153f3 a9425bf5 a94363f7 a8c57bfd d65f03c0
[    2.005792]a1ac  a9be7bfd 910003fd a90153f3 aa0003f3 b4000341 aa0103f4 97d3934b 91000401
[    2.006800]a1cc  8b010261 eb14003f 54000081 528007a0 381ff020 14000011 91000800 8b000273
[    2.007809]a1ec  eb14027f 54000141 528007a0 381fe260 aa1303e0 97d3933c 91000402 aa1303e1
[    2.008818]a20c  d1000660 97d36ddc 14000004 d29bd5a0 52815dc1 b9000001 52800000 a94153f3
[    2.009831]<0>-(0)[1:swapper/0]
[    2.009831]<0>X27: 0xffffffc000e94c10:
[    2.010685]4c10  00e59118 ffffffc0 00e5a07c ffffffc0 00e5a41c ffffffc0 00e5a438 ffffffc0
[    2.011692]4c30  00e5a454 ffffffc0 00e5ab50 ffffffc0 00e5ab88 ffffffc0 00e5aba4 ffffffc0
[    2.012701]4c50  00e5abc0 ffffffc0 00e5abdc ffffffc0 00e5ac00 ffffffc0 00e5cd80 ffffffc0
[    2.013710]4c70  00e5d13c ffffffc0 00e5d274 ffffffc0 00365b90 ffffffc0 00e5d730 ffffffc0
[    2.014718]4c90  00e5d740 ffffffc0 00e5d748 ffffffc0 00368f0c ffffffc0 00e5db88 ffffffc0
[    2.015727]4cb0  00e5db90 ffffffc0 00e5dc14 ffffffc0 00e5de28 ffffffc0 00e5de74 ffffffc0
[    2.016736]4cd0  00e5df0c ffffffc0 00e5e18c ffffffc0 00e5e2d8 ffffffc0 00e5e454 ffffffc0
[    2.017744]4cf0  00e5e498 ffffffc0 00e5e4e8 ffffffc0 00a10380 ffffffc0 00e5f220 ffffffc0
[    2.018758]<0>-(0)[1:swapper/0]
[    2.018758]<0>X29: 0xffffffc07f85bd50:
[    2.019611]bd50  00e5d730 ffffffc0 00e89248 ffffffc0 6e2ba4d0 00000000 00fe0000 ffffffc0
[    2.020618]bd70  7f858000 ffffffc0 00000000 00000000 00e3a1ac ffffffc0 00e94c90 ffffffc0
[    2.021627]bd90  00000000 00000000 7f85bdd0 ffffffc0 00e3a8a8 ffffffc0 7f85bdd0 ffffffc0
[    2.022636]bdb0  00e5d738 ffffffc0 60000105 00000000 00e89248 ffffffc0 00ee6000 ffffffc0
[    2.023645]bdd0  7f85be40 ffffffc0 00e3ab0c ffffffc0 00000127 00000000 00e37a18 ffffffc0
[    2.024653]bdf0  00e89248 ffffffc0 00000006 00000000 00fdf000 ffffffc0 00fdf000 ffffffc0
[    2.025662]be10  00e89230 ffffffc0 00e3a1ac ffffffc0 00e94c90 ffffffc0 00ebfea8 ffffffc0
[    2.026670]be30  00000006 00000006 6e15aae9 00000000 7f85bea0 ffffffc0 00a0a07c ffffffc0
[    2.027680]<0>-(0)[1:swapper/0]
[    2.028076]<0>-(0)[1:swapper/0]Process swapper/0 (pid: 1, stack limit = 0xffffffc07f858068)
[    2.029118]<0>-(0)[1:swapper/0]Stack: (0xffffffc07f85bdd0 to 0xffffffc07f85c000)
[    2.030042]<0>-(0)[1:swapper/0]bdc0:                                     7f85be40 ffffffc0 00e3ab0c ffffffc0
[    2.031271]<0>-(0)[1:swapper/0]bde0: 00000127 00000000 00e37a18 ffffffc0 00e89248 ffffffc0 00000006 00000000
[    2.032499]<0>-(0)[1:swapper/0]be00: 00fdf000 ffffffc0 00fdf000 ffffffc0 00e89230 ffffffc0 00e3a1ac ffffffc0
[    2.033728]<0>-(0)[1:swapper/0]be20: 00e94c90 ffffffc0 00ebfea8 ffffffc0 00000006 00000006 6e15aae9 00000000
[    2.034957]<0>-(0)[1:swapper/0]be40: 7f85bea0 ffffffc0 00a0a07c ffffffc0 00fdf000 ffffffc0 00fdf000 ffffffc0
[    2.036184]<0>-(0)[1:swapper/0]be60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.037412]<0>-(0)[1:swapper/0]be80: 00000000 00000000 00000000 00000000 00000000 00000000 02104833 c1259420
[    2.038641]<0>-(0)[1:swapper/0]bea0: 00000000 00000000 00084480 ffffffc0 00a0a060 ffffffc0 00000000 00000000
[    2.039869]<0>-(0)[1:swapper/0]bec0: 00000000 00000000 7f85c000 ffffffc0 00000000 00000000 00000000 00000000
[    2.041097]<0>-(0)[1:swapper/0]bee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.042324]<0>-(0)[1:swapper/0]bf00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.043552]<0>-(0)[1:swapper/0]bf20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.044780]<0>-(0)[1:swapper/0]bf40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.046008]<0>-(0)[1:swapper/0]bf60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.047235]<0>-(0)[1:swapper/0]bf80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.048463]<0>-(0)[1:swapper/0]bfa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.049690]<0>-(0)[1:swapper/0]bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000005 00000000
[    2.050919]<0>-(0)[1:swapper/0]bfe0: 00000000 00000000 00000000 00000000 19ba0300 14469880 00492290 9d903780
[    2.052139]<0>-(0)[1:swapper/0]Call trace:
[    2.052654]<0>-(0)[1:swapper/0][<ffffffc000e5d738>] hello_init+0x8/0x10
[    2.053484]<0>-(0)[1:swapper/0][<ffffffc000e3ab08>] kernel_init_freeable+0x138/0x1e0
[    2.054452]<0>-(0)[1:swapper/0][<ffffffc000a0a078>] kernel_init+0x18/0x154
[    2.055309]<0>-(0)[1:swapper/0]Code: a8c37bfd d65f03c0 d2800001 52800000 (b900003f)
[    2.056273]<0>-(0)[1:swapper/0]---[ end trace 1b75b31a2719ed1e ]---
[    2.057052]<0>-(0)[1:swapper/0]Kernel panic - not syncing: Fatal exception
[    2.057913]<2>-(2)[0:swapper/2]CPU2: stopping
[    2.058458]<2>-(2)[0:swapper/2]CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D W    3.10.65 #3
[    2.059528]<2>-(2)[0:swapper/2]Call trace:
[    2.060047]<2>-(2)[0:swapper/2][<ffffffc00008902c>] dump_backtrace+0x0/0x16c
[    2.060929]<2>-(2)[0:swapper/2][<ffffffc0000891a8>] show_stack+0x10/0x1c
[    2.061766]<2>-(2)[0:swapper/2][<ffffffc000a13848>] dump_stack+0x1c/0x28
[    2.062604]<2>-(2)[0:swapper/2][<ffffffc00008f8e4>] handle_IPI+0x260/0x294
[    2.063461]<2>-(2)[0:swapper/2][<ffffffc000081658>] gic_handle_irq+0x7c/0x84
[    2.064338]<2>-(2)[0:swapper/2]Exception stack(0xffffffc00309be20 to 0xffffffc00309bff4)
[    2.065351]<2>-(2)[0:swapper/2]be20: 03098000 ffffffc0 03098000 ffffffc0 0309bf60 ffffffc0 00085570 ffffffc0
[    2.066580]<2>-(2)[0:swapper/2]be40: 0008556c ffffffc0 00000000 00000000 00000000 00000000 00010000 00000000
[    2.067808]<2>-(2)[0:swapper/2]be60: 00000000 00000000 00000000 00000000 000c79b0 ffffffc0 00ee6000 ffffffc0
[    2.069036]<2>-(2)[0:swapper/2]be80: 00000000 00000000 0309bd60 ffffffc0 00ee6000 ffffffc0 03098000 ffffffc0
[    2.070265]<2>-(2)[0:swapper/2]bea0: 00fe0000 ffffffc0 0130f000 ffffffc0 00000001 00000000 0000001a 00000000
[    2.071492]<2>-(2)[0:swapper/2]bec0: 00000014 00000000 0000000e 00000000 00000007 00000000 03098000 ffffffc0
[    2.072720]<2>-(2)[0:swapper/2]bee0: 03098000 ffffffc0 03098000 ffffffc0 00fb5000 ffffffc0 010078c4 ffffffc0
[    2.073949]<2>-(2)[0:swapper/2]bf00: 00a31000 ffffffc0 00fb4488 ffffffc0 00c834a8 ffffffc0 00000001 00000000
[    2.075177]<2>-(2)[0:swapper/2]bf20: 400801d8 00000000 0309bf60 ffffffc0 0008556c ffffffc0 0309bf60 ffffffc0
[    2.076405]<2>-(2)[0:swapper/2]bf40: 00085570 ffffffc0 60000145 00000000 0309bf60 ffffffc0 0008556c ffffffc0
[    2.077633]<2>-(2)[0:swapper/2]bf60: 0309bf70 ffffffc0 00108204 ffffffc0 0309bfd0 ffffffc0 00a0d9e4 ffffffc0
[    2.078861]<2>-(2)[0:swapper/2]bf80: 00000002 00000000 00000000 00000000 00fdf210 ffffffc0 410fd033 00000000
[    2.080089]<2>-(2)[0:swapper/2]bfa0: 40eb7028 00000000 40000000 00000000 4007b000 00000000 4007d000 00000000
[    2.081318]<2>-(2)[0:swapper/2]bfc0: 000801f0 ffffffc0 400801d8 00000000 00000000 00000000 400801d8 00000000
[    2.082543]<2>-(2)[0:swapper/2]bfe0: 00000000 00000000 00000000 00000000 ffffffff
[    2.083478]<2>-(2)[0:swapper/2][<ffffffc000083e44>] el1_irq+0x84/0xf4
[    2.084284]<2>-(2)[0:swapper/2][<ffffffc000108200>] cpu_startup_entry+0x1b8/0x27c
[    2.085218]<2>-(2)[0:swapper/2][<ffffffc000a0d9e0>] secondary_start_kernel+0x208/0x228
[    2.086203]<1>-(1)[0:swapper/1]CPU1: stopping
[    2.086749]<1>-(1)[0:swapper/1]CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D W    3.10.65 #3
[    2.087817]<1>-(1)[0:swapper/1]Call trace:
[    2.088338]<1>-(1)[0:swapper/1][<ffffffc00008902c>] dump_backtrace+0x0/0x16c
[    2.089220]<1>-(1)[0:swapper/1][<ffffffc0000891a8>] show_stack+0x10/0x1c
[    2.090058]<1>-(1)[0:swapper/1][<ffffffc000a13848>] dump_stack+0x1c/0x28
[    2.090895]<1>-(1)[0:swapper/1][<ffffffc00008f8e4>] handle_IPI+0x260/0x294
[    2.091753]<1>-(1)[0:swapper/1][<ffffffc000081658>] gic_handle_irq+0x7c/0x84
[    2.092630]<1>-(1)[0:swapper/1]Exception stack(0xffffffc003097e20 to 0xffffffc003097ff4)
[    2.093643]<1>-(1)[0:swapper/1]7e20: 03094000 ffffffc0 03094000 ffffffc0 03097f60 ffffffc0 00085570 ffffffc0
[    2.094871]<1>-(1)[0:swapper/1]7e40: 0008556c ffffffc0 00000000 00000000 00000000 00000000 00010000 00000000
[    2.096100]<1>-(1)[0:swapper/1]7e60: 00000000 00000000 00000000 00000000 000c79b0 ffffffc0 00ee6000 ffffffc0
[    2.097328]<1>-(1)[0:swapper/1]7e80: 00000000 00000000 03097d60 ffffffc0 00000062 00000000 00000005 00000000
[    2.098556]<1>-(1)[0:swapper/1]7ea0: 00a37128 ffffffc0 0130f000 ffffffc0 00000001 00000000 0000001a 00000000
[    2.099784]<1>-(1)[0:swapper/1]7ec0: 00000014 00000000 0000000e 00000000 00000007 00000000 03094000 ffffffc0
[    2.101013]<1>-(1)[0:swapper/1]7ee0: 03094000 ffffffc0 03094000 ffffffc0 00fb5000 ffffffc0 010078c4 ffffffc0
[    2.102241]<1>-(1)[0:swapper/1]7f00: 00a31000 ffffffc0 00fb4488 ffffffc0 00c834a8 ffffffc0 00000001 00000000
[    2.103470]<1>-(1)[0:swapper/1]7f20: 400801d8 00000000 03097f60 ffffffc0 0008556c ffffffc0 03097f60 ffffffc0
[    2.104698]<1>-(1)[0:swapper/1]7f40: 00085570 ffffffc0 60000305 00000000 03097f60 ffffffc0 0008556c ffffffc0
[    2.105927]<1>-(1)[0:swapper/1]7f60: 03097f70 ffffffc0 00108204 ffffffc0 03097fd0 ffffffc0 00a0d9e4 ffffffc0
[    2.107154]<1>-(1)[0:swapper/1]7f80: 00000001 00000000 00000000 00000000 00fdf210 ffffffc0 410fd033 00000000
[    2.108383]<1>-(1)[0:swapper/1]7fa0: 40eb7028 00000000 40000000 00000000 4007b000 00000000 4007d000 00000000
[    2.109611]<1>-(1)[0:swapper/1]7fc0: 000801f0 ffffffc0 400801d8 00000000 00000000 00000000 400801d8 00000000
[    2.110837]<1>-(1)[0:swapper/1]7fe0: 00000000 00000000 00000000 00000000 ffffffff
[    2.111771]<1>-(1)[0:swapper/1][<ffffffc000083e44>] el1_irq+0x84/0xf4
[    2.112575]<1>-(1)[0:swapper/1][<ffffffc000108200>] cpu_startup_entry+0x1b8/0x27c
[    2.113510]<1>-(1)[0:swapper/1][<ffffffc000a0d9e0>] secondary_start_kernel+0x208/0x228
[    2.114494]<3>-(3)[0:swapper/3]CPU3: stopping
[    2.115041]<3>-(3)[0:swapper/3]CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D W    3.10.65 #3
[    2.116110]<3>-(3)[0:swapper/3]Call trace:
[    2.116630]<3>-(3)[0:swapper/3][<ffffffc00008902c>] dump_backtrace+0x0/0x16c
[    2.117513]<3>-(3)[0:swapper/3][<ffffffc0000891a8>] show_stack+0x10/0x1c
[    2.118350]<3>-(3)[0:swapper/3][<ffffffc000a13848>] dump_stack+0x1c/0x28
[    2.119187]<3>-(3)[0:swapper/3][<ffffffc00008f8e4>] handle_IPI+0x260/0x294
[    2.120045]<3>-(3)[0:swapper/3][<ffffffc000081658>] gic_handle_irq+0x7c/0x84
[    2.120921]<3>-(3)[0:swapper/3]Exception stack(0xffffffc0030a3e20 to 0xffffffc0030a3ff4)
[    2.121935]<3>-(3)[0:swapper/3]3e20: 030a0000 ffffffc0 030a0000 ffffffc0 030a3f60 ffffffc0 00085570 ffffffc0
[    2.123163]<3>-(3)[0:swapper/3]3e40: 0008556c ffffffc0 00000000 00000000 00000000 00000000 00010000 00000000
[    2.124391]<3>-(3)[0:swapper/3]3e60: 00000000 00000000 00000000 00000000 000c79b0 ffffffc0 00ee6000 ffffffc0
[    2.125619]<3>-(3)[0:swapper/3]3e80: 00000000 00000000 030a3d60 ffffffc0 0000004c 00000000 00000005 00000000
[    2.126848]<3>-(3)[0:swapper/3]3ea0: 00a37128 ffffffc0 00000078 00000000 00000001 00000000 0000001a 00000000
[    2.128076]<3>-(3)[0:swapper/3]3ec0: 00000014 00000000 0000000e 00000000 00000007 00000000 030a0000 ffffffc0
[    2.129304]<3>-(3)[0:swapper/3]3ee0: 030a0000 ffffffc0 030a0000 ffffffc0 00fb5000 ffffffc0 010078c4 ffffffc0
[    2.130532]<3>-(3)[0:swapper/3]3f00: 00a31000 ffffffc0 00fb4488 ffffffc0 00c834a8 ffffffc0 00000001 00000000
[    2.131761]<3>-(3)[0:swapper/3]3f20: 400801d8 00000000 030a3f60 ffffffc0 0008556c ffffffc0 030a3f60 ffffffc0
[    2.132989]<3>-(3)[0:swapper/3]3f40: 00085570 ffffffc0 60000105 00000000 030a3f60 ffffffc0 0008556c ffffffc0
[    2.134218]<3>-(3)[0:swapper/3]3f60: 030a3f70 ffffffc0 00108204 ffffffc0 030a3fd0 ffffffc0 00a0d9e4 ffffffc0
[    2.135446]<3>-(3)[0:swapper/3]3f80: 00000003 00000000 00000000 00000000 00fdf210 ffffffc0 410fd033 00000000
[    2.136674]<3>-(3)[0:swapper/3]3fa0: 40eb7028 00000000 40000000 00000000 4007b000 00000000 4007d000 00000000
[    2.137902]<3>-(3)[0:swapper/3]3fc0: 000801f0 ffffffc0 400801d8 00000000 00000000 00000000 400801d8 00000000
[    2.139128]<3>-(3)[0:swapper/3]3fe0: 00000000 00000000 00000000 00000000 ffffffff
[    2.140062]<3>-(3)[0:swapper/3][<ffffffc000083e44>] el1_irq+0x84/0xf4
[    2.140866]<3>-(3)[0:swapper/3][<ffffffc000108200>] cpu_startup_entry+0x1b8/0x27c
[    2.141800]<3>-(3)[0:swapper/3][<ffffffc000a0d9e0>] secondary_start_kernel+0x208/0x228
[    2.142785]Rebooting in 5 seconds..

根据log来看，明确指出了这是NULL指针引起的错误。

那么具体是哪一行代码引起的呢，可以根据Call trace，找到出错的地方应该是在hello_init()函数里面：

[    2.052654]<0>-(0)[1:swapper/0][<ffffffc000e5d738>] hello_init+0x8/0x10
[    2.053484]<0>-(0)[1:swapper/0][<ffffffc000e3ab08>] kernel_init_freeable+0x138/0x1e0
[    2.054452]<0>-(0)[1:swapper/0][<ffffffc000a0a078>] kernel_init+0x18/0x154

可以在hello_init()函数里面加上适当的打印语句，再重新根据log来看定位到是哪一行出现的问题。

出现oops以后，kernel不仅仅把Call trace打印出来了，还将当前的PC指针也打印出来了，例如这里的：

[    1.889795]<0>-(0)[1:swapper/0]PC is at hello_init+0x8/0x10
[    1.890489]<0>-(0)[1:swapper/0]LR is at do_one_initcall+0xd4/0x1fc
[    1.891259]<0>-(0)[1:swapper/0]pc : [<ffffffc000e5d738>] lr : [<ffffffc000e3a8a8>] pstate: 60000105
[    1.892384]<0>-(0)[1:swapper/0]sp : ffffffc07f85bdd0

我们可以根据PC指针这个值直接定位到是哪一行出的错误，使用addr2line命令，例如：

$ addr2line -f -e vmlinux ffffffc000e5d738
hello_init
/xxx/drivers/char/hello.c:7

直接定位到是hello.c的第7行出现了问题，也就是对一个空指针做了赋值操作。

addr2line命令的-f是打印出函数名，-e指定文件名，默认为a.out。

参考：http://elinux.org/Addr2line_for_kernel_debugging

// 2017-04-07 add

使用gdb

还是根据log来看，出错的地方是hello_init()函数：

[    2.052654]<0>-(0)[1:swapper/0][<ffffffc000e5d738>] hello_init+0x8/0x10
[    2.053484]<0>-(0)[1:swapper/0][<ffffffc000e3ab08>] kernel_init_freeable+0x138/0x1e0
[    2.054452]<0>-(0)[1:swapper/0][<ffffffc000a0a078>] kernel_init+0x18/0x154

出错的地方应该是hello_init+0x08这个地址处，使用gdb命令：

gdb vmlinux

然后使用gdb中的list命令：

(gdb) list *hello_init+0x08
0xffffffc000e5d738 is in hello_init (/xxx/drivers/char/hello.c:7).
2	#include <linux/module.h>
3
4	static int __init hello_init(void)
5	{
6		int *p = NULL;
7		*p = 0;
8		return 0;
9	}
10
11	static void __exit hello_exit(void)

很清晰的告诉我们出错的地方是hello.c的第7行。

参考：https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks