用gdb调试多进程和多线程的服务器程序

转自：http://bbs.chinaunix.net/viewthread.php?tid=1609486&page=3&authorid=20701329
1楼 发表于 2009-11-06 15:17 | 显示全部帖子

刚好这几天用gdb调试多进程和多线程的服务器程序，以往对于这类程序也都是打日志加core文件的方法， 但逻辑过于复杂的时候，屡次增加日志终是不爽，对于一般的服务器程序去掉编译优化选项gdb跟踪还是很好的， 对于多进程和多线程的daemon后台程序， set follow-fork-mode 或者attache的方式也一样可以直接跟出问题。

2楼 发表于 2009-11-07 23:04 | 显示全部帖子


QUOTE:
原帖由 benjiam 于 2009-11-7 13:06 发表


刚好这几天用gdb调试多进程和多线程的服务器程序，以往对于这类程序也 ...

来个直观的，echosvc是个多进程并使用多线程的daemon服务器程序：

QUOTE:
$ ls
config.ini echosvc log.txt runtest.sh
$ gdb echosvc
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
(gdb) l
95 }
96
97 apr_pollset_add(cs->pollset,cs->pfd);
98 return 0;
99 }
100
101 int main(int argc,const char * const argv[])
102 {
103
104 if(zevent_init(file,&pglobal)==-1)
(gdb) set follow-fork-mode child
(gdb) b zevent_process_connection
Breakpoint 1 at 0x80489b2: file echo.c, line 34.
(gdb) r
Starting program: echosvc
[Thread debugging using libthread_db enabled]
[Thread debugging using libthread_db enabled]
[New Thread 0xb7ce08d0 (LWP 8755)]
[Thread debugging using libthread_db enabled]
[New Thread 0xb7ce08d0 (LWP 8756)]
[New Thread 0xb637ab90 (LWP 8757)]
[New Thread 0xb5b79b90 (LWP 875

]
[New Thread 0xb5378b90 (LWP 8759)]
[New Thread 0xb4b77b90 (LWP 8760)]
[New Thread 0xb4376b90 (LWP 8761)]
[New Thread 0xb3b75b90 (LWP 8762)]
[New Thread 0xb3374b90 (LWP 8763)]
[New Thread 0xb2b73b90 (LWP 8764)]
[New Thread 0xb2372b90 (LWP 8765)]
[New Thread 0xb1b71b90 (LWP 8766)]
[New Thread 0xb1370b90 (LWP 8767)]
[New Thread 0xb0b6fb90 (LWP 876

]
[New Thread 0xb036eb90 (LWP 8769)]
[New Thread 0xafb6db90 (LWP 8770)]
[New Thread 0xaf36cb90 (LWP 8771)]
[New Thread 0xaeb6bb90 (LWP 8772)]
[New Thread 0xae36ab90 (LWP 8773)]
[New Thread 0xadb69b90 (LWP 8774)]
[tcsetpgrp failed in terminal_inferior: No such process]
[Thread 0xb637ab90 (LWP 8757) exited]
[Switching to Thread 0xae36ab90 (LWP 8773)]

Breakpoint 1, zevent_process_connection (cs=0x832cc40) at echo.c:34 当客户端发送消息成功进入断点函数
34 apr_size_t len=0;
(gdb) l
29 /*
30 * code for your app,this just an example for echo test.
31 */
32 apr_bucket *b;
33 char *msg;
34 apr_size_t len=0;
35 int olen = 0;
36 const char *buf;
37 apr_status_t rv;
38
(gdb)
39 cs->pfd->reqevents = APR_POLLIN;
40
41 if(cs->pfd->rtnevents & APR_POLLIN){
42 len = 4096;
43 msg = (char *)apr_bucket_alloc(len,cs->baout);
44 if (msg == NULL) {
45 return -1;
46 }
47 rv = apr_socket_recv(cs->pfd->desc.s,msg,&len);
48 if(rv != APR_SUCCESS)
(gdb) b 47 想看看我到底收到了什么
Breakpoint 2 at 0x8048a13: file echo.c, line 47.
(gdb) c
Continuing.

Breakpoint 2, zevent_process_connection (cs=0x832cc40) at echo.c:47
47 rv = apr_socket_recv(cs->pfd->desc.s,msg,&len);
(gdb) n
48 if(rv != APR_SUCCESS)
(gdb) p msg
$1 = 0x8332a78 "asdf/r/n"
(gdb) info threads
19 Thread 0xadb69b90 (LWP 8774) 0xb7f2a430 in __kernel_vsyscall ()
* 18 Thread 0xae36ab90 (LWP 8773) zevent_process_connection (cs=0x832cc40) at echo.c:48 我们正在调试的线程
17 Thread 0xaeb6bb90 (LWP 8772) 0xb7f2a430 in __kernel_vsyscall ()
16 Thread 0xaf36cb90 (LWP 8771) 0xb7f2a430 in __kernel_vsyscall ()
15 Thread 0xafb6db90 (LWP 8770) 0xb7f2a430 in __kernel_vsyscall ()
14 Thread 0xb036eb90 (LWP 8769) 0xb7f2a430 in __kernel_vsyscall ()
13 Thread 0xb0b6fb90 (LWP 876

0xb7f2a430 in __kernel_vsyscall ()
12 Thread 0xb1370b90 (LWP 8767) 0xb7f2a430 in __kernel_vsyscall ()
11 Thread 0xb1b71b90 (LWP 8766) 0xb7f2a430 in __kernel_vsyscall ()
10 Thread 0xb2372b90 (LWP 8765) 0xb7f2a430 in __kernel_vsyscall ()
9 Thread 0xb2b73b90 (LWP 8764) 0xb7f2a430 in __kernel_vsyscall ()
8 Thread 0xb3374b90 (LWP 8763) 0xb7f2a430 in __kernel_vsyscall ()
7 Thread 0xb3b75b90 (LWP 8762) 0xb7f2a430 in __kernel_vsyscall ()
6 Thread 0xb4376b90 (LWP 8761) 0xb7f2a430 in __kernel_vsyscall ()
5 Thread 0xb4b77b90 (LWP 8760) 0xb7f2a430 in __kernel_vsyscall ()
4 Thread 0xb5378b90 (LWP 8759) 0xb7f2a430 in __kernel_vsyscall ()
3 Thread 0xb5b79b90 (LWP 875

0xb7f2a430 in __kernel_vsyscall ()
1 Thread 0xb7ce08d0 (LWP 8756) 0xb7f2a430 in __kernel_vsyscall ()
(gdb) bt
#0 zevent_process_connection (cs=0x832cc40) at echo.c:48
#1 0xb7f2457c in zevent_run_process_connection (cs=0x832cc40) at zevent_hooks.c:14
#2 0xb7f20aef in process_socket (p=0x832ca50, sock=0x832ca90, cs=0x832cc40, my_child_num=0, my_thread_num=15)
at zevent.c:401
#3 0xb7f2145b in worker_thread (thd=0x8305f38, dummy=0x8307b5

at zevent.c:788
#4 0xb7f13736 in dummy_worker (opaque=0x8305f3

at threadproc/unix/thread.c:142
#5 0xb7d0f4ff in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#6 0xb7e4149e in clone () from /lib/tls/i686/cmov/libc.so.6
(gdb)

进程结构，类似如下：

QUOTE:
$ pstree -p 9788
echosvc(978

───echosvc(9789)─┬─{echosvc}(9791)
├─{echosvc}(9792)
├─{echosvc}(9793)
├─{echosvc}(9794)
├─{echosvc}(9795)
├─{echosvc}(9796)
├─{echosvc}(9797)
├─{echosvc}(979


├─{echosvc}(9799)
├─{echosvc}(9800)
├─{echosvc}(9801)
├─{echosvc}(9802)
├─{echosvc}(9803)
├─{echosvc}(9804)
├─{echosvc}(9805)
├─{echosvc}(9806)
└─{echosvc}(9807)

对于多个工作进程下图的情况则需要使用attach的方式，熟练后都不会太麻烦。
下面这种结构比较少见 但nginx和apache的event的模式下就是这种结构：

QUOTE:
echosvc(9894)─┬─echosvc(9895)─┬─{echosvc}(9897)
│ ├─{echosvc}(989


│ ├─{echosvc}(9907)
│ ├─{echosvc}(990


│ ├─{echosvc}(9909)
│ └─{echosvc}(9910)
├─echosvc(9899)─┬─{echosvc}(9901)
│ ├─{echosvc}(9902)
│ ├─{echosvc}(9903)
│ ├─{echosvc}(9904)
│ ├─{echosvc}(9905)
│ └─{echosvc}(9906)
├─echosvc(9911)─┬─{echosvc}(9913)
│ ├─{echosvc}(9914)
│ ├─{echosvc}(9915)
│ ├─{echosvc}(9916)
│ ├─{echosvc}(9917)
│ └─{echosvc}(9918)
├─echosvc(9919)─┬─{echosvc}(9921)
│ ├─{echosvc}(9922)
│ ├─{echosvc}(9923)
│ ├─{echosvc}(9924)
│ ├─{echosvc}(9925)
│ └─{echosvc}(9926)
├─echosvc(9927)─┬─{echosvc}(9929)
│ ├─{echosvc}(9932)
│ ├─{echosvc}(9933)
│ ├─{echosvc}(9934)
│ ├─{echosvc}(9935)



3楼 发表于 2009-11-11 17:33 | 显示全部帖子

上面的多数派观点真浪费了贴出详细例子的苦心，本想再贴些技巧，算拉。 我觉得某个为平台开发软件熟悉这个平台的基本编辑，开发，调试环境是基本能力， 如果程序出现问题 需要紧急现场调试开发 这时候你的环境不会有vc,log可能没有记下意外， 看到很多老手在紧急情况下艰难的编辑修改代码，就像刚学打字步履维艰 很着急， 稍花点时间让自己专业一点不行吗，所谓艺不压身，很多行业的开发特点不一样 ，尤其是很多行业软件环境， 不是你想什么环境甲方就会给你，从事互联网开发方面的问题比较突出，或许因为环境比较宽容，也就更 疏于这方面的技能。 [ 本帖最后由 zhoubug 于 2009-11-11 17:40 编辑 ]