Tips for debugging on Linux
2016-10-13 11:12
489 查看
Contents
Symbolized stack traceGDB
Basic browser process debugging
Allowing attaching to foreign processes
Multiprocess Tricks
Printing Chromium types
Graphical Debugging Aid for Chromium Views
Faster startup
Core files
Breakpad minidump files
Running Tests
Browser tests
Layout tests
UI tests
Timeouts
To replicate Window Manager setup on the bots
BROWSER_WRAPPER
Replicating Trybot Slowness
Logging
Seeing all LOG(foo) messages
Seeing IPC debug messages
Using valgrind
Profiling
i18n
Breakpad
Drag and Drop
Tracking Down Bugs
Isolating Regressions
Screen recording for bug reports
Version-specific issues
Google Chrome
Ubuntu Chromium
Fedora's Chromium
Xlib
Window Managers
Mozilla Tips
Symbolized stack trace
The sandbox can interfere with the internal symbolizer. Use--no-sandbox(but keep this temporary) or an external symbolizer (see
tools/valgrind/asan/asan_symbolize.py).
Generally, do not use
--no-sandboxon waterfall bots, sandbox testing is needed. Talk to
security@chromium.org.
GDB
GDB-7.7 is required in order to debug Chrome on Linux.Any prior version will fail to resolve symbols or segfault.
Basic browser process debugging
gdb -tui -ex=r --args out/Debug/chrome --disable-seccomp-sandbox \ http://google.com
Allowing attaching to foreign processes
On distributions that use theYama LSM (that includes Ubuntu and Chrome OS), process A can attach to process B only if A is an ancestor of B.
You will probably want to disable this feature by using
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
If you don‘t you’ll get an error message such as “Could not attach to process”.
Note that you'll also probably want to use
--no-sandbox, as explained below.
Multiprocess Tricks
Getting renderer subprocesses into
gdb
Since Chromium itself spawns the renderers, it can be tricky to grab a particular with gdb. This command does the trick:chrome --no-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb --args'
The
--no-sandboxflag is needed because otherwise the seccomp sandbox will kill the renderer process on startup, or the setuid sandbox will prevent xterm‘s execution. The “xterm” is necessary or gdb will run in the current terminal, which can
get particularly confusing since it’s running in the background, and if you‘re also running the main process in gdb, won’t work at all (the two instances will fight over the terminal). To auto-start the renderers in the debugger, send the “run” command to
the debugger:
chrome --no-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb \ -ex run --args
If you're using Emacs and
M-x gdb, you can do
chrome "--renderer-cmd-prefix=gdb --args"
Note: using the
--renderer-cmd-prefixoption bypasses the zygote launcher, so the renderers won‘t be sandboxed. It is generally not an issue, except when you are trying to debug interactions with the sandbox. If that’s what you
are doing, you will need to attach your debugger to a running renderer process (see below).
You may also want to pass
--disable-hang-monitorto suppress the hang monitor, which is rather annoying.
You can also use
--renderer-startup-dialogand attach to the process in order to debug the renderer code. Go to
http://www.chromium.org/blink/getting-started-with-blink-debugging for more information on how this can be done.
Choosing which renderers to debug
If you are starting multiple renderers then the above means that multiple gdb's start and fight over the console. Instead, you can set the prefix to point to this shell script:#!/bin/sh echo "**** Child $$ starting: y to debug" read input if [ "$input" = "y" ] ; then gdb --args $* else $* fi
Selective breakpoints
When debugging both the browser and renderer process, you might want to have separate set of breakpoints to hit. You can use gdb's command files to accomplish this by putting breakpoints in separate files and instructing gdb to load them.gdb -x ~/debug/browser --args chrome --no-sandbox --disable-hang-monitor \ --renderer-cmd-prefix='xterm -title renderer -e gdb -x ~/debug/renderer \ --args '
Also, instead of running gdb, you can use the script above, which let's you select which renderer process to debug. Note: you might need to use the full path to the script and avoid
$HOMEor
~/.
Connecting to a running renderer
Usuallyps aux | grep chromewill not give very helpful output. Try
pstree -p | grep chrometo get something like
| |-bash(21969)---chrome(672)-+-chrome(694) | | |-chrome(695)---chrome(696)-+-{chrome}(697) | | | \-{chrome}(709) | | |-{chrome}(675) | | |-{chrome}(678) | | |-{chrome}(679) | | |-{chrome}(680) | | |-{chrome}(681) | | |-{chrome}(682) | | |-{chrome}(684) | | |-{chrome}(685) | | |-{chrome}(705) | | \-{chrome}(717)
Most of those are threads. In this case the browser process would be 672 and the (sole) renderer process is 696. You can use
gdb -p 696to attach. Alternatively, you might find out the process ID from Chrome's built-in Task Manager (under the Tools menu). Right-click on the Task Manager, and enable “Process ID” in the list of columns.
Note: by default, sandboxed processes can't be attached by a debugger. To be able to do so, you will need to pass the
--allow-sandbox-debuggingoption.
If the problem only occurs with the seccomp sandbox enabled (and the previous tricks don‘t help), you could try enabling core-dumps (see the
Core files section). That would allow you to get a backtrace and see some local variables, though you won’t be able to step through the running program.
Note: If you‘re interested in debugging LinuxSandboxIPC process, you can attach to 694 in the above diagram. The LinuxSandboxIPC process has the same command line flag as the browser process so that it’s easy to identify it if you run
pstree -pa.
Getting GPU subprocesses into gdb
Use--gpu-launcherflag instead of
--renderer-cmd-prefixin the instructions for renderer above.
Getting
browser_tests
launched browsers into gdb
Use environment variable BROWSER_WRAPPERinstead of
--renderer-cmd-prefixswitch in the instructions above.
Example:
BROWSER_WRAPPER='xterm -title renderer -e gdb --eval-command=run \ --eval-command=quit --args' out/Debug/browser_tests --gtest_filter=Print
Plugin Processes
Same strategies as renderers above, but the flag is called--plugin-launcher:
chrome --plugin-launcher='xterm -e gdb --args'
Note: For now, this does not currently apply to PPAPI plugins because they currently run in the renderer process.
Single-Process mode
Depending on whether it‘s relevant to the problem, it’s often easier to just run in “single process” mode where the renderer threads are in-process. Then you can just run gdb on the main process.gdb --args chrome --single-process
Currently, the
--disable-gpuflag is also required, as there are known crashes that occur under TextureImageTransportSurface without it. The crash described in
http://crbug.com/361689 can also sometimes occur, but that crash can be continued from without harm.
Note that for technical reasons plugins cannot be in-process, so
--single-processonly puts the renderers in the browser process. The flag is still useful for debugging plugins (since it‘s only two processes instead of three) but you’ll still
need to use
--plugin-launcheror another approach.
Printing Chromium types
gdb 7 lets us use Python to write pretty-printers for Chromium types. The directorytools/gdb/contains a Python gdb scripts useful for Chromium code. There are similar scripts
in WebKit (in fact, the Chromium script relies on using it with the WebKit one).
To include these pretty-printers with your gdb, put the following into
~/.gdbinit:
python import sys sys.path.insert(0, "<path/to/chromium/src>/third_party/WebKit/Tools/gdb/") import webkit sys.path.insert(0, "<path/to/chromium/src>/tools/gdb/") import gdb_chrome
Pretty printers for std types shouldn‘t be necessary in gdb 7, but they’re provided here in case you're using an older gdb. Put the following into
~/.gdbinit:
# Print a C++ string. define ps print $arg0.c_str() end # Print a C++ wstring or wchar_t*. define pws printf "\"" set $c = (wchar_t*)$arg0 while ( *$c ) if ( *$c > 0x7f ) printf "[%x]", *$c else printf "%c", *$c end set $c++ end printf "\"\n" end
More STL GDB macros
Graphical Debugging Aid for
Chromium Views
The following link describes a tool that can be used on Linux, Windows and Mac under GDB.graphical_debugging_aid_chromium_views
Faster startup
Use thegdb-add-indexscript (e.g.
build/gdb-add-index out/Debug/browser_tests)
Only makes sense if you run the binary multiple times or maybe if you use the component build since most
.sofiles won't require reindexing on a rebuild.
See https://groups.google.com/a/chromium.org/forum/#!searchin/chromium-dev/gdb-add-index/chromium-dev/ELRuj1BDCL4/5Ki4LGx41CcJ for more info.
You can improve GDB load time significantly at the cost of link time by splitting symbols from the object files. In GN, set
use_debug_fission=falsein your “gn args”.
Core files
ulimit -c unlimitedshould cause all Chrome processes (run from that shell) to dump cores, with the possible exception of some sandboxed processes.
Some sandboxed subprocesses might not dump cores unless you pass the
--allow-sandbox-debuggingflag.
If the problem is a freeze rather than a crash, you may be able to trigger a core-dump by sending SIGABRT to the relevant process:
kill -6 [process id]
Breakpad minidump files
Seelinux_minidump_to_core.md
Running Tests
Many of our tests bring up windows on screen. This can be annoying (they steal your focus) and hard to debug (they receive extra events as you mouse over them). Instead, useXvfbor
Xephyrto run a nested X session to debug them, as outlined on
layout_tests_linux.md.
Browser tests
By default thebrowser_testsforks a new browser for each test. To debug the browser side of a single test, use a command like
gdb --args out/Debug/browser_tests --single_process --gtest_filter=MyTestName
note the underscore in
single_process-- this makes the test harness and browser process share the outermost process.
To debug a renderer process in this case, use the tips above about renderers.
Layout tests
Seelayout_tests_linux.md for some tips. In particular, note that it‘s possible to debug a layout test via
sshing to a Linux box; you don’t need anything on screen if you use
Xvfb.
UI tests
UI tests are run in forked browsers. Unlike browser tests, you cannot do any single process tricks here to debug the browser. See below aboutBROWSER_WRAPPER.
To pass flags to the browser, use a command line like
--extra-chrome-flags="--foo --bar".
Timeouts
UI tests have a confusing array of timeouts in place. (Pawel is working on reducing the number of timeouts.) To disable them while you debug, set the timeout flags to a large value:--test-timeout=100000000
--ui-test-action-timeout=100000000
--ui-test-terminate-timeout=100000000
To replicate Window Manager
setup on the bots
Chromium try bots and main waterfall's bots run tests under Xvfb&openbox combination. Xvfb is an X11 server that redirects the graphical output to the memory, and openbox is a simple window manager that is running on top of Xvfb. The behavior of openboxis markedly different when it comes to focus management and other window tasks, so test that runs fine locally may fail or be flaky on try bots. To run the tests on a local machine as on a bot, follow these steps:
Make sure you have openbox:
apt-get install openbox
Start Xvfb and openbox on a particular display:
Xvfb :6.0 -screen 0 1280x1024x24 & DISPLAY=:6.0 openbox &
Run your tests with graphics output redirected to that display:
DISPLAY=:6.0 out/Debug/browser_tests --gtest_filter="MyBrowserTest.MyActivateWindowTest"
You can look at a snapshot of the output by:
xwd -display :6.0 -root | xwud
Alternatively, you can use testing/xvfb.py to set up your environment for you:
testing/xvfb.py out/Debug out/Debug/browser_tests \ --gtest_filter="MyBrowserTest.MyActivateWindowTest"
BROWSER_WRAPPER
You can also get the browser under a debugger by setting theBROWSER_WRAPPERenvironment variable. (You can use this for
browser_teststoo, but see above for discussion of a simpler way.)
BROWSER_WRAPPER='xterm -e gdb --args' out/Debug/browser_tests
Replicating Trybot Slowness
Trybots are pretty stressed, and can sometimes expose timing issues you can't normally reproduce locally.You can simulate this by shutting down all but one of the CPUs (http://www.cyberciti.biz/faq/debian-rhel-centos-redhat-suse-hotplug-cpu/) and running a
CPU loading tool (e.g., http://www.devin.com/lookbusy/). Now run your test. It will run slowly, but any flakiness found by the trybot should replicate locally now - and often nearly 100% of the time.
Logging
Seeing all LOG(foo) messages
Default log level hidesLOG(INFO). Run with
--log-level=0and
--enable-logging=stderrflags.
Newer versions of chromium with VLOG may need --v=1 too. For more VLOG tips, see
the chromium-dev thread.
Seeing IPC debug messages
Run withCHROME_IPC_LOGGING=1eg.
CHROME_IPC_LOGGING=1 out/Debug/chrome
or within gdb:
set environment CHROME_IPC_LOGGING 1
If some messages show as unknown, check if the list of IPC message headers in
chrome/common/logging_chrome.cc is up to date. In case this file reference goes out of date, try looking for usage of macros like
IPC_MESSAGE_LOG_ENABLEDor
IPC_MESSAGE_MACROS_LOG_ENABLED.
Using valgrind
To run valgrind on the browser and renderer processes, with our suppression file and flags:$ cd $CHROMIUM_ROOT/src $ tools/valgrind/valgrind.sh out/Debug/chrome
You can use valgrind on chrome and/or on the renderers e.g
valgrind --smc-check=all ../sconsbuild/Debug/chromeor by passing valgrind as the argument to
--render-cmd-prefix.
Beware that there are several valgrind “false positives” e.g. pickle, sqlite and some instances in webkit that are ignorable. On systems with prelink and address space randomization (e.g. Fedora), you may also see valgrind errors in libstdc++ on startup
and in gnome-breakpad.
Valgrind doesn't seem to play nice with tcmalloc. To disable tcmalloc set the GN arg:
use_allocator="none"
and rebuild.
Profiling
See https://sites.google.com/a/chromium.org/dev/developers/profiling-chromium-and-webkit andLinux Profiling.
i18n
We obey your system locale. Try something like:LANG=ja_JP.UTF-8 out/Debug/chrome
If this doesn‘t work, make sure that the
LANGUAGE,
LC_ALLand
LC_MESSAGEenvironment variables aren’t set -- they have higher priority than LANG in the order listed. Alternatively, just do this:
LANGUAGE=fr out/Debug/chrome
Note that because we use GTK, some locale data comes from the system -- for example, file save boxes and whether the current language is considered RTL. Without all the language data available, Chrome will use a mixture of your system language and the language
you run Chrome in.
Here's how to install the Arabic (ar) and Hebrew (he) language packs:
sudo apt-get install language-pack-ar language-pack-he \ language-pack-gnome-ar language-pack-gnome-he
Note that the
--langflag does not work properly for this.
On non-Debian systems, you need the
gtk20.mofiles. (Please update these docs with the appropriate instructions if you know what they are.)
Breakpad
See the last section ofLinux Crash Dumping; you need to set a gyp variable and an environment variable for the crash dump tests to work.
Drag and Drop
If you break in a debugger during a drag, Chrome will have grabbed your mouse and keyboard so you won't be able to interact with the debugger! To work around this, run viaXephyr. Instructions for how to use
Xephyrare on the
Running layout tests on Linux page.
Tracking Down Bugs
Isolating Regressions
Old builds are archived here: http://build.chromium.org/buildbot/snapshots/chromium-rel-linux/ (TODO: does not exist).tools/bisect-builds.pyin the tree automates bisecting through the archived builds. Despite a computer science education, I am still amazed how quickly binary search will find its target.
Screen recording for bug reports
sudo apt-get install gtk-recordmydesktop
Version-specific issues
Google Chrome
Google Chrome binaries don't include symbols. Googlers can read where to get symbols fromthe Google-internal wiki.
Ubuntu Chromium
Since we don‘t build the Ubuntu packages (Ubuntu does) we can’t get useful backtraces from them. Direct users tohttps://wiki.ubuntu.com/Chromium/Debugging
Fedora's Chromium
Like Ubuntu, but direct users to https://fedoraproject.org/wiki/TomCallaway/Chromium_DebugXlib
If you're trying to track down X errors like:The program 'chrome' received an X Window System error. This probably reflects a bug in the program. The error was 'BadDrawable (invalid Pixmap or Window parameter)'.
Some strategies are:
pass
--syncon the command line to make all X calls synchronous
run chrome via xtrace
turn on IPC debugging (see above section)
Window Managers
To test on various window managers, you can use a nested X server likeXephyr. Instructions for how to use
Xephyrare on the
Running layout tests on Linux page.
If you need to test something with hardware accelerated compositing (e.g., compiz), you can use
Xgl(
sudo apt-get install xserver-xgl). E.g.:
Xgl :1 -ac -accel glx:pbuffer -accel xv:pbuffer -screen 1024x768
相关文章推荐
- 20 Killer Perl Programming Tips for Beginners on UNIX / Linux OS
- Linux: 提示 Tips for shell programming
- Single Sign On for Windows and Linux
- Unable to start debugging on the web server. The web server is not configured correctly. See help for common configuration errors. Running the web page outside of the debugger may provide further information.
- Debugging Memory on Linux
- Setting Up A PXE Install Server For Multiple Linux Distributions and WinPE On Debian Lenny
- [转] Intel release open source OpenGL API-based 3D graphics drivers for Linux on 965 Express chipset
- Tips for Optimization Linux Memory Usage
- Knoppix Hacks: Tips and Tools for Using the Linux Live CD to Hack, Repair, and Enjoy Your PC
- Linux Server Hacks, Volume Two: Tips & Tools for Connecting, Monitoring, and Troubleshooting
- How to improve the performance networking (TCP & UDP) on Linux 2.4+ for high-bandwidth applications
- 使用 OProfile for Linux on POWER 识别性能瓶颈
- MySQL for Linux on POWER,第 1 2 部分: 数据库创建简介
- Unable to start debugging on the web server. You do not have permission to debug the application. The URL for this project is in
- [已解决]在VMware-server for linux 下装centos错误:an error has occurred. - no valid devices were found on which to create new file system
- Top 10 Tips for Linux Users
- A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux
- DB2 SQL PL : Essential Guide for DB2 UDB on Linux, UNIX, Windows, i5/OS, and z/OS, Second Edition
- Linux kernel tweaking for performance and security on a busy Linux
- Development tools for C++ on Linux development environment