Introduction and BreakPad overview
Breakpad is a tool from Google that helps generate crash reports. From its description:
Breakpad is a library and tool suite that allows you to distribute an application to users with compiler-provided debugging information removed, record crashes in compact "minidump" files, send them back to your server, and produce C and C++ stack traces from these minidumps. Breakpad can also write minidumps on request for programs that have not crashed.
It works by stripping the debug information from the executable and saving it into "symbol files." When a crash occurs or upon request, the Breakpad client library generates the crash information in these "minidumps." The Breakpad minidump processor combines these files with the symbol files and generates a human-readable stack trace. The following picture, also from Breakpad's documentation, describes this process:
In WPE, Breakpad support was added initially for the downstream 2.28 branch by Vivek Arumugam and backported upstream. It'll be available in the soon-to-be-released 2.38 version, and the WebKit Flatpak SDK bundles the Breakpad client library since late May 2022.
Enabling Breakpad in WebKit
As a developer feature, Breakpad support is disabled by default but can be enabled by passing -DENABLE_BREAKPAD=1 to cmake when building WebKit. Optionally, you can also set -DBREAKPAD_MINIDUMP_DIR=<some path> to hardcode the path used by Breakpad to save the minidumps. If not set during build time, BREAKPAD_MINIDUMP_DIR must be set as an environment variable pointing to a valid directory when running the application. If defined, this variable also overrides the path defined during the build.
Generating the symbols
To generate the symbol files, Breakpad provides the dump_syms tool. It takes a path to the executable/library and dumps to stdout the symbol information.
Once generated, the symbol files must be laid out in a specific tree structure so minidump_stackwalk can find them when merging with the crash information. The folder containing the symbol files must match a hash code generated for that specific binary. For example, in the case of the libWPEWebKit:
$ dump_syms WebKitBuild/WPE/Release/lib/libWPEWebKit-1.1.so > libWPEWebKit-1.1.so.0.sym
$ head -n 1 libWPEWebKit-1.1.so.0.sym
MODULE Linux x86_64 A2DA230C159B97DC00000000000000000 libWPEWebKit-1.1.so.0
$ mkdir -p ./symbols/libWPEWebKit-1.1.so.0/A2DA230C159B97DC00000000000000000
$ cp libWPEWebKit-1.1.so.0.sym ./symbols/libWPEWebKit-1.1.so.0/A2DA230C159B97DC00000000000000000/
Generating the crash log
Besides the symbol files, we need a minidump file with the stack information, which can be generated in two ways. First, by asking Breakpad to create it. The other way is, well, when the application crashes :)
To generate a minidump manually, you can either call google_breakpad::WriteMiniDump(path, callback, context) or send one of the crashing signals Breakpad recognizes. The former is helpful to generate the dumps programmatically at specific points, while the signal approach might be helpful to inspect hanging processes. These are the signals Breakpad handles as crashing ones:
SIGSEGVSIGABRTSIGFPE-
SIGILL(Note: this is for illegal instruction, not the ordinarySIGKILL) SIGBUSSIGTRAP
Now, first we must run Cog:
$ BREAKPAD_MINIDUMP_DIR=/home/lauro/minidumps ./Tools/Scripts/run-minibrowser --wpe --release https://www.wpewebkit.org
Crashing the WebProcess using SIGTRAP:
$ ps aux | grep WebProcess
<SOME-PID> ... /app/webkit/.../WebProcess
$ kill -TRAP <SOME-PID>
$ ls /home/lauro/minidumps
5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp
$ file ~/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp
/home/lauro/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp: Mini DuMP crash report, 13 streams, Thu Aug 25 20:29:11 2022, 0 type
Note: In the current form, WebKit supports Breakpad dumps in the WebProcess and NetworkProcess, which WebKit spawns itself. The developer must manually add support for it in the UIProcess (the browser/application using WebKit). The exception handler should be installed as early as possible, and many programs might do some initialization before initializing WebKit itself.
Translating the crash log
Once we have a .dmp crash log and the symbol files, we can use minidump_stackwalk to show the human-readable crash log:
$ minidump_stackwalk ~/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp ./symbols/
<snip long header>
Operating system: Linux
0.0.0 Linux 5.15.0-46-generic #49-Ubuntu SMP Thu Aug 4 18:03:25 UTC 2022 x86_64
CPU: amd64
family 23 model 113 stepping 0
1 CPU
GPU: UNKNOWN
Crash reason: SIGTRAP
Crash address: 0x3e800000000
Process uptime: not available
Thread 0 (crashed)
0 libc.so.6 + 0xf71fd
rax = 0xfffffffffffffffc rdx = 0x0000000000000090
rcx = 0x00007f6ae47d61fd rbx = 0x00007f6ae4f0c2e0
rsi = 0x0000000000000001 rdi = 0x000056187adbf6d0
rbp = 0x00007fffa9268f10 rsp = 0x00007fffa9268ef0
r8 = 0x0000000000000000 r9 = 0x00007f6ae4fdc2c0
r10 = 0x00007fffa9333080 r11 = 0x0000000000000293
r12 = 0x0000000000000001 r13 = 0x00007fffa9268f34
r14 = 0x0000000000000090 r15 = 0x000056187ade7aa0
rip = 0x00007f6ae47d61fd
Found by: given as instruction pointer in context
1 libglib-2.0.so.0 + 0x585ce
rsp = 0x00007fffa9268f20 rip = 0x00007f6ae4efc5ce
Found by: stack scanning
2 libglib-2.0.so.0 + 0x58943
rsp = 0x00007fffa9268f80 rip = 0x00007f6ae4efc943
Found by: stack scanning
3 libWPEWebKit-1.1.so.0!WTF::RunLoop::run() + 0x120
rsp = 0x00007fffa9268fa0 rip = 0x00007f6ae8923180
Found by: stack scanning
4 libWPEWebKit-1.1.so.0!WebKit::WebProcessMain(int, char**) + 0x11e
rbx = 0x000056187a5428d0 rbp = 0x0000000000000003
rsp = 0x00007fffa9268fd0 r12 = 0x00007fffa9269148
rip = 0x00007f6ae74719fe
Found by: call frame info
<snip remaining trace>
Final words
This article briefly overviews enabling and using Breakpad to generate crash dumps. In a future article, we'll cover using Breakpad to get crashdumps while running WPE on embedded boards like RaspberryPis.
o/
Top comments (1)
Thank you this took me ages to figure out. I'd much rather be doing car towing service than coding but it's so satisfying once you find a great guide to help you and it becomes a breeze. You're incredible for sharing this, Lauro!