Using Breakpad to generate crash dumps with WPE WebKit

Introduction and BreakPad overview

Breakpad is a tool from Google that helps generate crash reports. From its description:

Breakpad is a library and tool suite that allows you to distribute an application to users with compiler-provided debugging information removed, record crashes in compact "minidump" files, send them back to your server, and produce C and C++ stack traces from these minidumps. Breakpad can also write minidumps on request for programs that have not crashed.

It works by stripping the debug information from the executable and saving it into "symbol files." When a crash occurs or upon request, the Breakpad client library generates the crash information in these "minidumps." The Breakpad minidump processor combines these files with the symbol files and generates a human-readable stack trace. The following picture, also from Breakpad's documentation, describes this process:

In WPE, Breakpad support was added initially for the downstream 2.28 branch by Vivek Arumugam and backported upstream. It'll be available in the soon-to-be-released 2.38 version, and the WebKit Flatpak SDK bundles the Breakpad client library since late May 2022.

Enabling Breakpad in WebKit

As a developer feature, Breakpad support is disabled by default but can be enabled by passing -DENABLE_BREAKPAD=1 to cmake when building WebKit. Optionally, you can also set -DBREAKPAD_MINIDUMP_DIR=<some path> to hardcode the path used by Breakpad to save the minidumps. If not set during build time, BREAKPAD_MINIDUMP_DIR must be set as an environment variable pointing to a valid directory when running the application. If defined, this variable also overrides the path defined during the build.

Generating the symbols

To generate the symbol files, Breakpad provides the dump_syms tool. It takes a path to the executable/library and dumps to stdout the symbol information.

Once generated, the symbol files must be laid out in a specific tree structure so minidump_stackwalk can find them when merging with the crash information. The folder containing the symbol files must match a hash code generated for that specific binary. For example, in the case of the libWPEWebKit:

$ dump_syms WebKitBuild/WPE/Release/lib/libWPEWebKit-1.1.so > libWPEWebKit-1.1.so.0.sym
$ head -n 1 libWPEWebKit-1.1.so.0.sym
MODULE Linux x86_64 A2DA230C159B97DC00000000000000000 libWPEWebKit-1.1.so.0
$ mkdir -p ./symbols/libWPEWebKit-1.1.so.0/A2DA230C159B97DC00000000000000000
$ cp libWPEWebKit-1.1.so.0.sym ./symbols/libWPEWebKit-1.1.so.0/A2DA230C159B97DC00000000000000000/

Generating the crash log

Besides the symbol files, we need a minidump file with the stack information, which can be generated in two ways. First, by asking Breakpad to create it. The other way is, well, when the application crashes :)

To generate a minidump manually, you can either call google_breakpad::WriteMiniDump(path, callback, context) or send one of the crashing signals Breakpad recognizes. The former is helpful to generate the dumps programmatically at specific points, while the signal approach might be helpful to inspect hanging processes. These are the signals Breakpad handles as crashing ones:

• SIGSEGV
• SIGABRT
• SIGFPE
• SIGILL (Note: this is for illegal instruction, not the ordinary SIGKILL)
• SIGBUS
• SIGTRAP

Now, first we must run Cog:

$ BREAKPAD_MINIDUMP_DIR=/home/lauro/minidumps ./Tools/Scripts/run-minibrowser --wpe --release https://www.wpewebkit.org

Crashing the WebProcess using SIGTRAP:

$ ps aux | grep WebProcess
<SOME-PID> ... /app/webkit/.../WebProcess
$ kill -TRAP <SOME-PID>
$ ls /home/lauro/minidumps
5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp
$ file ~/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp
/home/lauro/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp: Mini DuMP crash report, 13 streams, Thu Aug 25 20:29:11 2022, 0 type

Note: In the current form, WebKit supports Breakpad dumps in the WebProcess and NetworkProcess, which WebKit spawns itself. The developer must manually add support for it in the UIProcess (the browser/application using WebKit). The exception handler should be installed as early as possible, and many programs might do some initialization before initializing WebKit itself.

Translating the crash log

Once we have a .dmp crash log and the symbol files, we can use minidump_stackwalk to show the human-readable crash log:

$ minidump_stackwalk ~/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp ./symbols/
<snip long header>
Operating system: Linux
                  0.0.0 Linux 5.15.0-46-generic #49-Ubuntu SMP Thu Aug 4 18:03:25 UTC 2022 x86_64
CPU: amd64
     family 23 model 113 stepping 0
     1 CPU

GPU: UNKNOWN

Crash reason:  SIGTRAP
Crash address: 0x3e800000000
Process uptime: not available

Thread 0 (crashed)
 0  libc.so.6 + 0xf71fd
    rax = 0xfffffffffffffffc   rdx = 0x0000000000000090
    rcx = 0x00007f6ae47d61fd   rbx = 0x00007f6ae4f0c2e0
    rsi = 0x0000000000000001   rdi = 0x000056187adbf6d0
    rbp = 0x00007fffa9268f10   rsp = 0x00007fffa9268ef0
     r8 = 0x0000000000000000    r9 = 0x00007f6ae4fdc2c0
    r10 = 0x00007fffa9333080   r11 = 0x0000000000000293
    r12 = 0x0000000000000001   r13 = 0x00007fffa9268f34
    r14 = 0x0000000000000090   r15 = 0x000056187ade7aa0
    rip = 0x00007f6ae47d61fd
    Found by: given as instruction pointer in context
 1  libglib-2.0.so.0 + 0x585ce
    rsp = 0x00007fffa9268f20   rip = 0x00007f6ae4efc5ce
    Found by: stack scanning
 2  libglib-2.0.so.0 + 0x58943
    rsp = 0x00007fffa9268f80   rip = 0x00007f6ae4efc943
    Found by: stack scanning
 3  libWPEWebKit-1.1.so.0!WTF::RunLoop::run() + 0x120
    rsp = 0x00007fffa9268fa0   rip = 0x00007f6ae8923180
    Found by: stack scanning
 4  libWPEWebKit-1.1.so.0!WebKit::WebProcessMain(int, char**) + 0x11e
    rbx = 0x000056187a5428d0   rbp = 0x0000000000000003
    rsp = 0x00007fffa9268fd0   r12 = 0x00007fffa9269148
    rip = 0x00007f6ae74719fe
    Found by: call frame info
<snip remaining trace>

Final words

This article briefly overviews enabling and using Breakpad to generate crash dumps. In a future article, we'll cover using Breakpad to get crashdumps while running WPE on embedded boards like RaspberryPis.

o/

Comments

[Jerry Chicen]

Thank you this took me ages to figure out. I'd much rather be doing car towing service than coding but it's so satisfying once you find a great guide to help you and it becomes a breeze. You're incredible for sharing this, Lauro!