█ SecBSD: Crash Reports
Minimum information for kernel problems

Familiarize yourself with the general bug reporting procedures first. All of that will apply. When reporting a kernel panic or crash, please remember:

- We need the console output on the screen. Capture it and save it. Serial consoles are best, but if you are on a VGA console you can scroll the console back and take readable pictures with a phone or camera.
- If the kernel panicked we need the traceback. It may be displayed on the screen. If you are at a ddb> prompt, type `trace`. If you are running SMP, use the `mach ddbcpu N` command for each of the N processors you have and repeat the `trace` command for each processor.
- We need the process list. Use the command `ps` to get that.

Reports without the above information are useless. This is the minimum we need to be able to track down the issue.

Additional information you can send

In some situations more information is desirable. Below are outlined some additional steps you can take in certain situations:

- If your crash appears to involve filesystems. The following additional things would be helpful
  - The output of the ddb> command `show uvm`
  - The output of the ddb> command `show bcstats`
  - The output of the `mount` command from your running machine, so we know what filesystems are mounted and how.
- ... XXX boot crash? XXX
- ... XXX show regs? XXX

Lost the panic message?

Under some circumstances, you may lose the very first message of a panic, stating the reason for the panic.

    ddb> show panic
    0:      kernel: page fault trap, code=0
    ddb>

Note for SMP systems

You should get a trace from each processor as part of your report:

    ddb{0}> trace
    pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226
    fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5
    fxp_intr(d0a62000) at fxp_intr+0x1e7
    Xintr_ioapic0() at Xintr_ioapic0+0x6d
    --- interrupt ---
    idle_loop+0x21:
    ddb{0}> machine ddbcpu 1
    Stopped at      Debugger+0x4:   leave
    ddb{1}> trace
    Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4
    i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb
    i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x
    4a
    Xintripi() at Xintripi+0x47
    --- interrupt ---
    i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37
    Xintrltimer() at Xintrltimer+0x47
    --- interrupt ---
    idle_loop+0x21:
    ddb{1}>

Repeat the `machine ddbcpu x` followed by `trace` for each processor in your machine.

How do I gather further information from a kernel crash?

A typical kernel crash on SecBSD might look like this:

    kernel: page fault trap, code=0
    Stopped at    pf_route+0x263:        mov     0x40(%edi),%edx
    ddb>

This crash happened at offset `0x263` in the function `pf_route`.

The first command to run from the ddb(4) prompt is `trace`:

    ddb> trace
    pf_route(e28cb7e4,e28bc978,2,1fad,d0b8b120) at pf_route+0x263
    pf_test(2,1f4ad,e28cb7e4,b4c1) at pf_test+0x706
    pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at pf_route+0x207
    pf_test(2,d0a65440,e28cbb00,d023c282) at pf_test+0x706
    ip_output(d0b6a200,0,0,0,0) at ip_output+0xb67
    icmp_send(d0b6a200,0,1,a012) at icmp_send+0x57
    icmp_reflect(d0b6a200,0,1,0,3) at icmp_reflect+0x26b
    icmp_input(d0b6a200,14,0,0,d0b6a200) at icmp_input+0x42c
    ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at ipv4_input+0x6eb
    ipintr(10,10,e289f140,e289f140,e28cbd38) at ipintr+0x8d
    Bad frame pointer: 0xe28cbcac
    ddb>

This tells us what function calls lead to the crash.

To find out the particular line of C code that caused the crash, you can do the following:

Find the source file where the crashing function is defined. In this example, that would be `pf_route()` in `/sys/net/pf.c`. Use objdump(1) to get the disassembly:

    $ cd /sys/arch/$(uname -m)/compile/GENERIC
    $ objdump -dlr obj/pf.o >/tmp/pf.dis

In the output, grep for the function name:

    $ grep "<pf_route>:" /tmp/pf.dis
    00007d88 <pf_route>:

Take this first hex number `7d88` and add the offset `0x263` from the `Stopped at` line:

    $ printf '%x\n' $((0x7d88 + 0x263))
    7feb

Scroll down to the line `7feb`. The assembler instruction should match the one quoted in the `Stopped at` line. Then scroll up to the nearest C line number:

    $ more /tmp/pf.dis
    /sys/net/pf.c:3872
        7fe7:       0f b7 43 02             movzwl 0x2(%ebx),%eax
        7feb:       8b 57 40                mov    0x40(%edi),%edx
        7fee:       39 d0                   cmp    %edx,%eax
        7ff0:       0f 87 92 00 00 00       ja     8088 <pf_route+0x300>

So, it's precisely line `3872` of `pf.c` that crashes:

    $ nl -ba /sys/net/pf.c | sed -n 3872p
      3872		if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {

The kernel that produced the crash output and the object file for objdump must be compiled from the exact same source file, otherwise the offsets won't match.

If you provide both the ddb trace output and the relevant objdump section, that's very helpful.