Setting up OS X for kernel debugging

Here are the steps to prepare an OS X system for debugging kernel modules such as drivers. The instructions are deliberately brief to save the reader from tons of text.
If you search the web, you will come across may resources that point to using GDB to debug the kernel. These resources were written in the early days of OS X when versions 10.4 or 10.5 were prevalent. Since around v10.9 (or thereabouts), Mac moved to lldb as their debugger of choice and have continually improved it to enable easier setup & more efficient workflow.

This short post has the instructions to setting up lldb for debugging OS X kernel versions 10.11 and above.

My setup is using a Parallels VM as the target on the development system. But it should work on a separate hardware that is connected via network to the host.

Debuggee is the target system where drivers are deployed and tested. This also referred to as the target system. Debugger is the host system, typically running your development tools with the driver source code that connects to the target system through network. This is also referred to as the host.

1. Download the Kernel Debug Kit matching the target VM OS, including its build number (clicking on the OS About box's version number field would reveal the build number next to it) and install it (installs to /Library/Developer/KDKs). Copy kernel.development or kernel.debug to the target’s kernel file folder. (Defaults to /System/Library/Kernels) 

2. Define the VM’s boot arguments as:

   devices.mac.boot_args="debug=0x146 -v watchdog=0 pmuflags=1 kcsuffix=development"
   

   [debug=0x146 ]is to enable NMI as well as wait for debugger to connect on crash.

-v enable verbose mode boot which shows Linux like text messages during boot up process.

[watchdog=0] disables macOS watchdog from firing when returning from the kernel debugger.

Kcsuffix specifies the kernel to load during bootup. [kcsuffix=development]instructs bootloaded to start kernel.development as the OS kernel.

If you're using a separate machine, use nvram command:

    $ sudo nvram boot-args="debug=0x146 -v watchdog=0 pmuflags=1 kcsuffix=development"

1. Ensure that your driver kext is loaded by attaching the associated USB device to the VM. Only when a driver kext is loaded that we can add its symbols in lldb.

2. Start lldb with kernel as the debug target

   lldb /Library/Developer/KDKs/KDK_10.14.1_18B75.kdk/System/Library/Kernels/kernel.development
   

3. Issue an NMI on the target. On a VM this can be done through the dtrace utility:

   $ sudo dtrace -w -n “BEGIN { breakpoint(); }”
   

   If the OS picked up the boot-args that you set earlier, system will halt waiting for debugger connection. OTOH, if the boot-args were not properly picked by the kernel, your system will reset.

4. From the Host, connect to the waiting target (waiting for debugger as it’s using the development kernel):

   (lldb) kdp-remote <target IP address>
   

5. Add the kext symbols to lldb

   (lldb) target symbols add ~/Library/Developer/Xcode/DerivedData/mydriver-fyooammjpvckpndqflrqugeldfjm/Build/Products/Debug/mydriver.kext.dSYM
   

6. Set the necessary breakpoints and issue continue to run the halted target.

7. Wait for breakpoints to hit and do source level tracing.

Happy days!