LiveKd for Virtual Machine Debugging..!!!

  I pondered if there was a way to enable kernel debuggers like Windbg and Kd (part of the free Debugging Tools for Windows package that’s available in the Windows Platform SDK) to provide a local interactive view of a running system. Dave had introduced kernel debugger experiments in the 2nd edition, Inside Windows NT, that solidified the concepts presented by the book. For example, the chapter on memory management describes the page frame database, the data structure the system uses to keep track of the state of every page of physical memory, and an accompanying experiment shows how to view the actual data structure definition and contents of PFN entries on a running system using the kernel debugger. At the time, however, the only way to use Windbg and Kd to view kernel information was to attach a second computer with a serial “null modem” cable to the target system booted in debugging mode. The inconvenience of having to purchase an appropriate serial cable and configure two systems for kernel debugging meant that many readers skipped the experiments, but otherwise might have followed along and deepened their understanding if it was easier. 

After giving it some thought, I realized that I could fool the debuggers into thinking that they were looking at a crash dump file by implementing a file system filter driver that presented a “virtual” crash dump file debuggers could open. Since a crash dump file is simply a file header followed by the contents of physical memory, the driver could satisfy reads of the virtual dump file with the contents of physical memory, which the driver could easily read from the \Device\Physical Memory section object the memory manager creates. A couple of weeks later, LiveKd was born. We expanded the number of kernel debugger experiments in the book and began using LiveKd in our live Windows Internals seminars and classes as well.  LiveKd’s usage went beyond merely being an educational tool and over time became an integral part of IT pros and Microsoft support engineers troubleshooting toolkit. Microsoft even added local kernel debugging capability to Windows XP, but LiveKd can still do a few things that the native support can’t, like saving a copy of the system’s state to a dump file that can be examined on a different system and it works on Windows Vista/Server 2008 and higher without requiring the system to be booted in debug mode.

Virtual Machine Troubleshooting

The rise of virtualization has introduced a new scenario for live kernel debugging: troubleshooting virtual machines. While LiveKd works just as well inside a virtual machine as on a native installation, the ability to examine a running virtual machine without having to install and run LiveKd in the machine would add additional convenience and make it possible to troubleshoot virtual machines that are unresponsive or experiencing issues that would make it impossible to even launch LiveKd. Over the last few years I received requests from Microsoft support engineers for the feature and had started an initial investigation of the approach I’d take to add the support to LiveKd, but I hadn’t gotten around to finishing it.

Then a couple of months ago, I came across Matthieu Suiche’s LiveCloudKd tool, which enables Hyper-V virtual machine debugging and showed that there was general interest in the capability. We were so impressed that we invited Matthieu to speak about live kernel debugging and LiveCloudKd at this year’s BlueHat Security Briefings, held every year on Microsoft’s campus and taking place this week where I met him. Spurred on by LiveCloudKd, I decided it was time to finish the LiveKd enhancements and sent an email to Ken Johnson, formerly Skywing of Uninformed.org and now a developer in Microsoft’s security group (he had published articles revealing holes in 64-bit Windows “Patchguard” kernel tampering protection several times, so we hired him to help make Windows more secure), asking if he was interested in collaborating. Ken had previously contributed some code to LiveKd that enabled it to run on 64-bit Windows Vista and Windows 7 systems, so working with him was certain to speed the project – little did I know how much. He responded that he’d prototyped a tool for live virtual machine debugging a year before and thought he could incorporate it into LiveKd in a few days. Sure enough, a few days later and the beta of LiveKd 5.0 was finished, complete with the Hyper-V live debugging feature.

We picked this week to publish it to highlight Matthieu’s tool, which offers some capabilities not present in LiveKd. For example, just like it does for local machine debugging, LiveKd provides a read-only view of the target virtual machine, whereas LiveCloudKd lets you modify it as well.

LiveKd Hyper-V Debugging

LiveKd’s Hyper-V support introduces three new command line switches, -p, -hv, and -hvl:

 

When you’re want to troubleshoot a virtual machine, use –hvl to list the names and IDs of the ones that are active:

Next, use the -hv switch to specify the one you want to examine. You can use either the GUID or the virtual machine’s name, but it’s usually more convenient to use the name if it’s unique:

And that’s all there is to it. You can now perform the same commands as you can when using LiveKd on a native system, listing processes and threads, dumping memory, and generating crash dump files for later analysis.

The final switch, -p, pauses the virtual machine while LiveKd is connected. Normally, LiveKd reads pages of physical memory as they’re referenced by the debugger, which means that different pages can represent different points in time. That can lead to inconsistencies, for example, when you view a data structure on a page and then later one the structure references since second structure might have since been deleted. The pause option simply automates the Pause operation you can perform in the Hyper-V Virtual Machine management interface, giving you a frozen-in-time view of the virtual machine while you poke around.