I pondered if there was a way to enable kernel debuggers
like Windbg and Kd (part of the free Debugging Tools for Windows
package that’s available in the Windows Platform SDK) to provide a local interactive view of a running system. Dave had introduced kernel debugger experiments in the 2nd edition, Inside Windows NT,
that solidified the concepts presented by the book. For example, the
chapter on memory management describes the page frame database, the data
structure the system uses to keep track of the state of every page of
physical memory, and an accompanying experiment shows how to view the
actual data structure definition and contents of PFN entries on a
running system using the kernel debugger. At the time, however, the only
way to use Windbg and Kd to view kernel information was to attach a
second computer with a serial “null modem” cable to the target system
booted in debugging mode. The inconvenience of having to purchase an
appropriate serial cable and configure two systems for kernel debugging
meant that many readers skipped the experiments, but otherwise might
have followed along and deepened their understanding if it was easier.
After
giving it some thought, I realized that I could fool the debuggers into
thinking that they were looking at a crash dump file by implementing a
file system filter driver that presented a “virtual” crash dump file
debuggers could open. Since a crash dump file is simply a file header
followed by the contents of physical memory, the driver could satisfy
reads of the virtual dump file with the contents of physical memory,
which the driver could easily read from the \Device\Physical Memory
section object the memory manager creates. A couple of weeks later, LiveKd
was born. We expanded the number of kernel debugger experiments in the
book and began using LiveKd in our live Windows Internals seminars and
classes as well. LiveKd’s usage went beyond merely being an educational
tool and over time became an integral part of IT pros and Microsoft
support engineers troubleshooting toolkit. Microsoft even added local
kernel debugging capability to Windows XP, but LiveKd can still do a few
things that the native support can’t, like saving a copy of the
system’s state to a dump file that can be examined on a different system and it works on Windows Vista/Server 2008 and higher without requiring the system to be booted in debug mode.
Virtual Machine Troubleshooting
The
rise of virtualization has introduced a new scenario for live kernel
debugging: troubleshooting virtual machines. While LiveKd works just as
well inside a virtual machine as on a native installation, the ability
to examine a running virtual machine without having to install and run
LiveKd in the machine would add additional convenience and make it
possible to troubleshoot virtual machines that are unresponsive or
experiencing issues that would make it impossible to even launch LiveKd.
Over the last few years I received requests from Microsoft support
engineers for the feature and had started an initial investigation of
the approach I’d take to add the support to LiveKd, but I hadn’t gotten
around to finishing it.
Then a couple of months ago, I came across Matthieu Suiche’s LiveCloudKd
tool, which enables Hyper-V virtual machine debugging and showed that
there was general interest in the capability. We were so impressed that
we invited Matthieu to speak about live kernel debugging and LiveCloudKd
at this year’s BlueHat Security Briefings,
held every year on Microsoft’s campus and taking place this week where I
met him. Spurred on by LiveCloudKd, I decided it was time to finish the
LiveKd enhancements and sent an email to Ken Johnson, formerly Skywing
of Uninformed.org and now a
developer in Microsoft’s security group (he had published articles
revealing holes in 64-bit Windows “Patchguard” kernel tampering
protection several times, so we hired him to help make Windows more
secure), asking if he was interested in collaborating. Ken had
previously contributed some code to LiveKd that enabled it to run on
64-bit Windows Vista and Windows 7 systems, so working with him was
certain to speed the project – little did I know how much. He responded
that he’d prototyped a tool for live virtual machine debugging a year
before and thought he could incorporate it into LiveKd in a few days.
Sure enough, a few days later and the beta of LiveKd 5.0 was finished,
complete with the Hyper-V live debugging feature.
We picked
this week to publish it to highlight Matthieu’s tool, which offers some
capabilities not present in LiveKd. For example, just like it does for
local machine debugging, LiveKd provides a read-only view of the target
virtual machine, whereas LiveCloudKd lets you modify it as well.
LiveKd Hyper-V Debugging
LiveKd’s Hyper-V support introduces three new command line switches, -p, -hv, and -hvl:
When you’re want to troubleshoot a virtual machine, use –hvl to list the names and IDs of the ones that are active:
Next,
use the -hv switch to specify the one you want to examine. You can use
either the GUID or the virtual machine’s name, but it’s usually more
convenient to use the name if it’s unique:
And
that’s all there is to it. You can now perform the same commands as you
can when using LiveKd on a native system, listing processes and
threads, dumping memory, and generating crash dump files for later
analysis.
The final switch, -p, pauses the virtual machine
while LiveKd is connected. Normally, LiveKd reads pages of physical
memory as they’re referenced by the debugger, which means that different
pages can represent different points in time. That can lead to
inconsistencies, for example, when you view a data structure on a page
and then later one the structure references since second structure might
have since been deleted. The pause option simply automates the Pause
operation you can perform in the Hyper-V Virtual Machine management
interface, giving you a frozen-in-time view of the virtual machine while
you poke around.