This class provides a distilled understanding of some OS mechanisms such as memory management, segmentation, paging, interrupts, and port IO. This knowledge is directly relevant for the future Rootkits and Advanced x86: Virtualization with Intel VT-x classes.
Building upon the Introductory Intel x86 class, this class goes into more depth on topics already learned, and introduces more advanced topics that dive deeper into how Intel-based systems work.
Topics include, but are not limited to:
•Physical and virtual memory and how a limited amount of physical memory is represented as much more virtual memory through a multilevel paging system. We will also talk about memory segmentation.
•The hardware basis for kernel versus userspace separation and how software transitions between the two. This portion answers the question of why does x86 have 4 “rings”, with ring 0 being the most privileged, and ring 3 being the least.
•Hardware and software interrupts, and how they are the basis for debugging.
•Input/Output instructions and how these allow the CPU to talk to peripherals.
Example applications include showing how hardware and memory mechanisms are used for software exploits, anti-debug techniques, rootkit hiding, and direct hardware access for keystroke logging.
This material includes labs on:
•Using WinDbg to perform kernel debugging on a virtual machine (which is equally applicable for debugging a real machine.)
•Using a custom WinDbg plugin to examine the Local (memory segment) Descriptor Table (LDT), and Global (memory segment) Descriptor Table (GDT) in order to understand how Windows sets memory segment ranges and permissions for userspace and kernel space.
•Using WinDbg and the !pte command to understand how Windows organizes its paging structures which map physical memory to virtual memory.
•Investigating where exactly the XD/NX bit is set in order to make memory as non-executable (which Microsoft calls Data Execution Prevention (DEP)), to prevent some types of exploits from succeeding.
•Using the Read Timestamp Counter (RDTSC) instruction to profile code execution time. Also, using a profile of code execution time to change a program’s behavior in the presence of a debugger (e.g. executing different code if the code appears to have been stopped at a breakpoint.)
•Printing information about task state segments, which hold information that is used to find the kernel stack when an interrupt occurs.
•Watching what does and doesn’t change when a software interrupt is used to transfer control from userspace to kernel.
•Reading the Interrupt Descriptor Table (IDT) and understanding the security implications of changes to it.
•Understanding how RedPill uses the IDT in order to detect that a system is virtualized.
•Having a process read its own memory when a software breakpoint is set, in order to see how a debugger will change memory to set the breakpoint but hide the change from the user.
•Watch how hardware-based breakpoints manipulate dedicated debug registers.
•Using port input/output to access the backdoor communications channel that VMWare uses in order to send copy/paste, mouse movement, and other events in and out of a VM.
•Using port I/O in order to talk directly to the PS2 keyboard controller in order to sniff keystrokes or flash keyboard LEDs.
Knowledge of this material is strongly encouraged for future classes such as Rootkits and Advanced x86: Virtualization with Intel VT-x.
