Thursday, August 18, 2016

Xen altp2m on ARM

The introduction in one of my previous posts shortly describes the idea behind GSoC and mentions the topic I have been working on during the "Summer of Code". This post gives a general idea of altp2m on ARM and talks about my achievements.

Recap on Virtualization and Virtual Machine Introspection

Before we can talk about Xen's subsystem altp2m, we need to recap our knowledge about virtual machines (VMs) or hypervisors and Virtual Machine Introspection (VMI). Without going into details, system virtualization is about introducing isolated execution environments for entire systems -- each with its own operating system (OS)[0]. Similar to x86, the ARM architecture provides its own set of hardware virtualization extensions that simplifies the process of system virtualization. These virtualization extensions strongly differ from architecture to architecture but have the following properties in common: An additional, high privileged mode that is specially designed for the hypervisor (VMX-root on x86; HYP mode on ARM) and the virtualization of the guest's physical memory through a hardware supported set of second level address translation tables (EPT/NPT on x86; 2nd stage translation tables on ARM). Most common hypervisors, including the open source KVM and Xen hypervisor, make use of these virtualization extensions to virtualize guest systems.

Virtualization technology entails various benefits, such as flexibility and resource utilization. Besides, in the recent years, virtualization technology has been increasingly leveraged for security purposes. In this context, security applications were moved out of the operating systems into a hypervisor with a smaller attack surface and higher privileges to perform Virtual Machine Introspection[1]. In this way, the omniscient character of hypervisors lends security applications the necessary means for observation, analysis, and control of guest virtual machines. The term VMI has been coined by Garfinkel et al.[1] and used since 2003 to describe the ability of analyzing and manipulating the guest's state from the outside of the VM. To prevent, e.g., split-personality malware to register that it is being analyzed, the entire VMI process must be performed in a stealthy way.

By its nature, VMI achieves a high degree of stealthiness as it operates from a level below the actual operating system. Yet, when it comes to VMI-based dynamic (malware) analysis or debuggers, the guest system needs to be modified. For instance, instead of using hardware breakpoints that are limited in number and potentially used by the guest system itself (for benign or obfuscation purposes), one could apply software breakpoints or perform further instrumentation of a specific application in question so that the application eventually traps into the underlying hypervisor for monitoring purposes. Obviously, this mechanism almost screams for exposure. In this scenario a simple integrity checker might be sufficient to reveal the fact that an application is being monitored or at least has been modified. In the face of a malicious application, such exposures would potentially lead to a different behavior of the malware instance itself impeding analysis. In case of benign OS-level integrity checkers, such exposure might even lead to a system crash, as it is the case with Windows and PatchGuard. Consequently, these issues demand a mechanism that cannot be revealed that easily. And this is where Xen's subsystem altp2m comes into play.

Xen altp2m and its application

The Xen subsystem altp2m stands for alternate p2m. In this context, the term p2m refers to a second level address translation table translating guest physical addresses (p) to host physical or in Xen's jargon machine addresses (m). Usually, there is one p2m table per guest domain that is managed by Xen. This has been changed with alternate p2m tables. On x86 or more precisely on modern Intel architectures, a p2m table can be regarded as an EPT table represented by the EPTP pointer within the hardware defined data structure VMCS -- whereas the VMCS holds the host's and the guest's state and further VM control information. The interesting thing about the VMCS is that it is capable of holding up to 512 EPTPs. That is, one could define up to 512 EPT tables (and hence p2m's). Besides, the Intel architecture allows to dynamically switch between the EPTs by means of the VMFUNC instruction and/or the recent Virtualization Exceptions (#VE). That is, the idea behind altp2m on Xen is to maintain multiple (currently up to 10) p2m's representing different guest memory views.

Guest physical to machine physical address translation through p2m (left) and altp2m (right). Both figures have been taken from [2].

One example of how these views can be used in practice is given by DRAKVUF. DRAKVUF first duplicates host physical or machine code-pages of an application executing inside a specific domain (VM). Then, DRAKVUF injects, e.g., previously discussed software breakpoints into one of this page, while the other one remains integer. By employing Xen's altp2m, DRAKVUF defines two guest memory views by marking the modified machine physical page as execute-only and the integer page as read-only. That is, prohibited read accesses on the execute-only page induce an EPT violation. The same applies to instruction fetches on the read-only page. An EPT violation is then handled either by the underlying hypervisor or even directly within the guest context (through the #VE) by switching the memory views. In this simplified scenario, read accesses performed, e.g., by integrity checkers are satisfied by providing the integer view, while the guest continues execution of the modified, shadow copy of this page. This way, previously discussed software breakpoints or even further instrumentation can be effectively hidden from the guest.

The VMM maintains a shadow copy of the original code-page providing two different views of the guest memory. The figure has been taken from [2].


Alternate p2m on ARM

As opposed to the x86 architecture, ARM does not provide hardware support for p2m switching. Nevertheless, a similar behavior can be emulated in software, which is exactly what I did during my Summer of Code. While trying to preserve the original functionality of altp2m, I ported the entire altp2m code base to ARM and adjusted the implementation according to requests of the open source community. The implemented altp2m functionality thus acts as a foundation for DRAKVUF on ARM.

During the GSoC development period, I have submitted three patch series, which can be found in the xen-devel mailing list archieves:


The code to the patches can be found on my github repository:


While the patches have not yet been merged into the Xen mainline code base, we hope that it will be part of the upcoming Xen v4.8 release.

[0] J. Smith and R. Nair. Virtual Machines: Versatile Platforms for Systems and Processes. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2005.
[1] T. Garfinkel and M. Rosenblum. A Virtual Machine Introspection Based Architecture for Intrusion Detection. In: NDSS. Vol. 3. 2003, pp. 191–206.
[2] Stealthy monitoring with Xen altp2m
Posted by at