Out of Hypervisor (OoH): When Nested Virtualization Becomes Practical

TL;DR

This paper proposes Out of Hypervisor (OoH), a novel approach that exposes hardware virtualization features directly to guest OS processes, improving performance for tasks like VM live migration and process checkpointing.

Contribution

It introduces the OoH concept, demonstrating its implementation with Intel PML, and compares two designs, SPML and EPML, showing EPML's efficiency and minimal overhead.

Findings

EPML reduces CRIU checkpointing time by 14%.

EPML incurs only 0.5% overhead compared to other solutions.

OoH enables hardware features to benefit processes inside VMs.

Abstract

This paper introduces Out of Hypervisor (OoH), a new research axis close to nested virtualization. Instead of emulating a full virtual hardware inside a VM to support a hypervisor, the OoH principle is to individually expose current hypervisor-oriented hardware virtualization features to the guest OS so that its processes could also take benefit from those features. In fact, several hardware virtualization features such as Intel PML, SPP, CAT, and EPT which currently can only be used by the hypervisor also be beneficial for processes that run inside the VM. We illustrate OoH with Intel PML (Page Modification Logging), a feature which allows efficient dirty page tracking for improving VM live migration. According to the fact that dirty page tracking is at the heart of process checkpointing (CRIU) and concurrent garbage collection (Boehm), we present two OoH PML designs namely Shadow PML (SPML) and Extended PML (EPML). The former requires no hardware changes but incurs significant overhead, justifying EPML which extends PML. We evaluated and compared SPML and EPML with /proc and userfaultfd,t wo default solutions in Linux. We do this using a key-value store database as the benchmark. The results show that EPML reduces CRIU checkpointing time by about 14% while leading to a negligible overhead (of about 0.5%) compared to SPML, /proc, and userfaultfd.