Extracting Secrets from Encrypted Virtual Machines

TL;DR

This paper presents a method for hypervisors to efficiently and stealthily extract specific secrets like cryptographic keys from AMD SEV-encrypted virtual machines by inferring secret locations and analyzing memory regions in real-time.

Contribution

It introduces a novel approach combining activity inference and targeted memory extraction to quickly locate and steal secrets from encrypted VMs on current hardware.

Findings

Enables rapid identification of secret-containing memory regions.

Increases success rate of secret theft without extensive memory analysis.

Operates stealthily, reducing detection risk.

Abstract

AMD SEV is a hardware extension for main memory encryption on multi-tenant systems. SEV uses an on-chip coprocessor, the AMD Secure Processor, to transparently encrypt virtual machine memory with individual, ephemeral keys never leaving the coprocessor. The goal is to protect the confidentiality of the tenants' memory from a malicious or compromised hypervisor and from memory attacks, for instance via cold boot or DMA. The SEVered attack has shown that it is nevertheless possible for a hypervisor to extract memory in plaintext from SEV-encrypted virtual machines without access to their encryption keys. However, the encryption impedes traditional virtual machine introspection techniques from locating secrets in memory prior to extraction. This can require the extraction of large amounts of memory to retrieve specific secrets and thus result in a time-consuming, obvious attack. We present an approach that allows a malicious hypervisor quick identification and theft of secrets, such as TLS, SSH or FDE keys, from encrypted virtual machines on current SEV hardware. We first observe activities of a virtual machine from within the hypervisor in order to infer the memory regions most likely to contain the secrets. Then, we systematically extract those memory regions and analyze their contents on-the-fly. This allows for the efficient retrieval of targeted secrets, strongly increasing the chances of a fast, robust and stealthy theft.

Tables

Table 1. Table 1 . Statistics for the median length of the observation and search phases, and for the median number of extracted pages with the median absolute deviation for the different scenarios and load levels.

\pbox2cmUse Case	\pbox2cmLoad Level	\pbox2cmMedian Page No	\pbox2cm    MAD Page No	Median Time
				Observ.	Search
\pbox2cmTLS (nginx)	1	102	5	1.46s	17.72s
	9	116	19	0.37s	15.48s
	17	165	69	0.32s	18.61s
	25	301	160	0.31s	32.71s
\pbox2cmTLS (Apache)	1	128	21	1.42s	21.90s
	9	137	40	0.37s	17.95s
	17	154	80	0.33s	17.44s
	25	171	109	0.32s	18.65s
FDE	1	70	8	2.43s	12.24s
	9	71	9	2.15s	9.34s
	17	70	8	2.08s	7.84s
	25	69	9	2.04s	7.37s
SSH	1	7	1	193.36s	1.33s
	9	7	1	27.23s	0.97s
	17	7	1	16.19s	0.83s
	25	7	1	14.41s	0.80s

Full text

AMD SEV; virtual machine encryption; virtual machine introspection; memory extraction; data confidentiality

††journalyear: 2019††copyright: acmlicensed††conference: Ninth ACM Conference on Data and Application Security and Privacy; March 25–27, 2019; Richardson, TX, USA††booktitle: Ninth ACM Conference on Data and Application Security and Privacy (CODASPY ’19), March 25–27, 2019, Richardson, TX, USA††price: 15.00††doi: 10.1145/3292006.3300022††isbn: 978-1-4503-6099-9/19/03††ccs: Security and privacy Virtualization and security

1. Introduction

On common multi-tenant systems, the confidentiality of sensitive Virtual Machine (VM) data depends on both the Hypervisor’s (HV) integrity and on the operator’s trustworthiness. Unfortunately, these strong requirements are prone to getting infringed by different attack vectors. Examples are attacks by other tenants exploiting software-level vulnerabilities to escape their sandboxed VMs (Microsoft, 2017; VMware, 2017; Xenproject.org Security Team, 2017), attackers with physical access conducting a memory attack, e.g., via Direct Memory Access (DMA) (Boileau, 2006; Devine and Vissian, 2009; Becher et al., 2005) or cold boot (Halderman et al., 2009), or simply a malicious operator using the HV to read the VM’s memory. In order to protect the VM’s memory in such scenarios, AMD introduced Secure Encrypted Virtualization (SEV) (Advanced Micro Devices, 2018; David Kaplan, 2017) on recent server systems. SEV is a hardware extension for main memory encryption on a per-VM granularity. With SEV enabled, AMD’s Secure Processor (SP) transparently encrypts the main memory of each VM with individual SP-bound keys. The goal is to protect VMs from memory attacks and from a malicious or compromised HV. To attest tenants that their VMs’ memory is indeed encrypted, SEV includes a cryptographic protocol to verify VM encryption on an SEV-enabled platform.

SEVered (Morbitzer et al., 2018) is a recent attack on AMD SEV, which showed that it is nevertheless possible for a HV to extract plaintext contents from SEV-encrypted VMs. SEVered exploits SEV’s missing integrity protection for VM memory pages, previously discovered by (Payer, 2016; Hetzelt and Buhren, 2017), to modify the memory mapping of a non-colluding service inside a VM. The modification causes the service to access and return an arbitrary portion of plaintext memory when serving requests. This allows an attacker in the HV to extract all the VM’s main memory in plaintext by repeatedly requesting the same resource and changing its mapping. However, the encryption prevents the attacker from locating the VM’s most valuable resources in memory prior to extraction, such as keys for Transport Layer Security (TLS), Secure Shell (SSH) or Full Disk Encryption (FDE). In the worst case, extracting those secrets requires a full dump of the VM’s memory. This can take a significant amount of time, depending on the size of the attacker-controlled resource and throughput of the service. For example, an extraction speed of about 80 KB/s was reached with web servers providing a resource covering exactly one memory page. In this scenario, it takes more than 7 hours and requires 524,288 requests to extract all memory contents of a VM with 2 GB of main memory. During this time, other clients requesting the same resource also receive arbitrary contents, making full memory extraction conspicuous.

In this paper, we show that it is possible to overcome these downsides and present an approach that makes HVs capable of quickly locating and extracting specific secrets from SEV-enabled VMs. Our approach has two phases, the observation and the retrieval phase. In the observation phase, we exploit the fact that the HV is able to observe certain events triggered by VMs. These observable events can, for instance, be page faults which the HV handles but also I/O events like network traffic or disk writes. We observe and combine such events to identify a minimal set of VM memory pages likely to contain the targeted secrets. Second, in the retrieval phase, we iteratively extract and analyze the identified set of pages on the fly until we find the targeted secret. For this phase, we use the SEVered attack, but could potentially leverage other vectors allowing memory extraction from SEV-encrypted VMs.

Our targeted extraction approach offers an inconspicuous, reliable and efficient method to steal various secrets from encrypted VMs. We demonstrate the potential of our approach by extracting TLS and SSH keys from a VM’s user space memory, and FDE keys stored in the VM’s kernel space. We conduct our experiments on an SEV-enabled EPYC processor, running Apache and nginx web servers as well as the OpenSSH server. To show that our approach can cope with real-world scenarios where VMs can be under varying levels of load, we base our experiments on a load model in which multiple independent clients concurrently access the VM’s services during our attack.

2. AMD SEV and the SEVered Attack

This section provides background information on AMD SEV, the Second Level Address Translation (SLAT) concepts of HVs, and the SEVered attack.

SEV

The AMD SEV technology allows for the transparent encryption of main memory of individual VMs. SEV primarily targets server systems and builds on the AMD Secure Memory Encryption (SME) technology, which provides transparent full main memory encryption. While the goal of SME is to protect systems against physical attacks on the main memory, SEV tries to additionally protect memory of individual VMs against attacks from other VMs and from a malicious HV. The SEV encryption is executed by a hardware AES engine located in the memory controller. The keys for the encryption are created and managed by an additional component, the AMD SP. All keys are ephemeral and never exposed to software on the main CPU. In contrast to SME, SEV uses different keys for each VM and for the HV. Additionally, a VM running on an SEV-protected system can request encryption and receive proof that its memory contents are being encrypted, which establishes trust between its owner and the remote operator. While SME was first integrated into AMD’s Ryzen CPUs, SEV was introduced onto the market with the EPYC processor family. The mainline Linux kernel provides necessary software-level support for SEV.

SLAT

AMD SEV integrates with the existing AMD hardware virtualization technologies marketed as AMD-V. An integral component of hardware virtualization is an additional address translation, often named nested paging or SLAT (Advanced Micro Devices, 2008). While non-virtualized systems simply translate virtual addresses directly to physical addresses, a hardware-virtualized system distinguishes between three different types of addresses. When the VM accesses a Guest Virtual Address (GVA), the guest-controlled first level translation translates the address to a Guest Physical Address (GPA). The GPA is then translated to a Host Physical Address (HPA) using the second-level translation controlled by the HV. SLAT is completely transparent to the VM. This allows running multiple VMs that use the same GPA space while separating them in physical memory. With SEV enabled, the first level translation from GVA to GPA in the encrypted VM is non-accessible to the HV. But the HV is still responsible for managing physical memory for its VMs and is therefore able to restrict access and change second-level mappings from GPAs to HPAs. Since there is no integrity protection in SEV, the HV can use SLAT to transparently switch a GPA to HPA mapping to a different HPA page belonging to the same VM.

SEVered

The SEVered attack (Morbitzer et al., 2018) enables a malicious or compromised HV to extract the full memory of SEV-encrypted VMs in plaintext by exploiting SEV’s missing integrity protection. SEVered requires a (non-colluding) service in the targeted VM, e.g., a web server, offering a remotely accessible resource. The first step of SEVered is to identify the HPAs, i.e., the physical pages, at which the accessible resource is located in the VM’s encrypted memory. The number of pages containing (parts of) the resource depends on the size of the resource as well as on the page size. The knowledge about the resource’s location allows SEVered to modify the VM’s GPA to HPA mappings to point to arbitrary other HPAs of the VM instead of to the service’s resource. The modified mapping causes the service to access different memory pages instead of the real resource when handling requests. In the second step, SEVered repeatedly requests the resource while remapping the memory (using the SLAT feature). This leads to the iterative extraction of an encrypted VM’s full memory contents. The throughput of SEVered depends on the service and on the amount of pages that can be extracted at once. Our attack uses SEVered for the extraction of main memory from SEV-encrypted VMs. Like SEVered, our attack neither requires breaking SEV’s cryptographic primitives, nor control over the SP. Likewise, our attack requires control over the HV, i.e., a malicious administrator or a compromise of the HV. We refer to (Morbitzer et al., 2018) for further information about SEVered.

3. Finding and Extracting Secrets

Our concept for the targeted extraction of secrets from SEV-encrypted VMs has two phases: In the first phase, we start our attack by observing the page accesses of the targeted VM in the HV until an event occurs which indicates the VM’s recent use of the targeted secret. In the second phase, we search the VM’s memory for the secret by systematically extracting and analyzing the set of observed pages accesses. This section describes both phases in detail.

3.1. Observation Phase

The goal of the observation phase is to narrow down the set of VM memory pages possibly containing the targeted secret. We start the phase at an arbitrary point in time by tracking the VM’s page accesses in the HV until observing the end of a particular activity. This activity must make use of the targeted secret at least once. The start of the activity, denoted by ActivityStart, does not need to be observable by the HV. In contrast, the end of an activity, called ActivityEnd, must be a HV-observable event. This event indicates that the VM recently used the secret one or multiple times, denoted by Use1..Usen. As soon as we observe ActivityEnd, we stop tracking, denoted by TrackingEnd. We do not actively attempt to trigger ActivityStart in order to interfere as little as possible.

To start page access tracking, denoted by TrackingStart, the HV invalidates all the target VM’s GPA to HPA mappings. As a consequence, each of the VM’s page accesses causes an observable event, a SLAT page fault. For each SLAT page fault, we record the GPA as well as the time and type of the page access (read, write, execute) in a list and re-validate the mapping. The re-validation clears the page from tracking. This means that each accessed page triggers exactly one page fault and that we track the page exactly once, namely the first time it is accessed after TrackingStart. The tracking enforces that accesses to the secret will inevitably be recorded. Note that the secret can be contained in a single page or span over multiple pages and can have multiple occurrences on different pages.

An example for an activity is a TLS handshake as part of a request to a web server. The server uses the targeted secret, in this case its TLS private key, to authenticate itself to a client during the handshake. The HV can observe ActivityEnd by monitoring network traffic, waiting for the packet the VM sends to complete the handshake.

Figure 1 depicts an attack scenario with the target VM and the HV in the upper and lower box, respectively. The illustration shows the start and end of a VM’s activity along with events triggered by the activity, such as Use1..Usen. The vertical arrows crossing the upper and lower box represent the events observable from the HV. These are, for instance, SLAT page faults, network packets or disk I/O. Some of the vertical arrows do not cross the boundary of the VM. These are events not observable by the HV, for example, page faults handled by the VM or possibly ActivityStart. Some of the events may be related to concurrent activities, and multiple other activities may potentially make use of the secret as well, cases which are not depicted in Figure 1. The illustration emphasizes that TrackingEnd concludes the observation phase right after ActivityEnd.

When starting tracking between Usen and ActivityEnd, we do not observe any of the page accesses to the secret. This means that we are unable to find the secret in the later search phase, requiring to repeat the attack. This is why we call the timespan between Usen and ActivityEnd the critical window. The critical window size is an important factor regarding the quality of the attack. The smaller the critical window the higher the probability that the attack succeeds. Further, a small critical window means quick termination of page access tracking after Usen. This causes Usen to be tracked at the very end of the phase, and likely only a few more pages to be tracked after Usen. We evaluate the critical window size for different scenarios with various levels of load in Section 5.

It is not necessary to synchronize the start of the observation phase with a possibly non-observable ActivityStart. If TrackingStart takes place long before ActivityStart, the observation phase might take longer, but since every page is tracked only once, this does not lead to a persistent performance impact. On the other hand, if TrackingStart takes place after ActivityStart (but not inside the critical window), the tracking period will be shorter and likely output less tracked page accesses.

To conclude, the result of the observation phase is a list of pages in which the page with the targeted secret is contained at least once as long as TrackingStart is not inside the critical window. The set of pages in the list is significantly smaller than the whole set of the VM’s pages.

3.2. Search Phase

The goal of the search phase is to extract the targeted secret from the VM’s memory as quickly as possible, i.e., with a minimal number of memory requests. The input to the search phase is the list of tracked pages acquired during the observation phase. It is unknown which of the page accesses in the list correspond to Use1..Usen. The naive extraction of all pages in the list would still require a fairly high number of memory requests to find the secret. In the following, we describe our approach for a more efficient extraction.

The search phase starts right after TrackingEnd, as depicted at the bottom of Figure 1. We know that ActivityEnd indicates recent use of the secret. This means that Usen must have occurred shortly before TrackingEnd. For this reason, we consecutively extract the tracked pages in backward order until we find the secret. We thus start the extraction with the most recently tracked pages. This backward search is shown by the arrow directed to the left at the bottom of Figure 1. We analyze extracted memory chunks for the presence of the secret on the fly to be able to terminate the extraction procedure as early as possible. On the fly means we search the latest extracted memory chunk for the secret while we request the next chunk. When finding the secret in the chunk, we terminate the search phase, otherwise we request another chunk. The actual analysis is specific to the targeted secret and described in Section 4 for different secrets.

We propose an optional preprocessing step before the extraction to further minimize the number of memory requests. Preprocessing filters page accesses from the list, which cannot represent a use of the secret, and prioritizes accesses that are likely to represent a use. The ability to filter and prioritize depends on the use case, in particular, on the specific activity and secret. In most cases, the secret is a data structure on a page in non-executable memory, allowing to filter all execute-accesses from our list. The page is likely to be read, but may also be written during an activity. Depending on the use case, it is also possible that the secret resides in a read-only area, or represents confidential code. The information about this can often be acquired prior to the attack. A further possibility for preprocessing is to conduct a representative offline access pattern analysis for the activity to observe the expected timing of Use1..Usen. An offline analysis is more representative the more the hardware platform and the software configuration inside the VM resemble the attack target. With the gained timing information, an attacker can further filter or re-prioritize pages in the list.

Extracting the secret from the encrypted VM using SEVered requires the secret to remain at the same location during the attack. This means that the secret must not be erased or moved to a different HPA by the VM’s kernel before the search phase terminates. We show that the secrets we chose for extraction always fulfilled this requirement and investigate preprocessing possibilities as part of our evaluation in Section 5.

4. Key Extraction Scenarios

In the following, we describe the application of our concept for the extraction of targeted secrets at the example of private keys and symmetric FDE keys. We focus on the aspects from Section 3 that are specific for the type of secret. These aspects are the activities with their events and the on the fly analysis.

4.1. Private Keys

For the extraction of private keys, we focus on the example of web server TLS keys. These keys are resources located in a VM’s user space and highly sensitive. Web servers use these keys to establish authenticated TLS channels with clients. An attacker can make use of a stolen private key for identity spoofing and deceive clients for fraud or data exfiltration.

**Events.: **

ActivityStart is the start of a TLS handshake. The handshake can be part of an HTTPS request or be directly triggered by a client. Usei represents a server’s use of the TLS key for its authentication during the handshake. The exact moment of use depends on the key exchange method. For instance, in case of an Elliptic-Curve Diffie-Hellman Ephemeral (ECDHE)-based key exchange algorithm, this is the moment of signing curve parameters. For an RSA-based key exchange, this moment is the decryption of the premaster secret encrypted by the client with the server’s public key. ActivityEnd happens when the VM sends the client a specific network packet during the handshake. We observe these packets with network monitoring tools. The change cipher spec packet is an indicator independent of the specific key exchange algorithm. Depending on the algorithm, packets sent earlier may be usable indicators as well. Note that we can also observe or even trigger ActivityStart ourselves in this scenario. We discuss this aspect in Section 6.

**On the fly analysis.: **

The public key and its length are part of the server’s certificate and known in advance. When using RSA, the private components of the key are the factors p or q of known length dividing the modulus of the public key. For every extraction request we make, we traverse the extracted chunk of memory and check if it contains a contiguous bit sequence that divides the modulus without remainder. If so, we found either p or q and can instantly determine the other factor. Otherwise, we request the next chunk of memory. Analyzing a chunk this way usually takes less time than memory extraction with SEVered, see Section 5.

The same approach can be used for extracting SSH private keys. In the SSH scenario, the SSH server must also use its private key for authentication during the SSH handshake when establishing a session. We evaluate the extraction of TLS and SSH keys using the Apache, nginx and OpenSSH servers in Section 5.

4.2. FDE Keys

The normal approach when using SEV is to first perform an attestation of the platform. The attestation proves to the tenant that the VM has been started with SEV enabled. After a successful attestation, the tenant provides the FDE key in encrypted form to the VM (Advanced Micro Devices, 2018). This protects the key from eavesdropping adversaries in the network and from being read by the HV. Thereafter, the FDE key is present in the VM’s memory and can be extracted with our approach. The FDE key is particularly important, because it allows attackers to decrypt the VM’s persistent storage gaining access to further valuable secrets.

**Events.: **

The corresponding activity is a disk I/O operation. The trigger for ActivityStart is not observable by the HV and unlike in the TLS key scenario, ActivityStart can have many different triggers. The trigger can, for instance, be data uploaded to a service, a request to a web server being logged, or an operation of the VM’s OS involving disk I/O. The event Usei is the VM’s use of the FDE key to en- or decrypt disk content to be read or written. We observe ActivityEnd by monitoring the VM’s disk image file in the HV.

**On the fly analysis.: **

We can be sure that we found the secret as soon as we are able to successfully decrypt the VM’s persistent storage. Traversing extracted memory chunks and naively trying each possible sequence as key leads to an inefficient approach. Our goal is thus to first identify key candidates in extracted memory chunks. For this purpose, we search the extracted memory for characteristics specific to FDE keys based on the following two criteria.

First, the FDE key is stored in the VM’s kernel in a specific data structure. This structure has various fields, some of which must have certain value ranges, for instance, kernel addresses pointing to other kernel objects. Our first criterion for a key candidate is thus the identification of possible FDE key structures in extracted memory chunks.

Our second criterion is based on the statistical properties of the FDE key. Because FDE is usually AES-based, the kernel derives round keys from the FDE key and keeps them in AES key schedules in memory. The round keys have common statistical properties that can be identified with linear complexity. The first-round key is the AES key itself. We use aeskeyfind (Center for Information Technology Policy at Princeton University, 2008) to search memory chunks for AES key schedules. Note that candidates that turn out to be false positives are possibly symmetric keys used for other purposes and might also be valuable secrets. The traversal of memory chunks based on these two criteria takes considerably less time than the extraction of memory with SEVered, see Section 5.

We evaluate the FDE key extraction scenario as part of the following section.

5. Implementation and Evaluation

In the following, we first define performance indicators and then present our prototype and test setup. Based on that, we evaluate the extraction of TLS, FDE and SSH keys, as discussed in Section 4. In the final part of our evaluation, we present strategies for optimization with preprocessing and summarize our results.

5.1. Performance Indicators

The key factors we investigate are the success probability and the attack time.

Success Probability

As discussed in Section 3, the critical window size is the factor determining the success probability of our attack. The smaller the critical window, the smaller the probability that the observation phase ends without having tracked the access to the secret. In our evaluation, we present the success probability for the tested scenarios and provide an upper bound on the size of the critical window. We call the upper bound the reaction time of our attack. The reaction time is the sum of the critical window (the time frame between Usen and ActivityEnd) and the time our prototype requires to detect ActivityEnd and stop tracking (the time frame between ActivityEnd and TrackingEnd).

Attack Time

We divide the total attack time of a full attack into its three components: the time required to setup SEVered prior to extraction, the duration of the observation phase and the duration of the search phase.

**Setup of SEVered.: **

The time required to setup SEVered is evaluated in (Morbitzer et al., 2018) and is thus not subject of our evaluation. Setting up SEVered usually takes less than 20 seconds, depending on the load of the VM. After setting up SEVered once, we can arbitrarily extract the victim VM’s memory and repeat our attack when necessary.

**Observation phase.: **

The main factor for the duration of the observation phase is the frequency of the targeted activity. For instance, a web server under high load will often make TLS handshakes while SSH logins generally occur less frequently.

**Search phase.: **

The duration of the search phase is mainly determined by the amount of memory that has to be extracted until the secret is found. This is driven by the number of pages we track within the reaction time frame. The reaction time not only provides an upper bound on the critical window, but also serves as indicator for the expected number of tracked pages.

In our evaluation, we investigate the number of pages that have to be extracted, and the duration of the observation and search phase. We call the combined duration of both phases the attack time.

5.2. Prototype and Test Setup

We implemented our prototype including the functionality required for SEVered based on Kernel-based Virtual Machine (KVM). To start and stop page tracking and change mappings, we extended the KVM API with additional calls, in particular, with KVM* system ioctls* (The Linux Kernel Organization, 2018). This allows us to launch the attack from user space by communicating with the KVM kernel module. For page access tracking in KVM we used the technique from (Morbitzer et al., 2018; Guangrong, 2016). While tracking is active, we record all tracked pages in a list in kernel memory. Upon the call to stop tracking, KVM returns the list of tracked pages to user space.

We ran KVM on Debian with a page size of 4 KB using an SEV-enabled Linux kernel in version 4.18.13 and QEMU 3.0.50. We used an AMD EPYC 7251 processor with full support for SEV. We created a victim VM with 2 GB of memory and one of the four available CPU cores. We deployed Apache 2.4.25-3 and nginx 1.10.3-1 for the TLS key scenarios, and OpenSSH 7.4 for the SSH scenario in the VM. The FDE scenario is independent of a service, because the FDE key is a kernel resource exclusively used by the OS. We deployed eleven different web resources on each web server. We used 4096-bit private keys for TLS and SSH and a 256-bit symmetric FDE key for storage encryption with AES-XTS. As target for memory extraction with SEVered, we used a page-sized web resource served by nginx.

To capture the handshake messages for the TLS and SSH key scenarios, we used tcpdump with libpcap, a library for network packet capturing. For TLS, we captured the change cipher spec packet the services send to conclude a TLS handshake (filter tcp[37] == 0x04). For SSH, we captured the new keys message, which concludes the SSH handshake (filter tcp[37] == 0x15). We patched libpcap to execute a system call for TrackingStart the moment packet capturing begins, and a call for TrackingEnd the moment the filtered packet is captured. This tight interconnection minimizes the reaction time. To monitor disk I/O events of the VM in the FDE key scenario, we used the tool inotifywait to observe inotify events. In particular, the notify option allows to detect disk writes on the VM’s disk image file. We modified inotifywait to issue the calls for TrackingStart right before starting to watch events and for TrackingEnd as soon as an inotify event is identified.

In real-world scenarios, a tenant’s VM can show higher or less activity depending on the load caused by its clients. To simulate this behavior, we executed all our tests based on a load model with various load levels, representing low to high load. In our model, a load level of nine, for instance, refers to nine requests per second to the VM. We randomly alternate between the services for each request. With a probability of $\frac{300}{301}$, we make a request to one of the resources offered by one of the two web servers with equal probability. With a probability of $\frac{1}{301}$, we initiate an SSH login with a user remaining logged in for two minutes. Compared to the number of web server requests, we execute only few SSH logins, as these usually happen less frequently than requests to a web server. The average duration of the observation phase thus lies in the range of a few seconds to a few hundreds of milliseconds for the web servers and in the range of a few minutes to tens of seconds for SSH. Note that sshd forks a new process for each new SSH connection. When the session terminates, the process exits and purges its SSH key. This means that the search time must be less than two minutes to extract the SSH key before the forked process exits.

We conducted 2,000 independent iterations of our attack for each of the four scenarios on four load levels: level 1, 9, 17, and 25. We started our attacks at random points in time while the VM processed requests according to the specific load level of our model. As an initial preprocessing step before the search phase, we filtered all execute-accesses. In our scenarios, all secrets are data structures located on non-executable memory pages.

5.3. Success Probability and Reaction Time

In this part, we investigate the success probability and reaction times. The four diagrams in Figure 2 illustrate the distribution of measured reaction times for each scenario. The four graphs in each diagram represent the four load levels. The X-axes are discretized in steps of one millisecond, and the Y-axes are normalized to one. The vertical dashed lines show the median reaction times over all repetitions for each level, providing an upper bound on the median critical window size.

The results for Apache and nginx TLS handshakes are depicted in the top row of Figure 2. Both diagrams show a clear peak for the two lower load levels, indicating a reliable reaction time when the VM is not under high load. For the lowest load level, we can even observe that the reaction time never exceeded 21 milliseconds with Apache and 22 milliseconds with nginx. For higher load levels, more concurrent activities are executed by the VM, and the measurements are more dispersed over time. Consequently, it becomes likely that more pages have to be extracted in the search phase until the secret is found. This led to a maximum reaction time of around 50 milliseconds for both nginx and Apache in rare cases. However, the median reaction time increased to about only 20 milliseconds for nginx, and to about 16 milliseconds for Apache. As the reaction time is an upper bound for the critical window, the latter is smaller than tens of milliseconds for both Apache and nginx. We achieved a very high success rate of around 99.99% for both web servers on all load levels, meaning that we started tracking inside the critical window only in a few cases. The high success rate indicates that the upper bound we measured is a very conservative estimate. This comes from the fact that our prototype requires some time to actually stop tracking and (especially for the TLS scenarios) to recognize ActivityEnd. Note that if TrackingStart occurs inside the critical window of a TLS handshake, we still have the chance to observe Usei of other handshakes being concurrently processed on higher load levels where lots of handshakes are made each second. The critical window can thus be even smaller on higher load levels.

The bottom left diagram in Figure 2 for disk write events shows that our implementation achieved an extremely fast reaction time of one millisecond in the median for each load level. Only in a few cases, we encountered a slightly higher reaction time. In contrast to the TLS key scenarios, the behavior was generally independent of the load level. In the TLS key scenarios, the network packets must first be sent by the VM to the network interface, on which the HV executes more time-consuming network packet capturing. The interception of disk write events is less complex and introduces less delay. The success rate for FDE key extraction was about 99.99%, indicating a very small critical window, as confirmed by the upper bound in the graph.

The bottom right diagram in Figure 2 shows that the reaction time for SSH handshakes was four milliseconds in the median and mostly independent of the load level. We encountered only a few samples going up to about 30 milliseconds. As for the TLS scenario, this indicates a small upper bound on the critical window and a possibly quick extraction. Accordingly, our attack had a success rate of 99.98%.