CUP: Comprehensive User-Space Protection for C/C++

TL;DR

CUP is an LLVM sanitizer that offers comprehensive spatial and probabilistic temporal memory safety for C/C++ programs on 64-bit architectures, covering all data types and system libraries, significantly reducing false negatives and eliminating false positives.

Contribution

CUP introduces a hybrid metadata scheme supporting all program data and maintains ABI, providing complete user-space memory safety in C/C++ with improved accuracy over existing sanitizers.

Findings

Reduces false negatives by 10x compared to existing sanitizers.

Produces no false positives in testing.

Supports all program data including globals, heap, and stack.

Abstract

Memory corruption vulnerabilities in C/C++ applications enable attackers to execute code, change data, and leak information. Current memory sanitizers do no provide comprehensive coverage of a program's data. In particular, existing tools focus primarily on heap allocations with limited support for stack allocations and globals. Additionally, existing tools focus on the main executable with limited support for system libraries. Further, they suffer from both false positives and false negatives. We present Comprehensive User-Space Protection for C/C++, CUP, an LLVM sanitizer that provides complete spatial and probabilistic temporal memory safety for C/C++ program on 64-bit architectures (with a prototype implementation for x86_64). CUP uses a hybrid metadata scheme that supports all program data including globals, heap, or stack and maintains the ABI. Compared to existing approaches with the NIST Juliet test suite, CUP reduces false negatives by 10x (0.1%) compared to the state of the art LLVM sanitizers, and produces no false positives. CUP instruments all user-space code, including libc and other system libraries, removing them from the trusted code base.

Tables

Table 1. Table 1: Allocation distribution in SPEC CPU2006.

Memory Type	Allocations
global	0.0006%
heap	0.07%
stack	99.9%

Table 2. Table 2: Performance Results. † † \dagger indicates C++ benchmark

	401.bzip2	429.mcf	4435.gobmk	456.hmmer	458.sjeng	462.libquantum	464.h264ref	473.astar$†$	483.xalancbmk$†$	433.milc	444.namd$†$	450.soplex$†$	470.lbm	482.sphinx3	Geomean
musl-baseline	382	213	386	278	371	369	367	327	166	431	278	183	201	403	297
ASan	-	261	-	596	630	264	-	464	369	444	377	261	243	-	369
SoftBound+CETS	2004	-	-	-	1372	-	-	-	-	1543	-	-	409	-	1148
CUP	1131	503	1255	1268	1192	391	1196	955	-	638	905	-	370	1185	839

Table 3. Table 3: Percent Overhead over Baseline

CUP	158%
ASan	38%
SoftBound+CETS	53%

Table 4. Table 4: Percent Overhead over ASan

CUP	83%
SoftBound+CETS	134%

Table 5. Table 5: Juliet Suite Results

	False Neg.	False Pos.
CUP	4 (0.1%)	0 (0%)
SoftBound+CETS	1032 (25%)	12 (0.3%)
AddressSanitizer	315 (8%)	0 (0%)
Total Tests	4038

Equations

$$\text{0x}\underset{\text{makebox}[0.0pt]CapabilityID}{\underbrace{\text{80000001}}}\stackrel{\text{Offset}}{\overbrace{\text{00000000}}}$$

$$base\le ptr+element\_size-1<base+length$$

Full text

CUP: Comprehensive User-Space Protection for C/C++

Nathan Burow

Purdue University

  

Derrick McKee

Purdue University

  

Scott A. Carr

Purdue University

  

Mathias Payer

Purdue University

Abstract

Memory corruption vulnerabilities in C/C++ applications enable attackers to execute code, change data, and leak information. Current memory sanitizers do not provide comprehensive coverage of a program’s data. In particular, existing tools focus primarily on heap allocations with limited support for stack allocations and globals. Additionally, existing tools focus on the main executable with limited support for system libraries. Further, they suffer from both false positives and false negatives.

We present Comprehensive User-Space Protection for C/C++, CUP, an LLVM sanitizer that provides complete spatial and probabilistic temporal memory safety for C/C++ programs on 64-bit architectures (with a prototype implementation for x86_64). CUP uses a hybrid metadata scheme that supports all program data including globals, heap, or stack and maintains the ABI. Compared to existing approaches with the NIST Juliet test suite, CUP reduces false negatives by 10x (0.1%) compared to the state of the art LLVM sanitizers, and produces no false positives. CUP instruments all user-space code, including libc and other system libraries, removing them from the trusted code base.

1 Introduction

Despite extensive research into memory safety techniques [38], exploits of memory corruptions remain common [40, 18, 36]. These attacks rely on the fact that C/C++ require the programmer to manually enforce spatial safety (bounds checks) and temporal safety (lifetime checks). As the continuing stream of memory corruption Common Vulnerabilities and Exposures (CVEs) shows, these programmer added checks are often inadequate. Many of these bugs are in network facing code such as browsers [28, 29] and servers [30, 31], allowing attackers to illicitly gain arbitrary execution on remote systems. Consequently, a memory safety sanitizer that comprehensively protects user-space is necessary to find and fix these bugs.

To correctly address memory safety in user-space, there are four main requirements. Precision addresses spatial safety by requiring that exact bounds are maintained for all allocations. Object Awareness prevents temporal errors by tracking whether the pointed-to object is currently allocated or not. These two requirements are sufficient to enforce memory safety. Adding Comprehensive Coverage expands this protection to all of user space by requiring that all data on the stack, heap and globals be protected. Comprehensive Coverage implies that all code must be instrumented with the sanitizer, including system libraries like libc. A sanitizer that meets these three requirements is powerful enough to find all memory corruption vulnerabilities in user-space programs. To be useful, such a sanitizer must also be usable in practice. Requiring Exactness – no false positives and minimal false negatives – ensures that bugs reported by a sanitizer are real, and that all spatial and most temporal violations are found. We discuss these challenges in § 3.

The research community has come up with many approaches that attempt to address memory safety. Initial efforts to address spatial safety relied on “fat pointers” that store bounds information inline with pointers [26, 14], which unfortunately breaks the Application Binary Interface (ABI). SoftBound [24] moves metadata to a disjoint table, maintaining the ABI, but adding overhead to lookup the metadata associated with tables. Low-Fat Pointers [9] reduces overhead by partitioning and aligning memory allocations, allowing alignment based bounds checks. This however rounds object sizes, and alters the memory layout of the program. AddressSanitizer [35] (ASan) provides probabilistic spatial safety by relying on poison zones between objects, but is vulnerable to “long strides” that skip these zones. There are two main approaches to temporal safety. Probabilistic approaches [2, 32] change the memory allocator to reduce the frequency with which memory is reallocated. Deterministic schemes [25, 16] either maintain per pointer metadata that knows whether an object is still allocated, or invalidate all pointers when the object is deallocated.

Comprehensive Coverage is largely an open problem, with prior work mostly ignoring the stack and neglecting support for system libraries like libc. Low-Fat Pointers [9] is the only work to protect the stack – providing only spatial safety. No existing work provides spatial and temporal safety comprehensively for all user-space data (stack, heap, globals) and code (program code, libc, libraries). Doing so requires a large amount of additional metadata to protect the extra allocations § 2.2, which existing metadata schemes are unable to handle. Further, the performance of existing tools does not allow them to scale to handle the additional memory surface of the stack and libc.

libc is a particularly critical part of user-space to protect. It is prone to memory errors, notably the mem* and str* family of functions (memcpy, strcpy, …). Memory errors in libc are not limited to these functions, however as shown by, e.g., GHOST [18] and a stack overflow in getaddrinfo [36]. Consequently, the entire libc needs to be protected, not just certain interfaces.

Exactness shows how well a memory safety solution protects against vulnerabilities in practice. The U.S. National Institute of Standards and Technology (NIST) maintains the Juliet test suite. Juliet consists of thousands of examples of bugs, grouped by class from the Common Weakness Enumeration (CWE). Juliet reveals that existing, open source memory safety solutions [35, 24, 25] have both false positives and a non-trivial number of false negatives § 5.2.

CUP satisfies all four requirements for a powerful, usable memory sanitizer. We introduce a new hybrid metadata scheme which is capable of storing and using per object metadata for the stack, libc, heaps, and globals. Our metadata is precise and does not require altering the program’s memory layout. Additionally, we introduce a new way to check bounds that leverages hardware to increase our check’s performance. Hybrid metadata allows us to meet the Precision and Object Awareness requirements. CUP presents a novel use of escape analysis to reduce the amount of stack allocations without loss of protection. This reduction allows scaling our mechanism to include all user-space data, satisfying the Comprehensive Coverage requirement. Further, CUP successfully handles all system libraries, including libc, the first memory sanitizer to do so. Our evaluation on Juliet § 5.2 shows that we have no false positives and 0.1%false negatives, considerably advancing the state of the art for Exactness.

We present the following contributions:

• •

A new hybrid metadata scheme capable of tracking any runtime information about object allocations, and show how it can be applied to memory safety.

• •

The first sanitizer to fully protect user-space, including libc

• •

A new static analysis for determining what stack variables require active protection, and present a local protection scheme for non-escaping stack variables

• •

Evaluation of a CUP prototype that, using our hybrid metadata model, results in (i) no false positives and 0.1% false negatives on the NIST Juliet C/C++ test suite and (ii) reasonably low overhead (in line with other sanitizers).

2 Background and Challenges

Our requirements for Precision and Object Awareness are designed to enforce spatial and temporal memory safety, which we define here and then use to introduce the notion of a capability ID.

Spatial Vulnerabilities

also known as bounds-safety violations – are over- or under-flows of an object. Over/under-flows occur when a pointer is incremented/decremented beyond the bounds of the object that it is currently associated with. Even if the out-of-bounds pointer still points to a valid object, it does not have the capability for the referenced object, and the operation results in a spatial memory safety violation. However, this violation is only triggered on a dereference of an out-of-bounds pointer. The C standard specifically allows out-of-bounds pointers to exist.

Temporal Vulnerabilities

also known as lifetime-safety violations – occur when the object that a pointer’s capability refers to is no longer allocated and that pointer is dereferenced. For stack objects, this is because the stack frame of the object is no longer valid (the function it was created in returned); for heap objects, this happens as a result of a free. These errors do not necessarily cause segmentation faults (accesses to unmapped memory), because the memory may have been reallocated to a new object. Similarly, we cannot simply track what memory is currently allocated, because the object at a particular address can change, which still results in a temporal safety violation. Temporal bugs are at the heart of many recent exploits, e.g., for Google Chrome or Mozilla Firefox as shown in the pwn2own contests [39]

Violating either type of memory safety can be formulated as a capability violation. In our terminology, an object is a discrete memory area, created by an allocation regardless of location (stack, heap, data, bss under the Linux ELF format). A capability identifies a specific object, along with information about its bounds and allocation status. Pointers retain a capability ID that identifies the capability of the object that was most recently assigned – either directly from the allocation or indirectly by aliasing another pointer [12]. Capabilities form a contract, upon dereference: (i) the pointer must be in bounds, and (ii) the referenced object must still be allocated. Violating the terms of this contract leads to spatial or temporal memory safety errors respectively.

2.1 Vulnerable Objects

Comprehensive Coverage requires that we protect all memory objects. However, some objects are inherently safe, so it is sufficient to protect all vulnerable objects. In particular, an object is vulnerable if accesses to it are calculated dynamically, and not by fixed offsets. This happens with pointers, and array accesses (which are just syntactic sugar for pointer arithmetic). Dynamic address calculation does not happen for variables on the stack which are not arrays. Such local variables are accessed by fixed offsets, calculated at compile time, from the stack frame, and can only be used maliciously after an initial memory corruption. In particular, accessing a pointer on the stack does not need to be protected, only the pointer’s dereference, which dynamically looks up memory.

2.2 Comprehensive Coverage Challenges

Understanding the scope of the challenge presented by Comprehensive Coverage is critical to understanding CUP’s design. To illustrate this challenge, we show how intensively programs use different logical regions of memory. While the operating system presents applications with a contiguous virtual memory address space, that address space is partitioned into three logical groups for data: global, heap, and stack spaces. Global memory is allocated at application load-time, and is available for the entire lifetime of the application, (i.e. global memory is never explicitly deallocated). Heap memory is requested by the application through the new operator or a call to malloc, and is deallocated via the delete operator or a free call. Stack memory space is implicitly allocated with function calls, and is again implicitly deallocated with a return call.

Stack allocations account for almost all (99.9%) of memory allocations in SPEC CPU2006 (see Table 1). This measurement includes allocations made in libc. Further, the latest data from van der Veen, et. al. [40, 43] show that stack-based vulnerabilities are responsible for an average of over 15% of memory related CVEs annually since tracking began in November 2002. By comparison, heap-based vulnerabilities account for an average of 25% of memory related CVEs over the same time period. Given the stack’s exploitability and prevalence, which stresses memory safety designs, protecting it is a key design challenge for memory safety solutions.

3 Design

CUP provides precise, complete spatial memory safety and stochastic temporal memory safety by protecting all program data, including libc (and any other library). Safety is enforced, for all program data, by dynamically maintaining information about the size and allocation status of all objects that are vulnerable to memory safety errors. This information is recorded through our novel hybrid metadata scheme § 3.1. A compiler-based instrumentation pass is used to add code that records and checks metadata at runtime § 3.2. We provide a detailed argument for why our instrumentation guarantees memory safety in § 3.3.

A powerful usable memory sanitizer must comply to the following requirements:

1. I

Precision. The solution must enforce exact object bounds, ideally without changing memory layout (i.e., spatial safety). 2. II

Object Aware. The solution must remember the allocation state of any object accessed through pointer (i.e., temporal safety). 3. III

Comprehensive Coverage. The solution must fully protect a program’s user-space memory including the stack, heap, and globals, requiring instrumentation and analysis of all code, including system libraries such as libc (i.e., completeness). 4. IV

Exactness. The solution must have no false positives, and any false negatives must be the result of implementation limitations, not design limitations (i.e., usefulness).

These requirements drive the design of CUP. Fully complying with the Precision and Object Awareness requirements relies on creating metadata for all allocated objects. While it is possible [1, 9] to do alignment based spatial checks without metadata, these schemes loose precision, alter memory layout, and cannot support Object Awareness. Object Awareness for temporal checks requires metadata to either lookup whether the object is still valid [25] or to find all pointers associated with an object and mark them invalid upon deallocations [16]. Consequently, CUP is a metadata based sanitizer.

CUP provides Comprehensive Coverage, and in particular protects globals, the heap, and stack by instrumenting all code, including libc. Our hybrid metadata scheme scales to handle the required number of allocations § 2.2, and our bounds check leverages the x86_64 architecture § 4.2.2 to perform the required volume of checks quickly enough to be usable. Additionally, our compiler pass is robust enough to handle libc § 4.3, making CUP the first memory sanitizer to do so.

Exactness is achieved, in part, failing closed, making a missing check equivalent to a failed check. We modify the initial pointer returned by object allocation § 3.2, and our modification marks it illegal for dereference. This modification propagates through aliasing and all other operations naturally. Consequently, we must check all uses of the pointer for the program to execute correctly. Such an approach results in optimal precision at the cost of higher engineering burden (as shown in § 4 and § 5), but dramatically reduces false negatives. False positives are prevented by maintaining accurate metadata, and having it propagate automatically.

3.1 Hybrid Metadata Scheme

To provide Precision, Object Awareness, and Comprehensive Coverage, we introduce a new hybrid metadata scheme that lets us embed a capability ID in a pointer without changing its bit width. This capability ID ties a pointer to the capability metadata for its underlying object. Precision is provided by the metadata containing exact bounds for every object, and by not rearranging the memory layout. Object Awareness results from having a unique metadata entry for each capability ID.

Providing Comprehensive Coverage requires assigning a capability ID to all vulnerable objects in order to associate their pointers with the object’s capability metadata. However, the capability ID space is fundamentally limited by the width of pointers. To address this limit, we allow capability IDs to be reused. Consequently, our capability ID space only needs to support the maximum number of simultaneously allocated objects. This allows CUP to comprehensively cover globals, the heap, and stack for all allocations in long running applications.

Our metadata scheme that draws inspiration from both fat-pointers and disjoint metadata (and is thus a “hybrid” of the two) for 64-bit architectures. We conceptually reinterpret the pointer as a structure with two fields, as illustrated by 1. The first field contains the pointer’s capability ID. The second field stores the offset into the object. This does not change the size of the pointer, thus maintaining the ABI. Further, when pointers are assigned, the capability ID automatically transfers to the assigned pointer without further instrumentation.

Hybrid metadata rewrites pointers to include the capability ID of their underlying object and current offset, creating enriched pointers. The size of the offset field limits the size of supported object allocations. The tradeoffs of the field sizes and our implementation decisions are discussed in § 4.1. While we use it for memory safety, this design allows access to arbitrary metadata, and could be applied for, e.g., type safety, or any property that requires runtime information about object allocations.

Capability IDs in our hybrid-metadata scheme are indexes into a metadata table. Each entry in this table is a tuple of the base and end addresses for the memory object, required for spatial safety checks. Each object that is currently allocated has an entry in the table, leading to a memory overhead of 16 bytes per allocated object. Note that we do not require per-pointer metadata due to our hybrid scheme. To reduce the number of required IDs to the number of concurrently active objects, we allow capability IDs to be reused. Allowing ID reuse thus allows us to protect long running programs, as our limit is on concurrent pointers, not total allocations supported. The security impact of ID reuse is evaluated in § 6.

The metadata table provides strong probabilistic Object Awareness. For a temporal safety violation to go undetected, two conditions must hold. First, the capability ID must have been reused. Second, the accessed memory must be within the bounds of the new object. Current heap grooming techniques [11, 37] already require a large number of allocations to manipulate heap state. Adding the requirement that the same capability ID also be used makes temporal violations harder. § 6 contains other suggestions to further increase the difficulty.

We are aware of two memory safety concerns for hybrid metadta: (i) arithmetic overflows from the offset to the capability ID, and (ii) protecting the metadata table. The first concern is addressed by operating on the two fields of the pointer separately. By treating them like separate variables – while maintaining them as one entity in memory – we prevent under/over flows from the offset field modifying the capability ID. The second concern is not relevant for CUP– if all memory accesses are checked, then the metadata table cannot be modified through a memory violation.

3.2 Compiler Modifications

Our compiler adds instrumentation to create entries in the metadata table and perform runtime checks. In particular, we first analyze all allocations (Comprehensive Coverage), and filter them to the ones we must protect to provide Precision and Object Awareness. This section also shows how CUP fails closed, and checks all required pointers, both of which contribute to its Exactness.

The analysis phase identifies when objects are allocated or deallocated, and when pointers are dereferenced through an intra-procedural analysis. All pointers that are passed inter-procedurally are instrumented using our metadata scheme, including all heap allocations.

We manually annotate libc § 4.3 to mark heap allocations. For global variables we protect arrays as entries are referenced indirectly (i.e., with pointer arithmetic). Similarly for stack allocations, we only protect arrays, as well as any address taken variable. We leverage existing LLVM analysis to find address taken variables.

Our analysis further divides protected stack allocations into (i) escaping and (ii) non-escaping allocations. An allocation does not escape if the following holds: (i) it does not have any aliases, (ii) it is not assigned to the location referenced by a pointer passed in as a function argument, (iii) it is not assigned to a global variable, (iv) it is not passed to a sub-function (our analysis is intra-procedural excluding inlining), and (v) is not returned from the function. For those that escape, we use our usual metadata scheme so that the bounds information can be looked up in other functions. For those that do not escape, we use an alternate instrumentation scheme.

The optimized instrumentation for non-escaping stack variables scheme creates local variables with base and bounds information. Since these allocations are only used within the body of the function, we use local variables for checks instead of looking up the bounds in the metadata table. This reduces pressure on our capability IDs, helping us to achieve Comprehensive Coverage.

All other allocation sites requiring metadata are instrumented to assign the object the next capability ID and to create metadata (recording its precise base and end addresses) – returning an enriched pointer. We create metadata at allocation because it is the only time that we are guaranteed to know the size of the object.

Identifying deallocations for objects is straightforward. Global objects are never deallocated over the lifetime of the program. Heap objects are explicitly deallocated by, e.g., free() or delete. Stack objects are implicitly deallocated when their dominating function returns. Deallocations are instrumented to mark associated metadata invalid and to reclaim the capability ID.

Pointer derefences are found by traversing the use-def chain of identified pointers. Dereferences are analyzed intra-procedurally, so we include pointers from function arguments (including variadic arguments) and pointers returned by called functions in the set of allocations for this analysis. We instrument dereferences with a bounds check. Note that the bounds check implicitly checks that the pointer’s capability ID identifies the correct object. See § 3.3 for a discussion of the safety guarantees.

CUP also inserts instrumentation to handle int to pointer casts. These are commonly inserted by LLVM during optimization, and have matching pointer to int casts in the same function. In this case, and any others where we can identify a matching pointer to int cast, we restore the original capability ID to the pointer. If we are unable to find a matching pointer to int cast, we default to capability ID zero, which is all of user-space.

3.3 Memory Safety Guarantees

We discuss how CUP guarantees spatial memory safety and probabilistically provides temporal safety. We assume that all code is instrumented and capability IDs are protected against arithmetic overflow (as proposed).

For code that we instrument, we keep a capability ID (and thus metadata) for every memory object that can be accessed via a pointer. This subset is sufficient to enforce spatial memory safety. Objects that are not accessed via pointers are guaranteed to be safe by the compiler (if you are reading an int, it will always emit instructions to read the correct 4 bytes from memory).

Pointers can be used to read or write arbitrary memory. Further, the address that they reference is often determined dynamically. Thus, pointers require dynamic checks at runtime for memory safety guarantees. As defined in § 2, memory objects define capabilities for pointers. These capabilities include the size and validity of the object. We only create capabilities when objects are allocated at runtime. Objects can change size due to, e.g., realloc() calls, in which case we update our metadata appropriately by changing base and end to the new values § 4.2.2. Thus, we always have correct metadata for every object that has been created since the start of execution. The metadata for objects that have not been created yet is invalid by default.

Pointers can receive values in five ways. First, pointers can be directly assigned from the memory allocation, e.g., through a call to malloc(). We have instrumented all allocations to return instrumented pointers. Second, they can receive the address of an existing object, via the & operator. We treat this as a special case of object allocation and instrument it. Third, pointers can be assigned to the value of another pointer. As all existing pointers have been instrumented under the first two scenarios, this case is covered as well. Fourth, pointers can be assigned the result of pointer arithmetic. This is handled naturally, with our separate loads preventing overflows into the capability ID. The fifth scenario is a cast from an int to a pointer. This is exceedingly rare in well written user-space code. However, the compiler frequently inserts these operations in optimized code. As a result, we have to allow these operations. We assume that all ints being cast to pointer were previously a pointers, and thus instrumented.

So far we have established that all pointers are enriched with capabilities that accurately reflect the state of the underlying memory object. Memory safety violations occur when pointers are dereferenced [38]. We instrument every dereference to check the pointers capability and ensure that the dereference is valid. Because each pointer has a capability and each capability is up-to-date this ensures full memory safety.

A programs is memory safe before any pointer dereference happens. We have shown that each type of pointer dereference is protected. Consequently, every pointer dereference is valid. Thus, our scheme preserves memory safety for the entire program.

4 Implementation

We implemented CUP on top of LLVM version 4.0.0-rc1. Our compiler pass is $\approx$2,500 LoC (lines of code), the runtime is another $\approx$300 LoC for $\approx$2,800 LoC total. The line count excludes modifications to our libc, which required only light annotations § 4.3. Our pass runs after all optimizations, so that our instrumentation does not prevent compiler optimizations. This also reduces the total amount of memory locations that must be protected, reducing capability ID pressure.

Here we discuss the technical details of how we implemented CUP in accordance with our design § 3. We first discuss how our hybrid metadata scheme is implemented. Next we present how we find the sets of allocations and dereferences required by our design. With the metadata implementation in mind, we then show how we instrument allocations and dereferences. With these details established, we discuss the modifications required to libc for it to work with CUP.

4.1 Metadata Implementation

Our metadata scheme consists of four elements: (i) a table of information, (ii) a bookkeeping entry for the next entry to use in that table, (iii) a free list (encoded in the table) that enables us to reuse entries in the table, and (iv) how to divide the 64 bits in a pointer between the capability ID and offset in our enriched pointers § 3.1. Our metadata table is maintained as a global pointer to a mmap’d region of memory. Similarly, the next entry in that table is a global variable known as next_entry.

By mmap’ing our metadata table, we allow the kernel to lazily allocate pages, limiting our effective memory overhead. Further, our ID reuse scheme reduces fragmentation of our metadata since it will always reuse a capability ID before allocating a new one. This also helps improve the locality of our metadata lookups, reducing cache pressure. Alternative reuse schemes with better temporal security are discussed in § 6

To implement our capability ID reuse scheme, we update next_entry using our free list. The first entry in our metadata table is reserved § 4.2.2, so next_entry is initially one. The free list is encoded in the base fields of each free entry in the table. These are all initialized to zero. When an entry is free’d, the base field is set to the offset to the next available table entry. When we add a metadata entry, next_entry is incremented, and the offset is added. When an object is deallocated, we have to update the base field for its corresponding capability ID (ID) to maintain the free list correctly. This requires calculating the offset to the next free entry. C code illustrating these operations is in 2.

The final implementation decision for our metadata scheme is how to divide the 64 bits of the pointer between the capability ID and offset. We use the high order 32 bits to store our enriched flag and capability ID Figure 1. This leaves the low order 32 bits for the offset. Limiting the offset to 32 bits does limit individual object size to 4GB under our current design (with up to $2^{31}$ such allocations). However, hardware naturally supports 32-bit manipulations, improving the performance of our implementation. Further, having a 31-bit capability ID space is crucial for protecting the entire application § 2.2. The enriched bit helps make our engineering effort easier - with it we can safely check an unenriched pointer dereferences § 4.2.2, allowing us to be be conservative and over-approximate in the set of pointer dereferences that we check § 4.2.2. The most common use case for this is pointers returned from the kernel in, e.g., malloc(). Being able to handle non-enriched pointers allows us to intervene only when the pointer is returned to the user from libc § 4.3, and keep dynamic allocation outside the trusted computing base.

With a minimal allocation size of 8 bytes, a 31-bit ID allows for at least 16 GB of allocated memory. In practice, much more memory can be allocated as objects are usually larger than 8 bytes. When fully allocated, our metadata table uses 2GB * sizeof(struct Metadata), see 3. Note that CUP only allocates pages for ID’s that are actually used.

4.2 Compiler Pass

Our LLVM compiler pass operates in two phases: (i) analysis, and (ii) instrumentation. As per our design, the analysis phase first determines a set of code points that require us to add code to perform our runtime checks, and the instrumentation phases adds these checks. These checks have been optimized to let the hardware detect bounds violations rather than doing comparisons in software. 3 has a running example for stack objects.

4.2.1 Analysis Implementation

The first task of our pass is to find the set of object allocations that we must protect to guarantee spatial safety § 3. Heap-based allocations via malloc are found by our instrumented musl libc as detailed in § 4.3. Stack-based allocations are found by examining alloca instructions in the LLVM Intermediate Representation (IR). These are used to allocate all stack local variables. However, as detailed in § 2.1 we only target allocations which can be indirectly accessed via, e.g., pointers. In practice, this means that we need to protect arrays and address-taken variables on the stack, all others are accessed via fixed offsets from the frame pointer. Arrays are trivially found by checking the type of alloca instruction as LLVM’s type system for their IR includes arrays. LLVM’s IR has no notion of the & operator. However, clang (the C/C++ front end) inserts markers – llvm.lifetime.start – which we use to identify stack allocations that have their address taken.

CUP also protects global variables. As shown in § 2.1, we only need to protect global arrays (logic is identical to protecting stack arrays). Global arrays present a challenge for our instrumentation scheme. CUP relies on changing pointer values. Unfortunately in C/C++ it is illegal to assign to a global array once it has been allocated. This means that we cannot change the pointer’s value. To address this challenge, we create a new global pointer to the first element in the array, and instrument that pointer. We then replace all uses of the global array with our new pointer that can be manipulated as described above. The new pointer must be initialized at runtime, once the address of the global array it replace has been done this. To do this, we add a new global constructor that initializes our globals. The constructor is given priority such that it runs before any code that relies on our globals.

Once the analysis pass has identified the set of allocations that need to be protected, it must find all dereferences of pointers to those objects. For stack and heap variables, it uses an intra-procedural analysis to do this. In particular, the analysis pass iterates over the set of protected stack allocations in the function, pointers passed in as arguments (including variadic arguments), and pointers returned by called functions. For each such pointer, it traces through the use-def chain looking for dereferences. In LLVM IR, this corresponds to load and store instructions for read and write derefences respectively. Once these are found, they are added either to the list of local checks (if the originating allocation is non-escaping), or the list of checks using our metadata that need to be inserted.

Global variables are handled slightly differently. LLVM maintains their use-def chains at the Module level, corresponding to a source file. Therefore we apply the same analysis used for stack / heap variables, but it operates over the entire module instead of intra-procedurally.

4.2.2 Instrumentation

Our compiler is required to add two types of instrumentation to the code: (i) allocation, and (ii) dereference. Allocation instrumentation is responsible for assigning a capability ID to the resulting pointer, creating metadata for it, and returning the enriched pointer. A subcategory of allocation instrumentation is handling deallocations – where metadata must be invalidated and the free list updated. Dereference instrumentation is responsible for performing our bounds check, and returning a pointer that can be dereferenced. While our runtime library provides functions for the functionality described in this section (for use in manual annotations), all of our compiler added checks are done inline. 2 and 3 contain examples for these operations for stack based allocations.

Allocation Instrumentation

CUP requires us to rewrite the pointer for every allocation that we identify as potentially unsafe. Our rewritten or “enriched” pointer contains the assigned capability ID, has the enriched bit set, and the lower 32 bits (which encode the distance from the base pointer) are set to 0. All uses of the original pointer are then replaced with the new, instrumented pointer. At allocation time, we use the next_entry global variable as the capability ID, and then update next_entry as described in § 4.1. See the escapes variable in main() in 3.

Note that this creates a pointer which cannot be dereferenced on x86_64, which requires that the high order 16 bits all be 1 (kernel-space) or 0 (user-space). As a result, any dereference without a check will cause a hardware fault. Consequently, for any program that runs correctly we can guarantee that all pointers to protected allocations are checked on dereference. This is in contrast to other schemes [24] that fail open, i.e., silently continue, when a check is missed, sacrificing precision.

When an object is deallocated, we insert code to update the free list in our metadata table as per § 4.1 and 2. Further, we mark the end address 0 to invalidate the entry.

Dereference Instrumentation

To dereference a pointer, two things need to be done. First, we need to recreate the unenriched pointer. Then, using the metadata from the enriched pointer’s capability ID, we need to make sure that the unenriched pointer is in bounds.

To create the unenriched pointer, we first retrieve the high order bit (which indicates whether the pointer is enriched or not). We create a 64-bit mask with the value of this bit. We then extract the capability ID, and AND it with this mask. If the pointer was not enriched, this yields an ID of 0 and otherwise preserves the capability ID. We then lookup the base pointer for the capability ID, and add the offset to it. See check in 3.

In the case where the pointer was not enriched, we lookup the reserved entry 0 in our metadata table. This entry has base and end values that reflect all of user-space (0 to 0xffffffffffff). Thus, performing our unenrichment on an unenriched pointer has no effect, and our spatial check below simply sandboxes it in user-space.

Our spatial check performs the requisite lower and upper bounds check. Note, however, that on the upper bound we have to adjust for the number of bytes being read or written. This adjustment adds significantly to our improved precision against other mechanisms § 5.2. To illustrate its importance consider the following. An int pointer is being dereferenced, meaning the last byte used is the pointer base + 4 bytes

• 1, while for a char pointer, the last byte used is the pointer base + 1 byte - 1. Equation 1 shows our bounds check formula, the size of the pointer dereferenced is element_size.

[TABLE]

Hardware Enforcement

The check in Equation 1 could naively be implemented with comparison instructions and a jump – resulting in additional overhead. We propose a novel way to leverage hardware to perform the check for us. We observe that the difference between the adjusted pointer and the base address should always be greater than or equal to 0. Similarly, the end address minus the adjusted pointer should always be greater than or equal to 0. Consequently, the high order bit in the differences should always be 0 (x86_64 with two’s complement arithmetic). We OR these two differences together, mask off the low order 63 bits, and then OR the result into the unenriched pointer. If it passed the check, this changes nothing. If it failed the check, it results in a invalid pointer, causing a segfault when dereferenced. 3 shows this optimized check in check_bounds().

4.3 LIBC

Libc is the foundation of nearly every C program and therefore linked with nearly every executable. Unfortunately, many of its most popular functions such as the str* and mem* functions are highly prone to memory safety errors. They make assumptions about program state (e.g., null terminated strings, buffer sizes) and rely on them without checking that they hold. Prior work [24, 35, 9] assumes that libc is part of the trusted code base (TCB).

In contrast, CUP removes libc from the TCB by instrumenting libc with our compiler. The majority of the instrumentation is automatic, with few exceptions such as the memory allocator, system calls, and functions implemented in assembly code. The most mature libc implementation that we are aware of that compiles with Clang-4.0.0-rc1 is musl [20]. Our instrumented musl libc is part of CUP.

4.3.1 Dynamic Memory Allocator

The dynamic memory allocator is responsible for requesting memory for the process from the kernel, and returning it. To do so efficiently, most allocators – including musl – modify user requests. In particular, musl rounds up the number of bytes requested. Further, musl maintains metadata inline on the heap in the form of headers before each allocated segment. Consequently, the allocator’s view of memory is different than that of the program.

To compensate for this difference, we manually instrumented musl’s allocator. We ensure that pointers are instrumented with the programmer specified length, not the rounded length. Further, we manually unenrich pointers as necessary to allow musl to read the header blocks that proceed heap data segments. Without our intervention, these would appear to be out of bounds.

An interesting corner case is realloc(). By design it changes the end address. Additionally, it can change the base address if it was forced to move the allocation to find sufficient room. We manually intervene in both cases to keep our metadata table up-to-date.

4.3.2 System Calls

System calls are made through a dedicated API containing inline assembly in musl. We initially instrumented this API to ensure that no enriched pointers are passed to the kernel. Unfortunately, this is insufficient as structs containing pointers are passed to the kernel (FILE structs in particular). Consequently, we required more context than the narrow system call API provided. This forced us to find the actual system call sites and add additional instrumentation to ensure that all pointers passed to the kernel are unenriched.

4.3.3 Assembly Functions

musl implements many of the mem*, str* functions in assembly for x86. As a result, clang is unaware of these functions as they are directly assembled and linked in. We therefore manually instrumented these functions. This required minimal intervention to call the relevant function in our runtime library, while preserving the register state and return value of the functions we annotated.

4.3.4 Memory Safety in Musl

musl itself is not memory safe. As an example, strlen() reads 4 bytes at a time as an optimization. This means that it can overread by as many as 3 bytes. To prevent this, we modify strlen() to read byte by byte. Our experiments show that this does not effect performance for strings of length less than $\mathcal{O}(10^{5})$. Other str* functions overread as well when looking for the NULL terminator.

5 Evaluation

We evaluate CUP along two axes. First, we show that its performance is competitive with existing sanitizers. Further, our performance is markedly better than other tools that provide both Precision and Object Awareness, given that CUP provides Comprehensive Coverage. Second, we demonstrate our Exactness, showing that we have no false positives and two orders of magnitude fewer false negatives than existing open source tools on the NIST Juliet suite. Furthermore, the false negatives that we do register are architecturally dependent, and do not actually represent a failure to fulfill memory safety requirements.

All experiments were run on Ubuntu 14.04 with a 3.40GHz Intel Core i7-6700 CPU and 16GB of memory.

5.1 Performance

Performance is an important requirement for any usable sanitizer. We evaluate the performance of CUP with musl. For comparison, we also measure the performance overhead of similar sanitizers, using glibc as the baseline. AddressSanitizer is the closest open source related work that is compatible with LLVM-4.0.0-rc1. SoftBound+CETS is open source, but relies on LLVM-3.4 and can only run a small subset of SPEC CPU2006 benchmarks. Its performance results are reported here, but are not directly comparable. Low-Fat Pointers is not yet open source.

Table 3 summarizes the performance results for CUP. We have 158% overhead vs baseline, compared to 38% for AddressSanitizer. Comparing CUP directly with AddressSanitizer Table 4 shows that we have 83% overhead relative to AddressSanitizer. On benchmarks where both run, CUP is faster than SoftBound+CETS. Further, compared to these existing tools, we offer stronger, more precise coverage § 5.2. As we show in § 5.2, CUP has 10x the coverage of SoftBound+CETS, for less overhead. Low-Fat Pointers [9] reports 16% to 62% overhead depending on their optimization level. They achieve this by using clever alignment tricks to avoid metadata look ups. This has two drawbacks: (i) they round allocations up to the nearest power of two, losing precision for bounds checks, and (ii) their design cannot support temporal checks which require metadata.

5.2 Juliet Suite

NIST maintains the Juliet test suite, a large collection of C source code that demonstrates common programming practices that lead to memory vulnerabilities, organized by Common Weakness Enumeration (CWE) numbers [15]. Every example comes with two versions: one that exhibits the bug and one that is patched. We extracted the subset of tests for heap and stack buffers111The tests that match the following regular expression: CWE(121$|$122)+((?!socket).)*(.c)$. We compiled all sources with our pass, as well as with SoftBound+CETS222git commit 9a9c09f04e16f2d1ef3a906fd138a7b89df44996 and with AddressSanitizer [35]. Every patched test should execute normally. If a memory protection mechanism prematurely kills a patched test, we call it a false positive. Conversely, every buggy test should be stopped by the memory safety mechanism. All three memory protection mechanisms kill the process in case of a memory violation. If the process is not killed, we report a false negative.

Table 5 summarizes the results. Out of 4,038 tests that should not fail, we incur no false positives. ASan and SoftBound+CETS show a 0% and 0.3% respectively. We produce a 0.1% false negative rate, while ASan produces an 8% false negative rate, and SoftBound+CETS has a 25% false negative rate.

The false positives that SoftBound+CETS registers comes from variations in how the alloca() function call is handled. alloca() is compiler dependent [17]. The failing examples use alloca which is wrapped around a static function. SoftBound+CETS uses clang 3.4 as the underlying compiler, and CUP uses clang 4.0. Consequently, SoftBound+CETS handles the examples differently, and sees the memory from alloca as invalid, while CUP does not.

Our false negatives are architecturally dependent. The examples we fail to catch attempt to allocate memory for a structure containing exactly two ints. However, erroneously, the examples use the size of a pointer to the two int structure when allocating memory. (e.g. malloc(sizeof(TwoIntStruct*))). On architectures which do not define pointers as twice the size of ints (including 32-bit x86), such a mistake would lead to a memory violation. With the x86_64 architecture, though, the size of a pointer and the size of the two int structure are the same. Thus, while semantically incorrect, no true memory violation occurs. No false positives, combined with no true false negatives shows that CUP fulfills the Exactness requirement.

ASan and SoftBound+CETS higher false negative rate results from their incomplete coverage. In particular, many of the Juliet examples involve calling libc functions to copy strings and buffers (e.g. strcpy and memcpy). Neither ASan nor SoftBound+CETS are able to protect against unsafe libc calls. Our instrumentation of libc § 4.3 allows us to properly detect memory violations in these calls. Further, our adjustment for read / write size § 4.2.2 allows us to catch additional memory safety violations.

4 provides source based on a Juliet example 333CWE121_Stack_Based_Buffer_Overflow__CWE129_rand_22_bad() that CUP properly handles, and which ASan handles incorrectly. The code allocates 10 bytes on the stack, sets the bytes to ASCII A, and then sets a random character to ASCII B. ASan protects the stack by surrounding stack objects with poison zones. However, depending on the value of data, line 5 could be outside the allocated objects and outside the poison region – a classic buffer overwrite. CUP’s analysis detects that buffer leaves main through the call to memset, and inserts the appropriate runtime checks. If data is greater than 10, those runtime checks fail.

6 Discussion

We discuss several design aspects of CUP, how we handle specific corner cases, potential for optimization, and further extensions.

Reducing instrumentation.

Prior work on reducing the amount of required runtime checks has focused on type systems. CCured [26] infers three types of pointers: safe, sequential, and wild. Safe pointers are statically proven to stay in bounds. Sequential pointers are only incremented (or decremented) – e.g., iterating over an array in a loop. All remaining pointers are classified as wild. Leveraging CCured-style type systems to further optimize memory safety solutions is left as an open problem.

Optimization through LTO.

CUP does not depend on Link Time Optimization (LTO). However, LTO makes it possible to inline functions across source files, and generally flattens code. Inlining would increase the effectiveness of our stack optimization and further reduce the amount of instrumented stack variables, reducing the number of IDs that a program consumes. Reducing the IDs a program uses, reduces the overhead for ID management and resources used by CUP.

Arithmetic overflow.

Our hybrid metadata scheme stores the capability ID as part of the pointer. Pointer arithmetic can potentially modify the ID, allowing an adversary to chose the metadata the pointer is checked against. To prevent this attack vector, the upper and lower 32 bits of the pointer are loaded separately. The compiler enforces that any arithmetic operations only happen on the lower 32 bits, which contain the pointer’s offset. This protects our capability ID from manipulation by an adversary.

Stronger temporal protection.

As discussed in § 3.1, it is possible (albeit difficult) for an attacker to perform a temporal attack on software instrumented with CUP. If CUP were deployed as an active defense, the difficulty of a successful temporal attack could be increased by utilizing a randomized memory allocator like DieHard [2] or DieHarder [32]. Such allocators randomize heap allocations, making heap grooming [11, 37] much more difficult. Beyond getting the addresses to line up, the increased number of required allocations makes matching the capability IDs even harder. This renders a successful use-after-free attack highly unlikely, with minimal additional overhead. Additionally, a “lock and key” scheme [25] can be added to our metadata. Alternatively, our metadata can be extended to include a dangnull [16]-style approach that records how many references are still pointing to an object and either explicitly invalidating those references or waiting until the last reference has been overwritten before reusing IDs.

Uninstrumented code.

CUP supports linking with uninstrumented libraries. Enriched pointers are the same size as regular pointers, maintaining the ABI. Dereferencing enriched pointers in uninstrumented code results, by design, in a segmentation fault. A segmentation fault handler can, on demand, dereference individual pointers. As the memory allocator is instrumented, memory allocations in uninstrumented code will return enriched pointers. Stack allocations on the other hand will not be protected in uninstrumented code. While this option allows compatibility, it clearly results in high performance overhead. Note that, thanks to support for recompiling all user-space code, this situation is limited to legacy code.

Uninstrumented globals.

We currently cannot handle the corner case of external global arrays defined in uninstrumented code. Our pass assumes such arrays have been instrumented, and so adds an extern pointer to them § 4.2.1. If the global was defined in uninstrumented code, this assumption does not hold. Consequently, our extern pointer does not exist, and the code will not link. Note that such a situation is unlikely, as we support full user-space instrumentation.

Assembly code.

CUP automatically instruments all code written in high level languages; our analysis pass runs on LLVM IR. Our analysis does not (and cannot) instrument inline assembly and assembly files due to missing type information. We rely on the programmer to either instrument the assembly code accordingly or to fall back on supporting uninstrumented code as mentioned above.

Data races.

CUP does not protect against inter-thread races of updates to metadata (e.g., one thread frees an object while the other thread is dereferencing a pointer). We leave the design of a low-overhead metadata locking scheme as future work. Note that this limitation is shared with other sanitizers.

7 Related Work

Precision

is required to enforce spatial memory safety (bounds checks). There is a class of memory safety solutions that only approximately enforce this property [1, 35, 9]. These solutions make use of techniques such as poisoned zones – detecting spatial violations within limits, or rounding allocation size – which causes the executed program to differ from the programmers intent and results in challenges when trying to handle intra-array and intra-struct checks. By changing the memory layout and not enforcing exact bounds, these solutions are not faithful to the programmer’s intent. [24] is the existing solution which best satisfies this requirement, though it has other limitations.

Object-based memory checking [5, 7, 10] keeps track of metadata on a per-object basis. Since the meta-data is associated on a per object level, every pointer to the object shares the same metadata. Object layout in memory is generally left unchanged, which increases ABI compatibility. However, pointer casts and pointers to subfield struct members are unhandled [23]. SAFECode [8] is an example efficient object-based memory checking.

Recently, Intel has started to add memory safety extensions, called Intel MPX, to their processors, starting with the Skylake architecture [13]. These extensions add additional registers to store bounds information at runtime. While effective at detecting spatial memory violations, MPX is incapable of detecting temporal violations [33], and typecasts to integers are not protected. In addition, current implementations of MPX incur a large memory penalty of up to 4 times normal usage [19]. Other ISA extensions include Watchdog [22], WatchdogLite [21], HardBound [6], and Chuang et al. [4]. As a software only solution, CUP does not require extra hardware or ISA extensions.

Object Awareness

is required to prevent temporal memory safety violations (lifetime errors). This property requires remembering for every pointer whether the object to which it is assigned is still allocated. AddressSanitizer [35] and Low-Fat Pointers [9] make no attempt to do this (and their metadata does not support this property), while SoftBound+CETS [25] enforces this property. AddressSanitizer and Low-Fat Pointers do not maintain metadata either per object or per pointer. Object Awareness requires either per objecct or per pointer metadata. Consequently, they fundamentally cannot enforce temporal safety because their (current) metadata cannot be object aware.

Temporal only detectors include DangNull [16] and Undangle [3] from Microsoft. DangNull automatically nullifies all pointers to an object when it is freed. Undangle uses an early detection technique to identify unsafe pointers when they are created, instead of being used. CUP only provides a probabilistic temporal defense, however, DangNull and Undangle lack any spatial protection.

Comprehensive Coverage

is required to fully protect the program. As shown in § 2.2 stack objects are the overwhelming majority of allocations, and to this day a significant portion of memory safety Common Vulnerabilities and Exposures (CVE) are stack related. Our evaluation of SoftBound+CETS § 5.2 shows that it has poor coverage – missing many stack vulnerabilities. AddressSanitizer and Low-Fat Pointers do better. AddressSanitizer protects the stack through the use of poisoned zones, and, as illustrated in  § 5.2 cannot handle all invalid stack memory accesses. Additionally, neither of them supports compiling libc – leaving the window open for vulnerabilities such as GHOST [18]. Tripwires [27, 34, 41, 44] are a way to detect some spatial and temporal memory errors [35, 42]. Tripwires place a region of invalid memory around objects to avoid small stride overflows and underflows. Temporal violations are caught by registering memory freed as invalid, until reclaimed. Tripwires, however, miss long stride memory errors, and thus cannot be said to be completely secure.

The state-of-the-art C/C++ pointer-based memory safety scheme is SoftBound+CETS [23]. Other pointer-based schemes include CCured [26] and Cyclone [14]. CCured uses a fat pointer to store metadata, as well as programmer annotations for indicating safe casts. Unfortunately, fat pointers break the ABI, and programmer annotations can significantly increase developer time. Even with annotations, CCured fails to handle structure changes. Cyclone also uses a fat pointer scheme, but does not guarantee full memory safety. We refer to [23] for a survey of other related work on memory safety and a discussion on different pointer metadata schemes.

8 Conclusion

We present CUP, a C/C++ memory safety mechanism that provides full user-space protection, including libc, and strong probabilistic temporal protection. It is the first such mechanism that satisfies all requirements for a complete memory safety solution, while incurring only modest performance overhead compared with the state-of-the-art. CUP is exact, object aware, comprehensive in its coverage, and precise. We fully protect all user-space memory, including the stack, which, despite being the largest source of pointers, remained largely unprotected. Finally, we produce zero false negatives and zero authentic false positives in the NIST Juliet Vulnerability example suite, which represents a significant advancement over existing memory safety mechanisms.