Beyond Good and Evil: Formalizing the Security Guarantees of Compartmentalizing Compilation

TL;DR

This paper introduces a formal security property called secure compartmentalizing compilation (SCC) that characterizes the guarantees of compartmentalizing compilation, enhancing understanding of low-level attack defenses.

Contribution

It formalizes the SCC property, extending fully abstract compilation to better suit compartmentalization, and demonstrates its proof through a compiler example.

Findings

Defines the SCC security property for compartmentalizing compilation.

Adapts proof techniques from fully abstract compilation to establish SCC.

Provides a compiler example demonstrating SCC in practice.

Abstract

Compartmentalization is good security-engineering practice. By breaking a large software system into mutually distrustful components that run with minimal privileges, restricting their interactions to conform to well-defined interfaces, we can limit the damage caused by low-level attacks such as control-flow hijacking. When used to defend against such attacks, compartmentalization is often implemented cooperatively by a compiler and a low-level compartmentalization mechanism. However, the formal guarantees provided by such compartmentalizing compilation have seen surprisingly little investigation. We propose a new security property, secure compartmentalizing compilation (SCC), that formally characterizes the guarantees provided by compartmentalizing compilation and clarifies its attacker model. We reconstruct our property by starting from the well-established notion of fully abstract compilation, then identifying and lifting three important limitations that make standard full abstraction unsuitable for compartmentalization. The connection to full abstraction allows us to prove SCC by adapting established proof techniques; we illustrate this with a compiler from a simple unsafe imperative language with procedures to a compartmentalized abstract machine.