FlexOS: Towards Flexible OS Isolation

TL;DR

FlexOS is a flexible operating system that allows customizable safety and isolation strategies at deployment, enabling tailored security-performance trade-offs and easier adaptation to new hardware mechanisms.

Contribution

It introduces a modular OS architecture that can be configured post-deployment and provides an exploration technique for safety/performance optimization.

Findings

FlexOS supports numerous configurations for applications like Redis, Nginx, and SQLite.

The exploration technique efficiently identifies the safest configurations within performance constraints.

FlexOS performs comparably or better than existing baselines under similar configurations.

Abstract

At design time, modern operating systems are locked in a specific safety and isolation strategy that mixes one or more hardware/software protection mechanisms (e.g. user/kernel separation); revisiting these choices after deployment requires a major refactoring effort. This rigid approach shows its limits given the wide variety of modern applications' safety/performance requirements, when new hardware isolation mechanisms are rolled out, or when existing ones break. We present FlexOS, a novel OS allowing users to easily specialize the safety and isolation strategy of an OS at compilation/deployment time instead of design time. This modular LibOS is composed of fine-grained components that can be isolated via a range of hardware protection mechanisms with various data sharing strategies and additional software hardening. The OS ships with an exploration technique helping the user navigate the vast safety/performance design space it unlocks. We implement a prototype of the system and demonstrate, for several applications (Redis/Nginx/SQLite), FlexOS' vast configuration space as well as the efficiency of the exploration technique: we evaluate 80 FlexOS configurations for Redis and show how that space can be probabilistically subset to the 5 safest ones under a given performance budget. We also show that, under equivalent configurations, FlexOS performs similarly or better than several baselines/competitors.