DuckDB-SGX2: The Good, The Bad and The Ugly within Confidential Analytical Query Processing

TL;DR

This paper evaluates the performance of analytical query processing using DuckDB within Intel SGX enclaves, demonstrating acceptable overheads and providing tuning guidelines for secure, efficient execution.

Contribution

It combines columnar encryption with SGX for confidential analytics and offers practical insights and best practices for optimizing performance in this environment.

Findings

TPC-H SF30 runs with under 2x overhead

Columnar compression and encryption are effective together

Identifies key performance hazards and tuning strategies

Abstract

We provide an evaluation of an analytical workload in a confidential computing environment, combining DuckDB with two technologies: modular columnar encryption in Parquet files (data at rest) and the newest version of the Intel SGX Trusted Execution Environment (TEE), providing a hardware enclave where data in flight can be (more) securely decrypted and processed. One finding is that the "performance tax" for such confidential analytical processing is acceptable compared to not using these technologies. We eventually manage to run TPC-H SF30 with under 2x overhead compared to non-encrypted, non-enclave execution; we show that, specifically, columnar compression and encryption are a good combination. Our second finding consists of dos and don'ts to tune DuckDB to work effectively in this environment. There are various performance hazards: potentially 5x higher cache miss costs due to memory encryption inside the enclave, NUMA penalties, and highly elevated cost of swapping pages in and out of the enclave -- which is also triggered indirectly by using a non-SGX-aware malloc library.