TENNOR: Trustworthy Execution for Neural Networks through Obliviousness and Retrievals

TL;DR

TENNOR is a system that enables privacy-preserving neural network training in untrusted environments by eliminating access-pattern leakage through doubly oblivious primitives and efficient locality-sensitive hashing techniques.

Contribution

It introduces a novel co-designed training pipeline with doubly oblivious primitives and a new multi-probe LSH scheme, MP-WTA, to reduce memory and improve efficiency.

Findings

Achieves 13x to 470x speedup over Path ORAM baseline.

Reduces training time from 208 hours to about 26 minutes.

Maintains model accuracy while ensuring access-pattern privacy.

Abstract

Training wide neural networks on sensitive data in untrusted cloud environments requires simultaneously achieving computational efficiency and rigorous privacy guarantees. Sparsification techniques, essential for scalable training of wide layers, expose input-dependent memory-access patterns (i.e., leakage) that are visible and can be exploited by a host OS/hypervisor, even when computation is protected by a Trusted Execution Environment. We present TENNOR, a system that resolves this tension by co-designing the neural network training pipeline with doubly oblivious primitives, eliminating access-pattern leakage while also utilizing adaptive sparsification. TENNOR recasts sparse neuron activation as a locality-sensitive hashing (LSH) retrieval problem, reducing secure sparsification to doubly oblivious accesses over an LSH data structure. To eliminate the prohibitive storage cost of ``multi-table'' LSH, we introduce Multi-Probe Winner-Take-All (MP-WTA): the first multi-probe scheme for rank-based LSH, achieving a 50x reduction in (hash table) memory while preserving model accuracy. We evaluate TENNOR on extreme multi-label classification benchmarks with output layers of up to 325K neurons inside an Intel TDX Trusted Domain, achieving speedups of 13x--470x over a Path ORAM baseline and reducing a 208-hour run to about 26 minutes.