A Framework for Double-Blind Federated Adaptation of Foundation Models

TL;DR

BlindFed is a privacy-preserving framework for collaborative adaptation of foundation models using homomorphic encryption, enabling data owners and service providers to protect sensitive information while fine-tuning models.

Contribution

It introduces a novel homomorphic encryption-compatible architecture and a split learning approach for secure, privacy-preserving foundation model adaptation.

Findings

Feasible on four image classification datasets

Demonstrates privacy protection against model extraction attacks

Highlights high communication and computational costs

Abstract

Foundation models (FMs) excel in zero-shot tasks but benefit from task-specific adaptation. However, privacy concerns prevent data sharing among multiple data owners, and proprietary restrictions prevent the learning service provider (LSP) from sharing the FM. In this work, we propose BlindFed, a framework enabling collaborative FM adaptation while protecting both parties: data owners do not access the FM or each other's data, and the LSP does not see sensitive task data. BlindFed relies on fully homomorphic encryption (FHE) and consists of three key innovations: (i) FHE-friendly architectural modifications via polynomial approximations and low-rank adapters, (ii) a two-stage split learning approach combining offline knowledge distillation and online encrypted inference for adapter training without backpropagation through the FM, and (iii) a privacy-boosting scheme using sample permutations and stochastic block sampling to mitigate model extraction attacks. Empirical results on four image classification datasets demonstrate the practical feasibility of the BlindFed framework, albeit at a high communication cost and large computational complexity for the LSP.