Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers

TL;DR

This paper investigates combining differential privacy with parameter-efficient fine-tuning of TabTransformers for tabular data, demonstrating improved privacy-accuracy trade-offs and efficiency through extensive experiments.

Contribution

It introduces a framework for differentially private transfer learning of TabTransformers using PEFT methods, showing superior performance over traditional approaches.

Findings

PEFT methods outperform traditional fine-tuning in privacy-accuracy trade-offs

Significant reduction in trainable parameters with maintained accuracy

Enhanced privacy preservation in tabular data modeling

Abstract

For machine learning with tabular data, Table Transformer (TabTransformer) is a state-of-the-art neural network model, while Differential Privacy (DP) is an essential component to ensure data privacy. In this paper, we explore the benefits of combining these two aspects together in the scenario of transfer learning -- differentially private pre-training and fine-tuning of TabTransformers with a variety of parameter-efficient fine-tuning (PEFT) methods, including Adapter, LoRA, and Prompt Tuning. Our extensive experiments on the ACSIncome dataset show that these PEFT methods outperform traditional approaches in terms of the accuracy of the downstream task and the number of trainable parameters, thus achieving an improved trade-off among parameter efficiency, privacy, and accuracy. Our code is available at github.com/IBM/DP-TabTransformer.