FairNVT: Improving Fairness via Noise Injection in Vision Transformers

TL;DR

FairNVT introduces a lightweight framework that enhances fairness in transformer models by injecting calibrated noise into sensitive embeddings, reducing bias while maintaining accuracy across vision and language tasks.

Contribution

It proposes a novel debiasing method that connects representation and prediction fairness, using noise injection and regularization in pretrained transformers.

Findings

Reduces sensitive-attribute attacker accuracy across datasets.

Improves demographic parity and equalized odds metrics.

Maintains high task performance while reducing bias.

Abstract

This paper presents FairNVT, a lightweight debiasing framework for pretrained transformer-based encoders that improves both representation and prediction level fairness while preserving task accuracy. Unlike many existing debiasing approaches that address these notions separately, we argue they are inherently connected: suppressing sensitive information at the representation level can facilitate fairer predictions. Our approach learns task-relevant and sensitive embeddings via lightweight adapters, applies calibrated Gaussian noise to the sensitive embedding, and fuses it with the task representation. Together with orthogonality constraints and fairness regularization, these components jointly reduce sensitive-attribute leakage in the learned embeddings and encourage fairer downstream predictions. The framework is compatible with a wide range of pretrained transformer encoders. Across three datasets spanning vision and language, FairNVT reduces sensitive-attribute attacker accuracy, improves demographic-parity and equalized-odds metrics, and maintains high task performance.