Bias In, Bias Out? Finding Unbiased Subnetworks in Vanilla Models

TL;DR

This paper presents BISE, a method to extract bias-free subnetworks from vanilla-trained models through pruning, enabling bias mitigation without retraining or additional data, and demonstrating efficiency and effectiveness.

Contribution

Introducing BISE, a novel pruning-based technique to identify and extract unbiased subnetworks from standard models without retraining or dataset manipulation.

Findings

Subnetworks can be extracted via pruning without retraining.

Extracted subnetworks operate with less biased features.

Approach improves efficiency and maintains robust performance.

Abstract

The issue of algorithmic biases in deep learning has led to the development of various debiasing techniques, many of which perform complex training procedures or dataset manipulation. However, an intriguing question arises: is it possible to extract fair and bias-agnostic subnetworks from standard vanilla-trained models without relying on additional data, such as unbiased training set? In this work, we introduce Bias-Invariant Subnetwork Extraction (BISE), a learning strategy that identifies and isolates "bias-free" subnetworks that already exist within conventionally trained models, without retraining or finetuning the original parameters. Our approach demonstrates that such subnetworks can be extracted via pruning and can operate without modification, effectively relying less on biased features and maintaining robust performance. Our findings contribute towards efficient bias mitigation through structural adaptation of pre-trained neural networks via parameter removal, as opposed to costly strategies that are either data-centric or involve (re)training all model parameters. Extensive experiments on common benchmarks show the advantages of our approach in terms of the performance and computational efficiency of the resulting debiased model.