Invariant Representations through Adversarial Forgetting

TL;DR

This paper introduces an adversarial forgetting method that encourages neural networks to forget unwanted data factors, leading to invariant representations and improved performance across various datasets and tasks.

Contribution

It presents a novel adversarial forgetting mechanism that acts as an information bottleneck to induce invariance in deep neural networks.

Findings

Achieves state-of-the-art invariance learning performance.

Effective in handling nuisance and bias factors.

Demonstrates versatility across multiple datasets and tasks.

Abstract

We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism. We show that the forgetting mechanism serves as an information-bottleneck, which is manipulated by the adversarial training to learn invariance to unwanted factors. Empirical results show that the proposed framework achieves state-of-the-art performance at learning invariance in both nuisance and bias settings on a diverse collection of datasets and tasks.