Adversarial Representation Learning With Closed-Form Solvers

TL;DR

This paper introduces OptNet-ARL, a stable, efficient, closed-form solution for adversarial representation learning that improves convergence speed and balances utility with fairness and privacy.

Contribution

It proposes a novel closed-form solver-based approach for adversarial representation learning, avoiding unstable iterative methods.

Findings

OptNet-ARL converges three to five times faster than traditional methods.

It effectively balances utility and fairness in classification tasks.

The approach generalizes to multiple tasks and attributes.

Abstract

Adversarial representation learning aims to learn data representations for a target task while removing unwanted sensitive information at the same time. Existing methods learn model parameters iteratively through stochastic gradient descent-ascent, which is often unstable and unreliable in practice. To overcome this challenge, we adopt closed-form solvers for the adversary and target task. We model them as kernel ridge regressors and analytically determine an upper-bound on the optimal dimensionality of representation. Our solution, dubbed OptNet-ARL, reduces to a stable one one-shot optimization problem that can be solved reliably and efficiently. OptNet-ARL can be easily generalized to the case of multiple target tasks and sensitive attributes. Numerical experiments, on both small and large scale datasets, show that, from an optimization perspective, OptNet-ARL is stable and exhibits three to five times faster convergence. Performance wise, when the target and sensitive attributes are dependent, OptNet-ARL learns representations that offer a better trade-off front between (a) utility and bias for fair classification and (b) utility and privacy by mitigating leakage of private information than existing solutions.