Stable Distribution Alignment Using the Dual of the Adversarial Distance

TL;DR

This paper introduces a dual formulation for distribution alignment that improves stability and convergence over traditional adversarial methods, demonstrated on synthetic and real-image domain adaptation tasks.

Contribution

It proposes replacing the adversarial maximization with its dual, enhancing stability and convergence in distribution alignment tasks, especially with linear discriminators.

Findings

Dual formulation yields more stable convergence than primal min-max objectives.

The method performs well on synthetic point cloud alignment.

It improves domain adaptation results on digit image datasets.

Abstract

Methods that align distributions by minimizing an adversarial distance between them have recently achieved impressive results. However, these approaches are difficult to optimize with gradient descent and they often do not converge well without careful hyperparameter tuning and proper initialization. We investigate whether turning the adversarial min-max problem into an optimization problem by replacing the maximization part with its dual improves the quality of the resulting alignment and explore its connections to Maximum Mean Discrepancy. Our empirical results suggest that using the dual formulation for the restricted family of linear discriminators results in a more stable convergence to a desirable solution when compared with the performance of a primal min-max GAN-like objective and an MMD objective under the same restrictions. We test our hypothesis on the problem of aligning two synthetic point clouds on a plane and on a real-image domain adaptation problem on digits. In both cases, the dual formulation yields an iterative procedure that gives more stable and monotonic improvement over time.