Learning ABCs: Approximate Bijective Correspondence for isolating factors of variation with weak supervision

TL;DR

This paper introduces a novel weak supervision algorithm that learns to isolate active factors of variation by establishing correspondences between data sets, enabling effective representation learning even with limited or domain-shifted data.

Contribution

The proposed method leverages set-based weak supervision to isolate active factors of variation, improving representation learning and enabling domain transfer without explicit annotations.

Findings

Effective in synthetic-to-real pose transfer without pose labels.

Enhances supervised learning by strengthening intermediate representations.

Operates well with limited set-supervised natural images.

Abstract

Representational learning forms the backbone of most deep learning applications, and the value of a learned representation is intimately tied to its information content regarding different factors of variation. Finding good representations depends on the nature of supervision and the learning algorithm. We propose a novel algorithm that utilizes a weak form of supervision where the data is partitioned into sets according to certain inactive (common) factors of variation which are invariant across elements of each set. Our key insight is that by seeking correspondence between elements of different sets, we learn strong representations that exclude the inactive factors of variation and isolate the active factors that vary within all sets. As a consequence of focusing on the active factors, our method can leverage a mix of set-supervised and wholly unsupervised data, which can even belong to a different domain. We tackle the challenging problem of synthetic-to-real object pose transfer, without pose annotations on anything, by isolating pose information which generalizes to the category level and across the synthetic/real domain gap. The method can also boost performance in supervised settings, by strengthening intermediate representations, as well as operate in practically attainable scenarios with set-supervised natural images, where quantity is limited and nuisance factors of variation are more plentiful.