Towards efficient representation identification in supervised learning

TL;DR

This paper investigates how to achieve disentangled representations in supervised learning with limited auxiliary information, challenging existing assumptions that extensive auxiliary data is necessary for effective disentanglement.

Contribution

It introduces a theoretical and experimental framework demonstrating that disentanglement is possible with less auxiliary information than previously assumed, even without conditional independence.

Findings

Disentanglement can be achieved with auxiliary information less than the latent dimension.

Theoretical results support disentanglement without conditional independence.

Experimental validation confirms the effectiveness of reduced auxiliary information.

Abstract

Humans have a remarkable ability to disentangle complex sensory inputs (e.g., image, text) into simple factors of variation (e.g., shape, color) without much supervision. This ability has inspired many works that attempt to solve the following question: how do we invert the data generation process to extract those factors with minimal or no supervision? Several works in the literature on non-linear independent component analysis have established this negative result; without some knowledge of the data generation process or appropriate inductive biases, it is impossible to perform this inversion. In recent years, a lot of progress has been made on disentanglement under structural assumptions, e.g., when we have access to auxiliary information that makes the factors of variation conditionally independent. However, existing work requires a lot of auxiliary information, e.g., in supervised classification, it prescribes that the number of label classes should be at least equal to the total dimension of all factors of variation. In this work, we depart from these assumptions and ask: a) How can we get disentanglement when the auxiliary information does not provide conditional independence over the factors of variation? b) Can we reduce the amount of auxiliary information required for disentanglement? For a class of models where auxiliary information does not ensure conditional independence, we show theoretically and experimentally that disentanglement (to a large extent) is possible even when the auxiliary information dimension is much less than the dimension of the true latent representation.