Disentanglement by means of action-induced representations

TL;DR

This paper introduces action-induced representations (AIRs) and a variational AIR (VAIR) architecture, enabling provable disentanglement of physical system factors through experiments, surpassing standard VAEs in interpretability and action dependence modeling.

Contribution

The paper proposes AIRs and VAIR, a novel framework and architecture that achieve provable disentanglement by leveraging action-based experiments, addressing limitations of traditional VAEs.

Findings

VAIR can extract AIRs with provable disentanglement.

VAIR captures action dependence of generative factors.

Standard VAEs fail to achieve similar disentanglement.

Abstract

Learning interpretable representations with variational autoencoders (VAEs) is a major goal of representation learning. The main challenge lies in obtaining disentangled representations, where each latent dimension corresponds to a distinct generative factor. This difficulty is fundamentally tied to the inability to perform nonlinear independent component analysis. Here, we introduce the framework of action-induced representations (AIRs) which models representations of physical systems given experiments (or actions) that can be performed on them. We show that, in this framework, we can provably disentangle degrees of freedom w.r.t. their action dependence. We further introduce a variational AIR architecture (VAIR) that can extract AIRs and therefore achieve provable disentanglement where standard VAEs fail. Beyond state representation, VAIR also captures the action dependence of the underlying generative factors, directly linking experiments to the degrees of freedom they influence.