Measuring Orthogonality in Representations of Generative Models

TL;DR

This paper introduces two new metrics, IWO and IWR, to evaluate the orthogonality of generative factors in representations, showing they better correlate with downstream task performance than traditional disentanglement metrics.

Contribution

The paper proposes novel metrics for measuring orthogonality in representations, challenging the focus on disentanglement and providing a new perspective on representation quality evaluation.

Findings

IWO and IWR metrics correlate more strongly with downstream performance.

Orthogonality of generative factors is more indicative of representation quality than disentanglement.

Extensive experiments validate the effectiveness of the proposed metrics across datasets and models.

Abstract

In unsupervised representation learning, models aim to distill essential features from high-dimensional data into lower-dimensional learned representations, guided by inductive biases. Understanding the characteristics that make a good representation remains a topic of ongoing research. Disentanglement of independent generative processes has long been credited with producing high-quality representations. However, focusing solely on representations that adhere to the stringent requirements of most disentanglement metrics, may result in overlooking many high-quality representations, well suited for various downstream tasks. These metrics often demand that generative factors be encoded in distinct, single dimensions aligned with the canonical basis of the representation space. Motivated by these observations, we propose two novel metrics: Importance-Weighted Orthogonality (IWO) and Importance-Weighted Rank (IWR). These metrics evaluate the mutual orthogonality and rank of generative factor subspaces. Throughout extensive experiments on common downstream tasks, over several benchmark datasets and models, IWO and IWR consistently show stronger correlations with downstream task performance than traditional disentanglement metrics. Our findings suggest that representation quality is closer related to the orthogonality of independent generative processes rather than their disentanglement, offering a new direction for evaluating and improving unsupervised learning models.