Contrastive Factor Analysis

TL;DR

This paper introduces Contrastive Factor Analysis, a novel framework combining factor analysis with contrastive learning to enhance interpretability, robustness, and uncertainty estimation in representation learning.

Contribution

It presents the first integration of factor analysis with contrastive learning, including a non-negative extension for disentangled representations.

Findings

Improves expressiveness and robustness of learned representations

Enhances interpretability through non-negative factor analysis

Demonstrates accurate uncertainty estimation in experiments

Abstract

Factor analysis, often regarded as a Bayesian variant of matrix factorization, offers superior capabilities in capturing uncertainty, modeling complex dependencies, and ensuring robustness. As the deep learning era arrives, factor analysis is receiving less and less attention due to their limited expressive ability. On the contrary, contrastive learning has emerged as a potent technique with demonstrated efficacy in unsupervised representational learning. While the two methods are different paradigms, recent theoretical analysis has revealed the mathematical equivalence between contrastive learning and matrix factorization, providing a potential possibility for factor analysis combined with contrastive learning. Motivated by the interconnectedness of contrastive learning, matrix factorization, and factor analysis, this paper introduces a novel Contrastive Factor Analysis framework, aiming to leverage factor analysis's advantageous properties within the realm of contrastive learning. To further leverage the interpretability properties of non-negative factor analysis, which can learn disentangled representations, contrastive factor analysis is extended to a non-negative version. Finally, extensive experimental validation showcases the efficacy of the proposed contrastive (non-negative) factor analysis methodology across multiple key properties, including expressiveness, robustness, interpretability, and accurate uncertainty estimation.