Multiview Self-Representation Learning across Heterogeneous Views

TL;DR

This paper introduces a multiview self-representation learning method that leverages heterogeneous pretrained models to learn invariant features from unlabeled visual data, improving transfer learning performance.

Contribution

The paper proposes a novel MSRL approach that uses self-representation and an assignment consistency scheme to unify features from diverse pretrained models in an unsupervised manner.

Findings

MSRL outperforms state-of-the-art methods on benchmark datasets.

The information-passing mechanism effectively aggregates multiview features.

Theoretical analysis supports the robustness of the proposed scheme.

Abstract

Features of the same sample generated by different pretrained models often exhibit inherently distinct feature distributions because of discrepancies in the model pretraining objectives or architectures. Learning invariant representations from large-scale unlabeled visual data with various pretrained models in a fully unsupervised transfer manner remains a significant challenge. In this paper, we propose a multiview self-representation learning (MSRL) method in which invariant representations are learned by exploiting the self-representation property of features across heterogeneous views. The features are derived from large-scale unlabeled visual data through transfer learning with various pretrained models and are referred to as heterogeneous multiview data. An individual linear model is stacked on top of its corresponding frozen pretrained backbone. We introduce an information-passing mechanism that relies on self-representation learning to support feature aggregation over the outputs of the linear model. Moreover, an assignment probability distribution consistency scheme is presented to guide multiview self-representation learning by exploiting complementary information across different views. Consequently, representation invariance across different linear models is enforced through this scheme. In addition, we provide a theoretical analysis of the information-passing mechanism, the assignment probability distribution consistency and the incremental views. Extensive experiments with multiple benchmark visual datasets demonstrate that the proposed MSRL method consistently outperforms several state-of-the-art approaches.