Heterogeneous Matrix Factorization: When Features Differ by Datasets

TL;DR

This paper introduces Heterogeneous Matrix Factorization (HMF), a method for separating shared and source-specific factors in heterogeneous data sources, with theoretical guarantees and practical applications in various domains.

Contribution

The paper proposes a novel HMF algorithm that maintains orthogonality and provides convergence guarantees, addressing a gap in theoretical understanding of heterogeneous factorization methods.

Findings

HMF converges to optimal solutions close to ground truth.

HMF is easy to implement and suitable for distributed computing.

HMF demonstrates benefits in video segmentation, time-series analysis, and recommender systems.

Abstract

In myriad statistical applications, data are collected from related but heterogeneous sources. These sources share some commonalities while containing idiosyncratic characteristics. One of the most fundamental challenges in such scenarios is to recover the shared and source-specific factors. Despite the existence of a few heuristic approaches, a generic algorithm with theoretical guarantees has yet to be established. In this paper, we tackle the problem by proposing a method called Heterogeneous Matrix Factorization to separate the shared and unique factors for a class of problems. HMF maintains the orthogonality between the shared and unique factors by leveraging an invariance property in the objective. The algorithm is easy to implement and intrinsically distributed. On the theoretic side, we show that for the square error loss, HMF will converge into the optimal solutions, which are close to the ground truth. HMF can be integrated auto-encoders to learn nonlinear feature mappings. Through a variety of case studies, we showcase HMF's benefits and applicability in video segmentation, time-series feature extraction, and recommender systems.