Joint Dimensionality Reduction for Two Feature Vectors

TL;DR

This paper introduces a simple SVD-based method for joint dimensionality reduction of two feature vectors in supervised learning, providing theoretical guarantees on sample complexity without needing to specify nonlinear link functions.

Contribution

It proposes a novel, straightforward algorithm for joint dimensionality reduction in multi-modal data with theoretical analysis of its sample complexity.

Findings

SVD-based algorithm accurately estimates embeddings under sample complexity conditions

Sample complexities are similar across different link functions, differing only by constants

The method applies to various supervised learning scenarios with two feature sets

Abstract

Many machine learning problems, especially multi-modal learning problems, have two sets of distinct features (e.g., image and text features in news story classification, or neuroimaging data and neurocognitive data in cognitive science research). This paper addresses the joint dimensionality reduction of two feature vectors in supervised learning problems. In particular, we assume a discriminative model where low-dimensional linear embeddings of the two feature vectors are sufficient statistics for predicting a dependent variable. We show that a simple algorithm involving singular value decomposition can accurately estimate the embeddings provided that certain sample complexities are satisfied, without specifying the nonlinear link function (regressor or classifier). The main results establish sample complexities under multiple settings. Sample complexities for different link functions only differ by constant factors.