ORCCA: Optimal Randomized Canonical Correlation Analysis

TL;DR

ORCCA introduces a task-specific scoring rule for selecting random features in Canonical Correlation Analysis, leading to improved approximation and performance over traditional kernel methods, validated through theoretical guarantees and numerical experiments.

Contribution

The paper proposes ORCCA, a novel method that optimally selects random features for CCA, outperforming existing kernel approximation techniques.

Findings

ORCCA outperforms kernel CCA in expectation.

Numerical experiments show ORCCA's superior accuracy.

The method provides a principled, task-specific feature scoring rule.

Abstract

Random features approach has been widely used for kernel approximation in large-scale machine learning. A number of recent studies have explored data-dependent sampling of features, modifying the stochastic oracle from which random features are sampled. While proposed techniques in this realm improve the approximation, their suitability is often verified on a single learning task. In this paper, we propose a task-specific scoring rule for selecting random features, which can be employed for different applications with some adjustments. We restrict our attention to Canonical Correlation Analysis (CCA), and we provide a novel, principled guide for finding the score function maximizing the canonical correlations. We prove that this method, called ORCCA, can outperform (in expectation) the corresponding Kernel CCA with a default kernel. Numerical experiments verify that ORCCA is significantly superior than other approximation techniques in the CCA task.