Two Step CCA: A new spectral method for estimating vector models of words

TL;DR

This paper introduces a novel spectral method called Two Step CCA for learning low-dimensional word representations from co-occurrence data, demonstrating theoretical advantages and empirical effectiveness in NLP tasks.

Contribution

The paper proposes a new two-step CCA-based spectral method that improves sample complexity and representation quality over traditional single-step approaches.

Findings

Lower sample complexity than single-step methods

Effective in POS tagging tasks

Improves sentiment classification accuracy

Abstract

Unlabeled data is often used to learn representations which can be used to supplement baseline features in a supervised learner. For example, for text applications where the words lie in a very high dimensional space (the size of the vocabulary), one can learn a low rank "dictionary" by an eigen-decomposition of the word co-occurrence matrix (e.g. using PCA or CCA). In this paper, we present a new spectral method based on CCA to learn an eigenword dictionary. Our improved procedure computes two set of CCAs, the first one between the left and right contexts of the given word and the second one between the projections resulting from this CCA and the word itself. We prove theoretically that this two-step procedure has lower sample complexity than the simple single step procedure and also illustrate the empirical efficacy of our approach and the richness of representations learned by our Two Step CCA (TSCCA) procedure on the tasks of POS tagging and sentiment classification.