CoVeR: Learning Covariate-Specific Vector Representations with Tensor Decompositions

TL;DR

CoVeR introduces a tensor decomposition model that learns covariate-specific word embeddings, capturing additional document information such as author or venue, with improved data efficiency and interpretability over existing methods.

Contribution

The paper presents CoVeR, a novel tensor decomposition approach for jointly learning base and covariate-specific embeddings, enhancing interpretability and effectiveness in modeling covariate effects.

Findings

Outperforms standard covariate-specific embedding methods

Embeddings are topic-aligned with independent dimension meanings

Enables covariate effect analysis through topic comparison

Abstract

Word embedding is a useful approach to capture co-occurrence structures in large text corpora. However, in addition to the text data itself, we often have additional covariates associated with individual corpus documents---e.g. the demographic of the author, time and venue of publication---and we would like the embedding to naturally capture this information. We propose CoVeR, a new tensor decomposition model for vector embeddings with covariates. CoVeR jointly learns a \emph{base} embedding for all the words as well as a weighted diagonal matrix to model how each covariate affects the base embedding. To obtain author or venue-specific embedding, for example, we can then simply multiply the base embedding by the associated transformation matrix. The main advantages of our approach are data efficiency and interpretability of the covariate transformation. Our experiments demonstrate that our joint model learns substantially better covariate-specific embeddings compared to the standard approach of learning a separate embedding for each covariate using only the relevant subset of data, as well as other related methods. Furthermore, CoVeR encourages the embeddings to be "topic-aligned" in that the dimensions have specific independent meanings. This allows our covariate-specific embeddings to be compared by topic, enabling downstream differential analysis. We empirically evaluate the benefits of our algorithm on datasets, and demonstrate how it can be used to address many natural questions about covariate effects. Accompanying code to this paper can be found at http://github.com/kjtian/CoVeR.