Correlation-based Intrinsic Evaluation of Word Vector Representations

TL;DR

This paper presents QVEC-CCA, a new intrinsic evaluation metric for word vectors that correlates learned representations with linguistic features, showing better alignment with downstream task performance than existing methods.

Contribution

The paper introduces QVEC-CCA, an innovative correlation-based intrinsic evaluation method for word vectors that improves consistency with extrinsic task performance.

Findings

QVEC-CCA scores correlate well with semantic and syntactic tasks.

QVEC-CCA outperforms existing intrinsic evaluation methods.

The proposed metric provides a more reliable assessment of word vector quality.

Abstract

We introduce QVEC-CCA--an intrinsic evaluation metric for word vector representations based on correlations of learned vectors with features extracted from linguistic resources. We show that QVEC-CCA scores are an effective proxy for a range of extrinsic semantic and syntactic tasks. We also show that the proposed evaluation obtains higher and more consistent correlations with downstream tasks, compared to existing approaches to intrinsic evaluation of word vectors that are based on word similarity.