Intrinsic Subspace Evaluation of Word Embedding Representations

TL;DR

This paper proposes a new intrinsic evaluation method for word embeddings that assesses whether representations contain subspaces necessary for natural language understanding, revealing limitations of existing point-based evaluations.

Contribution

It introduces a novel subspace-based evaluation framework for word embeddings, addressing limitations of traditional similarity-based methods.

Findings

Count vector and neural models differ in subspace properties

Existing evaluations may overlook important representational features

The methodology reveals specific strengths and weaknesses of models

Abstract

We introduce a new methodology for intrinsic evaluation of word representations. Specifically, we identify four fundamental criteria based on the characteristics of natural language that pose difficulties to NLP systems; and develop tests that directly show whether or not representations contain the subspaces necessary to satisfy these criteria. Current intrinsic evaluations are mostly based on the overall similarity or full-space similarity of words and thus view vector representations as points. We show the limits of these point-based intrinsic evaluations. We apply our evaluation methodology to the comparison of a count vector model and several neural network models and demonstrate important properties of these models.