Embedding Lexical Features via Low-Rank Tensors

TL;DR

This paper introduces a tensor-based model for lexical features in NLP that captures conjunctions among word, context, and label parts, reducing parameters and enhancing prediction speed, achieving state-of-the-art results.

Contribution

It proposes a low-rank tensor approach to represent complex lexical features, improving efficiency and performance in NLP tasks.

Findings

Achieved state-of-the-art results on relation extraction, PP-attachment, and preposition disambiguation.

Reduced parameter space and increased prediction speed through low-rank tensor approximations.

Effectively handled features with mixed-length n-grams.

Abstract

Modern NLP models rely heavily on engineered features, which often combine word and contextual information into complex lexical features. Such combination results in large numbers of features, which can lead to over-fitting. We present a new model that represents complex lexical features---comprised of parts for words, contextual information and labels---in a tensor that captures conjunction information among these parts. We apply low-rank tensor approximations to the corresponding parameter tensors to reduce the parameter space and improve prediction speed. Furthermore, we investigate two methods for handling features that include $n$-grams of mixed lengths. Our model achieves state-of-the-art results on tasks in relation extraction, PP-attachment, and preposition disambiguation.