Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study

TL;DR

This study empirically evaluates how character and word-level representations can be combined using gating mechanisms to improve semantic encoding, especially for infrequent words, but finds limited impact on sentence-level tasks.

Contribution

It introduces a feature-wise sigmoid gating method for combining character and word representations and provides empirical evidence of its effectiveness for semantic similarity tasks.

Findings

Character modeling improves word and sentence representations.

Sigmoid gating effectively encodes semantic similarity.

Improved semantic similarity does not always enhance sentence-level task performance.

Abstract

In this paper we study how different ways of combining character and word-level representations affect the quality of both final word and sentence representations. We provide strong empirical evidence that modeling characters improves the learned representations at the word and sentence levels, and that doing so is particularly useful when representing less frequent words. We further show that a feature-wise sigmoid gating mechanism is a robust method for creating representations that encode semantic similarity, as it performed reasonably well in several word similarity datasets. Finally, our findings suggest that properly capturing semantic similarity at the word level does not consistently yield improved performance in downstream sentence-level tasks. Our code is available at https://github.com/jabalazs/gating