Improving Sequence Modeling Ability of Recurrent Neural Networks via Sememes

TL;DR

This paper introduces methods to incorporate sememes into RNNs, significantly enhancing their sequence modeling capabilities and robustness across various NLP tasks.

Contribution

It proposes three novel sememe incorporation techniques for RNNs and demonstrates their effectiveness across multiple benchmark datasets.

Findings

Sememe-incorporated RNNs outperform vanilla models in language modeling and NLP tasks.

Models with sememes show higher robustness against adversarial attacks.

The proposed methods are effective across different RNN architectures, including LSTM and GRU.

Abstract

Sememes, the minimum semantic units of human languages, have been successfully utilized in various natural language processing applications. However, most existing studies exploit sememes in specific tasks and few efforts are made to utilize sememes more fundamentally. In this paper, we propose to incorporate sememes into recurrent neural networks (RNNs) to improve their sequence modeling ability, which is beneficial to all kinds of downstream tasks. We design three different sememe incorporation methods and employ them in typical RNNs including LSTM, GRU and their bidirectional variants. In evaluation, we use several benchmark datasets involving PTB and WikiText-2 for language modeling, SNLI for natural language inference and another two datasets for sentiment analysis and paraphrase detection. Experimental results show evident and consistent improvement of our sememe-incorporated models compared with vanilla RNNs, which proves the effectiveness of our sememe incorporation methods. Moreover, we find the sememe-incorporated models have higher robustness and outperform adversarial training in defending adversarial attack. All the code and data of this work can be obtained at https://github.com/thunlp/SememeRNN.