Revisiting the Importance of Encoding Logic Rules in Sentiment Classification

TL;DR

This paper critically evaluates sentiment classification models on complex sentences, showing that contextualized embeddings like ELMo outperform logic rule-based models, and emphasizes the importance of proper experimental reproducibility.

Contribution

It demonstrates that explicit logic rules are less effective than contextualized embeddings like ELMo for sentiment analysis, highlighting the importance of reproducibility and implicit learning.

Findings

ELMo embeddings outperform logic rule-based models.

Proper averaging over random seeds is crucial for fair comparison.

ELMo implicitly learns logic rules and handles ambiguous sentiment sentences.

Abstract

We analyze the performance of different sentiment classification models on syntactically complex inputs like A-but-B sentences. The first contribution of this analysis addresses reproducible research: to meaningfully compare different models, their accuracies must be averaged over far more random seeds than what has traditionally been reported. With proper averaging in place, we notice that the distillation model described in arXiv:1603.06318v4 [cs.LG], which incorporates explicit logic rules for sentiment classification, is ineffective. In contrast, using contextualized ELMo embeddings (arXiv:1802.05365v2 [cs.CL]) instead of logic rules yields significantly better performance. Additionally, we provide analysis and visualizations that demonstrate ELMo's ability to implicitly learn logic rules. Finally, a crowdsourced analysis reveals how ELMo outperforms baseline models even on sentences with ambiguous sentiment labels.