Amobee at IEST 2018: Transfer Learning from Language Models

TL;DR

This paper presents an ensemble system using language models and LSTM networks with attention for emotion prediction in tweets, achieving first place in the WASSA 2018 shared task.

Contribution

It introduces a novel application of large Twitter-trained language models combined with LSTM and attention mechanisms for emotion classification.

Findings

Achieved top macro F1 score of 0.7145

Developed a system that effectively predicts emotions in tweets

Demonstrated the utility of transfer learning from language models

Abstract

This paper describes the system developed at Amobee for the WASSA 2018 implicit emotions shared task (IEST). The goal of this task was to predict the emotion expressed by missing words in tweets without an explicit mention of those words. We developed an ensemble system consisting of language models together with LSTM-based networks containing a CNN attention mechanism. Our approach represents a novel use of language models (specifically trained on a large Twitter dataset) to predict and classify emotions. Our system reached 1st place with a macro $\text{F}_1$ score of 0.7145.