Improving Agreement and Disagreement Identification in Online Discussions with A Socially-Tuned Sentiment Lexicon

TL;DR

This paper introduces an isotonic CRF model combined with a socially-tuned sentiment lexicon to improve agreement and disagreement detection in online discussions, outperforming previous methods on Wikipedia Talk pages and online debates.

Contribution

The paper presents a novel isotonic CRF model and a socially-tuned lexicon that together enhance agreement/disagreement detection in online discussions.

Findings

The isotonic CRF model achieves F1 scores of 0.74 for agreement and 0.67 for disagreement.

The socially-tuned lexicon improves detection performance over general sentiment lexicons.

The model outperforms state-of-the-art approaches on two diverse online discussion datasets.

Abstract

We study the problem of agreement and disagreement detection in online discussions. An isotonic Conditional Random Fields (isotonic CRF) based sequential model is proposed to make predictions on sentence- or segment-level. We automatically construct a socially-tuned lexicon that is bootstrapped from existing general-purpose sentiment lexicons to further improve the performance. We evaluate our agreement and disagreement tagging model on two disparate online discussion corpora -- Wikipedia Talk pages and online debates. Our model is shown to outperform the state-of-the-art approaches in both datasets. For example, the isotonic CRF model achieves F1 scores of 0.74 and 0.67 for agreement and disagreement detection, when a linear chain CRF obtains 0.58 and 0.56 for the discussions on Wikipedia Talk pages.