DialBERT: A Hierarchical Pre-Trained Model for Conversation Disentanglement

TL;DR

DialBERT is a hierarchical pre-trained model that effectively disentangles overlapping conversations by integrating local and global semantics, achieving state-of-the-art results with minimal parameter increase.

Contribution

The paper introduces DialBERT, a novel hierarchical model combining BERT and BiLSTM for conversation disentanglement, with improved accuracy over previous methods.

Findings

12% improvement in F1-Score over BERT

Achieves state-of-the-art results on a new IBM dataset

Only 3% increase in model parameters

Abstract

Disentanglement is a problem in which multiple conversations occur in the same channel simultaneously, and the listener should decide which utterance is part of the conversation he will respond to. We propose a new model, named Dialogue BERT (DialBERT), which integrates local and global semantics in a single stream of messages to disentangle the conversations that mixed together. We employ BERT to capture the matching information in each utterance pair at the utterance-level, and use a BiLSTM to aggregate and incorporate the context-level information. With only a 3% increase in parameters, a 12% improvement has been attained in comparison to BERT, based on the F1-Score. The model achieves a state-of-the-art result on the a new dataset proposed by IBM and surpasses previous work by a substantial margin.