Leveraging Linguistic Coordination in Reranking N-Best Candidates For End-to-End Response Selection Using BERT

TL;DR

This paper improves dialogue response selection by reranking BERT-generated candidates using linguistic coordination, leading to better top-choice accuracy in retrieval-based systems.

Contribution

It introduces a novel reranking method based on linguistic coordination to enhance BERT's response selection performance.

Findings

Improved R@1 accuracy over BERT baseline

Demonstrated effectiveness of linguistic coordination in reranking

Showed potential for repairing machine-generated dialogue outputs

Abstract

Retrieval-based dialogue systems select the best response from many candidates. Although many state-of-the-art models have shown promising performance in dialogue response selection tasks, there is still quite a gap between R@1 and R@10 performance. To address this, we propose to leverage linguistic coordination (a phenomenon that individuals tend to develop similar linguistic behaviors in conversation) to rerank the N-best candidates produced by BERT, a state-of-the-art pre-trained language model. Our results show an improvement in R@1 compared to BERT baselines, demonstrating the utility of repairing machine-generated outputs by leveraging a linguistic theory.