Intent-calibrated Self-training for Answer Selection in Open-domain Dialogues

TL;DR

This paper introduces intent-calibrated self-training (ICAST), a novel approach that uses predicted intent labels to enhance answer selection in open-domain dialogues, reducing reliance on large labeled datasets.

Contribution

The paper proposes ICAST, a new self-training method that leverages intent labels to improve answer selection accuracy with limited labeled data.

Findings

ICAST outperforms baselines with 1%, 5%, and 10% labeled data.

It improves F1 scores by 2.06% and 1.00% on two datasets with only 5% labeled data.

Extensive experiments validate the effectiveness of intent calibration in answer selection.

Abstract

Answer selection in open-domain dialogues aims to select an accurate answer from candidates. Recent success of answer selection models hinges on training with large amounts of labeled data. However, collecting large-scale labeled data is labor-intensive and time-consuming. In this paper, we introduce the predicted intent labels to calibrate answer labels in a self-training paradigm. Specifically, we propose the intent-calibrated self-training (ICAST) to improve the quality of pseudo answer labels through the intent-calibrated answer selection paradigm, in which we employ pseudo intent labels to help improve pseudo answer labels. We carry out extensive experiments on two benchmark datasets with open-domain dialogues. The experimental results show that ICAST outperforms baselines consistently with 1%, 5% and 10% labeled data. Specifically, it improves 2.06% and 1.00% of F1 score on the two datasets, compared with the strongest baseline with only 5% labeled data.