A Confidence-based Acquisition Model for Self-supervised Active Learning and Label Correction

TL;DR

CAMEL is a novel active learning framework that reduces annotation effort and improves data quality for sequential tasks by combining expert labeling, self-supervision, and label correction.

Contribution

It introduces a confidence-based acquisition model that minimizes expert annotation, enables self-supervision, and incorporates label correction for data cleaning in sequential problems.

Findings

CAMEL outperforms baseline methods in efficiency.

It improves dataset quality through label correction.

Effective on dialogue belief tracking tasks.

Abstract

Supervised neural approaches are hindered by their dependence on large, meticulously annotated datasets, a requirement that is particularly cumbersome for sequential tasks. The quality of annotations tends to deteriorate with the transition from expert-based to crowd-sourced labelling. To address these challenges, we present CAMEL (Confidence-based Acquisition Model for Efficient self-supervised active Learning), a pool-based active learning framework tailored to sequential multi-output problems. CAMEL possesses two core features: (1) it requires expert annotators to label only a fraction of a chosen sequence, and (2) it facilitates self-supervision for the remainder of the sequence. By deploying a label correction mechanism, CAMEL can also be utilised for data cleaning. We evaluate CAMEL on two sequential tasks, with a special emphasis on dialogue belief tracking, a task plagued by the constraints of limited and noisy datasets. Our experiments demonstrate that CAMEL significantly outperforms the baselines in terms of efficiency. Furthermore, the data corrections suggested by our method contribute to an overall improvement in the quality of the resulting datasets.