TrustAL: Trustworthy Active Learning using Knowledge Distillation

TL;DR

TrustAL introduces a novel active learning approach that mitigates knowledge forgetting by selecting optimal teacher models from previous iterations, enhancing label quality and data selection strategies.

Contribution

It proposes a new distillation objective based on model consistency, addressing knowledge forgetting and improving active learning effectiveness.

Findings

Consistency-based distillation reduces label forgetting.

Improves uncertainty and diversity in data selection.

Reduces impact of human annotation errors.

Abstract

Active learning can be defined as iterations of data labeling, model training, and data acquisition, until sufficient labels are acquired. A traditional view of data acquisition is that, through iterations, knowledge from human labels and models is implicitly distilled to monotonically increase the accuracy and label consistency. Under this assumption, the most recently trained model is a good surrogate for the current labeled data, from which data acquisition is requested based on uncertainty/diversity. Our contribution is debunking this myth and proposing a new objective for distillation. First, we found example forgetting, which indicates the loss of knowledge learned across iterations. Second, for this reason, the last model is no longer the best teacher -- For mitigating such forgotten knowledge, we select one of its predecessor models as a teacher, by our proposed notion of "consistency". We show that this novel distillation is distinctive in the following three aspects; First, consistency ensures to avoid forgetting labels. Second, consistency improves both uncertainty/diversity of labeled data. Lastly, consistency redeems defective labels produced by human annotators.