ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation

TL;DR

ARAIDA introduces an analogical reasoning approach that combines an annotation model with a KNN model to improve annotation accuracy and reduce human correction effort in interactive data annotation tasks.

Contribution

It presents a novel error-aware integration strategy that dynamically balances model predictions, enhancing annotation accuracy and efficiency over traditional methods.

Findings

Reduces human correction effort by 11.02% on average

Demonstrates adaptability across different annotation tasks and models

Improves annotation accuracy with analogical reasoning-based integration

Abstract

Human annotation is a time-consuming task that requires a significant amount of effort. To address this issue, interactive data annotation utilizes an annotation model to provide suggestions for humans to approve or correct. However, annotation models trained with limited labeled data are prone to generating incorrect suggestions, leading to extra human correction effort. To tackle this challenge, we propose Araida, an analogical reasoning-based approach that enhances automatic annotation accuracy in the interactive data annotation setting and reduces the need for human corrections. Araida involves an error-aware integration strategy that dynamically coordinates an annotation model and a k-nearest neighbors (KNN) model, giving more importance to KNN's predictions when predictions from the annotation model are deemed inaccurate. Empirical studies demonstrate that Araida is adaptable to different annotation tasks and models. On average, it reduces human correction labor by 11.02% compared to vanilla interactive data annotation methods.