Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning

TL;DR

This paper presents a novel counterfactual data augmentation method inspired by Variation Theory to improve active learning efficiency, especially in low-data scenarios, by synthesizing artificial data points that highlight key features.

Contribution

It introduces a neuro-symbolic pipeline combining LLMs and rule-based models to generate synthetic data for active learning, addressing the cold start problem.

Findings

Significantly improves performance with fewer labeled data

Reduces the impact of data augmentation as data size increases

Addresses cold start problem in active learning

Abstract

Active Learning (AL) allows models to learn interactively from user feedback. This paper introduces a counterfactual data augmentation approach to AL, particularly addressing the selection of datapoints for user querying, a pivotal concern in enhancing data efficiency. Our approach is inspired by Variation Theory, a theory of human concept learning that emphasizes the essential features of a concept by focusing on what stays the same and what changes. Instead of just querying with existing datapoints, our approach synthesizes artificial datapoints that highlight potential key similarities and differences among labels using a neuro-symbolic pipeline combining large language models (LLMs) and rule-based models. Through an experiment in the example domain of text classification, we show that our approach achieves significantly higher performance when there are fewer annotated data. As the annotated training data gets larger the impact of the generated data starts to diminish showing its capability to address the cold start problem in AL. This research sheds light on integrating theories of human learning into the optimization of AL.