Responsible Active Learning via Human-in-the-loop Peer Study

TL;DR

This paper introduces Peer Study Learning (PSL), a responsible active learning approach that preserves data privacy by using a human-in-the-loop architecture with peer students, improving model stability and performance.

Contribution

The paper proposes a novel privacy-preserving active learning method with peer students and a discrepancy-based sampling criterion, enhancing data privacy and model robustness.

Findings

PSL outperforms existing active learning methods in accuracy.

PSL effectively preserves data privacy in sensitive settings.

Peer students improve active learner stability.

Abstract

Active learning has been proposed to reduce data annotation efforts by only manually labelling representative data samples for training. Meanwhile, recent active learning applications have benefited a lot from cloud computing services with not only sufficient computational resources but also crowdsourcing frameworks that include many humans in the active learning loop. However, previous active learning methods that always require passing large-scale unlabelled data to cloud may potentially raise significant data privacy issues. To mitigate such a risk, we propose a responsible active learning method, namely Peer Study Learning (PSL), to simultaneously preserve data privacy and improve model stability. Specifically, we first introduce a human-in-the-loop teacher-student architecture to isolate unlabelled data from the task learner (teacher) on the cloud-side by maintaining an active learner (student) on the client-side. During training, the task learner instructs the light-weight active learner which then provides feedback on the active sampling criterion. To further enhance the active learner via large-scale unlabelled data, we introduce multiple peer students into the active learner which is trained by a novel learning paradigm, including the In-Class Peer Study on labelled data and the Out-of-Class Peer Study on unlabelled data. Lastly, we devise a discrepancy-based active sampling criterion, Peer Study Feedback, that exploits the variability of peer students to select the most informative data to improve model stability. Extensive experiments demonstrate the superiority of the proposed PSL over a wide range of active learning methods in both standard and sensitive protection settings.