Improving Active Learning in Systematic Reviews

TL;DR

This paper enhances active learning techniques for systematic reviews, reducing manual effort by proposing a novel algorithm and feature selection method, validated on extensive datasets across disciplines.

Contribution

It introduces a new active learning algorithm and a feature selection mechanism tailored for systematic review screening tasks.

Findings

Improved screening performance with the new active learning algorithm

Identified bias towards selecting similar documents in naive methods

Proposed feature extraction method selection enhances accuracy

Abstract

Systematic reviews are essential to summarizing the results of different clinical and social science studies. The first step in a systematic review task is to identify all the studies relevant to the review. The task of identifying relevant studies for a given systematic review is usually performed manually, and as a result, involves substantial amounts of expensive human resource. Lately, there have been some attempts to reduce this manual effort using active learning. In this work, we build upon some such existing techniques, and validate by experimenting on a larger and comprehensive dataset than has been attempted until now. Our experiments provide insights on the use of different feature extraction models for different disciplines. More importantly, we identify that a naive active learning based screening process is biased in favour of selecting similar documents. We aimed to improve the performance of the screening process using a novel active learning algorithm with success. Additionally, we propose a mechanism to choose the best feature extraction method for a given review.