CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support

TL;DR

This paper explores using large language models to create hierarchical structures of scientific literature for aiding reviews, introduces a new dataset called CHIME, and demonstrates improvements with human-in-the-loop corrections.

Contribution

It presents a novel LLM-based pipeline for hierarchical organization of scientific studies, introduces the CHIME dataset, and develops a human feedback-based corrector model to enhance study assignment accuracy.

Findings

LLMs can generate promising hierarchies but need improvements in study assignment.

Expert corrections help quantify and improve LLM performance.

The corrector model improves study assignment F1 score by 12.6 points.

Abstract

Literature review requires researchers to synthesize a large amount of information and is increasingly challenging as the scientific literature expands. In this work, we investigate the potential of LLMs for producing hierarchical organizations of scientific studies to assist researchers with literature review. We define hierarchical organizations as tree structures where nodes refer to topical categories and every node is linked to the studies assigned to that category. Our naive LLM-based pipeline for hierarchy generation from a set of studies produces promising yet imperfect hierarchies, motivating us to collect CHIME, an expert-curated dataset for this task focused on biomedicine. Given the challenging and time-consuming nature of building hierarchies from scratch, we use a human-in-the-loop process in which experts correct errors (both links between categories and study assignment) in LLM-generated hierarchies. CHIME contains 2,174 LLM-generated hierarchies covering 472 topics, and expert-corrected hierarchies for a subset of 100 topics. Expert corrections allow us to quantify LLM performance, and we find that while they are quite good at generating and organizing categories, their assignment of studies to categories could be improved. We attempt to train a corrector model with human feedback which improves study assignment by 12.6 F1 points. We release our dataset and models to encourage research on developing better assistive tools for literature review.