LitBench: A Graph-Centric Large Language Model Benchmarking Tool For Literature Tasks

TL;DR

LitBench is a flexible benchmarking tool that creates domain-specific literature graphs and tasks to evaluate and improve large language models' ability to understand and reason across specialized literature domains.

Contribution

The paper introduces LitBench, a novel tool for generating domain-specific literature datasets and tasks, enabling targeted training and evaluation of literature-focused LLMs.

Findings

Small domain-specific LLMs trained on LitBench datasets perform competitively with larger models.

LitBench supports flexible curation of literature graphs across various domains.

Open-source release facilitates community adoption and further research.

Abstract

While large language models (LLMs) have become the de facto framework for literature-related tasks, they still struggle to function as domain-specific literature agents due to their inability to connect pieces of knowledge and reason across domain-specific contexts, terminologies, and nomenclatures. This challenge underscores the need for a tool that facilitates such domain-specific adaptation and enables rigorous benchmarking across literature tasks. To that end, we introduce LitBench, a benchmarking tool designed to enable the development and evaluation of domain-specific LLMs tailored to literature-related tasks. At its core, LitBench uses a data curation process that generates domain-specific literature sub-graphs and constructs training and evaluation datasets based on the textual attributes of the resulting nodes and edges. The tool is designed for flexibility, supporting the curation of literature graphs across any domain chosen by the user, whether high-level fields or specialized interdisciplinary areas. In addition to dataset curation, LitBench defines a comprehensive suite of literature tasks, ranging from node and edge level analyses to advanced applications such as related work generation. These tasks enable LLMs to internalize domain-specific knowledge and relationships embedded in the curated graph during training, while also supporting rigorous evaluation of model performance. Our results show that small domain-specific LLMs trained and evaluated on LitBench datasets achieve competitive performance compared to state-of-the-art models like GPT-4o and DeepSeek-R1. To enhance accessibility and ease of use, we open-source the tool along with an AI agent tool that streamlines data curation, model training, and evaluation.