BioCoref: Benchmarking Biomedical Coreference Resolution with LLMs

TL;DR

This paper evaluates large language models for biomedical coreference resolution, revealing their strengths and limitations, and compares generative approaches with traditional discriminative models using domain-specific prompts.

Contribution

It provides a comprehensive benchmark of LLMs on biomedical coreference resolution, introducing prompt-based techniques and comparing them with SpanBERT.

Findings

LLMs perform well on surface-level coreference tasks with domain prompts.

Long-range context and ambiguity remain challenging for LLMs.

Entity-augmented prompts improve LLM precision and F1 scores.

Abstract

Coreference resolution in biomedical texts presents unique challenges due to complex domain-specific terminology, high ambiguity in mention forms, and long-distance dependencies between coreferring expressions. In this work, we present a comprehensive evaluation of generative large language models (LLMs) for coreference resolution in the biomedical domain. Using the CRAFT corpus as our benchmark, we assess the LLMs' performance with four prompting experiments that vary in their use of local, contextual enrichment, and domain-specific cues such as abbreviations and entity dictionaries. We benchmark these approaches against a discriminative span-based encoder, SpanBERT, to compare the efficacy of generative versus discriminative methods. Our results demonstrate that while LLMs exhibit strong surface-level coreference capabilities, especially when supplemented with domain-grounding prompts, their performance remains sensitive to long-range context and mentions ambiguity. Notably, the LLaMA 8B and 17B models show superior precision and F1 scores under entity-augmented prompting, highlighting the potential of lightweight prompt engineering for enhancing LLM utility in biomedical NLP tasks.