UW-BioNLP at ChemoTimelines 2025: Thinking, Fine-Tuning, and Dictionary-Enhanced LLM Systems for Chemotherapy Timeline Extraction

TL;DR

This paper explores various strategies, including chain-of-thought, fine-tuning, and dictionary lookups, to improve the extraction of chemotherapy timelines from clinical notes using large language models, achieving competitive results.

Contribution

It introduces a two-step workflow combining LLM-based extraction with normalization, and compares different training and prompting strategies for timeline construction.

Findings

Fine-tuned Qwen3-14B achieved the best score of 0.678.

Multiple approaches showed competitive performance on the leaderboard.

Insights provided for future timeline extraction tasks.

Abstract

The ChemoTimelines shared task benchmarks methods for constructing timelines of systemic anticancer treatment from electronic health records of cancer patients. This paper describes our methods, results, and findings for subtask 2 -- generating patient chemotherapy timelines from raw clinical notes. We evaluated strategies involving chain-of-thought thinking, supervised fine-tuning, direct preference optimization, and dictionary-based lookup to improve timeline extraction. All of our approaches followed a two-step workflow, wherein an LLM first extracted chemotherapy events from individual clinical notes, and then an algorithm normalized and aggregated events into patient-level timelines. Each specific method differed in how the associated LLM was utilized and trained. Multiple approaches yielded competitive performances on the test set leaderboard, with fine-tuned Qwen3-14B achieving the best official score of 0.678. Our results and analyses could provide useful insights for future attempts on this task as well as the design of similar tasks.