HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways

TL;DR

HealthBranches is a comprehensive, clinically-grounded dataset for evaluating complex reasoning in medical question-answering by providing detailed decision pathways, diverse question formats, and a large collection of validated medical cases.

Contribution

It introduces a semi-automated pipeline to generate a large, structured medical Q&A dataset with reasoning paths, supporting advanced evaluation of LLMs in healthcare contexts.

Findings

Enables robust assessment of multi-step medical reasoning in LLMs

Supports both open-ended and multiple-choice question formats

Includes clinically validated reasoning chains for 4,063 cases

Abstract

HealthBranches is a novel benchmark dataset for medical Question-Answering (Q&A), specifically designed to evaluate complex reasoning in Large Language Models (LLMs). This dataset is generated through a semi-automated pipeline that transforms explicit decision pathways from medical source into realistic patient cases with associated questions and answers. Covering 4,063 case studies across 17 healthcare topics, each data point is based on clinically validated reasoning chains. HealthBranches supports both open-ended and multiple-choice question formats and uniquely includes the full reasoning path for each Q&A. Its structured design enables robust evaluation of LLMs' multi-step inference capabilities, including their performance in structured Retrieval-Augmented Generation (RAG) contexts. HealthBranches establishes a foundation for the development of more trustworthy, interpretable, and clinically reliable LLMs in high-stakes domains while also serving as a valuable resource for educational purposes.