Syn-STARTS: Synthesized START Triage Scenario Generation Framework for Scalable LLM Evaluation

TL;DR

Syn-STARTS is a framework that uses large language models to generate realistic triage scenarios for mass casualty incidents, enabling scalable evaluation of AI decision-making in critical medical situations.

Contribution

It introduces a novel synthetic data generation method for triage scenarios using LLMs, addressing data scarcity in MCI research and evaluation.

Findings

Generated cases are qualitatively similar to real datasets.

High stability in LLM accuracy across different triage categories.

Synthetic data can support high-performance AI model development.

Abstract

Triage is a critically important decision-making process in mass casualty incidents (MCIs) to maximize victim survival rates. While the role of AI in such situations is gaining attention for making optimal decisions within limited resources and time, its development and performance evaluation require benchmark datasets of sufficient quantity and quality. However, MCIs occur infrequently, and sufficient records are difficult to accumulate at the scene, making it challenging to collect large-scale realworld data for research use. Therefore, we developed Syn-STARTS, a framework that uses LLMs to generate triage cases, and verified its effectiveness. The results showed that the triage cases generated by Syn-STARTS were qualitatively indistinguishable from the TRIAGE open dataset generated by manual curation from training materials. Furthermore, when evaluating the LLM accuracy using hundreds of cases each from the green, yellow, red, and black categories defined by the standard triage method START, the results were found to be highly stable. This strongly indicates the possibility of synthetic data in developing high-performance AI models for severe and critical medical situations.