Development, Evaluation, and Deployment of a Multi-Agent System for Thoracic Tumor Board

TL;DR

This paper presents the development, evaluation, and deployment of AI methods for automated patient summary generation in thoracic tumor boards, enhancing clinical workflow efficiency.

Contribution

It introduces several automated AI chart summarization techniques, compares them to physician standards, and validates an LLM-based evaluation approach.

Findings

Automated AI summaries achieved high agreement with physician standards.

The deployed AI tool improved efficiency in tumor board discussions.

LLM-based evaluation correlated well with expert assessments.

Abstract

Tumor boards are multidisciplinary conferences dedicated to producing actionable patient care recommendations with live review of primary radiology and pathology data. Succinct patient case summaries are needed to drive efficient and accurate case discussions. We developed a manual AI-based workflow to generate patient summaries to display live at the Stanford Thoracic Tumor board. To improve on this manually intensive process, we developed several automated AI chart summarization methods and evaluated them against physician gold standard summaries and fact-based scoring rubrics. We report these comparative evaluations as well as our deployment of the final state automated AI chart summarization tool along with post-deployment monitoring. We also validate the use of an LLM as a judge evaluation strategy for fact-based scoring. This work is an example of integrating AI-based workflows into routine clinical practice.