# Benchmarking Large Language Models Using a Best Evidence Topic Report in a Patient With Early Non-Small Cell Lung Cancer

## TL;DR

This paper compares how well large language models (LLMs) and a best evidence topic (BET) approach answer a clinical question about lung cancer surgery outcomes.

## Contribution

The study introduces an LLM-BET protocol to benchmark LLMs against human-curated evidence for clinical questions.

## Key findings

- LLMs provided outputs instantly but suffered from hallucinations and copyright issues.
- BETs, though time-consuming, were more reliable and nuanced.
- No major differences were found between RATS and VATS lobectomy outcomes except for shorter hospital stay with RATS.

## Abstract

Large language models (LLMs) are generative-AI which generate text output like a human conversation. We wanted to assess the ability of LLMs to answer patient’s questions and benchmark their output using a best evidence topic (BET).

We asked LLMs whether robot-assisted thoracic surgery (RATS) or video-assisted thoracoscopic surgery (VATS) lobectomy had better perioperative outcomes for postoperative pain, length of hospital stay (LOS) and mortality. A BET was constructed according to a structured protocol for the same questions. An initial search yielded 324 papers, 12 represented the best evidence.

LLM outputs are almost instantaneous while a BET took many hours of searching a database for relevant evidence. However, current iterations and models of LLMs did not provide relevant outputs, suffered from hallucinations, and could be restricted by copyright and paywall issues. The BET, on the other hand, was tailored to the scenario by specialist human oversight and therefore more reliable and nuanced.

There were no major differences between RATS and VATS lobectomy for T1cN0M0 NSCLC apart from shorter LOS following RATS. Current LLMs may not be entirely reliable for answering clinical questions. An LLM-BET protocol could be used as a standardized process to compare LLM outputs for different clinical scenarios, each benchmarked with a BET. It can also be used to analyse outputs of different models of current and future LLMs.

Large language models (LLMs) are a subgenre of generative artificial intelligence (Gen AI), which produce textual output with a contextual and conversational format.