Evaluating Computational Accuracy of Large Language Models in Numerical Reasoning Tasks for Healthcare Applications

TL;DR

This paper evaluates the numerical reasoning accuracy of GPT-3 based large language models in healthcare, highlighting their strengths and challenges in clinical numerical tasks, and proposes methods to improve reliability.

Contribution

It introduces a comprehensive evaluation of LLMs in healthcare numerical reasoning and demonstrates the effectiveness of prompt engineering and fact-checking pipelines.

Findings

Overall accuracy of 84.10% in numerical tasks

Fact-checking pipeline improved accuracy by 11%

Better performance in simple tasks than multi-step reasoning

Abstract

Large Language Models (LLMs) have emerged as transformative tools in the healthcare sector, demonstrating remarkable capabilities in natural language understanding and generation. However, their proficiency in numerical reasoning, particularly in high-stakes domains like in clinical applications, remains underexplored. Numerical reasoning is critical in healthcare applications, influencing patient outcomes, treatment planning, and resource allocation. This study investigates the computational accuracy of LLMs in numerical reasoning tasks within healthcare contexts. Using a curated dataset of 1,000 numerical problems, encompassing real-world scenarios such as dosage calculations and lab result interpretations, the performance of a refined LLM based on the GPT-3 architecture was evaluated. The methodology includes prompt engineering, integration of fact-checking pipelines, and application of regularization techniques to enhance model accuracy and generalization. Key metrics such as precision, recall, and F1-score were utilized to assess the model's efficacy. The results indicate an overall accuracy of 84.10%, with improved performance in straightforward numerical tasks and challenges in multi-step reasoning. The integration of a fact-checking pipeline improved accuracy by 11%, underscoring the importance of validation mechanisms. This research highlights the potential of LLMs in healthcare numerical reasoning and identifies avenues for further refinement to support critical decision-making in clinical environments. The findings aim to contribute to the development of reliable, interpretable, and contextually relevant AI tools for healthcare.