Evaluating Prompting Strategies with MedGemma for Medical Order Extraction

TL;DR

This study evaluates different prompting strategies for extracting medical orders from doctor-patient conversations using MedGemma, finding that simpler one-shot prompting outperforms more complex reasoning frameworks in certain clinical data scenarios.

Contribution

It systematically compares prompting paradigms for clinical order extraction with MedGemma, highlighting the effectiveness of simple prompts over complex reasoning methods.

Findings

One-shot prompting achieved the best validation performance.

Complex reasoning prompts can introduce noise and reduce accuracy.

Simple prompts are more robust in manually annotated clinical transcripts.

Abstract

The accurate extraction of medical orders from doctor-patient conversations is a critical task for reducing clinical documentation burdens and ensuring patient safety. This paper details our team submission to the MEDIQA-OE-2025 Shared Task. We investigate the performance of MedGemma, a new domain-specific open-source language model, for structured order extraction. We systematically evaluate three distinct prompting paradigms: a straightforward one-Shot approach, a reasoning-focused ReAct framework, and a multi-step agentic workflow. Our experiments reveal that while more complex frameworks like ReAct and agentic flows are powerful, the simpler one-shot prompting method achieved the highest performance on the official validation set. We posit that on manually annotated transcripts, complex reasoning chains can lead to "overthinking" and introduce noise, making a direct approach more robust and efficient. Our work provides valuable insights into selecting appropriate prompting strategies for clinical information extraction in varied data conditions.