A Comparative Study of Decoding Strategies in Medical Text Generation

TL;DR

This study compares decoding strategies in medical text generation, revealing deterministic methods like beam search outperform stochastic ones, with larger models not necessarily being more robust or better performing across tasks.

Contribution

It provides a comprehensive evaluation of 11 decoding strategies across five medical tasks, highlighting the impact of decoding choices on output quality and model robustness.

Findings

Deterministic decoding strategies outperform stochastic ones.

Larger models are not more robust to decoding choices.

Decoding strategy effects can surpass model size in influence.

Abstract

Large Language Models (LLMs) rely on various decoding strategies to generate text, and these choices can significantly affect output quality. In healthcare, where accuracy is critical, the impact of decoding strategies remains underexplored. We investigate this effect in five open-ended medical tasks, including translation, summarization, question answering, dialogue, and image captioning, evaluating 11 decoding strategies with medically specialized and general-purpose LLMs of different sizes. Our results show that deterministic strategies generally outperform stochastic ones: beam search achieves the highest scores, while {\eta} and top-k sampling perform worst. Slower decoding methods tend to yield better quality. Larger models achieve higher scores overall but have longer inference times and are no more robust to decoding. Surprisingly, while medical LLMs outperform general ones in two of the five tasks, statistical analysis shows no overall performance advantage and reveals greater sensitivity to decoding choice. We further compare multiple evaluation metrics and find that correlations vary by task, with MAUVE showing weak agreement with BERTScore and ROUGE, as well as greater sensitivity to the decoding strategy. These results highlight the need for careful selection of decoding methods in medical applications, as their influence can sometimes exceed that of model choice.