HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs

TL;DR

HuatuoGPT-o1 is a specialized medical language model that leverages verifiable problems and reinforcement learning to enhance complex reasoning in medical diagnosis and problem-solving.

Contribution

The paper introduces a novel two-stage training approach using verifiable medical problems and RL to improve reasoning in a medical LLM, HuatuoGPT-o1.

Findings

HuatuoGPT-o1 outperforms baselines in medical reasoning tasks.

Verifiable problems effectively guide model training.

Reinforcement learning further enhances reasoning capabilities.

Abstract

The breakthrough of OpenAI o1 highlights the potential of enhancing reasoning to improve LLM. Yet, most research in reasoning has focused on mathematical tasks, leaving domains like medicine underexplored. The medical domain, though distinct from mathematics, also demands robust reasoning to provide reliable answers, given the high standards of healthcare. However, verifying medical reasoning is challenging, unlike those in mathematics. To address this, we propose verifiable medical problems with a medical verifier to check the correctness of model outputs. This verifiable nature enables advancements in medical reasoning through a two-stage approach: (1) using the verifier to guide the search for a complex reasoning trajectory for fine-tuning LLMs, (2) applying reinforcement learning (RL) with verifier-based rewards to enhance complex reasoning further. Finally, we introduce HuatuoGPT-o1, a medical LLM capable of complex reasoning, which outperforms general and medical-specific baselines using only 40K verifiable problems. Experiments show complex reasoning improves medical problem-solving and benefits more from RL. We hope our approach inspires advancements in reasoning across medical and other specialized domains.