Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branching

TL;DR

This paper introduces Savage Conversation Forests (SCF), a reinforcement learning framework with a branched architecture that improves fine-tuning of large language models for multi-turn medical dialogues, enhancing diagnostic accuracy.

Contribution

The paper proposes a novel branched conversation architecture, SCF, for fine-tuning LLMs in multi-turn dialogues, addressing limitations of previous methods in complex conversational tasks.

Findings

SCF outperforms linear architectures in diagnostic accuracy.

Branched architecture provides richer training signals.

Improves understanding of conversation dynamics in medical dialogues.

Abstract

Fine-tuning methods such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) have demonstrated success in training large language models (LLMs) for single-turn tasks. However, these methods fall short in multi-turn applications, such as diagnostic patient interviewing, where understanding how early conversational turns influence downstream completions and outcomes is essential. In medicine, a multi-turn perspective is critical for learning diagnostic schemas and better understanding conversation dynamics. To address this gap, I introduce Savage Conversation Forests (SCF), a reinforcement learning framework that leverages a branched conversation architecture to fine-tune LLMs for multi-turn dialogue. SCF generates multiple possible conversation continuations at each turn, enabling the model to learn how different early responses affect downstream interactions and diagnostic outcomes. In experiments simulating doctor-patient conversations, SCF with branching outperforms linear conversation architectures on diagnostic accuracy. I hypothesize that SCF's improvements stem from its ability to provide richer, interdependent training signals across conversation turns. These results suggest that a branched training architecture is an important strategy for fine tuning LLMs in complex multi-turn conversational tasks.