Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation

TL;DR

This paper introduces HRGR-Agent, a hybrid model combining retrieval and generation for medical report creation, achieving state-of-the-art results with diverse, coherent, and accurate reports guided by reinforcement learning.

Contribution

The novel HRGR-Agent framework integrates retrieval and generation with hierarchical decision-making and reinforcement learning for improved medical report generation.

Findings

Achieves state-of-the-art results on two datasets.

Generates well-structured and diverse reports.

Improves detection accuracy of medical terminologies.

Abstract

Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.