HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models

TL;DR

This paper introduces HSCR, a novel hierarchical self-contrastive rewarding method that improves alignment and trustworthiness of medical vision-language models by generating high-quality preference data and capturing nuanced preferences.

Contribution

HSCR presents a cost-effective way to generate preference data and a multi-level optimization strategy for better modality alignment in Med-VLMs.

Findings

Enhanced zero-shot performance across medical tasks

Significant improvement in modality alignment and trustworthiness

Effective with only 2,000 training entries

Abstract

Medical Vision-Language Models (Med-VLMs) have achieved success across various tasks, yet most existing methods overlook the modality misalignment issue that can lead to untrustworthy responses in clinical settings. In this paper, we propose Hierarchical Self-Contrastive Rewarding (HSCR), a novel approach that addresses two critical challenges in Med-VLM alignment: 1) Cost-effective generation of high-quality preference data; 2) Capturing nuanced and context-aware preferences for improved alignment. HSCR first leverages the inherent capability of Med-VLMs to generate dispreferred responses with higher sampling probability. By analyzing output logit shifts after visual token dropout, we identify modality-coupled tokens that induce misalignment and derive an implicit alignment reward function. This function guides token replacement with hallucinated ones during decoding, producing high-quality dispreferred data. Furthermore, HSCR introduces a multi-level preference optimization strategy, which extends beyond traditional adjacent-level optimization by incorporating nuanced implicit preferences, leveraging relative quality in dispreferred data to capture subtle alignment cues for more precise and context-aware optimization. Extensive experiments across multiple medical tasks, including Med-VQA, medical image captioning and instruction following, demonstrate that HSCR not only enhances zero-shot performance but also significantly improves modality alignment and trustworthiness with just 2,000 training entries.