Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning

TL;DR

This paper introduces Cross-modal Identity Mapping (CIM), a reinforcement learning framework that reduces information loss in image captioning by aligning visual content and text, improving caption quality without extra annotations.

Contribution

The paper proposes CIM, a novel reinforcement learning approach that minimizes information loss in image captioning through identity mapping, without requiring additional labeled data.

Findings

CIM outperforms supervised fine-tuning methods in image captioning.

On COCO-LN500, CIM achieves 20% better relation reasoning.

CIM enhances image captioning quality by focusing on image details.

Abstract

Large Vision-Language Models (LVLMs) often omit or misrepresent critical visual content in generated image captions. Minimizing such information loss will force LVLMs to focus on image details to generate precise descriptions. However, measuring information loss during modality conversion is inherently challenging due to the modal gap between visual content and text output. In this paper, we argue that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption. Based on this insight, we further propose Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations. Specifically, the method quantitatively evaluates the information loss from two perspectives: Gallery Representation Consistency and Query-gallery Image Relevance. Supervised under these metrics, LVLM minimizes information loss and aims to achieve identity mapping from images to captions. The experimental results demonstrate the superior performance of our method in image captioning, even when compared with Supervised Fine-Tuning. Particularly, on the COCO-LN500 benchmark, CIM achieves a 20% improvement in relation reasoning on Qwen2.5-VL-7B.The code will be released when the paper is accepted.