LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray

TL;DR

LoFi introduces a location-aware, fine-grained representation learning approach for chest X-ray analysis, improving retrieval and grounding by leveraging region-level supervision and dense captioning.

Contribution

The paper presents a novel joint optimization framework with location-aware captioning losses and integrates a fine-grained encoder for enhanced chest X-ray grounding.

Findings

Achieves superior retrieval performance on MIMIC-CXR and PadChest-GR.

Effectively incorporates region-level supervision through grounding and dense captioning.

Enhances fine-grained representation learning in chest X-ray analysis.

Abstract

Fine-grained representation learning is crucial for retrieval and phrase grounding in chest X-rays, where clinically relevant findings are often spatially confined. However, the lack of region-level supervision in contrastive models and the limited ability of large vision language models to capture fine-grained representations in external validation lead to suboptimal performance on these tasks. To address these limitations, we propose Location-aware Fine-grained representation learning (LoFi), which jointly optimizes sigmoid, captioning, and location-aware captioning losses using a lightweight large language model. The location-aware captioning loss enables region-level supervision through grounding and dense captioning objectives, thereby facilitating fine-grained representation learning. Building upon these representations, we integrate a fine-grained encoder into retrieval-based in-context learning to enhance chest X-ray grounding across diverse settings. Extensive experiments demonstrate that our method achieves superior retrieval and phrase grounding performance on MIMIC-CXR and PadChest-GR.