Visual Instruction-Finetuned Language Model for Versatile Brain MR Image Tasks

TL;DR

This paper introduces LLaBIT, a versatile brain MRI task model that integrates visual reasoning with language understanding, outperforming specialized models across multiple clinical tasks.

Contribution

The paper presents LLaBIT, a novel multi-task brain MRI model that effectively combines visual and language reasoning, addressing diverse clinical tasks within a single framework.

Findings

LLaBIT outperforms task-specific models in all evaluated brain MRI tasks.

Incorporating feature map reuse minimizes spatial information loss.

Text data augmentation enhances model performance with limited image-text pairs.

Abstract

LLMs have demonstrated remarkable capabilities in linguistic reasoning and are increasingly adept at vision-language tasks. The integration of image tokens into transformers has enabled direct visual input and output, advancing research from image-to-text descriptions to text-to-image generation. However, simple text-to-image generation holds limited clinical utility. In medical imaging, tasks such as image segmentation for localizing pathologies or image translation for reconstructing missing sequences have much greater clinical importance. Despite this, integrating these diverse, clinically relevant tasks within a single, versatile language model remains unexplored. Our method, LLaBIT (Large Language Model for Brain Image Translation), extends the visual reasoning of LLMs to these clinically meaningful tasks in the brain MRI domain. To mitigate the spatial information loss inherent in image tokenization, we incorporate a mechanism to reuse feature maps from the image encoder, minimizing data degradation. We also generate text data using LLMs with strict predefined instructions to augment limited image-text paired data in brain MRI. We comprehensively evaluated our method on five brain MRI datasets across four distinct tasks: report generation, visual question answering, image segmentation, and image translation. Our model not only demonstrated superior performance across all tasks but also outperformed specialized, task-specific models in direct comparisons, highlighting its efficacy and versatility