VVTRec: Radio Interferometric Reconstruction through Visual and Textual Modality Enrichment

TL;DR

VVTRec is a novel multimodal reconstruction method for radio interferometric data that uses visual and textual information to improve image quality without significant computational costs.

Contribution

It introduces a multimodal approach combining visibility data with visual and textual features, leveraging vision-language models for enhanced radio image reconstruction.

Findings

Improves image quality by integrating multimodal information.

Achieves better structural integrity and accuracy in reconstructed images.

Operates efficiently without high computational overhead.

Abstract

Radio astronomy is an indispensable discipline for observing distant celestial objects. Measurements of wave signals from radio telescopes, called visibility, need to be transformed into images for astronomical observations. These dirty images blend information from real sources and artifacts. Therefore, astronomers usually perform reconstruction before imaging to obtain cleaner images. Existing methods consider only a single modality of sparse visibility data, resulting in images with remaining artifacts and insufficient modeling of correlation. To enhance the extraction of visibility information and emphasize output quality in the image domain, we propose VVTRec, a multimodal radio interferometric data reconstruction method with visibility-guided visual and textual modality enrichment. In our VVTRec, sparse visibility is transformed into image-form and text-form features to obtain enhancements in terms of spatial and semantic information, improving the structural integrity and accuracy of images. Also, we leverage Vision-Language Models (VLMs) to achieve additional training-free performance improvements. VVTRec enables sparse visibility, as a foreign modality unseen by VLMs, to accurately extract pre-trained knowledge as a supplement. Our experiments demonstrate that VVTRec effectively enhances imaging results by exploiting multimodal information without introducing excessive computational overhead.