CLIP-SR: Collaborative Linguistic and Image Processing for Super-Resolution

TL;DR

CLIP-SR introduces a multi-modal framework combining textual and visual features to improve high-quality, large-scale super-resolution with semantic consistency, outperforming traditional CNN methods especially at 8× and 16× upscaling.

Contribution

The paper presents a novel multi-modal super-resolution method that integrates CLIP-based textual semantics with visual features for enhanced detail and semantic coherence at large upscaling factors.

Findings

Effective super-resolution at 16× scaling with high semantic fidelity.

Outperforms existing CNN and text-guided SR methods in quality and consistency.

Enables controllable SR with semantic preservation.

Abstract

Convolutional Neural Networks (CNNs) have significantly advanced Image Super-Resolution (SR), yet most CNN-based methods rely solely on pixel-based transformations, often leading to artifacts and blurring, particularly under severe downsampling rates (\eg, 8$\times$ or 16$\times$). The recently developed text-guided SR approaches leverage textual descriptions to enhance their detail restoration capabilities but frequently struggle with effectively performing alignment, resulting in semantic inconsistencies. To address these challenges, we propose a multi-modal semantic enhancement framework that integrates textual semantics with visual features, effectively mitigating semantic mismatches and detail losses in highly degraded low-resolution (LR) images. Our method enables realistic, high-quality SR to be performed at large upscaling factors, with a maximum scaling ratio of 16$\times$. The framework integrates both text and image inputs using the prompt predictor, the Text-Image Fusion Block (TIFBlock), and the Iterative Refinement Module, leveraging Contrastive Language-Image Pretraining (CLIP) features to guide a progressive enhancement process with fine-grained alignment. This synergy produces high-resolution outputs with sharp textures and strong semantic coherence, even at substantial scaling factors. Extensive comparative experiments and ablation studies validate the effectiveness of our approach. Furthermore, by leveraging textual semantics, our method offers a degree of super-resolution editability, allowing for controlled enhancements while preserving semantic consistency. The code is available at https://github.com/hengliusky/CLIP-SR.