LR0.FM: Low-Res Benchmark and Improving Robustness for Zero-Shot Classification in Foundation Models

TL;DR

This paper introduces LR0.FM, a benchmark and metric for evaluating and improving the robustness of visual-language foundation models against low-resolution images, revealing key factors affecting performance and proposing a simple enhancement strategy.

Contribution

The paper presents LR0.FM, a new benchmark and metric for low-resolution robustness, along with a novel method LR-TK0 to improve model robustness without retraining.

Findings

Model size correlates with robustness to resolution loss.

Pre-training dataset quality impacts robustness more than size.

Fine-tuned and high-resolution models are less robust to LR images.

Abstract

Visual-language foundation Models (FMs) exhibit remarkable zero-shot generalization across diverse tasks, largely attributed to extensive pre-training on largescale datasets. However, their robustness on low-resolution/pixelated (LR) images, a common challenge in real-world scenarios, remains underexplored. We introduce LR0.FM, a comprehensive benchmark evaluating the impact of low resolution on the zero-shot classification performance of 10 FM(s) across 66 backbones and 15 datasets. We propose a novel metric, Weighted Aggregated Robustness, to address the limitations of existing metrics and better evaluate model performance across resolutions and datasets. Our key findings show that: (i) model size positively correlates with robustness to resolution degradation, (ii) pre-training dataset quality is more important than its size, and (iii) fine-tuned and higher resolution models are less robust against LR. Our analysis further reveals that the model makes semantically reasonable predictions at LR, and the lack of fine-grained details in input adversely impacts the model's initial layers more than the deeper layers. We use these insights and introduce a simple strategy, LR-TK0, to enhance the robustness of models without compromising their pre-trained weights. We demonstrate the effectiveness of LR-TK0 for robustness against low-resolution across several datasets and its generalization capability across backbones and other approaches. Code is available at https://github.com/shyammarjit/LR0.FM