On the Generalizability of Iterative Patch Selection for Memory-Efficient High-Resolution Image Classification

TL;DR

This paper investigates the generalizability of iterative patch selection in memory-efficient high-resolution image classification, revealing how dataset size, patch size, and noise affect performance, and proposing improvements for low-data scenarios.

Contribution

It introduces a novel testbed extending megapixel MNIST to study patch selection effects, and analyzes how dataset size, patch size, and noise influence generalization in IPS-based models.

Findings

Lower O2I ratios hinder classifier generalization, especially with small datasets.

Tuning patch size smaller than ROI improves generalization in low-data settings.

Noise similarity to image content causes IPS to fail, affecting robustness.

Abstract

Classifying large images with small or tiny regions of interest (ROI) is challenging due to computational and memory constraints. Weakly supervised memory-efficient patch selectors have achieved results comparable with strongly supervised methods. However, low signal-to-noise ratios and low entropy attention still cause overfitting. We explore these issues using a novel testbed on a memory-efficient cross-attention transformer with Iterative Patch Selection (IPS) as the patch selection module. Our testbed extends the megapixel MNIST benchmark to four smaller O2I (object-to-image) ratios ranging from 0.01% to 0.14% while keeping the canvas size fixed and introducing a noise generation component based on B\'ezier curves. Experimental results generalize the observations made on CNNs to IPS whereby the O2I threshold below which the classifier fails to generalize is affected by the training dataset size. We further observe that the magnitude of this interaction differs for each task of the Megapixel MNIST. For tasks "Maj" and "Top", the rate is at its highest, followed by tasks "Max" and "Multi" where in the latter, this rate is almost at 0. Moreover, results show that in a low data setting, tuning the patch size to be smaller relative to the ROI improves generalization, resulting in an improvement of + 15% for the megapixel MNIST and + 5% for the Swedish traffic signs dataset compared to the original object-to-patch ratios in IPS. Further outcomes indicate that the similarity between the thickness of the noise component and the digits in the megapixel MNIST gradually causes IPS to fail to generalize, contributing to previous suspicions.