Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient Ultra-Fine-Grained Image Recognition

TL;DR

This paper introduces a parameter-efficient down-sampling inter-layer adapter for ultra-fine-grained image recognition, significantly reducing computational costs while improving accuracy in resource-limited settings.

Contribution

It proposes a novel dual-branch down-sampling adapter that enables effective fine-tuning with minimal parameters, outperforming existing methods in accuracy and efficiency.

Findings

Achieves at least 6.8% higher accuracy than comparable methods.

Uses 123x fewer trainable parameters than state-of-the-art.

Reduces FLOPs by 30% on average.

Abstract

Ultra-fine-grained image recognition (UFGIR) categorizes objects with extremely small differences between classes, such as distinguishing between cultivars within the same species, as opposed to species-level classification in fine-grained image recognition (FGIR). The difficulty of this task is exacerbated due to the scarcity of samples per category. To tackle these challenges we introduce a novel approach employing down-sampling inter-layer adapters in a parameter-efficient setting, where the backbone parameters are frozen and we only fine-tune a small set of additional modules. By integrating dual-branch down-sampling, we significantly reduce the number of parameters and floating-point operations (FLOPs) required, making our method highly efficient. Comprehensive experiments on ten datasets demonstrate that our approach obtains outstanding accuracy-cost performance, highlighting its potential for practical applications in resource-constrained environments. In particular, our method increases the average accuracy by at least 6.8\% compared to other methods in the parameter-efficient setting while requiring at least 123x less trainable parameters compared to current state-of-the-art UFGIR methods and reducing the FLOPs by 30\% in average compared to other methods.