One-Time Model Adaptation to Heterogeneous Clients: An Intra-Client and Inter-Image Attention Design

TL;DR

This paper introduces ICIIA, a one-time training module that enhances image recognition models' adaptiveness to heterogeneous clients by using intra-client and inter-image attention, improving accuracy with minimal overhead.

Contribution

The paper proposes a novel ICIIA module that enables one-time cloud training for client-specific adaptation in image recognition models, incorporating multi-head self-attention and efficient linear projections.

Findings

ICIIA improves ImageNet-1K accuracy by over 8% with minimal FLOP increase.

ICIIA achieves significant accuracy gains across multiple models and datasets.

Partitioned linear projection enhances efficiency without major accuracy loss.

Abstract

The mainstream workflow of image recognition applications is first training one global model on the cloud for a wide range of classes and then serving numerous clients, each with heterogeneous images from a small subset of classes to be recognized. From the cloud-client discrepancies on the range of image classes, the recognition model is desired to have strong adaptiveness, intuitively by concentrating the focus on each individual client's local dynamic class subset, while incurring negligible overhead. In this work, we propose to plug a new intra-client and inter-image attention (ICIIA) module into existing backbone recognition models, requiring only one-time cloud-based training to be client-adaptive. In particular, given a target image from a certain client, ICIIA introduces multi-head self-attention to retrieve relevant images from the client's historical unlabeled images, thereby calibrating the focus and the recognition result. Further considering that ICIIA's overhead is dominated by linear projection, we propose partitioned linear projection with feature shuffling for replacement and allow increasing the number of partitions to dramatically improve efficiency without scarifying too much accuracy. We finally evaluate ICIIA using 3 different recognition tasks with 9 backbone models over 5 representative datasets. Extensive evaluation results demonstrate the effectiveness and efficiency of ICIIA. Specifically, for ImageNet-1K with the backbone models of MobileNetV3-L and Swin-B, ICIIA can improve the testing accuracy to 83.37% (+8.11%) and 88.86% (+5.28%), while adding only 1.62% and 0.02% of FLOPs, respectively.