Matching High-Dimensional Geometric Quantiles for Test-Time Adaptation of Transformers and Convolutional Networks Alike

TL;DR

This paper introduces an architecture-agnostic test-time adaptation method that uses geometric quantile matching to correct distribution shifts in test data, applicable to both CNNs and transformers.

Contribution

It proposes a novel, architecture-independent adapter trained with a quantile loss to improve test-time adaptation for various neural network architectures.

Findings

Effective on CIFAR10-C, CIFAR100-C, TinyImageNet-C datasets.

Works with both convolutional and transformer models.

Theoretically proven to learn optimal adapters under certain conditions.

Abstract

Test-time adaptation (TTA) refers to adapting a classifier for the test data when the probability distribution of the test data slightly differs from that of the training data of the model. To the best of our knowledge, most of the existing TTA approaches modify the weights of the classifier relying heavily on the architecture. It is unclear as to how these approaches are extendable to generic architectures. In this article, we propose an architecture-agnostic approach to TTA by adding an adapter network pre-processing the input images suitable to the classifier. This adapter is trained using the proposed quantile loss. Unlike existing approaches, we correct for the distribution shift by matching high-dimensional geometric quantiles. We prove theoretically that under suitable conditions minimizing quantile loss can learn the optimal adapter. We validate our approach on CIFAR10-C, CIFAR100-C and TinyImageNet-C by training both classic convolutional and transformer networks on CIFAR10, CIFAR100 and TinyImageNet datasets.