GeneralizeFormer: Layer-Adaptive Model Generation across Test-Time Distribution Shifts

TL;DR

GeneralizeFormer is a lightweight, transformer-based approach that dynamically generates layer parameters during inference to improve test-time domain generalization without fine-tuning, effectively handling multiple distribution shifts.

Contribution

It introduces a novel method to generate layer parameters on the fly using a meta-learned transformer, avoiding fine-tuning and enhancing adaptability to distribution shifts.

Findings

Outperforms existing methods on six domain generalization datasets.

Effectively handles multiple target distributions without fine-tuning.

Reduces computational cost by fixing convolutional parameters.

Abstract

We consider the problem of test-time domain generalization, where a model is trained on several source domains and adjusted on target domains never seen during training. Different from the common methods that fine-tune the model or adjust the classifier parameters online, we propose to generate multiple layer parameters on the fly during inference by a lightweight meta-learned transformer, which we call \textit{GeneralizeFormer}. The layer-wise parameters are generated per target batch without fine-tuning or online adjustment. By doing so, our method is more effective in dynamic scenarios with multiple target distributions and also avoids forgetting valuable source distribution characteristics. Moreover, by considering layer-wise gradients, the proposed method adapts itself to various distribution shifts. To reduce the computational and time cost, we fix the convolutional parameters while only generating parameters of the Batch Normalization layers and the linear classifier. Experiments on six widely used domain generalization datasets demonstrate the benefits and abilities of the proposed method to efficiently handle various distribution shifts, generalize in dynamic scenarios, and avoid forgetting.