Automatic Domain Adaptation by Transformers in In-Context Learning

TL;DR

This paper introduces a Transformer-based approach for automatic domain adaptation in in-context learning, capable of approximating and selecting suitable adaptation algorithms without parameter updates, leading to improved adaptability.

Contribution

It demonstrates that Transformers can theoretically approximate various domain adaptation algorithms and automatically choose the best method for a dataset in in-context learning.

Findings

Transformers can approximate unsupervised domain adaptation algorithms.

The proposed method outperforms existing domain adaptation techniques.

Transformers automatically select the most suitable adaptation algorithm.

Abstract

Selecting or designing an appropriate domain adaptation algorithm for a given problem remains challenging. This paper presents a Transformer model that can provably approximate and opt for domain adaptation methods for a given dataset in the in-context learning framework, where a foundation model performs new tasks without updating its parameters at test time. Specifically, we prove that Transformers can approximate instance-based and feature-based unsupervised domain adaptation algorithms and automatically select an algorithm suited for a given dataset. Numerical results indicate that in-context learning demonstrates an adaptive domain adaptation surpassing existing methods.