IDANI: Inference-time Domain Adaptation via Neuron-level Interventions

TL;DR

This paper introduces IDANI, a novel inference-time domain adaptation method that modifies neuron activations to transform test examples into source-like representations, improving model performance on unseen domains.

Contribution

The paper presents a new neuron-level intervention technique for domain adaptation that operates during inference, unlike traditional methods that require retraining.

Findings

Improves accuracy on unseen domains

Operates efficiently during inference

Outperforms some existing domain adaptation methods

Abstract

Large pre-trained models are usually fine-tuned on downstream task data, and tested on unseen data. When the train and test data come from different domains, the model is likely to struggle, as it is not adapted to the test domain. We propose a new approach for domain adaptation (DA), using neuron-level interventions: We modify the representation of each test example in specific neurons, resulting in a counterfactual example from the source domain, which the model is more familiar with. The modified example is then fed back into the model. While most other DA methods are applied during training time, ours is applied during inference only, making it more efficient and applicable. Our experiments show that our method improves performance on unseen domains.