Doge Tickets: Uncovering Domain-general Language Models by Playing Lottery Tickets

TL;DR

This paper introduces a method called doge tickets to identify domain-general parameters in language models, improving out-of-domain generalization by leveraging the lottery ticket hypothesis.

Contribution

It is the first to propose using lottery tickets to uncover domain-general parameters in pretrained language models for better domain transfer.

Findings

Doge tickets improve out-of-domain generalization.

Existence of domain-general parameters is supported.

Method outperforms several baselines.

Abstract

Over-parameterized models, typically pretrained language models (LMs), have shown an appealing expressive power due to their small learning bias. However, the huge learning capacity of LMs can also lead to large learning variance. In a pilot study, we find that, when faced with multiple domains, a critical portion of parameters behave unexpectedly in a domain-specific manner while others behave in a domain-general one. Motivated by this phenomenon, we for the first time posit that domain-general parameters can underpin a domain-general LM that can be derived from the original LM. To uncover the domain-general LM, we propose to identify domain-general parameters by playing lottery tickets (dubbed doge tickets). In order to intervene the lottery, we propose a domain-general score, which depicts how domain-invariant a parameter is by associating it with the variance. Comprehensive experiments are conducted on the Amazon, Mnli and OntoNotes datasets. The results show that the doge tickets obtains an improved out-of-domain generalization in comparison with a range of competitive baselines. Analysis results further hint the existence of domain-general parameters and the performance consistency of doge tickets.