Learning by Ignoring, with Application to Domain Adaptation

TL;DR

This paper introduces a novel machine learning framework called learning by ignoring (LBI), which automatically excludes data points with large domain shifts during pretraining to improve domain adaptation performance.

Contribution

The paper proposes a new three-level optimization framework for learning by ignoring, enabling automatic identification and exclusion of less relevant data in domain adaptation tasks.

Findings

LBI improves domain adaptation performance across various datasets.

The gradient-based algorithm efficiently solves the three-level optimization problem.

Ignoring irrelevant data enhances model focus and generalization.

Abstract

Learning by ignoring, which identifies less important things and excludes them from the learning process, is broadly practiced in human learning and has shown ubiquitous effectiveness. There has been psychological studies showing that learning to ignore certain things is a powerful tool for helping people focus. In this paper, we explore whether this useful human learning methodology can be borrowed to improve machine learning. We propose a novel machine learning framework referred to as learning by ignoring (LBI). Our framework automatically identifies pretraining data examples that have large domain shift from the target distribution by learning an ignoring variable for each example and excludes them from the pretraining process. We formulate LBI as a three-level optimization framework where three learning stages are involved: pretraining by minimizing the losses weighed by ignoring variables; finetuning; updating the ignoring variables by minimizing the validation loss. A gradient-based algorithm is developed to efficiently solve the three-level optimization problem in LBI. Experiments on various datasets demonstrate the effectiveness of our framework.