Black-box Unsupervised Domain Adaptation with Bi-directional Atkinson-Shiffrin Memory

TL;DR

This paper introduces BiMem, a bi-directional memory mechanism for black-box unsupervised domain adaptation that improves robustness and generalization across multiple visual recognition tasks by correcting noisy pseudo labels.

Contribution

The paper presents BiMem, a novel bi-directional memorization framework with three memory types, enhancing black-box UDA robustness and accuracy across diverse visual tasks.

Findings

BiMem outperforms existing methods in image classification, semantic segmentation, and object detection.

BiMem effectively corrects noisy pseudo labels during training.

Extensive experiments demonstrate consistent superior performance of BiMem.

Abstract

Black-box unsupervised domain adaptation (UDA) learns with source predictions of target data without accessing either source data or source models during training, and it has clear superiority in data privacy and flexibility in target network selection. However, the source predictions of target data are often noisy and training with them is prone to learning collapses. We propose BiMem, a bi-directional memorization mechanism that learns to remember useful and representative information to correct noisy pseudo labels on the fly, leading to robust black-box UDA that can generalize across different visual recognition tasks. BiMem constructs three types of memory, including sensory memory, short-term memory, and long-term memory, which interact in a bi-directional manner for comprehensive and robust memorization of learnt features. It includes a forward memorization flow that identifies and stores useful features and a backward calibration flow that rectifies features' pseudo labels progressively. Extensive experiments show that BiMem achieves superior domain adaptation performance consistently across various visual recognition tasks such as image classification, semantic segmentation and object detection.