Online Continual Adaptation with Active Self-Training

TL;DR

This paper introduces a new online learning framework called Online Active Continual Adaptation, which enables models to adapt to changing environments using limited labels through a novel self-training method and active querying strategy.

Contribution

The paper proposes OSAMD, a new online self-training algorithm with active label querying, and provides theoretical regret bounds demonstrating its effectiveness in changing environments.

Findings

OSAMD achieves $O(T^{2/3})$ regret in the separable case.

OSAMD's regret bound is $O(T^{2/3} + ext{separability term})$ in the general case.

Empirical results show OSAMD performs well on simulated and real-world data.

Abstract

Models trained with offline data often suffer from continual distribution shifts and expensive labeling in changing environments. This calls for a new online learning paradigm where the learner can continually adapt to changing environments with limited labels. In this paper, we propose a new online setting -- Online Active Continual Adaptation, where the learner aims to continually adapt to changing distributions using both unlabeled samples and active queries of limited labels. To this end, we propose Online Self-Adaptive Mirror Descent (OSAMD), which adopts an online teacher-student structure to enable online self-training from unlabeled data, and a margin-based criterion that decides whether to query the labels to track changing distributions. Theoretically, we show that, in the separable case, OSAMD has an $O({T}^{2/3})$ dynamic regret bound under mild assumptions, which is aligned with the $\Omega(T^{2/3})$ lower bound of online learning algorithms with full labels. In the general case, we show a regret bound of $O({T}^{2/3} + \alpha^* T)$, where $\alpha^*$ denotes the separability of domains and is usually small. Our theoretical results show that OSAMD can fast adapt to changing environments with active queries. Empirically, we demonstrate that OSAMD achieves favorable regrets under changing environments with limited labels on both simulated and real-world data, which corroborates our theoretical findings.