Fine-Tuning without Performance Degradation

TL;DR

This paper introduces a novel fine-tuning algorithm that minimizes performance degradation and accelerates learning during the transition from offline to online policy adaptation.

Contribution

The paper proposes a new fine-tuning method based on Jump Start that enables gradual exploration and reduces initial performance drops.

Findings

Significantly reduces performance degradation during fine-tuning

Achieves faster fine-tuning compared to existing algorithms

Demonstrates effectiveness across various settings

Abstract

Fine-tuning policies learned offline remains a major challenge in application domains. Monotonic performance improvement during \emph{fine-tuning} is often challenging, as agents typically experience performance degradation at the early fine-tuning stage. The community has identified multiple difficulties in fine-tuning a learned network online, however, the majority of progress has focused on improving learning efficiency during fine-tuning. In practice, this comes at a serious cost during fine-tuning: initially, agent performance degrades as the agent explores and effectively overrides the policy learned offline. We show across a range of settings, many offline-to-online algorithms exhibit either (1) performance degradation or (2) slow learning (sometimes effectively no improvement) during fine-tuning. We introduce a new fine-tuning algorithm, based on an algorithm called Jump Start, that gradually allows more exploration based on online estimates of performance. Empirically, this approach achieves fast fine-tuning and significantly reduces performance degradations compared with existing algorithms designed to do the same.