Discounted Online Convex Optimization: Uniform Regret Across a Continuous Interval

TL;DR

This paper introduces a new online convex optimization algorithm that adaptively achieves uniform discounted regret across a continuous range of discount factors without prior knowledge of the discount parameter.

Contribution

The paper proposes a novel analysis showing that smoothed OGD combined with Discounted-Normal-Predictor achieves uniform regret bounds over all discount factors, adapting to unknown environments.

Findings

Achieves $O( oot{rac{ ext{log} T}{1- ext{lambda}}})$ discounted regret uniformly for all discount factors.

Uses multiple OGD instances with aggregation via Discounted-Normal-Predictor.

Demonstrates effective combination of decisions from experts with different discount factors.

Abstract

Reflecting the greater significance of recent history over the distant past in non-stationary environments, $\lambda$-discounted regret has been introduced in online convex optimization (OCO) to gracefully forget past data as new information arrives. When the discount factor $\lambda$ is given, online gradient descent with an appropriate step size achieves an $O(1/\sqrt{1-\lambda})$ discounted regret. However, the value of $\lambda$ is often not predetermined in real-world scenarios. This gives rise to a significant open question: is it possible to develop a discounted algorithm that adapts to an unknown discount factor. In this paper, we affirmatively answer this question by providing a novel analysis to demonstrate that smoothed OGD (SOGD) achieves a uniform $O(\sqrt{\log T/1-\lambda})$ discounted regret, holding for all values of $\lambda$ across a continuous interval simultaneously. The basic idea is to maintain multiple OGD instances to handle different discount factors, and aggregate their outputs sequentially by an online prediction algorithm named as Discounted-Normal-Predictor (DNP) (Kapralov and Panigrahy,2010). Our analysis reveals that DNP can combine the decisions of two experts, even when they operate on discounted regret with different discount factors.