Empirical Comparison of Forgetting Mechanisms for UCB-based Algorithms on a Data-Driven Simulation Platform

TL;DR

This paper compares different forgetting mechanisms for UCB algorithms in non-stationary bandit environments, introducing FDSW-UCB, which combines long-term and short-term views, and demonstrates its superior adaptability through data-driven simulations.

Contribution

It introduces FDSW-UCB, a novel dual-view algorithm that effectively combines discounting and sliding window techniques for non-stationary bandit problems.

Findings

SW-UCB is robust in non-stationary environments.

D-UCB suffers from linear regret due to learning failure.

FDSW-UCB with optimistic aggregation outperforms other methods.

Abstract

Many real-world bandit problems involve non-stationary reward distributions, where the optimal decision may shift due to evolving environments. However, the performance of some typical Multi-Armed Bandit (MAB) models such as Upper Confidence Bound (UCB) algorithms degrades significantly in non-stationary environments where reward distributions change over time. To address this limitation, this paper introduces and evaluates FDSW-UCB, a novel dual-view algorithm that integrates a discount-based long-term perspective with a sliding-window-based short-term view. A data-driven semi-synthetic simulation platform, built upon the MovieLens-1M and Open Bandit datasets, is developed to test algorithm adaptability under abrupt and gradual drift scenarios. Experimental results demonstrate that a well-configured sliding-window mechanism (SW-UCB) is robust, while the widely used discounting method (D-UCB) suffers from a fundamental learning failure, leading to linear regret. Crucially, the proposed FDSW-UCB, when employing an optimistic aggregation strategy, achieves superior performance in dynamic settings, highlighting that the ensemble strategy itself is a decisive factor for success.