Online Learning with Costly Features in Non-stationary Environments

TL;DR

This paper introduces a novel approach to online learning in non-stationary environments where features are costly to observe, extending contextual bandits to optimize long-term rewards while balancing observation costs.

Contribution

It proposes a new algorithm for costly feature observation in non-stationary settings, achieving sublinear regret and improving decision-making efficiency.

Findings

The algorithm outperforms existing methods in real-world scenarios.

It effectively balances observation costs and reward maximization.

Demonstrates robustness to abrupt changes in environment dynamics.

Abstract

Maximizing long-term rewards is the primary goal in sequential decision-making problems. The majority of existing methods assume that side information is freely available, enabling the learning agent to observe all features' states before making a decision. In real-world problems, however, collecting beneficial information is often costly. That implies that, besides individual arms' reward, learning the observations of the features' states is essential to improve the decision-making strategy. The problem is aggravated in a non-stationary environment where reward and cost distributions undergo abrupt changes over time. To address the aforementioned dual learning problem, we extend the contextual bandit setting and allow the agent to observe subsets of features' states. The objective is to maximize the long-term average gain, which is the difference between the accumulated rewards and the paid costs on average. Therefore, the agent faces a trade-off between minimizing the cost of information acquisition and possibly improving the decision-making process using the obtained information. To this end, we develop an algorithm that guarantees a sublinear regret in time. Numerical results demonstrate the superiority of our proposed policy in a real-world scenario.