Using Non-Stationary Bandits for Learning in Repeated Cournot Games with Non-Stationary Demand

TL;DR

This paper introduces a novel adaptive epsilon-greedy algorithm for non-stationary multi-armed bandit problems in repeated Cournot games, enabling agents to adapt to changing market demands and identify new optimal actions effectively.

Contribution

The paper proposes the AWE epsilon-greedy algorithm that detects demand changes and adjusts exploration and learning rates, improving decision-making in non-stationary Cournot game environments.

Findings

Agents swiftly adapt to demand changes.

The approach facilitates emergence of collusive behavior.

Scalability is demonstrated with multiple agents and large action spaces.

Abstract

Many past attempts at modeling repeated Cournot games assume that demand is stationary. This does not align with real-world scenarios in which market demands can evolve over a product's lifetime for a myriad of reasons. In this paper, we model repeated Cournot games with non-stationary demand such that firms/agents face separate instances of non-stationary multi-armed bandit problem. The set of arms/actions that an agent can choose from represents discrete production quantities; here, the action space is ordered. Agents are independent and autonomous, and cannot observe anything from the environment; they can only see their own rewards after taking an action, and only work towards maximizing these rewards. We propose a novel algorithm 'Adaptive with Weighted Exploration (AWE) $\epsilon$-greedy' which is remotely based on the well-known $\epsilon$-greedy approach. This algorithm detects and quantifies changes in rewards due to varying market demand and varies learning rate and exploration rate in proportion to the degree of changes in demand, thus enabling agents to better identify new optimal actions. For efficient exploration, it also deploys a mechanism for weighing actions that takes advantage of the ordered action space. We use simulations to study the emergence of various equilibria in the market. In addition, we study the scalability of our approach in terms number of total agents in the system and the size of action space. We consider both symmetric and asymmetric firms in our models. We found that using our proposed method, agents are able to swiftly change their course of action according to the changes in demand, and they also engage in collusive behavior in many simulations.