Algorithmic Collusion in Dynamic Pricing with Deep Reinforcement Learning

TL;DR

This paper investigates the potential for collusive pricing behavior emerging from deep reinforcement learning algorithms in competitive markets, revealing that algorithm choice and market conditions influence collusion severity.

Contribution

It extends prior research by analyzing off- and on-policy deep reinforcement learning algorithms, providing nuanced insights into algorithmic collusion in dynamic pricing.

Findings

TQL shows higher collusion and price dispersion than DRL algorithms.

Collusion severity varies with market environment and algorithm type.

Proximal Policy Optimization is less prone to collusive outcomes.

Abstract

Nowadays, a significant share of the Business-to-Consumer sector is based on online platforms like Amazon and Alibaba and uses Artificial Intelligence for pricing strategies. This has sparked debate on whether pricing algorithms may tacitly collude to set supra-competitive prices without being explicitly designed to do so. Our study addresses these concerns by examining the risk of collusion when Reinforcement Learning algorithms are used to decide on pricing strategies in competitive markets. Prior research in this field focused on Tabular Q-learning (TQL) and led to opposing views on whether learning-based algorithms can lead to supra-competitive prices. Our work contributes to this ongoing discussion by providing a more nuanced numerical study that goes beyond TQL by additionally capturing off- and on-policy Deep Reinforcement Learning (DRL) algorithms. We study multiple Bertrand oligopoly variants and show that algorithmic collusion depends on the algorithm used. In our experiments, TQL exhibits higher collusion and price dispersion phenomena compared to DRL algorithms. We show that the severity of collusion depends not only on the algorithm used but also on the characteristics of the market environment. We further find that Proximal Policy Optimization appears to be less sensitive to collusive outcomes compared to other state-of-the-art DRL algorithms.