Strategic bidding in freight transport using deep reinforcement learning

TL;DR

This paper develops a multi-agent reinforcement learning model for strategic bidding in freight markets, demonstrating that decentralized agents can approximate Nash equilibria and potentially enable autonomous, self-organizing logistics systems.

Contribution

It introduces a novel multi-agent deep reinforcement learning framework for freight bidding, analyzing its ability to reach market equilibrium without central control.

Findings

High adherence to Nash equilibria in deterministic settings (~95%)

Good performance in stochastic environments (~85%)

Risk-seeking strategies can increase reward share without excessive aggression

Abstract

This paper presents a multi-agent reinforcement learning algorithm to represent strategic bidding behavior in freight transport markets. Using this algorithm, we investigate whether feasible market equilibriums arise without any central control or communication between agents. Studying behavior in such environments may serve as a stepping stone towards self-organizing logistics systems like the Physical Internet. We model an agent-based environment in which a shipper and a carrier actively learn bidding strategies using policy gradient methods, posing bid- and ask prices at the individual container level. Both agents aim to learn the best response given the expected behavior of the opposing agent. A neutral broker allocates jobs based on bid-ask spreads. Our game-theoretical analysis and numerical experiments focus on behavioral insights. To evaluate system performance, we measure adherence to Nash equilibria, fairness of reward division and utilization of transport capacity. We observe good performance both in predictable, deterministic settings (~95% adherence to Nash equilibria) and highly stochastic environments (~85% adherence). Risk-seeking behavior may increase an agent's reward share, as long as the strategies are not overly aggressive. The results suggest a potential for full automation and decentralization of freight transport markets.