Markov Decision Process modeled with Bandits for Sequential Decision Making in Linear-flow

TL;DR

This paper introduces a novel MDP framework with Bandits for sequential decision making in linear-flow marketing scenarios, leveraging Thompson sampling for efficient action allocation and demonstrating superior performance in simulations.

Contribution

The paper formulates a new MDP with Bandits approach for linear-flow sequential decision making, combining Thompson sampling with dynamic programming for improved efficiency and robustness.

Findings

Proposed method outperforms Q-learning and independent Bandits in simulations.

The approach is most robust to changes in interdependence strength across pages.

Leverages Thompson sampling's exploration-exploitation balance effectively.

Abstract

For marketing, we sometimes need to recommend content for multiple pages in sequence. Different from general sequential decision making process, the use cases have a simpler flow where customers per seeing recommended content on each page can only return feedback as moving forward in the process or dropping from it until a termination state. We refer to this type of problems as sequential decision making in linear--flow. We propose to formulate the problem as an MDP with Bandits where Bandits are employed to model the transition probability matrix. At recommendation time, we use Thompson sampling (TS) to sample the transition probabilities and allocate the best series of actions with analytical solution through exact dynamic programming. The way that we formulate the problem allows us to leverage TS's efficiency in balancing exploration and exploitation and Bandit's convenience in modeling actions' incompatibility. In the simulation study, we observe the proposed MDP with Bandits algorithm outperforms Q-learning with $\epsilon$-greedy and decreasing $\epsilon$, independent Bandits, and interaction Bandits. We also find the proposed algorithm's performance is the most robust to changes in the across-page interdependence strength.