Deep Hierarchical Reinforcement Learning Based Recommendations via Multi-goals Abstraction

TL;DR

This paper introduces a deep hierarchical reinforcement learning framework with multi-goals abstraction for recommendation systems, improving conversion metrics by effectively capturing long-term sparse signals through a novel two-level agent structure.

Contribution

The paper proposes a new deep hierarchical reinforcement learning algorithm, HRL-MG, with multi-goals abstraction, shared encoders, and a benefit assignment function to enhance recommendation performance.

Findings

Improved conversion rates on real-world e-commerce data.

Faster convergence due to shared state encoders.

Effective handling of sparse long-term feedback signals.

Abstract

The recommender system is an important form of intelligent application, which assists users to alleviate from information redundancy. Among the metrics used to evaluate a recommender system, the metric of conversion has become more and more important. The majority of existing recommender systems perform poorly on the metric of conversion due to its extremely sparse feedback signal. To tackle this challenge, we propose a deep hierarchical reinforcement learning based recommendation framework, which consists of two components, i.e., high-level agent and low-level agent. The high-level agent catches long-term sparse conversion signals, and automatically sets abstract goals for low-level agent, while the low-level agent follows the abstract goals and interacts with real-time environment. To solve the inherent problem in hierarchical reinforcement learning, we propose a novel deep hierarchical reinforcement learning algorithm via multi-goals abstraction (HRL-MG). Our proposed algorithm contains three characteristics: 1) the high-level agent generates multiple goals to guide the low-level agent in different stages, which reduces the difficulty of approaching high-level goals; 2) different goals share the same state encoder parameters, which increases the update frequency of the high-level agent and thus accelerates the convergence of our proposed algorithm; 3) an appreciate benefit assignment function is designed to allocate rewards in each goal so as to coordinate different goals in a consistent direction. We evaluate our proposed algorithm based on a real-world e-commerce dataset and validate its effectiveness.