Similarity metrics for Different Market Scenarios in Abides

TL;DR

This paper investigates how to measure similarity between financial Markov Decision Processes (MDPs) and uses this to transfer policies across different market scenarios, aiming to improve learning efficiency in financial reinforcement learning.

Contribution

It introduces and analyzes three similarity metrics for financial MDPs and applies Probabilistic Policy Reuse to enhance policy transfer based on these similarities.

Findings

Three similarity metrics effectively compare financial MDPs.

Policy reuse based on similarity improves learning efficiency.

The approach balances exploration and exploitation in financial MDPs.

Abstract

Markov Decision Processes (MDPs) are an effective way to formally describe many Machine Learning problems. In fact, recently MDPs have also emerged as a powerful framework to model financial trading tasks. For example, financial MDPs can model different market scenarios. However, the learning of a (near-)optimal policy for each of these financial MDPs can be a very time-consuming process, especially when nothing is known about the policy to begin with. An alternative approach is to find a similar financial MDP for which we have already learned its policy, and then reuse such policy in the learning of a new policy for a new financial MDP. Such a knowledge transfer between market scenarios raises several issues. On the one hand, how to measure the similarity between financial MDPs. On the other hand, how to use this similarity measurement to effectively transfer the knowledge between financial MDPs. This paper addresses both of these issues. Regarding the first one, this paper analyzes the use of three similarity metrics based on conceptual, structural and performance aspects of the financial MDPs. Regarding the second one, this paper uses Probabilistic Policy Reuse to balance the exploitation/exploration in the learning of a new financial MDP according to the similarity of the previous financial MDPs whose knowledge is reused.