# Reusability and Transferability of Macro Actions for Reinforcement Learning

## TL;DR

This paper investigates macro actions in reinforcement learning, demonstrating their reusability across different algorithms and transferability to similar environments with different rewards, thereby enhancing learning efficiency.

## Contribution

It introduces the properties of reusability and transferability of macro actions and analyzes their benefits in reinforcement learning contexts.

## Key findings

- Macro actions can be reused across different RL algorithms.
- Macro actions transfer effectively between similar environments with different reward structures.
- Analysis confirms the beneficial properties of macro actions for RL.

## Abstract

Conventional reinforcement learning (RL) typically determines an appropriate primitive action at each timestep. However, by using a proper macro action, defined as a sequence of primitive actions, an agent is able to bypass intermediate states to a farther state and facilitate its learning procedure. The problem we would like to investigate is what associated beneficial properties that macro actions may possess. In this paper, we unveil the properties of reusability and transferability of macro actions. The first property, reusability, means that a macro action generated along with one RL method can be reused by another RL method for training, while the second one, transferability, means that a macro action can be utilized for training agents in similar environments with different reward settings. In our experiments, we first generate macro actions along with RL methods. We then provide a set of analyses to reveal the properties of reusability and transferability of the generated macro actions.