Combining Model-Free Q-Ensembles and Model-Based Approaches for Informed Exploration

TL;DR

This paper proposes integrating model-free Q-ensembles with model-based trajectory memory to enhance exploration in reinforcement learning, demonstrating improved performance over using Q-ensembles alone.

Contribution

It introduces a novel combination of Q-ensembles and model-based trajectory memory for better exploration in RL tasks.

Findings

Combined approach outperforms standalone Q-ensembles.

Model-based trajectory memory enhances exploration efficiency.

Results show significant performance gains in experiments.

Abstract

Q-Ensembles are a model-free approach where input images are fed into different Q-networks and exploration is driven by the assumption that uncertainty is proportional to the variance of the output Q-values obtained. They have been shown to perform relatively well compared to other exploration strategies. Further, model-based approaches, such as encoder-decoder models have been used successfully for next frame prediction given previous frames. This paper proposes to integrate the model-free Q-ensembles and model-based approaches with the hope of compounding the benefits of both and achieving superior exploration as a result. Results show that a model-based trajectory memory approach when combined with Q-ensembles produces superior performance when compared to only using Q-ensembles.