# Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning

## TL;DR

This paper investigates the impact of prior and likelihood assumptions in Bayesian deep Q-learning, revealing a cold posterior effect and proposing improved priors for enhanced algorithm performance.

## Contribution

It challenges common Gaussian likelihood assumptions, demonstrates the cold posterior effect, and offers practical solutions for better priors in Bayesian deep Q-learning.

## Key findings

- Gaussian likelihood often violated in practice
- Cold posterior effect observed in Bayesian deep Q-learning
- Proposed improved priors enhance performance

## Abstract

Uncertainty quantification in reinforcement learning can greatly improve exploration and robustness. Approximate Bayesian approaches have recently been popularized to quantify uncertainty in model-free algorithms. However, so far the focus has been on improving the accuracy of the posterior approximation, instead of studying the accuracy of the prior and likelihood assumptions underlying the posterior. In this work, we demonstrate that there is a cold posterior effect in Bayesian deep Q-learning, where contrary to theory, performance increases when reducing the temperature of the posterior. To identify and overcome likely causes, we challenge common assumptions made on the likelihood and priors in Bayesian model-free algorithms. We empirically study prior distributions and show through statistical tests that the common Gaussian likelihood assumption is frequently violated. We argue that developing more suitable likelihoods and priors should be a key focus in future Bayesian reinforcement learning research and we offer simple, implementable solutions for better priors in deep Q-learning that lead to more performant Bayesian algorithms.