Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations

TL;DR

DPFRL introduces a neural network framework with a discriminative particle filter for improved decision making under complex partial visual observations, outperforming existing models in Atari and real-world navigation tasks.

Contribution

It presents a novel discriminative particle filter integrated into reinforcement learning for complex partial observations, with end-to-end training and new belief features.

Findings

Outperforms state-of-the-art POMDP RL models in Flickering Atari Games.

Achieves superior results in Natural Flickering Atari Games, a new challenging benchmark.

Demonstrates effective visual navigation in real-world Habitat environment.

Abstract

Deep reinforcement learning is successful in decision making for sophisticated games, such as Atari, Go, etc. However, real-world decision making often requires reasoning with partial information extracted from complex visual observations. This paper presents Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for complex partial observations. DPFRL encodes a differentiable particle filter in the neural network policy for explicit reasoning with partial observations over time. The particle filter maintains a belief using learned discriminative update, which is trained end-to-end for decision making. We show that using the discriminative update instead of standard generative models results in significantly improved performance, especially for tasks with complex visual observations, because they circumvent the difficulty of modeling complex observations that are irrelevant to decision making. In addition, to extract features from the particle belief, we propose a new type of belief feature based on the moment generating function. DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark introduced in this paper. Further, DPFRL performs well for visual navigation with real-world data in the Habitat environment.