SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning

TL;DR

SNeRL introduces a semantic-aware neural radiance field approach that enhances reinforcement learning by learning 3D-aware, semantic, and object-centric representations from multi-view images, outperforming previous methods.

Contribution

The paper proposes a novel SNeRL framework that jointly optimizes semantic-aware NeRFs with a convolutional encoder for improved 3D understanding in reinforcement learning.

Findings

SNeRL outperforms previous pixel-based and 3D-aware representations.

It improves reinforcement learning performance in both model-free and model-based settings.

The method effectively learns semantic and object-centric 3D representations.

Abstract

As previous representations for reinforcement learning cannot effectively incorporate a human-intuitive understanding of the 3D environment, they usually suffer from sub-optimal performances. In this paper, we present Semantic-aware Neural Radiance Fields for Reinforcement Learning (SNeRL), which jointly optimizes semantic-aware neural radiance fields (NeRF) with a convolutional encoder to learn 3D-aware neural implicit representation from multi-view images. We introduce 3D semantic and distilled feature fields in parallel to the RGB radiance fields in NeRF to learn semantic and object-centric representation for reinforcement learning. SNeRL outperforms not only previous pixel-based representations but also recent 3D-aware representations both in model-free and model-based reinforcement learning.