SplatR : Experience Goal Visual Rearrangement with 3D Gaussian Splatting and Dense Feature Matching

TL;DR

This paper introduces a novel approach for Experience Goal Visual Rearrangement using 3D Gaussian Splatting and dense feature matching, enabling more accurate scene reconstruction and comparison for embodied AI tasks.

Contribution

The work presents a new framework combining 3D Gaussian Splatting with dense feature matching for improved scene understanding in rearrangement tasks.

Findings

Achieves better performance on AI2-THOR benchmark

Enables consistent view comparison between current and goal states

Demonstrates robustness and generalization in scene reconstruction

Abstract

Experience Goal Visual Rearrangement task stands as a foundational challenge within Embodied AI, requiring an agent to construct a robust world model that accurately captures the goal state. The agent uses this world model to restore a shuffled scene to its original configuration, making an accurate representation of the world essential for successfully completing the task. In this work, we present a novel framework that leverages on 3D Gaussian Splatting as a 3D scene representation for experience goal visual rearrangement task. Recent advances in volumetric scene representation like 3D Gaussian Splatting, offer fast rendering of high quality and photo-realistic novel views. Our approach enables the agent to have consistent views of the current and the goal setting of the rearrangement task, which enables the agent to directly compare the goal state and the shuffled state of the world in image space. To compare these views, we propose to use a dense feature matching method with visual features extracted from a foundation model, leveraging its advantages of a more universal feature representation, which facilitates robustness, and generalization. We validate our approach on the AI2-THOR rearrangement challenge benchmark and demonstrate improvements over the current state of the art methods