Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering

TL;DR

Visual-RRT is a novel motion planning algorithm that integrates differentiable rendering with RRTs to enable goal planning directly from visual inputs, improving flexibility in vision-based robot tasks.

Contribution

It introduces a unified framework combining gradient-based exploitation with sampling-based exploration for visual-goal planning, along with novel exploration strategies.

Findings

Effective in simulated environments across multiple robot manipulators.

Successfully applied to real-world robot manipulation tasks.

Bridges the gap between sampling-based planning and vision-centric applications.

Abstract

Rapidly-exploring random trees (RRTs) have been widely adopted for robot motion planning due to their robustness and theoretical guarantees. However, existing RRT-based planners require explicit goal configurations specified as numerical joint angles, while many practical applications provide goal specifications through visual observations such as images or demonstration videos where precise goal configurations are unavailable. In this paper, we propose visual-RRT (vRRT), a motion planner that enables visual-goal planning by unifying gradient-based exploitation from differentiable robot rendering with sampling-based exploration from RRTs. We further introduce (i) a frontier-based exploration-exploitation strategy that adaptively prioritizes visually promising search regions, and (ii) inertial gradient tree expansion that inherits optimization states across tree branches for momentum-consistent gradient exploitation. Extensive experiments across various robot manipulators including Franka, UR5e, and Fetch demonstrate that vRRT achieves effective visual-goal planning in both simulated and real-world settings, bridging the gap between sampling-based planning and vision-centric robot applications. Our code is available at https://sgvr.kaist.ac.kr/Visual-RRT.