Avi: Action from Volumetric Inference

TL;DR

Avi introduces a 3D perception-based architecture for robotic action generation that uses 3D point clouds and language understanding, avoiding traditional policy learning and enhancing robustness to occlusions and viewpoint changes.

Contribution

Avi is the first to leverage 3D point clouds and language-grounded reasoning for robotic actions, bypassing end-to-end policy training on action tokens.

Findings

Robustness to occlusions and viewpoint variations.

Effective use of 3D multi-modal large language models.

Potential for scalable and generalizable robotic systems.

Abstract

We propose Avi, a novel 3D Vision-Language-Action (VLA) architecture that reframes robotic action generation as a problem of 3D perception and spatial reasoning, rather than low-level policy learning. While existing VLA models primarily operate on 2D visual inputs and are trained end-to-end on task-specific action policies, Avi leverages 3D point clouds and language-grounded scene understanding to compute actions through classical geometric transformations. Most notably, Avi does not train on previous action tokens, rather, we build upon a 3D Multi-modal Large Language Model (MLLM) to generate the next point cloud and explicitly calculate the actions through classical transformations. This approach enables generalizable behaviors that are robust to occlusions, camera pose variations, and changes in viewpoint. By treating the robotic decision-making process as a structured reasoning task over 3D representations, Avi bridges the gap between high-level language instructions and low-level actuation without requiring opaque policy learning. Our preliminary results highlight the potential of 3D vision-language reasoning as a foundation for scalable, robust robotic systems. Check it out at https://avi-3drobot.github.io/.