LISA-3D: Lifting Language-Image Segmentation to 3D via Multi-View Consistency

TL;DR

LISA-3D introduces a novel framework that enhances language-guided 3D reconstruction by leveraging multi-view consistency and existing models, achieving significant accuracy improvements without extensive retraining.

Contribution

The paper presents LISA-3D, a two-stage method that adapts language-image segmentation models for 3D tasks using geometry-aware LoRA layers and a differentiable reprojection loss, enabling zero-shot 3D content creation.

Findings

Up to +15.6 points improvement in language-to-3D accuracy.

Efficient adaptation using only 11.6M parameters.

Supports zero-shot deployment on unseen categories.

Abstract

Text-driven 3D reconstruction demands a mask generator that simultaneously understands open-vocabulary instructions and remains consistent across viewpoints. We present LISA-3D, a two-stage framework that lifts language-image segmentation into 3D by retrofitting the instruction-following model LISA with geometry-aware Low-Rank Adaptation (LoRA) layers and reusing a frozen SAM-3D reconstructor. During training we exploit off-the-shelf RGB-D sequences and their camera poses to build a differentiable reprojection loss that enforces cross-view agreement without requiring any additional 3D-text supervision. The resulting masks are concatenated with RGB images to form RGBA prompts for SAM-3D, which outputs Gaussian splats or textured meshes without retraining. Across ScanRefer and Nr3D, LISA-3D improves language-to-3D accuracy by up to +15.6 points over single-view baselines while adapting only 11.6M parameters. The system is modular, data-efficient, and supports zero-shot deployment on unseen categories, providing a practical recipe for language-guided 3D content creation. Our code will be available at https://github.com/binisalegend/LISA-3D.