UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
Yueming Xu, Jiahui Zhang, Ze Huang, Yurui Chen, Yanpeng Zhou, Zhenyu Chen, Yu-Jie Yuan, Pengxiang Xia, Guowei Huang, Xinyue Cai, Zhongang Qi, Xingyue Quan, Jianye Hao, Hang Xu, Li Zhang
TL;DR
UniUGG is a pioneering unified framework that combines 3D understanding and generation using a geometric-semantic encoding and a latent diffusion model, enabling high-quality 3D scene creation and spatial reasoning.
Contribution
It introduces the first unified 3D understanding and generation framework employing an LLM and a novel geometric-semantic pretraining strategy.
Findings
Outperforms existing methods in 3D representation quality
Enhances spatial understanding and visual reasoning
Enables high-quality 3D scene generation from images
Abstract
Despite the impressive progress on understanding and generating images shown by the recent unified architectures, the integration of 3D tasks remains challenging and largely unexplored. In this paper, we introduce UniUGG, the first unified understanding and generation framework for 3D modalities. Our unified framework employs an LLM to comprehend and decode sentences and 3D representations. At its core, we propose a spatial decoder leveraging a latent diffusion model to generate high-quality 3D representations. This allows for the generation and imagination of 3D scenes based on a reference image and an arbitrary view transformation, while remaining supports for spatial visual question answering (VQA) tasks. Additionally, we propose a geometric-semantic learning strategy to pretrain the vision encoder. This design jointly captures the input's semantic and geometric cues, enhancing both…
Peer Reviews
Decision·ICLR 2026 Poster
- **S.1:** The qualitative results look great.
- **W.1:** I've worked with LLMs, I've worked on diffusion papers, and I know about the Mast3r paper. Yet, I have no clue what's going on in this work and why. I think the writing is a bit too dense to follow unless the reader is already intimately familiar with all the related works. And I also don't think the figures are doing a great job at illustrating the method. I can't say anything more about this work. Maybe an overarching, vastly simplified diagram would be helpful.
1. The encoder is trained using geometric supervision which enables spatial reasoning compared to models trained on 2D images. 2. A single latent representation is used for both 3D reasoning and 3D generation. 3. The method allows geometric control through text or parametric view prompts.
1. There’s no discussion or visualization of the 3D output from the model. 2. All the evaluations are done in 2D space and there is no 3D evaluation performed. It’s unclear if the model is actually learning the structure of the 3D world or simply rendering based on the distribution of the training data. 3. The model is not trained end-to-end. The encoder and spatial VAE are trained separately and kept frozen during unified training (lines 301-302).
1. Unifying 3D understanding and generation is an interesting and novel research direction. 2. The proposed geometry–semantic encoder is well-designed; incorporating the MASt3R framework is an effective and interesting choice. 3. The paper is well-written, and the experiments are thorough, covering a diverse set of novel view synthesis and vision–language spatial reasoning benchmarks.
1. The paper would benefit from a more detailed discussion of limitations and failure cases. 2. The method’s robustness to more extreme view transformations is not clearly evaluated.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsImage Processing and 3D Reconstruction · 3D Shape Modeling and Analysis · Computer Graphics and Visualization Techniques