Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models
Mateusz Michalkiewicz, Sheena Bai, Mahsa Baktashmotlagh and, Varun Jampani, Guha Balakrishnan

TL;DR
This paper investigates the viewpoint stability of vision foundation models, revealing their sensitivity to minor viewpoint changes and highlighting the importance of robust features for 3D reasoning tasks.
Contribution
It introduces a methodology to recognize and classify viewpoint instabilities in foundation models using feature representations alone, without needing actual images.
Findings
Foundation models encode accidental viewpoints consistently.
Models show variability in interpreting out-of-distribution viewpoints.
Viewpoint instability impacts classification, VQA, and 3D reconstruction tasks.
Abstract
In this paper, we analyze the viewpoint stability of foundational models - specifically, their sensitivity to changes in viewpoint- and define instability as significant feature variations resulting from minor changes in viewing angle, leading to generalization gaps in 3D reasoning tasks. We investigate nine foundational models, focusing on their responses to viewpoint changes, including the often-overlooked accidental viewpoints where specific camera orientations obscure an object's true 3D structure. Our methodology enables recognizing and classifying out-of-distribution (OOD), accidental, and stable viewpoints using feature representations alone, without accessing the actual images. Our findings indicate that while foundation models consistently encode accidental viewpoints, they vary in their interpretation of OOD viewpoints due to inherent biases, at times leading to object…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNonprofit Sector and Volunteering · Religion and Society Interactions · Religion, Society, and Development
