Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models

TL;DR

This paper investigates the viewpoint stability of vision foundation models, revealing their sensitivity to minor viewpoint changes and highlighting the importance of robust features for 3D reasoning tasks.

Contribution

It introduces a methodology to recognize and classify viewpoint instabilities in foundation models using feature representations alone, without needing actual images.

Findings

Foundation models encode accidental viewpoints consistently.

Models show variability in interpreting out-of-distribution viewpoints.

Viewpoint instability impacts classification, VQA, and 3D reconstruction tasks.

Abstract

In this paper, we analyze the viewpoint stability of foundational models - specifically, their sensitivity to changes in viewpoint- and define instability as significant feature variations resulting from minor changes in viewing angle, leading to generalization gaps in 3D reasoning tasks. We investigate nine foundational models, focusing on their responses to viewpoint changes, including the often-overlooked accidental viewpoints where specific camera orientations obscure an object's true 3D structure. Our methodology enables recognizing and classifying out-of-distribution (OOD), accidental, and stable viewpoints using feature representations alone, without accessing the actual images. Our findings indicate that while foundation models consistently encode accidental viewpoints, they vary in their interpretation of OOD viewpoints due to inherent biases, at times leading to object misclassifications based on geometric resemblance. Through quantitative and qualitative evaluations on three downstream tasks - classification, VQA, and 3D reconstruction - we illustrate the impact of viewpoint instability and underscore the importance of feature robustness across diverse viewing conditions.