Object Centric Concept Bottlenecks

TL;DR

This paper introduces Object-Centric Concept Bottlenecks (OCB), a framework that combines concept-based models with object-centric foundation models to improve interpretability and performance in complex vision tasks.

Contribution

The paper proposes OCB, integrating CBMs with object-centric foundation models, enhancing interpretability and performance in complex visual understanding.

Findings

OCB outperforms traditional CBMs on complex datasets.

Object aggregation strategies significantly impact model performance.

OCB enables interpretable decision-making in complex visual tasks.

Abstract

Developing high-performing, yet interpretable models remains a critical challenge in modern AI. Concept-based models (CBMs) attempt to address this by extracting human-understandable concepts from a global encoding (e.g., image encoding) and then applying a linear classifier on the resulting concept activations, enabling transparent decision-making. However, their reliance on holistic image encodings limits their expressiveness in object-centric real-world settings and thus hinders their ability to solve complex vision tasks beyond single-label classification. To tackle these challenges, we introduce Object-Centric Concept Bottlenecks (OCB), a framework that combines the strengths of CBMs and pre-trained object-centric foundation models, boosting performance and interpretability. We evaluate OCB on complex image datasets and conduct a comprehensive ablation study to analyze key components of the framework, such as strategies for aggregating object-concept encodings. The results show that OCB outperforms traditional CBMs and allows one to make interpretable decisions for complex visual tasks.