Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lens

TL;DR

This paper investigates the interpretability and reliability of concept-based neural models, revealing how reasoning shortcuts undermine their performance and proposing theoretical conditions for proper concept and inference layer identification.

Contribution

It establishes a novel connection between reasoning shortcuts and concept-based models, deriving conditions for identifying concepts and inference layers, and empirically evaluates their practical challenges.

Findings

Reasoning shortcuts significantly affect concept-based models.

Existing mitigation strategies often fail to meet the theoretical conditions.

Theoretical insights guide better understanding of model interpretability.

Abstract

Concept-based Models are neural networks that learn a concept extractor to map inputs to high-level concepts and an inference layer to translate these into predictions. Ensuring these modules produce interpretable concepts and behave reliably in out-of-distribution is crucial, yet the conditions for achieving this remain unclear. We study this problem by establishing a novel connection between Concept-based Models and reasoning shortcuts (RSs), a common issue where models achieve high accuracy by learning low-quality concepts, even when the inference layer is fixed and provided upfront. Specifically, we extend RSs to the more complex setting of Concept-based Models and derive theoretical conditions for identifying both the concepts and the inference layer. Our empirical results highlight the impact of RSs and show that existing methods, even combined with multiple natural mitigation strategies, often fail to meet these conditions in practice.