Concept Induction using LLMs: a user experiment for assessment

TL;DR

This paper investigates the use of GPT-4 to generate high-level, human-understandable concepts for image classification explanations, comparing its output with human and heuristic methods through a human study.

Contribution

It demonstrates the potential of LLMs like GPT-4 to produce interpretable concepts for XAI, addressing limitations of existing automatic concept discovery methods.

Findings

GPT-4 concepts are more comprehensible than ECII.

Human explanations are still more effective than LLM-generated ones.

The study introduces a human evaluation approach for concept understandability.

Abstract

Explainable Artificial Intelligence (XAI) poses a significant challenge in providing transparent and understandable insights into complex AI models. Traditional post-hoc algorithms, while useful, often struggle to deliver interpretable explanations. Concept-based models offer a promising avenue by incorporating explicit representations of concepts to enhance interpretability. However, existing research on automatic concept discovery methods is often limited by lower-level concepts, costly human annotation requirements, and a restricted domain of background knowledge. In this study, we explore the potential of a Large Language Model (LLM), specifically GPT-4, by leveraging its domain knowledge and common-sense capability to generate high-level concepts that are meaningful as explanations for humans, for a specific setting of image classification. We use minimal textual object information available in the data via prompting to facilitate this process. To evaluate the output, we compare the concepts generated by the LLM with two other methods: concepts generated by humans and the ECII heuristic concept induction system. Since there is no established metric to determine the human understandability of concepts, we conducted a human study to assess the effectiveness of the LLM-generated concepts. Our findings indicate that while human-generated explanations remain superior, concepts derived from GPT-4 are more comprehensible to humans compared to those generated by ECII.