CRAFT: A Neuro-Symbolic Framework for Visual Functional Affordance Grounding

TL;DR

CRAFT is a neuro-symbolic framework that combines language models, commonsense knowledge, and visual evidence to improve interpretability and accuracy in grounding object affordances for actions in scenes.

Contribution

It introduces a novel energy-based reasoning loop integrating structured priors and visual evidence for interpretable affordance grounding.

Findings

Enhanced accuracy in multi-object scenes

Improved interpretability of grounding decisions

Effective in label-free settings

Abstract

We introduce CRAFT, a neuro-symbolic framework for interpretable affordance grounding, which identifies the objects in a scene that enable a given action (e.g., "cut"). CRAFT integrates structured commonsense priors from ConceptNet and language models with visual evidence from CLIP, using an energy-based reasoning loop to refine predictions iteratively. This process yields transparent, goal-driven decisions to ground symbolic and perceptual structures. Experiments in multi-object, label-free settings demonstrate that CRAFT enhances accuracy while improving interpretability, providing a step toward robust and trustworthy scene understanding.