Bridging Neural and Symbolic Representations with Transitional Dictionary Learning

TL;DR

This paper presents a Transitional Dictionary Learning framework that implicitly learns symbolic knowledge and compositional patterns in visual data, outperforming existing unsupervised methods and aligning well with human evaluations.

Contribution

The introduction of a game-theoretic diffusion model and novel metrics for evaluating symbolic and compositional learning in visual datasets.

Findings

Outperforms state-of-the-art unsupervised part segmentation methods

Discoveries of meaningful compositional patterns in visual data

Metrics align well with human judgment

Abstract

This paper introduces a novel Transitional Dictionary Learning (TDL) framework that can implicitly learn symbolic knowledge, such as visual parts and relations, by reconstructing the input as a combination of parts with implicit relations. We propose a game-theoretic diffusion model to decompose the input into visual parts using the dictionaries learned by the Expectation Maximization (EM) algorithm, implemented as the online prototype clustering, based on the decomposition results. Additionally, two metrics, clustering information gain, and heuristic shape score are proposed to evaluate the model. Experiments are conducted on three abstract compositional visual object datasets, which require the model to utilize the compositionality of data instead of simply exploiting visual features. Then, three tasks on symbol grounding to predefined classes of parts and relations, as well as transfer learning to unseen classes, followed by a human evaluation, were carried out on these datasets. The results show that the proposed method discovers compositional patterns, which significantly outperforms the state-of-the-art unsupervised part segmentation methods that rely on visual features from pre-trained backbones. Furthermore, the proposed metrics are consistent with human evaluations.