A-I-RAVEN and I-RAVEN-Mesh: Two New Benchmarks for Abstract Visual Reasoning

TL;DR

This paper introduces two new benchmarks, A-I-RAVEN and I-RAVEN-Mesh, to evaluate the generalization and knowledge transfer capabilities of deep neural networks in abstract visual reasoning using Raven's Progressive Matrices.

Contribution

The paper presents A-I-RAVEN with 10 generalization regimes and I-RAVEN-Mesh with line-based patterns, providing new tools for systematic evaluation of AVR models.

Findings

Models show limited generalization to unseen attributes.

Models struggle with compositionality and complex configurations.

Evaluation reveals specific shortcomings in transfer learning.

Abstract

We study generalization and knowledge reuse capabilities of deep neural networks in the domain of abstract visual reasoning (AVR), employing Raven's Progressive Matrices (RPMs), a recognized benchmark task for assessing AVR abilities. Two knowledge transfer scenarios referring to the I-RAVEN dataset are investigated. Firstly, inspired by generalization assessment capabilities of the PGM dataset and popularity of I-RAVEN, we introduce Attributeless-I-RAVEN (A-I-RAVEN), a benchmark with 10 generalization regimes that allow to systematically test generalization of abstract rules applied to held-out attributes at various levels of complexity (primary and extended regimes). In contrast to PGM, A-I-RAVEN features compositionality, a variety of figure configurations, and does not require substantial computational resources. Secondly, we construct I-RAVEN-Mesh, a dataset that enriches RPMs with a novel component structure comprising line-based patterns, facilitating assessment of progressive knowledge acquisition in transfer learning setting. We evaluate 13 strong models from the AVR literature on the introduced datasets, revealing their specific shortcomings in generalization and knowledge transfer.