Understanding in Artificial Intelligence

TL;DR

This paper reviews how current AI benchmarks and methods measure and develop understanding capabilities, analyzing progress and limitations in assessing AI's true understanding in tasks like visual question answering.

Contribution

It provides a comprehensive analysis of existing benchmarks and research streams related to AI understanding, highlighting progress and gaps in measurement and development.

Findings

Benchmarks have evolved to better measure understanding capabilities.

Current AI methods show progress but still have limitations in true understanding.

Analysis identifies gaps between benchmark performance and genuine understanding.

Abstract

Current Artificial Intelligence (AI) methods, most based on deep learning, have facilitated progress in several fields, including computer vision and natural language understanding. The progress of these AI methods is measured using benchmarks designed to solve challenging tasks, such as visual question answering. A question remains of how much understanding is leveraged by these methods and how appropriate are the current benchmarks to measure understanding capabilities. To answer these questions, we have analysed existing benchmarks and their understanding capabilities, defined by a set of understanding capabilities, and current research streams. We show how progress has been made in benchmark development to measure understanding capabilities of AI methods and we review as well how current methods develop understanding capabilities.