Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Task

TL;DR

This paper investigates the limitations of large language models in demonstrating fluid intelligence, focusing on the ARC task, and identifies key challenges such as skill composition, abstract input formats, and decoding strategies.

Contribution

The study provides a detailed analysis of LLMs' deficiencies in fluid intelligence using the ARC task, highlighting specific challenges and limitations.

Findings

LLMs have limited skill composition abilities.

Unfamiliarity with abstract input formats hampers performance.

Intrinsic left-to-right decoding limits fluid intelligence assessment.

Abstract

While LLMs have exhibited strong performance on various NLP tasks, it is noteworthy that most of these tasks rely on utilizing the vast amount of knowledge encoded in LLMs' parameters, rather than solving new problems without prior knowledge. In cognitive research, the latter ability is referred to as fluid intelligence, which is considered to be critical for assessing human intelligence. Recent research on fluid intelligence assessments has highlighted significant deficiencies in LLMs' abilities. In this paper, we analyze the challenges LLMs face in demonstrating fluid intelligence through controlled experiments, using the most representative ARC task as an example. Our study revealed three major limitations in existing LLMs: limited ability for skill composition, unfamiliarity with abstract input formats, and the intrinsic deficiency of left-to-right decoding. Our data and code can be found in https://wujunjie1998.github.io/araoc-benchmark.github.io/.