Intelligence Analysis of Language Models

TL;DR

This study evaluates large language models' ability to perform abstract reasoning tasks from the ARC dataset, revealing their limitations in non-linguistic domains even with advanced prompting techniques.

Contribution

First to assess open-source language models on the ARC dataset using prompt-based and Chain-of-Thought methods, highlighting their current performance limitations.

Findings

LLMs struggle with non-linguistic reasoning tasks

Chain-of-Thought improves performance but does not fully solve the challenge

Open-source models show significant room for improvement in abstract reasoning

Abstract

In this project, we test the effectiveness of Large Language Models (LLMs) on the Abstraction and Reasoning Corpus (ARC) dataset. This dataset serves as a representative benchmark for testing abstract reasoning abilities, requiring a fundamental understanding of key concepts such as object identification, basic counting, and elementary geometric principles. Tasks from this dataset are converted into a prompt-based format for evaluation. Initially, we assess the models' potential through a Zero-shot approach. Subsequently, we investigate the application of the Chain-of-Thought (CoT) technique, aiming to determine its role in improving model performance. Our results suggest that, despite the high expectations placed on contemporary LLMs, these models still struggle in non-linguistic domains, even when dealing with simpler subsets of the ARC dataset. Our study is the first to concentrate on the capabilities of open-source models in this context. The code, dataset, and prompts supporting this project's findings can be found in our GitHub repository, accessible at: https://github.com/Lianga2000/LLMsOnARC.