OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI

TL;DR

OlympicArena is a comprehensive benchmark dataset designed to evaluate AI's multi-discipline cognitive reasoning abilities across text and image modalities, revealing current limitations and guiding future superintelligent AI development.

Contribution

The paper introduces OlympicArena, a large-scale, multi-modal benchmark with interdisciplinary problems to evaluate and analyze AI reasoning capabilities comprehensively.

Findings

GPT-4o achieves only 39.97% accuracy on the benchmark.

Current AI models show significant limitations in complex, interdisciplinary reasoning.

OlympicArena provides resources for advancing AI research in scientific discovery.

Abstract

The evolution of Artificial Intelligence (AI) has been significantly accelerated by advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning abilities in problem-solving and scientific discovery (i.e., AI4Science) once exclusive to human intellect. To comprehensively evaluate current models' performance in cognitive reasoning abilities, we introduce OlympicArena, which includes 11,163 bilingual problems across both text-only and interleaved text-image modalities. These challenges encompass a wide range of disciplines spanning seven fields and 62 international Olympic competitions, rigorously examined for data leakage. We argue that the challenges in Olympic competition problems are ideal for evaluating AI's cognitive reasoning due to their complexity and interdisciplinary nature, which are essential for tackling complex scientific challenges and facilitating discoveries. Beyond evaluating performance across various disciplines using answer-only criteria, we conduct detailed experiments and analyses from multiple perspectives. We delve into the models' cognitive reasoning abilities, their performance across different modalities, and their outcomes in process-level evaluations, which are vital for tasks requiring complex reasoning with lengthy solutions. Our extensive evaluations reveal that even advanced models like GPT-4o only achieve a 39.97% overall accuracy, illustrating current AI limitations in complex reasoning and multimodal integration. Through the OlympicArena, we aim to advance AI towards superintelligence, equipping it to address more complex challenges in science and beyond. We also provide a comprehensive set of resources to support AI research, including a benchmark dataset, an open-source annotation platform, a detailed evaluation tool, and a leaderboard with automatic submission features.