Towards A Measure Of General Machine Intelligence

TL;DR

This paper introduces a new benchmark, the g-index, to measure how efficiently AI systems acquire new skills across diverse real-world tasks, emphasizing generalization ability over brute-force training.

Contribution

It proposes a universal instruction language and a method to quantify task difficulty and skill acquisition efficiency, enabling comparison of AI systems' general intelligence capabilities.

Findings

The g-index effectively measures skill acquisition efficiency.

Current models show varying g-index scores indicating different levels of generalization.

The proposed benchmark can evaluate AI systems across multiple domains.

Abstract

To build general-purpose artificial intelligence systems that can deal with unknown variables across unknown domains, we need benchmarks that measure how well these systems perform on tasks they have never seen before. A prerequisite for this is a measure of a task's generalization difficulty, or how dissimilar it is from the system's prior knowledge and experience. If the skill of an intelligence system in a particular domain is defined as it's ability to consistently generate a set of instructions (or programs) to solve tasks in that domain, current benchmarks do not quantitatively measure the efficiency of acquiring new skills, making it possible to brute-force skill acquisition by training with unlimited amounts of data and compute power. With this in mind, we first propose a common language of instruction, a programming language that allows the expression of programs in the form of directed acyclic graphs across a wide variety of real-world domains and computing platforms. Using programs generated in this language, we demonstrate a match-based method to both score performance and calculate the generalization difficulty of any given set of tasks. We use these to define a numeric benchmark called the generalization index, or the g-index, to measure and compare the skill-acquisition efficiency of any intelligence system on a set of real-world tasks. Finally, we evaluate the suitability of some well-known models as general intelligence systems by calculating their g-index scores.