Universal Psychometrics Tasks: difficulty, composition and decomposition

TL;DR

This paper reexamines the concept of task difficulty in AI and proposes a general framework for understanding task composition, decomposition, and complexity based on computational steps required for agents to solve tasks.

Contribution

It introduces a new formalism for tasks as asynchronous-time stochastic processes, extending beyond Markov decision processes to better evaluate diverse AI and human tasks.

Findings

Defines task difficulty as the logarithm of computational steps needed

Introduces a formalism for asynchronous-time stochastic tasks

Clarifies concepts of task composition and decomposition

Abstract

This note revisits the concepts of task and difficulty. The notion of cognitive task and its use for the evaluation of intelligent systems is still replete with issues. The view of tasks as MDP in the context of reinforcement learning has been especially useful for the formalisation of learning tasks. However, this alternate interaction does not accommodate well for some other tasks that are usual in artificial intelligence and, most especially, in animal and human evaluation. In particular, we want to have a more general account of episodes, rewards and responses, and, most especially, the computational complexity of the algorithm behind an agent solving a task. This is crucial for the determination of the difficulty of a task as the (logarithm of the) number of computational steps required to acquire an acceptable policy for the task, which includes the exploration of policies and their verification. We introduce a notion of asynchronous-time stochastic tasks. Based on this interpretation, we can see what task difficulty is, what instance difficulty is (relative to a task) and also what task compositions and decompositions are.