Universal Psychometrics Tasks: difficulty, composition and decomposition
Jose Hernandez-Orallo

TL;DR
This paper reexamines the concept of task difficulty in AI and proposes a general framework for understanding task composition, decomposition, and complexity based on computational steps required for agents to solve tasks.
Contribution
It introduces a new formalism for tasks as asynchronous-time stochastic processes, extending beyond Markov decision processes to better evaluate diverse AI and human tasks.
Findings
Defines task difficulty as the logarithm of computational steps needed
Introduces a formalism for asynchronous-time stochastic tasks
Clarifies concepts of task composition and decomposition
Abstract
This note revisits the concepts of task and difficulty. The notion of cognitive task and its use for the evaluation of intelligent systems is still replete with issues. The view of tasks as MDP in the context of reinforcement learning has been especially useful for the formalisation of learning tasks. However, this alternate interaction does not accommodate well for some other tasks that are usual in artificial intelligence and, most especially, in animal and human evaluation. In particular, we want to have a more general account of episodes, rewards and responses, and, most especially, the computational complexity of the algorithm behind an agent solving a task. This is crucial for the determination of the difficulty of a task as the (logarithm of the) number of computational steps required to acquire an acceptable policy for the task, which includes the exploration of policies and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComputability, Logic, AI Algorithms · Evolutionary Algorithms and Applications · Machine Learning and Algorithms
