When Do Differences Matter? On-Line Feature Extraction Through Cognitive Economy

TL;DR

This paper introduces a principled algorithm for online feature extraction in autonomous agents, based on cognitive economy, which adaptively partitions state spaces and improves learning efficiency.

Contribution

It presents a novel algorithm combining cognitive economy principles with active Q-learning for adaptive state-space abstraction.

Findings

Algorithm learns effective representations in continuous state-spaces.

Outperforms other well-known methods in puck-on-a-hill task.

Uses learning curves to evaluate and compare representations.

Abstract

For an intelligent agent to be truly autonomous, it must be able to adapt its representation to the requirements of its task as it interacts with the world. Most current approaches to on-line feature extraction are ad hoc; in contrast, this paper presents an algorithm that bases judgments of state compatibility and state-space abstraction on principled criteria derived from the psychological principle of cognitive economy. The algorithm incorporates an active form of Q-learning, and partitions continuous state-spaces by merging and splitting Voronoi regions. The experiments illustrate a new methodology for testing and comparing representations by means of learning curves. Results from the puck-on-a-hill task demonstrate the algorithm's ability to learn effective representations, superior to those produced by some other, well-known, methods.