A Sensorimotor Perspective on Grounding the Semantic of Simple Visual Features

TL;DR

This paper introduces a sensorimotor approach to grounding the semantics of simple visual features, demonstrating how an agent can autonomously understand visual invariants through active interaction without prior knowledge.

Contribution

It formalizes how sensorimotor contingencies can be used to derive semantic understanding of visual features in an unsupervised manner, aligning with the Sensorimotor Contingencies Theory.

Findings

Agent can characterize visual feature properties like uniformity and edge-ness

Unsupervised method works without prior knowledge of sensory encoding

Formalization applied to a simulated visual system

Abstract

In Machine Learning and Robotics, the semantic content of visual features is usually provided to the system by a human who interprets its content. On the contrary, strictly unsupervised approaches have difficulties relating the statistics of sensory inputs to their semantic content without also relying on prior knowledge introduced in the system. We proposed in this paper to tackle this problem from a sensorimotor perspective. In line with the Sensorimotor Contingencies Theory, we make the fundamental assumption that the semantic content of sensory inputs at least partially stems from the way an agent can actively transform it. We illustrate our approach by formalizing how simple visual features can induce invariants in a naive agent's sensorimotor experience, and evaluate it on a simple simulated visual system. Without any a priori knowledge about the way its sensorimotor information is encoded, we show how an agent can characterize the uniformity and edge-ness of the visual features it interacts with.