# Where is my forearm? Clustering of body parts from simultaneous tactile and linguistic input using sequential mapping

## TL;DR

This paper introduces a sequential mapping algorithm for associating tactile body part stimuli with linguistic labels, demonstrating improved accuracy over traditional methods in a robotic context, and analyzing effects of data size and noise.

## Contribution

It proposes a novel sequential mapping approach for cross-modal learning of body part representations, extending existing one-step models with a publicly available implementation.

## Key findings

- Sequential mapping outperforms one-step mapping in accuracy.
- Larger datasets improve mapping performance.
- Noise in linguistic input reduces accuracy.

## Abstract

Humans and animals are constantly exposed to a continuous stream of sensory information from different modalities. At the same time, they form more compressed representations like concepts or symbols. In species that use language, this process is further structured by this interaction, where a mapping between the sensorimotor concepts and linguistic elements needs to be established. There is evidence that children might be learning language by simply disambiguating potential meanings based on multiple exposures to utterances in different contexts (cross-situational learning). In existing models, the mapping between modalities is usually found in a single step by directly using frequencies of referent and meaning co-occurrences. In this paper, we present an extension of this one-step mapping and introduce a newly proposed sequential mapping algorithm together with a publicly available Matlab implementation. For demonstration, we have chosen a less typical scenario: instead of learning to associate objects with their names, we focus on body representations. A humanoid robot is receiving tactile stimulations on its body, while at the same time listening to utterances of the body part names (e.g., hand, forearm and torso). With the goal at arriving at the correct "body categories", we demonstrate how a sequential mapping algorithm outperforms one-step mapping. In addition, the effect of data set size and noise in the linguistic input are studied.