SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarios

TL;DR

This paper introduces SToLa, a self-adaptive framework that enhances tactile and language reasoning in open-ended scenarios by using Mixture of Experts and a new comprehensive tactile commonsense dataset.

Contribution

SToLa is the first framework to dynamically unify tactile and language modalities using Mixture of Experts for improved reasoning in open-ended physical scenarios.

Findings

SToLa achieves competitive results on PhysiCLeAR benchmark.

The Mixture of Experts architecture effectively manages multimodal data.

The new dataset enables more diverse and complex tactile reasoning evaluation.

Abstract

This paper explores the challenges of integrating tactile sensing into intelligent systems for multimodal reasoning, particularly in enabling commonsense reasoning about the open-ended physical world. We identify two key challenges: modality discrepancy, where existing large touch-language models often treat touch as a mere sub-modality of language, and open-ended tactile data scarcity, where current datasets lack the diversity, open-endness and complexity needed for reasoning. To overcome these challenges, we introduce SToLa, a Self-Adaptive Touch-Language framework. SToLa utilizes Mixture of Experts (MoE) to dynamically process, unify, and manage tactile and language modalities, capturing their unique characteristics. Crucially, we also present a comprehensive tactile commonsense reasoning dataset and benchmark featuring free-form questions and responses, 8 physical properties, 4 interactive characteristics, and diverse commonsense knowledge. Experiments show SToLa exhibits competitive performance compared to existing models on the PhysiCLeAR benchmark and self-constructed datasets, proving the effectiveness of the Mixture of Experts architecture in multimodal management and the performance advantages for open-scenario tactile commonsense reasoning tasks.