Language guided machine action

TL;DR

This paper introduces LGMA, a hierarchical modular network inspired by human cortical functions, enabling language-guided actions, intention decomposition, and mental simulation for complex tasks.

Contribution

The paper presents a novel hierarchical modular network architecture that mimics human cortical processing for versatile, language-guided machine actions and cognitive functions.

Findings

LGMA can perform language-guided actions and intention decomposition.

The system integrates multimodal sensory information effectively.

LGMA demonstrates capabilities for mental simulation before action execution.

Abstract

Here we build a hierarchical modular network called Language guided machine action (LGMA), whose modules process information stream mimicking human cortical network that allows to achieve multiple general tasks such as language guided action, intention decomposition and mental simulation before action execution etc. LGMA contains 3 main systems: (1) primary sensory system that multimodal sensory information of vision, language and sensorimotor. (2) association system involves and Broca modules to comprehend and synthesize language, BA14/40 module to translate between sensorimotor and language, midTemporal module to convert between language and vision, and superior parietal lobe to integrate attended visual object and arm state into cognitive map for future spatial actions. Pre-supplementary motor area (pre-SMA) can converts high level intention into sequential atomic actions, while SMA can integrate these atomic actions, current arm and attended object state into sensorimotor vector to apply corresponding torques on arm via pre-motor and primary motor of arm to achieve the intention. The high-level executive system contains PFC that does explicit inference and guide voluntary action based on language, while BG is the habitual action control center.