WheelArm-Sim: A Manipulation and Navigation Combined Multimodal Synthetic Data Generation Simulator for Unified Control in Assistive Robotics

TL;DR

This paper introduces WheelArm-Sim, a simulation framework for generating synthetic multimodal data to develop integrated control models for assistive wheelchairs with robotic arms, aiming to improve independent living support.

Contribution

It presents a novel simulation environment, WheelArm-Sim, for collecting multimodal data to enable unified control of wheelchairs and robotic arms in assistive robotics.

Findings

Successfully collected a large multimodal dataset with 67,783 samples.

Demonstrated feasibility of using synthetic data for machine learning-based control.

Baseline model showed promising results in a manipulation task.

Abstract

Wheelchairs and robotic arms enhance independent living by assisting individuals with upper-body and mobility limitations in their activities of daily living (ADLs). Although recent advancements in assistive robotics have focused on Wheelchair-Mounted Robotic Arms (WMRAs) and wheelchairs separately, integrated and unified control of the combination using machine learning models remains largely underexplored. To fill this gap, we introduce the concept of WheelArm, an integrated cyber-physical system (CPS) that combines wheelchair and robotic arm controls. Data collection is the first step toward developing WheelArm models. In this paper, we present WheelArm-Sim, a simulation framework developed in Isaac Sim for synthetic data collection. We evaluate its capability by collecting a manipulation and navigation combined multimodal dataset, comprising 13 tasks, 232 trajectories, and 67,783 samples. To demonstrate the potential of the WheelArm dataset, we implement a baseline model for action prediction in the mustard-picking task. The results illustrate that data collected from WheelArm-Sim is feasible for a data-driven machine learning model for integrated control.