# MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments

## TL;DR

MateRobot is a wearable vision-based system that accurately recognizes objects and materials for visually impaired users, enhancing pre-touch perception with low cognitive load.

## Contribution

The paper introduces MateViT, a lightweight model for pixel-wise semantic segmentation, enabling real-time material and object recognition on mobile platforms.

## Key findings

- Achieved 40.2% and 51.1% mIoU on COCOStuff-10K and DMS datasets.
- Field tests showed low cognitive demand with a NASA-TLX score of 28.
- Surpassed previous methods by 5.7% and 7.0% in segmentation accuracy.

## Abstract

People with Visual Impairments (PVI) typically recognize objects through haptic perception. Knowing objects and materials before touching is desired by the target users but under-explored in the field of human-centered robotics. To fill this gap, in this work, a wearable vision-based robotic system, MateRobot, is established for PVI to recognize materials and object categories beforehand. To address the computational constraints of mobile platforms, we propose a lightweight yet accurate model MateViT to perform pixel-wise semantic segmentation, simultaneously recognizing both objects and materials. Our methods achieve respective 40.2% and 51.1% of mIoU on COCOStuff-10K and DMS datasets, surpassing the previous method with +5.7% and +7.0% gains. Moreover, on the field test with participants, our wearable system reaches a score of 28 in the NASA-Task Load Index, indicating low cognitive demands and ease of use. Our MateRobot demonstrates the feasibility of recognizing material property through visual cues and offers a promising step towards improving the functionality of wearable robots for PVI. The source code has been made publicly available at https://junweizheng93.github.io/publications/MATERobot/MATERobot.html.