MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments

TL;DR

MateRobot is a wearable vision-based system that accurately recognizes objects and materials for visually impaired users, enhancing pre-touch perception with low cognitive load.

Contribution

The paper introduces MateViT, a lightweight model for pixel-wise semantic segmentation, enabling real-time material and object recognition on mobile platforms.

Findings

Achieved 40.2% and 51.1% mIoU on COCOStuff-10K and DMS datasets.

Field tests showed low cognitive demand with a NASA-TLX score of 28.

Surpassed previous methods by 5.7% and 7.0% in segmentation accuracy.

Abstract

People with Visual Impairments (PVI) typically recognize objects through haptic perception. Knowing objects and materials before touching is desired by the target users but under-explored in the field of human-centered robotics. To fill this gap, in this work, a wearable vision-based robotic system, MateRobot, is established for PVI to recognize materials and object categories beforehand. To address the computational constraints of mobile platforms, we propose a lightweight yet accurate model MateViT to perform pixel-wise semantic segmentation, simultaneously recognizing both objects and materials. Our methods achieve respective 40.2% and 51.1% of mIoU on COCOStuff-10K and DMS datasets, surpassing the previous method with +5.7% and +7.0% gains. Moreover, on the field test with participants, our wearable system reaches a score of 28 in the NASA-Task Load Index, indicating low cognitive demands and ease of use. Our MateRobot demonstrates the feasibility of recognizing material property through visual cues and offers a promising step towards improving the functionality of wearable robots for PVI. The source code has been made publicly available at https://junweizheng93.github.io/publications/MATERobot/MATERobot.html.

Tables

Table 1. TABLE I : Results of single-task learning on DMS dataset val / test set. GFLOPs are measured in size of 512 × 512 512 512 512{\times}512 .

Method	Backbone	GFLOPs	MParams	Pixel Acc (%)	mIoU (%)
PSPNet [16]	MobileNetV2	06.12	02.13	66.5 / 66.3	26.1 / 25.9
DeepLabV3 [25]	MobileNetV2	07.52	02.58	68.4 / 68.1	29.8 / 29.7
DeepLabV3+ [15]	MobileNetV2	07.85	03.11	69.3 / 69.2	30.1 / 29.9
LR-ASPP [26]	MobileNetV3-s	05.41	01.14	66.8 / 66.5	26.5 / 26.4
LR-ASPP [26]	MobileNetV3	08.78	03.29	70.7 / 70.2	30.3 / 29.9
DMS [10]	ResNet-50	-	-	73.1 / 72.9	43.5 / 42.0
MateViT (ours)	ViT-Tiny	04.30	07.78	76.9 / 76.8	45.3 / 44.1
MateViT (ours)	ViT-Small	15.54	28.79	79.6 / 79.4	51.0 / 50.1
w.r.t. DMS				+6.5 / +6.5	+7.5 / +8.1

Table 2. TABLE II : Results (mIoU) of multi-task learning on COCOStuff-10k and DMS test sets. GFLOPs @ 512 × 512 512 512 512{\times}512 .

Method	Backbone	GFLOPs	MParams	COCOStuff (%)	DMS (%)
Trans4Trans [5]	MiT-B0	12.24	04.16	27.7	37.1
Trans4Trans [5]	MiT-B1	18.98	14.26	30.6	41.3
Trans4Trans [5]	MiT-B2	27.00	25.30	34.5	44.1
MateViT (ours)	ViT-Tiny	07.95	11.62	32.7	45.1
MateViT (ours)	ViT-Small	22.08	37.10	40.2	51.1
w.r.t. Trans4Trans				+5.7	+7.0

Table 3. TABLE III : Ablation study on COCOStuff-10K and DMS. All values are calculated with test sets.

Method	COCOStuff-10K		DMS
	Pixel Acc (%)	mIoU (%)	Pixel Acc (%)	mIoU (%)
Segmenter [22]	65.0	31.3	76.9	43.2
+ ViT-Small	69.1	38.2	79.0	49.2
+ LIS	70.2	38.9	79.4	50.1
+ MoE	70.8	39.4	79.9	50.7
+ MMoE (MateViT)	71.5	40.2	80.6	51.1

Equations

$$MHSA(\text{Q},\text{K},\text{V})=Softmax(\frac{{\text{QK}}^T}{\sqrt{C}})\text{V}$$

$$G(\text{x})$$

$$H(\text{x})$$

$$N(\text{x})$$

$$TopM(\text{v},m{)}_i$$

Full text

I Introduction

In 2020, it was estimated that approximately 43 million individuals were living with blindness. By 2050, this number is expected to rise significantly to 61 million [1]. Therefore, it is imperative to facilitate the development of assistive systems for helping People with Visual Impairments (PVI). In recent years, considerable progress has been witnessed in the field of human-centered assistive technology, such as systems for navigation [2], object localization [3], indoor understanding [4], and path orientation [5].

Material recognition is often a challenging task for PVI, who typically recognize objects through touch [6, 7]. However, material recognition is under-explored in the domain of assistive technology. Therefore, it is essential to develop a wearable system that can assist PVI in recognizing the materials of objects without touching them, i.e., a contact-free material recognition system.

In this work, we present the wearable MateRobot system to cover both object and material recognition. As shown in Fig. 1, the system consists of a pair of smart glasses with an RGB-D camera, a pair of bone-conducting earphones, and a portable image processor inside a small bag. In Fig. 1, the MateRobot can recognize materials (indicated by underline, e.g., snow) and general objects (indicated in italics, e.g., sidewalk). These two predictions can form feedback of object categories with material properties (e.g., “snowy sidewalk”). This information is conveyed to the blind user in the form of speech through bone-conduction headphones.

Due to computationally complex self-attention operations in vision transformers [8], it is, however, hard to deploy a resource-intensive ViT-based model on portable platforms. To address this, we propose an efficient material segmentation ViT-based model, namely MateViT, which includes a Learnable Importance Sampling (LIS) strategy to maintain only the informative tokens for the material segmentation, so as to reduce the computational cost. Thanks to the LIS method, a resource-friendly MateViT model is obtained, enabling the deployment of vision transformer on wearable robotic devices that have limited computational resources.

Apart from the model efficiency, to enlarge the model capacity, we introduce a Multi-gate Mixture-of-Experts (MMoE) method to combine the aforementioned general image semantic segmentation and material semantic segmentation on a single model. Compared to the previous method [5] using a straightforward dual-head structure, our MMoE is used to construct a high-performance and efficient multi-task learning architecture. The feature tokens from input images are forwarded to respective gates and experts to extract informative features for task-relevant decoder heads that generate final semantic segmentation masks.

In order to endow MateRobot with robust perception capability, including general object and material semantic segmentation, we perform model training on COCOStuff-10K [9] and DMS [10] datasets, both contain more than $10K$ training samples. Through extensive experiments, our small model obtains $40.2\%$ and $51.1\%$ of mIoU scores, surpassing the previous multi-task learning baseline [5] by absolute ${+}5.7\%$ and ${+}7.0\%$ on COCOStuff-10K and DMS, respectively. For single-task learning on materials segmentation, i.e., on DMS, our model reaches state-of-the-art performance, having ${+}8.1\%$ mIoU gains compared to previous CNN counterpart [10]. To verify the practicability of our MateRobot for recognizing material categories in real-world scenarios, we conduct a task-oriented user study with six blindfolded participants. On the post-study questionnaires, our system obtains respective $28$ and $77$ scores regarding the evaluation criteria of NASA-Task Load Index (NASA-TLX) and System Usability Scale (SUS), which indicates the ease of use and the usability of our MateRobot in practical scenarios. In summary, our main contributions are:

• •

For the first time, we integrate material recognition into assistive technology and built a wearable robot system, i.e., MateRobot. This system empowers People with Visual Impairments (PVI) to achieve contactless long-distance perception similar to that of sighted people.

• •

We propose an efficient MateViT model to enable the deployment of resource-intensive ViT-based counterparts on resource-constrained mobile platforms by using a Learnable Importance Sampling (LIS) strategy.

• •

A Multi-gate Mixture-of-Experts (MMoE) method is designed to simultaneously perform object and material semantic segmentation in one model.

• •

Through a task-oriented user study and qualitative analyses, we gain valuable insights into designing wearable material recognition systems for PVI.

II Related Work

II-A Wearable Assistive System

With the tremendous capability revealed by computer vision algorithms, vision-based wearable assistance systems [11, 12, 13] are becoming increasingly applicable. A vision-based navigation system [2], calculating precise positions and orientations, is proposed to help PVI stay on track while walking, and it can recognize unexpected dynamic obstacles. Due to the COVID-19 pandemic, an object-finding algorithm is introduced in [3] to build a robotic cane system. In [4], a lightweight system with a solid-state LiDAR sensor is proposed for indoor detection and avoidance. In a previous work [5], to cover the segmentation of transparent objects, a dual-head model is deployed on a wearable device. However, only limited recognizable materials are delivered in previous wearable assistance systems. In order to help blind users obtain a more comprehensive and humanized experience on material recognition, we design a wearable robotic system in this work, for the first time, which can not only recognize conventional object categories, such as cups, but also further recognize the material of the object, such as plastic cups, delivering contactless object recognition.

II-B Material Semantic Segmentation

Recently, Vision Transformer [8, 14] is proposed to utilize the self-attention operation in transformer layers to extract non-local features from a sequence of image patches, yielding an alternative backbone solution compared to convolutional counterparts [15, 16, 17]. In DMS [10], a model based on ResNet [17] is used to address dense material segmentation. In contrast to DMS [10], we adopt Vision Transformer as the backbone, e.g., ViT [8], to perform material semantic segmentation, with the aim of extracting long-distance dependencies between image patches, since the long-range contextual information is crucial for robust representation of diverse materials [5]. However, due to the high computational demand of self-attention operation, there is still a bottleneck when deploying a plain vision transformer on resource-constrained mobile platforms, e.g. mobile robots and wearable devices. In this work, we propose a novel importance sampling method to reduce the number of tokens and maintain only the informative ones for material segmentation to enable the deployment of plain vision transformers on wearable devices.

III MateRobot: A Wearable Robotic System

III-A Hardware Component

As shown in Fig. 2, there are three main hardware components in our MateRobot, including a pair of KRVision smart vision glasses, a portable processor, and a power bank inside a waist bag. Inside the smart glasses, there is an RGB-Depth camera RealSense R200 and a pair of bone-conduction headphones. For the concept of human-friendly design, there are three advantages of using bone-conduction earphones, which are comfortable to wear, clean and hygienic, and keep in touch with the outside world. Maintaining awareness of ambient sounds is especially important for PVI. To ensure higher portability of the system, we tried different processors and chose the smaller NVIDIA AGX Xavier due to its applicable energy efficiency and inference capabilities. Furthermore, a power bank with high energy capacity is selected to provide the system with up to $6$ hours of battery life, which greatly reduces the battery life anxiety of traditional wearable devices [4]. Through the above hardware components, a more portable MateRobot and better user experience can be delivered to PVI when performing contactless material recognition in real-world scenarios.

III-B User Interaction

Between the system and the user, we design an easy-to-use interface for PVI. First of all, in order to give users timely information feedback, we adopt an adjustable feedback frequency. For example, if users set a larger interval in a more familiar environment, e.g., home, they will get concise information. If they explore an unknown space, setting a high frequency can obtain more information. Besides, between the pixel-wise image segmentation results to the auditory output to users, only the detected object and material located in the middle of the input image will be selected. Their pre-defined texts are used to generate speeches jointly in the form of “material objects”, such as “ceramic cups” or “metal forks”.

III-C MateViT Model

Apart from the hardware components, we further detail the efficient model designed for MateRobot. To equip mobile platforms with plain vision transformers, we propose MateViT, which has ViT [8] with Learnable Importance Sampling (LIS) as the backbone, followed by an upsampling layer, a Multi-gate Mixture-of-Experts (MMoE) layer and two decoder heads. The architecture is shown in Fig. 3. Note that our method is flexible to include more tasks by adding decoder heads. Mixed by two datasets, each input sample is first encoded by the efficient ViT backbone and then fed into the upsampling layer. One gate in the MMoE layer corresponds to one task, receiving the data sample that is only relevant to the task. Depending on the output of the gate, different selected experts are activated to learn meaningful latent representations, which are finally decoded by the task-relevant decoder head. Different from the training process, one data sample is fed into all gates synchronously during inference, and latent representations from selected experts are then decoded by corresponding decoder heads, producing predictions for all tasks.

III-D Learnable Importance Sampling

To make plain ViT [8] more lightweight and feasible in real-world applications, we propose a Learnable Importance Sampling strategy for material semantic segmentation, yielding an efficient MateViT. Our approach does not require an additional class token in forward pass compared to EViT [18]; therefore, it further reduces the model complexity with high-resolution inputs. As illustrated in Fig. 3, all image patches are flattened and projected into tokens, forming Queries (Q), Keys (K), and Values (V). Multi-Head Self-Attention (MHSA) is then calculated as:

[TABLE]

where $\textbf{Q}{\in}\mathbb{R}^{N\times C}$, $\textbf{K}{\in}\mathbb{R}^{N\times C}$ and $\textbf{V}{\in}\mathbb{R}^{N\times C}$ are query, key and value matrices; $N$, $C$ are token number and dimension. Since softmax is introduced in attention map calculation, the summation of each value in rows is equal to $1$. However, the result does not always equal to $1$ when summing up all values in columns, indicating the importance of each token from an image. Based on this observation, we first calculate the summation in columns and then average the importance vectors among all heads. Top $k$ values are selected and tokens are downsampled according to the indices of these $k$ values after Add & Norm, which stand for a residual link [17] and layer normalization [19]. Fig. 3 illustrates the whole LIS process in detail. Following [20], the downsampled tokens are then sent to a Feed-Forward Network (FFN) followed by another Add & Norm. Since the token number is reduced after the importance sampling, an upsampling layer is then applied, which is a standard transformer decoder block [20]. Thanks to LIS, a plain ViT-based model is expedited and better qualified for real-world mobile applications.

III-E Multi-gate Mixture-of-Experts

Since the performance of a wearable robotic system is relevant to its model capacity, Mixture-of-Experts [21] is proposed for enlarging model capacity while maintaining invariant model complexity. However, the usage of MoE is less discussed on wearable systems [5]. For the first time, we adopt the MoE method to perform complementary training of both general object and material segmentation, i.e., the former prevents the latter from overfitting and vice versa. More specifically, we adopt a Multi-gate Mixture-of-Experts (MMoE) layer in our model for high efficiency and performance. Fig. 3 shows the detail of the MMoE layer for multi-task learning. Specifically, one gate is responsible for one task to select the experts. Similar to [21], the resulting selection vector $G(\textbf{x})$ can be described as:

[TABLE]

where x, $\textbf{W}_{g}$, and $\textbf{W}_{noise}$ are token, gate matrix, and noise matrix, respectively. The noise term $N(\textbf{x})$ is utilized for load balance, which is the multiplication of the standard normal distribution sampling $Normal()$ and the outcome of $Softplus(\cdot)$. During the training, for every token in one image, only one same gate is activated to produce the selection vector. According to the indices of the top $m$ values, $m$ experts are selected and the token is only fed into the $m$ experts. The output of the MMoE layer is a weighted sum of the top $m$ values in the selection vector and their corresponding outcomes from the $m$ experts. A task-relevant decoder head [22] is then applied to transform all output tokens from MMoE into a prediction mask. Note that we also employ the load and importance balancing loss following [21] besides the task loss. During the inference, every token from one image is fed into all gates synchronously. The resulting weighted sums from selected experts are then decoded by the corresponding decoder heads shown in Fig. 3. As the model is trained jointly on two tasks, the knowledge absorbed from object segmentation plays an important role in the high performance of material recognition, which provides PVI with accurate material information in their daily life.

IV Experiments

IV-A Settings and Datasets

Settings. We implement the model with PyTorch $1.12.1$ and CUDA $11.6$. The learning rate is initialized as $0.01$ and it is scheduled by a cosine annealing strategy [23]. SGD with momentum $0.9$ is adopted as the optimizer. We initialize our efficient ViT backbone with a pre-trained plain ViT [8] and keep other layers of the model randomly initialized. Data augmentations like random horizontal flipping, random resize with a ratio $0.5\text{-}2$, and random cropping to $512{\times}512$ are used during training. Note that we do not use other tricks such as OHEM, auxiliary loss, and class-weighted loss for a fair comparison to other methods. We train our model with a batch size of $4$ for $200$ epochs on four 1080Ti GPUs.

Datasets. We adopt COCOStuff-10K [9] and DMS [10] for general object and material segmentation, respectively. The COCOStuff-10K dataset [9] has $9000/1000$ images for training/testing. We conduct experiments following the implementation of mmsegmentation [24] with $171$ categories. The DMS dataset [10] has respective $21857/9057/9152$ images for training/validation/testing with $46$ categories.

IV-B Results on Material Segmentation

To verify the proposed method for material segmentation, we conduct experiments of material segmentation on the DMS dataset [10]. Results are reported in Table I. It can be observed that our model using the ViT-Tiny backbone still has the lowest computation expense ($4.30$ GFLOPs) with high performance ($44.1\%$ in mIoU), and the ViT-Small variant outperforms other methods in both pixel accuracy and mIoU. Furthermore, Fig. 4 presents the per-class IoU of all material categories. It is worth noting that our ViT-Small variant achieves performance gains in all $46$ categories, especially those are relevant for assisting PVI, e.g., fire (gain ${+}20.2\%$), snow (gain ${+}21.3\%$), plastic (gain ${+}22.6\%$), and ceramic (gain ${+}25.5\%$). The impressive pre-study test scores obtained in our evaluations serve as a testament to the effectiveness and reliability of our proposed system in assisting people with visual impairments (PVI) in accurately recognizing and distinguishing materials.

IV-C Results on Multi-task Learning

To deliver more information to PVI and to perform complementary training, we further conduct experiments on multi-task learning based on MateViT, covering both object and material segmentation at once. Table II illustrates the quantitative results. Compared to Trans4Trans MiT-B0 [5], our model with ViT-Tiny requires less computation expense (${-}35.0\%$ GFLOPs), while reaching higher performance on both datasets. Additionally, compared to the Trans4Trans MiT-B2 variant [5], it becomes evident that our ViT-Small variant has a higher mIoU on both COCOStuff-10K (gain ${+}5.7\%$) and DMS (gain ${+}7.0\%$). More important, lower computational complexity (i.e., GFLOPs) is intuitive to reflect the high efficiency of the model running on the mobile platform and, therefore, to improve the user experience of the system. Compared to the single-task results in Table I, multi-task learning brings further improvements. The reason for the gains is two-fold: (1) the MMoE layer enlarges the model capacity; (2) when applying MMoE to both object segmentation and material segmentation, the former prevents the latter from overfitting and vice versa.

IV-D Ablation Study

To fully understand the proposed components, an ablation study is conducted, as shown in Table III. The baseline model is Segmenter ViT-Tiny [22]. Learned from Table III, replacing ViT-Tiny with ViT-Small boosts the performance ($43.2\%{\rightarrow}49.2\%$ in mIoU). After applying LIS, the mIoU continuously increases ($49.2\%{\rightarrow}50.1\%$). We then add the MoE layer to our model, leading to an even higher mIoU of $50.7\%$. It can be observed that our MMoE model achieves the best performances on all metrics in both object segmentation and material segmentation tasks compared to the baseline model, i.e., $6.5\%$ and $3.7\%$ boosts in pixel accuracy, $8.9\%$ and $7.9\%$ boosts in mIoU. Through the extensive pre-study experiments, the effectiveness of the proposed method can be comprehensively proved.

IV-E Qualitative Analysis

Four groups of scenarios related to the daily life of PVI are shown in Fig. 5. The upper-left group describes a scenario where PVI are having their meals. The hot dog colored orange is perfectly recognized by our system in the second image, and it is tagged with a food property from the third image. The group in the lower left corner shows a scenario where blind people walk in a park. According to the predictions in the second and third images, blind people using our system know additionally there is an asphaltic road ahead. In the bottom-right case, the entrance is not recognized if only object recognition is performed, however, the material segmentation result can provide supplementary information to find the doors, which can further improve the mobility accessibility. As a result, our system can help PVI better understand their surroundings, and to support them to make correct interactions with the environment.

V User Study

Following detailed pre-study experiments based on datasets, for wearable robotic systems, a critical factor that needs to be addressed is whether the system can provide a positive user experience in real-world situations. To know about that, we conduct a user study in a structured manner, including task-oriented testing and a questionnaire session.

V-A Organization

To verify the system’s usability, we organize a user study with six blindfolded participants on real-world testing scenarios. We select seven categories of materials that are common in daily life, i.e., fabric, foliage, glass, metal, paper, plastic, and wood. To conduct a comparison between without and with using our system, there are two rounds of material recognition, i.e., contact and contactless round. The contact round is to recognize by touch, while the contactless round is to recognize by using our system. Note that, to conduct a fair comparison, the order of the objects in two rounds is randomized. The participants are required to name the material after touching an object. The reaction time from touching the object until naming the material is recorded by the organizer. After two rounds of material recognition, all participants take part in an anonymous questionnaire session. The questionnaires regarding the NASA-Task Load Index (NASA-TLX) and System Usability Scale (SUS) are filled out by all participants. Besides, there is space for participants to write down their open comments.

V-B Results and Discussion

Cognitive load. To learn about the cognitive load of our wearable system, NASA-TLX is a simple and effective method for cognitive load measurement. We first calculate the average score of every factor among all participants. There are six factors, which are Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration, as shown in Fig. 7. We then average the scores of all six factors, resulting in a final NASA-TLX value of $28$. According to [27], this value illustrates the workload caused by our system is in the 20th percentile of global workload scores from $6.21$ to $88.50$ among $1173$ observations, which can assist users without too much burden. From Fig. 7, we notice that the effort value is relatively smaller than the rest five factors, meaning that our system is user-friendly.

Efficiency. We utilize average reaction time to evaluate the efficiency. The average reaction time in the contactless round is ${3.11}$s (±$0.21$), while the contact one is $3.97$s (±$0.34$). Without any haptic perception, our wearable system can perform a faster recognition than the contact-based perception, which indicates our MateRobot is feasible and reliable to provide fast assistance in recognizing material properties. More importantly, contactless object and material recognition can provide psychological safety for PVI by allowing them to identify objects without touching them. This can help PVI avoid potential hazards and feel more confident and independent in their surroundings. Additionally, contactless systems can free up their hands and allow them to identify objects from a longer distance, just like sighted people.

Usability. Apart from NASA-TLX, we verify the usability of our system with SUS. Our system scores $77$ out of $100$, which is a relatively high score. According to Bangor et al. [28], who analyzed $2324$ surveys from $206$ studies, “the best quarter of studies range from $78.51$ to $93.93$”. Therefore, we find that our system is useful for recognizing not only general objects but also their material properties.

User comments. We analyze the open comments made by users during the testing and from the post-study questionnaire session. The insights are reported below: (1) $66.7\%$ participants were amazed by the fast response of the MateRobot system, which is one of the reasons why they would like to use the system. (2) The user experience with the system was impressive for all participants. They found our system useful and helpful in the daily lives of visually impaired people. (3) Some participants suggest that the voice feedback of the glasses should constantly inform the user of the detected objects and materials. We plan to improve this in future work.

Limitations. Due to the challenges posed by the pandemic, it was difficult to find a sufficient number of participants to perform a user study on the developed system. The system was tested by six blindfolded participants. While the field test provides insights into the ability of our system, the results cannot be considered representative of the experience of real blind users. For future work, we plan to conduct studies involving users who are blind or visually impaired.

VI Conclusion

In this work, we look into semantic material understanding for helping visually impaired people via a wearable robotic system MateRobot. We put forward MateViT, which unifies general object and material segmentation via a multi-gate mixture-of-experts architecture, whose efficiency is enhanced via learnable importance sampling to make plain-vision-transformer models suitable for mobile applications. The proposed model is ported to our established assistive MateRobot system designed for supporting people with visual impairments. Extensive experiments on DMS and COCOStuff-10K datasets and a user study demonstrate the effectiveness and usefulness of our recognition system.