ResQbot: A Mobile Rescue Robot with Immersive Teleperception for Casualty Extraction

TL;DR

This paper introduces ResQbot, a mobile rescue robot with immersive stereoscopic teleperception, demonstrating improved safety and task performance in casualty extraction through experimental evaluation with a sensorized dummy.

Contribution

The paper presents a novel rescue robot with immersive teleperception and a new loco-manipulation approach, validated through experiments showing enhanced safety and task efficiency.

Findings

Safety parameters remained below injury thresholds.

Immersive teleperception improved task performance.

ResQbot successfully performed casualty extraction in experiments.

Abstract

In this work, we propose a novel mobile rescue robot equipped with an immersive stereoscopic teleperception and a teleoperation control. This robot is designed with the capability to perform safely a casualty-extraction procedure. We have built a proof-of-concept mobile rescue robot called ResQbot for the experimental platform. An approach called "loco-manipulation" is used to perform the casualty-extraction procedure using the platform. The performance of this robot is evaluated in terms of task accomplishment and safety by conducting a mock rescue experiment. We use a custom-made human-sized dummy that has been sensorised to be used as the casualty. In terms of safety, we observe several parameters during the experiment including impact force, acceleration, speed and displacement of the dummy's head. We also compare the performance of the proposed immersive stereoscopic teleperception to conventional monocular teleperception. The results of the experiments show that the observed safety parameters are below key safety thresholds which could possibly lead to head or neck injuries. Moreover, the teleperception comparison results demonstrate an improvement in task-accomplishment performance when the operator is using the immersive teleperception.

Figures

a shows the ResQbot platform that has been developed in this project. This platform is designed to be able to perform a casualty-extraction task based on the proposed loco-manipulation approach. This platform consists of two main modules: a differential-drive mobile base and a motorised stretcher-bed conveyor module. The platform is also equipped with a range of perception devices, including an RGB-D camera and a stereoscopic camera rig; these devices provide the perception required during the operation. We used an Xbox joystick controller for the operator interface to control the robot. We al

Tables

Table 1. Table 1: The summary of two significantly different trials—i.e. smooth and rough operation—performed during the experiment.

		Smooth Trial		Rough Trial
Max. instant acceleration ($m/s^2$)		$\approx 0.154$		$\approx 4.042$
Max. Velocity ($m/s$)
during initial contact		$\approx 0.015$		$\approx 0.16$
Victim’s head displacement ($m$)		$\approx 0.004$		$\approx 0.051$
Max. impact force ($N$)		$\approx 23.63$		$\approx 41.12$

Equations

$$F_i=ma+F_s$$

Full text

1 Introduction

Catastrophic events, disasters, or local incidents generate hazardous and unstable environments in which there is an urgent need for timely and reliable intervention, mainly to save lives. A multi-storey building fire disaster—such as the recent Grenfell Tower inferno in London, United Kingdom [1]—is an example of such a scenario. Responding to such situations is a race against time—immediate action is required to reach all potential survivors in time. Such responses, however, are limited to the availability of trained first responders as well as prone to potential risks to their lives. The high risk to the lives of rescue workers means that, in reality, fast response on-site human intervention may not always be a possibility.

Using robots in search-and-rescue (SAR) missions offers a great alternative by potentially minimising the danger for the first responders. Moreover, it is more flexible—the number of these robots can be expanded to perform faster responses. Thus, various robotic designs have been proposed to suit several specific SAR applications [2, 3]. These robots are designed to assist with one or sometimes multiple tasks as part of a SAR operation—including reconnaissance, exploration, search, monitoring, and excavation.

Our aim in this study is to develop a mobile rescue robot system (see Fig. 1a) that is capable of performing a casualty-extraction procedure. This procedure includes loading and transporting a human victim—a.k.a. casualty—smoothly, which is essential for ensuring the victim’s safety, via teleoperation mode. We also aim to develop more immersive teleperception for the robot’s operators to improve their performance and produce higher operation accuracy and safer operations.

2 Related Work

Wide-ranging robotics research studies have been undertaken in the area of search, exploration, and monitoring, specifically with applications in SAR scenarios [4, 5, 6]. Despite the use of the term ’rescue’ in SAR, little attention has been given to the development of a rescue robot that is capable of performing a physical rescue mission, including loading and transporting a victim to a safe zone—a.k.a. casualty extraction.

Several research studies have been conducted to enable the use of robots in the rescue phase of SAR missions. The majority of these studies focused on developing mobile robots—mainly tracked mobile robots—with mounted articulated manipulators [7, 8] or other novel arm mechanisms, such as elephant trunk-like arms [9] or snake-like arms [3]. Such designs enable the mobile robots to move debris by using their manipulators and also perform other physical interventions during the rescue mission, such as a casualty-extraction procedure.

Battlefield Extraction Assist Robot (BEAR) is one of the most sophisticated robot platforms designed and developed specifically for casualty-extraction procedures [2, 10]. This robot was developed by Vecna Technologies and intended for the U.S. Army [11]. It has a humanoid form with two independent, tracked locomotion systems. It is also equipped with a heavy-duty, dual-arm system that is capable of lifting and carrying up to 227-kg loads [12]. This robot is capable of performing casualty-extraction procedures by lifting up and carrying the casualty using its two arms. However, such a procedure could cause additional damage to the already injured victim—such as a spinal or neck injury from the lack of body support during the procedure.

Several investigations have been conducted on the development of rescue robots capable of performing safer and more robust casualty-extraction routines. These robots were designed to be more compact by using stretcher-type constructions or litters [13, 14, 15, 16]. The robots presented in these studies are intended for performing safe casualty extractions with simpler mechanisms compared to arm mechanisms. The use of the stretcher-type design on this robot also ensures the victim’s safety during transportation.

The mobile rescue robot demonstrated by the Tokyo Fire Department is one of the robots that uses this design concept [17, 18]. This robot is equipped with a belt-conveyor mechanism and also a pair of articulated manipulators. It uses its manipulators to lift up the casualty and place it onto the conveyor during the casualty-extraction procedure [19]. Then the belt conveyor pulls up the casualty into the container inside the robot for safe transportation. Compared to BEAR, this robot offers a safer casualty-transportation process. However, lifting up the casualty during the loading process is still a procedure that is highly likely to cause additional damage to the victim.

Iwano et al. in [13, 14, 15] proposed a mobile rescue robot platform capable of performing casualty extraction without a ”lifting” process. This robot loads the casualty merely using a belt-conveyor mechanism. This belt conveyor pulls the casualty from the ground onto the mobile platform, while the mobile platform synchronises the movement toward the body [15]. Since there is no ”lifting” process during the casualty-extraction procedure, this method is expected to be safer than the methods applied in BEAR and the Tokyo Fire Department’s robot. However, to the best of our knowledge, no safety evaluation of performing this extraction method has been published.

In terms of controlling method, these robots are still manually operated or teleoperated by human operators. The BEAR robot and the rescue robot demonstrated by the Tokyo Fire Department are teleoperated by human operators using a conventional teleoperation control setup [18]. On the other hand, based on the reports presented in [13, 14, 15], the rescue robot developed by Iwano et al. still requires human operators to be present on the scene to perform casualty-extraction procedures.

3 Research Contributions

In this work, we present a mobile rescue robot we developed that is capable of performing a casualty-extraction procedure using the method presented in [14]. Moreover, we equip this robot with teleoperation control and an immersive stereoscopic teleperception. As a proof of concept, we have designed and built a novel mobile rescue robot platform called ResQbot [20] shown in Fig. 1. This robot is equipped with an onboard stereoscopic camera rig to provide immersive teleperception that is transmitted via a virtual-reality (VR) headset to the operator.

The contributions to this work are:

Proof-of-concept ResQbot, including teleoperation mode with immersive teleperception via HTC Vive111Virtual Reality Head Mounted Display by VIVE: https://www.vive.com/ headset; 2. 2.

Preliminary evaluation of loco-manipulation-based casualty extraction using the ResQbot platform, in terms of task accomplishment and safety; 3. 3.

Evaluation of the proposed immersive teleperception compared with the other teleperception methods, including the conventional teleperception setup and direct observation as a baseline.

4 Casualty Extraction via Loco-Manipulation Approach

To perform a casualty-extraction procedure using the ResQbot platform, we propose using a loco-manipulation approach. By using this approach, the robot can implicitly achieve a manipulation objective—which is loading of a victim onto the robot—through a series of locomotive manoeuvres. We utilise the conveyor module mounted on the mobile robot base to create a simple mechanism for the loading of a victim onto the conveyor surface by solely following a locomotive routine.

Figure 2 illustrates the proposed casualty-extraction operation using the loco-manipulation technique. By removing the need for high-complexity robotic manipulators or mechanisms, this technique greatly simplifies the underlying controls required for conducting complex casualty extraction in rescue missions. This simplicity is highly beneficial for intuitive teleoperation by human operators.

This casualty-extraction procedure involves four major phases:

Relative pose adjustment: the robot aligns its relative pose with respect to the victim in preparation for performing the loco-manipulation routine. 2. 2.

Approaching: the robot gently approaches the victim to safely make contact with the victim’s head for initiating the loading process. 3. 3.

Loading: by using a balance between the locomotion of the base and the motion of the belt conveyor, the robot smoothly loads the victim onboard. Smooth operation at this stage is crucial in order to minimise traumatic injury caused by the operation. The victim is fully onboard when the upper body is fully loaded onto the stretcher bed. We consider the upper body to be from the head to the hip, thus protecting the critical parts of the body, including the head and spinal cord. 4. 4.

Fastening: once the victim is fully onboard, the strapping mechanism fastens the victim using a stretcher-strap mechanism in preparation for safe transportation. The conveyor surface serves as a stretcher bed for transporting the victim to a safe zone where paramedics can provide further medical care.

5 Robot Platform

We have designed and built a novel mobile rescue robot platform called ResQbot as a proof of concept for the implementation of the proposed casualty-extraction procedure. Figure 1a shows the ResQbot platform that has been developed in this project. This platform is designed to be able to perform a casualty-extraction task based on the proposed loco-manipulation approach. This platform consists of two main modules: a differential-drive mobile base and a motorised stretcher-bed conveyor module. The platform is also equipped with a range of perception devices, including an RGB-D camera and a stereoscopic camera rig; these devices provide the perception required during the operation. We used an Xbox joystick controller for the operator interface to control the robot. We also proposed an immersive teleperception interface for this mobile robot using an HTC Vive virtual reality headset. This headset will provide teleperception for the operator by displaying a real-time visual image sent from the onboard stereo camera of the mobile robot. Figure 1b shows the operator teleoperates ResQbot using the proposed teleoperation and teleperception devices.

5.1 Mobile Base Module

The mobile base module used for the ResQbot platform is a differential drive module. This platform is a customised version of a commercially available powered wheelchair—Quickie Salsa-M—manufactured by Sunrise Medical222Quickie Salsa-M powered wheelchair by Sunrise Medical: www.sunrisemedical.co.uk. This mobile base is chosen for its versatile design and stability owing to its original design purpose, which was to carry the disabled both indoors and outdoors. This platform has a compact turning circle while ensuring stability and safety through its all-wheel independent suspension and anti-pitch technology over rough or uneven terrain. Its mobile base is also capable of manoeuvring through narrow pathways and confined spaces, due to its compact design (only 600 mm in width).

5.2 Motorised Stretcher Bed Module

ResQbot is equipped with an active stretcher-bed module that enables active pulling up of the victim’s body while the mobile platform is moving. This module is mounted at the back of the mobile base via hinges allowing it to fold (for compact navigation) or unfold (for loading and transporting the victim) on demand. This stretcher bed is composed of a belt-conveyor module that is capable of transporting a maximum payload of 100 kg at its maximum power. This belt conveyor is powered by a 240 VDC motor with 500 W maximum power. The motor is controlled through a driver module powered by a 240 VAC onboard power inverter, and the pulse-width modulation (PWM) control signal is used to control the motor’s speed.

During loading of the victim’s body, the active-pulling speed of the belt conveyor has to be synchronised with the locomotion speed of the mobile platform. Therefore, this module is equipped with a closed-loop speed control system to synchronise the conveyor speed and the mobile base locomotion speed. An incremental rotary encoder connected to the conveyor’s pulley is used to provide speed measurements of the belt conveyor as feedback to the controller. Another incremental rotary encoder is connected to an omnidirectional wheel attached to the floor. This encoder provides measurements of the mobile base linear speed. The measured mobile base linear speed is used for the speed reference of the conveyor controller.

For a safe transportation process, the victim has to be safely placed onboard the stretcher bed. Thus, this stretcher-bed module is also equipped with a motorised stretcher strap to enable fastening of the victim on the bed as a safety measure. This stretcher-strap module is powered by a 24 VDC motor. The motor is controlled to fasten and unfasten the stretcher strap during the casualty-extraction procedure.

6 Experimental Setups and Results

Experimental Setting. We have conducted a number of experimental trials to evaluate our proof-of-concept mobile-rescue-robot platform, ResQbot, in terms of task accomplishment, safety and teleperception comparison. In these experiments, we conducted a mock casualty-extraction procedure using ResQbot by teleoperation with three different teleperception modalities:

• •

Direct mode (baseline): user controls the robot while being present at the scene;

• •

Conventional mode: user receives visual feedback provided by a monocular camera through a display monitor;

• •

Immersive mode: the user receives stereoscopic vision provided by an onboard stereoscopic camera module and through a virtual reality headset.

Ninety series of trials in total were conducted, with 30 series for each teleperception modality. We conducted this number of trials to capture any possible problems encountered during the trials. For the whole experiment, we used the same setup of the victim and its relative position and orientation with respect to the ResQbot. These various victim positions are inside the area of the ResQbot perception device’s field of view.

To evaluate the safety of the proposed casualty-extraction procedure using the ResQbot platform, we conducted the trials using a sensorised dummy as the casualty. This dummy was equipped with an inertial measurement unit (IMU) sensor placed on its head. We used 3DM-GX4-25333LORD MicroStrain IMU: http://www.microstrain.com/inertial/3dm-gx4-25 IMU sensor, with resolution $<$ 0.1 mg and bias instability $\pm$ 0.04 mg. During the trials, we recorded the data from this sensor at 100 Hz sampling frequency.

Task Accomplishment. In general, we achieved successful task accomplishment of the casualty-extraction procedure in every trial. Screenshot images in Figure 3 demonstrate the procedure applied during the casualty-extraction operation performed by ResQbot in the experiments.

Safety Evaluation. In this experiment, we observed the impact applied to the dummy’s head during the casualty-extraction procedure. This impact caused force and displacement of the dummy’s head. The observation was focused on the part during the loading phase when the robot—i.e. the stretcher-bed conveyor—made first contact with the dummy’s head. Two extreme cases were selected (i.e. roughest and smoothest operation, respectively) for this safety evaluation. These two cases represent the largest and the smallest maximum instant acceleration of each test within the whole experiment.

Figure 4 shows the dummy’s head displacement, speed and acceleration during the loading process caused by the robot’s first contact. Two extreme cases are presented; one is the smoothest trial (in blue), and the other one is the roughest trial (in red). The dashed vertical line (in green) indicates the time-step at which the contact was initiated during the loading process.

The forces applied to the dummy’s head can be estimated based on these measured accelerations from the IMU, and it can be calculated via:

[TABLE]

in which, $F_{i}$ corresponds to the estimated instantaneous force applied to the dummy’s head, and $F_{s}$ corresponds to the static friction force between the dummy’s head and the ground [21]. The estimated force was calculated based on the measured maximum instant acceleration (a) during the operation and the approximated mass of the dummy’s head (m). Table 1 summarises the observations of the two significant cases—i.e. smoothest and roughest trial—during the experiment. These cases correspond to the maximum instantaneous accelerations of the dummy’s head caused by the robot during the casualty-extraction procedure.

We compared the results presented in table 1 with several key safety thresholds—which were reported in the literature as possible causes of head or neck injuries to the casualty [22, 23]. Figure 5 illustrates the comparison between the trial results and the thresholds from the literature. It can be seen that all evaluated parameters in the experiments are relatively small and below the threshold. Even though not conclusive yet, these preliminary results show high safety promise for the proposed platform of the casualty-extraction procedure, and for further development. Thus it also motivates more elaborate safety evaluations for the practical deployment of the platform.

Teleperception Comparison. In this work, we proposed immersive teleperception via an HTC Vive headset to provide visual perception for the operator when operating the ResQbot platform. We evaluated this proposed teleperception modality by comparing it to the conventional teleperception setup (i.e. providing visual perception via single monitor) and the direct-observation scenario as a baseline (i.e. controlling the robot while being present at the scene).

We compared these three perception modes in terms of the smoothness of the casualty-extraction procedure, which is represented by the maximum force applied to the dummy’s head during the experiment in both cases. Figure 6 shows a box plot of the distribution of the estimated maximal forces in the three different perception modes during the experiments. According to Figure 6, in terms of the smoothness of the procedure, we observed that the VR mode (i.e. immersive) results in a higher population of smoother trials than the conventional mode. In fact, the VR mode achieves smooth operations with low maximal estimated forces under 28 N for approximately 50$\%$ of its trials, similar to the direct-observation mode, in which the operator has direct access to observe the scene during the operation.

7 Conclusion

In this paper, we presented a proposed mobile rescue robot system that is capable of safely loading and transporting a casualty. We proposed an immersive stereoscopic teleperception modality via an HTC Vive headset to provide the teleoperator with more realistic and intuitive perception information during the operation. As a proof of concept of the proposed system, we designed and built a novel mobile rescue robot called ResQbot, which is controlled via teleoperation. We evaluated the proposed platform in terms of task accomplishment, safety and teleperception comparison for completing a casualty-extraction procedure. Based on the results of our experiments, the proposed platform is capable of performing a safe casualty-extraction procedure and offers great promise for further development. Moreover, the teleperception comparisons highlight that the proposed immersive teleperception can improve the performance of the teleoperator controlling the mobile robot’s performance during a casualty-extraction procedure.

Acknowledment

Roni Permana Saputra would like to thank Indonesia Endowment Fund for Education - LPDP, for the financial support of the PhD program. The authors would also like to show our gratitude to Arash Tavakoli and Nemanja Rakicevic for helpful discussions and inputs for the present work.