Active Robot-Assisted Feeding with a General-Purpose Mobile Manipulator: Design, Evaluation, and Lessons Learned

TL;DR

This paper presents a novel autonomous meal-assistance system using a general-purpose mobile robot, evaluated with users with motor impairments, demonstrating safety, usability, and effectiveness in aiding independent eating.

Contribution

The work introduces a versatile active feeding system with autonomous capabilities and safety features, specifically designed for users with severe motor impairments, and shares valuable lessons learned.

Findings

Participants successfully used the system for eating various foods.

High success rates in autonomous behaviors reported by users.

Users found the system comfortable, safe, and easy to operate.

Abstract

Eating is an essential activity of daily living (ADL) for staying healthy and living at home independently. Although numerous assistive devices have been introduced, many people with disabilities are still restricted from independent eating due to the devices' physical or perceptual limitations. In this work, we present a new meal-assistance system and evaluations of this system with people with motor impairments. We also discuss learned lessons and design insights based on the evaluations. The meal-assistance system uses a general-purpose mobile manipulator, a Willow Garage PR2, which has the potential to serve as a versatile form of assistive technology. Our active feeding framework enables the robot to autonomously deliver food to the user's mouth, reducing the need for head movement by the user. The user interface, visually-guided behaviors, and safety tools allow people with severe motor impairments to successfully use the system. We evaluated our system with a total of 10 able-bodied participants and 9 participants with motor impairments. Both groups of participants successfully ate various foods using the system and reported high rates of success for the system's autonomous behaviors. In general, participants who operated the system reported that it was comfortable, safe, and easy-to-use.

Tables

Table 1. Table 1 : A survey of recent robot-assisted feeding systems.

	Platform	Interface\tnotextnote:robots-r1	Tool\tnotextnote:robots-r2	Teaching/Movement Type		Base	Safety Tool
				scooping	delivery
	My Spoon [10]	Joystick	sf	Predefined	-	Fixed	-
	Bestic arm [11]	Button	s	Predefined	-	Fixed	-
	Meal Buddy [19]	Joystick	s	Predefined	-	Fixed	-
	Mealtime [12]	Button	s	Predefined	-	Fixed	A shatterproof spoon
Commercial	Obi [20]	Button	s	Predefined	Kinesthetic	Fixed	Collision detection
	Yamazaki and Masuda [21]	GUI(H)	sfc	User-selected\tnotextnote:robots-r3	Predefined	Fixed	Force detection
	Song and Kim [14]	Joystick & Button	sg	Predefined	Predefined	Fixed	-
	Schroer et al. [22]\tnotextnote:robots-r4	BCI\tnotextnote:robots-r5	N/A	N/A	Vision	Movable	-
	Kobayashi et al. [23]	Touch sensor	sc	Vision	Predefined	Fixed	Spring joint
	Perera et al. [24, 25]	BCI	s	Predefined	Predefined	Fixed	-
	Admoni and Srinivasa [26]	Joystick & Gaze	f	User-selected	Predefined	Fixed	-
	Candeias et al. [27]	N/A	s	Predefined	Vision	Fixed	-
Academic	Our Work	GUI(H)	sf	Vision	Vision	Movable	Execution monitor

Table 2. Table 2 : The summary of high-level operating procedures. Our meal-assistance system can perform three independent subtasks: scooping/stabbing, spoon wiping, and delivery. The multimodal information from the listed sensors enables the system to provide active and safe feeding assistance. The location of sensors are visualized in Fig.  12 .

Subtasks	Details	Sensors
Scooping / Stabbing	The user clicks the Scooping button on the GUI described in Sec. 4.1. The robot then autonomously scoops a spoonful of yogurt from a random or visually selected location in the bowl using our visual food-location estimator described in Sec. 4.2 and 4.4.	A head-mounted camera and joint encoders
Spoon wiping	The user clicks the Wiping button if the scooping task results in excess food on the top or bottom of the spoon. The robot wipes off the surface of the spoon using the wiping bar on the bowl (see details in Sec. 4.2).	Joint encoders
Delivery	The user clicks the Feeding button on the GUI when an adequate amount of yogurt is present on the spoon. The user turns his or her head toward the camera on the robot’s right wrist. The robot then estimates the pose of the user’s mouth and delivers yogurt inside the mouth. It then pulls the spoon back from the mouth. (see details in Sec. 4.2) To prevent potential injuries due to the robot’s anomalous execution, the robot monitors the pattern of multi-modal sensory signals (see details in Sec. 4.5). We also run an optimization-based low-gain impedance controller (see details in Sec. 4.3).	A wrist-mounted camera, joint encoders, a microphone array, tactile skin sensors, a force-torque sensor, and current sensors

Table 3. Table 3 : Five-point Likert type questionnaire items for 9 able-bodied participants. The last column provides the average and standard deviation of scores with 1=strongly disagree, 2=disagree, 3=neither, 4=agree, and 5=strongly agree. Results are considered to have reasonable accuracy when relative standard error (RSE) is 25% or less.

Questionnaires	Score
	Avg.	Std.	RSE
I am familiar with engineering.	4.78	0.67	4.65%
I am familiar with robotic applications.	4.89	0.33	2.27%
I successfully ate food using the system.	4.67	1.00	7.14%
I am satisfied with using the system.	3.89	0.93	7.95%
The system was easy-to-use.	4.00	1.00	8.33%
I felt safe while using the system.	4.22	0.83	6.58%
I was comfortable while using the system.	4.33	0.71	5.44%
The system delivered an adequate amount of food.	3.00	1.58	17.57%
The system delivered food with adequate speed.	3.11	1.69	18.12%
The system accurately placed food in my mouth	4.00	0.87	7.22%
The system provides sufficient safety tools or functions to prevent hazards.	4.56	0.89	6.45%

Table 4. Table 4 : Five-point Likert type questionnaire items for 8 people with motor impairments. The last column provides the average and standard deviation of scores with 1=strongly disagree, 2=disagree, 3=neither, 4=agree, and 5=strongly agree.

Questionnaires	Score
	Avg.	Std.	RSE
I feel comfortable using my current feeding system.	2.88	1.64	20.19%
I feel independent using my current feeding system.	3.25	1.49	16.19%
I expect this meal-assistance system to increase the independence of the user.	4.14	0.90	7.68%
I expect this meal-assistance system to be satisfactory.	4.38	0.74	6.01%
I expect this meal-assistance system to be comfortable.	4.38	0.74	6.01%
I am comfortable with using technology.	4.57	0.53	4.13%
I felt comfortable using the meal-assistance system.	4.75	0.46	3.45%
I felt independent using the meal-assistance system.	4.50	1.07	8.40%
The meal-assistance system provided significant help in eating.	4.88	0.35	2.56%
The meal-assistance system successfully accomplished tasks.	4.38	0.52	4.18%
The meal-assistance system was simple and easy to use.	4.75	0.71	5.26%
I felt safe while using the meal-assistance system.	4.50	0.76	5.94%

Table 5. Table 5 : Questionnaire items based on NASA-TLX for 8 people with motor impairments. Participants chose a number between 1 and 20 for each question. 1 indicates “Very Low” for Mental, Physical, Temporal, Effort, and Frustration, and “Perfect” for Performance. 20 indicates “Very High” for Mental, Physical, Temporal, Effort, and Frustration, and “Failure” for Performance.

	Questionnaires	Score
		Avg.	Std.	RSE
Mental demand	How mentally demanding was the task?	2.88	2.03	23.55%
Physical demand	How physically demanding was the task?	4.00	5.01	41.79%
Temporal demand	How hurried or rushed was the pace of the task?	2.25	2.38	35.17%
Performance	How successful were you in accomplishing what you were asked to do?	4.50	4.34	32.17%
Effort	How hard did you have to work to accomplish your level of performance?	3.38	4.50	44.46%
Frustration	How insecure, discouraged, irritated, stressed, and annoyed were you?	5.25	5.31	33.73%

Equations

$$\Delta {\theta }^{\ast }=\underset{\Delta \theta }{\mathrm{argmin}}$$

$$\Delta {\theta }_{min}\le \Delta \theta \le \Delta {\theta }_{max}$$

$${\theta }_d=\theta +\Delta \theta ,$$

$$\Delta {\theta }^{\ast }=\underset{\Delta \theta }{\mathrm{argmin}}$$

$$\Delta {\theta }_{min}\le \Delta \theta \le \Delta {\theta }_{max}$$

$${\theta }_d=\theta +\Delta \theta .$$

$$\tau =K({\theta }_d-\theta )-D\dot{\theta }+{\hat{\tau }}_g,$$

$$s^{\ast }$$

Full text

††journal: Robotics and Autonomous Systems

1 Introduction

Activities of daily living (ADLs), such as eating, toileting, and dressing, are important for quality of life [1]. Yet for many people with disabilities, including people with upper limb impairments, such tasks prove challenging without assistance from a human caregiver. However, shortages of healthcare workers and rising healthcare costs create a pressing need for innovations that make assistance more affordable and effective.

Technology interventions can be a solution by bridging the gap between physical capability and necessary functional ability [2]. Numerous specialized assistive devices, including robots, have been developed to help people with disabilities perform ADLs on their own [3]. Each device typically provides a narrow form of assistance suitable for people with particular impairments. Alternatively, researchers have applied general-purpose mobile manipulators to a variety of applications, such as rescue, assistance, and residential service [4, 5]. The robots often have a mobile base and human-like arms (e.g., PR2 robot from Willow Garage [6] and Jaco arm with a mobile base from Fattal et al. [7]), and help people to overcome their physical or perceptual limitations via teleoperation [8]. Although mobile manipulators have the potential to provide a wide variety of assistive services [9], their complexity creates challenges, including the risk of low usability.

A representative assistive task is meal assistance, which is an essential ADL for staying healthy. People with upper-body and limb impairments often have difficulty feeding themselves. Although a number of specialized meal-assistance robots are commercially available (e.g., My Spoon [10], Bestic arm [11], and Mealtime partner [12]), these robots provide limited meal assistance. Notably, we refer to the type of assistance these robots provide as passive feeding assistance, where the robot delivers food to a predefined location outside the users’ mouth and users take the food by using their upper body and limb movement. This is due in part to the robots’ (desk-mountable) fixed bases, low degree-of-freedom (DoF) arms, and limited sensing capabilities. Instead, we use a general-purpose mobile manipulator to provide active feeding assistance that autonomously delivers food inside a user’s mouth, taking advantage of the robot’s greater physical and sensing capabilities.

In this paper, we introduce a meal-assistance system that enables a general-purpose mobile manipulator, a PR2 robot, to provide safe, easy-to-use assistance with feeding (see Fig. 1). The system provides active feeding assistance in which the PR2 uses visually-guided movements to autonomously scoop/stab food and deliver the food inside a user’s mouth. The system also provides a graphical user interface (GUI) for people with various motor impairments to easily command three independent subtasks: scooping/stabbing (food acquisition), spoon-wiping (removing excess food), and delivery (feeding). Note that we group the scooping and stabbing subtasks in terms of their similar functionality (i.e., food acquisition), but the subtasks use different tools, motions, and foods. Further, our system provides state-of-the-art safety functions that proactively monitor and prevent potential anomalous executions.

Our primary contribution is that instead of a specialized meal-assistance device, our system uses a general-purpose mobile manipulator to provide active feeding assistance that addresses considerations found in the literature: convenience, comfort, speed, and safety as well as food acquisition and delivery functions [13, 14]. For convenience, our system allows 5 utensils and 2 types of bowls to adapt the functions to the user-selected food. The system has software and hardware interfaces to allow caregivers to replace the utensil and bowl depending on the type of food. The system also allows users to access its interface from a web browser, enabling the use of a variety of devices and increasing accessibility [8]. To improve safety, we designed the system to use compliant arm motions with a low-gain controller as well as a multimodal execution monitor to detect and react to anomalous events during assistance [15, 16].

Another contribution is the evaluation of the system with two groups, able-bodied participants and participants with motor impairments, for comparing the design factors. As a step towards use by people with motor impairments, we first evaluated our system with 9 able-bodied participants. We then evaluated the system with 8 people for whom unassisted feeding was difficult due to physical disabilities. We compare the two groups of evaluation results and show the system is safe, convenient, and easy-to-use. In addition to these two laboratory studies, the first author performed a long-term self evaluation while developing the system, and we evaluated the system with Henry Evans111Henry Evans became quadriplegic and mute after a stroke in 2003. As our main collaborator, he has participated in several of our assistive robotics studies since 2010., a person with quadriplegia, who operated the system to feed himself at his home. We also discuss learned lessons and design insights toward potential meal-assistance systems for people with motor impairments.

The new and previously unpublished content in the current paper includes the following:

• •

We present a detailed description of an improved meal-assistance system with visually-guided behaviors for autonomous food acquisition and delivery, and an updated graphical user interface (GUI).

• •

We present a wholly new evaluation of the meal-assistance system with 8 people with disabilities who have difficulty feeding themselves.

• •

We present new results and analyses based on the new study with 8 people with disabilities, a new long-term self evaluation, and a previous study with 9 able-bodied participants [15].

• •

We share learned lessons and design insights for assistive robots.

The rest of this paper is organized as follows: Section 2 shows related work including the examples of assistive robots, particularly assistive feeding devices. Section 3 presents the outline of our meal-assistance system. Section 4 describes the individual components of the system. Then, Sections 5 and 6 show our experimental setup and results, respectively. Finally, we present design insights and conclusions in Sections 7 and 8, respectively.

2 Related Work

Assistive robots are a type of devices that can provide physical, mental, or social assistance to people with disabilities or seniors [17, 18]. In this section, we review assistive robots, particularly manipulators, for ADLs. We then go over meal-assistance devices including feeding robots.