Trends, Challenges and Adopted Strategies in RoboCup@Home (2019 version)

TL;DR

This paper reviews current trends, challenges, and strategies in RoboCup@Home, providing insights and milestones to guide future research and benchmarking efforts in service robotics competitions.

Contribution

It offers a comprehensive summary of recent approaches, discusses key achievements and challenges, and proposes milestones to steer future developments in RoboCup@Home.

Findings

Identification of current trending solutions in RoboCup@Home

Discussion of key challenges and limitations faced by robots

Proposal of milestones for future competitions

Abstract

Scientific competitions are crucial in the field of service robotics. They foster knowledge exchange and benchmarking, allowing teams to test their research in unstandardized scenarios. In this paper, we summarize the trending solutions and approaches used in RoboCup@Home. Further on, we discuss the attained achievements and challenges to overcome in relation with the progress required to fulfill the long-term goal of the league. Consequently, we propose a set of milestones for upcoming competitions by considering the current capabilities of the robots and their limitations. With this work we aim at laying the foundations towards the creation of roadmaps that can help to direct efforts in testing and benchmarking in robotics competitions.

Figures

Full text

With this work we aim at laying the foundations towards the creation of roadmaps that can help to direct efforts in testing and benchmarking in robotics competitions.

††publicationid: pubid: XXX-X-XXXX-XXXX-X/XX/$NN.00 © 2019 IEEE

I Introduction

Since its foundation in 2006, the RoboCup@Home league has played an important role fostering knowledge exchange and research in service robotics. Nowadays the competition influences —and sometimes directs— the course of research in the area of domestic service robotics.

Having such impact is not a minor thing. The RoboCup@Home league must take the responsibility of planning ahead and establish milestones for the competition. This, of course, can only be done after analyzing the grounds in which the league is standing.

In response, we present a brief survey of the approaches and technical solutions reported by teams in each core functions required to accomplish a task, considering:

a) claims made in the \IfSubStrnTeam Description Papers (TDPs††margin:

TDPs **)**Team Description Papers (TDPs),

b) relevant publications,

c) rulebooks,

d) multimedia material available on-line,

e) conducted polls targeting potential customers necessities, and

f) our cumulative experience as participants and referees in RoboCup@Home since 2009.

Based on that survey we contribute by discuss the unconquered challenges that we have identified throughout these first twelve years. Considering the studied material and the robot’s current capabilities and achievements, elaborate on the next steps to take towards achieving the RoboCup@Home goal, emphasizing those relevant in the short term.

This manuscript is organized as follows: Section II, briefly introduces to RoboCup@Home and its history. Section III, provides a brief summary of adopted hardware solutions. Section IV, addresses the strategies and software solutions used by the participant teams. In Section V we discuss the performance, challenges, and next steps. Finally, in Section VI we summarize the discussion and present our final conclusions.

II RoboCup@Home

The RoboCup@Home league was created in 2006 aiming to “develop service and assistive robot technology with high relevance for future personal domestic applications” 111Source: http://www.robocupathome.org/ Retrieved: Jan 1st, 2019..

The competition takes place in a testing arena shaped as a typical apartment. In most tests, the robots solve household-related tasks, having their abilities and performance evaluated [1, 2].

During the first two years, the tests were scored with boolean criteria, making difficult to analyze the robot’s performance. Therefore, the scoring system was redesigned in 2008 and tests were split in a sequential set of goals while considering difficulty increases every third year [3, 1, 2].

A sustained performance decrease (see Figure 1) motivated the \IfSubStrnTechnical Committee (TC††margin:

TC **)**Technical Committee (TC) to benchmark individual abilities. Based on the results, in 2014 the test scheme proposed by Wisspeintner et al. [3] and analyzed by Iocchi et al. [1] was replaced by a new one focused in measuring performance [2].

This new schema, considered ability benchmarking and repetitions to tackle the luck factor. The introduced changes showed competitors their weaknesses, producing a performance increase in 2016 as Figure 1 shows, decreasing again in 2017 when difficulty was risen [2].

Finally, in 2017 the league was split in three, the \IfSubStrnOpen Platform League (OPL††margin:

OPL **)**Open Platform League (OPL) with no hardware restrictions, the \IfSubStrnDomestic Standard Platform League (DSPL††margin:

DSPL **)**Domestic Standard Platform League (DSPL) featuring the Toyota HSR, and the \IfSubStrnSocial Standard Platform League (SSPL††margin:

SSPL **)**Social Standard Platform League (SSPL) featuring the SoftBank-Robotics Pepper. The division, veteran team migration, and the difficulty increase caused one of the lowest performances in the history of RoboCup@Home [2].

III Summary of Hardware Solutions

Here we summarize the hardware configurations most used in the \IfStrEqCasenOPL††margin:

OPL fOpen Platform League (OPL)fnOpen Platform League (OPL††margin:

OPL **)**xOpen Platform LeaguexnOpen Platform League††margin:

OPL [OPL]. The aspects have been chosen for their potential influence in the robot’s performance.

RGB-D Sensor: In 2017 and 2018, all teams used at least one RGB-D sensor, preferring the Microsoft Kinect 2 due to its incorporated Time of Flight sensor and better resolution, with Asus Xtion in second place (see Figure 2(a)). 2. 2.

Base: All robots in RoboCup@Home use wheels with no signs of upcoming changes (see Figure 2(b)). The most used configuration is Differential Pair followed by Omni-Drive and with a single reported case of Swerve Drive222Swerve drive is a special type of omni-directional configuration in which all four wheels rotate independently. 3. 3.

Head: To humans, head identification is intuitive, but robots can be deceiving. A robot can be bicephalous or feature a face displaced from its sensors. Hence, we consider a robot head as a device integrated by one or more cameras and a pan-tilt unit with a microphone. This criteria was met by 67% (10 out 15) robots in Nagoya 2017 and 71% (5 out 7) in Montreal 2018, all with 2 DoF. 4. 4.

Manipulator: To handle objects teams use either home-made or proprietary low-cost hardware [4, 5] since the size of professional arms often deems them unfit for domestic narrow spaces.333In 2016 team b-it-bots experienced issues due to the size of the 7-DOF KUKA LWR manipulator mounted on their Care-O-bot 3. The arm required a considerable amount of space to unfold and its size wouldn’t make it fit in some spaces.Consequently, no professional manipulator was used in Nagoya 2017 nor in Montreal 2018. In both competitions the number of \IfSubStrxnDegrees of Freedom (DoF††margin:

DoF **)**Degrees of Freedom (DoF) for manipulators ranged from 4 to 7 Figure 2(c) depicts with an average strength in the final effector of $1.25kg$. Only one third of the robots featured two manipulators. 5. 5.

Torso: We consider a robot torso as a device that provides panning and/or elevation to the robot’s head and upper limbs. In Nagoya 2017, only 60% (9 out of 15) of competing robots had a torso, all cases featuring elevation only.

IV Adopted Strategies and Software Solutions

In this section, we summarize the solutions most commonly adopted by teams to address each of the basic functionalities involved in the tests.

IV-A Frameworks and Middlewares

Nowadays, ROS has become a tacit standard in robotics with all teams using it at least partially [2]. Nonetheless, older frameworks like Orocos [6] are also used, while some teams continue using their own solutions [7], or vendor solutions like the NaoQi in \IfStrEqCasenSSPL††margin:

SSPL fSocial Standard Platform League (SSPL)fnSocial Standard Platform League (SSPL††margin:

SSPL **)**xSocial Standard Platform LeaguexnSocial Standard Platform League††margin:

SSPL [SSPL].

IV-B Navigation

In RoboCup@Home navigation involves path planning, obstacle avoidance, localization, and mapping.

Since safe, robust indoors navigation is taken for granted, path planning and localization are not scored inside the arena. Contrastingly, obstacle avoidance is tested for small, reflecting, and hard-to-see object. Finally, on-line mapping for unknown environments is being extensively tested.

With a couple of exceptions, it can be said that all teams rely on the ROS navigation stack (see Figure 4(a)) tuned to the robot and based on each teams preferences (see Figure 4(a)). Hence, the most broadly adopted solution sums up to OpenSlam’s Gmapping and \IfSubStrnAdaptive Monte Carlo Localization (AMCL††margin:

AMCL **)**Adaptive Monte Carlo Localization (AMCL) with an $A^{*}$ path planner [6, 5, 8], with the incorporation of a Kalman filter to Gmapping as a good second [9, 10].

Path Planning and Obstacle Avoidance

Several teams just report the use of ROS, so we assume an out-of-the-box configuration. Solutions other than ROS include Randomized Path Planners [11], and Wave-Propagation algorithms based on the Fast Marching Method [12]. Regarding obstacle avoidance, most teams reported building occupancy grids based on the the \IfSubStrnLaser Rangefinder Scanner (LRFS††margin:

LRFS **)**Laser Rangefinder Scanner (LRFS) and the RGB-D camera.

Localization and Mapping

These two abilities are often reported together. Regarding localization alone, there are two reported solution differing from the robot’s built-in localization and \IfStrEqCasenAMCL††margin:

AMCL fAdaptive Monte Carlo Localization (AMCL)fnAdaptive Monte Carlo Localization (AMCL††margin:

AMCL **)**xAdaptive Monte Carlo LocalizationxnAdaptive Monte Carlo Localization††margin:

AMCL [AMCL]. These solutions are addressed by different teams and implement \IfSubStrnSimultaneous Localization and Mapping (SLAM††margin:

SLAM **)**Simultaneous Localization and Mapping (SLAM) using the \IfSubStrnIterative Closest Point (ICP††margin:

ICP **)**Iterative Closest Point (ICP) algorithm [13, 14]. Both methods aim for accuracy and speed with limited resources. On the mapping side, reported solutions other than GMapping include hector \IfStrEqCasenSLAM††margin:

SLAM fSimultaneous Localization and Mapping (SLAM)fnSimultaneous Localization and Mapping (SLAM††margin:

SLAM **)**xSimultaneous Localization and MappingxnSimultaneous Localization and Mapping††margin:

SLAM [SLAM] [15], \IfSubStrnMobile Robot Programming ToolKit (MRPT††margin:

MRPT **)**Mobile Robot Programming ToolKit (MRPT) and \IfStrEqCasenICP††margin:

ICP fIterative Closest Point (ICP)fnIterative Closest Point (ICP††margin:

ICP **)**xIterative Closest PointxnIterative Closest Point††margin:

ICP [ICP] [16], and Omnimapper [17].

IV-C People Detection & Recognition

This section relates detection and recognition of static people using visual information.

Strategies for people detection combine face and skeleton detection to reduce false positives while limiting the detection range. Further, there are hybrid techniques like combining 3D object recognition with face detection (e.g. OpenFace), or analysis of thermal images [1, 18, 6]. Face detection strategies include Openface [19], Viola-Jones algorithm [20], and Haar-based algorithms [21].

In RoboCup@Home, person recognition implies recalling the person’s name, typically involving a facial recognition. Notable solutions include \IfSubStrnHistogram of Oriented Gradients (HOG††margin:

HOG **)**Histogram of Oriented Gradients (HOG) descriptors with \IfSubStrnSupport Vector Machine (SVM††margin:

SVM **)**Support Vector Machine (SVM) classifiers [22], Strands Perception People [8, 23], Viola-Jones and Eigenfaces [20], Siamese \IfSubStrnConvolutional Neural Network (CNN††margin:

CNN **)**Convolutional Neural Network (CNN)s [24], and Haar Cascades with either EigenFaces [7] or \IfSubStrnDeep Neural Network (DNN††margin:

DNN **)**Deep Neural Network (DNN) [21]. Some approaches consider texture and color segmentation as backup. Recently, cloud services are being incorporated.

Finally, other features addressed in the competition consider height, gender, age, pose, relative position, and clothing; which are mostly tacled using \IfStrEqCasenDNN††margin:

DNN fDeep Neural Network (DNN)fnDeep Neural Network (DNN††margin:

DNN **)**xDeep Neural NetworkxnDeep Neural Network††margin:

DNN [DNN]-based libraries and cloud services [25, 26].

IV-D Object Detection and Recognition

Although closely related, object detection is times faster than recognition, allowing continuous recognition. Often, the point-cloud of the RGB-D sensor is used to remove background, floor, and other surfaces (e.g. using Vector Quantization [27] or \IfSubStrnRandom Sample Consensus (RANSAC††margin:

RANSAC **)**Random Sample Consensus (RANSAC) [7, 12]). Then, color-depth images are clustered for its analysis by the object recognizer. This approach is preferred over deep-learning-based for being less expensive.

Most teams use multiple algorithms either choosing by consensus or processing in a pipeline to add robustness to their object recognition. Often depth information is used to recognize contours and shapes [7, 17, 28]. Some solutions combine

a) \IfSubStrnYou Only Look Once (YOLO††margin:

YOLO **)**You Only Look Once (YOLO) with \IfSubStrnScale-Invariant Feature Transform (SIFT††margin:

SIFT **)**Scale-Invariant Feature Transform (SIFT) and \IfSubStrnBinary Robust Independent Elementary Features (BRIEF††margin:

BRIEF **)**Binary Robust Independent Elementary Features (BRIEF) [11],

b) \IfSubStrnSpeeded Up Robust Features (SURF††margin:

SURF **)**Speeded Up Robust Features (SURF) with Continuous Hough-space voting with \IfSubStrnImplicit Shape Models (ISM††margin:

ISM **)**Implicit Shape Models (ISM) [26],

c) color, size and shape histograms and SIFT [7],

d) \IfSubStrnPoint Cloud Library (PCL††margin:

PCL **)**Point Cloud Library (PCL) with \IfStrEqCasenRANSAC††margin:

RANSAC fRandom Sample Consensus (RANSAC)fnRandom Sample Consensus (RANSAC††margin:

RANSAC **)**xRandom Sample ConsensusxnRandom Sample Consensus††margin:

RANSAC [RANSAC] and \IfStrEqCasenYOLO††margin:

YOLO fYou Only Look Once (YOLO)fnYou Only Look Once (YOLO††margin:

YOLO **)**xYou Only Look OncexnYou Only Look Once††margin:

YOLO [YOLO] [12], and

e) contours using LINEMOD with HSV color histograms and \IfStrEqCasenSURF††margin:

SURF fSpeeded Up Robust Features (SURF)fnSpeeded Up Robust Features (SURF††margin:

SURF **)**xSpeeded Up Robust FeaturesxnSpeeded Up Robust Features††margin:

SURF [SURF] considering joint models for occlusion [28].

Although some teams run their algorithms on demand, more often processes are running all the time, although normally results are simply discarded by the task planner. Very few teams have reported mechanisms to take advantage of continuous detection and recognition.

Beyond classification, object shape and feature recognition (e.g. color, size, position, shape) is fundamental for grasping and describing previously untrained objects. Unfortunately, the strategies used are not documented in the \IfStrEqCasenTDPs††margin:

TDPs fTeam Description Papers (TDPs)fnTeam Description Papers (TDPs††margin:

TDPs **)**xTeam Description PapersxnTeam Description Papers††margin:

TDPs [TDPs].

IV-E Audio, Speech, and Natural Language Processing

The most broadly adopted solution to deal with speech consists in a pipeline in which a filtered audio signal feeds an \IfSubStrfnAutomatic Speech Recognition (ASR††margin:

ASR **)**Automatic Speech Recognition (ASR) engine to get a text-transcript for further processing. Then, the transcript is sent to a \IfSubStrnNatural Language Processing (NLP††margin:

NLP **)**Natural Language Processing (NLP) unit that extracts all relevant information that is passed to a high-level task planner that triggers the pertinent behaviors[29, 30].

Often filtering is left to the microphone and the \IfStrEqCasenASR††margin:

ASR fAutomatic Speech Recognition (ASR)fnAutomatic Speech Recognition (ASR††margin:

ASR **)**xAutomatic Speech RecognitionxnAutomatic Speech Recognition††margin:

ASR [ASR] engine [31], being HARK444HARK (Honda Research Institute Japan Audition for Robots with Kyoto University) is an open-source robot audition software that includes modules for \IfStrEqCasenASR††margin:

ASR fAutomatic Speech Recognition (ASR)fnAutomatic Speech Recognition (ASR††margin:

ASR **)**xAutomatic Speech RecognitionxnAutomatic Speech Recognition††margin:

ASR [ASR] and sound-source localization and sound separation. Source: https://www.hark.jp/a notable exception [11, 32, 33].

Regarding \IfStrEqCasenASR††margin:

ASR fAutomatic Speech Recognition (ASR)fnAutomatic Speech Recognition (ASR††margin:

ASR **)**xAutomatic Speech RecognitionxnAutomatic Speech Recognition††margin:

ASR [ASR], the most adopted off-line solutions include Julius [15], the Microsoft Speech API [7], and CMU Sphinx [34, 8, 35] as Figure 3 shows. In \IfSubStrnStandard Platform Leagues (SPL††margin:

SPL **)**Standard Platform Leagues (SPL) most teams use cloud services due to the limited computing power of standard robots. The Google speech API is the most popular approach [11, 36, 24, 23, 37]. Due to network unreliability teams often have backup offline solutions.

Moving forward to \IfStrEqCasexnNLP††margin:

NLP fNatural Language Processing (NLP)fnNatural Language Processing (NLP††margin:

NLP **)**xNatural Language ProcessingxnNatural Language Processing††margin:

NLP [NLP], teams are migrating from keyword spotting and pattern matching with state machines [9, 34, 38]. Groups focusing in \IfStrEqCasenNLP††margin:

NLP fNatural Language Processing (NLP)fnNatural Language Processing (NLP††margin:

NLP **)**xNatural Language ProcessingxnNatural Language Processing††margin:

NLP [NLP] and \IfSubStrnNatural Language Understanding (NLU††margin:

NLU **)**Natural Language Understanding (NLU) are rare, and solutions are mentioned in less than 50% of the \IfStrEqCasenTDPs††margin:

TDPs fTeam Description Papers (TDPs)fnTeam Description Papers (TDPs††margin:

TDPs **)**xTeam Description PapersxnTeam Description Papers††margin:

TDPs [TDPs]. Notable approaches include Probabilistic Semantic Parsers [39], Multimodal Residual \IfStrEqCasenDNN††margin:

DNN fDeep Neural Network (DNN)fnDeep Neural Network (DNN††margin:

DNN **)**xDeep Neural NetworkxnDeep Neural Network††margin:

DNN [DNN] [28], Ontology-based parsers with inference engines [27], and $\lambda$–calculus-based semantic parsing [40]. The Stanford Parser [41] is the most adopted solution for POS-tagging and syntactic tree extraction along with LU4R [42], a Spoken Language Understanding Chain for HRI [23, 10].

IV-F Manipulation

A broadly adopted solution considers direct-inverse kinematic models with a closed-loop control and camera feedback as an alternative to the ROS manipulation stack, with many teams migrating to MoveIt!

Manipulation was reported in only 18 (56%) of the \IfStrEqCasenTDPs††margin:

TDPs fTeam Description Papers (TDPs)fnTeam Description Papers (TDPs††margin:

TDPs **)**xTeam Description PapersxnTeam Description Papers††margin:

TDPs [TDPs], from which 61% use MoveIt! and the rest custom solutions. These include super-ellipsoid fitting [17], multiple deep auto-encoders fed with RGB-D and audio data [43], and the proprietary built-in software [23].

While two-handed manipulation is often attempted in the \IfStrEqCasenOPL††margin:

OPL fOpen Platform League (OPL)fnOpen Platform League (OPL††margin:

OPL **)**xOpen Platform LeaguexnOpen Platform League††margin:

OPL [OPL], opening doors is still problematic. In contrast, \IfStrEqCasenDSPL††margin:

DSPL fDomestic Standard Platform League (DSPL)fnDomestic Standard Platform League (DSPL††margin:

DSPL **)**xDomestic Standard Platform LeaguexnDomestic Standard Platform League††margin:

DSPL [DSPL] robots can open doors with ease555Doors with handles and lose latches as requires by the league and as shown on the HSR live demonstration for DSPL (https://youtu.be/SwIkY1ffExI?t=30) and in this other videos: https://youtu.be/2nWX7-ccP8I?t=72 and https://youtu.be/wCJ9qQrr5M0?t=46. but lack of a second arm. As of \IfStrEqCasenSSPL††margin:

SSPL fSocial Standard Platform League (SSPL)fnSocial Standard Platform League (SSPL††margin:

SSPL **)**xSocial Standard Platform LeaguexnSocial Standard Platform League††margin:

SSPL [SSPL], manipulation is avoided due to the limitations of the robot.

Remarkably, there is a growing trend for deep-learning-based methods, especially in planning. Although computationally more expensive, they can be faster and more robust than the traditional methods when a non-optimal solution is acceptable. Besides, although their supporters claim they are easier to develop, the required data acquisition for training is their biggest weakness [44].

V Discussion on Intrinsic Challenges

The challenges described in this section correspond to the role of the RoboCup@Home in directing research in robotics. Here, we discuss the most relevant abilities.

V-A Navigation

Although closely related, indoor and outdoor navigation are approached with disjoint ability sets.

Indoor-Navigation

Despite the continuous improvements, steps, wet floors, rough carpets, and in general uneven surfaces present restrictions that haven’t been addressed yet. Furthermore, potential customers prefer silent carpet-friendly robots, specially when considering night cleaning.

Also troublesome are the amount of geometrical data required and the sensor’s perception range cap of 4m that might impede localization in wide spaces (e.g. big houses and facilities). Hence semantic localization must be incorporated, specially to move around regular buildings. We found reading signs and labels is also an important feature in offices and hospitals.

Finally important abilities discontinued or not addressed yet include using an elevator, navigate in narrow spaces, move furniture around, and functional touching (push aside objects with one’s body). People often rely on their body to move or stop objects when moving around.

Outdoor-Navigation

Even in urban environments, exteriors add an extra tier of complexity. Deal with rough terrain (e.g. grass, gravel, and sand), slopes, far distances, sunlight, and weather conditions is necessary to meet customers expectations. For instance, pet owners are interested in robots capable of walking and exercise the animals.

As distances and the number of stimuli to process grows, so does the computational power required. Robots need to handle occlusions and and react to fast-moving objects in real time with higher precision than autonomous cars, while at the same time consider signs, read street and shop names, and prevent collisions with children, elders, dogs, other robots, etc. Furthermore, the robot needs to synchronize with streetlights, automatic doors, and public transport.

Conclusions

Although RoboCup@Home nowadays features only wheeled-robots, this design is not suitable for most human environments. Consequently either environments must become robot-friendly, or robots will need other mechanisms (i.e. legs).

Finally, robots need to be programmed to recognize and map semantic information in their surroundings like people does, correlating perception with previous knowledge.

V-B People Detection & Recognition

Command retrieval, object delivery, and \IfSubStrnHuman-Robot Interaction (HRI††margin:

HRI **)**Human-Robot Interaction (HRI) make people detection a must-have function. As of 2018, robots must detect still people either standing, sitting or lying; exhibiting robust, faulty, and bad performance on each pose respectively.

In contrast, people recognition is barely addressed. In this analysis we split the feature in two aspects, people recalling and identification. Recalling usually consists in pairing the person’s features (typically the facial ones) with other information, such as a name or an order. Key problems strive in the reduced number of people (less than five) and the briefness of the data lifespan (information is disposed after a test).

Relatively new, people identification relates to finding a person matching a description, or describing individuals or groups of people. In 2018, provided descriptions weren’t accurate and description-based people search was tested at random. In consequence, it is impossible to give precise information regarding general performance. Nonetheless, it must be acknowledged that most robots perform good at counting groups of people and identifying their gender.

Next Steps

We believe the league should emphasize people recognition. Features like estimated age, gender, and relative position can be used to test awareness, decision making and planning, as well as integrating voice recognition (recalling). Furthermore, as with objects, detection ranges need to be increased, as well as the amount of people to remember.

We also consider important to introduce the detection of features like: emotions, moods, activities, attire styles, clothing elements, healthiness, and vital signs (inebriation, fatigue, sickness, sleep, etc). Also important, and shared with object recognition is addressing occlusions and translucent surfaces.

Conclusion

It’s clear that tests require to be redesigned to ensure benchmarking data is available for analysis. In addition, we find that visual and audio information are equally important when addressing people.

V-C Object Detection and Recognition

As of 2018, it is safe to state that robots are shortsighted. Despite the remarkable advances, detection is often constrained to 4 meters due to RGB-D sensor resolution and reach, causing robots to fail at seeing objects lying in direct line of sight. Further objects are always placed sparsely, being partially occluded at worst, and a robot must know about 25 of them, a situation that greatly differs from most people’s homes.

Remarkably, object detection is being used to aid in navigation by updating the grid of an occupancy map. In addition, robots are starting to successfully identify of objects of a kind (e.g. apples, regardless the color, size etc).

Notably, all reported strategies feed-forward process the input stream frame-by-frame, discarding used visual information.

Next Steps

Brain inspired models from neurosciences suggest the use of contextual information to build scene representations, so processes can rely on memory instead of visual search [45, 46], leading to a faster response. In addition, processed visual information could be used for reinforcement learning and update the world model instead of being discarded.

Other problems to address include dust, spots, and dirt detection, recognize stacked objects, position-based target selection among equals, increase the number of objects, and recognize objects in odd lightning conditions like direct sunlight and in the dark. Similarly, translucent and transparent objects present new challenges when it comes to occlusions and detection of spots.

Likewise, and borderline with manipulation, relevant problems include orientation identification, best placement location identification, and weight inference.

Conclusions

Like with other abilities, a roadmap is necessary to direct advances on object recognition. The first logical steps seems to be enhance the aspects that help with manipulation and navigation like placing surfaces, grasping orientation, and obstacle identification; as well as occlusions.

Finally, it is necessary to start integrating object recognition with high-level action planning so semantic, spatial and temporal relationships can be also recognized. A far-placed milestone would include objects that change over time (e.g. food).

V-D Manipulation

Manipulation is the most mature research area in robotics, nonetheless RoboCup@Home is far from reaching the feats of industrial robots due to the lack of a fixed inertial base and a concealed working space.

As of 2018 most manipulation tasks are special cases of fetch and pick-and-place, although robots in \IfStrEqCasenDSPL††margin:

DSPL fDomestic Standard Platform League (DSPL)fnDomestic Standard Platform League (DSPL††margin:

DSPL **)**xDomestic Standard Platform LeaguexnDomestic Standard Platform League††margin:

DSPL [DSPL] and \IfStrEqCasenOPL††margin:

OPL fOpen Platform League (OPL)fnOpen Platform League (OPL††margin:

OPL **)**xOpen Platform LeaguexnOpen Platform League††margin:

OPL [OPL] are required to open the door of a cupboard, a subtask that was skipped by all \IfStrEqCasenOPL††margin:

OPL fOpen Platform League (OPL)fnOpen Platform League (OPL††margin:

OPL **)**xOpen Platform LeaguexnOpen Platform League††margin:

OPL [OPL] participants in 2017 and by most of them in 2018 [2]. Also skipped were pouring, spot scrubbing, and tray transport [2]. Nonetheless, although skipped in 2017, several teams did it remarkably well at handling cutlery in 2018 [2]. Furthermore the competition considers mostly moving regular-shaped, small (5–25cm, 2lt max volume), non-fragile lightweight objects (75–950gr) like apples, small cereal boxes, and [empty] soda cans.

Market Requirements

Our polls reveal people are explicitly asking for robots that can clean the toilet, wipe windows, do the dishes (by hand), wash, iron and fold clothes, open flasks and jars, brush and wash the dog, and take out the garbage to name some examples.

Other important skills required in daily use include opening doors, move furniture, and operate the controls of electrical appliances. From these, opening doors has been addressed since 2006, but not really solved until team eR@sers impressed the league in 2016 with the proprietary robot now used in the \IfStrEqCasexnDSPL††margin:

DSPL fDomestic Standard Platform League (DSPL)fnDomestic Standard Platform League (DSPL††margin:

DSPL **)**xDomestic Standard Platform LeaguexnDomestic Standard Platform League††margin:

DSPL [DSPL]. Notwithstanding, it seems there is still a long way to go before this skill can be considered solved in \IfStrEqCasenOPL††margin:

OPL fOpen Platform League (OPL)fnOpen Platform League (OPL††margin:

OPL **)**xOpen Platform LeaguexnOpen Platform League††margin:

OPL [OPL] and \IfStrEqCasenSSPL††margin:

SSPL fSocial Standard Platform League (SSPL)fnSocial Standard Platform League (SSPL††margin:

SSPL **)**xSocial Standard Platform LeaguexnSocial Standard Platform League††margin:

SSPL [SSPL].

Next Steps

The manipulation capabilities of the robots must be expanded in several directions follows:

• •

Reach: As of 2018, objects are placed between $30cm$ and $1.8m$ height and no further than $5cm$ from the border. Next steps involve reaching the floor and up to $50cm$ in depth.

• •

Maneuverability: Nowadays robot’s movement is restricted to flat surfaces and hand-overs. Hence picking from/placing into boxes and bags is required. At the same time, twisting, uncapping, shaking, folding, levering, and turning are features yet to be tested in RoboCup@Home.

• •

Strength: Terminal-effector’s load must be gradually increased to reach at least $3kg$ when fully extended, and beyond. Heavy loads are required by elders for grocery transport and help when stand up.

• •

Precision: Torque, speed, trajectory, and acceleration control are required to spread butter, whip cream, and grasp pills. Besides, some people requested application of insulin injections.

Finally, other important tasks include manipulating a switch, taking out garbage, serve soup, mopping, unpacking, and passing towels.

Conclusion

The RoboCup@Home league would enormously benefit from a manipulation roadmap with which teams could plan hardware improvements with sufficient time. For now, gradually increasing weight and placing distance would suffice.

V-E Speech and Natural-Language Interaction

A key aspect to people’s communication, speech in RoboCup@Home is mostly restricted to issuing commands to robots [2, 29]. These commands are generated using grammars given in advance to teams under noise conditions which are far from those found in domestic environments [2, 29]. Further, experts take part in all interactions, meaning that the way to address a robot is not entirely natural [2, 29].

An thorough study, available in [29], was conducted addressing these troubling aspects in spoken \IfStrEqCasenHRI††margin:

HRI fHuman-Robot Interaction (HRI)fnHuman-Robot Interaction (HRI††margin:

HRI **)**xHuman-Robot InteractionxnHuman-Robot Interaction††margin:

HRI [HRI], followed by a roadmap and a set of strategies to tackle them and foster dialog-based natural language interactions in RoboCup@Home.

It’s worth mention, nonetheless, that so far available \IfStrEqCasenASR††margin:

ASR fAutomatic Speech Recognition (ASR)fnAutomatic Speech Recognition (ASR††margin:

ASR **)**xAutomatic Speech RecognitionxnAutomatic Speech Recognition††margin:

ASR [ASR] engines are an important limiting factor in natural language interaction.

V-F Roadmaps and milestones

We believe the RoboCup@Home must take the responsibility of leading and guide research in domestic service robotics. This implies carefully planning milestones and scheduling tests for the competition. Said in other works, the league should have roadmaps to direct research and help during test design.

To teams, knowing the challenges in advance will help them to prepare direct their research, and invest better their resources. As for \IfStrEqCasenTC††margin:

TC fTechnical Committee (TC)fnTechnical Committee (TC††margin:

TC **)**xTechnical CommitteexnTechnical Committee††margin:

TC [TC] an official roadmap can prevent newly elected members from overriding previous decisions, damping the league’s progress.

In consequence, appearing valuable ideas need to be analyzed, evaluated against the robot’s capabilities, and condensed in milestones and test drafts to be retaken in latter years.

As response, in this paper we have condensed teams’ ideas and concerns, the user’s needs, and the current capabilities and limitations of the robots; presenting them in Sections V-B, V-C, V-A and V-D as future steps or milestones . Nonetheless, this is just a first step, since roadmaps need to be still prepared and adopted by the league which, in the end, is peer-maintained.

VI Discussion and Conclusions

In this paper we conduct a thorough summary of the software solutions and strategies used by participating teams in RoboCup@Home to address some of the most important abilities required in the competition. Further on, we present an overview of the attained achievements since the league’s foundation based on our experience as long-time participants, contributors, and referees in the league. Finally, also organized per skill and along with the overview, we discuss these achievements while addressing what is expected by potential consumers, what needs to be done, and what would be the next logical steps based on the robot’s current capabilities.

This study result in two main contributions. First, we believe the presented summary can serve as quick reference guide for new competitors, or for experienced ones looking for alternatives to their current solutions. Second, our work sets the basis to build roadmaps to plan the direction and impact of robotics competitions like RoboCup@Home. Moreover, roadmaps can direct scientific research and help teams to prepare years in advance by establishing mid- and log-term goals, and perhaps a smarter resource management.

However, there is still work to be done. Not only the roadmaps have to be designed. This work has also allowed us to identify several important flaws that need to be addressed. For instance, the presence of certain rules might be undermining the development of certain features. At the same time, we have found that many successful approaches and strategies are never reported in the \IfStrEqCasenTDPs††margin:

TDPs fTeam Description Papers (TDPs)fnTeam Description Papers (TDPs††margin:

TDPs **)**xTeam Description PapersxnTeam Description Papers††margin:

TDPs [TDPs]. This has two important setbacks. First, it makes it much harder for the scientific community to compare the performance of the different approaches when the best performers are missing. Second, it leads to an eventual loss of knowledge that worsens as the lifetime of a good team shortens.

These insights are left to the competition organizers to analyze as part of future work, for which we trust this manuscript can come handy.