Simitate: A Hybrid Imitation Learning Benchmark

TL;DR

Simitate is a comprehensive benchmarking suite with a new dataset and evaluation tools designed to advance imitation learning research in realistic, simulated environments.

Contribution

It introduces a large, realistic dataset with synchronized RGB, depth, and ground truth data, along with a benchmarking suite for standardized evaluation of imitation learning methods.

Findings

Provides a new dataset with 1938 human activity sequences

Includes integrated simulation environment with ground truth data

Offers evaluation metrics for imitation quality

Abstract

We present Simitate --- a hybrid benchmarking suite targeting the evaluation of approaches for imitation learning. A dataset containing 1938 sequences where humans perform daily activities in a realistic environment is presented. The dataset is strongly coupled with an integration into a simulator. RGB and depth streams with a resolution of 960$\mathbb{\times}$540 at 30Hz and accurate ground truth poses for the demonstrator's hand, as well as the object in 6 DOF at 120Hz are provided. Along with our dataset we provide the 3D model of the used environment, labeled object images and pre-trained models. A benchmarking suite that aims at fostering comparability and reproducibility supports the development of imitation learning approaches. Further, we propose and integrate evaluation metrics on assessing the quality of effect and trajectory of the imitation performed in simulation. Simitate is available on our project website: \url{https://agas.uni-koblenz.de/data/simitate/}.

Tables

Table 1. TABLE I: Sequence overview

	# Seq	Avg. Length in s	Total Length in m
Basic Motions
Circle	104	6.83	11.58
Rectangle	105	6.84	11.97
Heart	85	6.85	9.70
Triangle	85	6.85	9.70
Zickzack	85	6.83	9.68
Motion
Reach	79	7.97	10.49
Move	79	7.96	10.48
Push	30	9.40	4.70
Pick	79	7.97	10.49
Place	79	7.96	10.49
Pour	224	8.25	30.83
Stack	63	14.63	15.36
Wipe	31	29.06	15.01
Mix	33	14.36	7.90
Complex
Ironing	92	31.74	48.66
Clean	92	28.11	43.11
Throw	50	6.84	5.70
Cut	49	19.37	15.82
Open	40	9.37	6.24
Close	20	4.34	1.44
Sequential
Rearrange	65	19.33	20.94
Pick and Place	409	14.21	96.91
Place into	60	10.67	10.67
Bring	82	21.02	28.73

Table 2. TABLE II: Evaluation of the absolute translation error (units are in m 𝑚 m )

	Min	Max	Mean	RMSE
Circle
TIAGo KP	0.006	0.673	0.105	0.160
TIAGo GT	0.007	0.108	0.034	0.038
Sawyer KP	0.011	0.755	0.110	0.174
Sawyer GT	0.003	0.333	0.030	0.041
Rectangle
TIAGo KP	0.010	0.548	0.065	0.086
TIAGo GT	0.005	0.139	0.028	0.032
Sawyer KP	0.009	0.769	0.061	0.086
Sawyer GT	0.005	0.386	0.026	0.033
Triangle
TIAGo KP	0.015	0.382	0.068	0.094
TIAGo GT	0.007	0.112	0.024	0.034
Sawyer KP	0.014	0.400	0.078	0.106
Sawyer GT	0.007	0.114	0.025	0.036
Heart
TIAGo KP	0.011	0.362	0.054	0.073
TIAGo GT	0.007	0.083	0.027	0.033
Sawyer KP	0.010	0.701	0.057	0.085
Sawyer GT	0.005	0.184	0.030	0.037
ZickZack
TIAGo KP	0.024	0.213	0.072	0.081
TIAGo GT	0.001	0.098	0.036	0.043
Sawyer KP	0.022	0.214	0.070	0.079
Sawyer GT	0.002	0.108	0.035	0.043

Equations

$$RPE:=\sum _{j=1}^n(p_e\ominus p_{g,j}{)}^2,$$

$$APE:=p_e\ominus p_g.$$

$$RMSE(RPE{}_{1:m}):=\sqrt{\frac{1}{m}\sum _{j=1}^m\Vert RPE{}_j{\Vert }^2}.$$

Full text

\captionof

figureOverview: This figure gives an overview of our benchmarking model. We provide a dataset recorded with a RGB-D camera and a motion capturing system. The sequences of the dataset are supposed to be interpreted by approaches for imitation learning which then have to execute the imitation in a simulated environment based on initial object positions of the provided ground truth. After the performance in simulation, results are automatically evaluated.

I Introduction

The application of robots in domestic environments is foreseeable and we believe that with the future spread of robots the demand for custom service robot tasks and therefore expert programmers will increase dramatically. We therefore publish a dataset that fosters imitation learning approaches just by visual observation of humans interacting with their environment. This supports the demonstrator when interacting in a natural way with its environment (let it be objects or humans). This idea stands in high contrast to current approaches that pull demonstrators out of their natural interaction by putting sensor suites or use kinesthetic teaching of robots. The Programming by Demonstration paradigm is most famous for various applications in industrial repetitive task programming.

Motivated by the increasing success of deep neural networks that recently opened up possibilities for reasonable accurate object recognition [1, 2, 3], detection [4, 5], semantic segmentation [6, 7, 8] and human pose estimations [9, 10, 11], we believe in advancing these fundamental approaches to actual scene understanding and even replication with a mobile domestic service robot.

As of now, imitation learning approaches are often empirically evaluated and show qualitative results that are commonly demonstrated on a small set of actions. There is no common dataset available that allows for comparison of approaches on a standardized dataset as is the case for many other topics like image classification [2], object detection [12], object tracking [13] or position estimation [14, 15]. This might be caused by the complexity of the evaluation process for the imitation learning task. We try to tackle this problem in this work and aim at providing a dataset and benchmarking combination that supports the development and evaluation of imitation learning tasks. To the best of our knowledge, there are no commonly used metrics for evaluation of imitation learning tasks. The importance of such a metric has been highlighted already in 2009 [16] and again in 2018 [17]. We found robotic imitation learning approaches that use custom collected data for experiments, but this data has not been published for general access. This makes reproduction and comparison harder or even impossible.

In high contrast to other currently available datasets we do not only focus on the recognition of actions, but also on a deeper understanding of the interaction between humans and objects. Even though we also recorded ground truth positions of the demonstrator’s hand and the interacting objects, the goal of the benchmark is to advance in markerless visual imitation learning approaches.

Simitate will be applicable for approaches in different fields like imitation learning through reinforcement learning [18], genetic programming [19] or generative adversarial networks [20]. Beside imitation learning, the dataset can be used for action recognition or object tracking, but does not primarily target this fields.

The main contributions of this paper are:

a novel publicly available dataset containing different individuals performing daily activity tasks 2. 2.

a novel benchmarking component which enables researchers to compare their results in a simulated environment 3. 3.

metrics for evaluation based on the imitated trajectory and the resulting effect are proposed.

The paper is structured as follows. First we introduce related work in terms of datasets as well as approaches in this field in Section II. In Section III, we then describe the dataset we provide as well as the setup and sensors we used during recording. We extend the dataset by providing a simulative benchmark and suggest metrics in Section IV. Finally, we conclude this paper in Section V.

II Related Work

Most approaches use custom datasets and methods for evaluation, making direct comparisons vague. Ross et al. [21] presented a supervised approach for imitation learning by dataset aggregation, called DAgger. Expert policies which gather a dataset of trajectories are used to train a second policy that aims at mimicking the trajectories well. Afterwards, more policies from expert demonstrations are used again to mimic the demonstrations but now the trained policies are added to the dataset. The next policy is then defined as the policy that best mimics the expert on the whole dataset. Laskey et al. [22] proposed an off-policy approach which injects noise into the demonstrator’s policy. By this the demonstrator is forced to correct the injected noise and a recovery behaviour from errors can be trained. In comparison to DAgger [21] they claim the approach to be faster and more robust. The data from the physical experiments on a real robot is not available. Ho et al. [23] presented an approach for extracting policies directly from data by a model-free imitation learning algorithm. Their approach has been proven to show same results as inverse reinforcement learning problems. One shot imitation learning approaches [24, 25, 26] have recently gained popularity. Further virtual reality approaches have been used for learning new activities by demonstration [27, 28]. A promising crowd sourcing approach of human-robot interactions was proposed by Mizuchi et al. [29]. This potentially could enable learning robot activities by demonstrations through virtual reality. All virtual reality approaches lack the direct transferability to real world robots because of the usage of simulated sensor data. We try to tackle this bottleneck in this paper. Comparable datasets mostly target action recognition or classification approaches. Weinzaepfel et al. [30] presented DALY, a dataset containing ten daily activity classes found in 500 youtube videos with a total duration of 31 hours. Pirsiavash et al. [31] created a first person dataset containing images from people fulfilling daily activity tasks. Most datasets focus on action recognition, a comprehensive survey is given by Zhang et al. [32]. Many published datasets focusing on imitation learning target autonomous driving [33] [34]. Gupta et al. presented a dataset [35] based on a subset of the COCO [12] dataset targeting semantic role labeling by verbs describing people interacting with objects. The dataset that comes closest to our proposed dataset is the CAD-120 by Koppula et al. [36] which contains 120 different RGB-D camera sequences where four individuals perform activities like making cereal, microwaving food and more. In addition, the dataset contains skeleton data provided by a skeleton tracker and manually annotated object tracks.

Benchmarking nowadays enables quantitative evaluation in many research topics like autonomous driving [14], RGB-D SLAM systems [15], object tracking [13, 37, 38]. Those benchmarks build a comfortable environment for evaluation as most commonly standard formats, evaluation metrics and scripts are specified for result comparison. Most of them even collect produced evaluation results online [14, 38] in a leader board. Some of the later benchmarks also integrate the replication by actual robotic systems i.e. for grasping [39]. Virtual reality environments have been previously used [40, 41] for evaluation of human robot interfaces. In form of competitions like RoboCup@Home [42] robotic systems are benchmarked in domestic environments, however, due to the biannual changes of the rules and not fully objective opinions of referees the comparison should be seen critical. Further, the focus is set on a time constrained one shot evaluation in most tasks. In contrast, the European Robotics League [43] puts a focus on benchmarking and uses explicit metrics. However, long term benchmarking and the limited amount of participating teams still makes long term comparability hard. Some metrics have been proposed mainly for the correspondence problem of imitation learning tasks [44]. A promising approach is to measure the effect based on [45] where demonstrated and imitated effects are compared by their displacements in relation to other objects. Most common for the evaluation of imitation learning tasks are qualitative observations [21, 22]. This is a major deficit in comparison to other well established fields.

III Dataset

In this section we describe the setup for the dataset acquisition, including the applied testbed and motion capturing system setup. Further, the dataset’s resulting sequences are introduced.

III-A Setup

To record the dataset we used a Kinect 2 RGB-D camera mounted on a tripod. Data was acquired in an exemplar apartment modelling common real world apartments, including different furniture and rooms. 12 ”OptiTrack PRIME 13” cameras were mounted on the ceiling. In total an area of 50$m^{2}$ is covered by the system. The optical center of the RGB-D camera is calibrated against the motion capturing system. Rigid body markers are attached to all relevant interacting objects and the human demonstrator. The demonstrator is completely visible during recording, except when he is occluded by objects or furniture that he is interacting with. The individual sequences were recorded at a number of different locations in the apartment. For inspection purposes we also recorded a camera stream giving an overview of the apartment.

III-B Calibration

The motion capturing system has been carefully calibrated before recording the sequences using OptiTrack Motive motion capturing software with a CW-500 marker. A common origin has been estimated using a CW-200 marker in a fixed point of the apartment. For the motion capturing calibration we achieved the following results. The mean overall wand error was $0.136mm$. For re-projection we got a mean 3D error of $0.523mm$, and a mean 2D error of $0.099$ pixels. The worst mean re-projection 3D error was at $0.642mm$ and the worst mean 2D error was at $0.143$ pixels. The RGB-D camera has been calibrated intrinsically and extrinsically. For calibration of the RGB-D camera against the Motion Capturing System we followed Sturm et al. [15] ideas. Reflective markers were attached at the corners of a checkerboard pattern. The centroid of the checkerboard was estimated using the Motion Capturing System and the central checkerboard pixel for corresponding image coordinates. It was ensured that the printed pattern was completely planar. We estimated the reflective marker height using the CW-200 marker and updated the centroid to be on the same planar surface as the printed pattern. Fig. 2 shows reprojected marker of the checkerboard center. The transformation between the centroid of the RGB-D camera’s rigid body and optical center of the RGB-D camera are then estimated [46]. In order not to interfere with the calibration result by motion we mounted the RGB-D camera and the checkerboard on tripods. The inverse transformation is used between the Motion Capturing System pose of the RGB-D camera and its optical center. A precise calibration is especially important for the alignment of real world data and later imitation in simulation. Too high residuals will lead to an inaccurate alignment between simulation and read world observation and can affect the imitation performance.

III-C Testbed

The testbed ISRoboNet@Home111http://welcome.isr.tecnico.ulisboa.pt/isrobonet/ has been set up for the European Robotics League to support the benchmarking of service robots. It aims at imitating a domestic environment separated in different rooms, including standard furniture and objects. The Motion Capturing System system described above is integrated in the testbed and allows to record ground truth data of interacting humans, robots, as well as objects. Besides the installed Motion Capturing System this testbed has the following benefits: its initial state can be recovered, it is similar to real apartments and it is open for use by research groups. This benefits also allow everyone to extend the set of recorded sequences.

III-D Human-Object interactions

We used common affordable home accessories that we got from a worldwide serving furniture retailer. The used objects are depicted in Fig. 3. For reproduction we also provide a list of objects, including their labeled training images and pre-trained models for two widely spread recent approaches [4, 6]. The images have been labeled with support of a recent guided image segmentation approach [47]. The provided data allows to easily reproduce the results and deminished the hurdles to develop approaches for this benchmark. We tried to get colorful objects too, as the focus of the presented benchmark should not be on object recognition, but on the imitation learning aspect.

We mount rigid body markers at the back of the right hand of the demonstrator. An exemplary setup for the human is shown in Fig. 1 $(a)$. We ensured that human pose estimates using a recent key-point detector are not interfered with by the marker setup. We provide human body keypoints extracted with OpenPose[11] and projected using the depth channel into world coordinates as well.

III-E Sequences

We recorded sequences for multiple purposes. First, we want to ensure that different categories of imitation learning can use this dataset. Therefore, we recorded sequences that aim at the interpretation of the demonstrations on a symbolic as well as on a trajectory level.

Sequences on a trajectory level are further divided into cloning tasks, where the human performs a movement and the goal is to mimic the movement. More challenging sequences contain object interactions. All sequences are performed by different individuals. We provide sequences that cover not only local demonstrations, but also movements between different places in the apartment of Fig. 1 $(d)$. For tasks like opening a door, we ensured to handle multiple doors of the apartment. We divide the sequences based on their level of difficulty. Basic Motion sequences contain drawn figures with the right hand. Its intention are to clone the observed movement. They also serve as testing sequences for the hand position estimation. Motion sequences contain activities like reaching for an object with the hand, picking, placing, moving or pushing it. More complex activities contain tasks that are categorized as Complex sequences. Sequential scenes contain multiple basic motions in various random combinations over a longer period of time. The complete list of sequences is given in Table I.

IV Benchmark

We propose a combined approach of real world observations and simulated environment for benchmarking imitation learning approaches. The initial object locations and positions of the observing sensors are propagated into a carefully reconstructed simulation of the testbed. This approach has multiple benefits: First, this enables evaluation methods for imitation learning and extends currently available datasets that focus on action recognition. Further, it supports generalization as the imitated behavior could be benchmarked with a wider variety of simulated robots and simplifies the transfer to real world robots. Furthermore, it enables generalization to verify the imitated behavior with a variety of objects and locations. Exemplary, we provide integration into two widely used simulations [48, 49] in the robotics and machine learning community. The benchmark in combination with the provided dataset therefore allows the evaluation of action recognition and task imitation on semantic and trajectory level. As action recognition is already addressed by many other datasets, we focus on the imitation aspect in the benchmark description.

To reduce the complexity in application of this benchmarking approach and to foster the development of imitation learning approaches we provide labeled training data for object segmentation and object detection as well as pre-trained models for current state of the art approaches [6, 4]. The benchmark is supposed to be executed sequentially. First, the individual sequences are played back. This sequence has to be analyzed by an approach either on semantic or trajectory level. After the analysis, the task is reproduce the observed actions. Generalization is evaluated by replication of the same tasks using different initial setups, but common actions on previously unseen sequences. In the observation step, sequences from the dataset will be analyzed and relevant information for the recognized action, interacting objects and arm trajectories should be extracted. We provide a class that simplifies this for later evaluation. The ground truth information from the sequence is used to initially setup the virtual representation of the testbed in simulation. A simulated robot should then execute the observed action. This allows evaluation of the achieved effect and trajectory error measurements.

IV-1 Effect

Using the effect has been proposed by Allisanrakis et al. [44]. We integrate effect evaluation for relative and absolute effects after performance of the imitation. Evaluating the relative object pose seems to be appropriate when objects are placed very close to each other. In this case we can measure the relative pose error $RPE\,$ between the final object pose $p_{e}$ and the relative ground truth poses between the object and the $j$th of $n$ surrounding objects $p_{g,j}$ like:

[TABLE]

where $\ominus$ is the inverse motion compensation operator [50] that can be imagined as the relative 3D transformation between two poses. This metric is inspired by suggestions for the accuracy of SLAM systems [50, 15]. The success of the imitation is evaluated based on post conditions that are modeled by the end state of the ground truth. In other cases it will be more relevant to aim for an effect in the humans coordinate frame. For that we use the absolute pose error:

[TABLE]

In the proposed benchmark we provide scripts for automatic evaluation of both metrics and weight their interest depending on the performed action. For many common everyday objects like bowls the rotation around their z axis is irrelevant because their symmetry is not distinguishable, even for humans . In this case we skip the angular component in the error calculation. This metric is used for motion sequences. Additionally, it could be applied on other sequences as well, but this is not primarily targeted by this benchmark.

IV-2 Trajectory Error

The other metric that we propose is based on the relative trajectory error between the robot’s end-effector and the interacting object over the period ($1:m$) of imitation. This results in a similar metric as proposed in [15] for visual odometry using the root mean square error ($RMSE$):

[TABLE]

To proof the validity of the proposed trajectory metrics and the benchmarking model, we implemented a simple approach for imitating human motions based on visual observation. Such a scenario is visualized in Fig. 5 $(a)$. For showing the validity of the effect metric we took exemplary sequences and compared them against other demonstrated sequences involving the same set of objects.

For the basic motion sequences we evaluated the absolute trajectory error of the imitation. We use a keypoint detector for human pose estimation [11] to estimate the hand positions in every frame of the sequence. The position of the right hand is projected in 3D space by using the depth channel of the corresponding pixel. The APE   of the first sequence of each set are with two robots shown in Table II. This table shows that the imitated hand poses with the robot’s end-effector are reasonably accurate but subject for further improvement. We show the applied metric for the approached estimated hand keypoints (KP) and also in contrast what could be potentially be reachable with the proposed same initial setup by the robot with the groundtruth hand position (GT). The keypoint results are heavily influenced by outliers that occurred through projection errors of the corresponding 2D estimation to the corresponding depth value i.e. in cases where no depth could be estimated.

We also verified the validity of the effect evaluation using the RPE for the imitation of a place sequence. The robots are placed in front of the table in a similar position as the RGB-D camera was placed. The goal is to replicate the final state of the scene. For simplicity we attach the moved object to the end-effector position and computed the inverse kinematics to the goal location in order to compute the RPE . For the TIAGo robot we got an average distance error of $0.047\,m$ and a rotational error of $0.013\,rad$ for the active object. The source code to reproduce the results is provided on the project page.

V Conclusion

We proposed a novel benchmark for imitation learning tasks. A dataset recorded with a RGB-D camera calibrated against a motion capturing system is coupled with a simulated representation of the environment. Metrics for evaluation are proposed. The goals of this benchmark are to foster comparability, reproducibility and the development of approaches for imitation learning tasks with a slight focus on visual imitation learning approaches. The dataset does not just contain toy examples (like reaching or moving objects), but also more complex challenges to solve, for example ironing cloths and sequences for imitation on a trajectory level without object interactions. Simitate aims at keeping the entrance barrier low by providing a complete suite with datasets, pre-trained models, integration into widely spread simulations and simple visual baseline approaches as a starting point. It can be extended by adding new tasks using an accessible testbed. The effect metric will come to a limit on imitation learning tasks with soft-bodies like bed sheets or more liquids whereas for the trajectory metric one could argue that the same effects will be reached with the same motions, when not influenced externally.

Acknowledgement

We want to thank the Institute for Systems and Robotics at the Instituto Superior Técnico, U. Lisboa, Portugal for enabling us to use the certified testbed and supporting us in the use of the motion capturing system.