Sequential Human Assembly and Disassembly Motions in Human-Robot Coexisting Environments

TL;DR

This paper introduces a new dataset capturing human assembly and disassembly motions in shared human-robot environments to improve robotic task understanding and collaboration.

Contribution

The novel dataset includes over 10,000 samples with synchronized RGB and skeleton data, addressing challenges like occlusions and repetitive motions.

Findings

The dataset includes diverse human motions with partial occlusions and repetitive actions.

Benchmarking with deep learning models shows the dataset's potential for practical applications.

Annotations and Python codes are provided for reproducibility and research flexibility.

Abstract

As human-robot systems and autonomous robots become increasingly prevalent, the need for task-oriented datasets to study human behaviors in shared spaces has grown significantly. We present a novel dataset focusing on sequential human assembly and disassembly motions in human-robot coexisting environments. It contains over 10,000 samples recorded from multi-view camera setups, each comprising synchronized RGB videos and 2D and 3D human skeletons. Data were collected from 33 participants with diverse physical characteristics and behavior preferences. This dataset highlights practical challenges such as partial occlusions, similar repetitive motions, and varying human behaviors, which are often overlooked in existing datasets and research. Technical validation using benchmarking with state-of-the-art deep learning models reveals significant potential in using this dataset for practical applications. To support diverse research applications, this dataset provides raw and processed data with detailed annotations, including precise timestamps, procedure annotations, and Python codes for reproducibility. It aims to advance research in human motion prediction, task-oriented robotic sequential decision-making, motion and task planning of autonomous robots, and human-robot collaborative policies.

Figures

Scenarios for data collection.

Height and weight distribution of all the participants.

Illustration of the protocol.

Full text

Background & Summary

A human-robot system (HRS) is a novel paradigm that comprises humans and robots working together as a unified workforce^1^. This paradigm is reflected in key research areas such as human-robot interaction (HRI) and human-robot collaboration (HRC)^2^. An HRS typically consists of fundamental modules, including perception, cognition, decision-making, and control. Among these modules, perception plays a central role. Moreover, the ability of robots to perceive task-oriented sequential manual procedures performed by humans distinguishes perception in HRS from conventional robotic perception.

Typical modalities for capturing human behaviors include RGB images, RGB-D images, videos, point clouds, and human skeletons. These modalities are often employed to track and recognize human activities. By leveraging the real-time states of human operators, task and motion planning can be implemented to enhance the collaborative capabilities of robots, thereby enabling efficient HRS, especially task-oriented ones. However, human motions in HRS, especially HRC, are predominantly task-related, particularly in collaborative assembly and disassembly.

Related datasets exist in both HRI and HRC. In HRI, MHRI^3^ is a multi-modal dataset containing human motion pointing and showing objects to a robot. While THOR^4^ is a motion trajectory dataset for human-robot navigation. Similar datasets in HRI also show up in other robotic-related fields such as hugging interaction^5^, personality and engagement^6^, games^7^, and assistive collaboration^8^. In HRC, the analysis of human motion is often industrial task-oriented, encompassing aspects such as body language and the task sequences involved in manipulation tasks. For instance, InHARD^9^ utilizes webcams and wearable motion capture devices in an HRC scenario where humans and robots operate side by side. HRI30^10^ collects data from an industrial environment in which a human collaborates with two robots, capturing human motions such as “pick up drill” and “move forward while drilling” from a wide camera view. Assembly101^11^ is a large-scale, multi-view dataset designed for sequential manual assembly tasks, primarily developed for augmented reality applications without the inclusion of physical robots, thereby focusing predominantly on forearm and hand movements above a table. ATTACH^12^ emphasizes two-hand assembly actions performed by humans, involving 42 participants in a cabinet assembly task, with data collected using three cameras. HA-ViD^13^ highlights the knowledge in assembly from 30 participants, with video data collected from three cameras.

These datasets released in recent years have significantly contributed to HRS research. However, they still present limitations in supporting HRS for industrial tasks. Firstly, common human motion datasets, such as Kinetics^14^ and NTU RGB+D^15^, do not account for task-oriented human behaviors, especially task sequences. They typically focus on simple daily life activities, such as crossing arms or raising hands. As for HRS-related datasets, InHARD relies on wearable devices, which limits its usage because of higher hardware requirements. HRI30 focuses more on humans walking between robots while holding tools, which is less task-oriented. The advantages of Assembly101 include its large scale and multiple camera views; however, it is not related to robotic environments. ATTACH concentrates on specific manual motions but does not address task sequences, and it lacks human-robot interference such as occlusions. HA-ViD stresses that the data annotation should be shared between humans and robots, yet it lacks human-robot interference on motion. The dynamic nature of HRS often results in human-robot occlusions in practical scenarios, for example, when a moving robot partially occludes the human body. Such occlusions are seldom present in publicly available datasets of daily human activities or in existing datasets about HRS, limiting the usage of these datasets in practical applications.

On the other hand, human motion processing is particularly pivotal for implementing HRS. From the related work, human motion processing can be categorized into several subfields, including gesture recognition, motion classification, motion prediction, task and procedure prediction, intention recognition, autonomous control of robotic systems, and robotic task procedure generation, among others^16,17^. However, the current limitations of existing work and their experiment conditions are evident in the following aspects:

• Number of Participants. Many studies validate proposed methods with few or even a single human due to the scarcity of available participants. For practical HRS applications, the model’s ability to generalize across individuals is crucial.
• Environmental Condition. Many precious publications employ relatively simple scenarios where a human interacts with a static environment, without the presence of industrial robots. Many studies only operate under ideal conditions, assuming high-quality data, clear vision, and the absence of occlusions. Human-robot occlusions are often neglected, resulting in methods that perform well only in controlled ideal environments.
• Hardware Barrier. Some datasets are constructed using specialized equipment like RGB-D cameras or wearable motion-capturing suits, which are impractical for large-scale deployment of models trained on such datasets in industrial settings.
• Task Sequences. In numerous published studies, the categories of human motions are overly simplistic, including actions such as raising hands, crossing arms, and waving hands, which are not typically encountered in real HRC scenarios. Additionally, similar and repetitive motions, along with their corresponding task procedures, receive insufficient attention. Many datasets focus on isolated human motions rather than sequential motions within complete tasks, hindering research on robust recognition across motion category transitions.
• Data Accessibility. Most studies in HRS, especially HRC, do not publicly share their datasets, making replication, benchmarking, and implementation challenging for the research community.

These limitations hinder the implementation of HRS in realistic environments and tasks. To this end, our dataset focuses on human motions in sequential assembly and disassembly tasks specially designed for HRC. It highlights samples from multiple individuals in human-robot coexisting environments with occlusions and similarity among task-oriented motion categories.

A comprehensive comparison between the proposed and existing datasets is summarized in Table 1. In detail, this dataset can be used to enhance generalization across diverse participants, incorporate multiple camera perspectives, handle real-world noise and occlusions, utilize accessible hardware setups, and focus on practical, repetitive, and sequential human motions. It includes raw video streams and human skeleton data, along with well-labeled and indexed annotations of various sequential human motions involved in assembly and disassembly tasks. Additionally, it provides complete Python scripts for video clipping, stitching, skeleton generation, data formatting, and labeling, ensuring ease of use and reproducibility. By offering these comprehensive features and being openly accessible, this dataset facilitates replication and benchmarking, thereby advancing the field of HRS, especially HRC. This dataset not only overcomes the limitations of existing datasets by providing robust, real-world scenarios with human-robot occlusions and sequential task procedures but also supports a wide range of research applications, including human motion prediction, robotic task planning, and collaborative system development. In summary, the proposed dataset has the following features.

• It involved 33 individuals of different gender (F/M = 1/2), clothing, height, and body shape, aged from 22 to 28.
• Two distinct scenarios for both assembly and disassembly tasks were set, each further involving static and dynamic settings depending on whether the robot affects the environment.
• Reflection of uncertainties was considered. All individuals behave according to their personal preferences, and the human body is partially blocked by the moving robot.
• Easy-to-use and easy-to-produce case. In this dataset, we used 3D-printed gear system assembly and disassembly tasks with nine procedures. The gear system (from a previous SIEMENS robot learning challenge) can be easily reproduced by users, and the procedures in this gear system are also feasible to be reassociated or reordered.
• Multiple camera views via contactless perception. Three off-the-shelf webcams were installed at different positions and heights without wearable devices.
• Flexibility. Raw videos with indexed frames, well-clipped human manual procedures, and Python scripts, based on which users can reclip the videos to form the task into different procedural sequences or apply different skeleton and mesh tracking algorithms.
• Multiple purposes. This dataset can support future HRS studies such as action recognition^18^, task sequence prediction, behavioral analysis considering uncertainties, robot task planning, robot motion planning^19^, turn-taking prediction in HRC, partially occluded human pose estimation, multi-source sensor fusion, etc. Table 1A comprehensive comparison between the proposed dataset and the existing related datasets.DatasetComparison of DatasetsTaskIndustrial EnvironmentModalitiesContactless Perception (wearable device free)Number of ParticipantsNumber of Camera ViewsHuman-robot OcclusionsMHRI^3^Human teaching robots with objects×Visual and audio*✓103×HRH^5^Human-robot hugging×Foot pressure, position, orientation×33N/A×MHHRI^6^Personality and engagement in HRI×Electrodermal activity and ECG✓184×THOR^4^Human-robot navigation×Motion trajectory and eye gaze×81×InHARD^9^Assembly✓Visual and skeleton×163×NEMO-Lowlands^7^Gameful HRI×Skeleton✓4281×HARMONIC^8^Assistive eating×Visual, eye gaze, body pose, hand pose, facial keypoints×243×HRI30^10^Industrial HRI✓Visual✓111×Assembly101^11^Assembly and disassembly✓Hand pose✓538×ATTACH^12^Assembly✓Visual and skeleton✓423×HA-ViD^13^Assembly✓Visual✓303×OursAssembly and disassembly✓Visual and skeleton✓333✓*

Furthermore, an extensive evaluation of 13 state-of-the-art deep learning models was conducted across two practical aspects: early human motion prediction (offline) and robot task procedure generation (online). The results indicate a noticeable performance gap between offline and online settings, highlighting the need for further research to enhance online inference with smoother transitions between task procedures. Additionally, the design trade-off between model capacity and computational efficiency expects a more detailed exploration.

Methods

The proposed dataset is collected from two scenarios. Scenario A is about fewer human-robot occlusions, while there are more occlusions in scenario B. In both scenarios, containers are placed on a workbench, and product components are in the containers. In scenario B, an industrial robot was moving arbitrarily in front of human operators. The two scenarios are shown in Fig. 1.Fig. 1. Scenarios for data collection.

There are 25 participants in each scenario. Because some of the participants only joined a single scenario, the complete dataset contains data from 33 student volunteers aged from 22 to 28. 8 of them are only in scenario A, 8 of them are only in scenario B, and 17 of them are in both scenarios. Among them, 11 are female and 22 are male. The height and weight distribution of all the participants are shown in Fig. 2. All participants were informed about the types of data to be collected, how those data would be stored and processed, and the overall purpose of the study. All participants provided consent for data collection and public release of de-identified videos. Participants had the option to wear face masks during data collection. To ensure privacy, all raw videos in the public dataset have been anonymized by automatically blurring faces. According to the Letter of Compliance (Dnr: HS-2025-2104 KS 4.4.1) provided by the KTH Research Ethics Advisor at the KTH Research Support Office, this research does not require institutional review. Although it involves processing information about living human beings, none of the data is traceable to specific individuals, either directly or indirectly, meaning no personal data is handled. Furthermore, the research does not involve any sensitive personal information such as health data, genetic or biometric data, or information related to ethnicity, beliefs, or legal offences. It also does not involve any physical or psychological procedures on human subjects, nor does it pose any risk of harm or use of biological samples. Therefore, the study falls outside the scope of the Swedish Ethics Review Act and does not require formal ethics approval.Fig. 2. Height and weight distribution of all the participants.

The assembly and disassembly tasks in this dataset use a gear system as the object. It contains seven components corresponding to seven task procedures. In addition, two more procedures are also contained, which are moving the plate from the holder to the workbench and moving the product installed on the plate from the workbench to the endpoint. In all, there are 9 procedures in both tasks. The CAD model of this gear system is also in the dataset as CAD_model.zip, which could be built using 3D printers.

Only off-the-shelf web cameras (Logitech C270 and HIKVISION E14a) were used for data collection. Participants were not required to use any wearable devices. Any video sample in this dataset was recorded by three cameras with different perspectives simultaneously. Cameras were located at different heights arbitrarily. For scenario A, cameras were located at 170 cm (left), 155 cm (middle), and 160 cm (right). For scenario B, cameras were located at 165 cm (left), 155 cm (middle), and 155 cm (right). There is no restriction on lighting conditions or coordinated lighting for multiple cameras.

During the assembly, the participants are required to first take a plate for components to be installed onto, then take components from containers and conduct assembly task procedures, and finally transport the assembled product with the plate to the endpoint. Disassembly is conducted inversely. Participants are required to conduct one task three times. Before data collection, every volunteer was taught how to do the assembly and disassembly tasks, and given one trial to practice. Apart from what has been mentioned, participants have no other rules to follow. For example, participants can use any hand for any procedure, use one hand or both hands for grasping, and are allowed to hesitate at any time. An illustration of such a protocol is shown in Fig. 3, in which anchors (the frame index dividing two adjacent procedures) are used to divide assembly and disassembly procedures, and the red arrows and blue curve are used to illustrate the reach-out and pull-back motion and trend, respectively. 5 annotators were required to label the start and end frame indexes for every procedure. One of them designed the annotation protocol, led the annotation team, supervised the work of the other four annotators, and finally verified all annotations by reviewing the segmented videos based on the provided frame indices.Fig. 3. Illustration of the protocol.

Data Records

The dataset is available at Dryad^20^. This dataset consists of 10,100 task procedure samples in total, that is 5,050 samples for each scenario and 2,025 samples for each task. In each scenario, data are from 25 participants, 3 cameras, 3 times of one task, 2 main tasks (assembly and disassembly), and 9 procedures. A summary of this dataset appears in Table 2. Concretely, this dataset consists of raw videos, clipped videos, and clipped 2D and 3D skeleton data.Table 2. Summary of the dataset.ScenarioTaskParticipantsProcessed SamplesTotal FramesTotal TimeAAssembly252,025224,5212:15:17Disassembly252,025205,4141:53:38BAssembly252,025236,5762:18:34Disassembly252,025212,9641:58:22Total338,100879,4758:25:51

Raw videos

The raw videos in this dataset are with 640 × 480 resolution. All videos are recorded at 30 frames per second, regardless of the camera hardware. All the videos are stored in AVI files, together with TXT files indicating the frame number and time in seconds from the start of the video. Each piece in the TXT files is a tuple containing three elements, that is (i, t, T), in which i (index of row) is the index of one frame, t is the time from the start in millisecond, and T is the clock time when this frame is recorded. There exist 900 raw videos in total, 450 for each scenario, and 225 for each task in one scenario.

Task procedure anchors

Task procedure anchors are the keyframe numbers that can annotate the start and end of any sequential human motions in the assembly and disassembly tasks. They are stored in CSV files. As for each row in such a CSV file, the first column is the name of one raw video. The remaining are 18 frame anchors (the start and end frames) for 9 task procedures in this raw video. For one task in each scenario, there is one CSV file of the task procedure anchors. The anchors of the annotation were done manually with Adobe Premiere Pro.

Video clips

Video clips were generated using the raw video and the task procedure anchors by the Python scripts. Every raw video is clipped into 9 procedural video clips. Video clips are also in AVI format and are renamed based on the file name of the raw video. There exist 10,100 video clips in total, 5,050 for each scenario, and 2,025 for each task in one scenario.

Skeleton frames

Skeleton frames are about the unmerged skeleton data generated by OpenPose (for 2D)^21^ and MMPose (for 3D)^22^ when taking video clips as input. We chose OpenPose for 2D skeleton generation due to its fast processing speed. We selected MMpose for 3D skeleton estimation due to its integration within the OpenMMLab ecosystem and its stronger modeling of human body priors. MMpose leverages parametric human models and applies kinematic constraints to enforce anatomical plausibility, which helps mitigate common issues such as joint floating, bone twisting, or limb penetration. End users of this dataset could switch to any skeleton generator for their usage. For every video clip, there is a series of JSON files representing the skeleton data of each frame. It is also possible to use other skeleton estimators on all the raw videos or video clips. The skeleton frames JSON files are formatted in the following way. This dataset provides skeleton data in both 2D and 3D formats. Within the 2D format, pose_keypoints_2d is heavily used, represented by (x, y, c), where x,y are the coordinate value, and c is the confidence value on this joint by the skeleton estimator. All 18 joints on one frame are listed in pose_keypoints_2d with an order defined by the Microsoft COCO dataset^23^. For the 3D skeleton, only the procedural skeleton sequences are provided. Skeleton frames are aimed at an easier understanding of the skeleton data on each frame, while the procedural skeleton sequences in the following paragraph are more often used.

Procedural skeleton sequences

A procedural skeleton sequence is a sequence of skeleton data from a single camera within one procedure in a task. One procedural skeleton sequence can be regarded as a sample, with a label indicating the category of this procedure. Procedural skeleton sequences are stored in JSON files. In such 2D skeleton JSON files, data contains frame indexes, as well as pose and score (confidence score generated by OpenPose) data at every frame. Finally, the label and its index are stored at the end of such a JSON file, as well as in the filename. The 3D skeleton JSON files have a similar structure, which can be seen as follows. Normalised procedural skeleton sequences are also provided in this dataset. The normalisation was conducted on camera resolution, for the user to be able to train models running on cameras with different resolutions. The users can certainly flexibly conduct the normalisation here or in the following steps, such as building the NumPy-based files before training.

Naming rules

All the AVI raw video files and their corresponding TXT files are with such a naming rule that is S**SP**PNKC, in which S**S ∈ {s1, s2}, P**P ∈ {p1, p1, …, p25},N ∈ {1, 2, 3}, K ∈ {a, d}, and C ∈ {1, 2, 3} are labels for scenarios, participant number, trial number, task, and camera number, respectively. Concretely, s1 is scenario A and s2 is scenario B. As for the task, a represents the assembly task while d is the disassembly task. Regarding cameras, camera 1 is the one to the right front of the human, camera 2 is the one to the left front of the human, and camera 3 is the one in the centre. All the AVI video clips were renamed by adding a final symbol to the original raw video name. It is shown as S**SP**PNKCR**R, where R**R ∈ {c1, c2, …, c9}. Skeleton frames are named by S**SP**PNKCR**RF, where F is the frame index starting from 0. Procedural skeleton sequences merged by skeleton frames are also named by S**SP**PNKCR**R.

Technical Validation

Extensive validations have been performed based on this dataset to demonstrate the practical usage of this dataset quantitatively and establish the benchmark for future research works. Concretely, a variety of state-of-the-art action recognition models in the computer vision field are implemented to realize the early prediction of human sequential motions for further task procedure generation for robots, inspired by previous work^24^ that used only the 2D skeleton data in the assembly task. These models include the RGB video-based ones (C3D^25^, I3D^26^, SlowFast^27^, TPN^28^, TimeSformer^29^, VideoSwin^30^, MViT^31^) and the skeleton-based ones (STGCN^32^, 2s-AGCN^33^, PoseC3D^34^, STGCNPP^35^). These models represent the major developments of human action recognition, e.g., from earlier Convolutional Neural Network (CNN)-based ones such as C3D and I3D to recent Transformer-based ones such as VideoSwin and MViT. They have also shown promising performances on various benchmarks in the general domain. For the skeleton-based models, Graph Convolutional Network (GCN) is commonly considered because it well captures the topological patterns of human body skeleton. Both 2D skeleton data generated from OpenPose and 3D skeleton data generated from MMPose are considered as the input modality. We benchmark both video and skeleton-based methods because of the consideration of the communication bandwidth cost of robotic applications. Often, skeleton-based modality has much lower data volume and requires fewer computational resources. The computer used for validation was equipped with an Intel i9-13900KF CPU, an RTX4090 GPU, and 64GB of RAM.

Human Motion Early Prediction

Human motion early prediction aims to recognize the task-related category of each human motion before it is completed, i.e., using a portion of video frames as the available observations. The fundamental problem is to classify the video segments based on the RGB or skeleton pose information.

The algorithmic performance for the early prediction of sequential human motions is evaluated for each of these models. Four subsets of this dataset are individually employed to report the outcomes (assembly and disassembly of scenario A and scenario B). Instead of randomly splitting each subset into the training and testing parts, two practical schemes are adopted that are similar to those in NTU RGB+D^15^: X Subject and X View. For X Subject, 70% of the entire participants are considered in training while the remaining 30% are used for testing. For X View, cameras 2 and 3 are considered in training while camera 1 is used for testing. Such schemes aim to fully exploit the diversity presented in this dataset, thus bridging the gap between the existing datasets and the HRC cases in the real world. Moreover, the first 25% and 50% of each of the clipped entire human motions are provided as the available information for the models, respectively. This is challenging yet meaningful because the robot task procedures need to be generated as early as possible to ensure smooth procedural transitions between human and robot. Top-1 and Top-2 recognition accuracies are reported for each of the configurations. The number of floating operations (FLOPs) of each implemented model is also presented (in the unit of Giga) to reflect the computational efficiency.

Tables 3–6, below show the results on the four subsets of this dataset, respectively. In general, it can be noticed that scenario B is more challenging than scenario A, regardless of the assembly or disassembly.Table 3. Human motion prediction results for all implemented models (Scenario B, assembly).ModelX SubjectX ViewFLOPs25%50%25%50%Top1Top2Top1Top2Top1Top2Top1Top2C3D98.7799.5498.9299.5449.1967.1152.3069.6338.50I3D98.6199.8599.23100.0077.1983.8566.8185.6343.50SlowFast91.2096.6097.0799.0757.7884.7468.8982.8165.71TPN98.75100.0099.23100.0073.0493.6373.7896.3066.01TimeSformer96.4599.5496.1499.3881.4895.1179.2694.52141.00VideoSwin98.9299.8599.3899.7986.2295.8988.59****96.61166.00MViT95.9997.9999.07100.0075.1189.6386.0793.0464.00STGCN-2D58.4975.4693.52****97.6813.6334.3721.7832.153.802s-AGCN-2D78.7088.5890.9096.9129.9339.7029.7846.524.40PoseC3D-2D76.5489.2081.9493.0629.3342.8133.9350.0720.60STGCNPP-2D83.80****91.8290.6197.0216.7427.8534.2248.741.95STGCN-3D72.3883.9581.6493.2171.4582.7279.6390.765.70STGCNPP-3D76.0888.5886.4296.3077.7888.2786.42****95.832.96Table 4Human motion prediction results for all implemented models (Scenario B, disassembly).ModelX SubjectX ViewFLOPs25%50%25%50%Top1Top2Top1Top2Top1Top2Top1Top2C3D95.2297.6999.0799.8561.4880.7454.3777.0438.50I3D97.3899.3897.0699.2363.8585.3384.7493.6343.50SlowFast96.7698.9296.9198.1585.1991.5689.9396.4465.71TPN97.8499.3897.6999.0487.8595.8589.7897.1966.01TimeSformer95.9998.9297.9999.6387.7096.5278.2291.77141.00VideoSwin97.6999.8199.38****100.0091.0896.4390.5298.64166.00MViT98.61****99.8598.7799.8993.3498.0692.7198.5764.00STGCN-2D65.9084.8883.6493.3628.1942.3733.4842.653.802s-AGCN-2D65.1281.7984.2695.9938.3756.5931.8540.894.40PoseC3D-2D70.8384.1083.4993.3738.0453.9340.6356.1520.60STGCNPP-2D70.6885.1986.5795.8339.9657.1939.8553.411.95STGCN-3D65.1282.1076.2387.6565.9080.5679.1792.135.70STGCNPP-3D71.14****85.4980.5691.6770.6885.3482.72****92.752.96Table 5Human motion prediction results for all implemented models (Scenario A, assembly).ModelX SubjectX ViewFLOPs25%50%25%50%Top1Top2Top1Top2Top1Top2Top1Top2C3D97.6999.3899.0799.6990.9698.5298.3799.7038.50I3D98.3099.5499.3899.7199.56100.0098.3499.5843.50SlowFast97.0798.6197.5398.3098.8199.8597.0499.2665.71TPN99.08****99.5498.9299.8599.41100.00100.00****100.0066.01TimeSformer96.1498.7797.5399.5496.3099.7099.11100.00141.00VideoSwin98.3099.5398.9299.86100.00100.0099.26100.00166.00MViT97.6999.2398.6199.3899.85100.0099.87100.0064.00STGCN-2D77.0188.1281.4890.9068.4480.8964.3079.263.802s-AGCN-2D87.9694.6090.2896.6065.6380.0463.4180.154.40PoseC3D-2D85.1993.2181.3392.5970.2287.1166.3788.1920.60STGCNPP-2D89.3594.4488.8996.3056.3073.6277.3390.371.95STGCN-3D88.7394.6090.7496.1484.2691.8292.9097.535.70STGCNPP-3D90.7596.9196.1397.8692.9096.1494.2996.912.96Table 6Human motion prediction results for all implemented models (Scenario A, disassembly).ModelX SubjectX ViewFLOPs25%50%25%50%Top1Top2Top1Top2Top1Top2Top1Top2C3D98.46100.0099.54100.0095.8599.7099.56100.0038.50I3D99.54100.0099.86100.0099.1199.4195.1799.1343.50SlowFast97.8499.52100.00100.0098.0799.7099.56100.0065.71TPN99.85100.00100.00100.0099.71100.00100.00****100.0066.01TimeSformer97.0699.6997.2299.8597.63100.0097.6699.89141.00VideoSwin100.00100.00100.00****100.0099.41100.00100.00100.00166.00MViT99.54100.0099.6999.85100.00****100.0099.70100.0064.00STGCN-2D70.5285.3483.6494.2942.3761.8455.8665.193.802s-AGCN-2D72.3887.1992.13****97.5351.4169.7867.5680.744.40PoseC3D-2D65.4378.5582.8791.9868.3081.0465.4885.9320.60STGCNPP-2D77.0187.0489.5195.9959.8581.1970.9689.481.95STGCN-3D73.3084.2684.4193.6765.1278.4085.4994.755.70STGCNPP-3D74.2388.8984.2693.8372.5386.4287.9694.602.96

Robot Task Procedure Generation

Robot task procedure generation indicates generating a procedure in a task sequence for the robot to conduct collaboratively, given the current status of the human motion. It stresses the online inference performance of each implemented model, which is directly related to the efficiency of HRC. Specifically, the models here are requested to execute the real-time prediction (inference) once every 5 video frames with a temporal sliding window of 32 video frames on each complete task video (instead of the procedural videos in the previous validation). The task sequence constraints (the gear system mentioned before) for the robot task procedure generation validation are from previous work^24^.

For each of these models, two important performance-related values are reported for each atomic human task procedure plus the average overall task procedures from Tables 7–10. The first value (the upper rows) corresponds to the temporally-aggregated prediction accuracy, while the second value (the lower rows) corresponds to the degrees of delay for the first correct prediction. Concretely, the first value quantifies the percentage of correct prediction times with respect to the overall attempts made. It is different from algorithmic performance in that it concerns continuous predictions as well as the overlap between successive procedures. The second value quantifies the percentage of running video frames before a correct prediction result first appears. It is intended to reflect how timely a meaningful robot task procedure can be generated based on human motion prediction. Tables 7–10 below present the results of the online inference performance for the challenging X View scheme and 50% observation of the complete videos. Considering the page limit, only the RGB video-based models are involved here because they generally perform better than the skeleton-based models.Table 7. The online inference results related to robotic procedure generation (Scenario B, assembly, X View, 50% Observation).ModelP0P1P2P3P4P5P6P7P8AllTemporally-aggregated prediction accuracy ↑C3D8.3773.6074.2719.970.2437.3477.858.7896.2544.08I3D38.7210.0971.4676.4037.0466.0775.7866.5577.2757.71SlowFast53.4769.1535.1313.2274.992.9879.279.8268.2945.15TPN99.7541.2234.3591.8746.3112.7181.7651.2560.7057.77TimeSformer66.9766.1325.9587.1234.2772.4969.9026.6151.1455.62VideoSwin98.7964.4880.3370.4266.5961.6897.7521.2582.3571.52MViT45.8084.0048.2316.9071.8464.4524.6335.5932.6647.12Degrees of delay for first correct prediction↓C3D19.708.945.2137.0570.0311.311.3751.172.4623.03I3D31.5869.5720.7122.4941.3418.536.1527.0223.8329.02SlowFast25.4314.6728.2057.2515.7146.005.1438.9315.5127.43TPN4.9225.994.801.5137.0637.461.237.8222.0415.87TimeSformer5.495.0310.342.9125.896.270.795.680.006.93VideoSwin21.7830.2814.2026.1026.1827.192.4561.311.6023.45MViT16.853.3916.7933.776.4116.7425.8327.6216.0018.16Table 8The online inference results related to robotic procedure generation (Scenario B, disassembly, X View, 50% Observation).ModelP0P1P2P3P4P5P6P7P8AllTemporally-aggregated prediction accuracy ↑C3D77.2075.6468.3118.749.7244.5042.5633.0221.1743.43I3D83.7749.1393.9753.4151.0357.9548.1835.3187.8662.29SlowFast82.5582.5659.5189.5874.6523.994.1548.4788.9361.60TPN81.8567.1028.1446.7789.9711.9951.586.6050.6648.30TimeSformer73.5367.1955.6075.5034.4567.0120.1064.2687.9260.62VideoSwin72.7599.4267.5261.1327.8594.5859.3669.4890.6571.41MViT50.5470.8945.6577.7650.4130.4555.3970.2261.3456.96Degrees of delay for first correct prediction ↓C3D7.713.4411.2632.6616.2629.7422.9922.4332.6219.90I3D16.1711.630.0039.5640.6938.8647.4345.3213.3728.12SlowFast15.948.1013.785.0115.6561.0534.3042.947.5422.70TPN3.613.009.895.163.4426.1110.4426.331.229.91TimeSformer3.741.862.994.559.994.0219.8515.510.416.99VideoSwin16.170.5722.5537.9742.765.1332.5229.7511.5522.11MViT14.730.876.579.0023.6834.3017.1411.6812.3714.48Table 9The online inference results related to robotic procedure generation (Scenario A, assembly, X View, 50% Observation).ModelP0P1P2P3P4P5P6P7P8AllTemporally-aggregated prediction accuracy ↑C3D76.7266.6687.3972.2990.8483.0684.1583.2986.5581.22I3D95.0761.2792.0892.3877.0746.5792.0765.6767.1776.59SlowFast91.3653.5760.8276.1581.1360.8173.2076.8347.3769.03TPN41.0275.5075.1270.3999.4428.2298.0268.9277.6670.48TimeSformer69.7871.5172.7368.4189.8161.1796.2855.1694.6575.50VideoSwin86.8296.7079.2777.2179.4384.6082.1384.1284.6883.88MViT48.2892.0383.3653.4494.4760.3997.2177.1479.7976.23Degrees of delay for first correct prediction ↓C3D9.5013.520.999.884.0511.1012.1021.8112.3810.59I3D19.9818.121.414.7124.5437.208.1956.6436.5623.04SlowFast20.3942.5228.1814.4419.6038.0023.5136.7734.0828.61TPN13.452.280.321.930.4823.720.5132.4119.2210.48TimeSformer4.412.280.085.753.047.121.0519.291.414.94VideoSwin19.910.6721.4020.2517.5410.0611.8121.8716.6015.57MViT24.243.047.7822.302.4222.851.9132.330.0012.23Table 10The online inference results related to robotic procedure generation (Scenario A, disassembly, X View, 50% Observation).ModelP0P1P2P3P4P5P6P7P8AllTemporally-aggregated prediction accuracy ↑C3D73.9696.0383.8786.4887.0892.2261.1565.8096.5882.58I3D93.2996.7376.4790.0368.8238.8616.0760.2362.1866.97SlowFast86.5996.2861.0794.6251.5248.1124.9043.9863.7763.42TPN72.9362.9478.1773.1653.0099.0758.7841.4362.0966.84TimeSformer70.5761.0772.8779.3595.5883.5970.3476.0393.8478.14VideoSwin86.2898.4587.6887.7969.8980.2944.9971.9685.8479.24MViT80.7496.5794.5774.9878.6172.2435.4564.5578.2875.11Degrees of delay for first correct prediction ↓C3D7.910.0713.036.335.905.3933.2322.612.3110.75I3D16.342.4311.974.6129.7861.5074.4539.2839.9631.15SlowFast16.383.7029.154.6044.4651.2772.8650.8133.3434.06TPN4.060.520.791.696.320.0028.3223.3820.659.53TimeSformer3.690.004.931.331.9013.0921.5817.561.967.34VideoSwin16.340.8611.5711.4829.6219.9754.8027.7416.0120.93MViT8.610.592.9416.8314.1317.0035.8817.5917.6314.58

Usage Notes

Users can use the data from different folders for their research purposes. For instance, for each task in one scenario, users can reach raw video and its corresponding frame index and timestamp from folders raw video and raw video frames. Data from these two folders can be used for video processing and further data generation with the specific needs of the users, just like skeletons. Data in folders procedural skeleton sequences and procedural skeleton sequences 3D are about the human skeleton sequences of every task procedure, which can be used for skeleton-based processing. Human skeletons were generated by estimators with no bias or further manual tuning. It means all the raw videos are the ground truth, while the quality of human skeletons relies on the human skeleton estimator. Occlusions affect the precision of skeleton joint position and may cause loss of joints. Users could switch to any other skeleton estimators for their research. Data in procedure_anchors.csv can be used to clip the raw video into procedural ones. Data from the aforementioned two scenarios can also be used jointly, for instance, to study the effects on parameter sensitivity by environmental changes. Because this dataset provides divided procedural data, rearranging the sequence can boost the diversity regarding the procedure sequence.