Human Pose Estimation for Real-World Crowded Scenarios

TL;DR

This paper enhances human pose estimation in crowded scenarios by using data augmentation, explicit occlusion detection, and synthetic datasets, achieving a 4.7% AP improvement and addressing real-world challenges.

Contribution

It introduces novel methods including occlusion augmentation, explicit occluded part detection, and synthetic dataset extension for improved crowded scene pose estimation.

Findings

Improved accuracy by 4.7% AP in crowded scenarios.

Effective use of synthetic datasets for real-world applications.

Explicit occlusion detection enhances pose estimation robustness.

Abstract

Human pose estimation has recently made significant progress with the adoption of deep convolutional neural networks. Its many applications have attracted tremendous interest in recent years. However, many practical applications require pose estimation for human crowds, which still is a rarely addressed problem. In this work, we explore methods to optimize pose estimation for human crowds, focusing on challenges introduced with dense crowds, such as occlusions, people in close proximity to each other, and partial visibility of people. In order to address these challenges, we evaluate three aspects of a pose detection approach: i) a data augmentation method to introduce robustness to occlusions, ii) the explicit detection of occluded body parts, and iii) the use of the synthetic generated datasets. The first approach to improve the accuracy in crowded scenarios is to generate occlusions at training time using person and object cutouts from the object recognition dataset COCO (Common Objects in Context). Furthermore, the synthetically generated dataset JTA (Joint Track Auto) is evaluated for the use in real-world crowd applications. In order to overcome the transfer gap of JTA originating from a low pose variety and less dense crowds, an extension dataset is created to ease the use for real-world applications. Additionally, the occlusion flags provided with JTA are utilized to train a model, which explicitly distinguishes between occluded and visible body parts in two distinct branches. The combination of the proposed additions to the baseline method help to improve the overall accuracy by 4.7% AP and thereby provide comparable results to current state-of-the-art approaches on the respective dataset.

Tables

Table 1. Table 1: Results of the different proposed cutout augmentation methods on the CrowdPose test set. While all approaches improve the performance over all crowding levels, the object cutout augmentation provides the best result, with an increase in 1.1% AP compared to the Baseline-50.

Cutout Method	AP	${\text{AP}}_{\text{Easy}}$	${\text{AP}}_{\text{Med}}$	${\text{AP}}_{\text{Hard}}$
-	61.9	72.7	63.1	49.1
Objects	63.0	73.4	64.1	50.5
Body Parts	62.8	73.0	63.9	50.4
Full Body	62.6	73.0	63.8	50.0
Parts and Obj.	61.3	71.7	62.5	48.7
Full and Obj.	61.5	71.6	62.7	49.0
Parts or Obj.	62.9	73.0	64.0	50.2
Full or Obj.	62.8	73.1	63.9	50.5

Table 2. Table 2: Overview of the results of the occluded joint detection networks in comparison to the baseline architecture on the subset of the JTA test set. OccNetCB provides results similar to the Baseline-50 model, whereas OccNet even degrades in performance.

Architecture	AP	${\text{AP}}_{\text{Easy}}$	${\text{AP}}_{\text{Med}}$	${\text{AP}}_{\text{Hard}}$
Baseline-50	88.1	90.8	86.5	85.7
OccNet	87.8	90.7	86.3	85.5
OccNetCB	88.0	90.7	86.5	85.7

Table 3. Table 3: Comparison of the results of the Baseline-50 model trained on JTA, JTA-Ext and CrowdPose, tested on the CrowdPose test set.

Training Set	AP	${\text{AP}}_{\text{Easy}}$	${\text{AP}}_{\text{Med}}$	${\text{AP}}_{\text{Hard}}$
CrowdPose	61.9	72.7	63.1	49.1
JTA	26.6	36.6	27.0	17.1
JTA-Ext	28.8	39.8	29.2	18.3

Table 4. Table 4: The results of the Baseline-50 model pre-trained on JTA, JTA-Ext and fine-tuned on CrowdPose in comparison to a model purely trained on CrowdPose. The CrowdPose test set is used for evaluation.

Pre-Training Set	AP	${\text{AP}}_{\text{Easy}}$	${\text{AP}}_{\text{Med}}$	${\text{AP}}_{\text{Hard}}$
-	61.9	72.7	63.1	49.1
JTA	63.3	73.7	64.4	50.6
JTA-Ext	63.0	73.1	64.3	50.4

Table 5. Table 5: Results on the CrowdPose test set including the proposed split of [ 6 ] in different crowding levels. Since the dataset is quite young, only few work reports results on its test set.

Method	AP	${\text{AP}}_{\text{Easy}}$	${\text{AP}}_{\text{Med}}$	${\text{AP}}_{\text{Hard}}$
Xiao et al. [11]	60.8	71.4	61.2	51.2
Li et al. [6]	66.6	75.7	66.3	57.4
Ours	65.5	75.2	66.6	53.1

Equations

$$\begin{array}{r}{\mathcal{L}}_o=\frac{1}{n}\sum _{i=1}^n\parallel P_i^{\text{vis}}-G_i^{\text{vis}}{\parallel }_2+\alpha \parallel P_i^{\text{occ}}-G_i^{\text{occ}}{\parallel }_2\end{array}$$

Full text

1 Introduction

Articulated human pose estimation has been a commonly addressed problem in computer vision for many years. It has attracted broad interest for its many applications such as video surveillance, action recognition, human computer interaction and motion capturing. Recent advances in human pose estimation have been achieved by harnessing the power of deep convolutional neural networks (CNNs) and large-scale pose estimation datasets like COCO [7] and MPII [1]. Up until now most approaches focus on pose estimation for images with few people in uncrowded scenarios. ††* Both authors contributed equally to this work. However, in many applications it is inevitable to handle larger groups or crowds of people which also introduces many new challenges, see Fig. 1.

Such challenges include partially occluded people, mutual occlusions by humans and low resolution of persons due to sensor limitations or distance to the camera. One prominent application is video based surveillance of public places, where pose estimation can be utilized to analyze crowd behavior. To be more clear at this point, our notion of crowded situations is different compared to the field of e.g. crowd counting. Whilst in the latter case most of the time only the area around the head of single persons is visible, we regard highly crowded situations as cases in which occlusions between persons are not permanently present but still dominate the situation. Pose information encoded as body joint keypoints is a low dimensional description and hence well suited for real-time analysis. This could enable to detect and prevent critical or dangerous situations by alarming security forces, if necessary. However, to the best of our knowledge there exists very little work on human pose estimation for crowded surveillance scenarios.

Thus, in this work we evaluate several aspects of state-of-the-art pose detection methods with respect to the characteristics of crowded surveillance imagery. Specifically, we focus on three distinct aspects in the pipeline of modern pose detection methods. We first evaluate the impact of a recently proposed occlusion data augmentation method on highly crowded data. We then propose our own architecture modification for explicitly modeling occluded keypoints within a CNN pose detector. And finally, we generate a new simulated dataset, which is specifically suited to evaluate recent pose detection methods under crowded surveillance scenario conditions. Our resulting surveillance pose recognition model, as well as the generated dataset will be made available to the community to spark further research.

This work is divided as follows. Starting with this introduction, this work then presents related literature for human pose estimation as well as the few existing works on human pose estimation for crowds. The subsequent section concentrates on the methods chosen for our experiments, wWihich includes data augmentation methods, the adaption of an enhanced architecture for detection of occluded keypoints and the use of synthetic training data. Finally, the described methods are evaluated separately and in a combined model.

2 Related Work

2.1 Top-Down and Bottom-Up Methods

Existing methods for human pose estimation can be divided into two opposing approach classes: bottom-up and top-down methods. Methods of the former class attempt to first locate all person keypoints (e.g. joints, eyes, ears) over the entire input image. These keypoints are then grouped to single persons, which is done by solving an assignment problem. In contrast, top-down methods start from person detections and identify keypoints within the detection bounding boxes, following the assumption that each bounding box contains at most one keypoint per keypoint class. Each of them comes with different advantages and disadvantages. Although bottom-up methods seem to be better suited for crowded scenes since they process whole images and their runtime is thus less dependent of the actual number of persons, literature shows that top-down methods like [6] and [11] perform comparably well on publicly available datasets.

2.2 Simple Baselines for Human Pose Estimation

The model introduced in [11] is the recent winner of the PoseTrack ECCV 2018111https://posetrack.net/workshops/eccv2018/posetrack_eccv_2018_results.html challenge in the category for multi-person pose tracking. It provides a simplistic baseline model based on a top-down pose estimation network and a novel method for tracking detected poses across multiple frames. The main difference to other state-of-the-art methods like CPN [2] and the hourglass network [8] lies in the method for generating high resolution feature maps using transposed convolutions.

The method proposed by [11] refines the common top-down pipeline by additionally using a second set of bounding boxes, which is determined by optical flow from previous frames. Therefore, the keypoints in the current frame are propagated to the next frame using the optical flow between the frames. With the help of the propagated keypoints, a bounding box proposal is generated. Non-maximum suppression is then used to select the best set of bounding boxes. Furthermore, the authors apply a flow-based pose similarity measure $S_{\text{flow}}$ to the greedy matching algorithm, instead of using Intersection-over-Union (IoU).

2.3 Further Work on Crowd Pose Estimation

As previously stated, human pose estimation in crowded scenarios is a rarely addressed problem. However, some recent works focus on this subject. Li et al. [6] introduce a new benchmark for evaluating pose estimation methods for this problem and proposed a method where human bounding box proposals obtained by human detector are fed into joint-candidate single person pose estimator (JCSPPE). JCSPPE locates the joint candidates with different response scores on the heatmap. Then the joint association algorithm takes these results and builds a person-joint connection graph. Finally, a graph matching problem is solved to find the best joint association result with a global maximum joints association algorithm. With their method the authors show robustness to interference in crowded scenes. Yet, their definition of "crowdedness" differs slightly from common situations in surveillance applications. Since the collected data originates from the COCO dataset, it has a relatively low person per image count. Furthermore, the perspective the images were taken from often differs strongly from typical views of surveillance cameras, which in most cases are mounted in higher spots leading to a steeper viewing angle.

Furthermore, the work of Fabbri et al. [3] is also of great interest for this kind of problem. Using Grand Theft Auto V, a widely known computer game with an active modding community, they created a synthetic dataset of humans annotated with highly accurate keypoint information alongside with a tool that allows to easily generate own synthetic datasets. Both, [3] and [6] proposed interesting datasets, which we used for our experiments, whereas [3] is more similar to surveillance situations than the dataset introduced by [6].

3 Methods

This section explores methods to optimize pose estimation for crowd applications. For this purpose we adapt the single-person pose estimator proposed by Xiao et al. [11] using the ResNet50 [5] network as backbone. The method is a two-staged top-down approach that first localizes every person using state-of-the-art object detection methods and then performs single-person pose estimation for every detection. This choice is motivated by the simplicity of the model and its performance compared to other state-of-the-art approaches.

For adaptation to crowded scenes, data augmentation methods are first discussed which improve the performance of the baseline model on both uncrowded and crowded scenarios. Next, the baseline architecture is extended in order to explicitly detect occluded keypoints with the help of the occluded keypoint annotation provided with the JTA dataset. Finally, the earlier mentioned JTA dataset is examined for the challenges of real world crowd applications and extended by a small dataset which was created specifically to alleviate some of these problems.

3.1 Synthetic Generated Occlusions

As Sárándi et al. [10] have shown, synthetic generated occlusions are an effective augmentation method to reduce occlusion-induced errors in pose estimation, while also improving results on data without any occlusions. Since the amount of real-world data that is currently available for pose estimation of human crowds is very limited, the question arises whether these augmentation methods are also helpful in the task of pose estimation for human crowds. In order to answer this question, two different kinds of synthetic occlusions are examined: occlusions by random objects and occlusions by person instances. For both augmentation methods the instance segmentation labels provided with the COCO dataset are used to get a high variety of objects and persons.

Similar to the approach of Sárándi et al. [10] randomly picked objects from the COCO dataset are cut out from their original images using the provided segmentation maps. These cutouts are then added onto the bounding boxes of people in the training image. The size and the position of the cutout is chosen at random, where the size can range from 8% to 70% of the size of the person's bounding box. Furthermore, the occlusion flag of a keypoint is adjusted accordingly, if it is occluded by the cutout.

While the segmentation labels for person instances are also available, they need to be handled separately, because they introduce different challenges than the occlusion by objects. For example, if a person is added exactly on top of another person, the labeled keypoints are likely to be occluded by the added person's body which can lead to faulty training data. To counterbalance this issue, two different methods are evaluated to add additional persons to the detections. The first one is instead of adding full person instances to the image only body parts are added onto the scene. Secondly, placement of cutouts is restricted to the outer bounds of the target persons bounding box to avoid confusing the top-down pose recognizer with two fully valid person images inside one bounding box.

3.2 Occlusion Detection Networks

In order to utilize the additional information provided by the occlusion flags in the JTA dataset, the baseline architecture is extended by integrating a branch exclusively for the detection of occluded keypoints. The idea behind that is that the detection of occluded and visible keypoints are connected but different tasks. Furthermore, this should enable the network to learn a better representation of the human body structure that is more robust at handling occlusions and provide explicit additional occlusion information for possible subsequent methods in an image analysis pipeline. Two architectures are proposed with an additional branch to detect the occluded keypoints, see Fig. 3.

The first proposed architecture, OccNet (Occlusion Net), splits after two transposed convolutions so a joint representation can be learned in the previous layers. The second architecture OccNetCB (Occlusion Net Cross Branch) splits after only one transposed convolution, however the output from both layers are shared to one another. So that both branches can utilize information extracted by the other one. JTA also provides the distinction between self-occluded and occluded keypoints, during training self-occluded keypoints are treated in the same way as visible keypoints.

The Occlusion Detection Networks output two sets of heatmaps per pose, one for visible keypoints and the other one for occluded keypoints. The mean squared error loss of the baseline model of Xiao et al. [11] is adjusted by adding a second set of heatmaps for occluded keypoints and a weighting factor $\alpha$ to account for the lesser share of occluded keypoints in the training data. The proposed loss can be defined as:

[TABLE]

where $G_{i}$ and $P_{i}$ denote the respective ground-truth and predicted heatmap of the corresponding keypoint $i$. Due to its definition, the loss explicitly penalizes keypoints predicted by the wrong branch, which is achieved by setting the ground truth heatmaps for visible keypoints to zero in the occluded branch and vice versa.

3.3 JTA-Ext

The synthetic dataset JTA introduced by Fabbri et al. [3] is currently the largest publicly available pose tracking dataset with a larger number of people within an image. However, the adaption of the dataset for real-world crowd applications introduces some challenges, mainly: the visual difference of synthetic images of a video game, limited pose variety and less crowded scenes in comparison to CrowdPose. Since JTA focuses primarily on urban settings, the images almost exclusively show people walking, which leads to far less variety of poses. In order to quantify this observation, Fig. 5 shows 2d histograms of the location of specific keypoint types. It is evident that the distribution of the JTA keypoints are much more concentrated in comparison to the approximately uniformly distributed CrowdPose dataset, which leads to the problem a model purely trained on JTA has a strong bias of walking and standing people. We addressed this issue by extending the existing dataset and including all activities provided with the JTA-Mod like sitting, doing yoga, doing push-ups, cheering and fighting. In this work, the proposed extended JTA version is referred to as JTA-Ext222Dataset available on https://github.com/thomasgolda/Human-Pose-Estimation-for-Real-World-Crowded-Scenarios. Another goal in creating an extension for the JTA dataset was to increase the share of highly crowded scenarios. In order to gauge the "crowdedness" of the datasets we utilized the CrowdIndex proposed by Li et al. [6]. The different CrowdIndex distributions are presented in Fig. 2. While [6] sampled the dataset to achieve a uniform CrowdIndex distribution, JTA has a bias on less crowded images. The proposed extension is aimed to increases the amount of strongly crowded images in the dataset. The final extended JTA dataset consists of 58 additional video sequences with a total of 812,742 poses distributed over 46,350 frames.

4 Experiments

4.1 Datasets

We conduct experiments on two different pose estimation datasets: the recently released real-world dataset CrowdPose [6] and the synthetical dataset JTA [3].

The CrowdPose dataset [6] contains about 20,000 images and a total of 80,000 human poses with 14 labeled keypoints. In the following experiments the approaches will be trained on the training set of CrowdPose and will be evaluated on the test set of CrowdPose which includes 8,000 images.

JTA is a synthetic dataset for pedestrian pose estimation and tracking which Fabbri et al. [3] collected by extracting videos from the video game Grand Theft Auto V. The annotations for human poses in JTA include 22 keypoints, a tracking id and visibility flags. The visibility flags distinguish whether a keypoints is visible, occluded or self-occluded. In order to compare the approaches across different datasets the annotations of JTA were matched to the format of CrowdPose by simply discarding and reordering the keypoints. For the evaluation on the JTA dataset only a subset of the test set is used which includes only every fifteenth frame of the JTA test-sequences. Furthermore, JTA-Ext is used for training which includes the extension of the JTA dataset and the original training set of JTA.

4.2 Evaluation Metrics

Following the procedure introduced in [6], the adjusted Object Keypoints Similarity (OKS) of CrowdPose is used. The OKS is used to describe the similarity of two poses and their keypoints by introducing a way to measure overlaps between keypoints in a manner as the IoU provides for bounding boxes. Afterwards, the results are reported on three different partitions of the dataset which cluster images based on their CrowdIndex $C=\min\{\frac{1}{N}\sum_{i=1}^{N}\frac{N_{a}^{i}}{N_{b}^{i}},1.0\}$. Thereby, $N$ is the number of persons in the image, $N_{b}^{i}$ is the number of keypoints within the bounding box of person $i$ corresponding to the person and $N_{a}^{i}$ is the number of keypoints inside bounding box belonging to all persons $j\neq i$. These levels are denoted as easy $[0.0,0.1)$, medium $[0.1,0.8)$, and hard $[0.8,1.0]$. Fig. 4 shows two examples for crowded images.

Although the metric describes crowded images from a shallow viewing angle in an appropriate way, it does not really capture the crowdedness of images taken from a higher viewing point. Due to this property, persons close to each other show a lower overlap of bounding boxes compared to image material recorded with lower located cameras. Consequently, this results in a CrowdIndex of $0.91$ and $0.69$ for Fig. 4a and Fig. 4b respectively, which suggests that the left image is more difficult.

4.3 Implementation

For the baseline model a TensorFlow re-implementation333https://github.com/mks0601/TF-SimpleHumanPose of the model of Xiao et al. [11] is used with a pre-trained ResNet50 backbone. For the input images the same resolution of $256\times 192$ is used while the resolution of the keypoint heatmaps is $64\times 48$ as proposed by [11]. While ground-truth boxes are used during training of the models, the evaluation requires person detections for a fair comparison to other approaches. Therefore, the person detection results of the YOLOv3 [9] model integrated in AlphaPose is used for all the following experiments444The code and the used weights of the used YOLOv3 provided by Xiu et al. [12] and Fang et al. [4] are available at: https://github.com/MVIG-SJTU/AlphaPose/tree/7eea793e5d497e7947bb8e6ce547d98c6a0059bf on CrowdPose. For our used loss function we set $\alpha$ to $1.5$.

4.4 Synthetic Generation of Occlusions

For comparison between the different cutout methods and the effect these methods have in the crowded scenario the Baseline-50 model is trained on the CrowdPose training set and evaluated respective test set. During training the Adam Optimizer is utilized with a learning rate of $10^{-3}$ which is dropped at epochs 90 and 120 by a factor of 10, similar to the training schedule proposed by Xiao et al. [11]. All the models were trained for a total number of 140 epochs with a mini-batch size of 64. Additionally, to testing every single cutout method separately, combinations of objects and the different person cutout methods are evaluated. Therefore, two different setups are proposed. The first is, to add cutouts independently that means it is possible, that both cutout methods are applied simultaneously. The second one is, that the cutout methods are selected randomly with a probability of 50% without both occurring at the same time.

The quantitative results in Table 1 show that all the proposed cutout methods improve the accuracy across all crowding levels, while the relative performance improvement increases as the scenes are more crowded. The object cutout method produces the best results across all crowding levels on the CrowdPose test set, but the increase in performance in comparison to the body cutout methods stagnates as the images are more crowded. Another interesting observation is that the full body cutouts degrade the performance compared to adding only body parts. The reason behind that could be, that by avoiding the center of the image for the full body cutout, the degree of occlusion is lower than on the body part cutout, where the position is not limited. Applying two different cutout methods simultaneously leads to a decrease in accuracy. This can be attributed to the fact that the combination of two different cutouts within in image occlude too much of the person's body, prohibiting the network to see unobstructed people during training. Which would also explain that the accuracy increases again when only one of the two cutout method is applied per image.

4.5 Detection of Occluded Keypoints

In order to study the effects of explicitly detecting occluded keypoints in crowded pose estimation, the Baseline-50 is compared to the proposed OccNet and OccNetCB architectures. Since the occluded joint detection networks need reliable ground truth annotations for visibility flags, the JTA dataset is chosen for training. All three architectures are trained for 15 epochs with a mini-batch size of 64 on the entire JTA training set. Again a learning rate of $10^{-3}$ is chosen which is dropped at epochs 8 and 10 by a factor of 10. For testing a subset of the JTA test sequences is used. Also it is important to note that during evaluation the ground-truth human detections are used to ensure that occluded body parts are included in the boxes.

The results in Table 2 show that the distinction between occluded and visible keypoints does not improve the accuracy, because the proposed methods achieve similar accuracy on the JTA test set. Nevertheless, the qualitative results presented in Fig. 6 show, that the proposed OccNet architectures learned to distinguish between visible and occluded keypoints, even on selected real-world images. That proves, that the distinction between occluded and visible keypoints is working, but simply does not provide a benefit to pose estimation in crowded scenarios. Therefore, the results also imply that the Baseline-50 model is sufficient to predict the location of occluded keypoints implicitly by inferring the location of visible keypoints.

4.6 Synthetic Training Data

Next, it is tested whether the use of synthetic data can provide comparable results on real-world datasets and if the extension of JTA can help to close the transfer-gap to a real-world dataset like CrowdPose. Therefore, the results of the Baseline-50 model trained on JTA, JTA-Ext and CrowdPose are compared to each other on the CrowdPose test set. The results are presented in Table 3. While it is apparent that the extension of JTA helped to improve the results, the accuracy severely lacks behind a model trained on CrowdPose. However, Table 4 shows that if the models are finetuned on CrowdPose is the AP increases across all crowding levels compared to a model purely trained on CrowdPose. Under these circumstances synthetic data has proven again to be a valuable addition to existing real-world training data, but should not be used as sole foundation for real-world applications.

4.7 Combined Experiments

Previous experiments explored approaches to improve the baseline architecture and evaluated them separately for the task of crowded pose estimation. The following experiment combines these approaches to compete with current approaches on the CrowdPose dataset. In contrast to previous experiments a ResNet101 backbone is used to provide a fair comparison to the approach of [11] and [6]. Additionally, the model is pre-trained on JTA for 8 epochs with a learning rate of $10^{-3}$ which is dropped at epochs 3 and 8 by a factor of 10. During training a mini-batch size of 80 was chosen and the object cutout method is utilized. For re-training on CrowdPose the same learning schedule and batch size as described in previous experiments was set. The results of Xiao et al. and Li et al. are taken from [6], in which the models are purely trained on the CrowdPose training set. As stated above the same human detector of [6] is used to ensure a fair comparison between the different methods.

Our proposed approach shows significant improvement compared to the adapted baseline approach of [11] and also provides comparable results to Li et al. [6] on the easy and medium crowding levels. However, on extremely crowded images [6] surpasses the final model on CrowdPose by 4.3% AP. This can be attributed to their joint candidate matching method designed especially for handling these kinds of crowded scenarios. However, to our best knowledge there are no further methods reporting results on CrowdPose. A qualitative comparison of the improvements of our combined model in relation to the Baseline-50 model is depicted in Fig. 7.

5 Conclusion

In this work, we examined the impact of different extensions to the training of human pose estimators for crowded scenes. We show that significant improvements were accomplished by utilizing simple data augmentation techniques like adding COCO [7] objects to the input images from crowded scenarios thus simulating occlusions. On the other hand, we observe that adding person instances or body parts is less effective and also requires additional attention to avoid creating erroneous poses in a top-down method. Our results further prove that JTA [3] is a valuable addition to the training data for crowd-level pose estimation, especially since no large datasets are available for pose estimation in crowded scenarios. The created extension of JTA, which includes a higher variety of poses and denser crowds, further improves the accuracy on CrowdPose. However, solely relying on synthetic training data is not sufficient, due to the domain gap between synthetic and real-world data, originating from limited pose variety and visual limitations of a real-time rendering engine. Finally, the combination of the investigated approaches improved the baseline model and achieves results comparable to state-of-the-art methods on the real-world dataset CrowdPose.

In future work, we will especially focus on the problem arising by the gap between synthetic and real-world data.