COMETH: Convex Optimization for Multiview Estimation and Tracking of Humans

TL;DR

COMETH is a lightweight, convex optimization-based algorithm that enhances multi-view human pose estimation and tracking by integrating biomechanical constraints, improving accuracy, temporal consistency, and scalability for real-time industrial applications.

Contribution

It introduces a novel convex optimization framework for multi-view human pose fusion that combines kinematic constraints, inverse kinematics, and state observation for improved accuracy and efficiency.

Findings

Outperforms state-of-the-art methods in localization and tracking accuracy.

Enables real-time, scalable human motion tracking in industrial settings.

Demonstrates robustness on public and industrial datasets.

Abstract

In the era of Industry 5.0, monitoring human activity is essential for ensuring both ergonomic safety and overall well-being. While multi-camera centralized setups improve pose estimation accuracy, they often suffer from high computational costs and bandwidth requirements, limiting scalability and real-time applicability. Distributing processing across edge devices can reduce network bandwidth and computational load. On the other hand, the constrained resources of edge devices lead to accuracy degradation, and the distribution of computation leads to temporal and spatial inconsistencies. We address this challenge by proposing COMETH (Convex Optimization for Multiview Estimation and Tracking of Humans), a lightweight algorithm for real-time multi-view human pose fusion that relies on three concepts: it integrates kinematic and biomechanical constraints to increase the joint positioning accuracy; it employs convex optimization-based inverse kinematics for spatial fusion; and it implements a state observer to improve temporal consistency. We evaluate COMETH on both public and industrial datasets, where it outperforms state-of-the-art methods in localization, detection, and tracking accuracy. The proposed fusion pipeline enables accurate and scalable human motion tracking, making it well-suited for industrial and safety-critical applications. The code is publicly available at https://github.com/PARCO-LAB/COMETH.

Tables

Table 1. Table 1 : Position and velocity limits for each joint angle. Position limits come from the biomechanics literature, while velocity ones are retrieved from the BioAMASS dataset  [ 24 ] .

Joint angle	$q_L$ $[$°$]$	$q_U$ $[$°$]$	${\dot{q}}_L$ $[$°/s$]$	${\dot{q}}_U$ $[$°/s$]$
Hip Flexion/Extension	$-40$	$140$	$-1.6$	$1.9$
Hip Adduction/Abduction	$-45$	$45$	$-0.6$	$0.5$
Hip Rotation	$-45$	$45$	$-0.6$	$0.5$
Knee Flexion/Extension	$-10$	$140$	$-2.0$	$2.1$
Lumbar Bending	$-20$	$20$	$-0.5$	$0.5$
Lumbar Twist	$-5$	$5$	$-0.3$	$0.3$
Thorax Bending	$-20$	$20$	$-0.4$	$0.4$
Thorax Twist	$-5$	$5$	$-0.3$	$0.3$
Shoulder Adduction/Abduction	$0$	$150$	$-0.8$	$0.8$
Shoulder Rotation	$-70$	$90$	$-0.9$	$0.9$
Shoulder Flexion/Extension	$-60$	$180$	$-1.4$	$1.4$
Elbow Flexion/Extension	$-11$	$154$	$-1.4$	$1.4$

Table 2. Table 2 : Results on the panoptic dataset.

Cams	Method	LocA (%)$\uparrow$	DetA (%)$\uparrow$	AssA (%)$\uparrow$	HOTA (%)$\uparrow$
	3D HPE	76.9	56.7	-	-
1	OpenPTrack [7]	74.9	40.7	43.1	40.6
	Befine [10]	76.9	56.6	26.4	34.9
	COMETH	78.4	58.9	56.1	56.1
2	OpenPTrack [7]	76.8	34.1	28.6	29.6
	Befine [10]	82.9	67.0	29.5	38.8
	COMETH	82.0	69.0	75.8	71.8
3	OpenPTrack [7]	77.1	25.1	22.4	22.0
	Befine [10]	84.6	68.4	30.7	39.1
	COMETH	85.1	72.0	84.1	77.3
4	OpenPTrack [7]	77.0	17.4	18.9	16.3
	Befine [10]	85.1	66.2	31.0	38.1
	COMETH	86.7	72.0	87.1	78.5
5	OpenPTrack [7]	76.7	11.4	18.4	12.7
	Befine [10]	85.5	63.6	31.2	37.1
	COMETH	87.4	69.3	86.7	76.7

Table 3. Table 3 : Localization and tracking accuracy of the fusion method varying number of subjects and cameras.

Cameras		1		2		3		4		5
Subjects	Method	LocA (%)$\uparrow$	HOTA (%)$\uparrow$	LocA (%)$\uparrow$	HOTA (%)$\uparrow$	LocA (%)$\uparrow$	HOTA (%)$\uparrow$	LocA (%)$\uparrow$	HOTA (%)$\uparrow$	LocA (%)$\uparrow$	HOTA (%)$\uparrow$
2	OpenPTrack [7]	73.4	38.0	75.8	23.8	76.8	18.6	76.9	14.9	76.6	12.6
	Befine[10]	75.8	35.5	82.2	38.0	83.5	38.2	83.7	37.0	83.7	36.0
	COMETH	77.4	50.4	81.3	72.5	84.5	81.9	86.3	85.5	87.3	83.6
3	OpenPTrack [7]	75.1	39.7	75.9	28.6	75.7	20.4	75.7	15.0	76.2	12.5
	Befine[10]	76.3	22.5	81.7	23.7	82.7	24.1	82.9	24.2	83.6	25.1
	COMETH	78.0	56.4	81.1	68.9	84.1	74.7	85.6	76.8	86.1	74.8
7	OpenPTrack [7]	78.1	43.6	79.7	37.1	80.0	27.7	80.0	21.5	79.3	17.0
	Befine[10]	80.2	17.5	84.3	17.2	85.7	16.5	86.1	15.9	86.2	15.5
	COMETH	81.1	58.8	84.5	75.2	87.2	82.1	88.5	84.8	89.2	85.8

Equations

$$W_i[k,j]={\kappa }_2(||X_i[k]-\text{FK}(B_j)|{|}_2)$$

$$\underset{\dot{q},\delta }{\mathrm{argmin}}{\dot{q}}^T\Lambda \dot{q}+{\delta }^{T}D\delta$$

$$\text{s.t.}\{\begin{array}{l}x+J(q)\dot{q}=x_T+\delta \\ -\gamma (q-q_L)\le \dot{q}\le \gamma (q_U-q)\\ {\dot{q}}_L-{\dot{q}}_0\le \dot{q}\le {\dot{q}}_U-{\dot{q}}_0\end{array}$$

$$\underset{\dot{q},{\delta }_0,\dots ,{\delta }_n}{\mathrm{argmin}}{\dot{q}}^T\Lambda \dot{q}+\sum _{i=1}^n{\delta }_i^{T}D{\delta }_i$$

$$\text{s.t.}\{\begin{array}{l}\forall i\in [1,n]:x_i+J_i(q)\dot{q}=x_{\text{Ti}}+{\delta }_i\\ -\gamma (q-q_L)\le \dot{q}\le \gamma (q_U-q)\\ {\dot{q}}_L-{\dot{q}}_0\le \dot{q}\le {\dot{q}}_U-{\dot{q}}_0\end{array}$$

$$Q_i$$

$$Q_i^{-}[t]=F{Q}_i^{+}[t-1]$$

$$P_i^{-}[t]=F{P}_i^{+}[t-1]F^T+\Sigma$$

$$K_i[t]=P_i^{-}[t]H^T(H{P}_i^{-}[t]H^T+R{)}^{-1}$$

$$Q_i^{+}[t]=Q_i^{-}[t]\cdot K_i[t](z_i[t]-H{Q}_i^{-}[t])$$

$$P_i^{+}[t]=(I-K_i[t]H)P_i^{-}[t]$$

$$\mathcal{S}(\text{pr},\text{gt})=\max \left(0,1-\frac{1}{|\text{kp}|}\sum _{i\in \text{kp}}\Vert {\text{pr}}_i-{\text{gt}}_i\Vert \right)$$

$$\text{DetA}={\int }_0^1\frac{|T{P}_{\alpha }|}{|T{P}_{\alpha }|+|F{N}_{\alpha }|+|F{P}_{\alpha }|}d\alpha$$

$$\text{LocA}={\int }_0^1\frac{1}{|T{P}_{\alpha }|}\sum _{c\in T{P}_{\alpha }}S(c,\text{gt})d\alpha$$

$$\mathcal{A}(c)=\frac{|TPA(c)|}{|TPA(c)|+|FNA(c)|+|FPA(c)|}$$

$$\text{AssA}={\int }_0^1\frac{1}{|T{P}_{\alpha }|}\sum _{c\in T{P}_{\alpha }}\mathcal{A}(c)d\alpha$$

$$\text{HOTA}={\int }_0^1\sqrt{{\text{DetA}}_{\alpha }\cdot {\text{AssA}}_{\alpha }}d\alpha$$

Full text

††journal: Information Fusion

\affiliation

organization=Department of Engineering for Innovation Medicine,addressline=University of Verona, city=Verona, state=Italy

\affiliation

organization=GRASP Laboratory, Department of Mechanical Engineering and Applied Mechanics,addressline=University of Pennsylvania, city=Philadelphia, state=Pennsylvania, country=USA

1 Introduction

The emergence of Industry 5.0 has accelerated the integration of non-contact IoT (Internet of Things) devices in industrial environments, enabling smarter, safer, and more efficient workplaces [1]. Human pose tracking is crucial in this transformation, enhancing ergonomic assessments, safety monitoring, and human-robot collaboration [2]. By accurately analyzing workers’ movements, industries can optimize workflows, prevent injuries, and ensure compliance with safety regulations [3].

Achieving reliable 3D human pose tracking presents significant technical challenges, particularly regarding accuracy, computational efficiency, and scalability [4]. Traditional multi-camera centralized setups reach high-accuracy 3D human pose estimation (HPE) by fusing information from multiple viewpoints. However, these methods are computationally expensive and require high-bandwidth network transmission, limiting their real-time applicability and scalability in large industrial environments. An alternative approach is edge computing, by which the overall processing is distributed across multiple devices to reduce network congestion and computational load [5]. While this strategy alleviates processing and network limitations, edge devices’ limited computational power and memory capacity often reduce model accuracy compared to more powerful centralized systems. Achieving a balance between efficiency and precision remains a key challenge in deploying scalable distributed HPE systems [6].

A variety of pose fusion algorithms have been developed to effectively tackle these challenges, integrating multiple sources of pose data to enhance accuracy and robustness. Carraro et al. [7] proposed a fusion algorithm using multiple Unscented Kalman Filters [8], each tracking a different joint for every person in the scene. The system uses the 2D skeletons from OpenPose [9] and projects the result in 3D using depth data. The system leverages a data association algorithm to match new detections with existing tracks, handle new tracks, or remove old ones. Boldo et al. [10] proposed BeFine, a distributed platform designed to perform Human Pose Estimation (HPE) in industrial settings. Their fusion module uses clustering and temporal association algorithms to provide a unique 3D skeleton for each subject in the scene. While these methods have been widely used, they do not provide guarantees for the accuracy of the fused human poses, as they are not built upon a human model.

We address this challenge by proposing COMETH (Convex Optimization for Multiview Estimation and Tracking of Humans), a novel and lightweight algorithm for multi-view human pose fusion for real-time, resource-constrained environments. COMETH is built upon three core concepts to enhance human pose estimation and tracking accuracy and consistency. First, it incorporates kinematic and biomechanical constraints to ensure anatomically plausible and physically consistent joint positioning, even in noisy or partial observations targeting low-power devices. Second, it leverages convex quadratic programming inverse kinematics (QPIK) to fuse pose estimates from multiple camera perspectives into a unified skeletal representation. Third, it implements a state observer, improving temporal stability and reducing jitter and drift across frames.

We evaluate our approach on the publicly available CMU Panoptic dataset [11] and real-world scenarios, demonstrating improved tracking accuracy and efficiency compared to state-of-the-art methods. Our findings suggest that COMETH is well-suited for real-time applications, including industrial monitoring and interactive systems, where accurate and efficient 3D human pose tracking is essential.

The article is organized as follows. Section 2 presents an analysis of the related work. Section 3 presents COMETH. Section 4 presents the experimental settings. Section 5 reports the experimental results, while Section 6 discusses the conclusions and future directions.

2 Background

2.1 3D Human Pose Tracking

Marker-based (MB) motion capture systems are widely employed in motion analysis for research and commercial use [12]. They detect physical markers attached to the subject’s body using specialized cameras. Although highly accurate, these systems have significant limitations, including high costs and the need for controlled environments. Marker-less (ML) motion capture systems overcome many challenges by being non-intrusive and operating in diverse environments without specialized hardware [13]. HPE is a computer vision-based ML technology that localizes the spatial positions of anatomical keypoints (e.g., elbows, knees, and wrists) of the human body from images or videos [14]. Early approaches employed background subtraction and edge detection [15], while state-of-the-art approaches predominantly utilize Deep Learning (DL) approaches [16].

HPE has limitations, including external occlusions and self-occluded poses, that can result in incorrect or missing key points [17]. Several methodologies utilize multi-camera setups to overcome limitations and cover a broader space. Generally, each camera captures input frames and transmits them to a centralized processing module that estimates the 3D pose representation of the subjects [18]. Although multi-view HPE mitigate some challenges of single-view methods, they have some limitations. They are resource-intensive, requiring substantial computational power to process and triangulate all the incoming streaming cameras in real-time. Additionally, network latency and data transfer bottlenecks can further affect the applicability and scalability of those solutions.

2.2 Digital Human Representation

Digital human modeling is critical for HPE as it provides a structured representation of the human body. These models enable more precise estimations of joint locations and body poses, correcting the keypoints detected through HPE.

Kinematic models represent the human body as a series of linked segments and joints, usually following the Denavit-Hartenberg convention [19]. This method is widely used in human motion analysis and humanoid robots [20] as it constrains the 3D joints’ position in space at a fixed distance from each other in correspondence with the bones. Xu et al. [6] modeled the human body as a combination of joints and limbs to limit the drift errors of wearable sensors. They used a geometrical constrained method to improve human motion, describing the forward kinematics structure with the Denavit–Hartenberg convention.

Surface models use deformable meshes based on shape and pose parameters to represent the body. The most popular surface body model is Skinned Multi-Person Linear Model (SMPL) [21]. It features a fixed topology of 6,890 vertices and 23 joints, enabling realistic animation and deformation. Owing to its efficacy and adaptability, SMPL is widely employed in computer vision, graphics, and applications such as motion capture, virtual reality, and human-centric artificial intelligence. However, it requires accurate parameter estimation, posing challenging computational costs in noisy scenarios. Furthermore, it fails to represent body structure accurately because it treats each joint as a ball joint with three angular degrees of freedom.

Biomechanical models are effective representations for simulating and analyzing human movement, dynamics, and forces [22]. Werling et al. [23] presented AddBiomechanics, a method to extract and standardize human movement dynamics. Given a motion capture dataset, the tool scales the body segments of a musculoskeletal model, registers marker locations, and computes body segment kinematics. The accuracy of these approaches heavily depends on the quality of the input data from the HPE, which may contain noise, occlusions, or scaling ambiguities, resulting in less reliable biomechanical predictions.

Hybrid models combine surface and biomechanical models, providing visual and biomechanical accuracy. Keller et al. [24] proposed SKEL, a combination of the SMPL body model with a biomechanical accurate skeleton. They fit the Biomechanical Skeleton Model (BSM) inside SMPL meshes of the AMASS dataset [25] and then trained a regressor from SMPL mesh vertices to the optimized joint locations, leading to a more biomechanically realistic representation of bone angles.

2.3 Inverse Kinematics

Inverse kinematics (IK) is a mathematical technique determining the joint-space configuration required for a manipulator to reach a desired end position [26]. Unlike forward kinematics (FK), which computes the end-effector’s position from given joint angles, IK works in reverse, solving for joint configurations based on target positions. For highly redundant manipulators, IK often involves optimization techniques such as gradient descent or Jacobian pseudoinverse algorithms. Convex optimization, a field of mathematical optimization, focuses on minimizing convex functions within convex sets. A specific type of convex optimization, quadratic programming (QP), seeks to minimize a quadratic function while satisfying linear constraints [27]. In convex QP, any local minimum is also the global minimum. Quadratic Programming Inverse Kinematics (QPIK) is an advanced IK method that formulates the IK problem as a QP problem. It is commonly used in robotics to enforce additional constraints, such as joint limits, balance, and collision avoidance [20].

3 COMETH

COMETH (Convex Optimization for Multiview Estimation and Tracking of Humans) is an aggregation algorithm that fuses information from multiple edge devices. At each time frame, the devices send sets of keypoints representing the 3D coordinates of human joints if they detect people in the scene. The aggregator has two steps: first, it synchronizes incoming detections in space and time and pairs them to existing bodies; second, it updates the positions of these bodies based on the associated measurements. Figure 1 shows an overview of the proposed fusion method, with each block described in detail over the following sections.

3.1 Pre-processing

At time $t$, each edge device may send a set of measurements $X$ over the network if a person is captured, where $|X|$ is the number of people detected by the HPE. Each measurement $x\in X$ comprises a set of labeled 3D points representing human joints, called keypoints. Since different edge devices with varying computing capabilities may be deployed in the scene, the information may arrive with various latencies and frequencies. When COMETH receives new measurements, they are stored in an ordered queue. At each iteration of the aggregator, measurements are removed from the queue if considered too old, using a time duration threshold $\varDelta$. Given the aggregator timestamp $t_{a}$, a set of measurements $X_{n}$ captured by the device $n$ at time $t_{X_{n}}$, if $t_{a}-t_{X}<\varDelta$ then $X$ is considered synchronized and ready to be fused. Otherwise, it is discarded.

The pre-processing function then analyzes all the selected measurements, removing outliers and false detections caused by the HPE or the depth sensor. This function eliminates keypoints that are too distant from the camera. This step is essential because even the most advanced depth sensors exhibit inaccuracies at greater distances from the target.

3.2 Association

All measurements with at least $k$ valid keypoints are fed into the association weights function to be associated with their body models.

Each body model $b\in B$ is derived from BSM [24], a digital human representation with 49 degrees of freedom (DOFs). The first 6 DOFs represent the absolute position and orientation in 3D space, while the remaining 43 correspond to anatomically accurate body joints connecting 24 rigid bone groups.

We use the Hungarian algorithm for each edge device $i\in[0,n)$ to find an assignment between the body models $B$ and the measurements $X_{i}$. Given a $|B|\times|X_{i}|$ cost matrix $W_{i}\in\mathbb{N}^{+}$, the weight matrix is computed as follows:

[TABLE]

where $\text{FK}(\cdot)$ is the forward kinematics of the body model, and $\kappa_{2}(\cdot)$ is a function that selects the second smallest element from a set. We select the second closest value between the two sets of 3D points based on their Euclidean distances, as the smallest distance could be an outlier.

Then, for each body model $b\in B$, we associate a set of measurements $X_{b}$ that might belong to the subject represented by $b$. Given that there may be at most one detection per subject for each edge device $i\in[0,n)$, at the end of the association process, each $b\in B$ may have a set of measurements $X_{b}$, where $|X_{b}|\in[0,n)$. After the association process, we create a new body model for each unassociated set of measurements.

3.3 Update

During this phase, COMETH updates each body’s information based on the associated measurements. We update the bones’ scale for each body, removing the incompatible keypoints. Next, we retrieve all joint displacement values through QPIK. A state observer then filters the QPIK results for temporally smoothed values. Finally, forward kinematics accurately reconstructs the poses in 3D space.

3.3.1 Body Scaling and Outlier Removal

For each body model $b\in B$, we update the scale of the bones. Scaling the bones of the body model to fit input data is a crucial step to ensure that the model accurately reflects the subject’s specific dimensions and geometry. Estimating body scale from clean input using an optimization algorithm is straightforward; however, handling partial or noisy data presents challenges. First, we begin by estimating the subject’s height. Since segment lengths have a well-defined relationship with overall human height [28, 29], we use them to approximate the subject’s height based on the inter-joint distances, as in [30]. Each distance between two physically connected joints is used to estimate height, and we compute a reasonable height by averaging all possible estimations. The average scaling factor, $s_{AVG}$, is derived from this value. To account for individual anatomical variations, we apply different scaling factors to each bone rather than a uniform scale. For instance, two people of the same height may have significantly different limb lengths, affecting joint positioning and movement patterns. Thus, for each distance between adjacent keypoints (e.g., knee and ankle), we compute the corresponding body part’s scaling factor and clip it within $\pm 5\%$ of $s_{AVG}$. For body parts that cannot be directly inferred from keypoints, we use $s_{AVG}$. Then, we remove outliers from the measurements $X_{b}$, excluding all keypoints that form bones with lengths incompatible with the bones of $b$. We store previous scaling factors and compute their average over time to ensure consistency in representing the same subject across frames. Figure 2 depicts the body scaling process and the removal of the outliers.

3.3.2 Multi-Source Inverse Kinematics

In traditional robotics IK problems, the objective is to determine a joint configuration that positions the end effector at a desired target location with a specific orientation. To determine the joint configuration of a human model from HPE data, each joint $i$ has its own positional target keypoint $kp_{i}$. Given a set of 3D points $x_{T}\in\mathbb{R}^{3|kp|}$, where $|kp|$ is the number of keypoints detected, we propose the following iterative QP formulation to solve the IK:

[TABLE]

[TABLE]

where $q\in\mathbb{R}^{49}$ is the angular displacement, $\dot{q}\in\mathbb{R}^{49}$ is the angular velocity, $\delta\in\mathbb{R}^{3|kp|}$ is a slack variable, $D\in\mathbb{R}^{3|kp|\times 3|kp|}_{++}$ is a symmetric positive definite matrix that sets weights for the slack variable $\delta$, and $\Lambda\in\mathbb{R}^{49\times 49}_{++}$ is a symmetric positive definite matrix equivalent to the damping term in least-squares IK methods. Equation 3 describes the three constraints to obtain a realistic configuration for the human model. The first one represents the linear approximation of the relationship between the Cartesian position and the angular displacement of each joint. The current position $x\in\mathbb{R}^{3|kp|}$, when updated by the motion determined by the product of the Jacobian matrix $J\in\mathbb{R}^{3|kp|\times 49}$ and joint velocities $\dot{q}\in\mathbb{R}^{49}$, should match the target position $x_{T}$ with a small error $\delta$.

The second constraint defines dynamic limitation on joint velocities to keep the joints within their physical range. It guarantees that the speed of a joint is appropriately reduced as it approaches its limits. A limited Range of Motion (ROM) ensures that our model accurately reflects human anatomical constraints. We define the lower limits $q_{L}\in\mathbb{R}^{49}$ and upper limits $q_{U}\in\mathbb{R}^{49}$ for each joint position based on normative data from the biomechanics literature. For instance, unlike BSM, COMETH restricts the elbow flexion to a range between -11° and 154°, following the normative data presented in [31]. This range accounts for physiological variations while preventing unrealistic joint movements and reducing the search space for the inverse kinematics solver. The scaling factor $\gamma\in(0,1]$ controls how aggressively the joint approaches its limits, acting as a barrier function [32].

In addition to joint positions, we impose velocity constraints. Joint velocity limitations arise from several anatomical factors, including muscle contraction speed and tendon elasticity. These constraints vary across joints; for example, the wrist can execute rapid small movements, while the knee moves more slowly due to greater mass and structural constraints. Velocity limits vary across individuals, influenced by flexibility, muscle strength, and training history. Since the biomechanics literature on every joint velocity is limited, we regress them from the BioAMASS dataset [24]. This comprehensive motion capture dataset extends the AMASS dataset [25], focusing on biomechanical applications. It includes high-quality kinematic and dynamic recordings of human motion from individuals with diverse anthropometric characteristics, including variations in age, gender, and physical condition. BioAMASS comes with BSM joint positions as pseudo-ground truth, so we apply the same scaling from BSM to each subject in COMETH. To align the COMETH body model with the ground truth, we employ an inverse kinematics solver based on the Levenberg-Marquardt algorithm [33]. For each sequence in the dataset, we extract instantaneous joint velocities. The lower bound of angular velocity $\dot{q}_{L}\in\mathbb{R}^{49}$ is determined using the 5th percentile, while the upper bound $\dot{q}_{U}\in\mathbb{R}^{49}$ is set using the 95th percentile. The third constraint sets dynamic bounds on the change in the final joint velocity, represented as $\dot{q}_{0}+\dot{q}$. Here, $\dot{q}_{0}$ denotes the aggregated velocity over a single timestep, accounting for multiple iterations of the QPIK solver. Considering the current state, this constraint ensures that the resulting velocity stays within the limits. Both ROM and velocity constraints are reported for each joint angle in Table 1.

Multi-source QPIK

When solving the IK with multiple targets, there may be conflicting constraints, requiring strategies such as constraint relaxation [27]. To determine the joint configuration of a human model from multiple HPE devices, each joint $i$ may have one or more target positions. We formulate a QPIK that considers mutiple sets of 3D points $x_{Ti}$ coming from $n$ sources as follows:

[TABLE]

[TABLE]

Differently from Equation 2, in 4 there are $n$ slack variables, one per input source. This formulation minimizes the difference between the final position of each joint and the targets. In Equation 5, we extend the first constraint from Equation 3 to handle multiple positional targets. The other two constraints remain as in Equation 3, since they do not depend on the number of target sources.

3.3.3 State Observer

When dealing with multiple streams of targets in the proposed human model, applying QPIK at each time step may lead to inconsistent solutions due to measurement errors. To deal with these limitations, we integrate QPIK into a state observer to ensure stable and realistic motion reconstruction. A state observer helps manage sensor noise, predict unmeasured joint states, and enforce temporal consistency. We implement a linear Kalman Filter (KF) [34], a popular state observer for estimating the state of a dynamic linear system in the presence of noise and uncertainty. For each joint $i$, we formulate the state $Q_{i}\in Q$, the second-order state transition matrix $F$, and the output matrix $H$ as:

[TABLE]

where $q_{i}$ is the positional value of joint angle $i$, $\dot{q}_{i}$ its velocity and $\ddot{q}_{i}$ its acceleration.

State observers work through two phases: prediction and correction. In the prediction phase, the current state estimates $X$ are projected forward in time, anticipating how the state will evolve. At time $t$, the predicted state $X^{-}_{i}[t]$ is obtained by:

[TABLE]

where $Q_{i}^{+}[t-1]$ is the state of the joint angle $i$ at time $t-1$. In this phase, the covariance estimate $P_{i}$ is also calculated, which propagates uncertainty over time:

[TABLE]

where $\varSigma$ represents the covariance matrix of the process noise. The prediction accuracy forms the basis for subsequent adjustments made during the correction phase, where the values are refined using real-time measurements.

In the correction phase, the filter computes the Kalman gain $K_{i}[t]$, determining how much the measurement should influence the estimate based on its reliability:

[TABLE]

where $R$ is the covariance matrix of the measurement noise. The updated state $Q^{+}_{i}[t]$ is then obtained by adjusting the predicted state $Q^{-}_{i}[t]$ with the difference between the result of the QPIK $z_{i}[t]$ and $Q^{-}_{i}[t]$:

[TABLE]

The covariance matrix is then updated to reduce the uncertainty after incorporating the new information:

[TABLE]

This method stabilizes the estimated state of each of the 49 joint angles over time.

4 Experimental Setup

4.1 Datasets

We quantitatively validated the proposed fusion method on the CMU Panoptic Dataset [11]. It includes multiple individuals engaged in social interactions, captured with 480 VGA cameras, 30 HD cameras, and 10 RGB-D Kinect v2 sensors. Our experiments used all sequences with valid ground truth and a stream of 5 RGB-D cameras at 30 Hz. We selected the upper ones as they more closely resemble surveillance cameras than those positioned at a lower height. Depending on the sequence, there may be at the same time 2,3, or 7 actors in the scene.

We also present the qualitative results of our method deployed in the ICE Laboratory [35], which is a modern, compact, and realistic production line environment. The setup includes 9 RGB cameras and 7 RGB-D cameras. We employed 3 StereoLabs ZED2 RGB-D cameras111https://www.stereolabs.com/zed-2/ at 15 Hz, each paired with an Nvidia Jetson Xavier NX222https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx running an HPE node.

4.2 Evaluation metrics

We quantitatively evaluated the proposed method using four metrics from the multiple objects tracking (MOT) literature [36]. First, we define a similarity score $\mathcal{S}$ between a predicted skeleton (pr) and its ground truth skeleton (gt) as:

[TABLE]

where kp is the common set of detected keypoints. Following this notation, when $\mathcal{S}$ is 1, the skeletons perfectly align, and when S is 0, there is no overlap between detections [36]. We perform a bijective matching between each pr and gt, utilizing the Hungarian algorithm [37]. Given a similarity threshold $\alpha\in(0,1)$, each matched pair is considered true positive ($TP$) if $\mathcal{S}(\text{{pr}},\text{{gt}})>\alpha$. Any gt that is not matched is a false negative ($FN$), while any pr that is not matched is a false positive ($FP$). The detection accuracy score (DetA) is defined as the standard Jaccard index:

[TABLE]

It measures how accurately the tracker detects skeletons in each frame over time.

The localization accuracy score (LocA) is calculated as follows:

[TABLE]

It measures how accurately the predicted keypoints positions match the ground truth locations.

When considering the ID association, each prediction pr is considered a True Positive Association (TPA) if pr is in the $TP$ set and has the same id as its gt. The set of False Negative Associations (FNAs) is the set of gt with the same id as pr but either wrongly associated or not associated with any pr. The set of False Positive Associations (FPAs) is the set of pr with the same ID as gt but either wrongly associated or not associated with any gt. Given a true positive candidate $c\in TP$, we define the association score $\mathcal{A}$ as:

[TABLE]

where $\mathcal{A}$ measures the alignment between gt and pr trajectories that are matched at the $TP$ $c$, calculated using the Jaccard index. The association accuracy score (AssA) is then defined as:

[TABLE]

It measures how accurately a tracking algorithm maintains the correct associations between detections across frames.

Higher Order Tracking Accuracy (HOTA) is the geometric mean of detection and association scores:

[TABLE]

HOTA evaluates how well tracking algorithms perform by measuring their accuracy in localization and association.

4.3 Compared methods and baselines

To maximize compatibility with most HPE methods, we consider the common 12 keypoints (shoulders, hips, elbows, wrists, knees, and ankles) as input data. We use TRTPose 333https://github.com/NVIDIA-AI-IOT/trt_pose as a baseline for evaluating the Panoptic dataset and OpenPose [9] for the real-case study. In both cases, we employ the 2D HPE to extract 2D keypoints representing human body joints from the RGB video stream. Then, we use the depth matrix to determine each keypoint’s 3D distance from the camera. To obtain the 3D coordinates of each keypoint, we back-project the 2D keypoints into 3D space. The final output is a set of 3D keypoints for each video frame, timestamped, representing the body joints of each detected person in the scene.

We compare the results of our method with two fusion algorithms that take as input 3D skeletons. The first one is OpenPTrack, proposed by Carraro et al. [7], which combines the Hungarian algorithm and multiple Unscented Kalman Filters to associate and track the subjects. The second method, called Befine, proposed by Boldo et al. [10], employs spatial clustering and specific temporal association algorithms.

4.4 Implementation details

We implement COMETH upon Nimble [38], a differentiable physics engine derived from DART [39], designed explicitly for articulated rigid body simulation. We set all the parameters once and kept the same values throughout the experimental results. To solve the QPIK formulation, we utilize cvxpy [40], a domain-specific language for convex optimization problems that generates efficient C codes tailored for the specific formulation of the optimization problem. All experiments are run on a desktop setup (AMD Ryzen 9 7950X, 64GB RAM DDR5, Nvidia RTX 4090), while the HPE runs on multiple Nvidia Jetson Xavier NX that act as edge devices.

We set all COMETH parameters through grid search, keeping the same values throughout the experiments in both case studies. In detail, in the pre-processing phase of the aggregator (Section 3.1), we set the duration timeout $\varDelta$ to 0.07 seconds. Regarding the multi-source QPIK (Section 3.3.2), we set the maximum number of iterations to 100. Both matrices $\Lambda$ and $D$ are set to identity matrices, and the scaling factor $\gamma$ is set to 1. In the state observer (Section 3.3.3), the covariance matrix $Q$ is set to a diagonal matrix with diagonal elements equal to 0.5, and the covariance matrix $R$ is set to the identity matrix.

5 Experimental Results

5.1 Analysis on CMU Panoptic Dataset

Table 2 presents the performance comparison of COMETH against the baseline approaches across different numbers of camera views. With a single camera, our approach reaches a LocA of 78.4%, which increases to 87.4% when using five cameras. For a lower number of cameras, the LocA of COMETH is similar or superior to that of the others. When using 2 cameras, Befine performs negligibly better (0.9%), because of its model-free clustering method that preserves better the initial detections of the HPE. However, compared to the other methods, the robustness of COMETH in correctly localizing human poses increases consistently as the number of cameras increases. Similarly, our approach shows superior DetA performance, starting at 58.9% for a single camera and reaching 69.3% for five cameras. This indicates a higher reliability in detecting individuals accurately compared to prior methods. The AssA score of our method shows significant improvement over competing approaches, particularly as the number of cameras increases. The AssA score rises from 56.1% with one camera to 86.7% with five cameras, demonstrating more robust multi-view association capabilities than the other methods. COMETH consistently achieves the best HOTA score, reaching 76.7% with five cameras. This metric reflects the overall tracking performance and confirms the superiority of COMETH in maintaining identity consistency over time.

Table 3 explores the effect of varying the number of tracked subjects while keeping different camera configurations constant. Our approach maintains high LocA and HOTA scores as the number of subjects increases. Even with seven subjects and five cameras, our method achieves a LocA of 89.2% and a HOTA of 85.8%, highlighting its robustness in crowded scenes. While OpenPTrack [7] experiences a decline in accuracy as the number of subjects increases, and Befine [10] struggles with association, COMETH remains reliable across all conditions.

5.2 Real-World Scenario

Figure 3 shows the fusion results of the proposed method on three scenes recorded in the ICE Laboratory [35]. We show both Befine [10] and our method’s results, where each row represents a separate moment. The detected joints of the HPEs are displayed in a color scale: lighter colors indicate keypoints where the HPE has lower confidence, while darker colors represent those with higher confidence. In the first scene, subject C is fully seen by Cam #1 and partially seen by Cam #3. The 3D HPE of Cam #2 wrongly detects the elbow of C on the surface of the robotic arm. Our aggregator’s outlier removal block effectively excludes this incorrect detection. In contrast, Befine [10] fails to correct the keypoint, resulting in an unrealistic configuration. In the second scene, the lower limbs of subject A are only partially seen by Cam #1. From only the upper body keypoints, our method accurately reconstructs the information about the ankles and knees. In contrast, the clustering algorithm of Befine assigns some keypoints of subject B to subject A. In the third scene, the self-occluding position of subject A poses a significant challenge. Even though the information is noisy and incomplete, and the resulting joint configurations are inaccurate from both aggregators, our method correctly localizes the subject’s position in the laboratory.

Figure 4 compares the association accuracy of the tracking methods using a top-down perspective of the ICE Laboratory. In 4(a), we manually annotated from video the reference trajectories of three subjects (A, B, and C), indicating expected movement paths. The other subfigures represent the results of the fusion methodologies. OpenPTrack [7] exhibits highly fragmented trajectories, with numerous abrupt directional changes and noisy patterns, suggesting significant association errors. Befine [10] also displays inconsistent tracking with a dense concentration of short, jittery trajectories, indicating frequent identity reassignments. Our method demonstrates a more stable and coherent tracking output, where trajectories closely resemble the reference traces, highlighting the robustness of the association algorithm. We uploaded the complete video comparisons as supplementary materials.

5.3 Performance Analysis

The main purpose of COMETH is to operate in real-time, so our goal was to ensure that the entire pipeline executes in less than 33 ms (i.e., $>$ 30 Hz). Latencies highly depend on the number of incoming measurements, the number of cameras, and the subjects in the scene. Figure 5 shows latencies of the compared aggregators when processing data from five edge devices, involving up to seven subjects. While OpenPTrack [7] is linearly dependent on the number of incoming measurements, both Befine [10] and our aggregator’s latency do not appear to depend on that.

6 Conclusion and Future Work

In this work, we proposed a fast, privacy-aware, multi-view 3D human pose tracking system that effectively integrates multiple camera sources to enhance tracking accuracy. Our approach significantly outperforms state-of-the-art methods in localization, detection, and tracking metrics. The system’s ability to robustly track multiple individuals across camera setups highlights its potential for real-world applications.

Future work will explore incorporating additional sensor modalities, such as IMUs, mmWave, or LiDAR, which could improve accuracy under challenging conditions.

Acknowledgments

This study was conducted within the MICS (Made in Italy – Circular and Sustainable) Extended Partnership and received funding from Next-Generation EU (Italian PNRR – M4 C2, Invest 1.3 – D.D. 1551.11-10-2022, PE00000004). CUP MICS D43C22003120001 - Cascade funding project CollaborICE. This work has also been supported by the “PREPARE” project (n. F/310130/05/X56 - CUP: B39J23001730005) - D.M. MiSE 31/12/2021; and the National Science Foundation (NSF) Foundational Research in Robotics (FRR) program under NSF CAREER Award Grant No. FRR-2443721.