Multi-View 3D Point Tracking

TL;DR

This paper presents the first data-driven multi-view 3D point tracker that accurately tracks points in dynamic scenes using a practical number of cameras, leveraging a novel neural network architecture and synthetic training data.

Contribution

Introduces a new multi-view 3D point tracking method that uses a feed-forward neural network with multi-view features and transformers, trained on synthetic data, for robust real-world application.

Findings

Achieves median trajectory errors of 3.1 cm and 2.0 cm on benchmarks.

Generalizes to 1-8 camera views with varying vantage points.

Operates effectively with video lengths of 24-150 frames.

Abstract

We introduce the first data-driven multi-view 3D point tracker, designed to track arbitrary points in dynamic scenes using multiple camera views. Unlike existing monocular trackers, which struggle with depth ambiguities and occlusion, or prior multi-camera methods that require over 20 cameras and tedious per-sequence optimization, our feed-forward model directly predicts 3D correspondences using a practical number of cameras (e.g., four), enabling robust and accurate online tracking. Given known camera poses and either sensor-based or estimated multi-view depth, our tracker fuses multi-view features into a unified point cloud and applies k-nearest-neighbors correlation alongside a transformer-based update to reliably estimate long-range 3D correspondences, even under occlusion. We train on 5K synthetic multi-view Kubric sequences and evaluate on two real-world benchmarks: Panoptic Studio and DexYCB, achieving median trajectory errors of 3.1 cm and 2.0 cm, respectively. Our method generalizes well to diverse camera setups of 1-8 views with varying vantage points and video lengths of 24-150 frames. By releasing our tracker alongside training and evaluation datasets, we aim to set a new standard for multi-view 3D tracking research and provide a practical tool for real-world applications. Project page available at https://ethz-vlg.github.io/mvtracker.

Tables

Table 1. Table 1 : Quantitative evaluation of multi-view 3D point tracking performance. We report results on Panoptic Studio  [ 18 ] , DexYCB  [ 3 ] , and Multi-View Kubric  [ 11 ] . Our method outperforms existing approaches across all datasets, achieving the highest accuracy ( δ avg \delta_{\text{avg}} ) and occlusion-aware tracking performance (AJ), while maintaining lower median trajectory error (MTE). Notably, our model surpasses the triplane-based baseline, indicating the effectiveness of kNN-directed correlation for multi-view tracking. We report MTE in centimeters. F denotes that the method is feed-forward, data-driven, and online. M denotes that the method supports fusing multi-view inputs. We use estimated depth  [ 37 ] for DexYCB, ground-truth optimization-based depth  [ 23 ] for Panoptic Studio, and simulated depth for Kubric.

Method	Train	F	M		Panoptic Studio [18]					DexYCB [3] (DUSt3R depth)					Multi-View Kubric [11]
					AJ $\uparrow$	${\delta }_{\text{avg}}$ $\uparrow$	OA $\uparrow$	MTE $\downarrow$		AJ $\uparrow$	${\delta }_{\text{avg}}$ $\uparrow$	OA $\uparrow$	MTE $\downarrow$		AJ $\uparrow$	${\delta }_{\text{avg}}$ $\uparrow$	OA $\uparrow$	MTE $\downarrow$
Dynamic 3DGS [23]	✗	✗	✓		66.5	92.4	74.1	3.9		45.7	57.1	81.3	11.3		30.4	48.6	68.3	11.2
Shape of Motion [35]	✗	✗	✓		72.6	89.2	82.1	4.8		36.2	53.6	63.3	8.0		57.8	63.4	86.2	5.3
SceneTracker [32]	LSFO	✓	✗		–	85.1	–	6.3		–	61.1	–	5.7		–	56.9	–	7.8
LocoTrack [5]	Kub	✓	✗		65.8	79.4	79.6	11.8		27.8	38.6	77.0	22.8		52.5	58.4	76.5	12.9
DELTA [24]	Kub	✓	✗		68.1	86.3	79.2	5.9		36.8	51.6	61.0	18.3		57.4	68.4	77.9	9.6
CoTracker2 [16]	Kub	✓	✗		69.5	83.2	80.7	10.4		28.8	40.5	76.2	20.8		54.6	60.3	79.8	10.4
CoTracker3 [17]	Kub+15k	✓	✗		74.5	84.5	86.4	8.6		29.4	40.3	78.6	22.0		55.1	60.6	77.7	11.9
SpaTracker [38]	Kub	✓	✗		55.5	70.8	81.7	9.9		48.1	62.6	73.9	5.5		43.6	55.1	76.3	5.1
SpaTracker [38]	MV-Kub	✓	✗		61.5	82.3	75.5	7.3		58.3	72.0	80.2	5.9		65.5	77.6	83.1	2.2
SpaTrackerV2 [39]	See [39]	✓	✗		75.3	85.1	91.2	6.9		35.5	45.1	91.3	9.8		58.6	69.3	86.3	3.9
TAPIP3D [43]	Kub	✓	✗		84.3	93.8	91.4	3.1		38.8	50.0	90.1	8.2		72.4	86.5	85.4	1.3
Triplane Baseline	MV-Kub	✓	✓		65.1	81.8	82.8	7.2		57.5	71.0	81.3	4.3		74.7	85.2	90.4	1.2
MVTracker (ours)	MV-Kub	✓	✓		86.0	94.7	92.3	3.1		71.6	80.6	91.3	2.0		81.4	90.0	93.7	0.7

Table 2. Table 2 : Point Correlation Components. “No offset” omits the offset vector entirely, “Offset + location” concatenates both the offset and the neighbor’s world-space coordinates, and “Offset only” encodes just the relative direction. Including only the offset vector (third row) yields the best overall performance. These experiments were trained for 25% of the total steps and on 8 GPUs.

	Multi-View Kubric [11]
	AJ $\uparrow$	${\delta }_{\text{avg}}$ $\uparrow$	OA $\uparrow$	MTE $\downarrow$
No offset	21.3	45.3	40.6	15.6
Offset + location	48.7	59.6	68.5	6.8
Offset only	53.6	64.9	73.4	4.3

Table 3. Table 3 : Impact of Camera Setups. Evaluation of AJ under varied camera setups. For Panoptic Studio  [ 18 ] , we select 4 out of the available 27 views as opposing views (setup A) or nearby views (setups B and C). For DexYCB  [ 3 ] , we form 4-view sets out of the available 8 views: (A) views 1–4, (B) views 3–6, (C) views 5–8.

Method	PStudio [18]				DexYCB [3]
	A	B	C		A	B	C
Dynamic 3DGS [23]	66.5	50.8	56.6		45.7	–	–
Shape of Motion [35]	72.6	64.3	66.8		36.2	–	–
LocoTrack [5]	65.8	57.9	63.7		27.8	40.9	42.9
DELTA [24]	68.1	61.1	65.9		36.5	43.3	47.6
CoTracker2 [16]	69.5	62.3	66.4		28.8	42.0	44.4
CoTracker3 [17]	74.5	66.3	70.9		29.4	43.8	46.3
SpaTracker [38]	61.5	54.8	57.8		58.3	57.9	63.8
Triplane Baseline	65.1	59.9	63.5		57.5	62.0	66.3
MVTracker (ours)	86.0	75.7	83.2		71.0	71.2	78.3

Table 4. Table A.1 : AJ for different depth sources. GT*: optimization-based depth. D: DUSt3R. V: VGGT. S: Sensor. ✗: depth estimation failed.

Method	Panoptic Studio			DexYCB			MV-Kubric
	GT*	D	V	S	D	V	GT	D	V
DELTA	68.1	✗	1.8	54.6	36.8	21.1	57.4	25.7	1.9
SpaTracker	61.5	✗	10.6	60.9	58.3	33.2	65.5	61.7	3.9
Triplane Baseline	65.1	✗	29.7	68.0	57.5	61.8	74.7	70.7	46.7
MVTracker (ours)	86.0	✗	47.7	82.7	71.6	68.0	81.4	70.0	48.7

Table 5. Table B.1 : Runtime Comparison (FPS). We measure frames per second for each method under a representative test setting, excluding depth-estimation overhead. Optimization-based methods are indicated with placeholders for their much lower throughput.

Method	FPS
Dynamic 3DGS [23]	(optimization-based)
Shape of Motion [35]	(optimization-based)
SpaTracker [38]	1.4
DELTA [24]	1.5
SceneTracker [32]	3.5
CoTracker2 [16]	5.7
Triplane Baseline	5.8
MVTracker (ours)	7.2
CoTracker3 [17]	18.4

Table 6. Table C.1 : Impact of Different Depth Sources on DexYCB. We report results using either DUSt3R-estimated  [ 37 ] or sensor-based depth for all methods. Sensor depth generally improves performance across the board, underscoring the importance of high-fidelity depth. Notably, our multi-view tracker benefits the most, indicating that our fused 3D point cloud representation effectively capitalizes on better depth quality.

Method	AJ$\uparrow$	${\delta }_{\text{avg}}$ $\uparrow$	OA$\uparrow$	MTE$\downarrow$
DexYCB [3] (DUSt3R depth)
LocoTrack [5]	27.8	38.6	77.0	22.8
DELTA [24]	36.8	51.6	61.0	18.3
CoTracker2 [16]	28.8	40.5	76.2	20.8
CoTracker3 [17]	29.4	40.3	78.6	22.0
SpaTracker [38]	58.3	72.0	80.2	5.9
SpaTrackerV2 [39]	35.5	45.1	91.3	9.8
TAPIP3D [43]	38.8	50.0	90.1	8.2
Triplane Baseline	57.5	71.0	81.3	4.3
MVTracker (ours)	71.6	80.6	91.3	2.0
DexYCB [3] (sensor depth)
LocoTrack [5]	55.1	64.6	76.7	27.5
DELTA [24]	54.6	67.8	72.6	42.6
CoTracker2 [16]	56.4	67.4	76.4	26.4
CoTracker3 [17]	57.3	68.5	76.7	23.4
SpaTracker [38]	60.9	75.0	82.5	4.0
SpaTrackerV2 [39]	69.2	77.5	91.3	4.2
TAPIP3D [43]	77.2	88.9	88.0	1.3
Triplane Baseline	68.0	79.5	85.9	2.6
MVTracker (ours)	82.7	91.0	91.6	0.8

Table 7. Table C.2 : Varying the Number of Input Views. Evaluation of AJ under a varied number of input views.

Method	Number of views
	1	2	3	4	5	6	7	8
	Panoptic Studio [18]
LocoTrack [5]	54.7	56.4	61.4	65.8	65.8	67.3	66.4	66.8
DELTA [24]	60.9	61.8	63.7	68.1	66.7	67.4	67.6	66.3
CoTracker2 [16]	59.5	60.7	64.4	69.5	68.8	69.3	68.4	69.9
CoTracker3 [17]	61.9	65.0	68.5	74.5	74.0	75.2	75.1	76.0
SpaTracker [38]	51.2	52.1	57.5	61.5	60.9	61.4	62.6	63.4
SpaTrackerV2 [39]	57.4	62.7	66.8	72.4	71.8	72.7	72.7	72.9
TAPIP3D [43]	68.8	72.9	78.5	84.3	84.6	85.6	86.0	86.8
Triplane Baseline	61.9	60.9	61.6	65.0	64.0	65.1	66.8	65.6
MVTracker (ours)	66.4	72.2	79.0	86.0	87.4	88.8	89.1	89.9
	DexYCB [3] (DUSt3R depth)
LocoTrack [5]	27.9	26.0	28.1	27.8	36.3	34.8	34.7	34.9
DELTA [24]	33.0	34.3	38.0	36.5	37.2	35.4	34.9	35.7
CoTracker2 [16]	29.8	26.4	29.2	28.8	37.8	36.2	36.0	36.0
CoTracker3 [17]	28.6	27.0	29.5	29.4	39.1	37.5	37.1	37.3
SpaTracker [38]	60.6	58.4	61.8	58.3	63.2	62.4	62.9	63.4
SpaTrackerV2 [39]	39.8	39.5	36.5	35.5	41.1	37.1	37.0	37.7
TAPIP3D [43]	36.6	35.6	40.5	38.8	57.7	54.2	55.2	56.4
Triplane Baseline	44.0	48.0	56.0	57.6	63.5	64.5	65.5	66.8
MVTracker (ours)	64.0	66.8	73.2	71.1	77.4	76.7	77.3	79.2
	Multi-View Kubric [11]
LocoTrack [5]	57.3	56.1	53.8	52.5	52.3	52.1	51.6	51.8
DELTA [24]	65.9	61.9	58.9	57.4	56.3	56.0	54.8	54.1
CoTracker2 [16]	61.4	59.1	56.1	54.6	53.8	53.7	53.1	52.8
CoTracker3 [17]	60.1	58.7	56.5	55.1	54.8	54.6	54.2	54.3
SpaTracker [38]	72.8	70.3	67.4	65.5	64.8	64.5	63.1	62.7
SpaTrackerV2 [39]	54.0	57.6	57.6	58.6	58.3	59.1	58.8	59.7
TAPIP3D [43]	76.4	75.4	72.8	72.5	71.7	71.4	70.9	70.6
Triplane Baseline	71.7	73.4	74.1	74.7	75.0	75.2	75.2	75.7
MVTracker (ours)	81.7	81.2	81.4	81.4	81.9	82.1	82.3	82.5

Table 8. Table C.3 : 2D Tracking Evaluation. Evaluation of projected 2D tracking performance using standard 2D metrics  [ 7 ] . Multi-view outputs are projected onto four views and the metrics are averaged across these views. Note that we can only report location accuracies since our multi-view methods report any-view visibility (not per-view).

Method	${\delta }_{\text{avg}}^{\text{2D}}$$\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$
	Panoptic Studio [18]
LocoTrack [5]	66.7	29.3	51.4	72.6	86.5	93.7
DELTA [24]	70.0	34.8	56.5	76.5	87.6	94.8
CoTracker2 [16]	64.3	28.0	49.3	69.7	82.9	91.4
CoTracker3 [17]	70.9	33.1	56.8	78.2	90.5	96.2
SpaTracker [38]	66.4	29.3	51.8	71.8	85.7	93.3
Triplane Baseline	51.9	8.5	24.1	52.0	80.3	94.5
MVTracker (ours)	70.5	26.6	52.9	79.5	94.6	98.9
	DexYCB [3] (sensor depth)
LocoTrack [5]	83.7	74.1	78.4	84.0	89.4	92.4
DELTA [24]	84.6	75.7	79.9	85.2	89.4	92.8
CoTracker2 [16]	83.7	73.7	78.9	84.5	89.3	92.1
CoTracker3 [17]	84.1	74.8	79.0	84.5	89.4	92.9
SpaTracker [38]	85.8	75.9	80.7	86.1	91.2	94.9
Triplane Baseline	75.9	52.2	70.4	78.9	85.9	92.0
MVTracker (ours)	87.5	73.3	81.2	88.3	95.5	99.2
	Multi-View Kubric [11]
LocoTrack [5]	75.3	53.6	66.8	78.0	86.1	91.9
DELTA [24]	78.4	61.9	72.0	80.3	86.5	91.2
CoTracker2 [16]	72.6	51.8	63.7	74.2	83.1	90.2
CoTracker3 [17]	74.8	54.2	66.3	76.8	85.4	91.5
SpaTracker [38]	75.8	51.4	65.8	79.3	88.5	94.0
Triplane Baseline	80.6	51.3	72.4	87.3	94.5	97.7
MVTracker (ours)	86.8	61.2	81.7	94.0	97.9	99.2

Table 9. Table C.4 : Evaluation of projected 2D tracking performance using standard 2D metrics  [ 7 ] on DexYCB with estimated depths. Single-view methods only require RGB for tracking and do not depend on the quality of the depth for achieving high 2D point tracking performance. However, the multi-view methods face challenges when dealing with the noisy depth estimates on DexYCB.

Method	${\delta }_{\text{avg}}^{\text{2D}}$$\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$
	DexYCB [3] (DUSt3R depth)
LocoTrack [5]	85.4	75.6	80.0	85.8	91.4	94.4
DELTA [24]	85.6	77.0	81.2	86.0	90.2	93.7
CoTracker2 [16]	86.2	75.7	81.3	87.1	92.1	95.0
CoTracker3 [17]	86.7	77.0	81.5	87.2	92.1	95.7
SpaTracker [38]	86.0	76.4	80.8	86.2	91.7	95.1
Triplane Baseline	64.3	31.8	53.6	71.4	79.3	85.4
MVTracker (ours)	71.3	42.1	59.8	76.0	85.2	93.5

Table 10. Table C.5 : Training Augmentation of the variable number of input views ranging from 1 to 8 (V) and varying depth sources between ground-truth and off-the-shelf depth estimation (D) during training on tracking performance.

V	D		AJ $\uparrow$	${\delta }_{\text{avg}}$ $\uparrow$	OA $\uparrow$	MTE $\downarrow$
			Panoptic Studio [18]
✓			80.4	94.4	87.8	3.1
	✓		80.7	93.3	88.7	3.5
✓	✓		80.7	94.3	88.2	3.2
			DexYCB [3]
✓			49.9	72.4	70.5	3.6
	✓		62.3	77.1	82.1	3.2
✓	✓		65.2	79.5	82.9	2.3
			Multi-View Kubric [11]
✓			80.2	91.1	91.1	0.7
	✓		77.2	88.0	91.0	0.8
✓	✓		80.4	90.5	92.1	0.7

Table 11. Table C.6 : Training Augmentation with evaluation on different numbers of views. We investigate the impact of using a variable number of views ranging from 1 to 8 (V) and varying depth sources between ground-truth and off-the-shelf depth estimation (D) during training on tracking performance. Results are reported in AJ.

V	D	Number of views
		1	2	3	4	5	6	7	8
		Panoptic Studio [18]
✓		69.3	72.0	76.9	80.4	81.4	84.4	84.8	85.1
	✓	67.0	71.3	77.2	80.7	81.9	84.2	84.9	85.8
✓	✓	68.1	73.1	76.9	80.7	81.5	83.4	84.4	84.3
		DexYCB [3] (DUSt3R depth)
✓		37.6	40.0	48.1	49.9	65.6	67.4	69.6	73.4
	✓	51.6	54.1	60.7	62.3	68.8	69.9	71.8	74.8
✓	✓	56.3	58.9	64.9	65.2	74.0	75.4	76.3	78.9
		Multi-View Kubric [11]
✓		80.7	79.9	80.1	80.2	80.9	81.4	81.8	81.9
	✓	75.9	76.3	76.6	77.2	77.2	78.0	78.1	78.4
✓	✓	80.4	79.6	80.0	80.4	81.1	81.6	81.8	82.0

Table 12. Table D.1 : Dynamic 3D Gaussians. Novel view synthesis scores for Dynamic 3DGS  [ 23 ] under varying numbers of cameras on Panoptic Studio  [ 18 ] .

	Number of Cameras
	27	17	9	4
PSNR	28.5	23.7	17.9	12.6
SSIM	0.91	0.83	0.74	0.52

Table 13. Table E.1 : TAPVid-2D Evaluation. Evaluation results in 2D point tracking on TAPVid-2D  [ 7 ] . Since our method requires depth as input, we report results for several monocular depth estimators. However, performance is far from satisfactory for most trajectories, which is in part attributable to out-of-distribution outdoor scenes with many far-away points, scenes not being normalized well to the training scale, failures in depth estimation, flickering in the estimated video depth, etc. We report both the original TAPVid-2D metrics as well as the metrics used elsewhere in this paper; the difference is that our metrics take the average of per-trajectory metrics, whereas TAPVid-2D computes each metric over all points, treating all trajectories as one trajectory. All metrics are in pixel scale ( e.g ., δ < 4 2D \delta_{<4}^{\text{2D}} is the within-a-radius-of-4-pixels location accuracy, MTE is in pixels, etc.). Best and second-best results per metric are bold and underlined , respectively.

Method	Depth		${\text{AJ}}^{\text{2D}}$							${\delta }_{\text{avg}}^{\text{2D}}$							OA	MTE
			TAPVid-2D Metrics
MVTracker (ours)	ZoeDepth [1]		9.5	1.6	3.9	8.5	14.3	19.4		20.4	3.0	7.5	16.7	29.7	45.3		46.5	–
MVTracker (ours)	MoGe [36]		13.1	2.0	4.4	9.5	19.5	30.0		25.0	3.4	8.0	18.4	37.5	57.6		54.6	–
MVTracker (ours)	MegaSAM [20]		31.6	5.7	14.4	31.3	47.4	59.1		46.0	11.0	25.3	47.8	66.8	79.1		78.6	–
CoTracker3 [17]	none required		64.1	28.2	51.3	72.2	82.4	86.4		77.0	42.1	67.1	85.3	93.3	97.0		91.0	–
			Our Metrics (where numbers are computed per track and then averaged)
MVTracker (ours)	ZoeDepth [1]		10.3	1.8	4.4	9.5	15.6	20.4		21.8	3.3	8.4	18.3	31.9	47.3		46.1	59.1
MVTracker (ours)	MoGe [36]		14.6	2.4	5.4	11.6	22.0	31.6		26.7	3.9	9.2	20.5	40.4	59.7		55.1	86.0
MVTracker (ours)	MegaSAM [20]		35.0	7.4	17.4	35.7	51.9	62.6		46.4	12.1	26.2	48.1	66.7	78.9		79.8	14.0
CoTracker3 [17]	none required		66.7	32.6	56.2	74.9	83.3	86.4		77.2	43.2	67.6	85.3	93.1	96.7		91.7	2.9

Equations

$$\displaystyle\mathbf{x}\;=\;\mathbf{E}_{t}^{v^{-1}}\bigl{(}\mathbf{K}_{t}^{v^{-1}}\left(\,u_{x},u_{y},1\,\right)^{\top}\cdot\mathbf{D}_{t}^{v}[\,u_{y},u_{x}\,]\bigr{)}.$$

$$\displaystyle\mathcal{X}_{t}^{s}\;=\;\bigl{\{}\bigl{(}\mathbf{x},\,\Phi_{t}^{v,s}[\,u_{y},u_{x}\,]\bigr{)}\;\big{|}\;v$$

$$\displaystyle\,(u_{x},u_{y})\in\Omega_{t}^{v}\bigr{\}},$$

$$\displaystyle\bigl{(}\mathbf{x}^{n,*},\mathbf{\phi}^{n,*}\bigr{)}\;=\;\!\operatorname*{\arg\!\min}_{\bigl{(}\mathbf{x},\,\mathbf{\phi}\bigr{)}\,\in\,\mathcal{X}_{\,t_{n}^{q}}^{\,1}}\|\mathbf{x}-\mathbf{p}_{t_{n}^{q}}^{n}\|,$$

$$\displaystyle\mathbf{C}_{t}^{n,s}\;=\;\bigl{\{}\langle\mathbf{f}_{t}^{n},\mathbf{\phi}_{k}\rangle\;\big{|}\;\bigl{(}\mathbf{x}_{k},\mathbf{\phi}_{k}\bigr{)}\in\mathcal{N}_{K}\!\bigl{(}\mathbf{\hat{p}}_{t}^{n},\mathcal{X}_{t}^{s}\bigr{)}\bigr{\}},$$

$${\hat{\mathbf{p}}}_t^{n,m+1}={\hat{\mathbf{p}}}_t^{n,m}+\Delta {\hat{\mathbf{p}}}_t^{n,m+1},$$

$${\mathbf{f}}_t^{n,m+1}={\mathbf{f}}_t^{n,m}+\Delta {\mathbf{f}}_t^{n,m+1}.$$

$$\mathcal{L}={\mathcal{L}}_{\text{xyz}}+{\lambda }_{\text{vis}}{\mathcal{L}}_{\text{vis}}.$$

$$\displaystyle\mathcal{L}_{\text{xyz}}\;=\;\sum_{j,m,n,t}\frac{\gamma^{\,M-m}}{JMNT}\;\bigl{\|}\,\mathbf{\hat{p}}_{t}^{\,n,m,j}-\mathbf{p}_{t}^{\,n,m,j}\bigr{\|}_{1}\,,$$

$$\displaystyle\mathcal{L}_{\text{vis}}\;=\;\sum_{j,t,n}\frac{1}{J\,T\,N}\operatorname{B{-}BCE}\bigl{(}\hat{v}_{t}^{\,n,j},\,v_{t}^{\,n,j}\bigr{)}.$$

$$\displaystyle MTE_{i}\;=\;\mathrm{median}\!\Bigl{\{}\|\hat{p}_{t}^{i}-p_{t}^{i}\|_{2}\;\big{|}\;t=1,\dots,T,\;v_{t}^{i}=1\Bigr{\}}.$$

$$MTE=\frac{1}{N}\sum _{i=1}^{N}MT{E}_i.$$

$${\mathrm{OA}}_i=$$

$$\mathrm{OA}=$$

$$\displaystyle\delta_{x}^{i}\;=\;\tfrac{1}{\sum_{t=1}^{T}v_{t}^{i}}\sum_{t=1}^{T}\mathds{1}\!\Bigl{(}v_{t}^{i}=1\Bigr{)}\mathds{1}\!\Bigl{(}\|\hat{p}_{t}^{i}-p_{t}^{i}\|_{2}<x\Bigr{)}.$$

$${\delta }_x=\frac{1}{N}\sum _{i=1}^N{\delta }_x^i,{\delta }_{\text{avg}}=\frac{1}{H}\sum _{h=1}^H{\delta }_{x_h}.$$

$$\displaystyle AJ_{x}^{i}\;=\;\frac{\sum_{t=1}^{T}v_{t}^{i}\hat{v}_{t}^{i}\alpha_{t,x}^{i}}{\sum_{t=1}^{T}\bigl{(}v_{t}^{i}+(1-v_{t}^{i})\hat{v}_{t}^{i}+v_{t}^{i}\hat{v}_{t}^{i}(1-\alpha_{t,x}^{i})\bigr{)}}.$$

$$A{J}_x=\frac{1}{N}\sum _{i=1}^{N}A{J}_x^i,AJ=\frac{1}{H}\sum _{h=1}^{H}A{J}_{x_h}.$$

Full text

1 Introduction

Tracking arbitrary points in 3D [30] is a fundamental problem in computer vision, with numerous applications in dynamic scene reconstruction [14], robotics [10], and augmented reality [25]. While remarkable progress has been made with the recent advancement of 2D point tracking methods [17, 16, 12, 7, 8, 9, 5, 45], they remain inherently limited in modeling 3D-consistent motion due to the fundamental ambiguity of the 3D-to-2D projection process. Thus, high-quality 3D point tracking remains a challenging task.

Scene flow approaches [30, 26, 31, 28] can estimate a dense 3D flow field for every 3D point, but they are usually limited to two consecutive video frames, whereas 3D point tracking operates on long sequences (e.g., tens or even hundreds of video frames). Recently, several methods [38, 18, 24] have been proposed to tackle the 3D point tracking task from monocular videos. However, their performance remains far from satisfactory for real-world applications that demand high quality and robustness, due to the difficulty of estimating 3D from single viewpoints in challenging scenarios such as occlusion and complex motion.

To address these challenges, we adopt a multi-camera setup and develop the first feed-forward model that can efficiently and robustly predict long-range 3D point trajectories from multi-view videos. Unlike previous multi-camera methods [30, 23, 44] that require more than 20 cameras and tedious per-sequence optimization, our approach enables feed-forward 3D point tracking with a practical and flexible number of cameras, such as a set of four camera views with arbitrary viewpoints. Thus, our method strikes a promising balance between accuracy and practicality, making it particularly suitable for real-world applications.

Key to our model is a dynamic fused 3D feature point cloud, constructed by combining the unprojected per-view depth maps, which not only effectively aggregates multi-view information to a global scene representation, but also facilitates reliable and efficient correspondence search with the k-nearest-neighbors (kNN) operation. This is different from the triplane representation used in the previous state-of-the-art method SpatialTracker [38], which inevitably suffers from information loss during the triplane splatting process and is therefore less effective in processing varying numbers of input cameras. Moreover, our model performs reasonably well with different sources of depth input, whether from accurate depth sensors or noisy estimates from methods such as DUSt3R [37] and VGGT [33].

Using the fused dynamic point cloud, we retrieve local neighbors for each tracked point using a kNN search and compute multi-scale correlation features that capture both appearance similarity and 3D offset information. A spatiotemporal transformer then iteratively refines each point’s 3D position and appearance over a sliding temporal window. Finally, the outputs from overlapping windows are merged to produce globally consistent 3D point trajectories.

To train our model, we construct a synthetic multi-view dataset using Kubric [11] and simulate 5K sequences. We evaluate the model on two real-world datasets, DexYCB [3] and Panoptic Studio [15], for which we construct 3D point trajectories by leveraging ground-truth object and hand pose estimation labels in DexYCB and merging existing monocular trajectory labels [18] in Panoptic Studio. We conduct extensive experiments on these multi-view video datasets, which suggest that our model outperforms baseline methods by a significant margin. Our model also performs well with different camera setups, different numbers of cameras, and different sources of depth maps, indicating the robustness of our proposed approach. We release our source code and models alongside the training and evaluation datasets to facilitate future research on multi-view 3D point tracking.

2 Related Work

Scene Flow.

Scene flow methods [30, 26, 31, 28] are usually designed to estimate the dense 3D motion between two consecutive video frames. Traditional methods [30, 26, 31] try to solve this task with an optimization framework, which is typically slow and requires many cameras (e.g., 31 in  [30]) to well-regularize the optimization process. Modern approaches [41, 28, 21] explore data-driven models to directly predict scene flow in a feed-forward manner. Despite recent progress, existing models are limited to two frames and are not able to track long-range correspondences in 3D.

2D Point Tracking.

Recent years have witnessed significant progress in long-term 2D point tracking [7, 12, 45, 34, 8, 16, 9, 17, 5, 13]. For instance, CoTracker2 [16] leverages self-attention to aggregate spatial context from supporting tracks, while LocoTrack [5] extends traditional 2D correlation into bidirectional 4D correlation for more robust local matching. These methods typically estimate long-range point trajectories over an entire video, handling occlusions effectively. In this work, we investigate their extension to 3D, but with a sparse multi-view setup and known camera parameters, rather than monocular input. Moreover, unlike recent trackers [17, 9] that reduce the synthetic-to-real domain gap via self-supervision or pseudolabeling, our approach focuses on direct supervision from synthetic data.

3D Point Tracking.

3D point tracking has seen innovative contributions from recent methods [42, 23, 38, 24, 39, 43, 35]. SpatialTracker [38] generalizes point tracking into 3D by integrating depth information through a Triplane representation [2] and DELTA [24] introduces a coarse-to-fine approach to estimate dense trajectories across the entire image plane, as opposed to sparse point tracking. Both methods assume monocular input. The concurrent SpatialTrackerV2 [39] and TAPIP3D [43] are also monocular. In a different approach, Dynamic 3DGS [23] leverages 3D Gaussian reconstructions over time to track dense scene elements in 3D and perform accurate 2D and 3D point tracking, but requires a multi-camera setup of 27 cameras in Panoptic Studio [15] in addition to sparse depth from depth sensors for point cloud initialization and segmentation masks for relevant objects. We also use multiple input views but assume fewer cameras (e.g., four), which is more practical for real-world applications. Works such as [35, 6, 27] use monocular tracking priors and our experiments suggest that the multi-view extension of [35] is constrained by this. In contrast, our method directly learns a multi-view tracking prior that is neither Gaussian-based nor optimization-based. VideoDoodles [42] presents an interactive system that anchors hand-drawn animations to objects within a reconstructed 3D scene, harnessing optical flow and depth maps in a novel 3D point tracking algorithm. However, it is primarily designed for video editing rather than competitive performance on standard benchmarks such as TAPVid-2D [7], whereas we focus on 3D point tracking accuracy.

3 Method

Our goal is to perform online 3D point tracking across multiple views, given a set of synchronized RGB frames with known camera parameters. The input consists of video sequences captured from $V$ different camera views, with each frame containing a set of query points that need to be tracked over time. We estimate temporally consistent 3D trajectories for these points while also predicting visibility, handling occlusions, and adapting to scene dynamics. Our method runs online at 7.2 FPS given RGB-D input. For RGB-only input, we rely on external depth estimation (e.g., DUSt3R [37] runs at 0.17 FPS and VGGT [33] at 3.1 FPS). An overview of our method is provided in Fig. 2.

Unlike single-view trackers that compute feature correlation on a 2D grid [16, 17, 5, 24, 8, 7, 12] or a triplane [38], our method leverages a fused multi-view point cloud representation to establish $k$-nearest neighbors (kNN) feature correlations. While 2D-grid-based correlation can match irrelevant background pixels and triplane-based correlation inherently suffers from information loss, our kNN targets geometrically relevant 3D neighborhoods. This enables more robust tracking by integrating geometric consistency across views and learning motion priors directly from data.

3.1 Problem Formulation

Given a calibrated multi-view video sequence $\mathbf{I}_{t}^{v}\in\mathbb{R}^{H\times W\times 3}$ captured by $V$ cameras over $T$ frames, our goal is to track $N$ query points in 3D space. We denote each of the $N$ query points as $\mathbf{q}^{n}=(t_{q}^{n},\,x_{q}^{n},\,y_{q}^{n},\,z_{q}^{n})\in\mathbb{R}^{4}$, where $t_{q}^{n}$ denotes the query frame and $(x_{q}^{n},\,y_{q}^{n},\,z_{q}^{n})$ its initial 3D location in world coordinates. Besides images and query points, the camera intrinsics $\mathbf{K}_{t}^{v}\in\mathbb{R}^{3\times 3}$ and extrinsics $\mathbf{E}_{t}^{v}\in\mathbb{R}^{4\times 4}$ are given for each view $v$ and frame $t$.

The goal is to predict the 3D trajectory {$\mathbf{p}_{t}^{n}=(x_{t}^{n},\,y_{t}^{n},\,z_{t}^{n})\in\mathbb{R}^{3}$, $t\geq t_{q}^{n}$}, along with visibilities {$v_{t}^{n}\in\{\,0,1\,\}$} where $v_{t}^{n}=1$ denotes that the point is visible in at least one camera view, and $v_{t}^{n}=0$ indicates occlusion in all views. Note that the query location matches the ground-truth track location at time $t_{q}^{n}$, i.e. $\mathbf{p}_{t_{q}^{n}}^{n}=(x_{q}^{n},\,y_{q}^{n},\,z_{q}^{n})$.

3.2 Point Cloud Encoding of Multi-view Videos

Feature Maps.

For each input frame $\mathbf{I}_{t}^{v}$, we extract $d$-dimensional per-view feature maps $\Phi_{t}^{v}=\mathcal{\varphi}(\mathbf{I}_{t}^{v})$ using a convolutional backbone (same as in [17, 16, 38]). We use a stride factor of $k=4$ to downscale the input resolution for computational efficiency, such that $\Phi_{t}^{v}\in\mathbb{R}^{\frac{H}{k}\times\frac{W}{k}\times d}$. We also compute the feature maps at $S=4$ different scales, i.e., $\Phi_{t}^{v,s}\in\mathbb{R}^{\frac{H}{k2^{s-1}}\times\frac{W}{k2^{s-1}}\times d}$, $s=1,\dots,S$. These are obtained by downscaling the base features (e.g., via average pooling) and will later be used in our pyramid correlation.

Fused 3D Feature Point Cloud.

We obtain multi-view-consistent depth maps $\mathbf{D}_{t}^{v}\in\mathbb{R}^{H\times W}$ for each view $v$ and frame $t$ using off-the-shelf depth estimation [37, 23, 33] (see Appendix A). Sensor-based depth (e.g., Kinect) can also be used if available and preferred. Using the depth and camera parameters, we lift each valid pixel $(u_{x},u_{y})$ into 3D:

[TABLE]

Invalid pixels are those that do not have a valid depth value due to missing Kinect depth values. For estimated depth, we retain all pixels and train for robustness (instead of thresholding by uncertainty predictions). Each of these lifted points $\mathbf{x}$ is associated with its corresponding feature from $\Phi_{t}^{v,s}$ and then fused across views into a single point cloud:

[TABLE]

where $\Omega_{t}^{v}$ is the set of valid pixels in view $v$. This unified 3D representation facilitates the computation of spatial correlation across views, enabling multi-view-consistent tracking.

An alternative to our fused point cloud is to combine multi-view features via a single global triplane [38], projecting 3D points onto three orthogonal planes (e.g., XY, YZ, ZX). However, this inevitably suffers from projection collisions: different surfaces map to the same planar coordinates, causing destructive feature averaging. It also requires choosing a fixed bounding region for the scene, which leads to wasted resolution or clipped content for large or uncentered scenes. By contrast, our point cloud preserves features directly in 3D, avoids collisions, and adapts naturally to different scenes, resulting in more robust multi-view tracking.

Track Features.

We assign a $d$-dimensional time-dependent appearance feature $\mathbf{f}_{t}^{n}$ to each tracked point. These features are initialized by sampling from the fused 3D feature point cloud at the query frame and 3D location:

[TABLE]

for every track $n$. We use the sampled feature $\mathbf{\phi}^{n,*}$ to initialize $\mathbf{f}_{t}^{n}$ for all $t\geq t_{n}^{q}$. These track features are subsequently used when computing track-centric spatial correlation with nearest neighbors from the fused 3D feature point cloud (see Sec. 3.3), but also refined by the spatiotemporal transformer to capture appearance changes over time (see Sec. 3.4).

3.3 Multi-Scale Spatial Correlation

Instead of computing correlations separately for each view [16, 17, 5, 24, 8, 7, 12] or on an auxiliary triplane [38], we establish correspondences directly in the fused feature point cloud using a multi-scale kNN approach. For each query point, we retrieve its $K$ nearest neighbors at different scales from $\mathcal{X}_{t}^{s}$ and compute local kNN feature correlations:

[TABLE]

where $\langle\cdot,\cdot\rangle$ is the dot product and $\mathcal{N}_{K}(\mathbf{\hat{p}}_{t}^{n},\mathcal{X}_{t}^{s})$ finds the k-nearest neighbors around the current location estimate $\mathbf{\hat{p}}_{t}^{n}$ in the fused 3D feature point cloud. These correlations capture spatial dependencies at multiple scales and are fed into our transformer-based tracking module, allowing it to search increasingly larger 3D neighborhoods. For reference, average neighbor distances at the four scales on Panoptic Studio (made-up of ~10m–wide scenes) are $12.5\pm 8.2$, $22.4\pm 12.2$, $42.7\pm 21.7$, and $85.8\pm 44.0$ cm. At the highest scale, this covers frame-to-frame motion of up to ~92 km/h at 30 FPS.

Unlike 2D correlation, where pixel offsets implicitly encode direction, our 3D point cloud representation requires explicit 3D offsets. Concretely, for each neighbor $(\mathbf{x}_{k},\mathbf{\phi}_{k})$ we concatenate the local similarity $\langle\mathbf{f}_{t}^{n},\mathbf{\phi}_{k}\rangle$ with the offset vector $(\mathbf{x}_{k}-\mathbf{\hat{p}}_{t}^{n})$ to encode direction and distance from $\mathbf{\hat{p}}_{t}^{n}$. This offset helps the model disambiguate different nearest neighbors and enables correlation-based matching in 3D.

3.4 Transformer-Based Iterative Tracking

Our transformer refines track estimates over a sliding window of $T$ frames. For each track $n$ at time $t$, we construct a token $G_{t}^{n}=(\,\eta(\,\mathbf{\hat{p}}_{t}^{n}-\mathbf{\hat{p}}_{\,t_{n}^{q}}^{\,n}),\,\mathbf{f}_{t}^{n},\,\mathbf{C}_{t}^{n,s},\,\hat{v}_{t}^{n})$, where $\eta(\cdot)$ is a sinusoidal positional encoding. The tokens $\{G_{t}^{n}\}_{t,n}$ are passed to a transformer $\mathcal{\psi}$, which applies temporal self-attention across time and cross-attention with a small set of learned virtual tracks to model spatial dependencies, following the design in CoTracker2 [17]. The transformer outputs residual position and feature updates $(\Delta\mathbf{\hat{p}}_{t}^{n},\Delta\mathbf{f}_{t}^{n})=\mathcal{\psi}(G_{t}^{n})$, which we apply iteratively for $m=1,\dots,M$:

[TABLE]

After each iteration $m$, we recompute $\mathbf{C}_{t}^{n,s,m}$ based on the refined position and feature. At the final iteration $m=M$, we estimate visibility as \hat{v}_{t}^{\,n}\;=\;\sigma\bigl{(}W\,\mathbf{f}_{t}^{n,M}\bigr{)}, where $\sigma(\cdot)$ is the sigmoid function and $W$ a learned projection matrix.

3.5 Windowed Inference and Unrolled Training

To process long videos, we adopt a windowed approach as in CoTracker [16]. Let $T$ be the maximum window size; for a longer video of length $T^{\prime}>T$, we divide it into $J$ overlapping windows of length $T$, each shifted by $T/2$ frames. Once the transformer completes $M$ iterative updates for window $j$, its final track estimates initialize window $j+1$, allowing refined trajectories to propagate. Let $\mathbf{\hat{p}}_{t}^{\,n,m,j}$ be the predicted location of query $n$ at time $t$ after iteration $m$ in window $j$, and $\hat{v}_{t}^{\,n,j}$ its visibility (predicted only at the final iteration). During training, we unroll these windowed updates so that each iteration learns to correct or refine predictions from previous windows and iterations.

3.6 Supervision

Training is supervised with ground-truth trajectories $\mathbf{p}_{t}^{n}$ and visibility labels $v_{t}^{n}$. The total loss consists of a position loss ($\mathcal{L}_{\text{xyz}}$) and a visibility loss ($\mathcal{L}_{\text{vis}}$), balanced by $\lambda_{\text{vis}}$:

[TABLE]

In particular, the position loss is the weighted $\ell_{1}$-norm error between the predicted and the ground-truth 3D positions:

[TABLE]

where $j$ indexes the windows $J$, $m$ the iterative updates $M$, $n$ the trajectories $N$, $t$ the number of frames $T$, and $\gamma$ is a weighting factor used to penalize the later iterations of the iterative refinement more. The visibility loss addresses class imbalance by minimizing the balanced binary cross-entropy (B-BCE) between predicted and ground-truth labels:

[TABLE]

4 Experiments

Datasets.

To train our model, we generate a synthetic multi-view dataset, MV-Kub, using Kubric [11], simulating 5K multi-view video sequences. Since no large-scale datasets exist for multi-view 3D point tracking, we rely on synthetic data for supervised training. For evaluation on the hand dexterity dataset DexYCB [3], we sample 10 scenes, leveraging ground-truth object and hand poses to generate track labels. We also construct a Panoptic Studio evaluation dataset by merging existing monocular labels from TAPVid-3D [18] for a total of 6 scenes. Evaluation query points are randomly sampled from both static and moving surfaces.

All results, unless otherwise stated, are reported assuming the availability of estimated depth [37] for DexYCB, ground-truth optimization-based depth [23] for Panoptic Studio, and simulated depth for MV-Kub. In Appendix C we ablate the impact of depth sources on the performance.

Evaluation Metrics.

As no standard metrics exist for multi-view 3D point tracking, we extend those from monocular benchmarks [7], reporting four key metrics: (1) Median Trajectory Error (MTE) for visible points, quantifying spatial accuracy; (2) $\delta_{\text{avg}}$, the average location accuracy over a set of threshold distances; (3) Occlusion Accuracy (OA), measuring the model’s binary visibility prediction across views; and (4) Average Jaccard (AJ), which jointly evaluates occlusion and position accuracy. All metrics are computed per track, averaged within a scene, and finally aggregated across all dataset scenes. See Appendix F for details.

Training.

We train our method on MV-Kub for 200K steps on 8 GH200 chips with 96GB GPU memory over 8 days, with a batch size of 8 and up to 2048 trajectories per sample. Our model is implemented in PyTorch (with Lightning Fabric for multi-node training) and optimized using AdamW [22]. The model operates with a latent feature dimension of $d=128$ and is trained on MV-Kub sequences of up to 24 frames using a sliding window of 12 frames. We use six-head self-attention layers with a hidden size of 256. We apply a range of augmentations to improve generalization and robustness. See Sec. C.4 for further details.

Baselines.

Since there currently exist no other feed-forward multi-view point trackers, we implement a triplane-based baseline by extending SpaTracker [38] to fuse multi-view features into a single global triplane, referred to as the “Triplane Baseline” in experiments. We also evaluate against two multi-view methods that require test-time optimization: Shape of Motion [35] and Dynamic 3DGS [23]; please refer to Appendix D for their implementation details.

We further evaluate against monocular 2D and 3D point trackers. For 2D point trackers (CoTracker2 [16], CoTracker3 [17], LocoTrack [5]), we run the models on each camera separately with the query points that are deemed to be best visible from that view (based on their projected depth), then lift the resulting 2D estimates to 3D using the known intrinsics and the same depth used by our model. We similarly run 3D trackers (DELTA [24], SpaTracker [38], SpaTrackerV2 [39], TAPIP3D [43]), but give depth as input to directly get 3D tracks. We finally merge per-camera predictions into the world space using known camera poses.

4.1 Method Comparisons

Tab. 1 compares MVTracker against several baselines on Panoptic Studio, DexYCB, and Multi-View Kubric. Our method consistently achieves higher location accuracy ($\delta_{\text{avg}}$), better occlusion-aware tracking (AJ), and lower trajectory errors (MTE). This is visually apparent in Fig. 3.

On Panoptic Studio, our tracker obtains an AJ of 86.0 and a $\delta_{\text{avg}}$ of 94.7, substantially outperforming single-view methods such as SpaTracker (61.5 AJ) and CoTracker3 (74.5 AJ). We observe similar trends on DexYCB, where our method reaches an AJ of 71.6 with a median trajectory error of only 2.0 cm, compared to the lower performance of LocoTrack, DELTA, and CoTracker3. This is because 2D point trackers are not robust to lifting their 2D trajectories into 3D estimates based on estimated depth maps. Finally, on the Multi-View Kubric validation set, our model attains an AJ of 81.4 and an exceptionally low MTE of 0.7 cm.

Compared to optimization-based methods, which require per-sequence training, our approach is fully feed-forward and runs at 7.2 FPS. While Shape of Motion achieves reasonable accuracy, its iterative optimization makes it impractical for large-scale or real-time applications. Dynamic 3DGS further requires dense camera setups, limiting its applicability to real-world scenarios with fewer cameras.

Impact of the kNN-based correlation.

Our method’s strong performance is largely due to its kNN-based correlation mechanism. Replacing it with a world-aligned triplane correlation leads to a significant drop in performance (see Tab. 1). Triplane-based methods compress multi-view features onto fixed 2D planes, causing destructive feature collisions when different 3D points from the same or different views project to the same grid cell. This leads to information loss that cannot easily be disentangled, especially as the number of views increases. Moreover, triplanes require placing the 2D planes in a fixed scene-aligned coordinate frame with a pre-defined scale and extent, which is difficult to generalize across diverse camera setups and scene scales. In contrast, our kNN-based correlation operates directly in 3D world space and dynamically selects relevant neighbors, avoiding these issues and enabling more robust tracking.

4.2 Ablations

We further evaluate the impact of the point correlation components, the varying number of input views, and different camera setups, as well as training augmentation strategies.

Point Correlation Components.

Tab. 2 analyzes the impact of different components in our correlation module. Unlike 2D correlation, where relative direction is implicitly encoded in the pixel grid, kNN in 3D retrieves neighbors from all directions, making explicit offset information crucial. We compare three variants: (i) no offset vector, (ii) offset vector plus explicit neighbor location, and (iii) offset vector only. We train each variant for 25% of total steps on 8 GPUs. The results suggest that (iii) “offset only” achieves the best performance across metrics, indicating that encoding only the relative offset, without concatenating absolute neighbor locations, yields most effective correspondences.

Effect of the Number of Input Views.

Fig. 4 shows how performance changes with the number of input views on DexYCB. Our tracker consistently improves as more views are added, achieving an AJ of 64.0 with a single view, 71.1 with four views, and 79.2 with eight views. This trend highlights the importance of multi-view information in reducing depth ambiguities and improving spatial consistency.

Compared to baseline methods, our approach benefits the most from additional views, thus suggesting better scalability. While SpatialTracker and Triplane show moderate improvements with more views, their performance plateaus earlier, indicating difficulty in fully leveraging additional multi-view information. In contrast, single-view methods like CoTracker3, DELTA, and LocoTrack show limited gains, reinforcing their inability to reliably reconstruct 3D correspondences. Similar trends are observed on Panoptic Studio as well as Multi-View Kubric (see Sec. C.2).

Impact of Camera Setups.

Tab. 3 shows the performance (measured in AJ) under different camera configurations on Panoptic Studio and DexYCB. For Panoptic Studio, we selected sets of four cameras from the available 27, ensuring they were either positioned opposite each other (Setup A) or placed nearby with varying baseline sizes (Setups B and C). For DexYCB, we sequentially chose four out of the available eight cameras: views 1–4 for setup A, views 3–6 for setup B, and views 5–8 for setup C. Note that on Multi-View Kubric the camera configuration is randomized (with cameras generally oriented toward the scene center), so separate evaluations are unnecessary; the main table (Tab. 1) already reflects a random sampling of camera placements. Our method consistently achieves high AJ across all tested setups, indicating robustness to varying camera positions.

Ablation of Training Augmentation.

In Sec. C.4, we examine how training augmentations impact performance, focusing on (i) the number of views used during training (ranging from 1 to 8) and (ii) the depth map source (ground truth vs. off-the-shelf estimation). Our results show that training with a variable number of views significantly improves robustness to different camera setups, ensuring better generalization across datasets. Additionally, incorporating both accurate and estimated depth maps helps mitigate domain shifts and enhances adaptability to real-world depth estimation errors. This is particularly noticeable on DexYCB, where DUSt3R depth quality varies significantly.

Overall, these ablations suggest that MVTracker remains robust across different camera configurations, benefits from diverse training conditions, and performs best with kNN-based offset encoding and carefully designed augmentations. Please refer to Appendix C for more ablation experiments and additional discussions. The inference speed measurements in Appendix B show that MVTracker runs at 7.2 FPS when provided with sensor depth, underscoring its suitability for near-real-time and large-scale applications.

5 Discussion and Future Work

While our method effectively tracks 3D points across multiple views, it has limitations. The key challenge is its reliance on the quality of sparse-view depth estimation. If no sensor depth is available, the approach assumes a reasonably accurate estimated depth. However, in sparse camera setups, such estimation can be unreliable or fail entirely, making tracking infeasible. While our learned motion priors help mitigate moderate noise or incompleteness, a more principled solution would involve jointly estimating depth and tracking for potential mutual refinement, or developing a foundation model for 4D reconstruction with integrated tracking capabilities. We view this as a critical future direction for the community. Appendix A provides additional analysis on the impact of depth quality, robustness to noise, comparisons across different estimators, and failure cases.

Another limitation is that our study focuses on tracking within bounded regions where camera overlap is sufficient. Extending to unbounded or outdoor environments presents additional challenges due to limited training data availability, scene scale variation, and less constrained viewpoints.

A related issue is scene normalization: our model is trained on a fixed dataset with randomized but similar scene scales and layouts. To bridge distribution gaps at test time, we currently apply manually or heuristically determined similarity transforms. While this works well on our benchmarks and a few additional qualitative datasets, more principled approaches are needed to support arbitrary new scenes.

Finally, long-term 3D point tracking faces the broader challenge of scarce large-scale real-world training data. Unlike optimization-based approaches, data-driven point trackers require diverse training distributions to generalize. However, most existing datasets are synthetic, limiting robustness. Recent community efforts [17, 9] show promise in leveraging self-supervised learning from real-world videos, which could be instrumental for improving generalization.

6 Conclusion

We introduce MVTracker, the first data-driven multi-view 3D point tracker, which combines kNN-based correlation in a fused 3D point cloud with a spatiotemporal transformer to track arbitrary points across multiple views. Our method outperforms single-view, multi-view, and optimization-based baselines, suggesting strong generalization across diverse camera configurations and occluded environments.

We hope that our work enables real-world use cases in robotics [29], provides a better data-driven multi-view prior for optimization-based reconstruction methods [35, 19, 23], and acts as a point tracking head in the upcoming age of scalable end-to-end 4D reconstruction and tracking models.

Acknowledgements

We are grateful for insightful discussions with Yiming Wang, Zador Pataki, Luigi Piccinelli, Paul‑Edouard Sarlin, and Philipp Lindenberger. We thank Ignacio Rocco and Skanda Koppula for clarifications regarding TAPVid-3D. This study was conducted within the national “Proficiency” research project (No. PFFS-21-19) funded by the Swiss Innovation Agency Innosuisse in 2021 as one of 15 flagship initiatives. This work was additionally supported as part of the Swiss AI initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project IDs A03 and A136 on Alps, enabling large-scale training and evaluation.

Appendix A Depth Estimation Analysis

Robustness to depth quality.

Our method is designed to be tolerant of moderate noise in estimated depth. As shown in Fig. A.1, tracking performance remains stable under additive Gaussian noise up to $\sigma$ = 2 cm. This robustness stems from extensive training augmentations and from the transformer’s ability to interpolate and re-identify points even when partial depth information is missing or corrupted.

Depth source comparisons.

Tab. A.1 summarizes performance across different depth sources, including sensor depth, optimization-based reconstruction, and learned estimators (DUSt3R and VGGT). Our method achieves strong performance with noisy but plausible depth maps from learned estimators, particularly VGGT, but degrades significantly when depth estimation produces misaligned geometry or fails.

DUSt3R alignment and runtime.

To align DUSt3R [37] reconstructions to our camera setup, we globally optimize the pairwise maps using its built-in alignment (following Sec. 3.4 in DUSt3R), while keeping our camera intrinsics and extrinsics fixed. This reconstruction step takes 300 iterations and runs at 0.17 FPS. VGGT offers a significantly faster alternative at 3.1 FPS.

Discussion.

Sparse-view reconstruction remains an open problem. In our evaluations, DUSt3R was the most reliable method for producing usable depth maps, but even it fails on some scenes. As depth estimation continues to improve, particularly with recent advances in foundation models for geometry (e.g., MegaSAM [20], Easi3R [4]), MVTracker can directly benefit (with or without retraining). Fig. A.2 visualizes our predicted tracks under different depth sources, and Tab. A.1 quantifies performance across depth sources.

Appendix B Inference Speed Comparison

We evaluate the runtime of various methods by measuring frames per second (FPS) on a representative test configuration. Tab. B.1 summarizes the results, excluding depth-estimation time. For optimization-based methods such as Shape of Motion [35] and Dynamic 3D Gaussians [23], the per-sequence runtime is significantly higher (e.g., about 30 minutes for Shape of Motion and about 50 minutes for Dynamic 3DGS on a single Panoptic Studio [18] sequence). Overall, the optimization-based approaches are orders of magnitude slower, making them less practical for large-scale applications and unusable for real-time applications. In contrast, our feed-forward model runs online at a comparable speed to the triplane-based baseline while outperforming it in accuracy, highlighting the efficiency of our kNN correlation in 3D point clouds. Note that we correctly adjust the effective FPS by taking into account the window size for sliding-window-based methods.

Appendix C Ablation Study

C.1 Impact of Different Depth Sources

Table C.1 reports the performance of our model and several baselines on DexYCB when using two different depth sources: DUSt3R [37] (estimated) and sensor-based depth. We observe that sensor-based depth leads to improved accuracy for all methods, highlighting the importance of depth quality. Notably, our multi-view tracker benefits substantially from sensor depth, surpassing single-view methods and the triplane baseline by a larger margin. This suggests that our fused 3D point cloud representation is robust to a wide range of depth sources, yet can further leverage higher-fidelity depth to achieve stronger 3D point tracking.

C.2 Varying the Number of Input Views

Tab. C.2 shows detailed per-dataset results for a varied number of input views. Our tracker consistently improves as more views are added.

C.3 2D Tracking Accuracy

Tab. C.3 compares the performance of 2D point tracking methods when their outputs are projected into 2D using standard metrics [7]. Although 2D methods do not rely on estimated depth quality, they generally achieve lower location accuracies compared to our multi-view approach, which fuses multi-view data to enhance performance. On DexYCB, we only report results using sensor depth maps to provide a more informative comparison in 2D point tracking. For results with depth estimated by DUSt3R, where single-view methods benefit from not having to handle noisy or incomplete depth maps, see Tab. C.4. Note that such depth noise or incompleteness affects our 2D tracking more than that of SpaTracker, due to SpaTracker’s XY-aligned feature plane compared to our fused 3D point cloud.

C.4 Training Augmentation

Tab. C.5 and C.6 evaluate the impact of training augmentation strategies. We consider two factors: (V) employing a variable number of views during training (ranging from 1 to 8) and (D) varying the source of depth maps (ground-truth versus off-the-shelf depth estimation). The results reveal that combining both variable view and depth augmentations leads to the best performance, especially as measured on DexYCB.

We apply a range of augmentations to improve generalization and robustness. These include photometric augmentations such as color jitter, Gaussian blur, and occlusion simulation via erasing and region replacement. Spatial augmentations involve random cropping, padding, flipping, and scaling. Depth perturbations include noise, rescaling, and occlusion-based erasure. Scene-level augmentations transform the world coordinate system through random rotation, translation, and scaling. Camera intrinsics and extrinsics are perturbed to simulate realistic variations. Additionally, we vary the number of tracks per sample, randomly sample the number of input views (between 1 and 6), and use either ground-truth depths or depths from an estimation method to improve model adaptability.

Appendix D Baselines

Dynamic 3D Gaussians. We use the original training code provided by the authors of the Dynamic 3D Gaussians [23]. The training process utilizes camera intrinsics, extrinsics, segmentation masks, images, and an initial point cloud. This initial point cloud is constructed from depth maps corresponding to selected viewpoints at the initial timestep. To differentiate between static and dynamic points within the point cloud, we leverage the provided segmentation masks.

For tracking, we identify the most influential Gaussian for a given point at time $t$ based on the influence computation described in the Dynamic 3DGS paper. To establish a trajectory, we then track the motion of this Gaussian across all time steps. The visibility of each point at time $t$ is determined by comparing the depth of the Gaussian with the corresponding depth value of the depth map rendered from each camera viewpoint. Trajectories that show sudden discontinuities or have a dominant Gaussian influence near zero for the query point are classified as static to ensure consistency in motion.

To evaluate the model’s performance in novel view synthesis, we conducted experiments on Panoptic Studio. We trained different models using a varying number of cameras. Then, we render images using these trained models from four test cameras. Our results in Tab. D.1 show that with 27 input views, we achieve reconstruction quality comparable to that reported in the original paper. However, as the number of views decreases, reconstruction quality reduces.

Shape of Motion. We adapt the monocular training pipeline of the original Shape of Motion [35] to a multi-view setting. The method requires segmentation masks for moving objects, depth maps, long-range 2D tracks, and camera poses. In our multi-view adaptation, we rely on the available depth maps, segmentation masks, and camera poses, while using TAPIR [8] to obtain long-range 2D tracks, as in the original approach.

For both static and dynamic part initialization, we use multi-view depth maps and segmentation masks to construct an initial 3D representation that captures the overall scene structure more effectively than the monocular approach. We follow the original paper’s strategy for choosing the canonical frame and optimizing motion bases. Additionally, we adapt the monocular loss function to a multi-view formulation, computing the original loss function separately for each view and averaging the results across all views. This adjustment ensures that the optimization effectively integrates multi-view information.

Appendix E Evaluation on TAPVid-2D

In Tab. E.1, we evaluate MVTracker on the TAPVid-2D benchmark by projecting our predicted 3D trajectories onto image views. As our method requires depth input, we report results using various monocular depth estimators. However, tracking performance is significantly degraded in this benchmark, which is in part attributable to TAPVid-2D containing outdoor unbounded scenes, our scene normalization not aligning scenes well to the training distribution, and monocular depth estimation frequently failing or flickering.

Appendix F Evaluation Metrics

There is no established set of metrics for multi-view point tracking in 3D. Thus, we adopt four metrics from prior monocular point-tracking benchmarks [7], extending them to our multi-view 3D setting. These metrics measure the quality of the predicted 3D trajectories ($\delta_{\text{avg}}$, MTE), the ability to predict whether a point is visible in any of the views (OA), or both simultaneously (AJ). We compute all metrics per-trajectory before averaging across all tracks in a scene, and then across all scenes in a dataset.

Let $N$ be the number of tracked points, $T$ the number of frames, and $\hat{p}_{t}^{i}$ (resp. $p_{t}^{i}$) the predicted (resp. ground-truth) 3D location of track $i$ at time $t$. Similarly, let $\hat{v}_{t}^{i}\in\{0,1\}$ and $v_{t}^{i}\in\{0,1\}$ be the predicted and ground-truth visibility flags.

Median Trajectory Error (MTE).

For each track $i$, the median trajectory error measures the median distance between predicted and ground-truth locations over timesteps where the track is visible:

[TABLE]

The overall MTE is the mean across all $N$ tracks:

[TABLE]

Occlusion Accuracy (OA).

Occlusion accuracy quantifies how well the model predicts visibility:

[TABLE]

Average Location Accuracy ($\delta_{\text{avg}}$).

We evaluate the fraction of points within each of $H$ distance thresholds $\{x_{1},\dots,x_{H}\}$ in centimeters. For a single threshold $x$, define the following:

[TABLE]

We then average over all tracks and thresholds:

[TABLE]

Average Jaccard (AJ).

This metric jointly assesses spatial and occlusion accuracy at each threshold $x$. Define \alpha_{t,x}^{i}=\mathds{1}\!\bigl{(}\|\hat{p}_{t}^{i}-p_{t}^{i}\|_{2}<x\bigr{)}. Then, for track $i$:

[TABLE]

Averaging over $N$ tracks yields

[TABLE]

F.1 Evaluation Details

For MV-Kubric and DexYCB, we sample 512 points uniformly from object surfaces and report metrics averaged across static and dynamic points. Panoptic Studio often lacks labeled static objects, so we sample 512 points and compute metrics over all of them. We use four views in the main evaluation tables: for DexYCB, these are the first four calibrated cameras; for Panoptic Studio, we select distant views; and for MV-Kubric, views are randomly sampled per scene. The sensitivity to the selected views is analyzed in Tab. 3. Jaccard and location accuracy thresholds are set to 1/2/5/10/20 cm for DexYCB, 0.65/1.3/2.6/5.2/10.4 simulation-scale-adjusted centimeters for MV-Kubric (1 Kubric unit is about 13 cm before simulation-scale adjustment), and 5/10/20/40 cm for Panoptic Studio.

Panoptic Studio trajectories were generated by merging per-view monocular labels from TAPVid-3D [7], filtered from Dynamic 3DGS [23] 27-view predictions. These labels are often noisy and erroneous, and for example, exhibit point drift (e.g., labels jumping from an elbow to a finger), which is why we use higher evaluation thresholds on this benchmark and recommend interpreting results as well as using the benchmark with caution. MV-Kubric labels are derived from simulated ground-truth trajectories, while DexYCB tracks are extracted from fitted object meshes provided with the dataset. For DexYCB, we sample points on annotated surfaces and track their position using barycentric coordinates of the deforming mesh triangles. Visibility labels on DexYCB can still be noisy due to imperfect Kinect depth maps and segmentation boundaries. The scripts used to process all datasets and generate labels are publicly available in our codebase.