KOM-SLAM: A GNN-Based Tightly Coupled SLAM and Multi-Object Tracking Framework

TL;DR

KOM-SLAM is a new framework that combines SLAM and multi-object tracking using GNNs to improve performance in dynamic scenes.

Contribution

The novel contribution is a tightly coupled framework using GNNs for joint keypoint and object association with differentiable pose estimation.

Findings

KOM-SLAM constructs a spatiotemporal graph for joint keypoint and object associations.

The framework uses a differentiable pose estimation method to directly supervise association learning.

Experiments on KITTI Tracking show improved localization and object tracking performance.

Abstract

Coupled simultaneous localization and mapping (SLAM) and multi-object tracking have been studied in recent years. Although these tasks achieve promising results, they mainly associate keypoints and objects across frames separately, which limits their robustness in complex dynamic scenes. To overcome this limitation, we propose KOM-SLAM, a tightly coupled SLAM and multi-object tracking framework based on a Graph Neural Network (GNN), which jointly learns keypoint and object associations across frames while estimating ego-poses in a differentiable manner. The framework constructs a spatiotemporal graph over keypoints and object detections for association, and employs a multilayer perceptron (MLP) followed by a sigmoid activation that adaptively adjusts association thresholds based on ego-motion and spatial context. We apply a soft assignment on keypoints to ensure differentiable pose estimation, enabling the pose loss to supervise the association learning directly. Experiments on the KITTI Tracking demonstrate that our method achieves improved performance in both localization and object tracking.

Figures

Full text

1. Introduction

Simultaneous localization and mapping (SLAM) and multi-object tracking are two essential tasks for autonomous systems. SLAM estimates the ego-pose (the current position and orientation of the sensor/robot) and builds a map of the surrounding environment [1,2], while multi-object tracking tracks the motion of detected objects to support safe decision-making [3,4]. Although these tasks are traditionally developed independently, the two tasks are naturally complementary. SLAM provides accurate ego-pose estimates for tracking, while object tracking helps identify dynamic objects, improving SLAM robustness in dynamic environments.

Tightly coupled SLAM and multi-object tracking systems usually build a factor graph of odometry and object states, and a joint optimization is applied to improve the accuracy and robustness of localization and object tracking using temporal and spatial constraints [5,6]. However, most existing approaches rely on manually designed heuristics for data association, such as descriptor similarity for keypoints and spatial proximity for objects. These heuristic associations can be brittle in cluttered or ambiguous environments, particularly when multiple dynamic objects occlude or overlap. Additionally, treating keypoint and object associations independently prevents full exploitation of the shared information between feature extraction and object detection.

Graph Neural Networks (GNNs) have emerged as powerful tools for learning associations in structured data by leveraging message passing to aggregate information from connected neighbors [7,8]. While GNNs have been applied to tasks like keypoint matching and object tracking independently [9,10], they have not been fully explored in a tightly coupled SLAM and multi-object tracking framework, where keypoint association and object association could be learned jointly. Moreover, most existing GNN-based approaches are not designed to support differentiable pose estimation, limiting their integration into end-to-end learning pipelines.

In response to these challenges, we propose KOM-SLAM, a tightly coupled SLAM and multi-object tracking framework, which jointly matches keypoints and objects across frames with a GNN. Our approach builds a graph that includes both keypoints and object detections across frames, enabling the GNN to simultaneously find the association between keypoints and between objects. Furthermore, to improve robustness under varying motion patterns, we apply a Multilayer Perceptron (MLP)–Sigmoid gating layer to restrict the association dynamically based on ego-motion and the spatial distance of each keypoint from the ego-pose. In addition, we adopt a soft assignment scheme over the keypoint similarity matrix, which enables differentiable pose estimation and allows the final odometry loss to directly supervise the GNN association learning.

The main contributions of the new proposed GNN-based, tightly coupled SLAM and multi-object tracking system are the following:

• 1.We propose KOM-SLAM, a GNN-based, tightly coupled SLAM and multi-object tracking framework, where the GNN jointly learns associations between keypoints and objects. A soft assignment mechanism is applied to backpropagate the pose estimation loss through the GNN. To the best of our knowledge, this is the first learning-based system that tightly integrates SLAM and multi-object tracking.
• 2.We embed the ego motion and spatial distance between the keypoint and the ego pose in the network to allow the dynamic adjustment of the keypoint matching range.
• 3.We validate the effectiveness of KOM-SLAM on the KITTI Tracking dataset, demonstrating improved performance in both pose estimation and object tracking.

The rest of the paper is organized as follows. Section 2 reviews the related work. Section 3 presents the proposed GNN-based, tightly coupled SLAM and multi-object tracking framework in detail. Experimental results and evaluations are demonstrated in Section 4. Finally, Section 5 concludes the paper and discusses directions of future work.

2. Related Work

2.1. Coupled SLAM and Multi-Object Tracking

Traditional SLAM systems rely on feature extraction and frame-to-frame correspondence to estimate ego-motion, using corner keypoints in visual inputs [11] or edge and planar points in Light Detection and Ranging (LiDAR) data [12]. However, including keypoints on dynamic objects in pose estimation introduces errors due to unmodeled motion. To address this, dynamic observation methods have been proposed to filter out potentially dynamic points—either by removing features associated with detected objects [13,14] or by segmenting dynamic regions using photometric or geometric cues [15]. The robustness of these methods is heavily dependent on the quality of the road scene understanding. Recent benchmarks, such as RSUD20K [16], have advanced the state-of-the-art in road scene understanding by providing diverse and high-resolution data for object detection. While effective in avoiding dynamic noise, these approaches often lack the ability to model or track object motion, limiting the SLAM system’s robustness in dynamic environments.

To improve robustness in dynamic environments, loosely coupled systems combine SLAM and multi-object tracking, where each component operates independently but shares information to enhance overall performance. In these frameworks, object tracking typically helps identify and remove dynamic points from sensor data, while SLAM uses the filtered static features to perform accurate ego-pose estimation. MaskFusion [17] and DOT [18], the visual dynamic object tracking methods for SLAM, rely on reprojection or photometric errors to track object motion with object masks and identify static keypoints. Instead of tracking image masks, ClusterSLAM [19] and ClusterVO [20] project keypoints into 3D, track 3D clusters of points, and estimate camera poses based on static features. LiDAR-based methods take advantage of the accurate depth measurement and object detection, often using Kalman filtering and point cloud registration to track objects [21]. Hybrid LiDAR-visual systems, such as Dynam-LVIO [22], benefit from sensor redundancy and use both the iterative closest point (ICP) error and reprojection error to update a Kalman filter to track objects. While these loosely coupled designs are modular and flexible, their ego-poses estimation only relies on the static features, which may limit their effectiveness in highly dynamic scenes where static features are not sufficient.

In contrast to loosely coupled methods, tightly coupled SLAM and multi-object tracking frameworks typically jointly optimize ego-poses, object poses, and object motions with a factor graph in the backend. This joint optimization allows dynamic object observations and motion models to directly support ego-pose estimation, which is particularly beneficial when static features are sparse or occluded. Visual systems usually calculate reprojection errors as constraint functions in the joint optimization. DynaSLAM II [5] and CubeSLAM [23] detect 2D boxes and optimize object points with factor graphs, while segmentation-based approaches like TwistSLAM [24] apply optical flow to track the object points. Recent advances also explore richer representations, including quadrics [6] and articulated models for non-rigid bodies [25], improving robustness across object types. The LiDAR-based methods, such as DL-SLOT [26] and LIMOT [27], benefit from accurate 3D detections and pay more attention to the object trajectory prediction and association. Instead of applying constant motion constraints, LIO-LOT [28] applies a constant acceleration and angular velocity motion model for objects. IMM-SLAMMOT [29] includes multiple motion models in the factor graph to be more adaptive to the complex real-world dynamic scenarios. The switching-coupled systems combine the loosely coupled factors to avoid unreliable objects causing performance degradation [30,31]. Despite their effectiveness, most existing tightly coupled methods rely on heuristic-based association strategies, leaving room for learning-based techniques to better capture complex correspondence patterns.

2.2. GNN-Based Matching

Graph Neural Networks (GNNs) have emerged as a powerful tool for learning associations in structured data, such as object tracking and keypoint matching across frames. In multi-object tracking, Wang et al. use the 2D appearance feature as nodes, define the edges between the nodes within pixel distance thresholds across frames, and apply GraphConv as their aggregation method [32]. Similarly, PTP [33] connects edges based on 3D distance and applies GraphConv; however, it utilizes Long Short-Term Memory (LSTM) and MLP to encode 3D detection results into positioning features as GNN nodes. Instead of using GraphConv, GNN3DMOT [3] improves the tracking performance by applying an attention-based weight to aggregate the difference in features between nodes and neighborhood nodes. Brasó and Leal-Taixé build a graph with multiple frames and design a time-aware message-passing network to update the node features [34]. Following it, SUSHI [10] and Bilgi et al. [35] perform detection association over different hierarchies of timespans to improve long-term association. At each hierarchical level, a shared-weights GNN is applied. Chen et al. apply GNN to multi-object tracking in satellite videos and focus on the tiny object tracking task [36]. Instead of using heuristic methods to define GNN edges, Gao et al. design NodeNet to generate the edge connections for GNN, constructed by an encoder and a decoder [37].

In keypoint matching, SuperGlue [9] represents keypoints as graph nodes with both descriptor and positioning features. It uses alternating self-attention layers, where nodes in the same frame are fully connected, and cross-attention layers, where nodes are fully connected across frames. Following SuperGlue, LightGlue [38] uses the positioning information to calculate the attention scores in self-attention layers and uses descriptor similarity in cross-attention layers. To improve the time efficiency of SuperGlue, SGMNet [39] applies a seeding module to generate a small set of matches as seeds. Subsequently, a Seeded GNN is applied to utilize seed matches to pass messages across frames and within frames through attentional aggregation. Similarly, ClusterGNN [40] reduces the number of edges by operating GNN on clusters of keypoints. They use the K-means algorithm to cluster the query and key vectors, and then apply a cluster-based sparse attention to aggregate the features. Furthermore, LoFTR [41] combines the GNN with a local feature detector to build an end-to-end trainable model. It enables the matching model to supervise the local feature convolutional neural network (CNN), which outperforms using a pre-built keypoint detector. These works collectively demonstrate GNNs’ strength in capturing contextual cues and finding correspondence in matching tasks. However, most existing works perform the object association and keypoint association independently, and the GNN approaches are not fully explored in the coupled object and keypoint matching.

3. Method

3.1. System Architecture

The overall structure of the proposed GNN-based coupled SLAM and multi-object tracking system is illustrated in Figure 1. The system takes the extracted keypoints and object detections from two frames of perception data at timestamps A and B as input. Its objective is to establish associations between keypoints and between objects across frames and estimate the relative transformation $[eqn]$ , which maps coordinates from frame A to frame B. In practice, we use frames A and B that are temporally adjacent (i.e., consecutive frames), as this ensures sufficient spatial overlap and stable geometric consistency for both keypoint and object associations. The framework does not strictly require consecutive frames, but larger temporal gaps generally increase motion magnitude and reduce association reliability. The impact of reduced frame density and increased time intervals is empirically evaluated in Section 4.4.3.

Each keypoint i in frame $[eqn]$ is represented as a tuple $[eqn]$ , where $[eqn]$ denotes the 3D position, and $[eqn]$ is the corresponding descriptor. Similarly, each object detection j in frame t is defined as $[eqn]$ , where $[eqn]$ denotes the 3D bounding box parameters (position, size, and yaw angle) and $[eqn]$ is the associated appearance feature.

In practice, the input data $[eqn]$ and $[eqn]$ are derived from stereo image inputs. Specifically, 2D keypoints and their descriptors are extracted from left images using SuperPoint [11], and their 3D coordinates $[eqn]$ are estimated by triangulation using depth inferred from stereo images and camera intrinsic matrix. Object detections are generated by PointRCNN [42], which provides 3D bounding box parameters $[eqn]$ . The associated appearance features $[eqn]$ are extracted by applying a pretrained ResNet backbone to the cropped image region of each detected object.

The features and positioning information of keypoint and object are encoded as nodes in an attention-based GNN, including self-attention, keypoint–object, and cross-attention layers. In the GNN layers, the positioning attributes remain fixed and are used to guide the attention mechanism, while the feature vectors are the only components updated. The updated features are then used to compute matching score matrices across frames. For keypoints, an auxiliary matrix is additionally introduced to guide the similarity computation by incorporating the estimated relative transformation and spatial distances between the keypoints and the ego-poses.

Let $[eqn]$ and $[eqn]$ denote the number of keypoints extracted in timestamps A and B, respectively. Soft assignment is applied to the keypoint matching score matrix $[eqn]$ to obtain the corresponding keypoints $[eqn]$ . These correspondences are then used to estimate the relative transformation $[eqn]$ via a differentiable optimization module. For object associations, a set of object correspondences $[eqn]$ is established through the object matching score matrix $[eqn]$ , where $[eqn]$ and $[eqn]$ are the number of detected objects in frames A and B, respectively. Unlike keypoints, soft assignment is not applied to objects due to their discrete and non-repeatable nature.

Finally, ego-pose optimization is performed by minimizing the alignment error of the matched keypoints. To further refine both pose and object trajectories, a factor graph is constructed over a sliding window to jointly optimize relative transformations, object poses, and object motions. The use of soft associations for keypoints, together with differentiable optimization, ensures that the entire framework is trainable.

3.2. Graph Neural Network

As illustrated in Figure 2, the proposed attention-based GNN jointly processes keypoints and object detections from two frames by propagating and refining node features through geometric and appearance-based relationships both within and across timestamps.

Each graph node is represented as a pair $[eqn]$ , where $[eqn]$ denotes the learnable feature vector (keypoint descriptor $[eqn]$ or object appearance feature $[eqn]$ ), and $[eqn]$ denotes the corresponding positional attribute (keypoints 3D position $[eqn]$ or object bounding box parameters $[eqn]$ ). While $[eqn]$ remains fixed throughout message passing, it is used in the attention mechanism to construct queries and keys, thereby guiding how information flows between nodes. Only the feature $[eqn]$ is updated layer by layer.

GNN edges are constructed differently across three layers to capture complementary contextual relationships. In the self-attention layer, keypoint nodes are connected to other keypoints within a predefined distance threshold in the same frame. Similarly, object nodes are connected to nearby object nodes in the same frame. This layer captures local spatial relationships. In the keypoint–object layer, the keypoint nodes and object nodes in the same frame are connected if a keypoint lies inside the 3D bounding box of the object, allowing keypoints to incorporate higher-level object context. Lastly, in the cross-attention layer, keypoints are connected to keypoints in the other frame within a spatial threshold. Likewise, object nodes are connected across frames based on spatial distance. This layer facilitates temporal correspondence reasoning across frames.

At each layer of GNN, node features are updated through attention-based message passing from its connected neighbors. Specifically, the updated feature $[eqn]$ for node i is calculated by combining the original feature $[eqn]$ with the aggregated message $[eqn]$ via a 2-layer MLP, as defined in Equation (1):

$$[eqn]$$

where $[eqn]$ is the updated GNN node, and $[eqn]$ denotes concatenation operation. The 2-layer architecture is chosen to provide sufficient expressive power for feature refinement while maintaining computational efficiency and preventing over-fitting.

The aggregated message $[eqn]$ is defined as a weighted sum over neighbors $[eqn]$ , calculated by Equations (2) and (3):

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ , $[eqn]$ , and $[eqn]$ denote the value, query, and key vectors, respectively. These vectors are obtained by applying different linear layers to the node features. The specific choice of input features depends on the type of layer, as shown in Figure 2. In the self-attention layers, keypoints embed positioning information for queries and keys, while their descriptors are applied to generate value vectors. Object nodes follow the same pattern, using bounding boxes to generate queries and keys. In the keypoint–object layers, queries are derived from keypoint positions and keys from object bounding boxes, while the values are taken from either keypoint descriptors or object features, enabling information exchange between keypoints and objects. Finally, in the cross-attention layers, queries, keys, and values are all constructed from descriptors (for keypoints) or appearance features (for objects), so that message passing emphasizes similarity in feature space across frames.

Each GNN layer is applied sequentially, and multiple iterations of message passing can be performed to allow contextual information to propagate throughout the graph. The updated keypoint and object features are then used for computing similarity matrices and downstream estimation tasks.

3.3. Matching Score Matrix Calculation

To determine correspondences between keypoints and between objects across two frames A and B, we compute two matching score matrices for keypoints and objects based on the updated node features obtained from the GNN. Since the processes are similar, we describe the calculation for keypoints, while the object case follows identically.

Let $[eqn]$ and $[eqn]$ denote the updated descriptors matrices for keypoints in frames A and B, respectively, where each row corresponds to $[eqn]$ , the updated features of keypoint i in frame t. We first calculate a similarity matrix $[eqn]$ using the inner product of the feature matrices, as shown in Equation (4):

$$[eqn]$$

We define the raw matching score matrix $[eqn]$ as the sum of the row-wise and column-wise softmax normalizations of $[eqn]$ in Equation (5):

$$[eqn]$$

In addition to descriptor similarity, we introduce an auxiliary matrix $[eqn]$ based on keypoint positions and estimated ego-motion. As shown in Equation (6), we compute a learned threshold $[eqn]$ using a lightweight network, which estimates the possible range that the corresponding keypoint might be in for each keypoint:

$$[eqn]$$

where $[eqn]$ is the Euclidean distance from keypoint i to the ego pose in frame A, and $[eqn]$ and $[eqn]$ represent the estimated translational and rotational velocities of ego-motion between frames A and B. The auxiliary matrix, which acts as a learned spatial prior, is then defined using the sigmoid function $[eqn]$ in Equation (7):

$$[eqn]$$

where $[eqn]$ is defined as the distance between the keypoint pair $[eqn]$ .

Finally, the keypoint matching score matrix $[eqn]$ is obtained by element-wise multiplication of the raw matching score matrix $[eqn]$ and the auxiliary matrix $[eqn]$ , as shown in Equation (8):

$$[eqn]$$

This fused matrix $[eqn]$ is used for soft keypoint matching and subsequent relative pose estimation. Note that for objects, the raw matching score matrix is directly used as the matching score matrix.

3.4. Correspondence Association and Joint Optimization

Given the matching score matrices computed from the updated keypoint and object features, our framework identifies inter-frame correspondences and jointly optimizes the relative ego-motion and object states.

Object correspondences are first determined through an argmax operation combined with a mutual consistency check. Let $[eqn]$ denote the object-level matching score matrix between the $[eqn]$ and $[eqn]$ detected objects in frames A and B, respectively. To ensure that both objects select each other as the most similar match, a candidate correspondence $[eqn]$ is selected if both conditions in Equations (9) and (10) are met:

$$[eqn]$$ $$[eqn]$$

Keypoint correspondences are derived using soft assignment. Given the keypoint score matrix $[eqn]$ , the corresponding location in frame B for each keypoint $[eqn]$ in frame A is estimated as a weighted average, as defined in Equation (11):

$$[eqn]$$

Crucially, this soft assignment mechanism is designed to maintain differentiability, allowing the matching scores to be directly supervised by the downstream odometry loss.

The resulting set of soft correspondences $[eqn]$ is then used to estimate the relative transformation $[eqn]$ by minimizing the sum of squared alignment errors $[eqn]$ defined in Equation (12):

$$[eqn]$$

This optimization is solved using the Levenberg–Marquardt (LM) algorithm [43], implemented in a differentiable form with Theseus [44]. This transformation aligns keypoints from frame A to frame B and provides an initial estimate of ego-motion. For the initial guess of $[eqn]$ in the optimization, we use a constant motion model. For the first frame, $[eqn]$ is initialized as an identity transformation.

To further refine the ego-motion estimate and incorporate object dynamics, we employ a joint optimization framework with a factor graph following [27]. The graph consists of ego-pose nodes, object pose nodes, and object motion nodes spanning a fixed-length sliding window (selected to be 5) of recent frames. Let $[eqn]$ denote the set of all variables in the graph. The following factors are defined: (i) odometry constraints $[eqn]$ between consecutive ego-poses; (ii) observation constraints $[eqn]$ linking ego-poses to observed object poses; (iii) motion constraints $[eqn]$ connecting consecutive object poses and motion; and (iv) constant velocity constraints $[eqn]$ between object motion nodes, as shown in Equation (13):

$$[eqn]$$

where $[eqn]$ denotes the optimized factor graph nodes. The optimization is again performed using the LM algorithm [43] with Theseus [44], maintaining full differentiability. This joint optimization allows for mutual refinement of ego-motion and object tracking, enhancing both consistency and accuracy. Crucially, both the soft assignment mechanism and the factor graph optimization are implemented in a differentiable manner, forming the foundation for our learning pipeline.

3.5. Training Loss

To supervise the learning process of KOM-SLAM, we design a composite loss function that supervises object association, keypoint correspondence estimation, and ego-motion prediction. The overall loss is defined in Equation (14):

$$[eqn]$$

where $[eqn]$ , $[eqn]$ , and $[eqn]$ are weighting coefficients.

Ground-truth tracking IDs are used to establish object correspondences between frames A and B. Let $[eqn]$ denote the ground truth object matching score matrix, where $[eqn]$ for the objects with the same tracking IDs, $[eqn]$ for objects with different tracking IDs. Following PTP [33], we apply the BCE loss for each entry of $[eqn]$ and the CE loss for each column of $[eqn]$ , as shown in Equation (15):

$$[eqn]$$

Ground truth matched keypoint pairs are identified using both spatial distance and descriptor similarity after aligning keypoints in frame A to the coordinate frame of B using the ground-truth relative transformation $[eqn]$ . A pair $[eqn]$ is considered positive if the spatial distance (Equation (16)) and descriptor similarity (Equation (17)) criteria are met:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ and $[eqn]$ are distance and descriptor similarity thresholds, respectively. Let $[eqn]$ denote the sets of ground truth matched keypoint pairs. As shown in Equation (18), following LightGlue [38], the keypoint matching loss is formulated as mean negative log-likelihood over the ground truth matched pairs:

$$[eqn]$$

To supervise ego-motion estimation, we penalize both translation and rotation errors between the predicted transformation $[eqn]$ and the ground-truth $[eqn]$ . Let $[eqn]$ denote the estimated and ground-truth unit quaternions, and $[eqn]$ denote the translation vectors. The transformation loss, $[eqn]$ , is defined as the sum of the rotation and translation errors, as shown in Equation (19):

$$[eqn]$$

where $[eqn]$ is a weighting factor to balance the contributions of rotational and translational errors. The factor $[eqn]$ is introduced for scale normalization, which is necessary because the rotational error (distance between unit quaternions) and the translational error (in meters) typically have different magnitudes. This ensures that both components contribute equitably to the overall loss gradient during training.

4. Experiments

We conduct experiments on the KITTI Tracking dataset [45] to evaluate the effectiveness of KOM-SLAM. Section 4.1 describes the implementation details and dataset setup. The experimental evaluation is organized into two main parts: Section 4.2 presents the results for odometry estimation, while Section 4.3 evaluates the multi-object tracking performance.

4.1. Experimental Details

For keypoint extraction, we apply SuperPoint [11], a self-supervised keypoint detector and descriptor, to generate keypoints from the images of the KITTI Tracking dataset. To obtain 3D object detections, we adopt PointRCNN [42], a two-stage 3D object detector that operates on LiDAR point clouds. For both models, we choose to use the pretrained weights available online. To fully evaluate KOM-SLAM on all the sequences of the KITTI Tracking dataset, we train two models. One is trained using sequences 0000–0010, while the other is trained with 0011, 0013, 0014, 0018, 0019, and 0020. This separation is necessary because we avoid evaluating a sequence using a model that has been trained on it. By training two models on complementary subsets, we ensure that every sequence can be used for evaluation without overlapping with its training data. Following prior works [5,24], the sequences 0012, 0015, 0016, and 0017 are excluded from the evaluation because they contain little or no camera motion. All models are trained using the Adam optimizer with a learning rate of $[eqn]$ for 80 epochs and a batch size of 1. The GNN module consists of two layers, each integrating self-attention, cross-attention, and keypoint–object interaction mechanisms, with a feature dimension of 256. The training loss is a weighted combination of object association loss, keypoint association loss, and odometry consistency loss, with weights set to $[eqn]$ , $[eqn]$ , and $[eqn]$ , respectively. The overall balance factor is set to $[eqn]$ . These hyperparameters were determined through a grid search. We observed that the system maintains stable performance within a $[eqn]$ range of the specified $[eqn]$ values. However, we found that reducing the factor $[eqn]$ causes an increase in rotation error, because rotational errors are numerically smaller than translational errors. The proposed framework runs at approximately 49 ms per frame, where the GNN module takes approximately 22 ms per frame, and the backend optimization costs approximately 24 ms per frame. All experiments are conducted on a desktop system equipped with an NVIDIA GeForce RTX 4070 GPU. Figure 3 illustrates an example of the joint keypoint and object association results generated by KOM-SLAM on the KITTI Tracking dataset, highlighting successful correspondence in a challenging dynamic scene.

4.2. Odometry Estimation

This subsection presents the evaluation of ego-pose estimation performance using KOM-SLAM on the KITTI Tracking dataset. Specifically, sequences 0000–0010 are evaluated using a model trained on sequences 0011, 0013, 0014, 0018, 0019, and 0020, while the remaining sequences are tested using the model trained on 0000–0010. This split ensures that the test sequences are not seen during training, enabling a fair evaluation of generalization performance. We use the Relative Pose Error (RPE) as the evaluation metric to assess the accuracy of the estimated ego motion. RPE measures the local accuracy of the transformation between two consecutive frames and is particularly suitable for evaluating methods that emphasize pairwise frame association, such as ours.

Table 1 compares our method with three established baselines: ORB-SLAM2 [1], DynaSLAM [46], and DynaSLAM2 [5]. KOM-SLAM outperforms the baselines in both translational (RPEt) and rotational (RPER) components on the majority of sequences. Notably, it achieves the lowest mean RPEt of 0.041 m/frame and the lowest mean RPER of 0.028°/frame across all test sequences. This demonstrates the effectiveness of incorporating both keypoint- and object-level associations for robust ego-motion estimation in dynamic environments. Additionally, KOM-SLAM achieves the lowest standard deviation in translation and a standard deviation in rotation comparable to the best-performing method (0.020 vs. 0.015 of DynaSLAM [46]), demonstrating stable and consistent pose estimates across different sequences.

4.3. Multi-Object Tracking

This subsection presents the evaluation of multi-object tracking performance using KOM-SLAM on the KITTI Tracking dataset. We follow the evaluation of DynaSLAM2 [5], and compare object pose estimation accuracy across 12 longest object trajectories of the KITTI tracking dataset. We adopt standard evaluation metrics to assess the tracking performance: True Positives (TP) and Multi-Object Tracking Precision (MOTP), computed in 2D image space, Bird’s-Eye View (BV), and full 3D space. TP reflects the percentage of correctly tracked frames with respect to ground-truth annotations, while MOTP measures the average spatial alignment error of matched objects. Higher TP and MOTP indicate better tracking performance and localization accuracy. Table 2 summarizes the comparison between KOM-SLAM and DynaSLAM2. KOM-SLAM achieves higher TP and MOTP across all three modalities—2D, BV, and 3D—demonstrating substantial improvements in both tracking continuity and pose accuracy.

4.4. Ablation Study

To systematically analyze the contribution of each key component in KOM-SLAM, we conduct an ablation study focusing on the keypoint–object layer, attention-based GNN, the MLP–Sigmoid gating layer, the keypoint soft assignment mechanism, and the robustness under varying keypoint densities and temporal spacing.

4.4.1. Ego Pose Estimation Ablation

Table 3 demonstrates the ego-pose estimation results for several ablated variants on the KITTI Tracking dataset. Specifically, we evaluate the following: (1) No Keypoint–Object Layer, where the keypoint–object interaction layer in the GNN is removed, and only the self-attention and cross-attention layers are used; (2) No GNN, where the attention-based GNN is removed, and associations are computed directly from raw keypoint features; (3) No Gating Layer, where the MLP–Sigmoid gating mechanism is disabled, and association scores are determined with the raw matching score matrix $[eqn]$ ; (4) No Soft Assignment, where hard assignment based on matching score matrix is used instead of soft assignment; and (5) KOM-SLAM, which corresponds to the full KOM-SLAM framework.

Across most sequences, the translational error (RPEt) remains relatively stable among different variants, indicating that coarse geometric alignment can still be obtained without learned association refinement. In contrast, the rotational error (RPER) exhibits a clear sensitivity to the association strategy. In particular, removing the keypoint–object layer leads to noticeably higher rotation errors and variance, highlighting its importance for stable and accurate pose estimation.

The full model consistently achieves the lowest mean rotation error and standard deviation, confirming that the combination of keypoint–object interaction, GNN-based feature refinement, MLP–Sigmoid gating, and differentiable soft assignment is essential for robust ego-motion estimation. These results validate the necessity of our core design choices rather than relying on any single component alone.

4.4.2. Object Tracking Association Ablation

To evaluate the effectiveness of the GNN-based association module for object tracking, we compare three variants using standard Multi-Object Tracking metrics: Multi-Object Tracking Accuracy (MOTA) and ID F1 Score (IDF1), as shown in Table 4. The compared methods include the following: (1) a Naive Approach, which associates objects solely based on the minimum Euclidean distance between the bounding box centers of the current detection and the predicted objects from the previous information; (2) No GNN, which uses raw appearance features without GNN-based refinement; (3) No Keypoint–Object Layer, which removes the keypoint–object interaction layer in GNN and only uses self-attention and cross-attention layers; and (4) KOM-SLAM, the full KOM-SLAM association pipeline.

For this ablation, ground-truth object detections are used to isolate the influence of the association mechanism and eliminate variance introduced by the upstream 3D detector. The naive distance-based approach performs poorly in sequences with frequent occlusions and interacting objects, leading to low MOTA and IDF1 scores. No GNN variant and no keypoint–object layer variant improve performance in most sequences by leveraging appearance cues, but still struggle with identity consistency under complex interactions.

The full model consistently achieves the highest MOTA and IDF1 scores in most sequences, demonstrating that the GNN architecture effectively captures relationships between objects. This results in more robust identity preservation and significantly improved tracking precision compared to heuristic or association strategies based on raw appearance feature matching.

4.4.3. Data Density and Time Gap Analysis

To assess the robustness of KOM-SLAM under practical sensing constraints, we further analyze the impact of reduced keypoint density and increased temporal spacing between frames, with results summarized in Table 5. For the Sparser Keypoints variant, 50% of the detected keypoints are randomly retained in each frame. For the Every 2 Frames variant, only every second frame in each sequence is used for pose estimation.

When keypoint density is reduced, KOM-SLAM maintains comparable mean and standard deviation of the relative pose error, demonstrating strong robustness to sparse visual features. This indicates that the learned association mechanism can effectively find correspondences even with limited observations.

In contrast, increasing the temporal gap between frames results in noticeable degradation, particularly in rotation error. This effect is most pronounced in sequences with sharp turns (e.g., 0000, 0004, 0007), where larger inter-frame motion violates the small-motion assumption implicitly used in pairwise matching. These results highlight the importance of temporal resolution for accurate motion estimation, while also demonstrating that KOM-SLAM degrades gracefully under challenging conditions rather than failing catastrophically in sequences without large inter-frame motion.

5. Conclusions and Future Work

In this work, we introduced KOM-SLAM, a GNN-based framework that tightly couples odometry estimation and multi-object tracking. Our method constructs a spatiotemporal graph over keypoints and objects across frames, enabling the GNN to jointly update their features for robust association. A differentiable soft assignment mechanism is integrated with odometry estimation, allowing pose loss to directly supervise the learning process. To further enhance the robustness of keypoint association, we incorporate ego-motion and spatial priors into the matching process through an MLP–Sigmoid gating layer. We evaluated KOM-SLAM on the KITTI Tracking dataset, where it consistently outperformed strong baselines. The results demonstrate notable gains in both odometry accuracy and multi-object tracking precision, particularly in challenging sequences with frequent object motion and occlusion.

While KOM-SLAM demonstrates improvements in ego-motion estimation and multi-object tracking, the current framework presents several limitations that motivate future work.

Firstly, the system relies on high-quality external perception modules (SuperPoint for keypoints and PointRCNN for 3D detection). This dependency means the system’s overall performance is bounded by the accuracy and completeness of these detectors. Our learning pipeline currently mainly supervises the association, leaving the feature extraction and detection unsupervised.

Additionally, the system relies on frame-independent object detection and feature extraction, which can suffer from temporal inconsistency in bounding box parameters. While the inherent redundancy of keypoints provides robustness to this inconsistency in ego-pose estimation, the detection inconsistency severely challenges the factor graph optimization, as the constant velocity constraints applied to dynamic objects struggle to reconcile the detection noise with the smooth motion model, thereby risking the degradation of object trajectory accuracy and potentially destabilizing the coupled ego-pose estimate.

Furthermore, the system’s reliance on consecutive frame matching and a constant motion model for initialization makes the method sensitive to large motion changes. As demonstrated in our ablation study (matching every 2 frames), extremely high-speed scenarios or sequences with significant rotational velocity changes between frames can degrade performance, as the geometric prediction becomes unreliable.

Finally, while we demonstrate the excellent RPE performance, the current framework lacks long-term robustness mechanisms such as global mapping consistency and loop closure. The integration of GNN-based re-localization is necessary to prevent long-term drift in the ego-pose and object trajectories.

Building upon these limitations, future efforts will concentrate on three main directions: (1) integrating the front-end keypoint extraction and temporally consistent object detection modules into the differentiable pipeline for a truly end-to-end trainable system. (2) Exploring extensions to global consistency and long-term tracking, potentially by introducing map nodes into the GNN and designing a global loop closure factor for the optimization backend. (3) Expanding the evaluation and robustness analysis by applying KOM-SLAM to other diverse and challenging datasets, such as nuScenes [47] or Waymo [48], and also comparing with additional state-of-the-art visual odometry methods, such as DROID-SLAM [49] and DeepV2D [50].