KPMapNet: Keypoint Representation Learning for Online Vectorized High-Definition Map Construction

TL;DR

KPMapNet improves the accuracy of creating high-definition maps for autonomous vehicles by learning keypoint representations and reducing errors in map construction.

Contribution

KPMapNet introduces a new keypoint-based representation and training framework for vectorized HD map construction.

Findings

KPMapNet achieves state-of-the-art performance with 75.1 mAP on nuScenes and 74.2 mAP on Argoverse2.

The new representation preserves geometric features better than conventional methods.

Visualizations confirm improved accuracy in map generation outcomes.

Abstract

Vectorized high-definition (HD) map construction is a critical task in the autonomous driving domain. The existing methods typically represent vectorized map elements with a fixed number of points, establishing robust baselines for this task. However, the inherent shape priors introduce additional shape errors, which in turn lead to error accumulation in the downstream tasks. Moreover, the subtle and sparse nature of the annotations limits detection-based frameworks in accurately extracting the relevant features, often resulting in the loss of fine structural details in the predictions. To address these challenges, this work presents KPMapNet, an end-to-end framework that redefines the ground truth training representation of vectorized map elements to achieve precise topological representations. Specifically, the conventional equidistant sampling method is modified to better preserve the geometric features of the original instances while maintaining a fixed number of points. In addition, a map mask fusion module and an enhanced hybrid attention module are incorporated to mitigate the issues introduced by the new representation. Moreover, a novel point-line matching loss function is introduced to further refine the training process. Extensive experiments on the nuScenes and Argoverse2 datasets demonstrate that KPMapNet achieves state-of-the-art performance, with 75.1 mAP on nuScenes and 74.2 mAP on Argoverse2. The visualization results further corroborate the enhanced accuracy of the map generation outcomes.

Figures

Full text

1. Introduction

High-definition (HD) maps are critical components in autonomous driving systems, as they provide essential environmental information for tasks such as trajectory forecasting [1,2,3], path planning [4,5,6], and other downstream applications [7,8]. Traditionally, the creation of these maps has depended on the manual annotation of LiDAR point clouds, a process that is labor-intensive, time-consuming, and costly. Moreover, manual annotation procedures are not readily updated to reflect dynamic changes in road conditions. In response, recent studies [9,10,11,12,13] have explored leveraging onboard sensor data to generate HD maps for the surrounding environment in real time.

Several works [9,11,12,14,15] have formulated real-time HD map generation as a semantic segmentation problem, aiming to learn and produce pixel-level rasterized maps from a bird’s-eye view (BEV) perspective. However, rasterized maps inherently contain redundant pixel-level semantic information and are not directly compatible with downstream tasks. Consequently, additional post-processing is required to vectorize these maps, a procedure that not only increases computational overhead but also may introduces cumulative errors. To overcome these limitations, VectorMapNet [16] represents each map element as a sequence of points within a two-stage framework that progressively refines the predictions from coarse to fine levels. Subsequently, MapTR [13] employs DETR to directly regress the point coordinates of map elements before grouping the predicted points to form complete road element instances. MapTRv2 [17] further improves model accuracy and training efficiency by refining the attention mechanism and incorporating auxiliary supervision.

Despite these promising advancements, the current map vectorization frameworks still exhibit several limitations. Representing map elements as point sets leads to varying numbers of points depending on their complexity, complicating model training. To address this variability, some studies [13,17] employ equidistant sampling on raw map data, converting each element into a fixed number of points regardless of shape complexity. However, as demonstrated in Figure 1b, this approach results in a loss of precision, particularly in regions characterized by right angles and curves, and introduces inherent errors that are not adequately captured by evaluation metrics, potentially impacting downstream tasks.

To overcome these issues and achieve a more precise map representation, the conventional equidistant sampling method has been refined by introducing a novel keypoint fine-tuning strategy. This approach selectively samples points to preserve the original shape of map elements as accurately as possible, thereby minimizing distortion. However, the resulting point sets no longer exhibit equidistant spacing, which introduces new challenges for network training. In response, KPMapNet is proposed as a novel framework designed to accurately model map elements through a redesigned loss function and an integrated prediction architecture that incorporates map masks. Specifically, the training ground truth is partitioned based on positional features into key points which delineate the shape and collinear points that supplement the representation. Based on this partitioning, the conventional point-to-point matching loss is reformulated into a point-to-line hybrid matching loss. Furthermore, a BEV mask fusion module is designed to enhance the localization and representational capabilities of BEV features by integrating the learned BEV mask into the feature map. Additionally, a hybrid query module is introduced within the decoder to improve query initialization and facilitate more effective detection.

Extensive experiments conducted on the NuScenes [18] and Argoverse2 [19] datasets demonstrate that KPMapNet achieves a state-of-the-art performance in terms of accuracy. Moreover, the visualization results indicate that KPMapNet yields more accurate shape recognition for map elements. The main contributions of this work can be summarized as follows:

• A keypoint fine-tuning strategy is proposed that optimizes the preservation of original ground truth shape features, establishing a novel method for ground truth representation in this field.
• A novel framework, KPMapNet, is introduced that integrates map mask feature fusion with an innovative point-to-line hybrid matching loss, thereby enabling the precise modeling of map elements.
• KPMapNet achieves 75.1 mAP on the NuScenes dataset and 74.2 mAP on the Argoverse2 dataset. Moreover, the visualization results demonstrate the framework’s improved predictive performance for instances exhibiting complex shapes.

2. Related Works

2.1. Online HD Map Construction

Traditional offline HD map generation typically relies on manual or semi-automated annotation [9,20]. With the emergence of methods that transition from picture view (PV) to BEV representations [21,22], recent studies [1,23,24,25] have investigated the feasibility of generating HD maps online directly using perspective view sensors. For example, HDMapNet [12] aims to directly generate vectorized map elements rather than rasterized maps, while VectorMapNet [16] employs a two-stage framework with an auto-regressive decoder to iteratively predict vector vertices. Furthermore, MapTR [13] and its enhanced version, MapTRv2 [17], introduce novel representations for map elements, enabling the simultaneous regression of both category and positional information. The proposed approach further extends these methods by incorporating generated map mask features into the model to better capture the specific vector shapes of map elements. In addition, KPMapNet introduces supplementary instance queries with shared decoder parameters during training to enhance overall model performance.

2.2. Map Instance Modeling

HD maps consist of diverse instances with varying geometric characteristics, such as lane dividers, pedestrian crossings, and road boundaries. In certain datasets [18], these map elements are typically stored and represented as polylines. Unlike the fixed-format bounding boxes used in detection tasks, the number of points in a polyline instance can vary, which poses significant challenges for network training. The conventional approaches [13,17,26] typically sample a fixed number of equidistant points from the ground truth, a strategy that may lead to a loss of shape information. In contrast, BeMapNet [24] represents ground truth using Bézier curves, while PivotNet [27] dynamically models and matches instances by employing pivotal points on a per-instance basis. In this paper, KpMapNet proposes a keypoint fine-tuning strategy that replaces equidistant sampling with a more precise representation that closely aligns with the original instances, thereby preserving detailed map features while maintaining model efficiency.

2.3. Map Mask for Segmentations

Image masks are widely employed in segmentation tasks to enhance the quality of both instance-level and semantic features. Previous studies [28] have examined the interaction between learned mask features and instance activation maps to effectively capture object-specific characteristics. Other works [29,30] have integrated mask features into transformer-based architectures, leveraging attention mechanisms for efficient feature extraction. In the context of HD map detection, KepMapNet utilizes high-quality map mask features to emphasize and enhance regions with detailed map annotations, thereby improving the model’s ability to capture fine-grained information.

3. The Proposed Method

3.1. Model Pipeline

Figure 2 illustrates the overall architecture of KPMapNet. Building upon MapTRv2 [17], KPMapNet first extracts features from the input of surrounding PV images and projects them into the BEV space. To enhance the representational capacity of the BEV features, a BEV mask fusion module is introduced, which reintegrates the generated rasterized map into the BEV features. These enriched BEV features are subsequently fed into a carefully designed hybrid attention decoder that generates the corresponding map instances. Moreover, this method has been redesigned to optimize the processing of the ground truth data to obtain more accurate training targets; the loss function has also been redefined accordingly.

The remainder of this section is organized as follows: Section 3.2 details the proposed keypoint fine-tuning sampling method, Section 3.3 describes the architecture of KPMapNet, and Section 3.4 introduces the overall training loss function.

3.2. Ground Truth Representation

Each map instance is defined by its class label $[eqn]$ and an ordered sequence of points $[eqn]$ that describe its shape, where N specifies the number of points representing each instance. The connectivity between the points is implicitly encoded in their ordering. As illustrated in Figure 1, the number of points N in the original representation varies with the complexity of the map element, complicating the training process. To standardize this representation, equidistant sampling is applied, resulting in a fixed number of points with equal spacing. While this approach simplifies the learning task, it can introduce geometric errors.

To mitigate this issue, this research decomposes the original point set $[eqn]$ into the following two subsets: a set of key points $[eqn]$ , which capture the essential shape features, and a set of redundant collinear points $[eqn]$ , which serve as fillers. The keypoint set $[eqn]$ is an ordered series of points that delineate the overall shape, typically marking changes in direction, whereas $[eqn]$ does not contribute additional shape information.

Let $[eqn]$ denote the point set obtained via equidistant sampling. Then, adjust $[eqn]$ using $[eqn]$ to preserve the original shape. The keypoint fine-tuning procedure is as follows: First, simplify $[eqn]$ using methods such as Douglas–Peucker [31] or Visvalingam–Whyatt [32] to derive $[eqn]$ . Then, apply equidistant sampling to $[eqn]$ to obtain $[eqn]$ . A custom matching algorithm is designed to align these two sets and obtain a fine-tuned configuration.

Consider matching the keypoint sequence $[eqn]$ with the equidistant sequence $[eqn]$ , where K is the number of key points and M is a predefined fixed number. While M is fixed, K varies with the map element’s shape complexity. Let $[eqn]$ denote the set of all possible combinations given the ordering constraints; there are $[eqn]$ possible combinations. For the i-th key point $[eqn]$ , let $[eqn]$ be its corresponding point in $[eqn]$ . For a given matching combination $[eqn]$ , the matching cost is defined as follows:

$$[eqn]$$

where $[eqn]$ denotes $[eqn]$ distance. The optimal matching $[eqn]$ is then obtained by minimizing the matching cost, as in the following equation:

$$[eqn]$$

Due to the ordered nature of the point sets, a fixed correspondence is assumed, i.e., $[eqn]$ and $[eqn]$ . This matching problem is addressed using dynamic programming. The algorithm uses an array $[eqn]$ to store the matching costs; let $[eqn]$ denote the minimum matching cost for the first i key points and the first j equidistant points. The state transition can be expressed as follows:

$$[eqn]$$

The initial condition is defined as follows:

$$[eqn]$$

Finally, replacing $[eqn]$ with the optimal combination obtained $[eqn]$ yields the fine-tuned keypoint set $[eqn]$ , which maintains a consistent number of points while preserving a more accurate representation of the original shapes.

3.3. Architecture Detail

BEV Feature Extractor. Given the input surround images $[eqn]$ , KpMapNet employs the conventional CNN backbones [33,34,35], along with an FPN [36], to extract the multi-view PV features $[eqn]$ . These PV features are then integrated into a unified BEV representation using various strategies such as LSS [23,25,37,38,39], GKT [40], CVT [11] and Deformable Attention [13,41,42]. Following the settings in MapTRv2 [17], KpMapNet adopts LSS [38] as the default transformation method to leverage depth information during supervised learning. The resulting BEV feature is denoted as $[eqn]$ , where $[eqn]$ represents the spatial dimensions and C denotes the feature depth.

BEV Mask Fusion. Before passing the BEV features to the decoder, KpMapNet introduces an optimization module that refines the BEV features at a fine-grained, point-level resolution by leveraging a rasterized BEV feature mask. As shown in Figure 2, a convolutional operation is applied to $[eqn]$ to generate the BEV mask $[eqn]$ . During training, $[eqn]$ is supervised using the ground truth mask $[eqn]$ , with the loss function defined as the cross-entropy loss, as in the following equation:

$$[eqn]$$

Subsequently, a CNN $[eqn]$ upsamples $[eqn]$ to a channel dimensionality of 32. This upsampled mask is concatenated with the BEV feature $[eqn]$ and a two-dimensional normalized positional encoding $[eqn]$ that encodes spatial location. Finally, these features are fused via a convolutional operation, as follows:

$$[eqn]$$

The fused feature $[eqn]$ emphasizes the salient positional and semantic information, thereby enhancing the model’s ability to accurately predict detailed map shapes by effectively distinguishing instance features from background noise.

Hybrid Attention Decoder. For each map instance, the transformer decoder employs a set of queries to predict both the category and the geometric details through regression. As depicted in Figure 2, KpMapNet adopts a hybrid approach that combines instance queries $[eqn]$ with point queries $[eqn]$ , while leveraging BEV features to refine the instance queries for improved initialization. The instance queries $[eqn]$ , with D denoting the number of instance queries, are dynamically updated alongside their random initialization to incorporate instance-specific features. Specifically, a convolutional layer followed by a sigmoid activation predicts an instance mask, which is then fused with the randomly initialized instance queries, as expressed in the following equation:

$$[eqn]$$

The point queries $[eqn]$ are obtained via random initialization. These instance and point queries are then combined to form the hybrid queries $[eqn]$ . Finally, $[eqn]$ and $[eqn]$ are fed into an L-layer decoder to generate the predicted category $[eqn]$ and the point set $[eqn]$ . The iterative residual design of the decoder further enhances the network’s capacity to learn robust regression features, thereby improving both representation and prediction accuracy.

3.4. Training Loss

Point–line Matching Loss. The fine-tuned point set $[eqn]$ obtained via the keypoint fine-tuning strategy does not enforce equal spacing between points, which poses new challenges for training. To accommodate the characteristics of this new ground truth, the original point-to-point matching loss is modified. Specifically $[eqn]$ can be decomposed into key points $[eqn]$ and collinear filler points $[eqn]$ . For points corresponding to key features, the L1 distance is computed directly. For collinear points, the L1 distance is calculated relative to the perpendicular projection onto the predicted line, as illustrated in Figure 3. The final point-to-line matching loss is defined as follows:

$$[eqn]$$

where $[eqn]$ denotes the straight line on which the collinear points reside.

Overall Loss. Following the approach in MapTRv2 [17], KpMapNet also incorporates classification loss $[eqn]$ and an edge direction loss $[eqn]$ . The total loss function is defined as follows:

$$[eqn]$$

where $[eqn]$ , $[eqn]$ , $[eqn]$ and $[eqn]$ are the corresponding loss weights.

4. Experiments

4.1. Datasets

KpMapNet evaluates model performance through comprehensive experiments on two widely recognized datasets in autonomous driving research, namely NuScenes [18] and Argoverse2 [19]. The NuScenes dataset comprises 1000 driving scenarios collected in Singapore and Boston, USA, under diverse weather conditions and times of day. Out of these, 750 scenes are allocated for training, 100 for validation, and 150 for testing. Each scenario spans approximately 20 s and provides 40 keyframes sampled at 2 Hz, including 360-degree RGB images from six cameras, LiDAR point clouds, and precisely annotated 2D map instances. Argoverse2 consists of 1000 scenarios with 3D map annotations collected from six U.S. cities, featuring data from LiDAR, stereo cameras, and ring cameras. In line with the previous studies [16,17,26], our experiments focus on the following three static categories of map elements: pedestrian crossing (ped.), lane divider (div.), and road boundary (bou.).

4.2. Evaluation Metrics

To ensure fair comparisons, the quality of map instance predictions is evaluated using average precision (AP) based on the average Chamfer distance. The Chamfer distance quantifies the alignment between predictions and ground truth by computing the mean Euclidean distance between corresponding points. The method first samples points from the predicted instances and computes the forward Chamfer distance as the mean of the shortest distances from predictions to the ground truth, then computes the reverse distance in the same manner for the ground truth. The average Chamfer distance is obtained by averaging these two measures. Predictions with distances below a specified threshold are considered true positives (TPs). Following the previous studies [17,27], this research employs two threshold sets, $[eqn]$ m and $[eqn]$ m, and reports the mAP as the average precision across these thresholds. Consistent with MapTRv2 [17], 100 points are sampled for Chamfer distance calculations.

4.3. Implementation Details

Architecture. KPMapNet employs ResNet50 [33], EfficientNet-B0 [34], and VoVNetV2-99 [35] as its backbone networks. For the NuScenes dataset, the original image resolution is $[eqn]$ . These images are resized by a factor of 0.5 and padded to $[eqn]$ . In the Argoverse2 dataset, the seven camera images have differing resolutions (with the front view at $[eqn]$ and the others at $[eqn]$ ). The front view images are cropped to $[eqn]$ and padded to match the other views ( $[eqn]$ ), and then all seven images are resized to $[eqn]$ . The preprocessed images are subsequently passed through the backbone and FPN for feature extraction, followed by an encoder to generate BEV features. In the experiments, the BEV perception range is set to 30 m both ahead and behind the vehicle, as well as 15 m to both the left and right. With a feature resolution of $[eqn]$ m, this results in a grid of $[eqn]$ BEV queries, each with a feature dimension of 256, processed by six decoder layers. The default number of instance queries is 50, and the number of point queries is 20. All settings are adopted from the previous studies [13,17,27] to ensure a fair comparison.

Training and Inference. KpMapNet is trained on a single NVIDIA A100 Tensor Core GPU (Santa Clara, CA, USA) with a batch size of 16, utilizing the AdamW [43] optimizer with a weight decay of 0.01. Training is conducted for 24 epochs by default, starting with an initial learning rate of $[eqn]$ and applying a decay factor of 0.1 for the backbone. The learning rate follows a cosine annealing schedule with a linear warm-up phase. For the point-line matching loss, the loss weight is set to 6, while all other hyperparameters adhere to the configurations used in MapTRv2 [17]. Inference is executed on a single NVIDIA GeForce RTX 3090 GPU, with the inference time duly recorded.

4.4. Main Results

Results on nuScenes. KPMapNet is trained using various epoch schedules and backbone networks on the nuScenes dataset. Table 1 presents a performance comparison of KPMapNet, using only RGB image inputs, alongside the previous methods. KPMapNet achieves better performance (75.1, 53.2 mAP) under both threshold configurations. The reproducibility metrics for the comparison methods [12,13,16,17,27,44] are obtained by executing publicly available source code and model checkpoints. The results demonstrate that KPMapNet outperforms the previous approaches, yielding significant improvements in complex features, particularly for ped. and bou. Under the easy threshold $[eqn]$ m, KpMapNet-ResNet50 shows improvements of 3.9 mAP for ped. and 3.7 mAP for bou., compared to the previous SOTA MapTRv2. Additionally, KPMapNet outperforms PivotNet by 3.6 mAP for ped. and 3.7 mAP for bou. KpMapNet based on VoVNetV2-99 achieves 75.1 mAP—1.7 mAP higher than MapTRv2. Moreover, the visualizations in Figure 4 illustrate the improved shape representation achieved by our model.

Results on Argoverse2. As shown in Table 2, on the Argoverse2 dataset, KPMapNet consistently exceeds teh previous SOTA methods under both evaluation settings. Our method achieves competitive performance, with KpMapNet outperforming MapTRv2 by reaching 74.2 mAP, thereby further validating the effectiveness of our proposed approach.

4.5. Ablation Study

This section presents the ablation experiments used to evaluate the contributions of the proposed modules and design choices. Unless otherwise specified, the experiments were conducted using ResNet50 [33] as the backbone, with NuScenes camera images as input, and training was performed for 24 epochs under the easy threshold configuration.

Ablation on different modules. The ablation results in Table 3 validate the contribution of each design component to overall performance. Initially, the configurations of MapTRv2 were adopted as baselines for comparison. Although the introduction of keypoint fine-tuning initially increased the network’s learning complexity and caused a slight performance decline, the subsequent integration of additional components resolved this issue and enhanced overall performance. Specifically, the new ground truth representation significantly improved the network’s precision in detecting complex vectors. The point-line matching loss accelerated convergence while maintaining performance comparable to the baseline. Moreover, incorporating mask fusion and hybrid attention resulted in performance gains of 2.9 mAP and 1.5 mAP, respectively. Together, these components achieved a cumulative improvement of 3.8 mAP, which was the highest observed enhancement.

Point–line loss weight. This research explores various weight configurations for the point-line matching loss to determine the optimal setting. As shown in Table 4, performance was relatively insensitive to specific weight values; however, excessively high or low weights adversely affected network convergence, leading to performance degradation. Based on these observations, a weight of 6 was identified as the optimal choice.

Auxiliary mask loss. Table 5 demonstrates the effectiveness of BEV mask supervision. By enforcing constraints on BEV features to ensure they capture meaningful spatial information, the auxiliary mask loss $[eqn]$ yielded a 1.7 mAP improvement, thereby enhancing overall performance.

Discussion of different thresholds. Smaller threshold settings impose stricter performance requirements on the model. Given that HD maps for autonomous driving require error control at the centimeter level, performance improvements under these stricter thresholds are particularly valuable for practical applications. As shown in Table 6, this model consistently achieves superior results compared to the previous methods across all thresholds. The improvements are especially pronounced at the $[eqn]$ m and $[eqn]$ m thresholds, underscoring the model’s effectiveness under more rigorous evaluation criteria.

5. Conclusions

This paper addresses the challenges of accurately processing vector representations for map elements by introducing a novel keypoint fine-tuning strategy and an online HD vectorized map detection framework, KPMapNet. Our approach offers a unified method for refining map elements, effectively preserving intricate shape details that are typically compromised during traditional ground truth processing. By integrating innovative components—including a BEV mask fusion module, a hybrid attention mechanism, and a point-line matching loss function—KPMapNet mitigates the loss of fine structural information and enhances the precision of HD map construction. Experimental results on public datasets demonstrate that KPMapNet achieves state-of-the-art performance, establishing a new benchmark in the domain. Notably, our framework significantly improves the detection accuracy of complex map elements, such as pedestrian crossings and road boundaries, underscoring its potential for practical autonomous driving applications.

Future research will explore the integration of multimodal sensor data and temporal information to construct more holistic and robust representations of the driving environment. These advancements are expected to further enhance the performance of vectorized map detection systems and contribute to the development of more reliable autonomous driving solutions.