Streamlining the Development of Active Learning Methods in Real-World Object Detection

TL;DR

This paper introduces object-based set similarity (OSS), a novel metric that predicts active learning method effectiveness and selects robust validation sets in real-world object detection, reducing computational costs and improving evaluation reliability.

Contribution

The paper presents OSS, a detector-agnostic similarity metric that eliminates ineffective AL methods before training and enhances validation set selection for more reliable evaluation.

Findings

OSS accurately predicts AL method effectiveness without detector training

OSS enables selection of representative validation sets for robust evaluation

Validated on three autonomous driving datasets with multiple detectors

Abstract

Active learning (AL) for real-world object detection faces computational and reliability challenges that limit practical deployment. Developing new AL methods requires training multiple detectors across iterations to compare against existing approaches. This creates high costs for autonomous driving datasets where the training of one detector requires up to 282 GPU hours. Additionally, AL method rankings vary substantially across validation sets, compromising reliability in safety-critical transportation systems. We introduce object-based set similarity ($\mathrm{OSS}$), a metric that addresses these challenges. $\mathrm{OSS}$ (1) quantifies AL method effectiveness without requiring detector training by measuring similarity between training sets and target domains using object-level features. This enables the elimination of ineffective AL methods before training. Furthermore, $\mathrm{OSS}$ (2) enables the selection of representative validation sets for robust evaluation. We validate our similarity-based approach on three autonomous driving datasets (KITTI, BDD100K, CODA) using uncertainty-based AL methods as a case study with two detector architectures (EfficientDet, YOLOv3). This work is the first to unify AL training and evaluation strategies in object detection based on object similarity. $\mathrm{OSS}$ is detector-agnostic, requires only labeled object crops, and integrates with existing AL pipelines. This provides a practical framework for deploying AL in real-world applications where computational efficiency and evaluation reliability are critical. Code is available at https://mos-ks.github.io/publications/.

Figures

Tables

Table 1. TABLE I: KITTI (top) and BDD (bottom): Correlation coefficient r r between OSS \mathrm{OSS} and the AP for 10 AL methods at the first AL iteration (10%), including the corresponding class proportion (%).

	Car	Van	Truck	Pedest- rian	Person Sitting	Cyclist	Tram
%	62,4	7.7	3.2	17.2	1.5	5.3	2.8
$r\uparrow$	${0.90}^{\ast \ast \ast }$	${0.70}^{\ast }$	${0.87}^{\ast \ast \ast }$	${0.91}^{\ast \ast \ast }$	${0.89}^{\ast \ast \ast }$	${0.70}^{\ast }$	${0.96}^{\ast \ast \ast }$
	Car	Rider	Truck	Pedest- rian	Motor- cycle	Bicycle	Bus	Traffic Light	Traffic Sign
%	51.3	0.6	2.4	9.0	0.4	0.9	1.1	15.6	18.7
$r\uparrow$	${0.87}^{\ast \ast \ast }$	${0.97}^{\ast \ast \ast }$	${0.88}^{\ast \ast \ast }$	${0.95}^{\ast \ast \ast }$	${0.97}^{\ast \ast \ast }$	${0.97}^{\ast \ast \ast }$	${0.85}^{\ast \ast }$	0.47	0.10

Table 2. TABLE II: KITTI (top) and BDD (bottom): Training time in GPU hours for the warm-up step ( i = 0 i=0 ) and each of the six AL iterations ( i ∈ [ 1 , 6 ] i\in[1,6] ), including savings if stopping is activated at early iterations based on mAP post-training ( S mAP S_{\text{mAP}} ) and OSS \mathrm{OSS} pre-training ( S OSS S_{\mathrm{OSS}} ).

	$i=0$	$i=1$	$i=2$	$i=3$	$i=4$	$i=5$	$i=6$
Training	03	04	04	05	06	09	23
$S_{\text{mAP}}$		47	43	38	32	23
$S_{\mathrm{OSS}}$		51	47	43	38	32
Training	25	29	44	54	164	233	282
$S_{\text{mAP}}$		777	733	679	515	282
$S_{\mathrm{OSS}}$		806	777	733	679	515

Table 3. TABLE III: Impact of removing ( − - ) each term in Eq.   1 and using class count as is on the correlation between OSS \mathrm{OSS} and mAP via r r .

	All	$r_j^{\left(P_m\right)}$	$n_j^{\left(P_m\right)}$	JSD	$T_1$
	$✓$	$-$	$✓$	$-$	$-$
$r_{\mathrm{BDD}}\left({𝒳}_l^1\parallel {𝒳}_{\text{alt}}\right)$	${\text{0.81}}^{\ast \ast \ast }$	$0.45$	$0.61$	${0.80}^{\ast \ast \ast }$	${0.78}^{\ast \ast }$
$r_{\mathrm{BDD}}\left({𝒳}_l^3\parallel {𝒳}_{\text{alt}}\right)$	${\text{0.89}}^{\ast \ast \ast }$	$0.59$	${0.70}^{\ast }$	${0.82}^{\ast \ast \ast }$	${\text{0.89}}^{\ast \ast \ast }$
$r_{\mathrm{KITTI}}\left({𝒳}_l^1\parallel {𝒳}_{\text{val}}\right)$	${\text{0.74}}^{\ast \ast }$	$-0.21$	$-0.59$	${0.63}^{\ast }$	${0.69}^{\ast }$
$r_{\mathrm{KITTI}}\left({𝒳}_l^3\parallel {𝒳}_{\text{val}}\right)$	${\text{0.83}}^{\ast \ast \ast }$	$0.35$	$-0.35$	${0.65}^{\ast }$	${0.76}^{\ast \ast }$

Table 4. TABLE IV: Impact of features in Eq.   1 on r KITTI ​ ( 𝒳 l 1 ∥ 𝒳 v ​ a ​ l ) r_{\mathrm{KITTI}}(\mathcal{X}_{l}^{1}\|\mathcal{X}_{val}) via removing ( − - ), adding ( + + ) more features to the JSD, and using model-based features with CS.

	JSD	AR	CH	DCT	HUE	SAT	VAL	CS
	$✓$	$-$	$-$	$-$	+	+	+	$✓$
$r$	${\text{0.74}}^{\ast \ast }$	${0.68}^{\ast }$	${0.72}^{\ast }$	${0.66}^{\ast }$	${\text{0.74}}^{\ast \ast }$	${0.71}^{\ast }$	${0.69}^{\ast }$	$-0.06$

Table 5. TABLE V: Impact of balancing terms in Eq.   1 on evaluation reliability, measured by τ \tau post-subsampling.

	All	${\beta }^{\left(P_m\right)}$	$w_j$	$C_w$
	$✓$	$-$	$-$	0.7	1.0	1.4
${\tau }_{\mathrm{BDD}}({𝒳}_{\mathrm{val},\mathrm{subset}},{𝒳}_{\mathrm{alt}})$	0.80	$-0.60$	$-0.60$	0.80	0.80	$-0.66$
${\tau }_{\mathrm{KITTI}}({𝒳}_{\mathrm{val},\mathrm{subset}},{𝒳}_{\mathrm{alt}})$	0.60	$-0.20$	-	$-0.22$	0.60	0.60

Equations

$$\begin{gathered} \mathrm{OSS}(P_m\Vert Q)= \\ {\left(\frac{1}{c}\sum _{j=1}^c{w}_j\cdot \left(\text{JSD}(M_{{\hat{P}}_m}\Vert M_{\hat{Q}})+r_j^{(P_m)}\right)\right)}^{-1}. \end{gathered}$$

$$r_j^{(P_m)}$$

$${\beta }^{(P_m)}$$

$$w_j=\{\begin{array}{ll}1& \text{if }{\text{CV}}_{\mathbf{w}}\le C_w\\ \frac{1}{k}\sum _{m=1}^k\frac{n_j^{(P_m)}}{n_{\text{det}}^{(P_m)}}& \text{otherwise.}\end{array}$$

Full text

I Introduction

The effective deployment of object detectors in real-world applications relies on their ability to generalize across diverse and unfamiliar situations. While more data enhances generalizability, obtaining large, labeled, and diverse datasets is challenging due to redundancy, noise, and costly manual labeling. Active learning (AL) [1, 2] aims to address this challenge by identifying samples that maximize learning while minimizing labeling effort. A typical pool-based AL pipeline consists of training a model on an initial labeled subset of the pool, predicting on the remaining unlabeled data, selecting the most informative samples for labeling, and retraining with the expanded labeled subset. Since no universal definition of informativeness exists, each AL method quantifies it differently. AL methods are evaluated based on detector performance trained on the selected samples. Therefore, informativeness in AL refers to the contribution of a sample to improving model performance when added to the training set.

One common approach to quantifying informativeness is via uncertainty, which reflects the confidence of the model in its predictions [3, 4, 5, 6]. In this work, we utilize widely used uncertainty-based AL methods as a case study for our $\mathrm{OSS}$ metric. Multiple works hypothesize the potential of separating between epistemic (model), aleatoric (data), localization, and classification uncertainties to increase the informativeness of a training set selected based on uncertainty [7, 8, 9, 10, 11, 12, 13]. This is hereafter referred to as the separation hypothesis. However, uncertainty correlates with informativeness only if accurately estimated, highlighting the importance of its calibration [14, 15, 16]. Furthermore, real-world datasets naturally exhibit class imbalance. Relying only on uncertainty for selection could amplify the imbalance and harm performance by prioritizing problematic samples regardless of their class. This underscores the significance of class balancing [17, 18, 19]. We therefore investigate the impact of uncertainty separation hypothesis, calibration, and class balancing on informativeness and their implications for real-world AL deployment.

Such an investigation comparing $m$ methods across $n$ iterations with $k$ training seeds would result in $k\times m\times n$ training runs. Training object detectors on real-world datasets usually requires many days on high-end GPUs, making AL method development for real-world applications computationally expensive. $\mathrm{OSS}$ quantifies the informativeness of a training set based on the data only. As a result, $\mathrm{OSS}$ can rank AL methods before any training occurs, enabling practitioners to eliminate ineffective AL methods early and focus computational resources on the most promising approaches.

Furthermore, uncertainty-based AL methods for object detection typically rely on two well-curated datasets [20]: PASCAL VOC [21] and MS COCO [22], with the evaluation conducted separately on each. However, when training extends beyond these benchmarks, inconsistencies emerge due to variations in training and evaluation strategies across previous works [23, 20, 24]. This leads to poor generalization to the more diverse and challenging real-world. Therefore, it is crucial to ensure the robustness of AL evaluation results [20, 18, 24, 23].

Autonomous driving datasets exhibit significant domain variations due to geographic differences, weather conditions, traffic patterns, and infrastructure variations. An AL method that performs well on one validation subset may perform poorly on another, leading to unreliable deployment decisions. We visualize such inconsistencies in AL evaluation and motivate our approach in Fig. 1. The evaluation results of five AL methods on a validation set $\mathcal{X}_{\text{val}}$ fail to generalize to its subsets or an alternative evaluation set $\mathcal{X}_{\text{alt}}$ with domain shift, as reflected in the ranking variations. For instance, if a dataset mostly contains daytime images with few nighttime samples, an unstratified validation set may contain only nighttime images. Even if an AL method yields a strong daytime detector, its validation performance will appear poor due to domain shift. We mitigate this by identifying a validation subset with $\mathrm{OSS}$ that better aligns with the target domain, providing a more reliable indicator of real-world performance than the original validation set (see Fig. 1).

This paper introduces a novel similarity-based approach that addresses both the computational efficiency and evaluation reliability challenges in AL for real-world applications. We propose that the factors determining the informativeness of a training set also influence the representativeness of an evaluation set. We therefore propose unifying the AL training and evaluation strategies under object-based set similarity and introduce a novel metric to quantify it.

Our key insight is that both AL method training set informativeness and evaluation set representativeness can be predicted by measuring how similar the sets are to the target domain. For this, $\mathrm{OSS}$ analyzes the distribution of visual features (aspect ratios, color histograms, and texture patterns) across object crops and their class distributions. As a result, $\mathrm{OSS}$ is model- and framework-agnostic, relying only on images and labels.

In summary, our novel $\mathrm{OSS}$ streamlines the development of AL methods by avoiding the computational costs resulting from multiple training runs [26, 27, 24] while also improving the generalizability of the evaluation results, even to domain shift. $\mathrm{OSS}$ does not influence training sample selection. AL methods must still select samples based on their metric for informativeness, which motivates our case study using uncertainty-based AL methods on autonomous driving datasets. Therefore, our contributions cover the various stages of an AL pipeline, spanning from selection and training to evaluation and deployment:

• •

We establish that the informativeness of an AL training set correlates with its similarity to the data pool and the evaluation set, hence introducing an $\mathrm{OSS}$ metric that reliably predicts performance before training.

• •

We evaluate uncertainty-based AL methods as a case study, including the impact of various types of uncertainties, class balancing, and calibration w.r.t. similarity and informativeness on real-world datasets.

• •

We further utilize our $\mathrm{OSS}$ metric to improve the consistency of the AL evaluation results and their generalizability to the target domain despite domain shift.

II Related Work

AL aims to identify the most informative samples in the data pool, although recent works diverge on the definition of informativeness. Some assert that it translates to selecting samples with the highest predictive uncertainty under the assumption that samples that provide new information to the model will highlight its limitations [3, 4, 6, 20]. Others consider diversity since diverse samples generally contain a broader spectrum of information [16, 28]. Alternative definitions include difficulty via loss prediction on unlabeled data [29], localization disagreement between anchor boxes [3], and localization stability post-corruption [7].

Uncertainty vs. Informativeness: When assessing uncertainty-based AL, several works propose separating localization and classification uncertainties [6, 29, 7]. Hacohen et al. [16] observe that uncertain classification examples offer valuable insights, provided the uncertainty is accurately estimated, highlighting the importance of calibration [14, 15]. To address high uncertainty introduced by commonly challenging features such as crowded or nighttime scenes, Tang et al. [15] incorporate diversity through clustering based on feature maps. However, Kothawade et al. [18] argue that diversity and uncertainty may be ineffective at selecting rare classes. This is especially relevant in autonomous driving datasets with imbalanced classes such as cars vs. motorcycles. Class balancing can mitigate this through class-based weighting of the loss function [30, 31, 4], class-based weighting of the AL selection function with the number of instances and classes [17, 19], and considering the distance between class distributions in the selected and labeled sets [32]. Informativeness is therefore influenced by several factors, which we further investigate on real-world datasets.

Quantifying Informativeness: Broadening the scope, Yu et al. [32] argue that a valid metric for informativeness ensures that information decreases over AL iterations. Furthermore, the information in the selected set of each iteration should surpass that in the labeled set to enhance performance. However, they do not propose an approach to quantify the information content within a set. We address this by linking informativeness to similarity and introducing a novel $\mathrm{OSS}$ metric. One relevant work on similarity addresses the selection of instances specific to rare events [18]. Kothawade et al. [18] use submodular mutual information functions, which measure the amount of information in the data, to find images that are semantically similar to regions of interest in the query set. Yuan et al. [33] address distribution bias between labeled and unlabeled instances, thereby increasing their similarity, by minimizing prediction discrepancies and aligning features through an adversarial classifier setting. These approaches target sample informativeness and sample selection. In contrast, our approach measures set similarity post-selection based solely on images and corresponding labels. It therefore supports the development of AL selection methods and is efficient and well-suited for real-world object detection.

Evaluation Reliability: Previous works on AL have primarily focused on performance metrics rather than evaluation reliability. Mittal et al. [23] advocate for a more robust evaluation strategy in AL that encompasses a broader spectrum of datasets, ranging from low- to large-budget regimes. We reinforce and uphold this viewpoint by validating our findings on datasets differing in size by a factor of 10. Nevertheless, acquiring diverse datasets poses a challenge. Therefore, we utilize $\mathrm{OSS}$ to address inconsistent evaluation [20, 23] and enhance reliability without requiring additional data collection or labeling of more evaluation sets. $\mathrm{OSS}$ helps identify a subset of the evaluation set that is more representative of the target domain, which is crucial for the dynamic environment of real-world applications such as autonomous driving.

III Leveraging Similarity in Active Learning

This section establishes the theoretical foundation for our approach in three parts: we first formalize the AL setup (LABEL:{subsec:AL}), then demonstrate how similarity is linked to training set informativeness (Section III-B1) and evaluation set representativeness (Section III-B2) in AL, and finally introduce our $\mathrm{OSS}$ metric that quantifies these relationships (Section III-C).

III-A Active Learning Setup

Pool-based AL follows an iterative setting on a pre-defined labeling budget $b\in\mathbb{N}_{>0}$ with a number of iterations $n_{i}\in\mathbb{N}_{>0}$, $n_{i}\leq b$. It aims to find a subset of size $b$ that contains the most informative samples of a pool $\mathcal{D_{\text{pool}}}$. Given $\mathcal{D_{\text{pool}}}$ of size $n_{\text{pool}}\geq b$, we denote the labeled subset at iteration $i$ for $i=1,\ldots,n_{i}$, as $\mathcal{X}_{l}^{i}\subseteq\mathcal{D_{\text{pool}}}$ with the corresponding labels $\mathcal{Y}_{l}^{i}$ and the unlabeled subset at iteration $i$ as $\mathcal{X}_{u}^{i}\subseteq\mathcal{D_{\text{pool}}}$ such that $\mathcal{D_{\text{pool}}}=\mathcal{X}_{l}^{i}\bigcup\mathcal{X}_{u}^{i}$. Formally, each AL iteration $i$ consists of predicting on $\mathcal{X}_{u}^{i}$, selecting the $\lfloor\frac{b}{n_{i}}\rfloor\in\mathbb{N}_{>0}$ most informative samples for the model from $\mathcal{X}_{u}^{i}$, labeling them by an oracle, and adding them to $(\mathcal{X}_{l}^{i},\mathcal{Y}_{l}^{i})$, which results in $(\mathcal{X}_{l}^{i+1},\mathcal{Y}_{l}^{i+1})$ as the new training set at iteration $i+1$. The validation set denoted by $(\mathcal{X}_{\text{val}},\mathcal{Y}_{\text{val}})$ is selected and labeled from $\mathcal{D_{\text{pool}}}$ at the start and remains fixed.

III-B Similarity Link

We observe that the informativeness of an AL training set and the representativeness of an evaluation set are linked to similarity, as detailed in Section III-B. Therefore, we introduce a novel object-based set similarity ($\mathrm{OSS}$) metric in Section III-C.

III-B1 Linking Similarity to Informativeness

We propose that the objective in AL of identifying the most informative subset can be framed as finding the subset that maximizes similarity to $\mathcal{D_{\text{pool}}}$: $\max(\mathrm{OSS}(\mathcal{X}_{l}\|\mathcal{D_{\text{pool}}}))$. Object detectors are trained to localize and classify specific objects. We contend that similarity quantification should be confined to these objects, therefore requiring labeled data.

Since the majority of $\mathcal{D_{\text{pool}}}$ is unlabeled, $\mathrm{OSS}(\mathcal{X}_{l}\|\mathcal{D_{\text{pool}}})$ is approximated by $\mathrm{OSS}(\mathcal{X}_{l}\|\mathcal{X}_{\text{val}})$. The performance of an AL method is ultimately measured on $\mathcal{X}_{\text{val}}$, typically via the mean Average Precision (mAP) for object detection [22]. Therefore, the mAP at an iteration $i$ should positively and linearly correlate with $\mathrm{OSS}(\mathcal{X}_{l}^{i}\|\mathcal{X}_{\text{val}})$, emphasizing the need for an $\mathcal{X}_{\text{val}}$ that is representative of $\mathcal{D_{\text{pool}}}$.

III-B2 Linking Similarity to Representativeness

A well-defined similarity metric should not only predict training set informativeness before training, but also support the identification of an evaluation set representative of the target domain. For example, if the class “Pedestrian” is infrequent in $\mathcal{X}_{\text{val}}$ but is prevalent in $\mathcal{D}_{\text{pool}}$, or if its characteristics differ significantly between sets, the mAP-based ranking of AL methods on $\mathcal{X}_{\text{val}}$ may not reflect their general effectiveness.

Since $\mathrm{OSS}(\mathcal{X}_{l}^{i}\|\mathcal{X}_{\text{val}})$ correlates with performance, we assert that the similarity between $\mathcal{X}_{\text{val}}$ and an alternative evaluation set $\mathcal{X}_{\text{alt}}$ reflects the reliability of the performance-based ranking of AL methods on $\mathcal{X}_{\text{val}}$. Here, $\mathcal{X}_{\text{alt}}$ can be a subset of $\mathcal{X}_{\text{val}}$ itself, an $\mathcal{X}_{\text{val}}$ from another dataset, or a test set $\mathcal{X}_{\text{test}}$ depending on the AL setup. Thus, the representativeness of $\mathcal{X}_{\text{val}}$ is linked to its similarity to other evaluation sets, suggesting greater generalizability to $\mathcal{D}_{\text{pool}}$.

Evaluation Set Identification: We propose that a more reliable evaluation is achieved by identifying a subset $\mathcal{X}_{\text{val,subset}}\subseteq\mathcal{X}_{\text{val}}$ with $\mathrm{OSS}(\mathcal{X}_{\text{val}}\|\mathcal{X}_{\text{alt}})\leq\mathrm{OSS}(\mathcal{X}_{\text{val,subset}}\|\mathcal{X}_{\text{alt}})$ via subsampling before starting the AL iterations. Let $\mathcal{X}_{\text{val,subset}}^{1},\ldots,\mathcal{X}_{\text{val,subset}}^{z}$ be randomly drawn, equally-sized subsets of $\mathcal{X}_{\text{val}}$, with $z$ manually determined based on the size of $\mathcal{X}_{\text{val}}$. The subset with the highest similarity to $\mathcal{X}_{\text{alt}}$, given by $\mathrm{OSS}_{z}=\max_{l\in{1,\ldots,z}}\mathrm{OSS}(\mathcal{X}_{\text{val,subset}}^{l}\|\mathcal{X}_{\text{alt}})$, is selected as the new validation set. This approach requires no additional labeling since representative subsets can be identified from existing evaluation sets. The validation sets remain fixed across AL iterations, so their superscripts differ from those of the training sets.

III-C Object-based Set Similarity ($\mathrm{OSS}$) Metric

Having established the theoretical links between similarity, informativeness, and representativeness, we now introduce the $\mathrm{OSS}$ metric that translates these theoretical insights into a practical similarity measure.

III-C1 Design Principles

We introduce a novel similarity metric based on the two links in Section III-B. The process is visualized in Fig. 2.

• •

Object-Centric Analysis: Real-world datasets often contain redundant information within and across images. Detectors are typically evaluated using mAP, which averages detection performance per class. We therefore measure similarity based on the object crops per class $j=1,\ldots,c$, prioritizing relevant features while reducing computation time due to lower resolution. We calculate similarity between two sets of images $P$ (the comparison set) and $Q$ (the reference set) by (I) cropping the images within these sets. This enables object-centric analysis focused on detection-relevant content.

• •

Multi-Modal Feature Representation: Real-world objects exhibit diverse visual characteristics requiring combined spatial, frequency, and color analysis. For each class $j$, we (II) extract three complementary features from object crops:

• –

Aspect ratio (AR) captures object shape characteristics critical for real-world applications, e.g., vehicle orientations and pedestrian poses vary significantly.

• –

Mean of the 2D discrete cosine transform (DCT) [34] coefficients quantifies texture patterns that distinguish object classes.

• –

Mean value of the flattened 3D color histogram (CH) represents color distributions affected by lighting conditions, weather, and object materials.

In summary, objects of each class are encoded via complementary features covering color, texture, and shape.

• •

Class-Balanced Evaluation: Real-world data exhibits class imbalance requiring weighted similarity computation. We (III) calculate the class-weighted multivariate Jensen-Shannon divergence (JSD) [35] over the empirical distributions of the features and the class count ratio between the two sets to achieve class-balanced evaluation.

III-C2 Similarity Computation

The $\mathrm{OSS}$ metric combines distributional divergence, class count ratios, and class-based weighting.

Distribution Divergence: JSD requires equal sample sizes from $P$ and $Q$, which may not hold (e.g., different sizes of training and evaluation sets). We therefore use Gaussian Kernel Density Estimation (KDE) [36, 37] to model their probability density functions $\hat{f}_{P}(x)$ for $P$ and $\hat{f}_{Q}(x)$ for $Q$. We then draw equal-sized samples, forming $\hat{P}$ and $\hat{Q}$. Let $M_{\hat{P}}\in\mathbb{R}^{c\times 3}$ and $M_{\hat{Q}}\in\mathbb{R}^{c\times 3}$ denote the feature matrices calculated for $\hat{P}$ and $\hat{Q}$, respectively. Each row corresponds to an object class, and each column to one of the three features. The JSD is estimated directly on $M_{\hat{P}}$ and $M_{\hat{Q}}$ as outlined in Eq. 14 of [38], a practical non-parametric estimation between continuous distributions without assuming a restrictive parametric form.

$\mathrm{OSS}$** Formulation:** To compare multiple AL methods simultaneously, we calculate $\mathrm{OSS}(P_{m}\|Q)\in\mathbb{R}^{+}$ in Eq. 1 between multiple comparison sets $P_{m}$ and $Q$, $m=1,\ldots,k$, and $k\geq 1$ as:

[TABLE]

Class Count: The class count ratio $r_{j}^{(P_{m})}$ accounts for training set scalability, as more object samples typically yield a more representative set. It is defined as the smoothed ratio of the class counts $n_{j}^{(P_{m})}$ and $n_{j}^{(Q)}$ for a class $j$, that is,

[TABLE]

For smoothing, we use the first-degree Taylor polynomial $T_{1}(x)=\frac{1}{2}+\frac{1}{4}x$ as a linear approximation of the sigmoid function at $x=0$. This approximation retains sensitivity to ratio changes while ensuring numerical stability and preventing extreme values [39].

Scale Normalization: To address significant detection count variations among the $k$ comparison sets, $\beta^{(P_{m})}$ normalizes $n_{\text{det}}^{(P_{m})}$ by the 25th percentile ($Q_{1}$) of $n_{\text{det}}^{(P_{m})}$ across all $k$ sets, denoted $Q_{1_{m=1}^{k}}(n_{\text{det}}^{(P_{m})})$ in:

[TABLE]

Imbalance Compensation: Since class imbalance affects the similarity metric, we introduce a weighting factor to ensure each class contributes proportionally to its frequency in scenarios with substantial class imbalances. The class weight $w_{j}$ represents the average class proportion across the $k$ comparison sets:

[TABLE]

$\text{CV}_{\mathbf{w}}=\frac{\sigma_{\mathbf{w}}}{\mu_{\mathbf{w}}}$ is the coefficient of variation, with $\textbf{w}=(w_{1},w_{2},\ldots,w_{c})^{\top}$, the mean $\mu_{\mathbf{w}}$, and the standard deviation $\sigma_{\mathbf{w}}$, and the manually defined activation threshold $C_{w}$. In case of activation, classes with a small weight $w_{j}<Q_{1_{j=1}^{c}}(w_{j})$ are filtered out due to their negligible and unreliable contribution to the similarity metric.

IV Experiments

IV-A Implementation Details

IV-A1 Experimental Setup

We select EfficientDet-D0 [40, 41] pre-trained on COCO [22] as the main baseline detector and three autonomous driving datasets for our experiments. These are KITTI (7 classes, 20% random split for validation) [42] and BDD100K (10 classes, 12.5% official split) [43] for fine-tuning ($\mathcal{X}_{l}$ and $\mathcal{X}_{u}$) and evaluation ($\mathcal{X}_{\text{val}}$). The third dataset, CODA (4 classes in common with KITTI and 8 with BDD) [25], is our alternative evaluation set ($\mathcal{X}_{\text{alt}}$) with domain shift. Each AL iteration consists of a training from scratch for 200 epochs with 8 batches and an input resolution of 1024$\times$512 pixels. All other hyperparameters of EfficientDet maintain their default value [40]. We initiate all iterations on a random 5% of $\mathcal{D_{\text{pool}}}$ as a warm-up [44], corresponding to 299 images for KITTI (reasonable minimum covering all classes) and 3,492 images for BDD. Results represent four training iterations (two identical and two distinct warm-up seeds) as minimal variations are observed.

IV-A2 Uncertainty Estimation

For the case study, we estimate epistemic classification ($\sigma_{\text{ep,cls}}$) and localization ($\sigma_{\text{ep,loc}}$) uncertainties via 2D spatial Monte Carlo (MC) dropout [45] with a dropout rate of 0.05 selected based on best performance and 10 MC samples, cf. [46]. Higher dropout rates significantly reduce performance, e.g., increasing the rate to 0.1 reduces mAP by $3.49\pm 0.38$. Our dropout rate and number of samples were also selected based on the low expected calibration error (ECE) of $0.11\pm 0.02$ for $\sigma_{\text{ep,loc}}$. To estimate aleatoric uncertainty ($\sigma_{\text{al}}$), we use loss attenuation (LA) [47, 48] in the localization head [48]. We extract and apply softmax on the predicted classification logits to calculate entropy ($\sigma_{\text{ent}}$) as $\sigma_{\text{ent}}=-\sum_{j=1}^{c}p_{j}\log_{2}(p_{j})$ over the confidences $p_{j}$ of the $c$ classes. We use isotonic regression per-class to calibrate $\sigma_{\text{cls}}$ and per-class and per-coordinate for $\sigma_{\text{loc}}$ [48], and denote them with the subscript ${}_{\text{cal}}$. $\sigma_{\text{loc}}$ is also normalized by the width or height depending on whether it pertains to an $x$- or $y$-coordinate. The uncertainties per object are averaged as $\sigma_{\text{loc}}=\frac{1}{4}\sum_{i=1}^{4}\sigma_{\text{loc},i}$ and $\sigma_{\text{cls}}=\frac{1}{c}\sum_{i=1}^{c}(\sigma_{\text{cls},i})$.

IV-A3 AL methods

We consider as baselines entropy-based sampling [3], random sampling, and the widely adopted probabilistic approach [6]. The latter standardizes all uncertainties and selects the maximum value per image. We compare these baseline AL methods to methods that cover the intuitions described in Sections I and II. These include $\sigma_{\text{ep,cls}}$, $\sigma_{\text{ep,loc}}$, $\sigma_{\text{al}}$, $\sigma_{\text{ep}}=\sigma_{\text{ep,cls}}+\sigma_{\text{ep,loc}}$, $\sigma_{\text{ep}}-\sigma_{\text{al}}$, $\sigma_{\text{ent}}+\sigma_{\text{al}}$, and a calibrated version for each uncertainty $\sigma$. We apply min-max scaling to each $\sigma$ across all images so that all types of $\sigma$ $\in[0,1]$. We aggregate the uncertainties per image using maximum across all objects, as we find that it outperforms both the mean and sum. We then apply top-$k$ selection across all images to select the most informative images for the next AL iteration [20]. Additionally, we test the $n$-bins formulation with $n=5$ by dividing the score space into equal-sized bins and then querying from the top ($n$-1) bins (exploration) and the last bin (exploitation) [3]. We address class imbalance by weighting uncertainties (subscript ${}_{\text{cb}}$) according to the inverse frequency of each corresponding class in the predictions on $\mathcal{X}_{u}^{i}$, thereby prioritizing underrepresented classes [17].

IV-A4 Evaluation Metrics

• •

Correlation Analysis: Pearson correlation coefficient $r$ [49] and the p-value111Asterisks indicate the level of statistical significance: no asterisk $(\text{p}>0.05)$, ${}^{*}(\text{p}\leq 0.05)$, ${}^{**}(\text{p}\leq 0.01)$, ${}^{***}(\text{p}\leq 0.001)$. measure the linear relationship between $\mathrm{OSS}$ and mAP performance across AL methods.

• •

Ranking Consistency: Kendall’s tau ($\tau$) [50] measures the consistency of mAP-based AL method rankings across multiple evaluation sets.

The following subsections present our experimental results in four parts: we first validate the similarity-informativeness and similarity-representativeness relationships in AL, followed by a comprehensive case study using uncertainty-based AL methods, and finally provide detailed ablation studies of the $\mathrm{OSS}$ components and its detector agnosticism.

IV-B Linking Similarity to Informativeness

We first demonstrate that similarity is linked to informativeness (see Section III-B1). For 10 AL methods from Section IV-A at iterations $i=1,2,3$ in Fig. 3, a strong linear correlation exists between the mAP on $\mathcal{X}_{\mathrm{val}}$ and $\mathrm{OSS}(\mathcal{X}_{l}^{1-3}\|\mathcal{D_{\text{pool}}})$ and $\mathrm{OSS}(\mathcal{X}_{l}^{1-3}\|\mathcal{X}_{\mathrm{val}})$. This holds for both the randomly selected $\mathcal{X}_{\mathrm{val}}$ of KITTI and the curated one of BDD. The correlation is particularly strong for larger subsets (smallest deviation at $i=3$) but also exists despite a modest subset size of 598 images from KITTI (10%) at $i=1$ and low disparity in mAP (below 1%).

The same experiments with CODA ($\mathrm{OSS}(\mathcal{X}_{l}^{i}\|\mathcal{X}_{\mathrm{alt}})$) yield an average $r=0.85^{***}$††footnotemark: for the three iterations on BDD and a weaker $r=0.27$††footnotemark: on KITTI due to stronger class distribution shifts and fewer classes between KITTI and CODA. Additionally, CODA includes a larger proportion of BDD and contains night images, unlike KITTI [25].

Table I extends the analysis towards per-class AP, showing a strong linear correlation with the AP despite an even lower deviation in AP ($<0.2$%) and count (0.4% of $\mathcal{X}_{\mathrm{val}}$). However, correlation strength varies based on class distinctiveness, with less descriptive classes (e.g., “Traffic Sign”) exhibiting weaker correlation across comparison sets.

Computational Cost Analysis: In summary, $\mathrm{OSS}(\mathcal{X}_{l}^{i}\|\mathcal{X}_{\mathrm{val}})$ serves as an estimator of informativeness of an AL training set $\mathcal{X}_{l}^{i}$, since informativeness in AL refers to how much a set increases model performance (see Section I). $\mathrm{OSS}$ requires no additional labeling and is computationally efficient, with a processing time of 1 minute for $\mathrm{OSS}(\mathcal{X}_{l}^{1}\|\mathcal{X}_{\mathrm{val}})$ on KITTI and 12 minutes for 100 times more crops in BDD on an Intel Xeon Platinum 8168 CPU. In contrast, the training for a single AL method on BDD (six iterations) requires up to around 34 days on an NVIDIA Tesla V100 GPU (806 GPU hours), highlighting the time-saving advantage of our approach. $\mathrm{OSS}$ can be used to eliminate ineffective AL methods at each iteration before training, ensuring that only the most promising ones proceed to subsequent iterations. We demonstrate the cost benefits of the latter in our case study.

IV-C Linking Similarity to Representativeness

We further demonstrate that similarity positively correlates with representativeness (see Section III-B2). An evaluation set is considered representative if the performance-based ranking of AL methods on it generalizes across other evaluation sets and ultimately to $\mathcal{D_{\text{pool}}}$.

We first split the evaluation sets into 100 subsets with $1/10$ of the original size each. Next, we compute $\tau$ between the ranking of the five AL methods in Fig. 1 based on $\mathrm{OSS}$ and those based on the mAP. We select $\tau$ as we transition from measuring the linear correlation between the continuous values $\mathrm{OSS}$ and mAP to comparing the rankings of the AL methods. A $\tau$ value of 1.0 indicates complete agreement in ranks, while a $\tau$ value of -1.0 indicates complete disagreement. We analyze 100 subsets as well as the original $\mathcal{X}_{\mathrm{val}}$ and $\mathcal{X}_{\mathrm{alt}}$ (CODA).

Evaluation Reliability Analysis: First, $\mathrm{OSS}$ correlates with $\tau$ (Fig. 4 left), even despite the domain shift in $\mathcal{X}_{\mathrm{alt}}$ (right) and the sensitivity of $\tau$ to similar performance ($<0.2\%$ mAP difference). Second, the performance of the AL methods on one evaluation set is unreliable (strong fluctuation of $\tau$ in Fig. 4). Third, as depicted in Fig. 1, the majority rank on the 100 $\mathcal{X}_{\mathrm{val,subsets}}$ may not reflect the performance on $\mathcal{X}_{\mathrm{val}}$ and $\mathcal{X}_{\mathrm{alt}}$.

Subsampling Analysis: However, a subset may exist within $\mathcal{X}_{\mathrm{val}}$ where the ranking exactly matches that of $\mathcal{X}_{\mathrm{val}}$ and even $\mathcal{X}_{\mathrm{alt}}$ (visualized in Fig. 1 and Fig. 4). This representative subset is directly identified based on its similarity ($\mathrm{OSS}_{z}$) to the reference set $\mathcal{X}_{\mathrm{val}}$ or $\mathcal{X}_{\mathrm{alt}}$, hence reducing the computational expense of evaluating on multiple subsets for a robust evaluation. We observe that subsampling a subset with higher similarity across sets improves the consistency of evaluation results when sampling and labeling more data is costly. Our similarity-based approach increases $\tau_{\text{KITTI}}(\mathcal{X}_{\mathrm{val}},\mathcal{X}_{\mathrm{alt}})$ from -0.2 to 0.6 and $\tau_{\text{BDD}}(\mathcal{X}_{\mathrm{val}},\mathcal{X}_{\mathrm{alt}})$ from -0.6 to 1.0, without requiring any additional labeling or computationally expensive steps. Meanwhile, it only increases $\tau_{\text{KITTI}}$ from -0.2 to 0.0 when subsampling from CODA in contrast to $\tau_{\text{BDD}}$ from -0.6 to 0.6 due to the previously mentioned limited similarity between KITTI and CODA, showcasing the limitations of subsampling. Nevertheless, it proves crucial to match the evaluation set with the deployment domain for a consistent and reliable evaluation of AL methods. The variability in $\tau$ cautions against relying on rankings from a single evaluation set with limited $\mathcal{D_{\text{pool}}}$ coverage. A 1% difference in mAP, as seen in state-of-the-art comparisons, may not reliably reflect the efficacy of the proposed AL methods.

IV-D Case Study: Uncertainty-Based AL

We complete the analysis on AL by investigating the intuitions in Sections I and II and resulting AL methods (see Section IV-A) through a real-world case study. We observe in Fig. 5 that the 5% warm-up has a negligible effect, cf. [51], as AL methods converge despite $\pm 7\%$ mAP initial performances. $\sigma_{\text{ent}}$ consistently outperforms standard uncertainty-based methods across all iterations (up to +7% mAP over [6] and random), a conclusion supported by previous works [27, 52]. This holds regardless of whether $\sigma_{\text{ent}}$ stems from a model trained with MC+LA ($\sigma_{\text{ent}}$), LA only, or the baseline (B) without uncertainty estimation ($<0.5$% mAP deviation), avoiding the need for costly epistemic uncertainty estimation. This also indicates that the classification confidence of the detector can effectively approximate its epistemic uncertainty without localization information.

However, Fig. 5 (mid and right) shows that this may be caused by the quality of the uncertainty at $\leq 25\%$ data, as calibrating the uncertainty consistently improves its performance as a selection strategy. We find no advantage to the aggregation strategies [3] (Section IV-A) or separation hypothesis ($\sigma_{\text{ep}}-\sigma_{\text{al}}$, $\sigma_{\text{ent}}+\sigma_{\text{al}}$, and [6]). Since $\mathcal{X}_{\mathrm{val}}$ may contain samples with inherently high $\sigma_{al}$, removing them does not improve performance. This shows that AL for real-world applications requires domain-specific adaptation, as theoretically sound methods may not always translate and approaches that work on benchmarking datasets do not necessarily generalize. Fig. 5 (top) reveals that favoring images with high epistemic but low aleatoric uncertainty decreases similarity. Meanwhile, class balancing ($\sigma_{\text{ent,cb}}$) proves particularly impactful on both datasets (+1-1.5% mAP over $\sigma_{\text{ent}}$) with a stronger effect on BDD due to the stronger class imbalance (see class proportions in Table I).

We observe that $\sigma_{\text{ent,cal,cb}}$ performs best (+1.5-2% mAP over $\sigma_{\text{ent}}$). The latter can be retraced to both class balancing and calibration inherently increasing the informativeness of the selected samples, as they naturally align the selected set more closely with $\mathcal{X}_{\mathrm{val}}$. This is demonstrated by the higher $\mathrm{OSS}(\mathcal{X}_{l}^{2}\|\mathcal{X}_{\mathrm{val}})$ in the similarity plots of Fig. 5 (top), further proving that similarity is indeed linked to informativeness. Overall, the performance of AL at 45% approaches that of 100%, with $70.05\pm 0.23$ on KITTI and $23.67\pm 0.13$ on BDD. This highlights the potential of AL for real-world applications, especially with its strategies unified under our similarity-based approach.

Method Elimination Simulation: To demonstrate the practical benefits of $\mathrm{OSS}$, we simulate an $\mathrm{OSS}$-based elimination of AL methods. While Fig. 5 presents mAP results, obtaining these results requires training detectors first, then evaluating them. The challenge for AL method developers is waiting for training and evaluation to complete to determine if a method is effective, as this is a purely empirical process. Each training takes the times shown in Table II, plus an additional 20 minutes for each evaluation on BDD with a V100 GPU. We evaluate every 20 epochs during training to monitor convergence, adding around three GPU hours per training session.

Table II presents computational savings achievable through $\mathrm{OSS}$-based method elimination ($S_{\mathrm{OSS}}$) compared to mAP-based elimination ($S_{\text{mAP}}$). For KITTI, this approach saves up to $S_{\mathrm{OSS}}=204$ GPU hours per eliminated method (51 GPU hours per training with four training runs for statistical robustness across multiple seeds). For BDD, the savings increase to 3,224 GPU hours per eliminated method via $\mathrm{OSS}$. Assuming a price of 2.48 per GPU hour for a V100 [[53](https://arxiv.org/html/2508.19906v1#bib.bib53)], this reduces cost by up to 7995,52 per eliminated method via $\mathrm{OSS}$. Meanwhile, elimination based on mAP post-training would save up to 34 GPU hours less on KITTI (9 GPU hours $\times$ 4 training runs at $i=5$) and 932 hours on BDD (233 GPU hours $\times$ 4 training runs at $i=5$), which is up to 2311,36less savings per eliminated method. [Fig. 5](https://arxiv.org/html/2508.19906v1#S4.F5) shows that already at the first iteration, low-performing methods typically retain their rank even as iterations proceed on both datasets (e.g.,\sigma_{\text{ep}}$and$\sigma_{\text{ep}}-\sigma_{\text{al}}$can be safely eliminated at$i=1$).

IV-E Ablation Studies

IV-E1 $\mathrm{OSS}$ Component Analysis

Our first ablation study focuses on the impact of individual terms in Eq. 1, with the results summarized in Table III.

Main Components: Our analysis reveals that the correlation between $\mathrm{OSS}$ and mAP is significantly impacted by the class count term $r_{j}^{(P_{m})}$ in Eq. 1, as its removal ($-$) inversely affects the correlation. Moreover, utilizing only $n_{j}^{(P_{m})}$ highlights the importance of a suitable metric comparison (JSD). This is particularly evident in KITTI due to less significant count differences between the comparison sets. Removing smoothing ($T_{1}$) lowers correlation in three use-cases, confirming its usefulness. Overall, using all terms of Eq. 1 results in the strongest correlation between $\mathrm{OSS}$ and mAP.

Feature Selection: The second ablation study expands on the impact of different features on the correlation with performance. Table IV shows that removing any of the three features reduces correlation. Including other image-based features in the JSD such as hue, saturation (SAT), value (VAL), average and standard deviation of pixel intensity, average pairwise structural similarity index measure, and average pairwise Hamming distance of perceptual hashes also prove to be less effective. Switching to model-based features with the average cosine similarity (CS) reverses the correlation and increases computation time. We use for the latter a pre-trained VGG16 model on ImageNet with an output of shape $512\times 7\times 7$ and an input of shape $224\times 224$ [54, 55]. While $\mathcal{X}_{l}^{1}$ of KITTI consists of 1300 crops, there are 128763 crops at $i=1$ for BDD. Calculating CS scales quadratically in contrast to our image-based features distributions. Using CS without our count ratio from Eq. 1 further reduced the correlation to $r=-0.35$††footnotemark: compared to the results in Table IV, which also underscores the efficacy of Eq. 1.

Balancing Terms: The third ablation study is on the balancing terms in Eqs. 2 and 3. Setting $C_{w}$ around 1.0 activates class weighting through $w_{j}$ exclusively for the strongly class-imbalanced BDD. This strengthens the correlation between similarity and $\tau$ and results in higher $\tau$ post-subsampling (see Table V). In contrast, removing $w_{j}$, i.e., $w_{j}=1$ for BDD or applying it on KITTI decreases correlation. This highlights the importance of only incorporating the class balancing terms $w_{j}$ when necessary. Moreover, removing $\beta^{(P_{m})}$ causes the similarity for $\mathcal{X}_{\mathrm{val}}$ to dominate over the subsets due to the 1/10 count factor. This leads to its selection as $\mathrm{OSS}_{z}$ (see Sections III-B2 and 4) and therefore a significant drop in the correlation and post-subsampling $\tau$.

IV-E2 Cross-Detector Validation

The fourth ablation study evaluates the impact of detector choice on $\mathrm{OSS}$. We replace EfficientDet with YOLOv3 [56, 57], fine-tuned on KITTI (50 epochs, $1024\times 1024$ resolution, batch size 8, learning rate 0.0001, k-means optimized anchors). The strong linear correlation between $\mathrm{OSS}$ and mAP holds across detectors, with average $r_{\mathrm{KITTI}}(\mathcal{X}_{l}^{i}\|\mathcal{X}_{\text{val}})=0.92^{***}$ for YOLOv3 ($r=0.97^{***}$ for EfficientDet; see Fig. 3), confirming $\mathrm{OSS}$ is detector-agnostic despite YOLOv3’s lower mAP (1–-9%).

V Conclusion

We introduce a similarity-based approach that unifies training and evaluation strategies in AL by defining and quantifying informativeness and representativeness. Our approach addresses three key challenges in real-world AL: reducing computational costs, improving evaluation reliability and generalizability despite domain shift, and understanding informativeness in AL from a practical perspective with a case study using uncertainty-based AL methods. We find that uncertainty contributes to informativeness only when it is calibrated and considers the class distribution, thereby naturally increasing similarity between the selected training set and the target domain. Comprehensive experiments on KITTI, BDD100K, and CODA demonstrate the effectiveness of our novel $\mathrm{OSS}$ metric across diverse autonomous driving scenarios

Ultimately, our approach improves the practical usability of AL at minimal computational cost, supporting broader adoption of AL in real-world applications.

VI Discussion and Future Work

Most AL works focus on minimizing labeling efforts. Although reducing labeling costs is the primary objective, the required training runs during AL method development are also computationally expensive, an objective often dismissed in previous works. $\mathrm{OSS}$ mitigates this by leveraging already labeled sets to eliminate ineffective AL methods at each iteration. Nevertheless, we also contribute to minimizing labeling efforts in this work by improving uncertainty-based AL methods through uncertainty calibration and class balancing. Future work could explore diversity-based AL and pseudo-labeling in semi-supervised learning as case studies for $\mathrm{OSS}$. We use CODA as an alternative evaluation set to address dataset-specific biases and ensure robust evaluation. Future work could explore other sets with varying levels of domain shift to further understand their effects on evaluation reliability. Additionally, while $\mathrm{OSS}$ finds a reliable evaluation set through subsampling, future work could investigate generative upsampling. The correlation between mAP and $\mathrm{OSS}$ relies on sufficient diversity in class characteristics, struggling with less descriptive rare classes. Future work could use class-balancing augmentations and employ foundation models for more refined feature extraction. While $\mathrm{OSS}$ is not used in this work to select samples, future work could investigate using similarity instead of uncertainty for optimal sample selection.