SSRT-DETR: Domain-Adaptive Semi-Supervised Detector

TL;DR

SSRT-DETR is a new framework for domain-adaptive object detection that improves performance in challenging conditions like adverse weather.

Contribution

SSRT-DETR introduces Domain-Aware Matching and Class-/Scene-Adaptive Pseudo-Labeling to stabilize and enhance cross-domain object detection.

Findings

SSRT-DETR improves mAP@0.5 from 51.0 to 54.3 on Cityscapes→Foggy Cityscapes.

It achieves 67.3 AP on KITTI→Cityscapes and 64.9 AP on Sim10K→Cityscapes for car detection.

The framework shows consistent gains in rare categories and adverse weather scenarios.

Abstract

Domain-adaptive object detection under set-prediction paradigms remains challenging, as Hungarian matching is sensitive to domain shift and fixed pseudo-label thresholds cannot simultaneously handle class imbalance and scene variability. This paper presents SSRT-DETR, a semi-supervised, domain-adaptive framework built on the real-time detector RT-DETR. We adopt a mean teacher–student architecture with style-transferred images to jointly model source and target domains. To stabilize the assignment process during the early stages of cross-domain training, Domain-Aware Matching (DAM) is formulated to augment the Hungarian matching cost with a teacher-guided decoder-query consistency term. Leveraging the more stable EMA teacher representations, DAM guides early matching toward domain-consistent assignments and is gradually annealed to recover standard matching as training converges. In parallel, we introduce Class-/Scene-Adaptive Pseudo-Labeling (CAP) to address a key limitation of existing DAOD methods that rely on fixed or globally tuned pseudo-label thresholds, which struggle with class imbalance and scene-dependent difficulty under domain shift. CAP leverages per-class confidence statistics and multi-view consistency to adapt classification and IoU thresholds across classes and scenes, while temperature scaling and quality-weighted losses provide soft control over pseudo-label reliability. Experiments on standard benchmarks demonstrate the robustness of SSRT-DETR. On Cityscapes→Foggy Cityscapes, SSRT-DETR improves mAP@0.5 from 51.0 to 54.3. On KITTI→Cityscapes and Sim10K→Cityscapes, it achieves 67.3 AP and 64.9 AP on the car category, respectively, clearly outperforming the RT-DETR baseline while maintaining real-time efficiency. Notably, consistent gains are observed in rare categories and adverse weather scenarios, validating the effectiveness of the proposed DAM and CAP modules.

Figures

Full text

1. Introduction

In many real-world applications, such as autonomous driving, intelligent surveillance, and human–computer interaction, object detectors [1,2,3] are required to operate under conditions that differ substantially from those seen during training. Weather, illumination, sensor characteristics, and scene style can all change dramatically. This mismatch between training and test distributions, commonly referred to as domain shift [4,5,6], often leads to a significant drop in detection accuracy and has become a key bottleneck for large-scale deployment. Although modern deep learning-based detectors can achieve human-level or even super-human performance on standard benchmarks, their accuracy tends to degrade noticeably once they are transferred to new cities, seasons, or imaging conditions.

To address this issue, domain-adaptive object detection (DAOD) [7,8,9] aims to exploit labeled source-domain data and unlabeled target-domain data to reduce the feature distribution gap between domains and improve target-domain performance. Existing approaches can be roughly grouped into three families: methods that explicitly minimize statistical measures of distribution discrepancy, reconstruction- or generation-based methods that capture shared structure, and adversarial learning-based methods that perform alignment at the image and instance levels. In particular, frameworks built on adversarial feature alignment and pseudo-label self-training have produced promising results for CNN-based two-stage detectors. However, most of these methods are tailored for architectures with region proposal networks (RPNs) [1,10], and their applicability and effectiveness for the rapidly emerging Transformer-based set-prediction detectors, such as DETR and RT-DETR [11], remain less explored.

Unlike traditional proposal-based detectors, DETR and its variants [12,13] perform one-to-one set prediction via Hungarian matching, thereby avoiding anchor design and post-processing heuristics. While this paradigm is conceptually elegant, it introduces new challenges in cross-domain scenarios. On the one hand, under severe domain shift, classification scores and bounding-box regression are highly unstable in the early stages of training, and a matching cost that relies solely on these signals is prone to incorrect assignments, which can amplify pseudo-label noise. On the other hand, existing pseudo-label-based semi-supervised or self-training methods typically use fixed confidence and IoU thresholds, which cannot simultaneously accommodate long-tailed categories, small objects, and complex scenes such as foggy or nighttime conditions, leading to ongoing debates on how to choose appropriate thresholds. These issues become particularly critical for the real-time detector RT-DETR [11], whose design emphasizes efficiency and simplicity, leaving limited room for heavy domain adaptation components.

To tackle these challenges, this paper proposes SSRT-DETR, a semi-supervised domain-adaptive framework built upon RT-DETR. The central idea is to enhance cross-domain robustness from two complementary perspectives—matching stability and pseudo-label quality control—while preserving real-time inference. Concretely, we introduce Domain-Aware Matching (DAM), which augments the Hungarian matching cost with a training-annealed domain discrepancy term driven by teacher–student decoder query similarity, thereby mitigating incorrect assignments at the early stages of cross-domain training. In parallel, we design Class-/Scene-Adaptive Pseudo-Labeling (CAP), which leverages per-class confidence distributions, multi-view consistency, temperature scaling, and quality-weighted supervision to perform fine-grained pseudo-label filtering and soft weighting, thus improving the effectiveness of supervision for long-tailed categories and adverse scenarios.

The present study aims to address two key questions: how to design a matching mechanism that is more robust to domain shift under the set-prediction paradigm, and how to exploit noisy target-domain pseudo-labels effectively without incurring additional inference cost. Through extensive experiments on standard adaptation benchmarks such as Cityscapes→Foggy Cityscapes, KITTI→Cityscapes, and Sim10K→Cityspaces, we show that SSRT-DETR can consistently improve cross-domain detection performance without sacrificing the real-time efficiency of RT-DETR, with particularly pronounced gains on rare categories and adverse weather conditions. Overall, the contributions of this work lie in providing an efficient yet robust domain-adaptation scheme for real-time Transformer-based detectors and in offering a new perspective on the interplay between matching and pseudo-label design within set-prediction frameworks.

In summary, our contributions are as follows:

• SSRT-DETR is introduced as a semi-supervised domain-adaptive framework for set-prediction-based object detectors, effectively mitigating matching instability and pseudo-label noise under domain shift while preserving real-time performance.
• We introduce Domain-Aware Matching (DAM), a teacher-guided matching strategy that stabilizes early-stage Hungarian assignment through an annealed domain-consistency term.
• We present Class-/Scene-Adaptive Pseudo-Labeling (CAP), which overcomes the limitations of fixed pseudo-label thresholds by adapting classification and IoU constraints across classes and scenes, combined with quality-weighted supervision.

2. Related Work

2.1. Transformer-Based Object Detection

Transformer-based object detection [14] has attracted extensive interest since the introduction of DETR [12], which replaces proposal-based pipelines with an end-to-end set prediction paradigm. DETR uses Hungarian matching for one-to-one assignment, eliminating anchor design and non-maximum suppression. However, vanilla DETR suffers from slow convergence and strong dependence on large-scale data. Subsequent works have introduced various enhancements: Deformable DETR [13] incorporates deformable attention to focus on sparse spatial regions and accelerates training; DINO [15] improves query initialization and localization precision through hybrid query selection and a two-stage architecture; and the RT-DETR family [11] achieves real-time performance through efficient multi-scale encoders and lightweight query selection, establishing itself as a practical choice for Transformer-based detection. Nevertheless, applying Transformer detectors to cross-domain scenarios remains challenging, as their global attention mechanisms often exacerbate feature distribution shifts, necessitating dedicated domain-adaptation strategies to ensure robustness.

2.2. Domain-Adaptive Object Detection

Unsupervised domain-adaptive Object Detection (UDA-OD) aims to reduce the distribution discrepancy between source and target domains, thereby mitigating performance degradation caused by domain shift. Early efforts were based on Faster R-CNN [8,9,16,17], leveraging gradient reversal layers or adversarial training to perform image-level and instance-level alignment, with the DANN family playing a pivotal role [4,5,18,19]. Subsequent studies introduced more refined alignment schemes, such as strong-weak alignment, task-specific alignment, and multi-level alignment, improving robustness for both classification and localization. However, these methods rely heavily on region proposal networks and are primarily used with two-stage detectors. With the rise of DETR, Transformer-based DAOD has become an emerging trend. Several works attempt to reduce domain gaps by introducing domain queries, encoder feature alignment, or masked reconstruction, whereas others integrate adversarial alignment, instance-level matching, or deformable attention into DETR frameworks [20,21]. Nevertheless, existing approaches have not fully addressed the matching instability inherent to set prediction under domain shift, nor have they adequately modeled how pseudo-label noise propagates within Transformer detectors, leaving room for further advancements.

2.3. Semi-Supervised and Pseudo-Label Learning for Set-Prediction Detectors

In semi-supervised and pseudo-label learning, teacher–student frameworks have become increasingly popular due to their ability to exploit unlabeled target-domain data without manual annotation [22,23,24,25,26,27,28]. Mean Teacher (MT) [29] and its variants employ consistency regularization and EMA-updated teachers to generate stable pseudo-labels for the student model. Subsequent works incorporate image-level alignment, confidence calibration, and other techniques to enhance pseudo-label reliability [30,31,32]. However, applying pseudo-label learning directly to set-prediction detectors introduces two major challenges [32,33]: first, the label assignment in DETR-like models depends on Hungarian matching, and unstable early-stage predictions can propagate matching noise into pseudo-labels; second, traditional pseudo-label filtering with fixed thresholds struggles to accommodate long-tailed class distributions and diverse scene conditions, resulting in uneven-quality pseudo-labels liable to mislead training. Although recent studies explore structured pseudo-labels, multi-view consistency, or masked reconstruction for Transformer detectors, few have focused on the coupled issue of matching stability and pseudo-label selection inherent to set prediction under domain shift. This gap motivates our SSRT-DETR framework, which explicitly addresses these intertwined challenges.

In summary, existing works have made significant progress in Transformer-based detection, domain adaptation, and pseudo-label learning. However, their combination under the set-prediction paradigm remains under-explored, particularly regarding the coupled issues of Hungarian matching instability and pseudo-label noise propagation under domain shift. This work focuses on this specific scope and proposes SSRT-DETR to address these challenges in a unified framework.

3. Materials and Methods

In this section, we introduce SSRT-DETR, a semi-supervised domain-adaptive framework built upon the real-time detector RT-DETR. We first formalize the problem setup and notations, then describe the base RT-DETR loss, the proposed Domain-Aware Matching (DAM), the Class-/Scene-Adaptive Pseudo-Labeling (CAP), the multi-view quality scoring, and finally the overall training objective.

As illustrated in Figure 1, SSRT-DETR follows a mean teacher–student training paradigm with real/fake style-transferred views. The teacher model generates pseudo-labels for target-domain images, while the student is trained using both labeled source data and quality-weighted target supervision. During training, Domain-Aware Matching (DAM) stabilizes Hungarian assignment in the early stages of cross-domain optimization, and Class-/Scene-Adaptive Pseudo-Labeling (CAP) filters and weights pseudo-labels to mitigate noise caused by class imbalance and scene variability. All components are jointly optimized in an end-to-end manner without introducing additional inference-time overhead.

3.1. Problem Setup and Notation

We consider the standard unsupervised domain adaptation setting for object detection, with a labeled source domain $[eqn]$ and an unlabeled target domain $[eqn]$ . To reduce image-level domain discrepancy, both source and target samples are augmented with a style transfer network (e.g., CUT [34]), yielding “real/fake” dual views [35]. We denote

$$[eqn]$$ $$[eqn]$$

where each

$$[eqn]$$

is a set of box-label pairs. The student detector is denoted by $[eqn]$ , and the teacher by $[eqn]$ , updated via EMA:

$$[eqn]$$

For each target image, the teacher produces a pseudo-label set

$$[eqn]$$

where $[eqn]$ is the predicted box, $[eqn]$ the class, and $[eqn]$ the quality score. The native RT-DETR detection loss is written as $[eqn]$ , comprising multi-level classification, L1 regression, GIoU, and denoising losses. On top of this base, SSRT-DETR introduces three key components: Domain-Aware Matching (DAM) to improve assignment robustness under domain shift, Class-/Scene-Adaptive Pseudo-Labeling (CAP) to enhance pseudo-label quality under long-tailed distributions and complex scenes, and multi-view quality scoring with temperature scaling to provide continuous weighting for pseudo-labels, thereby improving target-domain generalization without additional inference cost.

3.2. RT-DETR Base Detection Loss

In RT-DETR [36], the detection loss at each layer is determined by set prediction and Hungarian matching. Given an input image x and a target set

$$[eqn]$$

the RT-DETR detection loss is as follows:

$$[eqn]$$

For layer $[eqn]$ , it corresponds to the encoder head, and the remaining layers belong to the decoder; $[eqn]$ are scalar weights, and $[eqn]$ is the denoising loss. At layer ℓ, the Hungarian algorithm produces a matching set $[eqn]$ , where $[eqn]$ indicates that query i is matched to ground truth j. The classification loss is modeled with an IoU-weighted Focal/Varifocal Loss, yielding the following:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ denotes the number of object queries in the decoder.

The L1 box loss and GIoU loss are defined as follows:

$$[eqn]$$ $$[eqn]$$

Under severe domain shift and unstable early predictions, the matching set $[eqn]$ becomes a major source of error propagation, which motivates our domain-aware modification of the matching cost.

3.3. Domain-Aware Hungarian Matching (DAM)

The key challenge addressed by DAM is that Hungarian matching in set-prediction detectors relies entirely on student predictions, which are highly unreliable in the early stages of cross-domain training. This often leads to incorrect assignments that propagate errors to subsequent supervision. DAM alleviates this issue by introducing a teacher-guided regularization term into the matching cost, where the EMA-updated teacher provides more stable query representations to guide early assignments. Here, $[eqn]$ , $[eqn]$ , and $[eqn]$ denote the weighting coefficients for classification, box regression, and GIoU costs, respectively, and $[eqn]$ controls the relative contribution of the domain-aware term.

As shown in Figure 2, DAM augments the Hungarian matching cost with a teacher–student query consistency term, which stabilizes assignments under domain shift.

During early cross-domain training, student predictions on target-domain samples are highly unreliable, whereas vanilla Hungarian matching relies solely on these predictions to build the matching cost, making it vulnerable to incorrect assignments and amplifying subsequent supervision errors. To mitigate this, we introduce a domain-aware term that regularizes matching via consistency between teacher and student decoder queries. Let the student and teacher decoder queries at layer ℓ be

$$[eqn]$$

We define the domain cost via cosine distance:

$$[eqn]$$

Note that $[eqn]$ is a row-wise regularization term that depends only on the query index i and is broadcast to all candidate matches $[eqn]$ during Hungarian assignment. The full matching cost at layer ℓ is then extended to

$$[eqn]$$

where the first three terms are derived from classification, L1, and GIoU losses, and $[eqn]$ is an annealed weight depending on the normalized training progress $[eqn]$ . To keep the domain term at a stable ratio $[eqn]$ of the total cost, we estimate the average standard and domain costs within each batch,

$$[eqn]$$ $$[eqn]$$

and set

$$[eqn]$$

where $[eqn]$ is a hyper-parameter controlling the relative contribution of the domain-aware cost term.

$$[eqn]$$

where $[eqn]$ is the normalized training progress. Thus, the domain term dominates matching during early training and gradually vanishes, yielding domain-consistent assignments without affecting final convergence behavior.

DAM introduces a teacher-guided regularization term into Hungarian matching, which is gradually annealed to stabilize early assignments under domain shift.

3.4. Class-/Scene- and Prototype-Aware Pseudo-Labeling (CAP)

Existing pseudo-labeling strategies in domain-adaptive detection typically rely on fixed or globally tuned thresholds, which cannot simultaneously address class imbalance and scene-dependent difficulty under domain shift. As a result, pseudo-labels for rare classes or complex scenes are often either excessively noisy or overly suppressed. CAP is designed to address this limitation by adaptively adjusting confidence and IoU thresholds across classes and scenes, and by further integrating multi-view consistency and prototype-aware semantic verification. In single-class adaptation scenarios (e.g., KITTI→Cityscapes and Sim10K→Cityscapes), the class-adaptive components of CAP naturally degenerate to a class-agnostic form and therefore do not affect the training dynamics or inference behavior.

As illustrated in Figure 3, CAP generates pseudo-label candidates from the EMA teacher on both real/fake views, then filters and weights them using adaptive thresholds, multi-view consistency, and prototype-aware semantic verification.

Pseudo-label quality is the key factor that determines whether target-domain learning remains stable in semi-supervised domain-adaptive detection. While conventional CAP reduces noise via class-adaptive confidence filtering and real–fake geometric consistency, it may still retain class-confusion pseudo-labels that are confident and geometrically stable yet semantically incorrect under severe domain shift. To address this, we fully integrate a prototype-aware semantic consistency mechanism into CAP. The resulting CAP enforces pseudo-label reliability with three coupled constraints: (i) frequency-adaptive confidence filtering, (ii) frequency-adaptive geometric consistency, and (iii) prototype-based semantic consistency in feature space. All constraints are unified into a continuous quality score used for filtering and distillation weighting, without requiring any additional target annotations or introducing inference-time overhead.

Relative-Frequency Normalization

To avoid dataset-specific hard thresholds (e.g., $[eqn]$ ), we define all class-adaptive thresholds as continuous functions of the relative class frequency. Let $[eqn]$ be the number of source samples for class c. We compute a (log-)normalized frequency

$$[eqn]$$

where $[eqn]$ is a numerical stabilizer. Intuitively, $[eqn]$ indicates frequent classes and $[eqn]$ indicates rare classes.

We define a class-specific confidence threshold as a monotone decreasing function of $[eqn]$ , so that frequent classes use looser thresholds while rare classes use stricter ones:

$$[eqn]$$

Similarly, to impose stronger geometric stability for rare classes, we define the class-specific real–fake IoU threshold (also monotone decreasing in $[eqn]$ ):

$$[eqn]$$

For each target image, the EMA teacher predicts on both $[eqn]$ and $[eqn]$ . For view $[eqn]$ , we denote the candidate set as

$$[eqn]$$

where $[eqn]$ is the box, $[eqn]$ the predicted class, $[eqn]$ the temperature-scaled confidence, and $[eqn]$ the decoder query feature at layer ℓ. We first apply class-adaptive confidence filtering by retaining candidates satisfying

$$[eqn]$$

Then, for each remaining fake-view candidate $[eqn]$ , we find the best same-class match in the real view:

$$[eqn]$$

and keep the pair only if the class-adaptive IoU constraint holds:

$$[eqn]$$

We set the pseudo-label class as $[eqn]$ and use the real-view box $[eqn]$ as supervision since the student is trained on $[eqn]$ .

Confidence and IoU constraints alone may still retain semantically wrong yet stable pseudo-labels. We therefore introduce a prototype-aware semantic consistency term. On the labeled source branch, we maintain one prototype vector per class using high-confidence matched decoder queries. Let $[eqn]$ denote the Hungarian matching set at layer ℓ. For $[eqn]$ , query i is matched to ground-truth j of class $[eqn]$ , with student query feature $[eqn]$ . We update prototypes only when the student confidence on the GT class satisfies $[eqn]$ . Define

$$[eqn]$$

where $[eqn]$ is a confidence threshold used to select reliable source-domain queries for prototype updating. And compute the batch mean of normalized queries:

$$[eqn]$$

We then update the class prototype with EMA and re-normalize

$$[eqn]$$

where $[eqn]$ is the prototype update rate.

For each target pseudo-label candidate that passes the confidence and IoU constraints, we compute its query–prototype cosine similarity. We represent the candidate by averaging normalized teacher queries from the real and fake views:

$$[eqn]$$

and compute

$$[eqn]$$

To enforce stricter prototype constraints for frequent classes while being more tolerant for rare classes, we define a frequency-adaptive prototype threshold as a monotone increasing function of $[eqn]$ :

$$[eqn]$$

We then obtain a soft prototype weight by a sigmoid mapping:

$$[eqn]$$

where $[eqn]$ controls the softness of the gate.

We apply temperature scaling to teacher logits z to obtain calibrated confidences,

$$[eqn]$$

and define the multi-view base quality using the geometric mean:

$$[eqn]$$

Finally, CAP integrates prototype semantics into the quality score:

$$[eqn]$$

We retain pseudo-labels with $[eqn]$ to form $[eqn]$ , and use $[eqn]$ (or its batch average) to weight the distillation loss. By unifying confidence, geometry, and prototype-based semantic consistency, CAP effectively reduces class-confusion noise and improves pseudo-label reliability and balance, particularly for rare categories.

3.5. Pseudo-Label Quality Weighting with Temperature Scaling

After CAP filtering, pseudo-labels still present a continuous spectrum of reliability. Instead of using a hard accept/reject decision only, we leverage the CAP quality score to softly weight the contribution of pseudo-label supervision.

Let the teacher’s logits for the m-th candidate in the fake/real views be $[eqn]$ and $[eqn]$ . We obtain calibrated confidences via temperature scaling:

$$[eqn]$$

where $[eqn]$ is the temperature. Denote by $[eqn]$ the IoU of the matched real–fake boxes computed in CAP. Following the multi-view design in Section 3.4, we define the base multi-view quality as the geometric mean:

$$[eqn]$$

To further suppress semantically inconsistent yet geometrically stable pseudo-labels under domain shift, we incorporate the prototype-aware weight $[eqn]$ computed in CAP:

$$[eqn]$$

We retain pseudo-labels with $[eqn]$ to form $[eqn]$ , where $[eqn]$ denotes the minimum quality threshold for pseudo-label selection.

For each target image, the average pseudo-label quality is computed as follows:

$$[eqn]$$

The student is distilled on $[eqn]$ using pseudo-labels $[eqn]$ through

$$[eqn]$$

When pseudo-label quality is high, $[eqn]$ approaches 1 and target supervision becomes effective; when pseudo-labels are unreliable, $[eqn]$ automatically down-weights the distillation signal, thereby suppressing noise in a continuous fashion.

3.6. Source Branches and Overall Objective

On the source domain, the student detector minimizes the RT-DETR detection loss on both real and stylized images,

$$[eqn]$$ $$[eqn]$$

To encourage consistent detection performance under style changes, we introduce a source–fake consistency constraint:

$$[eqn]$$

where $[eqn]$ is a hyper-parameter. Combining source supervision, target-domain pseudo-label distillation, and consistency regularization, the overall loss of SSRT-DETR in each iteration is given by

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ balances supervised and unsupervised losses. Importantly, Domain-Aware Matching (DAM) modifies the internal matching cost of $[eqn]$ and thus acts as a structural change to the detection loss rather than an additional explicit term, which implies no extra cost at inference time.

Compared with existing domain-adaptive object detection methods, SSRT-DETR introduces two key improvements specifically tailored for set-prediction detectors: (1) DAM stabilizes Hungarian matching during early cross-domain training, effectively reducing error propagation; (2) CAP improves pseudo-label reliability by jointly addressing class imbalance and scene variability through adaptive thresholding and quality-weighted supervision. These improvements are tightly integrated into the RT-DETR framework, leading to more stable optimization and better generalization under domain shifts.

4. Experiments

We conducted extensive experiments to validate the effectiveness of the proposed method on a variety of cross-domain adaptation benchmarks. These benchmarks cover diverse adaptation scenarios, including adverse weather adaptation (Cityscapes→Foggy Cityscapes), synthetic-to-real adaptation (Sim10K→Cityscapes), and cross-camera adaptation (KITTI→Cityscapes). In addition, comprehensive ablation studies were performed to analyze the contribution of each proposed module. Experimental results consistently demonstrate that our method is highly robust and adaptable across domain shifts, leading to notable improvements in generalization performance across multiple scene understanding tasks.

4.1. Datasets

Cityscapes→Foggy Cityscapes. Cityscapes [37] is a widely used urban scene understanding benchmark, consisting of 2975 training images and 500 validation images with high-quality pixel-level annotations. Foggy Cityscapes [38] is derived from Cityscapes by synthetically introducing fog effects with varying intensities. Specifically, three fog density levels (0.005, 0.01, and 0.02) are provided, corresponding to light, moderate, and heavy fog conditions, respectively. In our experiments, we adopt the most severe setting (fog level 0.02) to evaluate performance on the eight object categories shared by both domains.

Sim10K→Cityscapes. Sim10K [39] is a large-scale synthetic dataset generated using a game engine, containing 10,000 images annotated with object-level bounding boxes. In this adaptation scenario, Sim10K serves as the source domain, and Cityscapes as the target domain. Following standard practice, evaluation is conducted exclusively on the “car” category.

KITTI→Cityscapes. KITTI [3] is another real-world street scene dataset captured using different cameras and collected from cities distinct from those in Cityscapes, resulting in a noticeable domain gap. In this setting, we employ 7481 KITTI images for training and focus on the “car” category, which is the only class shared between the two datasets.

4.2. Implementation Details

To provide a clear overview of the proposed training pipeline, Algorithm 1 summarizes the complete optimization procedure of SSRT-DETR. Different from online style translation, the fake-view images used in our framework are generated offline using CUT and stored together with their corresponding real images. During training, each mini-batch therefore consists of paired real/fake samples from both source and target domains, enabling consistent dual-view learning without additional runtime cost.

The student model is optimized using supervised losses on the labeled source domain and quality-weighted pseudo-label supervision on the unlabeled target domain. The teacher model is updated by an exponential moving average (EMA) and is only used to generate pseudo-labels and guide matching. Notably, Domain-Aware Matching (DAM) is applied inside the Hungarian matching process when computing the detection loss, while Class-/Scene-Adaptive Pseudo-Labeling (CAP) is applied only on the target branch during pseudo-label generation and filtering. Algorithm 1: Training Procedure of SSRT-DETR (offline fake images)

It is worth emphasizing that DAM and CAP play fundamentally different roles in the proposed framework. DAM modifies the internal assignment mechanism of set-prediction detectors by augmenting the Hungarian matching cost, and therefore affects how supervision signals are constructed during training. In contrast, CAP operates at the pseudo-label level and aims to control the reliability and contribution of unlabeled target supervision. This clear separation allows SSRT-DETR to jointly stabilize early-stage assignment and suppress noisy pseudo-label propagation, without introducing additional modules or inference-time overhead.

Table 1 reports all key hyperparameters used in our experiments to ensure reproducibility. Unless otherwise specified, the same configuration is shared across all adaptation benchmarks. The EMA momentum $[eqn]$ is applied at every training iteration to update the teacher model. The DAM scaling factor $[eqn]$ controls the relative contribution of the domain-aware matching cost and is kept fixed throughout training, while the annealing schedule is implicitly determined by the training progress. CAP-related parameters ( $[eqn]$ , $[eqn]$ , $[eqn]$ , and T) are selected based on preliminary experiments and kept fixed across all benchmarks.

All experiments were conducted using PyTorch 2.8 with CUDA 12.8. The implementation is based on the Ultralytics RT-DETR codebase. Experiments were run on a single NVIDIA RTX 5090 GPU, under Ubuntu 22.04 and Python 3.12. All experiments in this paper are conducted with a fixed random seed (seed = 0). Due to computational constraints, results are reported from a single run. yield

4.3. Comparison with Other Methods

We compare SSRT-DETR with representative domain-adaptive detection baselines on three benchmarks. Table 2 reports results on Cityscapes→Foggy Cityscapes, Table 3 on KITTI→Cityscapes, and Table 4 on Sim10K→Cityscapes.

Table 2 reports adaptation performance on Cityscapes→Foggy Cityscapes. Compared with the RT-DETR-L source-only baseline, DAM* and CAP* each yields consistent gains, indicating that stabilizing Hungarian assignment and improving pseudo-label reliability are both beneficial under adverse weather shift. Notably, combining DAM + CAP* achieves the best overall mAP, suggesting that the two modules provide complementary improvements: DAM mainly reduces early-stage assignment noise, while CAP further suppresses pseudo-label errors under class imbalance and scene variability.

Table 5 presents a component-wise ablation study to analyze the contributions of the two key modules in SSRT-DETR. DAM and CAP are evaluated both individually and jointly on top of the same RT-DETR-L baseline.

When applied alone, DAM yields a consistent improvement over the baseline, indicating that stabilizing Hungarian matching is beneficial under domain shift. CAP delivers a greater performance gain when used independently, underscoring the importance of adaptive pseudo-label selection and weighting. Combining DAM and CAP yields further improvement, demonstrating that the two modules are complementary and address different failure modes in set-prediction, domain-adaptive detection.

Table 6 reports the per-class AP for two representative categories, bicycle and train, which are known to be sensitive to domain shift and class imbalance under foggy conditions.

For the bicycle class, introducing DAM improves AP over the baseline, indicating that stabilizing Hungarian matching is beneficial for objects with ambiguous boundaries. CAP alone yields a comparable but slightly lower AP, suggesting that adaptive pseudo-label filtering may be more conservative for this class. When DAM and CAP are combined, the bicycle AP recovers and reaches the highest level, showing that CAP benefits from the more stable assignments provided by DAM.

For the train class, which is relatively rare and prone to class confusion in Foggy Cityscapes, DAM alone leads to a noticeable AP drop. This behavior is expected, as stabilizing matching without explicitly controlling pseudo-label noise may still propagate semantically incorrect assignments for rare categories. In contrast, CAP significantly improves train AP by enforcing class-adaptive thresholds and semantic consistency. Combining DAM and CAP effectively compensates for the weaknesses of each component, resulting in a substantial recovery of train AP to near baseline levels while maintaining the overall performance gain. These observations further confirm the complementary nature of DAM and CAP.

As shown in Table 3, SSRT-DETR achieves the highest car AP on KITTI→Cityscapes. The strong improvement supports our motivation that DAM helps mitigate assignment errors in set-prediction detectors, thereby enabling more reliable self-training on the unlabeled target domain.

Table 4 presents results on synthetic-to-real adaptation (Sim10K→Cityscapes). The performance gain indicates that SSRT-DETR effectively handles large appearance discrepancies. CAP alleviates this issue by adapting confidence/IoU constraints and using quality-weighted supervision, leading to more stable target-domain learning.

Overall, SSRT-DETR consistently improves adaptation performance across adverse weather, cross-camera, and synthetic-to-real settings. These results validate the effectiveness of DAM in stabilizing Hungarian matching under domain shift and CAP in improving pseudo-label quality. Importantly, the proposed modules are training-time modifications and do not introduce additional inference-time overhead.

5. Discussion

This work targets a practical yet under-explored setting: adapting real-time set-prediction detectors to unlabeled target domains, where the one-to-one Hungarian assignment and pseudo-label self-training are both prone to error amplification under domain shift. Building upon RT-DETR, SSRT-DETR integrates a mean teacher–student paradigm with dual-view (real/fake) training, and further introduces two complementary mechanisms—Domain-Aware Matching (DAM) and Class-/Scene-Adaptive Pseudo-Labeling (CAP)—to enhance matching stability and pseudo-label reliability without altering the inference pipeline. It is worth noting that in single-class adaptation scenarios such as KITTI→Cityscapes, the class-adaptive components of CAP naturally degenerate to a class-agnostic form and therefore do not actively influence pseudo-label selection. In such cases, the observed performance gains primarily stem from the stabilization of Hungarian assignment provided by DAM, while CAP introduces no additional training or inference overhead.

5.1. Overall Effectiveness Across Adaptation Scenarios

The experimental results indicate that SSRT-DETR improves cross-domain detection performance on challenging scene shifts. On Cityscapes→FoggyCityscapes, SSRT-DETR yields consistent gains over representative DAOD baselines, with particularly notable improvements on fog-sensitive or visually ambiguous categories. For instance, large relative gains are observed for categories such as bicycle and train, which are typically vulnerable to contrast degradation and boundary ambiguity in adverse weather. On KITTI→Cityscapes (car-only adaptation), SSRT-DETR achieves a substantial improvement in target-domain AP compared with the source-only RT-DETR baseline, suggesting that the proposed training scheme effectively transfers localization and classification knowledge across camera/scene domains. These results collectively support the central claim that stabilizing assignment and calibrating pseudo-label selection are both critical for set-prediction DAOD.

5.2. Why DAM Matters for Set-Prediction Domain Adaptation

A key failure mode in DETR-like models under domain shift lies in early-stage assignment noise: when classification confidence and box regression are unreliable, the vanilla matching cost may select suboptimal query–object pairs, and the resulting supervision can further deteriorate training. DAM addresses this issue by augmenting the matching cost with a teacher–student query consistency term whose contribution is annealed over training progress. Conceptually, this domain-aware regularizer biases the matcher toward queries with more stable cross-view representations, helping mitigate spurious assignments in the early phase. Importantly, DAM operates inside the matching procedure and does not introduce additional inference-time components, making it compatible with real-time detectors.

5.3. Why CAP Improves Pseudo-Label Quality Under Class and Scene Variability

Pseudo-labeling with fixed thresholds is known to be brittle in cross-domain settings, especially when class frequencies are imbalanced, and scene conditions vary dramatically. CAP explicitly addresses this by (i) employing frequency-adaptive confidence thresholds, (ii) enforcing real–fake geometric consistency through class-adaptive IoU constraints, and (iii) incorporating prototype-based semantic verification to suppress class-confusion pseudo-labels that are geometrically stable yet semantically incorrect. Furthermore, the use of temperature scaling and quality-weighted distillation provides a soft mechanism to modulate the contribution of pseudo-label supervision, preventing low-quality pseudo-labels from dominating optimization. These design choices align with the empirical observation that the largest improvements tend to appear on categories and scenarios where domain shift most severely disrupts confidence calibration and semantic separability.

6. Conclusions

This paper presented SSRT-DETR, a semi-supervised domain-adaptive framework for real-time set-prediction detectors built upon RT-DETR. The proposed approach targets two key challenges in domain-adaptive detection: stabilizing Hungarian matching under domain shift and effectively exploiting noisy target-domain pseudo-labels without introducing additional inference cost.

To address the first challenge, Domain-Aware Matching (DAM) is formulated by augmenting the Hungarian matching cost with an annealed teacher–student query consistency term.

Experimental results across all adaptation scenarios demonstrate that DAM effectively stabilizes early-stage assignment and reduces error propagation, particularly under severe domain shift, while preserving the original inference pipeline.

To address the second challenge, we introduced Class-/Scene-Adaptive Pseudo-Labeling (CAP), which improves pseudo-label reliability through frequency-adaptive thresholds, multi-view geometric consistency, prototype-based semantic verification, and quality-weighted distillation. The effectiveness of CAP is validated in multi-class adaptation scenarios such as Cityscapes→FoggyCityscapes, where class imbalance and scene variability significantly affect pseudo-label quality. In single-class scenarios, CAP naturally degenerates to a class-agnostic form and does not alter training or inference behavior.

Overall, these results highlight the importance of jointly addressing assignment stability and pseudo-label quality control for domain adaptation in DETR-like detectors, and demonstrate that SSRT-DETR provides an effective and efficient solution for adapting real-time Transformer-based detection systems to unlabeled target domains.

7. Patents

The authors declare that no patents resulted from the work reported in this manuscript.