Make me an Expert: Distilling from Generalist Black-Box Models into Specialized Models for Semantic Segmentation

TL;DR

This paper introduces a novel method called ATGC for distilling specialized semantic segmentation models from generalist black-box models using only one-hot predictions, addressing the challenge of scale sensitivity and improving performance.

Contribution

The paper proposes ATGC, a scale selection method leveraging attention maps for effective black-box model distillation in semantic segmentation, under realistic API access constraints.

Findings

Significant performance improvements over baseline methods.

Effective scale selection using attention map entropy.

Robustness across multiple datasets.

Abstract

The rise of Artificial Intelligence as a Service (AIaaS) democratizes access to pre-trained models via Application Programming Interfaces (APIs), but also raises a fundamental question: how can local models be effectively trained using black-box models that do not expose their weights, training data, or logits, a constraint in which current domain adaptation paradigms are impractical ? To address this challenge, we introduce the Black-Box Distillation (B2D) setting, which enables local model adaptation under realistic constraints: (1) the API model is open-vocabulary and trained on large-scale general-purpose data, and (2) access is limited to one-hot predictions only. We identify that open-vocabulary models exhibit significant sensitivity to input resolution, with different object classes being segmented optimally at different scales, a limitation termed the "curse of resolution". Our method, ATtention-Guided sCaler (ATGC), addresses this challenge by leveraging DINOv2 attention maps to dynamically select optimal scales for black-box model inference. ATGC scores the attention maps with entropy to identify informative scales for pseudo-labelling, enabling effective distillation. Experiments demonstrate substantial improvements under black-box supervision across multiple datasets while requiring only one-hot API predictions. Our code is available at https://github.com/yasserben/ATGC.

Tables

Table 1. Table 1: Different model adaptation settings. ( \big{(} , , ) \big{)} denote white-box, gray-box and black-box models, respectively. B 2 UDA methods assume access to logits, whereas our B 2 D does not. ( \big{(} , ) \big{)} denote closed-set and open-vocabulary pre-trained source model. Due to the use of open-vocabulary models, B 2 D allows adaptation to any desired set of classes, making it highly flexible for deployment on downstream tasks.

Settings	Source data	Pretrained model		Deployment feasibility
UDA [26]	$✓$			Low
SFUDA [43]	$\times$			Moderate
B2UDA [18]	$\times$			Moderate
B2D (Ours)	$\times$			High

Table 2. Table 2 : Specialization to Cityscapes with SAN. The best score for each column is highlighted in bold . Results are obtained using prompt engineering and averaged over 3 random seeds. † Reimplemented by us as the original code was not available.

Method	Logits	Road	SW	Buil.	Wall	Fence	Pole	TL	TS	Veg.	Ter.	Sky	PR	Rider	Car	Truck	Bus	Train	Mot.	Bike	mIoU
SAN [64]	-	88.7	50.1	82.0	29.2	33.4	3.1	22.5	29.8	81.3	20.9	86.1	53.5	0.7	73.1	23.9	60.0	50.3	46.8	53.0	46.8
CoRTE†[18]	✓	88.6	49.9	82.4	29.9	28.9	0.0	29.4	32.2	82.6	26.3	81.1	55.3	0.0	76.0	26.4	67.8	56.0	47.2	58.4	48.4
Naive Transfer	$\times$	89.2	50.4	82.3	28.3	31.9	0.9	32.1	33.1	82.5	28.8	82.7	54.8	0.0	75.1	25.0	67.7	56.4	48.0	57.8	48.8
Average	$\times$	88.4	45.6	80.5	25.0	27.7	0.3	25.9	28.2	80.4	21.6	81.5	51.9	0.0	74.3	24.3	62.4	29.6	44.7	55.6	44.6
Random	$\times$	88.2	48.2	82.3	27.9	31.9	5.5	31.2	37.0	82.9	26.0	83.1	55.8	0.0	75.8	24.6	67.3	48.2	47.9	59.4	48.6
ATGC (Ours)	$\times$	86.6	46.3	82.4	26.2	35.3	10.5	34.6	41.7	83.8	28.6	84.7	58.0	0.0	76.6	23.8	67.9	52.2	50.8	61.3	50.1
Oracle	-	91.8	56.8	84.4	40.7	41.7	9.4	35.6	41.3	84.8	35.0	83.7	57.6	0.0	79.2	28.4	70.0	60.8	50.8	61.1	53.3
Supervised	-	95.9	71.3	86.0	52.2	50.1	26.4	38.5	52.2	86.2	55.6	82.2	63.5	42.9	88.2	76.5	78.5	70.3	51.5	61.3	64.7

Table 3. Table 3 : Specialization to Cityscapes with CLIP-DINOiser. The best score for each column is highlighted in bold . Results are obtained using prompt engineering and averaged over 3 random seeds. † Reimplemented by us as the original code was not available.

Method	Logits	Road	SW	Buil.	Wall	Fence	Pole	TL	TS	Veg.	Ter.	Sky	PR	Rider	Car	Truck	Bus	Train	Mot.	Bike	mIoU
CLIP-DINOiser [62]	-	68.5	17.0	73.7	23.4	24.0	10.1	2.2	3.5	78.8	10.2	43.3	45.2	9.2	69.3	26.1	52.9	26.8	27.4	42.3	34.4
CoRTE† [18]	✓	64.9	18.5	75.8	23.1	18.5	10.0	0.0	0.0	80.5	4.5	36.9	52.0	3.0	72.9	30.2	57.1	24.6	35.8	47.2	34.5
Naive Transfer	$\times$	64.0	17.8	75.2	21.4	21.4	10.2	0.0	0.0	80.6	5.0	37.6	49.7	4.0	71.3	27.6	56.0	28.5	35.0	45.6	34.3
Average	$\times$	65.5	18.3	72.2	24.3	25.3	8.4	0.0	0.0	76.4	8.6	25.6	48.8	2.4	67.9	27.2	55.9	44.5	32.5	43.5	34.1
Random	$\times$	64.8	18.7	74.5	23.9	24.3	10.3	0.0	0.0	79.1	11.5	35.6	51.2	1.6	70.6	28.2	58.2	41.4	33.3	47.3	35.5
ATGC (Ours)	$\times$	47.7	15.6	78.0	22.5	24.2	16.3	0.2	4.7	83.1	27.4	62.1	53.0	0.7	73.1	28.9	56.1	35.8	35.5	54.3	37.9
Oracle	-	76.7	24.4	78.8	30.5	31.3	14.2	0.0	2.5	81.5	25.4	49.3	52.8	6.5	74.1	33.4	62.3	48.6	34.1	49.9	40.9
Supervised	-	95.9	71.3	86.0	52.2	50.1	26.4	38.5	52.2	86.2	55.6	82.2	63.5	42.9	88.2	76.5	78.5	70.3	51.5	61.3	64.7

Table 4. Table 4 : Specialization to ACDC. Results of training ATGC on the unlabelled ACDC dataset compared to state-of-the-art specialization methods across varying weather conditions. The best score for each column is highlighted in bold .

	Method	rain	fog	night	snow	Average
SAN	SAN [64]	45.5	45.9	31.8	48.0	42.8
	CoRTE [18]	42.7	45.7	31.9	42.0	40.6
	Naive Transfer	40.9	45.3	32.0	41.2	39.9
	Random	41.1	45.2	32.5	41.0	40.0
	ATGC	42.9	45.1	33.6	42.3	41.0
	Oracle	45.2	49.1	36.6	44.8	44.0
	Domain Generalization Evaluation
	Naive Transfer (DG)	46.1	47.2	33.3	47.0	43.4
	Random (DG)	45.9	48.0	33.3	47.8	43.8
	ATGC (DG)	48.0	49.4	34.3	48.5	45.1
CLIP-DINOiser	CLIP-DINOiser [62]	32.0	31.6	13.7	30.4	26.9
	CoRTE [18]	24.9	24.5	10.8	26.6	21.7
	Naive Transfer	25.1	21.6	11.1	22.4	20.1
	Random	26.1	24.0	12.5	23.4	21.5
	ATGC	27.7	27.9	17.2	27.0	25.0
	Oracle	34.6	38.5	23.1	34.7	32.7
	Domain Generalization Evaluation
	Naive Transfer (DG)	29.4	31.4	18.2	31.7	27.7
	Random (DG)	32.0	32.6	19.7	32.9	29.3
	ATGC (DG)	36.8	37.2	23.6	39.7	34.3

Table 5. Figure 10 : Qualitative comparison of different methods. From left to right: input RGB image, predictions from CoRTE  [ 18 ] , Naive Transfer, our method ATGC (Ours), and ground-truth segmentation maps. Our method shows improved segmentation quality of small objects, such as “poles” (highlighted with white rectangular boxes), compared to the other approaches.

Input	CoRTe [18]	Naive Transfer	ATGC (Ours)	Ground truth

Equations

$${\mathcal{L}}_{\text{KD}}=\alpha \text{KL}(p_t\Vert p_s)+(1-\alpha ){\mathcal{L}}_{\text{CE}}(p_s,Y),$$

$$\mathbf{S}(A_j)=-\sum _{u,v}{A}_j(u,v)\log A_j(u,v),$$

$$s^{\ast }=\arg \underset{s_j\in \mathcal{S}}{\min }\mathbf{S}(A_j)$$

Full text

1 Introduction

The paradigm of pre-training neural networks on large datasets [57, 13, 4] has produced powerful “Foundation Models” (FMs) [6] with broad applicability across numerous domains [19, 7, 58, 35, 40, 68]. Many of these FMs are commercialized as Artificial Intelligence as a Service (AIaaS) and accessed through APIs, such as GPT-4 [1] and Gemini [15]. While AIaaS simplifies infrastructure management, it presents users with significant challenges, including substantial costs at scale [11], network latency, and potential downtime.

A viable alternative is knowledge distillation (KD) [22, 56], where knowledge from a generalist teacher is distilled into a compact student model. This creates a local “expert”111Our usage of the term “expert” differs from the KD literature, where expert model refers to the teacher, but closer to mixture of experts [31]. model that is cost-effective by eliminating calls to external APIs and can be finetuned for downstream tasks. Such models can be created through standard KD if the teacher is open-weight [26], or via black-box adaptation for closed-weight teachers accessible only through an API [18]. We focus on the latter as it is both more challenging since it excludes techniques like weight initialization or intermediate layer distillation [65, 32], and more realistic given the growing prevalence of commercial API-based foundation models.

Existing black-box adaptation methods often leverage pseudo-labels (PLs) from an API model while assuming access to its output logits [70, 18]. While these approaches are termed “black-box”, we believe that “gray-box” (see Fig. 1(a)) would be a more appropriate description, given the access to logits. The “gray-box” assumption is a notable limitation, as commercial APIs increasingly withhold logits to protect intellectual property. This makes truly black-box approaches, which have seen recent interest for Large Language Models (LLMs) [24, 39, 45, 55], more relevant.

With a similar goal in mind, in this work, we formalize and study the problem of distilling a generalist black-box API-based foundation model into an expert model, but for the task of semantic segmentation. In this new setting, what we call Black-Box Distillation (B2D), we assume that the API model is truly black-box and serves an open-vocabulary semantic segmentation model, which we can prompt using natural language [63, 33] to extract pixel-level semantic labels. As shown in Fig. 1(b), we use these PLs from the teacher to train the student model. The newly introduced B2D is more challenging because: (i) strict black-box assumption limits the scope of PL filtering, and (ii) the resolution of the image crop presented to the API governs the quality of PLs (see Fig. 2). Since the pre-training data and their statistics are never disclosed for strictly-private API models, querying the API with optimal resolution to obtain high-quality PL is non-trivial.

To tackle B2D, we propose a student-teacher self-training framework designed to obtain higher quality PLs from the API model. In the absence of confidence thresholding, we employ a simple strategy of prompt ensembling, where we prompt the API using an averaged embedding derived from various synonyms of a noun (e.g. , “train”, “tram”, “locomotive”). To determine the optimal image crop resolution for the API, we introduce a novel module, ATtention-Guided sCaler (ATGC), that dynamically identifies the most suitable crop resolution. Our approach leverages the common finding in semantic segmentation literature that different object classes are best segmented at different image resolutions [29, 12]. For instance, “distant traffic lights” are better segmented with zoomed-in crops, while “closer trucks” benefit from the larger context provided by zoomed-out crops. We objectively assess the goodness of a crop resolution using the attention maps of the student model, which we score using Shannon entropy. A lower entropy value indicates a more focused map, suggesting that the resolution is well-suited to the objects within the crop. To ensure meaningful attention maps, we initialize the student’s encoder with an open-weight model (e.g. , DINOv2 [46]) and keep it frozen during training. ATGC then selects the crop yielding the lowest entropy score and feeds it to the API model to obtain the PLs. Our contributions can be summarized as follows:

• •

We formalize Black-Box Distillation (B2D), a novel setting which assumes no access to API logits in contrast to previous “gray-box” approaches, better reflecting the reality of commercial AI services.

• •

We propose ATtention-Guided sCaler (ATGC), a novel model which uses DINOv2 attention maps to dynamically select optimal image resolutions for API queries, yielding higher-quality pseudo-labels.

We benchmark our framework on Cityscapes [16] and ACDC [53], demonstrating superior performance over state-of-the-art methods. Our findings highlight both the challenges and potential of specializing models from truly black-box APIs.

2 Related work

We review the literature related to B2D, but limit our discussion to works that tackle the downstream task of semantic segmentation, which is the scope of this work.

Black-box adaptation refers to adapting from a pretrained model to a new domain or task without accessing or modifying its internal parameters, treating the model as a “black-box”. In semantic segmentation, the precursors to black-box adaptation include unsupervised domain adaptation (UDA), where both the source dataset and source model are available during adaptation [26, 25, 27, 9, 47], and source-free UDA (SFUDA) [43, 20, 36], which assumes access to only the source model, but not the source data. Both of these settings can be considered as “white-box”, as they allow access to the weights of the source model. Although white-box adaptation offers maximum flexibility and control, it also introduces several challenges such as security risks [59, 38] or weights being unavailable due to commercialization [1].

In contrast, black-box UDA (B2UDA) discards the assumption of access to the source pretrained model’s weights, and instead treats it as a black box, accessible only via an API. B2UDA has mainly been studied in the context of image classification [70, 38, 71], and has only recently been investigated for semantic segmentation [18]. In detail, Cuttano et al.  [18] proposed a mechanism to extract reliable PLs from the black-box model through confidence-based filtering. A downside of this approach is that it involves accessing the logits from the API, which may not always be guaranteed, and thus can more accurately be considered as a “gray-box” approach [55]. Differently, in our proposed B2D setting, we adhere to the true definition of a black-box setting – neither the source model’s weights nor the output logits are available. The key differences among the various settings have been summarized in Tab. 1.

Distilling from foundation models. Foundation models (FMs) are large-scale general-purpose models [48, 35, 46, 3] trained on massive, diverse datasets, designed to serve as a solid “foundation” or starting point for various downstream tasks [3]. While the FMs offer many opportunities [6], their large memory footprint hinder deployment on resource-constrained devices [50]. To balance performance and efficiency, Knowledge Distillation (KD) [23] has been adopted as a go-to technique to train a smaller student model using a FM as a teacher. At its core, KD-based approaches assume that the teacher is open-weight (i.e., white-box), and thus distill knowledge either using the teacher’s output logits [44, 5, 46], internal feature representations [66, 67, 69, 60] or their combination [42, 56]. Very recently, this idea has been extended to distill knowledge from multiple FMs into a single student model [49, 54, 21], and has demonstrated strong performance in multiple tasks, including semantic segmentation. Despite the progress enabled by KD, the white-box (or gray-box) assumption makes it inapplicable in truly restrictive APIs. The B2D setting proposed in this work is challenging as it allows only a “hard” variant of KD, through the use of one-hot PLs.

Unlike B2UDA [18], which allows adaptation to a fixed set of categories (or vocabulary), distilling from open-vocabulary FMs is more appealing, as joint training with vision and language modalities allow for adaptation beyond a fixed vocabulary through flexible natural language prompts (e.g. , a photo of a [CLASS] [48, 33]), facilitating deployment of the student model to any domain/task of interest. However, this advantage may get eclipsed in certain scenarios when the target domain (e.g. , medical images) exhibits a significant domain gap with respect to the FM pre-training dataset. This will lead to very noisy PLs [10], especially since the pre-training dataset is often not disclosed by API providers. To balance flexibility and performance, we propose two strategies to extract more reliable PLs, while staying within the scope of the true black-box setting.

3 Problem formulation and preliminaries

The goal of Black-Box Distillation (B2D) is to transfer knowledge from a generalist black-box model, accessed via an API, to a local model. Next, we formulate the problem and introduce some preliminaries.

Problem setup. We are interested in training a local model for the task of semantic segmentation. We assume that we have access to an API that serves an open-vocabulary segmentation model ${\bm{f}}_{\text{API}}$, which can produce one-hot semantic segmentation map when presented with an image $X\in\mathbb{R}^{3\times h\times w}$. We denote the one-hot segmentation map as $\mathcal{M}=\{0,1\}^{\mid\mathcal{C}\mid\times h\times w}$, where $\mathcal{C}=\{c_{1},c_{2},\cdots,c_{K}\}$ is the set of $K$ class names (or vocabulary). Note, $\mathcal{C}$ is not a set of class indices, but strings (e.g. , “car”, “pedestrian”), and is provided by the user.

Given a dataset of unlabelled images from the target domain, $\mathcal{D}=\{X_{i}\}^{n}_{i=1}$, our goal is to learn a local segmentation model $\mathcal{F}_{\theta}\colon X\mapsto[0,1]^{\mid\mathcal{C}\mid\times h\times w}$ that infers per-pixel class probabilities. The only supervision available for learning the parameters $\theta$ of $\mathcal{F}_{\theta}$, is the API model ${\bm{f}}_{\text{API}}$.

Challenges in B2D. While black-box adaptation is challenging in itself, B2D is more challenging than B2UDA [18] in two key aspects: (i) B2UDA assumes that the API model was trained on a niche labelled source dataset (e.g. , GTA [51]) that has a relatively small domain gap with the target domain (e.g. , Cityscapes). We argue that this strong assumption does not reflect real-world domain gaps, and as a result, the adaptation strategy may not generalize beyond academic benchmarks. Instead, in B2D, we propose to leverage an open-vocabulary segmentation model as the API. This choice can be viewed as a “double-edged sword”, since, unlike B2UDA, it offers the flexibility to distill information for any vocabulary, but at the same time may widen the domain gap when the target domain is very different from the (unknown) pre-training data distribution. (ii) Our setting operates under the stricter assumption that the API provides only one-hot segmentation maps ($\mathcal{M}$). This constraint is motivated by the practical limitations of many real-world APIs, which often return only final predictions. It was shown in [23] that KD derives its strength from the use of “soft targets” alongside hard labels, preventing overfitting. Thus, the absence of privileged information in B2D prevents the use of effective techniques, such as confidence-based pseudo-labelling [18] or soft-dillation [56], to mitigate the impact of noisy PLs, making it a more challenging task.

Preliminaries. Knowledge distillation [23] consists in training a smaller (student) model to mimic the behaviour of a larger (teacher) model. The idea is to compress the knowledge of the teacher into a lightweight, faster model, without losing much performance. This is achieved through a distillation loss $\mathcal{L}_{\text{KD}}$ that is a sum of KL-divergence loss, computed between the teacher’s $p_{t}$, and the student’s $p_{s}$ predicted probability distributions, and a cross-entropy loss between $p_{s}$ and the true hard label $Y$ of the sample:

[TABLE]

where $\alpha$ is a hyperparameter. In B2D, we do not have access to $p_{t}$, preventing us from directly using the formulation in Eq. (1) to train $\mathcal{F}_{\theta}$.

4 Methods

Motivation. In this work, we argue that the quality of the segmentation maps $\mathcal{M}$ derived from the API model ${\bm{f}}_{\text{API}}$ is influenced by the resolution of the input image. To verify this, we conducted a preliminary study in which we varied the resolution (or scale222An image is scaled by an arbitrary factor, and crops of the size expected by the network are extracted using a sliding window. We use scale and resolution interchangeably.) of Cityscapes images and fed them to an API model that serves an open-vocabulary segmentation model (SAN [64]). We prompted the API using the class names of Cityscapes and report in Fig. 3 the class-wise intersection over union (IoU) performance for a selected few classes. From Fig. 3 we observe that IoU for a class varies across scales, and there is no single scale that produces the best IoU across classes. For example, classes such as “road” and “bus” that cover a significant area of the scene require larger contextual information and are hence better segmented at zoomed-out resolution (or scale factors $<1$). Conversely, some classes that are typically distant and cover a tiny area of the scene (e.g. , “traffic sign” and “pole”) show peak performance when zoomed in (or higher resolution, with scale factors $>1$). This happens because small or distant objects do not require very long-range information to be well segmented. More qualitative and quantitative examples are provided in  Secs. C.2 and C.4

This phenomenon called the curse of resolution, also noted in [29, 72], is exacerbated when using an open-vocabulary segmentation model as the API, since the predominant resolution of its training images are likely to be different from the target dataset. The high variability in performance (e.g. , IoU for the class “road” drops from $\sim 70\%$ to $\sim 20\%$ when scale is varied) clearly indicates that identifying an optimal resolution can significantly improve the quality of PLs, ultimately leading to a high-performing local segmentation model. This insight forms the foundation of our proposed method. Next, we give a brief overview of our method and then describe the novel component in detail.

Method overview. Our goal is to train a local segmentation model that becomes an “expert” at segmenting target images by using supervision from a “generalist” black-box API model, which in our case is an open-vocabulary segmentation model. We adopt a student-teacher framework [28, 18], where the API model acts as a teacher and the local segmentation model acts as a student. The teacher supervises the student model by providing one-hot PLs to train its parameters. However, differently from [18] we do not employ an exponential moving average (EMA) teacher derived from the student. Instead, we simply focus on extracting higher quality PLs to supervise the student.

As shown in Fig. 4, at the core of our student-teacher framework is a novel module, ATGC (ATtention-Guided sCaler). We designed ATGC to identify (or mine) the optimal scale that produces less noisy PLs than those obtained through naive approaches such as random scaling. ATGC exploits the attention maps of the student as a proxy to determine whether a given scale is suitable for segmenting the objects contained in a crop. We find that well-defined and crisp attention maps strongly correlate with superior dense prediction performance. Through a scoring mechanism, we pick the scale corresponding to the best attention map, and use that scale to preprocess the image crop and query the API to get PLs for supervision.

4.1 ATtention-Guided sCaler (ATGC)

The proposed ATGC is a plug-and-play module, applied on a given image crop before being fed to the API, whose goal is to choose the optimal scale. While a naive approach would be to perform random scaling (as in random resized crop data augmentation), we argue that it has two downsides: (i) as discussed before, not all objects are well-segmented at every scale, and this will contribute to noisier PLs, which will be detrimental for the student when trained for a longer period [41]; and (ii) it incurs unnecessary API calls, increasing cost due to inefficient queries. Therefore, ATGC facilitates learning in terms of both stable training and budget.

We design ATGC drawing inspirations from self-supervised learning methods [46, 2], which have shown that Vision Transformers (ViTs) trained with self-supervision can yield strong localization properties. To exploit this property of ViTs we employ a pre-trained DINOv2 [46] as the student encoder. In particular, we leverage the attention maps between the [CLS] token and the patch tokens from the final layer of DINOv2, and consider them a proxy indicator of the suitability of a given image crop for pseudo-labelling. Next, we describe how we extract the attention maps, score them, and use the scores to find the optimal scale. We subsequently discuss why our design philosophy of utilizing attention maps works.

Extracting Attention Maps. Given an image crop $\mbox{\bf X}\sim\mathcal{D}$, we scale it to ${\mathbb{N}}$ different resolutions using a set of scale factors defining the support $\mathcal{S}=\{{\textnormal{s}}_{j}\mid j\in\mathbb{N},{\textnormal{s}}_{min}\leq{\textnormal{s}}_{j}\leq{\textnormal{s}}_{max}\}$, where ${\textnormal{s}}_{max}$, ${\textnormal{s}}_{min}\in{\mathbb{R}}^{+}$ are the maximum and minimum values. We feed each scaled image $\mbox{\bf X}_{j}={\mathcal{T}}_{{\textnormal{s}}_{j}}(\mbox{\bf X})$ (where ${\mathcal{T}}_{{\textnormal{s}}_{j}}:\mbox{\bf X}\in{\mathbb{R}}^{3\times h\times w}\mapsto\mbox{\bf X}\in{\mathbb{R}}^{3\times h\cdot{\textnormal{s}}_{j}\times w\cdot{\textnormal{s}}_{j}}$ represents a transformation by scaling) to the DINOv2 feature extractor ${\mathcal{E}}(\cdot)$ to get the attention maps from the final transformer block. Due to multi-headed attention, we get as many attention maps as the number of heads, which we average to get a single attention map per scale $s_{j}$ as $A_{j}\in{\mathbb{R}}^{h_{j}\times w_{j}}$, where $h_{j},w_{j}$ are the scaled spatial dimensions.

Scoring Attention Maps. To find the optimal scale $s^{*}$ for a given image crop $X$, we score the attention map $A_{j}$ with a metric $\mathbf{S}(\cdot)$ that can quantify the objectness of the crop. To this end, we choose this metric to be Shannon entropy:

[TABLE]

where the attention map $A_{j}$ is normalized to form a probability distribution over spatial locations $(u,v)$. We then select the scale $s_{j}$ that produces the lowest entropy attention map, as the optimal scale $s^{*}$:

[TABLE]

The rationale is that an attention map with lower entropy exhibits peaked activations, signaling the presence of detected objects. In contrast, a higher entropy indicates a more diffuse attention map, suggesting that the model did not identify any particular semantic object. In Fig. 5 we visualize the attention-maps and the PLs and observe that a lower entropy attention map corresponds to better PL.

Discussion. A natural question arises: Why should the student encoder serve as a good proxy for a black-box API model? At first glance, this may seem counterintuitive, given that the student encoder and the API model are two separate and independently trained entities. Viewing through the lens of the Platonic Representation Hypothesis [30] – which suggests that as models scale, they tend to converge toward learning the same underlying features – we argue that both the API model and the student encoder (in our case, a pre-trained DINOv2), being trained on large-scale datasets, may develop internal representations that approximate the same statistical structures in representation space. Thus, both the models will react similarly to the same input, a property that we exploit in the absence of the access to the API’s internal parameters.

4.2 Learning local segmentation model

Pseudo-labelling. We use the optimal scale $s^{*}$ from Eq. (3) to scale the input image crop as $X^{*}={\mathcal{T}}_{{\textnormal{s}}^{*}}(X)$. We then feed $X^{*}$ to the API model to get the PL, denoted by \hat{Y}={\bm{f}}_{\text{API}}\big{(}X^{*},\mathcal{C}\big{)}, where the vocabulary $\mathcal{C}$ are converted into standard text prompts [64, 14], such as “a photo of a [CLASS]”. More details are provided in Appendix B

Although ATGC ensures better PL quality than using random scales, some pixels can still have erroneous PLs, causing the student to overfit on noisy PLs. To further improve the quality of supervision, we perform another round of PL-filtering at the pixel-level. In detail, we compute the agreement in class predictions (or PL accuracy) between the API and the student model. The PLs provided by the API for a given image are used to supervise the student only when the PL accuracy exceeds a predefined threshold $\tau$. We study the effect of varying $\tau$ in Sec. C.1.

Student training. We train the student model following Eq. (1) by setting $\alpha=0$ and replacing $Y$ with $\hat{Y}$, which is equivalent to a cross-entropy loss with hard PLs. To preserve its rich internal representation, we train only the linear decoder and the last transformer block of the student model. Further implementation details are provided in Appendix B.

5 Experiments

5.1 Experimental setup

Implementation. We conduct experiments on two unlabelled datasets following standard practices [61, 29, 28]. We use Cityscapes [16] with $2975$ training and $500$ validation images at $2048\times 1024$ resolution. We also use ACDC [53] with $400$ training and $200$ validation images per weather condition (Night, Snow, Fog, Rain) at $1920\times 1080$ resolution. We employ two open-vocabulary segmentation models as black-box models: SAN [64] with a ViT-L/14 backbone trained on COCO-Stuff [8], and CLIP-DINOiser, a training-free CLIP-based method with a ViT-B/16 backbone that has not been trained on segmentation data. For the local model, we use a DINOv2-S/14 encoder with registers [46] and a simple linear decoder.. We use thirteen different scale factors $\mathcal{S}$ = {$0.25$, $0.28$, $0.34$, $0.38$, $0.44$, $0.47$, $0.5$, $0.75$, $1.0$, $1.25$, $1.5$, $1.75$, $2.0$} for resolution selection. Additional training details are provided in Appendix B.

**Evaluation Protocol and Baselines. ** We evaluate performance using mean Intersection over Union (mIoU) across 19 classes for all datasets. Since this work introduces a new B2D setting, no established methods exist for direct comparison. We therefore compare ATGC against CoRTE [18], the closest available approach. Since the official CoRTE code was not available, we reimplemented it for our experiments. We also include several baselines: (i) “Naive Transfer” trains the student model using pseudo-labels from the API model at the default scale (${\textnormal{s}}_{j}\!=\!1$); (ii) “Average” uses the average value of attention maps for scoring, with higher values preferred; (iii) “Random” randomly selects a pseudo-label corresponding to one of the available scales for each image crop; (iv) “Oracle” uses ground truth to evaluate all pseudo-labels at different scales and selects the one with highest pixel accuracy for training, providing an upper bound for our method; (v) “Supervised” is trained with real ground truth labels. Note that “Oracle” differs from “Supervised” : Oracle uses the best API-generated pseudo-labels (selected using ground truth), while “Supervised” uses actual ground truth labels for training.

5.2 Main results

Specialization to Cityscapes. We evaluate ATGC using two different open-vocabulary models as APIs: SAN [64] and CLIP-DINOiser [62]. Tabs. 2 and 3 show the results for both settings, respectively. For SAN as the API model, our method achieves 50.1 mIoU, outperforming all baselines including CoRTE† (48.4), Naive Transfer (48.8), Random (48.6), and Average (44.6). With CLIP-DINOiser as the API, our method reaches 37.9 mIoU, again surpassing CoRTE† (34.5), Naive Transfer (34.3), Average (34.1), and Random (35.5). Notably, our ATGC operates without access to API logits, while CoRTE*†* requires them, making our setting more realistic. Although our method falls short of Oracle performance (53.3 and 40.9 mIoU), it consistently outperforms existing approaches. The Oracle represents an upper bound achieved by always selecting optimal pseudo-labels, while the Supervised baseline (64.7 mIoU) shows potential performance when training with ground truth labels. These results validate our scale selection strategy’s effectiveness for black-box model distillation.

**Specialization to ACDC. ** Tab. 4 demonstrates ATGC’s superior performance across both evaluation settings and black-box API models (SAN and CLIP-DINOiser). In the standard setting where models train on unlabelled ACDC images, ATGC achieves 41.0 mIoU with SAN and 25.0 mIoU with CLIP-DINOiser, consistently outperforming all baselines. Remarkably, in the domain generalization scenario, models trained on unlabelled Cityscapes and evaluated on ACDC achieve superior performance compared to direct ACDC training, a phenomenon we examine in
3 .

5.3 In-depth analysis

1

Does ATGC always find the “optimal” scales?. To investigate this, we perform a grid search over several scales, where each scale is used to train a separate student model for the entire training duration. We then compare the performance of these fixed-scale models against ATGC. As reported in Fig. 6, we observe that for the “sky” class, the performance of a student trained using PLs from a fixed scale of $\times 1.5$ surpasses that of ATGC. Similarly, class “bicycle” benefits from scale $\times 1.25$, a scale that ATGC fails to always find. In contrast, for certain classes such as “terrain” and “truck”, ATGC achieves comparable or even superior results compared with the grid search scales. These results indicate that ATGC does not universally find the optimal scales for all the classes, which could be attributed to the presence of multiple competing classes within an image crop. However, ATGC is more compute-efficient and grid search is infeasible in practice due to the lack of any labelled data.

2

How does the API model’s training data affect performance?. The quality of pseudo-labels, and thus student performance, is greatly dependent on the domain gap between the API model’s training data and the target domain [52]. This is evidenced by the performance gap between the original CoRTE results (55.5 mIoU on Cityscapes) [18], which used an API model trained on the closely-aligned GTA dataset, and our CoRTE*†* baseline using SAN as an open-vocabulary API (48.4 in Tab. 2). This discrepancy arises because Cityscapes shares more similarities in scene layout and semantic classes with GTA than with COCO-Stuff, on which SAN was trained. The performance drop is even more pronounced with CLIP-DINOiser as the API, where CoRTE*†*’s performance decreases to 34.5 in Tab. 3, as it is a training-free model that has not been exposed to any segmentation dataset. We argue that achieving high performance as in CoRTE’s original setting can be illusory, as finding a target-aligned API model is often difficult in practice, particularly since the training data of commercial APIs is typically not disclosed. Therefore, assuming the API model is a generalist open-vocabulary model that covers a wide variety of target domains aligns better with real-world constraints.

3

Does dataset size play a role? We investigate why the DG results reported in Tab. 4 consistently outperform direct training on ACDC, an observation that is counterintuitive in the DG literature [17]. We believe that this difference is attributed to the smaller size of the ACDC dataset (2,975 in Cityscapes versus 1,600 in ACDC), as also shown in [37]. Other factors could be at play, such as weather-corrupted images in ACDC, which are typically long-tailed in open-vocabulary pre-training datasets, resulting in noisier PLs. Thus, we conclude that it is important to have a sufficiently large dataset for effective B2D, and caution must be exercised when working with target domains that are characterized by heavy corruptions or very long-tailed distributions.

6 Conclusions and Limitations

In this work, we introduced Black-Box Distillation in semantic segmentation (B2D), a realistic adaptation paradigm that operates with only one-hot predictions from open-vocabulary API models, removing impractical assumptions of existing black-box approaches. We identified the “curse of resolution”, whereby different object classes achieve optimal segmentation at different input scales, and proposed ATGC (ATtention-Guided sCaler), which leverages DINOv2 attention maps and entropy scoring to dynamically select optimal scales for black-box inference, demonstrating effectiveness in generating high-quality pseudo-labels.

Although our experiments show promising results, several limitations remain. As discussed in Sec. 5.3, ATGC operates at the image level rather than the class level and uses static prompts that do not adapt to varying object scales. This can produce noisier pseudo-labels when multiple classes benefiting from different scales appear in the same crop. Future work could address this by introducing class-level scaling or dynamic prompts. Additionally, our current approach does not account for API call budgets; efficiency could be improved by focusing on the most informative regions of an image to accelerate convergence.

Acknowledgements

This work has been supported by the French National Research Agency (ANR) with the ANR-20-CE23-0027 project and was granted access to the HPC resources of IDRIS under the allocations AD011013071 and AD011014921 made by GENCI. We also thank Nacereddine Laddaoui for his invaluable help with infrastructure.

Appendix A Pseudo-Code

We present the pseudo-code for ATGC in Algorithm 1 and Algorithm 2. Algorithm 1 details the offline construction of Attention Maps for the entire dataset, while Algorithm 2 describes the training procedure of ATGC that leverages these attention maps for resolution mining and pseudo-label generation. Note that in our implementation, we construct the attention maps offline, and do it only once for the entire dataset because it does not depend on the API model. Offline computation speeds up the training process, as multiple forward passes with varying crop resolutions are not needed to mine the optimal resolution.

Appendix B Implementation details

We train our model with the AdamW optimizer [34] using a weight decay of 0.05 and a learning rate of $10^{-5}$ with polynomial decay. We set training iterations to 10K for Cityscapes and 6K for ACDC, with a batch size of 4.

Following standard evaluation protocols, the student model is evaluated on resized validation images to maintain consistent inference conditions across datasets, specifically resizing Cityscapes images from 2048×1024 to 1024×512, ACDC images from 1920×1080 to 960×540 for all weather conditions (Night, Snow, Fog, Rain). All reported performance metrics reflect the student model’s capabilities after the full training schedule, ensuring fair comparisons with existing methods and maintaining computational efficiency.

Prompt engineering.

In this work, we have considered two open-vocabulary segmentation models –CLIP-DINOiser [62] and SAN [64] – as the API model. Following standard practice in open-vocabulary segmentation literature, we evaluate on 19 semantic classes from the Cityscapes dataset, using both primary class names and their synonyms to enhance model robustness (see class list below). For prompting the open-vocabulary models, we used the commonly used 80 templates from CLIP for zero-shot image classification, commonly referred to as ImageNet templates  [48]. We computed the average of the text embeddings from all templates to obtain the final text embedding for each class during inference. Classes: [ “road, street, highway”, “sidewalk, pavement, footpath”, “building, structure, house”, “wall, brick wall, stone wall”, “fence, barrier, hedge”, “pole, post, pillar”, “traffic light, red light, green light”, “traffic sign, stop sign, warning sign”, “vegetation, plants, trees”, “terrain, ground, grass”, “sky, air, clouds”, “person, pedestrian, people”, “rider, biker, driver”, “car, automobile, vehicle”, “truck, pickup, van”, “bus, shuttle, minibus”, “train, tram, locomotive”, “motorcycle, motorbike, scooter”, “bicycle, bike, cycle”, ]

Appendix C Experimental results

C.1 Pseudo-label filtering.

We ablate the pseudo-label filtering threshold $\tau\in[0.0,0.9]$ to study its impact on performance when using CLIP-DINOiser as the API model. As shown in Fig. 11, ATGC achieves optimal performance at $\tau=0.7$ with 37.9% mIoU.

Notably, ATGC demonstrates robust performance even when pseudo-label filtering is completely deactivated ($\tau\!=\!0.0$), achieving 37.2% mIoU with only a modest 0.7% decrease compared to the optimal threshold. This indicates that ATGC works effectively regardless of whether PL filtering is activated or not, showcasing the inherent quality of our scale-optimized pseudo-labels. Performance degrades more significantly when $\tau$ is too restrictive ($\tau\!=\!0.9$: 31.5%), as overly strict filtering removes valuable training signals, reducing effective pseudo-label coverage. The optimal threshold $\tau\!=\!0.7$ achieves the right balance between pseudo-label quality and quantity. For consistency, we report all main results using $\tau\!=\!0.7$ throughout our experiments.

C.2 Effect of varying resolutions

Figures 7 and 8 show the performance of SAN and CLIP-DINOiser, respectively, when used as API models, across various scaling factors applied to the initial image resolution ($1024\times 2048$) of the Cityscapes [16] validation set. The subplots illustrate the effect of image resolution on segmentation accuracy for each semantic class and the average mIoU. Each subplot highlights scaling factors ranging from $0.25$ to $2.0$, with blue star indicating the optimal resolution for each class.

The first subplot shows that the average mIoU across all classes peaks at the scaling factor of $1.0$, reflecting the best overall performance. However, peak performance varies across classes at different scale factors. For example, classes such as “person” and “motorcycle” perform optimally at scaling factors of $2.0$ and $1.5$, respectively. In contrast, “sidewalk” and “truck” are better segmented at scaling factors of $0.5$ and $0.75$, respectively.

In general, the results emphasize that resolution scaling plays a critical role in achieving optimal performance for each class. Optimal resolution choices vary across different object types, with some benefiting from detailed crops taken from high resolution base images and others from large context crops taken from low resolution base images. This suggests that a tailored resolution strategy can improve segmentation outcomes using the API model, highlighting the importance of careful resolution scaling rather than adopting a one-size-fits-all approach.

C.3 Qualitative comparison of methods

In Fig. 10, we present the qualitative comparison of different methods. We have compared our proposed ATGC with a B2UDA approach CoRTe [18], and the Naive Transfer baseline that directly distills from the API model without considering varying resolutions. From the figure we observe that for larger object classes, such as “car” and “bus”, and stuff classes, such as “road” and “vegetation” the segmentation quality is reasonably fair among all the baselines and our ATGC. However, when it comes to smaller objects, such as “poles” (highlighted with a white box in all the segmentation maps), the segmentation quality of our ATGC is much better. Furthermore, in row 3 of Fig. 10 we observe in the highlighted region of the segmentation map that CoRTe is worse at segmenting “sidewalk” that is farther in the scene, compared to the nearby “sidewalk”. Contrarily, both the Naive Transfer and ATGC is better at segmenting farther objects, with ATGC doing well on the farthest “pole” in the image. This observation underscores the need of obtaining pseudo-labels at the optimal crop resolution to reasonably segment objects that occupy a very small portion of the image.

C.4 Effectiveness of Attention Maps

Building upon the motivational experiment in Sec. 4 and Fig. 4 of the main paper, which demonstrated a positive correlation between attention map quality (computed between DINO [CLS] and patch tokens) and segmentation performance, we present additional qualitative validation in Fig. 12.

These examples confirm that the relationship between attention map entropy and pseudo-label (PL) quality is consistent across different scenarios. Specifically, attention maps with lower entropy display concentrated, peaked activations that correspond to better object localization and, consequently, higher-quality pseudo-labels. Conversely, higher entropy values indicate spatially diffuse attention patterns that correlate with poor-quality pseudo-labels. Our experimental analysis validates that Shannon entropy of attention maps serves as an effective and reliable proxy for identifying the optimal scale for pseudo-label generation, which forms the core mechanism our method uses to distill knowledge from the API model.

C.5 Failure cases

In Fig. 13, we report some failure modes of our ATGC, where we show that optimal crop resolution may not always result in the best pseudo-label predictions from the API model. In detail, for a given random crop, we visualize the prediction of the API model at the original crop resolution ($\bm{x}_{c}$) and the one with the optimal scale resolution ($\bm{x}_{c}^{*}$) as determined by the ATGC module. We observe from the figure that the quality of segmentation maps predicted by the API at the optimal scale is inferior when compared with the one obtained with the original crop resolution. Specifically, from Fig. 13(c) and (d) we observe that the optimal crop resolution introduces strange artifacts in the segmentation maps, e.g. , sharp discontinuities. Similar kind of observation holds for the other figures, where incorrect classes are predicted within other objects (e.g. , Fig. 13(a), where a “truck” pixels encroaches onto a “car” class pixels). This phenomenon occurs when extreme zoom-in operations (with high scale factors) cause attention maps to be computed over homogeneous image regions, leading to highly concentrated activations for a single class and consequently producing misleadingly low entropy values.

Despite these pathological predictions, we employ a PL-filtering technique (as discussed in Sec. 4 of the main paper) that uses the consistency between the API predictions and the student model predictions to avoid training on erroneous PLs. Since the student model processes the image crop at original resolution, it is less likely to make the same errors as the API model that always generates PLs at the optimal scale resolution (see Fig. 3 of the main paper). We compute the pixel-level agreement between the API pseudo-labels and the student predictions using pixel accuracy, and only use pseudo-labels for supervision when this agreement exceeds a predefined threshold $\tau$. This filtering mechanism ensures that pathological API predictions are discarded and do not negatively impact the student network training.

Appendix D Limitations and future works

In this work, we introduced the task of B2D, which distills knowledge from a generalist black-box model into a local segmentation model by mining optimal scales for querying the API. This approach represents a significant improvement over using default or random crop resolutions for pseudo-label generation. However, our current framework operates under the assumption that optimal scale selection is the primary determinant of pseudo-label quality. While our experiments validate that optimal scaling plays a crucial role in determining PL quality, several other factors may influence performance that remain unexplored, such as adaptive prompt engineering tailored to specific crop resolutions. Additionally, our framework lacks a systematic mechanism for selecting the most informative samples or image regions for API queries. This uniform treatment of all image regions is inefficient, particularly given the inherent class imbalance in driving datasets where “stuff” classes (road, sky, vegetation) occupy significantly larger pixel areas than “things” classes (vehicles, pedestrians, traffic signs).

As future work, we will explore other crucial factors that can further improve the quality of PLs and select most informative regions, thereby reducing the number of calls to the API, making black-box distillation from API calls more economically and computationally viable. Furthermore, we will also explore making API calls for pseudo-supervision while ensuring the privacy of sensitive local client data.