BAOD: Budget-Aware Object Detection

TL;DR

This paper introduces BAOD, a budget-aware object detection framework that optimally selects images and annotations to train detectors efficiently under annotation constraints, achieving near-supervised performance with less labeling effort.

Contribution

It proposes a hybrid supervised learning approach combined with an optimization-based data selection method for budget-aware object detection, outperforming other sampling techniques.

Findings

Optimization-based selection outperforms random and active learning methods.

Achieves comparable performance to fully supervised detectors with 12.8% less annotation.

Surpasses fully supervised performance when using the entire annotation budget.

Abstract

We study the problem of object detection from a novel perspective in which annotation budget constraints are taken into consideration, appropriately coined Budget Aware Object Detection (BAOD). When provided with a fixed budget, we propose a strategy for building a diverse and informative dataset that can be used to optimally train a robust detector. We investigate both optimization and learning-based methods to sample which images to annotate and what type of annotation (strongly or weakly supervised) to annotate them with. We adopt a hybrid supervised learning framework to train the object detector from both these types of annotation. We conduct a comprehensive empirical study showing that a handcrafted optimization method outperforms other selection techniques including random sampling, uncertainty sampling and active learning. By combining an optimal image/annotation selection scheme with hybrid supervised learning to solve the BAOD problem, we show that one can achieve the performance of a strongly supervised detector on PASCAL-VOC 2007 while saving 12.8% of its original annotation budget. Furthermore, when $100\%$ of the budget is used, it surpasses this performance by 2.0 mAP percentage points.

Tables

Table 1. Table 1 : Budget-Average mAP using fully and hybrid training pipelines with random and uncertainty selection. Uncertainty sampling is always better than random sampling selection, and hybrid training is always better than FSOD.

Selection Method	FSOD		Hybrid
Sampling Function	RS	US	RS	US
Low Budget Range	52.5	53.1	56.4	55.1
Mid Budget Range	62.5	63.6	64.5	65.8
High Budget Range	67.9	68.7	68.7	69.3

Table 2. Table 2 : Budget-Average mAP using simple hybrid training and optimization methods. US based optimization is slightly better than LAL one. The optimization methods perform better than the simple hybrid random selection and uncertainty selection methods in the three budget ranges.

Selection Method	Hybrid		Optimization
Sampling Function	RS	US	LAL	US (BAOD)
Low Budget Range	56.4	55.1	56.3	57.1
Mid Budget Range	64.5	65.8	65.9	66.0
High Budget Range	68.7	69.3	69.3	69.5

Table 3. Table 3 : Budget-Average mAP using a lower cost for strong annotations. If we assume the weak and strong annotation costs are more close (7 seconds and 1.5 seconds), US based optimization (BAOD) still performs better than the simple hybrid random selection and uncertainty selection methods in the three budget ranges.

Selection Method	FSOD	Hybrid	Optimization
Sampling Function	RS	US	US (BAOD)
Low Budget Range	53.1	52.3	54.2
Mid Budget Range	62.8	63.1	63.6
High Budget Range	68.3	68.6	69.3

Table 4. Table 4 : Simulation of a larger unlabeled image pool. With 87.2 % percent 87.2 87.2\% budget, BAOD achieves the same performance as a detector trained on fully annotated VOC07. If the budget equals to the total budget of VOC07, BAOD achieves 2 % percent 2 2\% mAP improvement over FSOD with the same budget. Further annotating all the images can only improve 3.4 % percent 3.4 3.4\% mAP.

Train Set Pool	VOC07	VOC0712
Max Budget*	100%	87.2%	100%	330%
Final mAP	0.710	0.710	0.730	0.764

Table 5. Table 5 : Budget-Average mAP using Largest and Smallest Uncertainty Scores. When the budget is low (10%-30%), collecting images with small uncertainty score (shown as SUS) is more effective. However, when the dataset size is large enough, choosing large uncertainty scores (shown as LUS) is preferable.

Selection Method	FSOD		RL	Opt.
Sampling Function	LUS	SUS		US (BAOD)
Low Budget Range	52.3	53.1	56.6	57.1
Mid Budget Range	63.1	62.8	64.1	66.0
High Budget Range	68.6	67.7	68.5	69.5

Table 6. Table 6 : The mapping from twenty categories in PASCAL VOC to Easy, Medium and Hard classes . This mapping is based on the final mAP of each category.

Class	Easy								Medium						Hard
mAP∗	88.9	88.4	88.1	87.6	86.6	86.3	85.9	84.3	82.3	82.0	80.4	80.1	78.9	77.7	75.8	65.7	68.9	70.8	63.6	53.6
Category	cat	car	bus	horse	train	cow	dog	areo	person	bike	sheep	mbike	tv	bird	sofa	table	boat	bottle	chair	plant
* data is from the last setup of Table 6 in the report [58].

Table 7. Table 7 : Table Representation for Raw Experiment Data

			Budget Percentages													Full
	H∗	CR∗∗	10.80	15.79	20.77	25.76	30.75	35.74	40.73	45.72	50.71	55.70	60.69	65.68	80.70	100
RS	✗	$H$	0.414	0.505	0.547	0.572	0.597	0.622	0.630	0.643	0.652	-	-	-	0.692	0.71
US	✗	$H$	0.430	0.506	0.519	0.552	0.599	0.621	0.623	0.651	0.657	-	-	-	0.695	0.71
SUS	✗	$H$	0.397	0.502	0.546	0.575	0.602	0.62	0.633	0.636	0.648	-	-	-	0.682	0.71
LAL	✗	$H$	0.429	0.495	0.545	0.569	0.58	0.606	0.634	0.64	0.644	-	-	-	0.692	0.71
RS	✓	$H$	0.408	0.562	0.576	0.608	0.612	0.637	0.651	0.655	0.663	-	-	-	-	0.71
US	✓	$H$	0.433	0.462	0.586	0.62	0.638	0.647	0.658	0.672	0.676	-	-	-	-	0.71
OPT-US	✓	$H$	0.433	0.529	0.594	0.624	0.643	0.646	0.665	0.668	0.679	0.689	0.687	0.691	-	0.71
OPT-LAL	✓	$H$	0.389	0.529	0.59	0.62	0.638	0.645	0.661	0.674	0.677	-	-	-	-	0.71
US	✓	$L$	0.412	0.498	0.509	0.527	0.573	0.613	0.626	0.653	0.664	0.675	0.673	-	-	0.71
RS	✓	$L$	0.412	0.492	0.519	0.517	0.539	0.587	0.614	0.633	0.641	0.662	0.664	-	-	0.71
OPT	✓	$L$	0.42	0.518	0.548	0.586	0.609	0.62	0.64	0.648	0.665	0.67	0.683	0.689	-	0.71
RS	✓	$M$	0.443	0.554	0.576	0.6	0.62	0.62	0.654	0.652	0.655	0.672	0.675	0.689	-	0.71
OPT	✓	$M$	0.433	0.544	0.582	0.608	0.632	0.643	0.652	0.664	0.677	0.684	0.687	0.693	-	0.71
Budget	-	-	10.8	20.8	30.8	40.8	50.8	60.8	70.8	80.8	90.8	92.8	96.8	-	-	-
VOC0712	✓	$M$	0.356	0.566	0.611	0.641	0.674	0.672	0.696	0.701	0.715	0.717	0.72	-	-	0.73
Budget	-	-	11.09	13.62	18.1	22.58	27.06	31.53	36.15	40.49	44.97	49.45	53.93	-	-	-
RL	✓	$H$	0.433	0.525	0.561	0.579	0.603	0.615	0.634	0.645	0.648	0.657	0.664	-	-	0.71
* H for Hybrid Learning
** CR for cost ratio of strong and weak annotations

Equations

$$s_k=\frac{1}{M}\sum _{i=1}^M\sum _{p\in {\mathbf{p}}_i}-p\log (p)$$

$${\delta }_t=f_1(P_t{)}^{{}^{\top }}{\mathbf{x}}_1+f_2(P_t{)}^{{}^{\top }}{\mathbf{x}}_2+f_3(P_t{)}^{{}^{\top }}{\mathbf{x}}_3$$

$${\delta }_t\approx {\mathbf{s}}^{\top }({\mathbf{x}}_1+{\mathbf{x}}_3)+(\mu 1-\mathbf{s}{)}^{\top }{\mathbf{x}}_2$$

$$\underset{{\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3\in \{0,1{\}}^N}{\max }{\mathbf{s}}^{\top }({\mathbf{x}}_1+{\mathbf{x}}_3)+(\mu 1-\mathbf{s}{)}^{\top }{\mathbf{x}}_2$$

$$\text{s.t.}\{\begin{array}{l}{\mathbf{x}}_3\le \psi \\ {\mathbf{x}}_1+{\mathbf{x}}_2\le 1-\psi \\ 1^{\top }(a{\mathbf{x}}_1+b{\mathbf{x}}_2+c{\mathbf{x}}_3)\le d\\ 1^{\top }(a{\mathbf{x}}_1+b{\mathbf{x}}_2+c{\mathbf{x}}_3)\ge d-a\end{array}$$

$${\delta }_t\approx {\mathbf{h}}_w^{\top }{\mathbf{x}}_2+{\mathbf{h}}_s^{\top }({\mathbf{x}}_1+{\mathbf{x}}_3)$$

$$\underset{\mathbf{y}\in \{0,1{\}}^M}{\max }{\mathbf{y}}^{\top }\mathbf{q}$$

$$\text{s.t.}\{\begin{array}{l}{\mathbf{y}}^{\top }\mathbf{A}(1-\mathbf{w})=0\\ \text{IoU}({\mathbf{P}}_i,{\mathbf{P}}_j)\le \alpha \forall y_i,y_j=1\\ 1\le \Vert \mathbf{y}{\Vert }_0\le \beta \end{array}$$

$$\begin{array}[]{rrclcl}\displaystyle\hat{\mathbf{x_{1}}},\hat{\mathbf{x_{2}}},\hat{\mathbf{x_{3}}}=\operatorname*{arg\,max}_{\mathbf{x_{1}},\mathbf{x_{2}},\mathbf{x_{3}}\in[0,1]^{N}}&\lx@intercol\mathbf{s}^{\top}(\mathbf{x_{1}}+\mathbf{x_{3}})+(\mathbf{\mu}-\mathbf{s})^{\top}\mathbf{x_{2}}\hfil\lx@intercol\\ \textrm{s.t.}&\displaystyle\mathbf{x_{3}}&\leq&\mathbf{\psi}\\ &\displaystyle\mathbf{x_{1}}+\mathbf{x_{2}}&\leq&\mathbf{1}-\mathbf{\psi}\\ &\displaystyle\mathbf{1}^{\top}(a\mathbf{x_{1}}+b\mathbf{x_{2}}+c\mathbf{x_{3}})&\leq&d\\ &\displaystyle\mathbf{1}^{\top}(a\mathbf{x_{1}}+b\mathbf{x_{2}}+c\mathbf{x_{3}})&\geq&d-a\\ \end{array}$$

$$\begin{array}[]{rcl}\mathbf{x_{1}}^{*}&=&1\{\hat{\mathbf{x_{1}}}>\epsilon_{1}\}\\ \mathbf{x_{2}}^{*}&=&1\{\hat{\mathbf{x_{2}}}>\epsilon_{2}\}\\ \mathbf{x_{3}}^{*}&=&1\{\hat{\mathbf{x_{3}}}>\epsilon_{3}\}\end{array}$$

$$\widehat{x_3}(k)\le 0⟹\widehat{x_3}(k)=0.$$

$$\widehat{x_1}(k)+\widehat{x_2}(k)\le 1⟹1-\widehat{x_2}(k)\ge \widehat{x_1}(k)⟹1\{1-\widehat{x_2}(k)<ϵ\}\le 1\{\widehat{x_1}(k)<ϵ\}$$

$$\begin{array}[]{rcccl}x_{3}^{*}(k)&=&1\{\hat{x_{3}}(k)>\epsilon\}=1\{0>\epsilon_{3}\}=0&=&\psi(k)\\ x_{1}^{*}(k)+x_{2}^{*}(k)&=&1\{\hat{x_{1}}(k)>\epsilon\}+1\{\hat{x_{2}}(k)>1-\epsilon\}\\ &=&1\{\hat{x_{1}}(k)>\epsilon\}+1\{1-\hat{x_{2}}(k)<\hat{x_{2}}(k)\}\\ &\leq&1\{\hat{x_{1}}(k)>\epsilon\}+1\{\hat{x_{1}}(k)<\hat{x_{2}}(k)\}\leq 1&=&1-\psi(k)\end{array}$$

$$\begin{array}[]{rclcrcl}\hat{x_{3}}(k)&\leq&1&\implies&\hat{x_{3}}(k)&=&1\\ \hat{x_{1}}(k)+\hat{x_{2}}(k)&\leq&0&\implies&\hat{x_{1}}(k),\hat{x_{2}}(k)&=&0\end{array}$$

$$\begin{array}[]{rrclcl}\displaystyle\min_{\mathbf{y}\in\{0,1\}^{M}}&\lx@intercol\hfil-\mathbf{y}^{\top}\mathbf{q}\hfil\lx@intercol\\ \textrm{s.t.}&\mathbf{y}^{\top}A(\mathbf{1}-\mathbf{w})&=&0\\ &\forall_{{y}_{i},{y}_{j}=1}IoU(P_{i},P_{j})&\leq&\alpha\\ &\displaystyle|\mathbf{y}|_{0}&\in&[1,\beta]\end{array}$$

Full text

1 Introduction

Object detection in images is a fundamental computer vision problem with applications in many tasks including face/pedestrian detection [44, 1, 21, 41, 12, 56], counting [6, 23], and visual search [10, 39]. Building a successful object detector encompasses three main dimensions: (1) the image dataset to be annotated for training the detector. A larger dataset allows for a more accurate detector, but the number of training images is limited by the annotation budget; (2) the annotation scheme used to label the training images. One could annotate either image-level labels (the categories of the objects are known but their locations are unknown, denoted weakly supervised annotation) or instance-level labels (both categories and locations are known, denoted strongly supervised annotation) [41, 35, 29, 40]; (3) the detection model. Most works on object detection fix the first two dimensions and only explore the third. In fact, they tend to focus on optimizing the detection model based on one or more training datasets that typically provide the same kind of annotations. In this paper, we fix the detection model and investigate solutions in the first two dimensions of the problem.

A large group of object detectors fall under the umbrella of Fully-Supervised Object Detection (FSOD) [45, 46, 15, 14, 11, 47, 18, 34]. It has been shown in recent years that these techniques can reach high detection performance, especially with the introduction of large datasets with strong annotations [50, 13, 35]. This requirement makes FSOD methods expensive and time consuming. In contrast, Weakly-Supervised Object Detection (WSOD) [4, 9, 24, 5, 61, 16, 32, 20, 48, 54, 22] aims at building object detectors from cheaper but less informative image-level or weak annotations.

In this paper, we propose a trade-off between dataset annotation cost and model precision in order to combine both weak and strong annotations and train an object detector with hybrid supervision. We put all the detectors on the same footing when different annotation schemes are available. Ideally, detectors should only be compared when they are trained using image datasets that offer the same amount of information (not necessarily the same number of images). Since this notion is difficult to define quantitatively, we take the training budget of a detector as a unifying surrogate measure. Here, we define budget as the effort, or cost, to annotate a dataset, thus combining the first two dimensions of the detection problem: dataset scale and annotation scheme. In fact, with a fixed budget and a set of unlabeled images, the number of images we can label depends highly on the annotation cost for each image. This cost varies significantly between image- and instance-level annotations. Typically, annotating a bounding box around an object in an image is significantly more expensive than simply annotating its category [41, 50]. Therefore, an FSOD method with the same budget as a WSOD method contains fewer images in its training set.

We explore strategies to build better object detector models when constrained with a training budget, a novel problem we coin budget-aware object detection (BAOD). As such, we focus on choosing the best images for training and how to annotate them. For this, we survey several selection methods to sequentially choose both the image and type of annotation following an active learning paradigm. Additionally, we propose a novel hybrid training procedure for object detectors that can use both strong (instance-level) and weak (image-level) annotations. Figure 1 shows that actively selecting images and their annotations to sequentially train a hybrid supervised detector outperforms its FSOD and WSOD counterparts, when the budget is larger than 20% of the budget needed to annotate VOC2007 trainval at the instance-level. Moreover, the yellow curve shows that we can reach an even higher mAP if we use the same budget of VOC07 to annotate images from both datasets VOC07 and VOC12.

Contributions.

(1) We propose the BAOD problem and present a new evaluation criterion Budget-Average mAP for object detection algorithms. This criterion takes into account both detection performance and budget. (2) We study several strategies (e.g. optimization and learning-based) to select both training images and their annotation scheme. (3) We propose a hybrid supervised learning strategy that combines category and localization information to train a robust detection model that handles both weak and strong annotations. (4) Following a BAOD approach (i.e. combining intelligent image and annotation scheme selection with hybrid supervised learning), we show that the mAP test performance on VOC07 can be improved by 2 percentage points for the same budget used to annotate the training set of VOC07 (by combining it with VOC12). We also show the opposite, i.e. that state-of-the-art test performance on VOC07 can be achieved, while saving $12.8\%$ of the budget used in strongly annotating its training set.

2 Related Work

2.1 Fully Supervised Object Detection (FSOD)

With the development of deep learning, many CNN based methods have been proposed to solve the FSOD problem, such as YOLO [46], SSD [37], RCNN [15], and its variants [47, 14, 11, 18, 34]. Although these methods achieve impressive detection results, providing them with large-scale instance-level (bounding-box) annotations for training is costly.

In this paper, we constrain the annotation budget and focus on how to reach the best detection performance within this budget by intelligently selecting between both weak and strong annotations. Su et al. [52] reported that it takes $26$ seconds to draw one bounding-box without quality control, and $42$ seconds with it. There has been recent progress in developing tools to further reduce annotation time (most recently to $7$ seconds) [40, 41, 29]. Our work investigates a complementary aspect of annotation to emphasize that some images do not need to be strongly annotated to achieve excellent performance. While our discussion of budget aware object detection only considers the labor cost for image annotation, we note that sequentially training detection models also consume computational resources. However, given the continual development of hardware acceleration and re-organization, which leads to steady decrease in their cost, manual annotation remains the more expensive component in the detection problem.

2.2 Weakly Supervised Object Detection (WSOD)

If there are no instance-level labels available at training time, a WSOD method can be trained. Most classical approaches cast WSOD as a Multiple Instance Learning (MIL) problem [51, 32, 20, 48, 4, 22]. Bilen et al.[5] were the first to utilize a deep CNN (denoted as the weakly supervised deep detection network or WSDDN) to solve this problem. WSDDN selects positive samples by multiplying the score of recognition and detection. Zhang et al.[59] proposed a simple yet effective post-processing step to mine pseudo ground truth bounding boxes used for iterative FSOD training.

We use weak supervision to improve pseudo object labels. These labels are generated from a previously trained detector, and are post-processed through a pseudo label mining process. The image-level labels help remove false predictions that correspond to a wrong object category.

2.3 Hybrid Supervised Learning

In our work, we study hybrid supervised learning for object detection. This type of learning exploits multiple types of supervision during training. Semi-supervised learning is a special case of this family, since it learns a model from a set of labeled and unlabeled data. Several works [33, 49, 7, 30, 43] try to solve the generic semi-supervised learning problem through teacher-student learning. They first train a teacher model from the strongly annotated subset, then the predictions obtained from the teacher model on the unlabeled data are used to regress a second model that is called the student model. Chéron et al. [8] also used several kinds of annotations to train an activity recognition model, and they revealed that strongly annotating every training sample is not necessary to achieve noteworthy localization results in the video domain.

In our case, since images can be labeled either weakly or strongly, a hybrid supervised dataset is always considered. Inspired by Rosenberg’s work [43], we train a teacher- student model to use the hybrid dataset. Our teacher detector is learned from strong annotations only, while the student detector is learned from both ground-truth and/or processed pseudo object labels.

2.4 Active Learning

In general, active learning is a sequential decision making process that iteratively selects the most useful examples an oracle should annotate and add to the labeled training set. It aims at training more accurate models with the minimum data required. This field has been widely studied in the context of image and video classification [42, 2, 28, 26, 60, 17], object and action localization [19, 29, 27, 40, 25], human pose estimation [36] and visual question answering [38]. A commonly used approach to selecting new training images is by means of their entropy score [55]. The intuition is that higher entropy examples attribute to more learning information. More recent research directly predicts the improvements of adding a new sample to the training set, and uses this measurement as a selection criteria [28]. As specified in previous work, active learning aims at maximizing performance while minimizing the human cost in labeling the training samples [42, 36, 19].

The BAOD problem can also be formulated as a sequential decision making process. We study a few well-known selection techniques in our new active learning pipeline, which contains two active processes: (1) select the next batch of training images to annotate and (2) decide the type of annotation for each selected image. Thus, we focus on the annotation sequence that can provide the most useful information to the detector. To the best of our knowledge, we are the first to use active learning and hybrid training to study object detection.

3 Budget Aware Object Detection (BAOD)

In this section, we explore several ways to sequentially create an object detection dataset with a fixed budget, which includes both image-level and instance-level annotations. Then, we introduce a hybrid supervised learning procedure to take advantage of this hybrid labeled training set. A complete overview of the whole active learning procedure is shown in Figure 2.

3.1 Hybrid Dataset using Active Learning

Our hybrid supervised dataset should consider which images to include in training and how to label them. To tackle this problem, we propose an active hybrid learning framework which acts as an active learning agent to simulate the annotation process. At every active step $\mathit{t}$, we have a budget constraint $\mathit{d}$ to spend on annotations. Each image in this set belongs to one of three pools: unlabeled $\mathit{U_{t}}$, weakly labeled $\mathit{W_{t}}$, or strongly labeled $\mathit{S_{t}}$. At each active step, the hybrid supervised dataset $\mathit{W_{t}}\cup\mathit{S_{t}}$ is used to train a hybrid detector, which will be described in detail in Section 3.2. This detector is used to find a selection function that takes an action on $\mathit{U_{t}}$ or $\mathit{W_{t}}$ pools. As shown in Figure 2, we have three possible actions with their associated cost: (1) annotate strongly $\mathbf{x}_{1}$ (i.e. send the image from $\mathit{U_{t}}$ to $\mathit{S_{t}}$), (2) annotate weakly $\mathbf{x}_{2}$ (i.e. send the image from $\mathit{U_{t}}$ to $\mathit{W_{t}}$), or (3) strongly annotate a weakly annotated image $\mathbf{x}_{3}$ (i.e. send the image from $\mathit{W_{t}}$ to $\mathit{S_{t}}$). Once the actions have been made, the image sets $\mathit{U_{t}}$, $\mathit{W_{t}}$, and $\mathit{S_{t}}$ are updated. We proceed iteratively until we either run out of images in $\mathit{P_{t}}=\mathit{U_{t}\cup W_{t}}$ or run out of budget $\mathit{d}$.

To this end, we study three active learning sampling functions (Random, Uncertainty, and Learning Active Learning [28]) within four action selection methods: Random Sampling (RS), Uncertainty Sampling (US), Optimization based on US and Optimization based on Learning Active Learning (LAL).

For RS, we randomly choose an active batch of images at each active step to include into $\mathit{W}_{t}$ or $\mathit{S}_{t}$. For US, images are sorted in descending order of uncertainty (measured by entropy) to choose high uncertainty images to train on with full supervision111We also conducted an experiment in which images in US were sorted in ascending order and low uncertainty images were chosen in every step. We discuss this in greater detail in the supplementary material., based on an uncertainty score denoted as $s_{k}$, and the top images are included into $\mathit{W}_{t}$ or $\mathit{S}_{t}$. We adhere to the following annotation policy for RS and US when a budget constraint is enforced. For both RS and US, we prioritize weak labels first. In other words, so long as the budget constraint is not exceeded, (1) the image batch that is selected in each active step for these two methods will only contain images from $\mathit{U}_{t}$ that will be weakly labeled or, (2) if $\mathit{U}_{t}$ becomes empty, images from $\mathit{W}_{t}$ will be selected and get a strongly label.

There are several ways to evaluate $s_{k}$ for an image [55, 41]. We follow convention and model this score in Eq. 1:

[TABLE]

This is an entropy measure for each of the $M$ bounding boxes in an image using the classification score predicted by the current detection model. Collecting $M$ predictions from an image $I_{k}$, each prediction has a probability score vector $\mathbf{p}_{i}\in[0,1]^{c}$ for the $c$ object categories.

Below, we explain the remaining active selection methods: Optimization based on US and LAL.

3.1.1 Optimization Based Active Selection using US

At the $t$-th active step, let $\mathit{P}_{t}=\{I_{k}\}_{k=1}^{N}$ be the $N$ images that can be further annotated, and $\mathbf{s}\in\mathbb{R}^{N}$ be their corresponding uncertainty scores (as defined above). We have three possible action vectors $\mathbf{x}_{1},\mathbf{x}_{2},\mathbf{x}_{3}\in\{0,1\}^{N}$, as defined earlier. If the $k$-th element of $\mathbf{x}_{i}$ is $1$, we annotate the image $I_{k}$ using option $i$. Assume the next active batch of annotations has a linear impact on the model performance increment $\delta_{t}$.

[TABLE]

To quantitatively show the contribution of new images, we also assume that the uncertainty score is a complete statistic of an unlabeled or weakly labeled set of images. Then, we can simply approximate $(f_{1},f_{2},f_{3})$ as linear functions. We observe that many active learning studies [31, 19] have experimentally shown that incorporating images with a higher uncertainty score in training may further improve the detector, but at the risk of having more difficulty in producing true positive predictions. Therefore, we model $(f_{1},f_{2},f_{3})$ as linear functions that tend to favor actions $\mathbf{x_{1}}$, $\mathbf{x_{3}}$ over action $\mathbf{x_{2}}$ for images with high enough uncertainty score (i.e. higher than the median). As such, and by combining the intuitions above, the expected increment in performance is modeled as:

[TABLE]

For the active selection method based on optimization using US, we seek to maximize $\delta_{t}$, while staying within the budget constraint. To model this latter constraint, we define $a$ as the cost of strongly annotating an unlabeled image, $b$ as the cost of weakly annotating an unlabeled image, and $c$ as the cost of strongly annotating an already weakly labeled image. Therefore, this selection method seeks to solve the following binary optimization problem for $(\mathbf{x_{1},x_{2},x_{3}})$:

[TABLE]

Here, we define a vector $\psi$ as the indicator vector for images that have already been weakly annotated, i.e. the $k$-th component of $\psi$ is 1 if $I_{k}\in\mathit{W}_{t}$ and 0 otherwise. Using this indicator, the first two constraints in Eq (4) enforce that only one action is performed on each image, i.e. among all the $k$-th components of $(\mathbf{x_{1},x_{2},x_{3}})$, only one can be 1. The third and fourth constraints enforce that the budget be used as much as possible in each active step.

The linear binary problem in Eq (4) is NP-hard in general, so exact solvers (e.g. the Branch and Bound Algorithm) tend to have long run-times especially when $N$ is large. As a tradeoff between optimization accuracy and per active step run-time, we employ a conventional linear relaxation of the original problem to form an approximate linear program (LP), which can be efficiently solved at large-scale with off-the-shelf LP solvers. More details about this problem are shown and proven in the supplementary material.

3.1.2 Optimization Based Active Selection using LAL

The uncertainty score $\mathbf{s}$ is not the only selection measure that has been studied in the active learning literature. In the Learning Active Learning (LAL) method proposed by Konyushkova et al. [28], the increment in performance function $\delta_{t}$ is learned to be a function of the current model state as well. More concretely, given images $I_{k}\in\mathit{P}_{t}$ and the current detection model, we build a feature vector $\mathbf{v}=[\mathbf{O}_{t},\mathbf{s}]$ that concatenates both the current model state $\mathbf{O}_{t}$, represented as the average precision curves under five different Intersection over Union (IoU) thresholds, and the uncertainty scores $\mathbf{s}$. Following the LAL method in [28], we train a Support Vector Regression (SVR) model to regress the actual increment in mAP performance from $\mathbf{v}$ at each active step for both weak and strong annotation actions. Obviously, these SVRs are trained on a separate detection dataset than the one used to evaluate BAOD. At each active step and by denoting the output predictions of these SVRs as $\mathbf{h}_{w}$ and $\mathbf{h}_{s}$, we formulate the same constrained optimization problem in Eq (4) but with the learned objective:

[TABLE]

3.2 Hybrid Supervision for Object Detection

At each active step, we train a hybrid supervised detector using both strong and weak supervision, i.e. using images from $\mathit{W}_{t}$ and $\mathit{S}_{t}$ (refer to Figure 3 for an illustration).

Teacher-Student Model. In this framework, we train an initial detector with the initial strongly annotated image set $\mathit{S}_{0}$. In each active step afterwards and to overcome undesirable local minima, we learn this detector using pretraining on Imagenet [50]. However, the model can also be fine-tuned from previous active steps to reduce computation time. Training on $\mathit{S}_{t}$ at each active step can only learn a decent detector, but it can also transfer knowledge to weakly annotated images. This detector works as a teacher that predicts objects in every image in the weakly labeled image set $\mathit{W}_{t}$. If these predictions are post-processed properly, they can be viewed as pseudo labels, which we merge with the ground truth instance-level labels in $\mathit{S}_{t}$ to train a fully supervised student detector. The student detector is also pretrained from Imagenet. Since it has both strong and weak annotations from training samples in $\mathit{W}_{t}\cup\mathit{S}_{t}$, we expect the student detector to perform better than its teacher.

Post-processing. The predictions from the teacher model at each step have many redundant or erroneous pseudo labels. We use minimal knowledge to post-process them so as to focus our study on the active selection methods and the benefit of hybrid training. More concretely, given an image $I_{k}$ with a weak annotation $\mathbf{\omega}\in\{0,1\}^{c}$, where $c$ is the number of categories, we assume that the teacher model gives $M$ positive predictions with localization $\mathbf{P}\in\mathbb{R}^{M\times 4}$ (4 components defining a bounding box), classification $\mathbf{A}\in\{0,1\}^{M\times c}$, and the confidence score for each of the $M$ positive predictions $\mathbf{q}\in[0,1]^{M}$ . $\mathbf{P}$, $\mathbf{A}$ and $\mathbf{q}$ are obtained from the teacher model. Among these $M$ predictions, we seek to mine the pseudo labels in the form of a sparse $M$-dimensional binary vector $\mathbf{y}$ that solves the following constrained optimization:

[TABLE]

The intuition behind solving this problem is that we seek to maximize the confidence score across all pseudo labels indexed by $\mathbf{y}$. The first constraint enforces image-level consistency of the pseudo labels with the ground truth weak annotations. The second constraint removes predictions that are highly overlapping (similar to an NMS post-processing step). The third constraint enforces a sparsity condition on $\mathbf{y}$, thus limiting the number of possible pseudo labeled bounding boxes (potential objects) to be between 1 and $\beta$. In our experiments, we take $(\alpha=0.3,\beta=3)$. Details on this optimization are in the supplementary material.

4 Experiments and Analysis

4.1 Experimental Setup

Datasets. As in most WSOD methods, we use the PASCAL VOC 2007 (VOC07) or 2012 (VOC12) datasets [13] to perform most of the experiments. Given the active learning pipeline of our method, we emulate the active learning procedure using VOC07-12 annotations to selectively annotate 20 categories in 5011 images in VOC07 or 16551 images in the union of VOC07 and VOC12 (VOC0712). All detection models are evaluated on the VOC07 test dataset. For each annotation type (weak or strong), we assume that each image has a fixed annotating cost/time, which is not necessarily true in practice but it simplifies the analysis. In most of the experiments, we set $(a=34.5,b=1.6,c=a-b)$, in unit seconds, according to the annotation procedure of [50]. In Section 4.4, we vary these cost value to $(a=7,b=1.6,c=a-b)$ following the more efficient annotation procedure of [41]222We left more discussion on the annotating time assumption in the supplementary material.. The cost of fully annotating VOC07 trainval is denoted as $100\%$ (or total) budget for every experiment.

Evaluation Metrics. To evaluate the active selection methods with the hybrid detector, we compute a budget-performance curve at various budget limits. The budget axis varies the percentage of the total budget, and the model detection performance is taken to be mAP. In doing so, we propose a new budget-aware metric denoted as Budget-Average mAP, measured as the normalized area under the budget-mAP curve for a certain budget range. We take three ranges $[10\%,30\%]$, $[30\%,50\%]$, and $[50\%,100\%]$ to evaluate our experiment such that three to five data points are located in each range and the metric is less affected by noise. The first range starts from $10\%$ since we need a small fully labeled warm up set to initialize the fully supervised detector, and start our pipeline. This warm up set is randomly selected and fixed in all experiments. The last budget range is wider because the performance curve saturates at high budgets, and we observe more subtle changes in performance. All budget-performance curves are shown in the supplementary material.

Implementation Details. In every active step, we choose Faster-RCNN as the object detection model (teacher and student), which is trained in the hybrid supervised way described in Section 3.2. Both teacher and student models use VGG16 as the backbone network pre-trained on ImageNet [50]. We follow the default setup in [58] to train Faster-RCNN. During training, the total number of epochs is set to $10$, the learning rate is $0.01$ for the first $8$ epochs and $0.001$ for the remaining. The batchsize is set to 16 on four-GPU cluster nodes equipped with Titan Xp. The student model is cloned from the ImageNet pretrained VGG16 in every active step and has the same training schedule as the teacher model333The source code of the framework will be made publicly available. . For LAL experiment, we collect 10 categories from the MS-COCO dataset [35] to train the SVRs. Since the LAL method is dataset agnostic [28], we take the SVR training categories to be different from the 20 categories in PASCAL VOC.

4.2 Advantages of Uncertainty Sampling and Hybrid Training

Uncertainty Sampling. In order to explore the influence of the uncertainty score on model performance, we emulate the annotation process of the oracle using VOC07 and incorporate it into the active learning pipeline with only strong annotations and use a FSOD method to train our model. We use two selection methods (RS and US) to build the dataset for FSOD. From the first two columns in Table 5 that compare these two sampling methods, we see that collecting images with large uncertainty score is more effective and preferable than random sampling in the three budget ranges.

Hybrid Training. Table 5 also shows the influence of the active selection scheme on model performance but when the hybrid supervised learning process is used to combine weak and strong annotations at each active step. We again use RS and US to select the images in each step for hybrid training. It is clear that both hybrid methods outperform their corresponding FSOD counterpart at every budget range, especially when the budget range is low (under $50\%$).

4.3 Optimization-Based Active Selection

Here, we evaluate the optimization-based selection methods (using US and LAL) described in Section 3.1, when they are combined with hybrid supervised training. Table 2 compares these methods by measuring their average mAP at each budget range and compares them to the RS and US selection methods for references. These results indicate that the optimization methods are the best ways to combine and use the hybrid annotations at every budget, where the optimization-based US method is slightly better than its LAL counterpart. As such, we denote the former method as the BAOD approach, which was mentioned and highlighted in Figure 1. Interestingly, we observe that the RS method requires $62\%$ of the total VOC07 budget (all images are strongly annotated) to achieve $95\%$ of the detection performance at that budget (i.e. $67.4\%$ mAP). In comparison, the BAOD method requires only $48.5\%$ of the total budget to reach the same performance. This performance gap attests to the effectiveness of this method. We include more results in the supplementary material.

4.4 Effect of Per-Image Annotation Cost

The cost of strong annotations can vary due the annotation strategy that is used. For instance, Papadopoulos et al. [41] created a method that reduces the time needed to draw bounding boxes to 7 seconds per image on average. Here, we use this cost ($a=7$) to study the behavior of our BAOD method given a smaller gap between strong and weak annotation. We report these results in Table 3, which depicts the best performing FSOD method and the RS hybrid baseline for reference. We observe the same relative behavior between the methods as in the case when $a$ was several times higher. However, the performance gap between these methods decreases in this case because the number of images that can be weakly annotated for the cost of one strong annotation is much smaller.

4.5 Improving Detection using Fixed Budget

This experiment simulates a real-world application of our method. We combine VOC07 and VOC12 data to simulate a larger pool of unlabeled images (called VOC0712) to choose from. As reported in Table 4, when the budget is set at $87.2\%$ of the total budget (annotation budget for VOC07), we learn an object detector whose performance is the same as an FSOD detector trained on the entire strongly labeled VOC07 training set. This is a budget saving of $12.8\%$. Now, if we choose to use this total budget on VOC0712, our method achieves a $73.0\%$ mAP, which is $2\%$ improvement over the aforementioned FSOD detector on the same VOC07 test set. If all the 16551 images in the union of VOC07 and VOC12 set are strongly labeled, we use an extra $230\%$ of budget and only improve $3.4\%$ mAP.

4.6 Easy Images and Weak Annotation First

We analyze the cost and number of images selected at every active selection step to investigate which type of training examples are more helpful in the sequential training process. Based on the final mAP of each category in VOC07 [14], we divide the twenty categories into three groups: Easy, Medium, and Hard.The left plot of Figure 4 illustrates that BAOD spends more budget to annotate Easy categories (shown in the green area) in the first several steps, while the cost of Hard categories (red area) increases when the detector becomes more accurate. These results align with concepts from curriculum learning, in which a larger number of easy samples can be trained on first to bootstrap the model and then hard samples are introduced progressively.

The right plot of Figure 4 measures the percentage of selected images per active step that belong to various subsets of the data (combinations of annotation type chosen and difficulty). Interestingly, the BAOD model tends to select more weak annotations at the beginning from all three groups of objects, since this kind of label is cheap and informative. Given a larger budget, the agent increases the proportion of strong annotations to further improve model performance. Note that there are no new weakly labeled images after active step 9 because all the images are already annotated.

4.7 Qualitative Results of the Active Selection

Figure 5 shows some selected strongly and weakly labeled images based on the BAOD experiment in section 4.5. Each row of the images is from the dog category in the VOC07 or VOC12 trainval set, and each column of images is selected at the same active step.

We show that in the first five active steps, strongly annotated images contain dog instances that are difficult to detect due to occlusion, multiple close instances or small-scale. In contrast, the selected weakly annotated images during the same steps are relatively easier to locate. As the model gets mature the difficulty for both levels of annotations increases. For example, the image chosen in step 4 contains three small dogs, and the dogs appeared in step 5’s images are small and black which makes them barely visible.

5 Conclusion

In summary, we introduce a novel budget-aware perspective to study the unexplored dimensions of the object detection problem. With a fixed budget, we compare both optimization and learn based sample methods to build diverse hybrid supervised object detection datasets which consist of both image level supervision and instance level supervision. The evaluation of detectors that learned from these budget fixed datasets shows that the handcrafted optimization method on uncertainty score outperforms other general active learning methods including random sampling, active learning, and reinforcement learning (shown in supplementary material). With the optimal set-up, our proposed budget-aware approach can achieve the performance of a strongly supervised detector on PASCAL-VOC 2007 while saving $12.8\%$ of its original annotation budget. Furthermore, when $100\%$ of the budget is used, our approach surpasses this performance by 2 percentage points of mAP.

Appendix A Solutions to Optimization Functions

A.1 Image Sampling

In section 3.1.1 and 3.1.2 of the submission, we propose two linear functions to approximate the mAP increment. Then we maximize it with several constraints. However, the proposed solution is an integer programming problem, and it cannot be solved in polynomial time. For example, the default solver uses Branch and Bound Algorithm (B&B) and it takes more than 24 hours to find a global maximum. To solve this problem more efficiently, we take the relaxation of the original integer program shown as Eq.7. Noted that the solution space of $\mathbf{x}_{1},\mathbf{x}_{2},\mathbf{x}_{3}$ becomes an interval $[0,1]$, but not a discrete set $\{0,1\}$, the relaxation Eq.7 can be solved in linear time.

[TABLE]

To project $\mathbf{x}_{1},\mathbf{x}_{2},\mathbf{x}_{3}$ back to the discrete set $\{0,1\}$, we take three float numbers $\epsilon_{1},\epsilon_{2},\epsilon_{3}$ as the thresholds of $\hat{\mathbf{x_{1}}},\hat{\mathbf{x_{2}}},\hat{\mathbf{x_{3}}}$, respectively. In another word, every element in $\mathbf{x_{k}}$ larger than $\epsilon_{k}$ is set as $1$, otherwise it is [math].

[TABLE]

The solution needs to be feasible after thresholding. More concretely, the first two constraints must be satisfied because we cannot give an invalid action (e.g. It’s impossible to annotate an image both weakly and strongly, but it might appear from the solution).

We set $\epsilon_{1}=\epsilon,\epsilon_{2}=1-\epsilon,\epsilon_{3}=\epsilon$, where $\epsilon$ exhaustively goes from [math] to $1$ to satisfy the last two budget constraints. In this case, it can be proven that the solution of Eq.8 is also feasible from the original constraints.

Proof.

(1) If $\psi(k)=0$, where $k$ can be the index of any element of vector $\hat{\mathbf{x_{1}}}$. Since $\hat{\mathbf{x_{1}}},\hat{\mathbf{x_{2}}},\hat{\mathbf{x_{3}}}$ is feasible, the first inequality in Eq.7 shows

[TABLE]

The second inequality in Eq.7 gives the inequality for $\hat{x_{1}}$ and $\hat{x_{2}}$.

[TABLE]

Here $1\{\}$ is the indicator function. When we apply Eq.8 to check the two restrictions, the discrete solutions $\mathbf{x_{1}}^{*},\mathbf{x_{2}}^{*},\mathbf{x_{3}}^{*}$ are still feasible.

[TABLE]

(2) If $\psi(k)=1$, we have

[TABLE]

Similar to previous discussion, we can have $x_{3}^{*}(k)\leq 1$ and $x_{1}^{*}(k)=0,x_{2}^{*}(k)=0$ to fit the two constraints. ∎

A.2 Post Processing

In the article, we give another optimization function to do post processing. This procedure checks the consistency between the image and instance-level annotations, and it removes abundant pseudo labels.

Given an image with the weakly annotation $\mathbf{\omega}\in\{0,1\}^{K}$, where $K$ is the number of categories. We assume the first detection model (teacher model) gives $M$ positive predictions for the localization $P\in\mathbb{R}^{M\times 4}$, classification $A\in\{0,1\}^{M\times K}$, and the positive class confidence vector $\mathbf{q}\in[0,1]^{M}$. Eventually, the post processing returns a sparse $M$-dimensional binary vector $\mathbf{y}$ as Eq.13, where $\alpha=0.3,\beta=3$.

[TABLE]

In this problem, our objective function takes high confidence predictions by choosing $\mathbf{y}\in\{0,1\}^{M}$. In the first constraint, $\mathbf{y}^{\top}A$ accumulates the selected predictions by their categories. The following inner product with $(\mathbf{1}-\mathbf{w})$ returns zero only when all the predicted classes are in the weak annotation. On the other hand, the second constraint removes the heavily overlapped predictions while the third one makes the binary vector $\mathbf{y}$ sparse. We develop Alg.1 to solve the above optimization.

Appendix B More Budget-Aware Methods and Experiments

B.1 Sample from Small Uncertainty Images

In the Section 3.1 of the submission, to apply Uncertainty Sampling (US) method, we sort images in descending order of uncertainty and choose high uncertainty images. Moreover, we also conducted an experiment in which images were sorted by the uncertainty score in an ascending order, and we pick low uncertainty images in every step. The other settings are the same to US’s. Table 5 compares these two methods.

When the budget is in low range ($10\%-30\%$), collecting images with small uncertainty score is more effective. However, when the dataset size is larger, choosing large uncertainty scores is preferable. This behaviour follows the conclusions of Bengio et al. [3], who suggested that it is better to train models using curriculum learning, i.e. giving easy images to the model in early training and then giving hard images once the model has learned better representations. This experiment also shows that there are better solutions to the BAOD problem. For example, we can find a better sampling function which also considers the current state of the detection model. Assembling the sampling function into the optimization selection method will produce a better budget-mAP performance.

B.2 Reinforcement Learning

B.2.1 Method

The problem of finding the next active batch of training examples can be naturally formulated as a reinforcement learning problem. This approach allows us to learn a strategy directly from the trial-and-error experience and to avoid the specific mathematical model of the previous sections. The reinforcement learning methods do not need any prior knowledge about the environment, but they can still construct an optimal strategy. Thus, it is easily extensible to stochastic environments with variable set-ups such as strongly-weakly annotation costs ratio and a total amount of budget.

Suppose that we already have a training set, and we regress a detector with performance $m_{0}$. In the next active step, we evaluate the uncertainties of a subset of unlabeled images. The active learning agent interacts with the environment in the form of the difficulty preference and annotation type. We use $O_{t}$ as defined in section 3.1.2 of the submission as the observed state, which considers both the classification and localization performance. The agent returns two actions from current state: (1) asking for low/medium/high uncertainty images and (2) giving a strongly/weakly annotation. In conclusion, the RL agent gives a choice $\mathit{a}$ from three by two possible actions matrix.

In the RL training process, we take the detection model increment performance $\Delta_{t}$ as a reward. If there is still enough budget, the agent chooses the next active batch from the new observed state $O_{t+1}$. If we run out of budget after $T$ active steps, the total reward is $R=\sum_{t=1}^{T}\Delta_{t}=m_{T}-m_{0}$. Here $m_{0}$ is the performance of a randomized warm-up model with a fixed seed. By maximizing the return $R$, the agent learns a policy $\pi$ that maximizes the final model performance $m_{T}$.

To train the agent, we learn a policy with $Q$-learning[57] which learns to approximate $Q$-function $Q^{\pi}(\mathit{a},O_{t})$ with a three-layer neural network. Given an action $\mathit{a}$ and the current state $O_{t}$, it returns the expected value from policy $\pi$.

B.2.2 Experiment

At every active step, the $Q$ agent gives insights on which images to label and which type of annotations is required. Different from the previous pipeline, the RL method does not evaluate all the unlabeled images, but renders a preference for easy, middle or hard images. This significantly expedites the training process. We firstly compute the number of required images from the budget and annotation types, then we stop evaluating unlabeled images’ uncertainties, once we get enough images. Table 5 shows that RL performs better than FSOD, since it learns a policy to weakly or strongly annotate images and takes advantage of hybrid training. Nevertheless, it performs worse than US and LAL optimization. The reinforcement learning environment is built by VOC12 training set and validation set. We simulate the active learning process on four Titan Xp GPUs for five days to train the $Q$-learning agent, and the agent is used to solve the budget aware object detection problem on VOC07 trainval set and test set.

Appendix C Definition of Easy, Medium and Hard Classes

We analyze the cost and number of images selected at every active selection step to investigate which type of training examples is more helpful in the sequential training process. Based on the final mAP of each category in VOC07, we divide the twenty categories into three groups: Easy, Medium and Hard. The mapping is motivated and shown in Table 2.

Appendix D Discussion on Annotating Time

In the submission, we assume $a=34.5$ or $a=7$, and $b=1.6$. We explain these values in this section.

First of all, annotating an image does not take a fixed time. From the technique used in ImageNet[50], the Median of annotating a bounding box is $25.5s$, but an extra of $9.0s$ and $7.8s$ is required to control its quality. However, using the state-of-the-art method extreme clicking for efficient object annotation [41] can reduce the strongly annotating time to $7s$. To consider most bounding box annotation methods, we assume this value to be $7s$, $21s$ or $34.5s$. On the other hand, answering a binary question takes around $1.6$ seconds[40], but answering twenty binary question does not take $1.6\times 20=32s$. [40] finds that clicking on one of five on-screen buttons takes around $2.4$ second. In the original ImageNet paper, people select a sequence of queries for humans such that we achieve the same labeling results with only a fraction of the cost of the naive approach. It is more efficient to organize the semantic concepts into hierarchies [53]. More importantly images are collected from the internet through keyword search, which implies that many image level labels are free.

In our Budget-Aware Object Detection problem, we only care about budget ratios between strong annotations and weak annotations, although there are multiple instances in one image. For this reason, we have experiments with different ratios to compensate the cost assumptions and to show that at different ratios our method still works. In the last chapter of the supplementary material, we have a table to show the experiments with different cost ratios (CR) in $\{34.5:1.6,21:1.6,7:1.6\}$ of strong and weak annotations.

Appendix E Budget-mAP Curves

We present the original Budget vs mAP figures for Table 1,2 and 3 in the submission, respectively. The curves shown here are mAP scores before integration, which are more detailed but noisy. Please check section F for the table representation.

Appendix F Raw Experiment Data

Appendix G More Visualizations

Below we show some visualization figures of strongly labelled and weakly images for each of the classes chosen by US based Optimization. Each column indicates one step of the active learning pipeline and each of the rows represents a class. The warm up set is represented on the first column showing only strongly labeled images chosen randomly. For the next 5 steps we can observe on the top sub-row strongly labeled images and on the bottom sub-row weakly labeled images. The bounding boxes shown in this figure are not the original annotations from PASCAL, they were drawn manually only for this visualization purposes.

[TABLE]