C-RPNs: Promoting Object Detection in real world via a Cascade Structure of Region Proposal Networks

TL;DR

This paper introduces C-RPNs, a cascade framework for region proposal networks that enhances real-world object detection by mining hard samples and learning stronger classifiers, showing improved performance on multiple datasets.

Contribution

The paper proposes a novel cascade structure of RPNs with feature and score chains, and a specialized loss function, to better handle data imbalance and hard samples in real-world detection tasks.

Findings

Achieves competitive results on Pascal VOC and challenging datasets.

Demonstrates all-sided improvements in error analysis.

Validates effectiveness for real-world object detection.

Abstract

Recently, significant progresses have been made in object detection on common benchmarks (i.e., Pascal VOC). However, object detection in real world is still challenging due to the serious data imbalance. Images in real world are dominated by easy samples like the wide range of background and some easily recognizable objects, for example. Although two-stage detectors like Faster R-CNN achieved big successes in object detection due to the strategy of extracting region proposals by region proposal network, they show their poor adaption in real-world object detection as a result of without considering mining hard samples during extracting region proposals. To address this issue, we propose a Cascade framework of Region Proposal Networks, referred to as C-RPNs. The essence of C-RPNs is adopting multiple stages to mine hard samples while extracting region proposals and learn stronger classifiers. Meanwhile, a feature chain and a score chain are proposed to help learning more discriminative representations for proposals. Moreover, a loss function of cascade stages is designed to train cascade classifiers through backpropagation. Our proposed method has been evaluated on Pascal VOC and several challenging datasets like BSBDV 2017, CityPersons, etc. Our method achieves competitive results compared with the current state-of-the-arts and all-sided improvements in error analysis, validating its efficacy for detection in real world.

Tables

Table 1. Table 1: Results on PASCAL VOC 2007 test set. 07+12: union of Pascal VOC07 trainval and VOC12 trainval.

Method	Trainset	mAP	aero	bike	bird	boat	bottle	bus	car	cat	chair	cow	table	dog	horse	mbike	person	plant	sheep	sofa	train	tv
Fast R-CNN [8]	07+12	70.0	77.0	78.1	69.3	59.4	38.3	81.6	78.6	86.7	42.8	78.8	68.9	84.7	82.0	76.6	69.9	31.8	70.1	74.8	80.4	70.4
Faster R-CNN [4]	07+12	73.2	76.5	79.0	70.9	65.5	52.1	83.1	84.7	86.4	52.0	81.9	65.7	84.8	84.6	77.5	76.7	38.8	73.6	73.9	83.0	72.6
SSD500 [11]	07+12	75.1	79.8	79.5	74.5	63.4	51.9	84.9	85.6	87.2	56.6	80.1	70.0	85.4	84.9	80.9	78.2	49.0	78.4	72.4	84.6	75.5
ION [19]	07+12	75.6	79.2	83.1	77.6	65.6	54.9	85.4	85.1	87.0	54.4	80.6	73.8	85.3	82.2	82.2	74.4	47.1	75.8	72.7	84.2	80.4
RON [46]	07+12	77.6	86.0	82.5	76.9	69.1	59.2	86.2	85.5	87.2	59.9	81.4	73.3	85.9	86.8	82.2	79.6	52.4	78.2	76.0	86.2	78.0
SIN [23]	07+12	76.0	77.5	80.1	75.0	67.1	62.2	83.2	86.9	88.6	57.7	84.5	70.5	86.6	85.6	77.7	78.3	46.6	77.6	74.7	82.3	77.1
C-RPNs (ours)	07+12	76.4	78.6	79.5	76.3	66.5	63.2	84.6	87.8	87.8	60.2	83.3	71.7	85.5	86.1	81.4	79.2	49.2	75.2	73.9	83.1	75.7

Table 2. Table 2: Performance Comparison on BSBDV 2017.

Method	Backbone Network	AP(%)
SSD500 [11]	VGG16 reduce	42.0
Faster R-CNN [4]	VGG16	59.3
RON [46]	ResNet-101	58.0
R-FCN [9]	ResNet-50	61.5
FPN [13]	ResNet-50	66.9
SIN [23]	VGG16	58.4
C-RPNs (ours)	VGG16	70.3

Table 3. Table 3: Performance Comparison on Caltech Pedestrian Benchmark and CityPersons.

Method	Caltech pedestrian benchmark	CityPersons
Faster R-CNN [4]	44.0	49.1
C-RPNs (ours)	48.1	51.4

Table 4. Table 4: The impact of cascade stages (BSBDV 2017).

AP of C-RPNs(%)	69.5	69.9	70.3
C-RPNs with Stage 4	✓	✓	✓
C-RPNs with Stage 3	✓	✓	✓
C-RPNs with Stage 2		✓	✓
C-RPNs with Stage 1			✓

Table 5. Table 5: The impact of feature/score chain (BSBDV 2017).

AP of C-RPNs(%)	69.4	70.0	69.8	70.3
Feature Chain		✓		✓
Score Chain			✓	✓

Equations

$$h_{t}=\left\{\begin{array}[]{ccc}f_{t}&when&t=1\\ \lambda_{f}*h_{t-1}\bigoplus\lambda_{p}*f_{t}&when&t>1\end{array}\right.$$

$$s_{t}=\left\{\begin{array}[]{ccc}c_{t}&when&t=1\\ \lambda_{f}*s_{t-1}+\lambda_{p}*c_{t}&when&t>1\end{array}\right.$$

$${\text{L}}_{C-RPNs}(s,k^{\ast },l,l^{\ast })=L_{cls}(s,k^{\ast })+L_{loc}(l,l^{\ast },k^{\ast })$$

$${\text{L}}_{cls}(s,k^{\ast })=-\sum _{t=1}^T{\alpha }_t{\mu }_{t}log(s_{t,k^{\ast }})$$

$${\alpha }_t=\frac{{\alpha }_T}{10^{T-t}}$$

$$\mu_{t}=\left\{\begin{array}[]{ccc}\prod_{i=1}^{t-1}[s_{t,k^{*}}<r]&when&t>1\\ 1&when&t=1\end{array}\right.$$

$$L_{detection}=L_{C-RPNs}+L_{roi}$$

Full text

††journal: Journal of LaTeX Templates

url]http://web.pkusz.edu.cn/adsp/ myfootnotemyfootnotefootnotetext: Address: Peking University, Xili Road, Nanshan District, Shenzhen, P.R.China

1 Introduction

Object detection is a most fundamental step in visual understanding, which aims at identifying and localizing objects of certain categories in images. To promote the development of object detection, plenty of benchmarks have been developed, i.e., PASCAL VOC [1] and MS COCO [2]. Most of object detection approaches are trained and tested on these common object detection benchmarks, which typically assume that objects in images are with good visibility and balance. Obviously, this assumption is usually not satisfied in real world.

Taking littoral bird images from developed benchmarks and wild scenes as examples, the former are usually collected with better visibility, while the latter are collected via monitoring cameras with different background and camera distance. Moreover, different illumination and weather conditions may appear in wild scenes. For more intuitive observation, several examples of littoral birds are illustrated in Figure 1. The image from BSBDV 2017 [3] shows birds from wild scenes, while images from PASCAL VOC [1] show birds from common benchmarks. The image from BSBDV 2017 is with resolution of 49123264, in which the heights of birds vary from 80 to 300 pixels. Images from PASCAL VOC 2007 and 2012 are with average resolution of 400400, where the heights of birds are from 150 to 480 pixels. Apparently, the easily recognizable background in wild scenes take more prominent position compared with that in common benchmarks. Besides, bird objects obtained from the wild scenes are with smaller sizes and less texture information. For object detection techniques, such a distribution mismatch from common benchmarks to real world have been observed to lead to a significant performance degradation.

Although enriching training data could possibly alleviate the performance degradation, it is not favored since annotating data is expensive and time consuming. Therefore, developing object detectors towards real world is desirable. To figure out the crucial elements of performance degradation in real-world object detection, plenty of experiments have been conducted. We list the conclusions as follows:

Data imbalance frequently occurs in real world. From an image in real world, the number of negative samples (also called background samples) is much larger than that of positive samples (As shown in Figure 1), and most of them are easy samples. Easy samples do not contribute useful learning information during training while hard samples benefit the convergence and the detection accuracy. Thus, the overwhelming number of easy samples during training leads to moronic classifiers and degenerate models. 2. 2.

As mentioned above, because of the smaller size, poor shooting conditions and poor abundance of objects in real-world scenes, classifiers in detection algorithms are unable to learn discriminative features from ground truth.

In this work, we aim to improve the precision of object detection in real world. Based on observations above, mining hard samples from abundant easy samples for training is a crucial route to address this issue. Based on the brilliant object detector Faster R-CNN [4], we firstly propose a cascade framework of region proposal networks, referred to as C-RPNs. While extracting region proposals, C-RPNs are adopted to mine hard samples and learn stronger classifiers. Multi-stage classifiers at early stages discard most of easy samples so that classifiers at latter stages focus on handling hard samples. Also, we design a feature chain and a score chain to generate more discriminative representations for proposals. Finally, a loss function of cascade stages is built to jointly learn cascade classifiers.

The contributions of this work are summarized as follows:

Based on the Faster R-CNN, a cascade structure of region proposal networks for object detection was firstly proposed, referred to as C-RPNs.

• 2.

A feature chain and a score chain were designed in C-RPNs to further improve the classification capacity of multi-stage classifiers.

• 3.

A loss function of multi-stage was constructed to jointly learn cascade classifiers.

• 4.

Integrating the proposed components into the Faster R-CNN, our resulting model can be trained end-to-end.

Extensive experiments have been conducted on several datasets, including PASCAL VOC [1] , BSBDV 2017 [3], Caltech Pedestrian Benchmark [5] and CityPersons [6]. Our approach have provided competitive performance compared with the current state-of-the-arts. Besides, error analyses have shown that our approach achieved all-sided improvements compared with the baseline Faster R-CNN. The experimental results demonstrate the effectiveness of our proposed approach for object detection in real world.

2 Related work

2.1 Related Work On Object Detection

We all have witnessed tremendous progresses in object detection using convolutional neural networks (CNNs) in recent years [7, 8, 4, 9, 10, 11, 12]. Region-based CNN approaches [7, 8, 4] are referred as two-stage detectors, which have received great attention due to their effectiveness. At the outset, R-CNN [7] was constrained by a selected search region. To reduce the computational complexity of R-CNN, Fast R-CNN [8] shared the convolutional feature maps among region of interest (RoI) and accelerated spatial pyramid pooling using RoI pooling layer. Renetal [4] introduced Region Proposal Network (RPN) to generate high-quality region proposals and then merged them with Fast R-CNN into a single network, referred to as Faster R-CNN. Besides, for faster detection, one-stage detectors such as YOLO [12] and SSD [11] were proposed to accomplish detection without region proposals, although this strategy reduced the detection performance. Researches showed that Faster R-CNN achieved a big success in object detection and laid the foundation for many follow-up works [10, 13, 14, 15]. For example, feature pyramid and fusion operations were adopted [13] to enhanced precision of detection. Deeper [16, 17, 18] or wider [19, 20] networks also benefited the performance. Deformable CNN [15] and Receptive Field Block Net [21] enhanced the convolutional features using deformable convolutional operation and Receptive Field Block respectively. In addition, using large batch size [22] during training provided improvement in detection. SIN [23] jointly used scene context and object relationships to promote detection performance.

Although reasonable detection performances have been achieved on benchmarks like PASCAL VOC [1] and MS COCO [2], object detection in real world still suffers from poor precision. Works mentioned above mostly focused on the conventional setting while rarely considered the adaptation issues for object detection in real world such as data imbalance.

2.2 Related Work On Hard Example Mining and Cascade CNN

Gradually updating the set of background samples by selecting those from samples which are detected as false positives, bootstrapping [24] was the earliest solution to automatic employ hard samples for training. The strategy in bootstrapping led to an iterative process that alternates between updating the trained model and finding new false positives to add to the bootstrapped training set. Bootstrapping techniques were then successfully applied on detectors driven by CNN and SVMs for object detection [7, 25], generally referred to as hard negative mining. After that, CNN detectors like Fast R-CNN [8] and its descendants were trained with stochastic gradient descent (SGD) on millions of samples, in which bootstrapping as an offline progress was no longer been adopted. To balance positive and negative training samples but without thinking of mining hard ones, Faster R-CNN [4] randomly used 256 samples in an image to compute the loss function of a mini-batch, where the positive and negative ones have a ratio of up to 1:1. A number of methods [26, 27, 28] then focused on mining hard samples online for training convolutional networks. Rowley [26] selected hard positive and negative samples from a larger set of random samples based on their loss independently. Sermanet[27] and Shrivastava [28] focused on online hard sample selection strategies for mini-batch SGD methods and then OHEM [28] were introduced for region-based detectors which built mini-batches with the highest-loss samples. Recently, Focal Loss [29] has been proposed to address the extreme foreground-background class imbalance problem in object detection with one-stage detectors, which applied a modulating term to the cross entropy loss in order to focus learning on hard negative examples. Analyzing of previous works shows that inchoate bootstrapping techniques are inappropriate for CNN-based detectors. Some online hard example mining strategies selected hard examples based on their loss, which are innovative but time-consuming. Focal Loss focused on dealing with data imbalance with one-stage detectors, while our works pay more attention to two-stage detectors with region proposals.

From another perspective, cascade structure is a widely used technique to discard easy samples at early stages for learning better classification models. Before the prosperity of CNNs, cascade structure were applied to SVM [30] and boosted classifiers [31, 32] with hand-crafted features. Multi-stage classifiers have been proved to be effective in generic object detection [30] and face detection [32, 33], although these multiple classifiers were not trained jointly. It was showed that CNNs with cascade structure performed effectively on classification [34, 35, 36] as well, in which multiple but separate CNNs were trained. After that, Qin [37] proposed a method to jointly train a cascade CNNs. The recent method Cascade R-CNN [38] trained Faster R-CNN with cascade increasing IoU thresholds, which was innovative, but without considering the data imbalance issue. Based on observations above, cascade structures are potential, but existing works either cannot be aggregated in the R-CNN based detection framework or have not considered building cascade structure on RPN to help extracting hard region proposals. Thus, confronting with the data imbalance problem in object detection in real world, the existing outcomes are very limited.

In this work, we propose C-RPNs to mine hard samples while extracting region proposals and learn more discriminative features for object detection in real world. Integrating with Faster R-CNN model, our proposed method, to the best of our knowledge, is the first cascade model of region proposal networks for object detection.

3 Proposed Method

3.1 Overview Of C-RPNs

Faster R-CNN consists of a shared backbone convolutional network, a region proposal network (RPN) and a final classifier based on region-of-interest (RoI), in which the RPN is employed to extract region proposals. Without considering mining hard samples in the process of RPN, Faster R-CNN shows its limited capacity in detection in realistic scenes. Our novel C-RPNs are firstly proposed to address this problem.

For performance comparison fairness, VGG16 is taken as the backbone network [16]. Figure 2 shows an overview of our proposed C-RPNs model. At first, several shared bottom convolutional layers are used for extracting convolutional features from the image (Conv1-Conv4_1). Then, C-RPNs is adopted upon four different convolutional layers, which are Conv4_2, Con4_3, Conv5_2 and Conv5_3. Since feature maps from Conv5 have the same channels but half size compared with those from Conv4, we employ an average pooling with size of 2*2 upon Conv4_2 and Conv4_3 to obtain feature maps of same resolutions for these four stages. At stage 1, the feature map extracted from Conv4_2 are used for generating region proposals and obtaining binary classification scores by a softmax function. The binary classification scores estimate a sample’s probabilities belonging to background and objects. With the classification scores and a reject threshold r, part of easy samples will be rejected at this stage, which are detailed in Section 3.3. At stage 2, if a proposal has not been rejected at the former stage, then the feature map from Conv4_3 for this proposal is used for further binary classification. Similar processes are applied at stage 3 and stage 4. Since there is no constrain that the rejected samples must be background, few easy positive samples might also be rejected at early stages during training. It is worth to point out that the stage 4 achieves not only binary classification but also bounding box regression. After these four stages, the proposals have not been rejected are sent to RoI pooling layer for final detection. In this study, we set batch of each stage as 1024, 768, 512 and 256 respectively so that the stage 4 has the same batch size with RPN from Faster R-CNN. It is worth mentioning that the reason why we set only 4 stages not 5 or more is that employing the shallow and bigger feature maps from Conv3 contributes very limited performance gain but is time-consuming according to our experiments.

From Figure 2, it can be seen that C-RPNs takes different convolutional features stage-by-stage which enables it obtains different semantic information and receptive field. It is also noted that, in C-RPNs, the classifiers at shallow stages handle easier samples so that the classifiers at deeper stages focus on handling more difficult samples. The easy samples rejected by a classifier from shallow stage will not participate in the latter stages. With this design, abundant samples can be used but only hard samples been mined will go for final classification and bounding box regression, which benefits to alleviate the data imbalance problem.

To further enhance the classification capacity, a feature chain and a score chain are designed in C-RPNs, which are detailed in Section 3.2. In the end, the multi-stage classifications and bounding box regressions are learned in an end-to-end manner through backpropagation via a joint loss function, details are given in Section 3.3.

3.2 Feature Chain and Score Chain

Literature studies show that FPN [13] and DSSD [39] are effective for object detection using multiple convolutional layers. In this study, in order to capture the variation of features from different layers, a feature chain and a score chain at cascade stages are designed which are able to make use of features at previous stages as the prior knowledge for the classification at current stage. Not like the top-down pathway and lateral connections from FPN, our feature fusion operation follows the bottom-up pathway, which is the feed-forward computation of the VGG16. The description of feature chain and score chain is shown in Figure 3.

We define the number of stages as T and t is the stage index. At stage t, we denote the features from convolutional layer as $f_{t}$ while features for classification as $h_{t}$. The feature chain is formulated as following:

[TABLE]

where $\oplus$ denotes the summarized point to point. $\lambda=\{\lambda_{f},\lambda_{p}\}$ are hyper parameters controlling the weight of features from former stage and present convolutional layer to generate fusional features for classification. $\lambda_{f}$ and $\lambda_{p}$ add up to 1. Considering features from present convolutional layer are more helpful for classification, we set $\lambda_{f}$ as 0.1 and $\lambda_{p}$ as 0.9 according to our empirical tests (detailed in Section 4.5). The fused features $h_{t}$ are then used for classification.

At stage t, for each proposal have not been rejected at the t-1 stage, we denote the score from classifier t as $c_{t}$ while the output score of this stage as $s_{t}$. The designed score chain has the following formulation.

[TABLE]

In this implementation, features and scores at current stage make use of those from previous stages which enhance the capacity of the classifiers at current stage.

3.3 Cascade Loss Function with Samples Mining

In Faster R-CNN, training loss is composed of loss from RPN and ROIs. The former contains a binary classification loss and a regression loss. In our method, illustrated in Figure 4, C-RPNs contains four binary classification losses and a regression loss.

In C-RPNs, the cascade classifiers assign a sample’s probabilities to background and objects. k={0,1} is denoted to express these two class respectively. At stages $t\epsilon\{1,2,3,4\}$, the set of class scores for a sample are denoted by $s=\{s_{t}|t=1,\ldots,T\}$. $s_{t}=\{s_{(t,0)},s_{(t,1)}\}$ are scores at stage t for background and objects respectively. Another layer at stage 4 outputs bounding box regression offsets $l=\{l^{k}|k=1\}$, $l^{k}=(l_{x}^{k},l_{y}^{k},l_{w}^{k},l_{h}^{k})$ for objects. Our proposed loss function of C-RPNs has the following formulation:

[TABLE]

[TABLE]

where $L_{cls}(*)$ is the loss for classification and $L_{loc}$ is the loss for bounding box regression. For $L_{loc}$, we use the $smoothed$ $L_{1}$ loss [8]. For $L_{cls}(*)$, $\alpha_{t}$ and $\mu_{t}$ are defined as follows, where $\alpha_{t}$ is a parameter that controls the weight of loss from cascade classifiers and $\mu_{t}$ evaluates whether the sample is rejected at previous stages.

[TABLE]

[TABLE]

Here, we set $\alpha_{T}=1$, where T=4 in C-RPNs. Since scores from deeper classifiers are more crucial for final classification than those from shallow classifiers, $\alpha_{t}$ from deeper classifiers has been distributed more weight with a tenfold increase based on our experience. For $\mu_{t}$, we set the r as a threshold value at each stage. $[s_{(t,k^{*})}<r]$ will output 1 if it is true or output 0 if it is false. If a sample has been rejected at previous stages, it will no longer be used for training the classifier at current stage. We set r as 0.99 according to our empirical tests (detailed in Section 4.5). If $\alpha_{t}=\mu_{t}=1$ and $T=1$, then $L_{cls}(*)$ is a normal cross entropy loss.

For the object detection with the proposed model, the final training loss is designed to compose the loss from C-RPNs and the loss from ROIs:

[TABLE]

where $L_{C-RPNs}$ and $L_{roi}$ both are composed of classification loss and regression loss. The former contains four cascade binary classification losses while the latter contains a multi-class classification loss. With this loss function, multiple classifiers and bounding box regressions are learned jointly through backpropagation.

4 Experiments and Evaluations

4.1 Experimental setup

Datasets and Evaluation Metrics

We evaluated our approach on several public object detection datasets, including PASCAL VOC [1] , BSBDV 2017 [3], Caltech Pedestrian Benchmark [5] and CityPersons [6]. For evaluation, we used the standard average precision (AP) and mean average precision (mAP) scores with IoU thresholds at 0.5.

Pascal VOC. Pascal VOC involves 20 categories. VOC 2007 dataset consists of about 5k trainval images and 5k test images, while VOC 2012 dataset includes about 11k trainval images and 11k test images. Following the protocol in [8], we perform training on the union of VOC 2007 trainval and VOC 2012 trainval. The test is conducted on VOC 2007 test set.

BSBDV2017. The Birds Dataset of Shenzhen Bay in Distant View [3] is a great challenging dataset in wild scenes, consisting of 1,421 trainval images and 351 test images. BSBDV2017 contains three kinds of image resolutions, which are 27361824, 42882848 and 54723648 respectively. Size of birds varies greatly from 1830 to 1274*632.

Caltech Pedestrian Benchmark. The Caltech Pedestrian Benchmark [5] includes a total of 350,000 bounding boxes of pedestrians. Approximately 2,300 unique pedestrians were annotated in roughly 250,000 frames. Following the protocol in [40], one frame from every five frames of Caltech Benchmark and all frames of the ETH [41] and TUD-Brussels [42] are extracted as training data, which includes 27,021 images in total. 4,024 images in the standard test set are used for evaluation.

CityPersons. The CityPersons [6] consists of images recorded across 27 cities, 3 seasons, various weather conditions and more common crowds. It creates high quality bounding box annotations for pedestrians in 5000 images, which is a subset of the Cityscapes dataset [43]. 2975 images from train set and 500 images from val set are used for training and testing respectively.

Implementation Details

Faster R-CNN is taken as our baseline, where all parameters are set according to the original publication [4] if not specified. We initialize the backbone network using a VGG16 pre-trained model on ImageNet [44] while all new layers are initialized by drawing weights from a zero-mean Gaussian distribution with standard deviation 0.01. For training on Pascal VOC, we use a learning rate of 0.001 for 80k iterations and 0.0001 for 30k iterations. For training on the other datasets, we use a learning rate of 0.001 for 50k iterations and 0.0001 for 20k iterations. We trained our model in the end-to-end manner with Stochastic Gradient Descent (SGD), where the momentum is 0.9, and the weight decay is 0.0005. Our program is implemented by Tensorflow [45] on a GPU of GeForce GTX TITAN X.

4.2 Overall Performance

Performance on Pascal VOC benchmark

We compare our approach with several state-of-the-arts in this subsection. Results in terms of mean average precision (mAP) are shown in Table 1. Our model achieves the second best performance among all methods, which is 1.2% lower than that of RON [46] but 3.2% higher than that of baseline Faster R-CNN. Besides, it is happy to see that our method outperforms ION [19] with the same backbone network which used features from Conv3_3, Conv4_3 and Conv5_3 to leverage context and multi-scale knowledge for object detection. From the table, we can see that although C-RPNs is designed aiming to improve detection in real world with imbalance data, it gets competitive performance on Pascal VOC benchmark.

Performance on BSBDV 2017

Table 2 shows the comparisons of C-RPNs with state-of-the-arts on BSBDV 2017. As shown in Table 2, our method achieves the best performance and its average precision (AP) is 3.4% higher than the second best (FPN [13]). More specifically, the AP of C-RPNs is 70.3%, which obtains 11% performance gain compared with that of Faster R-CNN. It is noted that our C-RPNs gets slightly lower mAP than that of RON [46] on VOC 2007, but it outperforms RON by a margin of 12.3% on BSBDV 2017. Also, the AP of C-RPNs is 8.8% and 3.4% higher than that of R-FCN [9] and FPN [13] respectively. These results demonstrate that our C-RPNs is more competitive in object detection in real world.

Comparison with baseline Faster R-CNN on pedestrian datasets

Pedestrian datasets like Caltech pedestrian benchmark [5] and CityPersons [6] are more challenging then Pascal VOC, which are collected via monitoring cameras on realistic street scenes. Performances on these two datasets are helpful to verify the efficiency of our approach since the scales and occlusion of pedestrians are changed frequently. Table 3 shows the comparisons of our C-RPNs with the baseline Faster R-CNN on these pedestrian datasets. Our C-RPNs achieves average precision of 48.1% and 51.4% on Caltech pedestrian benchmark and CityPersons, bringing 4.1% and 2.3% performance gain upon baseline Faster R-CNN, respectively, which indicates its robustness in intricate realistic scenes.

4.3 Quantitive Examples

Qualitative Examples on wild bird detection

For visualization purpose, several examples of detection results on BSBDV 2017 are given in Figure 5. The rows from the top to the bottom are respectively expressed as the results of Faster R-CNN and C-RPNs. Detection boxes from detectors are marked red. For better observation, we marked boxes of miss detection cases in yellow. According to the ground truth, there are 46 and 22 birds in the top and bottom images, respectively. From the results, we can see that our method shows significantly improved recall for object detection in wild scenes, where 40 and 17 birds have been detected, respectively. Compared with the results detected with Faster R-CNN, our method brings 16 and 2 more birds detected in two images respectively. Meanwhile, dotted boxes in blue show samples are detected with more than one boxes, three in the top images and none in the bottom images, which indicates that our method is able to generate more precise bounding boxes.

Qualitative Examples on practical pedestrian detection

In Figure 6, our proposed approach is trained on Caltech Pedestrian Benchmark and tested in realistic environments with random pedestrian flows. We show some detection images with different shooting angles such as looking down and looking up or with poor illumination, which are collected in subway, park and campus. Compared with the results from Faster R-CNN, our method brings more true positive and less false positive detections in these images respectively. It is found that some hard samples are falsely detected as background from Faster R-CNN, while those are detected aright as pedestrians from C-RPNs. According to the results in Figure 6, our proposed C-RPNs can adapt to harsh and complex environments to provide high quality object detection in real world.

4.4 Improvement analysis on false detections

To further examine the improvement of our C-RPNs upon baseline Faster R-CNN, the analysis tools [47] upon Pascal VOC are employed to produce a detailed error analysis. In Pascal VOC, animal categories include ‘bird’, ‘cat’, ‘cow’, ‘dog’, ‘horse’, ‘sheep’ and ‘person’. ‘Plane’, ‘bicycle’, ‘boat’, ‘bus’, ‘car’, ‘motorbike’ and ‘train’ make up the vehicle categories. Figure 7 takes animals and vehicles as examples to show the frequency and impact on the performance of each type of false positive. As shown, C-RPNs reduces detection errors compared with Faster R-CNN when detecting both animals and vehicles. It is found that C-RPNs has less BG errors as well as Loc errors compared with the baseline, indicating that C-RPNs can classify and localize objects better because it mined hard samples during training and learned stronger classifiers. However, just like Faster R-CNN, detection results from C-RPNs have same confusions with similar object categories, partly because binary classifiers in cascade RPN module only indicate samples to be background or object, which has limited promotion on distinguishing the categories of an object.

Figure 8 visualizes the analysis of different bird characteristics on VOC2007 test set. Performance improvements on these characteristics are explicit since C-RPNs achieves higher average precision on all characteristics of Occlusion, Truncation, Bounding Box Area, Aspect Ratio, Viewpoint and Part Visibility. It is worth mentioning that when occlusion is High, C-RPNs can still recognize some birds while baseline Faster R-CNN detects nothing. Furthermore, extra annotations of seven categories(‘airplane’, ‘bicycle’, ‘bird’, ‘boat’, ‘cat’, ‘chair’, ‘table’) are created in [47] for evaluating robustness of detection approaches. Figure 9 provides a compact summary of the sensitivity to each characteristic and the potential impact of improving robustness on seven categories. Overall, our C-RPNs achieves higher normalize average precision than Faster R-CNN against all characteristics, indicating its robustness in various scenes. Moreover, sensitivity against all these characteristics are decreased, which verifies that C-RPNs realizes an all-sided improvements upon Faster R-CNN. On the other side, we can see that C-RPNs is sensitive to the bounding box size just like Faster R-CNN and there is still some room to improve.

4.5 Ablation Studies

In previous sections, we have shown the efficiency of C-RPNs on several datasets. To further evaluate the individual effect of components of our C-RPNs, we analyze the object detection performance affected by the cascade stages as well as feature chain and score chain. We use BSBDV 2017 in this section.

Effects of cascade stages

To learn the efficiency of our C-RPNs with different number of cascade stages, results are summarized in Table 4. We remove different stages of C-RPNs to demonstrate their individual effect. It can be seen that, with stage 3 and stage 4, C-RPNs achieves AP of 69.5% which already outperforms the baseline Faster R-CNN. Adding stage 2 and stage 1 yields AP of 69.9% and 70.3% respectively, and it brings 0.4% and 0.4% performance gain respectively. Finally, the 4-stage cascade RPNs achieves the best performance. These results validate that employing more cascade stages and classifiers in the C-RPNs benefits the detection performance.

Effects of feature chain and score chain

To learn the impact of feature chain and score chain more specifically, Table 5 shows the results of our C-RPNs with or without feature chain and score chain. We set the same parameters for C-RPNs with previous sections but control the usage of feature chain and score chain separately. As shown in Table 5, feature chain is found to be effective in C-RPNs, which brings 0.6% performance gain. When we adapt score chain but without feature chain, the AP is 0.4% higher, which illustrates the efficiency of using score chain as well. The adjustment boosts the performance by 0.9% while both feature chain and score chain are used. These results verify that using features and scores at previous stages as the prior knowledge for the latter stages promotes the final detection.

Selection of reject threshold and fusion rate

To find the best hyper parameters, empirical tests were conducted using different reject threshold r and fusion rate $\lambda_{f}$ on BSBDV 2017 through one-dimensional grid search. Figure 10 shows the impacts of these two factors. As shown, reject threshold r=0.99 achieved the best AP of 70.31% when the fusion rate was fixed at 0.1. We then fixed the reject threshold as 0.99 and applied a grid search by changing the fusion rate $\lambda_{f}$. From Figure 10, the best $\lambda_{f}$ is observed as 0.1 with the AP of 70.31%.

5 Conclusion

In this paper, we have constructed C-RPNs, an effective approach for object detection in real world. The essence of our C-RPNs lies in adopting a cascade structure of region proposal networks to discard easy samples during training and learn stronger classifiers. Moreover, a feature chain and a score chain at multiple stages have been proposed to help generating more discriminative representations for proposals. Finally, a loss function of cascade stages is designed to jointly learn cascade classifiers. Extensive experiments have been conducted to evaluate our C-RPNs on a common benchmark (Pascal VOC) and several challenging datasets collected in wild scenes or realistic traffic scenes. Our C-RPNs achieves competitive results compared with the current state-of-the-arts and outperforms the baseline Faster R-CNN by an obvious margin, demonstrating its efficacy for object detection in real world.