ScarfNet: Multi-scale Features with Deeply Fused and Redistributed Semantics for Enhanced Object Detection

TL;DR

ScarfNet enhances object detection by deeply fusing multi-scale features with biLSTM and redistributing semantics via channel attention, significantly improving performance on PASCAL VOC and COCO benchmarks.

Contribution

The paper introduces ScarfNet, a novel method that fuses multi-scale features using biLSTM and redistributes semantics with attention, outperforming existing feature pyramid approaches.

Findings

Outperforms existing feature pyramid methods

Achieves state-of-the-art results on PASCAL VOC

Improves baseline detectors like Faster R-CNN, SSD, RetinaNet

Abstract

Convolutional neural network (CNN) has led to significant progress in object detection. In order to detect the objects in various sizes, the object detectors often exploit the hierarchy of the multi-scale feature maps called feature pyramid, which is readily obtained by the CNN architecture. However, the performance of these object detectors is limited since the bottom-level feature maps, which experience fewer convolutional layers, lack the semantic information needed to capture the characteristics of the small objects. In order to address such problem, various methods have been proposed to increase the depth for the bottom-level features used for object detection. While most approaches are based on the generation of additional features through the top-down pathway with lateral connections, our approach directly fuses multi-scale feature maps using bidirectional long short term memory (biLSTM) in effort to generate deeply fused semantics. Then, the resulting semantic information is redistributed to the individual pyramidal feature at each scale through the channel-wise attention model. We integrate our semantic combining and attentive redistribution feature network (ScarfNet) with baseline object detectors, i.e., Faster R-CNN, single-shot multibox detector (SSD) and RetinaNet. Our experiments show that our method outperforms the existing feature pyramid methods as well as the baseline detectors and achieve the state of the art performances in the PASCAL VOC and COCO detection benchmarks.

Equations

$$X_l=B_l\left(X_{l-1}\right)$$

$$\text{Detection Outputs}=D_l(X_l),$$

$$X_l^{\prime }=L_l\left(X_l\right)\oplus T_l\left(X_{l+1}^{\prime }\right)$$

$$\text{Detection Outputs}=D_l(X_l^{\prime })$$

$$X_l^{\prime }$$

$$=X_l\oplus {\text{ArNet}}_l(\text{ScNet}(X_{n-k+1:n}))$$

$$\text{Detection Outputs}=D_l(X_l^{\prime })$$

$$X_{n-k+1:n}^f=\text{ScNet}(X_{n-k+1:n}),$$

$${\bar{X}}_l=GlobalAveragePooling(X_l)$$

$${\bar{X}}_{l-1}^f=GlobalAveragePooling(X_{l-1}^f)$$

$$i_l=\sigma \left(W_{xi}{\bar{X}}_l+W_{x^{f}i}{\bar{X}}_{l-1}^f+b_i\right)$$

$$f_l=\sigma \left(W_{xf}{\bar{X}}_l+W_{x^{f}f}{\bar{X}}_{l-1}^f+b_f\right)$$

$$o_l=\sigma \left(W_{xo}{\bar{X}}_l+W_{x^{f}o}{\bar{X}}_{l-1}^f+b_o\right)$$

$$G_l=\tanh \left(W_{xc}\ast X_l+W_{x^{f}c}\ast X_{l-1}^f+b_c\right)$$

$$C_t=X_l\circ C_{l-1}+i_l\circ G_l$$

$$X_l^f=o_l\circ \tanh \left(C_l\right),$$

$$X_l^{\prime }=X_l\oplus {\text{ArNet}}_l(X_{n-k+1:n}^f),$$

Full text

1 Introduction

Object detection refers to the task of deciding whether there are any instances of objects in the image and returns the estimate of the location and the category of the objects [17], [15]. Historically, object detection has been one of the most challenging computer vision problems. Recently, deep learning has led unprecedented advances in object detection techniques [15]. Convolutional neural networks (CNNs) can produce a hierarchy of abstract feature maps through a cascade of convolution operations followed by a nonlinear function. Using CNN as a backbone network, object detectors can effectively infer the location of the bounding box and the category of the instances based on the abstract feature maps. Various object detection network structures have been proposed to date. The CNN-based object detectors are roughly categorized into two groups: two-stage detectors and single-stage detectors. The two-stage detectors detect the objects using two separate subnetworks; 1) the region proposal network for finding the bounding boxes containing the object, and 2) the object classifier network for identifying the class of the objects and refining the bounding boxes. The well-known two-stage detectors include R-CNN [7], Fast R-CNN [6], Faster R-CNN [20], and Mask R-CNN [8]. Single-stage detectors directly estimate the bounding boxes and the object classes from the feature maps in one shot and include single-shot multibox detector (SSD) [16], YOLO [18], YOLOv2 [19], and RetinaNet [13].

Recent advances in object detection are achieved by the CNN’s capability to produce the abstract features containing strong semantic cues. The deeper the convolutional layers are, the higher the level of abstraction is for the resulting feature maps. As a result, the features produced at the end of the CNN pipeline (called top-level features) contain rich semantics but lack spatial resolution, whereas the features placed at the input layers (called bottom-level features) lack semantic information but have detailed spatial information. The hierarchy of such multiscale features constitutes so-called feature pyramids, which are used to detect the objects of different scales in many object detectors (e.g., SSD [16], MS-CNN [2], and RetinaNet [13]). The structure designed to use such feature pyramids for object detection is described in Fig. 1 (a). Note that the attributes of the large objects tend to be captured on the top-level features of small size while those of the small objects are well represented by the shallow bottom-level features of large size.

One limitation of the feature pyramid method is the disparity of the semantic information between the multiscale feature maps used for object detection. The bottom-level features are not deep enough to exhibit high-level semantics underlying in the objects and their surroundings. This results in the accuracy loss in detecting the small objects. In order to address this problem, several approaches, which have attempted to reduce the semantic gap between the pyramidal features in different scales, have been proposed. One notable direction is to provide the contextual information to the bottom-level features by generating the highly semantic features in the top-down pathway with latent connections. As illustrated in Fig. 1 (c), based on the top-level pyramidal feature obtained from the bottom-up network, the additional features are subsequently generated with increased depth and resolution. In order to avoid losing the spatial information, lateral connections are used to take the bottom-level features and combine them with the high-level semantic features. Various object detectors including DSSD [5], FPN [12], and StairNet [22] follow this principle, and significant improvement has been reported in terms of detection accuracy.

Our work is motivated by the observation that the capacity of the current architectures for generating top-down features might not be large enough to generate strong semantics for all scales. Thus, we propose a new architecture, which deeply fuses the multiscale features for enhanced object detection. The proposed feature pyramid method, referred to as semantic combining and attentive redistribution feature network (ScarfNet), combines the multiscale feature maps using the recurrent neural networks and then redistributes the fused semantics to each level, generating the new multiscale feature maps. The structure of ScarfNet is depicted in Fig. 2 (d). First, the bidirectional long short-term memory (biLSTM) model [24] is used to combine the multiscale pyramidal features. Although biLSTM is widely used to extract the temporal features from the sequential data, it can effectively combine the semantics in multiscale features. Our conjecture is that compared to the convolutive fusion methods, biLSTM requires significantly reduced number of weights due to parameter sharing, and the only relevant semantic information is selectively aggregated through the gating function of biLSTM. The fused feature maps are distributed through the channel-wise attention model, generating highly semantic features tailored for each pyramid scale. The final multiscale feature maps are used for object detection. Note that our framework can be readily applied to various feature pyramid-based CNN architectures, which require strong semantic information.

In the experiments, we integrate ScarfNet with the baseline detectors including Faster R-CNN [20], SSD [16], and RetinaNet [13]. The evaluation conducted over PASCAL VOC [4] and MS COCO [14] datasets shows that our method offers significant improvement over the baseline detectors as well as other competitive detectors in terms of detection accuracy. Our code will be made publicly available. The contributions of our paper are summarized as follows.

• •

We introduce a new deep architecture for closing the semantic gaps between the multiscale feature maps. The proposed ScarfNet generates new multiscale feature maps with deeply fused and redistributed semantics by using the combination of biLSTM and the channel-wise attention model.

• •

For the first time in the literature, the biLSTM is used to combine the multiscale features to incorporate strong semantics for feature pyramids. The biLSTM model can produce deeply fused semantic information using the recurrent connection over different pyramid scales. Furthermore, ScarfNet benefits from the selective information gating mechanism inherent in the biLSTM model. Due to parameter sharing, the overhead due to ScarfNet is small. In addition, ScarfNet is easy to train and is also end-to-end trainable.

2 Related Work

In this section, we review the basic object detectors and several existing feature pyramid methods used to decrease the semantic gap between the scales.

2.1 CNN-based Object Detectors

Recently, CNN has brought an order of magnitude performance improvement in object detection. Various CNN-based object detectors have been proposed to date. The current object detectors can be categorized into two groups: two-stage detectors and single-stage detectors. Two-stage detectors detect the objects in two steps; finding the region proposals based on the objectness of the regions and conducting the classification and bound regression for detected region proposals. R-CNN [7] is the first CNN-based detector where the traditional selective search is employed to find the region proposals, and CNN is applied to the image patch in each region proposal. Fast R-CNN [6] and Faster R-CNN [20] reduced the computation time of R-CNN by employing the region of interest (ROI) pooling for using full-image feature maps and replacing the selective search with the region proposal network (RPN). Single-stage detectors directly perform classification and box regression based on the feature maps. These detectors compute the confidence score on the object category and the regression results for the candidate boxes while sweeping the feature maps spatially. The well-known single-stage detectors include SSD [16], YOLO [18], and YOLOv2 [19]. Recently, RetinaNet [13] has achieved the-state-of-the-art performance using ResNet [9] as a backbone and various latest training tricks. Refer to [15] for the comprehensive review of contemporary object detectors.

2.2 Object Detectors Using Multiscale Features

Several object detectors including SSD [16] and RetinaNet [13] rely on hierarchical feature pyramids to detect the objects of various sizes (Fig. 1 (a)). One problem with using multiscale features directly produced by CNNs is the gap of the semantic information between them caused by the different depths of the layers passed by the input. Due to the relatively low level of abstraction for bottom-level features, detection accuracy for small objects is often limited. Fig. 1 (b), (c), and (d) describe the strategies proposed to overcome this problem. Fig. 1 (b) depicts the strategy of combining the multiscale features into the single high-resolution feature map with strong semantics. HyperNet [11] and ION [1] improved the performance of RPN by aggregating the hierarchical features with the appropriate resizing of the feature maps. Fig. 1 (c) shows the strategy of generating highly semantic features through the top-down pathway with lateral connections. Note that the semantic information is generated through the top-down connections while the detailed spatial information is provided through lateral connections. Several detectors based on this structure include DSSD [5], StairNet [22], TDM [21], FPN [12], and RefineDet [25]. DSSD [5] and StairNet [22] use the deconvolutional layer-based top-down connections for the SSD baseline [16]. TDM [21] employs the top-down structure specified for the RPN of the Faster R-CNN [20]. FPN [12] uses the simplified structure using 2x up-sampling and 1x1 convolution for top-down and lateral connections, respectively. RefineDet [25] employs two-step cascade regression for top-down connections.

3 Proposed Object Detector

In this section, we introduce the details of the proposed ScarfNet architecture.

3.1 Existing Feature Pyramid Methods

The feature pyramid-based object detectors performs detection based on the $k(>1)$ feature maps across different pyramid levels to detect the various sizes of objects. As shown in Fig. 1 (a), the baseline detectors use the feature map $X_{l}$ at the $l$th pyramidal level

[TABLE]

where $l=n-k+1,...,n$. Note that $X_{1:n-k}(=[X_{1},X_{2},...,X_{n-k}])$ are the feature maps produced by the backbone network, and $X_{n-k+1:n}$ are the bottom-up features from the subsequent convolutional layers. ${B}_{l}(\cdot)$ denotes the operation performed by the $l$th convolutional layer, and ${D}_{l}(\cdot)$ denotes the detection subnetwork that often applies a single 3x3 convolutional layer to produce the output of classification and box regression. Due to the different depths from the input to each pyramidal feature, shallow bottom-level features suffer from the lack of semantic information.

In order to reduce the semantic gap between different pyramid levels, several works proposed the top-down structure using lateral connections as illustrated in Fig. 1 (c). This structure propagates the high-level semantics from the top layers to the bottom layers with increased resolution while keeping the spatial high resolution through lateral connections. The $l$th feature map $X^{\prime}_{l}$ generated using this method is expressed as

[TABLE]

where $l=n-k+1,...,n$. Note that $L_{l}(\cdot)$ is the operation for the $l$th lateral connection, and $T_{l}(\cdot)$ is the operation for the $l$th top-down connection. The operator $\oplus$ represents the operation of combining two feature maps, e.g., channel-wise concatenation and addition. Different methods (e.g., DSSD [5], StairNet [22], TDM [21], FPN [12], and RefineDet [25]) employ the slightly different structures for $L_{l}(\cdot)$ and $T_{l}(\cdot)$. While these methods promote the abstraction level for pyramidal features, they still have some limitations. Since the top-down connection propagates the semantic information in a unidirectional way, the semantics are not evenly distributed among all pyramid levels. As a result, the semantic gap between the pyramidal features still remains. Next, such unilateral processing of the features has limited capacity to produce rich contextual information for increasing the semantic levels in all scales. In order to address these problems, we developed a new architecture that uses biLSTM to generate the deeply fused semantics through bi-lateral connections between all pyramid scales. In the following subsections, we present the details of our design.

3.2 ScarfNet: Overall Architecture

ScarfNet attempts to resolve the discrepancy of the semantic information in two steps: 1) combining the scattered semantic information using biLSTM and 2) redistributing the fused semantics back to each pyramid level using the channel-wise attention model. The overall architecture of ScarfNet is depicted in Fig. 2. Taking the $k$ pyramidal features $X_{n-k+1:n}$ as input, ScarfNet produces the new $l$th pyramidal feature map $X^{\prime}_{l}$ as

[TABLE]

[TABLE]

where $l=n-k+1,...,n$. As in (6), ScarfNet consists of two subnetworks: semantic combining network (ScNet) and attentive redistribution network (ArNet). First, ScNet merges the pyramidal features $X_{n-k+1:n}$ through biLSTM and produces the output features with the fused semantics. Second, ArNet collects the output features from biSLTM and applies the channel-wise attention model to produce highly semantic multiscale features, which are concatenated to the original pyramidal features. Finally, the resulting feature maps are individually processed by the detection subnetwork ${D}_{l}(\cdot)$ to produce the results for object detection.

3.3 Semantic Combining Network (ScNet)

The feature maps $X^{f}_{n-k+1:n}$ produced by ScNet are obtained as follows:

[TABLE]

where $X^{f}_{l}$ is the output feature map for the $l$th layer. Fig. 3 depicts the detailed structure of ScNet. ScNet uniformly fuses the semantics scattered in different pyramid levels using biLSTM. The biLSTM model can selectively fuse the contextual information in multiscale features through the gating function.

As shown in Fig. 3, ScNet consists of the matching block and the biLSTM block. The matching block first resizes the pyramidal features $X_{n-k+1:n}$ such that they have the same size as the largest pyramidal feature. Then, it adjusts the channel dimension of the input using the 1x1 convolutional layer. As a result, the matching block produces the feature maps of the same spatial and channel dimensions for biLSTM. Note that the resizing operation is performed by bilinear interpolation. The biLSTM model used in SCNet follows the structure of [23], which has significantly saved the computation time by using convolutional layers for input connection and computing the gating parameters based on the results of global average pooling. The operations performed by biLSTM in [23] are summarized as follows:

[TABLE]

where $\circ$ denotes the Hadamard product. The state update of biLSTM is conducted in both forward and backward directions. Note that we only provide the forward update, and the equations are similar for the backward update.

3.4 Attentive Redistribution Network (ArNet)

The ArNet aims to produce the high-level semantic feature map, which is concatenated with the original pyramidal feature map $X_{l}$ as follows:

[TABLE]

where operator $\oplus$ denotes the channel-wise concatenation. The detailed structure of ArNet is depicted in Fig. 4. ArNet concatenates the outputs $X^{f}_{n-k+1:n}$ of biLSTM and apply the channel-wise attention to them. The attention weights are obtained by constructing the 1x1 vector using global average pooling [10] and passing it through two fully connected layers followed by the sigmoid function. Note that this channel-wise attention model allows for selective propagation of the semantics to each pyramid level. Once the attention weights are applied, the matching block down-samples the resulting feature maps to the original size of the pyramidal features and applies 1x1 convolution to match the channel dimensions with those of the original pyramidal features. Finally, the output of the matching block is concatenated with the original feature $X_{l}$ to produce the highly semantic feature $X^{\prime}_{l}$.

4 Experiments

We evaluated the performance of the proposed ScarfNet model by comparing our detector with other multiscale detection methods and conducting an extensive performance analysis to understand the behavior of our architecture.

4.1 Experimental Setup

ScarfNet was applied to the baseline object detectors including Faster R-CNN [20], SSD [16], and RetinaNet [13]. In the case of Faster R-CNN and RetinaNet, we replaced the original FPN part with the feature generation by ScarfNet. We compared our method with the baselines, Faster R-CNN [20], SSD [16], and RetinaNet [13], as well as the other competitive algorithms including ION [1], R-FCN [3], DSSD [5], and StairNet [22]. We measured the mean average precision (mAP) in % on three widely used datasets for object detection benchmark: PASCAL VOC 2007, PASCAL VOC 2012 [4], and MS COCO [14].

4.2 Network Configuration

The advantage of ScarfNet is that there are not many hyperparameters to be determined. Note that the spatial dimensions of the feature maps are readily determined based on those of the baseline detectors. The channel dimensions of the intermediate feature maps are fixed over the pipeline between two matching blocks in ScNet and ArNet. Thus, we only need to determine this channel dimension. According to our empirical results, we set the channel dimension to 256.

4.3 Performance Evaluation

4.3.1 PASCAL VOC Results

Training on PASCAL VOC 2007 Dataset: The object detectors under consideration were trained with the VOC 2007 trainval and the VOC 2012 trainval sets and evaluated with the VOC 2007 test set. When ScarfNet was combined with the SSD baseline, we trained our model over 120k iterations ($\sim$240 epochs). We used a learning rate of $10^{-3}$ for the first $80$k iterations, $10^{-4}$ for the next $20$k iterations, and $10^{-5}$ for the last $20$k iterations. The mini-batch size was set to $32$, the momentum for the stochastic gradient descent (SGD) update was set to $0.9$, and the weight decay was set to $0.0005$. When our method was combined with the RetinaNet baseline, we used a learning rate of $5\times 10^{-3}$ for the first $60$k iterations, $5\times 10^{-4}$ for the next $20$k iterations, and $5\times 10^{-5}$ for the last $10$k iterations. Other parameters were equally set except for the weight decay of $0.0001$.

     Training on PASCAL VOC 2012 Dataset: The object detectors were trained with the VOC 2007 trainval, the VOC 2007 test and the VOC 2012 trainval sets and evaluated with the VOC 2012 test set. When our model was applied to the SSD baseline, a total of 200k iterations were run with the same training parameters as in the VOC 2007 case. Note that we used a learning rate of $10^{-3}$ for the first $120$k iterations, $10^{-4}$ for the next $40$k iterations, and $10^{-5}$ for the rest.

     Performance Comparison: Table 1 shows the mAP performance of the object detectors under comparison evaluated on the PASCAL VOC 2007 and 2012 test sets. For both PASCAL 2007 and 2012 cases, the semantic features generated by ScarfNet offer a significant performance gain over the baseline detectors. In the case of PASCAL VOC 2007, the proposed method achieves 1.9%, 1.8%, and 1.2% mAP gains over the SSD300, SSD512, and Faster R-CNN baselines, respectively. The proposed method also outperforms the RetinaNet baseline by 0.5%. Since the RetinaNet baseline employs the top-down structure based on FPN [12], we conclude that the features generated by our method are superior to those generated by FPN. Our object detector also achieves better performance than the other competing algorithms including, StairNet [22], DSSD [5], ION [1], R-FCN [3]. Although the detection accuracy with the PASCAL VOC 2012 dataset is slightly degraded compared to that with PASCAL VOC 2017, the tendency of the detection results observed for PASCAL VOC 2007 remains the same. Note that the proposed detector maintains the performance gain of 1.4% and 1.3% mAP over the SSD300 and SSD500 baselines, respectively.

4.3.2 COCO Results

Training: The object detectors under comparison were trained with the MS COCO trainval35k split [1] (union of 80k images from the training set and a random 35k subset of images from 40k image val split). The evaluation was performed using the MS COCO test-dev. To train the proposed structure based on RetinaNet [13], we used the learning rate of $10^{-2}$ for the first $60$k iterations, $10^{-3}$ for the next $20$k iterations, and $10^{-5}$ for the last $20$k iterations. The mini-batch size was set to $16$, the momentum was set to $0.9$, and the weight decay was set to $0.0001$.

     Performance comparison: Table 2 provides the detection accuracy of the algorithms tested on the MS COCO dataset. The experiment was conducted on various baseline detectors and feature pyramid modules. The proposed Scarf SSD513 and Scarf RetinaNet achieve the significant performance gain over the baselines. Our method beats the Faster R-CNN baseline [20] by 0.9% AP. Note also that the Scarf SSD513 achieves 1.3% performance gain over DSSD513 and the Scarf RetinaNet offers the 0.8% performance gain over the RetinaNet baseline [13].

4.4 Performance Analysis

4.4.1 Ablation Study

Benefits of biLSTM: It is worth investigating the effectiveness of biLSTM and channel-wise attention for fusing the multi-scale features. Table 3 shows how the performance of our method improves as we add bi-LSTM and channel-wise attention to the baseline one by one. We see that the biLSTM offers the 1.6% AP gain over the baseline and combination of biLSTM and channel-wise attention adds 1.9% AP gain. Table 3 also compares the different fusion strategies including the 1x1 convolutional layer, the top-down structure, and the unidirectional LSTM. Our biLSTM achieves better performance than the others; thus, parameter sharing, gating units, and bilateral processing of biLSTM effectively control high-level information to reduce the subtle semantic gap between the hierarchical features.

Network Parameter Search We need to determine the channel dimension of the intermediate feature maps. We should also determine whether the element-wise addition or channel-wise concatenation is better in combining the output of ScarfNet with the original feature pyramid. Table 4 shows the evaluation of the performance of our detector for various combinations of channel dimensions (64, 128, 256, 512 versus 1024) and feature combining strategies (element-wise addition versus channel-wise concatenation). According to Table 4, the combination of 512 channel dimensions with element-wise addition leads to the best detection accuracy. However, using 512 channels significantly increases the computational complexity of the entire network; thus, we chose 256 channel dimensions with channel-wise concatenation.

4.4.2 Feature Visualization

We investigated the effectiveness of ScarfNet via feature visualization. Fig. 5 compares the original pyramidal feature map $X_{1}$ of the largest size (middle row) with the semantic feature map $X^{\prime}_{1}$ from ScarfNet (bottom row). In order to obtain the heat map, we took the channel with the highest average activation in the spatial domain. Due to the lack of the semantic cues in the original feature map $X_{1}$, it often fails to activate on the objects properly. On the contrary, the feature map $X^{\prime}_{1}$ has strong activation on the entire region occupied by the objects, which would lead to an improvement in the overall detection performance.

5 Conclusions

In this study, we developed a deep architecture that generates multiscale features with strong semantics to reliably detect the objects in various sizes. Our ScarfNet method transforms the pyramidal features produced by the baseline detector into evenly abstract features. ScarfNet fuses the pyramidal features using biLSTM and distributes the semantics back to each multiscale feature. We verified through experiments conducted with PASCAL VOC and MS COCO datasets that the proposed ScarfNet method significantly increases the detection performance over the baseline detectors. Our object detector achieves the state-of-the-art performance on the PASCAL VOC and COCO benchmarks.