Res2Net: A New Multi-scale Backbone Architecture

TL;DR

Res2Net introduces a novel multi-scale feature representation block for CNNs, enhancing their ability to capture features at various scales within a single residual block, leading to improved performance across multiple vision tasks.

Contribution

The paper proposes the Res2Net block, enabling hierarchical multi-scale feature extraction within a single residual block, compatible with existing CNN architectures.

Findings

Consistent performance improvements on CIFAR-100 and ImageNet datasets.

Enhanced results in object detection, class activation mapping, and salient object detection.

Res2Net outperforms baseline models across various tasks.

Abstract

Representing features at multiple scales is of great importance for numerous vision tasks. Recent advances in backbone convolutional neural networks (CNNs) continually demonstrate stronger multi-scale representation ability, leading to consistent performance gains on a wide range of applications. However, most existing methods represent the multi-scale features in a layer-wise manner. In this paper, we propose a novel building block for CNNs, namely Res2Net, by constructing hierarchical residual-like connections within one single residual block. The Res2Net represents multi-scale features at a granular level and increases the range of receptive fields for each network layer. The proposed Res2Net block can be plugged into the state-of-the-art backbone CNN models, e.g., ResNet, ResNeXt, and DLA. We evaluate the Res2Net block on all these models and demonstrate consistent performance gains over baseline models on widely-used datasets, e.g., CIFAR-100 and ImageNet. Further ablation studies and experimental results on representative computer vision tasks, i.e., object detection, class activation mapping, and salient object detection, further verify the superiority of the Res2Net over the state-of-the-art baseline methods. The source code and trained models are available on https://mmcheng.net/res2net/.

Figures

Tables

Table 1. TABLE I: Top-1 and Top-5 test error on the ImageNet dataset.

	top-1 err. ($\%$)	top-5 err. ($\%$)
ResNet-50 [27]	23.85	7.13
Res2Net-50	22.01	6.15
InceptionV3 [62]	22.55	6.44
Res2Net-50-299	21.41	5.88
ResNeXt-50 [68]	22.61	6.50
Res2NeXt-50	21.76	6.09
DLA-60 [72]	23.32	6.60
Res2Net-DLA-60	21.53	5.80
DLA-X-60 [72]	22.19	6.13
Res2NeXt-DLA-60	21.55	5.86
SENet-50 [30]	23.24	6.69
SE-Res2Net-50	21.56	5.94
bLResNet-50 [5]	22.41	-
bLRes2Net-50	21.68	6.00
Res2Net-v1b-50	19.73	4.96
Res2Net-v1b-101	18.77	4.64
Res2Net-200-SSLD [50]	14.87	-

Table 2. TABLE II: Top-1 and Top-5 test error ( % percent \% ) of deeper networks on the ImageNet dataset.

	top-1 err.	top-5 err.
DenseNet-161 [31]	22.35	6.20
ResNet-101 [27]	22.63	6.44
Res2Net-101	20.81	5.57

Table 3. TABLE III: Top-1 and Top-5 test error ( % percent \% ) of Res2Net-50 with different scales on the ImageNet dataset. Parameter w 𝑤 w is the width of filters, and s 𝑠 s is the number of scale, as described in Equation ( 4 ).

	Setting	FLOPs	Runtime	top-1 err.	top-5 err.
ResNet-50	64w	4.2G	149ms	23.85	7.13
Res2Net-50 ( Preserved complexity)	48w$\times$2s	4.2G	148ms	22.68	6.47
	26w$\times$4s	4.2G	153ms	22.01	6.15
	14w$\times$8s	4.2G	172ms	21.86	6.14
Res2Net-50 ( Increased complexity)	26w$\times$4s	4.2G	-	22.01	6.15
	26w$\times$6s	6.3G	-	21.42	5.87
	26w$\times$8s	8.3G	-	20.80	5.63
Res2Net-50-L	18w$\times$4s	2.9G	106ms	22.92	6.67

Table 4. TABLE IV: Top-1 test error ( % percent \% ) and model size on the CIFAR-100 dataset. Parameter c 𝑐 c indicates the value of cardinality, and w 𝑤 w is the width of filters.

	Params	top-1 err.
Wide ResNet [74]	36.5M	20.50
ResNeXt-29, 8c$\times$64w [68] (base)	34.4M	17.90
ResNeXt-29, 16c$\times$64w [68]	68.1M	17.31
DenseNet-BC (k = 40) [31]	25.6M	17.18
Res2NeXt-29, 6c$\times$24w$\times$4s	24.3M	16.98
Res2NeXt-29, 8c$\times$25w$\times$4s	33.8M	16.93
Res2NeXt-29, 6c$\times$24w$\times$6s	36.7M	16.79
ResNeXt-29, 8c$\times$64w-SE [30]	35.1M	16.77
Res2NeXt-29, 6c$\times$24w$\times$4s-SE	26.0M	16.68
Res2NeXt-29, 8c$\times$25w$\times$4s-SE	34.0M	16.64
Res2NeXt-29, 6c$\times$24w$\times$6s-SE	36.9M	16.56

Table 5. TABLE V: Object detection results on the PASCAL VOC07 and COCO datasets, measured using AP ( % percent \% ) and AP@IoU=0.5 ( % percent \% ). The Res2Net has similar complexity compared with its counterparts.

Dataset	Backbone	AP	AP@IoU=0.5
VOC07	ResNet-50	72.1	-
	Res2Net-50	74.4	-
COCO	ResNet-50	31.1	51.4
	Res2Net-50	33.7	53.6
	Res2Net-v1b-101	43.0	63.5

Table 6. TABLE VI: Average Precision (AP) and Average Recall (AR) of object detection with different sizes on the COCO dataset.

		Object size
		Small	Medium	Large	All
ResNet-50	AP ($\%$)	13.5	35.4	46.2	31.1
Res2Net-50		14.0	38.3	51.1	33.7
Improve.		+0.5	+2.9	+4.9	+2.6
ResNet-50	AR ($\%$)	21.8	48.6	61.6	42.8
Res2Net-50		23.2	51.1	65.3	45.0
Improve.		+1.4	+2.5	+3.7	+2.2

Table 7. TABLE VII: Performance of semantic segmentation on PASCAL VOC12 val set using Res2Net-50 with different scales. The Res2Net has similar complexity compared with its counterparts.

Backbone	Setting	Mean IoU ($\%$)
ResNet-50	64w	77.7
Res2Net-50	48w$\times$2s	78.2
	26w$\times$4s	79.2
	18w$\times$6s	79.1
	14w$\times$8s	79.0
ResNet-101	64w	79.0
Res2Net-101	26w$\times$4s	80.2

Table 8. TABLE VIII: Performance of instance segmentation on the COCO dataset using Res2Net-50 with different scales. The Res2Net has similar complexity compared with its counterparts.

Backbone	Setting	$AP$	$A{P}_{50}$	$A{P}_{75}$	$A{P}_S$	$A{P}_M$	$A{P}_L$
ResNet-50	64w	33.9	55.2	36.0	14.8	36.0	50.9
Res2Net-50	48w$\times$2s	34.2	55.6	36.3	14.9	36.8	50.9
	26w$\times$4s	35.6	57.6	37.6	15.7	37.9	53.7
	18w$\times$6s	35.7	57.5	38.1	15.4	38.1	53.7
	14w$\times$8s	35.3	57.0	37.5	15.6	37.5	53.4
Res2Net-v1b-101	64w	38.7	61.0	41.4	20.6	42.0	53.2

Table 9. TABLE IX: Salient object detection results on different datasets, measured using F-measure and Mean Absolute Error (MAE). The Res2Net has similar complexity compared with its counterparts.

Dataset	Backbone	F-measure$\uparrow$	MAE $\downarrow$
ECSSD	ResNet-50	0.910	0.065
	Res2Net-50	0.926	0.056
PASCAL-S	ResNet-50	0.823	0.105
	Res2Net-50	0.841	0.099
HKU-IS	ResNet-50	0.894	0.058
	Res2Net-50	0.905	0.050
DUT-OMRON	ResNet-50	0.748	0.092
	Res2Net-50	0.800	0.071

Table 10. TABLE X: Performance of key-points estimation on the COCO validation set. The Res2Net has similar complexity compared with its counterparts.

Backbone	$AP$	$A{P}_{50}$	$A{P}_{75}$	$A{P}_M$	$A{P}_L$
ResNet-50	70.4	88.6	78.3	67.1	77.2
Res2Net-50	71.5	89.0	79.3	68.2	78.4
ResNet-101	71.4	89.3	79.3	68.1	78.1
Res2Net-101	72.2	89.4	79.8	68.9	79.2
Res2Net-v1b-50	72.2	89.5	79.7	68.5	79.4
Res2Net-v1b-101	73.0	89.5	80.3	69.5	80.0

Equations

$$\displaystyle\mathbf{y}_{i}=\left\{\begin{array}[]{lll}\mathbf{x}_{i}&i=1;\\ \mathbf{K}_{i}(\mathbf{x}_{i})&i=2;\\ \mathbf{K}_{i}(\mathbf{x}_{i}+\mathbf{y}_{i-1})&2<i\leqslant s.\end{array}\right.$$

Full text

1 Introduction

Visual patterns occur at multi-scales in natural scenes as shown in Fig. 1. First, objects may appear with different sizes in a single image, *e.g., *the sofa and cup are of different sizes. Second, the essential contextual information of an object may occupy a much larger area than the object itself. For instance, we need to rely on the big table as context to better tell whether the small black blob placed on it is a cup or a pen holder. Third, perceiving information from different scales is essential for understanding parts as well as objects for tasks such as fine-grained classification and semantic segmentation. Thus, it is of critical importance to design good features for multi-scale stimuli for visual cognition tasks, including image classification [33], object detection [53], attention prediction [55], target tracking [76], action recognition [56], semantic segmentation [6], salient object detection [29, 2], object proposal [53, 12], skeleton extraction [80], stereo matching [52], and edge detection [69, 45].

Unsurprisingly, multi-scale features have been widely used in both conventional feature design [1, 48] and deep learning [61, 10]. Obtaining multi-scale representations in vision tasks requires feature extractors to use a large range of receptive fields to describe objects/parts/context at different scales. Convolutional neural networks (CNNs) naturally learn coarse-to-fine multi-scale features through a stack of convolutional operators. Such inherent multi-scale feature extraction ability of CNNs leads to effective representations for solving numerous vision tasks. How to design a more efficient network architecture is the key to further improving the performance of CNNs.

In the past few years, several backbone networks, *e.g., *[33, 57, 61, 27, 31, 15, 68, 10, 72, 30], have made significant advances in numerous vision tasks with state-of-the-art performance. Earlier architectures such as AlexNet [33] and VGGNet [57] stack convolutional operators, making the data-driven learning of multi-scale features feasible. The efficiency of multi-scale ability was subsequently improved by using conv layers with different kernel size (*e.g., *InceptionNets [61, 62, 60]), residual modules (*e.g., *ResNet [27]), shortcut connections (*e.g., *DenseNet [31]), and hierarchical layer aggregation (*e.g., *DLA [72]). The advances in backbone CNN architectures have demonstrated a trend towards more effective and efficient multi-scale representations.

In this work, we propose a simple yet efficient multi-scale processing approach. Unlike most existing methods that enhance the layer-wise multi-scale representation strength of CNNs, we improve the multi-scale representation ability at a more granular level. Different from some concurrent works [9, 5, 11] that improve the multi-scale ability by utilizing features with different resolutions, the multi-scale of our proposed method refers to the multiple available receptive fields at a more granular level. To achieve this goal, we replace the $3\times 3$ filters111 Convolutional operators and filters are used interchangeably. of $n$ channels, with a set of smaller filter groups, each with $w$ channels (without loss of generality we use $n=s\times w$). As shown in Fig. 2, these smaller filter groups are connected in a hierarchical residual-like style to increase the number of scales that the output features can represent. Specifically, we divide input feature maps into several groups. A group of filters first extracts features from a group of input feature maps. Output features of the previous group are then sent to the next group of filters along with another group of input feature maps. This process repeats several times until all input feature maps are processed. Finally, feature maps from all groups are concatenated and sent to another group of $1\times 1$ filters to fuse information altogether. Along with any possible path in which input features are transformed to output features, the equivalent receptive field increases whenever it passes a $3\times 3$ filter, resulting in many equivalent feature scales due to combination effects.

The Res2Net strategy exposes a new dimension, namely scale (the number of feature groups in the Res2Net block), as an essential factor in addition to existing dimensions of depth [57], width222 Width refers to the number of channels in a layer as in [74]., and cardinality [68]. We state in Sec. 4.4 that increasing scale is more effective than increasing other dimensions.

Note that the proposed approach exploits the multi-scale potential at a more granular level, which is orthogonal to existing methods that utilize layer-wise operations. Thus, the proposed building block, namely Res2Net module, can be easily plugged into many existing CNN architectures. Extensive experimental results show that the Res2Net module can further improve the performance of state-of-the-art CNNs, *e.g., *ResNet [27], ResNeXt [68], and DLA [72].

2 Related Work

2.1 Backbone Networks

Recent years have witnessed numerous backbone networks [33, 57, 61, 27, 31, 15, 68, 72], achieving state-of-the-art performance in various vision tasks with stronger multi-scale representations. As designed, CNNs are equipped with basic multi-scale feature representation ability since the input information follows a fine-to-coarse fashion. The AlexNet [33] stacks filters sequentially and achieves significant performance gain over traditional methods for visual recognition. However, due to the limited network depth and kernel size of filters, the AlexNet has only a relatively small receptive field. The VGGNet [57] increases the network depth and uses filters with smaller kernel size. A deeper structure can expand the receptive fields, which is useful for extracting features from a larger scale. It is more efficient to enlarge the receptive field by stacking more layers than using large kernels. As such, the VGGNet provides a stronger multi-scale representation model than AlexNet, with fewer parameters. However, both AlexNet and VGGNet stack filters directly, which means each feature layer has a relatively fixed receptive field.

Network in Network (NIN) [38] inserts multi-layer perceptrons as micro-networks into the large network to enhance model discriminability for local patches within the receptive field. The 1 $\times$ 1 convolution introduced in NIN has been a popular module to fuse features. The GoogLeNet [61] utilizes parallel filters with different kernel sizes to enhance the multi-scale representation capability. However, such capability is often limited by the computational constraints due to its limited parameter efficiency. The Inception Nets [62, 60] stack more filters in each path of the parallel paths in the GoogLeNet to further expand the receptive field. On the other hand, the ResNet [27] introduces short connections to neural networks, thereby alleviating the gradient vanishing problem while obtaining much deeper network structures. During the feature extraction procedure, short connections allow different combinations of convolutional operators, resulting in a large number of equivalent feature scales. Similarly, densely connected layers in the DenseNet [31] enable the network to process objects in a very wide range of scales. DPN [10] combines the ResNet with DenseNet to enable feature re-usage ability of ResNet and the feature exploration ability of DenseNet. The recently proposed DLA [72] method combines layers in a tree structure. The hierarchical tree structure enables the network to obtain even stronger layer-wise multi-scale representation capability.

2.2 Multi-scale Representations for Vision Tasks

Multi-scale feature representations of CNNs are of great importance to a number of vision tasks including object detection [53], face analysis [4, 51], edge detection [45], semantic segmentation [6], salient object detection [42, 78], and skeleton detection [80], boosting the model performance of those fields.

2.2.1 Object detection.

Effective CNN models need to locate objects of different scales in a scene. Earlier works such as the R-CNN [22] mainly rely on the backbone network, *i.e., *VGGNet [57], to extract features of multiple scales. He et al. propose an SPP-Net approach [26] that utilizes spatial pyramid pooling after the backbone network to enhance the multi-scale ability. The Faster R-CNN method [53] further proposes the region proposal networks to generate bounding boxes with various scales. Based on the Faster R-CNN, the FPN [39] approach introduces feature pyramid to extract features with different scales from a single image. The SSD method [44] utilizes feature maps from different stages to process visual information at different scales.

2.2.2 Semantic segmentation.

Extracting essential contextual information of objects requires CNN models to process features at various scales for effective semantic segmentation. Long et al. [47] propose one of the earliest methods that enables multi-scale representations of the fully convolutional network (FCN) for semantic segmentation task. In DeepLab, Chen et al. [6, 7] introduces cascaded atrous convolutional module to expand the receptive field further while preserving spatial resolutions. More recently, global context information is aggregated from region-based features via the pyramid pooling scheme in the PSPNet [77].

2.2.3 Salient object detection.

Precisely locating the salient object regions in an image requires an understanding of both large-scale context information for the determination of object saliency, and small-scale features to localize object boundaries accurately [79]. Early approaches [3] utilize handcrafted representations of global contrast [13] or multi-scale region features [64]. Li et al.[34] propose one of the earliest methods that enables multi-scale deep features for salient object detection. Later, multi-context deep learning [81] and multi-level convolutional features [75] are proposed for improving salient object detection. More recently, Hou et al.[29] introduce dense short connections among stages to provide rich multi-scale feature maps at each layer for salient object detection.

2.3 Concurrent Works

Recently, there are some concurrent works aiming at improving the performance by utilizing the multi-scale features  [9, 5, 11, 59]. Big-Little Net [5] is a multi-branch network composed of branches with different computational complexity. Octave Conv [9] decomposes the standard convolution into two resolutions to process features at different frequencies. MSNet [11] utilizes a high-resolution network to learn high-frequency residuals by using the up-sampled low-resolution features learned by a low-resolution network. Other than the low-resolution representations in current works, the HRNet [58, 59] introduces high-resolution representations in the network and repeatedly performs multi-scale fusions to strengthen high-resolution representations. One common operation in  [9, 5, 11, 58, 59] is that they all use pooling or up-sample to re-size the feature map to $2^{n}$ times of the original scale to save the computational budget while maintaining or even improving performance. While in the Res2Net block, the hierarchical residual-like connections within a single residual block module enable the variation of receptive fields at a more granular level to capture details and global features. Experimental results show that Res2Net module can be integrated with those novel network designs to further boost the performance.

3 Res2Net

3.1 Res2Net Module

The bottleneck structure shown in Fig. 2(a) is a basic building block in many modern backbone CNNs architectures, *e.g., *ResNet [27], ResNeXt [68], and DLA [72]. Instead of extracting features using a group of $3\times 3$ filters as in the bottleneck block, we seek alternative architectures with stronger multi-scale feature extraction ability, while maintaining a similar computational load. Specifically, we replace a group of $3\times 3$ filters with smaller groups of filters, while connecting different filter groups in a hierarchical residual-like style. Since our proposed neural network module involves residual-like connections within a single residual block, we name it Res2Net.

Fig. 2 shows the differences between the bottleneck block and the proposed Res2Net module. After the $1\times 1$ convolution, we evenly split the feature maps into $s$ feature map subsets, denoted by $\mathbf{x}_{i}$, where $i\in\{1,2,...,s\}$. Each feature subset $\mathbf{x}_{i}$ has the same spatial size but $1/s$ number of channels compared with the input feature map. Except for $\mathbf{x}_{1}$, each $\mathbf{x}_{i}$ has a corresponding $3\times 3$ convolution, denoted by $\mathbf{K}_{i}()$. We denote by $\mathbf{y}_{i}$ the output of $\mathbf{K}_{i}()$. The feature subset $\mathbf{x}_{i}$ is added with the output of $\mathbf{K}_{i-1}()$, and then fed into $\mathbf{K}_{i}()$. To reduce parameters while increasing $s$, we omit the $3\times 3$ convolution for $\mathbf{x}_{1}$. Thus, $\mathbf{y}_{i}$ can be written as:

[TABLE]

Notice that each $3\times 3$ convolutional operator $\mathbf{K}_{i}()$ could potentially receive feature information from all feature splits $\{\mathbf{x}_{j},j\leq i\}$. Each time a feature split $\mathbf{x}_{j}$ goes through a $3\times 3$ convolutional operator, the output result can have a larger receptive field than $\mathbf{x}_{j}$. Due to the combinatorial explosion effect, the output of the Res2Net module contains a different number and different combination of receptive field sizes/scales.

In the Res2Net module, splits are processed in a multi-scale fashion, which is conducive to the extraction of both global and local information. To better fuse information at different scales, we concatenate all splits and pass them through a $1\times 1$ convolution. The split and concatenation strategy can enforce convolutions to process features more effectively. To reduce the number of parameters, we omit the convolution for the first split, which can also be regarded as a form of feature reuse.

In this work, we use $s$ as a control parameter of the scale dimension. Larger $s$ potentially allows features with richer receptive field sizes to be learnt, with negligible computational/memory overheads introduced by concatenation.

3.2 Integration with Modern Modules

Numerous neural network modules have been proposed in recent years, including cardinality dimension introduced by Xie et al.[68], as well as squeeze and excitation (SE) block presented by Hu et al.[30]. The proposed Res2Net module introduces the scale dimension that is orthogonal to these improvements. As shown in Fig. 3, we can easily integrate the cardinality dimension  [68] and the SE block [30] with the proposed Res2Net module.

3.2.1 Dimension cardinality.

The dimension cardinality indicates the number of groups within a filter [68]. This dimension changes filters from single-branch to multi-branch and improves the representation ability of a CNN model. In our design, we can replace the 3 $\times$ 3 convolution with the 3 $\times$ 3 group convolution, where $c$ indicates the number of groups. Experimental comparisons between the scale dimension and cardinality are presented in Sec. 4.2 and Sec. 4.4.

3.2.2 SE block.

A SE block adaptively re-calibrates channel-wise feature responses by explicitly modelling inter-dependencies among channels [30]. Similar to [30], we add the SE block right before the residual connections of the Res2Net module. Our Res2Net module can benefit from the integration of the SE block, which we have experimentally demonstrated in Sec. 4.2 and Sec. 4.3.

3.3 Integrated Models

Since the proposed Res2Net module does not have specific requirements of the overall network structure and the multi-scale representation ability of the Res2Net module is orthogonal to the layer-wise feature aggregation models of CNNs, we can easily integrate the proposed Res2Net module into the state-of-the-art  models, such as ResNet [27], ResNeXt [68], DLA [72] and Big-Little Net [5]. The corresponding models are referred to as Res2Net, Res2NeXt, Res2Net-DLA, and bLRes2Net-50, respectively.

The proposed scale dimension is orthogonal to the cardinality [68] dimension and width [27] dimension of prior work. Thus, after the scale is set, we adjust the value of cardinality and width to maintain the overall model complexity similar to its counterparts. We do not focus on reducing the model size in this work since it requires more meticulous designs such as depth-wise separable convolution [49], model pruning [23], and model compression [14].

For experiments on the ImageNet [54] dataset, we mainly use the ResNet-50 [27], ResNeXt-50 [68], DLA-60 [72], and bLResNet-50 [5] as our baseline models. The complexity of the proposed model is approximately equal to those of the baseline models, whose number of parameters is around $25M$ and the number of FLOPs for an image of $224\times 224$ pixels is around $4.2G$ for 50-layer networks. For experiments on the CIFAR [32] dataset, we use the ResNeXt-29, 8c$\times$64w [68] as our baseline model. Empirical evaluations and discussions of the proposed models with respect to model complexity are presented in Sec. 4.4.

4 Experiments

4.1 Implementation Details

We implement the proposed models using the Pytorch framework. For fair comparisons, we use the Pytorch implementation of ResNet [27], ResNeXt [68], DLA [72] as well as bLResNet-50 [5], and only replace the original bottleneck block with the proposed Res2Net module. Similar to prior work, on the ImageNet dataset [54], each image is of 224$\times$224 pixels randomly cropped from a re-sized image. We use the same data argumentation strategy as [27, 62]. Similar to [27], we train the network using SGD with weight decay 0.0001, momentum 0.9, and a mini-batch of 256 on 4 Titan Xp GPUs. The learning rate is initially set to 0.1 and divided by 10 every 30 epochs.

All models for the ImageNet, including the baseline and proposed models, are trained for 100 epochs with the same training and data argumentation strategy. For testing, we use the same image cropping method as [27]. On the CIFAR dataset, we use the implementation of ResNeXt-29 [68]. For all tasks, we use the original implementations of baselines and only replace the backbone model with the proposed Res2Net.

4.2 ImageNet

We conduct experiments on the ImageNet dataset [54], which contains 1.28 million training images and 50k validation images from 1000 classes. We construct the models with approximate 50 layers for performance evaluation against the state-of-the-art methods. More ablation studies are conducted on the CIFAR dataset.

4.2.1 Performance gain.

Table I shows the top-1 and top-5 test error on the ImageNet dataset. For simplicity, all Res2Net models in Table I have the scale $s=4$. The Res2Net-50 has an improvement of 1.84$\%$ on top-1 error over the ResNet-50. The Res2NeXt-50 achieves a 0.85$\%$ improvement in terms of top-1 error over the ResNeXt-50. Also, the Res2Net-DLA-60 outperforms the DLA-60 by 1.27$\%$ in terms of top-1 error. The Res2NeXt-DLA-60 outperforms the DLA-X-60 by 0.64$\%$ in terms of top-1 error. The SE-Res2Net-50 has an improvement of 1.68$\%$ over the SENet-50. bLRes2Net-50 has an improvement of 0.73$\%$ in terms of top-1 error over the bLResNet-50. The Res2Net module further enhances the multi-scale ability of bLResNet at a granular level even bLResNet is designed to utilize features with different scales as discussed in Sec. 2.3. Note that the ResNet [27], ResNeXt [68], SE-Net [30], bLResNet [5], and DLA [72] are the state-of-the-art CNN models. Compared with these strong baselines, models integrated with the Res2Net module still have consistent performance gains.

We also compare our method against the InceptionV3 [62] model, which utilizes parallel filters with different kernel combinations. For fair comparisons, we use the ResNet-50 [27] as the baseline model and train our model with the input image size of 299$\times$299 pixels, as used in the InceptionV3 model. The proposed Res2Net-50-299 outperforms InceptionV3 by 1.14$\%$ on top-1 error. We conclude that the hierarchical residual-like connection of the Res2Net module is more effective than the parallel filters of InceptionV3 when processing multi-scale information. While the combination pattern of filters in InceptionV3 is dedicatedly designed, the Res2Net module presents a simple but effective combination pattern.

4.2.2 Going deeper with Res2Net.

Deeper networks have been shown to have stronger representation capability [27, 68] for vision tasks. To validate our model with greater depth, we compare the classification performance of the Res2Net and the ResNet, both with 101 layers. As shown in Table II, the Res2Net-101 achieves significant performance gains over the ResNet-101 with 1.82$\%$ in terms of top-1 error. Note that the Res2Net-50 has the performance gain of 1.84$\%$ in terms of top-1 error over the ResNet-50. These results show that the proposed module with additional dimension scale can be integrated with deeper models to achieve better performance. We also compare our method with the DenseNet [31]. Compared with the DenseNet-161, the best performing model of the officially provided DenseNet family, the Res2Net-101 has an improvement of 1.54$\%$ in terms of top-1 error.

4.2.3 Effectiveness of scale dimension.

To validate our proposed dimension scale, we experimentally analyze the effect of different scales. As shown in  Table III, the performance increases with the increase of scale. With the increase of scale, the Res2Net-50 with 14w$\times$8s achieves performance gains over the ResNet-50 with 1.99$\%$ in terms of top-1 error. Note that with the preserved complexity, the width of $\mathbf{K}_{i}()$ decreases with the increase of scale. We further evaluate the performance gain of increasing scale with increased model complexity. The Res2Net-50 with 26w$\times$8s achieves significant performance gains over the ResNet-50 with 3.05$\%$ in terms of top-1 error. A Res2Net-50 with 18w$\times$4s also outperforms the ResNet-50 by 0.93$\%$ in terms of top-1 error with only 69$\%$ FLOPs. Table III shows the Runtime under different scales, which is the average time to infer the ImageNet validation set with the size of 224 $\times$ 224. Although the feature splits $\{\mathbf{y}_{i}\}$ need to be computed sequentially due to hierarchical connections, the extra run-time introduced by Res2Net module can often be ignored. Since the number of available tensors in a GPU is limited, there are typically sufficient parallel computations within a single GPU clock period for the typical setting of Res2Net, *i.e., *$s=4$.

4.2.4 Stronger representation with ResNet.

To further explore the multi-scale representation ability of Res2Net, we follow the ResNet v1d [28] to modify Res2Net, and train the model with data augmentation techniques *i.e., *CutMix [73]. The modified version of Res2Net, namely Res2Net v1b, greatly improve the classification performance on ImageNet as shown in Table I. Res2Net v1b further improve the model performance on downstream tasks. We show the performance of Res2Net v1b on object detection, instance segmentation, key-points estimation in Table V,  Table VIII, and Table X, respectively.

The stronger multi-scale representation of Res2Net has been verified on many downstream tasks *i.e., *vectorized road extraction [63], object detection [35], weakly supervised semantic segmentation [46], salient object detection [21], interactive image segmentation [41], video recognition [37], concealed object detection [18], and medical segmentation [66, 19, 20]. Semi-supervised knowledge distillation solution [50] can also be applied to Res2Net, to achieve the 85.13$\%$ top.1 acc. on ImageNet.

4.3 CIFAR

We also conduct some experiments on the CIFAR-100 dataset [32], which contains 50k training images and 10k testing images from 100 classes. The ResNeXt-29, 8c$\times$64w [68] is used as the baseline model. We only replace the original basic block with our proposed Res2Net module while keeping other configurations unchanged. Table IV shows the top-1 test error and model size on the CIFAR-100 dataset. Experimental results show that our method surpasses the baseline and other methods with fewer parameters. Our proposed  Res2NeXt-29, 6c$\times$24w$\times$6s outperforms the baseline by 1.11$\%$. Res2NeXt-29, 6c$\times$24w$\times$4s even outperforms the ResNeXt-29, 16c$\times$64w with only 35$\%$ parameters. We also achieve better performance with fewer parameters, compared with DenseNet-BC ($k=40$). Compared with Res2NeXt-29, 6c$\times$24w$\times$4s, Res2NeXt-29, 8c$\times$25w$\times$4s achieves a better result with more width and cardinality, indicating that the dimension scale is orthogonal to dimension width and cardinality. We also integrate the recently proposed SE block into our structure. With fewer parameters, our method still outperforms the ResNeXt-29, 8c$\times$64w-SE baseline.

4.4 Scale Variation

Similar to Xie et al.[68], we evaluate the test performance of the baseline model by increasing different CNN dimensions, including scale (Equation (4)), cardinality [68], and depth [57]. While increasing model capacity using one dimension, we fix all other dimensions. A series of networks are trained and evaluated under these changes. Since [68] has already shown that increasing cardinality is more effective than increasing width, we only compare the proposed dimension scale with cardinality and depth.

Fig. 5 shows the test precision on the CIFAR-100 dataset with regard to the model size. The depth, cardinality, and scale of the baseline model are $29,6$ and $1$, respectively. Experimental results suggest that scale is an effective dimension to improve model performance, which is consistent with what we have observed on the ImageNet dataset in Sec. 4.2. Moreover, increasing scale is more effective than other dimensions, resulting in quicker performance gains. As described in  Equation (4) and Fig. 2, for the case of scale $s=2$, we only increase the model capacity by adding more parameters of $1\times 1$ filters. Thus, the model performance of $s=2$ is slightly worse than that of increasing cardinality. For $s=3,4$, the combination effects of our hierarchical residual-like structure produce a rich set of equivalent scales, resulting in significant performance gains. However, the models with scale 5 and 6 have limited performance gains, about which we assume that the image in the CIFAR dataset is too small (32$\times$32) to have many scales.

4.5 Class Activation Mapping

To understand the multi-scale ability of the Res2Net, we visualize the class activation mapping (CAM) using Grad-CAM [55], which is commonly used to localize the discriminative regions for image classification. In the visualization examples shown in Fig. 4, stronger CAM areas are covered with lighter colors. Compared with ResNet, the Res2Net based CAM results have more concentrated activation maps on small objects such as ‘baseball’ and ‘penguin’. Both methods have similar activation maps on the middle size objects, such as ‘ice cream’. Due to stronger multi-scale ability, the Res2Net has activation maps that tend to cover the whole object on big objects such as ‘bulbul’, ‘mountain dog’, ‘ballpoint’, and ‘mosque’, while activation maps of ResNet only cover parts of objects. Such ability of precisely localizing CAM region makes the Res2Net  potentially valuable for object region mining in weakly supervised semantic segmentation tasks [65].

4.6 Object Detection

For object detection task, we validate the Res2Net on the PASCAL VOC07 [17] and MS COCO [40] datasets, using Faster R-CNN [53] as the baseline method. We use the backbone network of ResNet-50 vs. Res2Net-50, and follow all other implementation details of [53] for a fair comparison. Table V shows the object detection results. On the PASCAL VOC07 dataset, the Res2Net-50 based model outperforms its counterparts by $2.3\%$ on average precision (AP). On the COCO dataset, the Res2Net-50 based model outperforms its counterparts by $2.6\%$ on AP, and $2.2\%$ on AP@IoU=0.5.

We further test the AP and average recall (AR) scores for objects of different sizes as shown in Table VI. Objects are divided into three categories based on the size, according to [40]. The Res2Net based model has a large margin of improvement over its counterparts by $0.5\%$, $2.9\%$, and $4.9\%$ on AP for small, medium, and large objects, respectively. The improvement of AR for small, medium, and large objects are $1.4\%$, $2.5\%$, and $3.7\%$, respectively. Due to the strong multi-scale ability, the Res2Net based models can cover a large range of receptive fields, boosting the performance on objects of different sizes.

4.7 Semantic Segmentation

Semantic segmentation requires a strong multi-scale ability of CNNs to extract essential contextual information of objects. We thus evaluate the multi-scale ability of Res2Net on the semantic segmentation task using PASCAL VOC12 dataset [16]. We follow the previous work to use the augmented PASCAL VOC12 dataset [24] which contains 10582 training images and 1449 val images. We use the Deeplab v3+ [8] as our segmentation method. All implementations remain the same with Deeplab v3+ [8] except that the backbone network is replaced with ResNet and our proposed Res2Net. The output strides used in training and evaluation are both 16. As shown in  Table VII, Res2Net-50 based method outperforms its counterpart by 1.5$\%$ on mean IoU. And Res2Net-101 based method outperforms its counterpart by 1.2$\%$ on mean IoU. Visual comparisons of semantic segmentation results on challenging examples are illustrated in Fig. 6. The Res2Net based method tends to segment all parts of objects regardless of object size.

4.8 Instance Segmentation

Instance segmentation is the combination of object detection and semantic segmentation. It requires not only the correct detection of objects with various sizes in an image but also the precise segmentation of each object. As mentioned in Sec. 4.6 and Sec. 4.7, both object detection and semantic segmentation require a strong multi-scale ability of CNNs. Thus, the multi-scale representation is quite beneficial to instance segmentation. We use the Mask R-CNN [25] as the instance segmentation method, and replace the backbone network of ResNet-50 with our proposed Res2Net-50. The performance of instance segmentation on MS COCO [40] dataset is shown in Table VIII. The Res2Net-26w$\times$4s based method outperforms its counterparts by 1.7$\%$ on $AP$ and 2.4$\%$ on $AP_{50}$. The performance gains on objects with different sizes are also demonstrated. The improvement of $AP$ for small, medium, and large objects are 0.9$\%$, 1.9$\%$, and 2.8$\%$, respectively. Table VIII also shows the performance comparisons of Res2Net  under the same complexity with different scales. The performance shows an overall upward trend with the increase of scale. Note that compared with the Res2Net-50-48w$\times$2s, the Res2Net-50-26w$\times$4s has an improvement of 2.8 $\%$ on $AP_{L}$, while the Res2Net-50-48w$\times$2s has the same $AP_{L}$ compared with ResNet-50. We assume that the performance gain on large objects is benefited from the extra scales. When the scale is relatively larger, the performance gain is not obvious. The Res2Net module is capable of learning a suitable range of receptive fields. The performance gain is limited when the scale of objects in the image is already covered by the available receptive fields in the Res2Net module. With fixed complexity, the increased scale results in fewer channels for each receptive field, which may reduce the ability to process features of a particular scale.

4.9 Salient Object Detection

Pixel level tasks such as salient object detection also require the strong multi-scale ability of CNNs to locate both the holistic objects as well as their region details. Here we use the latest method DSS [29] as our baseline. For a fair comparison, we only replace the backbone with ResNet-50 and our proposed Res2Net-50, while keeping other configurations unchanged. Following [29], we train those two models using the MSRA-B dataset [43], and evaluate results on ECSSD [70], PASCAL-S [36], HKU-IS [34], and DUT-OMRON [71] datasets. The F-measure and Mean Absolute Error (MAE) are used for evaluation. As shown in Table IX, the Res2Net based model has a consistent improvement compared with its counterparts on all datasets. On the DUT-OMRON dataset (containing 5168 images), the Res2Net based model has a $5.2\%$ improvement on F-measure and a $2.1\%$ improvement on MAE, compared with ResNet based model. The Res2Net based approach achieves greatest performance gain on the DUT-OMRON dataset, since this dataset contains the most significant object size variation compared with the other three datasets. Some visual comparisons of salient object detection results on challenging examples are illustrated in Fig. 7.

4.10 Key-points Estimation

Human parts are of different sizes, which requires the key-points estimation method to locate human key-points with different scales. To verify whether the multi-scale representation ability of Res2Net can benefit the task of key-points estimation, we use the SimpleBaseline [67] as the key-points estimation method and only replace the backbone with the proposed Res2Net. All implementations including the training and testing strategies remain the same with the SimpleBaseline [67]. We train the model using the COCO key-point detection dataset [40], and evaluate the model using the COCO validation set. Following common settings, we use the same person detectors in SimpleBaseline [67] for evaluation.  Table X shows the performance of key-points estimation on the COCO validation set using Res2Net. The Res2Net-50 and Res2Net-101 based models outperform baselines on $AP$ by 3.3$\%$ and 3.0$\%$, respectively. Also, Res2Net based models have considerable performance gains on human with different scales compared with baselines.

5 Conclusion and Future Work

We present a simple yet efficient block, namely Res2Net, to further explore the multi-scale ability of CNNs at a more granular level. The Res2Net exposes a new dimension, namely “scale”, which is an essential and more effective factor in addition to existing dimensions of depth, width, and cardinality. Our Res2Net module can be integrated with existing state-of-the-art methods with no effort. Image classification results on CIFAR-100 and ImageNet benchmarks suggested that our new backbone network consistently performs favourably against its state-of-the-art competitors, including ResNet, ResNeXt, DLA, etc.

Although the superiority of the proposed backbone model has been demonstrated in the context of several representative computer vision tasks, including class activation mapping, object detection, and salient object detection, we believe multi-scale representation is essential for a much wider range of application areas. To encourage future works to leverage the strong multi-scale ability of the Res2Net, the source code is available on https://mmcheng.net/res2net/.

Acknowledgments

This research was supported by NSFC (NO. 61620106008, 61572264), the national youth talent support program, and Tianjin Natural Science Foundation (17JCJQJC43700, 18ZXZNGX00110).