Fine-Grained Categorization via CNN-Based Automatic Extraction and Integration of Object-Level and Part-Level Features

TL;DR

This paper introduces a CNN-based method for fine-grained categorization that automatically extracts and integrates object-level and part-level features using only object-level labels, achieving state-of-the-art results without requiring detailed annotations.

Contribution

The study presents a novel approach that leverages hierarchical CNN features for fine-grained categorization using only object-level labels, reducing annotation costs.

Findings

Achieves superior or comparable performance to state-of-the-art methods.

Effectively utilizes hierarchical CNN features without additional annotations.

Demonstrates wide applicability beyond fine-grained categorization.

Abstract

Fine-grained categorization can benefit from part-based features which reveal subtle visual differences between object categories. Handcrafted features have been widely used for part detection and classification. Although a recent trend seeks to learn such features automatically using powerful deep learning models such as convolutional neural networks (CNN), their training and possibly also testing require manually provided annotations which are costly to obtain. To relax these requirements, we assume in this study a general problem setting in which the raw images are only provided with object-level class labels for model training with no other side information needed. Specifically, by extracting and interpreting the hierarchical hidden layer features learned by a CNN, we propose an elaborate CNN-based system for fine-grained categorization. When evaluated on the Caltech-UCSD Birds-200-2011, FGVC-Aircraft, Cars and Stanford dogs datasets under the setting that only object-level class labels are used for training and no other annotations are available for both training and testing, our method achieves impressive performance that is superior or comparable to the state of the art. Moreover, it sheds some light on ingenious use of the hierarchical features learned by CNN which has wide applicability well beyond the current fine-grained categorization task.

Tables

Table 1. Table 1: The choice of 𝒮 o ​ b ​ j ​ e ​ c ​ t l ​ a ​ y ​ e ​ r ​ _ ​ i ​ d ​ x subscript superscript 𝒮 𝑙 𝑎 𝑦 𝑒 𝑟 _ 𝑖 𝑑 𝑥 𝑜 𝑏 𝑗 𝑒 𝑐 𝑡 \mathcal{S}^{layer\_idx}_{object} and 𝒮 p ​ a ​ r ​ t ​ s l ​ a ​ y ​ e ​ r ​ _ ​ i ​ d ​ x subscript superscript 𝒮 𝑙 𝑎 𝑦 𝑒 𝑟 _ 𝑖 𝑑 𝑥 𝑝 𝑎 𝑟 𝑡 𝑠 \mathcal{S}^{layer\_idx}_{parts} for GoogLeNet, VGG19 and VGG-CNN-S.

CNN	${𝒮}_{object}^{layer\_idx}$	${𝒮}_{parts}^{layer\_idx}$
GoogLeNet [31]	inception_4e/output, inception_5a/output	inception_4d/output, inception_4e/output
VGG19 [30]	conv5_4	conv5_2, conv5_3
VGG-CNN-S [51]	conv5	conv4, conv5

Table 2. Table 2: The 15 parts annotated with ground truth location coordinates in Caltech-UCSD Birds-200-2011  [ 56 ] .

ID	1	2	3	4	5	6	7	8
name	back	beak	belly	breast	crown	forehead	left eye	left leg
ID	9	10	11	12	13	14	15
name	left wing	nape	right eye	right leg	right wing	tail	throat

Table 3. Table 3: Fine-grained categorization results on Caltech-UCSD Birds-200-2011 dataset of using proposed system with various settings. Detailed description can be found in the text.

Method	GoogLeNet-ft (baseline)	2 CNNs ensemble	2 CNNs: raw+obj	3 CNNs ensemble	3 CNNs: raw+2 parts	4 CNNs ensemble	4 CNNs: raw+3 parts
Test Accuracy ($\%$)	$73$	$72.7$	$77$	$74$	$81.38$	$72.1$	$78.5$

Table 4. Table 4: Fine-grained categorization results on FGVC Aircraft dataset of using proposed system with various settings. Detailed description can be found in the text.

Method	GoogLeNet-ft (baseline)	2 CNNs ensemble	2 CNNs: raw+obj	3 CNNs ensemble	3 CNNs: raw+2 parts	4 CNNs ensemble	4 CNNs: raw+3 parts
Test Accuracy ($\%$)	$75$	$77.0$	$80$	$78.57$	$82.7$	$76.6$	$80.1$

Table 5. Table 5: Comparison of different classification methods on Caltech-UCSD Birds-200-2011 dataset [ 56 ] . ‘Anno’ stands for ‘annotations’, ‘B’ for ‘bounding box’, ‘P’ for ‘part location’ and ‘ft’ for ‘fine-tune’. The rounding precision of other methods may differ a little in other citations.

Method	Train Anno	Test Anno	Accuracy ($\%$)
POOF [2]	$B+P$	$B+P$	$73.3$
PN-CNN [4]	$B+P$	$B+P$	$85.4$
Symbiotic [3]	$B+P$	$B+P$	$69.5$
Part R-CNN [41]	$B+P$	$B$	$76.37$
POOF [2]	$B+P$	$B$	$56.8$
DPD+DeCAF [5]	$B+P$	$B$	$64.96$
Nonparametric [6]	$B+P$	$B$	$57.84$
Symbiotic [3]	$B+P$	$B$	$61.60$
Deep LAC [7]	$B+P$	$B$	$80.26$
Spda-cnn [8]	$B+P$	$B$	$85.14$
PBC [11]	$B+P$	$B$	$83.7$
No parts [14]	$B$	$B$	$82.8$
Symbiotic [3]	$B$	$B$	$59.4$
CNNaug-SVM [29]	$B$	$B$	$61.8$
Bilinear [54]	$B$	$B$	$85.1$
multi-stage [64]	$B$	$B$	$67.86$
Coarse-to-fine [9]	$B$	$B$	$82.9$
Simple tech [65]	$B$	$B$	$66.87$
FOAF [23]	$B$	$B$	$86.34$
PS-CNN [10]	$B+P$	$B$	$76.6$
PN-CNN [4]	$B+P$	$-$	$75.7$
Part R-CNN [41]	$B+P$	$-$	$73.89$
No parts [14]	$B$	$-$	$82.0$
Coarse-to-fine [9]	$B$	$-$	$82.5$
Mul-granularity [16]	$B$	$-$	$83.0$
Task-driven [21]	$B$	$-$	$81.69$
PBC [11]	$B$	$-$	$83.3$

Table 6. Table 6: Comparison of different classification methods on Caltech-UCSD Birds-200-2011 dataset [ 56 ] . It is continued from Table  5 . The rounding precision of other methods may differ a little in other citations.

Method	Train Anno	Test Anno	Accuracy ($\%$)
Two-level [53]	$-$	$-$	$69.7$
Constellation [15]	$-$	$-$	$81.01$
Spatial trans [15]	$-$	$-$	$84.1$
W-supervised [18]	$-$	$-$	$79.34$
Bilinear [54]	$-$	$-$	$84.1$
Mul-granularity [16]	$-$	$-$	$81.7$
Picking deep [22]	$-$	$-$	$84.54$
Annotation-modi [20]	$-$	$-$	$75.36$
Fused [23]	$-$	$-$	$84.63$
Friend-or-Foe [24]	$-$	$-$	$77.4$
GoogLeNet-ft	$-$	$-$	$73.0$
Ours-GoogLeNet-based	$-$	$-$	$81.38$
VGG-19-ft	$-$	$-$	$70.79$
Ours-VGG-19-based	$-$	$-$	$72.0$
VGG-CNN-S-ft	$-$	$-$	$56.21$
Ours-VGG-CNN-S-based	$-$	$-$	$58.83$

Table 7. Table 7: Comparison of different classification methods on FGVC-Aircraft dataset  [ 57 ] . ‘Anno’ stands for ‘annotations’, ‘B’ for ‘bounding box’, ‘P’ for ‘part location’ and ‘ft’ for ‘fine-tune’. The rounding precision of other methods may differ a little in other citations.

Method	Train Anno	Test Anno	Accuracy ($\%$)
Coarse-to-fine [9]	$B$	$B$	$87.7$
Simple tech [65]	$B$	$B$	$72.18$
Mining Triplets [19]	$B$	$B$	$88.4$
Coarse-to-fine [9]	$B$	$-$	$86.9$
Mul-granularity [16]	$B$	$-$	$86.6$
Bilinear [54]	$-$	$-$	$84.1$
Mul-granularity [16]	$-$	$-$	$82.5$
GoogLeNet-ft	$-$	$-$	$75.0$
Ours-GoogLeNet-based	$-$	$-$	$82.7$
VGG-19-ft	$-$	$-$	$77.88$
Ours-VGG-19-based	$-$	$-$	$81.8$
VGG-CNN-S-ft	$-$	$-$	$66.48$
Ours-VGG-CNN-S-based	$-$	$-$	$69.5$

Table 8. Table 8: Comparison of different classification methods on cars dataset  [ 61 ] . ‘Anno’ stands for ‘annotations’, ‘B’ for ‘bounding box’, ‘P’ for ‘part location’ and ‘ft’ for ‘fine-tune’. The rounding precision of other methods may differ a little in other citations.

Method	Train Anno	Test Anno	Accuracy ($\%$)
No parts [14]	$B$	$B$	$92.8$
Mining Triplets [19]	$B$	$B$	$92.5$
Learn parts [13]	$B$	$-$	$73.9$
No parts [14]	$B$	$-$	$92.6$
Bilinear [54]	$-$	$-$	$91.3$
GoogLeNet-ft	$-$	$-$	$81.84$
Ours-GoogLeNet-based	$-$	$-$	$85.42$
VGG-19-ft	$-$	$-$	$78.74$
Ours-VGG-19-based	$-$	$-$	$83.0$
VGG-CNN-S-ft	$-$	$-$	$60.49$
Ours-VGG-CNN-S-based	$-$	$-$	$74.75$

Table 9. Table 9: Comparison of different classification methods on Stanford dogs dataset  [ 62 ] . ‘Anno’ stands for ‘annotations’, ‘B’ for ‘bounding box’, ‘P’ for ‘part location’ and ‘ft’ for ‘fine-tune’. The rounding precision of other methods may differ a little in other citations.

Method	Train Anno	Test Anno	Accuracy ($\%$)
Symbiotic [3]	$B$	$B$	$45.6$
multi-stage [64]	$B$	$B$	$70.31$
Simple tech [65]	$B$	$B$	$43.15$
FOAF [23]	$B$	$B$	$74.49$
PBC [11]	$B$	$B$	$78.3$
Attention [12]	$-$	$-$	$76.8$
Constellation [15]	$-$	$-$	$68.61$
Picking deep [22]	$-$	$-$	$71.96$
FOAF [23]	$-$	$-$	$68.66$
W-supervised [18]	$-$	$-$	$80.43$
Friend-or-Foe [24]	$-$	$-$	$71.4$
GoogLeNet-ft	$-$	$-$	$82.6$
Ours-GoogLeNet-based	$-$	$-$	$83.6$
VGG-19-ft	$-$	$-$	$80.97$
Ours-VGG-19-based	$-$	$-$	$83.0$
VGG-CNN-S-ft	$-$	$-$	$72.90$
Ours-VGG-CNN-S-based	$-$	$-$	$73.7$

Equations

$${\mathbf{M}}_{layer,sum}^j=\sum _{i\in {\mathcal{S}}_{layer\_j}^{map\_idx}}{\mathbf{F}}_i^j$$

$${\mathbf{M}}_{layer,sum}^j=\mathcal{N}({\mathbf{M}}_{layer,sum}^j)$$

$${\mathbf{M}}_{layer,sum}^j=\mathcal{R}({\mathbf{M}}_{layer,sum}^j)$$

$${\mathbf{M}}_{object}=\prod _{j\in {\mathcal{S}}_{object}^{layer\_idx}}{\mathbf{M}}_{layer,sum}^j$$

$${\mathbf{M}}_{object}=\mathcal{N}({\mathbf{M}}_{object})$$

$${\mathbf{M}}_{object}^{binary}(x,y)=\{\begin{array}{ll}1,& \text{if }{\mathbf{M}}_{object}(x,y)>T_{object}\\ 0,& \text{otherwise.}\end{array}$$

$${\mathbf{F}}_i^{j,binary}(x,y)=\{\begin{array}{ll}1,& \text{if }{\mathbf{F}}_i^j(x,y)>T_{parts}\\ 0,& \text{otherwise.}\end{array}$$

$${\mathbf{M}}_{part\_k}^{binary}(x,y)=\{\begin{array}{ll}1,& \text{if }{\mathbf{M}}_{part\_k}(x,y)>T_{parts}\\ 0,& \text{otherwise.}\end{array}$$

$$DB=\frac{1}{k}\sum _{i=1}^{k}ma{x}_{j\ne i}\{D_{i,j}\}$$

$$D_{i,j}=\frac{\text{within-cluster distance}}{\text{between-cluster distance}}=\frac{{\bar{d}}_i+{\bar{d}}_j}{d_{i,j}}$$

$$Si=\frac{bi-ai}{max(ai,bi)}$$

Full text

††journal: Journal of Image and Vision Computing

1 Introduction

Fine-grained visual categorization refers to a special type of image classification tasks in which the object categories generally have the same constituent parts and topology and hence they are visually and semantically very similar to each other. Some examples include fine distinction of birds according to species and human faces according to age or gender. Because different categories are very similar, fine-grained categorization often requires identifying subtle part-based differences between categories. This is particularly challenging when there exist large within-category variations such as pose and scale variance, or when only a small amount of training data is available.

Consequently, many fine-grained categorization systems put their emphasis on exploiting part-based features to boost the classification performance. One common approach is to make use of manually provided annotations, such as a bounding box around the whole object and the part locations indicated by coordinates, to extract fine features from the object parts [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]. The main limitation of this approach is that it is laborious to obtain data with part annotations to provide part-based features for the classifier. Moreover, for images outside the dataset with no part annotations available, applying the trained classifier on them requires using a part detector first. Although having such a part detector makes it unnecessary to provide part annotations for the test images, training the part detector still needs images with part annotations available. Some recent attempts have been made to relax this requirement to use less annotations by training a part detector in a weakly supervised manner [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. As a result, part locations are no longer needed for both training and testing. However, inaccurate part locations returned by the part detector can affect the quality of the part-based features extracted and hence impair the classification performance. To reduce this effect, computationally demanding procedures are often integrated into the system to improve the part detection accuracy. Without part annotations, a typical way to identify the object parts is to group or select them from randomly or exhaustively generated part proposals. It is costly to verify massive part proposals without much guidance. It would be beneficial if we could develop a more informed process for part detection by exploiting the features automatically learned solely from the raw images and the corresponding category labels. This leads us to a more promising alternative which aims at developing an automatic yet efficient process for part detection with minimal supervision required.

Deep learning [25, 26] has an important role to play here because it focuses on learning features or representations directly from raw data and it learns the features in a hierarchical manner supervised by the category labels only. In particular, convolutional neural networks (CNN or ConvNet) [27] provide a powerful end-to-end framework which tightly integrates feature extraction and classification to achieve state-of-the-art performance in many challenging computer vision tasks [28, 5, 29, 30, 31, 17]. It should be noted that the excitement does not just come from the superior performance achieved. The rich features learned by deep CNN ranging from low-level to high-level representations in the hidden layers have also aroused much research interest in investigating how to take advantage of them [32, 18, 33, 16, 34, 35, 36, 37, 38]. On one hand, interpreting the hidden layer features may help us understand the workings of CNNs and monitor the learning process. On the other hand, exploiting the features appropriately may help to further boost the performance of various tasks. However, due to the complexity of the highly encapsulated CNN architecture especially when it is deep, exploiting the hidden layer features learned turns out to be highly nontrivial. The recent attempts either show some general results without concrete methods for specific tasks [39, 5, 40] or exhaustively use the features learned by a CNN to come up with region proposals to assist some other manually designed tools [4, 29, 41, 14, 15, 21, 22].

We believe a well-trained CNN has potential that remains to be more fully explored and this has motivated us to pursue the current research. Our contribution in this work is twofold. First we explore the hidden layer feature maps of a well-trained CNN more thoroughly and make good use of them in simple but nontrivial ways. We visualize not only the hidden layer feature maps as they are to conclude their properties, but also the intermediate results of each step when we manipulate them. Second, guided by the general observation of CNN, we propose a pure CNN-based system tailed for fine-grained categorization. Partially, our system obtains robust object and parts detection through interpreting the hidden layers of a trained CNN rather then calculating them, which often requires a lot more computation, parameters and annotations.

The remainder of this paper is organized as follows. Sec. 2 reviews some previous work on fine-grained categorization and the related CNN study. Our proposed method is presented in Sec. 3 which is then followed by experiment results in Sec. 4. Sec. 5 concludes the paper.

2 Related Work

The approaches to handle fine-grained categorization vary a lot while the essentialness of part-based feature is well recognized. Various handcrafted features have been involved in related part-based methods [1, 42, 43, 2, 44]. For example, part-based one-vs-one features (POOF) [2, 45, 46] are based on histograms of oriented gradients (HOG) [47, 48] and color histograms. Each POOF is defined based on the locations of two parts which are used to align the object and a POOF is extracted around one of the two parts. The success of POOF features demonstrates the effectiveness of part-based features for fine-grained categorization, but its need for manual selection and location of the parts makes it somewhat unappealing.

After CNN was demonstrated to give superior performance in the ImageNet challenge [28], we have witnessed a resurgence of research interest in CNN [39, 5, 32, 49]. Moreover, the public availability of efficient CNN implementations [50] and powerful pre-trained CNN models [28, 51, 30, 31] has further popularized the pervasive use of CNN for various computer vision tasks. Not surprisingly, fine-grained categorization also turns to CNN for automatic feature extraction and part detection. Branson *et al. * [4] use CNN to extract local features from pose-normalized images. An object is first warped to a pose prototype where both the warping function and the pose prototype are learned by minimizing some manually designed objective functions with the part locations needed. Based only on segmentation, Krause *et al. * [14] locate the parts through alignment. No part annotations are needed for both training and testing, but complex co-segmentation, alignment and part selection methods are involved and the ground-truth object bounding box is crucial to proper initialization and refinement of the segmentation procedure. Simon *et al. * [40, 15] use the hidden layer feature maps of a CNN for part detection. Assuming that some feature maps of the CNN correspond to object parts, they use the gradient map of each feature map to offer a part proposal. A likelihood function is used to select the maps that correspond to valid object parts. However, although a filter (corresponding to a feature map) can detect a certain pattern in an object part, there often does not exist a simple one-to-one correspondence between the feature maps and the object parts. Thus, selecting one map to locate one object part may not be sufficient. Sermanet *et al. * [52] use a CNN to automatically find a bounding box for the object by combining the results obtained from many sliding windows. Unlike selecting only one choice in [40, 15], combining multiple feature maps provides a more robust approach which is also adopted by our method.

Some recent methods achieve promising performance without requiring additional information other than the object labels [53, 12, 54, 15, 18, 16]. The motivation of the bilinear CNN model [54] is to use two CNNs to factor out the variance due to location and the appearance of the object parts. However, it is mentioned in [54] that the roles of the two networks are not clearly separated. A CNN is used to detect both the whole object and its parts in [18] and [53]. The initial object proposals in [18], which include many noisy background regions, are generated by selective search [55]. The two-level attention model [53] is based on the intuition that performing fine-grained categorization requires first to “see” the object and then the most discriminative parts of it. This intuition is also crucial in our system design. In [53] the feature maps are clustered according to their filter coefficients, but the same object part may not have similar texture for different classes.

Although CNN has been used recently for fine-grained categorization, we believe there is still much room for further investigation especially on using CNN for automatic part detection. Our preliminary investigation shows that the hidden layer features extracted from a raw image by a well-trained CNN are rich but implicit. Unlike the classifier, the hidden layers are not trained to output an explicit target. Consequently, using them for part detection requires a carefully designed procedure to interpret the hidden layer features learned by a CNN. The filters respond to the corresponding patterns that appear in their field of perception, but those patterns may reside in both the foreground and the background and they may only correspond to partial regions inside object parts. In the previous work, a simplifying assumption is adopted by using one feature map as one part proposal and all the proposals are verified extensively. Instead, we take advantage of the hierarchical features extracted by a CNN based on the object detection result to impose constraints on part detection and sum over all the selected feature maps to achieve robust part detection.

3 Proposed System

Our fine-grained categorization system is entirely based on CNNs trained using raw images and their class labels only. A CNN is first trained using raw images as input. The hidden layer feature maps of the trained CNN are then used to detect the object and its parts. Based on the detection result, we crop some image patches each of which tightly contains either the whole object or one object part. These generated images are referred to as object-focused and part-focused images. As such, the original images are augmented by focusing on the targets at different levels. We then construct an additional CNN for each object part detected and initialize it with the CNN previously trained using raw images, and then fine-tune it with the corresponding part-focused images. All the CNNs are combined into an integrated CNN which extracts and combines both object-level and part-level features before classification. The whole system is shown in Figure 1 with two parts detected.

In the following subsections, we will first present clearly the notation and the initialization setting of our algorithm. We will then highlight the major steps and the key ideas behind them.

3.1 Notation and System Initialization

When describing our algorithm below, the image, mask, index set, hidden layer feature map, and 2-D centroid coordinates (in an image) are denoted by $\mathbf{I}$, $\mathbf{M}$, $\mathcal{S}$, $\mathbf{F}$, and $\mathbf{c}$, respectively. Superscripts and subscripts are used to identify different variables of the same kind and also give information about their properties. $\mathbf{I}_{orig}$ represents the original image. Its $k$th part-focused image generated is $\mathbf{I}_{part\_k}$. $\mathcal{S}^{layer\_idx}_{object}$ specifies the set of indices of the layers from whose features the whole object is detected while $\mathcal{S}^{layer\_idx}_{parts}$ specifies those used to detect the parts. Figure 1 shows the case with $|\mathcal{S}^{layer\_idx}_{object}|=1$ and $|\mathcal{S}^{layer\_idx}_{parts}|=2$, where $|\cdot|$ denotes the cardinality of a set. $\mathbf{F}^{j}_{i}$ means the $i$th feature map in layer $j$ and $\mathbf{c}_{j,i}$ is its weighted centroid. In addition to the given input, $\mathcal{S}^{map\_idx}_{layer\_j}$ contains all the feature map indices in layer $j$. All the other sets like $\mathcal{S}^{map\_idx}_{valid}$ and $\mathcal{S}^{centroid}_{valid}$ are used to store the intermediate results and are initialized as empty sets. $N_{part}$ denotes the number of parts to detect. $T_{object}$ and $T_{parts}$ are two scalar thresholds. $\mathcal{N}(\cdot)$ is a normalization function which scales each entry of its input matrix (representing a grayscale image) to the range $[0,1]$, and $\mathcal{R}(\cdot)$ is a function which resizes its input (both color and grayscale images) so that the images have the same resolution as the CNN input.

3.2 Object Detection

An active region in a feature map indicates the existence of a certain pattern detected at that location. A well-trained CNN should make good use of the patterns that are useful for distinguishing between different categories. Since the background is generally irrelevant to object categorization, the active regions usually reside within the object. Although some background regions may contain patterns similar to those in the object, the background patterns usually correspond to low-level features and the corresponding regions are unlikely to remain active in the higher-level feature maps. This phenomenon will be illustrated in Subsec 3.4.1. The object detection result is given by a saliency mask, which is essentially a $[0,1]$ grayscale image resized to the same resolution as the CNN input. The brighter the region in a mask, the stronger the indication of existence of its target. In our method, the object saliency mask $\mathbf{M}_{object}$ and its binary version $\mathbf{M}^{binary}_{object}$ are calculated based on steps (1) through (6) below:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

For each layer $j\in\mathcal{S}^{layer\_idx}_{object}$ (How we choose $\mathcal{S}^{layer\_idx}_{object}$ and other parameters are detailed in Subsec 3.4), all its feature maps are summed and then normalized to obtain $\mathbf{M}_{layer,sum}^{j}$. In so doing, all the patterns are aggregated and the activations in the background are suppressed after normalization since they are overshadowed by those within the object. In (4), the notation $\prod\limits_{j\in\mathcal{S}^{layer\_idx}_{object}}$ means applying element-wise multiplication to all the summed maps in all the layers indexed by $\mathcal{S}^{layer\_idx}_{object}$ to further increase the confidence. Consequently, an active region in the resulting mask $\mathbf{M}_{object}$ has to be active in all the layers in $\mathcal{S}^{layer\_idx}_{object}$.

3.3 Part Detection

The key component of our system is to automatically detect the object parts using the hidden layer features of the CNN trained with raw images and their class labels only. We first choose the feature maps which are likely to be activated by the object parts and then cluster them into $N_{part}$ groups. The mask of the $k$th part $\mathbf{M}_{part\_k}$ is obtained by summing over all the feature maps within the corresponding cluster. The detailed steps involved in part detection are listed in Algorithm 1.

The layers used to detect object parts are from $\mathcal{S}^{layer\_idx}_{parts}$ (how to choose $\mathcal{S}^{layer\_idx}_{parts}$ is detailed in Subsec 3.4.1), but not all their feature maps are useful for part detection because those with activations in the background may hurt the part detection accuracy during clustering. Thus we choose the maps that are likely to contribute to part detection based on two constraints. First, the activated region in the thresholded map should be a connected region. Second, this region should have its weighted centroid within the object detected previously. The selection results are stored in the set $\mathcal{S}^{map\_idx}_{valid}$. Figure 2 illustrates this step using two examples, with each example containing $7$ columns. In the first column of each example, from top to bottom shows the original input image, the soft masked image, and the thresholded masked image given by object detection with $T_{object}=0.3$, respectively. (Our system is quite robust to the choice of $T_{object}$ and $T_{parts}$. More details can be found in Sec. 4.) For column $2$ to $7$, the first row shows some normalized feature maps, the second row shows their thresholded version with $T_{parts}=0.3$, and the last row shows the results of applying the mask on the image with the mask’s weighted centroid marked in red. Column $2$ and $3$ show two maps that contain multiple disconnected active regions and thus fail to meet the first constraint. Although these maps may be activated by the object parts, it is likely that they correspond to some patterns shared by multiple parts instead of just one, making them ineffective for part detection. Column $4$ and $5$ show two feature maps that do contain only one connected region in their thresholded version, but their weighted centroids fail to fall within the detected object region. Notice that we only constrain the centroids but the actual active regions are allowed to extend beyond the detected object region. This strategy increases the robustness and flexibility of our method. The last two columns show two feature maps which are successfully selected to detect object parts. In fact the hidden layer feature maps of the CNN are very rich. Even when some feature maps are ignored by mistakes, the aggregation result is still quite robust.

We cluster the selected feature maps’ centroids by $k$-means so that the maps activated by the same part can be combined together to offer more robust detection of one part. The clustering results of $N_{part}=1,2,3,4,5,6$ are shown in Figure 3, and some generated part-focused images for $N_{part}=3,5$ are shown in Figure 4. It can be seen that our part detection method works well for various number of parts.

3.4 Choice of Parameters

The parameters we need to specify for our proposed system include the set of layers used to detect the object ($\mathcal{S}^{layer\_idx}_{object}$) and the parts ($\mathcal{S}^{layer\_idx}_{parts}$), the number of parts we want to detect ($N_{part}$), and the thresholds used to binarize the masks ($T_{object}$ and $T_{parts}$). Our method is robust to varying $T_{object}$ and $T_{parts}$ as we will see in Sec. 4. In the next two subsections, we will provide some insights on choosing $\mathcal{S}^{layer\_idx}_{object}$, $\mathcal{S}^{layer\_idx}_{parts}$ and $N_{part}$.

3.4.1 Choice of $\mathcal{S}^{layer\_idx}_{object}$ and $\mathcal{S}^{layer\_idx}_{parts}$

Part detectors that only check the local texture without other constraints may lead to false alarm in the background. For example, when seeing a single black dot, even humans may not be able to tell an eye from a stone. Seeing the whole object first provides the context to focus on specific parts. Consequently, our algorithm first detects the object region and then uses it to constrain the subsequent part detection task. For both object and part detection, one result is given by one mask which is the normalized version of the sum of some selected hidden layer feature maps. $\mathcal{S}^{layer\_idx}_{object}$ and $\mathcal{S}^{layer\_idx}_{parts}$ specify the layers whose feature maps are most likely to contribute to object detection and part detection, respectively. Figure 5 shows some resized feature maps from different hidden layers in the GoogLeNet [31], which has $22$ layers with parameters or $27$ layers if the pooling layers are also counted. (The original image is the first example in Figure 2.) Each column in this figure contains five feature maps of one layer and the last one is the normalized sum of all the maps in that layer. Considering the GoogLeNet with $22$ layers, from left to right shows seven of them which are the $\nth{1}$, $\nth{8}$, $\nth{9}$, $\nth{11}$, $\nth{13}$, $\nth{15}$, and $\nth{18}$ layers.

From the sample feature maps shown above, we can see that the lower layers tend to always respond to local patterns and the activated regions are scattered. An extreme case is the $\nth{1}$ layer maps which respond strongly to the edges of the bird and the branches alike and hence the sum map of this layer is like an edge detection result. As we move to higher levels, the activated regions become more compact in each map and the sum map is more focused on the object as a whole. This also justifies our part detection procedure described in Subsec. 3.3, by using lower layer feature maps selected by object detection result form higher layer, we leverage both the precision of lower layer and the robustness of higher layer.

Another observation is that in every layer, we can often find some feature maps that are activated by the background, and the object and parts are often partially detected in one map. However, the simple summation and normalization operations can sufficiently suppress the spurious activations and integrate the partial detections into a complete one. The boundary is complete in the $\nth{1}$ layer normalized sum map. As we move to higher layers, the background branches fade away in their normalized sum maps. The whole object region is clearly detected in the sum map of the $\nth{18}$ layer although its individual maps have both false alarm and incomplete object detection.

Since $\mathcal{S}^{layer\_idx}_{object}$ specifies the layer(s) whose features are most likely to be activated by object-level patterns, it is reasonable to pick the layer(s) high enough to perceive global view of the input image. From Figure. 5 we can see that the sum maps in lower layers include more background activation and consequently the object detection is not tight. Similar consideration applies to the choice of $\mathcal{S}^{layer\_idx}_{parts}$, where we hope to see that the activations are compact and fine enough to highlight one object part or part of it. Considering how we select hidden layer feature maps to detect parts in Subsec. 3.3 together with the results in Figure. 5, we can see that too high layers will not distinguish the parts but the object, while the activations in low layers’ feature maps tend to contain multiple disconnected regions instead of one, so few of them are valid to detect parts, making the detection result not robust. The choice of $\mathcal{S}^{layer\_idx}_{object}$ and $\mathcal{S}^{layer\_idx}_{parts}$ for the CNNs we use are listed in Table 1. We have tried similar choices and they also works.

3.4.2 Choice of $N_{part}$

As described in Algorithm 1, we cluster the selected feature maps’ centroids by $k$-means and we choose $k$ according to Davies-Bouldin criterion [58] and Silhouette criterion [59, 60].

The Davies-Bouldin criterion [58] is based on the ratio of the distances between samples of the same and different clusters. The Davies-Bouldin index ($DB$) of a clustering with $k$ groups is defined as:

[TABLE]

[TABLE]

$\bar{d}_{i}$ is the average distance between points and their own centroid in the ith cluster. $d_{i,j}$ is the distance between the centroids of the ith and jth clusters.

The maximum value of $D_{i,j}$ over $j$ gives a score of the most ambiguous case for group $i$ and $DB$ is obtained by averaging the worst case scores of all clusters. The smaller the $DB$ is, the better the clustering result.

The examples of using Davies-Bouldin criterion to measure our k-means clustering with different $k$ ($N_{part}$) on Caltech-UCSD Birds-200-2011 [56] and FGVC-Aircraft [57] datasets are shown in Figure 6 and Figure 7. The left figure shows the mean Davies-Bouldin index of all the images in the dataset and the right figure shows the histogram of best the $k$ choice. The cars [61] and Stanford dogs [62] datasets give similar results where $k=2$ is preferred.

The silhouette value $Si$ for the ith data point is defined as:

[TABLE]

where $ai$ is the average distance from the ith point to the other points within the same cluster and $bi$ is the minimum average distance from the ith point to the points in different clusters, and the minimization is taken over all the other clusters.

It can be seen that $-1\leq Si\leq 1$ and the higher the silhouette value the safer the ith point is in its own cluster. If most points have a high silhouette value, then the clustering solution is appropriate. So a high average value of $Si$ over all the data is an indication of good clustering.

The results of applying Silhouette criterion on Caltech-UCSD Birds-200-2011 [56] and FGVC-Aircraft [57] datasets are shown in Figure 8 and Figure 9. The left figure shows the mean Silhouette value of all the images in the dataset and the right figure shows the histogram of best $k$ choice. The cars [61] and Stanford dogs [62] datasets again give similar results.

It can be seen that both criteria suggest $k=2$ is preferred by the majority. However, the above mentioned two criteria are not appliable to $k=1$ case. We found during the experimets that for cars [61] and Stanford dogs [62] datasets, $k=1$ is more suitable. This is due to the property of the datasets. Two typical images from Stanford dogs [62] dataset and the clustering of the selected feature maps’ centroids are shown in Figure 10 where most of the feature maps are focused on the face of the dogs for the following reasons: First unlike birds, dogs do not have colorful feather. The furs of dogs are mostly white, brown and black without special texture. Second, most of the dogs in the images are pets that are facing the camera, with their bodies occluded by the head, human hands or clothes. Thus, dogs are mainly distinguished by their heads, so only one part is focused. Similarly, cars [61] of different models differ from each other mainly by their shapes of a large chunk rather than special local texture.

The choice of $N_{part}$ is further justified by the classification results in Subsec. 4.3. A small $N_{part}$ with each part-focused image containing a reasonable large portion of the object is actually quite preferable considering some practical issues of our fine-grained categorization framework. The purpose of part detection is to facilitate fine-grained local feature extraction by inputing cropped part-focused images into CNNs. As discussed previously, patterns are better interpreted in context. Even if the eye of a bird is precisely detected, cropping just a black dot does not provide very useful features, but including the feather around the eye can often help a lot. Another reason for preferring a small $N_{part}$ is that in many cases it is hard to identify the exact boundary between parts, e.g., between the head and neck, neck and chest, belly and back, etc. Moreover, sometimes one part may partially occlude another part, e.g., the wings covering the body. Besides, the order in which we concatenate the features of the parts is based on the size of the part region detected. A consequence of having a larger $N_{part}$ would be increasing the chance of getting parts with similar size and hence mis-ordering them. A practical consideration is that more parts require more memory in the integrated CNN. Last but not least, if the problem setting changes and part annotations are used, our method can still play an important role, e.g.,generating high-quality part proposals, etc.

3.5 Part-Focused Image Generation

Based on the saliency mask for each object part, the most intuitive way to generate a part-focused image is to directly crop a patch around the salient region. However, we observe that sometimes both the object and part detection results are biased towards the regions which have complex patterns and are crucial for classification while some less discriminative regions with simpler patterns are underestimated. This is because we aim at achieving robust detection by combining many feature maps but the regions with relatively simple and indistinctive patterns have few feature maps voting for them. Figure 11 (better viewed in color) demonstrates this phenomenon in applying GoogLeNet [31] on Caltech-UCSD Birds-200-2011 [56] with $N_{part}=2$. The first image of each row is the original image, with the detected object region represented roughly by an ellipse. The weighted centroids of the object region and two part regions are marked with red stars. We can see that the two parts detected correspond roughly to the head and body, but the salient region and the centroid of the object are biased towards the head. The third and fifth images of each row are the two part-focused images generated by directly cropping image patches around the active region in part detection masks. It can be seen that part of the head is sometimes included in the body-focused image while part of the tail is often ignored.

Another inspiring observation from Figure 11 is that although the detection results are biased, they are sufficient to indicate the pose of the bird. Thus, the bias can be easily compensated by shifting and extending the cropped region along the pose of the bird, e.g, in order to include the mistakenly ignored tail, we can simply shift and extend the cropped image patch to the pointing direction of the tail.

For the case of $N_{part}=2$, given the ellipse which roughly contains the bird, the centroid of the bird and those of its head and body, we can represent the pose of the bird by one vector pointing from the head to the body (or the opposite direction), or two vectors from the center of the bird to the two parts. Intuitively, we have two choices of the pose represented by a single vector: the vector that starts from the ellipse’s focal point closer to the centroid of the head and ends at the other focal point; the vector that starts from the centroid of the head and ends at the centroid of the body. There is one choice of the pose represented by two vectors: both vectors start at the centroid of the object, with one ending at the centroid of the head while the other ending at the centroid of the body. The three pose choices are illustrated in Figure 12 (better viewed in color) with each column showing one example. We can see that the three methods give very consistent pose estimation results. We have tried all these three versions of poses to adjust part-focused image generation and they indeed offer similar results. In our experiment we use the two-vector version, which can be extended to more parts cases intuitively.

In our system, we shift the cropped image patches along the pose to compensate potential bias and augment the training images. Figure 13 illustrates this procedure and the cropped image patches after adjustment are the forth and sixth columns in Figure 11.

3.6 Integration of Specialized CNNs

After the object-focused and part-focused images are generated, we construct one more CNN initialized with that previously trained using the raw images for each part. We fine-tune the CNNs for the object parts using the corresponding part-focused images so the new CNNs are specialized to extract discriminative features from different parts. To integrate different CNNs specialized for object-level and part-level features, we concatenate the feature vectors followed by a dropout layer with dropout ratio 0.3, then a fully-connected layer with the out put number matching the number of classes is added. The integrated CNN is fine-tuned for the last time. In Figure 1, the newly added module (blue rectangle) comprises the concatenation layer and dropout layer mentioned above. The classifiers in Figure 1 are simply a fully-connected layer with the number of output units the same as that of the classes followed by a softmax layer.

Thus, we specialize our feature extractors but changing the network structure is unappealing. In fact we only need to construct one CNN for these two phases, and adjust the $loss\_weight$ parameter in [50] once. We design our CNN for training as shown in Figure 14 and 15 (the same network with different parameter settings shown in different colors). Compared with Figure 1, the difference is that three more classifiers are plugged in directly after each feature extractor. The strength of supervision imposed by each classifier is adjusted by the $loss\_weight$ parameter in [50]. If the $loss\_weight$ of one classifier is set to [math], the error calculated by this classifier is discarded and no gradient information is propagated to the path below. In Figure 14 and 15, the classifiers with $loss\_weight=0$ are marked by an orange frame, the classifiers with $loss\_weight=1$ are marked by a blue frame, and the ‘silent’ backward propagation paths are shown in gray. Figure 14 shows the integrated CNN conducting completely separate fine-tuning for its three specialized feature extractors and Figure 15 shows the integrated CNN conducting overall fine-tuning based on the errors from only one final classifier. Another option is to set a non-zero $loss\_weight$ for all the classifiers simultaneously. Then both the overall loss and separate supervision are considered during the update of the feature extractors. In our experiments, we first specialize the three feature extractors as in Figure 14, and then fine-tune the overall network with $loss\_weight=1$ for the final classifier. During the overall fine-tuning, we try both $loss\_weight=0$ and $loss\_weight=0.3$ for the classifiers directly following the three feature extractors, and these two strategies achieve almost the same test accuracy.

During the concatenation of the feature vectors of the two parts, we order them according to the sizes of their detected regions. Although this appears to be too simplistic, our experimental results to be presented in Sec. 4 show that it works surprisingly well. When there are only a small number of parts, the different parts usually differ in size, e.g., the head and body of humans or animals. This scheme may also be generalized to more parts by detecting them in a hierarchical way. To do so, we first roughly decompose an object into a few major parts. This is then followed by detecting finer parts within each major part. Not only does this simplify the identification of detected parts, but it also increases the robustness of the detection results since the finer parts are constrained in their location by the major part to which they belong. In Subsec 3.5, we mention that the object pose can be estimated based on our object and part detection results, these results can assist in identifying and ordering the parts as well.

4 Experiments

In this section, we report the experimental validation of our proposed method. Subsec 4.1 shows the object detection and part detection results. Subsec 4.2 uses some examples to illustrate that the detection results are not sensitive to the thresholds $T_{object}$ and $T_{parts}$. Since our proposed method integrates multiple CNNs, the experiments in Subsec 4.3 are conducted to explain away that the performance gain does not comes from ensemble. Finally, comparison with our baselines and other methods are listed in Subsec 4.4.

We apply our method on the famous Caltech-UCSD Birds-200-2011 [56], FGVC-Aircraft [57], cars [61] and Stanford dogs [62] datasets. Caltech-UCSD Birds-200-2011 dataset [56] contains 11,788 images in total from 200 species of birds. Each of the 200 categories has about 60 images. We use the default data split which randomly assigns about half of the images for each category to the training set. FGVC-Aircraft dataset [57] has 100 kinds of aircrafts and contains 10,000 images in total, and we use 6,667 images for training and 3,333 for testing. The cars [61] dataset contains 16,185 images of 196 classes of cars. The data is split into 8,144 training images and 8,041 testing images, where each class has been split roughly in a 50-50 split. The Stanford Dogs dataset contains 20,580 images of 120 breeds of dogs from around the world and 12,000 of them are used for training, leaving 8,580 images for testing. Some sample images from the four datasets are shown in Figure, where each row shows seven images from a dataset. Our CNN-based system is implemented using Caffe [50], built and tested on GoogLeNet [31], VGG 19-layers [30] and VGG-CNN-S [51] in Caffe model zoo.

4.1 Object Detection and Part Detection Results

Our algorithm detects the object by summing the feature maps of layers in $\mathcal{S}^{layer\_idx}_{object}$, and the bounding box can be obtained by directly cropping the rectangle that tightly contains the thresholded active region. However, to be conservative, we extend the height and width of the rectangle by $5\%$. Both the results with and without conservative extension are evaluated using Recall-to-IoU shown in Figure 17 (better viewed in color). The results with extended boundary use star markers and are annotated by ‘margin’ in the legend. From left to right are the plots of Caltech-UCSD Birds-200-2011 [56], FGVC-Aircraft [57], cars [61] and Stanford dogs [62] datasets respectively. Each plot contains the results based on three CNN models: GoogLeNet [31], VGG 19-layers [30] and VGG-CNN-S [51]. It can be seen that the detection results vary between network models and datasets. In general, relatively shallower networks like VGGs [30, 51] outperforms the deep GoogLeNet [31], since pooling layers reduce the resolution of the feature maps. The results with conservative extension are slightly better than that without.

Among the four datasets only Caltech-UCSD Birds-200-2011 [56] offers the ground truth part annotations. The locations of 15 parts listed in Table 2 are specified by 2-D coordinates if visible. Our system detects two parts of the bird. For each detected part, we calculate the difference between its coordinates to all the available annotated parts. Next ,we first normalize the x and y coordinates of the difference by the width and height of the ground truth bounding box respectively, then calculate its norm. The averaged results are shown in Figure 18. The left column in Figure 18 shows the detection results directly obtained from the activation of the CNN hidden layer feature maps, the right column shows the results after shifting based on the pose estimated in Subsec 3.5. The three plots in each column from top to bottom show the detection results using GoogLeNet [31], VGG 19-layers [30] and VGG-CNN-S [51] respectively. It can be seen that both the curves of part 1 (i.e. the head) and part 2 (i.e. the body) have their featured shapes which are consistently kept through all the cases. Part 1 is always very close to the eyes, the forehead, the throat and the crown, while very far away from the belly, the wings, the legs and the tail. On the contrary, part 2 behaves almost oppositely. In the left column, both part 1 and part 2 are relatively far away from the tail and part 1 is much farther than part 2. After shifting based on the pose, the center of part 2 move much closer to the tail. In general, the cures in the right column are lower than that in the left column, which means that to adjust the detection results based on the estimated pose increases the part detection accuracy.

4.2 Robustness of Detection Results to $T_{object}$ and $T_{parts}$

Figure 19 illustrates our object detection result and how it varies with $6$ different choices of $T_{object}$. The first two images in each row show a resized original image and its soft-masked version. We threshold the soft mask $\mathbf{M}_{object}$ with $T_{object}=0.2,0.25,\cdots,0.45$ to obtain $\mathbf{M}^{binary}_{object}$ and display the result of applying $\mathbf{M}^{binary}_{object}$ to the original images. It can be seen that, as expected, the higher $T_{object}$ is, the smaller is the active region in $\mathbf{M}^{binary}_{object}$. In general, the object detection result is not sensitive to $T_{object}$, which means that the object saliency masks are rather confident to distinguish between the foreground and background. Similar observation can be found in Figure 20 for part detection of $N_{part}=2$, in which each example is shown in two consecutive rows. The original image is displayed in the upper-left corner and below it shows the object-focused image masked by $\mathbf{M}^{binary}_{object}$. Column $2$ shows the masks of two detected parts with the one having a smaller region on top. Columns $3$ to $8$ show the part-focused images generated with $T_{parts}=0.2,0.25,\cdots,0.45$. To be conservative, when we crop the object-focused and part-focused images, we extend the height and width of the rectangle by $5\%$. It is clear that the two parts focused are the head and body of a bird and this result is quite robust.

4.3 Explain away Ensemble

Though our method integrates multiple CNNs, the performance gain does not come from ensemble. This is proofed with the classification results in Table 3 and Table 4, which include our base line, the result using proposed method with $N_{part}=1,2,3,4$, and the ensemble results using the same network structure. ‘GoogLeNet-ft’ refers to the fine-tuned GoogLeNet [31] pre-trained on the ImageNet dataset [63], with its classification layer replaced to suit our fine-grained categorization datasets; ‘2 CNNs ensemble’ means that the input of the two integrated CNNs are both the raw images, similarly for ‘3 CNNs ensemble’ and ‘4 CNNs ensemble’; ‘2 CNNs: raw + obj’ means using the concatenated feature of two CNNs, one input is original raw images, the other input is object-focused images ($N_{part}=1$); ‘3 CNNs: raw+2 parts’ means the inputs of the 3 integrated CNNs are raw images and 2 parts-focused images, similarly for ‘4 CNNs: raw+3 parts’. Comparing the results of ‘GoogLeNet-ft’ and ‘3 CNNs: raw+2 parts’, it can be seen that the proposed method greatly outperforms our baseline ($81.38\%$ vs. $73\%$, $82.7\%$ vs. $75\%$). With the same number of CNNs integrated, the performance achieved by simple ‘ensemble’ method largely lags behind the proposed method. This shows the effectiveness of our CNN-based part feature extraction on fine-grained classification. From the results of using different $N_{part}$, we can see that the best performance is achieved by $N_{part}=2$, which is consistent with our analysis in Subsec 3.4.2.

4.4 Comparison with Other Methods

Table 5, Table 7, Table 6, Table 8 and Table 9 compares many existing methods applied to the Caltech-UCSD Birds-200-2011 [56], FGVC-Aircraft [57], cars [61] and Stanford dogs [62] datasets. The proposed method is built and tested on GoogLeNet [31], VGG 19-layers [30] and VGG-CNN-S [51]. Directly fine-tuning these CNNs can be viewed as our baselines. The result of proposed method together with their baseline results (in total 6 terms) are listed at the bottom. For all our fine-tuning operations, we adopt the two-step fine-tuning method in [4]. Similar as [28], during testing, we crop five images from the original image at the four corners and the center, flipping them to form 10 crops. Doing so leads to an increase in accuracy by about $1\%$. We can see that our proposed method improves the performance of our baselines in all cases with maximum categorization accuracy increased by $14.26\%$ (using VGG-CNN-S [51] on cars [61]), and achieves superior or comparable performance with that of the state-of-the-art under the general problem setting that only class labels are available during training with no other annotations. Importantly, instead of assembling complex algorithms, our proposed method only takes advantage of the CNN feature maps themselves to achieve good though not the best performance.

5 Conclusion

In this paper, we have proposed a novel CNN-based fine-grained categorization method for the general problem setting that only class labels are available during training with no other annotations. Not only does it achieve accuracy comparable to the state-of-the-art, but it also sheds some light on ingenious use of the features learned by CNN which can find wide applications well beyond the current fine-grained categorization task. Instead of trying to deepen the network architecture as many other researchers did, we seek to exploit the hidden layer feature maps learned to achieve robust object detection, part detection, and pose estimation which together can boost the classification accuracy significantly. Our future work will consider extending this approach to other tasks.

6 Acknowledgements

This work took me long time to complete and my supervisor Prof. D.Y. Yeung and senior labmate Lin Sun have offered me grate guidance and support all the way. Many thanks to them.

7 Funding sources

I am fully supported by the Hong Kong University of Science and Technology.