CHIP: Channel-wise Disentangled Interpretation of Deep Convolutional Neural Networks

TL;DR

CHIP introduces a channel-wise disentangled interpretation method for deep CNNs, providing high-resolution, class-discriminative visual explanations without re-training, and outperforms existing methods in classification and localization tasks.

Contribution

The paper proposes a novel CHIP model that distills class-discriminative channel importance and offers layer-wise, high-resolution visual interpretations without re-training.

Findings

Provides promising visual explanations for CNN predictions.

Outperforms existing methods in weakly-supervised localization.

Capable of interpreting different network layers without re-training.

Abstract

With the widespread applications of deep convolutional neural networks (DCNNs), it becomes increasingly important for DCNNs not only to make accurate predictions but also to explain how they make their decisions. In this work, we propose a CHannel-wise disentangled InterPretation (CHIP) model to give the visual interpretation to the predictions of DCNNs. The proposed model distills the class-discriminative importance of channels in networks by utilizing the sparse regularization. Here, we first introduce the network perturbation technique to learn the model. The proposed model is capable to not only distill the global perspective knowledge from networks but also present the class-discriminative visual interpretation for specific predictions of networks. It is noteworthy that the proposed model is able to interpret different layers of networks without re-training. By combining the distilled interpretation knowledge in different layers, we further propose the Refined CHIP visual interpretation that is both high-resolution and class-discriminative. Experimental results on the standard dataset demonstrate that the proposed model provides promising visual interpretation for the predictions of networks in image classification task compared with existing visual interpretation methods. Besides, the proposed method outperforms related approaches in the application of ILSVRC 2015 weakly-supervised localization task.

Tables

Table 1. TABLE I: Properties of different visual interpretation methods

	Instance interpretation		General	Class-discriminative	High-resolution	Multi-layer	Post-hoc
	Perturbation-based	Gradient-based	interpretation	interpretation	interpretation	interpretation	interpretation
Deconvolution[13]		✓			✓	✓	✓
Guided		✓			✓		✓
Backpropagation[12]
CAM[10]		✓		✓
Grad-CAM[11]		✓		✓			✓
Guided		✓		✓	✓		✓
Grad-CAM[11]
LIME[14]	✓			✓			✓
DeepDraw[15]			✓	✓	✓		✓
Guided CHIP	✓	✓	✓	✓	✓	✓	✓
CHIP &	✓		✓	✓	✓	✓	✓
Refined CHIP

Table 2. TABLE II: Localization and classification errors on ILSVRC 2015 validation set.

Localization method	Top 1 loc error	Top 5 loc error	Classification network	Top 1 cls error	Top 5 cls error
Backpropagation [22]	61.11	51.43
c-MWP [23]	70.92	63.01
			VGG16	30.12	10.85
Grad-CAM [11]	56.47	46.35
CHIP	51.45	40.16

Equations

$${\hat{\mathbf{A}}}_k=d_k{\mathbf{A}}_k$$

$$z_k=\frac{1}{uv}\sum _{i,j}{\hat{\mathbf{A}}}_k(i,j)$$

$$g({\mathbf{X}}_s,{\mathbf{d}}_n)=[g^1({\mathbf{X}}_s,{\mathbf{d}}_n)\cdots g^c({\mathbf{X}}_s,{\mathbf{d}}_n)\cdots g^C({\mathbf{X}}_s,{\mathbf{d}}_n){]}^T$$

$$f({\mathbf{X}}_s)=[f^1({\mathbf{X}}_s)\cdots f^c({\mathbf{X}}_s)\cdots f^C({\mathbf{X}}_s){]}^T$$

$$\begin{array}{rl}\arg \underset{\mathbf{W}}{\min }& \frac{1}{2}\sum _{n=1}^{N}h({\mathbf{d}}_n)\sum _{s=1}^S\Vert \pi ({\mathbf{X}}_s)\odot [\mathbf{W}{\mathbf{z}}_{s,n}-g({\mathbf{X}}_s,{\mathbf{d}}_n)]{\Vert }_2^2\\ & +\lambda \Vert \mathbf{W}{\Vert }_1\end{array}$$

$$\mathcal{L}=\frac{1}{2}\sum _{n=1}^{N}h({\mathbf{d}}_n)\sum _{s=1}^S\Vert \pi ({\mathbf{X}}_s)\odot [\mathbf{W}{\mathbf{z}}_{s,n}-g({\mathbf{X}}_s,{\mathbf{d}}_n)]{\Vert }_2^2$$

$$\pi ({\mathbf{X}}_s)=[\sqrt{f^1({\mathbf{X}}_s)}\cdots \sqrt{f^c({\mathbf{X}}_s)}\cdots \sqrt{f^C({\mathbf{X}}_s)}{]}^T$$

$$h({\mathbf{d}}_n)=\exp (-\frac{1}{{\sigma }^2}\Vert {\mathbf{d}}_n-1{\Vert }_2^2).$$

$$\Phi (\mathbf{W})=\Vert \mathbf{W}{\Vert }_1$$

$${\tilde{\mathbf{A}}}^c=\sum _k{w}_c^k{\mathbf{A}}_k$$

$$\begin{array}{rl}\arg \underset{{\mathbf{w}}_c}{\min }& \frac{1}{2}\sum _{n=1}^{N}h({\mathbf{d}}_n)\sum _{s=1}^S{f}^c({\mathbf{X}}_s)[{\mathbf{w}}_c{\mathbf{z}}_{s,n}-g^c({\mathbf{X}}_s,{\mathbf{d}}_n){]}^2\\ & +\lambda \Vert {\mathbf{w}}_c{\Vert }_1\end{array}$$

$$\begin{array}{rl}\arg \underset{{\mathbf{w}}_c,{\mathbf{m}}_c}{\min }& \frac{1}{2}\sum _{n=1}^{N}h({\mathbf{d}}_n)\sum _{s=1}^S{f}^c({\mathbf{X}}_s)[{\mathbf{w}}_c{\mathbf{z}}_{s,n}-g^c({\mathbf{X}}_s,{\mathbf{d}}_n){]}^2\\ & +\lambda \Vert {\mathbf{m}}_c{\Vert }_1\\ \text{subject to}& {\mathbf{w}}_c={\mathbf{m}}_c\end{array}$$

$$\begin{array}{rl}\arg \underset{{\mathbf{w}}_c,{\mathbf{m}}_c,\mathbf{p}}{\min }& \frac{1}{2}\sum _{n=1}^{N}h({\mathbf{d}}_n)\sum _{s=1}^S{f}^c({\mathbf{X}}_s)[{\mathbf{w}}_c{\mathbf{z}}_{s,n}-g^c({\mathbf{X}}_s,{\mathbf{d}}_n){]}^2\\ & +\lambda \Vert {\mathbf{m}}_c{\Vert }_1+{\mathbf{p}}^T({\mathbf{w}}_c-{\mathbf{m}}_c)+\frac{\rho }{2}\Vert {\mathbf{w}}_c-{\mathbf{m}}_c{\Vert }_2^2\end{array}$$

$$\begin{array}{rl}\arg \underset{{\mathbf{w}}_c,{\mathbf{m}}_c,\mathbf{q}}{\min }& \frac{1}{2}\sum _{n=1}^{N}h({\mathbf{d}}_n)\sum _{s=1}^S{f}^c({\mathbf{X}}_s)[{\mathbf{w}}_c{\mathbf{z}}_{s,n}-g^c({\mathbf{X}}_s,{\mathbf{d}}_n){]}^2\\ & +\lambda \Vert {\mathbf{m}}_c{\Vert }_1+\frac{\rho }{2}\Vert {\mathbf{w}}_c-{\mathbf{m}}_c-\mathbf{q}{\Vert }_2^2\end{array}$$

$$\mathbf{q}\equiv -\frac{1}{\rho }\mathbf{p}$$

$$\begin{array}{rl}{\mathbf{w}}_c\leftarrow \arg \underset{{\mathbf{w}}_c}{\min }& \frac{1}{2}\sum _{n=1}^{N}h({\mathbf{d}}_n)\sum _{s=1}^S{f}^c({\mathbf{X}}_s)[{\mathbf{w}}_c{\mathbf{z}}_{s,n}-g^c({\mathbf{X}}_s,{\mathbf{d}}_n){]}^2\\ & +\frac{\rho }{2}\Vert {\mathbf{w}}_c-{\mathbf{m}}_c-\mathbf{q}{\Vert }_2^2\end{array}$$

$$\begin{array}{rl}{\mathbf{w}}_c^{i+1}\leftarrow & (\sum _{s,n}{f}^c({\mathbf{X}}_s)h({\mathbf{d}}_n)g^c({\mathbf{X}}_s,{\mathbf{d}}_n){{\mathbf{z}}_{s,n}}^T+\rho {\mathbf{m}}_c^i\\ & +\rho {\mathbf{q}}^i)(\sum _{s,n}{f}^c({\mathbf{X}}_s)h({\mathbf{d}}_n){\mathbf{z}}_{s,n}{{\mathbf{z}}_{s,n}}^T+\rho \mathbf{I}{)}^{-1}\end{array}$$

$${\mathbf{m}}_c\leftarrow \arg \underset{{\mathbf{m}}_c}{\min }\lambda \Vert {\mathbf{m}}_c{\Vert }_1+\frac{\rho }{2}\Vert {\mathbf{w}}_c-{\mathbf{m}}_c-\mathbf{q}{\Vert }_2^2$$

$${\mathbf{m}}_c^{i+1}\leftarrow soft({\mathbf{w}}_c^{i+1}-{\mathbf{q}}^i,\frac{\lambda }{\rho })$$

$${\mathbf{q}}^{i+1}\leftarrow {\mathbf{q}}^i-({\mathbf{w}}_c^{i+1}-{\mathbf{m}}_c^{i+1})$$

$$\text{IoU}=\frac{\text{Area of Overlap}}{\text{Area of Union}}$$

Full text

I Introduction

Deep convolutional neural networks (DCNNs) have achieved impressive performances in many computer vision tasks such as image classification [1], object detection [2], image captioning [3], and visual question answering [4]. However, the complex internal working mechanism of DCNNs remains elusive and makes them highly non-transparent to human [5]. The lack of interpretability of DCNNs may result in a general mistrust of their results and further hinder their adoption, especially for critical applications such as medical diagnosis and criminal justice [6].

Interpretability study has drawn increasing attention, e.g., for building trust for real-world users and offering insight into black-box models for practitioners [7, 8]. Visual interpretation is one prevalent interpretation method. A number of visual interpretation models were proposed to interpret the decisions of DCNNs [9, 10, 11]. A desired visual interpretation model should possess the following properties:

1. $\bullet$

The model should give class-discriminative visual interpretation, because the DCNN being explained is designed to distinguish different classes. The class-discriminative visual interpretation should highlight the critical region of the input for a specific class prediction. 2. $\bullet$

The model should be able to present the fine-grained information that is important to the class prediction. 3. $\bullet$

The model is supposed to obtain interpretable knowledge from different layers and lend insight into the roles of different layers of DCNNs. 4. $\bullet$

Given an image, the model should give a specific explanation for the network decision. People can decide whether to trust that decision based on the instance interpretation. 5. $\bullet$

The model should give global interpretation of networks. General interpretation knowledge should be distilled from a large image dataset rather than from a single image.

It should also be noted that there is a trade-off between the performance and the interpretability of DCNNs. Pursuing the interpretability of DCNNs may sacrifice their accuracies, which is undesirable. We therefore focus on the post-hoc interpretability which refers to the extraction and analysis of information from a trained network. In this situation, the network usually does not have to sacrifice its performance in order to be interpretable.

For visual interpretation, several approaches have been proposed to highlight a portion of internal features that are critical to the decision by ascribing saliency [10, 11, 12, 13]. However, these methods have certain limitations. For example, Guided Backpropagation [12] cannot give class-discriminative visualization. Class Activation Mapping (CAM) [10] and Grad-CAM [11] are only constrained in the last convolutional layer to present the visual interpretation. Table I shows the properties of different visual interpretation methods.

In order to alleviate the above issues, we propose a CHannel-wise disentangled InterPretation (CHIP) model which can build trust in networks without sacrificing their accuracies. The main contributions of this work are summarized as follows:

1. $\bullet$

The proposed CHIP model can disentangle channels in a network to interpret internal features in different layers. Based on that, CHIP can give visual interpretations to network decisions without re-training. 2. $\bullet$

In CHIP, we first introduce the channel-wise network perturbation method by perturbing network features. The underlying principle is that the class prediction would drop dramatically if the forward propagation of class-specific important channels is blocked. Our model also utilizes the inherent sparse property of net-features as a regularization. 3. $\bullet$

CHIP can distill the class-discriminative knowledge of a network from a large dataset and utilize the distilled knowledge to interpret the net decision for an instance. It means our model can not only explore the general interpretation information without data bias but also give a visual interpretation for the prediction of a particular input. 4. $\bullet$

Through combining CHIP visual interpretations in the shallow and the deep layers, we extend CHIP to Refined CHIP which can give a high-resolution class-discriminative interpretation. 5. $\bullet$

The proposed CHIP model can be applied to tackle the weakly-supervised localization task. It achieves better performance on ILSVRC 2015 dataset when compared with previous related work.

II Related Work

The most related studies to our work are visual interpretation of DCNNs and its application in the weakly-supervised localization, as described below:

Visual interpretation of DCNNs. Visual interpretation of DCNNs can be classified into two major categories: one is instance interpretation which means the interpretation for a particular input; and the other is general interpretation which denotes the interpretation of the overall network. Both focus on providing visual interpretation to explore the internal working mechanism of DCNNs. Table I shows different visual interpretation methods and their properties.

Methods in the first category can be further divided into gradient-based and perturbation-based. Gradient-based approaches give interpretation by using the gradient (or gradient variant) of the output or the internal unit. For instance, Guided Backpropagation [12] and Deconvolution [13] interpret the network by visualizing internal units. They design different backward pass ways to map neuron activation down to the input space, visualizing the input image pattern that is most discriminative to the neuron activation. Although they can give high-resolution visualization for a specific input image, the visualization is not class-discriminative. In contrast, CAM [10] and Grad-CAM [11] are able to give the class-discriminative interpretation. Specifically, they visualize the linear combination of activations and class-specific weights in the last convolutional layer. However the obtained visualization is not high-resolution and does not show fine-grained details. To tackle this problem, Guided Grad-CAM was designed by combining Guided Backpropagation visualization. Another common drawback is that their visual interpretation is only effective for the last convolutional layer. In addition, CAM can only provide interpretation for a specific kind of DCNN architecture.

Perturbation-based methods mainly focus on the input image perturbation [9, 14]. To be specific, they involve perturbing input images and observing the change of predictions. The intuitive reason is that the prediction would drop by the maximum amount when pixels contribute maximally to the prediction are modified. Although these methods can obtain the class-discriminative visualization, they do not explore the internal mechanism of networks or interpret internal features. Meanwhile, their performances are affected by the size and the shape of occluded pieces in perturbed images. For instance, image patches in regular grids are used for occlusion in [13] while super-pixels from segmentation are used in [14].

Methods in the second category focus on visualizing DCNNs in a global perspective by synthesizing the optimal image that gives maximum activation of a unit [15, 16]. Although these methods are able to give class-discriminative interpretation, they are not designed for interpreting specific input images. In some sense, they only provide high-level abstract interpretation and are not helpful to understand specific decisions of DCNNs.

There is a compromise between instance interpretation and general interpretation: Instance interpretation can explain network predictions for a given input; while the knowledge obtained by the general interpretation is more stable and representative without data bias. To balance the desired characteristics of two interpretation categories, our proposed interpretation model is designed to distill the class-discriminative knowledge of a network from a large dataset and then utilize the knowledge to explain specific predictions. Further, Refined CHIP is designed to provide visual interpretation that is both class-discriminative and high-resolution.

Weakly-supervised Localization. This task refers to localizing the object only with the image-level class label. One approach is to utilize the visual interpretation distilled from the network to localize the target object [10], [11]. From this point of view, visual interpretation models can be applied to weakly-supervised localization.

In [10], a network should be modified to a particular kind of architecture to learn class activation maps which can be used to generate predicted bounding boxes for weakly-supervised localization. In the modified architecture, the convolutional layer is followed by the global average pooling layer and then the softmax layer. Compared with the original network, the modified network architecture may achieve inferior classification accuracy. Therefore, the localization accuracy would also be limited by the inferior accuracy of the modified network. In Grad-CAM, the gradients of the output to feature maps are used as the weights of feature maps to obtain class activation maps. Grad-CAM does not need to modify the network architecture and can be applied to different networks. It should be noted that the class-discriminative weights in Grad-CAM are only obtained from a single image. In contrast, our proposed CHIP model distills the class-discriminative knowledge from a large dataset, which provides more insight into the network and can also enhance the localization accuracy.

III Methodology

In this section, we present the proposed CHIP model whose objective is to provide insight into DCNNs based on the distilled class-discriminative knowledge. In section III-A, we describe the disentangled channels based on perturbed networks. In section III-B, we present the CHIP model to distill important channels for different classes. With this model, we also describe the way to obtain class-discriminative visual interpretations for the decisions of networks and apply it to weakly-supervised localization. Section III-C describes the proposed CHIP algorithm.

III-A Disentangled channels based on Perturbed Networks

Class-discriminative importance of channels. To disentangle the roles of channels, our interpretation model is designed to learn the importance of channels for different classes. Through turning off the forward propagation of partial channels each time, we can learn the class-discriminative importance of channels by analyzing the variation of predictions. The underlying principle is that the class prediction would drop dramatically if the forward propagation of important channels is blocked.

Specifically, for the $l$-th layer, the class-discriminative importance matrix is denoted as $\mathbf{W}=[\mathbf{w}_{1}~{}\mathbf{w}_{2}\cdots\mathbf{w}_{c}\cdots\mathbf{w}_{C}]^{T}$, where $C$ is the number of image category. For the $c$-th class, the class-discriminative importance is $\mathbf{w}_{c}=[w_{c}^{1}~{}w_{c}^{2}\cdots w_{c}^{k}\cdots w_{c}^{K}]$, where $w_{c}^{k}$ represents the importance of the $k$-th channel and $K$ is the number of channels in the $l$-th layer.

Channel-wise perturbed networks. Inspired by the perturbation-based method [14], we perturb a pre-trained network $f(\cdot)$ by channel gates to learn the class-discriminative importance of channels. As shown in Fig. 1(top), each original layer is associated with a gate layer in which each channel gate controls the state of the corresponding channel in the former layer. Here, in the $l$-th layer, we use a binary vector $\mathbf{d}=[d_{1}~{}d_{2}\cdots d_{k}\cdots d_{K}]^{T}$ to denote the channel gate layer. The $k$-th channel is turned off if $d_{k}$ is zero.

The perturbed network is generated by adding the channel gate layer after each original layer. We denote the original features in the $l$-th layer as $\mathbf{A}\in\mathcal{R}^{u,v,K}$, where $u$ and $v$ are the width and height of a channel. For the $k$-th channel in the $l$-th layer, the output of the channel gate layer is

[TABLE]

For the $l$-th layer, the global average pooling of the channel gate layer is $\mathbf{z}=[z_{1}~{}z_{2}\cdots z_{k}\cdots z_{K}]^{T}$. For the $k$-th channel, the global average pooling is

[TABLE]

In the perturbed network, we add control gate layers behind each original layer without changing the original weight of the pre-trained network.

Perturbed dataset. In order to learn the class-discriminative importance of channels, we need to generate the perturbed dataset, as shown in Fig. 1(bottom). Specifically, to learn $\mathbf{W}$ for the $l$-th layer, the perturbed dataset is obtained as follows: We first generate the perturbed networks. For the chosen $l$-th layer, we generate a channel gate sample $\mathbf{d}_{n}\in\{0,1\}^{K}$ by selecting elements of $\mathbf{d}_{n}$ as 1 uniformly at random. Here, the number of the selected elements with value 1 is also uniformly sampled. For other layers, we freeze the channel gates to be open. We obtain the channel gate dataset $\mathcal{D}_{gate}={\{\mathbf{d}_{n}\}}_{n=1}^{N}$ by generating $N$ channel gate samples.

Secondly, we feed each image from image dataset $\mathcal{D}_{img}={\{\mathbf{X}_{s}\}}_{s=1}^{S}$ into each perturbed network and get the results. For example, the result of the $n$-th perturbed network for the $s$-th image is denoted as

[TABLE]

where $g^{c}(\mathbf{X}_{s},\mathbf{d}_{n})$ is the prediction for the $c$-th class. Meanwhile, its global average pooling in the $l$-th channel gate layer is recorded as $\mathbf{z}_{s,n}$.

Finally, we get the perturbed dataset $\mathcal{D}\{\mathbf{X}_{s},\mathbf{d}_{n},\mathbf{z}_{s,n},f(\mathbf{X}_{s}),g(\mathbf{X}_{s},\mathbf{d}_{n})\}$, where the results of the original network are denoted as

[TABLE]

III-B CHIP Model

Given the perturbed dataset $\mathcal{D}$, we formulate the CHIP model as

[TABLE]

where $\lambda$ is the regularization parameter, and $\odot$ denotes Hadamard product. The first term is the loss function,

[TABLE]

In order to learn the class-discriminative importance weight, we approximate $\mathcal{L}$ by drawing perturbed samples which are weighted by $\pi(\mathbf{X}_{s})$ and $h(\mathbf{d}_{n})$.

The perturbed dataset contains the data of different images from the image dataset $\mathcal{D}_{img}={\{\mathbf{X}_{s}\}}_{s=1}^{S}$. To learn the class-discriminative importance, we denote $\pi(\mathbf{X}_{s})$ as the loyalty measure of image $\mathbf{X}_{s}$ to different classes. For a given image, its loyalty for learning the class-discriminative importance of net-units to the $c$-th class is large when its $f^{c}(\mathbf{X}_{s})$ is high. Specifically, it is defined as

[TABLE]

where $f^{c}(\mathbf{X}_{s})$ denotes the network prediction that $\mathbf{X}_{s}$ belongs to the $c$-th class.

The perturbed dataset also contains the data of different perturbed nets from the channel gate dataset $\mathcal{D}_{gate}={\{\mathbf{d}_{n}\}}_{n=1}^{N}$. For a given perturbed net, $h(\mathbf{d})$ is denoted as the proximity measure between a binary channel gate vector $\mathbf{d}$ in the perturbed net and the all-one vector $\mathbf{1}$ in the original net. In the perturbed dataset, we sample perturbed nets both in the vicinity of the original net ( high $h(\mathbf{d}_{n})$ weight in Eq. (6)) and at a distance from the original net (low $h(\mathbf{d}_{n})$ weight), where

[TABLE]

The second term in our model is the sparse regularization term,

[TABLE]

which measures the sparsity of the weight. We use sparsity to measure the complexity of the interpretation model because of the inherent sparse property of features in DCNNs. Also, the interpretation model needs to be simple enough to be interpretable.

Fig. 2 illustrates the workflow of the proposed CHIP model. CHIP learns the class-discriminative importance of channels for different classes. The distilled knowledge can be further applied to visual interpretation and weakly-supervised object localization.

CHIP Interpretation. By combining the class-discriminative importance and corresponding feature maps, we can get the class-discriminative visual interpretation for a specific network decision of an input image. Specifically, for a certain layer, the visual interpretation for the $c$-th class prediction of a particular instance is denoted as

[TABLE]

where $w_{c}^{k}$ represents the optimal importance of the $k$-th channel to the $c$-th class, and ${\mathbf{A}_{k}}$ denotes the feature in the $k$-th channel for the given image.

Refined CHIP Interpretation. The CHIP model can give the visual interpretation for a network decision. Further, we design the Refined CHIP to present a high-resolution visual interpretation which can show detailed features distilled from the network. It is commonly known that features in a shallow layer are of a higher resolution than that in a deep layer. Conversely, the semantic representation in a deep layer is at a higher level than that in a shallow layer. Likewise, CHIP interpretation for shallow layers and deep layers also possess similar properties. We can combine the distilled interpretation in different layers to obtain the Refined CHIP visual interpretation.

Specifically, the Refined CHIP result is obtained by the point-wise multiplication of CHIP visual interpretations for the first and the last convolutional layers. Therefore, Refined CHIP interpretation is not only high-semantic but also high-resolution. It utilizes the distilled interpretation knowledge in different layers, which also reveals the roles of different layers. Recall that Guided Grad-CAM combines Guided Backpropagation visualization with Grad-CAM to give a high-resolution class-discriminative interpretation. For comparison, we also combine Guided Backpropagation [12] and CHIP interpretation to get Guided CHIP interpretation.

Weakly-supervised Localization. It is well known that deep layers in DCNNs capture high-level semantic information which can be regarded as the object saliency information for localization. Therefore, the intuition is that our model distills the class-discriminative knowledge in the deep layer and transfers this knowledge from the pre-trained classification network to the localization task. Here, the visual interpretation can be regarded as a saliency map to localize the object. The bounding box can be obtained based on the saliency map distilled from the last convolutional layer. Because the visual interpretation obtained by our model is learned without the ground truth bounding box annotation, our localization method is also a weakly-supervised approach.

III-C CHIP Algorithm

The proposed CHIP model is used to distill important channels of a pre-trained network for different image categories. Here, we propose a channel-wise disentangled interpretation algorithm to optimize the CHIP model.

The optimization of CHIP model in Eq. (5) can be divided into solving the optimization problem for each class separately. For the $c$-th class, the optimization problem turns into

[TABLE]

The optimization problem in Eq. (11) is convex. Here, we design a channel-wise disentangled interpretation algorithm by adopting the alternating iteration rule to learn $\mathbf{w}_{c}$.

The optimization problem can be converted into the equivalent formulation

[TABLE]

The augmented Lagrangian for the above problem is

[TABLE]

The equation can be rewritten as

[TABLE]

where

[TABLE]

Through a careful choice of the new variable, the initial problem is converted into a simple problem. Given that the optimization is considered over the variable $\mathbf{w}_{c}$, the optimization function can be reduced to

[TABLE]

The solution is

[TABLE]

To calculate $\mathbf{m}_{c}$, the optimization problem to be solved is

[TABLE]

The solution is

[TABLE]

Lagrange multipliers update to

[TABLE]

The pseudocode of the CHIP algorithm is shown in Algorithm 1.

IV Experiments

In this section, we evaluate the performances of the CHIP model and the Refined CHIP interpretation. We first describe experiment settings of our interpretation model in section IV-A. We then conduct experiments in five aspects to qualitatively and quantitatively evaluate the proposed model.

Specifically, in section IV-B1, we show the CHIP and the Refined CHIP visual interpretation of the VGG16 network [17] for object classification based on the ILSVRC 2015 object classification dataset [18]. In section IV-B2, the performance of the CHIP model on the VGG16 network for object classification is quantitatively evaluated based on the ILSVRC 2015 object localization dataset. In section IV-C, we qualitatively and quantitatively evaluate the performance of the proposed model on the Inception-V3 network [19] for scene classification task based on the ADE20K dataset [20]. In section IV-D, we show the visual interpretation for different layers of the network. In section IV-E, the class-discriminative importance of channels for different classes are compared.

IV-A Experiment Settings

IV-A1 Networks Being Interpreted

The CHIP model is applied to interpret the decisions of VGG16 and Inception-V3.

VGG16 on the ILSVRC 2015 object classification dataset: To be consistent with the previous work, we use the off-the-shelf VGG16 model from the Caffe Model Zoo as one network to be interpreted. VGG16 involves 16 layers with learnable weight. Different layers have different number of channels. For example, the first convolutional layer contains 64 channels while the last convolutional layer has 512 channels. The object classification dataset contains 1000 object classes.

Inception-V3 on the Places365 scene classification dataset: Inception-V3 is a DCNN that belongs to the GoogLeNet series. The network is 48 layers deep and can learn rich feature representations. Places365 scene classification dataset involves 365 scene classes [21].

IV-A2 Quantitative Evaluation

The following human-annotated datasets are used for quantitative evaluation.

ILSVRC 2015 object localization dataset: To quantitatively evaluate the CHIP model for VGG16 on object classification, we utilize the ILSVRC 2015 object localization dataset and the Intersection over Union (IoU) metric. In this dataset, images are labeled with ground-truth classes and corresponding bounding boxes.

ADE20K scene understanding dataset: To quantitatively evaluate the CHIP model for Inception-V3 on scene classification, we utilize the ADE20K scene understanding dataset with pixel-wise annotations. ADE20K and Places365 have many overlapped scene classes. Therefore, these overlapped scene classes in ADE20K with pixel-wise annotations can be used to assess the CHIP interpretation for Inception-V3 on Places365. ADE20K contains object segmentation annotation for each image. Based on the IoU metric, quantitative evaluation is computed by comparing the CHIP interpretation with the critical object annotation for the target scene class.

In the quantitative comparison, the IoU metric is defined as:

[TABLE]

where the numerator refers to the area of overlap between the CHIP interpretation region and the ground-truth region for the target category, and the denominator denotes the area of the union of the two regions.

IV-A3 The Learning of CHIP Model

By adding gate layers on top of each original layer in the network being interpreted, we can build perturbed networks for the learning of our interpretation model.

In order to learn the class-discriminative importance of channels in a specific layer for different classes, we generate $100\times Image\,Number\,in\,Dataset$ perturbed networks. For the interpretation of VGG16 on the ILSVRC 2015 object classification dataset, we learn the CHIP model by using the training dataset which includes about 1.3 million images of total 1000 classes. For each class, the number of images ranges from 732 to 1300. Therefore, for the 1000 object classes, the number of perturbed networks is around a hundred million. In each perturbed network, we only perturb the layer of interest, while keeping other gate layers in open states. We get the perturbed dataset by feeding images into perturbed networks, as described in Section III-A. After the optimization, we can get the class-discriminative importance of channels in the layer of interest for each class.

IV-B Visual Interpretation for object classification

In this section, we qualitatively and quantitatively evaluate the visual interpretation performances of CHIP and Refined CHIP for VGG16 in object classification task. We learn class-discriminative importance of channels in the first and the last convolutional layers for 1000 classes. Qualitatively, the CHIP model provides visual interpretation results to explain network predictions. Quantitatively, it is evaluated by IoU in weakly-supervised localization task.

IV-B1 Qualitative evaluation on the ILSVRC 2015 object classification dataset

For qualitative evaluation, we choose images from ILSVRC 2015 object classification validation dataset as the test images.

Previous work has demonstrated that the semantic representation in a deep convolutional layer is more class-specific than that in a shallow convolutional layer. In light of that, CHIP model focuses on the class-discriminative visual interpretation in the last convolutional layer. Meanwhile, features in a shallow layer are of a higher resolution than that in a deep layer. Therefore, the distilled interpretation in the first and the last convolutional layers can be combined to obtain the Refined CHIP interpretation which is both high-semantic and high-resolution. We compare the CHIP and Refined CHIP with LIME [14], Grad-CAM [11], Guided Backpropagation [12] and Guided Grad-CAM [11]. LIME and Grad-CAM can only present coarse class-discriminative visual interpretation.

Fig. 3 shows the comparison of visual interpretations of different methods for simple images, each of which only contains a single target of the selected class. E.g., the first original image contains a dog, and visual interpretation results of different methods are provided for the dog class. For LIME, the class-discriminative importance is calculated for each super-pixel. Then, LIME explains the prediction by selecting image regions that is important to the target class. In the experiment, LIME shows the top 5 important super-pixels for the target class. For Grad-CAM and CHIP, the visual interpretation is designed at the pixel level, which can also be regarded as the saliency map for the target class. Similar to the previous work, visual interpretations of Grad-CAM and CHIP are displayed by superimposing saliency maps on the original image. Because the results of Grad-CAM and CHIP are not high-resolution, this displaying way makes it easier to evaluate whether the important pixels in visual interpretation belong to the object region of the target class. However, this displaying way is not needed for Refined CHIP because it is high-resolution interpretation containing fine-scale features. Therefore, we directly show the interpretation results of Refined CHIP.

From Fig. 3, we can observe that LIME, in most cases, is able to capture partial target object as the visual interpretation, except for the fourth original image where a fish is hiding near coral. The flaw of LIME is that its visual interpretation is limited by the image segmentation result. LIME also cannot explore inside the network, which restricts its performance. Its results usually contain image region that does not belong to the target object, such as the sixth LIME explanation where the sky is included as the critical image region for the airplane.

Compared with LIME and Grad-CAM, CHIP is more reasonable since it always captures the target object. For instance, in the explanation of the fourth original image, LIME and Grad-CAM both miss the fish object region and highlight the background, while the proposed CHIP and Refined CHIP both highlight the correct object region. It illustrates that CHIP and Refined CHIP have better class-discriminative characteristics. On top of CHIP, Refined CHIP presents fine-grained details for the target object, which further illustrates the inherent characteristics of features in different layers. For example, for the last original image in Fig. 3, Refined CHIP depicts the zebra-stripes in the visual interpretation for zebra. In contrast, other compared methods can only provide coarse visual interpretation, such as the super-pixel in LIME and the salient region in Grad-CAM.

Fig. 4 shows the visual interpretation in complex images each of which contains multiple objects of the target class. In this situation, a good visual interpretation is expected to highlight all objects of target category, which is more difficult. In this figure, the compared methods neglect some target objects in some cases. As shown in the second row of Fig. 4, LIME and Grad-CAM both miss one fish. In contrast, CHIP provides more comprehensive visual interpretation which captures all objects of the target category. Refined CHIP further shows more detailed visual interpretation for the target category, such as the explanation for the fourth original image where the contour and texture of airplanes are presented.

Fig. 5 shows the visual interpretation for an image where there are multiple target categories. The target categories are lemon and banana in the original image. The visual interpretation for banana and lemon are respectively in the top and bottom half of Fig. 5. In this figure, we also compare Refined CHIP with Guided Backpropagation, Guided Grad-CAM, and Guided CHIP.

Fig. 5 indicates that CHIP performs more reasonable than Grad-CAM. Grad-CAM does not provide a reasonable explanation for lemon, because it also highlights partial region of banana. Fig. 5 shows Guided Backpropagation provides high-resolution but not class-discriminative visualization, because its visualization results for lemon and banana are similar. The visual interpretation of Guided Grad-CAM and Guided CHIP involve rich details by combining Guided Backpropagation. However, because of the inferior visual interpretation of Grad-CAM, Guided Grad-CAM is also less reasonable than Guided CHIP. As shown in Fig. 5, Guided Grad-CAM for lemon highlights banana region, while Guided CHIP only highlights lemon region. In Fig. 5, Refined CHIP presents visual interpretation that is both high-resolution and class-discriminative without using Guided Backpropagation. Except for the correctly highlighted object region, the fine-grained information in Refined CHIP further identifies the important fine-scale features in the network for the predicted class.

IV-B2 Quantitative evaluation on the ILSVRC 2015 object localization dataset

In this experiment, we assess the CHIP model by the ILSVRC 2015 object localization dataset, the ground-truth of which is human-annotated bounding-boxes. In total, there are 50000 images for the 1000 classes in the validation dataset. The network being interpreted is the VGG16 on the ILSVRC 2015 object classification dataset.

In weakly-supervised object localization task, competing methods should localize the target object without the ground-truth localization annotation. In this experiment, competing approaches are evaluated in the ILSVRC 2015 weakly-supervised localization task where they are required to provide object bounding boxes together with classification predictions. Specifically, given an input image, the original classification network gives its class prediction. And different competing methods are used to learn the saliency map from the interpretation of the classification network for the predicted class. The obtained saliency map is binarized with the optimal threshold of the maximal intensity. The corresponding bounding box is obtained around the largest partition in the binarized saliency map. For each competing method, a grid search is implemented to select the optimal threshold for the best localization performance. Finally, the compared methods can get the bounding boxes as the localization results. Here, quantitative and qualitative analyses are provided to evaluate the competing methods.

For the quantitative comparison, we evaluate the localization error using the IoU metric in which the numerator refers to the area of overlap between the predicted bounding box and the ground-truth bounding box for the target category, and the denominator denotes the area of their union.

Table II shows the localization errors of competing methods in the ILSVRC 2015 validation dataset. Here, the proposed method is compared with Grad-CAM [11], c-MWP[23], and Backpropagation methods [22]. To evaluate the pre-trained VGG16 neural network in the context of image classification, both top 1 and top 5 classification errors in validation dataset are reported in Table II. Because all competing methods are based on the same pre-trained VGG16 network, their classification accuracies are the same. Table II shows both top 1 and top 5 localization errors in validation dataset. Table II indicates that the localization error of our model is lower than other methods. It also indicates that our model provides better class-discriminative saliency maps than other visual interpretation models in this task.

Fig. 6 shows the qualitative and quantitative comparisons of Grad-CAM and the proposed approach. Because Grad-CAM achieves better object localization result than c-MWP and Backpropagation, here we only show the result of Grad-CAM for comparison. It can be seen that the object localization of CHIP with high IoU outperforms that of Grad-CAM. As shown in Fig. 6, the predicted bounding boxes of Grad-CAM often cover the whole image region, such as the left three images.

The intuitive reason is that the saliency map of Grad-CAM sometimes highlights the background region rather than the target object, and thus results in the inaccurate localization. In comparison, the predicted bounding boxes of the proposed CHIP method have a better overlap with ground-truth annotations, resulting in a lower localization error.

IV-C Visual Interpretation for scene classification

In this section, we assess the performance of the CHIP model on the ADE20K scene understanding dataset. The ground-truth of each image in ADE20K is a fine-scale human-annotated label. Therefore, the experiment on this dataset provides an evaluation of the CHIP model by testing whether the visual interpretation of CHIP model matches human knowledge. From this point of view, the CHIP model is evaluated by the subjective assessment in this experiment.

Specifically, the CHIP model is applied to interpret the outputs of Inception-V3 network on the Places365 scene classification dataset. To quantitatively evaluate its performance, we adopt the ADE20K scene understanding dataset in which the critical object for the scene class is labeled by the pixel-wise annotation. In the quantitative evaluation, the IoU metric is measured by comparing the CHIP interpretation with the critical object annotation for the target scene class.

For each testing image, Inception-V3 network outputs its scene classification prediction. Different competing interpretation methods are applied to explain the network decision. The proposed CHIP model can distill the saliency map of the critical object for the target scene class. The interpretation results of the compared methods are obtained by binarizing saliency maps with the corresponding optimal threshold of the maximal intensity.

Fig. 7 shows the qualitative and quantitative evaluations of CHIP, when compared with LIME, Grad-CAM and the ground truth of the critical object for the target scene class. We can see that CHIP outperforms the compared methods. For example, in terms of the interpretation of valley in Fig. 7, Grad-CAM and LIME select the mountain region as the interpretation with low IoUs. In comparison, CHIP can interpret the net prediction with higher IoUs by distilling the valley class-discriminative knowledge from the network.

IV-D Visual Interpretation at Different Layers

In this section, we conduct experiments to show visual interpretations in different layers. We select five pooling layers from VGG16 as the target layers. For each pooling layer, the class-discriminative importance of channels for different classes are learned by the CHIP model. Visual interpretations are then obtained by combining the learned importance of channels with the corresponding features. We choose images from ILSVRC 2015 validation dataset as test images.

Fig. 8 shows visual interpretations for simple images. Here, "simple" means the object of target class is single and can be easily distinguished from background. In Fig. 8, we select images from three classes, cat, dog, and bird. Visual interpretations in the pool1 layer highlight fine-grained object details of the target classes, such as edges and textures shown in the interpretation for cat. In contrast, interpretations in deeper layers capture higher-level semantic features of objects, such as eyes and ears of dog shown in the interpretation of the pool4 layer. This observation is consistent with the previous visualization work [24]. We note that interpretations in different layers highlight the image region that is specific to the class of interest, which qualitatively demonstrates the effectiveness of the class-discriminative property of CHIP.

Fig. 9 compares our visual interpretation with Grad-CAM in different layers for zebra class in complex images. Here, "complex" means there are multiple objects of different classes in a complex background. E.g., the first original image has two zebras and some bisons in the grassland.

In Grad-CAM, visual interpretations are obtained by the weighted sum of features in the last convolutional layer. Grad-CAM is only applied to the last convolutional layer in [11]. Meanwhile, the channel weight in Grad-CAM is obtained from an individual image. Fig. 9 shows that interpretations of Grad-CAM in former layers are not reasonable, which always concentrate on the background. For example, for the top image in Fig. 9, Grad-CAM highlights the background but rather zebra from the pool1 layer to the pool4 layer.

In contrast, because the proposed CHIP model is based on network perturbation, our model can give visual interpretations in different layers. The class-discriminative importance of the CHIP model is obtained from the image dataset. Therefore, CHIP captures more reliable class-discriminative interpretations in different layers. In Fig. 9, the CHIP model can always highlight the zebra region in these layers.

Fig. 8 and Fig. 9 both illustrate that the visualization in a deeper layer captures better class-discriminative representation, which means the saliency of the target object is more obvious in a deeper layer.

IV-E Class-discriminative Importance of channels for Different Classes

In the section, we compare the class-discriminative importance of channels between different classes in the interpretation of VGG16 for object classification. Here, we select three classes as examples: bullfrog, tree frog, and zebra. The former two classes belong to the different species of frog, while the third class barely shares similarity with frog. Because a deep layer can capture a high-level semantic representation, we select the last convolutional layer for comparison.

Fig. 10 shows the comparison of the class-discriminative importance of channels for different classes, where the vertical axis and horizontal axis represent the channel index and the class respectively. We note the sparsity of the class-discriminative importance of channels for different classes, since only a small subset of channels are important for each class. It also shows that similar classes (bullfrog and tree frog) have some common important channels, while different classes (tree frog and zebra) seldom have overlapped important channels.

To further illustrate this observation, we plot two Venn diagrams comparing the number of overlapped important channels among three classes. In Fig. 11, the left Venn diagram shows the number of overlapped channels within the top 10 important channels. The right one shows the number of overlapped channels in the important channels whose importance are larger than a thousandth of the highest one for each class. In the right Venn diagram, the number of important channels for each class is not uniform. Fig. 11 illustrates that the number of overlapped important channels between similar classes (bullfrog and tree frog) is larger than that between dissimilar classes (tree frog and zebra).

V Conclusion

In the work, we proposed a novel CHIP model, which can provide visual interpretations for the predictions of networks without requiring re-training. Further, we combine visual interpretations in the first and the last convolutional layers to obtain Refined CHIP visual interpretation that is both class-discriminative and high-resolution. Through experiment evaluation, we have demonstrated that the proposed interpretation model can provide more reasonable visual interpretation compared with previous interpretation methods. The proposed CHIP model also outperforms other visual interpretation methods in weakly-supervised object localization task.