Semantically Interpretable and Controllable Filter Sets

TL;DR

This paper introduces a method for generating semantically interpretable filters from natural images in an unsupervised way, enabling controllable sensitivity for applications like image recognition and quality assessment.

Contribution

It presents a novel approach to learn interpretable filter sets that can be tuned for different tasks, demonstrating state-of-the-art results on recognition and quality assessment datasets.

Findings

Achieves state-of-the-art performance on CURE-TSR and TID 2013 datasets.

Demonstrates robustness of filters across progressive distortions.

Enables controllable sensitivity to color degradations.

Abstract

In this paper, we generate and control semantically interpretable filters that are directly learned from natural images in an unsupervised fashion. Each semantic filter learns a visually interpretable local structure in conjunction with other filters. The significance of learning these interpretable filter sets is demonstrated on two contrasting applications. The first application is image recognition under progressive decolorization, in which recognition algorithms should be color-insensitive to achieve a robust performance. The second application is image quality assessment where objective methods should be sensitive to color degradations. In the proposed work, the sensitivity and lack thereof are controlled by weighing the semantic filters based on the local structures they represent. To validate the proposed approach, we utilize the CURE-TSR dataset for image recognition and the TID 2013 dataset for image quality assessment. We show that the proposed semantic filter set achieves state-of-the-art performances in both datasets while maintaining its robustness across progressive distortions.

Tables

Table 1. Table 1 : Pearson correlation coefficients and Spearman correlation coefficients for different decolorization level

Metric	Pearson Correlation Coefficient
	Lv. 1	Lv. 2	Lv. 3	Lv. 4	Lv. 5
PSNR-HMA	0.643	0.626	0.280	0.046	0.486
MS-SSIM	0.248	0.143	0.302	0.525	0.744
SR-SIM	0.370	0.260	0.301	0.497	0.732
FSIMc	0.391	0.253	0.303	0.553	0.778
PerSIM	0.126	0.085	0.304	0.554	0.804
AE	0.716	0.725	0.765	0.775	0.577
AE(L1)	0.557	0.406	0.542	0.682	0.619
AE(L2)	0.079	0.004	0.275	0.454	0.568
Sem-AE	0.772	0.795	0.801	0.816	0.730
Metric	Spearman Correlation Coeffcieint
	Lv. 1	Lv. 2	Lv. 3	Lv. 4	Lv. 5
PSNR-HMA	0.505	0.475	0.140	0.229	0.732
MS-SSIM	0.471	0.345	0.111	0.224	0.691
SR-SIM	0.505	0.401	0.098	0.234	0.732
FSIMc	0.432	0.347	0.013	0.395	0.793
PerSIM	0.306	0.160	0.143	0.479	0.825
AE	0.648	0.764	0.795	0.786	0.389
AE(L1)	0.451	0.378	0.541	0.660	0.480
AE(L2)	0.084	0.120	0.188	0.381	0.543
Sem-AE	0.725	0.815	0.802	0.797	0.615

Equations

$$\begin{array}{c}s=\sigma (W_1^{T}P+b_1),\\ \forall s\in {\Re }^{h\times n},W_1\in {\Re }^{d\times h},P\in {\Re }^{d\times n},b_1\in {\Re }^h,\end{array}$$

$$\tilde{P}=W_2^{T}s+b_2.$$

$$J_f(W,b)=\Vert W_2^T(\sigma (W_1^{T}P+b_1))+b_2-P{\Vert }_2^2+f(W,b),$$

$$J_{{\ell }_1}(W,b)=\Vert W_2^T(\sigma (W_1^{T}P+b_1))+b_2-P{\Vert }_2^2+\beta \Vert W{\Vert }_1.$$

$$J_{{\ell }_2}(W,b)=\Vert W_2^T(\sigma (W_1^{T}P+b_1))+b_2-P{\Vert }_2^2+\lambda \Vert W{\Vert }_2^2.$$

$$J(W,b)=\Vert \tilde{P}-P{\Vert }_2^2+\beta \Vert W{\Vert }_1+\lambda \Vert W{\Vert }_2^2.$$

$$\kappa [W_1]=\frac{E[W_1-\mu {]}^4}{{\sigma }^4},$$

Full text

Index Terms—  Interpretability, Robustness, Semantic Filters, Unsupervised Learning, Autoencoders.

1 Introduction

Visual understanding is a research field that aims to provide semantically meaningful interpretation of the visual cues in data [1]. The objective of visual understanding algorithms is to compute mappings that lead to a representation space in which non-informative and informative representations are distinguishable. Traditional visual understanding algorithms are based on handcrafted approaches in which mappings between input pixel spaces and distinguishable representation spaces are tractable based on data dependent characteristics [2]. Because of the tractable nature of the mapping functions, the representation space spanned by the handcrafted mappings is interpretable. The interpretability and tractability of the representation spaces as well as their mappings have enabled handcrafted visual understanding algorithms to achieve application generalizability. Hence, based on the same underlying representation space, multiple tasks on various applications can be performed using handcrafted feature mappings. For instance, mappings that rely on keypoint detection and feature extraction such as SIFT are used for numerous application including image retrieval [3] and image quality assessment [4].

Recently, the availability of big data and advancements in computational resources have enabled the development of powerful data-driven algorithms for various vision tasks [5, 6]. In data-driven approaches, representation spaces are directly learned from data and labels. A commonly used supervised data-driven method is Convolutional neural network (CNN) which learns a set of discriminative features between classes  [6]. In this method, the discriminative features are extracted from the input pixel space using a set of learned convolutional filters and mapped to a latent representation space. To interpret the data-driven representation space, the latent representation is visualized in different ways. The authors in [7] proposed a visualization technique that projects a feature map of interest back to input pixel space. In [8], the authors visualized image patches that maximally activate hidden units and quantified the interpretability of individual filters. These techniques help to interpret the representation space. However, we cannot directly leverage the information obtained from the interpretation because data-driven representations normally contain mixture of abstract features [9]. Therefore, filters learned in a task-dependent manner are not easily application generalizable and such algorithms require additional training or fine-tuning for different tasks.

In this paper, we generate controllable semantic filter sets in an unsupervised fashion. The authors in [10] describe semantic filters as tools to extract subjectively meaningful structures from natural images. We use an autoencoder, an unsupervised neural network, to generate these filters. We demonstrate that, by using such filters, the performance gains of a data-driven approach and the application-generalizability of interpretable models can be combined. The contributions of this paper are threefold:

• •

We analyze various methods to control the training phase of an autoencoder. The filters learned from considered methods are visualized and validated based on their structural interpretability.

• •

We group interpretable filter sets into semantically meaningful visual concepts that are based on color and edge characteristics.

• •

We demonstrate the feasibility of semantic filter sets on two contrasting applications including image recognition and image quality assessment. Specifically, we test the robustness of these filters under mild to severe color degradation.

In [11] and [12], we proposed objective image quality estimators based on the representation space spanned by autoencoder filter sets, which were weighed with perceptually inspired formulations. In this paper, we develop further insight into the representation space by delving into the generation of the filter sets to embed semantic meaning within them. In particular, we investigate different regularization techniques for semantically meaningful filter sets and generalize them to applications including recognition under challenging domain shifted conditions. In addition, we analyze the robust performance of image quality estimators under varying challenge levels. Such results provide further insight into understanding the correlation between objective predictors and subjective quality opinions.

2 Background

In this section, we describe a vanilla autoencoder network. An autoencoder is an unsupervised learning network that is trained to copy inputs to outputs [13]. The network consists of encoder and decoder functions as shown in Fig. 1. The encoder maps a matrix $P$, consisting of $n$ input feature vectors each of size $d\times 1$, to a hidden layer with $h$ neurons. The encoder can also be considered as a set of $h$ forward filters each of size $d\times 1$, whose responses are passed through a non-linearity to obtain $s$. Mathematically, $s$ can be represented as,

[TABLE]

where $s$ is the set of non-linear hidden layer responses to an affine filter set parameterized by weights $W_{1}$ and bias $b_{1}$. The sigmoidal non-linearity is used through the rest of this work. The responses $s$ are mapped back to the input data space to obtain $\tilde{P}$ using a decoder. The decoder is a set of backward affine filters parameterized by weights $W_{2}$ and bias $b_{2}$. The reconstructed output $\tilde{P}$ is obtained as,

[TABLE]

Note that we use a linear decoder in our experiments. The forward and backward filters are simultaneously trained using backpropagation by constructing and minimizing a cost function $J_{f}(w,b)$, between $P$ and $\tilde{P}$ given by,

[TABLE]

where the first term is the Mean Square Error (MSE) between the input and reconstructed outputs, and $f(W,b)$ is a regularization term. Regularization is a modification of an optimization objective function to reduce the generalization error without impacting the training error [13]. Practically, adding regularization to an autoencoder means that the network is restricted to copying only an approximate input as its output. This forces the autoencoder to prioritize the necessary aspects of the input thereby learning a dictionary of useful properties that characterize the input data. The need for regularization is well recognized within the machine learning community [14] and a number of regularization techniques have been proposed. The commonly used techniques for autoencoders include weight penalties, derivative penalties, and training tricks including early stopping [15]. In this work, we concentrate on analyzing weight penalties. This is in keeping with the overall theme of generating semantic filters that are parameterized by weights. While these techniques have already been proposed, our focus is to to control the semantic meaning and visual concepts learned by the weights.

3 Regularization analysis

We train an autoencoder on ImageNet database [5] to obtain encoder filters. 100 patches of dimension $8\times 8\times 3$ are sampled from each of $1,000$ images and preprocessed using Zero-Phase Component Analysis (ZCA) whitening [16]. These whitened patches are fed into multiple autoencoders, each with different weight penalties, $f(W,b)$ from eq. 3. The different penalties are $\ell_{1}$ or LASSO [17], $\ell_{2}$ or ridge regression [18], and elastic net [19] regularization which is a weighted combination of $\ell_{1}$ and $\ell_{2}$ penalties. While all these penalties have been explored in regularization theory, we provide a qualitative analysis based on the visual concepts that each of the regularization techniques learns.

3.1 $\ell_{1}$ penalty

The $\ell_{1}$ constraint promotes sparsity within the hidden layer responses $s$. The training cost function for $\ell_{1}$ regularization is given by,

[TABLE]

where $\beta$ is a positive regularization parameter. The trained filters are visualized in Fig. 2(d). An example peppers image is passed through the autoencoder and its reconstructed image is visualized in Fig. 2(a). The fidelity of the reconstructed image is $23.57$ dB.

Theoretically, $\ell_{1}$ regularization can help with interpretability since, for every train and test case, only a few filters are activated. However, in practice the $\ell_{1}$ penalty suffers when there are correlated weights or filters. If there is a group of filters among which the pairwise correlations are very high, then the $\ell_{1}$ penalty selects any one of those correlated filters [19]. This filter selection issue is not suitable for applications where the representation space is used as features for further analysis. Filters learned from images have large correlations between them because of the inherent spatial correlation within training data. Also, as shown in Fig. 2(d), the interpretability of the mapping functions is not obvious.

3.2 $\ell_{2}$ penalty

The $\ell_{2}$ penalty in the cost function is also called ridge regression, weight decay or Tikhnov regularization [17]. This is a well studied regularization technique that promotes shrinkage between filters. The cost function with $\ell_{2}$ penalty on weights is given by,

[TABLE]

By penalizing the filters in this manner, the input data produces dense responses. However, a mixture of large and small responses causes instability while learning [19]. Practically, this means that the representation space changes rapidly even with slight domain shift between train and test data. For application-generalizability, the $\ell_{2}$ regularization may not be suitable. The filter sets learned from this technique along with the reconstructed peppers image are provided in Fig. 2(b) and  2(e) respectively. Again, the filters are able to reconstruct the image with a PSNR score of $29.09$ dB. However, there is no semantic interpretability that can be associated with either the filters or the activations.

3.3 Elastic Net penalty

The elastic net was proposed in [19] to correct for the filter selection challenge faced by $\ell_{1}$ penalty. Instead of one filter being activated from a group of correlated filters, all the correlated filters are activated at the same time. This is achieved by adding a weighted $\ell_{2}$ norm penalty to obtain within-group dense responses. A thorough explanation and analysis is provided in [19]. The minimization cost function for this regularization technique is given by,

[TABLE]

The values of $\beta$ and $\lambda$ are set to $5$ and $3e^{-3}$ respectively as suggested by the authors in [20]. The weights learned using the elastic net regularization along with the reconstructed peppers image are visualized in Fig. 2(c) and  2(f) respectively. The lower fidelity of the reconstructed image is expected because of the higher regularization penalties. However, it is obvious that each filter in Fig. 2(c) has a visual concept associated with it. These concepts include different colors, color gradients, and edges with multiple orientations. These weights are structurally meaningful and semantic in nature. While other works in sparse coding [16] and CNNs have shown that edges dominate the first layer with just $\ell_{2}$ penalty, the complete demarcation between color and edge filters using elastic net penalty is very interesting and deems further investigation.

4 Semantic Visual Concepts

To leverage these semantic filters for different tasks, we further group them based on individual visual concepts. The grouping of semantic filter sets is performed based on the kurtosis measure of each filter. Kurtosis, $\kappa$, is defined as,

[TABLE]

where $W_{1}$ are vectorized, zero-centered and normalized encoder filter values, $\mu$ and $\sigma$ are respectively mean and the standard deviation of $W_{1}$. For every $\kappa[W_{1}]$ greater than a threshold of $5$, the filter is classified as an edge filter and if $\kappa[W_{1}]<2$, the filter is classified as a color filter. The thresholds are empirically determined to ensure a complete demarcation between filters that represent color and edges. Edge filters have a higher kurtosis value because they have values in a localized area. Therefore, most of the filter values are located away from the mean of distribution leading to higher kurtosis. While the authors in [21] and [22] use elastic net regularization for learning, they do not go further to group filters by their representative visual concepts. We note that the entire process from obtaining interpretable filters to grouping semantic filters is conducted in an unsupervised fashion.

5 Experiments and Results

In this section, we demonstrate the advantages of learning semantic filters and categorizing them based on visual concepts with two contrasting applications:

• •

Recognition under progressive decolorization challenge where the objective is to recognize decolorized images when trained on color images of the same class.

• •

Image Quality Assessment (IQA) under color distortion where the goal is to objectively estimate the subjective scores of images affected by color distortion.

In these two applications, we show proposed semantic filter-based methods outperform state-of-the-art algorithms while maintaining its robust performance under different levels of color distortion. For validation, we use the Challenging Unreal and Real Environments for Traffic Sign Recognition (CURE-TSR) dataset [23] and the TID 2013 [24] dataset, for recognition and IQA respectively. They contain images with decolorization and color distortion challenges across $5$ progressive levels. After the semantic filters are grouped based on the visual concepts, we prune the filters in an application dependent manner. Pruning is carried out by applying application-dependent weights $(w_{c},w_{e})$ on color and edge filter sets. We refer to this model as a semantic autoencoder (Sem-AE) and a general framework is illustrated in 3. The $p_{i}$ refers to the whitened patches extracted from images. The filter set $W_{1}$ is grouped into visual concepts and their responses are processed in an application-dependent manner.

5.1 Recognition

The CURE-TSR dataset consists of $7,292$ challenge-free traffic sign images over $14$ classes for training. Testing is performed on $20,004$ images that includes $3,334$ images in each of distortion-free and five different levels of decolorization category (total $3,334\times 6=20,004$ images). An example of distortion-free, decolorization levels $3$ and $5$ are shown in the Fig. 4(a). In the Fig. 4(b), we visualize filters which show maximum activations for each patch in distortion-free, level $3$, and level $5$ decolorization images among all $W_{1}$ filters including color and edge concepts. In the Fig. 4(c), only the edge concept filters are used for visualization. It is apparent that the representation obtained only from edge filters are not adversely affected by decolorization while the representation obtained from all the filters change significantly. Hence, we assign $(w_{c},w_{e})$ to be $(0,1)$ and only the edge filters are used to obtain features. A softmax classifier is trained on these edge features as part of the application-dependent post processing block in Fig. 3. Performance accuracies of the Sem-AE-based recognition algorithm across decolorization levels are plotted in Fig. 5. We compare proposed algorithm with other baseline algorithms detailed in [23]. In addition, we show performance of $l_{1}$ (AE(L1)), $l_{2}$ (AE(L2)), and elastic net (AE) regularized filters on which softmax classifiers are trained. The performance of two intensity-based methods (I-Softmax, I-SVM) is not affected by decolorization since they do not use color information. However, by discarding color information, these methods achieve lower performance among the first $4$ levels than all the other methods. It can be seen that the Sem-AE with $(w_{c},w_{e})=(0,1)$ shows high and steady accuracy across decolorization levels while the accuracy of all other RGB based methods degrades as decolorzation becomes severe. The steady performance indicates that the representation space spanned by Sem-AE filters is robust to color distortions, because the edge-based filter responses are invariant to color degradations.

5.2 Image Quality Assessment

For the task of Image Quality Assessment (IQA), we use TID 2013 database to evaluate the performance of Sem-AE. In particular, we analyze the categories of color saturation, color quantization with dither and chromatic aberrations. We choose these categories to specifically illustrate the worth of learning color and edge concepts for IQA. Each category has $5$ levels of progressive color based distortions. Previous studies  [25, 26, 27, 28, 29] in this field, including our own [11, 12] concentrated on and provided results on all the distortion levels together. In this section, we report results individually for each level. Ideally, a good quality estimator is correlated with subjective scores across varying distortions.

The Sem-AE scores are obtained for both reference and distorted images with $(w_{c},w_{e})=(0.5,2)$. The higher weighting is given to edges because the human visual system is more sensitive to sharpness of images and distortion in the edge components of images causes higher visual discomfort [30]. Spearman correlation is calculated between the weighted responses of distorted and reference images to obtain the objective scores. In Table 1, we compare (Sem-AE) with other commonly compared metrics. Pearson (PCCs) and Spearman Correlation Coefficients (SCCs) which measure linearity and monotonic behavior between subjective and objective image quality scores are used to validate the objective metrics. We observe that Sem-AE follows the subjective scores closer than the other metrics. Not only does it exhibit the highest correlation in $4$ of the $5$ categories in both PCC and SCC, it maintains its steady performance across levels. This is in contrast to other methods which show low correlations in levels $1-3$. Note that both FSIMc and PerSIM take color characteristics into consideration. The tested quality estimators exhibit uneven correlations across distortion levels, which should be addressed to obtain robust methods.

6 Conclusion

In this paper, we analyzed existing regularization techniques to obtain semantic filter sets using an unsupervised learning technique. The semantic filters which represent characteristic visual concepts were learned jointly from natural images. While the learned semantic filters succeeded in their primary task of reconstructing an input image, their worth was illustrated in two applications that employed their color and structural groupings. This work provides a promising step towards defining perceptual visual concepts that can be used to learn, interpret, and leverage deep learning models.