Deep Generalized Max Pooling

TL;DR

Deep Generalized Max Pooling is a novel pooling method for CNNs that re-weights activations to balance contributions from different regions, improving classification and writer identification accuracy.

Contribution

It introduces a new pooling layer that balances activations across regions, outperforming traditional average and max pooling in specific classification tasks.

Findings

Outperforms average and max pooling in classification tasks

Effective in medieval manuscript and writer identification

Balances contributions of all activations across regions

Abstract

Global pooling layers are an essential part of Convolutional Neural Networks (CNN). They are used to aggregate activations of spatial locations to produce a fixed-size vector in several state-of-the-art CNNs. Global average pooling or global max pooling are commonly used for converting convolutional features of variable size images to a fix-sized embedding. However, both pooling layer types are computed spatially independent: each individual activation map is pooled and thus activations of different locations are pooled together. In contrast, we propose Deep Generalized Max Pooling that balances the contribution of all activations of a spatially coherent region by re-weighting all descriptors so that the impact of frequent and rare ones is equalized. We show that this layer is superior to both average and max pooling on the classification of Latin medieval manuscripts (CLAMM'16, CLAMM'17), as well as writer identification (Historical-WI'17).

Tables

Table 1. TABLE I : Impact of different triplet loss margins evaluated on the Icdar17-WI dataset for global average pooling (Avg), global max pooling (Max), and deep generalized max pooling (DGMP) with different initializations of λ 𝜆 \lambda .

	margin 0.1		margin 0.2		margin 0.5
	mAP	top-1	mAP	top-1	mAP	top-1
Avg	50.44	69.19	48.75	67.61	48.60	67.19
Max	50.32	69.63	46.63	65.47	40.04	57.03
DGMP, $\lambda =1$	53.76	72.19	52.16	71.08	48.55	67.69
DGMP, $\lambda ={10}^3$	54.81	73.94	53.54	72.56	49.37	68.83
DGMP, $\lambda ={10}^5$	52.41	71.22	48.98	68.00	45.24	64.47

Table 2. TABLE II : Comparison with different global pooling methods (global average pooling (Avg), global max pooling (Max), mixed pooling (Mixed), log-sum-exp pooling (LSE), deep generalized max pooling (DGMP)) and state of the art using the Icdar17-WI dataset dataset. All methods marked by (*) are taken from  [ 31 ] .

	mAP	rank-1
Avg	50.4	69.2
Max	50.3	69.6
Mixed	50.2	69.6
LSE	51.4	70.0
DGMP (ours)	54.8	73.9
Triplet Network (Fribourg)∗	30.7	47.8
CoHinge features (Groningen)∗	54.2	76.1
oBIF (Tébessa)∗	55.6	76.4
Unsupervised [3]	76.2	88.9

Table 3. TABLE III : Classification results showing the average accuracy [%] of five runs (Avg) and an ensemble (Ens) computed by all five models. Note that the state of the art approaches (bottom) use multiple crops while we only use a single view.

Pooling	Avg	Ens
Avg	85.9	87.9
Max	85.5	86.8
DGMP (ours)	87.1	88.6
FAU [18]	83.9	–
ResNet50 [21]	86.3	–
ResNet50 [1]	–	86.6

Table 4. (a) Clamm16

Pooling	Avg	Ens
Avg	85.9	87.9
Max	85.5	86.8
DGMP (ours)	87.1	88.6
FAU [18]	83.9	–
ResNet50 [21]	86.3	–
ResNet50 [1]	–	86.6

Table 5. (b) Clamm17

Pooling	Avg	Ens
Avg	81.3	82.6
Avg + ${\ell }^2$	81.2	82.2
Max	81.3	83.2
DGMP (ours)	83.3	85.1
CK2 [20]	78.9	–
T-DeepCNN [20]	–	85.2

Equations

$${\mathbf{\xi }}_{\text{mix}}=\alpha \cdot {\mathbf{\xi }}_{\text{max}}+(1-\alpha )\cdot {\mathbf{\xi }}_{\text{avg}}$$

$${\mathbf{\xi }}_{\text{lse,i}}=\frac{1}{r}\cdot \log \left[\frac{1}{|{\mathbf{\Phi }}_i|}\sum _{x\in {\mathbf{\Phi }}_i}\exp (rx)\right],$$

$${\mathbf{\xi }}_{\text{sum}}=\psi (\varphi (\mathcal{X}))=\sum _{\mathbf{x}\in \mathcal{X}}\varphi (\mathbf{x}).$$

$$\mathcal{K}(\mathcal{X},\mathcal{Y})=\sum _{\mathbf{x}\in \mathcal{X}}\sum _{\mathbf{y}\in \mathcal{Y}}\varphi (\mathbf{x}{)}^T\varphi (\mathbf{y}).$$

$${\mathbf{\xi }}_{\text{gmp}}=\sum _{\mathbf{x}\in \mathcal{X}}\alpha (\mathbf{x})\varphi (\mathbf{x})=\mathbf{\Phi }\mathbf{\alpha },$$

$${\mathbf{\Phi }(\mathbf{x})}^{\top }{\mathbf{\xi }}_{\text{gmp}}=1_N,$$

$${\mathbf{\xi }}_{\text{gmp}}=\underset{\mathbf{\xi }}{\mathrm{argmin}}\Vert {\mathbf{\Phi }}^{\top }\mathbf{\xi }-1_N{\Vert }_2^2+\lambda \Vert \mathbf{\xi }{\Vert }_2^2,$$

$$\begin{array}{rl}{\mathbf{\alpha }}_{gmp,\lambda }=& \underset{\mathbf{\alpha }}{\mathrm{argmin}}\Vert {\mathbf{\Phi }}^T\mathbf{\Phi }\mathbf{\alpha }-1_n{\Vert }^2+\lambda \Vert \mathbf{\Phi }\mathbf{\alpha }{\Vert }^2\\ =& \underset{\mathbf{\alpha }}{\mathrm{argmin}}\Vert \mathbf{K}\mathbf{\alpha }-1_n{\Vert }^2+\lambda {\mathbf{\alpha }}^{\top }\mathbf{K}\mathbf{\alpha },\end{array}$$

$$\mathbf{\alpha }=(\mathbf{K}+\lambda {\mathbf{I}}_n{)}^{-1}{1}_n,$$

$${\mathrm{AP}}_q=\frac{1}{R}\sum _{r=1}^S{\mathrm{Pr}}_q(r)\cdot {\mathrm{rel}}_q(r),$$

$$L_{\text{tri}}=\sum _{\begin{array}{c}a,p,n\\ y_a=y_p,y_a\ne y_n\end{array}}[m+D(\varphi (a),\varphi (p))-D(\varphi (a),\varphi (n)){]}_{+},$$

Full text

I Introduction

Convolutional neural networks (CNN) have demonstrated remarkable performance in many computer vision tasks in the past years, and are also predominant in the field of document analysis, such as font recognition [1], word spotting [2] or writer identification [3]. Many successful CNN architectures compute a global image descriptor at the final stage of their computational graph by aggregating the feature maps using global average pooling [4, 5] or global max pooling [6, 2]. These layers also allow the use of images of arbitrary dimensions.

Conceptually, one has to differentiate between average/max pooling used for downsampling that pools over local descriptors extracted from different image regions, and global average/max pooling used in CNNs that pool the activations over the entire activation map. Commonly, each activation map is pooled independently. It follows for global pooling that the final representation contains activations from different locations. For highly structured images, global average pooling might give too much emphasis on frequently occurring patches in the input image, e. g., the document background or the script. While global max pooling does not have this problem, it might get influenced by noise.

We propose a new global pooling that aggregates activations in a spatially coherent way. Therefore, a weighted aggregation of local activation vectors is applied, where a local activation vector is defined to be a position in the activation volume along the depth dimension, see Fig. 1 (left). In contrast to global average/max pooling, activations of specific locations are balanced jointly with each other.

The weights for each local vector are computed through an optimization process, where we chose the use of Generalized Max Pooling (GMP) [7]. It was originally developed to balance local descriptors in a traditional Bag-of-Words (BoW) setting, where a global representation is computed from embeddings of local descriptors. Average pooling is commonly used to aggregate local embeddings. Yet, average pooling may suffer from over-frequent local descriptors computed from very similar or identical image regions. GMP balances the influence of frequent and rare descriptors according to the global descriptor similarity. We propose the use of GMP in a deep learning setting, and phrase GMP as a network layer, denoted as Deep Generalized Max Pooling (DGMP).

This can also be regarded as a form of statistical attention. While attention depends on the content, we balance activation locations independent of the content but enforcing that the image-to-image similarity becomes independent of frequent and rare image locations. We believe that a combination with attention could further improve the accuracy.

In detail, our contributions are as follows:

1. We propose a spatially coherent pooling by weighting local activation vectors. For the computation of the weights, we propose to use DGMP. It balances frequent and rare embeddings that consist of locally coherent activations. The code for the DGMP layer is publicly available.111https://github.com/VChristlein/dgmp

2. Furthermore, the proposed DGMP layer is evaluated on two different scenarios:

(i) writer identification/retrieval and

(ii) script type classification

. For writer identification, we directly use the output of the DGMP layer to compute a similarity while we add a fully-connected layer on top of the DGMP layer for the classification of cursive Latin medieval fonts.

3. For both scenarios, we show that DGMP is superior to other global pooling methods, such as global average or max pooling.

4. While not being the focus of the paper, our results achieve better or comparable results to state-of-the-art methods for font classification.

The paper is organized as follows. After giving an overview of the related work in pooling techniques and historical document classification in Sec. II, we explain the methodology and how we phrase GMP as a network layer in Sec. III. In Sec. IV, the proposed pooling layer is compared against global average and global max pooling using different scenarios and datasets. We show that DGMP is a powerful replacement for both of them by adding only one single additional parameter to the network.

II Related Work

II-A Pooling

In the traditional Bag-of-(visual)-Words (BoW) model, local descriptors are aggregated to form a global image descriptor. Commonly, this is achieved by using average pooling. However, average pooling assumes that the descriptors in an image are independent and identically distributed. Thus, more frequently occurring descriptors will be more influential in the final representation and dominate the similarity. This phenomenon is known as visual burstiness [8]. Bursty descriptors can dominate the similarity metric while being irrelevant to distinguish different samples. For example in document images, the image background, i. e., areas without script, commonly do not contribute to the class decision. A common technique to counter burstiness is to normalize the final global descriptor, e. g., by power normalization [9]. Exceptions of this are the works of Murray et al. [10] and Jégou et al. [11] who proposed Generalized Max pooling and Democratic Aggregation, respectively. Both methods re-weight the descriptors such that their influence regarding the final similarity score is equalized. When comparing both methods, Murray et al. [7] found that GMP is superior to democratic aggregation.

In neural networks, we have to differentiate between spatial pooling and global pooling. Spatial pooling fuses information of input across spatial locations. Thereby the number of parameters is decreased and thus the computational cost. It also provides some affine invariance and mitigates the risk of overfitting. Spatial max pooling was already used in the seminal work of LeCun et al. [12] and was a typical building block of CNNs until recently. Nowadays, it is more common to increase the stride of the convolution.

The use of a global average layer as a last layer was proposed by Lin et al. [13], and got its breakthrough by the well-known residual network (ResNet) [4]. Global max pooling was for example proposed for weakly-supervised learning [14] and is also used in the PHOCNet for the task of word spotting [2].

It is also possible to learn a pooling function from the data using the neural network optimization framework. Lee et al. [15] suggest a tree-based spatial pooling, where pooling filters are learned that are structured as a tree. The authors compare the method also with a mixed pooling layer, a simple weighted sum of average and max pooling, i. e.,:

[TABLE]

where $\alpha$ is learned by backpropagation and $\bm{\xi}_{\text{max}}$, $\bm{\xi}_{\text{avg}}$ the max and average pooling representations. Another pooling method [16], denoted as log-sum-exp (LSE) pooling, computes a smooth approximation of the max function. It is computed for each feature map $\bm{\Phi}_{i}$ as follows:

[TABLE]

where $r$ is a pooling parameter (commonly not learned during training) that allows a transition from average pooling for $r\to 0$ to max pooling for $r\to\infty$. All these methods work on the basis of a single individual activation map without notion of the other activation maps. In contrast, we propose DGMP, a network layer that balances frequent and infrequent local activation vectors along the depth dimension.

II-B Historical Document Image Classification

While we propose a general pooling layer, which can be used in most common network architectures by replacing the global average pooling layer, we believe that the classification of writings benefits the most of our DGMP layer due to the repetitive character of script. Balancing frequent activations from the background with script features that are used to classify the document can be helpful to improve the image embedding.

We mainly focus on two specific tasks: Script type or font classification and writer identification/retrieval. Script type classification is an important aspect of paleographic research [17, 18], and has recently gained more attention through the ICFHR’16 and ICDAR’17 competitions in the classification of Latin medieval manuscripts (CLaMM) [19, 20]. Tensmeyer et al. [1] proposed the use of an ensemble of CNNs trained with different network architectures (for Clamm16) and image scales (for Clamm17). They employ a patch-based approach for training as well as testing. They show that they could improve the recognition accuracy by means of individual foreground/background brightness augmentation. Another patch-based approach achieves similar results by standardizing the images using zero component analysis [21]. In contrast, we show that we can achieve superior results for the Clamm16 dataset and comparable results for the Clamm17 dataset by single image views.

Another line of research is writer identification, where the task is to find the correct writer of a query sample given a dataset of samples with known scribes [3]. The evaluation procedure for writer retrieval is similar, however, the full ranking of the dataset is evaluated in terms of mean average precision (mAP). The most deep learning approaches compute CNN activation features for local image patches using the penultimate layer of a CNN using the writers of the training set as targets [22, 23, 3, 24, 25]. These local feature descriptors are subsequently aggregated to form a global embedding and then used for comparison. A semi-supervised learning scheme is suggested by Chen et al. [25] which makes use of additional unlabeled data. In comparison, Christlein et al. [3] proposed to use an unsupervised learning scheme to compute deep activation features that are eventually encoded using VLAD [26]. In a subsequent work [27], they show that GMP improves the encoding consistently. He et al. [28] employ auxiliary tasks to improve writer identification of single word images. Direct embeddings computed for example via triplet networks [29, 20] are also used. In these cases, no additional feature encoding such as VLAD is needed. We also rely on a triplet network, i. e., we use a triplet loss to learn an image embedding that can directly be used for comparison.

III Methodology

Our network layer is based upon generalized max pooling (GMP) [7]. It is a pooling technique that generalizes sum pooling and max pooling. The goal is to compute a single vector representation, from a set of local embeddings. Generalized max pooling was proposed as an aggregation function that summarizes feature embeddings obtained by embedding functions. The embedding function can be VLAD, Fisher vectors or CNN codes. However, GMP was originally not proposed as a neural network layer. We reinterpret generalized max pooling as a global pooling layer and incorporate our GMP layer in neural network architectures – replacing the commonly used global average pooling. Other than the original formulation, this allows an end-to-end training using backpropagation.

III-A Generalized Max Pooling

The standard approach for computing a global descriptor of an image given a set of local descriptors consists of two steps: an embedding step and an aggregation phase. In the embedding phase, an embedding function $\phi$ maps each local descriptor into a high-dimensional space. During the aggregation step an aggregation function $\psi$, e. g., sum pooling or max pooling, computes a single vector from the embedded feature vectors.

If we use sum pooling, the global descriptor for a set of local descriptors $\mathcal{X}=\{\bm{x}_{i}\in\mathbb{R}^{D},i=1,\ldots,N\}$ becomes

[TABLE]

Since we sum over all descriptors, the aggregated descriptors can suffer from interference of unrelated descriptors that influence the similarity, even if they have low individual similarity. Assume the dot-product is used to compute the similarity between two images that are represented by the sets $\mathcal{X}$ and $\mathcal{Y}$, respectively. Their similarity $\mathcal{K}(\mathcal{X},\mathcal{Y})$ follows as:

[TABLE]

Hence, the similarity is defined by the sum of pairwise similarities between all descriptors, i. e., unrelated descriptors with low similarity contribute to the overall expression.

Murray et al. [7] propose to perform a weighted sum pooling. For every $\bm{x}\in\mathcal{X}$ a weight $\alpha(\bm{x})$ is introduced. The weights shall equalize the contribution of each element to the similarity scores. The aggregated representation $\bm{\xi}$ becomes:

[TABLE]

where $\bm{\Phi}\in\mathbb{R}^{D\times N}$ and $\bm{\alpha}\in\mathbb{R}^{N\times 1}$.

Generalized max pooling is a procedure to obtain the weights $\bm{\alpha}$ by solving an optimization problem. The goal is to enforce the dot-product similarity between the aggregated representation $\bm{\xi}_{gmp}$ and each patch encododing $\phi$ to be a constant value. The value of the constant (we used $1$) has no influence, since the final encoding will be normalized. Thus, we end up with a linear system of $N$ equations and $D$ unknowns.

[TABLE]

where $\bm{1}_{N}$ denotes the vector of all constants set to $1$. This linear system can be turned into a least-squares ridge regression problem:

[TABLE]

with $\lambda$ being a regularization term that stabilizes the solution. An interesting property of $\lambda$ is that the pooling resembles sum-pooling when $\lambda\to\infty$. In case of $\lambda\to 0$, we turn the ridge regression to a least-squares regression problem.

III-B Deep GMP Layer

We adopt generalized max pooling as a trainable neural network layer following the known-operator paradigm [30]. This allows for an end-to-end training given images and a loss function, such as softmax or triplet loss. Both generalized max pooling and average pooling are at their core variants of weighted sum pooling. Therefore, we can easily interpret GMP as a form of global pooling, similar to global average pooling. Assume the final activation volume of a convolutional neural network with height $h$, width $w$ and depth $d$ (= number of activation maps), see Fig. 1. We interpret the vectors consisting of the activations at a location $i$ with $i=\{1,\ldots,h\cdot w\}$ across the whole depth $d$ as the local descriptor embedding $\bm{\phi}_{i}$. That means, there are $N=h\cdot w$ embeddings $\bm{\phi}$ of dimension $D=d$ that are aggregated.

In other words, we compute a weighted sum over all locations in the activation volume, yielding a $D$-dimensional vector – the global descriptor $\bm{\xi}$. This is substantially different from average or max pooling that average or max-pool among one activation map without considering the other activation maps.

We follow Murray et al. [7] and solve the dual formulation for each sample to explicitly compute the weights. Let $\bm{\xi}=\bm{\Phi}\bm{\alpha}$ and insert it into Equ. 7, it follows:

[TABLE]

where $\bm{K}$ is the Gram matrix, yielding the closed-form solution:

[TABLE]

which does not rely on the embeddings but on a similarity kernel. The GMP representation follows by Equ. 5. Note that for $N\gg D$ the solution of the primal (where also a closed-form solution exists) is more efficient. In our case $D<N$ for the normal ResNet50 version with $400\times 400$ input image sizes and $N\approx D$ for the modified ResNet50 version that we used for script type classification.

In the original formulation, $\lambda$ is a hyperparameter that has to be cross-validated. Since all operations involved are differentiable, we can optimize for $\lambda$. Yet, the right initialization of $\lambda$ still matters to some degree, see our experiments. The output of the DGMP layer is eventually normalized such that its $\ell^{2}$ norm equals to one.

IV Evaluation

We evaluate DGMP on two different tasks:

1. writer identification/retrieval and
2. script type classification.

For both scenarios, we show that DGMP is superior to other pooling mechanisms, such as average or max pooling.

IV-A Datasets

We evaluate the following publicly available datasets:

IV-A1 Icdar17-WI

is a dataset composed of letters used in the ICDAR 2017 Writer Identification competition [31]. The training set contains 394 writers contributing three samples each. 720 other writers with five samples each are provided for testing. In Fig. 2, indicative examples of the Icdar17-WI dataset can be seen. The task consists of training a representation given the writer-disjoint training set and use that representation to effectively retrieve the test-set. It can be seen that writer identity is hard to infer as general look and feel of the document dominates the visual similarity.

IV-A2 Clamm16 and Clamm17

were introduced in the competitions on Classification of Medieval Latin Manuscripts (CLaMM) of 2016 [19] and 2017 [20], respectively. The task is image classification and both datasets contain twelve classes representing script types. The Clamm16 dataset consists of 2000 training and 1000 test samples while the Clamm17 dataset contains 3540 and 2000 samples, respectively. Example images can be seen in Fig. 3.

IV-B Error Metrics

We report top-1 accuracy for writer identification and font classification. For writer retrieval, the mean average precision (mAP) is given. The latter is computed by taking the mean of computing the average precision per query sample. The average precision for a specific query sample $q$ and $R$ relevant samples in a dataset of size $S$ is given as:

[TABLE]

where $\mathrm{rel}_{q}(r)$ is an indicator function returning $1$ if the document at rank $r$ is relevant, and [math] otherwise, and $\mathrm{Pr}_{q}(r)$ is the precision at rank $r$.

IV-C Experiments

For all our experiments, we use the ResNet-50 [4] architecture with weights pretrained on ImageNet (provided by PyTorch). The last fully connected layer is either changed to represent the twelve classes of the CLaMM datasets or omitted in case of writer identification. The global pooling layer (= penultimate or last layer) is set to the respective pooling layer to be evaluated. We first evaluate GMP for writer identification, and then for script type classification.

IV-C1 Writer Identification

For writer identification, we are interested in global descriptors that discriminate different writers, i. e., we use directly the pooled representation as our image descriptor. However, exploratory experiments on a validation split of the Icdar17-WI training dataset using the full, colored images did not achieve good results. Since the images in the Icdar17-WI dataset are rather large, and we want to distinguish the authors on the individual characteristics of their handwriting, which are typically only present in small image portions, the image is subdivided into patches of size $400\times 400$ with a stride of 256. Crops that do not contain any handwriting are filtered out by setting a threshold (= 2000) on the sum of the Canny edge detector [32] output. We use the binarized version of the images to be independent of pen and background variations, and do not employ any additional transformations or augmentation techniques apart from standardizing the data by enforcing a zero-mean and standard deviation of one.

During test time, images are also sub-divided into overlapping patches and standardized. The respective descriptors computed by our model are then averaged to obtain the overall image descriptor for computing the ranking based on the query images.

The networks are trained by means of the triplet loss [33]:

[TABLE]

where $[\cdot]_{+}=\max(0,\cdot)$ and $D(x,y)$ denotes a distance metric between two samples $x$ and $y$. The triplet loss enforces that for a given anchor $a$, the embedding of a positive sample $p$ that belongs to the same class $y_{p}=y_{a}$ is closer to the encoding of the anchor than a negative sample $n$ belonging to a different class $y_{n}$ by at least a margin $m$.

We use the hard-batch variant of the triplet loss [34], i. e., the hardest positive and hardest negative pairs per (mini-)batch are selected for the triplet creation. Therefore, we construct the (mini-)batches of size $P\cdot K$ by randomly sampling $P$ writers and $K$ image crops for each writer. In all our experiments on the historical writer identification dataset, we set $P=14$ and $K=4$. We evaluate the impact of different margins for the triplet loss on the performance of our model.

For optimization, we use the AMSGrad variant [35] of Adam [36] with default parameters ($\beta_{1}=0.9,\beta_{2}=0.999$) and a weight decay of $10^{-5}$. We employ a learning rate of $\eta=2\cdot 10^{-4}$. We recognized a slight improvement when using a higher learning rate for the GMP pooling parameter (learning rate is multiplied by $10^{3}$). We run the experiments for 300 epochs and employ an exponential learning rate decay starting at epoch 150 as suggested by Hermans et al. [34]. As validation set, we choose 40 writers (\approx$$10\text{\,}\mathrm{\char 37\relax}) of the official training set. The best model is selected according to the best validation error along the 300 epochs.

Global average pooling and global max pooling serve as baselines for our experiments. Not only are these used in practice, these pooling operators also represent special cases of the other generalized and learned pooling functions. We evaluate different margin parameters for the triplet loss $m\in\{0.1,0.2,0.5\}$ and different initializations for the DGMP pooling parameter $\lambda\in\{1,10^{3},10^{5}\}$, afterwards $\lambda$ is optimized during training. The results are summarized in Tab. I. Generalized max pooling performs almost always better than average pooling and max pooling for a given margin. For DGMP, the initialization with $\lambda=10^{3}$ achieves the overall best results.

The choice of the margin parameter influences all models. All our models show the best performance at $m=0.1$, regardless of the pooling function. However, some pooling layers seem more sensitive to the change in margin than others. While average pooling is less susceptible to a change in the triplet loss margin, max pooling suffers the most, dropping from $50.32\text{\,}\mathrm{\char 37\relax}$ mAP to $40.04\text{\,}\mathrm{\char 37\relax}$ mAP. As a result, DGMP outperforms both average pooling and max pooling considerably for a margin of $m=0.1$, for $m=0.5$, however, the performance of DGMP relative to the average pooling baseline depends on the proper choice of $\lambda$.

Finally, we compare the proposed DGMP layer with other pooling methods, see Tab. II (top) and the state of the art (bottom). Therefore, we also evaluated a weighted mixed pooling [37] where a weighted sum of both pooling methods is computed and log-sum-exp pooling (LSE), cf. Equ. 1 and Equ. 2, respectively. The initial regularization parameters were set to $\alpha=0.5$ and $r=10$, which will get optimized during training. All competing pooling techniques are clearly behind our proposed Deep GMP layer. When comparing with the state of the art, our method is inferior to the current state-of-the-art method of Christlein et al. [3] where the features are learned in an unsupervised manner, which are explicitly aggregated. Yet, our method is superior to a competing deep learning method that uses the same triplet loss [31].

IV-C2 Script Type Classification

Next, we want to see how our pooling layer performs for a classification task. Therefore, we classified images in twelve font classes. In contrast to writer identification, we evaluated them on the full images at a resolution of $384\times 384$. To compensate the lack of training data, we made use of data augmentation. In addition to data standardization, we applied random resizing ($8\text{\,}\mathrm{\char 37\relax}100\text{\,}\mathrm{\char 37\relax}$), random aspect ratios (0.75–1.33), random rotations of up to $5\text{\,}\mathrm{\SIUnitSymbolDegree}$, and random color jitter of up to $10\text{\,}\mathrm{\char 37\relax}$ brightness and contrast variations.

We modified the ResNet architecture slightly by reducing the stride of the last convolutional block to one (instead of two) to increase the spatial resolution before global pooling [38] resulting into activation maps of size $21\times 21$. Thus, more activation vectors are pooled and the solution for the optimization problem becomes more reliable. For optimization, we used the same setting as for writer identification, but we chose to use a higher learning rate ($\eta=4\cdot 10^{-4}$) in the beginning. Though, the exponential decay was triggered earlier: starting at epoch 30 for Clamm16 and at epoch 16 for Clamm17, because both datasets are comparably small, and since we did not employ a patch-wise classification. The regularization parameter ($\lambda$) was set to 5k a learning rate multiplier of 100 was used in the Clamm17 experiment, which showed a slightly improved validation accuracy (no multiplier for the Clamm16 experiment).

Fig. 4 shows the mean and standard deviation (bright color) of the validation error against the epoch number for five runs. In both cases, we see that DGMP achieves overall lower validation errors. Since the validation set is quite small for the Clamm16 dataset (204 samples), the beginning of the validation curve of Fig. 4 is quite wriggled. Interestingly, max pooling has a lower validation error than average pooling. For Fig. 4, the difference becomes clearer and we see that GMP generalizes faster than max or average pooling.

For the Clamm17 dataset, we also experimented with another average pooling variant that enforces the pooled representation to have an $\ell^{2}$ norm of one. The experiments show that this needs a longer convergence time in comparison to normal global average pooling due to the bounded gradients, but results in about the same accuracy.

The average results of the five best models (obtained by taking for each run the model with lowest validation error) can be seen in Tab. III. The standard deviations are: (a) Clamm16: $1.4\text{\,}\mathrm{\char 37\relax}$, $0.5\text{\,}\mathrm{\char 37\relax}$, and $0.6\text{\,}\mathrm{\char 37\relax}$; (b) Clamm17: $0.4\text{\,}\mathrm{\char 37\relax}$, $0.8\text{\,}\mathrm{\char 37\relax}$, and $1\text{\,}\mathrm{\char 37\relax}$ for average pooling, max pooling, and DGMP, respectively.

Additionally, we provide the results when computing an ensemble of the five models. In particular, we compute the predictions by averaging all activations of the last linear layers. In advance, we make the outcomes more comparable by applying the softmax function. Furthermore, each individual output is weighted by the validation accuracy of the respective model.

Similar to the writer identification results, our proposed DGMP layer outperforms the other pooling layers in both CLaMM datasets. For the Clamm16 dataset, we even achieve a new highscore with a single model. Using an ensemble, we are substantially better than the approach of Tensmeyer et al. [1], who use an ensemble of two different models. For the Clamm17 dataset, our method is comparable to the T-DeepCNN submission [20], where an ensemble of five models trained with different image sizes is computed.

V Conclusion

We presented Deep Generalized Max Pooling (DGMP), a new pooling layer that balances on a per-sample basis frequent and rare activations. While common average and max pooling operate individually on each activation map, DGMP considers all locations in the activation volume as input while adding only one additional parameter to the network. We show that DGMP is superior to the commonly used global average or max pooling. In writer identification/retrieval and font classification, it achieves higher accuracies than those global pooling techniques. While not the focus of this work, we also outperform another method on the Clamm16 dataset while performing comparable on the Clamm17 dataset. For future work, we would like to improve our work on writer identification, e. g., by integrating this layer in the work of Christlein et al. [3]. We would also like to expand our evaluation on other scenarios, such as word spotting where the commonly-used PHOCNet [2] uses a global max pooling layer.