PoreNet: CNN-based Pore Descriptor for High-resolution Fingerprint Recognition

TL;DR

This paper introduces PoreNet, a CNN-based method for high-resolution fingerprint recognition that detects pores and computes descriptors, achieving state-of-the-art accuracy on benchmark datasets.

Contribution

The paper presents PoreNet, a novel residual CNN model for pore feature extraction, improving fingerprint recognition accuracy over existing methods.

Findings

Achieves 2.91% and 0.57% EERs on benchmark datasets.

Outperforms current state-of-the-art in FMR1000 and FMR10000.

Effective pore-based biometric recognition in high-resolution fingerprints.

Abstract

With the development of high-resolution fingerprint scanners, high-resolution fingerprint-based biometric recognition has received increasing attention in recent years. This paper presents a pore feature-based approach for biometric recognition. Our approach employs a convolutional neural network (CNN) model, DeepResPore, to detect pores in the input fingerprint image. Thereafter, a CNN-based descriptor is computed for a patch around each detected pore. Specifically, we have designed a residual learning-based CNN, referred to as PoreNet that learns distinctive feature representation from pore patches. For verification, the match score is generated by comparing pore descriptors obtained from a pair of fingerprint images in bi-directional manner using the Euclidean distance. The proposed approach for high-resolution fingerprint recognition achieves 2.91% and 0.57% equal error rates (EERs) on partial (DBI) and complete (DBII) fingerprints of the benchmark PolyU HRF dataset. Most importantly, it achieves lower FMR1000 and FMR10000 values than the current state-of-the-art approach on both the datasets.

Tables

Table 1. TABLE I: Detailed architecture of PoreNet

Layer name	Output shape	Kernel
conv1	$41\times 41,16$	$3\times 3$, $16$, stride 1, padding same
conv2_x	$41\times 41,64$	[ $1\times 1$, $32$ $3\times 3$, $32$ $1\times 1$, $64$ ] $\times 2$
conv3_x	$41\times 41,128$	[ $1\times 1$, $64$ $3\times 3$, $64$ $1\times 1$, $128$ ] $\times 2$
conv4	$41\times 41,1$	$3\times 3$, $1$
Flatten	$1681$	$-$
$l_2$-norm	$1681$	$-$
Total parameters	142,881	$-$

Table 2. TABLE II: Details of the PolyU HRF dataset

Dataset	Resolution (dpi)	Image size (pixels)	Fingers	Images per finger per session	Total images
DBI:Training set	1200	$320\times 240$	35	3	210
DBI:Test set	1200	$320\times 240$	148	5	1480
DBII	1200	$640\times 480$	148	5	1480

Table 3. TABLE III: EERs on PolyU datasets

Method	EER (%)
	DBI	DBII
Jain et al. [8]	$30.45\%$	$7.83\%$
Zhao et al. [9]	$15.42\%$	$7.05\%$
Liu et al. [12]	$6.59\%$	$0.97\%$
Liu et al. [13]	$3.25\%$	$0.53\%$
Segundo and Lemes [15]	$3.74\%$	$0.76\%$
Dahia and Segundo [19]	$4.18\%$	1.14%
Xu et al. [21]	1.73%	0.51%
Proposed approach	$2.91\%$	$0.57\%$

Table 4. TABLE IV: A comparative analysis using FMRs

Method	FMR10000		FMR1000
	DBI	DBII	DBI	DBII
Dahia and Segundo [19]	$16.20\%$	$3.33\%$	$10.82\%$	$2.25\%$
Xu et al. [21]	$6.94\%$	$2.01\%$	$5.92\%$	$1.31\%$
Proposed approach	6.47%	1.24%	5.42%	0.96%

Table 5. TABLE V: Details of the IITI-HRF dataset

Dataset	Resolution (dpi)	Image size (pixels)	Fingers	Total images
IITI-HRFP	1000	$320\times 240$	800	6400
IITI-HRFC	1000	$1000\times 1000$	800	6400

Table 6. TABLE VI: EERs on IITI-HRF dataset

Method	EER (%)
	IITI-HRFP	IITI-HRFC
Segundo and Lemes [15]	$8.37\%$	$2.90\%$
Proposed approach	$9.58\%$	$4.28\%$

Equations

$$\centering O_{n}(i,j)=max\Bigg{(}0,\frac{\partial I}{\partial n}\Bigg{)}\vspace{-0.5em}\@add@centering$$

$$R=\operatorname*{arg\,max}_{i\in\{1,2,\ldots,N\}}\bigg{(}\sum_{j=1,j\neq i}^{N}S_{ij}\bigg{)}$$

$$L_{triplet}=\max\big{\{}d(a,p)-d(a,n)+margin,0\big{\}}$$

Full text

I Introduction

Fingerprint is one of the most widely explored biometric traits, mainly due its distinctiveness and permanence [1]. The features extracted from a fingerprint image are broadly classified into level-1, level-2 and level-3 features. Level-1 features, which include global ridge orientation, are commonly used for fingerprint classification. Level-2 fingerprint features include finer details such as ridge endings and ridge bifurcations, which are collectively called minutiae [1]. Level-3 fingerprint features, on the other hand, include very fine details such as pores, incipient ridges, dots, and ridge contours. Level-1 and level-2 features can be observed in 500 dpi fingerprint images, while level-3 features are generally observable in fingerprint images having a resolution greater than 800 dpi [2].

Commercially available automated fingerprint recognition systems (AFRS) and a majority of the methods reported in the literature employ level-1 and level-2 features. However, with the advent of high-resolution fingerprint sensors, there has been a focus shift and several methods that employ level-3 features have been developed for fingerprint recognition. In addition to enhancing the recognition performance, level-3 features provide higher level of security, as they are difficult to forge. Further, the level-3 features have also been included in the extended feature set for fingerprint recognition [3]. Over the last few years, there has been growing interest in level-3 fingerprint features, especially the pores and several methods have been proposed for pore feature based automated fingerprint recognition [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. A pore-based AFRS typically consists of two major steps namely, pore detection in high-resolution fingerprint images and matching fingerprints using the detected pores. Stosz and Alyea [4] in their pioneering work proposed a fingerprint recognition approach that uses both pores and minutiae. Their approach involves detecting pores by tracing the ridges in skeletonized fingerprint image, followed by a multi-level matching using pores and minutiae. Roddy and Stosz [5] provided a detailed discussion on the statistics of the pores and examined its effectiveness in improving the performance of the existing AFRS. Krysczuk et al. [6, 7] demonstrated the efficacy of pore features for fragmentary fingerprint recognition. In their approach, closed pores are detected by applying a set of thresholds to the binarized fingerprint image and open pores are detected by skeletonizing the valleys and finding the spurs having a sufficient number of white pixels in the neighbourhood. Their experimental results demonstrated that pore features are vital in recognizing partial fingerprint. The early studies [4, 5, 6, 7] employed skeletonization-based approaches to detect pores. Such approaches are suitable only for very high-resolution ($\sim$ 2000 dpi) fingerprint images and their performance is likely to be adversely affected by fingerprint degradation caused by skin conditions. To circumvent these challenges, Jain et al. [8] presented a hierarchical fingerprint recognition approach that utilizes features from all the three levels. In their approach, pores are detected by applying Mexican-hat wavelet transform on the linear combination of the original and the enhanced fingerprint image. The fingerprints are first matched using minutiae and level-3 features are extracted in the neighbourhood of the matched minutiae points. The extracted level-3 features are then matched using the iterative closest point (ICP) algorithm. Later on, Zhao et al. [9] proposed an approach, in which the pores are extracted using the adaptive pore filtering [17]. For each pore, a descriptor is formed by considering the pixel intensities in the neighbourhood. The initial correspondences are established through dot product and are refined using the random sample consensus (RANSAC) algorithm. The authors have demonstrated the usefulness of pores for biometric recognition using partial fingerprint images, which may not contain sufficient level-2 features [10, 11]. Liu et al. [12] proposed an improved direct pore matching approach, which employs the same pore descriptor as in [9]. The coarse pore correspondences obtained through sparse representation are refined using the weighted RANSAC (WRANSAC) [18]. This work has been extended in [13], which employs the tangent distance and sparse representation to compare the pores extracted from the reference and probe fingerprint images.

Recently, Lemes et al. [14] proposed a pore detection approach with a low computational cost. Their approach is adaptive and handles variations in the pore size. Firstly, a binary fingerprint image is obtained through global thresholding. For every white pixel, the average valley width is then estimated by computing the distance to neighbouring dark pixels in each of the four directions. The average valley width is used to define the size of a mask centered on each white pixel. Bright pixels inside the mask are then used to define a local threshold $T_{local}$ and a local radius $r_{local}$. Finally, a circle centered at each bright pixel with its local radius $r_{local}$ is used to determine whether the bright pixel is part of a pore or not. Segundo and Lemes [15] improved the dynamic pore filtering approach [14] by considering the average ridge width in place of the average valley width to obtain the global and local radii, which are used in the same manner as in [14] to estimate the pore coordinates. The authors in [15] performed ridge reconstruction from the detected pores by employing Kruskal’s minimum spanning tree algorithm. In the matching stage, a scale invariant feature transform (SIFT) based descriptor is obtained for each pore and the pores with bidirectional correspondences are used to compute the matching score. The ridge structure and ridge consistency of the corresponding pores are also used to generate the matching score. Most recently, Dahia and Segundo [19] presented an approach to generate pore annotation by aligning fingerprint images in the training set, followed by learning a descriptor for each of the pore patches by using an existing CNN-based patch matching model, HardNet [20]. The two-step method [21] to compare fingerprint pores aligns the fingerprint images using a data-driven descending algorithm. The alignment process utilizes the fingerprint ridges and orientation field. Once the fingerprint images are aligned, the pores present in the overlapping area between the two fingerprint images are matched using a graph comparison method.

A review of the literature indicates that there is room for improvement in level-3 feature detection and the subsequent matching. The objectives of this work is to explore deep pore-descriptors and to advance the state-of-the-art in high-resolution fingerprint recognition. To this end, we have utilized CNN-based deep learning, which has proven successful for various computer vision problems [22, 23, 24]. The key contribution of this paper is a residual learning-based convolutional neural network, referred to as PoreNet, that learns distinctive feature representation from pore patches in high-resolution fingerprint images. In addition, we have developed an automated approach to generate labels for the pores that are common to different impressions of a finger belonging to the training set. We have also studied the effect of cross-sensor data on the proposed approach. Specifically, this is the first study that examines the performance of a learning based fingerprint recognition approach by testing the model on cross-sensor fingerprint images. The in-house high-resolution fingerprint dataset used in this study will be made publicly available to further research in this area.

The rest of this paper is organized as follows: Section II presents an introduction of the proposed methodology followed by a detailed description of the pore label generation approach and the pore descriptor learning approach. Experimental results and discussion are presented in Section III. Finally, our concluding remarks are presented in Section IV.

II Proposed Method

The proposed method employs a CNN model, DeepResPore [25], to detect pores in the input fingerprint image. It generates a pore intensity map, which is processed to obtain a binary pore map. Thereafter, another CNN model is used to compute a deep descriptor for a patch around each detected pore. This residual learning-based [26] CNN model, referred to as PoreNet, has been trained to compute distinctive feature representation from pore patches. For verification, a Euclidean distance-based matching score is generated by comparing the pore descriptor sets obtained from the probe and reference fingerprint images. The schematic diagram of the proposed method is presented in Fig. 1.

II-A Pore label generation

Given a training dataset, we first detect pores in the fingerprint images using DeepResPore model, as shown in Fig. 1a. For a given input fingerprint image, DeepResPore generates a pore intensity map, from which a binary pore map is obtained through local maxima filtering. To train a CNN model in a supervised manner, one requires a labeled dataset. Specifically, to train PoreNet, we require pore patches and their corresponding labels. To this end, we have first obtained labels for the pores that are common to different impressions of a finger.

We have generated pore labels using a handcrafted key-point descriptor namely, DAISY [27], which has been shown to be very effective in representing pores [28]. To obtain DAISY descriptor for a given image $I$, firstly, a set of orientation maps $O_{n}$, $1\leq n\leq N$, are computed as follows [27]:

[TABLE]

where $\frac{\partial I}{\partial n}$ is the image derivative computed at $(i,j)$ along the direction $n$. The orientation maps $O_{n}$ are then convolved with a set of Gaussian kernels having different standard deviation ($\sigma$) values. Each pixel’s neighborhood in the convolved orientation map is then divided into overlapping circular regions on a series of concentric rings around the center pixel. Next, a normalized histogram of values from the convolved orientation maps is computed for each of the circular regions. Finally, the histograms computed from each of the circular regions are concatenated to obtain the DAISY descriptor. A detailed description of the DAISY descriptor can be found in [27].

The steps involved in our pore-label generation process are as follows: firstly, we identify a reference fingerprint image for each of the fingers present in the training dataset. The objective is to select a reference fingerprint image that has the maximum number of pores common to all other impressions of that finger. To this end, we treat the detected pore coordinates as key-points and compute DAISY-based descriptor for each of the detected pores. Since the training set contains multiple impressions of a finger, we consider one impression ($i$) at a time and perform pair-wise comparisons with rest of the impressions ($j$) using the DAISY-based pore descriptor. The match score (i.e. the number of matched descriptors) generated from each comparison is stored in $S_{ij}$. This is repeated for every impression of a finger and the index $R$ of the reference fingerprint image is determined as follows:

[TABLE]

where $N$ is the number of impressions of a finger. The impression, for which the sum of match scores is maximum, is considered to be the reference image.

Once the reference fingerprint image is identified, the remaining impressions are aligned with the reference fingerprint to find the common pores that are present in all the impressions of that finger. The affine transformation [29, 30] for alignment is estimated using pore correspondences established based on the earlier comparison of DAISY-based pore descriptors. After aligning all impressions of a finger with its reference image, we identify the reference fingerprint image pore coordinates for which there are corresponding pores within the image boundaries of every transformed impression. We consider them to be common pores. At the end of this process, we have the coordinates of the common pores that are present in all impressions of each finger. Finally, a patch of $41\times 41$ pixels centered at each of the common pores is extracted and assigned a unique label. It is important to note that a finger having $P$ pores common to all its $r$ impressions will have $P\times r$ pore patches having $P$ unique labels in the training set.

II-B Pore descriptor learning

On completion of the pore-label generation process, we have a set of pore patches $X=\{x_{1},x_{2},\ldots,x_{T}\}$ and the corresponding set of labels $Y=\{y_{1},y_{2},\ldots,y_{T}\}$ for each of the fingerprint images in the training set. To learn a pore-patch descriptor, we have designed a customized residual learning-based CNN model, referred to as PoreNet. The detailed architecture of PoreNet is presented in Table I. As can be seen, the proposed model has 14 learnable layers with 4 residual blocks consisting of 4 shortcut connections. The PoreNet takes a pore-patch $I_{p}$ of size $41\times 41$ pixels as input and converts it into a 1681-dimensional pore descriptor. As shown in Fig. 2, the PoreNet does not perform any pooling operation in order that the size of the output feature map remains the same as that of the input. At the end of the network, a convolutional layer containing a single filter is introduced to generate the final feature map, which is flattened and normalized such that $l_{2}$-norm of the output embedding is equal to one [22]. All convolution operations in PoreNet are performed with a stride of one. At every stage, zero padding is employed to maintain the size of the feature map. Furthermore, all convolutional layers, except the last one, perform convolution followed by batch normalization [31] and ReLU activation.

In the training phase, the PoreNet is trained from the scratch. It is trained end-to-end in a supervised learning manner with the objective of minimizing the value of a triplet loss function. A batch of pore patches are fed to PoreNet, which generates the corresponding high-dimensional embeddings. Next, an online triplet mining scheme namely, batch-hard triplet mining [22], is applied to the obtained embeddings. In this method, for each anchor sample $(a)$, we first obtain the hardest positive sample $(p)$ having the same label as that of the anchor and the hardest negative sample $(n)$ having a label different from the anchor. Finally, we form the following triplet loss [22]:

[TABLE]

where $d(a,p)$ is the distance between $a$ and $p$ and $d(a,n)$ is the distance between $a$ and $n$. The PoreNet is trained in an end-to-end manner to minimize the above triplet loss.

Once the PoreNet is trained, the task is to match a pair of fingerprint images using the generated pore-patch descriptors. Considering two fingerprint images $I_{1}$ and $I_{2}$ containing $n$ and $m$ number of pore patches, respectively, the PoreNet will generate two sets of pore descriptors $P_{1}\in\mathds{R}^{n\times 1681}$ and $P_{2}\in\mathds{R}^{m\times 1681}$. The pore descriptors in $P_{1}$ are compared with those in $P_{2}$ using the Euclidean distance and the pairs of descriptors matched bidirectionally are retained. Finally, the matches are refined using a distance ratio ($d_{r}$) criterion involving the distance to the second nearest neighbour [32].

III Experimental results and discussion

In this section, we first present the details of preparation of the dataset for training the PoreNet model, followed by results of our experiments.

III-A Dataset preparation

We have performed experiments on the publicly available PolyU HRF dataset [33], which contains high-resolution fingerprint images of resolution 1200 dpi in two different sets DBI and DBII. DBI contains two subsets: DBI-train containing 210 partial fingerprint images from 35 fingers, each contributing 6 impressions and DBI-test containing 1480 partial fingerprint images. The size of these partial fingerprint images is $320\times 240$ pixels. On the other hand, DBII contains 1480 complete fingerprint images of size $640\times 480$ pixels. Fingerprints images in DBI-test and DBII are collected from 148 fingers in two different sessions, with each finger contributing 5 impressions per session. The detailed description of PolyU HRF dataset is provided in Table II. While the pore patches for training PoreNet have been obtained from DBI-train, the datasets DBI-test and DBII have been used to evaluate the proposed approach using the experimental protocol adopted in the previous works [15], [19], [21]. A training set $(80\%)$ and a validation set $(20\%)$ have been obtained by randomly partitioning DBI-train. We have obtained pore patches from every image in the training set and generated their labels using the method detailed in Section II-A. We have also employed data augmentation techniques to increase the amount of training data. Specifically, we have generated twenty (through ten rotations with angles randomly selected between $-20^{\circ}$ and $20^{\circ}$ and ten translations with shifts randomly selected between $-5$ and $5$ pixels) geometrically transformed images from every training sample. Further, we have generated an additional ten images by varying the contrast using the gamma transformation with gamma value selected in the range 0.45 to 0.9 in steps of 0.05. Overall, we have generated 815,982 labelled pore patches to train the PoreNet.

III-B Experimental results

We have trained PoreNet for 100 epochs using a batch size of 256 and learning rate of 0.0001. The loss function has been optimized using the adaptive moment estimation (ADAM) [34]. The margin for the triplet loss function was set to 0.8. All our experiments have been performed on a computer with 3.60 GHz Intel core i7-6850K processor, 48 GB RAM and Nvidia GTX 1080 8 GB GPU. While the pore patches have been generated in MATLAB environment, the PoreNet has been trained and tested using TensorFlow [35] in Python environment.

To make a fair comparison with the existing approaches, we have followed the same experimental protocol as in [15, 19, 21]. A set of genuine scores has been obtained by comparing every fingerprint image from the second session with all fingerprint images belonging to the same finger collected in the first session. On the other hand, impostor scores have been generated by comparing the first sample of every finger from the second session with the first sample of all other fingers from the first session. Thus, we have a total of 3700 $(148\times 25)$ genuine scores and 21,756 $(148\times 147)$ impostor scores. We report the following performance metrics: equal error rate (EER), FMR1000 and FMR10000 [36]. In addition, we present the detection error trade-off (DET) curve to help ascertain the performance of the proposed approach. The $d_{r}$ value was empirically set to 0.8 using the validation set. Table III 111The EERs of the existing approaches [8], [9],[12], [15] are taken directly from the results presented in [15] and those of the approaches [13] and [21] are taken from [21]. EERs values of [19] are obtained from the source code provided by the authors. presents EERs of the proposed and the existing approaches on PolyU datasets. Since the recent methods show comparable performance in terms of the EER, we have further analysed their performance using FMR10000 and FMR1000. These results are presented in Table IV. As can be seen, the proposed approach achieves lower FNMRs on both the datasets. To ascertain this superior performance, we have plotted DET curves (please see Fig. 3). These curves clearly show that the proposed approach achieves better FNMR for low FMRs, specifically, FMR in the range $10^{-4}$ to $10^{-3}$. The performance improvement in this region translates into increased security and convenience for users. Furthermore, the proposed approach requires on average $1.41$ seconds and $2.80$ seconds to compare a pair of fingerprint images from DBI and DBII, respectively. On the other hand, the existing approach [21] takes $8.46$ seconds to compare a pair of fingerprint images.

Overall, the experimental results presented in this section indicate that the proposed PoreNet model provides improvement in performance over the current state-of-the-art approaches. Specifically, it achieves lower FMR10000 and FMR1000 on both benchmark datasets.

III-C Performance in a cross-sensor scenario

In this section, we present results of the experiments conducted to study the effect of cross-sensor fingerprint data on the performance of the proposed approach. As discussed previously, our model has been trained on PolyU HRF dataset [33]. The in-house IITI-HRF high-resolution fingerprint (1000 dpi resolution) dataset [28], which has been expanded to include fingerprint images of 100 subjects, is used for testing. The expanded IITI-HRF high-resolution fingerprint dataset contains images of 8 fingers (all fingers except the little fingers on both the hands), each contributing 8 impressions. These images were acquired using the commercially available Biometrika HiScan-Pro fingerprint scanner. IITI-HRF dataset is partitioned into two subsets. The first one contains partial fingerprint images of size $320\times 240$ pixels, while the second one contains full fingerprint images of size $1000\times 1000$ pixels. The details of IITI-HRF dataset is presented in Table V.

To make a comparison, we have also evaluated the performance of the hand-crafted feature based approach proposed by Segundo and Lemes [15] on IITI-HRF dataset. For both these approaches, genuine and the impostor scores on IITI-HRF dataset have been generated using the following protocol: for each finger, 16 genuine scores have been generated by comparing the last four impressions with each of the first four impressions of the same finger. On the other hand, impostor scores have been generated by comparing the fifth impression of each finger with the first impression of all other fingers. This way, we have a total of 12,800 ($800\times 16$) genuine scores and 639,200 ($800\times 799$) impostor scores. The EERs and DET curves of both the approaches are presented in Table VI and Fig. 4, respectively. As can be observed, the hand-crafted feature-based approach [15] provides lower EER values as compared to the proposed learning-based approach on IITI-HRFP as well as IITI-HRFC. This may be due to the fact that our approach has been trained on PolyU HRF dataset and there exists a considerable variation in terms of resolution and the quality of pores between fingerprint images belonging to PolyU HRF and IIT-HRF. Therefore, the proposed learning based approach appears to suffer from the problem of domain adaptability.

The results presented in this section underline the key challenge facing learning-based fingerprint recognition approaches, specifically, to overcome domain variability in cross-sensor scenarios. Employing domain adaptation techniques such as the one proposed in [37] in the training phase is likely to improve the cross-sensor performance of the learning-based approaches.

IV Conclusion

In this paper, we have presented a deep pore-descriptor based method for high-resolution fingerprint recognition. Specifically, we have developed a residual learning-based CNN named PoreNet to learn a descriptor from pore patches in fingerprint images. The trained PoreNet generates deep embeddings from a given fingerprint image. To train PoreNet efficiently, we have also developed a method that generates pore labels by transforming the images based on the matched DAISY-based pore descriptors and finding the common pores. The results of our evaluations on the benchmark PolyU HRF datasets demonstrate the effectiveness of the proposed PoreNet in generating pore descriptors for high-resolution fingerprint recognition. Most importantly, the proposed PoreNet model achieves state-of-the-art performance in terms of FMR10000 and FMR1000. In future, we plan to work on the domain adaptability of the PoreNet model for cross-sensor fingerprint matching.

Acknowledgment

We thank Maurício P. Segundo for sharing the source codes of their approach. We also thank Yuanrong Xu for providing us with the data points of the DET curve of their approach.