Generative Convolutional Networks for Latent Fingerprint Reconstruction

TL;DR

This paper introduces a generative convolutional network approach to enhance fingerprint images by denoising and reconstructing missing ridge patterns, improving the accuracy of fingerprint recognition especially in corrupted or partial images.

Contribution

It presents a novel application of generative convolutional networks for latent fingerprint enhancement, combining deep learning with traditional feature extraction methods.

Findings

Improved ridge pattern reconstruction in corrupted fingerprint images.

Enhanced fingerprint recognition accuracy on multiple datasets.

Effective pre-processing step for latent fingerprint recognition systems.

Abstract

Performance of fingerprint recognition depends heavily on the extraction of minutiae points. Enhancement of the fingerprint ridge pattern is thus an essential pre-processing step that noticeably reduces false positive and negative detection rates. A particularly challenging setting is when the fingerprint images are corrupted or partially missing. In this work, we apply generative convolutional networks to denoise visible minutiae and predict the missing parts of the ridge pattern. The proposed enhancement approach is tested as a pre-processing step in combination with several standard feature extraction methods such as MINDTCT, followed by biometric comparison using MCC and BOZORTH3. We evaluate our method on several publicly available latent fingerprint datasets captured using different sensors.

Tables

Table 1. Table 1: Latent-to-Latent matching on IIIT-Delhi latent database. Matching was performed on images obtained with the proposed fingerprint enhancement (Our) and with no enhancement (Raw).

Method		Accuracy
Enhancement	Extract + Match	Rank-1	Rank-10
Raw	ABR11 + MCC	35.58%	58.27%
Raw	NBIS	57.69%	71.92%
Our	ABR11 + MCC	57.12%	79.04%
Our	NBIS	62.69%	78.85%

Table 2. Table 2: Latent-to-Sensor matching on IIIT-Delhi MOLF database. Shown as the performance of matching Lumidigm images to latent fingerprints with enhancement (Our) and with no enhancement (Raw). Two more methods listed in [ 31 ] are compared.

Method		Accuracy
Enhancement	Extract + Match	Rank-25	Rank-50
Raw	MINDTCT + MCC	8.02%	12.59%
Raw	ABR11 + MCC	3.07%	5.45%
Our	MINDTCT + MCC	12.55%	18.36%
Our	ABR11 + MCC	16.14%	22.36%
[31]	NBIS	N/A	6.06%
[31]	VeriFinger	N/A	6.80%

Table 3. Table 3: Cross-dataset Latent-to-Sensor matching on IIIT-Delhi MOLF database. Performance of matching Lumidigm, Secugen, and Crossmatch sensor images to latent fingerprints enhanced by our model (Our) and with no enhancement (Raw). In all cases, combination ABR11 + MCC was used for feature extraction and matching.

Method		Accuracy
Enhancement	Gallery dataset	Rank-25	Rank-50
Raw	Lumidigm	3.07%	5.45%
Raw	Secugen	2.59%	5.18%
Raw	Crossmatch	2.68%	5.32%
Our	Lumidigm	16.14%	22.36%
Our	Secugen	13.27%	19.50%
Our	Crossmatch	12.66%	19.07%

Equations

$$G_{\theta }(I)=I\star S_{\theta },$$

$$E_{grad}(I_t,I_r)=\frac{{\sum }_{\theta \in \mathcal{T}}\frac{1}{n}\Vert (I_t-I_r)\star S_{\theta }{\Vert }_2^2}{\left|\mathcal{T}\right|},$$

$$\mathrm{Cov}(I(x,y))=\left(\begin{array}{cc}{\mu }_{20}^{\prime }& {\mu }_{11}^{\prime }\\ {\mu }_{11}^{\prime }& {\mu }_{02}^{\prime }\end{array}\right)=\left(\begin{array}{cc}{G}_{xx}& G_{xy}\\ G_{xy}& G_{yy}\end{array}\right),$$

$$\Theta =\frac{1}{2}{\tan }^{-1}\left(\frac{2G_{xy}}{G_{xx}-G_{yy}}\right).$$

$$({\Phi }_x,{\Phi }_y)=(\cos (2\Theta ),\sin (2\Theta )),$$

$$({\Phi }_x^{\prime },{\Phi }_y^{\prime })=(g_{{\Sigma }_o}\star {\Phi }_x,g_{{\Sigma }_o}\star {\Phi }_y).$$

$$I_{min}=\frac{(G_{yy}+G_{xx})-(G_{xx}-G_{yy}){\Phi }_x^{\prime }-G_{xy}{\Phi }_y^{\prime }}{2},$$

$$I_{max}=G_{yy}+G_{xx}-I_{min},$$

$$R=1-\frac{I_{min}}{I_{max}}.$$

$$E_{ori}=\frac{1}{n}\Vert \Theta (I_t)-\Theta (I_r){\Vert }_2^2,$$

$$E_{rel}=\frac{1}{n}\Vert R(I_t)-R(I_r){\Vert }_2^2.$$

$$E=E_{grad}+\lambda (E_{ori}+E_{rel}),$$

Full text

1 Introduction

Fingerprints have been in used as a means of person authentication since ancient time. The first use of fingerprints in a criminal investigation dates back to 1880’s, when Francis Galton devised the first method for classifying fingerprints. In 1892, the first successful fingerprint-based identification helped convict a murderer. Since then, fingerprinting has underwent massive development and is nowadays an important part of crime scene investigation as well as modern biometric identification systems.

Based on their acquisition process, fingerprints can be categorized as inked, sensor-scan (e.g. optical, capacitive, etc.), or latent. The first two categories are typically further subdivided into rolled (nail-to-nail fingerprints), flat (single finger) or slap (four finger flat). Those have been heavily researched in the past yielding several state-of-the-art methods [20, 19, 22].

On the other hand, latent fingerprints are impressions of the papillary lines that are unintentionally left by a subject at crime scenes. They are typically partial, blured, noisy, and exhibit poor ridge quality, as opposed to inked or sensor-scan fingerprints. Such fingerprints are usually lifted from the surface by means of special chemical procedures and photographed using high-resolution camera for further processing. Latent fingerprint matching is a challenging problem and state-of-the-art methods developed for inked and sensor-scan fingerprints do not work well (they typically fail to detect the minutiae or detect many false minutiae due to the imperfections mentioned above).

In practice, latent fingerprints are analyzed with the help of forensic examiners who perform a manual latent fingerprint identification procedure called ACE-V (Analysis, Comparison, Evaluation, and Verification) [3]. This process is however very demanding and time consuming. In order to make identification efficient, forensic experts tend to restrict the population against which compare the latent fingerprints (focusing, for instance, only on suspects selected by witnesses or other evidence). This obviously reduces the likelihood of effectively identifying the culprit, thus making the overall process less reliable.

Nowadays, several systems supporting the work of forensic examiners are available. The biggest one, AFIS (Automated Fingerprint Identification System), allows examiners to match latent fingerprints against large databases in a semi-automatic manner. The process usually consists of manually marking the minutiae points; launching the AFIS matcher; and visually verifying the top candidate fingerprints. Such a process requires considerable amount of tedious manual labour.

In this paper, we improve latent fingerprint recognition by enhancing the input fingerprint image that allows using standard feature extraction methods more reliably. Inspired by the previous success of convolutional autoencoders (CAE) for image processing tasks [24, 35, 23, 28, 13, 9], we design a convolutional autoencoder neural network capable at reconstructing high-quality fingerprint images from latent fingerprints. The neural network is trained on a synthetic dataset consisting of partial and blurry fingerprint impressions containing background noise; the output of the network is compared to the groundtruth fingerprint image. The training set was generated using the open source implementation [2] of the SFinGE fingerprint generator [8].

We evaluate our method on two publicly available latent fingerprint datasets: IIIT-Delhi latent fingerprint [30] (for latent-to-latent fingerprints matching) and IIIT-Delhi MOLF [31] (for latent-to-sensor-scan fingerprints matching). We show the broad applicability of our approach and discuss the possible future directions. All the evaluations were done using only open source software. We used MINDTCT [27] and an implementation of [1] for features extraction, and BOZORTH3 [27] and MCC (Minutiae Cylinder Code) [6, 7, 11, 12] for fingerprints matching. We show that using our fingerprint enhancement method, the performance of non-commercial fingerprint matching algorithms can be improved and becomes comparable to some of the commercial ones.

The rest of the paper is organized as follows. In Section II, previous works are reviewed. Section III describes the objective function used for training the model. Section IV describes the convolutional autoencoder architecture. Section V presents the experimental results. Finally, Section VI concludes the paper and discusses possible future research directions.

2 Related work

Different solutions for improving latent fingerprint matching systems have been explored in the past. Many works have improved latent fingerprint matching by using extended features that require manual annotation [16, 33]. However, performing the annotation in low-quality latent fingerprints may be very time-consuming and even infeasible in some settings. Other works have strived towards reducing the amount of manual input to only selection region of interest (ROI) and singular points [33, 34]. Approaches performing fusion of multiple matchers or multiple latent fingerprints were explored in [30].

A different direction for improving fingerprint matching concerns the enhancement of the acquired latent fingerprint images. Yoon et al. [33] proposed a semi-automated method for enhancing the ridge information using the estimated orientation image. A more robust orientation field estimation technique based on Short-Time Fourier Transform and RANSAC has been proposed in [34]. Feng et al. [10] proposed an approach capable of using prior knowledge on the fingerprint ridge structure employing a dictionary of reference orientated patches. Cao et al. [5] introduced a coarse-to-fine dictionary-based ridge enhancement technique.

Most recent work similar to the proposed approach is Schuch et al. [32], who used a convolutional autoencoder for inked and sensor-scan fingerprints ridge enhancement. Their network had a rather simple architecture and was not evaluated in challenging latent fingerprint recognition settings.

3 Gradient-based fingerprint similarity

Since our method is based on Convolutional Neural Networks (CNN), the gradient analysis of the fingerprint image ridge pattern becomes a natural choice as the computation of the image gradient can be implemented using the convolutional operator.

Gradient filters.

We compute the directional derivatives of the fingerprint image $I$ by convolving it with the directional kernel $S_{\theta}$

[TABLE]

where $\theta$ is the direction in which we want to compute the gradient (we use 0, 45, 90, and 135 degrees).

As a criterion of similarity of two fingerprint images (target image $I_{t}$ and reconstructed image $I_{r}$), we used the average Mean Squared Error (MSE) on all the directions,

[TABLE]

where $\mathcal{T}=\{0,45,90,135\}$ is the set of considered orientations and $n$ is the number of image pixels.

Ridge pattern orientation.

Orientation of the ridge pattern can be defined through image moments using the already computed image gradients. Considering $G_{x}=G_{0}$ and $G_{y}=G_{90}$, we define the covariance matrix of the image using second order central moments ($\mu_{20}^{\prime}$, $\mu_{02}^{\prime}$, $\mu_{11}^{\prime}$) [14]:

[TABLE]

where $G_{xx}=g_{\Sigma_{s}}\star(G_{x}\cdot G_{x})$, $G_{yy}=g_{\Sigma_{s}}\star(G_{y}\cdot G_{y})$ and $G_{xy}=g_{\Sigma_{s}}\star(G_{x}\cdot G_{y})$. $g_{\Sigma}$ represents a Gaussian smoothing kernel with covariance ${\Sigma}$ and $\cdot$ denotes the element-wise product.

Eigenvectors of the covariance matrix $\mathrm{Cov}(I(x,y))$ point in the directions of the major and minor intensity of image $I$. This information is enough to compute the orientation as the angle between the eigenvector corresponding to the largest eigenvalue and the x-axis is expressed by formula:

[TABLE]

In order to strengthen the similarity between the reconstructed image and the associated ground-truth, we further calculate the reliability orientation field [17]. The ridge orientation image is converted into a continuous vector field as

[TABLE]

Subsequently, we apply Gaussian low-pass filter on the resulting vector field,

[TABLE]

The reliability measure $R$ is then defined by means of minimum inertia $I_{min}$ and maximum inertia $I_{max}$ as follows:

[TABLE]

[TABLE]

[TABLE]

The intuition behind is that when the ratio $I_{min}/I_{max}$ close to 1, there is very little orientation information at that point.

Finally, we define the orientation energy $E_{ori}$ and reliability energy $E_{rel}$ as the MSE of the orientation and reliability measures computed on the target image $I_{t}$ and the reconstructed image $I_{r}$,

[TABLE]

[TABLE]

4 Fingerprint reconstruction model

Following the principles described in the previous sections and the guidelines presented in [29], we approached the fingerprint reconstruction problem using a fully convolutional autencoder network.

Architecture.

The encoding part corresponds to a fully convolutional neural network comprising five convolutional layers. It receives as input tensors of dimensions 1@260x320 and produces as output tensors of dimensions 1024@8x10 (F@WxH is a compact notation for tensors with width W, height H and F different feature channels). The convolutional layers have stride equal to 2 and are equipped with REctified Linear Unit (RELU) [25]. Batch normalization is applied to each layer for faster convergence [15].

The output of the encoder is directly fed as an input to the decoding part. The decoder copies the architecture of the encoder performing the following changes. Each convolutional layer is replaced with de-convolutional layer (i.e. a fractionally-strided convolutional layer with stride equal to 0.5 [29]), and each RELU with a Leaky-RELU [21]. In order to reproduce the original grey scale image, a further convolutional layer equipped with a sigmoid activation function is applied at the end of the decoder. Figure 2 depicts our architecture.

Training objective.

Based on the analysis described in Section 3, we define an objective function that allows to efficiently train our convolutional autoencoder to reconstruct corrupted parts of the fingerprint images. The objective function corresponds to a weighted average of three different losses:

[TABLE]

where $E_{grad}$, $E_{ori}$ and $E_{rel}$ are as defined in the previous section, and $\lambda$ is a parameter weighting the contribution of the orientation and reliability regularizers.

Training.

For the purpose of training our model, we generated a dataset of synthetic fingerprints. From these fingerprint images, we simulated latent fingerprint images by applying rotation, translation, directional blur and morphological dilation, and blend the results with several different backgrounds. This way, we aim to simulate the latent fingerprint formation as reliably as possible. We binarize the groudtruth (target) images using approach based on [4]. The network is applied to the synthetic latent fingerprint images and tries to reconstruct the underlying groundtruth image by minimizing the above objective function between the output and the groundtruth image. Since the groundtruth images are binarized, our model learns this way not only to reconstruct the fingerprints, but also to directly produce binary images.

Overall, we used 15000 synthetic fingerprint images for training. Data augmentation was performed by adding random Gaussian i.i.d. noise with $\mu=0$ and $\sigma=3.5\cdot 10^{-3}$. The network was trained for 400 epochs (each epoch consisting of 64 iterations). In each iteration, we fed the network with a batch of 12 latent/groundtruth image pairs. We employed Adam [18] updates with $\beta_{1}=0.5$ and weight decay using $L_{2}$ regularization with $\mu=10^{-4}$. The learning rate is set to $2\cdot 10^{-4}$. The weighting parameter $\lambda$ was set to $0.1$.

5 Experiments

We demonstrate the proposed method on several settings of latent fingerprint recognition. The experiments were carried out using publicly available datasets. Latent fingerprint enhancement was evaluated using the IIIT-Delhi Latent fingerprint [30] and IIIT-Delhi MOLF [31] datasets.

For the first evaluation, we applies standard fingerprint recognition algorithms on original fingerprint images and those enhanced by our method. We employed two different feature extraction methods: ABR11 proposed by Abraham et al. [1] and MINDTCT from NBIS [27]. Extracted features were subsequently compared using two different methods, BOZORTH3 [27] (abbreviated to BOZ in the following) and Minutia Cylinder Code (MCC) [6, 7, 11, 12]. For the other evaluations, we used the combination ABR11 + MCC as the best performing one for latent fingerprint matching. Results of latent fingerprint enhancement were presented using TopX measure and Cumulative Match Characteristic (CMC) curves.

Latent-to-Latent matching.

We evaluated latent-tolatent fingerprint matching using the protocol described in [30]. The whole IIIT-Delhi latent fingerprint dataset contained 1046 samples of all the ten fingerprints collected from 15 different subjects. We followed the dataset split strategy proposed in [30], randomly choosing 395 images as gallery and 520 as probes, making sure that each class contained at least one gallery sample. 131 images were left out since Sankaran et al. used them for training. We performed 10-fold cross-validation in order to ensure the random splitting does not influence the reported performance.

Table 1, shows Rank-1 and Rank-10 accuracy for both recognition methods with and without our enhancement. Our approach significantly improves the matching accuracy. Comparing to [30], we outperform all the fingerprint matching methods they have evaluated. It is worth emphasizing that differently from Sankaran et al. we do not need to train on the subset of the data as they do.

The CMC curves for this experiment are shown in Figure 3. Our model boosts the performance of ABR11 + MCC much more than that of NBIS. We attribute this to the fact that the energy we minimize while training our model performs very similar operations as the ridge binarization part of the ABR11 feature extraction algorithm.

Latent-to-Sensor matching.

Our second set of experiments tackled an even more challenging task. We used the MOLF dataset containing all ten fingerprints of 100 different subjects. The samples in this dataset were of very different quality, including some very poor samples where no ridge structure was visible at all. Fingerprints of each participant were captured with several commercial fingerprint scanners (Lumidigm, Secugen and Crossmatch). In addition, each participant provided a set of latent fingerprints. It is therefore possible to match latent fingerprints to those acquired by a sensor. Following the testing protocol by Sankaran et al. [31], we considered the first and second instance fingerprints for each user from a sensor scanned database as the gallery. The whole latent fingerprint database consisting of 4400 samples was considered as probe set.

For this experiment, we refer to the case from [31], where minutiae were extracted automatically and afterwards matched using one of the standard algorithms. Sankaran et al. evaluated the performance of publicly available NBIS and commercial VeriFinger [26] fingerprint matching methods, reporting very poor performance for both. Here, we used the ABR11 + MCC method for matching. Table 2 shows that using our enhancement algorithm, ABR11 + MCC performs very poorly, consistently with the finding of [31]. Performing the enhancement with our model significantly improves the performance. CMC curves for this experiment are shown Figure 4.

Cross-dataset Latent-to-Sensor matching.

To support the fact that our method works not only matching latent fingerprints to Lumidigm (MOLF DB1) sensor scanned fingerprints, we provide a comparison of matching latent fingerprints to Secugen (MOLF DB2) and Crossmatch (MOLF DB3) sensor samples as well using the combination of ABR11 feature extractor and MCC matching method. As samples from various sensors are of different quality, the performance can differ slightly. However, Table 3 indicates that independently of the source sensor, our enhancement algorithm provides significant performance improvement. The CMC curves for this experiment are shown in Figure 5.

Fingerprint quality.

In order to qualitatively demonstrate how well our model reconstructs the latent fingerprints, we perform quality assessment using the NFIQ utility (part of NBIS [27]), which assigns a fingerprint image a numerical score from 1 (best quality) to 5 (poorest quality). The distribution of score values shown in Figure 6 clearly indicates that our method improves the fingerprint quality.

Moreover, we show several successful and several unsuccessful latent fingerprint reconstructions for both real (Figure 7) and synthetic (Figure 8) data. It is clearly visible that our model enhances the ridge information well when it is present and has problems in places where the original images does not contain discernible ridges. This suggests that very poor samples might not contain any meaningful information for the reconstruction process.

6 Conclusion

Convolutional autoencoders are amongst popular methods extensively used in image processing tasks for image denoising and inpainting. Inspired by the previous applications, we have explored the possibility of using the convolutional autoencoders to reconstruct latent or damaged fingerprints. With regard to principles of some of the fingerprint feature extraction and matching algorithms, we have carefully designed an objective function that should both well reflect on the important fingerprint properties and be efficient to optimize as it is based on gradient analysis, which has been already implemented on the GPUs. Our method is based on learning, however, in comparison to some of the previous research, we do not need any real training data and we effectively train our model on well designed synthetic dataset, which gives us the advantage of no training set size limitation.

We obtain state-of-the-art results on several challenging tasks such as latent-to-latent fingerprint matching and latent-to-sensor database fingerprint matching on IIIT-D standard datasets, outperforming the existing results using on the same data by a margin. Especially evaluation on very challenging IIIT-D MOLF dataset compensates for the fact that we cannot evaluate on NIST-SD27 as it is discontinued and no longer provided by NIST.

On the other hand, we observe that the reconstruction is not always successful, showing some of the failure cases that are prone to generate false minutiae. We would like to further examine this issue as future work.

As we do not aim to directly extract minutiae or perform matching, but rather reconstruct the correct ridge pattern of the poor-quality fingerprint, our method has broad applications ranging from latent fingerprint enhancement to possibly reconstruction of fingerprints affected by diseases, which is another of our desired future directions.

7 Acknowledgements

The authors are supported in part by ERC Starting Grant No. 307047 (COMET), ERC Consolidator Grant No. 724228 (LEMAN) and Nvidia equipment grant.