Pooling Facial Segments to Face: The Shallow and Deep Ends

TL;DR

This paper introduces two face detection methods, SegFace and DeepSegFace, designed for mobile environments with occluded and partial faces, demonstrating superior accuracy and robustness through extensive experiments.

Contribution

The paper proposes two novel face detectors based on facial segments, with DeepSegFace utilizing a deep neural network and a new data augmentation scheme for improved performance.

Findings

DeepSegFace outperforms state-of-the-art detectors in accuracy.

DeepSegFace requires less training data due to a novel augmentation scheme.

Both methods are effective in mobile face detection scenarios.

Abstract

Generic face detection algorithms do not perform very well in the mobile domain due to significant presence of occluded and partially visible faces. One promising technique to handle the challenge of partial faces is to design face detectors based on facial segments. In this paper two such face detectors namely, SegFace and DeepSegFace, are proposed that detect the presence of a face given arbitrary combinations of certain face segments. Both methods use proposals from facial segments as input that are found using weak boosted classifiers. SegFace is a shallow and fast algorithm using traditional features, tailored for situations where real time constraints must be satisfied. On the other hand, DeepSegFace is a more powerful algorithm based on a deep convolutional neutral network (DCNN) architecture. DeepSegFace offers certain advantages over other DCNN-based face detectors as it requires relatively little amount of data to train by utilizing a novel data augmentation scheme and is very robust to occlusion by design. Extensive experiments show the superiority of the proposed methods, specially DeepSegFace, over other state-of-the-art face detectors in terms of precision-recall and ROC curve on two mobile face datasets.

Figures

Tables

Table 1. TABLE I: Structure of DeepSegFace’s Convolutional layers (feature extraction and dimensionality reduction)

Segment	Input	Feature	Dim. Reduce	Flatten
$Nose$	$3\times 69\times 81$	$512\times 2\times 2$	$50\times 2\times 2$	$200$
$Eye$	$3\times 54\times 162$	$512\times 1\times 5$	$50\times 1\times 5$	$250$
$U{L}_{34}$	$3\times 147\times 147$	$512\times 4\times 4$	$50\times 4\times 4$	$800$
$U{R}_{34}$	$3\times 147\times 147$	$512\times 4\times 4$	$50\times 4\times 4$	$800$
$U_{12}$	$3\times 99\times 192$	$512\times 3\times 6$	$50\times 3\times 6$	$900$
$L_{34}$	$3\times 192\times 147$	$512\times 6\times 4$	$50\times 6\times 4$	$1200$
$U{L}_{12}$	$3\times 99\times 99$	$512\times 3\times 3$	$50\times 3\times 3$	$450$
$R_{12}$	$3\times 192\times 99$	$512\times 6\times 3$	$50\times 6\times 3$	$900$
$L_{12}$	$3\times 192\times 99$	$512\times 6\times 3$	$50\times 6\times 3$	$900$

Table 2. TABLE II: Comparision of the proposed methods

Component	SegFace	DeepSegFace
Proposal Generation	Clustering detections from cascade classifiers for facial segments	Clustering detections from cascade classifiers for facial segments
Low level features	HoG features	Deep CNN features
Intermediate stage	SVM for segment $i$ outputs a score on HoG features of segment $i$	Dimension reduction and concatenation to single 6400D vector
Final classifier	SVM trained on scores from part SVMs and priors	Fully connected layer, followed by a softmax layer
Using priors	Used as features in the final SVM	Used for re-ranking of face probabilities in post processing
Trade offs	Fast but less accurate	Slower but more accurate

Table 3. TABLE III: Comparison at 50 % percent 50 50\% overlap on AA-01-FD and UMDAA-02-FD datasets

Methods	AA-01		UMDAA-02
	TAR at	Recall at	TAR at	Recall at
	$1\%$ FAR	$99\%$ Prec.	$1\%$ FAR	$99\%$ Prec.
NPD [12]	$29.51$	$11.0$	$33.49$	$26.79$
DPMBaseline [16]	$85.08$	$83.25$	$78.48$	$72.79$
DeepPyramid [20]	$66.17$	$42.35$	$71.19$	$66.07$
HyperFace [21]	$90.52$	$90.32$	$73.01$	$71.14$
FSFD $C_{\text{best}}$ [13]	$59.06$	$55.65$	$55.74$	$26.88$
SegFace	$67.12$	$63.09$	$66.44$	$61.47$
DeepSegFace	$87.16$	$86.49$	$82.26$	$76.28$

Equations

$$\frac{|P\in {\Theta }^F|}{|{\Theta }^F|},$$

$$\frac{|P\in {\Theta }^{\overline{F}}|}{|{\Theta }^{\overline{F}}|},$$

$$\frac{|s_k\in S_p;S_p\in {\Theta }^F|}{|{\Theta }^F|},\text{ where, }k=1,2,\dots ,M.$$

$$\frac{|s_k\in S_p;S_p\in {\Theta }^{\overline{F}}|}{|{\Theta }^{\overline{F}}|},\text{ where, }k=1,2,\dots ,M.$$

Full text

I Introduction

In recent years, there has been substantial progress in the development of efficient and robust face detection techniques mostly because of the remarkable progress in convolutional neural network (CNN) architectures for face detection and the availability of large amount of face data [21][29][5]. Though the general trend of developing new face detectors is centered around detecting faces in unconstrained environments with large variations in pose and illumination [19][9][8][10], there is also a growing interest for developing face detectors optimized for detecting occluded and partially visible faces from images captured with mobile devices [13][25]. Reliable and fast detection of faces from the front-camera captures of a mobile device is a fundamental step for applications such as active/continuous authentication of the user of a mobile device [18][14][33][17].

Although, face-based authentication systems on mobile devices rely heavily on accurate detection of faces prior to verification, most state-of-the-art techniques are ineffective for mobile devices because of the following reasons:

Face images captured by the front camera of the phone are, in many cases, only partially visible [14]. Traditional face detectors such as Viola- Jones’s [30] and Deformable part model (DPM) [19] and more advanced face detectors based on deep convolutional neural networks (DCNN) such as Hyperface [21], yahoo multiview [5] and CUHK [29] are usually trained on full faces. While they work well on detecting multiple frontal or profile faces of various resolution, they frequently fail to detect partial faces. 2. 2.

For active authentication the recall rate needs to be high at very high precision. Many of the available face detectors have a low recall rate even though the precision is high. When operated at high recall, their precision drops rapidly because of excessive false positive detection. 3. 3.

The algorithm needs to be simple, fast and customizable in order to operate in real-time on a cellular device. While in [25] and [24] the authors deploy CNNs on mobile GPUs for face detection and verification, most CNN-based detectors, such as [21] and some generic methods like [19] are too complex to run on the mobile platform.

On the other hand, images captured for active authentication offer certain advantages for the face detection problem because of its semi-constrained nature [13]. Usually there is a single user in the frame, hence there is no need to handle multiple face detection. The face is in close proximity of the camera, within a certain range of dimensions and of high resolution, thus eliminating the need for detecting at multiple scales or resolution.

Partial faces such as ones present in images captured by the front camera of mobile devices can be handled if the algorithm is able to effectively combine detections of facial parts into a full face detection. Therefore to address this requirement, two algorithms SegFace and DeepSegFace are proposed in this paper, that detect faces from proposals made of face segments. Sample results for DeepSegFace are shown in Fig. 1. The proposal are generated using weak adaboost cascade classifiers following the method described in [13].

This paper makes the following contributions:

SegFace, a fast real-time face segment to face detector is proposed. 2. 2.

DeepSegFace, a novel deep CNN-based architecture, that detects faces from facial segments-based proposals is developed. 3. 3.

A principled scheme is developed that helps augment data as well as regularize the classifier.

In section II a summary of works done on face detection in general and in the mobile domain is given. In section III, the proposed face detection techniques are described in detail. A brief description of the two mobile-face datasets that are used for experimental validation are described in section IV . All the analysis, experimental results and comparisons for the proposed methods with state-of-the-art methods are provided in section V. Finally, a brief summary of this work as well as future directions of research are included in section VI.

II Related Works

Face detection is one of the earliest applications of computer vision dating back several decades [3][23]. However, most methods before 2004 performed poorly in unconstrained conditions, and therefore were not applicable in real-world settings [31]. Viola and Jones’s seminal work on boosted cascaded classification-based face detection [30] was the first algorithm that made face detection feasible in real-world applications and is still used widely in digital cameras, smartphones and photo organization software. The method, however, works reasonably well only for near-frontal faces under normal illumination without occlusion [32]. Extensions of the boosted architecture for multi-view face detection are found in literature, such as in [7][11], but these detectors are difficult to train, and do not perform well because of inaccuracies introduced by viewpoint estimation and quantization [32]. A more robust face detector is introduced in [19] that uses facial components or parts to construct a deformable part model (DPM) of a face. Similar geometrical modeling approaches are found in [1][15]. In [26], the authors introduced an examplar-based face detection method that does not require multi-scale shifting windows. As support vector machines (SVMs) became effective for classification and robust image features like SURF, local binary pattern (LFP) histogram of oriented gradient (HoG) and their variants were designed, researchers proposed different combinations of features with SVM for robust face detection [31]. Recently, in [16] the authors improved the performance of the DPM-based method and also introduced Headhunter, a new face detector that uses Integral Channel Features (ICF) with boosting to achieve state-of-the-art performance in face detection in the wild. A fast face detector that uses the scale invariant and bounded Normalized Pixel Difference (NPD) features is proposed in [12] that uses a single soft-cascade classifier to handle unconstrained face detection. The method is claimed to achieve state-of-the-arts performance on FDDB, GENKI, and CMU-MIT datasets.

The performance break-through observed after the introduction of Deep Convolutional Neural Networks (DCNN) can be attributed to the availability of large labeled datasets, availability of GPUs, the hierarchical nature of the deep networks and regularization techniques such as dropout space [31].

Continuous authentication of mobile devices requires partially visible face detection and verification to operate reliably [14]. In [13], the authors introduced a face detection method based on facial segments to detect partial faces on images captured for active authentication with smartphones. The idea was to produce face proposals by employing a number of weak Adaboost facial segment detectors on each image and clustering them. The authors proposed to form at most $\zeta$ subsets of facial segments from each cluster. Unique proposals were obtained by filtering out the redundant clusters that have exactly the same facial segments with the same bounding boxes. Statistical features from the unique proposals were then used to train a support vector machine classifier for face detection. The method worked well on AA-01-FD[33] and UMDAA-02-FD[14] mobile face detection datasets compared to other non-CNN methods [14]. [25] and [27] are two other methods that explicitly address the partial face detection problem. Specially in [27] the authors achieve state-of-the-arts performance on the FDDB, PASCAL and AFW datasets by generating face parts responses from an attribute-aware deep network and refining the face hypothesis for partial faces. Among other recent works, HyperFace [21] is a deep multi-task learning framework for face detection, landmark localization, pose Estimation, and gender recognition. The method exploits the synergy among related tasks by fusing the intermediate layers of a deep CNN using a separate CNN and thereby boosting their individual performances.

III Proposed Method

There are multiple paradigms of face detection for the mobile platform, one of which is to detect a face from facial segments. This is because of significant presence of partial faces in this domain as discussed earlier. Therefore, SegFace a simpler traditional feature-based scheme and DeepSegFace, a DCNN-based architecture, both of which detect faces from proposals composed of face segments, are proposed in this paper.

III-A Proposal generation

The set of facial segments is denoted by $S=\{a_{k}\mid k=1,2,\dots M\}$, where $M$ is the number of segments under consideration and $a_{k}$ is a particular facial segment. $M$ weak Adaboost facial segment detectors are trained to detect each of the segments in $S$. After running all the segment detectors on an image, the detected face segments which produce nearby estimated face center are grouped into clusters $\text{CL}_{j}$, $j=\{1,2,\dots c_{I}\}$ as discussed in [13]. Here, $c_{I}$ is the number of clusters formed for image $I$. A bounding box for the whole face $B_{\text{CL}_{j}}$ is calculated based on the constituent segments. For the generation of a proposal set, duplicate clusters that yield exactly same bounding boxes are eliminated and at most $\zeta$ face proposals are generated from each cluster by selecting random subsets of face segments constituting that cluster.

Therefore, each proposal $P$ is composed of a set of face segments $S_{P}$, where $S_{P}\in\mathcal{P}(S)-\{\varnothing\}$ and $\mathcal{P}$ denotes the power set. To get better proposals, one can impose extra requirements such as $|S_{P}|>c$, where $|\cdot|$ denotes cardinality and $c$ is a threshold. Each proposal is also associated with a bounding box for the whole face, which is the smallest bounding box that encapsulates all the segments in that proposal.

In our proposal generation scheme, $M=9$ is used. The nine parts under consideration are nose ($Nose$), eye-pair ($Eye$), upper-left three-fourth ($UL_{34}$), upper-right three-fourth ($UR_{34}$), upper-half ($U_{12}$), left three-fourth ($L_{34}$), upper-left-half ($UL_{12}$), right-half ($R_{12}$) and left-half ($L_{12}$. These nine parts, constituting the best combination $C_{best}$ [13] according to the analysis of effectiveness of each part in detecting faces, are considered in this experiment since the same adaboost classifiers are adapted in this work for proposal generation. The threshold $c$ is set to $2$. A small value is chosen to get high recall, at the cost of low precision. This lets one generate a large number of proposals, so that any face is not missed in this stage. $\zeta$ is set to $10$.

The general facial segment to face detector pipeline is shown schematically in Fig. 2. The figure depicts the integration of the proposal generation block into the pipeline. In the following sections, two instances of the detection model training block shown in the figure namely, SegFace and DeepSegFace, are discussed.

III-B SegFace

SegFace is a fast and shallow face detector built from segments proposal. For each segment in $s_{k}\in S$, a classifier $C_{s_{k}}$ is trained to accept features $f(s_{k})$ from the segment and generate a score denoting if a face is present. Output scores of $C_{s_{k}}$ are stored in an $M$ dimensional feature vector $F_{C}$, where, elements in $F_{C}$ corresponding to segments that are not present in a proposal are set to [math].

Another feature vector of size $2M+2$ is constructed using several prior probability values as features from the training proposal set that represents the likeliness of certain segments and certain combinations. These values are

• •

Fraction of total true faces constituted by proposal $P$, i.e.

[TABLE]

where $\Theta^{F}$ is the set of all proposals that return a true face.

• •

The fraction of total mistakes constituted by proposal $P$, i.e.

[TABLE]

, where, $\Theta^{\overline{F}}$ is the set of all proposals that are not faces.

• •

For each of the $M$ facial segment $s_{k}\in S$, the fraction of total true face proposals of which $s_{k}$ is a part of, i.e.

[TABLE]

• •

For each of the $M$ facial segment $s_{k}\in S$, the fraction of total false face proposals of which $s_{k}$ is a part of, i.e.

[TABLE]

$F_{C}$ and $F_{S}$ are appended together to form the full feature vector $F$ of length $3M+2$. Then a master classifier $C$ is trained on the training set of such labeled vectors $\{F_{i},Y_{i}\}$, where $Y_{i}$ denotes the label (face or no-face). Thus, $C$ learns how to assign relative importance to different segments and likeliness of certain combinations of segments occurring in deciding if a face is present in a proposal. Thus, SegFace extends the face detection from segments concept in [13] using traditional methods to obtain reasonably accurate results. In our implementation of SegFace, HoG [4] features are used as $f$ and Support Vector Machine (SVM) classifiers [2] are used as both $C_{s_{k}}$ and $C$ for generating segment-wise scores, as well as the final detection score, respectively.

III-C DeepSegFace: CNN Architecture for detecting faces from face segments

DeepSegFace is an architecture to integrate deep CNNs and segments-based face detection. Thus, it allows for end-to-end training and for exploiting the superior capabilities of deep CNNs for segment-based detection. At first, proposals, consisting of subsets of the $M=9$ parts as discussed earlier, are generated for each image. DeepSegFace is then trained to calculate the probability values of the proposal being a face. Finally, a re-ranking step adjusts the probability values from the network. The proposal with the maximum re-ranked score is deemed as the detection for that image.

The architecture of DeepSegFace is arranged according to the classic paradigm in pattern recognition: feature extraction, dimensionality reduction followed by a classifier. Training occurs end-to-end (proposal to face probability). A simple block diagram of the architecture is shown in Fig. 3. Different components of the figure are discussed here.

Convolutional Feature Extraction: There are nine convolutional networks for each of the nine segments. Each of the nine columns is structurally similar to and initialized with the convolution layers of VGG16 network [28]. Thus each network has thirteen convolution layers arranged in five blocks. Each segment in the proposal is resized to standard dimensions for that segment, then the VGG mean value is subtracted from each channel. For segments not present in the proposal, zero-input is fed into the networks corresponding to those segments, as shown for the $Nose$ segment in the figure.

Dimensionality reduction: Since the last convolutional layer of each network outputs $512$ feature maps, in total the number of features is quite large. Therefore, a randomly initialized convolutional layer with filter size $1\times 1$ and fifty feature maps is added to learn an appropriate dimension reduction.

Classifier: The outputs from the dimensionality reduction block for each segment-network are flattened and concatenated side by side to yield a $6400$ dimensional feature vector. A fully connected (fc) layer of $250$ nodes, followed by a softmax layer of two nodes (both randomly initialized) is added on top of the feature vector. The two outputs of the softmax layer corresponds to the probability of being a face or not being a face and hence they sum to one.

Re-ranking: The DeepSegFace network outputs the face detection probabilities for each proposal in an image, which can be used to rank the proposals and then declare the highest probability proposal as the face in that image. However there is some prior knowledge that some segments are more effective at detecting the presence of faces than others. This information is available from the prior probability values discussed in section III-B. In the case of SegFace, the statistical features are incorporated in the feature vector of the master SVM. Since these feature are similar to fixed priors for the segments and their differnt combinations, for DeepSegFace, these values are used to re-rank the final score by multiplying it with the mean of the statistical features.

More details on the dimensions of DeepSegFace architecture at different stages are presented in Table I. Key insights about the structure of the network are provided in the next subsection.

III-D Interpretations

III-D1 SegFace vs. DeepSegFace

One can think of SegFace and DeepSegFace belonging to the same school of face detection algorithms, namely, detecting faces by pooling detections of facial segments. Both have broad structural similarities as discussed in table II. However SegFace focuses on resource constrained execution (no GPU) while DeepSegFace targets performance.

Currently both of their input proposals come from the same proposal generation scheme described in section III-A. Hence, both strategies will benefit from improved proposal generation algorithms. Here a very fast proposal generation scheme is chosen over more sophisticated ones mostly for processing speed. However if processing power is not a bottleneck, one can customize more advanced proposal generation schemes such as Faster R-CNN [22] for this purpose.

III-D2 Facial Segment Drop-out for Better Generalization

As mentioned in the proposal generation scheme, subsets of face segments in a cluster are used to generate new proposals. For example, if a cluster of face segments contains $n$ segments and each proposal must contain atleast $c$ segments, then it is possible to generate $\sum\limits_{k=t}^{n}{{n}\choose{k}}$ proposals. Now, if all the facial segments are present, the network’s task is easier. However, all the nine parts are redundant for detecting a face, since there are significant overlaps among the parts. Also, in a given face, often many segments are not detected by the weak segment detectors. Thus, one can interpret the missing segments as ‘dropped-out’, i.e. some of the input signals are randomly missing (they are set to zero). Thus the network must be robust to face segments ‘dropping out’ and generalize better to be able to identify faces.

III-E Data augmentation

Training with subsets of detected proposals also has the additional effect of augmenting the data. It has been observed that around sixteen proposals are generated per image. Many of these proposals are actually training the network to detect the same face using different combination of segments. This is a more principled data augmentation technique compared to other commonly used methods like adding additional noise, in which case it is not explicitly clear how the augmentation is helping the network learn better.

IV Dataset

Given the sensitive nature of smartphone usage data, there has been a scarcity of large dataset of front camera images in natural settings. However, the following two datasets have been published in recent years that provided a platform to evaluate partial face detection methods in real-life scenarios.

IV-A Active Authentication Dataset-01 (AA-01)

The AA-01 dataset [33] is a challenging dataset for front-camera face detection task which contains the front-facing camera face video for forty three male and seven female IPhone users under three different ambient lighting conditions: well-lit, dimly-lit, and natural daylight. In each session, the users performed five different tasks: enrollment, scrolling, picture count, document reading and picture dragging. To evaluate the face detector, face bounding boxes were annotated in a total of $8036$ frames of the fifty users. This dataset, denoted as AA-01-FD, contains $1607$ frames without faces and $6429$ frames with faces [13], [25]. The images in this are semi-constrained as the subjects perform a set task during the data collection period. However they are not required or encouraged to maintain a certain posture, hence the dataset is sufficiently challenging due to pose variations, occlusions and partial faces.

IV-B University of Maryland Active Authentication Dataset-02 (UMDAA-02)

The UMDAA-02 contains usage data of more than fifteen smartphone sensors obtained in a natural settings for an average of ten days per user [14]. The UMDAA-02 Face Detection Dataset (UMDAA-02-FD) dataset contains a total of $33,209$ images, manually annotated for face bounding box, from all sessions of the $44$ users ($33$ male, $10$ female) of UMDAA-02 sampled at an interval of $7$ seconds. This dataset is truly unconstrained as data is collected during real-time phone usage over a period of one week. The face images have wide variation of pose and illumination, and it is observed that the faces are mostly large in size and a good proportion of the face images are only partially visible.

V Experimental Results

V-A Experimental Setup

The experimental results, presented in this section, demonstrate the effectiveness of the proposed methods over other state-of-the-art face detection algorithms. In particular, experimental results on the AA-01-FD and UMDAA-02-FD datasets are compared with a) Normalized Pixel Difference (NPD)-based detector [12], b) Hyperface detector [21], c) Deep Pyramid Deformable Part Model detector [20], d) DPM baseline detector [16], and e) Facial Segment-based Face Detector (FSFD) [13]. Both SegFace and DeepSegFace are trained on $3964$ images from AA-01-FD and trained models are validated using $1238$ images. The data augmentation process produces $57,756$ proposals from the training set, that is around $14.5$ proposals per image. The remaining $2835$ images of AA-01-FD are used for testing. For UMDAA-02-FD, $32,642$ images are used for testing. In all experiments with SegFace and DeepSegFace, $c=2$ and $\zeta=10$ is considered.

The results are evaluated by comparing the ROC curve and precision-recall curves of these detectors since all of them return a confidence score for detection. The goal is to achieve high True Acceptance Rate (TAR) at a very low False Acceptance Rate (FAR) and also a high recall at a very high precision. Hence, numerically, the value of TAR at $1\%$ FPR and recall achieved by a detector at $99\%$ precision are the two metrics that are used to compare different methods.

In table III, the performance of SegFace and DeepSegFace are compared with state-of-the-arts methods for both datasets. From the measures on the AA-01-FD dataset, it can be seen that SegFace, in spite of being a traditional feature based algorithm, outperforms several algorithms like FSFD and even DCNN based algorithms such as NPD and DeepPyramid. On the other hand, DeepSegFace outperforms all the other methods except HyperFace in terms of the two evaluation measures on the AA-01-FD dataset. Hyperface is a state-of-the-art algorithm that is trained on over $20,000$ images, compared to only $5202$ images used to train DeepSegFace. Also Hyperface uses R-CNN to generate face proposals, compared to the fast weak classifiers used by DeepSegFace. Furthur analysis reveals that one of the bottlenecks of DeepSegFace’s performance is the proposal generation phase, thus its performance can increase if it uses a more powerful proposal generation scheme, such as R-CNN[6].

In Fig. 4, some images are shown for which the proposal generator did not return any proposals or returned proposals without sufficient overlap, even though there are somewhat good, visible faces or facial segments in them. The percentage of true faces that are represented by at least one proposal in the list of proposals for the training and test sets are counted. The result of this analysis is shown in Fig. 5. The bar graphs denote the percentage of positive samples and negative samples present in the proposal list generated for a certain overlap ratio. For example, out of the $55,756$ proposals generated for training, there are approximately $62\%$ positive samples and $35\%$ negative samples at an overlap ratio of $50\%$. Considering the overlap ratio fixed to $50\%$ for this experiment, it can be seen from the line plot in Fig. 5(b), corresponding to the AA-01-FD test set, that the proposal generator actually represent $89.18\%$ of the true faces successfully and fails to generate a single good proposal for the rest of the images. Hence, the performance of the proposed detectors are upper-bounded by this number on this dataset, a constraint that can be mitigated by using advanced proposal schemes like R-CNN which generates around $2000$ proposals per image for Hyperface, compared to just around sixteen proposals that are generated by the fast proposal generator employed by DeepSegFace.

However, when considering the UMDAA-02-FD test set, which is completely unconstrained and has almost ten times more images than AA-01-FD test set, this upper bound might not be so bad. From Fig. 5(c) it can be seen that the upper bound for UMDAA-02-FD is $87.57\%$ true positive value. Now, in Fig. 6, the ROC for this dataset is shown. It can be seen that the DeepSegFace method outperforms all the other methods, including HyperFace, with a large margin even with the upper bound (the curve flattens around $87.5\%$). This is because all the traditional methods suffer so much more when detecting mobile faces in truly unconstrained settings that a true acceptance rate of even $87\%$ is hard to achieve. It is to be noted that the data collection process for AA-01-FD was task-based [33] and hence, supervised, while UMDAA-02-FD was collected in a completely natural setting [14]. The precision-recall curve for UMDAA-02-FD dataset is shown in Fig. 7. It can be seen that the DeepSegFace method has much better recall at $99\%$ precision than any other method. In both figures, the performance of SegFace is found satisfactory given its dependency on traditional features. In fact, the curves are not too far off from the DeepPyramid method, which is DCNN-based.

The proposals for both test sets are analyzed to reveal that on an average only three segments per proposal are present for both datasets. Thus, while there are nine convolutional networks in the architecture, only three of them need to fire on an average for generating scores from the proposals. When forwarding proposals in batch sizes of $256$, DeepSegFace takes around $0.02$ seconds per proposal on a GTX Titan-X GPU. SegFace takes around $0.49$ seconds when running on a Intel Xeon CPU E-2623 v4 ($2.604$ GHz) machine with $32$GB Memory without multi-threading, hence it is possible to optimize it to run on mobile devices in reasonable time without specialized hardware.

VI Conclusion and Future Directions

This paper proposes two schemes, DeepFaceSeg and FaceSeg, for detecting faces captured by front cameras of smartphones, primarily for the purpose of active authentication. By detecting faces from facial segments, the algorithms are well equipped to handle partial faces which are prevalent in the mobile face domain. Also a principled data augmentation for this class of algorithms is proposed that makes the network generalize well. DeepFaceSeg, a DCNN architecture, performs very well on two mobile face datasets, while FaceSeg, which is designed for speed, works reasonably well, but is simple and fast enough to be implemented on mobile platforms for real time operations.

DeepSegFace and FaceSeg are but two instantiations of the general concept of detecting faces from proposals with facial segment. The general idea can be extended by using improved segment proposals or better classifiers. These detected faces comes with several facial segments, which opens up the research opportunity to investigate facial segment-based fiducial estimation, attribute detection and face verification. Future research direction may also include customizing and optimizing both algorithms to implement them in mobile-platforms for real-life applications.

Acknowledgement

This work was done in partnership with and supported by Google Advanced Technology and Projects (ATAP), a Skunk Works-inspired team chartered to deliver breakthrough innovations with end-to-end product development based on cutting edge research and a cooperative agreement FA8750-13-2-0279 from DARPA.