Few-Shot Adaptive Gaze Estimation

TL;DR

This paper introduces FAZE, a novel few-shot learning framework for personalized gaze estimation that achieves state-of-the-art accuracy with minimal calibration samples by combining a disentangling encoder-decoder architecture and meta-learning.

Contribution

It presents a new framework that effectively personalizes gaze estimation networks with fewer than 10 calibration samples, outperforming previous methods.

Findings

Achieves 3.18° error on GazeCapture dataset

Improves gaze estimation accuracy by 19% over prior art

Capable of adapting with as few as 3 calibration samples

Abstract

Inter-personal anatomical differences limit the accuracy of person-independent gaze estimation networks. Yet there is a need to lower gaze errors further to enable applications requiring higher quality. Further gains can be achieved by personalizing gaze networks, ideally with few calibration samples. However, over-parameterized neural networks are not amenable to learning from few examples as they can quickly over-fit. We embrace these challenges and propose a novel framework for Few-shot Adaptive GaZE Estimation (FAZE) for learning person-specific gaze networks with very few (less than or equal to 9) calibration samples. FAZE learns a rotation-aware latent representation of gaze via a disentangling encoder-decoder architecture along with a highly adaptable gaze estimator trained using meta-learning. It is capable of adapting to any new person to yield significant performance gains with as few as 3 samples, yielding state-of-the-art performance of 3.18 degrees on GazeCapture, a 19% improvement over prior art. We open-source our code at https://github.com/NVlabs/few_shot_gaze

Equations

$${\mathbf{R}}^{\left(\theta ,\varphi \right)}=\left[\begin{array}{ccc}\cos \varphi & 0& \sin \varphi \\ 0& 1& 0\\ -\sin \varphi & 0& \cos \varphi \end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \theta & -\sin \theta \\ 0& \sin \theta & \cos \theta \end{array}\right].$$

$${\mathcal{L}}_{\mathrm{full}}={\lambda }_{\mathrm{recon}}{\mathcal{L}}_{\mathrm{recon}}+{\lambda }_{\mathrm{EC}}{\mathcal{L}}_{\mathrm{EC}}+{\lambda }_{\mathrm{gaze}}{\mathcal{L}}_{\mathrm{gaze}},$$

$${\mathcal{L}}_{\mathrm{recon}}\left({\mathbf{x}}_b,{\hat{\mathbf{x}}}_b\right)=\frac{1}{|{\mathbf{x}}_b|}\sum _{u\in {\mathbf{x}}_b,\hat{u}\in {\hat{\mathbf{x}}}_b}\left|\hat{u}-u\right|,$$

$${\mathcal{L}}_{\mathrm{EC}}=\frac{1}{B}\sum _{i=1}^B\underset{\begin{array}{c}j=1\dots B\\ id(i)=id(j)\end{array}}{\max }d\left(f({\mathbf{z}}_i^g),f({\mathbf{z}}_j^g)\right),$$

$${\mathcal{L}}_{\mathrm{gaze}}\left(\hat{\mathbf{g}},\mathbf{g}\right)=\arccos \left(\frac{\hat{\mathbf{g}}\cdot \mathbf{g}}{\Vert \hat{\mathbf{g}}\Vert \Vert \mathbf{g}\Vert }\right).$$

$${\theta }_n^{\prime }=f({\theta }_n)={\theta }_n-\alpha \nabla {\mathcal{L}}_{{\mathcal{P}}^{train}}^c({\theta }_n).$$

$${\theta }_{n+1}={\theta }_n-\eta \nabla {\mathcal{L}}_{{\mathcal{P}}^{train}}^v(f({\theta }_n)).$$

$${\theta }_{{\mathcal{P}}^{test}}={\theta }^{\ast }-\alpha \nabla {\mathcal{L}}_{{\mathcal{P}}^{test}}^c({\theta }^{\ast }),$$

Full text

00footnotetext: The first two authors contributed equally.

1 Introduction

Estimation of human gaze has numerous applications in human-computer interaction [8], virtual reality [34], automotive [47] and content creation [52], among others. Many of these applications require high levels of accuracy (cf. [3, 43, 19, 2]). While appearance-based gaze estimation techniques that use Convolutional Neural Networks (CNN) have significantly surpassed classical methods [57] for in-the-wild settings, there still remains a significant gap towards applicability in high-accuracy domains. The currently lowest reported person-independent error of $4.3^{\circ}$ [7] is equivalent to 4.7cm at a distance of 60cm, which restricts use of such techniques to public display interactions [60] or crowd-sourced attention analysis [30].

High-accuracy gaze estimation from images is difficult because it requires either explicit or implicit fitting of a person-specific eye-ball model to the image data and the estimation of their visual and optical axes. Moreover, it is well understood that inter-subject anatomical differences affect gaze estimation accuracy [11]. Classical model-based techniques can often be personalized via few (9 or less) samples (e.g., [11, 13]) but are not robust to image variations in uncontrolled settings. While feasible, subject-specific training of CNNs requires thousands of samples and is clearly impractical [59]. Few-shot personalization of CNNs is difficult because training of highly overparametrized models with only few training samples will lead to over-fitting.

We tackle these many-fold challenges by proposing Faze, a framework for learning gaze estimation networks for new subjects using very few calibration samples (Fig. 1). It consists of:

i) learning a rotation-aware latent representation of gaze via a disentangling transforming encoder-decoder architecture

ii) with these features learning a highly adaptable gaze estimator using meta-learning, and

iii) adapting it to any new person to yield significant performance gains with as few as $3$ samples.

In order to learn a robust representation for gaze, we take inspiration from recent work on transforming encoder-decoder architectures [15, 53] and design a rotation-equivariant pair of encoder-decoder functions. We disentangle the factors of appearance, gaze and head pose in the latent space and enforce equivariance by decoding explicitly rotated latent codes to images of the same person but with a different gaze direction compared to the input (via a $\ell_{1}$ reconstruction loss). The equivariance property of our gaze representation further allows us to devise a novel embedding consistency loss term that further minimizes the intra-person differences in the gaze representation. We then leverage the proposed latent embedding to learn person-specific gaze estimators from few samples. To this end we use a meta-learning algorithm to learn how to learn such estimators. We take inspiration from the recent success of meta-learning [1] for few-shot learning in several other vision tasks [6, 12, 31]. To the best of our knowledge, we are the first to cast few-shot learning of person-specific gaze estimators as one of multi-task learning where each subject is seen as a new task in the meta-learning sense.

We evaluate the proposed framework on two benchmark datasets and show that our meta-learned network with its latent gaze features can be successfully adapted with very few ($k\leq 9$) samples to produce accurate person-specific models. We demonstrate the validity of our design choices with detailed empirical evidence, and demonstrate that our proposed framework outperforms state-of-the-art methods by significant margins. In particular, we demonstrate improvements of $13\%$ ($3.94^{\circ}\rightarrow 3.42^{\circ}$) on the MPIIGaze dataset, and $19\%$ ($3.91^{\circ}\rightarrow 3.18^{\circ}$) on the GazeCapture dataset over the approach of [26] using just $3$ calibration samples.

To summarize, the main contributions of our work are:

• •

Faze, a novel framework for learning person-specific gaze networks with few calibration samples, fusing the benefits of rotation-equivariant feature learning and meta-learning.

• •

A novel encoder-decoder architecture that disentangles gaze direction, head pose and appearance factors.

• •

A novel and effective application of meta-learning to the task of few-shot personalization.

• •

State-of-the-art performance ($3.14^{\circ}$ with $k=9$ on MPIIGaze), with consistent improvements over existing methods for $1\leq k\leq 256$.

2 Related Work

Gaze Estimation. Appearance-based gaze estimation [46] methods that map images directly to gaze have recently surpassed classical model-based approaches [13] for in-the-wild settings. Earlier approaches in this direction assume images captured in restricted laboratory settings and use direct regression methods [28, 27] or learning-by-synthesis approaches combined with random forests to separate head-pose clusters [45]. More recently, the availability of large scale datasets such as MPIIGaze [57] and GazeCapture [22], and progress in CNNs have rapidly moved the field forward. MPIIGaze has become a benchmark dataset for in-the-wild gaze estimation. For the competitive person-independent within-MPIIGaze leave-one-person-out evaluation, gaze errors have progressively decreased from $6.3^{\circ}$ for naively applying a LeNet-5 architecture to eye-input [57] to the current best of $4.3^{\circ}$ with an ensemble of multi-modal networks based on VGG-16 [7]. Proposed advancements include the use of more complex CNNs [59]; more meaningful use of face [58, 22] and multi-modal input [22, 7, 54]; explicit handling of differences in the two eyes [4]; greater robustness to head pose [61, 36]; improvements in data normalization [55]; learning more informed intermediate representations [32]; using ensembles of networks [7]; and using synthetic data [42, 51, 24, 33, 36].

However, person-independent gaze errors are still insufficient for many applications [3, 43, 19, 2]. While significant gains can be obtained by training person-specific models, it requires many thousands of training images per subject [59]. On the other hand, CNNs are prone to over-fitting if trained with very few ($k\leq 9$) samples. In order to address this issue, existing approaches try to adapt person-independent CNN-based features [22, 33] or points-of-regard (PoR) [56] to person-specific ones via hand-designed heuristic functions. Some methods also train a Siamese network with pairs of images of the same subject [26].

Learned Equivariance. Generalizing models learned for regression tasks to new data is a challenging problem. However, recent works show improvements from enforcing the learning of equivariant mappings between input, latent features, and label spaces [16, 39], via so-called transforming encoder-decoder architectures [15]. In [53], this idea is expanded to learn complex phenomena such as the orientation of synthetic light sources and in [39] the method is applied to real-world multi-view imagery to improve semi-supervised human pose estimation. In contrast, we learn from very noisy real-world data while successfully disentangling the two noisily-labeled phenomena of gaze direction and head orientation.

Few-shot Learning. Few-shot learning aims to learn a new task with very few examples [23]. This is a non-trivial problem for highly over-parameterized deep networks as it leads to over-fitting. Recently, several promising meta-learning [44, 50, 38, 40, 6, 29, 37] techniques have been proposed that learn how to learn unique but similar tasks in a few-shot manner using CNNs. They have been shown to be successful for various few-shot visual learning tasks including object recognition [6], segmentation [35], viewpoint estimation [48] and online adaptation of trackers [31]. Inspired by their success, we use meta-learning to learn how to learn person-specific gaze networks from few examples. To the best of our knowledge we are the first to cast person-specific gaze estimation as a multi-task problem in the context of meta-learning, where each subject is seen as a new task for the meta-learner. Our insight is that meta-learning lends itself well to few-shot gaze personalization and leads to performance improvements.

3 Method

In this section, we describe how we perform gaze estimation from challenging in-the-wild imagery, with minimal burden to the user. The latter objective can be fulfilled by designing our framework to adapt well even with very few calibration samples ($k\leq 9$). We first provide an overview of the Faze framework and its three stages.

3.1 The Faze framework

We design Faze (Fig. 1) with the understanding that a person-specific gaze estimator must encode factors particular to the person, yet at the same time, leverage insights from observing the eye-region appearance variations across a large number of people with different head pose and gaze direction configurations. The latter is important for building models that are robust to extraneous factors such as poor image quality. Thus, the first step in Faze is to learn a generalizable latent embedding space that encodes information pertaining to the gaze-direction, including person-specific aspects. We detail this in Sec. 3.2.

Provided that good and consistent features can be learned, we can leverage meta-learning to learn how to learn few-shot person-specific gaze estimators for these features. This results in few-shot learners which generalize better to new people (tasks) without overfitting. Specifically, we use the MAML meta-learning algorithm [6]. For our task, MAML learns a set of initial network weights which allow for fine-tuning without the usual over-fitting issues that occur with low $k$. Effectively, it produces a highly Adaptable Gaze Estimation Network (AdaGEN). The final step concerns the adaptation of MAML-initialized weights to produce person-specific models (PS-GEN) for each user. We describe this in Sec. 3.3.

3.2 Gaze-Equivariant Feature Learning

In this section, we explain how the learning of a function, which understands equivalent rotations in input data and output label can lead to better generalization in the context of our final task of person-specific gaze estimation. In addition, we:

(a) show how to disentangle eyeball and head rotation factors leading to better distillation of gaze information, and

(b) introduce a frontalized embedding consistency loss term to specifically aid in learning consistent frontal gaze codes for a particular subject.

3.2.1 Architecture Overview

In learning a generalizable latent embedding space representing gaze, we apply the understanding that a relative change in gaze direction is easier to learn in a person-independent manner [26]. We extend the transforming encoder-decoder architecture [15, 53] to consider three distinct factors apparent in our problem setting: gaze direction, head orientation, and other factors related to the appearance of the eye region in given images (Fig. 2). We disentangle the three factors by explicitly applying separate and known differences in rotations (eye gaze and head orientation) to the respective sub-codes. We refer to this architecture as the Disentangling Transforming Encoder-Decoder (DT-ED).

For a given input image $\mathbf{x}$ , we define an encoder $\mathcal{E}:\mathbf{x}\rightarrow\mathbf{z}$ and a decoder $\mathcal{D}:\mathbf{z}\rightarrow\hat{\mathbf{x}}$ such that $\mathcal{D}\left(\mathcal{E}(\mathbf{x})\right)=\hat{\mathbf{x}}$. We consider the latent space embedding $\mathbf{z}$ as being formed of 3 parts representing: appearance ($\mathbf{z}^{a}$), gaze direction or eyeball rotation ($\mathbf{z}^{g}$), and head pose ($\mathbf{z}^{h}$), which can be expressed as: $\mathbf{z}=\left\{\mathbf{z}^{a};\,\mathbf{z}^{g};\,\mathbf{z}^{h}\right\}$ where gaze and head codes are flattened to yield a single column. We define $\mathbf{z}^{g}$ as having dimensions $\left(3\times F^{g}\right)$ and $\mathbf{z}^{h}$ as having dimensions $\left(3\times F^{h}\right)$ with $F\in\mathbb{N}$. With these dimensions, it is possible to apply a rotation matrix to explicitly rotate these $3D$ latent space embeddings using rotation matrices.

The frontal orientation of eyes and heads in our setting can be represented as $\left(0,\,0\right)$ in Euler angles notation for azimuth and elevation, respectively assuming no roll, and using the $x-y$ convention. Then, the rotation of the eyes and the head from the frontal orientation can be described using $\left(\theta^{g},\phi^{g}\right)$ and $\left(\theta^{h},\phi^{h}\right)$ in Euler angles and converted to rotation matrices defined as,

[TABLE]

While training DT-ED, we input a pair of images of a person $\mathbf{x}_{a}$ and $\mathbf{x}_{b}$. We can calculate $\mathbf{R}^{g}_{ba}=\mathbf{R}^{g}_{b}\left(\mathbf{R}^{g}_{a}\right)^{-1}$ to describe the change in gaze direction in going from sample $a$ to sample $b$ of the same person. Likewise for head rotation, $\mathbf{R}^{h}_{ba}=\mathbf{R}^{h}_{b}\left(\mathbf{R}^{h}_{a}\right)^{-1}$. This can be done using the ground-truth labels for gaze ($\mathbf{g}_{a}$ and $\mathbf{g}_{b}$) and head pose ($\mathbf{h}_{a}$ and $\mathbf{h}_{b}$) for the pair of input samples. The rotation of the latent code $\mathbf{z}^{g}_{a}$ can then be expressed via the operation $\hat{\mathbf{z}}^{g}_{b}=\mathbf{R}_{ab}^{g}\mathbf{z}^{g}_{a}$. At training time, we enforce this code to be equivalent to the one extracted from image $\mathbf{x}_{b}$, via a reconstruction loss (Eq. 3). We assume the rotated codes $\hat{\mathbf{z}}^{h}_{b}$ and $\hat{\mathbf{z}}^{g}_{b}$, along with the appearance-code $\mathbf{z}^{a}_{a}$, to be sufficient for reconstructing $\mathbf{x}_{b}$ through the decoder function such that, $\mathcal{D}\left(\hat{\mathbf{z}}_{b}\right)=\mathbf{x}_{b}$. More specifically, given the encoder output $\mathcal{E}\left(\mathbf{x}_{a}\right)=\mathbf{z}_{a}=\left\{\mathbf{z}^{a}_{a};\,\mathbf{z}^{g}_{a};\,\mathbf{z}^{h}_{a}\right\}$, we assume the rotated version of $\mathbf{x}_{a}$ to match the embedding of $\mathbf{x}_{b}$, that is we assume $\left\{\hat{\mathbf{z}}^{a}_{b};\,\hat{\mathbf{z}}^{g}_{b};\,\hat{\mathbf{z}}^{h}_{b}\right\}=\left\{\mathbf{z}^{a}_{a};\,\mathbf{R}^{g}_{ba}\mathbf{z}^{g}_{a};\,\mathbf{R}^{h}_{ba}\mathbf{z}^{h}_{a}\right\}$ (See Fig. 2).

This approach indeed applies successfully to noisy real-world imagery, as shown in Fig. 3 where we map a sample into the gaze and head pose latent spaces, rotate to the frontal orientation, and then again rotate by a pre-defined set of 15 yaw and pitch values and reconstruct the image via the decoder. We can see that the factors of gaze direction and head pose are fully disentangled and DT-ED succeeds in the challenging task of eye-region frontalization and re-direction from monocular RGB input.

We train the Faze transforming encoder-decoder architecture using a multi-objective loss function defined as,

[TABLE]

where we empirically set $\lambda_{\mathrm{recon}}=1$, $\lambda_{\mathrm{EC}}=2$, and $\lambda_{\mathrm{gaze}}=0.1$. The individual loss terms are explained in the following sub-sections.

3.2.2 Reconstruction Loss

To guide learning of the encoding-decoding process, we apply a simple $\ell_{1}$ reconstruction loss. Given an input image $\mathbf{x}_{b}$ and reconstructed $\hat{\mathbf{x}}_{b}$ obtained by decoding the rotated embeddings $\hat{\mathbf{z}}_{b}$ of image $\mathbf{x}_{a}$, the loss term is defined as,

[TABLE]

where $u$ and $\hat{u}$ are pixels of images $\mathbf{x}_{b}$ and $\hat{\mathbf{x}}_{b}$ respectively.

3.2.3 Embedding consistency Loss

We introduce a novel embedding consistency term, which ensures that the encoder network always embeds images of a person with different appearance but identical gaze direction to similar features. Usually this would require paired images with only gaze directions changed. However, it is intractable to collect such data in the real world, so we instead exploit the learned equivariance of DT-ED. Before measuring the consistency between latent gaze features from different samples, we first frontalize them by applying the inverse of the rotation matrix $\mathbf{R}^{g}_{a}$ using ground-truth gaze direction $\mathbf{g}_{a}$. It should be noted that naively enforcing all gaze features to be similar across persons may disregard the inter-subject anatomical differences which should result in different latent embeddings. Hence, we apply the embedding consistency to intra-subject pairs of images only. We validate this choice through experiments in Sec. B.1.

Given a batch of $B$ image samples during training, we formally compute the embedding consistency loss using,

[TABLE]

where $f(\mathbf{z}^{g})=\left(\mathbf{R}^{g}\right)^{-1}\mathbf{z}^{g}$ corresponds to frontalized latent gaze features. The function $id(i)$ provides the person-identity of the $i$-th sample in the batch, and $d$ is a function based on mean column-wise angular distance (between 3D vectors). The $\max$ function minimizes differences between intra-person features that are furthest apart, and is similar to the idea of batch-hard online triplet mining [41].

During training, we linearly increase $\lambda_{\mathrm{EC}}$ from [math] until sufficient mini-batches to cover $10^{6}$ images have been processed, to allow for more natural embeddings to arise before applying consistency.

3.2.4 Gaze Direction Loss

Lastly, we add the additional objective of gaze estimation via $\mathcal{G}:\mathbf{z}^{g}\rightarrow\hat{\mathbf{g}}$, parameterized by a simple multi-layer perceptron (MLP). The gaze direction loss is calculated using,

[TABLE]

3.3 Person-specific Gaze Estimation

Having learned a robust feature extractor, which is tailored specifically for gaze estimation, our final goal is to learn a personalized gaze estimator with as few calibration samples as possible. A straightforward solution for doing this is to train a person-independent model with the training data used to train DT-ED and simply fine-tune it with the available calibration samples for the given subject. However, in practical setups where only a few calibration samples are available, this approach can quickly lead to over-fitting (see experiments in Fig. 7(a)). In order to alleviate this problem, we propose to use the meta-learning method MAML [6], which learns a highly adaptable gaze network (AdaGEN).

Adaptable Gaze Estimator (AdaGEN) Training.

We wish to learn weights $\theta^{*}$ for the AdaGEN gaze prediction model $\mathcal{M}$ such that it becomes a successful few-shot learner. In other words, when $\mathcal{M}_{\theta^{*}}$ is fine-tuned with only a few “calibration” examples of a new person $\mathcal{P}$ not present in the training set, it can generalize well to unseen “validation” examples of the same person. We achieve this by training it with the MAML meta learning algorithm adapted for few-shot learning.

In conventional CNN training the objective is to minimize the training loss for all the examples of all training subjects. In contrast, for few-shot learning, MAML explicitly minimizes the generalization loss of a network after minimizing its training loss for a few examples of a particular subject via a standard optimization algorithm, e.g., stochastic gradient descent (SGD). Additionally, MAML repeats this procedure for all subjects in the training set and hence learns from several closely related “tasks” (subjects) to become a successful few shot learner for any new unseen task (subject). We identify that person-specific factors may have few parameters, with only slight but important variations across people such that all people constitute a set of closely related tasks. Our insight is that meta-learning lends itself well to such a problem of personalization.

The overall procedure of meta-learning to learn the optimal $\theta^{*}$ is as follows. We divide the entire set of persons $\mathcal{S}$ into meta-training ($\mathcal{S}^{train}$) and meta-testing ($\mathcal{S}^{test}$) subsets of non-overlapping subjects. During each meta-training iteration $n$, we randomly select one person $\mathcal{P}^{train}$ from $\mathcal{S}^{train}$ and create a meta-training sample for the person (via random sampling), defined as $\mathcal{P}^{train}=\{\mathcal{D}_{c}^{train},\mathcal{D}_{v}^{train}\}$, containing a calibration set $\mathcal{D}_{c}^{train}=\{(\mathbf{z^{g}}_{i},\mathbf{g}_{i})|i=1\ldots k\}$ of $k$ training examples, and a validation set $\mathcal{D}_{v}^{train}=\{(\mathbf{z^{g}}_{j},\mathbf{g}_{j})|j=1\ldots l\}$ of another $l$ examples for the same person. Here, $\mathbf{z^{g}}$ and $\mathbf{g}$ refer to the latent gaze representation learned by DT-ED and the ground-truth 3D gaze vector, respectively. Both $k$ and $l$ are typically small ($\leq 20$) and $k$ represents the “shot” size used in few-shot learning.

The first step in the meta-learning procedure is to compute the loss for the few-shot calibration set $\mathcal{D}_{c}^{train}$ and update the weights $\theta_{n}$ at step $n$ via one (or more) gradient steps and a learning rate $\alpha$ as,

[TABLE]

With the updated weights $\theta^{\prime}_{n}$, we then compute the loss for the validation set $\mathcal{D}_{v}^{train}$ of the subject $\mathcal{P}^{train}$ as $\mathcal{L}_{\mathcal{P}^{train}}^{v}(\theta^{\prime}_{n})=\mathcal{L}_{\mathcal{P}^{train}}^{v}(f(\theta_{n}))$ and its gradients w.r.t the initial weights of the network $\theta_{n}$ at that training iteration $n$. Lastly, we update $\theta_{n}$ with a learning rate of $\eta$ to explicitly minimize the validation loss as,

[TABLE]

We repeat these training iterations until convergence to get the optimal weights $\theta^{*}$.

Final Person-specific Adaptation.

Having learned our encoder and our optimal few-shot person-specific learner $\mathcal{M}_{\theta^{*}}$, we are now well poised to produce person-specific models for each new person $\mathcal{P}^{test}$ from $\mathcal{S}^{test}$. We fine-tune $\mathcal{M}_{\theta^{*}}$ with the $k$ calibration images $\mathcal{D}_{c}^{test}$ to create a personalized model for $\mathcal{P}^{test}$ as

[TABLE]

and test the performance of the personalized model $\mathcal{M}_{\theta_{\mathcal{P}^{test}}}$ on person $\mathcal{P}^{test}$’s validation set $\mathcal{D}_{v}^{test}$.

4 Implementation Details

4.1 Data pre-processing

Our data pre-processing pipeline is based on [55], a revision of the data normalization procedure introduced in [45]. In a nutshell, the data normalization procedure ensures that a common virtual camera points at the same reference point in space with the head upright. This requires the rotation, tilt, and forward translation of the virtual camera. Please refer to [55] for a formal and complete description, and our supplementary materials for a detailed list of changes.

4.2 Neural Network Configurations

DT-ED. The functions $\mathcal{E}$ and $\mathcal{D}$ in our transforming encoder-decoder architecture can be implemented with any CNN architecture. We select the DenseNet architecture [18] both for our DT-ED as well as for our re-implementation of state-of-the-art person-specific gaze estimation methods [26, 56]. The latent codes $\mathbf{z}_{a}$, $\mathbf{z}_{g}$, and $\mathbf{z}_{h}$ are defined to have dimensions $(64)$, $(3\times 2)$, and $(3\times 16)$ respectively. Please refer to supplementary materials for further details.

Gaze MLP. Our gaze estimation function $\mathcal{G}$ is parameterized by a multi-layer perceptron with $64$ hidden layer neurons and SELU [21] activation. The MLP outputs 3-dimensional unit gaze direction vectors.

4.3 Training

DT-ED. Following [9], we use a batch size of $1536$ and apply linear learning rate scaling and ramp-up for the first $10^{6}$ training samples. We use NVIDIA’s Apex library222https://github.com/NVIDIA/apex for mixed-precision training. and train our model for $50$ epochs with a base learning rate of $5\times 10^{-5}$, $l_{2}$ weight regularization of $10^{-4}$, and use instance normalization [49].

Gaze MLP. During meta-learning, we use $\alpha=10^{-5}$ with SGD (Eq. 6), and $\eta=10^{-3}$ (Eq. 7) with the Adam optimizer ($\alpha$ and $\beta$ in [6]), and do $5$ updates per inner loop iteration. For sampling $\mathcal{D}^{train}_{v}$ we set $l=100$. During standard eye-tracker calibration, one cannot assume the knowledge of extra ground-truth beyond the $k$ samples. Thus, we perform the final fine-tuning operation (Eq. 8) for $1000$ steps for all values of $k$ and for all people.

4.4 Datasets

GazeCapture [22] is the largest available in-the-wild gaze dataset. We mined camera intrinsic parameters from the web for the devices used, and applied our pre-processing pipeline (Sec. A.1) to yield input images. For training the DT-ED as well as for meta-learning, we use data from $993$ people in the training set specified in [22], each with $1766$ samples, on average, for a total of $1.7M$ samples. To ensure within-subject diversity of sampled image-pairs at training time, we only select subjects with $\geq 400$ samples. For computing our final evaluation metric, we use the last $500$ entries from $109$ subjects that have at least $1000$ samples each. We select the $k$-shot samples for meta-training and fine-tuning randomly from the remaining samples.

MPIIGaze [57] is the most established benchmark dataset for in-the-wild gaze estimation. In comparison to GazeCapture it has higher within-person variations in appearance including illumination, make-up, and facial hair changes, potentially making it more challenging. We use the images specified in the MPIIFaceGaze subset [58] only for evaluation purposes. The MPIIFaceGaze subset consists of $15$ subjects each with $2500$ samples on average. We reserve the last $500$ images of each subject for final evaluations as is done in [59] and select $k$ calibration samples for personalization by sampling randomly from the remaining samples.

5 Results

For all methods, we report person-specific gaze estimation errors for a range of $k$ calibration samples. For each data point, we perform the evaluation $10$ times using $k$ randomly chosen calibration samples. Each evaluation or trial yields a mean gaze estimation error in degrees over all subjects in the test set. The mean error over all such trials is plotted, with its standard deviation represented by the shaded areas above and below the curves. The values at $k=0$ are determined via $\mathcal{G}\left(\mathbf{z}^{g}\right)$. We train this MLP on the GazeCapture training subset, without any person-specific adaptation.

5.1 Ablation Study

We evaluate our method under different settings to better understand the impact of our various design choices. For all experiments, we train the models using the GazeCapture dataset’s training set and test on the MPIIGaze dataset. This challenging experiment allows us to demonstrate the generalization capability of our approach across different datasets. The ablation studies are summarized in Fig. 7. We provide additional plots of the results of this ablation study on the test partition of the GazeCapture dataset in the supplementary material.

MAML vs. Finetuning. In Fig. 7(a), we first evaluate the impact of meta-learning a few-shot person-adaptive gaze estimator using MAML (Sec. 3.3) and compare its performance with naive finetuning. When no person-specific adaptation is performed (i.e., $k=0$), the person-independent baseline model $\mathcal{G}\left(\mathbf{z}^{g}\right)$ with the features learned using a vanilla autoencoder (AE) results in a mean test error of $7.17\degree$. Using MAML for person-specific adaptation with only one calibration sample decreases the error to $6.61\degree$. The error reduces further as we increase $k$ and reaches a mean value of $5.38\degree$ for $k=32$. In contrast, naively finetuning (AE-Finetuning) the gaze estimator results in severe over-fitting and yields very high test errors, in particular, for very low $k$ values. In fact, for $k\leq 3$, the error increases to above the person-independent baseline model. Since the model initialized with weights learned by MAML clearly outperforms vanilla finetuning, in the rest of this section, we always use MAML unless specified otherwise.

Impact of feature representation. Fig. 7(a) also evaluates the impact of the features used to learn the gaze estimation model. Our proposed latent gaze features (Sec. 3.2) significantly decrease the error, e.g., $4.87\degree$ vs. $5.62\degree$ with $k=9$ for DT-ED (MAML) and AE (MAML), respectively. Note that the gain remains consistent across all values of $k$. The only difference between DT-ED and AE is that the latent codes are rotated in DT-ED before decoding. Despite this more difficult task, the learned code clearly better informs the final task of person-specific gaze estimation, showing that disentangling gaze, head pose, and appearance is importance for gaze estimation.

Contribution of loss terms. We evaluate the impact of each loss term described in Eq. (2) (Sec. 3.2) by incorporating them one at a time into the total loss used to train DT-ED. Fig. 7(b) summarizes the results. Using only the image reconstruction loss $\mathcal{L_{\mathrm{recon}}}$ in Eq. (3), the learned latent gaze features result in an error of $4.87\degree$ at $k=9$. Incorporating gaze supervision $\mathcal{L}_{\mathrm{gaze}}$ in Eq. (5) to obtain features that are more informed of the ultimate task of gaze-estimation gives an improvement of $26\%$ from $4.87\degree$ to $3.60\degree$. Adding the person-specific embedding consistency term $\mathcal{L}_{\mathrm{EC}}$ in Eq. (4) to $\mathcal{L}_{\mathrm{recon}}$ also reduces the error significantly from $4.87\degree$ to $3.32\degree$ at $k=9$ (an improvement of over $30\%$). Finally, combining all loss terms improves the performance even further to $3.14\degree$ (in total, an improvement of $36\%$).

Analysis of embedding consistency. In order to validate our choice of the embedding consistency loss, in Fig. 7(c), we compare its performance with two other possible variants. As described in Sec. 3.2.3, the embedding consistency loss term minimizes the intra-person differences of the frontalized latent gaze features. The main rationale behind this is that the gaze features for a unique person should be consistent while they can be different across subjects due to inter-subject anatomical differences. We further conjecture that preserving these inter-personal differences as opposed to trying to remove them by learning person-invariant embeddings is indeed important to obtaining high accuracy for gaze estimation. In order to validate this observation, we introduce a person-independent embedding consistency term which also minimizes the inter-person latent gaze feature differences. As is evident from Fig. 7(c), enforcing person-independent embedding consistency of the latent gaze features results in increased mean errors. In fact it performs worse than only using the reconstruction loss ($\mathcal{L}_{\mathrm{recon}}$).

One may argue the complete opposite i.e., the latent gaze features should be hugely different for every subject for the best possible subject-specific accuracy, but we did not find this to be the case. To demonstrate this, we apply a triplet loss ($\mathcal{L}_{\mathrm{triplet}}$) [41], which explicitly maximizes the inter-personal differences in gaze features in addition to minimizing the intra-person ones. As is evident from Fig 7(c) this results in a significant increase in the error. This suggests that perhaps factors that quantify the overall appearance of a person’s face and define their unique identity may not necessarily be correlated to the anatomical properties that define “person-uniqueness” for the task of gaze estimation.

5.2 Comparison with CNN + Meta-Learning

An alternative baseline to Faze can be created by replacing the DT-ED with a standard CNN architecture. We take an identically configured DenseNet (to Faze) and a VGG-16 architecture for the convolutional layers, then add $2$ fully-connected layers each with $256$ neurons and train the networks with the gaze objective (Eq. 5). The output of the convolutional layers are used as input to a gaze estimation network trained via MAML to yield the results in Fig. 5. Having been directly trained on the (cross-person) gaze estimation objective, it is expected that the encoder network would make better use of its model capacity as it does not have to satisfy a reconstruction objective. Thus, we can call these highly competitive baselines. Faze outperforms these baselines with statistical significance, demonstrating that the DT-ED training and our loss terms yield features which are more amenable to meta-learning, and thus to the final objective of personalized gaze estimation.

5.3 Comparison with State-of-the-Art

Few-shot personalization of CNN models in the context of gaze estimation for very low $k$ is very challenging. Two recent approaches [56, 26] are the most relevant in this direction, and we provide evaluations on highly competitive re-implementations. Our results are presented in Fig. 6 for both the test partition of the GazeCapture dataset and the MPIIGaze dataset. Overall, we show statistically significantly better mean errors over the entire range of $1\leq k\leq 256$ than all the existing state-of-the-art methods. In addition, our performance between trials is more consistent as shown by the narrower error bands. This indicates robustness to the choice of the $k$ calibration samples.

Ours vs Polynomial fit to PoR. In [56], Zhang et al. fit a 3rd order polynomial function to correct initial point-of-regard (PoR) estimates from a person-independent gaze CNN. To re-implement their method, we train a DenseNet CNN (identical to Faze) and intersect the predicted gaze ray (defined by gaze origin and direction in 3D with respect to the original camera) with the $z=0$ plane to estimate the initial PoR and later refine it with a person-specific 3rd order polynomial function. Though this approach performs respectably with $k=9$, yielding $4.19^{\circ}$ on MPIIGaze (Fig. 6(b)), it suffers with lower $k$ especially on GazeCapture. Nonetheless, its performance converges to our performance at $k\geq 128$ showing its effectiveness at higher $k$ despite its apparent simplicity.

Ours vs Differential Gaze Estimation. Liu et al. [26] introduce a CNN architecture for learning to estimate the differences in the gaze yaw and pitch values between pairs of images of the same subject. That is, in order to estimate the gaze their network always requires one reference image of a subject with known gaze values. Then given a reference image $I_{a}$ with a known gaze label $\mathbf{g}_{a}$ and another image $I_{b}$ with unknown gaze label, their approach outputs a $\Delta\mathbf{g}_{ba}$, from which the absolute gaze for $I_{b}$ can be computed as $\hat{\mathbf{y}}_{b}=\mathbf{y}_{a}+\Delta\mathbf{g}_{ba}$. Their original paper states a within-MPIIGaze error with $k=9$ at $4.67^{\circ}$ using a simple LeNet-5 style Siamese network and a pair of eye images as input. We use $256\times 64$ eye-region images from GazeCapture as input and use a DenseNet-based architecture to make their approach more comparable to ours. Our re-implementation yields $3.53^{\circ}$ for their method at $k=9$ on MPIIGaze, a $1.2^{\circ}$ improvement despite dataset differences. We show statistically significant improvements to [26] across all ranges of $k$ in our MPIIGaze evaluations, with our method only requiring $4$ calibration samples to compete with their best performance at $k=256$ (see the red and green curves in Fig. 6). The improvement from our final approach is further emphasized in Fig. 6(a) with evaluations on the test subset of GazeCapture. At $k=4$, we yield a performance improvement of $20.5\%$ or $0.8^{\circ}$ over [26].

6 Conclusion

In this paper we presented the first practical approach to deep-learning based high-accuracy personalized gaze estimation requiring only few calibration samples. Our Faze framework consists of a disentangling encode-decoder network that learns a compact person-specific latent representation of gaze, head pose and appearance. Furthermore, we show that these latent embeddings can be used in a meta-learning context to learn a person-specific gaze estimation network from very few (as low as $k=3$) calibration points. We experimentally showed that our approach outperforms other state-of-the-art approaches by significant margins and produces the currently lowest reported personalized gaze errors on both the GazeCapture and MPIIGaze datasets.

Acknowledgements.

Seonwook Park carried out this work during his internship at Nvidia. This work was supported in part by the ERC Grant OPTINT (StG-2016-717054).

Appendix

Due to constraints on the space available in the main paper, we were unable to include all the details there. Here we provide additional implementation details pertaining to our (a) data pre-processing pipeline and (b) the configuration of our DT-ED network. We also show additional results of the ablation study (Section 5.1 in the main paper) on the test partition of the GazeCapture dataset and the performance of Faze for the within MPIIGaze leave-one-person out setting. Finally, we show the sensitivity of Faze to various design configurations.

A Implementation Details

We describe further details in how we pre-process the datasets used, and the configuration of the DT-ED architecture. A reference implementation of both can be found as open-source software at https://github.com/NVlabs/few_shot_gaze.

A.1 Data Pre-processing

We employ a normalization procedure based on [55], which is a revision of [45], but with a few small changes. We utilize state-of-the-art open-source implementations for face detection333https://github.com/cydonia999/Tiny_Faces_in_Tensorflow [17] and facial landmarks detection444https://github.com/jiankangdeng/Face_Detection_Alignment [5], respectively. We use the Surrey Face Model [20] as the reference 3D face model, and select 4 eye corners and 9 nose landmarks as described by the Multi-PIE 68-points markup [10] for PnP-based [25] head pose estimation. This is in contrast to [45, 55] which instead use the 4 eye corners and 2 mouth corners. This is motivated by our observation that the mouth corner landmarks are not sufficiently static due to facial expression changes, and that the inherent ambiguity in determining head yaw with very few co-planar landmarks in 3D leads to less reliable head pose estimation.

In our work, we utilize a single image as input which contains both eyes. For this purpose, we select the mean of the 2 inner eye corner landmarks in 3D as the origin of our normalized camera coordinate system. We use a focal length of $1300mm$ for the normalized camera intrinsic parameters, and a distance of $600mm$ from the face to produce image patches of size $256\times 64$ to use as input for training.

A.2 Configuration of Disentangling Transforming Encoder-Decoder (DT-ED)

We use the DenseNet architecture to parameterize our encoder-decoder network [18]. We configure the DenseNet with a growth-rate of $32$, 4 dense blocks (each with 4 composite layers), and a compression factor of $1.0$. We neither use dropout nor $1\times 1$ convolutional layers. We use instance normalization [49] and leaky ReLU activation functions (with $\alpha=0.01$) throughout the network as they proved to improve performance for all architectures.

To project CNN features back from latent features $\mathbf{z}$, we apply a fully-connected layer to output values equivalent to $32$ feature maps of width $8$ and height $2$. The DenseNet decoder that we use to model $\mathcal{D}$ is identical in construction to a usual DenseNet but uses deconvolutional layers (with stride 1) in the place of normal convolutions, and $3\times 3$ deconvolutions (with stride 2) instead of average pooling at the transition layers. To be faithful to the original implementation, we do not apply bias layers to convolutions in our DenseNet-based DT-ED. We initialize all layers’ weights with MSRA initialization [14], while biases of the fully-connected layers are initialized with zeros.

B Additional Results

We provide additional results of the ablation study on the test partition of the GazeCapture dataset and evaluate the within-dataset performance of Faze on the MPIIGaze dataset.

B.1 Ablation Study on GazeCapture

In the main paper, we provide the results of the ablation study on the MPIIGaze dataset (Fig. 4 in the main paper). Our evaluation setting is a cross-dataset evaluation, where we train on the training partition of the GazeCapture dataset [22] and test on the test partition of the same dataset as well as on MPIIGaze [57]. Here we show additional results for the GazeCapture test partition (Fig. 7).

In Fig. 7(a) we observe the same trends for the GazeCapture test dataset that we observed for MPIIGaze. Our proposed DT-ED architecture learns latent representations that are better suited for gaze estimation than those learned by a naive encoder-decoder architecture. Additionally, for few-shot personalization significant gains in accuracy are obtained with meta-learning an adaptable network, as we propose, versus naively fine-tuning a network designed for person-independent gaze estimation (Fine-tuning versus MAML). The latter approach also leads to over-fits at very low $k$. Fig. 7(b) shows the value of our proposed loss terms of embedding consistency and of computing gaze from the latent representations while training DT-ED, for GazeCapture. Finally, Fig. 7(c) shows the consistent improvements obtained for the GazeCapture dataset by preserving inter-person differences versus not doing so.

B.2 Within-MPIIGaze Performance

So far Liu et al. [26] report the best known accuracy of $4.67^{\circ}$ with $9$ calibration samples on MPIIGaze with their differential network architecture. They use the within MPIIGaze leave-one-subject out evaluation protocol for their experiments. To directly compare against their method, we evaluate the performance of our Faze framework for this experimental protocol (Fig. 8). With $9$ calibration samples Faze obtains a gaze error of $3.88$, which is a $17\%$ improvement over Liu et al.’s method. Note, also, that within-MPIIGaze training performs worse than training with GazeCapture (see Fig. 6 in the main paper). This is expected, given the significantly larger diversity of subjects present in the GazeCapture training subset ($993$) versus MPIIGaze ($14$ in a leave-one-out setting), which benefits both DT-ED and MAML. This observation corroborates with similar ones previously made in [22].

C Sensitivity Analysis

We show the influence of various design parameters on the overall performance of our algorithm. These analyses help to determine the parameters’ optimal values.

C.1 Latent Gaze Code

Dimension

Our latent gaze code has the dimensions of $3\times F_{g}$. In order to empirically select the optimal value of $F_{g}$, we evaluate the performance of Faze for several different values of $F_{g}=\{16,3,2\}$ shown in Fig. 9, while keeping the dimensions of the appearance and head pose codes fixed at $64$ and $16$ respectively. Empirically we find $F_{g}=2$ to be optimal for both datasets and hence select it for our final implementation.

Normalization

In general we find that is beneficial to normalize our $3\times F_{g}$-sized latent gaze code to achieve the lowest gaze errors. We experiment with various methods for normalization, which involve computing an $\ell_{2}$ norm along a particular dimension and dividing all the observed values for that dimension with the norm. We compute norms along the $F_{g}$ dimension resulting in $3$ norms. Alternatively, one can normalize along the $3$ dimension, resulting in $F_{g}$ norms. We observe that normalizing along the $F_{g}$ dimension, produces lower gaze errors for GazeCapture and equivalent ones for MPIIGaze, versus the alternate approach (Fig. 10). Hence, we use it for our final implementation.