Dynamic Face Video Segmentation via Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning approach for online key-frame decision in real-time face video segmentation, outperforming baselines and generalizing to other datasets.

Contribution

It is the first to apply reinforcement learning to online key-frame decision in dynamic video segmentation and to focus on face videos.

Findings

Reinforcement learning-based scheduler outperforms baselines in key selection and speed.

Method generalizes well to Cityscapes dataset.

First application of RL in face video segmentation.

Abstract

For real-time semantic video segmentation, most recent works utilised a dynamic framework with a key scheduler to make online key/non-key decisions. Some works used a fixed key scheduling policy, while others proposed adaptive key scheduling methods based on heuristic strategies, both of which may lead to suboptimal global performance. To overcome this limitation, we model the online key decision process in dynamic video segmentation as a deep reinforcement learning problem and learn an efficient and effective scheduling policy from expert information about decision history and from the process of maximising global return. Moreover, we study the application of dynamic video segmentation on face videos, a field that has not been investigated before. By evaluating on the 300VW dataset, we show that the performance of our reinforcement key scheduler outperforms that of various baselines in terms of both effective key selections and running speed. Further results on the Cityscapes dataset demonstrate that our proposed method can also generalise to other scenarios. To the best of our knowledge, this is the first work to use reinforcement learning for online key-frame decision in dynamic video segmentation, and also the first work on its application on face videos.

Tables

Table 1. Table 1: The performances of various image segmentation models and the flow estimation model FlowNet2-s. For training FlowNet2-s, Deeplab-V3+ with ResNet-50 backbone is used as the key feature extractor 𝒩 f ​ e ​ a ​ t subscript 𝒩 𝑓 𝑒 𝑎 𝑡 \mathcal{N}_{feat} . FPS is evaluated on a Nvidia 1080Ti GPU. ‘N/A’ refers to “Not Applicable”.

Model	Eval Scales	mIoU(%)	FPS
FCN (VGG16)	N/A	63.54	45.5
Deeplab-V2 (VGG16)	N/A	65.80	3.44
Deeplab-V3+ (Xception-65)	1.0	68.25	24.4
	1.25, 1.75	68.98	6.4
Deeplab-V3+ (MobileNet-V2)	1.0	67.07	58.8
	1.25, 1.75	68.20	21.7
Deeplab-V3+ (ResNet-50)	1.0	67.50	33.3
	1.25, 1.75	69.61	10.1
FlowNet2-s	N/A	64.13	153.8

Equations

$$\begin{array}{rl}{r}_i& =\{\begin{array}{ll}0,& a_j=a_0.\\ U_{a_1}^i-U_{a_0}^i,& a_j=a_1.\end{array}\end{array}$$

$$\begin{array}{rl}{r}_i& =\{\begin{array}{ll}0,& a_j=a_0.\\ 1-Acc(y_{a_0}^i,y_{a_1}^i),& a_j=a_1.\end{array}\end{array}$$

$$\begin{array}{r}\mathcal{L}=J(\theta )+{\lambda }_1\mathcal{H}({\pi }_{\theta }(\mathbf{a}|\mathbf{s}))\end{array}$$

Full text

For real-time semantic video segmentation, most recent works utilised a dynamic framework with a key scheduler to make online key/non-key decisions. Some works used a fixed key scheduling policy, while others proposed adaptive key scheduling methods based on heuristic strategies, both of which may lead to suboptimal global performance. To overcome this limitation, we model the online key decision process in dynamic video segmentation as a deep reinforcement learning problem and learn an efficient and effective scheduling policy from expert information about decision history and from the process of maximising global return. Moreover, we study the application of dynamic video segmentation on face videos, a field that has not been investigated before. By evaluating on the 300VW dataset, we show that the performance of our reinforcement key scheduler outperforms that of various baselines in terms of both effective key selections and running speed. Further results on the Cityscapes dataset demonstrate that our proposed method can also generalise to other scenarios. To the best of our knowledge, this is the first work to use reinforcement learning for online key-frame decision in dynamic video segmentation, and also the first work on its application on face videos.

1 Introduction

In computer vision, semantic segmentation is a computationally intensive task which performs per-pixel classification on images. Following the pioneering work of Fully Convolutional Networks (FCN) [25], tremendous progress has been made in recent years with the propositions of various deep segmentation methods [5, 2, 52, 59, 24, 8, 22, 34, 6, 58, 30]. To achieve accurate result, these image segmentation models usually employ heavy-weight deep architectures and additional steps such as spatial pyramid pooling [59, 6, 5] and multi-scaled paths of inputs/features [7, 58, 23, 5, 4, 22, 35], which further increase the computational workload. For real-time applications such as autonomous driving, video surveillance, and facial analysis [49], it is impractical to apply such methods on a per-frame basis, which will result in high latency intolerable to those applications. Therefore, acceleration becomes a necessity for these models to be applied in real-time video segmentation.

Various methods [40, 64, 54, 20, 16, 29, 17, 31, 11] have been proposed to accelerate video segmentation. Because adjacent frames in a video often share a large proportion of similar pixels, most of these works utilise a dynamic framework which separates frames into key and non-key frames and produce their segmentation masks differently. As illustrated in Fig. 1 (up), a deep image segmentation model $\mathcal{N}$ is divided into a heavy feature extraction part $\mathcal{N}_{feat}$ and a light task-related part $\mathcal{N}_{task}$. To produce segmentation masks, key frames would go through both $\mathcal{N}_{feat}$ and $\mathcal{N}_{task}$, while a fast feature interpolation method is used to obtain features for the non-key frames by warping $\mathcal{N}_{feat}$’s output on the last key frame (LKF), thus to avoid the heavy cost of running $\mathcal{N}_{feat}$ on every frame. On top of that, a key scheduler is used to predict whether an incoming frame should be a key or non-key frame.

As an essential part of dynamic video segmentation, decisions made by the key scheduler could significantly affect the overall performance [20, 54, 62] of the video segmentation framework. However, this topic is somewhat underexplored by the community. Recent works have either adopted a fixed key scheduler [29, 64, 17, 16], or proposed adaptive schedulers [54, 20, 62] which are trained to heuristically predict similarities (or deviations) between two video frames. Those key schedulers lack awareness of the global video context and can lead to suboptimal performance in the long run.

To overcome this limitation, we propose to apply Reinforcement Learning (RL) techniques to expose the key scheduler to the global video context. Leveraging additional expert information about decision history, our scheduler is trained to learn key-decision policies that maximise the long-term returns in each episode, as shown in Fig. 1 (bottom).

We further study the problem of dynamic face video segmentation with our method. Comparing to semantic image/video segmentation, segmentation of face parts is a less investigated field [13, 61, 18, 45, 32, 50, 39, 55, 19, 12], and there are fewer works on face segmentation in videos [49, 37]. Existing works either used engineered features [18, 45, 50, 39, 55, 19], or employed outdated image segmentation models like FCN [25] on a per-frame basis [32, 37, 49] without a dynamic acceleration mechanism. Therefore, we propose a novel real-time face segmentation system utilising our key scheduler trained by Reinforcement Learning (RL).

We evaluate the performances of the proposed method on the 300 Videos in the Wild (300VW) dataset [41] for the task of real-time face segmentation. Comparing to several baseline approaches, we show that our reinforcement key scheduler can make more effective key-frame decisions at the cost of fewer resource. Through further experiment conducted on the Cityscapes dataset [10] for the task of semantic urban scene understanding, we demonstrate that our method can also generalise to other scenarios.

2 Related works

Semantic image segmentation   Fully Convolutional Networks (FCN) [25] is the first work to use fully convolutional layers and skip connections to obtain pixel-level predictions for image segmentation. Subsequent works have made various improvements, including the usage of dilated convolutions [4, 5, 6, 56, 57], encoder-decoder architecture [2, 22, 8], Conditional Random Fields (CRF) for post-processing [60, 4, 5], spatial pyramid pooling to capture multi-scale features [59, 5, 6] and Neural Architecture Search (NAS) [65] to search for the best-performing architectures [3, 24]. Nonetheless, such models usually require intensive computational resources, and thus may lead to unacceptably high latency in video segmentation.

Dynamic video segmentation   Clockwork ConvNet [40] promoted the idea of dynamic segmentation by fixing part of the network. Deep Feature Flow (DFF) [64] accelerated video recognition by leveraging optical flow (extracted by FlowNet [63, 15] or SpyNet [36]) to warp key-frame features. Similar ideas are explored in [54, 17, 31, 11]. Inter-BMV [16] used block motion vectors in compressed videos for acceleration. Mahasseni et al. [26] employed convolutions with uniform filters for feature interpolation, while Li et al. [20] used spatially-variant convolutions instead. Potential interpolation architectures were searched in [29].

On the other hand, studies of key schedulers are comparatively rare. Most existing works adopted fixed key schedulers [29, 64, 17, 16], which is inefficient for real-time segmentation. Mahasseni et al. [26] suggested a budget-aware, LSTM-based key selection strategy trained with reinforcement learning, which is only applicable for offline scenarios. DVSNet [54] proposed an adaptive key decision network based on the similarity score between the interpolated mask and the key predictions, i.e., low similarity scores leading to new keys and vice versa. Similary, Li et al. [20] introduced a dynamic key scheduler trained to predict the deviations between two video frames by the inconsistent low-level features, and [62] proposed to adaptively select key frames depending on the pixels with inconsistent temporal features. Those adaptive key schedulers only consider deviations between two frames, and therefore lack understandings of global video context, leading to suboptimal performances.

Semantic face segmentation   Semantic face parts segmentation received far less attention than that of image/video segmentation. Early works on this topic mostly used engineered features [18, 45, 50, 39, 55, 19] and were designed for static images. Saito et al. [37] employed graphic cut algorithm to refine the probabilistic maps from a FCN trained with augmented data. In [32], a semi-supervised data collection approach was proposed to generate more labelled facial images with random occlusions to train FCN. Recently, Wang et al. [49] integrated Conv-LSTM [53] with FCN [25] to extract face masks from video sequence, while the run-time speed did not improve. None of the se works considered to adopt video dynamics for accelerations, and we are the first to do so for real-time face segmentation.

Reinforcement learning   In model-free Reinforcement Learning (RL), an agent receives a state $\mathbf{s}_{t}$ at each time step $t$ from the environment, and learns a policy $\pi_{\theta}(a_{j}|\mathbf{s}_{t})$ with parameters $\theta$ that guides the agent to take an action $a_{j}\in\mathcal{A}$ to maximise the cumulative rewards $J=\sum_{t=1}^{\infty}\gamma^{t-1}r_{t}$. RL has demonstrated impressive performance on various fields such as robotics and complicated strategy games [21, 43, 28, 48, 42, 47]. In this paper, we show that RL can be seamlessly applied to online key decision problem in real-time video segmentation, and we chose the policy gradient with reinforcement [51] to learn $\pi_{\theta}$, where gradient ascend was used for maximising the objective function $J_{\pi}(\theta)$.

3 Methodology

3.1 System Overview

Our target is to develop an efficient and effective key scheduling policy $\pi_{\theta}(\mathbf{a}|\mathbf{s})$ for the dynamic video segmentation system. To this end, we use Deep Feature Flow [64] as the feature propagation framework, in which the optical flow is calculated by a light-weight flow estimation model $\mathcal{F}$ such as FlowNet [63, 15] or SpyNet [36]. Specifically, an image segmentation model $\mathcal{N}$ can be divided into a time-consuming feature extraction module $\mathcal{N}_{feat}$ and a task specified module $\mathcal{N}_{task}$. We denote the last key frame as $\mathbf{I}_{k}$ and its features extracted by $\mathcal{N}_{feat}$ as $\mathbf{f}_{k}$, i.e., $\mathbf{f}_{k}=\mathcal{N}_{feat}(\mathbf{I}_{k})$. For an incoming frame $\mathbf{I}_{i}$, if it is a key frame, the feature is $\mathbf{f}_{i}=\mathcal{N}_{feat}(\mathbf{I}_{i})$ and the segmentation mask is $\mathbf{y}_{i}=\mathcal{N}_{task}(\mathbf{f}_{i})$; if not, instead of using the resource-intensive module $\mathcal{N}_{feat}$ for feature extraction, its feature $\mathbf{f}_{i}$ will be propagated by a feature interpolation function $\mathcal{W}$, which involves the flow field $\mathbf{M}_{i\rightarrow k}$ from $\mathbf{I}_{i}$ to $\mathbf{I}_{k}$, the scale field $\mathbf{S}_{i\rightarrow k}$ from $\mathbf{I}_{i}$ to $\mathbf{I}_{k}$, and key frame feature $\mathbf{f}_{k}$, hence the predicted mask becomes $\mathbf{y}_{i}=\mathcal{N}_{task}(\mathbf{f}_{i})$. Please check [64] for more details on the feature propagation process.

On top of the DFF framework, we design a light-weight policy network $\pi_{\theta}$ to make online key predictions. The state $\mathbf{s}_{i}$ at frame $\mathbf{I}_{i}$ consists of two parts, the deviation information $\mathcal{D}_{i\rightarrow k}$ which describes the differences between $\mathbf{I}_{k}$ and $\mathbf{I}_{i}$, and the expert information $\mathcal{E}_{i}$ regarding key decision history (see Section 3.2 for details), i.e., $\mathbf{s}_{i}=\{\mathcal{D}_{i\rightarrow k},\mathcal{E}_{i}\}$. Feeding $\mathbf{s}_{i}$ as input, the policy network outputs the action probabilities $\pi_{\theta}(a_{j}|\mathbf{s}_{i})$ where $a_{j}\in\{a_{0},a_{1}\}$ and $\pi_{\theta}(a_{0}|\mathbf{s}_{i})+\pi_{\theta}(a_{1}|\mathbf{s}_{i})=1.0$ (we define $a_{0}$ for non-key action and $a_{1}$ for the key one). For an incoming frame $\mathbf{I}_{i}$, if $\pi_{\theta}(a_{1}|\mathbf{s}_{t})>\tau$ where $\tau$ is a threshold, it will be identified as a key frame, vice versa. In general, key action $a_{1}$ will lead to a segmentation mask with better quality than the ones given by action $a_{0}$.

In this work, we utilise the FlowNet2-s model [15] as the optical flow estimation function $\mathcal{F}$. DVSNet [54] has shown that the high-level features from FlowNet models contain sufficient information about the deviations between two frames, and it can also be easily fetched along with optical flow without additional cost. Therefore, we adopt the features of FlowNet2-s model for $\mathcal{D}_{i\rightarrow k}$. It is worthwhile to notice that by varying $\mathcal{D}_{i\rightarrow k}$ properly, our key scheduler can be easily integrated into other dynamic segmentation frameworks [17, 20, 29, 16, 62] which do not use optical flow. Fig. 2 gives an overview of our system.

3.2 Training Policy Network

Network structure   Our policy network comprises of one convolution layer and four fully connected (FC) layers. The FlowNet2-s feature $\mathcal{D}_{i\rightarrow k}$ is fed into the first convolution layer Conv0 with 96 channels, followed by FC layers (FC0, FC1 and FC2) with output size being 1024, 1024 and 128 respectively. Two additional channels containing expert information about decision history $\mathcal{E}_{i}$ are concatenated to the output of FC2 layer. The first channel records the Key All Ratio (KAR), which is the ratio between the key frame and every other frames in decision history, while the second channel contains the Last Key Distance (LKD), which is the interval between the current and the last key frame. KAR provides information on the frequency of historical key selection, and LKD gives awareness about the length of continuous non-key decisions. Hence, the insertion of KAR and LKD extends the output dimension of FC2 to 130, while FC3 layer summarises all these information and gives action probabilities $\pi_{\theta}(a_{j}|\mathbf{s}_{i})$ where $a_{j}\in\{a_{0},a_{1}\}$, $a_{0}$ and $a_{1}$ stand for non-key and key action correspondingly.

Reward definition   We use mean Intersection-over-Union (mIoU) as the metric to evaluate the segmentation masks. We denote the mIoU of $\mathbf{y}_{i}$ from a non-key action $a_{0}$ as $U^{i}_{a_{0}}$, the mIoU from key action $a_{1}$ as $U^{i}_{a_{1}}$, and the reward $r_{i}$ at frame $\mathbf{I}_{i}$ is defined in Eq. 1. Such definition encourages the scheduler to choose key action on the frames that would result in larger improvement over non-key action, and it also reduces the variances of mIoUs across the video.

[TABLE]

If no groundtruth is available (such that mIoU could not be calculated), we use the segmentation mask from key action as the pseudo groundtruth mask. In this case, the reward formulation is changed to Eq. 2, in which ${y}^{i}_{a_{0}}$ and ${y}^{i}_{a_{1}}$ denote the segmentation mask on $i^{th}$ frame from non-key action $a_{0}$ and key action $a_{1}$ respectively, and $Acc({y}^{i}_{a_{0}},{y}^{i}_{a_{1}})$ stands for the accuracy score with ${y}^{i}_{a_{0}}$ as the prediction and ${y}^{i}_{a_{1}}$ as the label.

[TABLE]

Constraining key selection frequency   The constraints on key selection frequency are necessary in our task. Since a key action will generally lead to a better reward than a non-key one, the policy network inclines to make all-key decisions if no constraint is imposed on the frequency of key selection. In this paper, we propose a stop immediately exceeding the limitation approach. Particularly, for one episode consisting of $M+1$ frames $\{\mathbf{I}_{t},\mathbf{I}_{t+1},...,\mathbf{I}_{t+M}\}$, the agent starts from $\mathbf{I}_{t}$ and explores continuously towards $\mathbf{I}_{t+M}$. At each time step, if the KAR in decision history has already surpassed a limit $\eta$, the agent will stop immediately and thus this episode ends, otherwise, it will continue until reaching the last frame $\mathbf{I}_{t+M}$. By using this strategy, a policy network should limit the use of key decision to avoid an over-early stopping, and also learn to allocate the limited key budgets on the frames with higher rewards. By varying the KAR limit $\eta$, we could train $\pi_{\theta}$ with different key decision frequencies.

Episode settings   Real-time videos usually contains enormous number of high-dimensional frames, thus it is impractical to include all of them in one episode, due to the high computational complexity and possible huge variations across frames. For simplicity, we limit the length of one episode $\{\mathbf{I}_{t},\mathbf{I}_{t+1},...,\mathbf{I}_{t+M}\}$ to 270 frames (9 seconds) for 300VW and 30 frames (snippet length) for Cityscapes respectively. We vary the starting frame $\mathbf{I}_{t}$ during training to learn the global policies across videos. For each episode, we let the agent run $K$ times (with the aforementioned key constraint strategy) to obtain $K$ trials to reduce variances. The return of each episode can be expressed as $J(\theta)=\frac{1}{K}\sum_{v=1}^{K}\sum_{u=t}^{t+p_{v}}\gamma^{u-t}r_{u}^{v}$, where $t$ is the starting frame index of the episode, and $p_{v}$ denotes the total step number at the $v^{th}$ trail (since agent may stop before $M$ steps), and $r_{u}^{v}$ refers to the reward of frame $u$ in $v^{th}$ trail. $J(\theta)$ is the main objective function to optimise.

Auxiliary loss   In addition to optimise the cumulative reward $J(\theta)$, we employ the entropy loss $\mathcal{H}(\pi_{\theta}(\mathbf{a}|\mathbf{s}))$ as in [27, 33] to promote the policy that retains high-entropy action posteriors so as to avoid over-confident actions. Eq. 3 shows the final objective function $\mathcal{L}$ to optimise using policy gradient with reinforcement method [51].

[TABLE]

Epsilon-greedy strategy   During training, agent may still fall into over-deterministic dilemmas with action posteriors approaching nearly 1, even though the auxiliary entropy loss have been added. To recover from such dilemma, we implement a simple strategy similar to epsilon-greedy algorithm for action sampling, i.e., in the cases that action probabilities $\pi_{\theta}(a_{j}|\mathbf{s})$ exceed a threshold $\epsilon$ (such as 0.98), instead of taking action ${a_{j}}$ with probability $\pi_{\theta}({a_{j}}|\mathbf{s})$, we use $\epsilon$ to stochastically pick action ${a_{j}}$ (and $1.0-\epsilon$ for picking action ${a_{1-j}}$).

4 Experiments

4.1 Datasets

We conducted experiments on two datasets: the 300 Videos on the Wild (300VW) dataset [41] and the Cityscapes dataset [10]. 300VW is used for evaluating the proposed real-time face segmentation system with the RL key selector. To the best of our knowledge, 300VW is the only publicly available face video dataset that provides per-frame segmentation labels. Therefore, to demonstrate the generality of our method, we also evaluate our method on Cityscapes [10], which is a widely used dataset for scene parsing, and thus we show how our RL key scheduler can generalise to other datasets and scenarios.

The 300VW dataset contains 114 face videos (captured at 30 FPS) with an average length of 64 seconds, all of which are taken in unconstrained environment. Following [49], we have cropped faces out of the video frames and generated the segmentation labels with facial skin, eyes, outer mouth and inner mouth for all the 218,595 frames. For experiment purpose, we divided the videos into three subject-independent parts, namely $A/B/C$ sets containing 51 / 51 / 12 videos. In detail, for training $\mathcal{N}$, we randomly picked 9,990 / 1,0320 / 2,400 frames from sets $A/B/C$ for training/validation/testing. To train $\mathcal{F}$, we randomly generate 32,410 / 4,836 / 6,671 key-current image pairs with a varying gap between 1 to 30 frames from sets $A/B/C$ for training/validation/testing. We intentionally excluded set $A$ for policy network learning, since this set has already been used to train $\mathcal{N}$ and $\mathcal{F}$, instead, we used the full $B$ set (51 videos with 98,947 frames) for training and validating the RL key scheduler, and evaluated it on the full $C$ set (12 videos with 22,580 frames).

The Cityscapes dataset contains 2,975 / 500 / 1,525 annotated urban scene images as training/validation/testing set, while each annotated image is the $20^{th}$ frame of a 30-frame (1.8 seconds) video snippet. To ensure a fair comparison on this dataset, we have adopted the same preliminary models ($\mathcal{N}$ and $\mathcal{F}$) and the model weights provided by the authors of DVSNet [54], such that we only re-trained the proposed RL key schedulers using the Cityscapes training snippets. Following DVSNet [54], our method and the baselines are evaluated on the validation snippets, where the initial frame is set as key and performances are measured on the $20^{th}$ annotated frame.

4.2 Experimental Setup

Evaluation metric We employed the commonly used mean Intersection-over-Union (mIoU) as the evaluation metric. For the performance evaluation of different key schedulers, we measure: 1. the relationship between Average Key Intervals (AKI) and mIoU, as to demonstrate the effectiveness of key selections under different speed requirements, and 2. the relationship between the actual FPS and mIoU.

Training preliminary networks   On 300VW, we utilised the state-of-the-art Deeplab-V3+ architecture [8] for image segmentation model $\mathcal{N}$, and we adopted the FlowNet2-s architecture [15] as the implementation of flow estimation function $\mathcal{F}$. For training $\mathcal{N}$, we initialised the weights using the pre-trained model provided in [8] and then fine-tuned it. We set the output stride and decoder output stride to 16 and 4, respectively. We divided $\mathcal{N}$ into $\mathcal{N}_{feat}$ and $\mathcal{N}_{task}$, where the output of $\mathcal{N}_{feat}$ is the posterior for each image pixel, we then fine-tuned the FlowNet2-s model $\mathcal{F}$ as suggested in [64] by freezing $\mathcal{N}_{feat}$ and $\mathcal{N}_{task}$. Also, we used the pre-trained weights provided in [15] as the starting point of training $\mathcal{F}$. The input sizes for $\mathcal{N}$ and $\mathcal{F}$ are both set to 513*513.

On Cityscapes, we have adopted identical $\mathcal{N}$ and $\mathcal{F}$ architectures as DVSNet [54] and directly use the weights provided by the authors, such that we only re-trained the proposed policy key scheduler. Besides, we have also adopted the frame division strategy from DVSNet and have divided the frame into four individual regions. We refer interested readers to [54] for more details.

Reinforcement learning settings   For state $\mathbf{s}_{i}=\{\mathcal{D}_{i\rightarrow k},\mathcal{E}_{i}\}$, following DVSNet [54], we leveraged the features from the Conv6 layer of the FlowNet2-s model as the deviation information $\mathcal{D}_{i\rightarrow k}$, and we obtained the expert information $\mathcal{E}_{i}=\{KAR,LKD\}$ from the last 90 decisions. During the training of policy network, $\mathcal{N}_{feat}$, $\mathcal{N}_{task}$ and $\mathcal{F}$ were frozen to avoid unnecessary computations. We chose RMSProp [46] as the optimiser and set the initial learning rate to 0.001. The parameters $\lambda_{1}$ in Eq. 3 were set to 0.14. We empirically decided the discount factor $\gamma$ to be 1.0, as the per frame performance was equally important in our task. The value of epsilon $\epsilon$ in epsilon-greedy strategy was set to 0.98. During training, we set the threshold value $\tau$ for determining the key action to 0.5. We used the reward formulation as defined in Eq. 1 for 300VW. For Cityscapes, the modified reward as defined in Eq. 2 was used because most frames in the Cityscapes dataset are not annotated. The maximum length of each episode was set to 270 frames (9 seconds) for 300VW and 30 frames (snippet length) for Cityscapes respectively, and we repeated a relatively large number of 32 trials for each episode with a mini-batch size of 8 episodes for back-propagation in $\pi_{\theta}$. We trained each model for 2,400 episodes and validated the performances of checkpoints on the same set. We also varied the KAR limit $\eta$ to obtain policy networks with different key decision tendencies.

Baseline comparison   We compared our method with three baseline approaches on both datasets: (1) The adaptive key decision model DVSNet [54]; (2) The adaptive key scheduler using flow magnitude difference in [54]; (3) Deep Feature Flow (DFF) with a fixed key scheduler as in [64]. We utilised the same implementations and settings for the baselines as described in DVSNet paper, and we refer the readers to [54] for details. Note that for the implementation of DVSNet on Cityscapes, we directly used the model weights provided by the authors, but we have re-trained the DVSNet model on 300VW. For our method, to obtain key decisions with different Average Key Intervals, we have trained multiple models with various KAR limit $\eta$, and also varied the key threshold values $\tau$ of those models.

Implementation   We implemented our method in Tensorflow [1] framework. Experiments were run on a cluster with eight NVidia 1080 Ti GPUs, and it took approximately 2.5 days to train a RL model per GPU.

4.3 Results

Preliminary networks on 300VW   We evaluated five image segmentation models on 300VW dataset: FCN [25] with VGG16 [44] architecture, Deeplab-V2 [5] of VGG16 version, the Deeplab-V3+ [8] with Xception-65 [9] / MobileNet-V2 [38] / RestNet-50 [14] backbones. We have also tested two different eval scales (refer [8] for details) for Deeplab-V3+ model. As can be seen from Table 1, Deeplab-V3+ with ResNet-50 backbone and multiple eval scales (1.25 and 1.75) has achieved the best mIoU with an acceptable FPS, therefore we selected it for our segmentation model $\mathcal{N}$. Its feature extraction part $\mathcal{N}_{feat}$ was used to extract key frame feature in key-current images pairs during the training of FlowNet2-s [15] model $\mathcal{F}$, whose performance was evaluated by the interpolation results on current frames. From Table 1 we can discover that the interpolation speed with $\mathcal{F}$ is generally much faster than those segmentation models at the cost of a slight drop in mIoU (from 69.61% to 64.13%). Under live video scenario, the loss of accuracy can be effectively remedied by a good key scheduler.

RL training visualisation on 300VW   In the upper row of Fig. 3, we demonstrate the average return during RL training with different KAR limits $\eta$ (0.04, 0.06, 0.14) on 300VW dataset. It can be seen that even though we select the starting frames of each episode randomly, those return curves still exhibit a generally increasing trend despite several fluctuations. This validates the effectiveness of our solutions for reducing variances and stabilising gradients, and it also verifies that the policy $\pi_{\theta}$ is improving towards more rewarding key actions. Besides, as the value of $\eta$ increases and allows for more key actions, the maximum return that each curve achieves also becomes intuitively higher.

We also visualised the influences of two expert information KAR and LDK by plotting their weights in $\pi_{\theta}$ during RL training on 300VW. In the bottom row of Fig. 3, we have plotted the weights of the two channels in $\pi_{\theta}$ that received KAR and LDK as input and contributed to the key posteriors $\pi_{\theta}(a_{1}|\mathbf{s})$, and we can observe that the weights of the LDK channel show a globally rising trend, while that of the KAR channel decrease continuously. Such trends indicate that the KAR/LDK channels become increasingly important in key decisions as training proceeds, since a large LDK value (or a small KAR) will encourage $\pi_{\theta}$ to take key action. This observation is consistent with the proposed key constraint strategy. Furthermore, we can also imply that the key scheduler relies more on the LDK channel than the KAR with a lower $\eta$ like 0.04, conversely, KAR becomes more significant with a higher $\eta$ like 0.14.

Performances evaluations   The upper plot of Fig. 4 shows the Average Key Intervals (AKI) versus mIoU of various key selectors on the 300VW dataset and the bottom plot depicts the corresponding FPS versus mIoU curves. Note that in the AKI vs. mIoU graph, we include two versions of DFF: the one with fixed key intervals and the variant with randomly selected keys. We can easily see that our key scheduler have shown superior performance than others in terms of both effective key selections and the actual running speed. Although the performance of all methods are similar for AKI less than 20, this is to be expected as the performance degradation on non-key frames can be compensated by dense key selections. Our method starts to show superior performance when the key interval increases beyond 25, where our mIoUs are consistently higher than that of other methods and decreases slower as the key interval increases.

The evaluation results on Cityscapes can be found in Fig. 5, which demonstrates a similar trend with those results on 300VW and therefore validates the generality of our RL key scheduler to other datasets and tasks. However, it should be noted that, in the case of face videos, selecting key frames by a small interval ($\ll 20$) does not significantly affect the performance, which is not the same as in the autonomous driving scenarios of Cityscapes. This could be attributed to the fact that variations between consecutive frames in face videos are generally less than those in autonomous driving scenes. As a result, we can gain more efficiency benefit when using key scheduling policy with relatively large interval for dynamic segmentation of face video.

4.4 Visualising Key Selections

To better understand why our RL-based key selection method outperforms the baselines, we visualise the distribution of intervals between consecutive keys (CKI) based on the key selections made by all evaluated methods. Without loss of generality, Fig. 6 shows the density curves plotted from the experiment on 300VW dataset at AKI=121. As DFF uses a fixed key interval, its CKI distribution takes the shape of a single spike in the figure. In contrast, the CKI distribution given by our method has the flattest shape, meaning that the key frames selected by our method are more unevenly situated in the test videos. Noticeably, there are more cases of large gaps (>200) between neighbouring keys selected by our method than by others. This indicates our method could better capture the dynamics of the video and only select keys that have larger global impact to the segmentation accuracy.

In addition, we also visualise the key frames selected by our method, DFF and DVSNet on a 30-second test video in Fig. 7 to provide insight on how the key selections can affect the mIoU. We can observe from this figure that the key frames selected by our method can better compensate for the loss of accuracy and retain higher mIoU over longer span of frames (such as frame 37 and 459), while those selected by DFF (fixed scheduler) are less flexible and the compensation to mIoU loss is generally worse than ours. Comparing DVSNet with ours, we can see that 1) our method can give key decisions with more stable non-key mIoUs (frames 37, 459 and 713), and 2) on hard frames such as frames 600 to 750, our reinforcement key scheduler has also made better compensations to performance loss with less key frames. These observations demonstrate the benefits brought by reinforcement learning, which is to learn key-decision policies from the global video context.

Last but not least, in Fig. 8, we plot the segmentation masks generated by different methods on several non-key frames during the experiment on 300VW dataset (AKI=121). It can be seen that DFF with fixed key schedulers usually leads to low-quality masks with missing facial components, while the DVSNet and the Flow Magnitude methods have shown better but still not satisfying results. In contrast, our method has produced non-key masks with the best visual qualities, which further validate the effectiveness of the proposed key schedulers.

5 Conclusions

In this paper, we proposed to learn an efficient and effective key scheduler via reinforcement learning for dynamic face video segmentation. By utilising expert information and appropriately designed training strategies, our key scheduler achieves more effective key decisions than baseline methods at smaller computational cost. We also show the method is not limited to face video but could also generalise to other scenarios. By visualising the key selections made by our method, we try to explain why our key scheduler can make better selections than others. This is the first work to apply dynamic segmentation techniques with RL on real-time face videos, and it can be inspiring to future works on real-time face segmentation and on dynamic video segmentation.

Acknowledgements

The work of Yujiang Wang has been partially supported by China Scholarship Council (No. 201708060212) and the EPSRC project EP/N007743/1 (FACER2VM). The work of Yang Wu has been supported, in part, by Microsoft Research Asia through MSRA Collaborative Research 2019 Grant.