Adversarial Framework for Unsupervised Learning of Motion Dynamics in Videos

TL;DR

This paper introduces an adversarial, unsupervised framework for learning motion dynamics in videos, enabling dense and global predictions, and reducing annotation needs for video segmentation.

Contribution

It proposes a GAN-based self-supervised method that models static content and motion separately, learning a coherent motion latent space for improved video understanding.

Findings

Achieves comparable segmentation performance with 50% fewer annotations.

Outperforms state-of-the-art in motion-related video generation metrics.

Learns both local and global video dynamics in an unsupervised manner.

Abstract

Human behavior understanding in videos is a complex, still unsolved problem and requires to accurately model motion at both the local (pixel-wise dense prediction) and global (aggregation of motion cues) levels. Current approaches based on supervised learning require large amounts of annotated data, whose scarce availability is one of the main limiting factors to the development of general solutions. Unsupervised learning can instead leverage the vast amount of videos available on the web and it is a promising solution for overcoming the existing limitations. In this paper, we propose an adversarial GAN-based framework that learns video representations and dynamics through a self-supervision mechanism in order to perform dense and global prediction in videos. Our approach synthesizes videos by 1) factorizing the process into the generation of static visual content and motion, 2) learning a suitable representation of a motion latent space in order to enforce spatio-temporal coherency of object trajectories, and 3) incorporating motion estimation and pixel-wise dense prediction into the training procedure. Self-supervision is enforced by using motion masks produced by the generator, as a co-product of its generation process, to supervise the discriminator network in performing dense prediction. Performance evaluation, carried out on standard benchmarks, shows that our approach is able to learn, in an unsupervised way, both local and global video dynamics. The learned representations, then, support the training of video object segmentation methods with sensibly less (about 50%) annotations, giving performance comparable to the state of the art. Furthermore, the proposed method achieves promising performance in generating realistic videos, outperforming state-of-the-art approaches especially on motion-related metrics.

Figures

Tables

Table 1. Table 1: Architecture of the generator. Bkg stream contains the layers included in the background stream of the model, that returns b ​ ( z C ) 𝑏 subscript 𝑧 𝐶 b(z_{C}) (see Eq. 1 in the paper). Motion features lists the layers in the initial shared part of the foreground stream. Fg features considers the layers used to process concatenation over the channel dimension of the output of motion features with the output from the third layer of the background stream . By ”3D Residual Block” we simply denote standard shape-preserving residual blocks using 3D convolution. The output of Fg raw is f ​ ( z C , z M ) 𝑓 subscript 𝑧 𝐶 subscript 𝑧 𝑀 f(z_{C},z_{M}) and the output of Fg mask is m ​ ( z C , z M ) 𝑚 subscript 𝑧 𝐶 subscript 𝑧 𝑀 m(z_{C},z_{M}) . LReLU stands for Leaky ReLU, while layers marked with ConvTran execute transposed convolution over their input.

Bkg stream	Kernel	Stride	Padding	Activation	BatchNorm	Output shape
$z_C$	$-$	$-$	$-$	$-$	-	100x1x1
ConvTran2D	4x4	1x1	$-$	LReLU($\alpha =0.2$)	Yes	512x4x4
ConvTran2D	4x4	2x2	1x1	LReLU($\alpha =0.2$)	Yes	256x8x8
ConvTran2D	4x4	2x2	1x1	LReLU($\alpha =0.2$)	Yes	128x16x16
ConvTran2D	4x4	2x2	1x1	LReLU($\alpha =0.2$)	Yes	64x32x32
ConvTran2D	4x4	2x2	1x1	Tanh	No	3x64x64

Table 2. Table 2: Discriminator architecture. Note that the structure of transposed convolutions used in Motion Stream is symmetrical to the one of convolutions in the Shared part. LReLU stands for Leaky ReLU, while the layers marked with ConvTran are transposed convolutions. The output of the motion stream has 3 channels: 1 for segmentation and 2 for optical flow.

Shared	Kernel Size	Stride	Padding	Activation	BatchNorm	Out shape
Input	$-$	$-$	$-$	$-$		3x64x64x32
Conv3D	4x4x4	2x2x2	1x1x1	LReLU($\alpha =0.2$)	No	64x32x32x16
Conv3D	4x4x4	2x2x2	1x1x1	LReLU($\alpha =0.2$)	Yes	128x16x16x8
Residual Layer
3D Residual Block	3x3x3	1x1x1	1x1x1	ReLU	Yes	128x16x16x8
3D Residual Block	3x3x3	1x1x1	1x1x1	ReLU	Yes	128x16x16x8
3D Residual Block	3x3x3	1x1x1	1x1x1	ReLU	Yes	128x16x16x8
Conv3D	4x4x4	2x2x2	1x1x1	LReLU($\alpha =0.2$)	Yes	256x8x8x4
Residual Layer
3D Residual Block	3x3x3	1x1x1	1x1x1	ReLU	Yes	256x8x8x4
3D Residual Block	3x3x3	1x1x1	1x1x1	ReLU	Yes	256x8x8x4
3D Residual Block	3x3x3	1x1x1	1x1x1	ReLU	Yes	256x8x8x4
Conv3D	2x2x2	1x1x1	1x1x1	LReLU($\alpha =0.2$)	Yes	512x9x9x5

Table 3. Table 3: User preference score (percentage) on video generation quality on different types of generated videos.

	User preference %
Golf course
VOS-GAN vs VGAN NIPS2016_6194	80.5 / 19.5
VOS-GAN vs TGAN Saito_2017_ICCV	65.1 / 34.9
VOS-GAN vs MoCoGAN Tulyakov_2018_CVPR	58.2 / 41.8
Weizmann Action dataset
VOS-GAN vs VGAN NIPS2016_6194	82.2 / 17.8
VOS-GAN vs TGAN Saito_2017_ICCV	70.3 / 29.7
VOS-GAN vs MoCoGAN Tulyakov_2018_CVPR	39.7 / 60.3

Table 4. Table 4: Ablation studies. Quantitative evaluation on the “Golf course” dataset of different configurations of the proposed model. The baseline is the GAN architecture described in Sect. 3 trained with traditional adversarial loss and using a random latent variable for foreground content in addition to the bakcground latent space.

						FCD $\downarrow$	FID $\downarrow$	IS $\uparrow$
	SLERP	LSTM	Segmentation	OF	$\alpha$
Baseline						5.41 $\pm$ 0.020	44.02 $\pm$ 0.45	1.98 $\pm$ 0.019
Model 1						5.27 $\pm$ 0.024	41.76 $\pm$ 0.53	2.07 $\pm$ 0.024
Model 2						4.77 $\pm$ 0.018	40.54 $\pm$ 0.41	2.29 $\pm$ 0.019
Model 3						4.61 $\pm$ 0.026	38.79 $\pm$ 0.53	2.73 $\pm$ 0.011
Model 4						4.22 $\pm$ 0.022	33.18 $\pm$ 0.43	3.09 $\pm$ 0.011
VOS-GAN						4.11 $\pm$ 0.018	31.32 $\pm$ 0.46	3.16 $\pm$ 0.032

Table 5. Table 5: Comparison of quantitative generation performance, in terms of foreground content distance (FCD), Fréchet Inception Distance (FID) and Inception Score (IS), against VGAN, TGAN and MoCoGAN, respectively, on the “Golf course” and the Weizmann action dataset.

	FCD $\downarrow$	FID $\downarrow$	IS $\uparrow$
Golf course
VGAN	10.61 $\pm$ 0.015	45.32 $\pm$ 0.31	1.74 $\pm$ 0.021
TGAN	5.74 $\pm$ 0.025	42.58 $\pm$ 0.39	2.02 $\pm$ 0.019
MoCoGAN	5.01 $\pm$ 0.023	34.53 $\pm$ 0.42	2.47 $\pm$ 0.013
VOS-GAN	4.11 $\pm$ 0.018	31.32 $\pm$ 0.46	3.16 $\pm$ 0.032
Weizmann Action dataset
VGAN	5.18 $\pm$ 0.021	7.64 $\pm$ 0.041	2.78 $\pm$ 0.027
TGAN	4.53 $\pm$ 0.029	7.00 $\pm$ 0.027	2.94 $\pm$ 0.016
MoCoGAN	4.07 $\pm$ 0.018	5.74 $\pm$ 0.031	3.76 $\pm$ 0.025
VOS-GAN	4.24 $\pm$ 0.016	5.71 $\pm$ 0.034	3.29 $\pm$ 0.020

Table 6. Table 6: Video object segmentation results (in percentage). The first two rows report the results obtained by training the model without annotations, while the third and forth rows report the performance when fine-tuning on the video benchmarks. The last row shows the results achieved by our baseline trained purely in a supervised way.

Learning	Model		DAVIS-16			DAVIS-17			SegTrack-v2
			${ℳ}_{ℱ}$	${ℳ}_{𝒥}$		${ℳ}_{ℱ}$	${ℳ}_{𝒥}$		${ℳ}_{ℱ}$	${ℳ}_{𝒥}$
Unsupervised	Synthetic VOS		25.41	33.66		19.83	21.87		20.32	23.72
	Adversarial VOS		31.22	38.11		22.57	27.01		24.11	27.42
Supervised	Synthetic VOS FT		60.85	64.66		52.54	55.12		56.43	59.95
	Adversarial VOS FT		67.35	71.24		56.10	61.65		61.14	65.02
	Baseline		57.05	62.85		50.96	53.41		53.26	57.03

Table 7. Table 7: Performance on DAVIS 2017 by unsupervised (i.e., not using any annotated data at inference time) video object segmentation methods, as well as of unsupervised models fine-tuned on DAVIS 2017.

	Method	$M_{𝒥}$	$M_{ℱ}$
Unsupervised	Optical Flow Farneback:2003	14.55	10.32
	Optical Flow IMSKDB17	17.21	15.43
	Unsupervised VondrickSFGM18	19.86	17.47
	Adversarial VOS	27.01	22.57
Supervised	OF estimation + FT	54.32	47.73
	Adversarial VOS + FT	61.65	56.10

Table 8. Table 8: Comparison to state-of-the-art unsupervised methods on DAVIS 2016, DAVIS 2017 and SegTrack-v2 benchmarks in terms of region similarity ( 𝒥 𝒥 \mathcal{J} ) contour accuracy ( ℱ ℱ \mathcal{F} ). On SegTrack-v2 we report only mean 𝒥 𝒥 \mathcal{J} as comparing methods employ only that measure since recall and contour accuracy metrics were introduced only in 7780454 . In bold best performance, in italic the second best performance.

DAVIS 2016
	BMVC.28.21	papazoglou2013fast	Tokmakov17	JainXG17	8100267	Ours
$𝒥$ - Region Similarity
Mean $ℳ$	64.1	57.5	70.0	70.7	76.3	71.2
Recall $𝒪$	73.1	65.2	85.0	83.5	89.2	84.6
$ℱ$ - Contour accuracy
Mean $ℳ$	59.3	53.6	65.9	65.3	71.1	67.3
Recall $𝒪$	65.8	57.9	79.2	73.8	82.8	77.9
- Average
Mean $ℳ$	61.7	55.5	67.9	68.0	73.7	69.2
Recall $𝒪$	69.4	61.5	82.1	78.6	86.0	81.2
DAVIS 2017
		BMVC.28.21	papazoglou2013fast	Jang2018VideoPW	8100267	Ours
$𝒥$ - Region Similarity
Mean $ℳ$		51.4	49.6	45.0	63.3	61.7
Recall $𝒪$		55.5	52.9	46.4	72.9	61.4
$ℱ$ - Contour accuracy
Mean $ℳ$		48.6	48.0	44.8	61.2	56.1
Recall $𝒪$		49.4	46.8	43.0	67.8	53.8
- Average
Mean $ℳ$		50.0	48.8	44.9	62.2	58.9
Recall $𝒪$		52.4	49.8	44.7	70.3	57.6
SegTrack-v2
		papazoglou2013fast	6248065	BMVC2015_186	HallerL17	Ours
$𝒥$ - Region Similarity
Mean $ℳ$		50.1	30.4	49.9	61.1	65.0

Table 9. Table 9: Video action recognition accuracy on UCF101 of the models originally trained on “Golf course” (classification accuracy, in percentage).

Settings	Models
	VGAN	TGAN	VOS-GANG
Transfer learning	39.19	32.45	41.02
Fine-tuning	45.43	36.94	49.33

Table 10. Table 10: Video action recognition accuracy on UCF101 of the models originally trained on Weizmann Action dataset (classification accuracy, in percentage).

Settings	Models
	MoCoGAN	VOS-GANW
Transfer learning	18.47	29.42
Fine-tuning	32.83	45.66

Table 11. Table 11: Video action recognition accuracy on Weizmann Action dataset of the models originally trained on “Golf course” (classification accuracy, in percentage).

Settings	Models
	VGAN	TGAN	VOS-GANG
Transfer learning	65.87	54.19	68.79
Fine-tuning	64.41	52.30	66.80

Table 12. Table 12: Video action recognition accuracy on Weizmann Action Dataset of the models originally trained on the same dataset (classification accuracy, in percentage).

Settings	Models
	MoCoGAN	VOS-GANW
Transfer learning	70.76	74.29
Fine-tuning	67.02	71.63

Equations

$$\begin{array}{r}G(z_C,z_M)=m(z_C,z_M)\odot f(z_C,z_M)\\ +(1-m(z_C,z_M))\odot b(z_C)\end{array}$$

$$\begin{array}{rl}{\mathcal{L}}_D=& -{\mathbb{E}}_{x\sim p_{\text{real}}}\left[\log D_{\text{adv}}\left(x\right)\right]\\ & -{\mathbb{E}}_{z\sim p_z}\left[\log \left(1-D_{\text{adv}}\left(G\left(z\right)\right)\right)\right]+\\ & +{\mathbb{E}}_{z\sim p_z}\left[{\text{NLL}}_{\text{2D}}\left(D_{\text{seg}}\left(G\left(z\right)\right),m(z)\right)\right]+\\ & +\alpha \left({\mathbb{E}}_{z\sim p_z}\left[{\Vert D_{\text{opt}}\left(G\left(z\right)\right)-\text{OF}\left(G\left(z\right)\right)\Vert }^2\right]\right)+\\ & +\alpha \left({\mathbb{E}}_{x\sim p_{\text{real}}}\left[{\Vert D_{\text{opt}}\left(x\right)-\text{OF}\left(x\right)\Vert }^2\right]\right)\end{array}$$

$${\mathcal{L}}_G=-{\mathbb{E}}_{z\sim p_z}\left[\log D_{\text{adv}}\left(G\left(z\right)\right)\right]$$

$${\mathcal{M}}_{\mathcal{C}}(R)=\frac{1}{|R|}\sum _{{\mathcal{S}}_i\in \mathcal{R}}\overline{\mathcal{C}}(S_i),$$

$${\mathcal{O}}_{\mathcal{C}}(R)=\frac{1}{|R|}\sum _{{\mathcal{S}}_i\in \mathcal{R}}\mathbb{I}\left[\overline{\mathcal{C}}(S_i)>\tau \right],$$

$${\mathcal{D}}_{\mathcal{C}}(R)=\frac{1}{|R|}\sum _{{\mathcal{Q}}_i\in \mathcal{R}}\overline{\mathcal{C}}(Q_i^1)-\overline{\mathcal{C}}(Q_i^4),$$

Full text

1 Introduction

Learning motion dynamics plays an important role in video understanding, which fosters many applications, such as object tracking, video object segmentation, event detection and human behavior understanding. The latter is a particularly complex task, due to the variability of possible scenarios, conditions, actions/behaviors of interest, appearance of agents and to a generic ambiguity in how behaviors should be defined, categorized and represented. Behavior understanding is also a key component in the direction toward visual intelligence: the identification of what happens in a given environment necessarily requires the capability to decode actions and intentions from visual evidence.

If behavior understanding in computer vision is a fundamental component for scene understanding, pixel-wise dense prediction for video object segmentation is one of the founding stones for the whole process, as it isolates relevant regions in a scene from unnecessary background elements, thus serving both as a way to focus analysis on a subset of the input data, and to compute a preliminary representation suitable for further processing. Unfortunately, although video object segmentation has been studied for decades, it is far from solved, as current approaches are not yet able to generalize to the variety of unforeseeable conditions that are found in real-world applications. Additionally, learning long-term spatio-temporal features directly for dense prediction greatly depends on the availability of large annotated video object segmentation benchmarks (e.g., the popular DAVIS 2017 benchmark dataset contains only 150 short video clips, barely enough for training end-to-end deep models from scratch). The alternative approach is to avoid resorting to manual annotations by explicitly defining a “cost” or “energy” function based on a priori considerations on the motion patterns that characterize objects of interest in a video papazoglou2013fast ; 7299114 . However, these approaches do not seem to be on par with deep learning methods.

Recently, generative adversarial networks (GANs) NIPS2014_5423 have become a successful trend in computer vision and machine learning, thanks to their results in advancing the state of the art on image generation to unprecedented levels of accuracy and realism NIPS2015_5773 ; RadfordMC15 ; Zhang_2017_ICCV ; pmlr-v70-arjovsky17a ; Mao_2017_ICCV ; NIPS2017_6797 ; Huang_2017_CVPR . The key idea of GANs is to have two models, a generator and a discriminator, compete with each other in a scenario where the discriminator learns to distinguish between real and fake samples, while the generator learns to produce more and more realistic images. As the models improve in parallel, they learn hierarchies of general feature representations that can be used for multiple tasks, e.g., image classification RadfordMC15 and semantic segmentation Souly_2017_ICCV . These characteristics have demonstrated GANs’ usefulness in training or supporting the training of models from unlabeled data NIPS2016_6125 , rising as one of the most promising paradigms for unsupervised learning111Note the distinction between unsupervised learning (the class of approaches that train machine learning models without annotations) and what is known in the video segmentation literature as unsupervised segmentation (the class of methods that perform segmentation in inference without additional input on object location other than the video frames)..

Given the success of GANs with images, the natural direction of research has been to attempt and extend their applicability to videos, both as a generative approach and as a way to disentangle video dynamics by learning features that leverage the vast amount of unlabeled data available on the web. For example, in a classification scenario such as action recognition, a simple way to employ unlabeled videos is to add an additional class for fake videos (i.e., produced by the generator) and have the discriminator both predict the realism of the input video and identify its class odena2016semi ; NIPS2016_6125 . However, naively extending image generation methods to videos by adding the temporal dimension to convolutional layers may be too simplistic, as it jointly attempts to handle both the spatial component of the video, which describes object and background appearance, and the temporal one, representing object motion and consistency across frames. Building on these considerations, recent generative efforts NIPS2016_6194 ; Saito_2017_ICCV have attempted to factor the latent representation of each video frame into two components that model a time-independent background of the scene and the time-varying foreground elements. We argue that the main limitation of these methods is that both factors are learned by mapping a single point of a latent space (sampled as random noise) to a whole video. This, indeed, over-complicates the generation task as two videos depicting the same scene with different object trajectories or the same trajectory on different scenes are represented as different points in the latent space, although they share a common factor (in the first case the background, in the second one object motion).

In this paper, we tackle both the problem of unsupervised learning for video object segmentation and that of video generation with disentangled background and foreground dynamics, combining both of them into an adversarial framework that guides the discriminator in performing video object segmentation through a self-supervision mechanism, by using ground-truth masks internally synthesized by the generator. In particular, our video generation approach employs two latent spaces (as shown in Fig. 1) to improve the video generation process: 1) a traditional random latent space to model the static visual content of the scene (background), and 2) a trajectory latent space suitable designed to ensure spatio-temporal consistency of generated foreground content. In particular, object motion dynamics are modeled as point trajectories in the second latent space, with each point representing the foreground content in a scene and each latent trajectory ensuring regularity and realism of the generated motion across frames. On top of the traditional adversarial framework, we extend the discriminator architecture in order to perform adversarial dense pixel-wise prediction in videos. In particular, besides the adversarial loss driving the generator/discriminator game, we add loss terms related to the discriminator’s estimation of optical flow (supervised by the output of a state-of-the-art algorithm) and segmentation masks (supervised by the foreground masks computed by the generator) from the generated videos. The three losses encourage the generator to produce realistic videos, while improving representation learning in the discriminator and unlocking the possibility to perform dense video predictions with no manual annotations. Experimentally, we verify that our video generation approach is able to effectively synthesize videos, outperforming existing solutions, especially in motion coherency metrics, thus suggesting that it indeed learns, in an unsupervised way, motion features. We further demonstrate that the features learned by the model’s discriminator can be used for effective unsupervised video object segmentation in different domains and allow for reducing significantly (about 50% less) the number of annotated frames required to achieve the same performance as through traditional supervision. Additionally, we find that the features learned through unsupervised learning encode general appearance and motion cues and can be also employed for global prediction tasks such as video action recognition.

To summarize, the main contributions of this paper are:

• •

We introduce a GAN-based video generation framework able to explicitly model object motion through learning a latent trajectory representation space that enforces spatio-temporal regularity of object motion in the generated videos as well estimating motion cues from the generated content;

• •

We demonstrate that our framework provides a useful means for video object segmentation — known as being an annotation-hungry task — by both employing a trained generator to create synthetic foreground masks and directly integrating dense prediction into the adversarial training procedure;

• •

We verify that our approach is able to successfully learn video features that can be used in a variety of computer vision tasks.

2 Related Work

This paper mainly tackles the problem of unsupervised learning of motion dynamics for pixel-wise dense prediction in a video object segmentation scenario through an adversarial video generation framework. Thus, we first review the recent literature on video object segmentation methods and then focus on video generation approaches.

Video object segmentation is the task of predicting each video pixel either as foreground or background and consequently to segment objects preserving their boundaries. Recent video object segmentation methods can be classified as unsupervised, i.e., methods that perform segmentation in inference without additional input on object location other than the video frames, semi-supervised, i.e., methods that, instead, employ annotations in the first frame for inference, and supervised, i.e. methods that require annotations for every frame or user interaction. In this paper we propose a framework that enables to learn motion features through unsupervised learning, i.e., without using annotations at training time. The learned representations are then employed to train a method for unsupervised video object segmentation; thus, in this section, we will focus on this last class of approaches.

Many of these methods formulate the problem as a spatio-temporal tube (representing motion trajectories) classification task papazoglou2013fast ; Brox:2010 ; 6247883 ; 6126471 ; keuper2015 ; 7298961 . The core idea is to track points or regions over consecutive frames in order to create coherent moving objects, by learning motion/appearance cues. Brox and Malik Brox:2010 propose a pairwise metric on trajectories across frames for clustering trajectories. Analogously, Fragkiadaki et al. 6247883 analyze embedding density variations between spatially contiguous trajectories to perform segmentation. Papazoglou and Ferrari papazoglou2013fast , instead, model motion, appearance and spatial feature density forward and backward in time across frames to extract moving objects. Keuper et al. keuper2015 also track points in consecutive frames and employ a multicut-based approach for trajectory clustering. Wang et al. 7298961 define spatio-temporal saliency, computed as geodesic distance of spatial edges and temporal motion boundaries, and employ such saliency as a prior for segmentation.

Recently, CNN-based methods for unsupervised video object segmentation have been proposed 8099855 ; JainXG17 ; Cae17 ; Tokmakov17 , based on the concept of making the network implicitly learn motion/appearance cues for object trajectory modeling. Most of CNN-based methods use pre-trained segmentation models 8099855 ; JainXG17 ; Cae17 or optical-flow models Tokmakov17 to perform either semi-supervised or unsupervised segmentation. Such methods are, however, profoundly different from the more challenging case of learning to segment moving objects without utilizing, or reducing significantly the need of, labeled data at training time. Thus, although our approach shares the strategy of these methods, i.e., making a CNN learn motion and appearance cues for segmentation, it significantly differs from them in that we learn motion cues in a completely unsupervised way during training.

To carry out unsupervised learning, we leverage the recent success of generative adversarial networks (GANs) NIPS2014_5423 for high-quality image NIPS2015_5773 ; RadfordMC15 ; Zhang_2017_ICCV ; NIPS2015_5773 ; pmlr-v70-arjovsky17a ; Mao_2017_ICCV ; NIPS2017_6797 , video synthesis NIPS2016_6194 ; Saito_2017_ICCV ; Tulyakov_2018_CVPR ; ohnishi2018hierarchical and prediction Jang2018VideoPW ; Vondrick_2017_CVPR , by devising a video generation model that performs, at the same time, pixel-wise dense prediction for video object segmentation. Existing adversarial deep image generation methods have attempted to increase the level of realism NIPS2015_5773 ; RadfordMC15 ; Zhang_2017_ICCV ; NIPS2015_5773 of the generated images, while providing solutions to cope with training stabilization issues pmlr-v70-arjovsky17a ; Mao_2017_ICCV ; NIPS2017_6797 . Most of these methods use a single input latent space to describe all possible visual appearance variations, with some exceptions, such as Stacked GAN Huang_2017_CVPR that, instead, employs multiple latent spaces for different image scales. Works reported in NIPS2016_6194 ; Saito_2017_ICCV ; Tulyakov_2018_CVPR extend the GAN image generation framework to the video generation problem. In particular, the work in NIPS2016_6194 replaces the GAN image generator with a spatio-temporal CNN-based network able to generate, from an input latent space, background and foreground separately, which are then merged into a video clip for the subsequent discriminator. Similarly, the work in Saito_2017_ICCV shares the same philosophy of NIPS2016_6194 with the difference that the input latent space is mapped into a set of subspaces, where each one is used for the generation of a single video frame. However, simply extending the traditional GAN framework to videos fails, because of the time-varying pixel relations that are found in videos due to structured object motion. Although the above generation methods exploit video factorization into a stationary and a temporally varying part, deriving the two components from a single input latent space greatly complicates the task, and needs more training data given that each point in the input latent space corresponds to a complete scene with specific object motion. Furthermore, object motion is only loosely linked to the scene, e.g., the motion of a person walking on two different environments is more or less independent from the specific environment. Thus, the assumption that the two video components are highly inter-correlated (so as to derive both from a single point in the latent space) is too strong, and recently Tulyakov_2018_CVPR performed video generation by disentangling the two video components into different latent spaces. While our approach is in the same spirit of Tulyakov_2018_CVPR , there are several crucial differences both in the problem formulation and in the motivation. In terms of motivation, our approach performs video generation with the main goal to learn motion dynamics in an unsupervised manner, and it is designed in order to implement a self-supervision mechanism for guiding the pixel-wise dense prediction process. In terms of problem formulation, we use two latent spaces as in Tulyakov_2018_CVPR , but our foreground latent space is a multidimensional space from which we sample trajectories, and not a sequence of uncorrelated and isolated points, as done in Tulyakov_2018_CVPR . These trajectories are fed to a recurrent layer that learns suitable temporal embeddings. Thus, we learn a latent representation for motion trajectories, which, as shown in the results, leads to a better spatio-temporal consistency than Tulyakov_2018_CVPR . Along the line of using two separate latent spaces for background and foreground modeling, OhnishiYUH18 propose a video generation approach that first generates optical flow and then converts it to foreground content for integration with a background model. Our approach differs from this work in the following ways:

• •

Although OhnishiYUH18 employs two separate latent spaces for motion and content, single samples are drawn from the two for generating a video; instead, we learn a motion latent space, which more naturally maps to the spatio-temporal nature of motion, and encodes it as input for the generation process in the foreground stream.

• •

It employs optical flow provided by a state-of-the-art algorithm as a condition, as done in standard conditional GANs, in the hierarchical structure of the generator. We, instead, estimate optical flow through the discriminator and use it to supervise, in the form of “self-supervision”, video generation in order to directly encourage a better understanding of motion dynamics.

• •

Lastly, this work does not address the video object segmentation problem, which remains the main objective of our work.

Video GANs have been also adopted for the video prediction task Vondrick017 ; VillegasYHLL17 ; pmlr-v80-jang18a , i.e., for conditioning the generation process of future frames given a set of input frames. While this is a different task than what we here propose, i.e., video generation for supporting spatio-temporal dense segmentation, the way we encode motion can resemble theirs with the key difference that they learn motion dynamics from real data and then draw samples from the learned data distribution; we, instead, learn a latent representation for motion trajectories by enforcing spatio-temporal consistency of generated content. Additionally, appearance and motion are also captured differently; for example, in pmlr-v80-jang18a , they are included by using explicit values as conditions for the generation of future frames, namely, the first frame of the video and some motion attributes; we, instead, model motion and appearance simply as samples drawn from latent spaces and provided as inputs to the generator, and not as quantities estimated by the discriminator.

As mentioned above, we adopt GANs mainly for unsupervised learning of object motion. GANs have been already employed for unsupervised domain adaption Bousmalis_2017_CVPR ; Tzeng_2017_CVPR , image-to-image translation Zhu_2017_ICCV ; Yi_2017_ICCV , for semi-supervised semantic image segmentation Souly_2017_ICCV , as well as for unsupervised feature learning in the image domain doersch2015unsupervised . In the video domain, GANs have been particularly useful for semi-supervised and unsupervised video action recognition NIPS2016_6194 ; Saito_2017_ICCV ; Tulyakov_2018_CVPR or representation learning Mahasseni_2017_CVPR , given their innate ability to learn video dynamics while discriminating between real and fake videos. Unlike existing approaches, our video generation framework supports unsupervised pixel-wise dense prediction for video object segmentation, which is a more complex task that requires learning contextual relations between time-varying pixels. To the best of our knowledge, this is the first attempt to perform adversarial unsupervised video object segmentation, although some GAN-based approaches NIPS2017_6639 perform dense prediction for the optical flow estimation problem. In particular, NIPS2017_6639 proposes a conditional GAN taking an image pair as input and predicting the optical flow. Flow-warped error images both on predicted flow and on ground-truth flow are then computed and a discriminator is trained to distinguish between the real and the fake ones. The network is trained both on labeled and unlabeled data and the adversarial GAN loss is extended with the supervised end-point-error loss, computed on the labeled data. Differently from this work, our dense-prediction network uses only unlabeled data and extends the traditional adversarial loss by including the error made by the discriminator in estimating motion as well as in predicting segmentation maps.

3 Adversarial Framework for Video Generation and Unsupervised Motion Learning

Our adversarial framework for video generation and dense prediction — VOS-GAN — is based on a GAN framework and consists of the following two modules:

• •

a generator, implemented as a hybrid deep CNN-RNN, that receives two inputs: 1) a noise vector from a latent space that models scene background; 2) a sequence of vectors that model foreground motion as a trajectory in another latent space. The output of the generator is a video with its corresponding foreground mask.

• •

a discriminator, implemented as a deep CNN, that receives an input video and 1) predicts whether it is real or not; 2) performs pixel-wise dense prediction to generate an object segmentation map; 3) performs pixel-wise dense prediction to estimate the optical flow between video frames.

The traditional adversarial loss is extended by having the discriminator learn to compute motion-related dense predictions for the input video, thus forcing the generator to produce more realistic motion trajectories. Additionally, this formulation makes the discriminator suitable as a stand-alone model for object segmentation and optical flow estimation.

3.1 Generator Architecture

The architecture of the generator, inspired by the two-stream approach in NIPS2016_6194 , is shown in Fig. 2. Specifically, our generation approach factorizes the process into separate background and foreground generation, on the assumption that a scene is generally stationary and the presence of informative motion can be constrained only to a set of objects of interest in a semi-static environment. However, unlike NIPS2016_6194 and similar to Tulyakov_2018_CVPR , we separate the latent spaces for scene and foreground generation, and explicitly represent the latter as a temporal quantity, thus enforcing a more natural correspondence between the latent input and the frame-by-frame motion output.

Hence, the generator receives two inputs: $z_{C}\in\mathcal{Z}_{C}=\mathbb{R}^{d}$ and $z_{M}=\{z_{M,i}\}_{i=1}^{t}$, with each $z_{M,i}\in\mathcal{Z}_{M}=\mathbb{R}^{d}$. A point $z_{C}$ in the latent space $\mathcal{Z}_{C}$ encodes the general scene to be applied to the output video, and is mainly responsible for driving the background stream of the model. This stream consists of a cascade of transposed convolutions, which gradually increase the spatial dimension of the input in order to obtain a full-scale background image $b(z_{C})$, that is used for all frames in the generated video.

The set of $z_{M,i}$ points from the latent space, $\mathcal{Z}_{M}$, defines the objects motion to be applied in the video. The latent sequence is obtained by sampling the initial and final points and performing a spherical linear interpolation (SLERP Shoemake:1985 ) to compute all intermediate vectors, such that the length of the sequence is equal to the length (in frames) of the generated video. Using an interpolation rather than sampling multiple random points enforces temporal coherency in the foreground latent space. The list of latent points is then encoded through a recurrent neural network (LSTM) in order to provide a single vector (i.e., the LSTM’s final state) summarizing a representation of the whole motion. After a cascade of spatio-temporal convolutions (i.e., with 3D kernels that also span the time dimension), motion features are conditioned on content by concatenation of intermediate features from the background that are replicated along the time dimension. Then, these activations are processed by a $1\times 1\times 1$ convolution and three 3D residual layers he2016deep , with three residual blocks in each layer. After another convolutional layer, this foreground stream outputs a set of frames $f(z_{C},z_{M})$ with foreground content and masks defining motion pixel location $m(z_{C},z_{M})$.

The two streams are finally combined as

[TABLE]

Foreground generation can be directly controlled acting on $z_{M}$. Indeed, varying $z_{M}$ for a fixed value of $z_{C}$ results in videos with the same background and different foreground appearance and motion.

3.2 Discriminator Architecture

The primary goal of the discriminator network is to distinguish between generated and real videos, in order to push the generator towards more realistic outputs. At the same time, we train the discriminator to perform dense pixel-wise predictions of foreground masks and optical flow. These two additional outputs have a twofold objective: 1) they force the discriminator to learn motion-related features, rather than (for example) learn to identify the visual features of objects that are more likely to be part of the foreground (e.g., people, animals, vehicles); 2) they enable the discriminator to perform additional tasks from unlabeled data.

Fig. 3 shows the architecture of the discriminator. The input to the model is a video clip (either real or produced by the generator), that goes first through a series of convolutional and residual layers, encoding the video dynamics into a more compact representation, which in turn is provided as input to two separate streams: 1) a discrimination stream, which applies 3D convolutions to the intermediate representation and then makes a prediction on whether the input video is real or fake; 2) a motion stream, feeding the intermediate representation to a cascade of 3D transposed convolutional and residual layers, which fork at the final layer and return the frame-by-frame foreground segmentation maps (each as a 2D binary map) and optical flow estimations (each as a two-channel 2D map) for the input video.

The discrimination path of the model (i.e., the initial shared convolutional layers and the discrimination stream) follows a standard architecture for video discrimination NIPS2016_6194 , while the motion path, based on transposed convolutions, decodes the video representation in order to provide the two types of dense predictions long2015fully . Formally, we define the outputs returned by the discriminator as $D_{\text{adv}}(x)$, $D_{\text{seg}}(x)$ and $D_{\text{opt}}(x)$, which are, respectively, the probability that the provided input is real, the foreground segmentation maps and the optical flow estimated for the video; input $x$ may be either a real video or the output of the generator.

3.3 Learning Procedure

We jointly train the generator and the discriminator in a GAN framework, with the former trying to maximize the probability that the discriminator predicts fake outputs as real, and the latter trying to minimize the same probability. Additionally, when training the discriminator, we also include loss terms related to the accuracy of the estimated foreground segmentation and optical flow.

The main problem in computing these additional losses is that, while optical flow ground truth for fake videos can be easily obtained by assuming the output of a state-of-the-art algorithm to be sufficiently accurate, there are no video segmentation approaches that provide the needed accuracy. To solve this problem, we propose what is — to the best of our knowledge — the first approach for video object segmentation trained in an unsupervised manner: since the architecture of the generator internally computes the foreground masks of the generated videos, we use those to supervise the prediction of the masks computed by the discriminator. Of course, this kind of self-referential ground-truth will not be very meaningful at first, but will become more and more akin to real ground-truth from real videos as the generator improves.

The discriminator loss is then defined as follows (for the sake of compactness, we will define $z=(z_{C},z_{M})$):

[TABLE]

In the equation above, the first two lines encode the adversarial loss, which pushes the discriminator to return high likelihood scores for real videos and low ones for the generated videos. The third line encodes the loss on foreground segmentation, and it computes the average pixel-wise negative log-likelihood of the predicted segmentation map, using the generator’s foreground mask $m(z)$ as source for correct labels (in our notation, $\text{NLL}_{\text{2D}}(\hat{y},y)$ is the negative log-likelihood of predicted label map $\hat{y}$ given correct label map $y$). The last two lines encode the loss on optical flow estimation, as the squared $L_{2}$ norm between the predicted optical flow and the one calculated on the input video using the $\text{OF}(\cdot)$ function, implemented as per Farneback:2003 . It should be noted that the non-adversarial term related to object segmentation is only computed on the generated videos (hence in a fully-unsupervised manner) for which foreground masks are provided by the generator in Fig. 2, since segmentation ground-truth may not be available for real videos. Optical flow is, instead, computed on both generated and real videos, and it serves to provide supervision to the discriminator (especially to the motion stream shown in Fig. 3) in learning motion cues from real videos. We additionally introduce an $\alpha$ term to control the influence of optical flow and segmentation estimations over the traditional discriminator loss. $\alpha$ is initially set to 1 and then increased by a step $s_{\alpha}$ at each epoch since a specific epoch $E_{\alpha}$. The role of $\alpha$ is specifically to stabilize the GAN training procedure by first letting it learn the general appearance of the scene and, as training goes on, focusing more on learning motion cues.

The generator loss is, more traditionally, defined as:

[TABLE]

In this case, the generator tries to push the discriminator to increase the likelihood of its output being real.

During training, we follow the common approach for GAN training, by sampling real videos (from an existing dataset) and generated videos (from the generator) and alternately optimizing the discriminator and the generator.

4 Performance Analysis

Our system is specifically designed for supporting unsupervised dense and global prediction in videos, but it requires first to train the GAN-based generation model. For this reason, we initially report the video generation performance followed by the performance obtained for video object segmentation (pixel-wise dense prediction) and video action recognition (global prediction).

4.1 Datasets and Metrics

4.1.1 Video Generation

For ease of comparison with publicly-available implementations of state-of-the-art methods, we train our video generation model on two different datasets: “Golf course” videos (over 600,000 video clips, each 32 frames long) from the dataset proposed in NIPS2016_6194 and also employed in Saito_2017_ICCV , and the Weizmann Action database gorelick2007 (93 videos of people performing 9 actions), employed in Tulyakov_2018_CVPR . To have data in the same format, we pre-process the Weizmann Action dataset by splitting the original videos into 32-frame partially-overlapping (i.e., shifted by 5 frames) sequences, resulting in 683 such video clips.

For testing video generation capabilities, we compute qualitative and quantitative results. For the former, we carry out a user study aiming at assessing how the generated videos are perceived by humans in comparison to other GAN-based generative methods, and measure the preference score in percentage. To assess video generation performance quantitatively, we evaluate the appearance and motion of the generated videos, using the following metrics:

• •

Foreground Content Distance (FCD). This score assesses foreground appearance consistency by computing the average $L_{2}$ distance between visual features of foreground objects in two consecutive frames. Feature vectors are extracted from a fully-connected layer of a pre-trained Inception network 43022 , whose input is the bounding box containing the foreground region, defined as the discriminator’s segmentation output.

• •

Fréchet Inception Distance (FID) NIPS2017_7240 suitably adapted to videos as in wang2018vid2vid . FID is a widely adopted metric for implicit generative models, as it has been demonstrated to correlate well with visual quality of generated content. We employ the variant for videos proposed in wang2018vid2vid , that projects generated videos into a feature space corresponding to a specific layer of a pre-trained video recognition CNN. The embedding layer is then considered as a continuous multivariate Gaussian, and consequently mean and covariance are computed for both generated data and real data. The Fréchet distance between these two Gaussians (i.e., Wasserstein-2 distance) is then used to quantify the quality of generated samples. As pre-trained CNN model for feature extraction we use ResNeXt xie2017aggregated ; hara2018can .

• •

Inception score (IS) NIPS2016_6125 is the most adopted quality score in GAN literature. In our case, we compute the Inception score by sampling a random frame from each video in a pool of generated ones.

4.1.2 Video Object Segmentation

The capabilities of the motion stream of our discriminator network (see Fig. 3) for pixel-wise dense prediction are tested on several benchmarks for video object segmentation: DAVIS 2016 and DAVIS 2017 7780454 and SegTrack-v2 Tsai2012 . Each dataset provides accurate pixel-level annotations for all video frames. The employed datasets show diverse features, useful to test the performance of video object segmentation methods in different scenarios. In particular, DAVIS 2017 contains 150 video sequences (50 of which constitute the DAVIS 2016 dataset), and includes challenging examples with occlusion, motion blur and appearance changes. SegTrack-v2 is a dataset containing 14 videos showing camera motion, slow object motion, object/background similarity, non-rigid deformations and articulated objects.

We employ standard metrics 7780454 for measuring video object segmentation performance to ease comparison to state-of-the-art methods, namely: a) region similarity $\mathcal{J}$, computed as pixel-wise intersection over union of the estimated segmentation and the ground-truth mask; b) contour accuracy $\mathcal{F}$, defined as the F1-measure between the contour points of the estimated segmentation mask and the ground-truth mask. For each of the two metrics, we compute the mean value as:

[TABLE]

where $\overline{\mathcal{C}}(S_{i})$ is the average of measure $\mathcal{C}$ (either $\mathcal{J}$ or $\mathcal{F}$) on $S_{i}$ and $R$ is the set of video sequences $S$. Also, for comparison with state of the art methods we compute recall $\mathcal{O}$ and decay $\mathcal{D}$ of the above metrics. The former quantifies the portion of sequences on which a segmentation method scores higher than a threshold, and is defined as:

[TABLE]

where $\mathbb{I}(p)$ is 1 if $p$ is true and 0 otherwise, and $\tau$ = 0.5 in our experiments. Decay measures the performance loss (or gain) over time:

[TABLE]

with $Q_{i}=\{Q_{i}^{1},Q_{i}^{2},Q_{i}^{3},Q_{i}^{4}\}$ being a partition of $S_{i}$ in quartiles.

The results are computed on test or validation sets where available; otherwise, we split the videos into training and test sets with proportions of 70% and 30%. In particular, ablation studies and analysis of the different architectural settings are done on DAVIS 2017 (because of its larger size). We use DAVIS 2016 and SegTrack-v2 for showing generalization capabilities of our approach and for comparison to state-of-the-art methods. All available videos are divided into 32-frame shots and downsampled to fit the input size allowed by the video segmentation network (i.e., the discriminator architecture in Fig. 3), i.e., 64$\times$64, while output segmentation maps are rescaled (through bi-linear interpolation) to ground-truth size for metrics computation.

Accurate evaluation of optical flow estimation is not performed, since our model is designed primarily for performing prediction by self-supervision (i.e., adversarial generation of foreground masks), while optical flow, provided by a state-of-the-art method, is used only to guide the discriminator towards learning motion features from real videos. It should be noted that we did not use any deep learning–based optical flow, e.g., IMSKDB17 , but, instead, employed the traditional approach in Farneback:2003 as it it exploits physical properties of object motion in a purely unsupervised way. This avoids to include any form of “human-supervision” in the segmentation pipeline, making the proposed approach fully unsupervised.

4.1.3 Action Recognition

The capabilities of our model (namely, the discriminator stream in Fig. 3) in learning global motion cues are tested in a video action recognition scenario. In particular, we evaluate its accuracy in classifying actions in videos on on two benchmarks, UCF101 ucf101 (13,320 videos from 101 action categories) and the Weizmann Action database gorelick2007 . We use average classification accuracy (the ratio of correctly-classified samples over the total number of test samples) as metric. In the evaluation on Weizmann Action dataset, due to the small number of video sequences, we perform 10-fold cross validation to average accuracy scores. Performance on video action recognition serves also to provide an additional metrics for quantitative evaluation for video generation.

4.2 Training Settings

The architectures of the generator and discriminator networks in terms of kernel sizes, padding, stride, activation functions and use of batch normalization ioffe2015batch are given, respectively, in Tables 1 and 2.

In the video generation and segmentation experiments, we performed gradient-descent using ADAM, with an initial learning rate of 0.0002, $\beta_{1}=0.5$, $\beta_{2}=0.999$ and batch size of 16. For video generator training we used $\alpha=1$, $s_{\alpha}=0.2$ and $E_{\alpha}=2$ (values set empirically). We trained the video generation models for 25 epochs and the video segmentation ones for 200 epochs. For video action recognition, we used SGD with momentum and dampening of 0.9, weight decay of 0.001 and learning rate of 0.1, reduced by a factor of 10 when no improvement in validation accuracy occurred for 10 epochs. Batch size was 128 and the number of epochs was 130.

4.3 Video Generation

4.3.1 Qualitative evaluation

To evaluate qualitatively our video generation approach, we used Amazon Mechanical Turk (MTurk) in order to measure how generated videos are perceived by humans. Our generated videos are compared to those synthesized by VGAN NIPS2016_6194 , TGAN Saito_2017_ICCV , MoCoGAN Tulyakov_2018_CVPR .

On MTurk, each job is created by randomly choosing two of the models under comparison and generating a 16-video batch from each of them; workers are then asked to choose which of the two sets of videos look more realistic. All workers have to provide answers for all the generated batches. We consider only the answers by workers with a lifetime Human Intelligent Task rate over than 90%. The achieved results are reported in Tab. 3 and show how our method generates more visually-plausible “golf” videos compared to VGAN, TGAN, and MoCoGAN. MoCoGAN, instead, outperforms our approach on the Weizmann Action dataset. It should be noted that MoCoGAN, differently from our approach, employs two discriminators — one for single frames and one for the whole video — and given the low scene variability in the Weizmann Action dataset, the frame generator is able to produce high-quality frames. In our approach, instead, background and foreground are integrated in videos and the discriminator is mainly trained (see Eq. 2) to capture motion features rather than visual appearance. A direct consequence from our learning schema is that VOS-GAN requires many samples to generate good resolution videos, and this explains why performance is better in the “Golf course” dataset (about 600,000 videos) than in Weizmann Action dataset (less than 100 videos). However, the capability of our approach to learn better motion cues than MoCoGAN is demonstrated by the results obtained on quantitative generation evaluation (see Sect. 4.3.2) and in action recognition task (see Sect. 4.5).

Samples of generated videos for VGAN, TGAN, MoCoGAN and our method are shown in Fig. 4, while comparisons with MoCoGAN on the Weizmann Action Dataset are shown in Fig. 5.

4.3.2 Quantitative evaluation

Quantitative evaluation of video generation performance is carried out by measuring FCD, FID and IS scores on 20 sets of 50,000 videos generated by the compared models trained on “Golf course” of NIPS2016_6194 , and on 20 sets of 500 videos generated on the Weizmann Action Dataset. Results are computed in terms of mean and standard deviation of each metrics over the sets of generated samples.

Firstly, we perform an ablation study on “Golf course” to understand how our GAN design choices affect the quality of the generated videos, by evaluating the above-mentioned metrics when each proposed term is included in the model. In particular, we computed the performance of VOS-GAN excluding all components (i.e., loss on segmentation and optical flow and replacing the RNN-SLERP based trajectory latent space modeling with a random latent space) — VOS-GAN (baseline) — and when gradually including individual components. The results of our ablation study are given in Tab. 4. Both loss terms (on segmentation and optical flow) contribute to the model’s accuracy: optical flow has the largest impact, likely because foreground regions usually correspond to clusters of oriented vectors in the optical flow, hence learning to compute it accurately is also informative from the segmentation perspective. Both SLERP and LSTM also contribute significantly to generating visually-plausible videos by modeling better motion as demonstrated by the increase in FID and FCD metrics.

Tab. 5 shows the comparison with existing methods on the “Golf course” and on the Weizmann Action datasets. The results show that VOS-GAN outperforms VGAN, TGAN and MoCOGAN on the three metrics on “Golf course”, while MoCoGAN is better than the proposed approach, on two out of the three adopted metrics, on the Weizmann Action Dataset. The reasons behind the different behavior of VOS-GAN and MoCoGAN on the two datasets are similar to those mentioned above in the qualitative analysis: 1) The Weizmann action dataset is characterized by few videos with constrained motion and background, thus the variability among frames is low and MoCoGAN’s image generator is able to model scene appearance with good quality. On the contrary, our approach is able to learn motion (as demonstrated by the results in action recognition given below), but it does not receive enough training data (given the size of the dataset) for learning well the scene; 2) “Golf course” contains many video sequences with high variability in terms of motion and appearance — variability that MoCoGAN is not able to learn. However, VOS-GAN confirms its capabilities to learn better motion, indeed, it achieves a higher FID than MoCoGAN, substantiating also our previous claims on the different performance of MoCoGAN on the two employed datasets.

4.4 Video Object Segmentation

The first part of this evaluation aims at investigating the contribution of adversarial training for video object segmentation, by assessing the quality of the foreground maps computed by our video object segmentation approach in four different settings:

• •

synthetic: we use the motion stream, i.e. segmentation subnetwork, from the architecture of our discriminator and train it from scratch with the foreground masks synthesized by our generator trained on “Golf course” video dataset;

• •

adversarial: we use segmentation subnetwork from the discriminator of VOS-GAN trained on the “Golf dataset”;

• •

fine-tuned synthetic: the segmentation network trained in the synthetic modality is then fine-tuned on ground-truth masks of the benchmark datasets’ (DAVIS 2016/2017 and SegTrack-v2) training sets;

• •

fine-tuned adversarial: analogously, we use the segmentation model from the adversarial training scenario and fine-tune it on real segmentation masks.

As baseline we select the segmentation network of our GAN model trained, in a supervised way (i.e., using annotations at training time), from scratch on the training splits of the employed datasets. As metrics, we use mean values for region similarity and contour accuracy, i.e., $\mathcal{M}_{\mathcal{J}}$ and $\mathcal{M}_{\mathcal{F}}$.

Tab. 6 shows the obtained performance: our segmentation network obtains fair results even when trained without labeled data (first two rows of the table). There is a significant gap between region similarity $\mathcal{M}_{\mathcal{J}}$ (higher) and contour accuracy $\mathcal{M}_{\mathcal{F}}$ (lower), meaning that the model performs better in motion localization than in object segmentation. Results also show that our adversarial model fine-tuned on real data (fourth line) significantly outperforms, by a margin of about 10%, our baseline (fifth line), indicating that pre-training, with self-supervision, the segmentation model obtains better accuracy. Pre-training the segmentation model with synthetic videos before fine-tuning on real data (third line) leads also to increased (about 2%) performance than the baseline. Thus, our video generation model acts as a data distillation approach RadosavovicDGGH18 , i.e., through injecting synthetic video masks into the training procedure, our approach combines predictions from arbitrary transformations (enabled by the generator) of unlabeled videos. We further quantify the percentage of training examples needed to obtain satisfactory results both for our baseline and the adversarial fine-tuned model. The results on the three employed datasets are given in Fig. 7 and indicate that our adversarial model requires fewer annotations to reach satisfactory performance than our baseline, which requires at least twice as many annotations. The obtained results, thus, demonstrate that our approach not only yields better segmentation results, but it also allows us to reduce the needed training data. Fine-tuning the segmentation subnetwork on real data (third and fourth rows in Tab. 6) has the effect to focus better on target objects and to cope with camera motion (that is not considered by the video generation model), as demonstrated by the performance improvement reported in rows three and four. This effect is also shown in Fig. 6, that shows that the segmentation maps predicted by our purely-unsupervised method highlight object motion, but object contours are not accurate. Training the segmentation network in an adversarial framework (rather than directly on the fake maps) leads to a significant performance increase, both with and without fine-tuning on real data. Fine-tuning the whole GAN video generation on the three employed video benchmarks, instead, does not yield a significant performance improvement, but relaxes the requirements on available manual annotations.

On the other hand, the advantage of using less annotated data comes at the cost of training the GAN for video generation using unlabeled videos. For this reason, we quantify how the amount of unlabeled videos for video generation affects video object segmentation performance. This evaluation is carried out on the adversarial configuration for video object segmentation, and uses region similarity and contour accuracy as metrics. Fig. 8 shows an almost linear dependence between video object segmentation accuracy and the number of unlabeled videos used for generation by our VOS-GAN, suggesting that using more unlabeled videos in training the adversarial models may ideally lead to better results as the approach effectively learns motion dynamics.

Since our proposed adversarial framework is used for unsupervised pre-training video segmentation models, we compare it, on DAVIS 2017, to: a) two approaches that perform optical flow estimation (both non-learning-based Farneback:2003 and learning-based IMSKDB17 optical flow) and assign foreground pixels based on the empirically-optimal threshold over flow magnitude; b) the approach proposed in VondrickSFGM18 , relaxing its constraint of having the first annotated label at inference time by replacing the needed frame with a binary image obtained by thresholding the optical flow between the first two frames, as done in the previous methods. We also compare the performance of our adversarial learning method fine-tuned on DAVIS 2017 to our discriminator network trained adversarially to estimate only optical flow and then fine-tuned on the video segmentation task (”OF estimation + FT” in Table 7). Results, in terms of $M_{\mathcal{J}}$ and $M_{\mathcal{F}}$, are given in Table 7. Our purely adversarial unsupervised approach learns better features (segmentation performance higher by about 8-10%) than both optical flow methods and VondrickSFGM18 . Also, when fine-tuned, our approach gives better segmentation performance than our generation model trained without segmentation mask self-supervision (i.e., the GAN model using only optical flow) suggesting that, indeed, our self-supervision strategy through synthetic masks during video generation learns better representations for the segmentation task.

We finally compare our approach to state-of-the-art unsupervised video object segmentation methods (i.e., not using any annotations at test time), namely, BMVC.28.21 , papazoglou2013fast , Tokmakov17 , JainXG17 , 8100267 on DAVIS 2017; BMVC.28.21 , papazoglou2013fast , Jang2018VideoPW and 8100267 on DAVIS 2016; papazoglou2013fast , 6248065 , BMVC2015_186 , HallerL17 on SegTrack-v2. For a fair comparison we did not test any of the existing code, but just report performance from the literature, specifically from Man+18b for DAVIS 2016, from 8100267 for DAVIS 2017 and from HallerL17 for SegTrack-v2. The achieved results are given in Tab. 8 and show that our approach is almost on par with the best performing method on the DAVIS benchmarks and outperforms existing methods on SegTrack-v2. Most importantly, according to Fig. 7 and results in Table 8, it leads to performance enhancement requiring less annotated data.

4.5 Video Action Recognition

Finally, in order to evaluate the representational power of features learned by the methods under comparison, we employ the models’ discriminators (after the initial training for video generation) to perform action recognition on the UCF101 and Weizmann Action datasets. Furthermore, results on video action recognition serve also as an additional means to verify video generation performance especially in learning motion dynamics.

This analysis is carried out on two different training settings: a) transfer learning: each model’s discriminator is used as a feature extractor by removing the final real/fake output layer and training a linear classifier on the exposed features; b) fine-tuning: the real/fake output layer is replaced with a softmax layer and the whole discriminator is fine-tuned for classification.

In order to make the comparison fair and consistent with the initial adversarial training, we separately evaluate the models originally trained on “Golf course” from those trained on Weizmann Action dataset: hence, Tab. 9 and 10 report the classification accuracy on UCF101 and Weizmann Action dataset by comparing the discriminators originally trained for video generation on “Golf course”; similarly, Tab. 11 and 12 report the classification accuracy by the discriminators originally trained on Weizmann Action dataset. The results show that our VOS-GAN discriminator outperforms VGAN, TGAN and MoCoGAN in all the considered scenarios. Note that all models quickly overfit on Weizmann Action dataset, due to the small size of the dataset, which likely explains why transfer learning performs better than fine-tuning.

5 Conclusion

We propose a framework for unsupervised learning of motion cues in videos. It is based on an adversarial video generation approach — VOS-GAN — that disentangles background and foreground dynamics. Motion generation is improved by learning a suitable object motion latent space and by controlling the discrimination process with pixelwise dense prediction of moving objects through a self-supervision mechanism, in which segmentation masks are internally synthesized by the generator.

Extensive experimental evaluation showed that our VOS-GAN outperforms existing video generation methods, namely, VGAN NIPS2016_6194 , TGAN Saito_2017_ICCV , MoCoGAN Tulyakov_2018_CVPR , especially in modeling object motion. The capability of our approach to better model object motion is further demonstrated by the fact that the learned (in an unsupervised way) features can be used for effective video object segmentation and video action recognition tasks.

Finally, it has to be noted that, although we introduce a new segmentation network as part of the adversarial framework, it is quite general and can easily integrated into any segmentation model, by adding the optical flow dense prediction stream and using masks synthesized by the generator for supervision.