A Multi-Pass GAN for Fluid Flow Super-Resolution

TL;DR

This paper introduces a multi-pass GAN approach for volumetric super-resolution, enabling high up-scaling factors by decomposing the problem into orthogonal slices and leveraging spatio-temporal supervision.

Contribution

The novel multi-pass GAN method allows volumetric up-scaling by a factor of eight, overcoming previous limitations in neural network size and training complexity.

Findings

Achieved volumetric up-scaling by a factor of eight.

Demonstrated generality across complex 3D datasets.

Compared favorably to previous super-resolution methods.

Abstract

We propose a novel method to up-sample volumetric functions with generative neural networks using several orthogonal passes. Our method decomposes generative problems on Cartesian field functions into multiple smaller sub-problems that can be learned more efficiently. Specifically, we utilize two separate generative adversarial networks: the first one up-scales slices which are parallel to the XY-plane, whereas the second one refines the whole volume along the Z-axis working on slices in the YZ-plane. In this way, we obtain full coverage for the 3D target function and can leverage spatio-temporal supervision with a set of discriminators. Additionally, we demonstrate that our method can be combined with curriculum learning and progressive growing approaches. We arrive at a first method that can up-sample volumes by a factor of eight along each dimension, i.e., increasing the number of degrees of freedom by 512. Large volumetric up-scaling factors such as this one have previously not been attainable as the required number of weights in the neural networks renders adversarial training runs prohibitively difficult. We demonstrate the generality of our trained networks with a series of comparisons to previous work, a variety of complex 3D results, and an analysis of the resulting performance.

Tables

Table 1. Table 1. Evaluation of Performance (avg. for 100 frames)

	Regular solver		Multi-grid solver		Multi-pass GAN
Resolution	256	512	256	512	256	512
Computation time (s)	116.90	1376.54	41.90	463.81	10.14	59.65

Table 2. Table 2. Architecture of G 1 , 8 ⁣ × subscript 𝐺 1 8 G_{1,8\times}

Layer	Output Shape
LR Input	16$\times$16$\times$4
ResBlock. 3$\times$3	16$\times$16$\times$16
ResBlock. 3$\times$3	16$\times$16$\times$64
Avg-Depool	32$\times$32$\times$64
ResBlock. 3$\times$3	32$\times$32$\times$128
ResBlock. 3$\times$3	32$\times$32$\times$64
Avg-Depool	64$\times$64$\times$64
ResBlock. 3$\times$3	64$\times$64$\times$64
ResBlock. 3$\times$3	64$\times$64$\times$32
Avg-Depool	128$\times$128$\times$32
ResBlock. 3$\times$3	128$\times$128$\times$32
ResBlock. 3$\times$3	128$\times$128$\times$16
Conv. 1$\times$1	128$\times$128$\times$1
total parameters	550k

Table 3. Table 4. Architecture of G 2 , 8 ⁣ × subscript 𝐺 2 8 G_{2,8\times}

Layer	Output Shape
LR Input	16$\times$16$\times k_i$
ResBlock. 5$\times$5	64$\times$64$\times$12
ResBlock. 5$\times$5	64$\times$64$\times$48
ResBlock. 5$\times$5	64$\times$64$\times$96
ResBlock. 5$\times$5	64$\times$64$\times$48
ResBlock. 5$\times$5	64$\times$64$\times$48
ResBlock. 5$\times$5	64$\times$64$\times$24
ResBlock. 5$\times$5	64$\times$64$\times$24
ResBlock. 5$\times$5	64$\times$64$\times$12
Conv. 1x1	64$\times$64$\times$1
total parameters	774k

Table 4. Table 6. Parameters for progressive growing

Networks	blend	decay	training	time	batch &
	iter.	iter.	parameters		tile size
$G_{1,8\times }$, $D_{1s,8\times }$	120k	160k	Adam: $[\eta =0.0005$, ${\beta }_1=0.0$,	8	16,
& $D_{1t,8\times }$			${\beta }_2=0.99$, $ϵ={10}^{-8}],$	days	${16}^2$
			${\lambda }_1=20,{\lambda }_W=10$
$G_{2,8\times }$, $D_{2s,8\times }$	-	600k	Adam: $[\eta =0.0005$, ${\beta }_1=0.0$,	5	16,
& $D_{2t,8\times }$			${\beta }_2=0.99$, $ϵ={10}^{-8}]$,	days	${64}^2$
			${\lambda }_1=20,{\lambda }_W=10$

Equations

$$\begin{array}[]{@{}ll}\begin{aligned} \min_{G}\max_{D}V(D,G))=&\mathbb{E}_{y\sim p_{\text{y}}(y)}[\log D(y)]+\mathbb{E}_{x\sim p_{\text{x}}(x)}[\log(1-D(G(x)))]\\ =&\sum_{m}[\log D(y_{m})]+\sum_{n}[\log(1-D(G(x_{n})))],\end{aligned}\end{array}$$

$$\begin{array}[]{@{}ll}\begin{aligned} \mathcal{L}_{G_{1}}(D_{1}s,D_{1}t,G_{1})=&-\sum_{n}[\log D_{1}s(x,G_{1}(x))]-\sum_{n}[\log D_{1}t\Big{(}\widetilde{G_{1}}_{\mathcal{A}}\Big{(}\widetilde{X}\Big{)}\Big{)}]\\ &+\sum_{n,j}\lambda_{f}^{j}\left\|F^{j}(G_{1}(x))-F^{j}({y})\right\|_{2}^{2}+\lambda_{L_{1}}\sum_{n}\left\|G_{1}(x)-{y}\right\|_{1}\end{aligned}\end{array}$$

$$\begin{array}[]{@{}ll}\begin{aligned} \mathcal{L}_{D_{1}t}(D_{1}t,G_{1})=&-\sum_{m}[\log D_{1}t(\widetilde{Y}_{\mathcal{A}})]-\sum_{n}[\log\Big{(}1-D_{1}t\Big{(}\widetilde{G_{1}}_{\mathcal{A}}\Big{(}\widetilde{X}\Big{)}\Big{)}\Big{)}]\\ \mathcal{L}_{D_{1}s}(D_{1}s,G_{1})=&-\sum_{m}[\log D_{1}s(x,y)]-\sum_{n}[\log(1-D_{1}s(x,G_{1}(x)))]{,}\end{aligned}\end{array}$$

$$\begin{array}[]{@{}ll}\begin{aligned} \mathcal{L}_{D_{1t,8\times}}(D_{1t,8\times},G_{1,8\times})=&-\sum_{n}[D_{1t,8\times}(\widetilde{Y}_{\mathcal{A}})]+\sum_{n}[D_{1t,8\times}\Big{(}\widetilde{G_{1,8\times}}_{\mathcal{A}}\Big{(}\widetilde{X}\Big{)}\Big{)}]\\ &+\lambda_{W}\sum_{n}[(||\nabla_{\hat{Y}_{\mathcal{A}}}D(\hat{Y}_{\mathcal{A}})||_{2}-1)^{2}]\end{aligned}\end{array}$$

$$\begin{array}[]{@{}ll}\begin{aligned} \mathcal{L}_{D_{1s,8\times}}(D_{1s,8\times},G_{1,8\times})=&-\sum_{m}[D_{1s,8\times}(x,y)]+\sum_{m}[D_{1s,8\times}(x,{G_{1,8\times}}(x))]\\ &+\lambda_{W}\sum_{m}[(||\nabla_{\hat{y}}D(\hat{y})||_{2}-1)^{2}]\end{aligned}\end{array}$$

$$\begin{array}[]{@{}ll}\begin{aligned} \mathcal{L}_{G_{1,8\times}}(D_{1t,8\times},D_{1s,8\times},G_{1,8\times})=&-\sum_{n}[D_{1t,8\times}\Big{(}\widetilde{G_{1,8\times}}_{\mathcal{A}}\Big{(}\widetilde{X}\Big{)}\Big{)}]-\sum_{m}[D_{1s,8\times}(x,{G_{1,8\times}}(x))]\\ &+\lambda_{L1}\sum_{m}[||(G_{1,8\times}(x)-y)||_{1}],\end{aligned}\end{array}$$

Full text

physics-based deep learning, generative models, computer animation, fluid simulation

††conference: SCA: Symposium on Computer Animation; July 26–28, 2019; Los Angeles, CA††ccs: Computing methodologies Neural networks††ccs: Computing methodologies Physical simulation

1. Introduction

Deep learning-based generative models, in particular generative adversarial networks (GANs) (Goodfellow et al., 2014), are widely used for synthesis-related learning tasks. GANs contain a generator, which can be trained to achieve a specific task, and a discriminator network that efficiently represents a learned loss function. The discriminator drives the generated data distribution to be close to the reference data distribution. There is no explicit content-based loss function for training the generator, and as a consequence, GANs perform very well for problems with multiple solutions, i.e., multimodal problems, such as super-resolution (SR) tasks. Most of the generative models focus on two-dimensional (2D) data, such as images, as processing higher dimensional data quickly becomes prohibitively expensive. However, as our world is full of three-dimensional (3D) objects, learning 3D relationships is an important and challenging task. In the following, we will propose a volumetric training pipeline called multi-pass GAN. Our method breaks down the generative task to infer large Cartesian density functions in space and time into multiple orthogonal passes, each of which represents a much more manageable inference problem.

We target a fluid flow setting where the multi-pass GAN solely uses 2D slices for training but is able to learn 3D relationships by processing the data multiple times. This is especially beneficial for training: finding a stable minimum for the coupled non-linear optimization of a GAN training grows in complexity with the number of variables to train. Our multi-pass network significantly reduces the number of variables and in this way stabilizes the training, which in turn makes it possible to train networks with complexities that would be infeasible with traditional approaches. Our approach can potentially be used in a variety of 3D training cases, such as 3D data SR, classification, or data synthesis. In the following, we demonstrate its capabilities for fluid flow SR, more specifically for buoyant smoke SR, in conjunction with the tempoGAN (Xie et al., 2018) and the progressive growing of GANs (Wang et al., 2018) architectures.

For SR problems, there are inherent challenges. First of all, most SR algorithms focus on $4\times$ 2D data SR. The large size of 3D volumetric data makes it difficult to directly employ 2D SR algorithms. Existing 3D SR algorithms, such as tempoGAN, can only be applied for relatively small up-scaling factors, whereas larger factors become utterly expensive and difficult to train. Larger factors directly imply that the unknown function to be learned, i.e., the detail missing in the low-resolution (LR) input, contains content with higher frequencies. As such, the content is more complex to represent and it is more challenging to ensure its temporal coherence. A popular direction within the field of GANs is the progressive growing approach (Wang et al., 2018), which can achieve large up-scaling factors. However, the progressive growing of GANs has only been demonstrated for 2D content, as 3D volumetric data requires processing data and representing functions that have orders of magnitude more degrees of freedom. This is one of the key motivations for our approach: leveraging multi-pass GANs to decrease the resource consumption for training in order to arrive at robust learning of 3D functions.

Simulating fluid flows at high-resolution (HR) is an inherently challenging process: the number of underlying numerical computations typically increases super-linearly when the discretization is refined. In addition to the spatial degrees of freedom, the temporal axis likewise needs to be refined to reduce numerical errors from time integration. While methods like synthetic turbulence methods (Kim et al., 2008) typically rely on a finely resolved advection scheme, the deep learning-based tempoGAN algorithm arrived at a frame-by-frame super-resolution scheme that circumvents the increase in computational complexity by working on independent frames of volumetric densities via a spatio-temporal discriminator supervision. Similar to tempoGAN, our multi-pass GAN also aims at fluid simulation SR, but targets two extra goals: reducing the unstable and intensive computations of the 3D training process and increasing the range of possible up-scaling factors. To achieve those targets, multi-pass GAN adopts a divide-and-conquer strategy for 3D training. Instead of training with 3D volume data directly, multi-pass GAN divides the main task, i.e., learning the 3D relationship between LR and HR, into two smaller sub-tasks by learning 2D relationships between LR and HR of two orthogonal planes. In a series of experiments, we will demonstrate that this strategy strongly reduces the required computational resources and stabilizes the coupled non-linear optimization of GANs.

To summarize, the main contributions of our work are:

• •

a novel multi-pass approach for robust network training with time sequences of 3D volume data,

• •

extending progressive training approaches for multi-pass GANs, and

• •

combining progressive growing with temporal discriminators.

In this way, we arrive at a first method for learning $8\times$ up-scaling of 3D fluid simulations with deep neural networks.

2. Related Work

Large disparities between HR screens in household devices and LR data have driven a large number of advances in SR algorithms, which also find applications in other fields like surveillance (Zhang et al., 2010) and medical image processing(Kennedy et al., 2006). As deep learning-based methods were shown to generate state-of-the-art results for SR problems, we focus on this class of algorithms in the following. At first, convolutional neural networks (CNNs) were used in SRCNN (Dong et al., 2014), and were shown to achieve better performance than traditional SR methods (Timofte et al., 2014). Theoretically, deeper CNNs can solve more arduous tasks, but overly deep CNNs typically cause vanishing gradient problems. Here, CNNs with skip connections, such as residual networks (He et al., 2016), were applied to alleviate this problem(Lim et al., 2017). Instead of generating images directly, VDSR  (Kim et al., 2016) and ProSR (Wang et al., 2018) apply the CNNs to learn the residual content, which largely decreases the workload for the networks and improves the quality of the results. In our structure, we also applied residual learning to improve performance and reduce resource consumption.

Beyond architectures, choosing the right loss function is similarly crucial. Pixel-wise differences are the most basic ones and are widely-used, e.g., in DRRN (Tai et al., 2017), LapSRN (Lai et al., 2017) and SRDensseNet (Tong et al., 2017). These methods achieve reasonable results and improve the performance in terms of the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM). However, minimizing vector norm differences typically blurs out small scale details and features, which leads to high perceptual errors. With the help of a pre-trained VGG network, perceptual quality can be improved by minimizing differences of feature maps and styles between generated results and references (Johnson et al., 2016). However, VGG is trained with large amounts of labeled 2D image data, and, for fluid-related data-sets, no pre-trained networks exist. Alternatively, GANs (Goodfellow et al., 2014) can improve perceptual quality significantly. In addition to the target model as generator, a discriminator is trained to distinguish generated results from references, which guides the generator to output more realistic results. Instead of minimizing pixel-wise differences, GANs aim to generate results that follow the underlying data distribution of the training data. As such, GANs outperform traditional loss functions for multimodal problems (Ledig et al., 2017). Original GANs, e.g., DC-GAN(Radford et al., 2016), adopt an unconditional structure, i.e., generators only process randomly-initialized vectors as inputs. By adding a conditional input, conditional GANs (CGAN) (Mirza and Osindero, 2014) learn the relationship between the condition and the target. Therefore, this input can be used to control the generated output. By using conditional adversarial training, EnhanceNet (Sajjadi et al., 2017) and SRGAN (Ledig et al., 2017) arrive at detailed results for natural images. tempoGAN (Xie et al., 2018) uses the velocity of the fluid simulation as a conditional input. This allows the network to learn the underlying physics and improves the resulting quality.

For SR problems, larger up-scale factors typically lead to substantially harder learning tasks and potentially more unstable training processes for recovering the details of the reference. Hence, most previous methods do not consider up-scaling factors larger than $2$ or $4$. However, ProSR (Wang et al., 2018) and progressive growing GANs (Karras et al., 2017) train the network in a staged manner, step-by-step. This makes it possible to generate HR results with up-scaling factors of $8$ and above. Here, we also apply the progressive training pipeline to show that our method can be applied to complex 3D flow data.

The works discussed above focus on single image SR and do not take temporal coherence into account, which, however, is crucial for sequential and animation-related SR applications. One solution to keep results temporally coherent is using a sequence as input data and generating an output sequence all at once (Saito et al., 2017; Yu et al., 2017). This improves the temporal relationships, but frames can only be generated sequentially, and an extension to 3D data would require processing 4D volumes. For models that generate individual frames independently, an alternative solution is to use additional temporal losses. E.g., L2 losses were used to minimize differences between warped nearby frames (Ruder et al., 2016; Chen et al., 2017). However, this typically leads to sub-optimal perceptual quality. Beyond the L2 loss, specifically designed loss terms can be used to penalize discontinuities between adjacent frames (Bhattacharjee and Das, 2017). Instead of using explicit loss functions to restrict temporal relationships, temporal discriminators (Xie et al., 2018; Chu et al., 2018) can automatically supervise w.r.t. temporal aspects. They also improve the perceptual quality of the generated sequences. In our work, a variant of such a temporal discriminator architecture is used.

To arrive at realistic fluid simulations, methods typically aim for solving the Navier-Stokes (NS) equations and inviscid Euler equations as efficiently and precisely as possible. Based on the first stable single-phase fluid simulation algorithm (Stam, 1999) for computer animation, a variety of extensions was proposed, e.g., more accurate advection schemes (Kim et al., 2005; Selle et al., 2008), coupling between fluid and solids (Batty et al., 2007; Teng et al., 2016), and fluid detail synthesis based on turbulence models (Kim et al., 2008; Narain et al., 2008). In recent years, deep learning methods also attracted attention in the field of computer graphics, such as rendering (Bako et al., 2017; Chaitanya et al., 2017), volume illumination (Kallweit et al., 2017), and character control (Peng et al., 2017). Since fluid simulations are very time-consuming, CNNs are also applied to estimate parts of numerical calculations, e.g., a CNN-based PDE solver (Farimani et al., 2017; Long et al., 2017; Tompson et al., 2017) was proposed, as well as a fast SPH method using regression forests for velocity prediction (Jeong et al., 2015). CNNs can learn the relationships between control parameters and simulations of interactive liquids (Prantl et al., 2017). Neural networks are also used for splash generation (Um et al., 2018) and descriptor-encoding of pre-computed fluid patches for fluid synthesis (Chu and Thuerey, 2017). Generative networks were additionally trained to pre-compute solution spaces for smoke and liquid flows (Kim et al., 2018). In contrast to these methods, we aim for particularly large up-scaling models that could not be realized with the aforementioned methods.

3. Method

Training neural networks with a large number of weights, as it is usually required for 3D problems, is inherently expensive. In an adversarial setting training large networks is particularly challenging, as stability is threatened by the size of the volumetric data. This in turn often leads to small mini-batch sizes and correspondingly unreliable gradients. In the following, we will explain how these difficulties of 3D GANs can be circumvented with our multi-pass GAN approach.

3.1. Multi-Pass GAN

To reduce the dimensionality of the learning problem, we decompose the 3D data into a stack of 2D slices and train our networks on those slices using convolutions along two spatial dimensions. As shown in Fig. 3, there are three different ways to split a 3D volume into stacked slices. If we split the 3D volume along the $X$-axis, we can retrieve motion information in the $YZ$-plane for every $YZ$-slice, while splitting along the $Y$-axis yields information about the $XZ$-planes.

In order to reconstruct the target 3D function when training with 2D slices, we perform the steps illustrated in Fig. 2. First, we have to ensure that there is no bias in the inference, i.e., all spatial dimensions are processed in the same manner. Thus, for the LR volume $x$ with resolution ($a,b,c$), we linearly interpolate along the $Z$-axis to increase the resolution from ($a,b,c$) to ($a,b,4c$). We then split the new volume along the $Z$-axis to obtain $4c$ slices of size ($a,b$). Here, we train a 2D network $G_{1}$ to up-scale these ($a,b$) slices to the target size of ($4a,4b$). This first generator produces HR volumes, but as it only works in the $XY$-plane, the results are only temporally coherent in this plane, and the model would still generate stripe-like artifacts in the $XZ$- or $YZ$-planes of the volumes. In order to allow the network to evaluate motion information along the $Y-$ and $Z$-dimensions, we use a second generator network $G_{2}$ to refine the details of the volume along the $Y$- and $Z$-axes. Here, we cut the volume along the $X$-axis to obtain $4a$ slices of size ($4b,4c$). $G_{2}$ is then trained to refine those slices and drive the distribution of the output closer to the target function, i.e., an HR simulation. Note that we have found it beneficial to not change the resolution with $G_{2}$, but instead let it process the full resolution data. In summary, combining the interpolation along $Z$, $G_{1}$ and $G_{2}$ jointly up-scale 3D volumes from ($a,b,c$) to ($4a,4b,4c$) and ensure that all voxels are consistently processed by the two generator networks.

Theoretically, as soon as we have motion from two orthogonal directions, we are able to infer the full 3D motion. Thus, we train $G_{1}$ and $G_{2}$ with $XY$- and $YZ$-slices, respectively, such that they – in combination – obtain full motion information. To verify that this is sufficient, we also employed a third network $G_{3}$ to refine the $XZ$-plane with an additional pass through training on $XZ$-slices. Results from the first, second, and this optional third network can be found in Fig. 4, where each row illustrates examples of a different slicing direction. In the first row, for example, the volume is cut along the $Z$-axis, which $G_{1}$ operates on. When comparing the second column to the third one, it is notable that there is not much improvement for the $XY$-plane. However, looking at the second row which is a sample of the $YZ$-plane that $G_{2}$ focuses on, it is clear that the artifacts generated by the linear interpolation are removed and the spatial coherence along the $Z$-axis is substantially improved. Even the slices in the $XZ$-plane are enhanced, as shown in the third row of Fig. 4. The fourth column in the Fig. 4 shows that $G_{3}$ leads to minimal changes along any of the axes, which matches our assumption that this additional direction of slicing is redundant. Therefore, we focus on two generator networks for our final structure, $G_{1},G_{2}$, while $G_{3}$ is not applied.

4. Multi-pass GAN with Temporal Coherence

In general, the super-resolution problem is a multimodal task and here our goal is to additionally obtain physically plausible results with a high perceptual quality. Thus, we employ adversarial training to train both $G_{1}$ and $G_{2}$. The full optimization problem for regular GANs can be formulated as (Goodfellow et al., 2014):

[TABLE]

where $m,n$ denote the number of reference samples $y$ and drawn inputs $x$, respectively. $y\sim p_{\text{y}}(y)$ states that reference data is sampled from the probability distribution $p_{\text{y}}$.

Our goal is similar to the one from the tempoGAN network (Xie et al., 2018). To illustrate the advantages of our approach, we will first employ our network in a tempoGAN pipeline for $4\times$ fluid SR in this section before targeting large up-scale factors in conjunction with a progressive growing scheme in Sec. 5.

4.1. Network Structure and Loss Function

Unlike regular GANs, which only contain one discriminator, tempoGAN uses two discriminators: one spatial discriminator $D_{s}$ to constrain spatial details, and one temporal discriminator $D_{t}$ to keep the sequence temporally coherent. For $D_{t}$, three warped adjacent frames’ densities are used as inputs to enforce learning temporal evolution in the discriminator. The tempoGAN generator furthermore requires velocity information as input in order to distinguish different amounts of turbulent detail to be generated. Similar to tempoGAN, our generator model consists of 4 residual blocks and takes the LR density and velocity fields as inputs, which, assuming that the fields are of size $16^{2}$, leads to an input of size $16\times 16\times 4$, with one density and three velocity channels.

For a multi-pass version of tempoGAN, there are two generators: $G_{1}$ and $G_{2}$. Every generator is paired with a spatial and a temporal discriminator, i.e., $D_{1s}$, $D_{1t}$ and $D_{2s}$, $D_{2t}$, respectively. The training process of $G_{1}$ is shown in Fig. 5. For $G_{2}$, a similar procedure is applied, with the only difference being the up-scale operation in the network and the input data which consists of the output of the first network and LR velocities.

During the training of $G_{1}$ and $G_{2}$, we use an additional L1 loss and feature space loss terms to stabilize the training. The resulting loss functions for training $G_{1}$, $D_{1s}$, $D_{1t}$ are then given by:

[TABLE]

[TABLE]

where $\mathcal{A}$ advects a given frame with the current velocity, s.t. $y^{t}=\mathcal{A}(y^{t-1},v_{y}^{t-1})$. \widetilde{G}_{\mathcal{A}}($$\widetilde{X}$$) denotes three advected, consecutive generated frames: \widetilde{G}_{\mathcal{A}}($$\widetilde{X}$$)=\{\mathcal{A}(G(x^{t-1}),v_{x}^{t-1}), $G(x^{t})$, $\mathcal{A}(G(x^{t+1}),-v_{x}^{t+1})\}$, while $\widetilde{Y}_{\mathcal{A}}$ denotes three ground truth frames: $\widetilde{Y}_{\mathcal{A}}=\{\mathcal{A}(y^{t-1},v_{x}^{t-1})$, $y^{t}$, $\mathcal{A}(y^{t+1},-v_{x}^{t+1})\}$. $j$ is a layer in our discriminator network, and $F^{j}$ denotes the activations of the corresponding layer. $G_{2}$, $D_{2s}$, and $D_{2t}$ are trained analogously. Note that the fluid is only advected in the $XY$-plane and in the $YZ$-plane for $G_{1}$ and $G_{2}$, respectively.

Exemplary outputs of $G_{1}$ and $G_{2}$ from a multi-pass tempoGAN are shown in Fig. 7. While the output of $G_{1}$ still contains visible and undesirable interpolation artifacts, they are successfully removed by $G_{2}$. Fig. 6 shows a comparison of the original tempoGAN model and our multi-pass version at the same target resolution. Our approach yields a comparable amount of detail and visual quality, while being the result of a much simpler and faster learning problem. We will evaluate this aspect of our approach in more detail below.

5. Multi-pass GAN with Growing

Due to the coupled non-linear optimization involving multiple networks, GANs are particularly hard to train. As this challenge grows with larger numbers of weights in the networks, techniques such as curriculum learning (Wang et al., 2018) and progressive growing (Karras et al., 2017) were proposed to alleviate these inherent difficulties. In the following, we will outline both techniques briefly before explaining how to adopt them for our multi-pass GAN in order to increase the upscaling factor to $8$.

Curriculum learning describes the process of increasing the difficulty of a training target over time. In our case, we first train the generator and discriminators on pre-computed density references that are twice as large as the input. After training for a number of iterations, typically $120$k, we double the up-scaling factor until the final goal of $8$ is reached. The progressive growing approach goes hand in hand with curriculum learning: after increasing the difficulty of training, additional layers are smoothly faded in for the generator and discriminator networks. This ensures that the gradients from the added stages don’t impair the existing learning progress. As shown in Fig. 8, the blending process is controlled by the parameter $\alpha\in[0,1]$, which is used to linearly interpolate between two current up-scaled density fields, e.g., combining generated outputs of scale $2$ and $4$. The procedure is similar regarding the discriminators. However, we use average pooling layers to down-scale the density field between the stages instead of up-scaling it. This growing technique is applied to the generator network $G_{1,8\times}$, the spatial discriminator $D_{1s,8\times}$, as well as the temporal discriminator $D_{1t,8\times}$. While fading in new layers, we increase $\alpha$ from [math] to $1$ over the course of $120$k iterations. The networks are then trained for another $120$k iterations on the current SR factor. Since the task for the second generator is to purely refine the volume along the axis that was invisible to $G_{1,8\times}$, and because the only training target is the HR reference, we disable progressive growing and curriculum learning for $G_{2,8\times}$, $D_{2s,8\times}$, and $D_{2t,8\times}$.

5.1. Network Architecture

The resulting generator network $G_{1,8\times}$ consists of a sequence of residual blocks (ResBlocks). Overall, for a total SR factor of $8$, eight ResBlocks are used which are divided into four growing blocks. Each growing block up-scales the previous layer by a factor of $2$, except for the first one which maintains the input resolution. For each growing block, a $1\times 1$ convolution is trained in addition to translate the intermediate outputs into the density field if the additional down-stream layers are not yet active as shown in Fig. 8. For $G_{1,8\times}$, in order to save computations, we concentrate most of the parameters in the earlier stages of the network which have smaller spatial resolutions. We use a Wasserstein GAN loss with gradient penalty (WGAN) and a weak L1-loss in combination with the proposed equalized learning rate and pixelwise normalization (PN) (Karras et al., 2017) instead of batch normalization to keep the magnitudes in the adversarial networks under control. Thus, each ResBlock consists of the sequence of Conv3x3-ReLU-PN-Conv3x3-ReLU-PN.

In addition, inspired by VDSR  (Kim et al., 2016) and ProSR (Wang et al., 2018), both our generators output residual details rather than the whole density field to improve the quality of results and to decrease computing resources. Additionally, a bi-cubic up-scaled LR density field is added to the output of $G_{1,8\times}$ and $G_{2,8\times}$. The architecture of $G_{1,8\times}$ leads to a receptive field of approximately $7$ LR pixels in both dimensions. For $G_{2,8\times}$, $D_{2s,8\times}$, and $D_{2t,8\times}$, we increase the kernel size of the convolutional layers to $5$ in order to increase the receptive range to $4$ LR cells in each direction, resulting in a receptive field of $16$ LR cells. Furthermore, we decrease the amount of feature maps to keep the number of parameters similar to $G_{1,8\times}$, $D_{1s,8\times}$, and $D_{1t,8\times}$.

Beyond our multi-pass approach, the resulting network differs from existing growing approaches in the sense that it targets functions of much larger dimensionality (3D plus time) via a temporal discriminator, while it also differs from the tempoGAN network in several ways: we employ a residual architecture with Wasserstein GAN loss and due to the progressive growing, the networks are inherently different. The exact network structure and all training parameters can be found in Appendix A and  B.

Fig. 9 shows that the combination of our method and the growing approach can yield highly detailed and coherent buoyant smoke. $G_{2,8\times}$ is able to add additional details to the volume, reconstruct the desired HR shape, and generally sharpen the output along the axis that was unseen by $G_{1,8\times}$.

5.2. Loss Function

The resulting Wasserstein GAN-based loss function for our multi-pass, multi-discriminator network is then given by:

[TABLE]

[TABLE]

[TABLE]

with $\hat{y}=(1-R)\cdot G_{1,8\times}(x)+R\cdot y$, $\hat{Y}_{\mathcal{A}}=(1-R)\cdot\widetilde{G_{1,8\times}}_{\mathcal{A}}(\widetilde{X})+R\cdot\widetilde{Y}_{\mathcal{A}}$, and $R\in[0,1]$, which is randomly sampled. $n$ and $m$ denote the mini-batch sizes used for training the temporal and spatial discriminator and are set to $15$ and $16$, respectively. We use $\lambda_{W}=10$ and $\lambda_{L1}=20$ as scaling factors for the loss terms. The networks are trained by using Adam (Kingma and Ba, 2014) with learning rate $\eta=0.0005$, $\beta_{1}=0.0$, $\beta_{2}=0.99$, and $\epsilon=10^{-8}$. Eq. (4) is used instead of Eq. (2) and Eq. (3) when training $G_{1,8\times}$, $D_{1s,8\times}$, and $D_{1t,8\times}$. Note that we do not employ the feature space loss since the training process stays stable by using the WGAN-GP loss.

Training processes of the spatial discriminators with different losses are shown in Fig. 10. Here, lower values indicate that the discriminator is more successful at differentiating between real and fake images. In a) and b), which employ a tempoGAN with the loss described in Eq. (2), (3) and the least squares GAN loss (Mao et al., 2016) (LSGAN), respectively, it becomes apparent that the classification task is too easy, especially after fading in new layers. Since the losses continue to drop for tempoGAN and LSGAN which means that the spatial discriminators are growing too strong too quickly, the gradients for the discriminators will become more and more ineffective, which in turn leads to the generator receiving unreliable guidance (Arjovsky and Bottou, 2017). In contrast, the stability of the WGAN with gradient penalty in c) is barely influenced by the blending process, and can recover quickly from the curriculum updates.

6. Data Generation

The data used for training is generated by a stable fluids solver (Stam, 1999) with MacCormack advection and MiC-preconditioned CG solver in mantaflow (Thuerey and Pfaff, 2018). Overall, we use $20$ 3D simulations with $120$ frames each. For the setup, we initialize between $3$ and $12$ random density and velocity inflow areas combined with a randomized buoyancy force between $(0,0,0)$ and $(0,3\cdot 10^{-4},0)$. The LR volume is of size $64^{3}$, whereas the HR references are of size $256^{3}$ for the tempoGAN $4\times$ up-scaling case, and of size $512^{3}$ for the progressively growing GAN $8\times$ up-scaling case. The inputs $x$ are generated by down-scaling the reference volumes $y$. When loading and converting the data to slices, we remove slices with average density below the threshold 0.005.. For the progressively growing version, we also generate the intermediate resolution frames of size $128^{3}$ and $256^{3}$ via down-scaling the references. We apply physical data augmentation methods in form of scaling and 90-degree rotations (Xie et al., 2018). When modifying an input density field, we also have to adjust the velocity field accordingly.

While fluids in general exhibit rotational invariance, we target phenomena with buoyancy which confines the invariance to rotations around the axis of gravity. To train for shift invariance, we cut out tile pairs using randomized offsets before applying any augmentation. The size of the tiles after augmentation is $16^{2}$ for our inputs, and $(16\cdot j)^{2}$ for the targets of the current stage with $j$ being the up-scaling factor. Overall, data augmentation is crucial for successful training runs, as illustrated with an example in Fig. 11. More varied training data encourages the generation of more coherent densities with sharper details.

7. Results and Discussion

In the following, we demonstrate that our method can generate realistic outputs in a variety of flow settings, and that our network generalizes to a large class of buoyant smoke motions. For all the following scenes, please also consider the accompanying video which contains the full sequences in motion.

In Fig. 12, a scene with three colliding smoke plumes is shown. The input of size $50^{3}$ is transformed into an output of $400^{3}$ by our network. The smooth streaks in the input are successfully sharpened and refined by our multi-pass GAN. In addition, the generated volumes are temporally coherent along all spatial dimensions. Fig. 13 on the other hand shows a scene of fine smoke filaments interacting with a complex obstacle in the flow. The underlying simulation is of size $100^{3}$, while the generated output has a resolution of $800^{3}$. Even though the training data did not contain any obstacles, our network manages to create realistic and detailed volumetric smoke effects.

7.1. Evaluation

In addition, it is interesting to compare the performance of our models trained for different up-sampling factors with outputs at the same target resolution. To compare the $8\times$ model with the $4\times$ one, we apply the former one to a simulation of size $32^{3}$, which was generated by down-sampling the actual simulation of size $64^{3}$. This way, both models generate a final volume of size $256^{3}$. Fig. 14 shows renderings of the outputs. Here it becomes apparent that our $8\times$ network manages to create even sharper and more pronounced details than the $4\times$ version, despite starting from a coarser input. For this simulation, we used the time step $\Delta t_{o}=0.25$ instead of $\Delta t_{t}=0.5$ as for training our networks, with $\Delta t_{t}$ and $\Delta t_{o}$ denoting the training and output time step, respectively. Most importantly, the range of velocities of the input simulation should not exceed the range the network was trained on, otherwise artifacts occasionally occur. One possibility is to re-scale the velocities to match the training time step by multiplying with $\frac{\Delta t_{t}}{\Delta t_{o}}$ which we applied in the former example.

As shown in Fig. 15, applying the $4\times$ networks to a smooth LR simulation of size $80^{3}$ sharpens only very few areas, e.g., the area of the plume which faces the direction of the buoyancy force. As desired, other parts such as the density inflow area are largely unchanged. Notably, however, are the few new density gradients at certain boundaries of the smoke which might be undesirable when targeting a very smooth scenarios. Since our training data did not contain such smooth simulations, the results shown here could be improved by fine-tuning the networks further.

As a simpler variant of our approach, we also evaluated applying the same network to all slices of a volume along all three dimensions, each of which was linearly up-sampled. This yields three volumes at full resolution which can be combined to obtain a single final output volume. For the combination, we have tested an averaging operation (AVG), taking the maximum per cell (MAX), or taking one of the volumes as a basis and then adding details by taking the difference between the the linearly up-sampled input and the output along another axis (RES). The latter variant represents a residual transfer. Here, we also tested a variant that only used additive details to be transferred, i.e., the residuals were clamped at zero (CRES).

As shown in Fig. 16, all of these simpler methods fail at producing sharp, round edges and typically generate staircasing artifacts. The (RES) version contains especially strong artifacts while the (CRES) version does not perform much better. In comparison, our approach yields smooth and detailed outputs. Thus, training specialized networks for multiple passes is preferable over re-using a single network. The additional work to train $G_{2}$ for a second refinement pass pays off in terms of quality of the generated structures.

7.2. Performance

Despite making it possible to achieve large up-scaling factors, our method also reduces training time. Training our $4\times$ multi-pass network took approximately $3$ days for the first generator and $2$ days for the second. This is almost twice as fast as training a 3D model reported by previous work (Xie et al., 2018). We additionally only employed a single GTX 1080 Ti instead of two GPUs. Training the progressively growing network for an $8\times$ up-scaling took about $8$ and $5$ days for the first and second generator network, respectively. As this scale is not feasible with previous work, we cannot compare training times for the $8\times$ case.

Similar to previous work, the memory available in current GPUs can be a bottleneck when applying the trained networks to new input, and can make it necessary to subdivide the inputs into tiles. Regarding our implementation, we are able to deal with full slices of up to approximately $256^{3}$. Therefore, we usually do not need to apply tiling. For larger volumes, the tiling process requires an additional overlap for the tiles ($4$ LR cells for $G_{1}$ and $16$ HR cells for $G_{2}$). It takes around $10.14$ seconds to apply our $4\times$ network to a single frame of size $64^{3}$ to up-scale the volume to $256^{3}$. Applying $G_{1,8\times}$ to a volume of size $64^{3}$ takes $7.58$ seconds while refining it with $G_{2,8\times}$ takes an additional $52.07$ seconds. We compared our multi-pass GAN with a regular CPU-based solver using CG or multi-grid pressure solvers which are shown in  Table 1. According to the CFL condition, we applied a $\frac{1}{f}$ smaller time step (where the up-scaling factor $f$ is $4$ or $8$ in our case) for stability and convergence. E.g., if we generate data of the same time length as with the $8\times$ multi-pass GAN, we run $8\times$ more iterations with a regular or multi-grid fluid solver with a $\frac{1}{8}$ time step. Note however, that this solver is CPU-based and therefore it is difficult to compare the actual timings. Applying our method scales linearly with the number of cells of the simulation, whereas fluid solvers with CG-based pressure projections typically scale super-linearly. Besides, for a regular solver, simulation time noticeably rises when the smoke volume increases in later frames while the content does not affect generation time for the multi-pass GAN. In  Table 1, we clearly see that the multi-pass GAN is significantly faster than regular and multi-grid fluid solvers when generating data for a given number of frames.

7.3. Limitations and Outlook

A first limitations of our approach is that it requires multiple passes over the full HR volume (two in our case). However, in practice, these passes often result in speed-ups compared to networks that have to process the full data set at once. E.g., the tempoGAN architecture relies on up to $128$ latent features per volumetric degree of freedom. These intermediate representations have to be stored on the GPU and quickly fill up the memory capacities of current GPU architectures. Hence, larger output volumes will require tiling of the inputs and induces increased workloads due to these ghost layers. In contrast, our multi-pass networks only build feature spaces for slices of reduced dimensionality that typically can be processed without tiling.

Our $8\times$ network in its current form can lead to artifacts caused by the initial LR sampling. This typically only happens in rare situations, two examples of which are shown in Fig. 17. Here, the network seems to reinforce steps from the input data. These most likely stem from the difficult inference task to generate a $512\times$ larger output. As we have not observed these artifacts in our $4\times$ versions, they potentially could be alleviated by larger and more varied training data sets.

8. Conclusion

We have presented a first multi-pass GAN method to achieve volumetric generative networks that can up-sample 3D spatio-temporal smoke effects by factors of eight. We have demonstrated this for the tempoGAN setting and demonstrated that our approach can be extended to curriculum learning and growing GANs. Most importantly, our method reduces the number of weights that need to be trained at once, which yields shorter and more robust training runs. While we have focused on single-phase flow effects in 3D, our method is potentially applicable to all kinds of high-dimensional field data, e.g., it will be highly interesting to apply our method to 4D space-time data sets.

9. Acknowledgments

This work was funded by the ERC Starting Grant realFlow (StG-2015- 637014). We would like to thank Marie-Lena Eckert for paper proofreading.

APPENDIX

Appendix A Network architecture

Architectures of $G_{1,8\times}$, $G_{2,8\times}$, $D_{s,8\times}$, and $D_{t,8\times}$ are listed in  Table 3, Table 3, Table 5, and Table 5, respectively. ReLU is used for the generator and an additional pixel-wise normalization layer is added after every convolutional layer. Each convolutional layer in the discriminator uses leaky ReLU as an activation function. The input to the spatial discriminator consists of the up-scaled LR and HR density fields, whereas the input to the temporal one is composed of three advected frames. Input to $G_{1,8\times}$ are the LR density and velocity, whereas the input to $G_{2,8\times}$ consists of LR density, velocity, and the output of $G_{1,8\times}$. Note that the tile sizes for $G_{2,8\times}$ and $G_{1,8\times}$ are $64^{2}$ HR and $16^{2}$ LR pixels, respectively.

Appendix B Training parameters

For an up-scaling factor of $8$, $3$ stages exist in which new layers are faded in. ’blend iter.’ describes the number of training iterations for $G_{1,8\times}$. The blending and stabilizing processes are applied in an alternating fashion: $120$k iterations for fading in, $120$k iterations for stabilizing, etc. This leads to $120k\cdot 6=720k$ iterations overall. After finishing the progressive growing of the networks, we slowly decay the learning rate for ’decay iter.’.