Related Applications

[0001] This application claims the benefit of U.S. Provisional Patent Application No.

62/762,293, filed April 27, 2018, the contents of which are incorporated herein in their entirety.

Government Rights

[0002] This invention was made with government support under contract 20131JCXK005 awarded by the Department of Justice and contract N6833516C0177 awarded by NavAir. The government has certain rights in the invention.

Background of the Invention

[0003] Generative adversarial networks (GAN) and many of its variants can learn from the tremendous amount of data available and generate new instances that are like the data they have observed, in any domain. This line of research is extremely important because it has the potential to provide meaningful insight into the physical world we human beings can perceive.

[0004] Take visual perception for instance, the generative models have much smaller number of parameters than the amount of visual data available in the world, which means that, for the generative models to come up with new instances that are like the actual true data, they have to search for intrinsic pattern and distill the essence. As a result, machines can be made to understand, describe, and model the visual world better. [0005] GANs simultaneously train a generator network for generating realistic images, and a discriminator network for distinguishing between the generated images and the samples from the training data (true distribution). The two players (generator and discriminator) play a two-player minimax game until Nash equilibrium where the generator is able to generate images as genuine as the ones sampled from the true distribution, and the discriminator is no longer able to distinguish between the two sets of images, or, equivalently, is guessing at random chance. In the traditional GAN formulation, the generator and the discriminator are updated by receiving gradient signals from the loss induced by observing discrepancies between the two

distributions by the discriminator. GANs are able to generate images with the highest visual quality by far. The image details are sharp as well as semantically sound.

[0006] While there have been many GAN variants that show some improvements over the original GAN, there has not been much work dedicated to self-improvement of GAN, i.e. improving the GAN performance solely based on the self-play with previous versions of itself.

[0007] The performance of the GAN generator is directly affected by the GAN discriminator.

In short, to make the generator stronger, a stronger opponent (discriminator) is needed. With a weak discriminator which does a poor job telling generated images from the true images, it takes only a little effort for the generator to win the two- player minimax game as described in the original work of GAN

[0008] To further improve upon the state-of-the-art GAN method, one possible direction is to enforce a maximum margin ranking loss in the optimization of the discriminator, which will result in a stronger discriminator that attends to the fine details of images. A stronger discriminator helps to develop a stronger generator. [0009] Perhaps the most seminal GAN-related work since the inception of the original GAN is the Wasserstein GAN (WGAN). Efforts have been made to fully understand the training dynamics of generative adversarial networks through theoretical analysis, which leads to the creation of the WGAN.

[0010] The two major issues with the original GAN and many of its variants are the

vanishing gradient issues and the mode collapse issue. By incorporating a smooth Wasserstein distance metric and objective, as opposed to the KL-divergence and JS- divergence used by GAN, the WGAN is able to overcome the vanishing gradient and mode collapse issues. WGAN also has made training and balancing between the generator and discriminator much easier in the sense that one can now train the discriminator till optimality, and then gradually improve the generator. Moreover, it provides an indicator (based on the Wasserstein distance) for the training progress, which correlates well with the visual image quality of the generated samples.

[0011] The GAN framework trains two networks, a generator Q _e (z) : z ® x, and a

discriminator T> _w (c ) : x ® [0,1] Q maps a random vector z, sampled from a prior distribution p _z (z), to the image space. T> maps an input image to a likelihood. The purpose of Q is to generate realistic images, while T> plays an adversarial role to discriminate between the image generated from Q, and the image sampled from data distribution p _data . The networks are trained by optimizing a minimax loss function

[0012] There are many ways to structure Q(z). One method uses fractionally-strided

convolutions to upsample images instead of fully-connected neurons. The generator Q is updated to fool the discriminator T> into wrongly classifying the generated sample, £(z), while the discriminator T> tries not to be fooled. Here, both Q and T> are deep convolutional neural networks and are trained with an alternating gradient descent algorithm. After convergence, T> is able to reject images that are too fake, and Q can produce high quality images faithful to the training distribution (true distribution

[0013] WGAN avoids the gradient vanishing and mode collapse issues in the original GAN and many of its variants by adopting a new distance metric: the Wasserstein-l distance, or the earth-mover distance. One of the biggest advantages of the

Wasserstein distance over KL and JS-divergence is that it is smooth, which is very important in providing meaningful gradient information when the two distributions have support contained in two closed manifolds that don't perfectly aligned don't have full dimension, in which case KL and JS-divergence would fail to provide gradient information successfully.

Summary of the Invention

[0014] Disclosed here is an improvement to the WGAN which incorporates a maximum

margin ranking criterion in the optimization, and with a progressive training paradigm that self-improve the WGANs at later stages. The improvement is referred to herein as GoGAN, short for Gang of GANs.

[0015] The improvement includes generalizing on the WGAN discriminator loss with a margin-based discriminator loss and providing a self-improving training paradigm involving multiple versions of GANs to contribute to the maximum margin ranking loss, such that the GAN, at a later GoGAN stage, will improve upon early stages. The improvements provide a theoretical guarantee that the GoGAN will bridge the gap between true data distribution and generated data distribution by at least half.

Brief Description of the Drawings

[0016] FIG. 1 is a graphical schematic showing the layout and operation of the GoGAN. [0017] FIG. 2 shows face generation using the present invention wherein latent vectors are obtained by passing faces through an encoder.

[0018] FIG. 3 shows face generation using the present invention wherein latent vectors are randomly generated without an encoder.

[0019] FIG. 4 shows graphs of discriminator scores during training in view (A) and during testing in view (B).

Detailed Description of the Invention

[0020] The GoGAN method is a progressive training paradigm to improve the GAN, by allowing GANs at later stages to contribute to a new ranking loss function that will improve the GAN performance further. At each GoGAN stage, the WGAN discriminator loss is generalized, and a margin-based discriminator loss is arrived at, which is referred to herein as the network margin GAN (MGAN).

[0021] The components of GoGAN will now be discussed with reference to FIG. 1, which shows the entire GoGAN flowchart.

[0022] As discussed above, WGAN has several advantages over the traditional GAN. Given that B _Wj (i) and D _W. (Q _W. (x)) are the discriminator scores for the real image x and generated image Q _Wj (x) in stage i respectively, then to further improve it, a margin- based WGAN discriminator loss is introduced as follows: where [x] ₊ = max(0, x) is the hinge loss. This MGAN loss function is a generalization of the discriminator loss in WGAN. When the margin e ® ¥, this loss becomes the WGAN discriminator loss. The margin ( e ) determines how big or small the gap is between the real image distribution and the generated image distribution at a particular stage.

[0023] The intuition behind the MGAN loss is as follows. WGAN loss treats a gap of 10 or 1 equally and it tries to increase the gap even further. The MGAN loss focuses on increasing separation of examples with gap 1 and leaving the samples with separation 10, which ensures a better discriminator, and hence a better generator. The MGAN loss can be extended even further by incorporating margin-based ranking when going beyond a single MGAN.

[0024] Encoder S

[0025] The encoder is connected to the rest of the GoGAN architecture as shown in FIG. 1.

The encoder is needed to rank the images generated at a later stage closer to the image from the true distribution. This means an entry point needs to be created for the image x to go through (1) the generator and the discriminator, such that the margin-based discriminator loss L _disc can be properly computed, and (2) the generator and the ranker, such that the ranking loss L _rank can be properly computed. Based on the design principles for GAN, the best way is to attach an encoder at the beginning of the network architecture that takes the input image x and maps it to the latent vector z = £(x), which further goes through the generator and the rest of the GoGAN.

[0026] Ranker 31:

[0027] The ranking loss is:

[0028] Combing equations (1) and (2), the L _disc and L _rank loss together are equivalent to enforcing the following ranking strategy. Notice that such ranking constraint only happens between adjacent GoGAN pairs, and it can be easily verified that it has intrinsically established an ordering among all the MGANs involved, which will be further discussed below.

[0029] The weights of the ranker 31 and the discriminator 2) are tied together. Conceptually, from Stage-2 and onward, the ranker is just the discriminator which takes in extra ranking loss in addition to the discriminator loss already in place for the MGAN.

[0030] In FIG. 1, the ranker is a separate block, but only for illustrative purpose. Different training stages are encircled by dotted lines with various transparency levels. The solid lines show the connectivity within the GoGAN, with various transparency levels in accordance with the progressive training stages. The arrows on both ends of the solid lines indicate forward and backward pass of the information and gradient signal. If the entire GoGAN is trained, the ranker will have achieved the following desired goal: fl(6 G ) < fl( G ) fl( G ) - < X(SK( ^Z ) < XM (5) where < indicates relative ordering.

[0031] In GoGAN, the entire model is updated by minimizing the following 3 loss functions:

(1) the cross-entropy discriminator loss L _disc as commonly used in traditional GANs;

(2) the ranking loss L _rank : and (3) the reconstruction fidelity loss L _mse measured by the mean-squared error (MSE) between the generated images Q _L (z) and the image from true distribution x, as commonly used in VAEs. The total loss for GoGAN is: ^GoGAN 4 ^mse T -i ^disc T T 2 ^rank ( ^~ ) where weighting parameters l ₀ , l ₁ and l ₂ control the relative strength. MSE loss is computed right after the generative network. It computes the image-level fidelity (i.e., dissimilarity) between the ground-truth input image and the generated image. A higher MSE loss implies a greater difference between the generated image and the ground-truth image.

[0032] GoGAN with ranking loss as specified by equation (2) trained at its equilibrium will reduce the gap between the real data distribution P _r and the generated data distribution P ₀ at least by half for a Wasserstein GAN trained at its optimality.

[0033] FIG. 2 shows open-set face generation visual results from various stages in GoGAN.

Here the latent vectors z's are obtained by passing the input faces through the encoder e. FIG. 3 also shows open-set face generation visual results from various stages in GoGAN and latent vectors z's are randomly generated without the encoder e. From both cases, it can clearly be seen that as the stage progresses, the GoGAN is able to generate sharper face images which are visually more appealing.

[0034] Both the FID and Inception score metrics show gradual improvement in the generated image quality with each stage, as can be observed visually from the generated images in FIG. 2 as well as FIG. 3. The Stage 1 (VAE) generated images are blurry. As the process proceeds through Stage 2 and Stage 3, the images get sharper and have more face-features. As can be seen in FIG. 4, the discriminator score of the real samples and of the samples generated by the previous stage generator are clamped, between which, the discriminator score of the current stage generator gradually increases. The clamping of discriminator scores is achieved by penalizing the difference between discriminator's scores between current and previous stage discriminators. The penalty added to discriminator's loss is given by:

[0035] Clamping forces the discriminator to get better at evaluating the generated samples, with each stage. As can be seen in FIG. 4, it takes a few iterations for clamping to stabilize when a new stage begins (this can be seen in the rise and fall of the upper and lower curves).