Published:

This post provides summary of the paper by Berthelot et al. 2017. They proposed a robust architecture for GAN with usual training procedure. In order to have stable convergence, they propose use to use equilibrium concept between Generator and Discriminator. The results are much imporoved in terms of both image diversity and visual quality.

paper : Berthelot et al. 2017

## Proposed Method

### Wasserstein distance for auto-encoders

To study the effects of matching the distribution of errors, an auto-encoder loss is approximated as normal distribution and further Wasserstein distance is computed between auto-encoder loss of real and generated samples.

Pixel-wise autoencoder training loss is given as $L(v) = |v - D(v) |^{\eta} \\$ where $D: \mathbb{R} ^{N_x} \rightarrow \mathbb{R}^{N_x}$ is autoencoder function, $\eta \in \{1,2\}$ is target norm, $v \in \mathbb{R}^{N_x}$ is a sample of dimension $N_x$ .

Using Central limit theorem, if number of pixels are assumed to be large and per pixel loss to i.i.d. the image wise losses approximately follow normal distribution.

#### Wasserstein distance

Given two normal distributions $\mu_1 = \mathcal{N}(m_1, C_1)$ and $\mu_2 = \mathcal{N}(m_2, C_2)$ with mean $m_{1,2} \in \mathbb{R}^p$ and covariances $C_{1,2} \in \mathbb{R}^{p*p}$ , the distance is given as

when $p=1$ , above equation reduces to

if $\frac{(c_1 +c_2 - 2\sqrt{c_1c_2})}{||m_1 - m_2||^2_2} \text{is constant or monotously increasing}$

the problem is simplified as

In this paper the aim is to reduce Wasserstein distance between loss distribution and not sample distribution.

### GAN Objective

$\mu_1$ be the normal distibution of the loss $\mathcal{L}(x)$ where $x$ are real samples

$\mu_2$ be the normal distribution of the loss $\mathcal{L}(G(z))$ where $G : \mathbb{R}^{N_z} \rightarrow \mathbb{R}^{N_x}$ is the generator function and $z \in [-1, 1]^{N_z}$ are random uniform samples

The possible solutions of maximizing eq(5)

a.

b.

the choice for objective in the paper is (a)

Overall GAN objective is given as

### Equilibrium

Usually, $G$ and $D$ are not well balanced and $D$ wins easily. To maintain a balance between generator and descriminator. The equilibrium condition is given as :

However, in general settings this is not the case and hence a new parameter is considered.

where $\gamma \in [0,1]$

## Training

both the objectives of generator and discriminator are trained simultaneously

### Model

For Generator

img —> conv(3,3) —> {[conv33, conv33] —>upsample(2,2) } x 3—> conv3,3 x 2 —> fc —> emb(h)

For discriminator

emb(h) —> {conv33, conv33, subsample(2,2)} x 3 —> {conv3,3 , conv3,3} -> conv3,3(img)

Tags: