Pix2Pix: How the Cat Generator Works

You've all probably already seen super-realistic cats that you can draw here .

Let's figure out what's inside.

Disclaimer: the post is written based on the edited chat logs of closedcircles.com , hence the presentation style and clarifying questions

All this is an implementation of the Image-to-Image Translation pager with Conditional Adversarial Networks from Berkeley AI Research.

So how does this all work?

In the pager, people solve the problem of transforming the picture into another so that the person does not need to invent a loss function.

One of the main problems with neural networks in image generation is that if you use just the average pixel difference, for example, L1 or L2 (aka mean squared error) as loss, then the network seeks to average all possible options. If there is some uncertainty in the final picture - for example, the edge may be in a different position, or the color may be in a certain range, then the optimal result from the point of view of L2 loss is something in between all possible cases, and not some specific one them.

Therefore, the pictures are very blurry spots.

For different individual tasks, people came up with other loss functions to express a certain structure that should be in the resulting picture (for segmentation, for example, Conditional Random Fields tried to add, etc., etc.), but this all helps very incrementally and is very dependent on the task.

Well, following the new trends, GAN (Generative Adversarial Network) is stuck in the pager as such an additional loss to L1. (read about GANs it is possible to read on Habré here and here )

Their general scheme is as follows:

The input image is given to the generator at the input - it is an additional condition for what needs to be generated. Based on it, the generator should generate a picture for the output.

The discriminator is given both an input image and what the generator generated (or, for positive examples, a real pair from a training dataset), and it should give out whether the generated image is real or generated. Thus, if the generator generates a picture that is not related to the input, the discriminator must determine this and discard it.

The generator is the result of an iterative training of this pair of networks.

In general, this is the standard approach of Conditional GANs - a variant of GAN, where the model should generate pictures corresponding to the additional input class vector.

Only here the input vector of the class is a picture, and the total loss is GAN loss + L1.

In the sense of "sticking GAN" in the context of the discussion of losses? Type add generator and solve the problem of finding minimax?
Well yes.

Everything is at a high level!

What are their interesting details?

• Unlike the classical approach to GANs, no noise vector is transmitted at all to the generator.
  All the diversity is only due to the fact that the network has a dropout, and they do not turn it off after training.

  
  
• The network architecture is U-Net, a fairly new architecture for segmentation that has many skip connections from encoder to decoder (here is a short description )

Here is a picture that shows that both GAN loss and U-net help.

Here, by the way, the initial problem is clearly visible using only L1 loss - even a powerful model generates blurry spots to minimize the average deviation.

• They train the model on patches 70x70, and then apply on large pictures through full convolution. It's funny that 70x70 gives on average better results than doing the whole 256x256 picture at once.

And where are the cats !!!

After that, there is a system that can be taught on arbitrary inputs and outputs, even if they are from completely different tasks.

From segmentation to photography, from daytime photography to nighttime, from black and white to color, etc.

And the last example is from edges to picture. The edges in the picture are generated by the standard algorithm from computer vision.

This means that you can just take a set of pictures, run off edge detection, and
train on these pairs . It is possible on cats:

And after that, the model can generate something for any sketches that people draw.

(send, by the way, what you remember)

So was eliminated the lack of bread-like cats in humanity!

All in all, this work is another example of how GANs have soared since last year. It turns out that this is a very powerful and flexible tool that expresses “I want it to be indistinguishable from the present, although I don’t know what this specifically means” as the goal of optimization.
I hope someone writes a complete review of the rest happening in the area! Everything is very cool there.

Thanks for attention.