Expanding the canvas of Van Gogh's “Starry Night” with the Wolfram Language (Mathematica)

Original author: Piotr Wendykier
  • Transfer

Translation of Piotr Wendykier's post " Extending Van Gogh's Starry Night with Inpainting "
Download the translation as a Mathematica document , which contains all the code used in the article, here (archive, ~ 8 MB).


Can computers learn how to draw like van gogh? Definitely yes, to some extent! To do this, like copyist artists, the algorithm will first need to take some original work, and then it will be able to create something on its basis itself. How well can he handle this? Please judge for yourself.

ExtendingVanGoghStarryNightInpainting_1.gif
Second Prize at the ZEISS Photo Contest

Recently, Cambridge University's Department of Engineering announced the winners of an annual photo contest entitled “The Art of Engineering: Images from the Frontiers of Technology”. Second place went to Yarin Gal, a graduate student in machine learning, for his extrapolation of Van Gogh's “Starry Night,” which is shown in the image above. You can see this and similar “enhanced” computer images on the website of Yarina Extrapolated Art . In the creation of these artificial paintings, an extrapolation sketching algorithm called PatchMatch was used and in this post I will tell you how you can create a similar effect using the Wolfram Language ( Mathematica ).

The term “digital inpainting” was first introduced in the article “ Image Inpainting ” at the SIGGRAPH 2000 conference. The main purpose of “sketching” is to repair damaged areas of images. However, this algorithm is also widely used to remove or move selected objects in images.

The Wolfram Language has an Inpaint function that implements this algorithm. The region that you want to “sketch” (retouch) can be given in the form of an image, a set of graphic primitives (vector graphics) or a matrix.

In [1]: =

ExtendingVanGoghStarryNightInpainting_2.gif

Out [1] =

ExtendingVanGoghStarryNightInpainting_3.gif

Inpaint function can work with 5 algorithms, while you can specify a specific algorithm using the Method option, these are: “Diffusion,” “TotalVariation,” “FastMarching,” “NavierStokes,” and “TextureSynthesis” (default). “TextureSynthesis”, unlike other algorithms, does not operate separately with each color channel and does not create any new pixel color values. In other words, each pixel involved in the “sketching” is taken from parts of the original image that have mask values ​​of zero (that is, in other words, they are black pixels of the mask). The example below clearly shows that the “TextureSynthesis” method is great for removing large objects from an image.

In [2]: =

ExtendingVanGoghStarryNightInpainting_4.gif

Out [2] =

ExtendingVanGoghStarryNightInpainting_5.gif

The “TextureSynthesis” method is based on the algorithm described in the dissertation “ Image Texture Tools”(P. Harrison). This algorithm is an improvement on the best approximation method, which was proposed in 1981 in the doctoral dissertation “ Computational Models for Texture Analysis and Texture Synthesis ” (D. Garber). The parameters of the “TextureSynthesis” algorithm can be set using two suboptions: “NeighborCount” (default, 30) and “MaxSamples” (default, 300). The first parameter determines the number of nearby pixels that will be used to compare the textures, the second - sets the maximum sample size for selecting the best texture.

Let's get back to extrapolating Van Gogh's painting. To get started, we import its image and remove the frame.

In [3]: =

ExtendingVanGoghStarryNightInpainting_6.gif

Out [4] =

ExtendingVanGoghStarryNightInpainting_7.gif

Now we need to expand the image field by adding a white field to it, and also create a mask for “sketching”.

In [5]: =

ExtendingVanGoghStarryNightInpainting_8.gif

Out [8] =

ExtendingVanGoghStarryNightInpainting_9.gif

Now we can extrapolate the image using the “TextureSynthesis” method.

In [9]: =

ExtendingVanGoghStarryNightInpainting_11.png

Out [9] =

ExtendingVanGoghStarryNightInpainting_12.gif

Not so bad. You can get a different result by changing the values ​​of the “NeighborCount” and “MaxSamples” parameters.

In [10]: =

ExtendingVanGoghStarryNightInpainting_13.png

Out [10] =

ExtendingVanGoghStarryNightInpainting_14.gif

You can also experiment with other parameter values ​​and other works of art. *

In [11]: =

ExtendingVanGoghStarryNightInpainting_15.gif

Out [16] =

ExtendingVanGoghStarryNightInpainting_16.gif

With this technique, I think you can create an original New Year gift. You can make really interesting things out of your personal photos or pictures. Or, you can surprise your child by showing him an improvisation on the subject of his drawing , as was done above based on the drawing of a 14-year-old girl. It all depends on your imagination!

Resources for learning Wolfram Language (Mathematica) in Russian: http://habrahabr.ru/post/244451

Also popular now: