How to distinguish birds from flowers. Or flowers from birds

As a weekend program, I wanted to play with a kind of “neural” network (spoiler - there are no neurons in it). And so that later it would not be excruciatingly painful for the hours spent aimlessly lived , I thought that in vain we feed it, let it bring benefits - let this grid at the same time take apart the home photo archive and at least put the photos of flowers in a separate folder.

The simplest network

The simplest network was found in the article " Neural network in 11 lines in Python " (this is a translation from SLY_G of the article " A Neural Network in 11 lines of Python (Part 1) ", in general, the author still has a continuation of " A Neural Network in 13 lines of Python (Part 2 - Gradient Descent) ", but here the first article is enough).

A brief description of the grid - this network has exactly one dependency - NumPy .

Many inputs are considered as a matrix.$$, many outputs - as a vector $$. In the original article, the network multiplies the input matrix, (4 x 3), by the input weight matrix$$ (3 x 4), applies the transfer function to the product, and gets the matrix of the layer $$ (4 x 4).

$$

$$

Next layer $$ multiplied by the output weight matrix $$ (4 x 1), also passed through the function, and the result is a layer $$ (4 x 1), which is the result of the network.

$$

Total, omitting the scalar transfer function, the network implements two matrix multiplications:

$$

The consequence of this, according to the rules of matrix multiplication, it turned out that one of the dimensions does not change during the operation of the network $$and it’s impossible to get a single number.

Therefore, I slightly modified the code from the article, added transposition after multiplication and work with an arbitrary number of layers in the grid. This gave me the opportunity to get any combination of dimensions of inputs and outputs.

For example, if you want the input matrix to be (3 x 4), and the output to be a single number, then we add two synapse matrices (4 x 1) and (3 x 1):

$$

Or, say, you can convert the input matrix (10 x 8) to the output (4 x 5):

$$

The resulting code is:

Upload photos

So, there is a grid, now we need to deal with loading photos. Photos are on disk, mostly in JPG, but there are other formats. They also have different sizes, depending on what they shot and how they were processed, from 3 Mpx to 16 Mpx.

At first I tried to upload photos via Qt, the QImage class, it can work with different formats, provides conversion and gives direct access to image data. Surely in Python there is a simpler way, but I did not have to deal with QImage. In order for the network to work with a picture, it should be converted to a monochrome image and reduced to a standard size.

def readImage(file, imageSize):
        img = QImage(file)
        if img.isNull():
            return 0
        img = img.convertToFormat(QImage.Format_Grayscale8)
        img = img.scaled(imageSize[0],imageSize[1],Qt.IgnoreAspectRatio)
        return img

To transfer to the grid, you need to convert the image to the numpy.ndarray matrix. QImage.bits () gives a pointer to image data, where each byte corresponds to a pixel. NumPy found a recarray function that can make an array of records from a buffer, and it has a view method that will make us a numpy.ndarray matrix without copying data.

        srcBi = img.bits()
        srcBi.setsize(img.width() * img.height())
        srcBy = bytes(srcBi)
        srcW, srcH = img.width(), img.height()
        srcArr = np.recarray((srcH, srcW), dtype=np.int8, buf=srcBy).view(dtype=np.byte,type=np.ndarray)

Network for images

Directly sending a picture to the network input will be too expensive, albeit a reduced one - I have already said that the network does matrix multiplication, so even one training cycle will lead to 400x400x400 = 64 million multiplications. Experts recommend the use of convolution . Wikipedia has a wonderful illustration of her work:


This animation shows that the dimension of the result is equal to the dimension of the original matrix. But I’ll simplify my life a little bit, I’ll move not in pixels, but I will break the image into pieces the size of the input matrix, and apply the grid to them one by one. In matrices, cutting a piece is done quite simply:

srcArr[x:x+dw, y:y+dw]

The result of processing the pieces by the network is added to a smaller matrix, this matrix is ​​transmitted to the input of the common network. That is, there will be two networks - the first works with pieces of the image, the second - with the result of the first network working on the pieces.

Create a primary network:

class ImgNN:
    def __init__(self, shape, resultShape = (16, 16), imageSize = (400,400)):
        self.resultShape = resultShape
        self.w = imageSize[0] // shape[0]
        self.h = imageSize[1] // shape[1]
        self.net = NN([shape, (1,shape[0]), (1,1)])
        self.shape = shape
        self.imageSize = imageSize

Self.net is created inside - the network itself, with a given size of the matrix of inputs shape and c output in the form of an elementary matrix 1x1. Yes, it was possible to inherit from the class of the NN network, but there was a day off, I wanted to get the result faster, and the architecture has not settled down yet. Time to market beats in our hearts!

Calculation of the image by the first network:

    def calc(self, srcArr):
        w = srcArr.shape[0] // self.shape[0]
        h = srcArr.shape[1] // self.shape[1]
        resArr = np.zeros(self.resultShape)
        for x in range(w):
            for y in range(h):
                a = srcArr[x:x+self.shape[0], y:y+self.shape[1]]
                if a.shape != (self.shape[0], self.shape[1]):
                    continue
                if x >= self.resultShape[0] or y >= self.resultShape[1]:
                    continue
                res = self.nn.calc(a)
                resArr[x,y] = res[0,0]
        return resArr

At the output, we have a resArr matrix, with a dimension equal to the number of pieces into which the image was divided. We pass this matrix to the input of the second network, which gives the final result.

    y = np.array([[1,0,1,0]])
    firstShape = (40, 40)
    middleShape = (5, 5)
    imageSize = firstShape[0]*middleShape[0], firstShape[1]*middleShape[1]
...
            nn = ImgNN(firstShape, resultShape=middleShape, imageSize=imageSize)
            nn2 = NN([middleShape, (y.shape[1], middleShape[0]), y.shape])
...
                i = readImage(f, imageSize)
                mid = nn.calc(i)
                res = nn2.calc(mid)

Here you should ask me where I got the first line, and what it means:

y = np.array([[1,0,1,0]])

This is the expected result of the network in the case of a positive answer, i.e. if the network believes that the input image is a flower. I chose the dimension from the principle of “neither less nor more” - if we take the dimension 1x1, then from one resulting number it is difficult to judge how much the network “doubts” the result. There is no point in asking a large dimension either — it will not provide more information. An equal number of zeros and ones gives a clear reference - the closer to it, the greater the coincidence. If we take all units or all zeros, then the network will have an incentive to retrain - increase all factors or, respectively, reset them to get the desired result regardless of the input data.

How to train a convolutional network?

I made a training sample from my own photographs, simply decomposing them into two directories:
flowers



and noflowers



I will collect the paths to the pictures in two arrays

            import os
            fl = [e.path for e in os.scandir('flowers')]
            nofl = [e.path for e in os.scandir('noflowers')]
            all = fl+nofl

It is usually proposed to train simple networks, including in the original article, by the traditional method - back propagation of errors . But in order to apply this method to a convolutional network consisting of two elementary ones, it is necessary to ensure the end-to-end transmission of the accumulated error from the second network to the first. In general, there are other methods for convolution networks . I was too lazy to redo the working network, at least for now, so I decided to train the second network, and do not teach the first one at all, leave it clogged with random values, judging that since the person’s eye nerves are not trained, then I have nothing to teach the primary network, "Looking" at the image.

        for epoch in range(100):
            print('Epoch =', epoch)
            nn = ImgNN(firstShape, resultShape=middleShape, imageSize=imageSize)
            nn2 = NN([middleShape, (y.shape[1], middleShape[0]), y.shape])
            for f in fl:
                i = readImage(f, imageSize)
                # nn.learn(i, yy, 1)
                mid = nn.calc(i)
                nn2.learn(mid, y, 1000)

In each era, right after training, I run through the network the entire sample and see what happened.

            for f in all:
                i = readImage(f, imageSize)
                mid = nn.calc(i)
                res = nn2.calc(mid)
                delta = abs(y-res)
                v = round(np.std(delta),3)

If the network has trained correctly, then its output should have a value close to the given [[1,0,1,0]], if the input is a flower, and as different as possible from the given one, for example [[0,1,0, 1]] if there is no flower at the entrance. The result is evaluated, empirically I accepted a deviation from a successful result of no more than 0.2 - this is also a successful result, and the number of errors is considered. Of all the runs, we select one where the least errors are made, and save the weights of the synapses of both grids into files. Further these files can be used for loading grids.

                if v > 0.2 and f in fl:
                    fails += 1
                    failFiles.append(f)
                elif v<0.2 and f in nofl:
                    fails +=1
                    failFiles.append(f)
            if minFails == None or fails < minFails:
                minFails = fails
                lastSyns = nn.net.syns
                lastSyns2 = nn2.syns
            print('fails =',fails, failFiles)
            print('min =',minFails)
            if minFails <= 1:
                print('found!')
                break
        for i in range(len(lastSyns)):
            np.savetxt('syns_save%s.txt'%i, lastSyns[i])
        for i in range(len(lastSyns2)):
            np.savetxt('syns2_save%s.txt'%i, lastSyns2[i])

Call it a rose, if not

With hope, I launch and ... wait ..., then wait another ..., and more ... I get complete nonsense - the grid does not learn:


Being the carrier of real living, and not artificial neurons, it dawned on me that the main difference between colors is color (yes, cap, thank you for being always there, although you are often late with your advice). Therefore, it would be necessary to translate it into some kind of color model, where the color component will be highlighted (HSV or HSL), and train the network in color.

But it turned out that the QImage class does not know such color spaces . I had to abandon it and upload pictures using OpenCV, where there is such an opportunity.

import cv2
def readImageCV(file, imageSize):
    img = cv2.imread(file)
    small = cv2.resize(img, imageSize)
    hsv = cv2.cvtColor(small, cv2.COLOR_BGR2HSV)
    return hsv[:,:,0]/255

True, OpenCV flatly refused to work with Russian letters in file names, I had to rename them.

Launched - the result is not pleased, almost the same.

I also thought, decided that the problem is in strongly random values ​​in the first grid, in vain I hoped that the stars would converge without my help, so I added a little pre-training to her, only 2 cycles per file. For a sample of a positive result, I took the identity matrix.

        yy = np.zeros(middleShape)
        np.fill_diagonal(yy,1)
...
            for f in fl:
                i = readImage(f, imageSize)
                nn.learn(i, yy, 2) # чуть-чуть обучаем первую сетку
                mid = nn.calc(i)
                nn2.learn(mid, y, 1000)

I started it again - it became much more interesting, the numbers began to change, although I did not reach the ideal.


Further ... And then the weekend ended, and it was time for me to do household work.

What to do next?

Of course, this network, the way I taught it, and the test dataset have very little relation to real networks and what data scientists do. This is just a toy for the gymnastics of the mind, do not have high hopes for it.

You can outline further steps on how to achieve the desired result (if you need it):

1. Add another intermediate layer or several to the second network - so she will have more freedom in learning. Still, the network on matrix multiplication is not entirely classical, since it has fewer synaptic links between layers, and the synapses themselves are not unique.
2. Use approximations to successful results as a blank for subsequent training - i.e. remember synapse weights of the most successful result, and not overwrite all random values.
3. Try genetic algorithms - mix and share, multiply the successful and reject the unsuccessful.
4. Try other learning methods, of which there is already a carriage and a small cart.
5. Use more information from the original image, for example, simultaneously apply color and monochrome to various networks, process the results in a common network.

Continuation