Non convolutional networks
The advantages, problems, and limitations of convolutional neural networks (CNNs) are currently fairly well understood. About 5 years have passed after the community of engineers recognized them and the first impression “now we will solve all the problems”, I want to believe, has already passed. So, it's time to look for ideas that will take the next step in AI. Hinton, for example, suggested CapsuleNet .
Together with Alexei Redozubov, relying on his ideas about the structure of the brain, we also decided to step back from the mainstream. And now I have something to show: architecture (it goes with a title picture to attract attention) and Tensorflow sources for MNIST.
More formally, the result is described in an article on arxiv .
What for?
Always and in everything we deal with ambiguity. The further the information leaves the sensory organs, the more options for its interpretation become. Vision doesn’t seem to fail us. But no, there are a number of illusions that demonstrate that this is not so: the illusion of a concave mask (you need to try very hard to realize it is concave in depth), the famous white-blue-brown dress. Rumor has the same story that strikes even more than visual illusions: I want to congratulate you on this . And when the question of perception of information concerns abstract concepts, one cannot help but recall the endless debate about politics.
This principle of changing the meaning of the observed depending on the context is universal, in one way or another it is implemented for any information that we are dealing with. And, at the same time, it is poorly or not explicitly reflected in most approaches in machine learning. Often, the formulation of machine learning tasks precludes the very possibility of various interpretations. Although, if you look closely at the idea of convolutional networks from a different perspective, it turns out that the problem statement for one convolutional layer is formulated as follows: it is necessary to select K features of size MxM, regardless of what position (spatial context) they are in. If, however, it is assumed that the transfer operation along X, Y is not programmed in advance, then the task becomes very non-trivial. Similarly with recurrent networks:
In addition, after training, we can use ideas about how data changes due to a change in context. Show the person a new car model from 4 sides, he will easily recognize it later in perspective from a random angle. The child will see a cartoon where some social model of interaction is demonstrated by animals, it is not difficult for him to recognize the same situation in his relationship with people. This is such a powerful and universal principle that it personally surprises me why so little attention is paid to it.
I am convinced that Strong Artificial Intelligence will be distinguished precisely by the ability to present any information at any processing level in various contexts, to choose the most suitable context for the observed situation and the experience gained as a result of training.
The source of inspiration for convolutional networks (or rather, the neocognitron) was the research of Hübel and Wiesel, who at one time conducted very effective studies of the visual cortex V1 and the function of minicolumns in it, for which they were awarded the Nobel Prize. But the results of their experiments can be interpreted differently. More details here .
Hinton's capsule network is also a way to rethink the function of minicolumns.
Suppose the function of mini-columns
The cerebral cortex structurally consists of zones with various functions, but at the same time, regardless of the zone, mini-columns stand out structurally - columns of neurons, vertical connections between which are denser than horizontal ones.
We assume that the minicolumn presents input in its own context. Then, well-known patterns are recognized in each mini-column, its activity is proportional to the degree of recognition reliability. Neighboring mini-columns represent similar transformations, due to this we can choose local activity maxima on the cortex, which will correspond to the best contexts in which the input information is interpreted. The output of the cortical area will be a list: (recognized patterns, context).
Model for MNIST
Only one level of information analysis was modeled (one zone), several zones will be the next step.
It is fundamentally important that here we abandoned spatial convolutions in the image (Conv2D) and max-pooling 2x2, which forgive small geometric distortions, due to which the accuracy on MNIST noticeably improves.
I’ll repeat the picture so that I don’t have to rewind far by reading the text:
There are 3 important parts:
1) Only a vector 32 in length falls into the zone - the result of compression by a classic auto encoder with 2 fully connected layers for decoding and encoding.
2) We define 405 geometric context-transformations: 5 turns X 9 offsets X 9 scales. We look at how the code (32 components) changes after the transformation of input images, we train 2 fully connected layers (32-> 256-> 32) for each contextual transformation, i.e. one hidden and one day off. In fact, of course, just remember.
3) We are looking for stable patterns, regardless of the context in which they appeared, for each pattern we select one best context (max-pooling layer).
Then we send the zone exit to a small fully connected network, which decides what kind of figure is in front of us.
Training
Yes, the principle of back propagation of errors was everywhere used here. The only thing that his "long-range action" was very limited. Those. studied in stages, there really wasn’t much “deep learning”. There is a feeling that everything should work on different approaches, but until the algorithms are written, we will use proven tools.
First, we train the auto-
encoder separately: And we fix the weights of the encoder and decoder layers.
In fact, this is a completely optional step. Moreover, due to the not perfect presentation of the data, the accuracy will slightly deteriorate. But at stage 3, this reduces the number of calculations.
Then we train all 405 transformations. Those. how the image code changes after these transformations. We’ll test what happened. We will restore the original images from the converted codes:
It seems that it is commonplace (and it is)! But there are a couple of very important points:
1) From the point of view of the model, the transformation is not programmed, but obtained by training. Those. absolutely no restrictions on what data we are dealing with and what contextual transformations. And what's even cooler, you can even convert not into the same space. But more on that later.
2) You can take and submit a sign that has never been shown before, and we will know how it changes depending on the context.
Give the input an inverted triangle and get:
It looks, of course, not so impressive. But ideologically this is very cool! Through previous experience, you know how an object that you have never seen in your life will look in all possible contexts. And this opens up the possibility for one-shoot learning. We saw the object in one view, and then you will find out in a completely different way! This drastically reduces the required training set. And if we talk about computer vision, then it is not only about geometric transformations, but also about color, brightness, etc.
And then we need detectors common to all contexts. This was implemented using a convolutional layer with a core size of 1x1 and a max-pooling layer, which selected a winner for each core. The convolutional layer here is just a way to use the same weights for different contexts; it does not fulfill its usual function (like max-pooling with an unusual size core 405x1).
results
First of all, it was important to check how things are going with reducing the size of the training sample. Take the first 1000 images and train only on them.
The learning curve is as follows:
Accuracy 96%.
It should be compared with a convolution network, for example, of such an architecture:
Accuracy after CNN training was 95.3%.
An attentive reader will notice that the comparison is unfair, as I used implicit augmentation (geometric distortion of input images). So we introduce a similar augmentation for the convolution network. We get already 96.2%.
Not bad, the result is comparable to a convolution network.
Complete training gave an accuracy of 99.1%. Which is noticeably worse than a convolutional network, but also better than a fully-connected network of several levels.
Avoiding the use of convolutional layers supposedly reduced accuracy. But, in any case, it was possible to debug the architecture and get adequate accuracy.
Does it all resemble something?
Yes, firstly, of course, it unexpectedly looks like Hinton’s CapsuleNet. He also offers his interpretation of the functionality of mini-columns. The main difference is that CapsuleNet does not contain the idea of presenting information in various contexts. But the fact that the architectures turned out to be similar leads to very interesting thoughts. In its implementation, the activity of the mini-column corresponds to the recognition of one of the numbers, and in our concept, it turns out that the number is a context. And what does this picture mean in the context of the fact that it is a seven? And in the context of the fact that this is a unit? And this question is really not without meaning. The context may be a geometric transformation, and the context may be a type of recognizable object. More on this later.
Another transformation idea is contained in the Spartial Transformation Networks. A layer of geometric transformation and another small grid are included in the layer break, which estimates the most suitable transformation using regression. At the same time, gaps in differentiability are not allowed and all this can be trained using the method of back propagation of errors. There are 2 differences here: it is not formulated what to do with discrete transformations (and, for example, they are discrete in the grammar of speech), and it seems that a small network with regression to evaluate the optimal transform is superfluous. The best context should be selected based on in which context the data forms a known pattern, and there is no other optimal way than to analyze each of the transformations.
And another thought that never lets go
There is the so-called "Two-stream hypothesis" in the visual tract (and in the auditory / speech). On one side of the brain (ventral stream), mini-columns respond primarily to the characteristics of objects and the types of objects themselves, and on the other side of the brain (dorsal stream) mainly to spatial characteristics.
In our interpretation, dorsal stream is about spatial and temporal transformations. Let's take a look at V2 and V3: Speed X Direction - transition to a frame of reference moving at a given speed with a given direction, Spartical Frequency - simply transforming the scale, Temporal Frequency - changing the scale in time.
But Ventral Stream looks completely counterintuitive. What does it mean to go into the context of “Simple shapes”, or the context of curvature? In the context of color, by the way, is no problem. Or in TE / AIT "in the context of the object"? This problem can be overcome by assuming that the mini-column code characterizes the properties and characteristics of the object. For example, the curvature of the border in the image is characterized by contrast, blur, color, etc. And these characteristics will be common to each of the mini-columns responsible for the curvature.
Very close to CapsuleNet, but the styles or characteristics inside the mini-columns should be common to some neighborhood. The only question that remains open is how to train this type of contextual transformation.
It is also remarkable that the recognition result obtained in one of the Stream-s can be used in another. By recognizing the shape in the context of a particular position in the dorsal stream, orientation and scale, it is possible to increase the likelihood of activation of the corresponding mini-columns in the ventral stream. And vice versa.
conclusions
There is still a lot of work ahead.
PS: the name ContextNet, which I used in the source code, is actually already taken , but it fits very well.
