DetectNet: Deep Neural Network for Object Detection in DIGITS

Hi Habr. Recently, I really enjoy reading articles on deep learning, convolution networks, image processing, etc. Indeed, there are very cool articles here that amaze and inspire your own "more modest" exploits. So, I want to bring to the attention of the Russian-speaking public a translation of an article from Nvidia written on August 11, 2016, in which their new DIGITS tool and DetectNet network for detecting objects in images are presented. The original article, of course, at first may seem a bit advertising, and the DetectNet network does not represent anything “revolutionary”, but the combination of the DIGITS tool and the DetectNet network seems to me to be interesting for everyone.

Today, with the help of the NVIDIA Deep Learning GPU Training System (DIGITS), research analysts have at their disposal the full power of deep learning to solve the most common tasks in this area, such as: preparing data, defining a convolutional network, parallel training of several models , monitoring the learning process in real time, as well as choosing the best model. The fully interactive DIGITS tool saves you from programming and debugging and you only do the design and training of the network.

DIGITS 4 introduces a new approach to the task of detecting objects, which allows you to train the network to find objects (such as faces, vehicles or pedestrians) in images and define bounding boxes around objects. Read the article Deep Learning for Object Detection with DIGITS for a more detailed introduction to the method.

Figure 1. DetectNet Network Discovery for Vehicle Detection

To quickly master the method of working with DIGITS, the tool includes a representative example of a neural network model called DetectNet. In fig. Figure 1 shows the result of a DetectNet network trained to detect vehicles in aerial photographs.

DetectNet Data Format

For the classification of images, the input data for the training sample are ordinary pictures (usually small in size and containing one object) and class labels (usually an integer class identifier or a class string name). On the other hand, for the task of detecting objects, more information is needed for training. Images from the input from the training sample for DetectNet are larger and contain several objects, and for each object in the image, the label must contain not only information about the class to which the object belongs, but also the location of the corners of its bounding box. In this case, a naive choice of the label format with a variable length and dimension leads to the fact that the definition of the loss function ( loss function) can be difficult, because the number of objects in the training image can vary.

DetectNet solves this problem using a fixed three-dimensional label format, which allows you to work with images of any size and a different number of objects present. This input representation for DetectNet was “inspired” by work [Redmon et al. 2015] .

In fig. 2 shows a diagram of the processing of images from a training sample with markup for training the DetectNet network. In the beginning, a fixed grid with a size slightly smaller than the smallest object that we want to detect is superimposed on the original image. Further, each square of the lattice is marked with the following information: the class of the object located in the square of the lattice, and the coordinates of the pixelthe angles of the bounding box relative to the center of the square of the grid. In the event that no object fell into the square of the lattice, then the special class "dontcare" is used to save a fixed data format. Also, an additional “coverage” value is added to the input data format, taking values ​​0 or 1 to indicate whether the object is present in the square of the grid or not. In the case when several objects fall into one square of the grid, DetectNet selects the object that occupies the largest number of pixels. If the number of pixels is the same, then the object with the smallest ordinate ( OY) bounding box. Such a choice of objects is not important for aerial photography, but it makes sense for images with a horizon, for example, images from a DVR, where the object with the smallest ordinate of the bounding rectangle is closer to the camera.

Figure 2. Presentation of input for DetectNet

Thus, the goal of DetectNet training is to predict a similar presentation of data for a given image. Or, in other words, DetectNet must predict for each square of the lattice whether an object is present in it, and also calculate the relative coordinates of the angles of the bounding box.

DetectNet Network Architecture

The DetectNet neural network has five parts defined in the Caffe framework network model file. In fig. Figure 3 shows the DetectNet network architecture used during training. It can distinguish 3 important processes:

1. Images and marks of the training set are input to the data layer. Further, the transform layer “on the fly” adds data.
2. A fully convolutional network (FCN) produces feature extraction and prediction of feature classes and bounding rectangles over grid squares.
3. The loss functions, at the same time, consider the error in the problems of predicting the coverage of an object and the angles of bounding rectangles by the squares of the lattice.

Figure 3. DetectNet network structure for training

In fig. Figure 4 shows the DetectNet network architecture for verification, which has two additional important processes:

1. Clustering predicted bounding boxes to get the final set.
2. Calculation of the simplified metric mAP (mean Average Precision) to measure the effectiveness of the model for the entire test sample.

Figure 4. DetectNet network structure for verification

We can resize the grid square for training marks by setting the stride in pixels for the layer detectnet_groundtruth_param. For instance,

detectnet_groundtruth_param {
    stride: 16
    scale_cvg: 0.4
    gridbox_type: GRIDBOX_MIN
    min_cvg_len: 20
    coverage_type: RECTANGULAR
    image_size_x: 1024
    image_size_y: 512
    obj_norm: true
    crop_bboxes: false
  }

In the parameters of this layer, you can also specify the size of the training images (image_size_x, image_size_y). Thus, when these parameters are specified, images that enter the DetectNet network during training are randomly cropped to these sizes. This can be useful if your training set consists of very large images in which the objects to be detected are very small.

The parameters for the layer that complements the on-the-fly input are defined in detectnet_augmentation_param. For instance,

detectnet_augmentation_param {
    crop_prob: 1.0
    shift_x: 32
    shift_y: 32
    scale_prob: 0.4
    scale_min: 0.8
    scale_max: 1.2
    flip_prob: 0.5
    rotation_prob: 0.0
    max_rotate_degree: 5.0
    hue_rotation_prob: 0.8
    hue_rotation: 30.0
    desaturation_prob: 0.8
    desaturation_max: 0.8
  }

The data addition procedure plays an important role for the successful training of a highly sensitive and accurate object detector using DetectNet. The parameters from detectnet_augmentation_paramdetermine various random transformations (displacement, reflections, etc.) over the training set. Such transformations of the input data lead to the fact that the network never processes the same image twice and, thus, becomes more resistant to retraining and natural change in the shape of objects from the test sample.

The DetectNet FCN subnet has a structure similar to a GoogLeNet network without an input layer, a final pool layer, and output layers [Szegedy et al. 2014] . This approach allows DetectNet to use the already trained GoogLeNet model, reduce training time and improve the accuracy of the full model. A fully convolutional network (FCN) is a convolutional neural network without completely connected layers. This means that the network can receive images of various sizes at the input and, in the usual way, count the response using the sliding window technique with a step. The output is a multidimensional array of real values, which can be superimposed on the input image, like the input marks and the square grid from DetectNet. As a result, a GoogLeNet network without a final pool layerrepresents a certain convolutional neural network with a sliding window of 555 x 555 pixels in size and a pitch of 16 pixels [1] .

DetectNet uses a linear combination of two independent loss functions to create the ultimate loss function and optimization. The first loss function coverage_lossis the quadratic error for all squares of the source data lattice between the present and the predicted coverage of the object:

The second function bbox_lossis the average error between the real and the predicted angles of the bounding rectangle over all the squares of the lattice:

The Caffe framework minimizes the weighted sum of the values ​​of these loss functions.

DetectNet Network Output

The last layers of the DetectNet network filter and cluster the set of generated bounding rectangles for grid squares. For this, the groupRectangles algorithm from the OpenCV library is used. Bounding rectangles are filtered by the threshold method according to the value of the predicted coverage of the object. The threshold value is set by parameter gridbox_cvg_thresholdin the file prototxt DetectNet model. Bounding rectangles are clustered using the equivalence criterion for rectangles, which combines shapes of similar location and size. The similarity of the rectangles is determined by the variableeps: with a zero value, the rectangles are not combined, but with a value tending to plus infinity, all the rectangles fall into one cluster. After combining the rectangles into clusters, there is a threshold filtering of small clusters with a threshold specified by the parameter gridbox_rect_thresh, and the remaining rectangles are counted as middle rectangles, which are recorded in the output list. The clustering method is implemented by the function in Python and is called in Caffe via the “Python Layers“ interface. Algorithm parameters groupRectanglesare set through a layer clusterin the DetectNet network model file.

In DetectNet, the Python Layers interface is also used to compute and output the simplified metric mAP (mean Average Precision), calculated from the final set of bounding rectangles. For the predicted and present bounding boxes, Intersection over Union (IoU) is calculated.- the ratio of the area of ​​intersection of the rectangles to the sum of their areas. When using the threshold value (default 0.7) for IoU, the predicted rectangle can be classified as true-positive or false-positive predictions. In the case when the IoU value for a pair of rectangles does not exceed the threshold value, the predicted rectangle falls into the category of false negative predictions - the object was not detected. Thus, the simplified mAP metric in DetectNet is calculated as the product of accuracy (precision is the ratio of true positive to the sum of true positive and false positive) and the measure of completeness (recall is the ratio of true positive to the sum of true positive and true negative).

Note trans. Useful picture from Wikipedia about accuracy and completeness

This metric is a convenient characteristic of the sensitivity of the DetectNet network to the detection of training sample objects, discarding false results and the accuracy of bounding boxes. More detailed information on the analysis of errors in object detection can be found in [Hoiem et al. 2012] .

Learning Effectiveness and Results

The main advantage of the DetectNet network for the object detection problem is the efficiency with which objects are detected and the accuracy of the generated bounding rectangles. The presence of a fully convolutional network (FCN) allows DetectNet to be more efficient by comparison using a classifier based on a neural network on a sliding window. This avoids unnecessary calculations associated with overlapping windows. This approach with a unified neural network architecture is also simpler and more elegant for solving the detection problem.

DetectNet training in DIGITS 4 with Nvidia Caffe 0.15.7 and cuDNN RC 5.1 on a sample of 307 training and 24 test images, 1536 x 1024 pixels in size, takes 63 minutes using a single Titan X graphics card.

DetectNet detects objects on images with a size of 1536 x 1024 pixels and a lattice size of 16 pixels, in the previous configuration (one TitanX, Nvidia Caffe 0.15.7, cuDNN RC 5.1) takes 41 ms (approximately 24 fps).

First Steps with DetectNet

If you want to try DetectNet on your own data, you can download DIGITS 4 . A step-by-step demonstration of the workflow for detecting objects in DIGITS is presented here .
Also read the Deep Learning for Object Detection with DIGITS post for a tutorial on how to use the object detection functionality in DIGITS 4.

Если вас интересуют плюсы и минусы различных подходов с использованием глубокого обучения в задаче обнаружения объектов, смотрите выступление Jon Barker на GTC 2016

Другие обучающие материалы по глубокому обучению, включая вебинары итд, можно найти в NVIDIA Deep Learning Institute.

Ссылки

Hoiem, D., Chodpathumwan, Y., and Dai, Q. 2012. Diagnosing Error in Object Detectors. Computer Vision – ECCV 2012, Springer Berlin Heidelberg, 340–353.

Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. 2015. You Only Look Once: Unified, Real-Time Object Detection. arXiv [cs.CV]. http://arxiv.org/abs/1506.02640.

Szegedy, C., Liu, W., Jia, Y., et al. 2014. Going Deeper with Convolutions. arXiv [cs.CV]. http://arxiv.org/abs/1409.4842.

Дополнительная информация и ссылки от переводчика

Оригинал статьи можно найти здесь. Проект DIGITS является открытым и источники можно найти здесь. Файл prototxt сети DetectNet можно найти здесь или в виде изображения тут.

Про установку DIGITS

Подробную инструкцию по установке можно найти здесь.
Приложению DIGITS для запуска необходим Caffe, а точнее форк от NVidia, куда добавлены слои Python, необходимые для сети DetectNet. Этот форк "без особых проблем" может устанавливается на Mac OSX и Ubuntu. С Windows проблема в том, что ветка windows от BVLC/Caffe отсутcтвует в форке, поэтому как пишут сами авторы: DIGITS for Windows does not support DetectNet. Таким образом, на Windows можно установить BVLC/Caffe и запускать "стандартные" сети.

Примечания

[1] Using GoogLeNet with it’s final pooling layer removed results in the sliding window application of a CNN with a receptive field of 555 x 555 pixels and a stride of 16 pixels.