We make a rating of Russian cities by road quality



Once again, driving a car around my hometown and going around another pit, I thought: did such “good” roads exist everywhere in our country and I decided that we should objectively assess the situation with the quality of roads in our country.

Task formalization


In Russia, requirements for the quality of roads are described in GOST R 50597-2017 “Roads and roads. Requirements for the operational state acceptable under the conditions of ensuring road safety. Control methods". This document defines the requirements for covering the carriageway, roadsides, dividing strips, sidewalks, pedestrian walkways, etc., and also establishes the types of damage.

Since the task of determining all parameters of the roads is quite extensive, I decided to narrow it down for myself and focus only on the problem of determining defects in the coverage of the roadway. In GOST R 50597-2017, the following defects in the coating of the roadway are distinguished:

  • potholes
  • breaks
  • drawdowns
  • shifts
  • combs
  • track
  • sweating binder

I decided to tackle these defects.

Data Collection


Where can I get photographs that depict sufficiently large sections of the roadway, and even with reference to geolocation? The answer came in rhinestones - panoramas on the maps of Yandex (or Google), however, after a bit of searching, I found several more alternative options:

  • issuing search engines for pictures for relevant requests;
  • photos on sites for receiving complaints (Rosyama, Angry citizen, Virtue, etc.)
  • Opendatascience prompted a project to detect road defects with a marked dataset - github.com/sekilab/RoadDamageDetector

Unfortunately, an analysis of these options showed that they are not very suitable for me: issuing search engines has a lot of noise (a lot of photos that are not roads, various renders, etc.), photos from sites for receiving complaints contain only photos with large violations of the asphalt surface , there are quite a few photos with small violations of coverage and without violations on these sites, the dataset from the RoadDamageDetector project is collected in Japan and does not contain samples with large violations of coverage, as well as roads without coverage at all.

Since alternative options are not suitable, we will use Yandex panoramas (I excluded the Google panorama option, since the service is presented in fewer cities in Russia and is updated less frequently). He decided to collect data in cities with a population of more than 100 thousand people, as well as in federal centers. I made a list of city names - there were 176 of them, later it turns out that only 149 of them have panoramas. I will not delve into the features of parsing tiles, I will say that in the end I got 149 folders (one for each city) in which there were a total of 1.7 million photos. For example, for Novokuznetsk, the folder looked like this:



By the number of downloaded photos, the cities were distributed as follows:

Table
Town
Number of photos, pcs
Moscow

86048

St. Petersburg

41376

Saransk

18880

Podolsk

18560

Krasnogorsk

18208

Lyubertsy

17760

Kaliningrad

16928

Kolomna

16832

Mytishchi

16192

Vladivostok

16096

Balashikha

15968

Petrozavodsk

15968

Ekaterinburg

15808

Velikiy Novgorod

15744

Naberezhnye Chelny

15680

Krasnodar

15520

Nizhny Novgorod

15488

Khimki

15296

Tula

15296

Novosibirsk

15264

Tver

15200

Miass

15104

Ivanovo

15072

Vologda

15008

Zhukovsky

14976

Kostroma

14912

Samara

14880

Korolev

14784

Kaluga

14720

Cherepovets

14720

Sevastopol

14688

Pushkino

14528

Yaroslavl

14464

Ulyanovsk

14400

Rostov-on-Don

14368

Domodedovo

14304

Kamensk-Uralsky

14208

Pskov

14144

Yoshkar-Ola

14080

Kerch

14080

Murmansk

13920

Tolyatti

13920

Vladimir

13792

Eagle

13792

Syktyvkar

13728

Dolgoprudny

13696

Khanty-Mansiysk

13664

Kazan

13600

Engels

13440

Arkhangelsk

13280

Bryansk

13216

Omsk

13120

Syzran

13088

Krasnoyarsk

13056

Shchelkovo

12928

Penza

12864

Chelyabinsk

12768

Cheboksary

12768

Nizhny Tagil

12672

Stavropol

12672

Ramenskoye

12640

Irkutsk

12608

Angarsk

12608

Tyumen

12512

Odintsovo

12512

Ufa

12512

Magadan

12512

Permian

12448

Kirov

12256

Nizhnekamsk

12224

Makhachkala

12096

Nizhnevartovsk

11936

Kursk

11904

Sochi

11872

Tambov

11840

Pyatigorsk

11808

Volgodonsk

11712

Ryazan

11680

Saratov

11616

Dzerzhinsk

11456

Orenburg

11456

Mound

11424

Volgograd

11264

Izhevsk

11168

Chrysostom

11136

Lipetsk

11072

Kislovodsk

11072

Surgut

11040

Magnitogorsk

10912

Smolensk

10784

Khabarovsk

10752

Kopeysk

10688

Maykop

10656

Petropavlovsk-Kamchatsky

10624

Taganrog

10560

Barnaul

10528

Sergiev Posad

10368

Elista

10304

Sterlitamak

9920

Simferopol

9824

Tomsk

9760

Orekhovo-Zuevo

9728

Astrakhan

9664

Evpatoria

9568

Noginsk

9344

Chita

9216

Belgorod

9120

Biysk

8928

Rybinsk

8896

Severodvinsk

8832

Voronezh

8768

Blagoveshchensk

8672

Novorossiysk

8608

Ulan-Ude

8576

Serpukhov

8320

Komsomolsk-on-Amur

8192

Abakan

8128

Norilsk

8096

Yuzhno-Sakhalinsk

8032

Obninsk

7904

Essentuki

7712

Bataysk

7648

Volzhsky

7584

Novocherkassk

7488

Berdsk

7456

Arzamas

7424

Pervouralsk

7392

Kemerovo

7104

Elektrostal

6720

Derbent

6592

Yakutsk

6528

Murom

6240

Nefteyugansk

5792

Reutov

5696

Birobidzhan

5440

Novokuybyshevsk

5248

Salekhard

5184

Novokuznetsk

5152

New Urengoy

4736

Noyabrsk

4416

Novocheboksarsk

4352

Dace

3968

Kaspiysk

3936

Stary Oskol

3840

Artyom

3744

Zheleznogorsk

3584

Salavat

3584

Prokopyevsk

2816

Gorno-Altaysk

2464



Preparing a dataset for training


And so, the dataset is assembled, how now, having a photo of the road section and the enclosing objects, find out the quality of the asphalt depicted on it? I decided to cut out a piece of the photo measuring 350 * 244 pixels in the center of the original photo just below the middle. Then reduce the cut piece horizontally to a size of 244 pixels. The resulting image (244 * 244 in size) will be the input for the convolutional encoder:



To better understand what data I deal with, I marked the first 2000 pictures myself, the remaining pictures were marked by Yandex.Tolki workers. Before them I posed a question in the following wording.

Indicate what road surface you see in the photo:

  1. Soil / Rubble
  2. Paving stones, tile, pavement
  3. Rails, railway tracks
  4. Water, large puddles
  5. Asphalt
  6. There is no road in the photo / Foreign objects / Coverage is not visible due to cars

If the performer chose “Asphalt”, a menu appeared that offered to evaluate its quality:

  1. Excellent coverage
  2. Slight single cracks / shallow single potholes
  3. Large cracks / Grid cracks / single minor potholes
  4. Large Potholes / Deep Potholes / Destroyed Coating

As test runs of the tasks showed, the performers of Y. Toloki do not differ in the integrity of the work - they accidentally click on the fields with the mouse and consider the task completed. I had to add control questions (in the assignment there were 46 photographs, 12 of which were control) and enable delayed acceptance. As control questions, I used those pictures that I marked out myself. I automated the delayed acceptance - Y. Toloka allows you to upload the work results to a CSV file, and load the results of verification of responses. Verification of answers worked as follows - if the task contains more than 5% incorrect answers to control questions, then it is considered to be unfulfilled. Moreover, if the contractor indicated an answer that is logically close to true, then his answer is considered correct.
As a result, I got about 30 thousand tagged photos, which I decided to distribute in three classes for training:

  • “Good” - photos labeled “Asphalt: Excellent Coating” and “Asphalt: Minor Single Cracks”
  • “Middle” - photos labeled “Paving stones, tiles, pavement”, “Rails, railway tracks” and “Asphalt: Large cracks / Grid cracks / single minor potholes”
  • “Large” - photos labeled “Soil / Crushed stone”, “Water, large puddles” and “Asphalt: A large number of potholes / Deep potholes / Destroyed pavement”
  • Photos tagged “There is no road in the photo / Foreign objects / Coverage is not visible due to cars” there were very few (22 pcs.) And I excluded them from further work

Classifier development and training


So, the data is collected and labeled, we proceed to the development of the classifier. Usually, for the tasks of image classification, especially when training on small datasets, a ready-made convolutional encoder is used, to the output of which a new classifier is connected. I decided to use a simple classifier without a hidden layer, an input layer of size 128 and an output layer of size 3. I decided to immediately use several ready-made options trained on ImageNet as encoders:

  • Xception
  • Resnet
  • Inception
  • Vgg16
  • Densenet121
  • Mobilenet

Here is the function that creates the Keras model with the given encoder:

def createModel(typeModel):
  conv_base = None
  if(typeModel == "nasnet"):
    conv_base = keras.applications.nasnet.NASNetMobile(include_top=False,
                                                    input_shape=(224,224,3),
                                                    weights='imagenet')
  if(typeModel == "xception"):
    conv_base = keras.applications.xception.Xception(include_top=False,
                                                    input_shape=(224,224,3),
                                                    weights='imagenet')
  if(typeModel == "resnet"):
    conv_base = keras.applications.resnet50.ResNet50(include_top=False,
                                                    input_shape=(224,224,3),
                                                    weights='imagenet')
  if(typeModel == "inception"):
    conv_base = keras.applications.inception_v3.InceptionV3(include_top=False,
                                                    input_shape=(224,224,3),
                                                    weights='imagenet')
  if(typeModel == "densenet121"):
    conv_base = keras.applications.densenet.DenseNet121(include_top=False,
                                                    input_shape=(224,224,3),
                                                    weights='imagenet')
  if(typeModel == "mobilenet"):
    conv_base = keras.applications.mobilenet_v2.MobileNetV2(include_top=False,
                                                    input_shape=(224,224,3),
                                                    weights='imagenet')
  if(typeModel == "vgg16"):
    conv_base = keras.applications.vgg16.VGG16(include_top=False,
                                                    input_shape=(224,224,3),
                                                    weights='imagenet')
  conv_base.trainable = False
  model = Sequential()
  model.add(conv_base)
  model.add(Flatten())
  model.add(Dense(128, 
                   activation='relu',
                   kernel_regularizer=regularizers.l2(0.0002)))
  model.add(Dropout(0.3))
  model.add(Dense(3, activation='softmax'))
  model.compile(optimizer=keras.optimizers.Adam(lr=1e-4),
                loss='binary_crossentropy',
                metrics=['accuracy'])
  return model

For training, I used a generator with augmentation (since the possibilities of the augmentation built into Keras seemed to me insufficient, then I used the Augmentor library ):

  • Slopes
  • Random distortion
  • Turns
  • Color swap
  • Shifts
  • Change contrast and brightness
  • Adding random noise
  • Crop

After augmentation, the photos looked like this:



Generator code:


def get_datagen():
  train_dir='~/data/train_img'
  test_dir='~/data/test_img'
  testDataGen = ImageDataGenerator(rescale=1. / 255)
  train_generator = datagen.flow_from_directory(
      train_dir,
      target_size=img_size,
      batch_size=16,
      class_mode='categorical')
  p = Augmentor.Pipeline(train_dir)  
  p.skew(probability=0.9)
  p.random_distortion(probability=0.9,grid_width=3,grid_height=3,magnitude=8)
  p.rotate(probability=0.9, max_left_rotation=5, max_right_rotation=5)
  p.random_color(probability=0.7, min_factor=0.8, max_factor=1)
  p.flip_left_right(probability=0.7)
  p.random_brightness(probability=0.7, min_factor=0.8, max_factor=1.2)
  p.random_contrast(probability=0.5, min_factor=0.9, max_factor=1)
  p.random_erasing(probability=1,rectangle_area=0.2)
  p.crop_by_size(probability=1, width=244, height=244, centre=True)  
  train_generator = keras_generator(p,batch_size=16)   
  test_generator  = testDataGen.flow_from_directory(
      test_dir,
      target_size=img_size,
      batch_size=32,
      class_mode='categorical')
  return (train_generator, test_generator)

The code shows that augmentation is not used for test data.

Having a tuned generator, you can start training the model, we will carry out it in two stages: first, only train our classifier, then completely the entire model.


def evalModelstep1(typeModel): 
  K.clear_session()
  gc.collect()
  model=createModel(typeModel)
  traiGen,testGen=getDatagen()
  model.fit_generator(generator=traiGen,  
                    epochs=4,
                    steps_per_epoch=30000/16,
                    validation_steps=len(testGen),
                    validation_data=testGen,                    
                   )
  return model
def evalModelstep2(model):
  early_stopping_callback = EarlyStopping(monitor='val_loss', patience=3)  
  model.layers[0].trainable=True
  model.trainable=True
  model.compile(optimizer=keras.optimizers.Adam(lr=1e-5),
              loss='binary_crossentropy',
              metrics=['accuracy'])
  traiGen,testGen=getDatagen()
  model.fit_generator(generator=traiGen,  
                  epochs=25,
                  steps_per_epoch=30000/16,
                  validation_steps=len(testGen),
                  validation_data=testGen,
                  callbacks=[early_stopping_callback]
                 ) 
  return model
def full_fit():
  model_names=[      
            "xception",
            "resnet",
            "inception",       
            "vgg16",
            "densenet121",       
            "mobilenet"
  ]
  for model_name in model_names:
    print("#########################################")
    print("#########################################")
    print("#########################################")
    print(model_name)
    print("#########################################")
    print("#########################################")
    print("#########################################")
    model = evalModelstep1(model_name)
    model = evalModelstep2(model)
    model.save("~/data/models/model_new_"+str(model_name)+".h5")

Call full_fit () and wait. We are waiting for a long time.

As a result, we will have six trained models, we will check the accuracy of these models on a separate portion of labeled data; I received the following:

Model name


Accuracy%


Xception


87.3


Resnet


90.8


Inception


90.2


Vgg16


89.2


Densenet121


90.6


Mobilenet


86.5



In general, not a lot, but with such a small training sample, one can not expect more. To slightly increase the accuracy, I combined the outputs of the models by averaging:


def create_meta_model():
  model_names=[      
            "xception",
            "resnet",
            "inception",       
            "vgg16",
            "densenet121",       
            "mobilenet"
  ]
  model_input = Input(shape=(244,244,3))
  submodels=[]
  i=0;
  for model_name in model_names:
    filename= "~/data/models/model_new_"+str(model_name)+".h5"   
    submodel = keras.models.load_model(filename)
    submodel.name = model_name+"_"+str(i)
    i+=1
    submodels.append(submodel(model_input))
  out=average(submodels)
  model = Model(inputs = model_input,outputs=out)
  model.compile(optimizer=keras.optimizers.Adam(lr=1e-4),
                loss='binary_crossentropy',
                metrics=['accuracy'])
  return model

The resulting accuracy was 91.3%. On this result, I decided to stop.

Using Classifier


Finally the classifier is ready and it can be put into action! I prepare the input data and run the classifier - a little more than a day and 1.7 million photos have been processed. Now the fun part is the results. Immediately bring the first and last ten cities in the relative number of roads with good coverage:



Full table (clickable picture)



And here is the road quality rating by federal subjects:



Full table


Rating by federal districts:



Distribution of road quality in Russia as a whole:



Well, that’s all, everyone can draw conclusions himself.

Finally, I will give the best photos in each category (which received the maximum value in their class):

Picture



PS In the comments, quite rightly pointed out the lack of statistics on the years of receipt of the photos. I correct and give a table:

Year


Number of photos, pcs


200837
2009thirteen
2010157030
201160724
201242387
201312148

2014141021

201546143

2016410385

2017324279

2018581961


Also popular now: