Artificial intelligence against lies and deceit

In all the tasks of learning artificial intelligence there is one most unpleasant phenomenon - errors in the marking of the training sequence. These errors are inevitable, since all the markup is done manually, because if there is a way to mark up real data programmatically, then why do we need someone else to teach them to mark and waste time and money to create an absolutely unnecessary construction!

The task of finding and removing fake masks in a large training sequence is quite complex, You can view them all manually, but this will not save you from repeated errors. But if you look closely at the neural network research tools proposed in previous posts , it turns out there is a simple and effective way to detect and extract all the artifacts from the training sequence.

And in this post there is a specific example, it is obvious that a simple, on ellipses and polygons, for an ordinary U-net, is again a Lego in the sandbox, but unusually concrete, useful and effective. We show how a simple method reveals and finds almost all the artifacts, all the lies of the training sequence.

So, let's begin!

As before, we will study the sequences of the picture / mask pairs. In the picture in different quarters chosen randomly we will place an ellipse of random size and a quadrilateral also of arbitrary size and both of them are painted in the same color, also randomly chosen from two of them. In the second remaining color we paint the background. Sizes of both ellipse and quad are of course limited.

But in this case, we’ll make changes to the steam generation program and prepare together with a completely correct mask an incorrect one, poisoned by a lie - in approximately one percent of cases we replace the quadrilateral with an ellipse in the mask, i.e. The true object for segmentation is the ellipse rather than the quadrilateral on the false masks.

Examples of random 10



Examples of random 10, but from erroneous markup. The upper mask is true, the lower is false and the numbers in the pictures are in the training sequence.



for segmentation, we take the same metrics and loss calculation programs and the same simple U-net, just don’t use Dropout.

Libraries
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import NoNorm
%matplotlib inline
import math
from tqdm import tqdm
#from joblib import Parallel, delayedfrom skimage.draw import ellipse, polygon
from keras import Model
from keras.optimizers import Adam
from keras.layers import Input,Conv2D,Conv2DTranspose,MaxPooling2D,concatenate
from keras.layers import BatchNormalization,Activation,Add,Dropout
from keras.losses import binary_crossentropy
from keras import backend as K
from keras.models import load_model
import tensorflow as tf
import keras as keras
w_size = 128
train_num = 10000
radius_min = 10
radius_max = 30


Metric and loss functions
defdice_coef(y_true, y_pred):
    y_true_f = K.flatten(y_true)
    y_pred = K.cast(y_pred, 'float32')
    y_pred_f = K.cast(K.greater(K.flatten(y_pred), 0.5), 'float32')
    intersection = y_true_f * y_pred_f
    score = 2. * K.sum(intersection) / (K.sum(y_true_f) + K.sum(y_pred_f))
    return score
defdice_loss(y_true, y_pred):
    smooth = 1.
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)
    intersection = y_true_f * y_pred_f
    score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)
    return1. - score
defbce_dice_loss(y_true, y_pred):return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)
defget_iou_vector(A, B):# Numpy version
    batch_size = A.shape[0]
    metric = 0.0for batch in range(batch_size):
        t, p = A[batch], B[batch]
        true = np.sum(t)
        pred = np.sum(p)
        # deal with empty mask firstif true == 0:
            metric += (pred == 0)
            continue# non empty mask case.  Union is never empty # hence it is safe to divide by its number of pixels
        intersection = np.sum(t * p)
        union = true + pred - intersection
        iou = intersection / union
        # iou metrric is a stepwise approximation of the real iou over 0.5
        iou = np.floor(max(0, (iou - 0.45)*20)) / 10
        metric += iou
    # teake the average over all images in batch
    metric /= batch_size
    return metric
defmy_iou_metric(label, pred):# Tensorflow versionreturn tf.py_func(get_iou_vector, [label, pred > 0.5], tf.float64)
from keras.utils.generic_utils import get_custom_objects
get_custom_objects().update({'bce_dice_loss': bce_dice_loss })
get_custom_objects().update({'dice_loss': dice_loss })
get_custom_objects().update({'dice_coef': dice_coef })
get_custom_objects().update({'my_iou_metric': my_iou_metric })


Normal U-net
defbuild_model(input_layer, start_neurons):# 128 -> 64
    conv1 = Conv2D(start_neurons * 1, (3, 3),
                   activation="relu", padding="same")(input_layer)
    conv1 = Conv2D(start_neurons * 1, (3, 3),
                   activation="relu", padding="same")(conv1)
    pool1 = Conv2D(start_neurons * 1, (2, 2),
                   strides=(2, 2), activation="relu", padding="same")(conv1)
#    pool1 = Dropout(0.25)(pool1)# 64 -> 32
    conv2 = Conv2D(start_neurons * 2, (3, 3),
                   activation="relu", padding="same")(pool1)
    conv2 = Conv2D(start_neurons * 2, (3, 3),
                   activation="relu", padding="same")(conv2)
    pool2 = Conv2D(start_neurons * 1, (2, 2),
                   strides=(2, 2), activation="relu", padding="same")(conv2)
#    pool2 = Dropout(0.5)(pool2)# 32 -> 16
    conv3 = Conv2D(start_neurons * 4, (3, 3),
                   activation="relu", padding="same")(pool2)
    conv3 = Conv2D(start_neurons * 4, (3, 3),
                   activation="relu", padding="same")(conv3)
    pool3 = Conv2D(start_neurons * 1, (2, 2),
                   strides=(2, 2), activation="relu", padding="same")(conv3)
#    pool3 = Dropout(0.5)(pool3)# 16 -> 8
    conv4 = Conv2D(start_neurons * 8, (3, 3),
                   activation="relu", padding="same")(pool3)
    conv4 = Conv2D(start_neurons * 8, (3, 3),
                   activation="relu", padding="same")(conv4)
    pool4 = Conv2D(start_neurons * 1, (2, 2),
                   strides=(2, 2), activation="relu", padding="same")(conv4)
#    pool4 = Dropout(0.5)(pool4)# Middle
    convm = Conv2D(start_neurons * 16, (3, 3),
                   activation="relu", padding="same")(pool4)
    convm = Conv2D(start_neurons * 16, (3, 3)
                   , activation="relu", padding="same")(convm)
    # 8 -> 16
    deconv4 = Conv2DTranspose(start_neurons * 8,
                              (3, 3), strides=(2, 2), padding="same")(convm)
    uconv4 = concatenate([deconv4, conv4])
#    uconv4 = Dropout(0.5)(uconv4)
    uconv4 = Conv2D(start_neurons * 8, (3, 3)
                    , activation="relu", padding="same")(uconv4)
    uconv4 = Conv2D(start_neurons * 8, (3, 3)
                    , activation="relu", padding="same")(uconv4)
    # 16 -> 32
    deconv3 = Conv2DTranspose(start_neurons * 4,
                              (3, 3), strides=(2, 2), padding="same")(uconv4)
    uconv3 = concatenate([deconv3, conv3])
#    uconv3 = Dropout(0.5)(uconv3)
    uconv3 = Conv2D(start_neurons * 4, (3, 3)
                    , activation="relu", padding="same")(uconv3)
    uconv3 = Conv2D(start_neurons * 4, (3, 3)
                    , activation="relu", padding="same")(uconv3)
    # 32 -> 64
    deconv2 = Conv2DTranspose(start_neurons * 2,
                              (3, 3), strides=(2, 2), padding="same")(uconv3)
    uconv2 = concatenate([deconv2, conv2])
#    uconv2 = Dropout(0.5)(uconv2)
    uconv2 = Conv2D(start_neurons * 2, (3, 3)
                    , activation="relu", padding="same")(uconv2)
    uconv2 = Conv2D(start_neurons * 2, (3, 3)
                    , activation="relu", padding="same")(uconv2)
    # 64 -> 128
    deconv1 = Conv2DTranspose(start_neurons * 1,
                              (3, 3), strides=(2, 2), padding="same")(uconv2)
    uconv1 = concatenate([deconv1, conv1])
#    uconv1 = Dropout(0.5)(uconv1)
    uconv1 = Conv2D(start_neurons * 1, (3, 3)
                    , activation="relu", padding="same")(uconv1)
    uconv1 = Conv2D(start_neurons * 1, (3, 3)
                    , activation="relu", padding="same")(uconv1)
#    uncov1 = Dropout(0.5)(uconv1)
    output_layer = Conv2D(1, (1,1), padding="same", activation="sigmoid")(uconv1)
    return output_layer
input_layer = Input((w_size, w_size, 1))
output_layer = build_model(input_layer, 27)
model = Model(input_layer, output_layer)
model.compile(loss=bce_dice_loss, optimizer=Adam(lr=1e-4), metrics=[my_iou_metric])
model.summary()


The program for generating images and masks - true and fake. In the array is placed the first layer of the picture, the second true mask and the third layer of the false mask.

defnext_pair_f(idx):
    img_l = np.ones((w_size, w_size, 1), dtype='float')*0.45
    img_h = np.ones((w_size, w_size, 1), dtype='float')*0.55
    img = np.zeros((w_size, w_size, 3), dtype='float')
    i0_qua = math.trunc(np.random.sample()*4.)
    i1_qua = math.trunc(np.random.sample()*4.)
    while i0_qua == i1_qua:
        i1_qua = math.trunc(np.random.sample()*4.)
    _qua = np.int(w_size/4)
    qua = np.array([[_qua,_qua],[_qua,_qua*3],[_qua*3,_qua*3],[_qua*3,_qua]])
    p = np.random.sample() - 0.5
    r = qua[i0_qua,0]
    c = qua[i0_qua,1]
    r_radius = np.random.sample()*(radius_max-radius_min) + radius_min
    c_radius = np.random.sample()*(radius_max-radius_min) + radius_min
    rot = np.random.sample()*360
    rr, cc = ellipse(
        r, c, 
        r_radius, c_radius, 
        rotation=np.deg2rad(rot), 
        shape=img_l.shape
    )
    p0 = np.rint(np.random.sample()*(radius_max-radius_min) + radius_min)
    p1 = qua[i1_qua,0] - (radius_max-radius_min)
    p2 = qua[i1_qua,1] - (radius_max-radius_min)
    p3 = np.rint(np.random.sample()*radius_min)
    p4 = np.rint(np.random.sample()*radius_min)
    p5 = np.rint(np.random.sample()*radius_min)
    p6 = np.rint(np.random.sample()*radius_min)
    p7 = np.rint(np.random.sample()*radius_min)
    p8 = np.rint(np.random.sample()*radius_min)
    poly = np.array((
        (p1, p2),
        (p1+p3, p2+p4+p0),
        (p1+p5+p0, p2+p6+p0),
        (p1+p7+p0, p2+p8),
        (p1, p2),
    ))
    rr_p, cc_p = polygon(poly[:, 0], poly[:, 1], img_l.shape)
    if p > 0:
        img[:,:,:1] = img_l.copy()
        img[rr, cc,:1] = img_h[rr, cc]
        img[rr_p, cc_p,:1] = img_h[rr_p, cc_p]
    else:
        img[:,:,:1] = img_h.copy()
        img[rr, cc,:1] = img_l[rr, cc]
        img[rr_p, cc_p,:1] = img_l[rr_p, cc_p]
    img[:,:,1] = 0.
    img[:,:,1] = 0.
    img[rr_p, cc_p,1] = 1.
    img[:,:,2] = 0.
    p_f = np.random.sample()*1000.if p_f > 10:
        img[rr_p, cc_p,2] = 1.else:
        img[rr, cc,2] = 1.
        i_false[idx] = 1return img

Program for the calculation of the cheat sheet
defmake_sh(f_imgs, f_msks, val_len):
    precision = 0.85
    batch_size = 50
    t = tqdm()
    t_batch_size = 50
    raw_len = val_len
    id_train = 1#id_select = 1
    v_false = np.zeros((train_num), dtype='float')
    whileTrue:
        if id_train == 1:
            fit = model.fit(f_imgs[m2_select>0], f_msks[m2_select>0],
                            batch_size=batch_size,
                            epochs=1,
                            verbose=0
                           )
            current_accu = fit.history['my_iou_metric'][0]
            current_loss = fit.history['loss'][0]
            if current_accu > precision:
                id_train = 0else:
            t_pred = model.predict(
                f_imgs[raw_len: min(raw_len+t_batch_size,f_imgs.shape[0])],
                batch_size=batch_size
                                  )
            for kk in range(t_pred.shape[0]):
                val_iou = get_iou_vector(
                    f_msks[raw_len+kk].reshape(1,w_size,w_size,1),
                    t_pred[kk].reshape(1,w_size,w_size,1) > 0.5)
                v_false[raw_len+kk] = val_iou
                if val_iou < precision*0.95:
                    new_img_test = 1
                    m2_select[raw_len+kk] = 1                
                    val_len += 1break
            raw_len += (kk+1)
            id_train = 1
        t.set_description("Accuracy {0:6.4f} loss {1:6.4f} selected img {2:5d} tested img {3:5d} ".
                          format(current_accu, current_loss, val_len, raw_len))
        t.update(1)
        if raw_len >= train_num:
            break
    t.close()
    return v_false


The main program of calculations. We made minor changes to the same program from the previous post and some variables require clarification and comment.

i_false = np.zeros((train_num), dtype='int')

Here is an indicator of the mask falsity. If 1, then the mask from F_msks does not match the mask from f_msks. This is an indicator of what we are actually looking for - false masks.

m2_select = np.zeros((train_num), dtype='int')

Indicator that this picture is selected in the cheat sheet.

batch_size = 50
val_len = batch_size + 1# i_false - false mask marked as 1
i_false = np.zeros((train_num), dtype='int')
# t_imgs, t_msks -test images and masks
_txy = [next_pair_f(idx) for idx in range(train_num)]
t_imgs = np.array(_txy)[:,:,:,:1].reshape(-1,w_size ,w_size ,1)
t_msks = np.array(_txy)[:,:,:,1].reshape(-1,w_size ,w_size ,1)
# m2_select - initial 51 pair
m2_select = np.zeros((train_num), dtype='int')
for k in range(val_len):
    m2_select[k] = 1# i_false - false mask marked as 1
i_false = np.zeros((train_num), dtype='int')
_txy = [next_pair_f(idx) for idx in range(train_num)]
f_imgs = np.array(_txy)[:,:,:,:1].reshape(-1,w_size ,w_size ,1)
f_msks = np.array(_txy)[:,:,:,1].reshape(-1,w_size ,w_size ,1)
# F_msks - mask array with ~1% false mask
F_msks = np.array(_txy)[:,:,:,2].reshape(-1,w_size ,w_size ,1)
fig, axes = plt.subplots(2, 10, figsize=(20, 5))
for k in range(10):
    kk = np.random.randint(train_num)
    axes[0,k].set_axis_off()
    axes[0,k].imshow(f_imgs[kk].squeeze(), cmap="gray", norm=NoNorm())
    axes[1,k].set_axis_off()
    axes[1,k].imshow(f_msks[kk].squeeze(), cmap="gray", norm=NoNorm())
plt.show(block=True)
false_num = np.arange(train_num)[i_false>0]
fig, axes = plt.subplots(3, 10, figsize=(20, 7))
for k in range(10):
    kk = np.random.randint(false_num.shape[0])
    axes[0,k].set_axis_off()
    axes[0,k].set_title(false_num[kk])
    axes[0,k].imshow(f_imgs[false_num[kk]].squeeze(), cmap="gray", norm=NoNorm())
    axes[1,k].set_axis_off()
    axes[1,k].imshow(f_msks[false_num[kk]].squeeze(), cmap="gray", norm=NoNorm())
    axes[2,k].set_axis_off()
    axes[2,k].imshow(F_msks[false_num[kk]].squeeze(), cmap="gray", norm=NoNorm())
plt.show(block=True)

We build sequences of pairs of picture / mask for training and another sequence for testing. Those. We will check on a new, independent sequence of 10,000 pairs. We display and visually selectively check random pictures with true and false masks. The pictures themselves are shown above.

In this particular case, there were 93 fake masks, in which an ellipse, not a quad, is marked as a mask.

We start training on the correct set, as a mask we use f_msks

input_layer = Input((w_size, w_size, 1))
output_layer = build_model(input_layer, 25)
model = Model(input_layer, output_layer)
model.compile(loss=bce_dice_loss, optimizer=Adam(lr=1e-4), metrics=[my_iou_metric])
v_false = make_sh(f_imgs, f_msks, val_len)
t_pred = model.predict(t_imgs,batch_size=batch_size)
print (get_iou_vector(t_msks,t_pred.reshape(-1,w_size ,w_size ,1)))

Accuracy 0.9807 loss 0.0092 selected img   404 tested img 10000 : : 1801it [08:13,  3.65it/s]
0.9895299999999841

The cheat sheet turned out to be only 404 pictures and obtained acceptable accuracy on an independent test sequence.

Now we re-compile the network and train on the same training sequence, but as masks, we feed F_msks with 1% false masks

input_layer = Input((w_size, w_size, 1))
output_layer = build_model(input_layer, 25)
model = Model(input_layer, output_layer)
model.compile(loss=bce_dice_loss, optimizer=Adam(lr=1e-4), metrics=[my_iou_metric])
v_false = make_sh(f_imgs, F_msks, val_len)
t_pred = model.predict(t_imgs,batch_size=batch_size)
print (get_iou_vector(t_msks,t_pred.reshape(-1,w_size ,w_size ,1)))

Accuracy 0.9821 loss 0.0324 selected img   727 tested img 10000 : : 1679it [25:44,  1.09it/s]
0.9524099999999959

We got a cheat sheet in 727 pictures, which is significantly more and the accuracy of the test predictions, the same as in the previous test, was reduced from 0.98953 to 0.9525. We added lies to the training sequence by less than 1%, only 93 out of 10,000 masks were lies, but the result worsened by 3.7%. And this is no longer just a lie, this is the real treachery! And the cheat sheet has increased from just 404 to 727 already.

Soothes and pleases only one

print (len(set(np.arange(train_num)[m2_select>0]).intersection(set(np.arange(train_num)[i_false>0]))))
93

Let me explain this long formula, we take the intersection of the set of pictures selected in the cheat sheet with a lot of false pictures and see that all 93 false pictures the algorithm chose in the cheat sheet.

The task was simplified significantly, it is not 10,000 images viewed manually, it’s only 727 and all the lies are concentrated here.

But there is a more interesting and useful way. When we compiled the cheat sheet, we included only those picture / mask pairs whose prediction is less than the threshold, and in this particular case we saved the prediction accuracy value into the v_false array. Let's look at the pairs from the training sequence which have a very small prediction, for example, less than 0.1 and see how many lies there are.

print (len(set(np.arange(train_num)[v_false<0.01]).intersection(set(np.arange(train_num)[i_false>0]))))
89


As you can see, the main part of the false masks 89 out of 93 fell into these masks
np.arange(train_num)[v_false<0.01].shape
(382,)

Thus, if we test only 382 masks manually, and this is from 10,000 pieces, then most of the false masks will be identified and destroyed by us without any pity.

If it is possible to view pictures and masks while making decisions about their inclusion in the cheat sheet, then starting with a certain step, all the false masks, all lies will be determined by the minimum level of prediction already a little trained network, and the correct masks will have a prediction greater than this level .

Let's sum up


If in some imaginary world truth is always quadrangular, and oval lies and some unknown entity decided to distort the truth and called some ellipses true, and quadrilaterals are false, then using artificial intelligence and natural ability to make cheat sheets, the local inquisition will quickly and easily find and eradicate lies and deceit completely and cleaned.

PS: The ability to detect ovals, triangles, simple polygons is a necessary condition for creating any AI driving a car. If you don’t know how to search for ovals and triangles, you won’t find all the road signs and your AI will not go there by car.

Also popular now: