# Artificial intelligence against lies and deceit

In all the tasks of learning artificial intelligence there is one most unpleasant phenomenon - errors in the marking of the training sequence. These errors are inevitable, since all the markup is done manually, because if there is a way to mark up real data programmatically, then why do we need someone else to teach them to mark and waste time and money to create an absolutely unnecessary construction!

The task of finding and removing fake masks in a large training sequence is quite complex, You can view them all manually, but this will not save you from repeated errors. But if you look closely at the neural network research tools proposed in previous posts , it turns out there is a simple and effective way to detect and extract all the artifacts from the training sequence.

And in this post there is a specific example, it is obvious that a simple, on ellipses and polygons, for an ordinary U-net, is again a Lego in the sandbox, but unusually concrete, useful and effective. We show how a simple method reveals and finds almost all the artifacts, all the lies of the training sequence.

So, let's begin!

As before, we will study the sequences of the picture / mask pairs. In the picture in different quarters chosen randomly we will place an ellipse of random size and a quadrilateral also of arbitrary size and both of them are painted in the same color, also randomly chosen from two of them. In the second remaining color we paint the background. Sizes of both ellipse and quad are of course limited.

But in this case, we’ll make changes to the steam generation program and prepare together with a completely correct mask an incorrect one, poisoned by a lie - in approximately one percent of cases we replace the quadrilateral with an ellipse in the mask, i.e. The true object for segmentation is the ellipse rather than the quadrilateral on the false masks.

Examples of random 10

Examples of random 10, but from erroneous markup. The upper mask is true, the lower is false and the numbers in the pictures are in the training sequence.

for segmentation, we take the same metrics and loss calculation programs and the same simple U-net, just don’t use Dropout.

Libraries
``````import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import NoNorm
%matplotlib inline
import math
from tqdm import tqdm
#from joblib import Parallel, delayedfrom skimage.draw import ellipse, polygon
from keras import Model
from keras.layers import Input,Conv2D,Conv2DTranspose,MaxPooling2D,concatenate
from keras.losses import binary_crossentropy
from keras import backend as K
import tensorflow as tf
import keras as keras
w_size = 128
train_num = 10000

Metric and loss functions
``````defdice_coef(y_true, y_pred):
y_true_f = K.flatten(y_true)
y_pred = K.cast(y_pred, 'float32')
y_pred_f = K.cast(K.greater(K.flatten(y_pred), 0.5), 'float32')
intersection = y_true_f * y_pred_f
score = 2. * K.sum(intersection) / (K.sum(y_true_f) + K.sum(y_pred_f))
return score
defdice_loss(y_true, y_pred):
smooth = 1.
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
intersection = y_true_f * y_pred_f
score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)
return1. - score
defbce_dice_loss(y_true, y_pred):return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)
defget_iou_vector(A, B):# Numpy version
batch_size = A.shape[0]
metric = 0.0for batch in range(batch_size):
t, p = A[batch], B[batch]
true = np.sum(t)
pred = np.sum(p)
# deal with empty mask firstif true == 0:
metric += (pred == 0)
continue# non empty mask case.  Union is never empty # hence it is safe to divide by its number of pixels
intersection = np.sum(t * p)
union = true + pred - intersection
iou = intersection / union
# iou metrric is a stepwise approximation of the real iou over 0.5
iou = np.floor(max(0, (iou - 0.45)*20)) / 10
metric += iou
# teake the average over all images in batch
metric /= batch_size
return metric
defmy_iou_metric(label, pred):# Tensorflow versionreturn tf.py_func(get_iou_vector, [label, pred > 0.5], tf.float64)
from keras.utils.generic_utils import get_custom_objects
get_custom_objects().update({'bce_dice_loss': bce_dice_loss })
get_custom_objects().update({'dice_loss': dice_loss })
get_custom_objects().update({'dice_coef': dice_coef })
get_custom_objects().update({'my_iou_metric': my_iou_metric })
``````

Normal U-net
``````defbuild_model(input_layer, start_neurons):# 128 -> 64
conv1 = Conv2D(start_neurons * 1, (3, 3),
conv1 = Conv2D(start_neurons * 1, (3, 3),
pool1 = Conv2D(start_neurons * 1, (2, 2),
#    pool1 = Dropout(0.25)(pool1)# 64 -> 32
conv2 = Conv2D(start_neurons * 2, (3, 3),
conv2 = Conv2D(start_neurons * 2, (3, 3),
pool2 = Conv2D(start_neurons * 1, (2, 2),
#    pool2 = Dropout(0.5)(pool2)# 32 -> 16
conv3 = Conv2D(start_neurons * 4, (3, 3),
conv3 = Conv2D(start_neurons * 4, (3, 3),
pool3 = Conv2D(start_neurons * 1, (2, 2),
#    pool3 = Dropout(0.5)(pool3)# 16 -> 8
conv4 = Conv2D(start_neurons * 8, (3, 3),
conv4 = Conv2D(start_neurons * 8, (3, 3),
pool4 = Conv2D(start_neurons * 1, (2, 2),
#    pool4 = Dropout(0.5)(pool4)# Middle
convm = Conv2D(start_neurons * 16, (3, 3),
convm = Conv2D(start_neurons * 16, (3, 3)
# 8 -> 16
deconv4 = Conv2DTranspose(start_neurons * 8,
uconv4 = concatenate([deconv4, conv4])
#    uconv4 = Dropout(0.5)(uconv4)
uconv4 = Conv2D(start_neurons * 8, (3, 3)
uconv4 = Conv2D(start_neurons * 8, (3, 3)
# 16 -> 32
deconv3 = Conv2DTranspose(start_neurons * 4,
uconv3 = concatenate([deconv3, conv3])
#    uconv3 = Dropout(0.5)(uconv3)
uconv3 = Conv2D(start_neurons * 4, (3, 3)
uconv3 = Conv2D(start_neurons * 4, (3, 3)
# 32 -> 64
deconv2 = Conv2DTranspose(start_neurons * 2,
uconv2 = concatenate([deconv2, conv2])
#    uconv2 = Dropout(0.5)(uconv2)
uconv2 = Conv2D(start_neurons * 2, (3, 3)
uconv2 = Conv2D(start_neurons * 2, (3, 3)
# 64 -> 128
deconv1 = Conv2DTranspose(start_neurons * 1,
uconv1 = concatenate([deconv1, conv1])
#    uconv1 = Dropout(0.5)(uconv1)
uconv1 = Conv2D(start_neurons * 1, (3, 3)
uconv1 = Conv2D(start_neurons * 1, (3, 3)
#    uncov1 = Dropout(0.5)(uconv1)
output_layer = Conv2D(1, (1,1), padding="same", activation="sigmoid")(uconv1)
return output_layer
input_layer = Input((w_size, w_size, 1))
output_layer = build_model(input_layer, 27)
model = Model(input_layer, output_layer)
model.summary()``````

The program for generating images and masks - true and fake. In the array is placed the first layer of the picture, the second true mask and the third layer of the false mask.

``````defnext_pair_f(idx):
img_l = np.ones((w_size, w_size, 1), dtype='float')*0.45
img_h = np.ones((w_size, w_size, 1), dtype='float')*0.55
img = np.zeros((w_size, w_size, 3), dtype='float')
i0_qua = math.trunc(np.random.sample()*4.)
i1_qua = math.trunc(np.random.sample()*4.)
while i0_qua == i1_qua:
i1_qua = math.trunc(np.random.sample()*4.)
_qua = np.int(w_size/4)
qua = np.array([[_qua,_qua],[_qua,_qua*3],[_qua*3,_qua*3],[_qua*3,_qua]])
p = np.random.sample() - 0.5
r = qua[i0_qua,0]
c = qua[i0_qua,1]
rot = np.random.sample()*360
rr, cc = ellipse(
r, c,
shape=img_l.shape
)
poly = np.array((
(p1, p2),
(p1+p3, p2+p4+p0),
(p1+p5+p0, p2+p6+p0),
(p1+p7+p0, p2+p8),
(p1, p2),
))
rr_p, cc_p = polygon(poly[:, 0], poly[:, 1], img_l.shape)
if p > 0:
img[:,:,:1] = img_l.copy()
img[rr, cc,:1] = img_h[rr, cc]
img[rr_p, cc_p,:1] = img_h[rr_p, cc_p]
else:
img[:,:,:1] = img_h.copy()
img[rr, cc,:1] = img_l[rr, cc]
img[rr_p, cc_p,:1] = img_l[rr_p, cc_p]
img[:,:,1] = 0.
img[:,:,1] = 0.
img[rr_p, cc_p,1] = 1.
img[:,:,2] = 0.
p_f = np.random.sample()*1000.if p_f > 10:
img[rr_p, cc_p,2] = 1.else:
img[rr, cc,2] = 1.
i_false[idx] = 1return img
``````

Program for the calculation of the cheat sheet
``````defmake_sh(f_imgs, f_msks, val_len):
precision = 0.85
batch_size = 50
t = tqdm()
t_batch_size = 50
raw_len = val_len
id_train = 1#id_select = 1
v_false = np.zeros((train_num), dtype='float')
whileTrue:
if id_train == 1:
fit = model.fit(f_imgs[m2_select>0], f_msks[m2_select>0],
batch_size=batch_size,
epochs=1,
verbose=0
)
current_accu = fit.history['my_iou_metric'][0]
current_loss = fit.history['loss'][0]
if current_accu > precision:
id_train = 0else:
t_pred = model.predict(
f_imgs[raw_len: min(raw_len+t_batch_size,f_imgs.shape[0])],
batch_size=batch_size
)
for kk in range(t_pred.shape[0]):
val_iou = get_iou_vector(
f_msks[raw_len+kk].reshape(1,w_size,w_size,1),
t_pred[kk].reshape(1,w_size,w_size,1) > 0.5)
v_false[raw_len+kk] = val_iou
if val_iou < precision*0.95:
new_img_test = 1
m2_select[raw_len+kk] = 1
val_len += 1break
raw_len += (kk+1)
id_train = 1
t.set_description("Accuracy {0:6.4f} loss {1:6.4f} selected img {2:5d} tested img {3:5d} ".
format(current_accu, current_loss, val_len, raw_len))
t.update(1)
if raw_len >= train_num:
break
t.close()
return v_false
``````

The main program of calculations. We made minor changes to the same program from the previous post and some variables require clarification and comment.

``i_false = np.zeros((train_num), dtype='int')``

Here is an indicator of the mask falsity. If 1, then the mask from F_msks does not match the mask from f_msks. This is an indicator of what we are actually looking for - false masks.

``m2_select = np.zeros((train_num), dtype='int')``

Indicator that this picture is selected in the cheat sheet.

``````batch_size = 50
val_len = batch_size + 1# i_false - false mask marked as 1
i_false = np.zeros((train_num), dtype='int')
# t_imgs, t_msks -test images and masks
_txy = [next_pair_f(idx) for idx in range(train_num)]
t_imgs = np.array(_txy)[:,:,:,:1].reshape(-1,w_size ,w_size ,1)
t_msks = np.array(_txy)[:,:,:,1].reshape(-1,w_size ,w_size ,1)
# m2_select - initial 51 pair
m2_select = np.zeros((train_num), dtype='int')
for k in range(val_len):
m2_select[k] = 1# i_false - false mask marked as 1
i_false = np.zeros((train_num), dtype='int')
_txy = [next_pair_f(idx) for idx in range(train_num)]
f_imgs = np.array(_txy)[:,:,:,:1].reshape(-1,w_size ,w_size ,1)
f_msks = np.array(_txy)[:,:,:,1].reshape(-1,w_size ,w_size ,1)
F_msks = np.array(_txy)[:,:,:,2].reshape(-1,w_size ,w_size ,1)
fig, axes = plt.subplots(2, 10, figsize=(20, 5))
for k in range(10):
kk = np.random.randint(train_num)
axes[0,k].set_axis_off()
axes[0,k].imshow(f_imgs[kk].squeeze(), cmap="gray", norm=NoNorm())
axes[1,k].set_axis_off()
axes[1,k].imshow(f_msks[kk].squeeze(), cmap="gray", norm=NoNorm())
plt.show(block=True)
false_num = np.arange(train_num)[i_false>0]
fig, axes = plt.subplots(3, 10, figsize=(20, 7))
for k in range(10):
kk = np.random.randint(false_num.shape[0])
axes[0,k].set_axis_off()
axes[0,k].set_title(false_num[kk])
axes[0,k].imshow(f_imgs[false_num[kk]].squeeze(), cmap="gray", norm=NoNorm())
axes[1,k].set_axis_off()
axes[1,k].imshow(f_msks[false_num[kk]].squeeze(), cmap="gray", norm=NoNorm())
axes[2,k].set_axis_off()
axes[2,k].imshow(F_msks[false_num[kk]].squeeze(), cmap="gray", norm=NoNorm())
plt.show(block=True)
``````

We build sequences of pairs of picture / mask for training and another sequence for testing. Those. We will check on a new, independent sequence of 10,000 pairs. We display and visually selectively check random pictures with true and false masks. The pictures themselves are shown above.

In this particular case, there were 93 fake masks, in which an ellipse, not a quad, is marked as a mask.

We start training on the correct set, as a mask we use f_msks

``````input_layer = Input((w_size, w_size, 1))
output_layer = build_model(input_layer, 25)
model = Model(input_layer, output_layer)
v_false = make_sh(f_imgs, f_msks, val_len)
t_pred = model.predict(t_imgs,batch_size=batch_size)
print (get_iou_vector(t_msks,t_pred.reshape(-1,w_size ,w_size ,1)))
``````

``````Accuracy 0.9807 loss 0.0092 selected img   404 tested img 10000 : : 1801it [08:13,  3.65it/s]
0.9895299999999841``````

The cheat sheet turned out to be only 404 pictures and obtained acceptable accuracy on an independent test sequence.

Now we re-compile the network and train on the same training sequence, but as masks, we feed F_msks with 1% false masks

``````input_layer = Input((w_size, w_size, 1))
output_layer = build_model(input_layer, 25)
model = Model(input_layer, output_layer)
v_false = make_sh(f_imgs, F_msks, val_len)
t_pred = model.predict(t_imgs,batch_size=batch_size)
print (get_iou_vector(t_msks,t_pred.reshape(-1,w_size ,w_size ,1)))
``````

``````Accuracy 0.9821 loss 0.0324 selected img   727 tested img 10000 : : 1679it [25:44,  1.09it/s]
0.9524099999999959``````

We got a cheat sheet in 727 pictures, which is significantly more and the accuracy of the test predictions, the same as in the previous test, was reduced from 0.98953 to 0.9525. We added lies to the training sequence by less than 1%, only 93 out of 10,000 masks were lies, but the result worsened by 3.7%. And this is no longer just a lie, this is the real treachery! And the cheat sheet has increased from just 404 to 727 already.

``````print (len(set(np.arange(train_num)[m2_select>0]).intersection(set(np.arange(train_num)[i_false>0]))))
93``````

Let me explain this long formula, we take the intersection of the set of pictures selected in the cheat sheet with a lot of false pictures and see that all 93 false pictures the algorithm chose in the cheat sheet.

The task was simplified significantly, it is not 10,000 images viewed manually, it’s only 727 and all the lies are concentrated here.

But there is a more interesting and useful way. When we compiled the cheat sheet, we included only those picture / mask pairs whose prediction is less than the threshold, and in this particular case we saved the prediction accuracy value into the v_false array. Let's look at the pairs from the training sequence which have a very small prediction, for example, less than 0.1 and see how many lies there are.

``````print (len(set(np.arange(train_num)[v_false<0.01]).intersection(set(np.arange(train_num)[i_false>0]))))
89``````

As you can see, the main part of the false masks 89 out of 93 fell into these masks
``````np.arange(train_num)[v_false<0.01].shape
(382,)``````

Thus, if we test only 382 masks manually, and this is from 10,000 pieces, then most of the false masks will be identified and destroyed by us without any pity.

If it is possible to view pictures and masks while making decisions about their inclusion in the cheat sheet, then starting with a certain step, all the false masks, all lies will be determined by the minimum level of prediction already a little trained network, and the correct masks will have a prediction greater than this level .

### Let's sum up

If in some imaginary world truth is always quadrangular, and oval lies and some unknown entity decided to distort the truth and called some ellipses true, and quadrilaterals are false, then using artificial intelligence and natural ability to make cheat sheets, the local inquisition will quickly and easily find and eradicate lies and deceit completely and cleaned.

PS: The ability to detect ovals, triangles, simple polygons is a necessary condition for creating any AI driving a car. If you don’t know how to search for ovals and triangles, you won’t find all the road signs and your AI will not go there by car.