Artificial intelligence against lies and deceit
In all the tasks of learning artificial intelligence there is one most unpleasant phenomenon - errors in the marking of the training sequence. These errors are inevitable, since all the markup is done manually, because if there is a way to mark up real data programmatically, then why do we need someone else to teach them to mark and waste time and money to create an absolutely unnecessary construction!
The task of finding and removing fake masks in a large training sequence is quite complex, You can view them all manually, but this will not save you from repeated errors. But if you look closely at the neural network research tools proposed in previous posts , it turns out there is a simple and effective way to detect and extract all the artifacts from the training sequence.
And in this post there is a specific example, it is obvious that a simple, on ellipses and polygons, for an ordinary U-net, is again a Lego in the sandbox, but unusually concrete, useful and effective. We show how a simple method reveals and finds almost all the artifacts, all the lies of the training sequence.
So, let's begin!
As before, we will study the sequences of the picture / mask pairs. In the picture in different quarters chosen randomly we will place an ellipse of random size and a quadrilateral also of arbitrary size and both of them are painted in the same color, also randomly chosen from both. In the second remaining color we paint the background. Sizes of both ellipse and quad are of course limited.
But in this case, we’ll make changes to the steam generation program and prepare together with a completely correct mask an incorrect one, poisoned by a lie - in approximately one percent of cases we replace the quadrilateral with an ellipse in the mask, The true object for segmentation is the ellipse rather than the quadrilateral on the false masks.
Examples of random 10
Examples of random 10, but from erroneous markup. The upper mask is true, the lower is false and the numbers in the pictures are in the training sequence.
for segmentation, we take the same metrics and loss calculation programs and the same simple U-net, just don’t use Dropout.
The program for generating images and masks - true and fake. In the array is placed the first layer of the picture, the second true mask and the third layer false mask.
The main program of calculations. We made minor changes to the same program from the previous post and some variables require clarification and comment.
There is an indicator of the falsity of the mask. If 1, then the mask from F_msks does not match the mask from f_msks. This is an indicator of what we are actually looking for - false masks.
Indicator that this picture is selected in the cheat sheet.
We build sequences of pairs of picture / mask for training and another sequence for testing. Those. We will check on a new, independent sequence of 10,000 pairs. We display and visually selectively check random pictures with true and false masks. The pictures themselves are shown above.
In this particular case, there were 93 fake masks, in which an ellipse, not a quad, is marked as a mask.
We start training on the correct set, as a mask we use f_msks
The cheat sheet turned out to be only 404 pictures and obtained acceptable accuracy on an independent test sequence.
Now we re-compile the network and train on the same training sequence, but as masks, we feed F_msks with 1% false masks
We got a cheat sheet in 727 pictures, which is significantly more and the accuracy of the test predictions, the same as in the previous test, was reduced from 0.98953 to 0.9525. We added lies to the training sequence by less than 1%, only 93 out of 10,000 masks were lies, but the result worsened by 3.7%. And this is no longer just a lie, this is the real treachery! And the cheat sheet has increased from just 404 to 727 already.
Soothes and pleases only one
Let me explain this long formula, we take the intersection of the set of pictures selected in the cheat sheet with a lot of false pictures and see that all 93 false pictures the algorithm chose in the cheat sheet.
The task was simplified significantly, it is not 10,000 images viewed manually, it’s only 727 and all the lies are concentrated here.
But there is a more interesting and useful way. When we compiled the cheat sheet, we included only those picture / mask pairs whose prediction is less than the threshold, and in this particular case we saved the prediction accuracy value into the v_false array. Let's look at the pairs from the training sequence which have a very small prediction, for example, less than 0.1 and see how many lies there are.
As you can see, the main part of the false masks 89 out of 93 fell into these masks
Thus, if we test only 382 masks manually, and this is from 10,000 pieces, then most of the false masks will be identified and destroyed by us without any pity.
If it is possible to view pictures and masks while making decisions about their inclusion in the cheat sheet, then starting with a certain step, all the false masks, all lies will be determined by the minimum level of prediction already a little trained network, and the correct masks will have a prediction greater than this level .
If in some imaginary world truth is always quadrangular, and oval lies and some unknown entity decided to distort the truth and called some ellipses true, and quadrilaterals are false, then using artificial intelligence and natural ability to make cheat sheets, the local inquisition will quickly and easily find and eliminate lies and deceit completely and cleaned.
PS: The ability to detect ovals, triangles, simple polygons is a necessary condition for creating any AI driving a car. If you don’t know how to search for ovals and triangles, you won’t find all the road signs and your AI will not go there by car.
The task of finding and removing fake masks in a large training sequence is quite complex, You can view them all manually, but this will not save you from repeated errors. But if you look closely at the neural network research tools proposed in previous posts , it turns out there is a simple and effective way to detect and extract all the artifacts from the training sequence.
And in this post there is a specific example, it is obvious that a simple, on ellipses and polygons, for an ordinary U-net, is again a Lego in the sandbox, but unusually concrete, useful and effective. We show how a simple method reveals and finds almost all the artifacts, all the lies of the training sequence.
So, let's begin!
As before, we will study the sequences of the picture / mask pairs. In the picture in different quarters chosen randomly we will place an ellipse of random size and a quadrilateral also of arbitrary size and both of them are painted in the same color, also randomly chosen from both. In the second remaining color we paint the background. Sizes of both ellipse and quad are of course limited.
But in this case, we’ll make changes to the steam generation program and prepare together with a completely correct mask an incorrect one, poisoned by a lie - in approximately one percent of cases we replace the quadrilateral with an ellipse in the mask, The true object for segmentation is the ellipse rather than the quadrilateral on the false masks.
Examples of random 10
Examples of random 10, but from erroneous markup. The upper mask is true, the lower is false and the numbers in the pictures are in the training sequence.
for segmentation, we take the same metrics and loss calculation programs and the same simple U-net, just don’t use Dropout.
Libraries
import numpy as np import matplotlib.pyplot as plt from matplotlib.colors import NoNorm %matplotlib inline import math from tqdm import tqdm #from joblib import Parallel, delayed from skimage.draw import ellipse, polygon from keras import Model from keras.optimizers import Adam from keras.layers import Input,Conv2D,Conv2DTranspose,MaxPooling2D,concatenate from keras.layers import BatchNormalization,Activation,Add,Dropout from keras.losses import binary_crossentropy from keras import backend as K from keras.models import load_model import tensorflow as tf import keras as keras w_size = 128 train_num = 10000 radius_min = 10 radius_max = 30 Metric and loss functions
def dice_coef(y_true, y_pred): y_true_f = K.flatten(y_true) y_pred = K.cast(y_pred, 'float32') y_pred_f = K.cast(K.greater(K.flatten(y_pred), 0.5), 'float32') intersection = y_true_f * y_pred_f score = 2. * K.sum(intersection) / (K.sum(y_true_f) + K.sum(y_pred_f)) return score def dice_loss(y_true, y_pred): smooth = 1. y_true_f = K.flatten(y_true) y_pred_f = K.flatten(y_pred) intersection = y_true_f * y_pred_f score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth) return 1. - score def bce_dice_loss(y_true, y_pred): return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred) def get_iou_vector(A, B): # Numpy version batch_size = A.shape[0] metric = 0.0 for batch in range(batch_size): t, p = A[batch], B[batch] true = np.sum(t) pred = np.sum(p) # deal with empty mask first if true == 0: metric += (pred == 0) continue # non empty mask case. Union is never empty # hence it is safe to divide by its number of pixels intersection = np.sum(t * p) union = true + pred - intersection iou = intersection / union # iou metrric is a stepwise approximation of the real iou over 0.5 iou = np.floor(max(0, (iou - 0.45)*20)) / 10 metric += iou # teake the average over all images in batch metric /= batch_size return metric def my_iou_metric(label, pred): # Tensorflow version return tf.py_func(get_iou_vector, [label, pred > 0.5], tf.float64) from keras.utils.generic_utils import get_custom_objects get_custom_objects().update({'bce_dice_loss': bce_dice_loss }) get_custom_objects().update({'dice_loss': dice_loss }) get_custom_objects().update({'dice_coef': dice_coef }) get_custom_objects().update({'my_iou_metric': my_iou_metric }) Normal U-net
def build_model(input_layer, start_neurons): # 128 -> 64 conv1 = Conv2D(start_neurons * 1, (3, 3), activation="relu", padding="same")(input_layer) conv1 = Conv2D(start_neurons * 1, (3, 3), activation="relu", padding="same")(conv1) pool1 = Conv2D(start_neurons * 1, (2, 2), strides=(2, 2), activation="relu", padding="same")(conv1) # pool1 = Dropout(0.25)(pool1) # 64 -> 32 conv2 = Conv2D(start_neurons * 2, (3, 3), activation="relu", padding="same")(pool1) conv2 = Conv2D(start_neurons * 2, (3, 3), activation="relu", padding="same")(conv2) pool2 = Conv2D(start_neurons * 1, (2, 2), strides=(2, 2), activation="relu", padding="same")(conv2) # pool2 = Dropout(0.5)(pool2) # 32 -> 16 conv3 = Conv2D(start_neurons * 4, (3, 3), activation="relu", padding="same")(pool2) conv3 = Conv2D(start_neurons * 4, (3, 3), activation="relu", padding="same")(conv3) pool3 = Conv2D(start_neurons * 1, (2, 2), strides=(2, 2), activation="relu", padding="same")(conv3) # pool3 = Dropout(0.5)(pool3) # 16 -> 8 conv4 = Conv2D(start_neurons * 8, (3, 3), activation="relu", padding="same")(pool3) conv4 = Conv2D(start_neurons * 8, (3, 3), activation="relu", padding="same")(conv4) pool4 = Conv2D(start_neurons * 1, (2, 2), strides=(2, 2), activation="relu", padding="same")(conv4) # pool4 = Dropout(0.5)(pool4) # Middle convm = Conv2D(start_neurons * 16, (3, 3), activation="relu", padding="same")(pool4) convm = Conv2D(start_neurons * 16, (3, 3) , activation="relu", padding="same")(convm) # 8 -> 16 deconv4 = Conv2DTranspose(start_neurons * 8, (3, 3), strides=(2, 2), padding="same")(convm) uconv4 = concatenate([deconv4, conv4]) # uconv4 = Dropout(0.5)(uconv4) uconv4 = Conv2D(start_neurons * 8, (3, 3) , activation="relu", padding="same")(uconv4) uconv4 = Conv2D(start_neurons * 8, (3, 3) , activation="relu", padding="same")(uconv4) # 16 -> 32 deconv3 = Conv2DTranspose(start_neurons * 4, (3, 3), strides=(2, 2), padding="same")(uconv4) uconv3 = concatenate([deconv3, conv3]) # uconv3 = Dropout(0.5)(uconv3) uconv3 = Conv2D(start_neurons * 4, (3, 3) , activation="relu", padding="same")(uconv3) uconv3 = Conv2D(start_neurons * 4, (3, 3) , activation="relu", padding="same")(uconv3) # 32 -> 64 deconv2 = Conv2DTranspose(start_neurons * 2, (3, 3), strides=(2, 2), padding="same")(uconv3) uconv2 = concatenate([deconv2, conv2]) # uconv2 = Dropout(0.5)(uconv2) uconv2 = Conv2D(start_neurons * 2, (3, 3) , activation="relu", padding="same")(uconv2) uconv2 = Conv2D(start_neurons * 2, (3, 3) , activation="relu", padding="same")(uconv2) # 64 -> 128 deconv1 = Conv2DTranspose(start_neurons * 1, (3, 3), strides=(2, 2), padding="same")(uconv2) uconv1 = concatenate([deconv1, conv1]) # uconv1 = Dropout(0.5)(uconv1) uconv1 = Conv2D(start_neurons * 1, (3, 3) , activation="relu", padding="same")(uconv1) uconv1 = Conv2D(start_neurons * 1, (3, 3) , activation="relu", padding="same")(uconv1) # uncov1 = Dropout(0.5)(uconv1) output_layer = Conv2D(1, (1,1), padding="same", activation="sigmoid")(uconv1) return output_layer input_layer = Input((w_size, w_size, 1)) output_layer = build_model(input_layer, 27) model = Model(input_layer, output_layer) model.compile(loss=bce_dice_loss, optimizer=Adam(lr=1e-4), metrics=[my_iou_metric]) model.summary() The program for generating images and masks - true and fake. In the array is placed the first layer of the picture, the second true mask and the third layer false mask.
def next_pair_f(idx): img_l = np.ones((w_size, w_size, 1), dtype='float')*0.45 img_h = np.ones((w_size, w_size, 1), dtype='float')*0.55 img = np.zeros((w_size, w_size, 3), dtype='float') i0_qua = math.trunc(np.random.sample()*4.) i1_qua = math.trunc(np.random.sample()*4.) while i0_qua == i1_qua: i1_qua = math.trunc(np.random.sample()*4.) _qua = np.int(w_size/4) qua = np.array([[_qua,_qua],[_qua,_qua*3],[_qua*3,_qua*3],[_qua*3,_qua]]) p = np.random.sample() - 0.5 r = qua[i0_qua,0] c = qua[i0_qua,1] r_radius = np.random.sample()*(radius_max-radius_min) + radius_min c_radius = np.random.sample()*(radius_max-radius_min) + radius_min rot = np.random.sample()*360 rr, cc = ellipse( r, c, r_radius, c_radius, rotation=np.deg2rad(rot), shape=img_l.shape ) p0 = np.rint(np.random.sample()*(radius_max-radius_min) + radius_min) p1 = qua[i1_qua,0] - (radius_max-radius_min) p2 = qua[i1_qua,1] - (radius_max-radius_min) p3 = np.rint(np.random.sample()*radius_min) p4 = np.rint(np.random.sample()*radius_min) p5 = np.rint(np.random.sample()*radius_min) p6 = np.rint(np.random.sample()*radius_min) p7 = np.rint(np.random.sample()*radius_min) p8 = np.rint(np.random.sample()*radius_min) poly = np.array(( (p1, p2), (p1+p3, p2+p4+p0), (p1+p5+p0, p2+p6+p0), (p1+p7+p0, p2+p8), (p1, p2), )) rr_p, cc_p = polygon(poly[:, 0], poly[:, 1], img_l.shape) if p > 0: img[:,:,:1] = img_l.copy() img[rr, cc,:1] = img_h[rr, cc] img[rr_p, cc_p,:1] = img_h[rr_p, cc_p] else: img[:,:,:1] = img_h.copy() img[rr, cc,:1] = img_l[rr, cc] img[rr_p, cc_p,:1] = img_l[rr_p, cc_p] img[:,:,1] = 0. img[:,:,1] = 0. img[rr_p, cc_p,1] = 1. img[:,:,2] = 0. p_f = np.random.sample()*1000. if p_f > 10: img[rr_p, cc_p,2] = 1. else: img[rr, cc,2] = 1. i_false[idx] = 1 return img Program for the calculation of the cheat sheet
def make_sh(f_imgs, f_msks, val_len): precision = 0.85 batch_size = 50 t = tqdm() t_batch_size = 50 raw_len = val_len id_train = 1 #id_select = 1 v_false = np.zeros((train_num), dtype='float') while True: if id_train == 1: fit = model.fit(f_imgs[m2_select>0], f_msks[m2_select>0], batch_size=batch_size, epochs=1, verbose=0 ) current_accu = fit.history['my_iou_metric'][0] current_loss = fit.history['loss'][0] if current_accu > precision: id_train = 0 else: t_pred = model.predict( f_imgs[raw_len: min(raw_len+t_batch_size,f_imgs.shape[0])], batch_size=batch_size ) for kk in range(t_pred.shape[0]): val_iou = get_iou_vector( f_msks[raw_len+kk].reshape(1,w_size,w_size,1), t_pred[kk].reshape(1,w_size,w_size,1) > 0.5) v_false[raw_len+kk] = val_iou if val_iou < precision*0.95: new_img_test = 1 m2_select[raw_len+kk] = 1 val_len += 1 break raw_len += (kk+1) id_train = 1 t.set_description("Accuracy {0:6.4f} loss {1:6.4f} selected img {2:5d} tested img {3:5d} ". format(current_accu, current_loss, val_len, raw_len)) t.update(1) if raw_len >= train_num: break t.close() return v_false The main program of calculations. We made minor changes to the same program from the previous post and some variables require clarification and comment.
i_false = np.zeros((train_num), dtype='int') There is an indicator of the falsity of the mask. If 1, then the mask from F_msks does not match the mask from f_msks. This is an indicator of what we are actually looking for - false masks.
m2_select = np.zeros((train_num), dtype='int') Indicator that this picture is selected in the cheat sheet.
batch_size = 50 val_len = batch_size + 1 # i_false - false mask marked as 1 i_false = np.zeros((train_num), dtype='int') # t_imgs, t_msks -test images and masks _txy = [next_pair_f(idx) for idx in range(train_num)] t_imgs = np.array(_txy)[:,:,:,:1].reshape(-1,w_size ,w_size ,1) t_msks = np.array(_txy)[:,:,:,1].reshape(-1,w_size ,w_size ,1) # m2_select - initial 51 pair m2_select = np.zeros((train_num), dtype='int') for k in range(val_len): m2_select[k] = 1 # i_false - false mask marked as 1 i_false = np.zeros((train_num), dtype='int') _txy = [next_pair_f(idx) for idx in range(train_num)] f_imgs = np.array(_txy)[:,:,:,:1].reshape(-1,w_size ,w_size ,1) f_msks = np.array(_txy)[:,:,:,1].reshape(-1,w_size ,w_size ,1) # F_msks - mask array with ~1% false mask F_msks = np.array(_txy)[:,:,:,2].reshape(-1,w_size ,w_size ,1) fig, axes = plt.subplots(2, 10, figsize=(20, 5)) for k in range(10): kk = np.random.randint(train_num) axes[0,k].set_axis_off() axes[0,k].imshow(f_imgs[kk].squeeze(), cmap="gray", norm=NoNorm()) axes[1,k].set_axis_off() axes[1,k].imshow(f_msks[kk].squeeze(), cmap="gray", norm=NoNorm()) plt.show(block=True) false_num = np.arange(train_num)[i_false>0] fig, axes = plt.subplots(3, 10, figsize=(20, 7)) for k in range(10): kk = np.random.randint(false_num.shape[0]) axes[0,k].set_axis_off() axes[0,k].set_title(false_num[kk]) axes[0,k].imshow(f_imgs[false_num[kk]].squeeze(), cmap="gray", norm=NoNorm()) axes[1,k].set_axis_off() axes[1,k].imshow(f_msks[false_num[kk]].squeeze(), cmap="gray", norm=NoNorm()) axes[2,k].set_axis_off() axes[2,k].imshow(F_msks[false_num[kk]].squeeze(), cmap="gray", norm=NoNorm()) plt.show(block=True) We build sequences of pairs of picture / mask for training and another sequence for testing. Those. We will check on a new, independent sequence of 10,000 pairs. We display and visually selectively check random pictures with true and false masks. The pictures themselves are shown above.
In this particular case, there were 93 fake masks, in which an ellipse, not a quad, is marked as a mask.
We start training on the correct set, as a mask we use f_msks
input_layer = Input((w_size, w_size, 1)) output_layer = build_model(input_layer, 25) model = Model(input_layer, output_layer) model.compile(loss=bce_dice_loss, optimizer=Adam(lr=1e-4), metrics=[my_iou_metric]) v_false = make_sh(f_imgs, f_msks, val_len) t_pred = model.predict(t_imgs,batch_size=batch_size) print (get_iou_vector(t_msks,t_pred.reshape(-1,w_size ,w_size ,1))) Accuracy 0.9807 loss 0.0092 selected img 404 tested img 10000 : : 1801it [08:13, 3.65it/s] 0.9895299999999841 The cheat sheet turned out to be only 404 pictures and obtained acceptable accuracy on an independent test sequence.
Now we re-compile the network and train on the same training sequence, but as masks, we feed F_msks with 1% false masks
input_layer = Input((w_size, w_size, 1)) output_layer = build_model(input_layer, 25) model = Model(input_layer, output_layer) model.compile(loss=bce_dice_loss, optimizer=Adam(lr=1e-4), metrics=[my_iou_metric]) v_false = make_sh(f_imgs, F_msks, val_len) t_pred = model.predict(t_imgs,batch_size=batch_size) print (get_iou_vector(t_msks,t_pred.reshape(-1,w_size ,w_size ,1))) Accuracy 0.9821 loss 0.0324 selected img 727 tested img 10000 : : 1679it [25:44, 1.09it/s] 0.9524099999999959 We got a cheat sheet in 727 pictures, which is significantly more and the accuracy of the test predictions, the same as in the previous test, was reduced from 0.98953 to 0.9525. We added lies to the training sequence by less than 1%, only 93 out of 10,000 masks were lies, but the result worsened by 3.7%. And this is no longer just a lie, this is the real treachery! And the cheat sheet has increased from just 404 to 727 already.
Soothes and pleases only one
print (len(set(np.arange(train_num)[m2_select>0]).intersection(set(np.arange(train_num)[i_false>0])))) 93 Let me explain this long formula, we take the intersection of the set of pictures selected in the cheat sheet with a lot of false pictures and see that all 93 false pictures the algorithm chose in the cheat sheet.
The task was simplified significantly, it is not 10,000 images viewed manually, it’s only 727 and all the lies are concentrated here.
But there is a more interesting and useful way. When we compiled the cheat sheet, we included only those picture / mask pairs whose prediction is less than the threshold, and in this particular case we saved the prediction accuracy value into the v_false array. Let's look at the pairs from the training sequence which have a very small prediction, for example, less than 0.1 and see how many lies there are.
print (len(set(np.arange(train_num)[v_false<0.01]).intersection(set(np.arange(train_num)[i_false>0])))) 89 As you can see, the main part of the false masks 89 out of 93 fell into these masks
np.arange(train_num)[v_false<0.01].shape (382,) Thus, if we test only 382 masks manually, and this is from 10,000 pieces, then most of the false masks will be identified and destroyed by us without any pity.
If it is possible to view pictures and masks while making decisions about their inclusion in the cheat sheet, then starting with a certain step, all the false masks, all lies will be determined by the minimum level of prediction already a little trained network, and the correct masks will have a prediction greater than this level .
Let's sum up
If in some imaginary world truth is always quadrangular, and oval lies and some unknown entity decided to distort the truth and called some ellipses true, and quadrilaterals are false, then using artificial intelligence and natural ability to make cheat sheets, the local inquisition will quickly and easily find and eliminate lies and deceit completely and cleaned.
PS: The ability to detect ovals, triangles, simple polygons is a necessary condition for creating any AI driving a car. If you don’t know how to search for ovals and triangles, you won’t find all the road signs and your AI will not go there by car.
Source: https://habr.com/ru/post/440120/