A simple neural network gives the result ~ 0.5

Question

I am trying to implement a simple neural network of direct distribution. But the result that it gives after training (the method of back-propagation of error) tends to a value of 0.5. Though after 1000 epochs of learning, even after 10,000. It has 3 inputs, 2 neurons in the hidden layer and 1 resultant neuron.

And if you do not train her, and with just generated weights, you can make a decision, then her answer ranges from 0.5 to 1.

Here is the code of what is at the moment.

/*var weight_1 = [[0.79, 0.44, 0.43], [0.85, 0.43, 0.29]], weight_2 = [[0.5, 0.52]],*/ var weight_1 = randArr(2, 3), weight_2 = randArr(1, 2), learning_rate = 0.05, data = [ [[0,0,0], 0], [[0,0,1], 1], [[0,1,0], 1], [[0,1,1], 0], [[1,0,0], 1], [[1,0,1], 0], [[1,1,0], 0], [[1,1,1], 1] ]; console.log("---------До тренировки---------"); console.groupCollapsed("Prediction before"); predictSet(data); console.groupEnd(); function randArr(rows, cols) { arr = []; for (var i = 0; i<rows; i++) { arr[i] = []; for (var j = 0; j<cols; j++) { arr[i][j] = Math.random(); } } return arr; } function activation(x) { return 1 / (1 + Math.exp(-x)); } function sumMultVector(A, B) { var C = 0; for (var i=0;i<A.length;i++) { C += A[i] * B[i]; } return C; } function getWeightLayer(inputs, weight) { var getWeightLayer = []; for (var i = 0; i<weight.length; i++) { getWeightLayer[i] = activation(sumMultVector(inputs,weight[i])); } return getWeightLayer; } function predict(inputs) { outputs_1 = getWeightLayer(inputs, weight_1); outputs_2 = getWeightLayer(outputs_1, weight_2); return outputs_2[0]; } function train(inputs, expected) { weights = getWeightLayer(inputs, weight_1); weights_2 = getWeightLayer(weights, weight_2); predict_2 = weights_2[0]; error_2 = predict_2 - expected; sigmoid_2 = predict_2 * (1 - predict_2); delta_2 = error_2 * sigmoid_2; weight_2[0][0] -= weights_2[0] * delta_2 * learning_rate; weight_2[0][1] -= weights_2[0] * delta_2 * learning_rate; for (var j = 0; j < 2; j++) for (var i = 0; i < 3; i++) { error_1 = weight_2[0][j] * delta_2; sigmoid_1 = weights[j] * (1 - weights[j]); delta_2 = error_1 * sigmoid_1; weight_1[j][i] -= inputs[i] * delta_2 * learning_rate; } } function compareRandom() { return Math.random() - 0.5; } function trainSet(data) { for (var k = 0; k < 5000; k++) { data.sort(compareRandom); for (var i = 0; i < data.length; i++) { train(data[i][0], data[i][1]) } } } function predictSet(data) { for (var i = 0; i < data.length; i++) { console.log("спрогнозировало:", predict(data[i][0]), "ожидается:",data[i][1]); } } trainSet(data); console.log("---------После тренировки---------"); console.groupCollapsed("Prediction after"); predictSet(data); console.groupEnd(); console.groupCollapsed("Weights"); console.log("От входного слоя к скрытому: ", weight_1); console.log("От скрытого к выходному: ", weight_2); console.groupEnd();

And my problem is that all the results that this algorithm produces are close to 0.5 and rather random, no matter what arguments I use.

It would be nice if someone has a similar, but working neural network. Then I could get the result of training on certain static scales and focus on the correctness of my neural network.

And tried to use a different algorithm for generating the initial ones?
@ yar85 I tried to manually set the weight, the result is the same.
Assume they initially look like this: From the input layer to the hidden [[0.34154, 0.48588, 0.36448] [0.47394, 0.79290, 0.52845]] From the hidden to the output [0.54270, 0.03027] After 1000 iterations of the workout, they changed to [[0.70796, 0.29439, 0.47045 ] [0.03797, 0.97974, 0.47748]] and [-0.3225, 0.31591] respectively
Even on the simplest data set, the network does not learn anything.
You need to look at what sizes of matrices you get on each of the layers.
Just a few neurons, you need not 3, but 5 or more in the hidden layer.
If you really want to study the whole topic of machine learning and neural networks in general, then you can go here: coursera.org/learn/machine-learning - the details here may not be entirely relevant, but all the mathematics are the same as now, as before (I'm talking about the beginning of 2018)

Accepted Answer · 2018-04-19T08:05:36

Take the framework for building networks, recreate the architecture of your network and ... We will see that even a deliberately working network cannot reveal patterns in your input data. For a knowingly negative sample [1,1,0] network shows the result within 0.5. If you look at the training error, then at 20,000 iterations it will be about 0.2, from which in itself it can be concluded that the network architecture is not enough.

You can try to fix the algorithm for as long as you want, but you will fail without changing and complicating the architecture. As an experiment, you can try to pick up weights with your hands that would give the correct answer. I am sure that it is impossible to do this even to a person, what can we say about algorithms that cannot even come down from the mountain properly .

 var myNetwork = new synaptic.Architect.Perceptron(3, 2, 1) var trainer = new synaptic.Trainer(myNetwork) var trainingSet = [ { input: [0,0,0], output: [0] }, { input: [0,0,1], output: [1] }, { input: [0,1,0], output: [1] }, { input: [0,1,1], output: [0] }, { input: [1,0,0], output: [1] }, { input: [1,0,1], output: [0] }, { input: [1,1,0], output: [0] }, { input: [1,1,1], output: [1] } ] var trainingOptions = { rate: .1, iterations: 20000, error: .005, } console.log(trainer.train(trainingSet, trainingOptions)); console.log(myNetwork.activate([0,0,1])); console.log(myNetwork.activate([1,0,0])); console.log(myNetwork.activate([1,1,1])); console.log(myNetwork.activate([1,1,0]));

 <script src="https://cdnjs.cloudflare.com/ajax/libs/synaptic/1.1.4/synaptic.min.js"></script>

If you complicate the structure by adding three neurons in the hidden layer ... Then the system for the same negative sample [1,1,0] shows a result close to 0.1, that is, the expected negative. The learning error is minimal. This, of course, is not necessarily good, but for our minimal and comprehensive error data set there are no questions.

(It would be enough to add two neurons, but more is better for reliability.)

 var myNetwork = new synaptic.Architect.Perceptron(3, 5, 1) var trainer = new synaptic.Trainer(myNetwork) var trainingSet = [ { input: [0,0,0], output: [0] }, { input: [0,0,1], output: [1] }, { input: [0,1,0], output: [1] }, { input: [0,1,1], output: [0] }, { input: [1,0,0], output: [1] }, { input: [1,0,1], output: [0] }, { input: [1,1,0], output: [0] }, { input: [1,1,1], output: [1] } ] var trainingOptions = { rate: .1, iterations: 20000, error: .005, } console.log(trainer.train(trainingSet, trainingOptions)); console.log(myNetwork.activate([0,0,1])); console.log(myNetwork.activate([1,0,0])); console.log(myNetwork.activate([1,1,1])); console.log(myNetwork.activate([1,1,0]));

 <script src="https://cdnjs.cloudflare.com/ajax/libs/synaptic/1.1.4/synaptic.min.js"></script>

This network can even be counted manually, so that you will see what your algorithm is correct or correctly count.

RealPeha RealPeha 341 one 12 · Answer 2 · 2018-04-20T20:02:25

Thanks to sanmai, I finally solved my problem! More than a week of effort, and now I have a ready neural network. Completely rewrote the function of training, it turned out I used the wrong formulas. I give the code, can someone come in handy

 var hiddenLayer = 4, weight_1 = randArr(hiddenLayer, 2), //(скрытий слой, входной слой) weight_2 = randArr(1, hiddenLayer), //(выходной слой, скрытый слой) learning_rate = 1, data = [ [[0,0], 1], [[0,1], 0], [[1,0], 0], [[1,1], 1] ]; function randArr(rows, cols) { arr = []; for (var i = 0; i<rows; i++) { arr[i] = []; for (var j = 0; j<cols; j++) { arr[i][j] = Math.random() - 0.5; } } return arr; } function activation(x) { return 1 / (1 + Math.exp(-x)); } function multVector(A, B) { var C = 0; for (var i=0;i<A.length;i++) { C += A[i] * B[i]; } return C; } function getWeightLayer(inputs, weight) { var getWeightLayer = []; for (var i = 0; i<weight.length; i++) { getWeightLayer[i] = activation(multVector(inputs,weight[i])); } return getWeightLayer; } function predict(inputs) { outputs_1 = getWeightLayer(inputs, weight_1); outputs_2 = getWeightLayer(outputs_1, weight_2); return outputs_2[0]; } //Обратное распространение ошибки function train(inputs, expected) { outputs_1 = getWeightLayer(inputs, weight_1); outputs_2 = getWeightLayer(outputs_1, weight_2); //Дельта выходного нейрона predict_2 = outputs_2[0]; delta_2 = predict_2 * (1 - predict_2) * (expected - predict_2); //От выходного слоя к скрытому delta_1 = []; for (var i = 0; i < hiddenLayer; i++) { delta_1[i] = outputs_1[i] * (1 - outputs_1[i]) * weight_2[0][i] * delta_2; grad = outputs_1[i] * delta_2; //Меняем вес weight_2[0][i] += grad * learning_rate; } //От скрытого к входному for (var j = 0; j < hiddenLayer; j++) for (var i = 0; i < 2; i++) { //Вычисляем смещение grad = inputs[i] * delta_1[j]; //Меняем вес weight_1[j][i] += grad * learning_rate; } } function compareRandom() { return Math.random() - 0.5; } //Тренировка сети function trainSet(data) { for (var k = 0; k < 5000; k++) { for (var i = 0; i < data.length; i++) { train(data[i][0], data[i][1]) } } } //Вывод результата function predictSet(data) { for (var i = 0; i < data.length; i++) { console.log("p:", predict(data[i][0]), "e:", data[i][1]); } } trainSet(data); predictSet(data);

It remains to bring the code in order and expand the architecture to solve problems more difficult. And she also seems to be coming out of the local minimum. I read that another parameter is used for this - a moment, which, like the learning rate, is determined by the selection method, but I could not realize it. I hope it does not affect badly in the future.

A simple neural network gives the result ~ 0.5

2 answers 2

More articles: