Speed ​​up complex calculations with minimal RAM

In the process of mastering ML, CNN, NN in Python, a novice almost always faces the problem of computation speed, and sometimes lack of RAM.

This is especially noticeable with large databases that are larger than 50% of free RAM. Thinking about buying more worthy hardware is just one of the possible solutions.

Another option to use one of the features in Python is to iterate over the function itself.

A simple and illustrative example. Suppose you need to build in 200,000 degree a series of numbers from 1 to 10 and add up their sum. As a result, you should get the number of long 200 thousand characters. This is 2 times more than google )

Here is the simple code for this function:

from time import time # импорт модуля времени # простая функция возведения в степень def power_1(x): power_1 = 0 start = time() power_1 = x**200000 end = time() total_1 = end - start return round(total_1,6) # простые операции сложения, для того что бы посчитать отдельно время операции power_1(1) + power_1(2) + power_1(3) + power_1(4) + power_1(5) + power_1(6) + power_1(7) + power_1(8) \ + power_1(9) + power_1(10) 

As a result, we get quite a “costly” operation for your computer, and for example mine it takes more than 13 ms. And what if there can be many such operations? Then everything gets complicated, and maybe you just do not have enough RAM.

But you can be more cunning and use the same function by running it through iteration over your variables.

Here is the code for such a simple solution of the same function:

 from time import time #получаем модуль времени # функция итерации def power_2(): for i in range(11): yield i # функции возведения в степень for i in power_2(): if i < 11: d = 0 total_2 = 0 start = time() d += i**200000 end = time() total_2 += end - start print('Ineration {} lasts {}'.format(i, round(total_2, 6))) if i == 10: print(' \n End with total time {} \n'.format(round(total_2,3))) 

Time spent on my computer



The difference is 0.13 - 0.024 = 0.106 ms!


If you simply add 1 to this huge number in the usual way, this one will take longer than the erection of the 200,000 degree itself. That is, the advantages of this method are obvious.

 def add_one(x): total_3 = 0 start = time() s = x + 1 end = time() total_3 += end - start print('Time to do add 1 to BIG number: {} \n'.format(total_3)) return print('Difference in time {}\n'.format(round(total_2 - total_3 ,3))) add_one(d) print('The size of the number {}!!!'.format(len(str(d)))) 

The result is:



Why is that? As far as I know, when iterating through a function, Python does not create temporary objects in memory, which in turn significantly speeds up any computational process. Therefore, by rewriting the function in this way, you save time and nerves.

Total - for complex calculations with a limited size of RAM, it is better to use the iteration of the function than just the functions themselves.

I hope this will help someone not to lose precious minutes or not to spend on additional, rapidly aging iron.

Inspired by the lecture