There is a list of say this type:

L = range(10000)

It is necessary to do something with each element of the list depending on the condition, if the list is very large, then standard Python cycles do not work as quickly as we would like. Are there any means in the numpy to iterate through the lists. For example, there is a function:

 def some_func(a): print('this is', a)

It is necessary that this function be applied, say to each non-even element of the list, how to do it by a non-standard loop, is there a similar possibility in numpy?

  • If you make a conclusion for every 2nd element, then the execution speed will be limited by the output speed to the external device (to a file or to the console, and the output to the console will be much slower than the file). No Numpy will help here. If just some calculations, then for 10,000 elements of the calculation (if they are not very complex) with the help of the usual for, they will be executed almost instantly. - insolor
  • 2
    Then show a more realistic example. - insolor
  • 2
    It is better to give a real concrete example on which there is a sense to accelerate something. And then write the number of elements really. - insolor
  • 2
    The fact is that in itself, the iteration is almost instantaneous; in addition, there is no sense in accelerating it, and there is nothing to accelerate there, everything has already been optimized to the limit. What really takes time is the code inside the loop. And it does not work faster, from the fact that you start to move faster between iterations. - Xander
  • 2
    @ IgorIgoryanych, do as you insolor advised. All the power and beauty of Numpy in "vectorized ufuncs" (optimized functions written in C / Cython and therefore very fast). In general, in Numpy there is apply_along_axis () , BUT it is the same for ... cycle and in speed it is unlikely to be faster. Knowing a specific problem, we could try to find a vectorized (fast) solution ... - MaxU

1 answer 1

All the power and beauty of Numpy in vectorized ufuncs (optimized functions written in C / Cython and therefore very fast). In general, Numpy has vectorize () for functions that work with scalars and apply_along_axis () for functions that work with lists, BUT it’s the same for ... cycle and in speed it is unlikely to be faster. Knowing a specific problem, we could try to find a vectorized (fast) solution ...

Example:

 In [130]: a = np.random.randint(10**4, size=(10**6)) In [131]: import math In [132]: def f(lst): ...: return [math.sin(x) * math.cos(x) for x in lst] ...: In [133]: a.shape Out[133]: (1000000,) In [134]: %timeit f(a[1::2]) 866 ms ± 886 µs per loop (mean ± std. dev. of 7 runs, 1 loop each) In [135]: %timeit np.apply_along_axis(f, 0, a[1::2]) 932 ms ± 183 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

If your function works with scalars :

 In [137]: def f(x): ...: return math.sin(x) * math.cos(x) ...: In [138]: vfunc = np.vectorize(f) In [139]: %timeit vfunc(a[1::2]) 969 ms ± 19.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Vector solution:

 In [136]: %timeit np.sin(a[1::2]) * np.cos(a[1::2]) 65.4 ms ± 92.7 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Conclusion: you must first try to find a vectorized solution. If this is not possible, then other options should be compared and, perhaps, " list comprehension " will be faster (as in my example) ...