I have 2 server Xeon processors with 32 cores and installed Linux Ubuntu 16.04. I have a Python script, when I run the script, it runs on 1 core and it takes quite a long time. Can anyone know how to parallelize the calculation of this script so that it can be processed by several cores at once?

    1 answer 1

    Required

    • understand what the existing script does
    • what parts are possible to perform independently
    • rewrite the script using processes, threads or anything else to separate one command flow into several

    There is no magic word or directive to use many cores in imperative programming languages. Because the program itself is a sequence of commands. One sequence only one CPU core executes and can. It is impossible to divide one sequence of commands in parallel into 10 cores and get a meaningful result as a result.

    Perhaps, the easiest way to load several cores will suit you - just run several copies of the programs, each of which will process some of the tasks. For example, if you need to compress 100 images - you can do this in one cycle sequentially, or you can run 10 programs, each of which will be processing the same cycle in the same cycle, but not all 100 images, but each process is separate 10 This will most likely not be very difficult to change the code to use not 1, but 10 cores.

    It may be more convenient to take any queue manager: the required number of (simpler) single-threaded programs are started, which periodically request a task from the queue - and perform this task. Accordingly, one task is performed on one core, but a group of tasks can be executed simultaneously by different copies of the script.