Hello,

I am new to GNU Parallel and will be happy if you point out errors and some misunderstandings. I read the manual but it says mostly about one-step action in which you need to specify a specific "action" in the GNU Parallel syntax (unpacking, moving) and nothing is indicated about the multi-step steps when you need to perform several actions without changing (significantly) the code ( unless of course this is possible at all)

Is it possible to implement parallel processing in code that does not actually support it? But there is a loop in it. That is, there is an incoming list of files in any format and at some point in the code it comes to a loop. (code example: 138 GMT line)

In short, is it possible that parallel processing is similar ( without changing the code significantly or only about 138 lines) when just every file starts to be processed at the same time regardless of what actions are written in the code in this moment? (and even multi-step actions , that is, when, for example, you need to add a file to the archive and then move and then something else, etc.)

# Π›ΡŽΠ±Ρ‹Π΅ дСйствия ΠΈ ΠΎΠ±ΡΠ·Π°Ρ‚Π΅Π»ΡŒΠ½ΠΎ "ΠΎΡ‚ΠΊΡ€Ρ‹Ρ‚ΠΈΠ΅ ΠΈΠ»ΠΈ ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΊΠ° наличия" списка Ρ„Π°ΠΉΠ»ΠΎΠ² # loop # foreach line (`awk '{print $0}' $1`) # Π›ΡŽΠ±Ρ‹Π΅ дСйствия

Maybe this can be implemented using a tool other than GNU Parallel? I will be glad to any help! thanks in advance

  • make a function from the loop body, and call it a parallel program. - aleksandr barakin
  • Could not be a bit more detailed (novice) and better, of course, an example of syntax. The example of the code that I threw is called intf_tops.csh intf.in batch_tops.config and it is absolutely not clear how to call it, for example, with GNU Parallel. And it is necessary that this moment, which is given at the top since 138, is β€œdivided”. Thank you very much. - user2899758
  • Yes, and if you make parallel all the code, then this is likely to cause problems. It is necessary at the moment of the loop - user2899758

1 answer 1

short answer:

minor changes in your parallelization case (at least with the gnu / parallel program) cannot be achieved.

long answer:

if the bash interpreter were used, then the construct in the csh interpreter language:

 foreach line (`ΠΊΠΎΠΌΠ°Π½Π΄Π° с Π°Ρ€Π³ΡƒΠΌΠ΅Π½Ρ‚Π°ΠΌΠΈ`) ΠΊΠ°ΠΊΠΈΠ΅-Ρ‚ΠΎ ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹ end

that sounds in the bash interpreter language like this:

 for line in `ΠΊΠΎΠΌΠ°Π½Π΄Π° с Π°Ρ€Π³ΡƒΠΌΠ΅Π½Ρ‚Π°ΠΌΠΈ` ΠΊΠ°ΠΊΠΈΠ΅-Ρ‚ΠΎ ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹ done

it would be possible to parallelize something like this:

 fff() { ΠΊΠ°ΠΊΠΈΠ΅-Ρ‚ΠΎ ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹ } export -f fff parallel line={} fff ::: `ΠΊΠΎΠΌΠ°Π½Π΄Π° с Π°Ρ€Π³ΡƒΠΌΠ΅Π½Ρ‚Π°ΠΌΠΈ`

but this would only work if the variables defined before the cycle in the same script were not used inside the loop body.

because the gnu/parallel program will launch new processes that will not β€œknow” about these local variables.

The next point is that your example uses the csh interpreter, which does not have support for functions:

The stdio file handles independently and support for functions .

those. all the code from the loop, to be able to call it with the gnu / parallel program, must be placed in a separate file.

  • That is, in fact, taking out a separate file of this, will it solve the problem? You described what needs to be done with bash, for csh it's easier to rewrite everything, so I understand? - user2899758
  • @ user2899758, I, like, wrote in the answer: there are no functions for csh , so the commands contained within the loop will have to be moved to a separate file. and it already has to be called by the program parallel , inserted instead of the entire cycle. if inside the loop the variables described before the loop are not used, then it should work. - aleksandr barakin
  • @user2899758, I added a response to a csh construct, a similar construct to bash . and export in the case of csh , of course, will not be needed. - aleksandr barakin
  • Thank you very much. All clear. - user2899758
  • Could you tell me how I could make the variables in the loop? Or can I add them to a separate file (which was discussed above) before the loop ? Or should it be indicated in it? - user2899758