I can not overcome the problem with copying files in several streams. Task: there are directories dir1 = /home/my_dir1 and dir2 = /home/my_dir2 . How, using a multithreading module, simultaneously copy several files from dir1 to dir2 .
In this case there is a list co with all directories inside dir1 .

 remote_dir1 = '/home/my_dir1/' def listdir_fullpath(d): return [os.path.join(d, f) for f in os.listdir(d)] listdirs = sorted(listdir_fullpath(remote_dir1))

2 answers 2

  1. Go through all the files in the directory and put the full paths to each of the files in the Queue .
  2. Create n threads ( threading ), each of which takes a file from the Queue and copies it to a new location until the queue is empty.

The links have examples, with code that demonstrates this approach. This is one of the standard methods of independent multithreaded processing of elements of a sysk.

    Thank you for the right direction lebedev-ilya! The solution of the problem:

     from threading import Thread import os from queue import Queue q = Queue(maxsize=0) q.put('/dir1/one') q.put('/dir1/two') q.put('/dir1/three') tread = 3 def scp(q): while True: os.system('scp ' + q.get() + ' ' + '/dir2/') q.task_done() for i in range(tread): worker = Thread(target=scp, args=(q,)) worker.setDaemon(True) worker.start() q.join()