I have a python script that bypasses folders in some directory on the server and deletes old files. Such a script loads the north very much. a lot of files. Is there any way to make this process easier for the processor? For example, if you rewrite a script into a compiled language like С++ or Java , would it be useful?

  • Comments are not intended for extended discussion; conversation moved to chat . - Nick Volynkin ♦

1 answer 1

iowait=100% says that other CPU metrics such as user, sys, idle are all zero. That is, the CPU does nothing for you, but only waits until the IO (input / output) operations are completed.

Please note: "iowait 100% iowait, or have a disk bottleneck with 0% iowait." that is, by itself, this metric may mean nothing. Look at the total number of IO operations (IOPS) and disk load (MB / s), for example, using iostat and / or csysdig (the latter is useful to get a broad picture of the current state of the system, looking at the details if necessary).

Methods that can reduce the disk load of the @Dave Cheney response :

  • add RAM so there is room for file cache
  • free up disk space (performance suffers with full disk on some file systems due to fragmentation)
  • tweak the file system settings (for example, use the noatime option when connecting)
  • suitable RAID
  • if you write / read data from a disk, then you can play with the buffer size

In your case, the cleaning task is most likely background, so you can simply try to limit the number of files that you delete every second:

 #!/usr/bin/env python3 """Usage: clean-dir-tree <root-dir>""" import sys import time import pathlib interval = 1 / 10 # delete no more than 10 files per second root_dir = pathlib.Path(sys.argv[1]) for path in root_dir.rglob('*.whatever'): try: path.unlink() except OSError as error: print("Can't delete {path}, reason: {error}".format(**vars()), file=sys.stderr) else: time.sleep(interval - time.monotonic() % interval)

In order not to struggle with other programs for disk access, you can try ionice :

 $ ionice -c3 clean-dir-tree .
  • Made ionice -c3 with the process. Everything worked well for about an hour, then the disk began to fail again and the service began to slow down sharply. How is it going? Doesn’t launch only in io idle guarantee us that as the server loads on the disk, my script will not load it even more? - faoxis
  • @faoxis in order to intelligently answer such questions, you need a lot more information (what kind of hardware, how many VMs are on this hardware, what do they do, versions, software settings, etc.). Find out the name of the scheduler . Also follow the links iostat, sysdig in response to find out the disk load (common IOPS metrics, MB / s read / write and what specific processes) at the time when your script does not load the system and when it loads. And it is especially useful to see when the "service slows down", and your script is not running at all. - jfs