Hello everyone and immediately to the point. There is some problem, let it be foo ()

def foo(): capture m, use m, leave m, do smth 

Next, there are n sets containing m elements. The set is one of the parameters for foo (). When the task starts, it must find a free element in the set given to it and capture it, perform some operations, release and continue to do some work. In theory, the elements of m are full-fledged mutex-objects, practically - these are files. In an hour about 30 thousand tasks of foo are filled in the broker. The implementation through file locking had some problems: All the “workers” engaged in processing tasks use one set, i.e. if there is one element in it, then everyone is idle and waiting for him to be released. At the same time, the broker has tasks for other sets that have free elements. If you return tasks to the broker with the help of raise, then in principle everyone is in business, but since the priority queue, this method does not always work. Once lock was hung on the file, but it did not appear and all the "workers" hung on waiting. It was not possible to reconstruct the error, but it is scary if this happens on the battle server. With KazooClient.Semaphore it was possible at least not to let the workers into the sets, which have all the elements blocked, and inside there are lock files again. Actually, it seems to me that the task is pretty clear.

Currently used technologies: Django1.11 + celery + RabbitMQ + kazoo?

Now, actually, the questions are: 1. Is it possible to turn m objects into mutex objects and pass them into a task in order to try to find a free one inside and, in this case, refuse a lock? It seems that the necessary technologies are present in kazoo, but, unfortunately, the deadline is near and I would like to work, not learn. 2. Sometimes a large pile of tasks for one set with high priority can fall into the queue, and if I understood everything correctly, then all the “workers” will hang up again, because will take the task, do not find a free "m" and fall back. It seems to be solved, if for such tasks to create a separate basket, but even here they can begin to juggle the first ten tasks.

If anyone has an idea, but rather a link to the code that solves my problem. I really do not want to write the task allocator myself. Thanks for attention.

  • What the question is not clear. You have mixed several levels of the application. Some slang is incomprehensible: "full-fledged mutex-objects, practically - these are files." If you start to describe the fragmentary implementation details, then you can bring a minimal example of code that illustrates the problem with a specific implementation (it’s better to start from scratch) ¶ And here the design is not clear (if you are interested in the design, describe what you want to work (with user’s point of view), what specific requirements do you have, what specific problems with the current architecture). - jfs
  • I apologize for the confusion, I tried to write extensively. - Olenin Sergey
  • I apologize for the confusion, I tried to write extensively. I have n-folders, one or more files in them. At start, the worker receives the path to a folder. There he must find a free file and take the settings from there. Question: how to make the workers idle as little as possible while waiting for the file and exclude the possibility of simultaneous use. Also, if the basket contains 1000 tasks for folder A, and 1 for B, then the following problem occurs: all n workers dig in folder A. They go in, do not find a free config, and throw the task back into the queue. Need balancing. Thank. - Olenin Sergey
  • again you are trying to describe the fragmentary implementation details ("as"), and not "what" ¶ Having defined the requirements, use the capabilities of the finished task queue (you mentioned celery), for example, look at -Ofair , --scheduler . If there are not enough opportunities or it doesn’t fit at all, see which ready-made job schedulers, workload automation systems are closest to your case (from different categories: Slurm, k8s) - jfs
  • The site has a profile that has m-accounts. You cannot take several simultaneous actions through one account. Accordingly, each task should search for free accounting. Hmm, I’m explaining very badly if you suggest that I turn off the preforms. - Olenin Sergey

0