Good day!

When trying to port my code from linux (arch) to windows (7) I had problems with creating child processes (multiprocessing.Process).

from stompy import stomp from multiprocessing import Process import json def tmp(q, w, e): print q, w, e return s = stomp.Stomp(amq_ip, amq_port) s.connect(username=amq_user, password=amq_pass) s.subscribe({'destination':'/queue/%s' % amq_queue, 'ack':'client'}) while True: try: frame = s.receive_frame() except: continue # тут костылирую # жду советов от гуру, как избавиться body = json.loads(frame.as_string().split('\n\n')[1][:-1]) p = Process(target=tmp, args=(frame, s, body)) p.daemon = True p.start() p.join()

I subscribe to the ActiveMQ queue, and after receiving a new message from it, I process it in a new thread. Everything works fine under the linux, under Windows - I get a TypeError template: can't pickle thread.lock objects.

 Traceback (most recent call last): File "./tmp.py", line 31, in <module> p.start() File "C:\Python27\lib\multiprocessing\process.py", line 130, in start self._popen = Popen(self) File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__ dump(process_obj, to_child, HIGHEST_PROTOCOL) File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump ForkingPickler(file, protocol).dump(obj) File "C:\Python27\lib\pickle.py", line 224, in dump self.save(obj) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 425, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 655, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 554, in save_tuple save(element) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 425, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 655, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 425, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 655, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 692, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 731, in save_inst save(stuff) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 655, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 425, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 655, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 306, in save rv = reduce(self.proto) TypeError: can't pickle thread.lock objects

In this case, if the queue descriptor (s) and frame (frame) are not passed to the parameters of the function of the new flow, then everything is working fine. Windows 7 x64. Python 2.7.12.

  • If you are given an exhaustive answer, mark it as correct (a daw opposite the selected answer). - Nicolas Chabanovsky ♦

1 answer 1

In this case, the problem is in the Stompy library. Here is a brief example illustrating the problem:

 from multiprocessing import Process, Manager from Queue import Queue def tmp(q, w, e): print q, w, e if __name__ == "__main__": q = Queue() p = Process(target=tmp, args=("fname", q, "body")) p.daemon = True p.start() p.join()

During the execution, a TypeError: can't pickle thread.lock objects exception will be thrown TypeError: can't pickle thread.lock objects . In order to transfer data from one process to another, the pickle module is used - all data is serialized by it and sent in a serialized form. At Stompy in the guts of somewhere created a queue and transfer such an object will not work. The corrected example will look like this:

 from multiprocessing import Manager q = Manager().Queue()
  • Correctly understand that this solution involves the rejection of the use of Stompy? - rrgang
  • Not necessarily, the essence of other solutions comes down to the fact that it is not necessary to transfer this object to other processes. - m9_psy