On 12/16/2010 02:29 PM, Michael Foord wrote: > On 16/12/2010 11:09, Dimitrios Pritsos wrote: >> >> Hello Core Developers, >> >> My name is Dimitrios and I am newbie in python. I am working on a >> Project (part of my PhD) that is called Synergeticprocessing module. >> Initially is imitating the multiprocessing built in module but the >> processes are distributed on a LAN and not Locally. The main issue I >> have is with Pickle module. And I think I found some kind of BUG in >> the built in multiprocessing module. >> > > Hello Dimitrios, > > Please post your bug report to the Python bug tracker. As you think > you have a fix for the issue it is much more likely to be applied > quickly if you can package it in the form of a test that demonstrates > the issue and a patch that fixes it. > > http://bugs.python.org/ > > All the best, > > Michael Foord > Hello Michael, OK I will do sent it to the bug tracker. But what about the last issue i.e. that even if the class is transfered-and-pickled-unpickled it raises an exception if the class is defined into the __main__ script and not imported from an other one. (see below) could you please guide me where should I post this for someone to help me fix it! Best Regards, Dimitrios > > >> (Synergeticprocessing module is located at GitHub: >> https://github.com/dpritsos/synergeticprocessing) >> >> Starting with the "BUG". In case someone uses the >> multiprocessing.Pool of processes he/she has to face the problem of >> types.MehtodType Impossible pickling. That is you cannot dispatch an >> class instance method to the to the Process Pool. However, digging in >> to the Source Code of the module there are few lines that resolve >> this issue however are not activated or they are faultily activated >> so they do not work. This is the 'BUG' >> >> _@ ....../multiprocessing/forking.py_ >> >> >> . >> . >> . >> >> # >> # Try making some callable types picklable >> # >> >> from pickle import Pickler >> class ForkingPickler(Pickler): >> dispatch = Pickler.dispatch.copy() >> >> @classmethod >> def register(cls, type, reduce): >> def dispatcher(self, obj): >> rv = reduce(obj) >> self.save_reduce(obj=obj, *rv) >> cls.dispatch[type] = dispatcher >> >> def _reduce_method(m): >> if m.im_self is None: >> return getattr, (m.im_class, m.im_func.func_name) >> else: >> return getattr, (m.im_self, m.im_func.func_name) >> ForkingPickler.register(type(ForkingPickler.save), _reduce_method) >> >> def _reduce_method_descriptor(m): >> return getattr, (m.__objclass__, m.__name__) >> ForkingPickler.register(type(list.append), _reduce_method_descriptor) >> ForkingPickler.register(type(int.__add__), _reduce_method_descriptor) >> >> #def _reduce_builtin_function_or_method(m): >> # return getattr, (m.__self__, m.__name__) >> #ForkingPickler.register(type(list().append), >> _reduce_builtin_function_or_method) >> #ForkingPickler.register(type(int().__add__), >> _reduce_builtin_function_or_method) >> . >> . >> . >> >> The RED lines are not doing the job, for some reason they are not >> managing to register the GREEN function as a global reduce/pickling >> function even if you call the registration function into you __main__ >> script. >> >> The solution I found is just to do this >> * >> import copy_reg >> import types* >> >> def _reduce_method(m): >> if m.im_self is None: >> return getattr, (m.im_class, m.im_func.func_name) >> else: >> return getattr, (m.im_self, m.im_func.func_name) >> >> *copy_reg.pickle(types.MethodType, _reduce_method)* >> . >> . >> . >> >> Doing that everything works FINE. But ONLY for local methods i.e. the >> ones that their class is defined on the __main__ script or other >> import-ed. >> >> In case you want to send something remotely (in an other machine) or >> to an other __main__ script running separately then you get a message >> like this: >> >> 'module' object has no attribute '<my_class>' >> >> The only way to resolve this is firstly to import a script that has >> <my_class> defined there and everything works fine. >> >> SO the problems it seems to be that the *m.im_class* (look code >> above) has some attribute __module__ defined as __module__ = >> '__main__' or something like that. And this is the reason why remote >> script cannot execute the function. I mean that the _reduce_method() >> above DOES is pickling the whole CLASS object so there is no reason >> not to be executed at the remote script. Besides it does as mentioned >> above in you just import this the user defined class form an other >> script. >> >> >> I have already spent about 12 weeks working on building my >> synergeticPool and resolve the issue of Pickling and only 2 days >> needed for the code of the Pool and the rest of the time was spent >> for the Pickling issues, and study all the Class related mechanics of >> python. That was the reason I ve started digging the >> multipocessessing module and found this say 'BUG', and finally sent >> this email. >> >> Best Regards, >> >> >> Dimitrios >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > -- > > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > May you share freely, never taking more than you give. > -- the sqlite blessinghttp://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20101216/b5fc7d38/attachment.html>
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4