Oops. I accidentally replied off-list: On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: > --- El jue, 9/12/10, Brian Quinlan escribió: >> On Dec 9, 2010, at 4:26 AM, Thomas Nagy wrote: >> >>> I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86 >>> ) >> by the futures module which was announced in python 3.2 >> beta. I am a bit stuck with it, so I have a few questions >> about the futures: >>> >>> 1. Is the futures API frozen? >> >> Yes. >> >>> 2. How hard would it be to return the tasks processed >> in an output queue to process/consume the results while they >> are returned? The code does not seem to be very open for >> monkey patching. >> >> You can associate a callback with a submitted future. That >> callback could add the future to your queue. > > Ok, it works. I was thinking the object was cleaned up immediately > after it was used. > >>> 3. How hard would it be to add new tasks dynamically >> (after a task is executed) and have the futures object never >> complete? >> >> I'm not sure that I understand your question. You can >> submit new work to an Executor at until time until it is >> shutdown and a work item can take as long to complete as you >> want. If you are contemplating tasks that don't complete >> then maybe you could be better just scheduling a thread. >> >>> 4. Is there a performance evaluation of the futures >> code (execution overhead) ? >> >> No. Scott Dial did make some performance improvements so he >> might have a handle on its overhead. > > Ok. > > I have a process running for a long time, and which may use futures > of different max_workers count. I think it is not too far-fetched to > create a new futures object each time. Yet, the execution becomes > slower after each call, for example with http://freehackers.org/~tnagy/futures_test.py > : > > """ > import concurrent.futures > from queue import Queue > import datetime > > class counter(object): > def __init__(self, fut): > self.fut = fut > > def run(self): > def look_busy(num, obj): > tot = 0 > for x in range(num): > tot += x > obj.out_q.put(tot) > > start = datetime.datetime.utcnow() > self.count = 0 > self.out_q = Queue(0) > for x in range(1000): > self.count += 1 > self.fut.submit(look_busy, self.count, self) > > while self.count: > self.count -= 1 > self.out_q.get() > > delta = datetime.datetime.utcnow() - start > print(delta.total_seconds()) > > fut = concurrent.futures.ThreadPoolExecutor(max_workers=20) > for x in range(100): > # comment the following line > fut = concurrent.futures.ThreadPoolExecutor(max_workers=20) > c = counter(fut) > c.run() > """ > > The runtime grows after each step: > 0.216451 > 0.225186 > 0.223725 > 0.222274 > 0.230964 > 0.240531 > 0.24137 > 0.252393 > 0.249948 > 0.257153 > ... > > Is there a mistake in this piece of code? There is no mistake that I can see but I suspect that the circular references that you are building are causing the ThreadPoolExecutor to take a long time to be collected. Try adding: c = counter(fut) c.run() + fut.shutdown() Even if that fixes your problem, I still don't fully understand these numbers because I would expect the runtime to fall after a while as ThreadPoolExecutors are collected. Cheers, Brian > Thanks, > Thomas > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20101210/1b45d817/attachment.html>
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4