On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote: > --- El vie, 10/12/10, Brian Quinlan escribió: >> On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: >>> I have a process running for a long time, and which >> may use futures of different max_workers count. I think it >> is not too far-fetched to create a new futures object each >> time. Yet, the execution becomes slower after each call, for >> example with http://freehackers.org/~tnagy/futures_test.py: >>> >>> """ >>> import concurrent.futures >>> from queue import Queue >>> import datetime >>> >>> class counter(object): >>> def __init__(self, fut): >>> self.fut = fut >>> >>> def run(self): >>> def >> look_busy(num, obj): >>> >> tot = 0 >>> >> for x in range(num): >>> >> tot += x >>> >> obj.out_q.put(tot) >>> >>> start = >> datetime.datetime.utcnow() >>> self.count = 0 >>> self.out_q = >> Queue(0) >>> for x in >> range(1000): >>> >> self.count += 1 >>> >> self.fut.submit(look_busy, self.count, >> self) >>> >>> while >> self.count: >>> >> self.count -= 1 >>> >> self.out_q.get() >>> >>> delta = >> datetime.datetime.utcnow() - start >>> >> print(delta.total_seconds()) >>> >>> fut = >> concurrent.futures.ThreadPoolExecutor(max_workers=20) >>> for x in range(100): >>> # comment the following line >>> fut = >> concurrent.futures.ThreadPoolExecutor(max_workers=20) >>> c = counter(fut) >>> c.run() >>> """ >>> >>> The runtime grows after each step: >>> 0.216451 >>> 0.225186 >>> 0.223725 >>> 0.222274 >>> 0.230964 >>> 0.240531 >>> 0.24137 >>> 0.252393 >>> 0.249948 >>> 0.257153 >>> ... >>> >>> Is there a mistake in this piece of code? >> >> There is no mistake that I can see but I suspect that the >> circular references that you are building are causing the >> ThreadPoolExecutor to take a long time to be collected. Try >> adding: >> >> c = counter(fut) >> c.run() >> + fut.shutdown() >> >> Even if that fixes your problem, I still don't fully >> understand this because I would expect the runtime to fall >> after a while as ThreadPoolExecutors are collected. > > The shutdown call is indeed a good fix :-) Here is the time response > of the calls to counter() when shutdown is not called: > http://www.freehackers.org/~tnagy/runtime_futures.png FWIW, I think that you are confusion the term "future" with "executor". A future represents a single work item. An executor creates futures and schedules their underlying work. Hmmm....that is very suspicious - it looks like the ThreadPoolExecutors are not being collected. If you are feeling bored you could figure out why not :-) > After trying to stop the program by using CTRL+C, the following > error may appear, after which the process cannot be interrupted: > > """ > 19:18:12 /tmp/build> python3.2 futures_test.py > 0.389657 > 0.417173 > 0.416513 > 0.421424 > 0.449666 > 0.482273 > ^CTraceback (most recent call last): > File "futures_test.py", line 36, in <module> > c.run() > File "futures_test.py", line 22, in run > self.fut.submit(look_busy, self.count, self) > File "/usr/local/lib/python3.2/concurrent/futures/thread.py", line > 114, in submit > self._work_queue.put(w) > File "/usr/local/lib/python3.2/queue.py", line 135, in put > self.not_full.acquire() > KeyboardInterrupt > """ > > It is not expected, is it? It isn't surprising. Python lock acquisitions are not interruptible and anytime you interrupt a program that manipulates locks you may kill the code that was going to cause the lock to be released. Cheers, Brian > Thomas > > > >
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4