[Barry] > Very interesting! I think I've noticed the same thing with some > Pipermail (Mailman's archiver) work I've been doing. When doing a > from-scratch re-generation of the archives of say, python-list > (280+MB), it creates some really big dicts, and when it goes to free > them (apparently in gc), the disk just gets hammered. Relatively > little CPU is being used (it's like 80-90% idle), but the machine is > almost unresponsive. This on a 2.4.2 kernel with 256MB. Maybe related, but not the same: Roeland was careful to run tests small enough that the disk didn't get involved. His slowdowns were on all-in-RAM cases, and he reported high CPU utilization throughout. Once the disk gets involved, it can be a true nightmare: while we traverse the dict in cache-friendly left-to-right sequential order, the memory it points *at* jumps all over creation from one slot to the next. Several years ago I amusedly waited 3 hours for a Python program to terminate on my old Win95 box; I would have waited longer except the disk grinding was interfering with hearing the TV <wink>. We could likely "fix that", via sorting the dict entries by pointed-at memory address before decref'ing (waste a second to save a day); note that there's already a superstitious feeble approximation to that in list_dealloc. > I'll try building a 2.2 with pymalloc and see what happens. Well, I'm waiting <wink>.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4