Mike Coleman wrote:
> Andrew, this is on an (intel) x86_64 box with 64GB of RAM. I don't
> recall the maker or details of the architecture off the top of my
> head, but it would be something "off the rack" from Dell or maybe HP.
> There were other users on the box at the time, but nothing heavy or
> that gave me any reason to think was affecting my program.
>
> It's running CentOS 5 I think, so that might make glibc several years
> old. Your malloc idea sounds plausible to me. If it is a libc
> problem, it would be nice if there was some way we could tell malloc
> to "live for today because there is no tomorrow" in the terminal phase
> of the program.
>
> I'm not sure exactly how to attack this. Callgrind is cool, but no
> way will work on something this size. Timed ltrace output might be
> interesting. Or maybe a gprof'ed Python, though that's more work.
Some malloc()s (notably FreeBSD's) can be externally tuned at runtime
via options in environment variables or other mechanisms - the malloc
man page on your system might be helpful if your platform has something
like this.
It is likely that PyMalloc would be better with a way to disable the
free()ing of empty arenas, or move to an arrangement where (like the
various type free-lists in 2.6+) explicit action can force pruning of
empty arenas - there are other usage patterns than yours which would
benefit (performance wise) from not freeing arenas automatically.
--
-------------------------------------------------------------------------
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au (pref) | Snail: PO Box 370
andymac at pcug.org.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4