On 8 September 2015 at 11:07, Gary Robinson <garyrob at me.com> wrote: >> I guess a third possible solution, although it would probably have >> meant developing something for yourself which would have hit the same >> "programmer time is critical" issue that you noted originally, would >> be to create a module that managed the data structure in shared >> memory, and then use that to access the data from the multiple >> processes. > > I think you mean, write a non-python data structure in shared memory, such as writing it in C? If so, you’re right, I want to avoid the time overhead for writing something like that. Although I have used C data in shared-memory in the past when the data structure was simple enough. It’s not a foreign concept to me — it just would have been a real nuisance in this case. > > An in-memory SQLLite database would have been too slow, at least if I used any kind of ORM. Without an ORM it still would have slowed things down while making for code that’s harder to read and write. While I have used in-memory SQLite code at times, I’m not sure how much slowdown it would have engendered in this case. > >> Your suggestion (2), of having a non-refcounted data structure is >> essentially this, doable as an extension module. The core data >> structures all use refcounting, and that's unlikely to change, but >> there's nothing to say that an extension module couldn't implement >> fast data structures with objects allocated from a pool of >> preallocated memory which is only freed as a complete block. > > Again, I think you’re talking about non-Python data structures, for instance C structures, which could be written to be “fast”? Again, I want to avoid writing that kind of code. Sure, for a production project where I had more programmer time, that would be a solution, but that wasn’t my situation. And, ideally, even if I had more time, I would greatly prefer not to have to spend it on that kind of code. I like Python because it saves me time and eliminates potential bugs that are associated with language like C but not with Python (primarily memory management related). To the extent that I have to write and debug external modules in C or C++, it doesn’t. I've used cffi to good effect to gain some of the benefits of the "share a lump of memory" model.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4