On 06Feb2019 0906, Antoine Pitrou wrote: > For the record there are number of initiatives currently to boost the > usefulness and efficiency of multi-process computation in Python. > > One of them is PEP 574 (zero-copy pickling with out-of-band buffers), > which I'm working on. > > Another is Pierre Glaser's work on allowing pickling of dynamic > functions and classes with the C-accelerated _pickle module (rather than > the slow pure Python implementation): > https://bugs.python.org/issue35900 > https://bugs.python.org/issue35911 > > Another is Davin's work on shared memory managers. > > There are also emerging standards like Apache Arrow that provide a > shared, runtime-agnostic, compute-friendly representation for in-memory > tabular data, and third-party frameworks like Dask which are > potentially able to work on top of that and expose nice end-user APIs. > > For maximum synergy between these initiatives and the resulting APIs, > it is better if things are done in the open ;-) Hopefully our steering council can determine (or delegate the determination of) the direction we should go here so we can all be pulling in the same direction :) That said, there are certainly a number of interacting components and not a lot of information about how they interact and overlap. A good start would be to identify the likely overlap of this work to see where they can build upon each other rather than competing, as well as estimating the long-term burden of standardising. Cheers, Steve
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4