Showing content from https://mail.python.org/pipermail/python-dev/2016-January.txt below:
Hi, I proposed a patch for the devguide to give the current status of all Python branches: active, bugfix, security only, end-of-line, with their end-of-life when applicable (past date or scheduled date) http://bugs.python.org/issue26165 What do you think? Does it look correct? We will have to update this table each time that the status of a branch change. Hopefully, it's not a common event, so it will not require a lot of work for release managers :-) Victor From brett at python.org Wed Jan 20 12:45:11 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 17:45:11 +0000 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: On Tue, 19 Jan 2016 at 19:33 Martin Panter wrote: > On 19 January 2016 at 20:12, Brett Cannon wrote: > > Here is a proposed update: > > > > diff -r 633f51d10a67 pep-0007.txt > > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800 > > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800 > > @@ -75,9 +75,9 @@ > > } > > > > * Code structure: one space between keywords like ``if``, ``for`` and > > - the following left paren; no spaces inside the paren; braces may be > > - omitted where C permits but when present, they should be formatted > > - as shown:: > > + the following left paren; no spaces inside the paren; braces are > > + strongly preferred but may be omitted where C permits, and they > > + should be formatted as shown:: > > > > if (mro != NULL) { > > ... > > This change seems to be accidentally smuggled in, in the guise of a > PEP 512 update :) > Darn, sorry about that; forgot I had that change sitting in my peps checkout. I'll revert it when I get home (unless the change is actually acceptable to Guido). -Brett > > My view is I prefer always using curly brackets in my own code. It is > easier to add printf() debugging without making mistakes. I support > ?strongly preferring? them in the style guide, which is as much as a > style guide can do anyway. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jan 20 12:53:33 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 17:53:33 +0000 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: On Wed, 20 Jan 2016 at 09:41 Victor Stinner wrote: > Hi, > > I proposed a patch for the devguide to give the current status of all > Python branches: active, bugfix, security only, end-of-line, with > their end-of-life when applicable (past date or scheduled date) > http://bugs.python.org/issue26165 > > What do you think? Does it look correct? > I would update it have a "first release" date column and also the projected EOL for Python 3.5. Otherwise LGTM. -Brett > > We will have to update this table each time that the status of a > branch change. Hopefully, it's not a common event, so it will not > require a lot of work for release managers :-) > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Jan 20 12:58:12 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 20 Jan 2016 09:58:12 -0800 Subject: [Python-Dev] Code formatter bot In-Reply-To: <85wpr4myvy.fsf@benfinney.id.au> References: <569EA3A6.4010802@email.de> <85wpr4myvy.fsf@benfinney.id.au> Message-ID: <2859591B-BAF8-4D29-9509-AF2E6C48D076@yahoo.com> On Jan 20, 2016, at 00:35, Ben Finney wrote: > > francismb writes: > >> what's your opinion about a code-formatter bot for cpython. > > What is the proposal? The opinions will surely depend on: ... plus: * How does the formatter bot deal with "legacy code"? Large parts of CPython predate PEPs 7 and 8, and the decision was made long ago not to reformat existing code unless that code is being substantially modified for some other reason. Similarly, when the PEPs are updated, the usual decision is to not reformat old code. * When code _is_ auto-reformatted, what tools do you have to help git's merge logic, Reitveld, human readers looking at diffs or blame/annotate locally or on the web, etc. look past the hundreds of trivial changes to highlight the ones that matter? * What's the argument for specifically automating code formatting instead of any of the other things a commit-triggered linter can catch just as easily? But one comment on Ben's comment: > * If on the other hand you propose to enforce only those rules which > are strict enough to be applied automatically (e.g. ?don't mix > spaces and TABs?, ?encode source using UTF-8 only?) then that's best > done by editor plug-ins like EditorConfig[0]. In my experience (although mostly with projects with a lot fewer contributors than CPython...), it can be helpful to have both suggested editor plugins that do the auto formatting on the dev's computer, and VCS-triggered checkers that ensure the formatting was correct. That catches those occasional cases where you do a quick "trivial" edit in nano instead of your usual editor and then forget you did so and try to check in), without the nasty side-effects you mention later (like committing code you haven't seen). (Of course writing plugins that understand "legacy code" in the exact same way as the commit filter can be tricky, but in that case, it's better to know that one or the other isn't working as intended--both so a human can decide, and so people can see the bug in the plugin or filter--than to automatically make changes that weren't wanted.) From ben+python at benfinney.id.au Wed Jan 20 13:09:31 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 21 Jan 2016 05:09:31 +1100 Subject: [Python-Dev] Code formatter bot References: <569EA3A6.4010802@email.de> <85wpr4myvy.fsf@benfinney.id.au> <2859591B-BAF8-4D29-9509-AF2E6C48D076@yahoo.com> Message-ID: <85oacgm8bo.fsf@benfinney.id.au> Andrew Barnert via Python-Dev writes: > [?] it can be helpful to have both suggested editor plugins that do > the auto formatting on the dev's computer, and VCS-triggered checkers > that ensure the formatting was correct. Right, I was not intending the different stages to be exclusive. I was seeking clarification from the OP on the intended stages proposed to be automated. It's also worth noting that neither of the above approaches you mention would qualify IMO to be termed a ?bot?, since they still leave it to the human operator to deal with formatting violations before the edits reach the VCS. But that's another clarification I'm seeking from the OP: what ?bot? is being proposed? -- \ ?Nothing is more sacred than the facts.? ?Sam Harris, _The End | `\ of Faith_, 2004 | _o__) | Ben Finney From yselivanov.ml at gmail.com Wed Jan 20 13:02:56 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 20 Jan 2016 13:02:56 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: Message-ID: <569FCBD0.4040307@gmail.com> On 2016-01-18 5:43 PM, Victor Stinner wrote: > Is someone opposed to this PEP 509? > > The main complain was the change on the public Python API, but the PEP > doesn't change the Python API anymore. > > I'm not aware of any remaining issue on this PEP. Victor, I've been experimenting with the PEP to implement a per-opcode cache in ceval loop (I'll share my progress on that in a few days). This allows to significantly speedup LOAD_GLOBAL and LOAD_METHOD opcodes, to the point, where they don't require any dict lookups at all. Some macro-benchmarks (such as chameleon_v2) demonstrate impressive ~10% performance boost. I rely on your dict->ma_version to implement cache invalidation. However, besides guarding against version change, I also want to guard against the dict being swapped for another dict, to avoid situations like this: def foo(): print(bar) exec(foo.__code__, {'bar': 1}, {}) exec(foo.__code__, {'bar': 2}, {}) What I propose is to add a pointer "ma_extra" (same 64bits), which will be set to NULL for most dict instances (instead of ma_version). "ma_extra" can then point to a struct that has a globally unique dict ID (uint64), and a version tag (unit64). A macro like PyDict_GET_ID and PyDict_GET_VERSION could then efficiently fetch the version/unique ID of the dict for guards. "ma_extra" would also make it easier for us to extend dicts in the future. Yury From fijall at gmail.com Wed Jan 20 13:15:21 2016 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 20 Jan 2016 19:15:21 +0100 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FCBD0.4040307@gmail.com> References: <569FCBD0.4040307@gmail.com> Message-ID: The easiest version is to have global numbering (as opposed to local). Anyway, I would strongly suggest getting some benchmarks done and showing performance benefits first, because you don't want PEPs to be final when you don't exactly know the details. On Wed, Jan 20, 2016 at 7:02 PM, Yury Selivanov wrote: > On 2016-01-18 5:43 PM, Victor Stinner wrote: >> >> Is someone opposed to this PEP 509? >> >> The main complain was the change on the public Python API, but the PEP >> doesn't change the Python API anymore. >> >> I'm not aware of any remaining issue on this PEP. > > > Victor, > > I've been experimenting with the PEP to implement a per-opcode > cache in ceval loop (I'll share my progress on that in a few > days). This allows to significantly speedup LOAD_GLOBAL and > LOAD_METHOD opcodes, to the point, where they don't require > any dict lookups at all. Some macro-benchmarks (such as > chameleon_v2) demonstrate impressive ~10% performance boost. > > I rely on your dict->ma_version to implement cache invalidation. > > However, besides guarding against version change, I also want > to guard against the dict being swapped for another dict, to > avoid situations like this: > > > def foo(): > print(bar) > > exec(foo.__code__, {'bar': 1}, {}) > exec(foo.__code__, {'bar': 2}, {}) > > > What I propose is to add a pointer "ma_extra" (same 64bits), > which will be set to NULL for most dict instances (instead of > ma_version). "ma_extra" can then point to a struct that has a > globally unique dict ID (uint64), and a version tag (unit64). > A macro like PyDict_GET_ID and PyDict_GET_VERSION could then > efficiently fetch the version/unique ID of the dict for guards. > > "ma_extra" would also make it easier for us to extend dicts > in the future. > > Yury > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com From brett at python.org Wed Jan 20 13:22:50 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 18:22:50 +0000 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FCBD0.4040307@gmail.com> References: <569FCBD0.4040307@gmail.com> Message-ID: On Wed, 20 Jan 2016 at 10:11 Yury Selivanov wrote: > On 2016-01-18 5:43 PM, Victor Stinner wrote: > > Is someone opposed to this PEP 509? > > > > The main complain was the change on the public Python API, but the PEP > > doesn't change the Python API anymore. > > > > I'm not aware of any remaining issue on this PEP. > > Victor, > > I've been experimenting with the PEP to implement a per-opcode > cache in ceval loop (I'll share my progress on that in a few > days). This allows to significantly speedup LOAD_GLOBAL and > LOAD_METHOD opcodes, to the point, where they don't require > any dict lookups at all. Some macro-benchmarks (such as > chameleon_v2) demonstrate impressive ~10% performance boost. > Ooh, now my brain is trying to figure out the design of the cache. :) > > I rely on your dict->ma_version to implement cache invalidation. > > However, besides guarding against version change, I also want > to guard against the dict being swapped for another dict, to > avoid situations like this: > > > def foo(): > print(bar) > > exec(foo.__code__, {'bar': 1}, {}) > exec(foo.__code__, {'bar': 2}, {}) > > > What I propose is to add a pointer "ma_extra" (same 64bits), > which will be set to NULL for most dict instances (instead of > ma_version). "ma_extra" can then point to a struct that has a > globally unique dict ID (uint64), and a version tag (unit64). > A macro like PyDict_GET_ID and PyDict_GET_VERSION could then > efficiently fetch the version/unique ID of the dict for guards. > > "ma_extra" would also make it easier for us to extend dicts > in the future. > Why can't you simply use the id of the dict object as the globally unique dict ID? It's already globally unique amongst all Python objects which makes it inherently unique amongst dicts. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Jan 20 13:36:53 2016 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 20 Jan 2016 19:36:53 +0100 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> Message-ID: On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon wrote: > > > On Wed, 20 Jan 2016 at 10:11 Yury Selivanov wrote: >> >> On 2016-01-18 5:43 PM, Victor Stinner wrote: >> > Is someone opposed to this PEP 509? >> > >> > The main complain was the change on the public Python API, but the PEP >> > doesn't change the Python API anymore. >> > >> > I'm not aware of any remaining issue on this PEP. >> >> Victor, >> >> I've been experimenting with the PEP to implement a per-opcode >> cache in ceval loop (I'll share my progress on that in a few >> days). This allows to significantly speedup LOAD_GLOBAL and >> LOAD_METHOD opcodes, to the point, where they don't require >> any dict lookups at all. Some macro-benchmarks (such as >> chameleon_v2) demonstrate impressive ~10% performance boost. > > > Ooh, now my brain is trying to figure out the design of the cache. :) > >> >> >> I rely on your dict->ma_version to implement cache invalidation. >> >> However, besides guarding against version change, I also want >> to guard against the dict being swapped for another dict, to >> avoid situations like this: >> >> >> def foo(): >> print(bar) >> >> exec(foo.__code__, {'bar': 1}, {}) >> exec(foo.__code__, {'bar': 2}, {}) >> >> >> What I propose is to add a pointer "ma_extra" (same 64bits), >> which will be set to NULL for most dict instances (instead of >> ma_version). "ma_extra" can then point to a struct that has a >> globally unique dict ID (uint64), and a version tag (unit64). >> A macro like PyDict_GET_ID and PyDict_GET_VERSION could then >> efficiently fetch the version/unique ID of the dict for guards. >> >> "ma_extra" would also make it easier for us to extend dicts >> in the future. > > > Why can't you simply use the id of the dict object as the globally unique > dict ID? It's already globally unique amongst all Python objects which makes > it inherently unique amongst dicts. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > Brett, you need two things - the ID of the dict and the version tag. What we do in pypy is we have a small object (called, surprisingly, VersionTag()) and we use the ID of that. That way you can change the version id of an existing dict and have only one field. From yselivanov.ml at gmail.com Wed Jan 20 13:38:49 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 20 Jan 2016 13:38:49 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> Message-ID: <569FD439.9040704@gmail.com> On 2016-01-20 1:15 PM, Maciej Fijalkowski wrote: > [..] > > Anyway, I would strongly suggest getting some benchmarks done and > showing performance benefits first, because you don't want PEPs to be > final when you don't exactly know the details. I agree 100%. Yury From tjreedy at udel.edu Wed Jan 20 13:39:58 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Jan 2016 13:39:58 -0500 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: On 1/20/2016 12:40 PM, Victor Stinner wrote: > Hi, > > I proposed a patch for the devguide to give the current status of all > Python branches: active, bugfix, security only, end-of-line, with > their end-of-life when applicable (past date or scheduled date) > http://bugs.python.org/issue26165 > > What do you think? Does it look correct? I thought end-of-life was 5 years after initial release, not 5 years after last bugfix. That would put eol for 3.4 in Feb 2019, I believe. > We will have to update this table each time that the status of a > branch change. Hopefully, it's not a common event, so it will not > require a lot of work for release managers :-) I believe there is some text describing current releases somewhere that also needs to be changed. The release pep or scripts should have a reminder in the sections about the transitions. -- Terry Jan Reedy From tjreedy at udel.edu Wed Jan 20 13:42:24 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Jan 2016 13:42:24 -0500 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: On 1/20/2016 12:45 PM, Brett Cannon wrote: > > > On Tue, 19 Jan 2016 at 19:33 Martin Panter > wrote: > > On 19 January 2016 at 20:12, Brett Cannon > wrote: > > Here is a proposed update: > > > > diff -r 633f51d10a67 pep-0007.txt > > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800 > > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800 > > @@ -75,9 +75,9 @@ > > } > > > > * Code structure: one space between keywords like ``if``, > ``for`` and > > - the following left paren; no spaces inside the paren; braces > may be > > - omitted where C permits but when present, they should be formatted > > - as shown:: > > + the following left paren; no spaces inside the paren; braces are > > + strongly preferred but may be omitted where C permits, and they > > + should be formatted as shown:: > > > > if (mro != NULL) { > > ... > > This change seems to be accidentally smuggled in, in the guise of a > PEP 512 update :) > > > Darn, sorry about that; forgot I had that change sitting in my peps > checkout. I'll revert it when I get home (unless the change is actually > acceptable to Guido). I thought that the above was your intentional compromise change given the range of opinions ;-). -- Terry Jan Reedy From yselivanov.ml at gmail.com Wed Jan 20 13:45:51 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 20 Jan 2016 13:45:51 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> Message-ID: <569FD5DF.2000700@gmail.com> Brett, On 2016-01-20 1:22 PM, Brett Cannon wrote: > > > On Wed, 20 Jan 2016 at 10:11 Yury Selivanov > wrote: > > On 2016-01-18 5:43 PM, Victor Stinner wrote: > > Is someone opposed to this PEP 509? > > > > The main complain was the change on the public Python API, but > the PEP > > doesn't change the Python API anymore. > > > > I'm not aware of any remaining issue on this PEP. > > Victor, > > I've been experimenting with the PEP to implement a per-opcode > cache in ceval loop (I'll share my progress on that in a few > days). This allows to significantly speedup LOAD_GLOBAL and > LOAD_METHOD opcodes, to the point, where they don't require > any dict lookups at all. Some macro-benchmarks (such as > chameleon_v2) demonstrate impressive ~10% performance boost. > > > Ooh, now my brain is trying to figure out the design of the cache. :) Yeah, it's tricky. I'll need some time to draft a comprehensible overview. And I want to implement a couple more optimizations and benchmark it better. BTW, I've some updates (html5lib benchmark for py3, new benchmarks for calling C methods, and I want to port some PyPy benchmakrs) to the benchmarks suite. Should I just commit them, or should I use bugs.python.org? > > I rely on your dict->ma_version to implement cache invalidation. > > However, besides guarding against version change, I also want > to guard against the dict being swapped for another dict, to > avoid situations like this: > > > def foo(): > print(bar) > > exec(foo.__code__, {'bar': 1}, {}) > exec(foo.__code__, {'bar': 2}, {}) > > > What I propose is to add a pointer "ma_extra" (same 64bits), > which will be set to NULL for most dict instances (instead of > ma_version). "ma_extra" can then point to a struct that has a > globally unique dict ID (uint64), and a version tag (unit64). > A macro like PyDict_GET_ID and PyDict_GET_VERSION could then > efficiently fetch the version/unique ID of the dict for guards. > > "ma_extra" would also make it easier for us to extend dicts > in the future. > > > Why can't you simply use the id of the dict object as the globally > unique dict ID? It's already globally unique amongst all Python > objects which makes it inherently unique amongst dicts. We have a freelist for dicts -- so if the dict dies, there could be a new dict in its place, with the same ma_version. While the probability of such hiccups is low, we still have to account for it. Yury From yselivanov.ml at gmail.com Wed Jan 20 14:00:31 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 20 Jan 2016 14:00:31 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> Message-ID: <569FD94F.2000808@gmail.com> On 2016-01-20 1:36 PM, Maciej Fijalkowski wrote: > On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon wrote: >> >> On Wed, 20 Jan 2016 at 10:11 Yury Selivanov wrote: [..] >>> "ma_extra" would also make it easier for us to extend dicts >>> in the future. >> >> Why can't you simply use the id of the dict object as the globally unique >> dict ID? It's already globally unique amongst all Python objects which makes >> it inherently unique amongst dicts. >> >> > Brett, you need two things - the ID of the dict and the version tag. > What we do in pypy is we have a small object (called, surprisingly, > VersionTag()) and we use the ID of that. That way you can change the > version id of an existing dict and have only one field. Yeah, that's essentially what I propose with ma_extra. Yury From fijall at gmail.com Wed Jan 20 14:02:53 2016 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 20 Jan 2016 20:02:53 +0100 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FD94F.2000808@gmail.com> References: <569FCBD0.4040307@gmail.com> <569FD94F.2000808@gmail.com> Message-ID: On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov wrote: > > > On 2016-01-20 1:36 PM, Maciej Fijalkowski wrote: >> >> On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon wrote: >>> >>> >>> On Wed, 20 Jan 2016 at 10:11 Yury Selivanov >>> wrote: > > [..] >>>> >>>> "ma_extra" would also make it easier for us to extend dicts >>>> in the future. >>> >>> >>> Why can't you simply use the id of the dict object as the globally unique >>> dict ID? It's already globally unique amongst all Python objects which >>> makes >>> it inherently unique amongst dicts. >>> >>> >> Brett, you need two things - the ID of the dict and the version tag. >> What we do in pypy is we have a small object (called, surprisingly, >> VersionTag()) and we use the ID of that. That way you can change the >> version id of an existing dict and have only one field. > > > > Yeah, that's essentially what I propose with ma_extra. > > Yury The trick is we use only one field :-) you're proposing to have both fields - version tag and dict id. Why not just use the id of the object (without any fields)? From yselivanov.ml at gmail.com Wed Jan 20 14:08:48 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 20 Jan 2016 14:08:48 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> <569FD94F.2000808@gmail.com> Message-ID: <569FDB40.9060906@gmail.com> On 2016-01-20 2:02 PM, Maciej Fijalkowski wrote: > On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov wrote: > [..] >>> Brett, you need two things - the ID of the dict and the version tag. >>> What we do in pypy is we have a small object (called, surprisingly, >>> VersionTag()) and we use the ID of that. That way you can change the >>> version id of an existing dict and have only one field. >> Yeah, that's essentially what I propose with ma_extra. >> >> Yury > The trick is we use only one field :-) > > you're proposing to have both fields - version tag and dict id. Why > not just use the id of the object (without any fields)? What if your old dict is GCed, its "VersionTag()" (1) object is freed, and you have a new dict, for which a new "VersionTag()" (2) object happens to be allocated at the same address as (1)? Yury From fijall at gmail.com Wed Jan 20 14:09:54 2016 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 20 Jan 2016 20:09:54 +0100 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FDB40.9060906@gmail.com> References: <569FCBD0.4040307@gmail.com> <569FD94F.2000808@gmail.com> <569FDB40.9060906@gmail.com> Message-ID: On Wed, Jan 20, 2016 at 8:08 PM, Yury Selivanov wrote: > > On 2016-01-20 2:02 PM, Maciej Fijalkowski wrote: >> >> On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov >> wrote: >> > [..] >>>> >>>> Brett, you need two things - the ID of the dict and the version tag. >>>> What we do in pypy is we have a small object (called, surprisingly, >>>> VersionTag()) and we use the ID of that. That way you can change the >>>> version id of an existing dict and have only one field. >>> >>> Yeah, that's essentially what I propose with ma_extra. >>> >>> Yury >> >> The trick is we use only one field :-) >> >> you're proposing to have both fields - version tag and dict id. Why >> not just use the id of the object (without any fields)? > > > What if your old dict is GCed, its "VersionTag()" (1) object is > freed, and you have a new dict, for which a new "VersionTag()" (2) > object happens to be allocated at the same address as (1)? > > Yury > You don't free a version tag that's stored in the guard. You store the object and not id From yselivanov.ml at gmail.com Wed Jan 20 14:23:43 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 20 Jan 2016 14:23:43 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> <569FD94F.2000808@gmail.com> <569FDB40.9060906@gmail.com> Message-ID: <569FDEBF.2090709@gmail.com> On 2016-01-20 2:09 PM, Maciej Fijalkowski wrote: >> > > You don't free a version tag that's stored in the guard. You store the > object and not id Ah, got it. Although for my current cache design it would be more memory efficient to use the dict itself to store its own unique id and tag, hence my "ma_extra" proposal. In any case, the current "ma_version" proposal is flawed :( Yury From fijall at gmail.com Wed Jan 20 14:31:42 2016 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 20 Jan 2016 20:31:42 +0100 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FDEBF.2090709@gmail.com> References: <569FCBD0.4040307@gmail.com> <569FD94F.2000808@gmail.com> <569FDB40.9060906@gmail.com> <569FDEBF.2090709@gmail.com> Message-ID: there is also the problem that you don't want it on all dicts. So having two extra words is more to pay than having extra objects (also, comparison is cheaper for guards) On Wed, Jan 20, 2016 at 8:23 PM, Yury Selivanov wrote: > > > On 2016-01-20 2:09 PM, Maciej Fijalkowski wrote: >>> >>> > >> >> You don't free a version tag that's stored in the guard. You store the >> object and not id > > > Ah, got it. Although for my current cache design it would be > more memory efficient to use the dict itself to store its own > unique id and tag, hence my "ma_extra" proposal. In any case, > the current "ma_version" proposal is flawed :( > > Yury From v+python at g.nevcal.com Wed Jan 20 14:45:42 2016 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 20 Jan 2016 11:45:42 -0800 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> Message-ID: <569FE3E6.303@g.nevcal.com> On 1/20/2016 10:36 AM, Maciej Fijalkowski wrote: >> Why can't you simply use the id of the dict object as the globally unique >> >dict ID? It's already globally unique amongst all Python objects which makes >> >it inherently unique amongst dicts. >> > >> >_______________________________________________ >> >Python-Dev mailing list >> >Python-Dev at python.org >> >https://mail.python.org/mailman/listinfo/python-dev >> >Unsubscribe: >> >https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com >> > > Brett, you need two things - the ID of the dict and the version tag. > What we do in pypy is we have a small object (called, surprisingly, > VersionTag()) and we use the ID of that. That way you can change the > version id of an existing dict and have only one field. For the reuse case, can't you simply keep the ma_version "live" in dict items on the free list, rather than starting over at (presumably) 0 ? Then if the dict is reused, it bumps the ma_version, and the fallback code goes on with (presumably) relocating the original dict (oops, it's gone), and dealing with the fallout. Then you can use the regular dict id as the globally unique dict id? And only need the one uint64, rather than a separately allocated extra pair of uint64? -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jan 20 15:18:53 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 20:18:53 +0000 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: On Wed, 20 Jan 2016 at 10:45 Terry Reedy wrote: > On 1/20/2016 12:45 PM, Brett Cannon wrote: > > > > > > On Tue, 19 Jan 2016 at 19:33 Martin Panter > > wrote: > > > > On 19 January 2016 at 20:12, Brett Cannon > > wrote: > > > Here is a proposed update: > > > > > > diff -r 633f51d10a67 pep-0007.txt > > > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800 > > > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800 > > > @@ -75,9 +75,9 @@ > > > } > > > > > > * Code structure: one space between keywords like ``if``, > > ``for`` and > > > - the following left paren; no spaces inside the paren; braces > > may be > > > - omitted where C permits but when present, they should be > formatted > > > - as shown:: > > > + the following left paren; no spaces inside the paren; braces > are > > > + strongly preferred but may be omitted where C permits, and they > > > + should be formatted as shown:: > > > > > > if (mro != NULL) { > > > ... > > > > This change seems to be accidentally smuggled in, in the guise of a > > PEP 512 update :) > > > > > > Darn, sorry about that; forgot I had that change sitting in my peps > > checkout. I'll revert it when I get home (unless the change is actually > > acceptable to Guido). > > I thought that the above was your intentional compromise change given > the range of opinions ;-). > It is, but Guido is the author of PEP 7 and so I didn't want to check in the change without his approval first. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jan 20 15:20:02 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 20:20:02 +0000 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: On Wed, 20 Jan 2016 at 10:40 Terry Reedy wrote: > On 1/20/2016 12:40 PM, Victor Stinner wrote: > > Hi, > > > > I proposed a patch for the devguide to give the current status of all > > Python branches: active, bugfix, security only, end-of-line, with > > their end-of-life when applicable (past date or scheduled date) > > http://bugs.python.org/issue26165 > > > > What do you think? Does it look correct? > > I thought end-of-life was 5 years after initial release, not 5 years > after last bugfix. It is, which is why I requested the first release date be a column. > That would put eol for 3.4 in Feb 2019, I believe. > > > We will have to update this table each time that the status of a > > branch change. Hopefully, it's not a common event, so it will not > > require a lot of work for release managers :-) > > I believe there is some text describing current releases somewhere that > also needs to be changed. The release pep or scripts should have a > reminder in the sections about the transitions. > PEP 101 would need a tweak to remind the RM to update the devguide. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jan 20 15:23:59 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 20:23:59 +0000 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FD5DF.2000700@gmail.com> References: <569FCBD0.4040307@gmail.com> <569FD5DF.2000700@gmail.com> Message-ID: On Wed, 20 Jan 2016 at 10:46 Yury Selivanov wrote: > Brett, > > On 2016-01-20 1:22 PM, Brett Cannon wrote: > > > > > > On Wed, 20 Jan 2016 at 10:11 Yury Selivanov > > wrote: > > > > On 2016-01-18 5:43 PM, Victor Stinner wrote: > > > Is someone opposed to this PEP 509? > > > > > > The main complain was the change on the public Python API, but > > the PEP > > > doesn't change the Python API anymore. > > > > > > I'm not aware of any remaining issue on this PEP. > > > > Victor, > > > > I've been experimenting with the PEP to implement a per-opcode > > cache in ceval loop (I'll share my progress on that in a few > > days). This allows to significantly speedup LOAD_GLOBAL and > > LOAD_METHOD opcodes, to the point, where they don't require > > any dict lookups at all. Some macro-benchmarks (such as > > chameleon_v2) demonstrate impressive ~10% performance boost. > > > > > > Ooh, now my brain is trying to figure out the design of the cache. :) > > Yeah, it's tricky. I'll need some time to draft a comprehensible > overview. And I want to implement a couple more optimizations and > benchmark it better. > > BTW, I've some updates (html5lib benchmark for py3, new benchmarks > for calling C methods, and I want to port some PyPy benchmakrs) > to the benchmarks suite. Should I just commit them, or should I > use bugs.python.org? > I actually emailed speed@ to see if people were interested in finally sitting down with all the various VM implementations at PyCon and trying to come up with a reasonable base set of benchmarks that better reflect modern Python usage, but I never heard back. Anyway, issues on bugs.python.org are probably best to talk about new benchmarks before adding them (fixes and updates to pre-existing benchmarks can just go in). > > > > > I rely on your dict->ma_version to implement cache invalidation. > > > > However, besides guarding against version change, I also want > > to guard against the dict being swapped for another dict, to > > avoid situations like this: > > > > > > def foo(): > > print(bar) > > > > exec(foo.__code__, {'bar': 1}, {}) > > exec(foo.__code__, {'bar': 2}, {}) > > > > > > What I propose is to add a pointer "ma_extra" (same 64bits), > > which will be set to NULL for most dict instances (instead of > > ma_version). "ma_extra" can then point to a struct that has a > > globally unique dict ID (uint64), and a version tag (unit64). > > A macro like PyDict_GET_ID and PyDict_GET_VERSION could then > > efficiently fetch the version/unique ID of the dict for guards. > > > > "ma_extra" would also make it easier for us to extend dicts > > in the future. > > > > > > Why can't you simply use the id of the dict object as the globally > > unique dict ID? It's already globally unique amongst all Python > > objects which makes it inherently unique amongst dicts. > > We have a freelist for dicts -- so if the dict dies, there > could be a new dict in its place, with the same ma_version. > Ah, I figured it would be too simple to use something we already had. > > While the probability of such hiccups is low, we still have > to account for it. > Yep. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Jan 20 15:27:12 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 20 Jan 2016 15:27:12 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FE3E6.303@g.nevcal.com> References: <569FCBD0.4040307@gmail.com> <569FE3E6.303@g.nevcal.com> Message-ID: <569FEDA0.1040506@gmail.com> On 2016-01-20 2:45 PM, Glenn Linderman wrote: > For the reuse case, can't you simply keep the ma_version "live" in > dict items on the free list, rather than starting over at (presumably) > 0 ? Then if the dict is reused, it bumps the ma_version, and the > fallback code goes on with (presumably) relocating the original dict > (oops, it's gone), and dealing with the fallout. Not all dicts are created from a freelist, and not all dicts go to the freelist when they are GCed. You still can have this situation: - dict "A" is used as f_locals for a frame, its ma_version is set to X - dict "A" is GCed, but the freelist is full, so it's just freed - after a while, you call the code object, a new dict "B" is allocated with malloc (since now the freelist happens to be empty) for f_locals - it happens that "B" is allocated where "A" was, and its ma_version happens to be X by an accident > > Then you can use the regular dict id as the globally unique dict id? > And only need the one uint64, rather than a separately allocated extra > pair of uint64? In my design only namespace dicts will have a non-NULL ma_extra, which means that only a fraction of dicts will actually have a separated pair of uint64s. I think that we should either use one global ma_version (as Maciej suggested) or ma_extra, as it gives more flexibility for us to extend dicts in the future. A global (shared between all dicts) unit64 ma_version is actually quite reliable -- if a program does 1,000,000 dict modifications per second, it would take it 600,000 years till wrap-around. Yury From brett at python.org Wed Jan 20 15:50:41 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 20:50:41 +0000 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FEDA0.1040506@gmail.com> References: <569FCBD0.4040307@gmail.com> <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com> Message-ID: On Wed, 20 Jan 2016 at 12:27 Yury Selivanov wrote: > > > On 2016-01-20 2:45 PM, Glenn Linderman wrote: > > For the reuse case, can't you simply keep the ma_version "live" in > > dict items on the free list, rather than starting over at (presumably) > > 0 ? Then if the dict is reused, it bumps the ma_version, and the > > fallback code goes on with (presumably) relocating the original dict > > (oops, it's gone), and dealing with the fallout. > > Not all dicts are created from a freelist, and not all dicts go to the > freelist when they are GCed. > > You still can have this situation: > > - dict "A" is used as f_locals for a frame, its ma_version is set to X > - dict "A" is GCed, but the freelist is full, so it's just freed > - after a while, you call the code object, a new dict "B" is allocated > with malloc (since now the freelist happens to be empty) for f_locals > - it happens that "B" is allocated where "A" was, and its ma_version > happens to be X by an accident > > > > > Then you can use the regular dict id as the globally unique dict id? > > And only need the one uint64, rather than a separately allocated extra > > pair of uint64? > > In my design only namespace dicts will have a non-NULL ma_extra, which > means that only a fraction of dicts will actually have a separated pair > of uint64s. > > I think that we should either use one global ma_version (as Maciej > suggested) or ma_extra, as it gives more flexibility for us to extend > dicts in the future. > > A global (shared between all dicts) unit64 ma_version is actually quite > reliable -- if a program does 1,000,000 dict modifications per second, > it would take it 600,000 years till wrap-around. > Since you're going to need to hold the GIL for the modifications there won't be any locking or contention problems, so it sounds like the global value is the best since that's simple, uses the least amount of memory, and will be easiest to use as a modification check since that will be a simple uint64 comparison instead of comparing a GUID + version. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hotpy.org Wed Jan 20 16:06:01 2016 From: mark at hotpy.org (Mark Shannon) Date: Wed, 20 Jan 2016 21:06:01 +0000 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: Message-ID: <569FF6B9.6000907@hotpy.org> On 11/01/16 16:49, Victor Stinner wrote: > Hi, > > After a first round on python-ideas, here is the second version of my > PEP. The main changes since the first version are that the dictionary > version is no more exposed at the Python level and the field type now > also has a size of 64-bit on 32-bit platforms. > > The PEP is part of a serie of 3 PEP adding an API to implement a > static Python optimizer specializing functions with guards. The second > PEP is currently discussed on python-ideas and I'm still working on > the third PEP. If anyone wants to experiment (at the C, not Python, level) with dict versioning to optimise load-global/builtins, then you can do so without adding a version number. A "version" can created by splitting the dict with "make_keys_shared" and then making the keys-object immutable by setting "dk_usable" to zero. This means that any change to the keys will force a keys-object change, but changes to the values will not. For many optimisations, this is want you want. Using this trick: To read a global, check that the keys is the expected keys and read the value straight out of the values array at the known index. To read a builtins, check that the module keys is the expected keys and thus cannot shadow the builtins, then read the builtins as above. I don't know how much help this will be for a static optimiser, but it could work well for a dynamic optimiser. I used this optimisation in HotPy for optimising object attribute lookups. Cheers, Mark. From victor.stinner at gmail.com Wed Jan 20 16:22:01 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 20 Jan 2016 22:22:01 +0100 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: I pushed my table, it will be online in a few hours (I don't know when the devguide is recompiled?): http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment By the way, it would be super cool to rebuild the PEPs with a post-commit hook server-side, rather than having to wait the crontab which requires to wait ~30 minutes (1h? I don't know exactly). 2016-01-20 21:20 GMT+01:00 Brett Cannon : > It is, which is why I requested the first release date be a column. I added a Scheduled column with a link to the Release Schedule PEP of each version. I also added a column with the date of the first date. I added 5 years to estimate the end-of-line. I used the same month and same date, with a comment above explaining that the release manager is free to adujst the end-of-line date. Thanks for the feedback. > PEP 101 would need a tweak to remind the RM to update the devguide. Can someone please mention this table in the PEP Victor From v+python at g.nevcal.com Wed Jan 20 16:18:25 2016 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 20 Jan 2016 13:18:25 -0800 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com> Message-ID: <569FF9A1.8010108@g.nevcal.com> On 1/20/2016 12:50 PM, Brett Cannon wrote: > > A global (shared between all dicts) unit64 ma_version is actually > quite > reliable -- if a program does 1,000,000 dict modifications per second, > it would take it 600,000 years till wrap-around. > But would invalidate everything, instead of just a fraction of things, on every update to anything that is monitored... -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Jan 20 17:01:03 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 21 Jan 2016 11:01:03 +1300 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FCBD0.4040307@gmail.com> References: <569FCBD0.4040307@gmail.com> Message-ID: <56A0039F.6070409@canterbury.ac.nz> Yury Selivanov wrote: > What I propose is to add a pointer "ma_extra" (same 64bits), > which will be set to NULL for most dict instances (instead of > ma_version). "ma_extra" can then point to a struct that has a > globally unique dict ID (uint64), and a version tag (unit64). Why not just use a single global counter for allocating dict version tags, instead of one per dict? -- Greg From brett at python.org Wed Jan 20 17:01:28 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 22:01:28 +0000 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: On Wed, 20 Jan 2016 at 13:22 Victor Stinner wrote: > I pushed my table, it will be online in a few hours (I don't know when > the devguide is recompiled?): > > http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment > > By the way, it would be super cool to rebuild the PEPs with a > post-commit hook server-side, rather than having to wait the crontab > which requires to wait ~30 minutes (1h? I don't know exactly). > This is a proposed optional, future feature leading from moving to GitHub: https://www.python.org/dev/peps/pep-0512/#web-hooks-for-re-generating-web-content -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From francismb at email.de Wed Jan 20 17:18:12 2016 From: francismb at email.de (francismb) Date: Wed, 20 Jan 2016 23:18:12 +0100 Subject: [Python-Dev] Code formatter bot In-Reply-To: <569EA3A6.4010802@email.de> References: <569EA3A6.4010802@email.de> Message-ID: <56A007A4.80705@email.de> Thanks again to all persons that commented so far. > what's your opinion about a code-formatter bot for cpython. > Pros, Cons, where could be applicable (new commits, new workflow, it > doesn't make sense), ... > > > - At least it should follow PEP 7 ;-) > - ... There seems to be too much implicit information on the lines above. Sorry for that. I'll try to clarify. >From my point of view (not a core-dev) the reviews seem to be one of the bottlenecks to the commit throughput, and I noticed on the other PEP 7 thread that time is taken to review such, IMHO, (and as Andrew also noted) trivialities. Thus the basic idea was to get that noise away from the reviews, somehow (better upfront, but why not after, accidental PEP 7 noise commits can happen). Well, people should first agree on that PEP and then some automatic could come. Please notice that the interaction is not just: core-dev committing to a repo or a bot committing to a repo but external-contributor that tries to minimize review iterations. I have no problem with some process (call it now a bot or script) that just changes a patch/file to reduce that cycle (but it could, run on all the workflow points you mentioned, plus on the PR-site as a kind of an advisor/mentor). Regards, francis From victor.stinner at gmail.com Wed Jan 20 17:28:00 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 20 Jan 2016 23:28:00 +0100 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: 2016-01-20 23:01 GMT+01:00 Brett Cannon : > This is a proposed optional, future feature leading from moving to GitHub: > https://www.python.org/dev/peps/pep-0512/#web-hooks-for-re-generating-web-content I'm using the free service ReadTheDocs.org and it's really impressive how fast it is to update the HTML page after a push. It's usually less than 10 seconds. Victor From guido at python.org Wed Jan 20 17:31:01 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 20 Jan 2016 14:31:01 -0800 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: The wording is totally fine! You might still want to revert it and re-commit it so it doesn't look like a mistake when reviewing the log. BTW When can we start using git for the peps repo? On Wed, Jan 20, 2016 at 12:18 PM, Brett Cannon wrote: > > > On Wed, 20 Jan 2016 at 10:45 Terry Reedy wrote: > >> On 1/20/2016 12:45 PM, Brett Cannon wrote: >> > >> > >> > On Tue, 19 Jan 2016 at 19:33 Martin Panter > > > wrote: >> > >> > On 19 January 2016 at 20:12, Brett Cannon > > > wrote: >> > > Here is a proposed update: >> > > >> > > diff -r 633f51d10a67 pep-0007.txt >> > > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800 >> > > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800 >> > > @@ -75,9 +75,9 @@ >> > > } >> > > >> > > * Code structure: one space between keywords like ``if``, >> > ``for`` and >> > > - the following left paren; no spaces inside the paren; braces >> > may be >> > > - omitted where C permits but when present, they should be >> formatted >> > > - as shown:: >> > > + the following left paren; no spaces inside the paren; braces >> are >> > > + strongly preferred but may be omitted where C permits, and >> they >> > > + should be formatted as shown:: >> > > >> > > if (mro != NULL) { >> > > ... >> > >> > This change seems to be accidentally smuggled in, in the guise of a >> > PEP 512 update :) >> > >> > >> > Darn, sorry about that; forgot I had that change sitting in my peps >> > checkout. I'll revert it when I get home (unless the change is actually >> > acceptable to Guido). >> >> I thought that the above was your intentional compromise change given >> the range of opinions ;-). >> > > It is, but Guido is the author of PEP 7 and so I didn't want to check in > the change without his approval first. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jan 20 17:41:16 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 22:41:16 +0000 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: On Wed, 20 Jan 2016 at 14:31 Guido van Rossum wrote: > The wording is totally fine! You might still want to revert it and > re-commit it so it doesn't look like a mistake when reviewing the log. > Sure thing! > > BTW When can we start using git for the peps repo? > Depends on how fast the parts covered in https://www.python.org/dev/peps/pep-0512/#requirements-for-code-only-repositories and https://www.python.org/dev/peps/pep-0512/#requirements-for-web-related-repositories takes. My hope is before PyCon US (although if we choose to not enforce the CLA on PEP contributions then it could happen even faster). -Brett > > On Wed, Jan 20, 2016 at 12:18 PM, Brett Cannon wrote: > >> >> >> On Wed, 20 Jan 2016 at 10:45 Terry Reedy wrote: >> >>> On 1/20/2016 12:45 PM, Brett Cannon wrote: >>> > >>> > >>> > On Tue, 19 Jan 2016 at 19:33 Martin Panter >> > > wrote: >>> > >>> > On 19 January 2016 at 20:12, Brett Cannon >> > > wrote: >>> > > Here is a proposed update: >>> > > >>> > > diff -r 633f51d10a67 pep-0007.txt >>> > > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800 >>> > > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800 >>> > > @@ -75,9 +75,9 @@ >>> > > } >>> > > >>> > > * Code structure: one space between keywords like ``if``, >>> > ``for`` and >>> > > - the following left paren; no spaces inside the paren; braces >>> > may be >>> > > - omitted where C permits but when present, they should be >>> formatted >>> > > - as shown:: >>> > > + the following left paren; no spaces inside the paren; braces >>> are >>> > > + strongly preferred but may be omitted where C permits, and >>> they >>> > > + should be formatted as shown:: >>> > > >>> > > if (mro != NULL) { >>> > > ... >>> > >>> > This change seems to be accidentally smuggled in, in the guise of a >>> > PEP 512 update :) >>> > >>> > >>> > Darn, sorry about that; forgot I had that change sitting in my peps >>> > checkout. I'll revert it when I get home (unless the change is actually >>> > acceptable to Guido). >>> >>> I thought that the above was your intentional compromise change given >>> the range of opinions ;-). >>> >> >> It is, but Guido is the author of PEP 7 and so I didn't want to check in >> the change without his approval first. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Jan 20 18:45:50 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 21 Jan 2016 00:45:50 +0100 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <569FF9A1.8010108@g.nevcal.com> References: <569FCBD0.4040307@gmail.com> <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com> <569FF9A1.8010108@g.nevcal.com> Message-ID: Hi, 2016-01-20 22:18 GMT+01:00 Glenn Linderman : > On 1/20/2016 12:50 PM, Brett Cannon wrote: >> >> A global (shared between all dicts) unit64 ma_version is actually quite >> reliable -- if a program does 1,000,000 dict modifications per second, >> it would take it 600,000 years till wrap-around. I think that Yury found a bug in FAT Python. I didn't test the case when the builtins dictionary is replaced after the definition of the function. To be more concrete: when a function is executed in a different namespace using exec(code, namespace). That's why I like the PEP process, it helps to find all issues before going too far :-) I like the idea of global counter for dictionary versions. It means that the dictionary constructor increases this counter instead of always starting to 0. FYI a fat.GuardDict keeps a strong reference to the dictionary. For some guards, I hesitated to store the object identifier and/or using a weak reference. An object identifier is not reliable because the object can be destroyed and a new object, completly different, or of the same type, can get the same identifier. > But would invalidate everything, instead of just a fraction of things, on > every update to anything that is monitored... I don't understand this point. In short, the guard only has to compare two 64 bit integers in the fast-path, when nothing changed. For a namespace, it means that no value was replaced in this namespace. If a different namespace is modified, the version of the watched namespace does not change, so we are still in the fast-path. If a value is replaced in the watched namespace, but not the watched variable, we have to take a slow-path, hopefully only once. The worst case is when a value different than the watched value is modified between each guard check. In this case, we always need a dict lookup. An heuristic can be chosen to decide to give up after N tries. Currently, fat.GuardDict always retries. Victor From brett at python.org Wed Jan 20 19:08:12 2016 From: brett at python.org (Brett Cannon) Date: Thu, 21 Jan 2016 00:08:12 +0000 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com> <569FF9A1.8010108@g.nevcal.com> Message-ID: On Wed, 20 Jan 2016 at 15:46 Victor Stinner wrote: > Hi, > > 2016-01-20 22:18 GMT+01:00 Glenn Linderman : > > On 1/20/2016 12:50 PM, Brett Cannon wrote: > >> > >> A global (shared between all dicts) unit64 ma_version is actually quite > >> reliable -- if a program does 1,000,000 dict modifications per second, > >> it would take it 600,000 years till wrap-around. > > I think that Yury found a bug in FAT Python. I didn't test the case > when the builtins dictionary is replaced after the definition of the > function. To be more concrete: when a function is executed in a > different namespace using exec(code, namespace). That's why I like the > PEP process, it helps to find all issues before going too far :-) > > I like the idea of global counter for dictionary versions. It means > that the dictionary constructor increases this counter instead of > always starting to 0. > > FYI a fat.GuardDict keeps a strong reference to the dictionary. For > some guards, I hesitated to store the object identifier and/or using a > weak reference. An object identifier is not reliable because the > object can be destroyed and a new object, completly different, or of > the same type, can get the same identifier. > > > But would invalidate everything, instead of just a fraction of things, on > > every update to anything that is monitored... > > I don't understand this point. > I think Glenn was assuming we had a single, global version # that all dicts shared *without* having a per-dict version ID. The key thing here is that we have a global counter that tracks the number of mutations for *all* dictionaries but whose value we store as a *per-dictionary* value. That ends up making the version ID inherently both a token representing the state of any dict but also the uniqueness of the dict since no two dictionaries will ever have the same version ID. > > In short, the guard only has to compare two 64 bit integers in the > fast-path, when nothing changed. For a namespace, it means that no > value was replaced in this namespace. > > If a different namespace is modified, the version of the watched > namespace does not change, so we are still in the fast-path. > > If a value is replaced in the watched namespace, but not the watched > variable, we have to take a slow-path, hopefully only once. > > The worst case is when a value different than the watched value is > modified between each guard check. In this case, we always need a dict > lookup. An heuristic can be chosen to decide to give up after N tries. > Currently, fat.GuardDict always retries. > Does "retries" mean "check if the value really changed, and if it hasn't then just update the version ID the guard checks"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jan 20 19:09:43 2016 From: brett at python.org (Brett Cannon) Date: Thu, 21 Jan 2016 00:09:43 +0000 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: On Wed, 20 Jan 2016 at 14:28 Victor Stinner wrote: > 2016-01-20 23:01 GMT+01:00 Brett Cannon : > > This is a proposed optional, future feature leading from moving to > GitHub: > > > https://www.python.org/dev/peps/pep-0512/#web-hooks-for-re-generating-web-content > > I'm using the free service ReadTheDocs.org and it's really impressive > how fast it is to update the HTML page after a push. It's usually less > than 10 seconds. > I have no idea if the way our docs are built would work on readthedocs.org, but if it could then I would definitely vote to move our docs there and have the PSF make a regular donation for the service. But this is a discussion to have on core-workflow@ and not here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Wed Jan 20 19:10:56 2016 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 20 Jan 2016 16:10:56 -0800 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com> <569FF9A1.8010108@g.nevcal.com> Message-ID: <56A02210.7030002@g.nevcal.com> On 1/20/2016 4:08 PM, Brett Cannon wrote: > > > On Wed, 20 Jan 2016 at 15:46 Victor Stinner > wrote: > > Hi, > > 2016-01-20 22:18 GMT+01:00 Glenn Linderman >: > > On 1/20/2016 12:50 PM, Brett Cannon wrote: > >> > >> A global (shared between all dicts) unit64 ma_version is > actually quite > >> reliable -- if a program does 1,000,000 dict modifications per > second, > >> it would take it 600,000 years till wrap-around. > > I think that Yury found a bug in FAT Python. I didn't test the case > when the builtins dictionary is replaced after the definition of the > function. To be more concrete: when a function is executed in a > different namespace using exec(code, namespace). That's why I like the > PEP process, it helps to find all issues before going too far :-) > > I like the idea of global counter for dictionary versions. It means > that the dictionary constructor increases this counter instead of > always starting to 0. > > FYI a fat.GuardDict keeps a strong reference to the dictionary. For > some guards, I hesitated to store the object identifier and/or using a > weak reference. An object identifier is not reliable because the > object can be destroyed and a new object, completly different, or of > the same type, can get the same identifier. > > > But would invalidate everything, instead of just a fraction of > things, on > > every update to anything that is monitored... > > I don't understand this point. > > > I think Glenn was assuming we had a single, global version # that all > dicts shared *without* having a per-dict version ID. The key thing > here is that we have a global counter that tracks the number of > mutations for *all* dictionaries but whose value we store as a > *per-dictionary* value. That ends up making the version ID inherently > both a token representing the state of any dict but also the > uniqueness of the dict since no two dictionaries will ever have the > same version ID. This would work. You were correct about my assumptions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jan 20 19:16:31 2016 From: brett at python.org (Brett Cannon) Date: Thu, 21 Jan 2016 00:16:31 +0000 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: I just checked with Van and we should have CLAs for even PEP contributors, so it will have to go through the same steps as the other ancillary repositories. On Wed, 20 Jan 2016 at 14:41 Brett Cannon wrote: > On Wed, 20 Jan 2016 at 14:31 Guido van Rossum wrote: > >> The wording is totally fine! You might still want to revert it and >> re-commit it so it doesn't look like a mistake when reviewing the log. >> > > Sure thing! > > >> >> BTW When can we start using git for the peps repo? >> > > Depends on how fast the parts covered in > https://www.python.org/dev/peps/pep-0512/#requirements-for-code-only-repositories > and > https://www.python.org/dev/peps/pep-0512/#requirements-for-web-related-repositories takes. > My hope is before PyCon US (although if we choose to not enforce the CLA on > PEP contributions then it could happen even faster). > > -Brett > > >> >> On Wed, Jan 20, 2016 at 12:18 PM, Brett Cannon wrote: >> >>> >>> >>> On Wed, 20 Jan 2016 at 10:45 Terry Reedy wrote: >>> >>>> On 1/20/2016 12:45 PM, Brett Cannon wrote: >>>> > >>>> > >>>> > On Tue, 19 Jan 2016 at 19:33 Martin Panter >>> > > wrote: >>>> > >>>> > On 19 January 2016 at 20:12, Brett Cannon >>> > > wrote: >>>> > > Here is a proposed update: >>>> > > >>>> > > diff -r 633f51d10a67 pep-0007.txt >>>> > > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800 >>>> > > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800 >>>> > > @@ -75,9 +75,9 @@ >>>> > > } >>>> > > >>>> > > * Code structure: one space between keywords like ``if``, >>>> > ``for`` and >>>> > > - the following left paren; no spaces inside the paren; braces >>>> > may be >>>> > > - omitted where C permits but when present, they should be >>>> formatted >>>> > > - as shown:: >>>> > > + the following left paren; no spaces inside the paren; >>>> braces are >>>> > > + strongly preferred but may be omitted where C permits, and >>>> they >>>> > > + should be formatted as shown:: >>>> > > >>>> > > if (mro != NULL) { >>>> > > ... >>>> > >>>> > This change seems to be accidentally smuggled in, in the guise of >>>> a >>>> > PEP 512 update :) >>>> > >>>> > >>>> > Darn, sorry about that; forgot I had that change sitting in my peps >>>> > checkout. I'll revert it when I get home (unless the change is >>>> actually >>>> > acceptable to Guido). >>>> >>>> I thought that the above was your intentional compromise change given >>>> the range of opinions ;-). >>>> >>> >>> It is, but Guido is the author of PEP 7 and so I didn't want to check in >>> the change without his approval first. >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> >> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>> >>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Jan 20 19:23:44 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 21 Jan 2016 01:23:44 +0100 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com> <569FF9A1.8010108@g.nevcal.com> Message-ID: 2016-01-21 1:08 GMT+01:00 Brett Cannon : > On Wed, 20 Jan 2016 at 15:46 Victor Stinner >> The worst case is when a value different than the watched value is >> modified between each guard check. In this case, we always need a dict >> lookup. An heuristic can be chosen to decide to give up after N tries. >> Currently, fat.GuardDict always retries. > > Does "retries" mean "check if the value really changed, and if it hasn't > then just update the version ID the guard checks"? If the dict version changes (because a value different than the watched value is modified) each time that the guard is checked, the guard always require a dict lookup to check if the watched value changed. Victor From victor.stinner at gmail.com Wed Jan 20 19:33:03 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 21 Jan 2016 01:33:03 +0100 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: 2016-01-21 1:09 GMT+01:00 Brett Cannon : > On Wed, 20 Jan 2016 at 14:28 Victor Stinner >> I'm using the free service ReadTheDocs.org and it's really impressive >> how fast it is to update the HTML page after a push. It's usually less >> than 10 seconds. > > I have no idea if the way our docs are built would work on readthedocs.org, > but if it could then I would definitely vote to move our docs there and have > the PSF make a regular donation for the service. Oh, I was talking about small documentations of personal projects. I didn't propose to move Python docs to readthedocs.org. I don't know if it makes sense. It's just to say that we can do better than 30 minutes of the current system :-) Victor From abarnert at yahoo.com Wed Jan 20 20:54:41 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 21 Jan 2016 01:54:41 +0000 (UTC) Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: Message-ID: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com> On Wednesday, January 20, 2016 4:10 PM, Brett Cannon wrote: >I think Glenn was assuming we had a single, global version # that all dicts shared without having a per-dict version ID. The key thing here is that we have a global counter that tracks the number of mutations for all dictionaries but whose value we store as a per-dictionary value. That ends up making the version ID inherently both a token representing the state of any dict but also the uniqueness of the dict since no two dictionaries will ever have the same version ID. This idea worries me. I'm not sure why, but I think because of threading. After all, it's pretty rare for two threads to both want to work on the same dict, but very, very common for two threads to both want to work on _any_ dict. So, imagine someone manages to remove the GIL from CPython by using STM: now most transactions are bumping that global counter, meaning most transactions fail and have to be retried, so you end up with 8 cores each running at 1/64th the speed of a single core but burning 100% CPU. Obviously a real-life implementation wouldn't be _that_ stupid; you'd special-case the version-bumping (maybe unconditionally bump it N times before starting the transaction, and then as long as you don't bump more than N times during the transaction, you can commit without touching it), but there's still going to be a lot of contention. And that also affects something like PyPy being able to use FAT-Python-style AoT optimizations via cpyext. At first glance that sounds like a stupid idea--why would you want to run an optimizer through a slow emulator? But the optimizer only runs once and transforms the function code, which runs a zillion times, so who cares how slow the optimizer is? Of course it may still be true that many of the AoT optimizations that FAT makes don't apply very well to PyPy, in which case it doesn't matter. But I don't think we can assume that a priori. Is there a way to define this loosely enough so that the implementation _can_ be a single global counter, if that turns out to be most efficient, but can also be a counter per dictionary and a globally-unique ID per dictionary? From brett at python.org Wed Jan 20 21:04:58 2016 From: brett at python.org (Brett Cannon) Date: Thu, 21 Jan 2016 02:04:58 +0000 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com> References: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Wed, 20 Jan 2016, 17:54 Andrew Barnert wrote: > On Wednesday, January 20, 2016 4:10 PM, Brett Cannon > wrote: > > > >I think Glenn was assuming we had a single, global version # that all > dicts shared without having a per-dict version ID. The key thing here is > that we have a global counter that tracks the number of mutations for all > dictionaries but whose value we store as a per-dictionary value. That ends > up making the version ID inherently both a token representing the state of > any dict but also the uniqueness of the dict since no two dictionaries will > ever have the same version ID. > > This idea worries me. I'm not sure why, but I think because of threading. > After all, it's pretty rare for two threads to both want to work on the > same dict, but very, very common for two threads to both want to work on > _any_ dict. So, imagine someone manages to remove the GIL from CPython by > using STM: now most transactions are bumping that global counter, meaning > most transactions fail and have to be retried, so you end up with 8 cores > each running at 1/64th the speed of a single core but burning 100% CPU. > Obviously a real-life implementation wouldn't be _that_ stupid; you'd > special-case the version-bumping (maybe unconditionally bump it N times > before starting the transaction, and then as long as you don't bump more > than N times during the transaction, you can commit without touching it), > but there's still going to be a lot of contention. > This is all being regarded as an implementation detail of CPython, so in this hypothetical STM world we can drop all of this (or lock it). > And that also affects something like PyPy being able to use > FAT-Python-style AoT optimizations via cpyext. At first glance that sounds > like a stupid idea--why would you want to run an optimizer through a slow > emulator? But the optimizer only runs once and transforms the function > code, which runs a zillion times, so who cares how slow the optimizer is? > Of course it may still be true that many of the AoT optimizations that FAT > makes don't apply very well to PyPy, in which case it doesn't matter. But I > don't think we can assume that a priori. > > Is there a way to define this loosely enough so that the implementation > _can_ be a single global counter, if that turns out to be most efficient, > but can also be a counter per dictionary and a globally-unique ID per > dictionary? > There's no need to if this is all under the hood and in no way affects anyone but the eval loop and those who choose to use it. We can make sure to preface all of this with underscores so it's obvious they are private and so use at your own peril. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Jan 20 21:13:38 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 20 Jan 2016 21:13:38 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com> References: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56A03ED2.3010204@gmail.com> On 2016-01-20 8:54 PM, Andrew Barnert via Python-Dev wrote: >> >I think Glenn was assuming we had a single, global version # that all dicts shared without having a per-dict version ID. The key thing here is that we have a global counter that tracks the number of mutations for all dictionaries but whose value we store as a per-dictionary value. That ends up making the version ID inherently both a token representing the state of any dict but also the uniqueness of the dict since no two dictionaries will ever have the same version ID. > This idea worries me. I'm not sure why, but I think because of threading. After all, it's pretty rare for two threads to both want to work on the same dict, but very, very common for two threads to both want to work on_any_ dict. So, imagine someone manages to remove the GIL from CPython by using STM: now most transactions are bumping that global counter, meaning most transactions fail and have to be retried, so you end up with 8 cores each running at 1/64th the speed of a single core but burning 100% CPU. Obviously a real-life implementation wouldn't be_that_ stupid; you'd special-case the version-bumping (maybe unconditionally bump it N times before starting the transaction, and then as long as you don't bump more than N times during the transaction, you can commit without touching it), but there's still going to be a lot of contention. Well, PEP 509 proposes to add ma_version only for CPython. It's an implementation detail of CPython. Victor's FAT optimizer is also tailored specifically for CPython, and making it work on PyPy would require a completely different set of hacks. To remove the GIL or implement an efficient STM one will have to rewrite (and potentially break) so much code in CPython, that ma_version won't be a concern. For now, though, ma_version will be protected by GIL, so threading shouldn't be a problem. > > And that also affects something like PyPy being able to use FAT-Python-style AoT optimizations via cpyext. At first glance that sounds like a stupid idea--why would you want to run an optimizer through a slow emulator? But the optimizer only runs once and transforms the function code, which runs a zillion times, so who cares how slow the optimizer is? Of course it may still be true that many of the AoT optimizations that FAT makes don't apply very well to PyPy, in which case it doesn't matter. But I don't think we can assume that a priori. The idea of FAT is that it will also generate optimized code objects with guards. I doubt it would make any sense to use it under PyPy or any jitted Python implementation. JITs have a far better understanding of the code than any static optimizer. > > Is there a way to define this loosely enough so that the implementation_can_ be a single global counter, if that turns out to be most efficient, but can also be a counter per dictionary and a globally-unique ID per dictionary? Defining it "loosely" means that you can't trust it. I'd just explicitly say that: - ma_version is an implementation detail of CPython and may not be implemented on other platforms; - ma_version can be removed from future CPython releases; - ma_version can be used by code optimizers tailored specifically for CPython and CPython itself. Yury From yselivanov.ml at gmail.com Thu Jan 21 00:08:57 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 21 Jan 2016 00:08:57 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <56A0039F.6070409@canterbury.ac.nz> References: <569FCBD0.4040307@gmail.com> <56A0039F.6070409@canterbury.ac.nz> Message-ID: <56A067E9.8090409@gmail.com> On 2016-01-20 5:01 PM, Greg Ewing wrote: > Yury Selivanov wrote: >> What I propose is to add a pointer "ma_extra" (same 64bits), >> which will be set to NULL for most dict instances (instead of >> ma_version). "ma_extra" can then point to a struct that has a >> globally unique dict ID (uint64), and a version tag (unit64). > > Why not just use a single global counter for allocating > dict version tags, instead of one per dict? > Yeah, I think that's what we agreed on: https://mail.python.org/pipermail/python-dev/2016-January/142837.html The only advantage of ma_extra pointer is that it allows to add more stuff to dicts in the future. Yury From ncoghlan at gmail.com Thu Jan 21 02:07:43 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 21 Jan 2016 17:07:43 +1000 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: On 21 January 2016 at 10:16, Brett Cannon wrote: > I just checked with Van and we should have CLAs for even PEP contributors, > so it will have to go through the same steps as the other ancillary > repositories. We should probably mention that in PEP 1 - I wasn't aware of that requirement, so I've never explicitly checked CLA status for folks contributing packaging related PEPs. (And looking at the just-checked-in PEP 513, I apparently have a CLA to chase up...) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Thu Jan 21 02:46:21 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 21 Jan 2016 08:46:21 +0100 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: 2016-01-20 22:22 GMT+01:00 Victor Stinner : > I pushed my table, it will be online in a few hours (I don't know when > the devguide is recompiled?): > http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment Hum ok, it takes more than a few hours in fact. It's still not online 10 hours after my push :-/ https://docs.python.org/devguide/ Victor From greg.ewing at canterbury.ac.nz Thu Jan 21 00:20:42 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 21 Jan 2016 18:20:42 +1300 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com> References: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56A06AAA.7020507@canterbury.ac.nz> Andrew Barnert via Python-Dev wrote: > imagine someone manages to remove the GIL from CPython by using > STM: now most transactions are bumping that global counter, meaning most > transactions fail and have to be retried, If this becomes a problem, the tag could be split into two parts of m and n bits, with m + n = 64. Use a global counter for allocating the high half, and increment the low half locally. When the low half overflows, allocate a new high half. A value of n = 16 or so ought to reduce contention for the global counter to something fairly negligible, I would think, without much risk of the high half ever wrapping around. -- Greg From victor.stinner at gmail.com Thu Jan 21 03:28:25 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 21 Jan 2016 09:28:25 +0100 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com> References: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com> Message-ID: 2016-01-21 2:54 GMT+01:00 Andrew Barnert : > This idea worries me. I'm not sure why, but I think because of threading. After all, it's pretty rare for two threads to both want to work on the same dict, but very, very common for two threads to both want to work on _any_ dict. So, imagine someone manages to remove the GIL from CPython by using STM: (...) That's a huge project :-) PyPy works one this, but PyPy is not CPython. > (...) now most transactions are bumping that global counter, meaning most transactions fail and have to be retried, Python has atomic types which work well with multiple concurrent threads. > And that also affects something like PyPy being able to use FAT-Python-style AoT optimizations via cpyext. I don't think that using a static optimizer with PyPy makes sense. Reminder that a dynamic optimize (JIT compiler) beats a static compiler on performance ;-) PyPy has a very different design, it's a very bad idea to implement optimizations on cpyext which is emulated and known to be slow. > Is there a way to define this loosely enough so that the implementation _can_ be a single global counter, if that turns out to be most efficient, but can also be a counter per dictionary and a globally-unique ID per dictionary? I don't see how a single global counter for all dictionary can be used to implement fast guards on namespaces. See the rationale of the PEP 509: I wrote it to implement fast guards on namespaces. Victor From victor.stinner at gmail.com Thu Jan 21 03:34:07 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 21 Jan 2016 09:34:07 +0100 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: <56A067E9.8090409@gmail.com> References: <569FCBD0.4040307@gmail.com> <56A0039F.6070409@canterbury.ac.nz> <56A067E9.8090409@gmail.com> Message-ID: 2016-01-21 6:08 GMT+01:00 Yury Selivanov : > Yeah, I think that's what we agreed on: > https://mail.python.org/pipermail/python-dev/2016-January/142837.html > > The only advantage of ma_extra pointer is that it allows to add more stuff > to dicts in the future. I don't agree on ma_extra since I don't understand it :-) What is the advantage compared to a simple integer? If it's a pointer, it requires to dereference the pointer? You say that it can be NULL, does it mean that we also have to first test if the pointer is NULL. Does it mean that we have to allocate a second memory block to store a version tag object? When do you create a version tag or not? Note: The PEP 509 proposes to use 64-bit integer for the version on 32-bit systems to avoid integer overflow after a few seconds. I first proposed to use the size_t type (which has the size of a pointer) but it doesn't work. I tried to fix FAT Python to handle your use case: function defined in a namespace and run in a different namespace (especially for the builtin dictionary). It looks like I don't need the discussion change to use a global version, FAT Python guards have a different design than your cache. But if a global counter doesn't make the slow more complex and opens new kinds of optimization, I agree to change my PEP 509. Victor From yselivanov.ml at gmail.com Thu Jan 21 11:12:16 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 21 Jan 2016 11:12:16 -0500 Subject: [Python-Dev] PEP 509: Add a private version to dict In-Reply-To: References: <569FCBD0.4040307@gmail.com> <56A0039F.6070409@canterbury.ac.nz> <56A067E9.8090409@gmail.com> Message-ID: <56A10360.1060604@gmail.com> On 2016-01-21 3:34 AM, Victor Stinner wrote: [..] > But if a global counter doesn't make the slow more complex and opens > new kinds of optimization, I agree to change my PEP 509. Please. That would allow us to use ma_version to implement caches in CPython itself. Yury From brett at python.org Thu Jan 21 12:12:10 2016 From: brett at python.org (Brett Cannon) Date: Thu, 21 Jan 2016 17:12:10 +0000 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: On Wed, 20 Jan 2016 at 23:07 Nick Coghlan wrote: > On 21 January 2016 at 10:16, Brett Cannon wrote: > > I just checked with Van and we should have CLAs for even PEP > contributors, > > so it will have to go through the same steps as the other ancillary > > repositories. > > We should probably mention that in PEP 1 Yep. > - I wasn't aware of that > requirement, so I've never explicitly checked CLA status for folks > contributing packaging related PEPs. (And looking at the > just-checked-in PEP 513, I apparently have a CLA to chase up...) > Yeah, I didn't know either until I asked Van, so I think it's new to everyone. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Jan 21 12:18:13 2016 From: brett at python.org (Brett Cannon) Date: Thu, 21 Jan 2016 17:18:13 +0000 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: It's live: https://docs.python.org/devguide/#status-of-python-branches On Wed, 20 Jan 2016 at 23:47 Victor Stinner wrote: > 2016-01-20 22:22 GMT+01:00 Victor Stinner : > > I pushed my table, it will be online in a few hours (I don't know when > > the devguide is recompiled?): > > > http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment > > Hum ok, it takes more than a few hours in fact. It's still not online > 10 hours after my push :-/ > https://docs.python.org/devguide/ > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Thu Jan 21 12:18:13 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Thu, 21 Jan 2016 11:18:13 -0600 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: On Thu, Jan 21, 2016 at 11:12 AM, Brett Cannon wrote: > On Wed, 20 Jan 2016 at 23:07 Nick Coghlan wrote: >> - I wasn't aware of that >> requirement, so I've never explicitly checked CLA status for folks >> contributing packaging related PEPs. (And looking at the >> just-checked-in PEP 513, I apparently have a CLA to chase up...) > > Yeah, I didn't know either until I asked Van, so I think it's new to > everyone. :) It's quite surprising to me, since all (as far as I know) PEPs are explicitly public domain. I am very much not a lawyer, though :) -- Zach From guido at python.org Thu Jan 21 12:40:34 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 21 Jan 2016 09:40:34 -0800 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: Thanks Victor for doing this. I'm starting a campaign to tell people about it on Twitter. :-) On Thu, Jan 21, 2016 at 9:18 AM, Brett Cannon wrote: > It's live: https://docs.python.org/devguide/#status-of-python-branches > > On Wed, 20 Jan 2016 at 23:47 Victor Stinner > wrote: > >> 2016-01-20 22:22 GMT+01:00 Victor Stinner : >> > I pushed my table, it will be online in a few hours (I don't know when >> > the devguide is recompiled?): >> > >> http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment >> >> Hum ok, it takes more than a few hours in fact. It's still not online >> 10 hours after my push :-/ >> https://docs.python.org/devguide/ >> >> Victor >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 21 13:42:40 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 21 Jan 2016 18:42:40 +0000 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: On 21 January 2016 at 17:18, Brett Cannon wrote: > It's live: https://docs.python.org/devguide/#status-of-python-branches Nice :-) Minor nit, the status column says "end of life", but the text below the table uses the term "end of line" (as does the comment "Versions older than 2.6 reached their end-of-line". From my experience, "end of life" is the more common term. Paul From steve.dower at python.org Thu Jan 21 14:50:37 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 21 Jan 2016 11:50:37 -0800 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: I'm still yet to meet a lawyer who trusts "public domain" statements... The CLA will ensure we have enough rights to republish the PEP on p.o or future sites, and it doesn't prevent authors from also releasing the text elsewhere under other terms. Top-posted from my Windows Phone -----Original Message----- From: "Zachary Ware" Sent: ?1/?21/?2016 9:23 To: "Python-Dev" Subject: Re: [Python-Dev] Update PEP 7 to require curly braces in C On Thu, Jan 21, 2016 at 11:12 AM, Brett Cannon wrote: > On Wed, 20 Jan 2016 at 23:07 Nick Coghlan wrote: >> - I wasn't aware of that >> requirement, so I've never explicitly checked CLA status for folks >> contributing packaging related PEPs. (And looking at the >> just-checked-in PEP 513, I apparently have a CLA to chase up...) > > Yeah, I didn't know either until I asked Van, so I think it's new to > everyone. :) It's quite surprising to me, since all (as far as I know) PEPs are explicitly public domain. I am very much not a lawyer, though :) -- Zach _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From emile at fenx.com Thu Jan 21 16:27:44 2016 From: emile at fenx.com (Emile van Sebille) Date: Thu, 21 Jan 2016 13:27:44 -0800 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: On 1/21/2016 10:42 AM, Paul Moore wrote: > On 21 January 2016 at 17:18, Brett Cannon wrote: >> It's live: https://docs.python.org/devguide/#status-of-python-branches > > Nice :-) > > Minor nit, the status column says "end of life", but the text below > the table uses the term "end of line" (as does the comment "Versions > older than 2.6 reached their end-of-line". From my experience, "end of > life" is the more common term. I'd prefer end-of-support -- bet you can't count how many pre 2.5 installations are still live. Emile From stephen at xemacs.org Fri Jan 22 00:25:38 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 22 Jan 2016 14:25:38 +0900 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: <22177.48466.360573.863828@turnbull.sk.tsukuba.ac.jp> Emile van Sebille writes: > I'd prefer end-of-support -- bet you can't count how many pre 2.5 > installations are still live. I see your point, but (having just been thinking about CLAs and Schneier's blog) have to suggest that software that has an explicit "security support" period and is in use after that ends isn't "live". It is "undead". Hope-the-admin-is-named-Alice-ly y'rs, From stephen at xemacs.org Fri Jan 22 00:32:40 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 22 Jan 2016 14:32:40 +0900 Subject: [Python-Dev] Update PEP 7 to require curly braces in C In-Reply-To: References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com> <569E8073.3010106@stoneleaf.us> Message-ID: <22177.48888.899524.150195@turnbull.sk.tsukuba.ac.jp> Zachary Ware writes: > It's quite surprising to me, since all (as far as I know) PEPs are > explicitly public domain. I am very much not a lawyer, though :) The reason for any explicit CLA is CYA: you can't be sure that the contributor DTRTs, so the CLA shows that the contribution was received in good faith. (Some CLAs also include indemnification clauses, such as the contributor warrants that they own the copyright, etc, but all CLAs are useful to show good faith, which may matter big time if the contributor turns out to have a all-IP-yours-is-ours employment contract or the like.) You don't really need to be a lawyer to understand this stuff, you just need to think like a security person. https://www.schneier.com/blog/archives/2008/03/the_security_mi_1.html From ncoghlan at gmail.com Fri Jan 22 02:54:05 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 22 Jan 2016 17:54:05 +1000 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: On 22 January 2016 at 07:27, Emile van Sebille wrote: > On 1/21/2016 10:42 AM, Paul Moore wrote: >> >> On 21 January 2016 at 17:18, Brett Cannon wrote: >>> >>> It's live: https://docs.python.org/devguide/#status-of-python-branches >> >> Nice :-) >> >> Minor nit, the status column says "end of life", but the text below >> the table uses the term "end of line" (as does the comment "Versions >> older than 2.6 reached their end-of-line". From my experience, "end of >> life" is the more common term. > > I'd prefer end-of-support -- bet you can't count how many pre 2.5 > installations are still live. I can count the number of folks contributing changes to the upstream Python 2.5 branch: zero. Even if somebody offered a patch for it, we wouldn't accept it - that maintenance branch is dead, which is what the "End of Life" refers to. Folks are still free to run it (all past Python releases remain online, all the way back to 1.1), and downstreams may still offer support for it, but that's their call. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Fri Jan 22 03:26:59 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 22 Jan 2016 09:26:59 +0100 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: 2016-01-21 19:42 GMT+01:00 Paul Moore : > Minor nit, the status column says "end of life", but the text below > the table uses the term "end of line" Ooops, it's a funny typo. Fixed. Thanks for the report! Victor From victor.stinner at gmail.com Fri Jan 22 11:03:39 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 22 Jan 2016 17:03:39 +0100 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: 2016-01-21 18:18 GMT+01:00 Brett Cannon : > It's live: https://docs.python.org/devguide/#status-of-python-branches There is a very strange bug in this website. This URL shows the table: https://docs.python.org/devguide/ This URL doesn't show the table: https://docs.python.org/devguide/index.html Outdated version of the guide? This bug can be seen without a browser, using wget: $ wget -O- https://docs.python.org/devguide/ 2>&1|grep 'Python branches' Status of Python branches<(...)
Status of Python branches $ wget -O- https://docs.python.org/devguide/index.html 2>&1|grep 'Python branches' Victor From berker.peksag at gmail.com Fri Jan 22 11:21:48 2016 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Fri, 22 Jan 2016 18:21:48 +0200 Subject: [Python-Dev] Devguide: Add a table summarizing status of Python branches In-Reply-To: References: Message-ID: On Fri, Jan 22, 2016 at 6:03 PM, Victor Stinner wrote: > 2016-01-21 18:18 GMT+01:00 Brett Cannon : >> It's live: https://docs.python.org/devguide/#status-of-python-branches > > There is a very strange bug in this website. > > This URL shows the table: > https://docs.python.org/devguide/ > > This URL doesn't show the table: > https://docs.python.org/devguide/index.html > > Outdated version of the guide? It looks like a cache issue. I purged the cache for /devguide/index.html: $ wget -O- https://docs.python.org/devguide/index.html 2>&1 | grep "Python branches" Status of Python branches?
Status of Python branches --Berker From status at bugs.python.org Fri Jan 22 12:08:33 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 22 Jan 2016 18:08:33 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20160122170833.EF922560CE@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-01-15 - 2016-01-22) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5354 (-33) closed 32585 (+89) total 37939 (+56) Open issues with patches: 2348 Issues opened (31) ================== #25791: Raise an ImportWarning when __spec__.parent/__package__ isn't http://bugs.python.org/issue25791 reopened by brett.cannon #26128: Let the subprocess.STARTUPINFO constructor take arguments http://bugs.python.org/issue26128 opened by cool-RR #26130: redundant local copy of a char pointer in classify in Parser\p http://bugs.python.org/issue26130 opened by Oren Milman #26131: Raise ImportWarning when loader.load_module() is used http://bugs.python.org/issue26131 opened by brett.cannon #26132: 2.7.11 Windows Installer issues on Win2008R2 http://bugs.python.org/issue26132 opened by David Rader #26133: asyncio: ugly error related to signal handlers at exit if the http://bugs.python.org/issue26133 opened by Alex Brandt #26134: HTTPPasswordMgrWithPriorAuth does not work with DigestAuthenti http://bugs.python.org/issue26134 opened by guesommer #26136: DeprecationWarning for PEP 479 (generator_stop) http://bugs.python.org/issue26136 opened by martin.panter #26137: [idea] use the Microsoft Antimalware Scan Interface http://bugs.python.org/issue26137 opened by Alexander Riccio #26140: inspect.iscoroutinefunction raises TypeError when checks Mock http://bugs.python.org/issue26140 opened by miyakogi #26141: typing module documentation incomplete http://bugs.python.org/issue26141 opened by Ben.Darnell #26143: Ensure that IDLE's stdlib imports are from the stdlib http://bugs.python.org/issue26143 opened by terry.reedy #26144: test_pkg test_4 and/or test_7 sometimes fail http://bugs.python.org/issue26144 opened by martin.panter #26145: PEP 511: Add sys.set_code_transformers() http://bugs.python.org/issue26145 opened by haypo #26146: PEP 511: Add ast.Constant to allow AST optimizer to emit const http://bugs.python.org/issue26146 opened by haypo #26148: String literals are not interned if in a tuple http://bugs.python.org/issue26148 opened by serhiy.storchaka #26149: Suggest PyCharm Community as an editor for Unix platforms http://bugs.python.org/issue26149 opened by John Hagen #26152: A non-breaking space in a source http://bugs.python.org/issue26152 opened by Drekin #26153: PyImport_GetModuleDict: no module dictionary! when `__del__` t http://bugs.python.org/issue26153 opened by minrk #26155: 3.5.1 installer issue on Win 7 32 bit http://bugs.python.org/issue26155 opened by TarotRedhand #26158: File truncate() not defaulting to current position as document http://bugs.python.org/issue26158 opened by fornax #26159: Unsafe to BaseEventLoop.set_debug(False) when PYTHONASYNCIODEB http://bugs.python.org/issue26159 opened by Bradley McLean #26160: Tutorial incorrectly claims that (explicit) relative imports d http://bugs.python.org/issue26160 opened by Kevin.Norris #26167: Improve copy.copy speed for built-in types (list/set/dict) http://bugs.python.org/issue26167 opened by josh.r #26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure http://bugs.python.org/issue26168 opened by squidevil #26173: test_ssl.bad_cert_test() exception handling http://bugs.python.org/issue26173 opened by martin.panter #26175: Fully implement IOBase abstract on SpooledTemporaryFile http://bugs.python.org/issue26175 opened by Gary Fernie #26176: EmailMessage example doesn't work http://bugs.python.org/issue26176 opened by Srujan Chaitanya #26177: tkinter: Canvas().keys returns empty strings. http://bugs.python.org/issue26177 opened by terry.reedy #26180: multiprocessing.util._afterfork_registry leak in threaded envi http://bugs.python.org/issue26180 opened by mzamazal #26181: argparse can't handle positional argument after list (help mes http://bugs.python.org/issue26181 opened by atpage Most recent 15 issues with no replies (15) ========================================== #26181: argparse can't handle positional argument after list (help mes http://bugs.python.org/issue26181 #26180: multiprocessing.util._afterfork_registry leak in threaded envi http://bugs.python.org/issue26180 #26176: EmailMessage example doesn't work http://bugs.python.org/issue26176 #26173: test_ssl.bad_cert_test() exception handling http://bugs.python.org/issue26173 #26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure http://bugs.python.org/issue26168 #26159: Unsafe to BaseEventLoop.set_debug(False) when PYTHONASYNCIODEB http://bugs.python.org/issue26159 #26153: PyImport_GetModuleDict: no module dictionary! when `__del__` t http://bugs.python.org/issue26153 #26141: typing module documentation incomplete http://bugs.python.org/issue26141 #26136: DeprecationWarning for PEP 479 (generator_stop) http://bugs.python.org/issue26136 #26131: Raise ImportWarning when loader.load_module() is used http://bugs.python.org/issue26131 #26128: Let the subprocess.STARTUPINFO constructor take arguments http://bugs.python.org/issue26128 #26122: Isolated mode doesn't ignore PYTHONHASHSEED http://bugs.python.org/issue26122 #26120: pydoc: move __future__ imports out of the DATA block http://bugs.python.org/issue26120 #26117: Close directory descriptor in scandir iterator on error http://bugs.python.org/issue26117 #26103: Contradiction in definition of "data descriptor" between (dott http://bugs.python.org/issue26103 Most recent 15 issues waiting for review (15) ============================================= #26177: tkinter: Canvas().keys returns empty strings. http://bugs.python.org/issue26177 #26175: Fully implement IOBase abstract on SpooledTemporaryFile http://bugs.python.org/issue26175 #26167: Improve copy.copy speed for built-in types (list/set/dict) http://bugs.python.org/issue26167 #26146: PEP 511: Add ast.Constant to allow AST optimizer to emit const http://bugs.python.org/issue26146 #26145: PEP 511: Add sys.set_code_transformers() http://bugs.python.org/issue26145 #26144: test_pkg test_4 and/or test_7 sometimes fail http://bugs.python.org/issue26144 #26140: inspect.iscoroutinefunction raises TypeError when checks Mock http://bugs.python.org/issue26140 #26130: redundant local copy of a char pointer in classify in Parser\p http://bugs.python.org/issue26130 #26125: Incorrect error message in the module asyncio.selector_events. http://bugs.python.org/issue26125 #26121: Use C99 functions in math if available http://bugs.python.org/issue26121 #26117: Close directory descriptor in scandir iterator on error http://bugs.python.org/issue26117 #26110: Speedup method calls 1.2x http://bugs.python.org/issue26110 #26098: PEP 510: Specialize functions with guards http://bugs.python.org/issue26098 #26089: Duplicated keyword in distutils metadata http://bugs.python.org/issue26089 #26082: functools.lru_cache user specified cachedict support http://bugs.python.org/issue26082 Top 10 most discussed issues (10) ================================= #26158: File truncate() not defaulting to current position as document http://bugs.python.org/issue26158 14 msgs #25702: Link Time Optimizations support for GCC and CLANG http://bugs.python.org/issue25702 12 msgs #19475: Add timespec optional flag to datetime isoformat() to choose t http://bugs.python.org/issue19475 9 msgs #25878: CPython on Windows builds with /W3, not /W4 http://bugs.python.org/issue25878 8 msgs #25907: Documentation i18n: Added trans tags in sphinx templates http://bugs.python.org/issue25907 8 msgs #26145: PEP 511: Add sys.set_code_transformers() http://bugs.python.org/issue26145 8 msgs #26152: A non-breaking space in a source http://bugs.python.org/issue26152 7 msgs #23883: __all__ lists are incomplete http://bugs.python.org/issue23883 6 msgs #25934: ICC compiler: ICC treats denormal floating point numbers as 0. http://bugs.python.org/issue25934 6 msgs #26146: PEP 511: Add ast.Constant to allow AST optimizer to emit const http://bugs.python.org/issue26146 6 msgs Issues closed (84) ================== #5626: misleading comment in socket.gethostname() documentation http://bugs.python.org/issue5626 closed by berker.peksag #8604: Adding an atomic FS write API http://bugs.python.org/issue8604 closed by haypo #9006: xml-rpc Server object does not propagate the encoding to Unmar http://bugs.python.org/issue9006 closed by serhiy.storchaka #12869: PyOS_StdioReadline is printing the prompt on stderr http://bugs.python.org/issue12869 closed by martin.panter #14046: argparse: assertion failure if optional argument has square/ro http://bugs.python.org/issue14046 closed by martin.panter #15809: 2.7 IDLE console uses incorrect encoding. http://bugs.python.org/issue15809 closed by serhiy.storchaka #16620: Avoid using private function glob.glob1() in msi module and to http://bugs.python.org/issue16620 closed by serhiy.storchaka #16907: Distutils fails to build extension in path with spaces http://bugs.python.org/issue16907 closed by zach.ware #16956: Allow signed line number deltas in the code object's line num http://bugs.python.org/issue16956 closed by haypo #17633: zipimport's handling of namespace packages is incorrect http://bugs.python.org/issue17633 closed by brett.cannon #18620: multiprocessing page leaves out important part of Pool example http://bugs.python.org/issue18620 closed by berker.peksag #21385: in debug mode, compile(ast) fails with an assertion error if a http://bugs.python.org/issue21385 closed by haypo #21847: Fix xmlrpc in unicodeless build http://bugs.python.org/issue21847 closed by serhiy.storchaka #21949: Document the Py_SIZE() macro. http://bugs.python.org/issue21949 closed by berker.peksag #23795: argparse -- incorrect usage for mutually exclusive http://bugs.python.org/issue23795 closed by martin.panter #23962: Incorrect TimeoutError referenced in concurrent.futures docume http://bugs.python.org/issue23962 closed by orsenthil #23965: test_ssl failure on Fedora 22 http://bugs.python.org/issue23965 closed by ncoghlan #24520: Stop using deprecated floating-point environment functions on http://bugs.python.org/issue24520 closed by haypo #24761: ERROR: test_dh_params (test.test_ssl.ThreadedTests) http://bugs.python.org/issue24761 closed by martin.panter #24832: Issue building viewable docs with newer sphinx (default theme http://bugs.python.org/issue24832 closed by benjamin.peterson #24985: Python install test fails - OpenSSL - "dh key too small" http://bugs.python.org/issue24985 closed by martin.panter #25058: Right square bracket argparse metavar http://bugs.python.org/issue25058 closed by martin.panter #25089: Can't run Python Launcher on Windows http://bugs.python.org/issue25089 closed by steve.dower #25366: test_venv fails with --without-threads http://bugs.python.org/issue25366 closed by berker.peksag #25613: fix ssl tests with sslv3 disabled http://bugs.python.org/issue25613 closed by martin.panter #25644: Unable to open IDLE on Windows 10 with Python 2.7.10 http://bugs.python.org/issue25644 closed by steve.dower #25694: test.libregrtest not installed http://bugs.python.org/issue25694 closed by steve.dower #25704: Update the devguide to 3.5 http://bugs.python.org/issue25704 closed by berker.peksag #25731: Assigning and deleting __new__ attr on the class does not allo http://bugs.python.org/issue25731 closed by python-dev #25759: Python 2.7.11rc1 not building with Visual Studio 2015 http://bugs.python.org/issue25759 closed by steve.dower #25765: Installation error http://bugs.python.org/issue25765 closed by steve.dower #25799: 2.7.11rc1 not added to Win10 app list (start menu) http://bugs.python.org/issue25799 closed by terry.reedy #25824: 32-bit 2.7.11 installer creates registry keys that are incompa http://bugs.python.org/issue25824 closed by steve.dower #25843: code_richcompare() don't use constant type when comparing code http://bugs.python.org/issue25843 closed by haypo #25850: Building extensions with MSVC 2015 Express fails http://bugs.python.org/issue25850 closed by steve.dower #25859: EOFError in test_nntplib.NetworkedNNTPTests.test_starttls() http://bugs.python.org/issue25859 closed by martin.panter #25876: test_gdb: use subprocess._args_from_interpreter_flags() to tes http://bugs.python.org/issue25876 closed by haypo #25909: Incorrect documentation for PyMapping_Items and like http://bugs.python.org/issue25909 closed by orsenthil #25925: Coverage support for CPython 2 http://bugs.python.org/issue25925 closed by zach.ware #25935: OrderedDict prevents garbage collection if a circulary referen http://bugs.python.org/issue25935 closed by serhiy.storchaka #25982: multiprocessing docs for Namespace lacks class definition http://bugs.python.org/issue25982 closed by orsenthil #25993: Crashed when call time.time() after using _mm_xor_si64 http://bugs.python.org/issue25993 closed by steve.dower #26013: Pickle protocol 2.0 not loading in python 3.5 http://bugs.python.org/issue26013 closed by serhiy.storchaka #26017: Update https://docs.python.org/3/installing/index.html to alwa http://bugs.python.org/issue26017 closed by orsenthil #26035: traceback.print_tb() takes `tb`, not `traceback` as a keyword http://bugs.python.org/issue26035 closed by orsenthil #26059: Integer Overflow in strop.replace() http://bugs.python.org/issue26059 closed by serhiy.storchaka #26065: python embedded 3.5 amd64 crash when using venv http://bugs.python.org/issue26065 closed by steve.dower #26070: Launcher fails to find in-place built binaries from earlier Py http://bugs.python.org/issue26070 closed by steve.dower #26071: bdist_wininst created binaries fail to start and find 32bit Py http://bugs.python.org/issue26071 closed by steve.dower #26073: Update the list of magic numbers in launcher http://bugs.python.org/issue26073 closed by steve.dower #26077: Make slicing of immutable structures return a view instead of http://bugs.python.org/issue26077 closed by serhiy.storchaka #26099: site ignores ImportError when running sitecustomize and usercu http://bugs.python.org/issue26099 closed by haypo #26100: PEP 511: Add test.support.optim_args_from_interpreter_flags() http://bugs.python.org/issue26100 closed by haypo #26101: Lib/test/test_compileall.py fails when run directly http://bugs.python.org/issue26101 closed by haypo #26106: Move licences to literal blocks http://bugs.python.org/issue26106 closed by haypo #26107: PEP 511: code.co_lnotab: use signed line number delta to suppo http://bugs.python.org/issue26107 closed by haypo #26108: Calling PyInitialize with 2.7.11 on Windows x64 terminates pro http://bugs.python.org/issue26108 closed by steve.dower #26114: Rewrite math.erf() and math.erfc() from scratch http://bugs.python.org/issue26114 closed by brett.cannon #26126: Possible subtle bug when normalizing and str.translate()ing http://bugs.python.org/issue26126 closed by SilentGhost #26127: Broken link in docs for tokenize http://bugs.python.org/issue26127 closed by martin.panter #26129: Difference in behaviour with grp.getgrgid and pwd.getpwuid http://bugs.python.org/issue26129 closed by larry #26135: Documentation Recommends Deprecated `imp` Module http://bugs.python.org/issue26135 closed by orsenthil #26138: Disable /W4 warning (non-standard dllimport behavior) http://bugs.python.org/issue26138 closed by skrah #26139: libmpdec: disable /W4 warning (non-standard dllimport behavior http://bugs.python.org/issue26139 closed by skrah #26142: Formatting bug on https://docs.python.org/2.7/c-api/intro.html http://bugs.python.org/issue26142 closed by orsenthil #26147: Encoding errors in xmlrpc http://bugs.python.org/issue26147 closed by serhiy.storchaka #26150: SequenceMatcher's algorithm is not correct http://bugs.python.org/issue26150 closed by tim.peters #26151: str(bytes) does __repr__() instead of __str__() http://bugs.python.org/issue26151 closed by haypo #26154: Add private _PyThreadState_UncheckedGet() to get the current t http://bugs.python.org/issue26154 closed by haypo #26156: Bad name into power operator syntax http://bugs.python.org/issue26156 closed by yselivanov #26157: Typo in asyncio documentation http://bugs.python.org/issue26157 closed by berker.peksag #26161: Use Py_uintptr_t instead of void* for atomic pointers http://bugs.python.org/issue26161 closed by haypo #26162: thread error http://bugs.python.org/issue26162 closed by eryksun #26163: FAIL: test_hash_effectiveness (test.test_set.TestFrozenSet) http://bugs.python.org/issue26163 closed by haypo #26164: test_with_pip() of test_venv fails on Windows buildbots http://bugs.python.org/issue26164 closed by haypo #26165: devguide: table summarizing status of Python branches http://bugs.python.org/issue26165 closed by haypo #26166: zlib compressor/decompressor objects should support copy proto http://bugs.python.org/issue26166 closed by serhiy.storchaka #26169: Pasting 900000 chars into a tk Entry widget fails http://bugs.python.org/issue26169 closed by serhiy.storchaka #26170: pip Crash on Unpacking in get_platform() line 119 http://bugs.python.org/issue26170 closed by dstufft #26171: heap overflow in zipimporter module http://bugs.python.org/issue26171 closed by python-dev #26172: iBook can't open ePub http://bugs.python.org/issue26172 closed by python-dev #26174: Exception alias cause destruction of existing variable http://bugs.python.org/issue26174 closed by brett.cannon #26178: Python C-API: __all__ Creator http://bugs.python.org/issue26178 closed by Devyn Johnson #26179: Python C-API "unused-parameter" warnings http://bugs.python.org/issue26179 closed by haypo From robertpancoast77 at gmail.com Fri Jan 22 14:56:37 2016 From: robertpancoast77 at gmail.com (=?UTF-8?Q?=C6=A6OB_COASTN?=) Date: Fri, 22 Jan 2016 14:56:37 -0500 Subject: [Python-Dev] python3 k1om dissociation permanence Message-ID: Hello, Enabling the build system for Intel MIC k1om is non-trivial using Python-3.4.4 Using Python2 for the k1om is very popular, but I require Python3 support on k1om. The first requirement to complete this build involved the download and extraction of pre-built MPSS RPM's. Then built required host python bins using GCC. Lastly, build MIC bins using ICC. The exacts are appended to the end of this message. I would like to discuss a few change requirements that trial and error has revealed. 1.) libffi requires the University OF Cantabria patch because the k1om is not binary compatible with x86_64. [attached] These libffi changes could be implemented using the __MIC__ or __KNC__ macros. *see https://software.intel.com/en-us/node/514528 2.) ./configure script halts during failure to locate readelf for the host. I simply commented out these lines in the ./configure file: #if test "$cross_compiling" = yes; then #case "$READELF" in #readelf|:) #as_fn_error $? "readelf for the host is required for cross builds" "$LINENO" 5 #;; #esac #fi Ideally, ./configure would support ICC and k1om. Am I missing something in the configure/make commands below? Is it possible to bypass the readelf requirement when cross-compiling for k1om? Additionally, are any of the command line parameters below unnecessary? PKG_CONFIG_LIBDIR PKG_CONFIG_PATH PYTHON_FOR_BUILD _PYTHON_HOST_PLATFORM HOSTPGEN HOSTARCH BUILDARCH Thanks, Rob #copy/unpack the k1om bins tarball cd /home/ wget mpss-3.4.6-k1om.tar tar xvf mpss-3.4.6-k1om.tar cd /home/mpss-3.4.6/k1om/ for rpm in *.rpm; do rpm2cpio $rpm | cpio -idm; done #vars PythonVersion=Python-3.4.4 k1om_rpm=/home/mpss-3.4.6/k1om/ INSTALLPREFIX=/home/Python/release/$PythonVersion-mic SRC=/home/Python/$PythonVersion echo "Compiling host Python" cd $SRC && make distclean cd $SRC && ./configure cd $SRC && make python Parser/pgen rm -f $SRC/hostpython mv $SRC/python $SRC/hostpython rm -f $SRC/Parser/hostpgen mv $SRC/Parser/pgen $SRC/Parser/hostpgen cd $SRC && make distclean echo "Configuring Python for MIC..." cd $SRC && CONFIG_SITE=config.site \ ./configure \ CC="icc -mmic" \ CFLAGS="-I$k1om_rpm/include -I$k1om_rpm/usr/include -wd10006" \ CXX="icpc -mmic" \ CPPFLAGS="-I$k1om_rpm/include -I$k1om_rpm/usr/include -wd10006" \ PKG_CONFIG_LIBDIR="$k1om_rpm/usr/lib64/pkgconfig" \ PKG_CONFIG_PATH="$k1om_rpm/usr/lib64/pkgconfig" \ --host=x86_64-k1om-linux \ --build=x86_64-linux-gnu \ --with-cxx-main="icpc -mmic" \ --disable-ipv6 echo "done" echo "Compiling Python for MIC..." cd $SRC && make \ PYTHON_FOR_BUILD=./hostpython \ _PYTHON_HOST_PLATFORM=x86_64-k1om-linux \ HOSTPGEN=./Parser/hostpgen \ HOSTARCH=x86_64-k1om-linux \ BUILDARCH=x86_64-linux-gnu \ EXTRA_CFLAGS="-fp-model precise -shared -fPIC" \ LDFLAGS="-L$k1om_rpm/lib64 -L$k1om_rpm/usr/lib64" echo "done" echo "Installing Python for MIC" mkdir -p $INSTALLPREFIX cd $SRC && make install \ PYTHON_FOR_BUILD=./hostpython \ _PYTHON_HOST_PLATFORM=x86_64-k1om-linux \ prefix=$INSTALLPREFIX echo "done" -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-k1om-libffi.patch Type: application/octet-stream Size: 18705 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-READELF.patch Type: application/octet-stream Size: 790 bytes Desc: not available URL: From brett at python.org Fri Jan 22 16:50:04 2016 From: brett at python.org (Brett Cannon) Date: Fri, 22 Jan 2016 21:50:04 +0000 Subject: [Python-Dev] python3 k1om dissociation permanence In-Reply-To: References: Message-ID: If you could, ?OB, can you open issues on bugs.python.org for each of these problems/changes? Otherwise we will lose track of this. On Fri, 22 Jan 2016 at 11:57 ?OB COASTN wrote: > Hello, > > Enabling the build system for Intel MIC k1om is non-trivial using > Python-3.4.4 > Using Python2 for the k1om is very popular, but I require Python3 > support on k1om. > > The first requirement to complete this build involved the download and > extraction of pre-built MPSS RPM's. > Then built required host python bins using GCC. > Lastly, build MIC bins using ICC. > The exacts are appended to the end of this message. > > I would like to discuss a few change requirements that trial and error > has revealed. > > 1.) libffi requires the University OF Cantabria patch because the k1om > is not binary compatible with x86_64. [attached] > > These libffi changes could be implemented using the __MIC__ or __KNC__ > macros. > *see https://software.intel.com/en-us/node/514528 > > 2.) ./configure script halts during failure to locate readelf for the host. > > I simply commented out these lines in the ./configure file: > #if test "$cross_compiling" = yes; then > #case "$READELF" in > #readelf|:) > #as_fn_error $? "readelf for the host is required for cross > builds" "$LINENO" 5 > #;; > #esac > #fi > > Ideally, ./configure would support ICC and k1om. > Am I missing something in the configure/make commands below? > Is it possible to bypass the readelf requirement when cross-compiling for > k1om? > > Additionally, are any of the command line parameters below unnecessary? > PKG_CONFIG_LIBDIR > PKG_CONFIG_PATH > PYTHON_FOR_BUILD > _PYTHON_HOST_PLATFORM > HOSTPGEN > HOSTARCH > BUILDARCH > > > Thanks, > Rob > > > > > #copy/unpack the k1om bins tarball > cd /home/ > wget mpss-3.4.6-k1om.tar > tar xvf mpss-3.4.6-k1om.tar > cd /home/mpss-3.4.6/k1om/ > for rpm in *.rpm; do rpm2cpio $rpm | cpio -idm; done > > #vars > PythonVersion=Python-3.4.4 > k1om_rpm=/home/mpss-3.4.6/k1om/ > INSTALLPREFIX=/home/Python/release/$PythonVersion-mic > SRC=/home/Python/$PythonVersion > > echo "Compiling host Python" > cd $SRC && make distclean > cd $SRC && ./configure > cd $SRC && make python Parser/pgen > rm -f $SRC/hostpython > mv $SRC/python $SRC/hostpython > rm -f $SRC/Parser/hostpgen > mv $SRC/Parser/pgen $SRC/Parser/hostpgen > cd $SRC && make distclean > > echo "Configuring Python for MIC..." > cd $SRC && CONFIG_SITE=config.site \ > ./configure \ > CC="icc -mmic" \ > CFLAGS="-I$k1om_rpm/include -I$k1om_rpm/usr/include -wd10006" \ > CXX="icpc -mmic" \ > CPPFLAGS="-I$k1om_rpm/include -I$k1om_rpm/usr/include -wd10006" \ > PKG_CONFIG_LIBDIR="$k1om_rpm/usr/lib64/pkgconfig" \ > PKG_CONFIG_PATH="$k1om_rpm/usr/lib64/pkgconfig" \ > --host=x86_64-k1om-linux \ > --build=x86_64-linux-gnu \ > --with-cxx-main="icpc -mmic" \ > --disable-ipv6 > echo "done" > > echo "Compiling Python for MIC..." > cd $SRC && make \ > PYTHON_FOR_BUILD=./hostpython \ > _PYTHON_HOST_PLATFORM=x86_64-k1om-linux \ > HOSTPGEN=./Parser/hostpgen \ > HOSTARCH=x86_64-k1om-linux \ > BUILDARCH=x86_64-linux-gnu \ > EXTRA_CFLAGS="-fp-model precise -shared -fPIC" \ > LDFLAGS="-L$k1om_rpm/lib64 -L$k1om_rpm/usr/lib64" > echo "done" > > echo "Installing Python for MIC" > mkdir -p $INSTALLPREFIX > cd $SRC && make install \ > PYTHON_FOR_BUILD=./hostpython \ > _PYTHON_HOST_PLATFORM=x86_64-k1om-linux \ > prefix=$INSTALLPREFIX > echo "done" > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jan 22 18:44:57 2016 From: brett at python.org (Brett Cannon) Date: Fri, 22 Jan 2016 23:44:57 +0000 Subject: [Python-Dev] do people use sys._mercurial? Message-ID: Since we are going to be switching over to Git, sys._mercurial is going to be made to return a dummy value of `('CPython', '', '')` once we switch to Git. But my question is do we bother to replace it with sys._git? I wanted to make sure that the effort is worth it to keep changing these VCS-specific attributes every time we change our VCS (and no, we are not going to adopt a generic one; already had that debate). So do please speak up if you actually have found value from sys._mercurial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jan 22 21:47:14 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 23 Jan 2016 12:47:14 +1000 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: References: Message-ID: On 23 January 2016 at 09:44, Brett Cannon wrote: > Since we are going to be switching over to Git, sys._mercurial is going to > be made to return a dummy value of `('CPython', '', '')` once we switch to > Git. But my question is do we bother to replace it with sys._git? I wanted > to make sure that the effort is worth it to keep changing these VCS-specific > attributes every time we change our VCS (and no, we are not going to adopt a > generic one; already had that debate). So do please speak up if you actually > have found value from sys._mercurial. It's incorporated into the output of "platform.python_(branch|revision|build)()", so I assume most people would use those if they needed to report exact build information, rather than accessing the attribute directly. We also use it ourselves in printing the appropriate banner when running an interactive prompt (note the first line of the banner): Python 3.6.0a0 (default:32a4e7b337c9, Jan 23 2016, 12:30:00) [GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys._mercurial ('CPython', 'default', '32a4e7b337c9') And the regression test suite: $ ./python -m test == CPython 3.6.0a0 (default:32a4e7b337c9, Jan 23 2016, 12:30:00) [GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] == Linux-4.3.3-301.fc23.x86_64-x86_64-with-fedora-23-Twenty_Three little-endian == hash algorithm: siphash24 64bit == /home/ncoghlan/devel/cpython/build/test_python_13167 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0) [ 1/401] test_grammar ... I guess that means "Update platform.py, and test the interactive prompt and regression test banners still work as expected when built from git" needs to be added to the CPython migration part of the PEP. (In looking into this, I found several of the docstrings in platform.py still referred to Subversion, even though the platform._sys_version() helping had been updated to handle Mercurial) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From nad at python.org Fri Jan 22 22:33:39 2016 From: nad at python.org (Ned Deily) Date: Fri, 22 Jan 2016 22:33:39 -0500 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: References: Message-ID: On Jan 22, 2016, at 18:44, Brett Cannon wrote: > Since we are going to be switching over to Git, sys._mercurial is going to be made to return a dummy value of `('CPython', '', '')` once we switch to Git. But my question is do we bother to replace it with sys._git? I wanted to make sure that the effort is worth it to keep changing these VCS-specific attributes every time we change our VCS (and no, we are not going to adopt a generic one; already had that debate). So do please speak up if you actually have found value from sys._mercurial. As long as the git revision tag (if any) and hash show up in sys.version and the interpreter interactive (REPL) mode header as they do today with hg and previously with svn (that is, when the interpreter is built from a vcs checkout), I'm happy: $ python3 Python 3.5.1 (v3.5.1:37a07cee5969, Dec 5 2015, 21:12:44) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> -- Ned Deily nad at python.org -- [] From guido at python.org Sat Jan 23 00:50:03 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 22 Jan 2016 21:50:03 -0800 Subject: [Python-Dev] Typehinting repo moved to python/typing Message-ID: This is just a note that with Benjamin's help we've moved the ambv/typehinting repo on GitHub into the python org, so its URL is now https://github.com/python/typing . This repo was used most intensely for discussions during PEP 484's drafting period. It also contains the code for typing.py, repackaged for earlier releases on PyPI. The issue tracker is still open for proposals to change PEP 484, which is not unheard of given its provisional status. If you find a pointer to the original location of this repo in a file you can update, please go ahead (though GitHub is pretty good at forwarding URLs from renamed repos). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Jan 23 01:03:12 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 23 Jan 2016 17:03:12 +1100 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? Message-ID: I just had a major crash on the system that hosts the angelico-debian-amd64 buildbot, and as usual, checked it carefully after bringing everything up. It seems now to be timing out after an hour of operation: http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/3132/steps/test/logs/stdio This is happening across all the 3.* branches on my buildbot, but NOT on 2.7, and not on any other buildbots. That makes it look like some kind of config problem at my end. In seeking to diagnose it, I duplicated the 3.x build directory and manually ran the commands to run a test, and it stalled out (I didn't let it go to the whole hour but it sat there for some minutes) at the same place: in test_socket. Running just that test file: $ ./python Lib/test/test_socket.py ... chomp lots of lines ... testRecvmsgPeek (__main__.RecvmsgUDP6Test) ... seems to indicate that the stall is due to IPv6 and UDP. The VM should have full IPv6 support, although my ISPs don't carry IPv6 traffic, so it won't be able to reach the internet proper; but it should be able to do all manner of local tests. The test runs just fine on my main system, so it's only failing in the buildbot VM. Any ideas as to what's going on or how to diagnose? By the way, this looks odd: make buildbottest TESTOPTS= TESTPYTHONOPTS= TESTTIMEOUT=3600 in dir /root/buildarea/3.x.angelico-debian-amd64/build (timeout 3900 secs) The parameter says 3600 (which corresponds to the error message at the end), but it echoes back that the timeout is 3900 seconds. ChrisA From zachary.ware+pydev at gmail.com Sat Jan 23 01:39:07 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Sat, 23 Jan 2016 00:39:07 -0600 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? In-Reply-To: References: Message-ID: On Sat, Jan 23, 2016 at 12:03 AM, Chris Angelico wrote: > By the way, this looks odd: > > make buildbottest TESTOPTS= TESTPYTHONOPTS= TESTTIMEOUT=3600 > in dir /root/buildarea/3.x.angelico-debian-amd64/build (timeout 3900 secs) > > The parameter says 3600 (which corresponds to the error message at the > end), but it echoes back that the timeout is 3900 seconds. I'm no help on the main issue, but to explain the timeout difference: TESTTIMEOUT is a makefile variable that sets the faulthandler timeout that tries to make Python bail out with a stack trace instead of letting buildbot kill Python silently. The 3900 second timeout is buildbot's "there's been no output in this long, assume it's hung and kill it" timeout. The difference between the two is to give faulthandler a chance to do its thing. -- Zach From rosuav at gmail.com Sat Jan 23 02:47:14 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 23 Jan 2016 18:47:14 +1100 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? In-Reply-To: References: Message-ID: On Sat, Jan 23, 2016 at 5:39 PM, Zachary Ware wrote: > On Sat, Jan 23, 2016 at 12:03 AM, Chris Angelico wrote: >> By the way, this looks odd: >> >> make buildbottest TESTOPTS= TESTPYTHONOPTS= TESTTIMEOUT=3600 >> in dir /root/buildarea/3.x.angelico-debian-amd64/build (timeout 3900 secs) >> >> The parameter says 3600 (which corresponds to the error message at the >> end), but it echoes back that the timeout is 3900 seconds. > > I'm no help on the main issue, but to explain the timeout difference: > TESTTIMEOUT is a makefile variable that sets the faulthandler timeout > that tries to make Python bail out with a stack trace instead of > letting buildbot kill Python silently. The 3900 second timeout is > buildbot's "there's been no output in this long, assume it's hung and > kill it" timeout. The difference between the two is to give > faulthandler a chance to do its thing. Ah, cool. That's one mystery cleared up, at least. ChrisA From victor.stinner at gmail.com Sat Jan 23 09:37:36 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 23 Jan 2016 15:37:36 +0100 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? In-Reply-To: References: Message-ID: 3600 seconds is the maximum duration of a single test file. We may reduce it since a single test file should not take longer than 30 min. Maybe we can do better and put the timeout on a single test function. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Jan 23 09:55:58 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 24 Jan 2016 01:55:58 +1100 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? In-Reply-To: References: Message-ID: On Sun, Jan 24, 2016 at 1:37 AM, Victor Stinner wrote: > 3600 seconds is the maximum duration of a single test file. We may reduce it > since a single test file should not take longer than 30 min. Maybe we can do > better and put the timeout on a single test function. I'd be inclined to put some strong timeouts on test_socket.py (at some level or other of granularity). Most of those tests should be finished in a fraction of a second; a few might take a few seconds, maybe. None should take a whole minute. But they might easily sit around that long. But I'd rather know what I messed up in my recreation of the VM's configs. ChrisA From brett at python.org Sat Jan 23 14:30:32 2016 From: brett at python.org (Brett Cannon) Date: Sat, 23 Jan 2016 19:30:32 +0000 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: References: Message-ID: It seems that people do use the values so there will be a sys._git attribute. On Fri, 22 Jan 2016 at 19:57 Ned Deily wrote: > On Jan 22, 2016, at 18:44, Brett Cannon wrote: > > Since we are going to be switching over to Git, sys._mercurial is going > to be made to return a dummy value of `('CPython', '', '')` once we switch > to Git. But my question is do we bother to replace it with sys._git? I > wanted to make sure that the effort is worth it to keep changing these > VCS-specific attributes every time we change our VCS (and no, we are not > going to adopt a generic one; already had that debate). So do please speak > up if you actually have found value from sys._mercurial. > > As long as the git revision tag (if any) and hash show up in sys.version > and the interpreter interactive (REPL) mode header as they do today with hg > and previously with svn (that is, when the interpreter is built from a vcs > checkout), I'm happy: > > $ python3 > Python 3.5.1 (v3.5.1:37a07cee5969, Dec 5 2015, 21:12:44) > [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> > > -- > Ned Deily > nad at python.org -- [] > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From francismb at email.de Sat Jan 23 14:45:19 2016 From: francismb at email.de (francismb) Date: Sat, 23 Jan 2016 20:45:19 +0100 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: References: Message-ID: <56A3D84F.70803@email.de> Hi, On 01/23/2016 12:44 AM, Brett Cannon wrote: > Since we are going to be switching over to Git, sys._mercurial is going to > be made to return a dummy value of `('CPython', '', '')` once we switch to > Git. for me sys._mercurial it's already returning that (?) : what should return now? (it's a bug?) $ ipython Python 2.7.11 (default, Dec 9 2015, 00:29:25) Type "copyright", "credits" or "license" for more information. IPython 2.4.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]: import sys In [2]: sys._mercurial Out[2]: ('CPython', '', '') In [3]: Regards, francis From brett at python.org Sat Jan 23 14:48:56 2016 From: brett at python.org (Brett Cannon) Date: Sat, 23 Jan 2016 19:48:56 +0000 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: <56A3D84F.70803@email.de> References: <56A3D84F.70803@email.de> Message-ID: On Sat, 23 Jan 2016 at 11:45 francismb wrote: > Hi, > > On 01/23/2016 12:44 AM, Brett Cannon wrote: > > Since we are going to be switching over to Git, sys._mercurial is going > to > > be made to return a dummy value of `('CPython', '', '')` once we switch > to > > Git. > > for me sys._mercurial it's already returning that (?) : what should > return now? (it's a bug?) > Depends on your OS and how CPython was built whether it returns that value or something more useful. IOW it's not a bug. -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Sat Jan 23 19:09:41 2016 From: random832 at fastmail.com (Random832) Date: Sat, 23 Jan 2016 19:09:41 -0500 Subject: [Python-Dev] do people use sys._mercurial? References: Message-ID: Brett Cannon writes: > (and no, we are not going to adopt a generic one; already had that > debate). Do you have a link to the relevant discussions? From brett at python.org Sat Jan 23 19:29:12 2016 From: brett at python.org (Brett Cannon) Date: Sun, 24 Jan 2016 00:29:12 +0000 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: References: Message-ID: Some discussion happened on core-workflow@, otherwise you can look through the python-dev archives for when we added sys._mercurial. On Sat, 23 Jan 2016, 16:25 Random832 wrote: > Brett Cannon writes: > > (and no, we are not going to adopt a generic one; already had that > > debate). > > Do you have a link to the relevant discussions? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 24 07:17:32 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Jan 2016 22:17:32 +1000 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: References: <56A3D84F.70803@email.de> Message-ID: On 24 January 2016 at 05:48, Brett Cannon wrote: > On Sat, 23 Jan 2016 at 11:45 francismb wrote: >> for me sys._mercurial it's already returning that (?) : what should >> return now? (it's a bug?) > > Depends on your OS and how CPython was built whether it returns that value > or something more useful. IOW it's not a bug. Linux distros tend to build Python from a tarball rather than a source checkout, for example, which means the build directory doesn't include any VCS details: $ python3 Python 3.4.3 (default, Jun 29 2015, 12:16:01) [GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys._mercurial ('CPython', '', '') While my local checkout does have those details: $ ./python Python 3.6.0a0 (default:32a4e7b337c9, Jan 23 2016, 12:30:00) [GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys._mercurial ('CPython', 'default', '32a4e7b337c9') Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Jan 24 07:13:32 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Jan 2016 22:13:32 +1000 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: References: Message-ID: On 24 January 2016 at 10:29, Brett Cannon wrote: > Some discussion happened on core-workflow@, otherwise you can look through > the python-dev archives for when we added sys._mercurial. We actually forgot one relevant point in those discussions: there's already a generic API for accessing this information in the platform module. At the sys module level, though, the key point is that the precise semantic interpretation is VCS dependent, so changing the variable name conveys immediately to users both that the semantics have changed, and which VCS is now being used to define them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From francismb at email.de Sun Jan 24 10:48:07 2016 From: francismb at email.de (francismb) Date: Sun, 24 Jan 2016 16:48:07 +0100 Subject: [Python-Dev] Code formatter bot In-Reply-To: <569EA3A6.4010802@email.de> References: <569EA3A6.4010802@email.de> Message-ID: <56A4F237.6060709@email.de> Hi, from your valuable feedback, here is what I thing could be a previous requirements list (lets call it for e.g. autopep7 script by now): - It should follow PEP 7 :-) - It should check PEP 7 compliance on a per file basis (for simplicity) - It should be embeddable on the test suite, returning PASS or FAILURE - It should be usable as a pre-commit hook - It could return a patch to apply to the non compliant file making it compliant - It could return the lines (an the reason why) that aren't compliant Some other details on using some autopep7 on the infrastructure (as part of a workflow, by wrapping or enhancing it): - It could be used it on a regular basis against the cpython repo to be able to see how much files doesn't follow PEP 7. The first time shows the legacy ones (plus the current failures). - It should be used in pair with a "skip" list to avoid checking the legacy code mentioned (or a list of pairs file:reason to skip). - It could be used on the CI-side to flag PEP 7 compliance (core-devs doesn't have to need to point to that in reviews). Or could be used as a part of series of checks that generate a "ready-to-review" flag. - It could be used to continually check and proposes patches (on bug.python.org). - It could be used to continually check, log the issue on the tracker and commit the patch to the cpython repository. Regards, francis From francismb at email.de Sun Jan 24 12:40:14 2016 From: francismb at email.de (francismb) Date: Sun, 24 Jan 2016 18:40:14 +0100 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: References: <56A3D84F.70803@email.de> Message-ID: <56A50C7E.3090209@email.de> Hi, On 01/24/2016 01:17 PM, Nick Coghlan wrote: > > Linux distros tend to build Python from a tarball rather than a source > checkout, for example, which means the build directory doesn't include > any VCS details: > Does that helps traceability (reproducibility)? If distros use (?) the tarball from the release why it doesn't have, at least, the information from where that tarball was generated from (the check out point) ? Regards, francis From raymond.hettinger at gmail.com Sun Jan 24 19:28:36 2016 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 24 Jan 2016 16:28:36 -0800 Subject: [Python-Dev] Code formatter bot In-Reply-To: <569EA3A6.4010802@email.de> References: <569EA3A6.4010802@email.de> Message-ID: > On Jan 19, 2016, at 12:59 PM, francismb wrote: > > Dear Core-Devs, > what's your opinion about a code-formatter bot for cpython. > Pros, Cons, where could be applicable (new commits, new workflow, it > doesn't make sense), ... > > > - At least it should follow PEP 7 ;-) Please don't do this. It misses the spirit of how the style-guides are intended to be used. "I personally hate with a vengeance that there are tools named after style guide PEPs that claim to enforce the guidelines from those PEPs. The tools' rigidity and simplicity reflects badly on the PEPs, which try hard not to be rigid or simplistic." -- GvR https://mail.python.org/pipermail/python-dev/2016-January/142643.html "PEP 8 unto thyself, not onto others" -- Me https://www.youtube.com/watch?v=wf-BqAjZb8M (the most popular talk from last year's Pycon) Almost nothing that is wrong with CPython is stylistic, the real issues are more substantive. That is where you should devote your talents. Raymond Hettinger From stephen at xemacs.org Sun Jan 24 21:07:17 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 25 Jan 2016 11:07:17 +0900 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: <56A50C7E.3090209@email.de> References: <56A3D84F.70803@email.de> <56A50C7E.3090209@email.de> Message-ID: <22181.33621.40740.909756@turnbull.sk.tsukuba.ac.jp> francismb writes: > Does that helps traceability (reproducibility)? If distros use (?) > the tarball from the release why it doesn't have, at least, the > information from where that tarball was generated from (the check > out point) ? The pointer goes in the other direction: there will be a tag in the repo named to indicate the version. From ncoghlan at gmail.com Sun Jan 24 21:24:34 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 25 Jan 2016 12:24:34 +1000 Subject: [Python-Dev] do people use sys._mercurial? In-Reply-To: <56A50C7E.3090209@email.de> References: <56A3D84F.70803@email.de> <56A50C7E.3090209@email.de> Message-ID: On 25 January 2016 at 03:40, francismb wrote: > On 01/24/2016 01:17 PM, Nick Coghlan wrote: >> Linux distros tend to build Python from a tarball rather than a source >> checkout, for example, which means the build directory doesn't include >> any VCS details: > Does that helps traceability (reproducibility)? If distros use (?) the > tarball from the release why it doesn't have, at least, the information > from where that tarball was generated from (the check out point) ? The main reason is that distro packaging processes long predate Subversion's popularisation of atomic commits in open source version control tools, and are designed to cope with release processes that involve uploading a source tarball to a web server, so they don't assume VCS tags or revision IDs will be available. However, distro processes also capture the source code itself, and often apply additional distro-specific patches, at which point claiming to correspond directly to any given upstream commit would be inaccurate. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Mon Jan 25 13:16:29 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 25 Jan 2016 19:16:29 +0100 Subject: [Python-Dev] FAT Python (lack of) performance Message-ID: Hi, Summary: FAT Python is not faster, but it will be ;-) -- When I started the FAT Python as a fork of CPython 3.6, I put everything in the same repository. Last weeks, I focused on splitting my giant patch (10k lines) into small reviewable patches. I wrote 3 PEP (509 dict version, 510 function specialziation, 511 code tranformers) and I enhanced the API to make it usable for more use cases than just FAT Python. I also created fatoptimizer (the AST optimizer) and fat (runtime dependency of the optimizer) projects on GitHub to separate clearly what should be outside Python core. For all links, see: http://faster-cpython.readthedocs.org/fat_python.html For the fatoptimizer project, my constraint is to be able to run the full Python test suite unmodified. In practice, I have to disable some optimizations by putting a "__fatoptimizer__= {...}" configuration to some test files. For example, I have to disable constant folding on test_bool because it tests that False+2 gives 2 at runtime, whereas the optimizer replaces directly False+2 with 2 during the compilation. Well, test_bool.py is not the best example because all tests pass with the constant folding optimization (if I comment my "__fatoptimizer__={...}" change). This constraint ensures that the optimizer "works" and doesn't break (too much ;-)) the Python semantics, but it's more difficult to implement powerful optimizations. I also found and fixed various kinds of bugs. In my code obviously, but also in the Python core, in various places. Some bugs only concern AST transformers which is a new feature, but I had to fix them. For example, Python didn't support negative line number delta in co_lntotab of code objects, and so line number were all wrong on optimized code. I merged my enhancement in the default branch of CPython (issue #26107). In short, I focused on having something working (respecting the Python semantics), rather than spending time on writing optimizations. -- When I asked explicitly "Is someone opposed to this PEP 509 [dict verion] ?", Barry Warsaw answered that a performance analysis is required. Extract of his mail: "I still think this is maintenance and potential performance overhead we don't want to commit to long term unless it enables significant optimization. Since you probably can't prove that without some experimentation, this API should be provisional." Last week, I ran some benchmarks and I have to admin that I was disappointed. Not only fatoptimizer doesn't make Python faster, but it makes it much slower on some tests! http://fatoptimizer.readthedocs.org/en/latest/benchmarks.html Quickly, I identified a major performance issue when nested functions are specialized, especially in Lib/json/encoder.py (tested by bm_json_v2.py benchmark). I fixed my optimizer to not specialize nested functions anymore. This simple change fixed the main performance issue. Reminder: in performance critical code, don't use nested functions! I will maybe propose patches for Lib/json/encoder.py to stop using nested functions. I only ran benchmarks with the optimizer enabled. I now have to measure the overhead of my patches (PEP 509, 510 and 511) adding the API fat AST optimizers. The overhead must be negligible. For me, it's a requirement of the whole project. Changes must not make Python slower when the optimizer is not used. fatoptimizer is faster on microbenchmarks, but I had to write manually some optimizations: http://fatoptimizer.readthedocs.org/en/latest/microbenchmarks.html IMHO fatoptimizer is not faster on macro benchmarks because it is not smart enough (yet) to generate the most interesting optimizations, like function inlining and specialization for argument types. You can estimate the speedup if you specialize manually your functions. -- Barry also wrote: "Did you address my suggestion on python-ideas to make the new C API optionally compiled in?" Well, it is an option, but I would prefer to have the API for AST optimizer directly built in Python. The first beta version of Python 3.6 is scheduled in September 2016 (deadline for new features in Python 3.6), so I still have a few months to implement more powerful optimizations and prove that it can be faster ;-) Victor From francismb at email.de Mon Jan 25 14:55:38 2016 From: francismb at email.de (francismb) Date: Mon, 25 Jan 2016 20:55:38 +0100 Subject: [Python-Dev] Code formatter bot In-Reply-To: References: <569EA3A6.4010802@email.de> Message-ID: <56A67DBA.8010005@email.de> Hi, On 01/25/2016 01:28 AM, Raymond Hettinger wrote: > >> - At least it should follow PEP 7 ;-) > > Please don't do this. It misses the spirit of how the style-guides are intended to be used. > > "I personally hate with a vengeance that there are tools named after style guide PEPs that claim to enforce the guidelines from those PEPs. The tools' rigidity and simplicity reflects badly on the PEPs, which try hard not to be rigid or simplistic." -- GvR > https://mail.python.org/pipermail/python-dev/2016-January/142643.html > Good point, tools need to get better (more context sensitive), maybe should work only as advisors and have just funny names (and not claim the spirit). > "PEP 8 unto thyself, not onto others" -- Me > https://www.youtube.com/watch?v=wf-BqAjZb8M > (the most popular talk from last year's Pycon) > > Almost nothing that is wrong with CPython is stylistic, the real issues are more substantive. That is where you should devote your talents. > Good advices and refactoring on your talk (and I love the gorilla part but I think that the "color_distraction" trick doesn't work for "non native" speakers as me ;-)) and indeed, that wanted to be my point: Let's review what the code does and not 007-Things. It's was really about the workflow not style, but I'll follow your advice. Regards, francis PS: Interesting is also how/why do people introduce bug's by reformatting or trying to beautify things (and how those can be avoided, by allowing change). From gmludo at gmail.com Mon Jan 25 16:20:24 2016 From: gmludo at gmail.com (Ludovic Gasc) Date: Mon, 25 Jan 2016 22:20:24 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: Message-ID: Hi, Just thanks for this big contribution. And maybe this project could give new ideas to optimize Python, who knows ? At least, you've win a beer for the FOSDEM event this week-end ;-) Have a nice week. -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/ 2016-01-25 19:16 GMT+01:00 Victor Stinner : > Hi, > > Summary: FAT Python is not faster, but it will be ;-) > > -- > > When I started the FAT Python as a fork of CPython 3.6, I put > everything in the same repository. Last weeks, I focused on splitting > my giant patch (10k lines) into small reviewable patches. I wrote 3 > PEP (509 dict version, 510 function specialziation, 511 code > tranformers) and I enhanced the API to make it usable for more use > cases than just FAT Python. I also created fatoptimizer (the AST > optimizer) and fat (runtime dependency of the optimizer) projects on > GitHub to separate clearly what should be outside Python core. For all > links, see: > > http://faster-cpython.readthedocs.org/fat_python.html > > For the fatoptimizer project, my constraint is to be able to run the > full Python test suite unmodified. In practice, I have to disable some > optimizations by putting a "__fatoptimizer__= {...}" configuration to > some test files. For example, I have to disable constant folding on > test_bool because it tests that False+2 gives 2 at runtime, whereas > the optimizer replaces directly False+2 with 2 during the compilation. > Well, test_bool.py is not the best example because all tests pass with > the constant folding optimization (if I comment my > "__fatoptimizer__={...}" change). > > This constraint ensures that the optimizer "works" and doesn't break > (too much ;-)) the Python semantics, but it's more difficult to > implement powerful optimizations. > > I also found and fixed various kinds of bugs. In my code obviously, > but also in the Python core, in various places. Some bugs only concern > AST transformers which is a new feature, but I had to fix them. For > example, Python didn't support negative line number delta in > co_lntotab of code objects, and so line number were all wrong on > optimized code. I merged my enhancement in the default branch of > CPython (issue #26107). > > In short, I focused on having something working (respecting the Python > semantics), rather than spending time on writing optimizations. > > -- > > When I asked explicitly "Is someone opposed to this PEP 509 [dict > verion] ?", Barry Warsaw answered that a performance analysis is > required. Extract of his mail: > > "I still think this is maintenance and potential performance > overhead we don't want to commit to long term unless it enables > significant optimization. Since you probably can't prove that without > some experimentation, this API should be provisional." > > Last week, I ran some benchmarks and I have to admin that I was > disappointed. Not only fatoptimizer doesn't make Python faster, but it > makes it much slower on some tests! > > http://fatoptimizer.readthedocs.org/en/latest/benchmarks.html > > Quickly, I identified a major performance issue when nested functions > are specialized, especially in Lib/json/encoder.py (tested by > bm_json_v2.py benchmark). I fixed my optimizer to not specialize > nested functions anymore. This simple change fixed the main > performance issue. Reminder: in performance critical code, don't use > nested functions! I will maybe propose patches for Lib/json/encoder.py > to stop using nested functions. > > I only ran benchmarks with the optimizer enabled. I now have to > measure the overhead of my patches (PEP 509, 510 and 511) adding the > API fat AST optimizers. The overhead must be negligible. For me, it's > a requirement of the whole project. Changes must not make Python > slower when the optimizer is not used. > > fatoptimizer is faster on microbenchmarks, but I had to write manually > some optimizations: > > http://fatoptimizer.readthedocs.org/en/latest/microbenchmarks.html > > IMHO fatoptimizer is not faster on macro benchmarks because it is not > smart enough (yet) to generate the most interesting optimizations, > like function inlining and specialization for argument types. You can > estimate the speedup if you specialize manually your functions. > > -- > > Barry also wrote: "Did you address my suggestion on python-ideas to > make the new C API optionally compiled in?" > > Well, it is an option, but I would prefer to have the API for AST > optimizer directly built in Python. > > The first beta version of Python 3.6 is scheduled in September 2016 > (deadline for new features in Python 3.6), so I still have a few > months to implement more powerful optimizations and prove that it can > be faster ;-) > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/gmludo%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Jan 25 16:43:16 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 25 Jan 2016 22:43:16 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: Message-ID: Hi, 2016-01-25 22:20 GMT+01:00 Ludovic Gasc : > Just thanks for this big contribution. > And maybe this project could give new ideas to optimize Python, who knows ? Sorry for my long email. I should try to summarize next time :-) In short: FAT Python is not fast today, but it will be faster if you give me a few more months to implement optimizations which will unlock the real power of AST optimizers ;-) I have a looooong list of ideas of optimizations: https://fatoptimizer.readthedocs.org/en/latest/todo.html According to microbenchmarks, the most promising optimizations are functions inlining (Python function calls are slow :-/) and specialize the code for the type of arguments. And I agree to wait until fatoptimizer is proven to be faster than the regular CPython before taking a decision of my 3 PEPs (509, 510, 511). Victor From srkunze at mail.de Mon Jan 25 16:51:11 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 25 Jan 2016 22:51:11 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: Message-ID: <56A698CF.3080402@mail.de> Hi Victor, I encourage you to proceed here. I would love to see your PEPs (509-511) incorporated into CPython. It's not that I consider Python slow (although some folks claim so), but performance improvements are always welcome; especially when I glance over diagrams like those: http://blog.carlesmateo.com/wp-content/uploads/2014/10/blog-carlesmateo-com-performance-several-languages-php7-phantomjs-nodejs-java-bash-go-perl-luajit-hhvm3_9-scale_mod5.png So, I join Barry when he says, we want more benchmarking and definite results, however, I might be less strict than he is and say: - even if FAT might not optimize significantly (whatever definition we apply), PEP 509 to 511 are a great win for CPython - they provide a great infrastructure for optimizing CPython AND extending/experimenting Python as an ecosystem - FAT provides interesting insights into the field of optimizing a dynamic language So, keep up the good work. I am eager to see where this goes. If there's anything I can do, let me know. :) Best, Sven On 25.01.2016 19:16, Victor Stinner wrote: > Hi, > > Summary: FAT Python is not faster, but it will be ;-) > > -- > > When I started the FAT Python as a fork of CPython 3.6, I put > everything in the same repository. Last weeks, I focused on splitting > my giant patch (10k lines) into small reviewable patches. I wrote 3 > PEP (509 dict version, 510 function specialziation, 511 code > tranformers) and I enhanced the API to make it usable for more use > cases than just FAT Python. I also created fatoptimizer (the AST > optimizer) and fat (runtime dependency of the optimizer) projects on > GitHub to separate clearly what should be outside Python core. For all > links, see: > > http://faster-cpython.readthedocs.org/fat_python.html > > For the fatoptimizer project, my constraint is to be able to run the > full Python test suite unmodified. In practice, I have to disable some > optimizations by putting a "__fatoptimizer__= {...}" configuration to > some test files. For example, I have to disable constant folding on > test_bool because it tests that False+2 gives 2 at runtime, whereas > the optimizer replaces directly False+2 with 2 during the compilation. > Well, test_bool.py is not the best example because all tests pass with > the constant folding optimization (if I comment my > "__fatoptimizer__={...}" change). > > This constraint ensures that the optimizer "works" and doesn't break > (too much ;-)) the Python semantics, but it's more difficult to > implement powerful optimizations. > > I also found and fixed various kinds of bugs. In my code obviously, > but also in the Python core, in various places. Some bugs only concern > AST transformers which is a new feature, but I had to fix them. For > example, Python didn't support negative line number delta in > co_lntotab of code objects, and so line number were all wrong on > optimized code. I merged my enhancement in the default branch of > CPython (issue #26107). > > In short, I focused on having something working (respecting the Python > semantics), rather than spending time on writing optimizations. > > -- > > When I asked explicitly "Is someone opposed to this PEP 509 [dict > verion] ?", Barry Warsaw answered that a performance analysis is > required. Extract of his mail: > > "I still think this is maintenance and potential performance > overhead we don't want to commit to long term unless it enables > significant optimization. Since you probably can't prove that without > some experimentation, this API should be provisional." > > Last week, I ran some benchmarks and I have to admin that I was > disappointed. Not only fatoptimizer doesn't make Python faster, but it > makes it much slower on some tests! > > http://fatoptimizer.readthedocs.org/en/latest/benchmarks.html > > Quickly, I identified a major performance issue when nested functions > are specialized, especially in Lib/json/encoder.py (tested by > bm_json_v2.py benchmark). I fixed my optimizer to not specialize > nested functions anymore. This simple change fixed the main > performance issue. Reminder: in performance critical code, don't use > nested functions! I will maybe propose patches for Lib/json/encoder.py > to stop using nested functions. > > I only ran benchmarks with the optimizer enabled. I now have to > measure the overhead of my patches (PEP 509, 510 and 511) adding the > API fat AST optimizers. The overhead must be negligible. For me, it's > a requirement of the whole project. Changes must not make Python > slower when the optimizer is not used. > > fatoptimizer is faster on microbenchmarks, but I had to write manually > some optimizations: > > http://fatoptimizer.readthedocs.org/en/latest/microbenchmarks.html > > IMHO fatoptimizer is not faster on macro benchmarks because it is not > smart enough (yet) to generate the most interesting optimizations, > like function inlining and specialization for argument types. You can > estimate the speedup if you specialize manually your functions. > > -- > > Barry also wrote: "Did you address my suggestion on python-ideas to > make the new C API optionally compiled in?" > > Well, it is an option, but I would prefer to have the API for AST > optimizer directly built in Python. > > The first beta version of Python 3.6 is scheduled in September 2016 > (deadline for new features in Python 3.6), so I still have a few > months to implement more powerful optimizations and prove that it can > be faster ;-) > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de From barry at python.org Mon Jan 25 17:14:51 2016 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Jan 2016 17:14:51 -0500 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: Message-ID: <20160125171451.49a43f0a@subdivisions.wooz.org> On Jan 25, 2016, at 07:16 PM, Victor Stinner wrote: >Barry also wrote: "Did you address my suggestion on python-ideas to >make the new C API optionally compiled in?" > >Well, it is an option, but I would prefer to have the API for AST >optimizer directly built in Python. In my plan, it would be, but it would have to be enabled with a configure switch, and provisionally protected by an #ifdef. >And I agree to wait until fatoptimizer is proven to be faster than the >regular CPython before taking a decision of my 3 PEPs (509, 510, 511). +1 - and just to be clear, I hope you succeed beyond your wildest imagination. :) Cheers, -Barry From abarnert at yahoo.com Mon Jan 25 17:28:57 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Jan 2016 14:28:57 -0800 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: Message-ID: On Jan 25, 2016, at 13:43, Victor Stinner wrote: > > According to microbenchmarks, the most promising optimizations are > functions inlining (Python function calls are slow :-/) and specialize > the code for the type of arguments. Can you specialize a function with a C API function, or only with bytecode? I'm not sure how much benefit you'd get out of specializing list vs. generic iterable or int vs. whatever from an AST transform, but substituting raw C code, on the other hand... From brett at python.org Mon Jan 25 17:34:20 2016 From: brett at python.org (Brett Cannon) Date: Mon, 25 Jan 2016 22:34:20 +0000 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: Message-ID: On Mon, 25 Jan 2016 at 14:30 Andrew Barnert via Python-Dev < python-dev at python.org> wrote: > On Jan 25, 2016, at 13:43, Victor Stinner > wrote: > > > > According to microbenchmarks, the most promising optimizations are > > functions inlining (Python function calls are slow :-/) and specialize > > the code for the type of arguments. > > Can you specialize a function with a C API function, or only with > bytecode? I'm not sure how much benefit you'd get out of specializing list > vs. generic iterable or int vs. whatever from an AST transform, but > substituting raw C code, on the other hand... > Victor's work is only manipulation of ASTs and bytecode. If you want something that low-level you need to either reach for Cython or hope a project like Pyjion pays off. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Jan 25 17:46:45 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 25 Jan 2016 23:46:45 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: Message-ID: Hi, 2016-01-25 23:28 GMT+01:00 Andrew Barnert : > On Jan 25, 2016, at 13:43, Victor Stinner wrote: >> >> According to microbenchmarks, the most promising optimizations are >> functions inlining (Python function calls are slow :-/) and specialize >> the code for the type of arguments. > > Can you specialize a function with a C API function, or only with bytecode? I'm not sure how much benefit you'd get out of specializing list vs. generic iterable or int vs. whatever from an AST transform, but substituting raw C code, on the other hand... As I wrote in the first part of my email, I redesigned to API to make it more generic. One of my change was to change PyFunction_Specialize() to not only accept code objects, but any callable object. The PEP 510 even contains an example using a builtin function as the specialized code: https://www.python.org/dev/peps/pep-0510/#using-builtin-function "On a microbenchmark, calling the C builtin takes 95 ns, whereas the original bytecode takes 155 ns (+60 ns): 1.6 times as fast. Calling directly chr(65) takes 76 ns." You can design an AST optimizer to compile some functions to C and then register them as specialized code at runtime. I have a side project to use Cython and/or pythran to specialize some functions using type annotation on parameters. Victor From abarnert at yahoo.com Mon Jan 25 17:50:31 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Jan 2016 14:50:31 -0800 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: Message-ID: <505C9ADE-804D-4EA9-97D2-A67800A6F5FD@yahoo.com> On Jan 25, 2016, at 14:46, Victor Stinner wrote: > > You can design an AST optimizer to compile some functions to C and > then register them as specialized code at runtime. I have a side > project to use Cython and/or pythran to specialize some functions > using type annotation on parameters. That last part is exactly what I was thinking of. One way in which cythonizing your code isn't 100% compatible is that if you, say, shadow or replace int or range, the cython code is now wrong. Which is exactly the kind of thing FAT can guard against. Which is very cool. Glad to see you already thought of that before me. :) From victor.stinner at gmail.com Mon Jan 25 17:58:12 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 25 Jan 2016 23:58:12 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <56A698CF.3080402@mail.de> References: <56A698CF.3080402@mail.de> Message-ID: 2016-01-25 22:51 GMT+01:00 Sven R. Kunze : > - they provide a great infrastructure for optimizing CPython AND > extending/experimenting Python as an ecosystem I hope that these API will create more optimizer projects than just fatoptimizer. For example, I expect more specialized optimizers like numba or pythran which are very efficient but more specific (ex: numeric computations) than fatoptimizer. Maybe not new optimizers, but just glue to existing static compilers (numba, pythran, cython, etc.). > If there's anything I can do, let me know. :) Oh, they are a lot of things to do! My patches for PEP 509, 510 and 511 still need some love (reviews): http://bugs.python.org/issue26058 http://bugs.python.org/issue26098 http://bugs.python.org/issue26145 I'm finishing my patch adding ast.Constant. This one is less controversal, it has no impact on performance nor the Python semantics: http://bugs.python.org/issue26146 But these patches are boring C code. You may prefer to work on the funny fatoptimizer project which is written in pure Python: https://fatoptimizer.readthedocs.org/en/latest/ Victor From songofacandy at gmail.com Mon Jan 25 21:21:45 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 26 Jan 2016 11:21:45 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: I'm very interested in it. Ruby 2.2 and PHP 7 are faster than Python 2. Python 3 is slower than Python 2. Performance is a attractive feature. Python 3 lacks it. How can I help your work? On Tue, Jan 26, 2016 at 7:58 AM, Victor Stinner wrote: > 2016-01-25 22:51 GMT+01:00 Sven R. Kunze : > > - they provide a great infrastructure for optimizing CPython AND > > extending/experimenting Python as an ecosystem > > I hope that these API will create more optimizer projects than just > fatoptimizer. > > For example, I expect more specialized optimizers like numba or > pythran which are very efficient but more specific (ex: numeric > computations) than fatoptimizer. Maybe not new optimizers, but just > glue to existing static compilers (numba, pythran, cython, etc.). > > > > If there's anything I can do, let me know. :) > > Oh, they are a lot of things to do! My patches for PEP 509, 510 and > 511 still need some love (reviews): > > http://bugs.python.org/issue26058 > http://bugs.python.org/issue26098 > http://bugs.python.org/issue26145 > > I'm finishing my patch adding ast.Constant. This one is less > controversal, it has no impact on performance nor the Python > semantics: > > http://bugs.python.org/issue26146 > > > But these patches are boring C code. You may prefer to work on the > funny fatoptimizer project which is written in pure Python: > > https://fatoptimizer.readthedocs.org/en/latest/ > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com > -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jan 25 21:34:43 2016 From: brett at python.org (Brett Cannon) Date: Tue, 26 Jan 2016 02:34:43 +0000 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: On Mon, 25 Jan 2016 at 18:22 INADA Naoki wrote: > I'm very interested in it. > > Ruby 2.2 and PHP 7 are faster than Python 2. > Python 3 is slower than Python 2. > Performance is a attractive feature. Python 3 lacks it. > That is not a fair statement to make about Python 3. It entirely depends on your workload whether it is faster or slower. https://gist.github.com/brettcannon/9d19cc184ea45b3e7ca0 -Brett > > How can I help your work? > > On Tue, Jan 26, 2016 at 7:58 AM, Victor Stinner > wrote: > >> 2016-01-25 22:51 GMT+01:00 Sven R. Kunze : >> > - they provide a great infrastructure for optimizing CPython AND >> > extending/experimenting Python as an ecosystem >> >> I hope that these API will create more optimizer projects than just >> fatoptimizer. >> >> For example, I expect more specialized optimizers like numba or >> pythran which are very efficient but more specific (ex: numeric >> computations) than fatoptimizer. Maybe not new optimizers, but just >> glue to existing static compilers (numba, pythran, cython, etc.). >> >> >> > If there's anything I can do, let me know. :) >> >> Oh, they are a lot of things to do! My patches for PEP 509, 510 and >> 511 still need some love (reviews): >> >> http://bugs.python.org/issue26058 >> http://bugs.python.org/issue26098 >> http://bugs.python.org/issue26145 >> >> I'm finishing my patch adding ast.Constant. This one is less >> controversal, it has no impact on performance nor the Python >> semantics: >> >> http://bugs.python.org/issue26146 >> >> >> But these patches are boring C code. You may prefer to work on the >> funny fatoptimizer project which is written in pure Python: >> >> https://fatoptimizer.readthedocs.org/en/latest/ >> >> Victor >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com >> > > > > -- > INADA Naoki > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Jan 25 22:02:52 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Jan 2016 19:02:52 -0800 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: On Jan 25, 2016, at 18:21, INADA Naoki wrote: > > I'm very interested in it. > > Ruby 2.2 and PHP 7 are faster than Python 2. > Python 3 is slower than Python 2. Says who? That was certainly true in the 3.2 days, but nowadays, most things that differ seem to be faster in 3.x. Maybe it's just the kinds of programs I write, but speedup in decoding UTF-8 that's usually ASCII (and then processing the decoded unicode when it's usually 1/4th the size), faster listcomps, and faster datetime seem to matter more than slower logging or slower imports. And that's just when running the same code; when you actually use new features, yield from is much faster than looping over yield; scandir blows away listdir; asyncio blows away asyncore or threading even harder; etc. Maybe if you do different things, you have a different experience. But if you have a specific problem, you'd do a lot better to file specific bugs for that problem than to just hope that everything magically gets so much faster that your bottleneck no longer matters. > Performance is a attractive feature. Python 3 lacks it. When performance matters, people don't use Python 2, Ruby, or PHP, any more than they use Python 3. Or, rather, they use _any_ of those languages for the 95% of their code that doesn't matter, and C (often through existing libraries like NumPy--and try to find a good equivalent of that for Ruby or PHP) for the 5% that does. From songofacandy at gmail.com Mon Jan 25 22:32:07 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 26 Jan 2016 12:32:07 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: On Tue, Jan 26, 2016 at 12:02 PM, Andrew Barnert wrote: > On Jan 25, 2016, at 18:21, INADA Naoki wrote: > > > > I'm very interested in it. > > > > Ruby 2.2 and PHP 7 are faster than Python 2. > > Python 3 is slower than Python 2. > > Says who? > For example, http://benchmarksgame.alioth.debian.org/u64q/php.html In Japanese, many people compares language performance by microbench like fibbonacci. > > That was certainly true in the 3.2 days, but nowadays, most things that > differ seem to be faster in 3.x. Python is little faster than ever in these years. But PHP and Ruby are much more faster than these years. Matz announced Ruby 3x3. Ruby hackers will make more effort to optimize ruby. http://engineering.appfolio.com/appfolio-engineering/2015/11/18/ruby-3x3 > Maybe it's just the kinds of programs I write, but speedup in decoding > UTF-8 that's usually ASCII (and then processing the decoded unicode when > it's usually 1/4th the size), faster listcomps, and faster datetime seem to > matter more than slower logging or slower imports. And that's just when > running the same code; when you actually use new features, yield from is > much faster than looping over yield; scandir blows away listdir; asyncio > blows away asyncore or threading even harder; etc. > I know. But people compares language speed by simple microbench like fibbonacci. They doesn't use listcomp or libraries to compare *language* speed. > Maybe if you do different things, you have a different experience. But if > you have a specific problem, you'd do a lot better to file specific bugs > for that problem than to just hope that everything magically gets so much > faster that your bottleneck no longer matters. > I did it sometimes. But I'd like to base language performance like function call more faster. > > > Performance is a attractive feature. Python 3 lacks it. > > When performance matters, people don't use Python 2, Ruby, or PHP, any > more than they use Python 3. Or, rather, they use _any_ of those languages > for the 95% of their code that doesn't matter, and C (often through > existing libraries like NumPy--and try to find a good equivalent of that > for Ruby or PHP) for the 5% that does. In the case of Web devs, many people choose main language from PHP, Ruby and Python. When peformance matters, they choose sub language from node.js, Go and Scala. While performance is not a matter when choosing first language, slowest of three makes bad impression and people feel less attractive about Python. -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Jan 25 22:34:35 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 25 Jan 2016 21:34:35 -0600 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: <8E5CCA97-D191-408A-8925-6A598397C622@gmail.com> On January 25, 2016 9:32:07 PM CST, INADA Naoki wrote: >On Tue, Jan 26, 2016 at 12:02 PM, Andrew Barnert >wrote: > >> On Jan 25, 2016, at 18:21, INADA Naoki >wrote: >> > >> > I'm very interested in it. >> > >> > Ruby 2.2 and PHP 7 are faster than Python 2. >> > Python 3 is slower than Python 2. >> >> Says who? >> > >For example, http://benchmarksgame.alioth.debian.org/u64q/php.html >In Japanese, many people compares language performance by microbench >like >fibbonacci. > ...does writing Fibonacci in a foreign language make a performance difference? Or did you mean "In Japan?" > >> >> That was certainly true in the 3.2 days, but nowadays, most things >that >> differ seem to be faster in 3.x. > > >Python is little faster than ever in these years. >But PHP and Ruby are much more faster than these years. > >Matz announced Ruby 3x3. Ruby hackers will make more effort to optimize >ruby. >http://engineering.appfolio.com/appfolio-engineering/2015/11/18/ruby-3x3 > > > >> Maybe it's just the kinds of programs I write, but speedup in >decoding >> UTF-8 that's usually ASCII (and then processing the decoded unicode >when >> it's usually 1/4th the size), faster listcomps, and faster datetime >seem to >> matter more than slower logging or slower imports. And that's just >when >> running the same code; when you actually use new features, yield from >is >> much faster than looping over yield; scandir blows away listdir; >asyncio >> blows away asyncore or threading even harder; etc. >> > >I know. >But people compares language speed by simple microbench like >fibbonacci. >They doesn't use listcomp or libraries to compare *language* speed. > > >> Maybe if you do different things, you have a different experience. >But if >> you have a specific problem, you'd do a lot better to file specific >bugs >> for that problem than to just hope that everything magically gets so >much >> faster that your bottleneck no longer matters. >> > >I did it sometimes. >But I'd like to base language performance like function call more >faster. > > >> >> > Performance is a attractive feature. Python 3 lacks it. >> >> When performance matters, people don't use Python 2, Ruby, or PHP, >any >> more than they use Python 3. Or, rather, they use _any_ of those >languages >> for the 95% of their code that doesn't matter, and C (often through >> existing libraries like NumPy--and try to find a good equivalent of >that >> for Ruby or PHP) for the 5% that does. > > >In the case of Web devs, many people choose main language from PHP, >Ruby >and Python. >When peformance matters, they choose sub language from node.js, Go and >Scala. > >While performance is not a matter when choosing first language, slowest >of >three makes bad impression >and people feel less attractive about Python. -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From rosuav at gmail.com Mon Jan 25 22:59:36 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 26 Jan 2016 14:59:36 +1100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: On Tue, Jan 26, 2016 at 2:32 PM, INADA Naoki wrote: > > I know. > But people compares language speed by simple microbench like fibbonacci. > They doesn't use listcomp or libraries to compare *language* speed. > Well, that's a stupid way to decide on a language. Here, look: Python is faster than C. Proof! rosuav at sikorsky:~$ time python3 fib.py 2880067194370816120 real 0m0.033s user 0m0.032s sys 0m0.000s rosuav at sikorsky:~$ cat fib.py import functools @functools.lru_cache() def fib(n): if n < 2: return n return fib(n-2) + fib(n-1) print(fib(90)) rosuav at sikorsky:~$ gcc fib.c && time ./a.out 1134903170 real 0m9.104s user 0m9.064s sys 0m0.000s rosuav at sikorsky:~$ cat fib.c #include unsigned long fib(unsigned long n) { if (n < 2) return n; return fib(n-2) + fib(n-1); } int main() { printf("%lu\n",fib(45)); } Algorithmic differences - even subtle ones - can easily outdo choice of language for run-time performance. And if you try to write a true C equivalent of that Python code, good luck - I'll have the Python one written and running while you're still trying to figure out how to write a cache, much less how to keep the syntax clean as you add a cache to an existing function. Of course, rewriting the whole thing to work iteratively instead of double-recursively will make a dramatic difference to both programs. That's an unsubtle algorithmic difference, and if you're coding like that, you probably can't see the difference in performance between any two languages at anything up to a machine word (about the 90th or so Fibonacci number, on a 64-bit system) - all you'll see is the startup performance. As soon as you go beyond a machine word, Python massively trumps C, because its default integer type is a bignum. Going beyond a machine word in C is a hassle. Going beyond a machine word in Python 2 is almost insignificant - hey look, now your repr has an 'L' on the end, and performance is immeasurably worse. In Python 3, there's no machine word whatsoever. So, yeah... Python beats C for Fibonacci calculation, too. You just have to not be stupid with your benchmarking. ChrisA From songofacandy at gmail.com Tue Jan 26 00:02:31 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 26 Jan 2016 14:02:31 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: Do you say I and many people are so fool? People use same algorithm on every language when compares base language performance [1]. [1] There are no solid definition about "Base language performance". But it includes function call, method lookup, GC. It may include basic string and arithmetic operations. See here for example: http://d.hatena.ne.jp/satosystems/20121228/1356655565 This article is written in 2012. In this article, php 5.3 takes 85sec, Python 2.7 takes 53sec and CRuby 1.8 takes 213sec. (!!) For now: $ python2 -V Python 2.7.11 $ time python2 -S fib.py 39088169 real 0m17.133s user 0m16.970s sys 0m0.055s $ python3 -V Python 3.5.1 $ time python3 -S fib.py 39088169 real 0m21.380s user 0m21.337s sys 0m0.028s $ php -v PHP 7.0.2 (cli) (built: Jan 7 2016 10:40:21) ( NTS ) Copyright (c) 1997-2015 The PHP Group Zend Engine v3.0.0, Copyright (c) 1998-2015 Zend Technologies $ time php fib.php 39088169 real 0m7.706s user 0m7.654s sys 0m0.027s $ ruby -v ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin14] $ time ruby fib.rb 39088169 real 0m6.195s user 0m6.124s sys 0m0.032s Fibonacci microbench measures performance of function call. When I said "Base language performance", I meant performance of function call, attribute lookup, GC, etc... PHP and Ruby made grate effort to improve base language performance. While I'm fan of Python, I respect people made PHP and Ruby faster. Of course, I respect people making Python faster too. But I wonder if CPython is more faster, especially about global lookup and function call. -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Jan 26 00:44:07 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Jan 2016 21:44:07 -0800 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: <9A44AFB5-CE53-4C5B-9A49-6E85F9C148A2@yahoo.com> On Jan 25, 2016, at 19:32, INADA Naoki wrote: > >> On Tue, Jan 26, 2016 at 12:02 PM, Andrew Barnert wrote: >> On Jan 25, 2016, at 18:21, INADA Naoki wrote: >> > >> > I'm very interested in it. >> > >> > Ruby 2.2 and PHP 7 are faster than Python 2. >> > Python 3 is slower than Python 2. >> >> Says who? > > For example, http://benchmarksgame.alioth.debian.org/u64q/php.html > In Japanese, many people compares language performance by microbench like fibbonacci. "In Japan, the hand is sharper than a knife [man splits board with karate chop], but the same doesn't work with a tomato [man splatters tomato all over himself with karate chop]." A cheap knife really is better than a karate master at chopping tomatoes. And Python 2 really is better than Python 3 at doing integer arithmetic on the edge of what can fit into a machine word. But so what? Without seeing any of your Japanese web code, much less running a profiler, I'm willing to bet that your code is rarely CPU-bound, and, when it is, it spends a lot more time doing things like processing Unicode strings that are almost always UCS-2 (about 110% slower on Python 2) than doing this kind of arithmetic (9% faster on Python 2), or cutting tomatoes (TypeError on both versions). -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Tue Jan 26 01:13:19 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 26 Jan 2016 15:13:19 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <9A44AFB5-CE53-4C5B-9A49-6E85F9C148A2@yahoo.com> References: <56A698CF.3080402@mail.de> <9A44AFB5-CE53-4C5B-9A49-6E85F9C148A2@yahoo.com> Message-ID: On Tue, Jan 26, 2016 at 2:44 PM, Andrew Barnert wrote: > On Jan 25, 2016, at 19:32, INADA Naoki wrote: > > On Tue, Jan 26, 2016 at 12:02 PM, Andrew Barnert > wrote: > >> On Jan 25, 2016, at 18:21, INADA Naoki wrote: >> > >> > I'm very interested in it. >> > >> > Ruby 2.2 and PHP 7 are faster than Python 2. >> > Python 3 is slower than Python 2. >> >> Says who? >> > > For example, http://benchmarksgame.alioth.debian.org/u64q/php.html > In Japanese, many people compares language performance by microbench like > fibbonacci. > > > "In Japan, the hand is sharper than a knife [man splits board with karate > chop], but the same doesn't work with a tomato [man splatters tomato all > over himself with karate chop]." > > A cheap knife really is better than a karate master at chopping tomatoes. > And Python 2 really is better than Python 3 at doing integer arithmetic on > the edge of what can fit into a machine word. But so what? Without seeing > any of your Japanese web code, much less running a profiler, I'm willing to > bet that your code is rarely CPU-bound, and, when it is, it spends a lot > more time doing things like processing Unicode strings that are almost > always UCS-2 (about 110% slower on Python 2) than doing this kind of > arithmetic (9% faster on Python 2), or cutting tomatoes (TypeError on both > versions). > > Calm down, please. I didn't say "microbench is more important than macrobench". While editor is not a main problem of software development, people likes comparing vim and emacs. Like that, Japanese dev people likes comparing speed. While it's not a real problem of typical application, new people should choose first (and probably main) editor and language. Slowest on such a basic microbench gives bad impression for them. Additionally, some application (e.g. traversing DOM) makes much function calls. Faster function call may makes some *real* application faster. -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Jan 26 02:16:15 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 26 Jan 2016 16:16:15 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: <22183.7487.315172.752001@turnbull.sk.tsukuba.ac.jp> INADA Naoki writes: > For example, http://benchmarksgame.alioth.debian.org/u64q/php.html > In Japanese, many people compares language performance by > microbench like fibbonacci. True enough. But as a teacher in a Japanese engineering school, I am ashamed to see that posted to a public list. We Japanese ;-) need to get over that tendency (and not just in programming, my own biggest battle is against those who worship goodness-of-fit statistics). Our universities are doing an awful job at getting "big picture thinking" across to our students. I see that you respond that you're talking about "base language performance" which is a big improvement over blind faith in micro- benchmarks, but still it's a matter of pulling your nose back from the bark so you can see the tree. At least from the point of view of the computations the people around me are doing with these languages, you still miss the forest. > While performance is not a matter when choosing first language, > slowest of three makes bad impression and people feel less > attractive about Python. My school is the one that Matz graduated from (different department), and as far as I can see (which is pretty local, I admit), Python is a clear first and growing as a teaching language and as a glue language for research computations. Java is second for both[1], Ruby is rare, and PHP just isn't on the map. I see some lab home pages using Ruby-on-Rails, and a couple of Mediawiki installations and the like based on PHP. For me personally, PHP and Ruby are both unusable for *performance* reasons: I need something like Pandas, which is based on NumPy, and at least the last time I checked a couple years ago Ruby had no equivalent to NumPy, let alone Pandas. (For PHP I asked the nearest PHP fanboy web programmer, so I could be way wrong about no efficient computation for PHP.) At least as used at this university, I just don't see a performance issue that Python needs to be specifically worried about. That said, optimization is important, and IMO anything that allows 1% of people currently writing (and debugging ...) modules in C for performance reasons to write them in pure Python instead is a win. But to me, relieving programmer pain, not microbenchmarks, should be the performance measure we care about (even if others do care about those benchmarks). Measure them, yes, but focus on them only when we're pretty sure that poor benchmark numbers are associated with programmer pain. Footnotes: [1] Java was clearly first as a teaching language until 2010 or so, and since most of our grad students and assistant profs were students here in that era, I suspect its persistence is just the inertia that goes with academic nepotism. From ncoghlan at gmail.com Tue Jan 26 05:06:03 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Jan 2016 20:06:03 +1000 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <22183.7487.315172.752001@turnbull.sk.tsukuba.ac.jp> References: <56A698CF.3080402@mail.de> <22183.7487.315172.752001@turnbull.sk.tsukuba.ac.jp> Message-ID: On 26 January 2016 at 17:16, Stephen J. Turnbull wrote: > Our universities are doing an awful job at getting "big picture thinking" > across to our students. That problem isn't specific to Japan - I'm not aware of *anywhere* that does a particularly good job of teaching developers not to get tribal about their programming language choice, and instead evaluate their options based on their specific problem, the team that will be working on it, and the pre-existing problem solving tools available in that ecosystem. As a result, folks making programming language choices based on criteria that aren't actually relevant to the problem they're trying to solve is going to be a fact of life. While improving those kinds of metrics isn't a *reason* to do anything, it does count as an added bonus when it comes as a beneficial side effect of working on something else (such as the function specialisation infrastructure underpinning Victor's optimiser project). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Tue Jan 26 08:38:13 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Jan 2016 08:38:13 -0500 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: On 1/26/2016 12:02 AM, INADA Naoki wrote: > People use same algorithm on every language when compares base language > performance [1]. The python code is NOT using the same algorithm. The proof is that the Python function will return the correct value for, say fib(50) while most if not all the other versions will not. The domain of an algorithm is part of what characterizes an algorithm. -- Terry Jan Reedy From rymg19 at gmail.com Tue Jan 26 10:28:42 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 26 Jan 2016 09:28:42 -0600 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: On January 25, 2016 9:59:36 PM CST, Chris Angelico wrote: >On Tue, Jan 26, 2016 at 2:32 PM, INADA Naoki >wrote: >> >> I know. >> But people compares language speed by simple microbench like >fibbonacci. >> They doesn't use listcomp or libraries to compare *language* speed. >> > >Well, that's a stupid way to decide on a language. Here, look: Python >is faster than C. Proof! > >rosuav at sikorsky:~$ time python3 fib.py >2880067194370816120 > >real 0m0.033s >user 0m0.032s >sys 0m0.000s >rosuav at sikorsky:~$ cat fib.py >import functools > >@functools.lru_cache() >def fib(n): > if n < 2: return n > return fib(n-2) + fib(n-1) > >print(fib(90)) > >rosuav at sikorsky:~$ gcc fib.c && time ./a.out >1134903170 > >real 0m9.104s >user 0m9.064s >sys 0m0.000s >rosuav at sikorsky:~$ cat fib.c >#include > >unsigned long fib(unsigned long n) >{ > if (n < 2) return n; > return fib(n-2) + fib(n-1); >} > >int main() >{ > printf("%lu\n",fib(45)); >} > *cough* -O3 *cough* > >Algorithmic differences - even subtle ones - can easily outdo choice >of language for run-time performance. And if you try to write a true C >equivalent of that Python code, good luck - I'll have the Python one >written and running while you're still trying to figure out how to >write a cache, much less how to keep the syntax clean as you add a >cache to an existing function. > >Of course, rewriting the whole thing to work iteratively instead of >double-recursively will make a dramatic difference to both programs. >That's an unsubtle algorithmic difference, and if you're coding like >that, you probably can't see the difference in performance between any >two languages at anything up to a machine word (about the 90th or so >Fibonacci number, on a 64-bit system) - all you'll see is the startup >performance. As soon as you go beyond a machine word, Python massively >trumps C, because its default integer type is a bignum. Going beyond a >machine word in C is a hassle. Going beyond a machine word in Python 2 >is almost insignificant - hey look, now your repr has an 'L' on the >end, and performance is immeasurably worse. In Python 3, there's no >machine word whatsoever. > >So, yeah... Python beats C for Fibonacci calculation, too. You just >have to not be stupid with your benchmarking. > >ChrisA >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From victor.stinner at gmail.com Tue Jan 26 10:32:45 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 26 Jan 2016 16:32:45 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: Hi, 2016-01-26 3:21 GMT+01:00 INADA Naoki : > How can I help your work? I don't know exactly yet, but I started to write a documentation to explain how to contribute: http://faster-cpython.readthedocs.org/fat_python.html#how-can-you-contribute You may contact me directly ;-) Victor From rosuav at gmail.com Tue Jan 26 10:40:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 27 Jan 2016 02:40:08 +1100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: On Wed, Jan 27, 2016 at 2:28 AM, Ryan Gonzalez wrote: >>rosuav at sikorsky:~$ gcc fib.c && time ./a.out >>1134903170 >> >>real 0m9.104s >>user 0m9.064s >>sys 0m0.000s >>rosuav at sikorsky:~$ cat fib.c >>#include >> >>unsigned long fib(unsigned long n) >>{ >> if (n < 2) return n; >> return fib(n-2) + fib(n-1); >>} >> >>int main() >>{ >> printf("%lu\n",fib(45)); >>} >> > > *cough* -O3 *cough* > Oh, I'm sorry. Let's try it again. rosuav at sikorsky:~$ gcc -O3 fib.c && time ./a.out 1134903170 real 0m3.153s user 0m3.088s sys 0m0.052s Cool! Seems to be linear. From which we can deduce that Python on my system was compiled at about -O275. ChrisA From srkunze at mail.de Tue Jan 26 12:10:59 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 26 Jan 2016 18:10:59 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: <56A7A8A3.5040705@mail.de> Hi, will look into it soon. :) Best, Sven On 26.01.2016 16:32, Victor Stinner wrote: > Hi, > > 2016-01-26 3:21 GMT+01:00 INADA Naoki : >> How can I help your work? > I don't know exactly yet, but I started to write a documentation to > explain how to contribute: > http://faster-cpython.readthedocs.org/fat_python.html#how-can-you-contribute > > You may contact me directly ;-) > > Victor From srkunze at mail.de Tue Jan 26 12:35:56 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 26 Jan 2016 18:35:56 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: <56A7AE7C.3080806@mail.de> I completely agree with INADA. It's like saying, because a specific crossroad features a higher accident rate, *people need to change their driving behavior*. *No!* People won't change and it's not necessary either. The crossroad needs to be changed to be safer. Same goes for Python. If it's slow using the very same piece of code (even superficially), you better make the language faster. Developers won't change and they won't change their code either. Just not necessary. Btw. it would be a great feature for Python 3 to be faster than Python 2. I've heard a lot of complaints of the scientific community that Python is slow. Would Python 3 be significantly faster than Python 2, that'll be a huge reason to upgrade (and would create pressure to upgrade libs as well). They are satisfied with Python so far, but would there be a language equally readable/maintainable and 10x faster (of course proven by some weird micro benchmarks - incomprehensible to most nervous subscribers to this list), they would readily switch over. I for one hope that *Python itself will be that language* in the foreseeable future. This is some sort of marketing but also requires hard facts indeed. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Jan 26 12:50:13 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 27 Jan 2016 02:50:13 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <22183.7487.315172.752001@turnbull.sk.tsukuba.ac.jp> Message-ID: <22183.45525.814657.897955@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > On 26 January 2016 at 17:16, Stephen J. Turnbull wrote: > > Our universities are doing an awful job at getting "big picture > > thinking" across to our students. > > That problem isn't specific to Japan - I'm not aware of *anywhere* > that does a particularly good job of teaching developers not to get > tribal about their programming language choice, But that's a different issue. The approach that Naoki describes isn't "tribal", in fact it's the exact opposite: it's an attempt to base such decisions on strictly objective measures. > and instead evaluate their options based on their specific problem, > the team that will be working on it, and the pre-existing problem > solving tools available in that ecosystem. One of which is the language itself, and the team's experience with it. "We're a C++/Java/Python/Ruby/Brainf!ck/assembler/COBOL shop" isn't a bad heuristic in most cases -- outside of research and/or "we also walk dogs" generalist consultancies -- especially when you're under pressure from the bean-counting department to reduce costs. That heuristic is hard to distinguish from "tribalism", though. AFAICT, in fact, the generalists (including massive entities like IBM and Google, as well as companies like Red Hat and cooperatives like Debian) are quite good at the kind of evaluation you describe. And Python has been quite good at targeting the kinds of improvements that make it appealing to people who can and do do that kind of evaluation, in more and more areas. > While improving those kinds of metrics isn't a *reason* to do > anything, it does count as an added bonus when it comes as a > beneficial side effect of working on something else (such as the > function specialisation infrastructure underpinning Victor's > optimiser project). Sure, but that's a "have your cake and eat it too" situation. Nobody's going to complain about that! If Naoki -- or anybody else -- wants to work on optimizations enabled by FAT Python, more power to them, I say. I just would like to see them reviewed with the goal of making Python a better language for solving a wide variety of problems, rather than strictly focusing on benchmarks. If the benchmarks can be improved without closing off potential syntax improvements or restricting the domain of algorithms (cf Terry's comment), wonderful! I thought Chris's point about efficient algorithms that would be hard to implement correctly in other languages but are easy to do in Python *because* of Python's carefully crafted, but not always maximally speedy, semantics was especially apropos here. Of course his claim that Python is faster than C is tongue in cheek, and a caching version of fibonacci wouldn't be that hard to write in C, and an iterative version even easier. But others have pointed out many other syntaxes (comprehensions, generators, yield from, and so on) that put together often make efficient computation TOOWTDI. That, to me, is The Python Advantage. From stephen at xemacs.org Tue Jan 26 12:51:54 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 27 Jan 2016 02:51:54 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp> Terry Reedy writes: > On 1/26/2016 12:02 AM, INADA Naoki wrote: > > > People use same algorithm on every language when compares base language > > performance [1]. > > The python code is NOT using the same algorithm. The proof is that the > Python function will return the correct value for, say fib(50) while > most if not all the other versions will not. True, but that's not a reasonable criterion for "same algorithm" in this context. Naoki's application ("base language performance" benchmarking) requires fib(n) only for n < 40, and run it in a loop 100 times if you want 2 more decimal places of precision ("40" is appropriate for an implementation with 32-bit ints). On that restricted domain the algorithm *is* the same. If you want to argue that the bigger domain is a better one to use for evaluating programming languages, be our guest. But then you're comparing apples (speed) to oranges (domain), and Naoki (or the Japanese benchmarkers) can argue that a smaller, more risky, domain is covered by "consenting adults" -- if you know there's a risk, you need to write code to deal with it, but if you know there isn't, you shouldn't have to accept lower performance. Obviously, I don't think that's an appropriate tradeoff myself, but that's based on "IMHO" not "comparison is invalid because algorithms differ". From truong_nguyen8 at yahoo.com Tue Jan 26 18:18:08 2016 From: truong_nguyen8 at yahoo.com (Truong Nguyen) Date: Tue, 26 Jan 2016 23:18:08 +0000 (UTC) Subject: [Python-Dev] Hoping to Find a Mentor References: <970197771.523029.1453850288839.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <970197771.523029.1453850288839.JavaMail.yahoo@mail.yahoo.com> Dear Everyone, My name is Truong Nguyen and I like web development. I'm writing hoping to find a mentor (who likes to teach), and opportunity to contribute to the python.org website to gain skills in full stack development. Python Software Foundation interest me because the language is used in a wide variety of fields. My background is not in computer science, but I've earned a B.S. in Applied Mathematics and have an equivalent minor in computer science. I did front end project with HTML, CSS3 & Media Query, and JavaScript myself, but hope to work with someone to learn from. I'm a hard worker and responsible person and do my best to contribute to the organization. Have a nice day, Truong Nguyen -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at snarky.ca Tue Jan 26 12:31:16 2016 From: brett at snarky.ca (Brett Cannon) Date: Tue, 26 Jan 2016 17:31:16 +0000 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (cbd4a6a2657e): sum=134 In-Reply-To: <20160126084734.122727.74008@psf.io> References: <20160126084734.122727.74008@psf.io> Message-ID: Looks like Victor's ast.Constant change introduced a refleak. On Tue, 26 Jan 2016 at 00:47 wrote: > results for cbd4a6a2657e on branch "default" > -------------------------------------------- > > test_ast leaked [39, 39, 39] references, sum=117 > test_ast leaked [5, 5, 5] memory blocks, sum=15 > test_collections leaked [-2, 0, 0] references, sum=-2 > test_functools leaked [0, 2, 2] memory blocks, sum=4 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', > '3:3:/home/psf-users/antoine/refleaks/reflogqIZGVY', '--timeout', '7200'] > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sjoerdjob at sjec.nl Wed Jan 27 00:23:41 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Wed, 27 Jan 2016 06:23:41 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> Message-ID: <20160127052341.GA14190@sjoerdjob.com> On Mon, Jan 25, 2016 at 11:58:12PM +0100, Victor Stinner wrote: > ... > Oh, they are a lot of things to do! ... Just wondering, do you also need a set of (abusive) test-cases which check 100% conformity to the CPython semantics? I'm sure many of us would be able to whip up some ideas of things that are possible with Python and are of the kind "but you should not do that! That's bad programming!" which may or may not break the optimizations (especially specialized functions). I'm thinking of things like def override_length_function_test(): global len value = len("abc") len = lambda obj: ord(obj[0])) value += len("abc") assert value == 3 + 97, "Outdated `len` used." And also cases where `len` was changed not in the function itself, but in another function that was called indirectly (maybe 4 functions down, one of which was monkey-patched in after the compilation step): module_a.py def test_func(callback): value = len("abc") callback() value += len("abc") assert value == 3 + 97, "Outdated `len` used." module_b.py import module_a def override_length_function_test(): def callback(): module_a.len = lambda obj: ord(obj[0]) assert module_a.test_func(callback) == 3 + 97, "Outdated `len` used." (I'm sure some of the other readers of this list can be even more creative in trying to show that making optimizations like this can break semantics.) Other fun tricks I'd like to try is overriding the `len` method from a signal handler, what happens when you monkey-patch a dependent method, having `__getitem__` and `__getattr__` on some value override `len`. Basically: trying things that I normally should not try during my working hours, on account of wanting to still have a job the next day. Kind regards, Sjoerd Job From victor.stinner at gmail.com Wed Jan 27 03:09:36 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 27 Jan 2016 09:09:36 +0100 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (cbd4a6a2657e): sum=134 In-Reply-To: References: <20160126084734.122727.74008@psf.io> Message-ID: Hi, I pushed a fix before you sent your message. At least test_ast should be fixed. https://hg.python.org/cpython/rev/c5df914e73ad FYI I'm unable to reproduce the test_collections leak. Victor Le mardi 26 janvier 2016, Brett Cannon a ?crit : > Looks like Victor's ast.Constant change introduced a refleak. > > On Tue, 26 Jan 2016 at 00:47 > wrote: > >> results for cbd4a6a2657e on branch "default" >> -------------------------------------------- >> >> test_ast leaked [39, 39, 39] references, sum=117 >> test_ast leaked [5, 5, 5] memory blocks, sum=15 >> test_collections leaked [-2, 0, 0] references, sum=-2 >> test_functools leaked [0, 2, 2] memory blocks, sum=4 >> >> >> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', >> '3:3:/home/psf-users/antoine/refleaks/reflogqIZGVY', '--timeout', '7200'] >> _______________________________________________ >> Python-checkins mailing list >> Python-checkins at python.org >> >> https://mail.python.org/mailman/listinfo/python-checkins >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Jan 27 04:28:15 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 27 Jan 2016 09:28:15 +0000 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <20160127052341.GA14190@sjoerdjob.com> References: <56A698CF.3080402@mail.de> <20160127052341.GA14190@sjoerdjob.com> Message-ID: On 27 January 2016 at 05:23, Sjoerd Job Postmus wrote: > On Mon, Jan 25, 2016 at 11:58:12PM +0100, Victor Stinner wrote: >> ... >> Oh, they are a lot of things to do! ... > > Just wondering, do you also need a set of (abusive) test-cases which > check 100% conformity to the CPython semantics? I'm sure many of us > would be able to whip up some ideas of things that are possible with > Python and are of the kind "but you should not do that! That's bad > programming!" which may or may not break the optimizations (especially > specialized functions). > > I'm thinking of things like > > def override_length_function_test(): > global len > value = len("abc") > len = lambda obj: ord(obj[0])) > value += len("abc") > assert value == 3 + 97, "Outdated `len` used." > > And also cases where `len` was changed not in the function itself, but > in another function that was called indirectly (maybe 4 functions down, > one of which was monkey-patched in after the compilation step): > > module_a.py > def test_func(callback): > value = len("abc") > callback() > value += len("abc") > assert value == 3 + 97, "Outdated `len` used." > > module_b.py > import module_a > def override_length_function_test(): > def callback(): > module_a.len = lambda obj: ord(obj[0]) > assert module_a.test_func(callback) == 3 + 97, "Outdated `len` used." > > (I'm sure some of the other readers of this list can be even more > creative in trying to show that making optimizations like this can break > semantics.) > > Other fun tricks I'd like to try is overriding the `len` method from a > signal handler, what happens when you monkey-patch a dependent method, > having `__getitem__` and `__getattr__` on some value override `len`. > > Basically: trying things that I normally should not try during my > working hours, on account of wanting to still have a job the next day. > > Kind regards, > Sjoerd Job Maybe I'm just nasty, but IMO those kinds of "torture tests" are just as valuable in general, so I'd encourage people to submit patches to the main Python test suite to add them. Paul From victor.stinner at gmail.com Wed Jan 27 04:31:54 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 27 Jan 2016 10:31:54 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <20160127052341.GA14190@sjoerdjob.com> Message-ID: Python has test_dynamic which tests such corner cases. For example, test_modify_builtins_while_generator_active(): "Modify the builtins out from under a live generator." https://hg.python.org/cpython/file/58266f5101cc/Lib/test/test_dynamic.py#l49 Victor 2016-01-27 10:28 GMT+01:00 Paul Moore : > On 27 January 2016 at 05:23, Sjoerd Job Postmus wrote: >> On Mon, Jan 25, 2016 at 11:58:12PM +0100, Victor Stinner wrote: >>> ... >>> Oh, they are a lot of things to do! ... >> >> Just wondering, do you also need a set of (abusive) test-cases which >> check 100% conformity to the CPython semantics? I'm sure many of us >> would be able to whip up some ideas of things that are possible with >> Python and are of the kind "but you should not do that! That's bad >> programming!" which may or may not break the optimizations (especially >> specialized functions). >> >> I'm thinking of things like >> >> def override_length_function_test(): >> global len >> value = len("abc") >> len = lambda obj: ord(obj[0])) >> value += len("abc") >> assert value == 3 + 97, "Outdated `len` used." >> >> And also cases where `len` was changed not in the function itself, but >> in another function that was called indirectly (maybe 4 functions down, >> one of which was monkey-patched in after the compilation step): >> >> module_a.py >> def test_func(callback): >> value = len("abc") >> callback() >> value += len("abc") >> assert value == 3 + 97, "Outdated `len` used." >> >> module_b.py >> import module_a >> def override_length_function_test(): >> def callback(): >> module_a.len = lambda obj: ord(obj[0]) >> assert module_a.test_func(callback) == 3 + 97, "Outdated `len` used." >> >> (I'm sure some of the other readers of this list can be even more >> creative in trying to show that making optimizations like this can break >> semantics.) >> >> Other fun tricks I'd like to try is overriding the `len` method from a >> signal handler, what happens when you monkey-patch a dependent method, >> having `__getitem__` and `__getattr__` on some value override `len`. >> >> Basically: trying things that I normally should not try during my >> working hours, on account of wanting to still have a job the next day. >> >> Kind regards, >> Sjoerd Job > > Maybe I'm just nasty, but IMO those kinds of "torture tests" are just > as valuable in general, so I'd encourage people to submit patches to > the main Python test suite to add them. > > Paul From victor.stinner at gmail.com Wed Jan 27 04:39:43 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 27 Jan 2016 10:39:43 +0100 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? In-Reply-To: References: Message-ID: 2016-01-23 7:03 GMT+01:00 Chris Angelico : > I just had a major crash on the system that hosts the > angelico-debian-amd64 buildbot, and as usual, checked it carefully > after bringing everything up. It seems now to be timing out after an > hour of operation: > > http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/3132/steps/test/logs/stdio I opened http://bugs.python.org/issue26206 to track this issue. > Running just that test file: > > $ ./python Lib/test/test_socket.py > ... chomp lots of lines ... > testRecvmsgPeek (__main__.RecvmsgUDP6Test) ... > > seems to indicate that the stall is due to IPv6 and UDP. The VM should > have full IPv6 support, although my ISPs don't carry IPv6 traffic, so > it won't be able to reach the internet proper; but it should be able > to do all manner of local tests. Try to apply attached patch and run: $ ./python -m test -v -m testRecvmsgPeek test_socket (...) testRecvmsgPeek (test.test_socket.RecvmsgUDP6Test) ... CLI SOCK SERV SOCK CLI SOCK ('::1', 44347, 0, 0) SERV SOCK ('::1', 40488, 0, 0) ok testRecvmsgPeek (test.test_socket.RecvmsgIntoUDP6Test) ... CLI SOCK SERV SOCK CLI SOCK ('::1', 52721, 0, 0) SERV SOCK ('::1', 43967, 0, 0) ok (...) As you can see: the test uses the local loopback interface. Inet6TestBase.host is "::1". You can try to run a UDP server using netcat: "nc -l -u ::1 12345". Keep the command running in a terminal, and then run in a different terminal: "echo abc | nc -u ::1 12345". You should receive abc in the server. Victor -------------- next part -------------- A non-text attachment was scrubbed... Name: debug.patch Type: text/x-patch Size: 928 bytes Desc: not available URL: From tjreedy at udel.edu Wed Jan 27 05:59:37 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 27 Jan 2016 05:59:37 -0500 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <56A7AE7C.3080806@mail.de> References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> Message-ID: On 1/26/2016 12:35 PM, Sven R. Kunze wrote: > I completely agree with INADA. I an not sure you do. > It's like saying, because a specific crossroad features a higher > accident rate, *people need to change their driving behavior*. > *No!* People won't change and it's not necessary either. The crossroad > needs to be changed to be safer. Safer crossroads tend to be slower unless one switched to alternate designs that eliminate crossing streams of traffic. Python is safer, in everyday use as well as in artificial benchmarks, and is slower as a result. Languages that don't have integers but use residue classes (with wraparound) or finite integer classes (with overflow) as a (faster) substitute have, in practice, lots of accidents (bugs) when used by non-experts. Guido noticed this, gave up on changing coder behavior, and put the expert behavior of checking for wraparound/overflow and switching to real integers (longs) into the language. (I forget when this was added.) The purpose of the artificially low input to fib() is to hide and avoid the bugginess of most languages. The analogous trick with testing crossroads would be to artificially restrict the density of cars to mask the accident-proneness of a 'fast, consenting-adults' crossroads with no stop signs and no stop lights. > Same goes for Python. If it's slow using the very same piece of code > (even superficially), you better make the language faster. > Developers won't change and they won't change their code either. Just > not necessary. Instead of making people rewrite fib to dramatically increase speed, we added the lru-cache decorator to get most of the benefit without a rewrite. But Inada rejected this Python speedup. An ast optimizer could potentially do the same speedup without the explicit decorator. (No side-effects? Multiple recursive calls? Add a cache!) > Btw. it would be a great feature for Python 3 to be faster than Python > 2. We all agree on that. One way for this to happen is to add optimizers that would make Python 'cheat' on micrebenchmarks -- Terry Jan Reedy From ncoghlan at gmail.com Wed Jan 27 06:16:19 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Jan 2016 21:16:19 +1000 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <56A7AE7C.3080806@mail.de> References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> Message-ID: On 27 January 2016 at 03:35, Sven R. Kunze wrote: > I completely agree with INADA. > > It's like saying, because a specific crossroad features a higher accident > rate, people need to change their driving behavior. > No! People won't change and it's not necessary either. The crossroad needs > to be changed to be safer. Umm, no, that's not how this works - developers contribute to community driven projects for their *own* reasons. Nobody gets to tell them what to do unless they're paying them. Micro-optimising a poor algorithm won't deliver macro level improvements because macro level code uses things like space-speed trade-offs to improve the algorithmic efficiency (as in the example of applying functools.lru_cache to a naive recursive fibonacci implementation). Victor's work on FAT optimiser is interesting because it offers opportunities to speed up even code that is already algorithmically efficient, as well as making CPython a better platform for experimenting with those kinds of changes. More generally though, much larger opportunities for improvement lie in persuading people to *stop writing code*, and instead spending more of their time on finding and assembling code other people have *already written* into solutions to interesting problems. *That's* the kind of improvement that turns enormously complex problems like facial recognition into 25 line Python scripts: https://realpython.com/blog/python/face-recognition-with-python/ Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Wed Jan 27 07:16:45 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 27 Jan 2016 07:16:45 -0500 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp> References: <56A698CF.3080402@mail.de> <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp> Message-ID: On 1/26/2016 12:51 PM, Stephen J. Turnbull wrote: > Terry Reedy writes: > > On 1/26/2016 12:02 AM, INADA Naoki wrote: > > > > > People use same algorithm on every language when compares base language > > > performance [1]. > > > > The python code is NOT using the same algorithm. The proof is that the > > Python function will return the correct value for, say fib(50) while > > most if not all the other versions will not. Let me try to be clearer. 1. Like everyone else, I would like Python function calls to be faster either in general or in special cases detected during compilation. This will require micro-benchmark for function calls that do just that. First time an empty loop, then time a loop with a call to an empty function. Do the same for various signatures, and maybe other special cases. 2. Cross-language micro-benchmarks aimed at timing specific operations are tough. To run on multiple languages, they must be restricted to a lowest-common-denominator of features. It is impossible to make every implementation perform exactly the same set of operations. Some languages may bundle features together. Some languages and implementations have optimizations that avoid unneeded operations. Not all optimizatons can be turned off. 3. While there are trends as to the speed of implementations of a language, benchmarks time particular implementations. Shedskin would compile fib to a C function that runs much faster than fib with the restricted subset of CPython allowed for the benchmark. > True, but that's not a reasonable criterion for "same algorithm" in > this context. Naoki's application ("base language performance" > benchmarking) requires fib(n) only for n < 40, and run it in a loop > 100 times if you want 2 more decimal places of precision ("40" is > appropriate for an implementation with 32-bit ints). So you agree that the limit of 39 is not intrinsic to the fib function or its uses, but is an after-the-fact limit imposed to mask the bug proneness of using substitutes for integers. To my mind, a fairer and more useful benchmark of 'base language performance' based on fib would use a wider domain. The report would say that CPython (with lru_cache disallowed) is slow but works over a wide range of inputs, while some other implementations of other languages run faster for small inputs but fail catastrophically for larger inputs. Users could then make a more informed pick. Also see my answer to Sven Kunze. -- Terry Jan Reedy From rosuav at gmail.com Wed Jan 27 07:47:33 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 27 Jan 2016 23:47:33 +1100 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? In-Reply-To: References: Message-ID: On Wed, Jan 27, 2016 at 8:39 PM, Victor Stinner wrote: > 2016-01-23 7:03 GMT+01:00 Chris Angelico : >> Running just that test file: >> >> $ ./python Lib/test/test_socket.py >> ... chomp lots of lines ... >> testRecvmsgPeek (__main__.RecvmsgUDP6Test) ... >> >> seems to indicate that the stall is due to IPv6 and UDP. The VM should >> have full IPv6 support, although my ISPs don't carry IPv6 traffic, so >> it won't be able to reach the internet proper; but it should be able >> to do all manner of local tests. > > Try to apply attached patch and run: > > $ ./python -m test -v -m testRecvmsgPeek test_socket > (...) > testRecvmsgPeek (test.test_socket.RecvmsgUDP6Test) ... CLI SOCK > type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 44347, 0, 0)> > SERV SOCK type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 40488, 0, 0)> > CLI SOCK ('::1', 44347, 0, 0) > SERV SOCK ('::1', 40488, 0, 0) > ok > testRecvmsgPeek (test.test_socket.RecvmsgIntoUDP6Test) ... CLI SOCK > type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 52721, 0, 0)> > SERV SOCK type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 43967, 0, 0)> > CLI SOCK ('::1', 52721, 0, 0) > SERV SOCK ('::1', 43967, 0, 0) > ok > (...) > > As you can see: the test uses the local loopback interface. > Inet6TestBase.host is "::1". Confirmed. It does two tests of IPv4 which run just fine, and then: testRecvmsgPeek (test.test_socket.RecvmsgUDP6Test) ... CLI SOCK CLI SOCK ('::1', 47518, 0, 0) SERV SOCK SERV SOCK ('::1', 42421, 0, 0) and hangs until I interrupt it. > You can try to run a UDP server using netcat: "nc -l -u ::1 12345". > Keep the command running in a terminal, and then run in a different > terminal: "echo abc | nc -u ::1 12345". You should receive abc in the > server. (Oddly, the default netcat-traditional doesn't seem to support this, but installing netcat-openbsd adds IPv6 support.) Yep, and that works flawlessly. It's nothing weird about that particular port, either - nc can use 42421 without a problem. After digging through test_socket.py for over an hour (the MRO for RecvmsgUDP6Test is enormous!!), I've boiled the issue down to this: import socket MSG = b'asdf qwer zxcv' serv = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM) serv.bind(("::1", 0)) cli = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM) cli.bind(("::1", 0)) cli.sendto(MSG, serv.getsockname()) print(serv.recvmsg(len(MSG) - 3, 0, socket.MSG_PEEK)) print(serv.recvmsg(len(MSG), 0, socket.MSG_PEEK)) print(serv.recvmsg(len(MSG))) On my main system, this produces three lines of output: the first has truncated text, the second has full text, and the third also has full text. This proves that MSG_PEEK is working correctly. On the buildbot, though, the first one stalls out. Commenting that line out produces correct results - peek the full data, then read it, and all is well. Any idea why partial read on a datagram socket would sometimes stall? ChrisA From stephen at xemacs.org Wed Jan 27 12:12:16 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 28 Jan 2016 02:12:16 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp> Message-ID: <22184.64112.8831.955609@turnbull.sk.tsukuba.ac.jp> Terry Reedy writes: > So you agree that the limit of 39 is not intrinsic to the fib function > or its uses, but is an after-the-fact limit imposed to mask the bug > proneness of using substitutes for integers. I don't know what the limit used in the benchmark is, but it must be quite a bit lower than 50 for 32-bit integers and could be greater than 90 for 64-bit integers. And it's not "masking bugs", it's "respecting the domain of valid input", s'il vous plait. > To my mind, a fairer and more useful benchmark of 'base language > performance' based on fib would use a wider domain. "Fair", maybe. But why play this game at all? These benchmarks are simply not useful to users choosing languages, unless they already know the difficulties of interpreting benchmarks and are willing to expend the effort to account for them. Without that knowledge and effort, choosing a programming language based on microbenchmarks is like choosing a car based on the leg-length of the model sitting on the hood in the TV commercial. > The report would say that CPython (with lru_cache disallowed) is > slow but works over a wide range of inputs, No, the report would say "Use of this benchmark for cross-language comparison of function call speed is more or less inaccurate due to differences in representation of integers and in handling the possibility of exceptions in 'integer' arithmetic." You are picking one tiny difference, but there are potentially many, some quite a bit larger on the tested domain (for example, some languages may be able to optimize fib() to unboxed integers, in which case they'll blow away all those that don't). > Users could then make a more informed pick. My point in my reply to Nick is that users aren't making informed picks. If they were, we wouldn't even be thinking about having this conversation. I'm not sure what they are doing (maybe, as Nick suggests, justifying their "tribal" prejudices?), but it's not that. ;-) Sure, other things being equal, better benchmarks will improve runtime performance, but other things are so far from being equal even an economist can't say "ceteris paribus" here. To expand that point: I don't really see a point in users (ie, developers in Python and other such languages) looking at these benchmarks except for the fun of feeling like implementers, to be honest. Even the implementers shouldn't much care about cross- language benchmarks, except that when a "similar"[1] language does significantly better on a particular benchmark, it's often useful to wonder "how dey do dat?!" Typically the answer is "they 'cheat'" == "fail one of the properties we consider required", but sometimes it's "ooh, that's cute, and I bet we could make Python work the same way" or "urkh, we can't do *that* (yuck!) but we could FATten up Python with similar effect". (Let me take this opportunity to say "Thank you, Victor!") Of course in the case of a controlled experiment like "configure in Victor's changes and run the benchmarks to make sure they're not detectably slower", they're invaluable regression tests, and more or less valuable (ie, YMMV) as measures of improvement to compare to costs they may impose in other features or (even more fuzzy) in developer time. Footnotes: [1] Whatever that means.... From srkunze at mail.de Wed Jan 27 13:11:27 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 27 Jan 2016 19:11:27 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> Message-ID: <56A9084F.4090200@mail.de> On 27.01.2016 11:59, Terry Reedy wrote: > On 1/26/2016 12:35 PM, Sven R. Kunze wrote: >> I completely agree with INADA. > > I an not sure you do. > I am sure I am. He wants to solve a problem the way that is natural to him as a unique human being. >> It's like saying, because a specific crossroad features a higher >> accident rate, *people need to change their driving behavior*. >> *No!* People won't change and it's not necessary either. The crossroad >> needs to be changed to be safer. > > Safer crossroads tend to be slower unless one switched to alternate > designs that eliminate crossing streams of traffic. So Python can be safer AND faster ( = different design) if we try hard enough. > Languages that don't have integers but use residue classes (with > wraparound) or finite integer classes (with overflow) as a (faster) > substitute have, in practice, lots of accidents (bugs) when used by > non-experts. Guido noticed this, gave up on changing coder behavior, > and put the expert behavior of checking for wraparound/overflow and > switching to real integers (longs) into the language. (I forget when > this was added.) I am glad he did because it helps humans solve their problems in a natural way without artificial boundaries. :) > The purpose of the artificially low input to fib() is to hide and > avoid the bugginess of most languages. The analogous trick with > testing crossroads would be to artificially restrict the density of > cars to mask the accident-proneness of a 'fast, consenting-adults' > crossroads with no stop signs and no stop lights. > I am completely with you here, however I disagree about suspected hiding/avoiding mentality. You say: Python -> *no problem with big integers but slow at small integers* Other Language -> *faster but breaks at big integers* Yes. That's it. We haven't solved the human side, however. A human AGAIN would need to compromise on either speed or safety. My point is: it would be insanely great if Python could be more like "*fast AND no problem with big integers*". No compromise here (at least no noticeable). So, people could entirely *concentrate on their problem domain* without every worrying about such tiny little, nitty-gritty computer science details. I love computer science but people of other domains don't have the time nor the knowledge to decide properly. That's the reason why they might decide by using some weird micro-benchmarks. Just humans. >> Same goes for Python. If it's slow using the very same piece of code >> (even superficially), you better make the language faster. >> Developers won't change and they won't change their code either. Just >> not necessary. > > Instead of making people rewrite fib to dramatically increase speed, > we added the lru-cache decorator to get most of the benefit without a > rewrite. But Inada rejected this Python speedup. An ast optimizer > could potentially do the same speedup without the explicit decorator. > (No side-effects? Multiple recursive calls? Add a cache!) > Bingo! That's the spirit. Why that decorator in the first place? Hey, I mean, if I ever want to write some cryptic-looking source code with 3-letters abbreviations (LRU), I use Assembler again. But I discovered and love Python and I never want to go back when my problem domain does not require me to. So, when a machine can detect such an optimization, hell, do it, please. It's more likely that I apply it at the wrong function AND only in 10% of the correct cases: missing 90% and introducing some wild errors. Again human stuff. >> Btw. it would be a great feature for Python 3 to be faster than Python >> 2. > > We all agree on that. One way for this to happen is to add optimizers > that would make Python 'cheat' on micrebenchmarks Then, we are all set. :) Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Jan 27 13:25:27 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 27 Jan 2016 13:25:27 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% Message-ID: <56A90B97.7090001@gmail.com> Hi, tl;dr The summary is that I have a patch that improves CPython performance up to 5-10% on macro benchmarks. Benchmarks results on Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are available at [1]. There are no slowdowns that I could reproduce consistently. There are twodifferent optimizations that yield this speedup: LOAD_METHOD/CALL_METHOD opcodes and per-opcode cache in ceval loop. LOAD_METHOD & CALL_METHOD ------------------------- We had a lot of conversations with Victor about his PEP 509, and he sent me a link to his amazing compilation of notes about CPython performance [2]. One optimization that he pointed out to me was LOAD/CALL_METHOD opcodes, an idea first originated in PyPy. There is a patch that implements this optimization, it's tracked here: [3]. There are some low level details that I explained in the issue, but I'll go over the high level design in this email as well. Every time you access a method attribute on an object, a BoundMethod object is created. It is a fairly expensive operation, despite a freelist of BoundMethods (so that memory allocation is generally avoided). The idea is to detect what looks like a method call in the compiler, and emit a pair of specialized bytecodes for that. So instead of LOAD_GLOBAL/LOAD_ATTR/CALL_FUNCTION we will have LOAD_GLOBAL/LOAD_METHOD/CALL_METHOD. LOAD_METHOD looks at the object on top of the stack, and checks if the name resolves to a method or to a regular attribute. If it's a method, then we push the unbound method object and the object to the stack. If it's an attribute, we push the resolved attribute and NULL. When CALL_METHOD looks at the stack it knows how to call the unbound method properly (pushing the object as a first arg), or how to call a regular callable. This idea does make CPython faster around 2-4%. And it surely doesn't make it slower. I think it's a safe bet to at least implement this optimization in CPython 3.6. So far, the patch only optimizes positional-only method calls. It's possible to optimize all kind of calls, but this will necessitate 3 more opcodes (explained in the issue). We'll need to do some careful benchmarking to see if it's really needed. Per-opcode cache in ceval ------------------------- While reading PEP 509, I was thinking about how we can use dict->ma_version in ceval to speed up globals lookups. One of the key assumptions (and this is what makes JITs possible) is that real-life programs don't modify globals and rebind builtins (often), and that most code paths operate on objects of the same type. In CPython, all pure Python functions have code objects. When you call a function, ceval executes its code object in a frame. Frames contain contextual information, including pointers to the globals and builtins dict. The key observation here is that almost all code objects always have same pointers to the globals (the module they were defined in) and to the builtins. And it's not a good programming practice to mutate globals or rebind builtins. Let's look at this function: def spam(): print(ham) Here are its opcodes: 2 0 LOAD_GLOBAL 0 (print) 3 LOAD_GLOBAL 1 (ham) 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 9 POP_TOP 10 LOAD_CONST 0 (None) 13 RETURN_VALUE The opcodes we want to optimize are LAOD_GLOBAL, 0 and 3. Let's look at the first one, that loads the 'print' function from builtins. The opcode knows the following bits of information: - its offset (0), - its argument (0 -> 'print'), - its type (LOAD_GLOBAL). And these bits of information will *never* change. So if this opcode could resolve the 'print' name (from globals or builtins, likely the latter) and save the pointer to it somewhere, along with globals->ma_version and builtins->ma_version, it could, on its second call, just load this cached info back, check that the globals and builtins dict haven't changed and push the cached ref to the stack. That would save it from doing two dict lookups. We can also optimize LOAD_METHOD. There are high chances, that 'obj' in 'obj.method()' will be of the same type every time we execute the code object. So if we'd have an opcodes cache, LOAD_METHOD could then cache a pointer to the resolved unbound method, a pointer to obj.__class__, and tp_version_tag of obj.__class__. Then it would only need to check if the cached object type is the same (and that it wasn't modified) and that obj.__dict__ doesn't override 'method'. Long story short, this caching really speeds up method calls on types implemented in C. list.append becomes very fast, because list doesn't have a __dict__, so the check is very cheap (with cache). A straightforward way to implement such a cache is simple, but consumes a lot of memory, that would be just wasted, since we only need such a cache for LOAD_GLOBAL and LOAD_METHOD opcodes. So we have to be creative about the cache design. Here's what I came up with: 1. We add a few fields to the code object. 2. ceval will count how many times each code object is executed. 3. When the code object is executed over ~900 times, we mark it as "hot". We also create an 'unsigned char' array "MAPPING", with length set to match the length of the code object. So we have a 1-to-1 mapping between opcodes and MAPPING array. 4. Next ~100 calls, while the code object is "hot", LOAD_GLOBAL and LOAD_METHOD do "MAPPING[opcode_offset()]++". 5. After 1024 calls to the code object, ceval loop will iterate through the MAPPING, counting all opcodes that were executed more than 50 times. 6. We then create an array of cache structs "CACHE" (here's a link to the updated code.h file: [6]). We update MAPPING to be a mapping between opcode position and position in the CACHE. The code object is now "optimized". 7. When the code object is "optimized", LOAD_METHOD and LOAD_GLOBAL use the CACHE array for fast path. 8. When there is a cache miss, i.e. the builtins/global/obj.__dict__ were mutated, the opcode marks its entry in 'CACHE' as deoptimized, and it will never try to use the cache again. Here's a link to the issue tracker with the first version of the patch: [5]. I'm working on the patch in a github repo here: [4]. Summary ------- There are many things about this algorithm that we can improve/tweak. Perhaps we should profile code objects longer, or account for time they were executed. Maybe we shouldn't deoptimize opcodes on their first cache miss. Maybe we can come up with better data structures. We also need to profile the memory and see how much more this cache will require. One thing I'm certain about, is that we can get a 5-10% speedup of CPython with relatively low memory impact. And I think it's worth exploring that! If you're interested in these kind of optimizations, please help with code reviews, ideas, profiling and benchmarks. The latter is especially important, I'd never imagine how hard it is to come up with a good macro benchmark. I also want to thank my company MagicStack (magic.io) for sponsoring this work. Thanks, Yury [1] https://gist.github.com/1st1/aed69d63a2ff4de4c7be [2] http://faster-cpython.readthedocs.org/index.html [3] http://bugs.python.org/issue26110 [4] https://github.com/1st1/cpython/tree/opcache2 [5] http://bugs.python.org/issue26219 [6] https://github.com/python/cpython/compare/master...1st1:opcache2?expand=1#diff-b253e61c56dfa646a6b1b9e7aaad418aR18 From brett at python.org Wed Jan 27 13:33:38 2016 From: brett at python.org (Brett Cannon) Date: Wed, 27 Jan 2016 18:33:38 +0000 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <56A9084F.4090200@mail.de> References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> <56A9084F.4090200@mail.de> Message-ID: On Wed, 27 Jan 2016 at 10:12 Sven R. Kunze wrote: > On 27.01.2016 11:59, Terry Reedy wrote: > > On 1/26/2016 12:35 PM, Sven R. Kunze wrote: > > I completely agree with INADA. > > > I an not sure you do. > > > I am sure I am. He wants to solve a problem the way that is natural to him > as a unique human being. > > > It's like saying, because a specific crossroad features a higher > accident rate, *people need to change their driving behavior*. > *No!* People won't change and it's not necessary either. The crossroad > needs to be changed to be safer. > > > Safer crossroads tend to be slower unless one switched to alternate > designs that eliminate crossing streams of traffic. > > > So Python can be safer AND faster ( = different design) if we try hard > enough. > > > Languages that don't have integers but use residue classes (with > wraparound) or finite integer classes (with overflow) as a (faster) > substitute have, in practice, lots of accidents (bugs) when used by > non-experts. Guido noticed this, gave up on changing coder behavior, and > put the expert behavior of checking for wraparound/overflow and switching > to real integers (longs) into the language. (I forget when this was > added.) > > > I am glad he did because it helps humans solve their problems in a natural > way without artificial boundaries. :) > > > The purpose of the artificially low input to fib() is to hide and avoid > the bugginess of most languages. The analogous trick with testing > crossroads would be to artificially restrict the density of cars to mask > the accident-proneness of a 'fast, consenting-adults' crossroads with no > stop signs and no stop lights. > > > I am completely with you here, however I disagree about suspected > hiding/avoiding mentality. You say: > > Python -> *no problem with big integers but slow at small integers* > Other Language -> *faster but breaks at big integers* > > Yes. That's it. > > We haven't solved the human side, however. A human AGAIN would need to > compromise on either speed or safety. > > > My point is: it would be insanely great if Python could be more like "*fast > AND no problem with big integers*". No compromise here (at least no > noticeable). > And this is why this entire email thread has devolved into a conversation that isn't really going anywhere. This whole thread has completely lost track of the point of Victor's earlier email saying "I'm still working on my FAT work and don't take any notice of the performance numbers until more stuff gets finished". And this discussion of what benchmarks to care about is rather pointless since the core team has an implicit understanding that any performance improvement is taken into consideration in terms of balancing complexity in CPython with how much improvement it gets us. So if someone wants to speed up Fibonacci then they are welcome to try, but the solution must be maintainable in proportion to the speed increase it buys Python as a whole. -Brett > > So, people could entirely *concentrate on their problem domain* without > every worrying about such tiny little, nitty-gritty computer science > details. I love computer science but people of other domains don't have the > time nor the knowledge to decide properly. That's the reason why they might > decide by using some weird micro-benchmarks. Just humans. > > > Same goes for Python. If it's slow using the very same piece of code > (even superficially), you better make the language faster. > Developers won't change and they won't change their code either. Just > not necessary. > > > Instead of making people rewrite fib to dramatically increase speed, we > added the lru-cache decorator to get most of the benefit without a > rewrite. But Inada rejected this Python speedup. An ast optimizer could > potentially do the same speedup without the explicit decorator. (No > side-effects? Multiple recursive calls? Add a cache!) > > > Bingo! That's the spirit. > > Why that decorator in the first place? Hey, I mean, if I ever want to > write some cryptic-looking source code with 3-letters abbreviations (LRU), > I use Assembler again. But I discovered and love Python and I never want to > go back when my problem domain does not require me to. So, when a machine > can detect such an optimization, hell, do it, please. It's more likely that > I apply it at the wrong function AND only in 10% of the correct cases: > missing 90% and introducing some wild errors. > > Again human stuff. > > > Btw. it would be a great feature for Python 3 to be faster than Python > 2. > > > We all agree on that. One way for this to happen is to add optimizers > that would make Python 'cheat' on micrebenchmarks > > > Then, we are all set. :) > > Best, > Sven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Jan 27 13:40:44 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 27 Jan 2016 19:40:44 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> Message-ID: <56A90F2C.9000401@mail.de> On 27.01.2016 12:16, Nick Coghlan wrote: > On 27 January 2016 at 03:35, Sven R. Kunze wrote: >> I completely agree with INADA. >> >> It's like saying, because a specific crossroad features a higher accident >> rate, people need to change their driving behavior. >> No! People won't change and it's not necessary either. The crossroad needs >> to be changed to be safer. > Umm, no, that's not how this works That's exactly how it works, Nick. INADA uses Python as I use crossroads each day. Daily human business. If you read his post carefully, you can discover that he just presented to you his perspective of the world. Moreover, I can assure you that he's not alone. As usual with humans it's not about facts or mathematically proven theorems but *perception*. It's more about marketing, little important details (or unimportant ones depending on whom you ask) and so on. Stating that he has a wrong perspective will not change anything. Believing that Python is treated unfair will not change that either. Most people believe what they see. When they see a "FUNCTION CALL", it's the same in every language. Why? Because it looks like a function call ( name + parentheses ), it's called "function call" even if it's implemented completely differently. It even doesn't matter if we use commas, 'def', return types, etc. Because people understand the bigger concept, so that is what people want to compare. Average Joe doesn't care and does not understand. He looks at the benchmarks. That is something he can understand. "While performance is not a matter when choosing first language, slowest of three makes bad impression and people feel less attractive about Python." << just like that Not saying that INADA is an Average Joe, but I think you get the idea. > - developers contribute to > community driven projects for their *own* reasons. Nobody gets to tell > them what to do unless they're paying them. Bit off-topic. > Micro-optimising a poor algorithm won't deliver macro level > improvements because macro level code uses things like space-speed > trade-offs to improve the algorithmic efficiency (as in the example of > applying functools.lru_cache to a naive recursive fibonacci > implementation). I completely agree, Nick. :) But that isn't the issue here. > Victor's work on FAT optimiser is interesting because it offers > opportunities to speed up even code that is already algorithmically > efficient, as well as making CPython a better platform for > experimenting with those kinds of changes. Exactly. :) > More generally though, much larger opportunities for improvement lie > in persuading people to *stop writing code*, and instead spending more > of their time on finding and assembling code other people have > *already written* into solutions to interesting problems. *That's* the > kind of improvement that turns enormously complex problems like facial > recognition into 25 line Python scripts: > https://realpython.com/blog/python/face-recognition-with-python/ Interesting post. :) Thanks. Btw. I completely agree with you on the "improve programming education", but not everybody can do it; and not everybody wants to learn and to practice it properly. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Jan 27 13:43:31 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 27 Jan 2016 19:43:31 +0100 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> <56A9084F.4090200@mail.de> Message-ID: <56A90FD3.80505@mail.de> On 27.01.2016 19:33, Brett Cannon wrote: > And this is why this entire email thread has devolved into a > conversation that isn't really going anywhere. This whole thread has > completely lost track of the point of Victor's earlier email saying > "I'm still working on my FAT work and don't take any notice of the > performance numbers until more stuff gets finished". And this > discussion of what benchmarks to care about is rather pointless since > the core team has an implicit understanding that any performance > improvement is taken into consideration in terms of balancing > complexity in CPython with how much improvement it gets us. So if > someone wants to speed up Fibonacci then they are welcome to try, but > the solution must be maintainable in proportion to the speed increase > it buys Python as a whole. +1 From brett at python.org Wed Jan 27 15:01:15 2016 From: brett at python.org (Brett Cannon) Date: Wed, 27 Jan 2016 20:01:15 +0000 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <56A90B97.7090001@gmail.com> References: <56A90B97.7090001@gmail.com> Message-ID: On Wed, 27 Jan 2016 at 10:26 Yury Selivanov wrote: > Hi, > > > tl;dr The summary is that I have a patch that improves CPython > performance up to 5-10% on macro benchmarks. Benchmarks results on > Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are available > at [1]. There are no slowdowns that I could reproduce consistently. > > There are twodifferent optimizations that yield this speedup: > LOAD_METHOD/CALL_METHOD opcodes and per-opcode cache in ceval loop. > > > LOAD_METHOD & CALL_METHOD > ------------------------- > > We had a lot of conversations with Victor about his PEP 509, and he sent > me a link to his amazing compilation of notes about CPython performance > [2]. One optimization that he pointed out to me was LOAD/CALL_METHOD > opcodes, an idea first originated in PyPy. > > There is a patch that implements this optimization, it's tracked here: > [3]. There are some low level details that I explained in the issue, > but I'll go over the high level design in this email as well. > > Every time you access a method attribute on an object, a BoundMethod > object is created. It is a fairly expensive operation, despite a > freelist of BoundMethods (so that memory allocation is generally > avoided). The idea is to detect what looks like a method call in the > compiler, and emit a pair of specialized bytecodes for that. > > So instead of LOAD_GLOBAL/LOAD_ATTR/CALL_FUNCTION we will have > LOAD_GLOBAL/LOAD_METHOD/CALL_METHOD. > > LOAD_METHOD looks at the object on top of the stack, and checks if the > name resolves to a method or to a regular attribute. If it's a method, > then we push the unbound method object and the object to the stack. If > it's an attribute, we push the resolved attribute and NULL. > > When CALL_METHOD looks at the stack it knows how to call the unbound > method properly (pushing the object as a first arg), or how to call a > regular callable. > > This idea does make CPython faster around 2-4%. And it surely doesn't > make it slower. I think it's a safe bet to at least implement this > optimization in CPython 3.6. > > So far, the patch only optimizes positional-only method calls. It's > possible to optimize all kind of calls, but this will necessitate 3 more > opcodes (explained in the issue). We'll need to do some careful > benchmarking to see if it's really needed. > > > Per-opcode cache in ceval > ------------------------- > > While reading PEP 509, I was thinking about how we can use > dict->ma_version in ceval to speed up globals lookups. One of the key > assumptions (and this is what makes JITs possible) is that real-life > programs don't modify globals and rebind builtins (often), and that most > code paths operate on objects of the same type. > > In CPython, all pure Python functions have code objects. When you call > a function, ceval executes its code object in a frame. Frames contain > contextual information, including pointers to the globals and builtins > dict. The key observation here is that almost all code objects always > have same pointers to the globals (the module they were defined in) and > to the builtins. And it's not a good programming practice to mutate > globals or rebind builtins. > > Let's look at this function: > > def spam(): > print(ham) > > Here are its opcodes: > > 2 0 LOAD_GLOBAL 0 (print) > 3 LOAD_GLOBAL 1 (ham) > 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > 9 POP_TOP > 10 LOAD_CONST 0 (None) > 13 RETURN_VALUE > > The opcodes we want to optimize are LAOD_GLOBAL, 0 and 3. Let's look at > the first one, that loads the 'print' function from builtins. The > opcode knows the following bits of information: > > - its offset (0), > - its argument (0 -> 'print'), > - its type (LOAD_GLOBAL). > > And these bits of information will *never* change. So if this opcode > could resolve the 'print' name (from globals or builtins, likely the > latter) and save the pointer to it somewhere, along with > globals->ma_version and builtins->ma_version, it could, on its second > call, just load this cached info back, check that the globals and > builtins dict haven't changed and push the cached ref to the stack. > That would save it from doing two dict lookups. > > We can also optimize LOAD_METHOD. There are high chances, that 'obj' in > 'obj.method()' will be of the same type every time we execute the code > object. So if we'd have an opcodes cache, LOAD_METHOD could then cache > a pointer to the resolved unbound method, a pointer to obj.__class__, > and tp_version_tag of obj.__class__. Then it would only need to check > if the cached object type is the same (and that it wasn't modified) and > that obj.__dict__ doesn't override 'method'. Long story short, this > caching really speeds up method calls on types implemented in C. > list.append becomes very fast, because list doesn't have a __dict__, so > the check is very cheap (with cache). > What would it take to make this work with Python-defined classes? I guess that would require knowing the version of the instance's __dict__, the instance's __class__ version, the MRO, and where the method object was found in the MRO and any intermediary classes to know if it was suddenly shadowed? I think that's everything. :) Obviously that's a lot, but I wonder how many classes have a deep inheritance model vs. inheriting only from `object`? In that case you only have to check self.__dict__.ma_version, self.__class__, self.__class__.__dict__.ma_version, and self.__class__.__class__ == `type`. I guess another way to look at this is to get an idea of how complex do the checks have to get before caching something like this is not worth it (probably also depends on how often you mutate self.__dict__ thanks to mutating attributes, but you could in that instance just decide to always look at self.__dict__ for the method's key and then do the ma_version cache check for everything coming from the class). Otherwise we can consider looking at the the caching strategies that Self helped pioneer (http://bibliography.selflanguage.org/) that all of the various JS engines lifted and consider caching all method lookups. > > A straightforward way to implement such a cache is simple, but consumes > a lot of memory, that would be just wasted, since we only need such a > cache for LOAD_GLOBAL and LOAD_METHOD opcodes. So we have to be creative > about the cache design. Here's what I came up with: > > 1. We add a few fields to the code object. > > 2. ceval will count how many times each code object is executed. > > 3. When the code object is executed over ~900 times, we mark it as > "hot". What happens if you simply consider all code as hot? Is the overhead of building the mapping such that you really need this, or is this simply to avoid some memory/startup cost? > We also create an 'unsigned char' array "MAPPING", with length > set to match the length of the code object. So we have a 1-to-1 mapping > between opcodes and MAPPING array. > > 4. Next ~100 calls, while the code object is "hot", LOAD_GLOBAL and > LOAD_METHOD do "MAPPING[opcode_offset()]++". > > 5. After 1024 calls to the code object, ceval loop will iterate through > the MAPPING, counting all opcodes that were executed more than 50 times. > Where did the "50 times" boundary come from? Was this measured somehow or did you just guess at a number? > > 6. We then create an array of cache structs "CACHE" (here's a link to > the updated code.h file: [6]). We update MAPPING to be a mapping > between opcode position and position in the CACHE. The code object is > now "optimized". > > 7. When the code object is "optimized", LOAD_METHOD and LOAD_GLOBAL use > the CACHE array for fast path. > > 8. When there is a cache miss, i.e. the builtins/global/obj.__dict__ > were mutated, the opcode marks its entry in 'CACHE' as deoptimized, and > it will never try to use the cache again. > > Here's a link to the issue tracker with the first version of the patch: > [5]. I'm working on the patch in a github repo here: [4]. > > > Summary > ------- > > There are many things about this algorithm that we can improve/tweak. > Perhaps we should profile code objects longer, or account for time they > were executed. Maybe we shouldn't deoptimize opcodes on their first > cache miss. Maybe we can come up with better data structures. We also > need to profile the memory and see how much more this cache will require. > > One thing I'm certain about, is that we can get a 5-10% speedup of > CPython with relatively low memory impact. And I think it's worth > exploring that! > Great! > > If you're interested in these kind of optimizations, please help with > code reviews, ideas, profiling and benchmarks. The latter is especially > important, I'd never imagine how hard it is to come up with a good macro > benchmark. > Have you tried hg.python.org/benchmarks? Or are you looking for new benchmarks? If the latter then we should probably strike up a discussion on speed@ and start considering a new, unified benchmark suite that CPython, PyPy, Pyston, Jython, and IronPython can all agree on. > > I also want to thank my company MagicStack (magic.io) for sponsoring > this work. > Yep, thanks to all the companies sponsoring people doing work lately to try and speed things up! -------------- next part -------------- An HTML attachment was scrubbed... URL: From damien.p.george at gmail.com Wed Jan 27 15:10:03 2016 From: damien.p.george at gmail.com (Damien George) Date: Wed, 27 Jan 2016 20:10:03 +0000 Subject: [Python-Dev] Speeding up CPython 5-10% Message-ID: Hi Yuri, I think these are great ideas to speed up CPython. They are probably the simplest yet most effective ways to get performance improvements in the VM. MicroPython has had LOAD_METHOD/CALL_METHOD from the start (inspired by PyPy, and the main reason to have it is because you don't need to allocate on the heap when doing a simple method call). The specific opcodes are: LOAD_METHOD # same behaviour as you propose CALL_METHOD # for calls with positional and/or keyword args CALL_METHOD_VAR_KW # for calls with one or both of */** We also have LOAD_ATTR, CALL_FUNCTION and CALL_FUNCTION_VAR_KW for non-method calls. MicroPython also has dictionary lookup caching, but it's a bit different to your proposal. We do something much simpler: each opcode that has a cache ability (eg LOAD_GLOBAL, STORE_GLOBAL, LOAD_ATTR, etc) includes a single byte in the opcode which is an offset-guess into the dictionary to find the desired element. Eg for LOAD_GLOBAL we have (pseudo code): CASE(LOAD_GLOBAL): key = DECODE_KEY; offset_guess = DECODE_BYTE; if (global_dict[offset_guess].key == key) { // found the element straight away } else { // not found, do a full lookup and save the offset offset_guess = dict_lookup(global_dict, key); UPDATE_BYTECODE(offset_guess); } PUSH(global_dict[offset_guess].elem); We have found that such caching gives a massive performance increase, on the order of 20%. The issue (for us) is that it increases bytecode size by a considerable amount, requires writeable bytecode, and can be non-deterministic in terms of lookup time. Those things are important in the embedded world, but not so much on the desktop. Good luck with it! Regards, Damien. From yselivanov.ml at gmail.com Wed Jan 27 15:26:35 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 27 Jan 2016 15:26:35 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: <56A90B97.7090001@gmail.com> Message-ID: <56A927FB.8080307@gmail.com> On 2016-01-27 3:01 PM, Brett Cannon wrote: > > > [..] > > > We can also optimize LOAD_METHOD. There are high chances, that > 'obj' in > 'obj.method()' will be of the same type every time we execute the code > object. So if we'd have an opcodes cache, LOAD_METHOD could then > cache > a pointer to the resolved unbound method, a pointer to obj.__class__, > and tp_version_tag of obj.__class__. Then it would only need to check > if the cached object type is the same (and that it wasn't > modified) and > that obj.__dict__ doesn't override 'method'. Long story short, this > caching really speeds up method calls on types implemented in C. > list.append becomes very fast, because list doesn't have a > __dict__, so > the check is very cheap (with cache). > > > What would it take to make this work with Python-defined classes? It already works for Python-defined classes. But it's a bit more expensive because you still have to check object's __dict__. Still, there is a very noticeable performance increase (see the results of benchmark runs). > I guess that would require knowing the version of the instance's > __dict__, the instance's __class__ version, the MRO, and where the > method object was found in the MRO and any intermediary classes to > know if it was suddenly shadowed? I think that's everything. :) No, unfortunately we can't use the version of the instance's __dict__ as it is very volatile. The current implementation of opcode cache works because types are much more stable. Remember, the cache is per *code object*, so it should work for all times when code object is executed. class F: def spam(self): self.ham() # <- version of self.__dict__ is unstable # so we'll endup invalidating the cache # too often __class__ version, MRO changes etc are covered by tp_version_tag, which I use as one of guards. > > Obviously that's a lot, but I wonder how many classes have a deep > inheritance model vs. inheriting only from `object`? In that case you > only have to check self.__dict__.ma_version, self.__class__, > self.__class__.__dict__.ma_version, and self.__class__.__class__ == > `type`. I guess another way to look at this is to get an idea of how > complex do the checks have to get before caching something like this > is not worth it (probably also depends on how often you mutate > self.__dict__ thanks to mutating attributes, but you could in that > instance just decide to always look at self.__dict__ for the method's > key and then do the ma_version cache check for everything coming from > the class). > > Otherwise we can consider looking at the the caching strategies that > Self helped pioneer (http://bibliography.selflanguage.org/) that all > of the various JS engines lifted and consider caching all method lookups. Yeah, hidden classes are great. But the infrastructure to support them properly is huge. I think that to make them work you'll need a JIT -- to trace, deoptimize, optimize, and do it all with a reasonable memory footprint. My patch is much smaller and simpler, something we can realistically tune and ship in 3.6. > > A straightforward way to implement such a cache is simple, but > consumes > a lot of memory, that would be just wasted, since we only need such a > cache for LOAD_GLOBAL and LOAD_METHOD opcodes. So we have to be > creative > about the cache design. Here's what I came up with: > > 1. We add a few fields to the code object. > > 2. ceval will count how many times each code object is executed. > > 3. When the code object is executed over ~900 times, we mark it as > "hot". > > > What happens if you simply consider all code as hot? Is the overhead > of building the mapping such that you really need this, or is this > simply to avoid some memory/startup cost? That's the first step for this patch. I think we need to profile several big applications (I'll do it later for some of my code bases) and see how big is the memory impact if we optimize everything. In any case, I expect it to be noticeable (which may be acceptable), so we'll probably try to optimize it. > We also create an 'unsigned char' array "MAPPING", with length > set to match the length of the code object. So we have a 1-to-1 > mapping > between opcodes and MAPPING array. > > 4. Next ~100 calls, while the code object is "hot", LOAD_GLOBAL and > LOAD_METHOD do "MAPPING[opcode_offset()]++". > > 5. After 1024 calls to the code object, ceval loop will iterate > through > the MAPPING, counting all opcodes that were executed more than 50 > times. > > > Where did the "50 times" boundary come from? Was this measured somehow > or did you just guess at a number? If the number is too low, then you'll optimize code in branches that are rarely executed. So I picked 50, because I only trace opcodes for 100 calls. All of those numbers can be (should be?) changed, and I think we should experiment with different heuristics. > [..] > > > If you're interested in these kind of optimizations, please help with > code reviews, ideas, profiling and benchmarks. The latter is > especially > important, I'd never imagine how hard it is to come up with a good > macro > benchmark. > > > Have you tried hg.python.org/benchmarks ? Yes: https://gist.github.com/1st1/aed69d63a2ff4de4c7be > Or are you looking for new benchmarks? If the latter then we should > probably strike up a discussion on speed@ and start considering a new, > unified benchmark suite that CPython, PyPy, Pyston, Jython, and > IronPython can all agree on. Yes, IMHO we need better benchmarks. Some of the existing ones are very unstable -- I can run them three times and get three completely different results. Benchmarking is hard :) I'll create a few issues on bugs.python.org with new/updated benchmarks, and will join the speed@ mailing list. Yury From v+python at g.nevcal.com Wed Jan 27 15:32:41 2016 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 27 Jan 2016 12:32:41 -0800 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <22184.64112.8831.955609@turnbull.sk.tsukuba.ac.jp> References: <56A698CF.3080402@mail.de> <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp> <22184.64112.8831.955609@turnbull.sk.tsukuba.ac.jp> Message-ID: <56A92969.2030600@g.nevcal.com> On 1/27/2016 9:12 AM, Stephen J. Turnbull wrote: > Without that knowledge and effort, choosing a programming language > based on microbenchmarks is like choosing a car based on the > leg-length of the model sitting on the hood in the TV commercial. +1 QOTD -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Jan 27 15:37:49 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 27 Jan 2016 15:37:49 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: Message-ID: <56A92A9D.70508@gmail.com> On 2016-01-27 3:10 PM, Damien George wrote: > Hi Yuri, > > I think these are great ideas to speed up CPython. They are probably > the simplest yet most effective ways to get performance improvements > in the VM. Thanks! > > MicroPython has had LOAD_METHOD/CALL_METHOD from the start (inspired > by PyPy, and the main reason to have it is because you don't need to > allocate on the heap when doing a simple method call). The specific > opcodes are: > > LOAD_METHOD # same behaviour as you propose > CALL_METHOD # for calls with positional and/or keyword args > CALL_METHOD_VAR_KW # for calls with one or both of */** > > We also have LOAD_ATTR, CALL_FUNCTION and CALL_FUNCTION_VAR_KW for > non-method calls. Yes, we'll need to add CALL_METHOD{_VAR|_KW|etc} opcodes to optimize all kind of method calls. However, I'm not sure how big the impact will be, need to do more benchmarking. BTW, how do you benchmark MicroPython? > > MicroPython also has dictionary lookup caching, but it's a bit > different to your proposal. We do something much simpler: each opcode > that has a cache ability (eg LOAD_GLOBAL, STORE_GLOBAL, LOAD_ATTR, > etc) includes a single byte in the opcode which is an offset-guess > into the dictionary to find the desired element. Eg for LOAD_GLOBAL > we have (pseudo code): > > CASE(LOAD_GLOBAL): > key = DECODE_KEY; > offset_guess = DECODE_BYTE; > if (global_dict[offset_guess].key == key) { > // found the element straight away > } else { > // not found, do a full lookup and save the offset > offset_guess = dict_lookup(global_dict, key); > UPDATE_BYTECODE(offset_guess); > } > PUSH(global_dict[offset_guess].elem); > > We have found that such caching gives a massive performance increase, > on the order of 20%. The issue (for us) is that it increases bytecode > size by a considerable amount, requires writeable bytecode, and can be > non-deterministic in terms of lookup time. Those things are important > in the embedded world, but not so much on the desktop. That's a neat idea! You're right, it does require bytecode to become writeable. I considered implementing a similar strategy, but this would be a big change for CPython. So I decided to minimize the impact of the patch and leave the opcodes untouched. Thanks! Yury From damien.p.george at gmail.com Wed Jan 27 16:20:27 2016 From: damien.p.george at gmail.com (Damien George) Date: Wed, 27 Jan 2016 21:20:27 +0000 Subject: [Python-Dev] Speeding up CPython 5-10% Message-ID: Hi Yury, (Sorry for misspelling your name previously!) > Yes, we'll need to add CALL_METHOD{_VAR|_KW|etc} opcodes to optimize all > kind of method calls. However, I'm not sure how big the impact will be, > need to do more benchmarking. I never did such fine grained analysis with MicroPython. I don't think there are many uses of * and ** that it'd be worth it, but definitely there are lots of uses of plain keywords. Also, you'd want to consider how simple/complex it is to treat all these different opcodes in the compiler. For us, it's simpler to treat everything the same. Otherwise your LOAD_METHOD part of the compiler will need to peek deep into the AST to see what kind of call it is. > BTW, how do you benchmark MicroPython? Haha, good question! Well, we use Pystone 1.2 (unmodified) to do basic benchmarking, and find it to be quite good. We track our code live at: http://micropython.org/resources/code-dashboard/ You can see there the red line, which is the Pystone result. There was a big jump around Jan 2015 which is when we introduced opcode dictionary caching. And since then it's been very gradually increasing due to small optimisations here and there. Pystone is actually a great benchmark for embedded systems because it gives very reliable results there (almost zero variation across runs) and if we can squeeze 5 more Pystones out with some change then we know that it's a good optimisation (for efficiency at least). For us, low RAM usage and small code size are the most important factors, and we track those meticulously. But in fact, smaller code size quite often correlates with more efficient code because there's less to execute and it fits in the CPU cache (at least on the desktop). We do have some other benchmarks, but they are highly specialised for us. For example, how fast can you bit bang a GPIO pin using pure Python code. Currently we get around 200kHz on a 168MHz MCU, which shows that pure (Micro)Python code is about 100 times slower than C. > That's a neat idea! You're right, it does require bytecode to become > writeable. I considered implementing a similar strategy, but this would > be a big change for CPython. So I decided to minimize the impact of the > patch and leave the opcodes untouched. I think you need to consider "big" changes, especially ones like this that can have a great (and good) impact. But really, this is a behind-the-scenes change that *should not* affect end users, and so you should not have any second thoughts about doing it. One problem I see with CPython is that it exposes way too much to the user (both Python programmer and C extension writer) and this hurts both language evolution (you constantly need to provide backwards compatibility) and ability to optimise. Cheers, Damien. From yselivanov.ml at gmail.com Wed Jan 27 16:53:55 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 27 Jan 2016 16:53:55 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: Message-ID: <56A93C73.6030606@gmail.com> Damien, On 2016-01-27 4:20 PM, Damien George wrote: > Hi Yury, > > (Sorry for misspelling your name previously!) NP. As long as the first letter is "y" I don't care ;) > >> Yes, we'll need to add CALL_METHOD{_VAR|_KW|etc} opcodes to optimize all >> kind of method calls. However, I'm not sure how big the impact will be, >> need to do more benchmarking. > I never did such fine grained analysis with MicroPython. I don't > think there are many uses of * and ** that it'd be worth it, but > definitely there are lots of uses of plain keywords. Also, you'd want > to consider how simple/complex it is to treat all these different > opcodes in the compiler. For us, it's simpler to treat everything the > same. Otherwise your LOAD_METHOD part of the compiler will need to > peek deep into the AST to see what kind of call it is. > >> BTW, how do you benchmark MicroPython? > Haha, good question! Well, we use Pystone 1.2 (unmodified) to do > basic benchmarking, and find it to be quite good. We track our code > live at: > > http://micropython.org/resources/code-dashboard/ The dashboard is cool! An off-topic: have you ever tried hg.python.org/benchmarks or compare MicroPython vs CPython? I'm curious if MicroPython is faster -- in that case we'll try to copy some optimization ideas. > You can see there the red line, which is the Pystone result. There > was a big jump around Jan 2015 which is when we introduced opcode > dictionary caching. And since then it's been very gradually > increasing due to small optimisations here and there. Do you use opcode dictionary caching only for LOAD_GLOBAL-like opcodes? Do you have an equivalent of LOAD_FAST, or you use dicts to store local variables? >> That's a neat idea! You're right, it does require bytecode to become >> writeable. I considered implementing a similar strategy, but this would >> be a big change for CPython. So I decided to minimize the impact of the >> patch and leave the opcodes untouched. > I think you need to consider "big" changes, especially ones like this > that can have a great (and good) impact. But really, this is a > behind-the-scenes change that *should not* affect end users, and so > you should not have any second thoughts about doing it. If we change the opcode size, it will probably affect libraries that compose or modify code objects. Modules like "dis" will also need to be updated. And that's probably just a tip of the iceberg. We can still implement your approach if we add a separate private 'unsigned char' array to each code object, so that LOAD_GLOBAL can store the key offsets. It should be a bit faster than my current patch, since it has one less level of indirection. But this way we loose the ability to optimize LOAD_METHOD, simply because it requires more memory for its cache. In any case, I'll experiment! > One problem I > see with CPython is that it exposes way too much to the user (both > Python programmer and C extension writer) and this hurts both language > evolution (you constantly need to provide backwards compatibility) and > ability to optimise. Right. Even though CPython explicitly states that opcodes and code objects might change in the future, we still have to be careful about changing them. Yury From yselivanov.ml at gmail.com Wed Jan 27 17:01:17 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 27 Jan 2016 17:01:17 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <56A90B97.7090001@gmail.com> References: <56A90B97.7090001@gmail.com> Message-ID: <56A93E2D.5060707@gmail.com> As Brett suggested, I've just run the benchmarks suite with memory tracking on. The results are here: https://gist.github.com/1st1/1851afb2773526fd7c58 Looks like the memory increase is around 1%. One synthetic micro-benchmark, unpack_sequence, contains hundreds of lines that load a global variable and does nothing else, consumes 5%. Yury From v+python at g.nevcal.com Wed Jan 27 15:46:15 2016 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 27 Jan 2016 12:46:15 -0800 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <56A92A9D.70508@gmail.com> References: <56A92A9D.70508@gmail.com> Message-ID: <56A92C97.30806@g.nevcal.com> On 1/27/2016 12:37 PM, Yury Selivanov wrote: > >> >> MicroPython also has dictionary lookup caching, but it's a bit >> different to your proposal. We do something much simpler: each opcode >> that has a cache ability (eg LOAD_GLOBAL, STORE_GLOBAL, LOAD_ATTR, >> etc) includes a single byte in the opcode which is an offset-guess >> into the dictionary to find the desired element. Eg for LOAD_GLOBAL >> we have (pseudo code): >> >> CASE(LOAD_GLOBAL): >> key = DECODE_KEY; >> offset_guess = DECODE_BYTE; >> if (global_dict[offset_guess].key == key) { >> // found the element straight away >> } else { >> // not found, do a full lookup and save the offset >> offset_guess = dict_lookup(global_dict, key); >> UPDATE_BYTECODE(offset_guess); >> } >> PUSH(global_dict[offset_guess].elem); >> >> We have found that such caching gives a massive performance increase, >> on the order of 20%. The issue (for us) is that it increases bytecode >> size by a considerable amount, requires writeable bytecode, and can be >> non-deterministic in terms of lookup time. Those things are important >> in the embedded world, but not so much on the desktop. > > That's a neat idea! You're right, it does require bytecode to become > writeable. Would it? Remember "fixup lists"? Maybe they still exist for loading function addresses from one DLL into the code of another at load time? So the equivalent for bytecode requires a static table of offset_guess, and the offsets into that table are allocated by the byte-code loader at byte-code load time, and the byte-code is "fixed up" at load time to use the correct offsets into the offset_guess table. It takes one more indirection to find the guess, but if the result is a 20% improvement, maybe you'd still get 19%... -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Jan 27 17:40:26 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 27 Jan 2016 17:40:26 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <56A90B97.7090001@gmail.com> References: <56A90B97.7090001@gmail.com> Message-ID: <56A9475A.3040300@gmail.com> BTW, this optimization also makes some old optimization tricks obsolete. 1. No need to write 'def func(len=len)'. Globals lookups will be fast. 2. No need to save bound methods: obj = [] obj_append = obj.append for _ in range(10**6): obj_append(something) This hand-optimized code would only be marginally faster, because of LOAD_METHOD and how it's cached. Yury From yselivanov.ml at gmail.com Wed Jan 27 17:58:11 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 27 Jan 2016 17:58:11 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <56A92C97.30806@g.nevcal.com> References: <56A92A9D.70508@gmail.com> <56A92C97.30806@g.nevcal.com> Message-ID: <56A94B83.5020705@gmail.com> On 2016-01-27 3:46 PM, Glenn Linderman wrote: > On 1/27/2016 12:37 PM, Yury Selivanov wrote: >> >>> >>> MicroPython also has dictionary lookup caching, but it's a bit >>> different to your proposal. We do something much simpler: each opcode >>> that has a cache ability (eg LOAD_GLOBAL, STORE_GLOBAL, LOAD_ATTR, >>> etc) includes a single byte in the opcode which is an offset-guess >>> into the dictionary to find the desired element. Eg for LOAD_GLOBAL >>> we have (pseudo code): >>> >>> CASE(LOAD_GLOBAL): >>> key = DECODE_KEY; >>> offset_guess = DECODE_BYTE; >>> if (global_dict[offset_guess].key == key) { >>> // found the element straight away >>> } else { >>> // not found, do a full lookup and save the offset >>> offset_guess = dict_lookup(global_dict, key); >>> UPDATE_BYTECODE(offset_guess); >>> } >>> PUSH(global_dict[offset_guess].elem); >>> >>> We have found that such caching gives a massive performance increase, >>> on the order of 20%. The issue (for us) is that it increases bytecode >>> size by a considerable amount, requires writeable bytecode, and can be >>> non-deterministic in terms of lookup time. Those things are important >>> in the embedded world, but not so much on the desktop. >> >> That's a neat idea! You're right, it does require bytecode to become >> writeable. > > Would it? > > Remember "fixup lists"? Maybe they still exist for loading function > addresses from one DLL into the code of another at load time? > > So the equivalent for bytecode requires a static table of > offset_guess, and the offsets into that table are allocated by the > byte-code loader at byte-code load time, and the byte-code is "fixed > up" at load time to use the correct offsets into the offset_guess > table. It takes one more indirection to find the guess, but if the > result is a 20% improvement, maybe you'd still get 19%... Right, in my current patch I have an offset table per code object. Essentially, this offset table adds 8bits per opcode. It also means that only first 255 LOAD_GLOBAL/LOAD_METHOD opcodes *per-code-object* are optimized (because the offset table only can store 8bit offsets), which is usually enough (I think you need to have more than a 500 lines of code function to reach that limit). Yury From stephen at xemacs.org Wed Jan 27 19:53:51 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 28 Jan 2016 09:53:51 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> <56A9084F.4090200@mail.de> Message-ID: <22185.26271.894312.852637@turnbull.sk.tsukuba.ac.jp> Brett Cannon writes: > the core team has an implicit understanding that any performance > improvement is taken into consideration in terms of balancing > complexity in CPython with how much improvement it gets us. EIBTI. I can shut up now. Thank you! From ncoghlan at gmail.com Thu Jan 28 02:58:07 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Jan 2016 17:58:07 +1000 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: <56A90F2C.9000401@mail.de> References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> <56A90F2C.9000401@mail.de> Message-ID: On 28 January 2016 at 04:40, Sven R. Kunze wrote: > On 27.01.2016 12:16, Nick Coghlan wrote: >> Umm, no, that's not how this works > That's exactly how it works, Nick. > > INADA uses Python as I use crossroads each day. Daily human business. > > If you read his post carefully, you can discover that he just presented to > you his perspective of the world. Moreover, I can assure you that he's not > alone. As usual with humans it's not about facts or mathematically proven > theorems but perception. It's more about marketing, little important details > (or unimportant ones depending on whom you ask) and so on. Stating that he > has a wrong perspective will not change anything. The only part I disagree with is requesting that *other people* care about marketing numbers if that's not something they're already inclined to care about. I'm not in any way disputing that folks make decisions based on inappropriate metrics, nor that it bothers some folks that there are dozens of perfectly viable programming languages people may choose to use instead of Python. The fact remains that contributors to open source projects work on what they want to work on or on what their employers pay them to work on (for a lucky few, those are the same thing), so telling other contributors that they're working on the "wrong thing" because their priorities differ from our priorities is almost always going to be irritating rather than helpful. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From songofacandy at gmail.com Thu Jan 28 03:30:43 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 28 Jan 2016 17:30:43 +0900 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> <56A90F2C.9000401@mail.de> Message-ID: Please stop. I'm sorry about messing up this thread. I just wanted to represent why I'm very interested in Victor's efforts. Regards. On Thu, Jan 28, 2016 at 4:58 PM, Nick Coghlan wrote: > On 28 January 2016 at 04:40, Sven R. Kunze wrote: > > On 27.01.2016 12:16, Nick Coghlan wrote: > >> Umm, no, that's not how this works > > That's exactly how it works, Nick. > > > > INADA uses Python as I use crossroads each day. Daily human business. > > > > If you read his post carefully, you can discover that he just presented > to > > you his perspective of the world. Moreover, I can assure you that he's > not > > alone. As usual with humans it's not about facts or mathematically proven > > theorems but perception. It's more about marketing, little important > details > > (or unimportant ones depending on whom you ask) and so on. Stating that > he > > has a wrong perspective will not change anything. > > The only part I disagree with is requesting that *other people* care > about marketing numbers if that's not something they're already > inclined to care about. I'm not in any way disputing that folks make > decisions based on inappropriate metrics, nor that it bothers some > folks that there are dozens of perfectly viable programming languages > people may choose to use instead of Python. > > The fact remains that contributors to open source projects work on > what they want to work on or on what their employers pay them to work > on (for a lucky few, those are the same thing), so telling other > contributors that they're working on the "wrong thing" because their > priorities differ from our priorities is almost always going to be > irritating rather than helpful. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From vadmium+py at gmail.com Thu Jan 28 03:35:59 2016 From: vadmium+py at gmail.com (Martin Panter) Date: Thu, 28 Jan 2016 08:35:59 +0000 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? In-Reply-To: References: Message-ID: > After digging through test_socket.py for over an hour (the MRO for > RecvmsgUDP6Test is enormous!!), I've boiled the issue down to this: > > import socket > MSG = b'asdf qwer zxcv' > serv = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM) > serv.bind(("::1", 0)) > cli = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM) > cli.bind(("::1", 0)) > cli.sendto(MSG, serv.getsockname()) > print(serv.recvmsg(len(MSG) - 3, 0, socket.MSG_PEEK)) > print(serv.recvmsg(len(MSG), 0, socket.MSG_PEEK)) > print(serv.recvmsg(len(MSG))) > > On my main system, this produces three lines of output: the first has > truncated text, the second has full text, and the third also has full > text. This proves that MSG_PEEK is working correctly. On the buildbot, > though, the first one stalls out. Commenting that line out produces > correct results - peek the full data, then read it, and all is well. > > Any idea why partial read on a datagram socket would sometimes stall? I think it would stall if there is no data to receive. Maybe check the return value of sendto(), to ensure it is sending the whole message. Attached is a C program which should do the equivalent of your boiled-down Python script, in case that helps: $ gcc -Wall peek-udp6.c -o peek-udp6 $ ./peek-udp6 Bytes sent: 14 Received [asdf qwer z] Received [asdf qwer zxcv] Received [asdf qwer zxcv] Other things that come to mind are to see if there is anything odd about the buildbot?s Linux kernel and glibc versions. Maybe run the Python script under ?strace? to see if anything strange is going on. -------------- next part -------------- A non-text attachment was scrubbed... Name: peek-udp6.c Type: text/x-csrc Size: 2404 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Jan 28 04:39:52 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Jan 2016 19:39:52 +1000 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> <56A90F2C.9000401@mail.de> Message-ID: On 28 January 2016 at 18:30, INADA Naoki wrote: > Please stop. > > I'm sorry about messing up this thread. Not your fault at all! This is just a particular bugbear of mine, since software architecture design (including appropriate programming language selection) is an even more poorly understood discipline than software development in general :) > I just wanted to represent why I'm very interested in Victor's efforts. And thanks for posting that, as it is indeed cool that the optimisation efforts currently being discussed may result in performance improvements on some of the simplified micro-benchmarks popular in programming language shootouts. There's no way you could have anticipated the subsequent tangential discussion on motives for contributing to open source projects, and the impact that has on what we can reasonably expect from fellow contributors. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Thu Jan 28 05:41:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 28 Jan 2016 21:41:08 +1100 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? In-Reply-To: References: Message-ID: On Thu, Jan 28, 2016 at 7:35 PM, Martin Panter wrote: > Other things that come to mind are to see if there is anything odd > about the buildbot?s Linux kernel and glibc versions. Maybe run the > Python script under ?strace? to see if anything strange is going on. > I did that, and a few other things. Most notably, commenting out the partial-read resulted in a flawless run, and strace showed it stalling in the partial read too. However, as I was doing so (and I just discarded a draft message where I'd been typing up notes), my entire system went kerblooie, and I've spent the last day rebuilding stuff from scratch. When I get around to it, I'll rebuild the buildbot VM - and it'll be Debian Jessie (current stable), because there's no particular reason to use Wheezy (oldstable). So the problem will almost certainly disappear. ChrisA From stefan at bytereef.org Thu Jan 28 08:49:50 2016 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 28 Jan 2016 14:49:50 +0100 Subject: [Python-Dev] [Python-checkins] BAD Benchmark Results for Python Default 2016-01-26 In-Reply-To: References: <2acd60eb-a5d1-4b31-853d-7c7f3f0841fa@irsmsx101.ger.corp.intel.com> Message-ID: <20160128134950.GA3196@bytereef.org> IMO the timings of the benchmark suite are a bit unstable -- this is not the fault of Intel's setup, I noticed it also when running the suite myself. On Tue, Jan 26, 2016 at 06:48:54PM +0000, Stewart, David C wrote: > Wow, what happened to Python default to cause such a regression? > > > > > On 1/26/16, 7:31 AM, "lp_benchmark_robot" wrote: > > >Results for project Python default, build date 2016-01-26 03:07:40 +0000 > >commit: cbd4a6a2657e > >previous commit: f700bc0412bc > >revision date: 2016-01-26 02:54:37 +0000 > >environment: Haswell-EP > > cpu: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz 2x18 cores, stepping 2, LLC 45 MB > > mem: 128 GB > > os: CentOS 7.1 > > kernel: Linux 3.10.0-229.4.2.el7.x86_64 > > > >Baseline results were generated using release v3.4.3, with hash b4cbecbc0781 > >from 2015-02-25 12:15:33+00:00 > > > >---------------------------------------------------------------------------------- > > benchmark relative change since change since current rev run > > std_dev* last run baseline with PGO > >---------------------------------------------------------------------------------- > >:-) django_v2 0.21% -2.93% 8.95% 16.19% > >:-| pybench 0.10% 0.05% -1.87% 5.40% > >:-( regex_v8 2.72% -0.02% -4.67% 4.57% > >:-| nbody 0.13% -0.92% -1.33% 7.40% > >:-| json_dump_v2 0.20% 0.87% -1.59% 11.48% > >:-| normal_startup 0.90% -0.57% 0.10% 5.35% > >---------------------------------------------------------------------------------- > >* Relative Standard Deviation (Standard Deviation/Average) > > > >If this is not displayed properly please visit our results page here: http://languagesperformance.intel.com/bad-benchmark-results-for-python-default-2016-01-26/ > > > >Note: Benchmark results are measured in seconds. > > > >Subject Label Legend: > >Attributes are determined based on the performance evolution of the workloads > >compared to the previous measurement iteration. > >NEUTRAL: performance did not change by more than 1% for any workload > >GOOD: performance improved by more than 1% for at least one workload and there > >is no regression greater than 1% > >BAD: performance dropped by more than 1% for at least one workload and there is > >no improvement greater than 1% > >UGLY: performance improved by more than 1% for at least one workload and also > >dropped by more than 1% for at least one workload > > > > > >Our lab does a nightly source pull and build of the Python project and measures > >performance changes against the previous stable version and the previous nightly > >measurement. This is provided as a service to the community so that quality > >issues with current hardware can be identified quickly. > > > >Intel technologies' features and benefits depend on system configuration and may > >require enabled hardware, software or service activation. Performance varies > >depending on system configuration. > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins From larry at hastings.org Thu Jan 28 09:57:20 2016 From: larry at hastings.org (Larry Hastings) Date: Thu, 28 Jan 2016 06:57:20 -0800 Subject: [Python-Dev] Fun with ancient unsupported platforms Message-ID: <56AA2C50.7050700@hastings.org> Check out and cd into Python trunk. % grep -Ri win16 * | wc 10 66 625 % grep -Ri nextstep | wc 23 119 1328 % grep -Ri rhapsody * | wc 47 269 3390 % grep -Ri msdos * | wc 56 381 3895 % grep -Ri ms-dos * | wc 20 180 1425 win16! *laughs* *wipes tear from eye* It's currently 2016. Perhaps it's time to remove all vestiges of these unsupported operating systems nobody's cared about since a year that started with a '1'? //arry/ p.s. I suspect some of the uses of "rhapsody" are actually live code used for OS X. So this isn't necessarily dead code, some of it is merely long-out-of-date comments. p.p.s. At least there were no references to "taligent", "copland", or "gershwin"...! -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Thu Jan 28 10:40:48 2016 From: christian at python.org (Christian Heimes) Date: Thu, 28 Jan 2016 16:40:48 +0100 Subject: [Python-Dev] Fun with ancient unsupported platforms In-Reply-To: <56AA2C50.7050700@hastings.org> References: <56AA2C50.7050700@hastings.org> Message-ID: <56AA3680.8060602@python.org> On 2016-01-28 15:57, Larry Hastings wrote: > > > Check out and cd into Python trunk. > > % grep -Ri win16 * | wc > 10 66 625 > > % grep -Ri nextstep | wc > 23 119 1328 > > % grep -Ri rhapsody * | wc > 47 269 3390 > > % grep -Ri msdos * | wc > 56 381 3895 > % grep -Ri ms-dos * | wc > 20 180 1425 > > > win16! *laughs* *wipes tear from eye* > > It's currently 2016. Perhaps it's time to remove all vestiges of these > unsupported operating systems nobody's cared about since a year that > started with a '1'? The platform module has more hilarious comments: Still needed: * more support for WinCE * support for MS-DOS (PythonDX ?) * support for Amiga and other still unsupported platforms running Python Christian From rymg19 at gmail.com Thu Jan 28 10:44:32 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 28 Jan 2016 09:44:32 -0600 Subject: [Python-Dev] Fun with ancient unsupported platforms In-Reply-To: <56AA2C50.7050700@hastings.org> References: <56AA2C50.7050700@hastings.org> Message-ID: win16 doesn't seem to have important stuff: https://github.com/python/cpython/search?utf8=?&q="win16" On January 28, 2016 8:57:20 AM CST, Larry Hastings wrote: > > >Check out and cd into Python trunk. > >% grep -Ri win16 * | wc > 10 66 625 > >% grep -Ri nextstep | wc > 23 119 1328 > >% grep -Ri rhapsody * | wc > 47 269 3390 > >% grep -Ri msdos * | wc > 56 381 3895 > % grep -Ri ms-dos * | wc > 20 180 1425 > > >win16! *laughs* *wipes tear from eye* > >It's currently 2016. Perhaps it's time to remove all vestiges of these > >unsupported operating systems nobody's cared about since a year that >started with a '1'? > > >//arry/ > >p.s. I suspect some of the uses of "rhapsody" are actually live code >used for OS X. So this isn't necessarily dead code, some of it is >merely long-out-of-date comments. > >p.p.s. At least there were no references to "taligent", "copland", or >"gershwin"...! > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Jan 28 11:00:31 2016 From: barry at python.org (Barry Warsaw) Date: Thu, 28 Jan 2016 11:00:31 -0500 Subject: [Python-Dev] Fun with ancient unsupported platforms In-Reply-To: References: <56AA2C50.7050700@hastings.org> Message-ID: <20160128110031.7e7e2295@subdivisions.wooz.org> Just as long as you can still build and run Python on Guido's ancient SGI machine . -Barry From victor.stinner at gmail.com Thu Jan 28 11:29:41 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 28 Jan 2016 17:29:41 +0100 Subject: [Python-Dev] Fun with ancient unsupported platforms In-Reply-To: <56AA2C50.7050700@hastings.org> References: <56AA2C50.7050700@hastings.org> Message-ID: We slowly remove old platforms, but only if the code specific to these old platforms is annoying to maintain. For example, I wrote the change: https://hg.python.org/cpython/rev/a1605d2508af """ Issue #22591: Drop support of MS-DOS Drop support of MS-DOS, especially of the DJGPP compiler (MS-DOS port of GCC). Today is a sad day. Good bye MS-DOS, good bye my friend :'-( """ => http://bugs.python.org/issue22591 Victor 2016-01-28 15:57 GMT+01:00 Larry Hastings : > > > Check out and cd into Python trunk. > > % grep -Ri win16 * | wc > 10 66 625 > > % grep -Ri nextstep | wc > 23 119 1328 > > % grep -Ri rhapsody * | wc > 47 269 3390 > > % grep -Ri msdos * | wc > 56 381 3895 > % grep -Ri ms-dos * | wc > 20 180 1425 > > > win16! *laughs* *wipes tear from eye* > > It's currently 2016. Perhaps it's time to remove all vestiges of these > unsupported operating systems nobody's cared about since a year that started > with a '1'? > > > /arry > > p.s. I suspect some of the uses of "rhapsody" are actually live code used > for OS X. So this isn't necessarily dead code, some of it is merely > long-out-of-date comments. > > p.p.s. At least there were no references to "taligent", "copland", or > "gershwin"...! > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From pmiscml at gmail.com Thu Jan 28 13:38:36 2016 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Thu, 28 Jan 2016 20:38:36 +0200 Subject: [Python-Dev] Fun with ancient unsupported platforms In-Reply-To: References: <56AA2C50.7050700@hastings.org> Message-ID: <20160128203836.7e65ed2d@x230> Hello, On Thu, 28 Jan 2016 17:29:41 +0100 Victor Stinner wrote: > We slowly remove old platforms, but only if the code specific to these > old platforms is annoying to maintain. For example, I wrote the > change: > > https://hg.python.org/cpython/rev/a1605d2508af > """ > Issue #22591: Drop support of MS-DOS > > Drop support of MS-DOS, especially of the DJGPP compiler (MS-DOS port > of GCC). > > Today is a sad day. Good bye MS-DOS, good bye my friend :'-( > """ > => http://bugs.python.org/issue22591 Well, MicroPython is ready to take the baton - FreeDOS (that's how they call MS-DOS now) support was added recently: https://github.com/micropython/micropython/commit/64a909ef5113925adef19f275f62473de8ee68c5 > > Victor > [] -- Best regards, Paul mailto:pmiscml at gmail.com From rosuav at gmail.com Thu Jan 28 22:47:03 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 29 Jan 2016 14:47:03 +1100 Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6? In-Reply-To: References: Message-ID: On Thu, Jan 28, 2016 at 9:41 PM, Chris Angelico wrote: > However, as I was doing so (and I just discarded a draft message where > I'd been typing up notes), my entire system went kerblooie, and I've > spent the last day rebuilding stuff from scratch. When I get around to > it, I'll rebuild the buildbot VM - and it'll be Debian Jessie (current > stable), because there's no particular reason to use Wheezy > (oldstable). So the problem will almost certainly disappear. Sure enough, the problem is no longer reproducible. Sorry folks. Hopefully it wasn't actually a bug anywhere, but was some bizarre piece of VM setup weirdness. The "AMD64 Debian root" buildbot (angelico-debian-amd64) is now running Debian Jessie, rather than the Wheezy it was before. ChrisA From pludemann at google.com Fri Jan 29 03:05:09 2016 From: pludemann at google.com (Peter Ludemann) Date: Fri, 29 Jan 2016 00:05:09 -0800 Subject: [Python-Dev] FAT Python (lack of) performance In-Reply-To: References: <56A698CF.3080402@mail.de> <56A7AE7C.3080806@mail.de> <56A90F2C.9000401@mail.de> Message-ID: About benchmarks ... I've been there and it's not benchmarks that decide whether something succeeds or fails. (I found this old discussion which mentions FIB (also TAK , which is rather more brutal) ... do you recognize the language that got an implementation that was competitive with C in performance, was vastly more expressive, yet failed to catch on?) OTOH, good performance is never a bad thing and sometimes is a necessity; so I applau this work. On 28 January 2016 at 01:39, Nick Coghlan wrote: > On 28 January 2016 at 18:30, INADA Naoki wrote: > > Please stop. > > > > I'm sorry about messing up this thread. > > Not your fault at all! This is just a particular bugbear of mine, > since software architecture design (including appropriate programming > language selection) is an even more poorly understood discipline than > software development in general :) > > > I just wanted to represent why I'm very interested in Victor's efforts. > > And thanks for posting that, as it is indeed cool that the > optimisation efforts currently being discussed may result in > performance improvements on some of the simplified micro-benchmarks > popular in programming language shootouts. > > There's no way you could have anticipated the subsequent tangential > discussion on motives for contributing to open source projects, and > the impact that has on what we can reasonably expect from fellow > contributors. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/pludemann%40google.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Jan 29 05:00:02 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jan 2016 11:00:02 +0100 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <56A90B97.7090001@gmail.com> References: <56A90B97.7090001@gmail.com> Message-ID: Yury Selivanov schrieb am 27.01.2016 um 19:25: > tl;dr The summary is that I have a patch that improves CPython performance > up to 5-10% on macro benchmarks. Benchmarks results on Macbook Pro/Mac OS > X, desktop CPU/Linux, server CPU/Linux are available at [1]. There are no > slowdowns that I could reproduce consistently. > > There are two different optimizations that yield this speedup: > LOAD_METHOD/CALL_METHOD opcodes and per-opcode cache in ceval loop. > > LOAD_METHOD & CALL_METHOD > ------------------------- > > We had a lot of conversations with Victor about his PEP 509, and he sent me > a link to his amazing compilation of notes about CPython performance [2]. > One optimization that he pointed out to me was LOAD/CALL_METHOD opcodes, an > idea first originated in PyPy. > > There is a patch that implements this optimization, it's tracked here: > [3]. There are some low level details that I explained in the issue, but > I'll go over the high level design in this email as well. > > Every time you access a method attribute on an object, a BoundMethod object > is created. It is a fairly expensive operation, despite a freelist of > BoundMethods (so that memory allocation is generally avoided). The idea is > to detect what looks like a method call in the compiler, and emit a pair of > specialized bytecodes for that. > > So instead of LOAD_GLOBAL/LOAD_ATTR/CALL_FUNCTION we will have > LOAD_GLOBAL/LOAD_METHOD/CALL_METHOD. > > LOAD_METHOD looks at the object on top of the stack, and checks if the name > resolves to a method or to a regular attribute. If it's a method, then we > push the unbound method object and the object to the stack. If it's an > attribute, we push the resolved attribute and NULL. > > When CALL_METHOD looks at the stack it knows how to call the unbound method > properly (pushing the object as a first arg), or how to call a regular > callable. > > This idea does make CPython faster around 2-4%. And it surely doesn't make > it slower. I think it's a safe bet to at least implement this optimization > in CPython 3.6. > > So far, the patch only optimizes positional-only method calls. It's > possible to optimize all kind of calls, but this will necessitate 3 more > opcodes (explained in the issue). We'll need to do some careful > benchmarking to see if it's really needed. I implemented a similar but simpler optimisation in Cython a while back: http://blog.behnel.de/posts/faster-python-calls-in-cython-021.html Instead of avoiding the creation of method objects, as you proposed, it just normally calls getattr and if that returns a bound method object, it uses inlined calling code that avoids re-packing the argument tuple. Interestingly, I got speedups of 5-15% for some of the Python benchmarks, but I don't quite remember which ones (at least raytrace and richards, I think), nor do I recall the overall gain, which (I assume) is what you are referring to with your 2-4% above. Might have been in the same order. Stefan From damien.p.george at gmail.com Fri Jan 29 07:38:53 2016 From: damien.p.george at gmail.com (Damien George) Date: Fri, 29 Jan 2016 12:38:53 +0000 Subject: [Python-Dev] Speeding up CPython 5-10% Message-ID: Hi Yury, > An off-topic: have you ever tried hg.python.org/benchmarks > or compare MicroPython vs CPython? I'm curious if MicroPython > is faster -- in that case we'll try to copy some optimization > ideas. I've tried a small number of those benchmarks, but not in any rigorous way, and not enough to compare properly with CPython. Maybe one day I (or someone) will get to it and report results :) One thing that makes MP fast is the use of pointer tagging and stuffing of small integers within object pointers. Thus integer arithmetic below 2**30 (on 32-bit arch) requires no heap. > Do you use opcode dictionary caching only for LOAD_GLOBAL-like > opcodes? Do you have an equivalent of LOAD_FAST, or you use > dicts to store local variables? The opcodes that have dict caching are: LOAD_NAME LOAD_GLOBAL LOAD_ATTR STORE_ATTR LOAD_METHOD (not implemented yet in mainline repo) For local variables we use LOAD_FAST and STORE_FAST (and DELETE_FAST). Actually, there are 16 dedicated opcodes for loading from positions 0-15, and 16 for storing to these positions. Eg: LOAD_FAST_0 LOAD_FAST_1 ... Mostly this is done to save RAM, since LOAD_FAST_0 is 1 byte. > If we change the opcode size, it will probably affect libraries > that compose or modify code objects. Modules like "dis" will > also need to be updated. And that's probably just a tip of the > iceberg. > > We can still implement your approach if we add a separate > private 'unsigned char' array to each code object, so that > LOAD_GLOBAL can store the key offsets. It should be a bit > faster than my current patch, since it has one less level > of indirection. But this way we loose the ability to > optimize LOAD_METHOD, simply because it requires more memory > for its cache. In any case, I'll experiment! Problem with that approach (having a separate array for offset_guess) is that how do you know where to look into that array for a given LOAD_GLOBAL opcode? The second LOAD_GLOBAL in your bytecode should look into the second entry in the array, but how does it know? I'd love to experiment implementing my original caching idea with CPython, but no time! Cheers, Damien. From yselivanov.ml at gmail.com Fri Jan 29 09:49:11 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 29 Jan 2016 09:49:11 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: <56A90B97.7090001@gmail.com> Message-ID: <56AB7BE7.9060902@gmail.com> On 2016-01-29 5:00 AM, Stefan Behnel wrote: > Yury Selivanov schrieb am 27.01.2016 um 19:25: >> [..] >> >> LOAD_METHOD looks at the object on top of the stack, and checks if the name >> resolves to a method or to a regular attribute. If it's a method, then we >> push the unbound method object and the object to the stack. If it's an >> attribute, we push the resolved attribute and NULL. >> >> When CALL_METHOD looks at the stack it knows how to call the unbound method >> properly (pushing the object as a first arg), or how to call a regular >> callable. >> >> This idea does make CPython faster around 2-4%. And it surely doesn't make >> it slower. I think it's a safe bet to at least implement this optimization >> in CPython 3.6. >> >> So far, the patch only optimizes positional-only method calls. It's >> possible to optimize all kind of calls, but this will necessitate 3 more >> opcodes (explained in the issue). We'll need to do some careful >> benchmarking to see if it's really needed. > I implemented a similar but simpler optimisation in Cython a while back: > > http://blog.behnel.de/posts/faster-python-calls-in-cython-021.html > > Instead of avoiding the creation of method objects, as you proposed, it > just normally calls getattr and if that returns a bound method object, it > uses inlined calling code that avoids re-packing the argument tuple. > Interestingly, I got speedups of 5-15% for some of the Python benchmarks, > but I don't quite remember which ones (at least raytrace and richards, I > think), nor do I recall the overall gain, which (I assume) is what you are > referring to with your 2-4% above. Might have been in the same order. > That's great! I'm still working on the patch, but so far it looks like adding just LOAD_METHOD/CALL_METHOD (that avoid instantiating BoundMethods) gives us 10-15% faster method calls. Combining them with my opcode cache makes them 30-35% faster. Yury From yselivanov.ml at gmail.com Fri Jan 29 10:06:38 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 29 Jan 2016 10:06:38 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: Message-ID: <56AB7FFE.7010004@gmail.com> Hi Damien, BTW I just saw (and backed!) your new Kickstarter campaign to port MicroPython to ESP8266, good stuff! On 2016-01-29 7:38 AM, Damien George wrote: > Hi Yury, > > [..] >> Do you use opcode dictionary caching only for LOAD_GLOBAL-like >> opcodes? Do you have an equivalent of LOAD_FAST, or you use >> dicts to store local variables? > The opcodes that have dict caching are: > > LOAD_NAME > LOAD_GLOBAL > LOAD_ATTR > STORE_ATTR > LOAD_METHOD (not implemented yet in mainline repo) > > For local variables we use LOAD_FAST and STORE_FAST (and DELETE_FAST). > Actually, there are 16 dedicated opcodes for loading from positions > 0-15, and 16 for storing to these positions. Eg: > > LOAD_FAST_0 > LOAD_FAST_1 > ... > > Mostly this is done to save RAM, since LOAD_FAST_0 is 1 byte. Interesting. This might actually make CPython slightly faster too. Worth trying. > >> If we change the opcode size, it will probably affect libraries >> that compose or modify code objects. Modules like "dis" will >> also need to be updated. And that's probably just a tip of the >> iceberg. >> >> We can still implement your approach if we add a separate >> private 'unsigned char' array to each code object, so that >> LOAD_GLOBAL can store the key offsets. It should be a bit >> faster than my current patch, since it has one less level >> of indirection. But this way we loose the ability to >> optimize LOAD_METHOD, simply because it requires more memory >> for its cache. In any case, I'll experiment! > Problem with that approach (having a separate array for offset_guess) > is that how do you know where to look into that array for a given > LOAD_GLOBAL opcode? The second LOAD_GLOBAL in your bytecode should > look into the second entry in the array, but how does it know? > > I've changed my approach a little bit. Now I have a simple function [1] to initialize the cache for code objects that are called frequently enough. It walks through the code object's opcodes and creates the appropriate offset/cache tables. Then, in ceval loop I have a couple of convenient macros to work with the cache [2]. They use INSTR_OFFSET() macro to locate the cache entry via the offset table initialized by [1]. Thanks, Yury [1] https://github.com/1st1/cpython/blob/opcache4/Objects/codeobject.c#L167 [2] https://github.com/1st1/cpython/blob/opcache4/Python/ceval.c#L1164 From steve.dower at python.org Fri Jan 29 12:05:18 2016 From: steve.dower at python.org (Steve Dower) Date: Fri, 29 Jan 2016 09:05:18 -0800 Subject: [Python-Dev] More optimisation ideas Message-ID: <56AB9BCE.2080000@python.org> Since we're all talking about making Python faster, I thought I'd drop some previous ideas I've had here in case (1) someone wants to actually do them, and (2) they really are new ideas that haven't failed in the past. Mostly I was thinking about startup time. Here are the list of modules imported on clean startup on my Windows, US-English machine (from -v and cleaned up a bit): import _frozen_importlib import _imp import sys import '_warnings' import '_thread' import '_weakref' import '_frozen_importlib_external' import '_io' import 'marshal' import 'nt' import '_thread' import '_weakref' import 'winreg' import 'zipimport' import '_codecs' import 'codecs' import 'encodings.aliases' import 'encodings' import 'encodings.mbcs' import '_signal' import 'encodings.utf_8' import 'encodings.latin_1' import '_weakrefset' import 'abc' import 'io' import 'encodings.cp437' import 'errno' import '_stat' import 'stat' import 'genericpath' import 'ntpath' import '_collections_abc' import 'os' import '_sitebuiltins' import 'sysconfig' import '_locale' import '_bootlocale' import 'encodings.cp1252' import 'site' Obviously the easiest first thing is to remove or delay unnecessary imports. But a while ago I used a native profiler to trace through this and the most impactful modules were the encodings: import 'encodings.mbcs' import 'encodings.utf_8' import 'encodings.latin_1' import 'encodings.cp437' import 'encodings.cp1252' While I don't doubt that we need all of these for *some* reason, aliases, cp437 and cp1252 are relatively expensive modules to import. Mostly due to having large static dictionaries or data structures generated on startup. Given this is static and mostly read-only information[1], I see no reason why we couldn't either generate completely static versions of them, or better yet compile the resulting data structures into the core binary. ([1]: If being able to write to some of the encoding data is used by some people, I vote for breaking that for 3.6 and making it read-only.) This is probably the code snippet that bothered me the most: ### Encoding table encoding_table=codecs.charmap_build(decoding_table) It shows up in many of the encodings modules, and while it is not a bad function in itself, we are obviously generating a known data structure on every startup. Storing these in static data is a tradeoff between disk space and startup performance, and one I think it likely to be worthwhile. Anyway, just an idea if someone wants to try it and see what improvements we can get. I'd love to do it myself, but when it actually comes to finding time I keep coming up short. Cheers, Steve P.S. If you just want to discuss optimisation techniques or benchmarking in general, without specific application to CPython 3.6, there's a whole internet out there. Please don't make me the cause of a pointless centithread. :) From status at bugs.python.org Fri Jan 29 12:08:30 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 29 Jan 2016 18:08:30 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20160129170830.60F3156645@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-01-22 - 2016-01-29) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5381 (+27) closed 32615 (+30) total 37996 (+57) Open issues with patches: 2359 Issues opened (39) ================== #26182: Deprecation warnings for the future async and await keywords http://bugs.python.org/issue26182 opened by marco.buttu #26184: raise an error when create_module() is not defined by exec_mod http://bugs.python.org/issue26184 opened by brett.cannon #26185: zipfile.ZipInfo slots can raise unexpected AttributeError http://bugs.python.org/issue26185 opened by Matthew Zipay #26186: LazyLoader rejecting use of SourceFileLoader http://bugs.python.org/issue26186 opened by brett.cannon #26187: sqlite3 trace callback prints duplicate line http://bugs.python.org/issue26187 opened by palaviv #26188: Provide more helpful error message when `await` is called insi http://bugs.python.org/issue26188 opened by Nicholas Chammas #26192: python3 k1om dissociation permanence: libffi http://bugs.python.org/issue26192 opened by mancoast #26193: python3 k1om dissociation permanence: readelf http://bugs.python.org/issue26193 opened by mancoast #26194: Undefined behavior for deque.insert() when len(d) == maxlen http://bugs.python.org/issue26194 opened by rhettinger #26195: Windows frozen .exe multiprocessing.Queue access is denied exc http://bugs.python.org/issue26195 opened by alex_python_org #26198: PyArg_ParseTuple with format "et#" and "es#" detects overflow http://bugs.python.org/issue26198 opened by hniksic #26200: SETREF adds unnecessary work in some cases http://bugs.python.org/issue26200 opened by rhettinger #26204: compiler: ignore constants used as statements? (don't emit LOA http://bugs.python.org/issue26204 opened by haypo #26205: Inconsistency concerning nested scopes http://bugs.python.org/issue26205 opened by Roscoe R. Higgins #26207: distutils msvccompiler fails due to mspdb140.dll error on debu http://bugs.python.org/issue26207 opened by haypo #26208: decimal C module's exceptions don't match the Python version http://bugs.python.org/issue26208 opened by encukou #26209: TypeError in smtpd module with string arguments http://bugs.python.org/issue26209 opened by lorenzo.ancora #26210: `HTMLParser.handle_data` may be invoked although `HTMLParser.r http://bugs.python.org/issue26210 opened by Hibou57 #26212: Python with ncurses6.0 will not load _curses module on Solaris http://bugs.python.org/issue26212 opened by jonesrw #26213: Document BUILD_*_UNPACK opcodes http://bugs.python.org/issue26213 opened by brett.cannon #26214: textwrap should minimize number of breaks in extra long words http://bugs.python.org/issue26214 opened by Tuomas Salo #26215: Make GC_Head a compile-time option http://bugs.python.org/issue26215 opened by yuriy_levchenko #26216: run runtktests.py error when test tkinter http://bugs.python.org/issue26216 opened by allensll #26218: Set PrependPath default to true http://bugs.python.org/issue26218 opened by Wallison Resende Santos #26219: implement per-opcode cache in ceval http://bugs.python.org/issue26219 opened by yselivanov #26221: asynco run_in_executor swallows StopIteration http://bugs.python.org/issue26221 opened by ikelly #26222: Missing code in linux_distribution python 2.7.11 http://bugs.python.org/issue26222 opened by Rasmus Rynning Rasmussen #26223: decimal.to_eng_string() does not implement engineering notatio http://bugs.python.org/issue26223 opened by serge.stroobandt #26224: Add "version added" for documentation of asyncio.timeout for d http://bugs.python.org/issue26224 opened by Udi Oron #26225: New misleading wording in execution model documenation http://bugs.python.org/issue26225 opened by abarnert #26226: Various test suite failures on Windows http://bugs.python.org/issue26226 opened by ebarry #26228: pty.spawn hangs on FreeBSD 9.3, 10.x http://bugs.python.org/issue26228 opened by chris.torek #26229: Make number serialization ES6/V8 compatible http://bugs.python.org/issue26229 opened by anders.rundgren.net at gmail.com #26231: HTTPResponse.close() should consume all remaining data in body http://bugs.python.org/issue26231 opened by Jacky #26233: select.epoll.poll() should avoid calling malloc() each time http://bugs.python.org/issue26233 opened by haypo #26234: The typing module includes 're' and 'io' in __all__ http://bugs.python.org/issue26234 opened by gvanrossum #26235: argparse docs: Positional * argument in mutually exclusive gro http://bugs.python.org/issue26235 opened by paul.j3 #26236: urllib2 initiate irregular call to gethostbyaddr http://bugs.python.org/issue26236 opened by juliadolgova #26238: httplib use wrong hostname in https request with SNI support http://bugs.python.org/issue26238 opened by lvhancy Most recent 15 issues with no replies (15) ========================================== #26238: httplib use wrong hostname in https request with SNI support http://bugs.python.org/issue26238 #26236: urllib2 initiate irregular call to gethostbyaddr http://bugs.python.org/issue26236 #26235: argparse docs: Positional * argument in mutually exclusive gro http://bugs.python.org/issue26235 #26224: Add "version added" for documentation of asyncio.timeout for d http://bugs.python.org/issue26224 #26216: run runtktests.py error when test tkinter http://bugs.python.org/issue26216 #26214: textwrap should minimize number of breaks in extra long words http://bugs.python.org/issue26214 #26209: TypeError in smtpd module with string arguments http://bugs.python.org/issue26209 #26200: SETREF adds unnecessary work in some cases http://bugs.python.org/issue26200 #26195: Windows frozen .exe multiprocessing.Queue access is denied exc http://bugs.python.org/issue26195 #26193: python3 k1om dissociation permanence: readelf http://bugs.python.org/issue26193 #26192: python3 k1om dissociation permanence: libffi http://bugs.python.org/issue26192 #26187: sqlite3 trace callback prints duplicate line http://bugs.python.org/issue26187 #26185: zipfile.ZipInfo slots can raise unexpected AttributeError http://bugs.python.org/issue26185 #26184: raise an error when create_module() is not defined by exec_mod http://bugs.python.org/issue26184 #26176: EmailMessage example doesn't work http://bugs.python.org/issue26176 Most recent 15 issues waiting for review (15) ============================================= #26233: select.epoll.poll() should avoid calling malloc() each time http://bugs.python.org/issue26233 #26228: pty.spawn hangs on FreeBSD 9.3, 10.x http://bugs.python.org/issue26228 #26224: Add "version added" for documentation of asyncio.timeout for d http://bugs.python.org/issue26224 #26219: implement per-opcode cache in ceval http://bugs.python.org/issue26219 #26204: compiler: ignore constants used as statements? (don't emit LOA http://bugs.python.org/issue26204 #26198: PyArg_ParseTuple with format "et#" and "es#" detects overflow http://bugs.python.org/issue26198 #26194: Undefined behavior for deque.insert() when len(d) == maxlen http://bugs.python.org/issue26194 #26192: python3 k1om dissociation permanence: libffi http://bugs.python.org/issue26192 #26185: zipfile.ZipInfo slots can raise unexpected AttributeError http://bugs.python.org/issue26185 #26177: tkinter: Canvas().keys returns empty strings. http://bugs.python.org/issue26177 #26175: Fully implement IOBase abstract on SpooledTemporaryFile http://bugs.python.org/issue26175 #26173: test_ssl.bad_cert_test() exception handling http://bugs.python.org/issue26173 #26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure http://bugs.python.org/issue26168 #26167: Improve copy.copy speed for built-in types (list/set/dict) http://bugs.python.org/issue26167 #26145: PEP 511: Add sys.set_code_transformers() http://bugs.python.org/issue26145 Top 10 most discussed issues (10) ================================= #26194: Undefined behavior for deque.insert() when len(d) == maxlen http://bugs.python.org/issue26194 17 msgs #26207: distutils msvccompiler fails due to mspdb140.dll error on debu http://bugs.python.org/issue26207 12 msgs #26198: PyArg_ParseTuple with format "et#" and "es#" detects overflow http://bugs.python.org/issue26198 11 msgs #26039: More flexibility in zipfile interface http://bugs.python.org/issue26039 7 msgs #26219: implement per-opcode cache in ceval http://bugs.python.org/issue26219 7 msgs #26223: decimal.to_eng_string() does not implement engineering notatio http://bugs.python.org/issue26223 7 msgs #19883: Integer overflow in zipimport.c http://bugs.python.org/issue19883 6 msgs #20598: argparse docs: '7'.split() is confusing magic http://bugs.python.org/issue20598 6 msgs #25314: Documentation: argparse's actions store_{true,false} default t http://bugs.python.org/issue25314 6 msgs #26204: compiler: ignore constants used as statements? (don't emit LOA http://bugs.python.org/issue26204 6 msgs Issues closed (27) ================== #18018: SystemError: Parent module '' not loaded, cannot perform relat http://bugs.python.org/issue18018 closed by brett.cannon #18898: Apply the setobject optimizations to dictionaries http://bugs.python.org/issue18898 closed by rhettinger #19023: ctypes docs: Unimplemented and undocumented features http://bugs.python.org/issue19023 closed by martin.panter #22363: argparse AssertionError with add_mutually_exclusive_group and http://bugs.python.org/issue22363 closed by martin.panter #24705: sysconfig._parse_makefile doesn't expand ${} vars appearing be http://bugs.python.org/issue24705 closed by berker.peksag #25296: Simple End-of-life guide covering all unsupported versions http://bugs.python.org/issue25296 closed by berker.peksag #26034: venv documentation out of date http://bugs.python.org/issue26034 closed by berker.peksag #26146: PEP 511: Add ast.Constant to allow AST optimizer to emit const http://bugs.python.org/issue26146 closed by haypo #26181: argparse can't handle positional argument after list (help mes http://bugs.python.org/issue26181 closed by martin.panter #26183: 2.7.11 won't clean install on Windows 10 x64 http://bugs.python.org/issue26183 closed by Roger Cook #26189: Interpreter returns control to cmd.exe early http://bugs.python.org/issue26189 closed by Ivan.Pozdeev #26190: GC memory leak using weak and cyclic references http://bugs.python.org/issue26190 closed by koehlma #26191: pip on Windows doesn't honor Case http://bugs.python.org/issue26191 closed by SilentGhost #26196: Argparse breaks when a switch is given an argument beginning w http://bugs.python.org/issue26196 closed by eric.smith #26197: arange from numpy function has some limits....I propose a pyth http://bugs.python.org/issue26197 closed by ebarry #26199: fix broken link to hamcrest.library.integration.match_equality http://bugs.python.org/issue26199 closed by berker.peksag #26201: Faster type checking in listobject.c http://bugs.python.org/issue26201 closed by rhettinger #26202: The range() object is deepcopied as atomic http://bugs.python.org/issue26202 closed by serhiy.storchaka #26203: nesting venv in virtualenv segfaults http://bugs.python.org/issue26203 closed by brett.cannon #26206: test_socket.testRecvmsgPeek() timeout on "AMD64 Debian root 3. http://bugs.python.org/issue26206 closed by martin.panter #26211: HTMLParser: ???AssertionError: we should not get here!??? http://bugs.python.org/issue26211 closed by Hibou57 #26217: Fatal error when importing ``test.test_os`` in debug mode on W http://bugs.python.org/issue26217 closed by ebarry #26220: Unicode HOWTO references a question mark that isn't in snippet http://bugs.python.org/issue26220 closed by martin.panter #26227: Windows: socket.gethostbyaddr(name) fails for non-ASCII hostna http://bugs.python.org/issue26227 closed by haypo #26230: Cookies do not correct if cookiename includes [ or ] http://bugs.python.org/issue26230 closed by tom_lt #26232: Mock(spec=spec) has no effect http://bugs.python.org/issue26232 closed by michael.foord #26237: UnboundLocalError error while handling exception http://bugs.python.org/issue26237 closed by haypo From storchaka at gmail.com Fri Jan 29 13:14:00 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 29 Jan 2016 20:14:00 +0200 Subject: [Python-Dev] How to resolve distinguishing between documentation and implementation Message-ID: How to resolve distinguishing between documentation and implementation if current implementation is incorrect, but third-party code can implicitly depends on it? For example see issue26198. Currently buffer overflow of predefined buffer for "es#" and "et#" format units causes TypeError (with misleading message, but this is other story). The correct and *documented* exception is ValueError. User code can depend on current behavior, because TypeError is what is raised now for this type of errors, and this is what is raised for other types of errors. Unlikely authors of such code read the documentation, otherwise this issue would be reported earlier. On other hand, looks these format units are rarely used with predefined buffer (never in the stdlib since 3.5). I think it is obvious that the code in the development branch should be changed to produce documented and more logical exception. But what about bugfix releases? Changing the documentation would be misleading, changing the code can break existing code (unlikely, but). From brett at python.org Fri Jan 29 13:26:32 2016 From: brett at python.org (Brett Cannon) Date: Fri, 29 Jan 2016 18:26:32 +0000 Subject: [Python-Dev] How to resolve distinguishing between documentation and implementation In-Reply-To: References: Message-ID: On Fri, 29 Jan 2016 at 10:14 Serhiy Storchaka wrote: > How to resolve distinguishing between documentation and implementation > if current implementation is incorrect, but third-party code can > implicitly depends on it? > > For example see issue26198. Currently buffer overflow of predefined > buffer for "es#" and "et#" format units causes TypeError (with > misleading message, but this is other story). The correct and > *documented* exception is ValueError. User code can depend on current > behavior, because TypeError is what is raised now for this type of > errors, and this is what is raised for other types of errors. Unlikely > authors of such code read the documentation, otherwise this issue would > be reported earlier. On other hand, looks these format units are rarely > used with predefined buffer (never in the stdlib since 3.5). > > I think it is obvious that the code in the development branch should be > changed to produce documented and more logical exception. But what about > bugfix releases? Changing the documentation would be misleading, > changing the code can break existing code (unlikely, but). > When the potential breakage is low, I would move to the more reasonable exception and add the appropriate note to the docs and What's New about how to port pre-existing code by either changing the exception caught or catching both exceptions until support < 3.6 can be dropped by the user. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jan 29 13:24:48 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 29 Jan 2016 10:24:48 -0800 Subject: [Python-Dev] How to resolve distinguishing between documentation and implementation In-Reply-To: References: Message-ID: Typically we fix this in the next feature release but not in bugfix releases, and that's what I recommend in this case. But deciding remains an art, not an exact science. People who catch specific lists of exceptions based on experience are bound to run into trouble occasionally, so I have little pity on them when they upgrade to the next feature release (especially when the new behavior is more logical and follows an established standard). For bugfix releases I like to be much more conservative, since people often don't have control over which bugfix release is installed. On Fri, Jan 29, 2016 at 10:14 AM, Serhiy Storchaka wrote: > How to resolve distinguishing between documentation and implementation if > current implementation is incorrect, but third-party code can implicitly > depends on it? > > For example see issue26198. Currently buffer overflow of predefined buffer > for "es#" and "et#" format units causes TypeError (with misleading message, > but this is other story). The correct and *documented* exception is > ValueError. User code can depend on current behavior, because TypeError is > what is raised now for this type of errors, and this is what is raised for > other types of errors. Unlikely authors of such code read the documentation, > otherwise this issue would be reported earlier. On other hand, looks these > format units are rarely used with predefined buffer (never in the stdlib > since 3.5). > > I think it is obvious that the code in the development branch should be > changed to produce documented and more logical exception. But what about > bugfix releases? Changing the documentation would be misleading, changing > the code can break existing code (unlikely, but). > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From francismb at email.de Fri Jan 29 16:54:35 2016 From: francismb at email.de (francismb) Date: Fri, 29 Jan 2016 22:54:35 +0100 Subject: [Python-Dev] More optimisation ideas In-Reply-To: <56AB9BCE.2080000@python.org> References: <56AB9BCE.2080000@python.org> Message-ID: <56ABDF9B.9040502@email.de> Hi, > > Storing these in static data is a tradeoff between > disk space and startup performance, and one I think it likely to be > worthwhile. it's really an important trade off? As far a I understand from your email those modules are always being loaded and the final data created. won't the space be there (on mem or disk)? Thanks in advance! francis From steve.dower at python.org Fri Jan 29 22:48:41 2016 From: steve.dower at python.org (Steve Dower) Date: Fri, 29 Jan 2016 19:48:41 -0800 Subject: [Python-Dev] More optimisation ideas In-Reply-To: <56ABDF9B.9040502@email.de> References: <56AB9BCE.2080000@python.org> <56ABDF9B.9040502@email.de> Message-ID: It doesn't currently end up on disk. Some tables are partially or completely stored on disk as Python source code (some are partially generated from simple rules), but others are generated by inverting those. That process takes time that could be avoided by storing the generated tables, and storing all of it in a format that doesn't require parsing, compiling and executing (such as a native array). Potentially it could be a win all around if we stopped including the (larger) source files, but that doesn't seem like a good idea for maintaining portability to other implementations. The main thought is making the compiler binary bigger to avoid generating encoding tables at startup. Top-posted from my Windows Phone -----Original Message----- From: "francismb" Sent: ?1/?29/?2016 13:56 To: "python-dev at python.org" Subject: Re: [Python-Dev] More optimisation ideas Hi, > > Storing these in static data is a tradeoff between > disk space and startup performance, and one I think it likely to be > worthwhile. it's really an important trade off? As far a I understand from your email those modules are always being loaded and the final data created. won't the space be there (on mem or disk)? Thanks in advance! francis _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Jan 29 23:28:35 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 30 Jan 2016 15:28:35 +1100 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <56A90B97.7090001@gmail.com> References: <56A90B97.7090001@gmail.com> Message-ID: <20160130042835.GJ4619@ando.pearwood.info> On Wed, Jan 27, 2016 at 01:25:27PM -0500, Yury Selivanov wrote: > Hi, > > > tl;dr The summary is that I have a patch that improves CPython > performance up to 5-10% on macro benchmarks. Benchmarks results on > Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are available > at [1]. There are no slowdowns that I could reproduce consistently. Have you looked at Cesare Di Mauro's wpython? As far as I know, it's now unmaintained, and the project repo on Google Code appears to be dead (I get a 404), but I understand that it was significantly faster than CPython back in the 2.6 days. https://wpython.googlecode.com/files/Beyond%20Bytecode%20-%20A%20Wordcode-based%20Python.pdf -- Steve From oscar.j.benjamin at gmail.com Sat Jan 30 08:54:54 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 30 Jan 2016 13:54:54 +0000 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> <56ABDF9B.9040502@email.de> Message-ID: On 30 January 2016 at 03:48, Steve Dower wrote: > > It doesn't currently end up on disk. Some tables are partially or completely > stored on disk as Python source code (some are partially generated from > simple rules), but others are generated by inverting those. That process > takes time that could be avoided by storing the generated tables, and > storing all of it in a format that doesn't require parsing, compiling and > executing (such as a native array). > > Potentially it could be a win all around if we stopped including the > (larger) source files, but that doesn't seem like a good idea for > maintaining portability to other implementations. The main thought is making > the compiler binary bigger to avoid generating encoding tables at startup. When I last tried to profile startup on Windows (I haven't used Windows for some time now) it seemed that the time was totally dominated by file system access. Essentially the limiting factor was the inordinate number of stat calls and small file accesses. Although this was probably Python 2.x which may not import those particular modules and maybe it depends on virus scanner software etc. Things may have changed now but I concluded that substantive gains could only come from improving FS access. Perhaps something like zipping up the standard library would see a big improvement. -- Oscar From storchaka at gmail.com Sat Jan 30 09:45:44 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 30 Jan 2016 16:45:44 +0200 Subject: [Python-Dev] More optimisation ideas In-Reply-To: <56AB9BCE.2080000@python.org> References: <56AB9BCE.2080000@python.org> Message-ID: On 29.01.16 19:05, Steve Dower wrote: > This is probably the code snippet that bothered me the most: > > ### Encoding table > encoding_table=codecs.charmap_build(decoding_table) > > It shows up in many of the encodings modules, and while it is not a bad > function in itself, we are obviously generating a known data structure > on every startup. Storing these in static data is a tradeoff between > disk space and startup performance, and one I think it likely to be > worthwhile. $ ./python -m timeit -s "import codecs; from encodings.cp437 import decoding_table" -- "codecs.charmap_build(decoding_table)" 100000 loops, best of 3: 4.36 usec per loop Getting rid from charmap_build() would save you at most 4.4 microseconds per encoding. 0.0005 seconds if you have imported *all* standard encodings! And how you expected to store encoding_table in more efficient way? From steve.dower at python.org Sat Jan 30 11:31:32 2016 From: steve.dower at python.org (Steve Dower) Date: Sat, 30 Jan 2016 08:31:32 -0800 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> Message-ID: <56ACE564.7080107@python.org> On 30Jan2016 0645, Serhiy Storchaka wrote: > $ ./python -m timeit -s "import codecs; from encodings.cp437 import > decoding_table" -- "codecs.charmap_build(decoding_table)" > 100000 loops, best of 3: 4.36 usec per loop > > Getting rid from charmap_build() would save you at most 4.4 microseconds > per encoding. 0.0005 seconds if you have imported *all* standard encodings! Just as happy to be proven wrong. Perhaps I misinterpreted my original profiling and then, embarrassingly, ran with the result for a long time without retesting. > And how you expected to store encoding_table in more efficient way? There's nothing inefficient about its storage, but as it does not change it would be trivial to store it statically. Then "building" the map is simply obtaining a pointer into an already loaded memory page. Much faster than building it on load, but both are clearly insignificant compared to other factors. Cheers, Steve From storchaka at gmail.com Sat Jan 30 13:20:08 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 30 Jan 2016 20:20:08 +0200 Subject: [Python-Dev] More optimisation ideas In-Reply-To: <56ACE564.7080107@python.org> References: <56AB9BCE.2080000@python.org> <56ACE564.7080107@python.org> Message-ID: On 30.01.16 18:31, Steve Dower wrote: > On 30Jan2016 0645, Serhiy Storchaka wrote: >> $ ./python -m timeit -s "import codecs; from encodings.cp437 import >> decoding_table" -- "codecs.charmap_build(decoding_table)" >> 100000 loops, best of 3: 4.36 usec per loop >> >> Getting rid from charmap_build() would save you at most 4.4 microseconds >> per encoding. 0.0005 seconds if you have imported *all* standard >> encodings! > > Just as happy to be proven wrong. Perhaps I misinterpreted my original > profiling and then, embarrassingly, ran with the result for a long time > without retesting. AFAIK the most time is spent in system calls like stat or open. Archiving the stdlib into the ZIP file and using zipimport can decrease Python startup time (perhaps there is an open issue about this). From brett at python.org Sat Jan 30 14:09:43 2016 From: brett at python.org (Brett Cannon) Date: Sat, 30 Jan 2016 19:09:43 +0000 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> <56ACE564.7080107@python.org> Message-ID: On Sat, 30 Jan 2016 at 10:21 Serhiy Storchaka wrote: > On 30.01.16 18:31, Steve Dower wrote: > > On 30Jan2016 0645, Serhiy Storchaka wrote: > >> $ ./python -m timeit -s "import codecs; from encodings.cp437 import > >> decoding_table" -- "codecs.charmap_build(decoding_table)" > >> 100000 loops, best of 3: 4.36 usec per loop > >> > >> Getting rid from charmap_build() would save you at most 4.4 microseconds > >> per encoding. 0.0005 seconds if you have imported *all* standard > >> encodings! > > > > Just as happy to be proven wrong. Perhaps I misinterpreted my original > > profiling and then, embarrassingly, ran with the result for a long time > > without retesting. > > AFAIK the most time is spent in system calls like stat or open. > Archiving the stdlib into the ZIP file and using zipimport can decrease > Python startup time (perhaps there is an open issue about this). > Check the archives, but I did trying freezing the entire stdlib and it didn't really make a difference in startup, so I don't know if this still holds true anymore. At this point I think all of our knowledge of what takes the most amount of time during startup is outdated and someone should try to really profile the whole thing to see where the hotspots are (e.g., is it stat calls from imports, is it actually some specific function, is it just so many little things adding up to a big thing, etc.). -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sat Jan 30 14:15:26 2016 From: steve.dower at python.org (Steve Dower) Date: Sat, 30 Jan 2016 11:15:26 -0800 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> <56ACE564.7080107@python.org> Message-ID: Brett tried freezing the entire stdlib at one point (as we do for parts of importlib) and reported no significant improvement. Since that rules out code compilation as well as the OS calls, it'd seem the priority is to execute less code on startup. Details of that work were posted to python-dev about twelve months ago, IIRC. Maybe a little longer. Top-posted from my Windows Phone -----Original Message----- From: "Serhiy Storchaka" Sent: ?1/?30/?2016 10:22 To: "python-dev at python.org" Subject: Re: [Python-Dev] More optimisation ideas On 30.01.16 18:31, Steve Dower wrote: > On 30Jan2016 0645, Serhiy Storchaka wrote: >> $ ./python -m timeit -s "import codecs; from encodings.cp437 import >> decoding_table" -- "codecs.charmap_build(decoding_table)" >> 100000 loops, best of 3: 4.36 usec per loop >> >> Getting rid from charmap_build() would save you at most 4.4 microseconds >> per encoding. 0.0005 seconds if you have imported *all* standard >> encodings! > > Just as happy to be proven wrong. Perhaps I misinterpreted my original > profiling and then, embarrassingly, ran with the result for a long time > without retesting. AFAIK the most time is spent in system calls like stat or open. Archiving the stdlib into the ZIP file and using zipimport can decrease Python startup time (perhaps there is an open issue about this). _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Sat Jan 30 15:29:20 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Sat, 30 Jan 2016 21:29:20 +0100 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> <56ACE564.7080107@python.org> Message-ID: <56AD1D20.5090205@mail.de> On 30.01.2016 19:20, Serhiy Storchaka wrote: > AFAIK the most time is spent in system calls like stat or open. > Archiving the stdlib into the ZIP file and using zipimport can > decrease Python startup time (perhaps there is an open issue about this). Oh, please don't. One thing I love about Python is the ease of access. I personally think that startup time is not really a big issue; even when it comes to microbenchmarks. Best, Sven From brett at python.org Sat Jan 30 15:32:55 2016 From: brett at python.org (Brett Cannon) Date: Sat, 30 Jan 2016 20:32:55 +0000 Subject: [Python-Dev] More optimisation ideas In-Reply-To: <56AD1D20.5090205@mail.de> References: <56AB9BCE.2080000@python.org> <56ACE564.7080107@python.org> <56AD1D20.5090205@mail.de> Message-ID: On Sat, Jan 30, 2016, 12:30 Sven R. Kunze wrote: > On 30.01.2016 19:20, Serhiy Storchaka wrote: > > AFAIK the most time is spent in system calls like stat or open. > > Archiving the stdlib into the ZIP file and using zipimport can > > decrease Python startup time (perhaps there is an open issue about this). > > Oh, please don't. One thing I love about Python is the ease of access. > It wouldn't be a requirement, just a nootion > I personally think that startup time is not really a big issue; even > when it comes to microbenchmarks. > You might not, but just about every command-line app does. -brett > Best, > Sven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Sat Jan 30 16:50:13 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Sat, 30 Jan 2016 22:50:13 +0100 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> <56ACE564.7080107@python.org> <56AD1D20.5090205@mail.de> Message-ID: <56AD3015.6010003@mail.de> On 30.01.2016 21:32, Brett Cannon wrote: > On Sat, Jan 30, 2016, 12:30 Sven R. Kunze > wrote: > > On 30.01.2016 19:20, Serhiy Storchaka wrote: > > AFAIK the most time is spent in system calls like stat or open. > > Archiving the stdlib into the ZIP file and using zipimport can > > decrease Python startup time (perhaps there is an open issue > about this). > > Oh, please don't. One thing I love about Python is the ease of access. > > > It wouldn't be a requirement, just a nootion > That's good. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Jan 31 10:11:06 2016 From: larry at hastings.org (Larry Hastings) Date: Sun, 31 Jan 2016 07:11:06 -0800 Subject: [Python-Dev] Fun with ancient unsupported platforms In-Reply-To: <56AA2C50.7050700@hastings.org> References: <56AA2C50.7050700@hastings.org> Message-ID: <56AE240A.1040807@hastings.org> On 01/28/2016 06:57 AM, Larry Hastings wrote: > It's currently 2016. Perhaps it's time to remove all vestiges of > these unsupported operating systems nobody's cared about since a year > that started with a '1'? We dropped support for Irix in 2.3. We dropped support for Irix threads in 3.2. All our supported platforms have Thread Local Storage (TLS) support. Maybe we can drop our 250-line portable TLS library from Python/thread.c? //arry/ p.s. Derpy code in Python/thread_nt.h. It literally looks like this: /* use native Windows TLS functions */ #define Py_HAVE_NATIVE_TLS #ifdef Py_HAVE_NATIVE_TLS It seems this developer had the short-term memory of a goldfish. -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Sun Jan 31 11:56:21 2016 From: antoine at python.org (Antoine Pitrou) Date: Sun, 31 Jan 2016 16:56:21 +0000 (UTC) Subject: [Python-Dev] More optimisation ideas References: <56AB9BCE.2080000@python.org> Message-ID: Hi, If you want to make startup time faster for a broad range of applications, please consider adding a lazy import facility in the stdlib. I recently tried to write a lazy import mechanism using import hooks (to make it portable from 2.6 to 3.5), it seems nearly impossible to do so (or, at least, for an average Python programmer like me). This would be much more useful (for actual users, not for architecture astronauts) than refactoring the importlib APIs in each feature version... Thanks in advance Antoine. From brett at python.org Sun Jan 31 12:02:30 2016 From: brett at python.org (Brett Cannon) Date: Sun, 31 Jan 2016 17:02:30 +0000 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> Message-ID: A lazy importer was added in Python 3.5 and it was not possible without the module spec refactoring. On Sun, 31 Jan 2016, 08:57 Antoine Pitrou wrote: > > Hi, > > If you want to make startup time faster for a broad range of applications, > please consider adding a lazy import facility in the stdlib. > I recently tried to write a lazy import mechanism using import hooks > (to make it portable from 2.6 to 3.5), it seems nearly impossible to do > so (or, at least, for an average Python programmer like me). > > This would be much more useful (for actual users, not for architecture > astronauts) than refactoring the importlib APIs in each feature version... > > Thanks in advance > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Sun Jan 31 12:09:49 2016 From: antoine at python.org (Antoine Pitrou) Date: Sun, 31 Jan 2016 17:09:49 +0000 (UTC) Subject: [Python-Dev] More optimisation ideas References: <56AB9BCE.2080000@python.org> Message-ID: Brett Cannon python.org> writes: > > > A lazy importer was added in Python 3.5 and it was not possible > without the module spec refactoring. Wow... Thank you, I didn't know about that. Now for the next question: how am I supposed to use it? The following documentation leaves me absolutely clueless: """This class only works with loaders that define exec_module() as control over what module type is used for the module is required. For those same reasons, the loader?s create_module() method will be ignored (i.e., the loader?s method should only return None). Finally, modules which substitute the object placed into sys.modules will not work as there is no way to properly replace the module references throughout the interpreter safely; ValueError is raised if such a substitution is detected.""" (reference: https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader) I want to import lazily the modules from package "foobar.*", but not other modules as other libraries may depend on import side effects. How do I do that? The quoted snippet doesn't really help. Regards Antoine. From donald at stufft.io Sun Jan 31 12:11:23 2016 From: donald at stufft.io (Donald Stufft) Date: Sun, 31 Jan 2016 12:11:23 -0500 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> Message-ID: > On Jan 31, 2016, at 12:02 PM, Brett Cannon wrote: > > A lazy importer was added in Python 3.5 Is there any docs on how to actually use the LazyLoader in 3.5? I can?t seem to find any but I don?t really know the import system that well. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Sun Jan 31 12:26:19 2016 From: brett at python.org (Brett Cannon) Date: Sun, 31 Jan 2016 17:26:19 +0000 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> Message-ID: There are no example docs for it yet, but enough people have asked this week about how to set up a custom importer that I will write up a generic example case which will make sense for a lazy loader (need to file the issue before I forget). On Sun, 31 Jan 2016, 09:11 Donald Stufft wrote: > > On Jan 31, 2016, at 12:02 PM, Brett Cannon wrote: > > A lazy importer was added in Python 3.5 > > > Is there any docs on how to actually use the LazyLoader in 3.5? I can?t > seem to find any but I don?t really know the import system that well. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jan 31 12:57:12 2016 From: brett at python.org (Brett Cannon) Date: Sun, 31 Jan 2016 17:57:12 +0000 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> Message-ID: I have opened http://bugs.python.org/issue26252 to track writing the example (and before ppl go playing with the lazy loader, be aware of http://bugs.python.org/issue26186). On Sun, 31 Jan 2016 at 09:26 Brett Cannon wrote: > There are no example docs for it yet, but enough people have asked this > week about how to set up a custom importer that I will write up a generic > example case which will make sense for a lazy loader (need to file the > issue before I forget). > > On Sun, 31 Jan 2016, 09:11 Donald Stufft wrote: > >> >> On Jan 31, 2016, at 12:02 PM, Brett Cannon wrote: >> >> A lazy importer was added in Python 3.5 >> >> >> Is there any docs on how to actually use the LazyLoader in 3.5? I can?t >> seem to find any but I don?t really know the import system that well. >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Sun Jan 31 13:43:20 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 31 Jan 2016 19:43:20 +0100 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> <56ACE564.7080107@python.org> Message-ID: <56AE55C8.8000807@egenix.com> On 30.01.2016 20:15, Steve Dower wrote: > Brett tried freezing the entire stdlib at one point (as we do for parts of importlib) and reported no significant improvement. Since that rules out code compilation as well as the OS calls, it'd seem the priority is to execute less code on startup. > > Details of that work were posted to python-dev about twelve months ago, IIRC. Maybe a little longer. Freezing the entire stdlib does improve the startup time, simply because it removes stat calls, which dominate the startup time at least on Unix. It also allows sharing the stdlib byte code in memory, since it gets stored in static C structs which the OS will happily mmap into multiple processes for you without any additional effort. Our eGenix PyRun does exactly that. Even though the original motivation is a different one, the gained improvement in startup time is a nice side effect: http://www.egenix.com/products/python/PyRun/ Aside: The encodings don't really make much difference here. The dictionaries aren't all that big, so generating them on the fly doesn't really create much overhead. The trade off in terms of maintainability/speed definitely leans toward maintainability. For the larger encoding tables we already have C implementations with appropriate data structures to make lookup speed vs. storage needs efficient. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Jan 31 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ > Top-posted from my Windows Phone > > -----Original Message----- > From: "Serhiy Storchaka" > Sent: ?1/?30/?2016 10:22 > To: "python-dev at python.org" > Subject: Re: [Python-Dev] More optimisation ideas > > On 30.01.16 18:31, Steve Dower wrote: >> On 30Jan2016 0645, Serhiy Storchaka wrote: >>> $ ./python -m timeit -s "import codecs; from encodings.cp437 import >>> decoding_table" -- "codecs.charmap_build(decoding_table)" >>> 100000 loops, best of 3: 4.36 usec per loop >>> >>> Getting rid from charmap_build() would save you at most 4.4 microseconds >>> per encoding. 0.0005 seconds if you have imported *all* standard >>> encodings! >> >> Just as happy to be proven wrong. Perhaps I misinterpreted my original >> profiling and then, embarrassingly, ran with the result for a long time >> without retesting. > > AFAIK the most time is spent in system calls like stat or open. > Archiving the stdlib into the ZIP file and using zipimport can decrease > Python startup time (perhaps there is an open issue about this). > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com > From brett at python.org Sun Jan 31 15:23:00 2016 From: brett at python.org (Brett Cannon) Date: Sun, 31 Jan 2016 20:23:00 +0000 Subject: [Python-Dev] More optimisation ideas In-Reply-To: <56AE55C8.8000807@egenix.com> References: <56AB9BCE.2080000@python.org> <56ACE564.7080107@python.org> <56AE55C8.8000807@egenix.com> Message-ID: So freezing the stdlib helps on UNIX and not on OS X (if my old testing is still accurate). I guess the next question is what it does on Windows and if we would want to ever consider freezing the stdlib as part of the build process (and if we would want to change the order of importers on sys.meta_path so frozen modules came after file-based ones). On Sun, 31 Jan 2016, 10:43 M.-A. Lemburg wrote: > On 30.01.2016 20:15, Steve Dower wrote: > > Brett tried freezing the entire stdlib at one point (as we do for parts > of importlib) and reported no significant improvement. Since that rules out > code compilation as well as the OS calls, it'd seem the priority is to > execute less code on startup. > > > > Details of that work were posted to python-dev about twelve months ago, > IIRC. Maybe a little longer. > > Freezing the entire stdlib does improve the startup time, > simply because it removes stat calls, which dominate the startup > time at least on Unix. > > It also allows sharing the stdlib byte code in memory, since it gets > stored in static C structs which the OS will happily mmap into > multiple processes for you without any additional effort. > > Our eGenix PyRun does exactly that. Even though the original > motivation is a different one, the gained improvement in > startup time is a nice side effect: > > http://www.egenix.com/products/python/PyRun/ > > Aside: The encodings don't really make much difference here. The > dictionaries aren't all that big, so generating them on the fly doesn't > really create much overhead. The trade off in terms of > maintainability/speed > definitely leans toward maintainability. For the larger encoding > tables we already have C implementations with appropriate data > structures to make lookup speed vs. storage needs efficient. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Jan 31 2016) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > > > Top-posted from my Windows Phone > > > > -----Original Message----- > > From: "Serhiy Storchaka" > > Sent: ?1/?30/?2016 10:22 > > To: "python-dev at python.org" > > Subject: Re: [Python-Dev] More optimisation ideas > > > > On 30.01.16 18:31, Steve Dower wrote: > >> On 30Jan2016 0645, Serhiy Storchaka wrote: > >>> $ ./python -m timeit -s "import codecs; from encodings.cp437 import > >>> decoding_table" -- "codecs.charmap_build(decoding_table)" > >>> 100000 loops, best of 3: 4.36 usec per loop > >>> > >>> Getting rid from charmap_build() would save you at most 4.4 > microseconds > >>> per encoding. 0.0005 seconds if you have imported *all* standard > >>> encodings! > >> > >> Just as happy to be proven wrong. Perhaps I misinterpreted my original > >> profiling and then, embarrassingly, ran with the result for a long time > >> without retesting. > > > > AFAIK the most time is spent in system calls like stat or open. > > Archiving the stdlib into the ZIP file and using zipimport can decrease > > Python startup time (perhaps there is an open issue about this). > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mal%40egenix.com > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aarmour at cipmail.org Sun Jan 31 15:16:27 2016 From: aarmour at cipmail.org (ty armour) Date: Sun, 31 Jan 2016 15:16:27 -0500 Subject: [Python-Dev] just a friendly developer Message-ID: I am looking for tutorials on basically how to write the python language. As well I want to know how to write wrappers for things like alsa and directx and coreaudio and jackd and pulseaudio. I would be looking to write wrappers for coreaudio in mac and coreaudio for windows as well as alsa. i think it would benefit you to have this information. I am looking to aid in development of computers and languages and tutorials on how to write the actual python language would be super useful. but yeah im looking to write applications like guitarix rakarrack blender ardour lmms. I am also going to build my own computers so these tutorials would help a lot. anyhow thanks for your time -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Jan 31 17:43:22 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 31 Jan 2016 14:43:22 -0800 Subject: [Python-Dev] just a friendly developer In-Reply-To: References: Message-ID: <56AE8E0A.2010906@stoneleaf.us> On 01/31/2016 12:16 PM, ty armour wrote: > I am looking for tutorials on basically how to write the python > language. Try asking on the python-list [1] mailing list, as that is for general discussion of Python. This list is for developing Python itself. -- ~Ethan~ [1] https://mail.python.org/mailman/listinfo/python-list From tjreedy at udel.edu Sun Jan 31 18:35:44 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 31 Jan 2016 18:35:44 -0500 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> Message-ID: On 1/31/2016 12:09 PM, Antoine Pitrou wrote: > The following documentation leaves me absolutely clueless: > > """This class only works with loaders that define exec_module() as control > over what module type is used for the module is required. No wonder. I cannot parse it as an English sentence. It needs rewriting. > For those same > reasons, the loader?s create_module() method will be ignored (i.e., the > loader?s method should only return None). Finally, modules which substitute > the object placed into sys.modules will not work as there is no way to > properly replace the module references throughout the interpreter safely; > ValueError is raised if such a substitution is detected.""" > > (reference: > https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader) -- Terry Jan Reedy From brett at python.org Sun Jan 31 20:09:27 2016 From: brett at python.org (Brett Cannon) Date: Mon, 01 Feb 2016 01:09:27 +0000 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> Message-ID: On Sun, 31 Jan 2016, 15:36 Terry Reedy wrote: > On 1/31/2016 12:09 PM, Antoine Pitrou wrote: > > > The following documentation leaves me absolutely clueless: > > > > """This class only works with loaders that define exec_module() as > control > > over what module type is used for the module is required. > > No wonder. I cannot parse it as an English sentence. It needs rewriting. > Feel free to open an issue to clarify the wording. -Brett > > For those same > > reasons, the loader?s create_module() method will be ignored (i.e., the > > loader?s method should only return None). Finally, modules which > substitute > > the object placed into sys.modules will not work as there is no way to > > properly replace the module references throughout the interpreter safely; > > ValueError is raised if such a substitution is detected.""" > > > > (reference: > > > https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader > ) > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sun Jan 31 21:27:35 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 01 Feb 2016 13:27:35 +1100 Subject: [Python-Dev] just a friendly developer References: Message-ID: <857fipdv1k.fsf@benfinney.id.au> ty armour writes: > I am looking for tutorials on basically how to write the python > language. Newcomers to Python are especially invited to the Python ?tutor? forum , for collaborative tutoring in a friendly environment. -- \ ?The internet's completely over.? Anyway, all these computers | `\ and digital gadgets are no good. They just fill your head with | _o__) numbers and that can't be good for you.? ?Prince, 2010-07-05 | Ben Finney From steve at pearwood.info Sun Jan 31 22:12:27 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 1 Feb 2016 14:12:27 +1100 Subject: [Python-Dev] More optimisation ideas In-Reply-To: References: <56AB9BCE.2080000@python.org> <56ACE564.7080107@python.org> <56AE55C8.8000807@egenix.com> Message-ID: <20160201031226.GF31806@ando.pearwood.info> On Sun, Jan 31, 2016 at 08:23:00PM +0000, Brett Cannon wrote: > So freezing the stdlib helps on UNIX and not on OS X (if my old testing is > still accurate). I guess the next question is what it does on Windows and > if we would want to ever consider freezing the stdlib as part of the build > process (and if we would want to change the order of importers on > sys.meta_path so frozen modules came after file-based ones). I find that being able to easily open stdlib .py files in a text editor to read the source is extremely valuable. I've learned much more from reading the source than from (e.g.) StackOverflow. Likewise, it's often handy to do a grep over the stdlib. When you talk about freezing the stdlib, what exactly does that mean? - will the source files still be there? - how will this affect people writing patches for bugs? -- Steve
RetroSearch is an open source project built by @garambo
| Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4