. Wouldn't that be expressed cleaner using if in : ? - Similar, this smells a bit; can you explain or improve? try: duck = sig.var_annotations[sig.var_kw_args] except (KeyError, AttributeError): That's all I have for now; on to reject^Wreview some more PEPs... -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Tue Mar 13 02:12:34 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Mar 2012 11:12:34 +1000 Subject: [Python-Dev] Review of PEP 362 (signature object) In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 10:51 AM, Guido van Rossum wrote: > - I don't quite understand how bind() is supposed to work. Maybe an > example would help? (It could also use some motivation. I think this > is meant to expose a canonical version of the algorithm that maps > arguments to parameters. What's a use case?) I can help with that part: one use case is to give early errors for bad arguments to delayed calls. Currently, if you have a delayed call (e.g. a callback of some kind) that accepts (callable, *args, **kwds), there's no parameter checking until the call actually happens. That can lead to quite a debugging hunt as you try to track down where the bad callback was originally registered. However, with bind() available, you can do an initial sanity check that the arguments can at least be used to invoke the callable, throwing an error on *registration* if the callback is simply never going to work. Another use case is more sophisticated protocol checking than a simple hasattr() check for a method name - you can check that the method will accept the arguments you want to pass, not just whether it exists or not. (For example, this can help generate better error messages if a protocol evolves to accept additional optional arguments, but supporting those arguments is *required* for a particular application) Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From brett at python.org Tue Mar 13 02:51:48 2012 From: brett at python.org (Brett Cannon) Date: Mon, 12 Mar 2012 21:51:48 -0400 Subject: [Python-Dev] Review of PEP 362 (signature object) In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 20:51, Guido van Rossum wrote: > I'm very sympathetic to this PEP. I would accept it outright except I > have a few quibbles on details, and some questions and remarks. > > - There are several examples of poor English grammar, perhaps from > your co-author. Can you fix these? (Do you need me to produce a list?) > Nope. I will do a re-read since it's an old PEP. > > - You're using an informal notation to indicate compound types, e.g. > dict(str, object). I'm not sure it's worth using this particular > notation without defining it (although maybe the time is ripe for > creating a PEP that proposes a standard use for parameter > annotations...). You're also not using it very consistently > > - sig.name is currently the unqualified function name. Now we have > __qualname__ maybe this should be the qualified name instead? > Probably. > > - Did you think about whether var_args and var_kw_args should be '' or > None or undefined if these aren't present in the actual definition? > It's an open issue in the PEP. Perhaps they can be set to their default values of () and {}, respectively? > > - If there is no return annotation, is return_annotation None or > undefined? (TBH I think undefined is awkward because you'd have to use > hasattr() to test for its presence. I'd be okay with equating None > with no return annotation. For parameter annotations I'm less sure, > it's not so bad to test for presence in the dict, and you can easily > use .get().) > I think it should be None since that is what the return value is > > - I don't quite understand how bind() is supposed to work. Maybe an > example would help? (It could also use some motivation. I think this > is meant to expose a canonical version of the algorithm that maps > arguments to parameters. What's a use case?) > Nick addressed this. > > - Why is bind() listed under attributes, while there's also a list of > methods? Is it something funky about self? > Probably just an oversight. > > - The PEP still seems to support tuple unpacking, which is no longer > supported in Python 3. Please take it out. > Sure thing. > > - I see it was my idea to give kw-only parameter a valid but > meaningless position. I think I want to revert my opinion; it would be > odd if there's a (kw-only) *parameter* 5 that cannot correspond to > *argument* 5. So let's set it to None if it's kw-only. Maybe > sig.parameters should not have these guys? Or it should have a > separate sig.kw_only_parameters which is a dict?????? > Yeah, trying to handle these odd cases is one of the reasons I have not pushed hard for this PEP before. =) > > - There is mention of has_annotation but no definition. Is this due to > a half revision of the PEP? I sort of see the point but maybe it's > more pragmatic to set it to None for an absent annotation? (Later: > maybe I see, there's a similar pattern for defaults, and it does make > sense to distinguish between "x" and "x = y".) > I will clarify in the PEP. > > - And why are there two ways of getting the annotations, one via > sig.var_annotations[v] and once via sig[v].annotation ? > Probably just made sense for some reason all those years ago. > > - Actually I now see there are also, kind of, two ways to access the > Parameter objects: sig[v] and sig.parameters[i]. But maybe that's more > defensible since we want to be able to access them by position or by > name. > > - You have some examples like "var_args(*[1,2,3])" -- I think that > should just be "var_args(1, 2, 3)" right? Similar "var_kw_args(**{'a': > 1})" should be "var_kw_args(a=1)"... > > Quite possibly. > - You have some example code using try: [ ] // > except KeyError: pass // else: . Wouldn't that be expressed > cleaner using if in : ? > Yes. =) > > - Similar, this smells a bit; can you explain or improve? > try: > duck = sig.var_annotations[sig.var_kw_args] > except (KeyError, AttributeError): > > Sure. -Brett > That's all I have for now; on to reject^Wreview some more PEPs... > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shazow at gmail.com Tue Mar 13 03:23:11 2012 From: shazow at gmail.com (Andrey Petrov) Date: Mon, 12 Mar 2012 19:23:11 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives Message-ID: Hi Pythonistas, I've had the pleasure of speaking with Guido at PyCon and it became evident that some of Python's included batteries are significantly lagging behind the rapidly-evolving defacto standards of the community?specifically in cases like urllib and urllib2, which lack important features provided by alternatives like httplib2, requests, and my own urllib3. Part 1: I propose we add a snippet to the top of the documentation of specific archaic standard modules which encourages users to investigate third-party alternatives if they find the standard module frustrating or otherwise lacking. These notes would target new users, including those coming from other languages where the third-party library choices are not nearly as amazing as they are in Python. (For what it's worth, Guido has verbally agreed to a proposal of this nature.) What such a snippet might look like: "Batteries are included with Python but sometimes they are old and leaky?this is one of those cases. Please have a look in PyPI for more modern alternatives provided by the Python community." Additionally, I would like for us as a community to identify which other standard libraries (cgi? ssl? others?) are candidates for this kind of messaging in the official Python documentation. Part 2: I propose we add a new category of package identifiers such as "Topic :: Standard Library Alternative :: {stdlib_package_name}" which authors of libraries can tag themselves under. The documentation warning snippet will provide a link to the appropriate PyPI query to list packages claiming to be alternatives to the stdlib package in question. Objections? Concerns? Improvements? What is the next step to making this happen? Pythonically yours, - Andrey (on behalf of Ori Livneh, Kenneth Reitz, Brandon Rhodes, David Wolever, and everyone else who contributed to this letter during our PyCon sprint.) P.S. Appendix: Here are some additional snippet alternatives that were proposed: > Batteries are included with Python but sometimes they are old and leaky?this is one of those cases. Please have a look in PyPI for more modern alternatives provided by the Python community. > While this module has served Python programmers faithfully for many years, there are now many powerful alternatives available as third-party modules. To learn more about them, view the Python Package Index results for the category "Topic :: Standard Library Alternative :: asyncore." ? [With the topic name as a hyperlink] > This module has been identified by the community as crusty, a signal that better alternatives exist outside the standard library. Because the Python standard library is constained to maintain backward-compatibility, it does not always reflect what is current or common in the Python community. If you are not constrained to support legacy code, you may wish to browse the Python Package Index for alternatives to this module. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Mar 13 04:07:03 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 12 Mar 2012 23:07:03 -0400 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: <20120313030704.46C512500ED@webabinitio.net> I don't like any of the suggested wordings. I have no problem with us recommending other modules, but most of the Python libraries are perfectly functional (not "leaky" or some other pejorative), they just aren't as capable as the wiz-bang new stuff that's available on PyPI. --David From eliben at gmail.com Tue Mar 13 04:22:45 2012 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 13 Mar 2012 05:22:45 +0200 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120313030704.46C512500ED@webabinitio.net> References: <20120313030704.46C512500ED@webabinitio.net> Message-ID: On Tue, Mar 13, 2012 at 05:07, R. David Murray wrote: > I don't like any of the suggested wordings. ?I have no problem with > us recommending other modules, but most of the Python libraries are > perfectly functional (not "leaky" or some other pejorative), they just > aren't as capable as the wiz-bang new stuff that's available on PyPI. > +1 to David's comment, and -0 on the proposal as a whole. The suggested wordings are simply offensive to those modules & their maintainers specifically, and to Python generally. Personally, I think an intelligent user should realize that a language's standard library won't provide all the latest and shiniest gadgets. Rather, it will focus on providing stable tools that have withstood the test of time and can serve as a basis for building more advanced tools. That intelligent user should also be aware of PyPI (and the main Python page makes it prominent enough), so I see no reason explicitly pointing to it in the documentation of several modules. Eli From ctb at msu.edu Tue Mar 13 04:25:49 2012 From: ctb at msu.edu (C. Titus Brown) Date: Mon, 12 Mar 2012 20:25:49 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: <20120313030704.46C512500ED@webabinitio.net> Message-ID: <20120313032548.GA29711@idyll.org> On Tue, Mar 13, 2012 at 05:22:45AM +0200, Eli Bendersky wrote: > On Tue, Mar 13, 2012 at 05:07, R. David Murray wrote: > > I don't like any of the suggested wordings. ?I have no problem with > > us recommending other modules, but most of the Python libraries are > > perfectly functional (not "leaky" or some other pejorative), they just > > aren't as capable as the wiz-bang new stuff that's available on PyPI. > > > > +1 to David's comment, and -0 on the proposal as a whole. > > The suggested wordings are simply offensive to those modules & their > maintainers specifically, and to Python generally. > > Personally, I think an intelligent user should realize that a > language's standard library won't provide all the latest and shiniest > gadgets. Rather, it will focus on providing stable tools that have > withstood the test of time and can serve as a basis for building more > advanced tools. That intelligent user should also be aware of PyPI > (and the main Python page makes it prominent enough), so I see no > reason explicitly pointing to it in the documentation of several > modules. I see the point, but as a reasonably knowledgeable Python programmer (intelligent? who knows...) I regularly discover nifty new modules that "replace" stdlib modules. It'd be nice to have pointers in the docs, although that runs the risk of having the pointers grow stale, too. --titus -- C. Titus Brown, ctb at msu.edu From eliben at gmail.com Tue Mar 13 04:42:55 2012 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 13 Mar 2012 05:42:55 +0200 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120313032548.GA29711@idyll.org> References: <20120313030704.46C512500ED@webabinitio.net> <20120313032548.GA29711@idyll.org> Message-ID: On Tue, Mar 13, 2012 at 05:25, C. Titus Brown wrote: > On Tue, Mar 13, 2012 at 05:22:45AM +0200, Eli Bendersky wrote: >> On Tue, Mar 13, 2012 at 05:07, R. David Murray wrote: >> > I don't like any of the suggested wordings. ?I have no problem with >> > us recommending other modules, but most of the Python libraries are >> > perfectly functional (not "leaky" or some other pejorative), they just >> > aren't as capable as the wiz-bang new stuff that's available on PyPI. >> > >> >> ?+1 to David's comment, and -0 on the proposal as a whole. >> >> The suggested wordings are simply offensive to those modules & their >> maintainers specifically, and to Python generally. >> >> Personally, I think an intelligent user should realize that a >> language's standard library won't provide all the latest and shiniest >> gadgets. Rather, it will focus on providing stable tools that have >> withstood the test of time and can serve as a basis for building more >> advanced tools. That intelligent user should also be aware of PyPI >> (and the main Python page makes it prominent enough), so I see no >> reason explicitly pointing to it in the documentation of several >> modules. > > I see the point, but as a reasonably knowledgeable Python programmer > (intelligent? who knows...) I regularly discover nifty new modules > that "replace" stdlib modules. ?It'd be nice to have pointers in the > docs, although that runs the risk of having the pointers grow stale, > too. > Exactly. It's not the job of the core developers to keep track of the latest and greatest gadgets and to diligently update the docs when something new comes out. Note that "the latest and coolest" changes frequently, so this may mean different "recommendations" between 3.x.y and 3.x.y+1, which is even more confusing. Wasn't a PyPI recommendation / voting system discussed a while ago? *That* would be much more appropriate than officially endorsing specific modules by pointing to them in the standard documentation. Eli From ctb at msu.edu Tue Mar 13 04:48:21 2012 From: ctb at msu.edu (C. Titus Brown) Date: Mon, 12 Mar 2012 20:48:21 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: <20120313030704.46C512500ED@webabinitio.net> <20120313032548.GA29711@idyll.org> Message-ID: <20120313034819.GC29711@idyll.org> On Tue, Mar 13, 2012 at 05:42:55AM +0200, Eli Bendersky wrote: > On Tue, Mar 13, 2012 at 05:25, C. Titus Brown wrote: > > I see the point, but as a reasonably knowledgeable Python programmer > > (intelligent? who knows...) I regularly discover nifty new modules > > that "replace" stdlib modules. ?It'd be nice to have pointers in the > > docs, although that runs the risk of having the pointers grow stale, > > too. > > > > Exactly. It's not the job of the core developers to keep track of the > latest and greatest gadgets and to diligently update the docs when > something new comes out. Note that "the latest and coolest" changes > frequently, so this may mean different "recommendations" between 3.x.y > and 3.x.y+1, which is even more confusing. > > Wasn't a PyPI recommendation / voting system discussed a while ago? > *That* would be much more appropriate than officially endorsing > specific modules by pointing to them in the standard documentation. I feel like there's a middle ground where stable, long-term go-to modules could be mentioned, though. I don't spend a lot of time browsing PyPI, but I suspect almost everyone spends a certain amount of time in the Python docs (which is a testimony to their quality IMO). So I'm in favor of conservative link-outs but without any deprecating language. --titus -- C. Titus Brown, ctb at msu.edu From brian at python.org Tue Mar 13 04:58:12 2012 From: brian at python.org (Brian Curtin) Date: Mon, 12 Mar 2012 20:58:12 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 19:23, Andrey Petrov wrote: > What such a snippet might look like: > > "Batteries are included with Python but sometimes they are old and > leaky?this is one of those cases. Please have a look in PyPI for more modern > alternatives provided by the Python community." What does "leaky" mean here? Someone's going to see that, think it has memory leaks, then rant on the internet about how we ship crap and just document it as so. > Part 2: > I propose we add a new category of package identifiers such as "Topic :: > Standard Library Alternative :: {stdlib_package_name}" which authors of > libraries can tag themselves under. The documentation warning snippet will > provide a link to the appropriate PyPI query to list packages claiming to be > alternatives to the stdlib package in question. Automating it to something on PyPI is the not the right answer. People will use it incorrectly, either in that they'll add it to packages for which it isn't accurate, and people just flat out won't use it or know about it. It won't be accurate this way, and anything that we're documenting needs to be vetted. It's not often that a great alternative comes up, so I don't see the manual burden being too great. From v+python at g.nevcal.com Tue Mar 13 05:01:31 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 12 Mar 2012 21:01:31 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120313034819.GC29711@idyll.org> References: <20120313030704.46C512500ED@webabinitio.net> <20120313032548.GA29711@idyll.org> <20120313034819.GC29711@idyll.org> Message-ID: <4F5EC69B.1090000@g.nevcal.com> On 3/12/2012 8:48 PM, C. Titus Brown wrote: > I feel like there's a middle ground where stable, long-term go-to modules could > be mentioned, though. I don't spend a lot of time browsing PyPI, but I suspect > almost everyone spends a certain amount of time in the Python docs (which is a > testimony to their quality IMO). So I'm in favor of conservative link-outs > but without any deprecating language. Any outward links will be somewhat intrusive, and their existence will admit that the stdlib module is limited in some fashion, such that someone invested time and effort to create an alternative. On the other hand, if there were a standard place for external links to alternatives, say, perhaps, at the bottom of the left-hand table of contents for the module, and if it were Wiki-like (anyone could add an alternative) then the core developers wouldn't need to monitor and approve the alternatives. The alternatives would not be listed in the TOC, only the link, if alternatives were submitted. At the link target Wiki, there could be various alternatives, pro/con comments, user votes, whatever seems useful. If there is truly a desire by core developers to recommend specific alternative modules, then wording like the following seems neutral to me: Alternatives to this module exist at [list of links], which may be updated more regularly than the stdlib. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shazow at gmail.com Tue Mar 13 05:14:56 2012 From: shazow at gmail.com (Andrey Petrov) Date: Mon, 12 Mar 2012 21:14:56 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 8:58 PM, Brian Curtin wrote: > On Mon, Mar 12, 2012 at 19:23, Andrey Petrov wrote: >> What such a snippet might look like: >> >> "Batteries are included with Python but sometimes they are old and >> leaky?this is one of those cases. Please have a look in PyPI for more modern >> alternatives provided by the Python community." > > What does "leaky" mean here? Someone's going to see that, think it has > memory leaks, then rant on the internet about how we ship crap and > just document it as so. I agree Brian and David, the choice of "leaky" in the wording is poor. It was supposed to be maintaining the "batteries" metaphor but it's clearly ambiguous. Perhaps something along the lines of... "Batteries are included with Python but for stability and backwards compatibility, some of the standard library is not always as modern as alternatives provided by the Python community?this is one of those cases. Please have a look at PyPI for more cutting-edge alternatives." >> Part 2: >> I propose we add a new category of package identifiers such as "Topic :: >> Standard Library Alternative :: {stdlib_package_name}" which authors of >> libraries can tag themselves under. The documentation warning snippet will >> provide a link to the appropriate PyPI query to list packages claiming to be >> alternatives to the stdlib package in question. > > Automating it to something on PyPI is the not the right answer. People > will use it incorrectly, either in that they'll add it to packages for > which it isn't accurate, and people just flat out won't use it or know > about it. It won't be accurate this way, and anything that we're > documenting needs to be vetted. > > It's not often that a great alternative comes up, so I don't see the > manual burden being too great. There are a dozen or more urllib/httplib/pycurl competitors on PyPI, and new ones spring up all the time. I'm not sure how we would go about objectively blessing the best "official" option at any given moment, or how frequently we would have to do this. With self-identifying, we could sort by some sort metric (monthly downloads? magical score?) and create a somewhat-actionable list. - Andrey From tjreedy at udel.edu Tue Mar 13 05:22:27 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 13 Mar 2012 00:22:27 -0400 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On 3/12/2012 10:23 PM, Andrey Petrov wrote: > I've had the pleasure of speaking with Guido at PyCon and it became > evident that some of Python's included batteries are significantly > lagging behind the rapidly-evolving defacto standards of the > community?specifically in cases like urllib and urllib2, which lack > important features provided by alternatives like httplib2, requests, and > my own urllib3. > > > Part 1: > I propose we add a snippet to the top of the documentation of specific > archaic standard modules which encourages users to investigate > third-party alternatives if they find the standard module frustrating or > otherwise lacking. These notes would target new users, including those > coming from other languages where the third-party library choices are > not nearly as amazing as they are in Python. I would rather we figure out how to encourage authors of advancing packages to contribute better implementations of existing features and well-tested new features back to the stdlib module. For instance, are you the same 'Andrey Petrov' who is'darkprokoba' on the tracker? As near as I can tell, that user has posted on one issue about free threading and nothing else, in particular, nothing about web protocols. If that is you, why not? > What such a snippet might look like: > "Batteries are included with Python but sometimes they are old and > leaky?this is one of those cases. Please have a look in PyPI for more > modern alternatives provided by the Python community." You have every right to work independently. develop alternative modules, and promote them. But suggesting that we denigrate our work to promote yours strikes me as inappropriate. If nothing else, it would discourage rather than encourage more contributions from more people. > Additionally, I would like for us as a community to identify which > other standard libraries (cgi? ssl? others?) are candidates for this > kind of messaging in the official Python documentation. To the degree feasible, stdlib modules should be the best possible in the area they cover. Then all who install Python would benefit. Do you disagree? I would like more of the community to help make that happen. Any messages in the stdlib doc should be about modules that do things intentionally not covered in the stdlib. -- Terry Jan Reedy From brian at python.org Tue Mar 13 05:23:20 2012 From: brian at python.org (Brian Curtin) Date: Mon, 12 Mar 2012 21:23:20 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 21:14, Andrey Petrov wrote: > On Mon, Mar 12, 2012 at 8:58 PM, Brian Curtin wrote: >> On Mon, Mar 12, 2012 at 19:23, Andrey Petrov wrote: >>> What such a snippet might look like: >>> >>> "Batteries are included with Python but sometimes they are old and >>> leaky?this is one of those cases. Please have a look in PyPI for more modern >>> alternatives provided by the Python community." >> >> What does "leaky" mean here? Someone's going to see that, think it has >> memory leaks, then rant on the internet about how we ship crap and >> just document it as so. > > I agree Brian and David, the choice of "leaky" in the wording is poor. > It was supposed to be maintaining the "batteries" metaphor but it's > clearly ambiguous. > > Perhaps something along the lines of... > > "Batteries are included with Python but for stability and backwards > compatibility, some of the standard library is not always as modern as > alternatives provided by the Python community?this is one of those > cases. Please have a look at PyPI for more cutting-edge alternatives." Sorry for another color choice on the bikeshed, but I would drop the word or references to "batteries". *We* know what "batteries included" means, but there are undoubtedly people who won't get it. It's just code - let's call it code. >>> Part 2: >>> I propose we add a new category of package identifiers such as "Topic :: >>> Standard Library Alternative :: {stdlib_package_name}" which authors of >>> libraries can tag themselves under. The documentation warning snippet will >>> provide a link to the appropriate PyPI query to list packages claiming to be >>> alternatives to the stdlib package in question. >> >> Automating it to something on PyPI is the not the right answer. People >> will use it incorrectly, either in that they'll add it to packages for >> which it isn't accurate, and people just flat out won't use it or know >> about it. It won't be accurate this way, and anything that we're >> documenting needs to be vetted. >> >> It's not often that a great alternative comes up, so I don't see the >> manual burden being too great. > > There are a dozen or more urllib/httplib/pycurl competitors on PyPI, > and new ones spring up all the time. I'm not sure how we would go > about objectively blessing the best "official" option at any given > moment, or how frequently we would have to do this. The same way we choose to accept libraries into the standard library. New ones spring up all the time - mature, proven, and widely used ones do not. If someone thinks libfoo is ready, they suggest it. If we haven't heard of it, the conversation ends. If we have people who know it, maybe we have them look deeper and figure out if it's something we can put our stamp on just like we might with the recent talk of "experimental package" inclusion. > With self-identifying, we could sort by some sort metric (monthly > downloads? magical score?) and create a somewhat-actionable list. Downloads don't mean the code is good. Voting is gamed. I really don't think there's a good automated solution to tell us what the high-quality replacement projects are. From guido at python.org Tue Mar 13 05:40:37 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Mar 2012 21:40:37 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 9:22 PM, Terry Reedy wrote: > I would rather we figure out how to encourage authors of advancing packages > to contribute better implementations of existing features and well-tested > new features back to the stdlib module. I would not. There are many excellent packages out there that should not be made into stdlib packages simply because their authors are not done adding new features. If you contribute something to the stdlib and also maintain a non-stdlib version of the same code to which you regularly add features, your code is not ready for inclusion into the stdlib. Not only should you be willing to wait 18 months (until the next feature release) before your features are released, but you should also accept that only the latest version of Python will see those features. This obviously makes it very unattractive for many authors to ever contribute to the stdlib. That's fine. There's a healthy ecosystem of 3rd party modules. For some areas (web stuff especially) there's just no way that the stdlib can keep up. Yes, the stdlib offerings work. But no, they are not very convenient and may not support popular idioms very well. For these types of modules I think it is a good idea to place some sort of pointer in the stdlib docs to an external page (maybe a wiki page) that collects a currently popular set of alternatives, or perhaps a few pointers and wiki pages. We should still be conservative with this, and we should word it to avoid implying that the stdlib code is buggy -- it just isn't as spiffy or featureful as the 3rd party options. (Agreed that the "leaky" wording was unfortunate. I may have inadvertently suggested this, taking the analogy with "batteries included". But I didn't mean it to be literally included into the stdlib.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Mar 13 05:43:49 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Mar 2012 21:43:49 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 9:23 PM, Brian Curtin wrote: > Downloads don't mean the code is good. Voting is gamed. I really don't > think there's a good automated solution to tell us what the > high-quality replacement projects are. Sure, these are imperfect metrics. But not having any metrics at all is flawed too. Despite the huge flamewar we had 1-2 years ago about PyPI comments, I think we should follow the lead of the many app stores that pop up on the web -- users will recognize the pattern and will tune their skepticism sensors as needed. -- --Guido van Rossum (python.org/~guido) From eliben at gmail.com Tue Mar 13 05:48:20 2012 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 13 Mar 2012 06:48:20 +0200 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 06:43, Guido van Rossum wrote: > On Mon, Mar 12, 2012 at 9:23 PM, Brian Curtin wrote: >> Downloads don't mean the code is good. Voting is gamed. I really don't >> think there's a good automated solution to tell us what the >> high-quality replacement projects are. > > Sure, these are imperfect metrics. But not having any metrics at all > is flawed too. Despite the huge flamewar we had 1-2 years ago about > PyPI comments, I think we should follow the lead of the many app > stores that pop up on the web -- users will recognize the pattern and > will tune their skepticism sensors as needed. > An additional bonus of such a system is that we won't have to maintain a separate Wiki page with "popular" choices. Pointing to the PyPI "rating" page, which can presumably be filtered by tags (i.e. web, scientific, XML, etc.) should be sufficient, given that such a rating page exists. Eli From anacrolix at gmail.com Tue Mar 13 06:10:18 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 13 Mar 2012 13:10:18 +0800 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: Definitely think some library vetting needs to occur. Superior alternatives do exist and are difficult to find and choose from. Stuff like LXML, Requests, Tornado are clear winners. The more of this done externally (ie PyPI the better). I still think a set of requirements for "official approval" would be good. This could outline things like requiring that certain stable Python versions are supported, interface stability, demonstrated user base, documentation etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From senthil at uthcode.com Tue Mar 13 06:10:27 2012 From: senthil at uthcode.com (Senthil Kumaran) Date: Mon, 12 Mar 2012 22:10:27 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: <20120313051027.GB2318@mathmagic> On Mon, Mar 12, 2012 at 07:23:11PM -0700, Andrey Petrov wrote: > I've had the pleasure of speaking with Guido at PyCon and it became evident > that some of Python's included batteries are significantly lagging behind the > rapidly-evolving defacto standards of the community specifically in cases like > urllib and urllib2, which lack important features provided by alternatives like > httplib2, requests, and my own urllib3. Well, I think I have address this because I am the maintainer of those modules in standard lib. First things first, it looks to me that trashing something gives good motivation to you (and others working on related modules). I don't have a problem with that. But on the other hand, if you think things can be improved in stdlib, you are welcome to contribute. Just remember that new features, refactoring with backwards compatibility, 'cool api' for new features should go in 3.3+ onwards. Bug fixes, confusions on what's RFC supported vs what's defacto standards, fine line between bugs and features, those can be considered for 2.7. I am personally in favor of constantly improving the standard library modules along with mention of any good libraries which can be useful for the purposes of the user. We already have lots of such references in standard library documentation. If there is a well maintained package, as long as the external package is active and maintained, we can have it as link in the docs. Sometimes those external packages become inactive too, in those cases, those should pruned. Its' all about maintaining libraries and docs and being helpful. -- Senthil From shazow at gmail.com Tue Mar 13 06:48:04 2012 From: shazow at gmail.com (Andrey Petrov) Date: Mon, 12 Mar 2012 22:48:04 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120313051027.GB2318@mathmagic> References: <20120313051027.GB2318@mathmagic> Message-ID: Dear authors of Python's standard library: Please accept my deepest apologies. We didn't mean for the messaging to come off as unappreciative for you work, and I am very sorry for that! Without the aforementioned urllib/httplib/etc I would have never made it as far as to build my own libraries which have suited my needs better than the foundations that originally allowed me to get off the ground. I am very grateful for all of those responsible for building the Python standard library and I appreciate your continued efforts. Some specific replies: @Senthil: I originally asked Guido for guidance on improving the standard library and perhaps including some of my favourite projects, but he pointed out that in a couple of years we might end up again in the same position as before but with one extra library people will complain about for being obsoleted yet remains impossible to deprecate. I agreed with Guido that embracing and even encouraging users to use the rapidly-evolving community-built packages alongside the tried-and-true standard library is the best move here. @Terry: I don't know who 'darkprokoba' is. Unfortunately 'Andrey Petrov' is a very common name. I go under the handle 'shazow' but I haven't participated in core Python discussions until today. I did not suggest for Python to endorse a specific module, even if it is my own. In fact, I generally oppose doing this as I feel that when Django was announced as the blessed 'winner' of the Python MVC frameworks was a harmful event to all other competing frameworks at the time. My suggestion is to inform the users when there are other potentially better suited alternatives available from the community PyPI and to educate them how to find these alternatives. - Andrey From senthil at uthcode.com Tue Mar 13 07:12:39 2012 From: senthil at uthcode.com (Senthil Kumaran) Date: Mon, 12 Mar 2012 23:12:39 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: <20120313051027.GB2318@mathmagic> Message-ID: <20120313061239.GE2318@mathmagic> On Mon, Mar 12, 2012 at 10:48:04PM -0700, Andrey Petrov wrote: > @Senthil: I originally asked Guido for guidance on improving the > standard library and perhaps including some of my favourite projects, > but he pointed out that in a couple of years we might end up again in > the same position as before but with one extra library people will > complain about for being obsoleted yet remains impossible to > deprecate. > > I agreed with Guido that embracing and even encouraging users to use the > rapidly-evolving community-built packages alongside the tried-and-true > standard library is the best move here. I agree with that too. I think, any improvements which can made by external libraries, if they can be considered as can be good improvement in the stdlib, It's a good idea to push them in. I looked at the requests, I think that apis for various HTTP verbs is something interesting, I think web applications authors see some advantage there explicit GET / POST and sending specific options to those. That facility, if it can be brought into urllib.request (3.3 onwards) it could be nice. Also you could have noticed an addition of method= parameter in 3.3 I have not had a chance to look at urllib3. Should do. I have been following httplib2 and use it. Thanks, Senthil From valhallasw at arctus.nl Tue Mar 13 12:44:58 2012 From: valhallasw at arctus.nl (Merlijn van Deen) Date: Tue, 13 Mar 2012 12:44:58 +0100 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) Message-ID: http://bugs.python.org/issue6784?("byte/unicode pickle incompatibilities between python2 and python3") Hello all, Currently, pickle unpickles python2 'str' objects as python3 'str' objects, where the encoding to use is passed to the Unpickler. However, there are cases where it makes more sense to unpickle a python2 'str' as python3 'bytes' - for instance when it is actually binary data, and not text. Currently, the mapping is as follows, when reading a pickle: python2 'str' -> python3 'str' (using an encoding supplied to Unpickler) python2 'unicode' -> python3 'str' or, when creating a pickle using protocol <= 2: python3 'str' -> python2 'unicode' python3 'bytes' -> python2 '__builtins__.bytes object' This issue suggests to add a flag to change the behaviour as follows: a) python2 'str' -> python3 'bytes' b) python3 'bytes' -> python2 'str' The question on this is how to pass this flag. To quote Antoine (with permission) on my mail about this issue on core-mentorship: > I haven't answered because I'm unsure about the approach itself - do we > want to add yet another argument to pickle methods, especially this late > in the 3.x development cycle? Currently, I have implemented it using an extra argument for the Pickler and Unpickler objects ('bytestr'), which toggles the behaviour. I.e.: >>> pickled = Pickler(data, bytestr=True); unpickled = Unpickler(data, bytestr=True). This is the approach used in pickle_bytestr.patch [1] Another option would be to implement a seperate Pickler/Unpickler object, such that >>> pickled = BytestrPickler(data, bytestr=True); unpickled = BytestrUnpickler(data, bytestr=True) This is the approach I initially implemented [2]. Alternatively, there is the option only to implement the Unpickler, leaving the Pickler as it is. This allows >>> unpickled = Unpickler(data, encoding=bytes) where the bytes type is used as a special 'flag'. And, of course, there is the option not to implement this in the stdlib at all. What are your ideas on this? Best, Merlijn [0] http://bugs.python.org/issue6784 [1] http://bugs.python.org/file24719/pickle_bytestr.patch [2] https://github.com/valhallasw/py2/blob/master/bytestrpickle.py From valhallasw at arctus.nl Tue Mar 13 12:45:59 2012 From: valhallasw at arctus.nl (Merlijn van Deen) Date: Tue, 13 Mar 2012 12:45:59 +0100 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: Message-ID: Oops. I should re-read my mails before I send them, not /after/ I send them. On 13 March 2012 12:44, Merlijn van Deen wrote: >>>> pickled = BytestrPickler(data, bytestr=True); unpickled = BytestrUnpickler(data, bytestr=True) should of course read >>>> pickled = BytestrPickler(data); unpickled = BytestrUnpickler(data) Sorry about that. Merlijn From p.f.moore at gmail.com Tue Mar 13 14:31:38 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Mar 2012 13:31:38 +0000 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120313034819.GC29711@idyll.org> References: <20120313030704.46C512500ED@webabinitio.net> <20120313032548.GA29711@idyll.org> <20120313034819.GC29711@idyll.org> Message-ID: On 13 March 2012 03:48, C. Titus Brown wrote: > I feel like there's a middle ground where stable, long-term go-to modules could > be mentioned, though. ?I don't spend a lot of time browsing PyPI, but I suspect > almost everyone spends a certain amount of time in the Python docs (which is a > testimony to their quality IMO). ?So I'm in favor of conservative link-outs > but without any deprecating language. I applaud the idea of promoting the many excellent packages available. It can be very hard to separate the good from the indifferent (or even bad) when browsing PyPI. I've found some very good packages recently which I'd never have known about without some random comment on a mailing list. However, I'm not keen on having the stdlib documentation suggest that I should be using something else. No code should ever be documenting "don't use me, there are better alternatives" unless it is deprecated or obsolete. On the other hand, I would love to see a community-maintained document that described packages that are acknowledged as "best of breed". That applies whether or not those packages replace something in the stdlib. Things like pywin32, lxml, and requests would be examples in my experience. There's no reason this *has* to be in the core documentation - it may be relevant that nothing has sprung up independently yet... Maybe a separate item in the Python documentation, "External Modules", could be created and maintained by the community? By being in the documentation, it has a level of "official recommendation" status, and by being a top-level document it's visible (more so than, for example, a HOWTO document would be). Because it's in the released documentation, it is relatively stable, which implies that external modules would need to have a genuine track record to get in there, but because it's community maintained it should reflect a wider consensus than just the core developers' views. Paul. From donald.stufft at gmail.com Tue Mar 13 14:34:33 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 13 Mar 2012 09:34:33 -0400 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: <20120313030704.46C512500ED@webabinitio.net> <20120313032548.GA29711@idyll.org> <20120313034819.GC29711@idyll.org> Message-ID: <273F4C10E0754BED9962CCD62D7D3882@gmail.com> On Tuesday, March 13, 2012 at 9:31 AM, Paul Moore wrote: > On 13 March 2012 03:48, C. Titus Brown wrote: > > I feel like there's a middle ground where stable, long-term go-to modules could > > be mentioned, though. I don't spend a lot of time browsing PyPI, but I suspect > > almost everyone spends a certain amount of time in the Python docs (which is a > > testimony to their quality IMO). So I'm in favor of conservative link-outs > > but without any deprecating language. > > > > > I applaud the idea of promoting the many excellent packages available. > It can be very hard to separate the good from the indifferent (or even > bad) when browsing PyPI. I've found some very good packages recently > which I'd never have known about without some random comment on a > mailing list. > > However, I'm not keen on having the stdlib documentation suggest that > I should be using something else. No code should ever be documenting > "don't use me, there are better alternatives" unless it is deprecated > or obsolete. > > On the other hand, I would love to see a community-maintained document > that described packages that are acknowledged as "best of breed". That > applies whether or not those packages replace something in the stdlib. > Things like pywin32, lxml, and requests would be examples in my > experience. There's no reason this *has* to be in the core > documentation - it may be relevant that nothing has sprung up > independently yet... > > http://python-guide.org ? > > Maybe a separate item in the Python documentation, "External Modules", > could be created and maintained by the community? By being in the > documentation, it has a level of "official recommendation" status, and > by being a top-level document it's visible (more so than, for example, > a HOWTO document would be). Because it's in the released > documentation, it is relatively stable, which implies that external > modules would need to have a genuine track record to get in there, but > because it's community maintained it should reflect a wider consensus > than just the core developers' views. > > Paul. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org (mailto:Python-Dev at python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Tue Mar 13 15:35:10 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 13 Mar 2012 10:35:10 -0400 Subject: [Python-Dev] Review of PEP 362 (signature object) In-Reply-To: References: Message-ID: <59118671-F1E5-4D32-833E-64C2B6517634@gmail.com> Guido, Brett, I've tried to use the proposed signature object, however, I found that the 'bind' method is incorrect, and came up with my own implementation of the PEP: https://gist.github.com/2029032 (If needed, I can change the licence to PSFL) I used my version to implement typechecking, arguments validation in various RPC dispatch mechanisms, and it is proven to work. First of all, in the current version of the PEP, "bind" doesn't work correctly with "varargs", as it returns a dictionary object: def foo(bar, *args): print(bar, args) s = signature(foo) bound = s.bind(1, 2 ,3 ,4) print('Bound args:', bound) foo(**bound) This code outputs the following: Bound args: {'args': (2, 3, 4), 'bar': 1} Traceback (most recent call last): File "test.py", line 286, in foo(**bound) TypeError: foo() got an unexpected keyword argument 'args' The conclusion is that ** form of unpacking is not always enough, so 'bind' should at least return a pair of (args, kwargs). Secondly, I don't think that even (args, kwargs) pair is enough, as some information about how arguments were mapped is lost. In my implementation, 'bind' method returns an instance of 'BoundArguments' class, which preserves the exact mapping, and has two convenience properties '.args' and '.kwargs', so the example above transforms into: bound = s.bind(1, 2, 3, 4) foo(*bound.args, **bound.kwargs) And that works as it should. When some advanced processing is required, you can work with its private fields: >>> bound._args (1,) >>> bound._varargs (2, 3, 4) I also think that keyword-only arguments deserve to land in a separate collection in the signature object, in my implementation it is 'signature.kwonlyargs' slot. It is easier to do some advanced processing of arguments this way, and even the 'signature.__iter__' is simpler. Finally, I also added 'render_args' method to the signature. It just renders function's arguments as in its definition: >>> signature(foo).render_args() bar, *args This is useful for printing detailed error messages and hints. - Yury From p.f.moore at gmail.com Tue Mar 13 15:52:19 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Mar 2012 14:52:19 +0000 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <273F4C10E0754BED9962CCD62D7D3882@gmail.com> References: <20120313030704.46C512500ED@webabinitio.net> <20120313032548.GA29711@idyll.org> <20120313034819.GC29711@idyll.org> <273F4C10E0754BED9962CCD62D7D3882@gmail.com> Message-ID: On 13 March 2012 13:34, Donald Stufft wrote: > http://python-guide.org ? Hmm, yes maybe. I had seen this before (it's where I found out about requests, IIRC). As it says, it "is mostly a skeleton at the moment". With some fleshing out, then it's probably a good start. I have some problems with its recommendations (notably "If you?re starting work on a new Python module, I recommend you write it for Python 2.5 or 2.6, and add support for Python3 in a later iteration." which is probably not appropriate for something that would be officially sanctioned by the core developers). Also, I don't think we want something advertised as "opinionated". And it covers a much wider area than the original suggestion. I'd envisaged something more like a simple list of "obvious" modules: """ Requests - URL Requests is a module designed to make getting resources from the web as easy as possible. It is a simpler and more powerful alternative to the stdlib urllib and urllib2 modules. ... Some code samples here giving basic usage. """ My ideal would be something I could scan in a few spare moments, and pick up pointers to particular modules that I'd find useful. Basically, take the "Scenario Guide" section of python-guide, flesh it out, and turn it into a flat list. (I don't like the "Scenario" approach as it tends to force people into a particular view of the world, but maybe that's just me, my applications tend to be more eclectic than any of the "normal" categories). Paul. From guido at python.org Tue Mar 13 15:52:17 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2012 07:52:17 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120313051027.GB2318@mathmagic> References: <20120313051027.GB2318@mathmagic> Message-ID: On Mon, Mar 12, 2012 at 10:10 PM, Senthil Kumaran wrote: > On Mon, Mar 12, 2012 at 07:23:11PM -0700, Andrey Petrov wrote: >> I've had the pleasure of speaking with Guido at PyCon and it became evident >> that some of Python's included batteries are significantly lagging behind the >> rapidly-evolving defacto standards of the community specifically in cases like >> urllib and urllib2, which lack important features provided by alternatives like >> httplib2, requests, and my own urllib3. > > Well, I think I have address this because I am the maintainer of those > modules in standard lib. > > First things first, it looks to me that trashing something gives good > motivation to you (and others working on related modules). I don't > have a problem with that. > > But on the other hand, if you think things can be improved in stdlib, > you are welcome to contribute. Just remember that new features, > refactoring with backwards compatibility, 'cool api' for new features > should go in 3.3+ onwards. Bug fixes, confusions on what's RFC > supported vs what's defacto standards, fine line between bugs and > features, those can be considered for 2.7. > > I am personally in favor of constantly improving the standard library > modules along with mention of any good libraries which can be useful > for the purposes of the user. > > We already have lots of such references in standard library > documentation. If there is a well maintained package, as long as the > external package is active and maintained, we can have it as link in > the docs. Sometimes those external packages become inactive too, in > those cases, those should pruned. Its' all about maintaining libraries > and docs and being helpful. Well said. Improving existing stdlib modules is always welcome of course! (And the bar is much lower than for adding new modules.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Mar 13 17:27:51 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2012 09:27:51 -0700 Subject: [Python-Dev] Exceptions in comparison operators In-Reply-To: References: <4F54B496.3030905@hotpy.org> Message-ID: Mark, did you do anything with my reply? On Mon, Mar 5, 2012 at 10:41 AM, Guido van Rossum wrote: > On Mon, Mar 5, 2012 at 4:41 AM, Mark Shannon wrote: >> Comparing two objects (of the same type for simplicity) >> involves a three stage lookup: >> The class has the operator C.__eq__ >> It can be applied to operator (descriptor protocol): C().__eq__ >> and it produces a result: C().__eq__(C()) >> >> Exceptions can be raised in all 3 phases, >> but an exception in the first phase is not really an error, >> its just says the operation is not supported. >> E.g. >> >> class C: pass >> >> C() == C() is False, rather than raising an Exception. >> >> If an exception is raised in the 3rd stage, then it is propogated, >> as follows: >> >> class C: >> ? def __eq__(self, other): >> ? ? ? raise Exception("I'm incomparable") >> >> C() == C() ?raises an exception >> >> However, if an exception is raised in the second phase (descriptor) >> then it is silenced: >> >> def no_eq(self): >> ? ?raise Exception("I'm incomparable") >> >> class C: >> ? __eq__ = property(no_eq) >> >> C() == C() is False. >> >> But should it raise an exception? >> >> The behaviour for arithmetic is different. >> >> def no_add(self): >> ? ?raise Exception("I don't add up") >> >> class C: >> ? __add__ = property(no_add) >> >> C() + C() raises an exception. >> >> So what is the "correct" behaviour? >> It is my opinion that comparisons should behave like arithmetic >> and raise an exception. > > I think you're probably right. This is one of those edge cases that > are so rare (and always considered a bug in the user code) that we > didn't define carefully what should happen. There are probably some > implementation-specific reasons why it was done this way (comparisons > use a very different code path from regular binary operators) but that > doesn't sound like a very good reason. > > OTOH there *is* a difference: as you say, C() == C() is False when the > class doesn't define __eq__, whereas C() + C() raises an exception if > it doesn't define __add__. Still, this is more likely to have favored > the wrong outcome for (2) by accident than by design. > > You'll have to dig through the CPython implementation and find out > exactly what code needs to be changed before I could be sure though -- > sometimes seeing the code jogs my memory. > > But I think of x==y as roughly equivalent to > > r = NotImplemented > if hasattr(x, '__eq__'): > ?r = x.__eq__(y) > if r is NotImplemented and hasattr(y, '__eq__'): > ?r = y.__eq__(x) > if r is NotImplemented: > ?r = False > > which would certainly suggest that (2) should raise an exception. A > possibility is that the code looking for the __eq__ attribute > suppresses *all* exceptions instead of just AttributeError. If you > change no_eq() to return 42, for example, the comparison raises the > much more reasonable TypeError: 'int' object is not callable. > > -- > --Guido van Rossum (python.org/~guido) -- --Guido van Rossum (python.org/~guido) From merwok at netwok.org Tue Mar 13 17:40:30 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 13 Mar 2012 17:40:30 +0100 Subject: [Python-Dev] [RELEASED] Distutils2 1.0a4 In-Reply-To: <4F5F77CB.1040600@netwok.org> References: <4F5F77CB.1040600@netwok.org> Message-ID: <4F5F787E.1090307@netwok.org> What would be a release email without errors? :) The wiki link I gave doesn?t work, it should be http://wiki.python.org/moin/Distutils2/Contributing From merwok at netwok.org Tue Mar 13 17:37:31 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 13 Mar 2012 17:37:31 +0100 Subject: [Python-Dev] [RELEASED] Distutils2 1.0a4 Message-ID: <4F5F77CB.1040600@netwok.org> Hello, On behalf of the distutils2 contributors, I am thrilled to announce the release of Distutils2 1.0a4. Distutils2 is the packaging library that supersedes Distutils. It supports distributing, uploading, downloading, installing and removing projects, and is also a support library for other packaging tools like pip and buildout. It will be provided as part of Python 3.3; this release is a backport compatible with Python 2.5 to 2.7. Distutils2 1.0a4 contains a number of known bugs, limitations and missing features, but we have released it to let users and developers download and try it. This means you! If you want to report new issues, request features or contribute, please read DEVNOTES.txt in the source distribution or http://wiki.python.org/Distutils2/Contributing More alpha releases will be cut when important bugs get fixed during the next few months, like Windows or PyPy compatibility. The first beta is planned for June, and 1.0 final for August (to follow Python 3.3.0). Until beta, the API is subject to drastic changes and code can get removed. Basic documentation is at http://docs.python.org/dev/packaging ; it will get updated, expanded and improved in the coming months. Enjoy! Repository: http://hg.python.org/distutils2 Bug tracker: http://bugs.python.org/ (component "Distutils2") Mailing list: http://mail.python.org/mailman/listinfo/distutils-sig/ From tarek at ziade.org Tue Mar 13 18:36:38 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 13 Mar 2012 10:36:38 -0700 Subject: [Python-Dev] [RELEASED] Distutils2 1.0a4 In-Reply-To: <4F5F77CB.1040600@netwok.org> References: <4F5F77CB.1040600@netwok.org> Message-ID: <4F5F85A6.1050802@ziade.org> Thanks a lot for your hard work and dedication on packaging ! On 3/13/12 9:37 AM, ?ric Araujo wrote: > Hello, > > On behalf of the distutils2 contributors, I am thrilled to announce the > release of Distutils2 1.0a4. > > Distutils2 is the packaging library that supersedes Distutils. It > supports distributing, uploading, downloading, installing and removing > projects, and is also a support library for other packaging tools like > pip and buildout. It will be provided as part of Python 3.3; this > release is a backport compatible with Python 2.5 to 2.7. > > Distutils2 1.0a4 contains a number of known bugs, limitations and > missing features, but we have released it to let users and developers > download and try it. This means you! If you want to report new issues, > request features or contribute, please read DEVNOTES.txt in the source > distribution or http://wiki.python.org/Distutils2/Contributing > > More alpha releases will be cut when important bugs get fixed during the > next few months, like Windows or PyPy compatibility. The first beta is > planned for June, and 1.0 final for August (to follow Python 3.3.0). > Until beta, the API is subject to drastic changes and code can get removed. > > Basic documentation is at http://docs.python.org/dev/packaging ; it will > get updated, expanded and improved in the coming months. > > Enjoy! > > Repository: http://hg.python.org/distutils2 > Bug tracker: http://bugs.python.org/ (component "Distutils2") > Mailing list: http://mail.python.org/mailman/listinfo/distutils-sig/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com From mark at hotpy.org Tue Mar 13 19:35:33 2012 From: mark at hotpy.org (Mark Shannon) Date: Tue, 13 Mar 2012 18:35:33 +0000 Subject: [Python-Dev] Exceptions in comparison operators In-Reply-To: References: <4F54B496.3030905@hotpy.org> Message-ID: <4F5F9375.8040007@hotpy.org> Guido van Rossum wrote: > Mark, did you do anything with my reply? Not yet. I noticed the difference when developing my HotPy VM (latest incarnation thereof) which substitutes a sequence of low-level bytecodes for the high-level ones when tracing. (A bit like PyPy but much more Python-specific and amenable to interpretation, rather than compilation) I generate all the code sequences for binary ops from a template and noticed the slight difference when running the test suite. My implementation of equals follows the same pattern as the arithmetic operators (which is why I was wondering if that were the correct behaviour). My definition of op1 == op2: def surrogate_eq(op1, op2): if $overrides(op1, op2, '__eq__'): if op2?__eq__: result = op2$__eq__(op1) if result is not NotImplemented: return result if op1?__eq__: result = op1$__eq__(op2) if result is not NotImplemented: return result else: if op1?__eq__: result = op1$__eq__(op2) if result is not NotImplemented: return result if op2?__eq__: result = op2$__eq__(op1) if result is not NotImplemented: return result return op1 is op2 Where: x$__op__ means special lookup (bypassing the instance dictionary): x?__op__ means has the named special method i.e. any('__op__' in t.__dict__ for t in type(op).__mro__)) and $overrides(op1, op2, 'xxx') means that type(op2) is a proper subtype of type(op1) *and* type(op1).__dict__['xxx'] != type(op2).__dict__['xxx'] It would appear that the current version is: def surrogate_eq(op1, op2): if is_proper_subtype_of( type(op1), type(op1) ): if op2?__eq__: result = op2$__eq__(op1) if result is not NotImplemented: return result if op1?__eq__: result = op1$__eq__(op2) if result is not NotImplemented: return result else: if op1?__eq__: result = op1$__eq__(op2) if result is not NotImplemented: return result if op2?__eq__: result = op2$__eq__(op1) if result is not NotImplemented: return result return op1 is op2 Which means that == behaves differently to + for subtypes which do not override the __eq__ method. Thus: class MyValue1: def __init__(self, val): self.val = val def __lt__(self, other): print("lt") return self.val < other.val def __gt__(self, other): print("gt") return self.val > other.val def __add__(self, other): print("add") return self.val + other.val def __radd__(self, other): print("radd") return self.val + other.val class MyValue2(MyValue1): pass a = MyValue1(1) b = MyValue2(2) print(a + b) print(a < b) currently prints the following: add 3 gt True Cheers, Mark. > > On Mon, Mar 5, 2012 at 10:41 AM, Guido van Rossum wrote: >> On Mon, Mar 5, 2012 at 4:41 AM, Mark Shannon wrote: >>> Comparing two objects (of the same type for simplicity) >>> involves a three stage lookup: >>> The class has the operator C.__eq__ >>> It can be applied to operator (descriptor protocol): C().__eq__ >>> and it produces a result: C().__eq__(C()) >>> >>> Exceptions can be raised in all 3 phases, >>> but an exception in the first phase is not really an error, >>> its just says the operation is not supported. >>> E.g. >>> >>> class C: pass >>> >>> C() == C() is False, rather than raising an Exception. >>> >>> If an exception is raised in the 3rd stage, then it is propogated, >>> as follows: >>> >>> class C: >>> def __eq__(self, other): >>> raise Exception("I'm incomparable") >>> >>> C() == C() raises an exception >>> >>> However, if an exception is raised in the second phase (descriptor) >>> then it is silenced: >>> >>> def no_eq(self): >>> raise Exception("I'm incomparable") >>> >>> class C: >>> __eq__ = property(no_eq) >>> >>> C() == C() is False. >>> >>> But should it raise an exception? >>> >>> The behaviour for arithmetic is different. >>> >>> def no_add(self): >>> raise Exception("I don't add up") >>> >>> class C: >>> __add__ = property(no_add) >>> >>> C() + C() raises an exception. >>> >>> So what is the "correct" behaviour? >>> It is my opinion that comparisons should behave like arithmetic >>> and raise an exception. >> I think you're probably right. This is one of those edge cases that >> are so rare (and always considered a bug in the user code) that we >> didn't define carefully what should happen. There are probably some >> implementation-specific reasons why it was done this way (comparisons >> use a very different code path from regular binary operators) but that >> doesn't sound like a very good reason. >> >> OTOH there *is* a difference: as you say, C() == C() is False when the >> class doesn't define __eq__, whereas C() + C() raises an exception if >> it doesn't define __add__. Still, this is more likely to have favored >> the wrong outcome for (2) by accident than by design. >> >> You'll have to dig through the CPython implementation and find out >> exactly what code needs to be changed before I could be sure though -- >> sometimes seeing the code jogs my memory. >> >> But I think of x==y as roughly equivalent to >> >> r = NotImplemented >> if hasattr(x, '__eq__'): >> r = x.__eq__(y) >> if r is NotImplemented and hasattr(y, '__eq__'): >> r = y.__eq__(x) >> if r is NotImplemented: >> r = False >> >> which would certainly suggest that (2) should raise an exception. A >> possibility is that the code looking for the __eq__ attribute >> suppresses *all* exceptions instead of just AttributeError. If you >> change no_eq() to return 42, for example, the comparison raises the >> much more reasonable TypeError: 'int' object is not callable. >> >> -- >> --Guido van Rossum (python.org/~guido) > > > From v+python at g.nevcal.com Tue Mar 13 19:58:13 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 13 Mar 2012 11:58:13 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: <20120313030704.46C512500ED@webabinitio.net> <20120313032548.GA29711@idyll.org> <20120313034819.GC29711@idyll.org> Message-ID: <4F5F98C5.1000309@g.nevcal.com> On 3/13/2012 6:31 AM, Paul Moore wrote: > It can be very hard to separate the good from the indifferent (or even > bad) when browsing PyPI. I've found some very good packages recently > which I'd never have known about without some random comment on a > mailing list. +1 > However, I'm not keen on having the stdlib documentation suggest that > I should be using something else. No code should ever be documenting > "don't use me, there are better alternatives" unless it is deprecated > or obsolete. +0 > On the other hand, I would love to see a community-maintained document > that described packages that are acknowledged as "best of breed". That > applies whether or not those packages replace something in the stdlib. > Things like pywin32, lxml, and requests would be examples in my > experience. +1 > There's no reason this*has* to be in the core > documentation - it may be relevant that nothing has sprung up > independently yet... Hmm. > Maybe a separate item in the Python documentation, "External Modules", > could be created and maintained by the community? By being in the > documentation, it has a level of "official recommendation" status, and > by being a top-level document it's visible (more so than, for example, > a HOWTO document would be). Because it's in the released > documentation, it is relatively stable, which implies that external > modules would need to have a genuine track record to get in there, but > because it's community maintained it should reflect a wider consensus > than just the core developers' views. +1 This is the best proposal I've seen for including references to external modules. It gets it in the core documentation, hopefully with enough keywords that search would typically find external modules that are supersets of stdlib modules in the same result set that the stdlib module would be found. Yet it doesn't intrude on the documentation for the stdlib module. And beyond a 1-2 paragraph description, would not be fully documented, except by referencing the external module's documentation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at kennethreitz.com Tue Mar 13 20:13:52 2012 From: me at kennethreitz.com (Kenneth Reitz) Date: Tue, 13 Mar 2012 12:13:52 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: I think the cheesehop trove classifiers would be the ideal way to agnostically link to a page of packages related to the standard package in question. No need for sort order. The beauty of this solution is that packages that aren't maintained won't add the appropriate classifier to their package, and therefore not show up in the list. -- Kenneth Reitz On Monday, March 12, 2012 at 9:23 PM, Brian Curtin wrote: > On Mon, Mar 12, 2012 at 21:14, Andrey Petrov wrote: > > On Mon, Mar 12, 2012 at 8:58 PM, Brian Curtin wrote: > > > On Mon, Mar 12, 2012 at 19:23, Andrey Petrov wrote: > > > > What such a snippet might look like: > > > > > > > > "Batteries are included with Python but sometimes they are old and > > > > leaky?this is one of those cases. Please have a look in PyPI for more modern > > > > alternatives provided by the Python community." > > > > > > > > > > > > > What does "leaky" mean here? Someone's going to see that, think it has > > > memory leaks, then rant on the internet about how we ship crap and > > > just document it as so. > > > > > > > > > I agree Brian and David, the choice of "leaky" in the wording is poor. > > It was supposed to be maintaining the "batteries" metaphor but it's > > clearly ambiguous. > > > > Perhaps something along the lines of... > > > > "Batteries are included with Python but for stability and backwards > > compatibility, some of the standard library is not always as modern as > > alternatives provided by the Python community?this is one of those > > cases. Please have a look at PyPI for more cutting-edge alternatives." > > > > > Sorry for another color choice on the bikeshed, but I would drop the > word or references to "batteries". *We* know what "batteries included" > means, but there are undoubtedly people who won't get it. It's just > code - let's call it code. > > > > > Part 2: > > > > I propose we add a new category of package identifiers such as "Topic :: > > > > Standard Library Alternative :: {stdlib_package_name}" which authors of > > > > libraries can tag themselves under. The documentation warning snippet will > > > > provide a link to the appropriate PyPI query to list packages claiming to be > > > > alternatives to the stdlib package in question. > > > > > > > > > > > > > Automating it to something on PyPI is the not the right answer. People > > > will use it incorrectly, either in that they'll add it to packages for > > > which it isn't accurate, and people just flat out won't use it or know > > > about it. It won't be accurate this way, and anything that we're > > > documenting needs to be vetted. > > > > > > It's not often that a great alternative comes up, so I don't see the > > > manual burden being too great. > > > > > > > > > There are a dozen or more urllib/httplib/pycurl competitors on PyPI, > > and new ones spring up all the time. I'm not sure how we would go > > about objectively blessing the best "official" option at any given > > moment, or how frequently we would have to do this. > > > > > The same way we choose to accept libraries into the standard library. > New ones spring up all the time - mature, proven, and widely used ones > do not. If someone thinks libfoo is ready, they suggest it. If we > haven't heard of it, the conversation ends. If we have people who know > it, maybe we have them look deeper and figure out if it's something we > can put our stamp on just like we might with the recent talk of > "experimental package" inclusion. > > > With self-identifying, we could sort by some sort metric (monthly > > downloads? magical score?) and create a somewhat-actionable list. > > > > > Downloads don't mean the code is good. Voting is gamed. I really don't > think there's a good automated solution to tell us what the > high-quality replacement projects are. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Tue Mar 13 20:38:04 2012 From: brian at python.org (Brian Curtin) Date: Tue, 13 Mar 2012 14:38:04 -0500 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 14:13, Kenneth Reitz wrote: > I think the cheesehop trove classifiers would be the ideal way to > agnostically link to a page of packages related to the standard package in > question. No need for sort order. Randomize the order for all I care. We still need to ensure we're suggesting quality projects. It doesn't make sense for us to suggest alternatives that we wouldn't want to use ourselves by just polling some list that anyone can get on. This is documentation that receives hundreds of thousands of views a month*. We need to be picky about what goes in it. > The beauty of this solution is that packages that aren't maintained won't > add the appropriate classifier to their package, and therefore not show up > in the list. Just because it's maintained doesn't mean it's not garbage. I think we really need to start every project off with a 0 and make them prove that they're a 10. Just being active means nothing. * http://www.python.org/webstats/usage_201202.html#TOPURLS - I don't know what page "Documentation" means since it doesn't have a specific link, but whatever page that is got hit 960K times in February. From fuzzyman at voidspace.org.uk Tue Mar 13 20:42:20 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 13 Mar 2012 12:42:20 -0700 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: Message-ID: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> On 13 Mar 2012, at 04:44, Merlijn van Deen wrote: > http://bugs.python.org/issue6784 ("byte/unicode pickle > incompatibilities between python2 and python3") > > Hello all, > > Currently, pickle unpickles python2 'str' objects as python3 'str' > objects, where the encoding to use is passed to the Unpickler. > However, there are cases where it makes more sense to unpickle a > python2 'str' as python3 'bytes' - for instance when it is actually > binary data, and not text. > > Currently, the mapping is as follows, when reading a pickle: > python2 'str' -> python3 'str' (using an encoding supplied to Unpickler) > python2 'unicode' -> python3 'str' > > or, when creating a pickle using protocol <= 2: > python3 'str' -> python2 'unicode' > python3 'bytes' -> python2 '__builtins__.bytes object' > It does seem unfortunate that by default it is impossible for a developer to "do the right thing" as regards pickling / unpickling here. Binary data on Python 2 being unpickled as Unicode on Python 3 is presumably for the convenience of developers doing the *wrong thing* (and only works for ascii anyway). All the best, Michael Foord > This issue suggests to add a flag to change the behaviour as follows: > a) python2 'str' -> python3 'bytes' > b) python3 'bytes' -> python2 'str' > > The question on this is how to pass this flag. To quote Antoine (with > permission) on my mail about this issue on core-mentorship: > >> I haven't answered because I'm unsure about the approach itself - do we >> want to add yet another argument to pickle methods, especially this late >> in the 3.x development cycle? > > > Currently, I have implemented it using an extra argument for the > Pickler and Unpickler objects ('bytestr'), which toggles the > behaviour. I.e.: >>>> pickled = Pickler(data, bytestr=True); unpickled = Unpickler(data, bytestr=True). > This is the approach used in pickle_bytestr.patch [1] > > Another option would be to implement a seperate Pickler/Unpickler > object, such that >>>> pickled = BytestrPickler(data, bytestr=True); unpickled = BytestrUnpickler(data, bytestr=True) > This is the approach I initially implemented [2]. > > Alternatively, there is the option only to implement the Unpickler, > leaving the Pickler as it is. This allows >>>> unpickled = Unpickler(data, encoding=bytes) > where the bytes type is used as a special 'flag'. > > And, of course, there is the option not to implement this in the stdlib at all. > > > What are your ideas on this? > > Best, > Merlijn > > [0] http://bugs.python.org/issue6784 > [1] http://bugs.python.org/file24719/pickle_bytestr.patch > [2] https://github.com/valhallasw/py2/blob/master/bytestrpickle.py > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From van.lindberg at gmail.com Tue Mar 13 20:43:27 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 13 Mar 2012 14:43:27 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 Message-ID: Following up on conversations at PyCon, I want to bring up one of my personal hobby horses for change in 3.3: Fix install layout on Windows, with a side order of making the PATH work better. Short version: 1) The layout for the python root directory for all platforms should be as follows: stdlib = {base/userbase}/lib/python{py_version_short} platstdlib = {base/userbase}/lib/python{py_version_short} purelib = {base/userbase}/lib/python{py_version_short}/site-packages platlib = {base/userbase}/lib/python{py_version_short}/site-packages include = {base/userbase}/include/python{py_version_short} scripts = {base/userbase}/bin data = {base/userbase} 2) On Windows, the python executable (python.exe) will be in the "bin" directory. That way the installer can optionally add just that directory to the PATH to pick up all python-related executables (like pip, easy_install, etc). I have talked with a number of people at PyCon, including Tarek and MvL. Nobody objected, and several thought it was a good idea. Long version: As a bit of background,the layout for the Python root directory is different between platforms, varying in capitalization ("Include" vs. "include") and sometimes in the names of directories used ("Scripts" on Windows vs. "bin" most everywhere else). Further, the python version may or may not be included in the path to the standard library or not. In times past, this layout was driven by an INSTALL_SCHEMES dict deep in the guts of distutils, but with distutils2 it has been lifted out and placed within a config file (sysconfig.cfg). [1] Proposal #1 above deals with this inconsistency [1]. More concretely, it also makes it so that I can check in an entire environment into source control and work on it cross platform. As an additional wrinkle on Windows, the main python binary (python.exe) is placed in the python root directory, but all associated runnable files are placed in the "Scripts" directory, so that someone who wants to run both Python and a Python script needs to add both $PYTHON_ROOT and $PYTHON_ROOT/Scripts to the PATH. To add just a little more complication, the python binary is placed within the binaries directory when a virtualenv is created, leading to an inconsistency between regular python installs and virtualenvs. Proposal #2 again provides consistency between virtualenvs and regular Python installs, and on windows it allows a single directory to be placed on the PATH to get all python-related binaries to run. [1] https://bitbucket.org/tarek/distutils2/src/6c3d67ed3adb/distutils2/_backport/sysconfig.cfg [2] It may be a foolish consistency, but I have a little mind. Thanks, Van From tjreedy at udel.edu Tue Mar 13 20:49:26 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 13 Mar 2012 15:49:26 -0400 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On 3/13/2012 12:40 AM, Guido van Rossum wrote: > On Mon, Mar 12, 2012 at 9:22 PM, Terry Reedy wrote: >> I would rather we figure out how to encourage authors of advancing packages >> to contribute better implementations of existing features and well-tested >> new features back to the stdlib module. > > I would not. I think you misunderstood me and are talking about something other than what I meant. There are about 3250 open issues (this slowly but steadily grows). Of them, 1450 are behavior (bug) issues. We need more people, especially people with specialized expertise, writing and reviewing patches. As you said in response to Senthil > Improving existing stdlib modules is always welcome Exactly. So I would like to figure out how to encourage more such improvements. > There are many excellent packages out there that should > not be made into stdlib packages simply because their authors are not > done adding new features. Or because the package is outside the reasonable scope of the stdlib, or requires a different type of expertise than most core development, or for other reasons. Authors of separately maintained packages are, from our viewpoint, as eligible to help with tracker issues as anyone else, even while they continue work on their external package. Some of them are more likely than most contributors to have the knowledge needed for some particular issues. -- Terry Jan Reedy From thomas at python.org Tue Mar 13 20:50:09 2012 From: thomas at python.org (Thomas Wouters) Date: Tue, 13 Mar 2012 12:50:09 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 12:38, Brian Curtin wrote: > On Tue, Mar 13, 2012 at 14:13, Kenneth Reitz wrote: > > I think the cheesehop trove classifiers would be the ideal way to > > agnostically link to a page of packages related to the standard package > in > > question. No need for sort order. > > Randomize the order for all I care. We still need to ensure we're > suggesting quality projects. It doesn't make sense for us to suggest > alternatives that we wouldn't want to use ourselves by just polling > some list that anyone can get on. > > This is documentation that receives hundreds of thousands of views a > month*. We need to be picky about what goes in it. > > > The beauty of this solution is that packages that aren't maintained won't > > add the appropriate classifier to their package, and therefore not show > up > > in the list. > > Just because it's maintained doesn't mean it's not garbage. I think we > really need to start every project off with a 0 and make them prove > that they're a 10. Just being active means nothing. > > > > * http://www.python.org/webstats/usage_201202.html#TOPURLS - I don't > know what page "Documentation" means since it doesn't have a specific > link, but whatever page that is got hit 960K times in February. > GroupURL /doc/* Documentation So it's anything that's in www.python.org/doc/. I don't believe it counts doc.python.org and docs.python.org. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Tue Mar 13 21:11:43 2012 From: brian at python.org (Brian Curtin) Date: Tue, 13 Mar 2012 15:11:43 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 14:43, VanL wrote: > Following up on conversations at PyCon, I want to bring up one of my > personal hobby horses for change in 3.3: Fix install layout on Windows, with > a side order of making the PATH work better. > > Short version: > > 1) The layout for the python root directory for all platforms should be as > follows: > > stdlib = {base/userbase}/lib/python{py_version_short} > platstdlib = {base/userbase}/lib/python{py_version_short} > purelib = {base/userbase}/lib/python{py_version_short}/site-packages > platlib = {base/userbase}/lib/python{py_version_short}/site-packages > include = {base/userbase}/include/python{py_version_short} > scripts = {base/userbase}/bin > data = {base/userbase} I'm familiar with the scripts/bin change. I take it the rest of that stuff matches *nix? Text later on seems to support this, so I think I'm on board with it. > 2) On Windows, the python executable (python.exe) will be in the "bin" > directory. That way the installer can optionally add just that directory to > the PATH to pick up all python-related executables (like pip, easy_install, > etc). I'm updating my installer patch to do exactly this. After talking with Dino from Microsoft's Python Tools team, we're also going to add an additional registry key for them to find that bin/ path. > I have talked with a number of people at PyCon, including Tarek and MvL. > Nobody objected, and several thought it was a good idea. Martin and I spoke on Friday and at least the bin/ folder and Path stuff are acceptable and I'm working on the code for those. > To add just a little more complication, the python binary is placed within > the binaries directory when a virtualenv is created, leading to an > inconsistency between regular python installs and virtualenvs. If that virtualenv PEP is also accepted for 3.3, I think we can take care of inconsistencies there (at least moving forward). From guido at python.org Tue Mar 13 22:13:31 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2012 14:13:31 -0700 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: On Tue, Mar 13, 2012 at 12:42 PM, Michael Foord wrote: > > On 13 Mar 2012, at 04:44, Merlijn van Deen wrote: > >> http://bugs.python.org/issue6784 ("byte/unicode pickle >> incompatibilities between python2 and python3") >> >> Hello all, >> >> Currently, pickle unpickles python2 'str' objects as python3 'str' >> objects, where the encoding to use is passed to the Unpickler. >> However, there are cases where it makes more sense to unpickle a >> python2 'str' as python3 'bytes' - for instance when it is actually >> binary data, and not text. >> >> Currently, the mapping is as follows, when reading a pickle: >> python2 'str' -> python3 'str' (using an encoding supplied to Unpickler) >> python2 'unicode' -> python3 'str' >> >> or, when creating a pickle using protocol <= 2: >> python3 'str' -> python2 'unicode' >> python3 'bytes' -> python2 '__builtins__.bytes object' >> > > > It does seem unfortunate that by default it is impossible for a developer to "do the right thing" as regards pickling / unpickling here. Binary data on Python 2 being unpickled as Unicode on Python 3 is presumably for the convenience of developers doing the *wrong thing* (and only works for ascii anyway). Well, since trying to migrate data between versions using pickle is the "wrong" thing anyway, I think the status quo is just fine. Developers doing the "right" thing don't use pickle for this purpose. -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Mar 13 22:16:40 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2012 14:16:40 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 12:49 PM, Terry Reedy wrote: > Authors of separately maintained packages are, from our viewpoint, as > eligible to help with tracker issues as anyone else, even while they > continue work on their external package. Some of them are more likely than > most contributors to have the knowledge needed for some particular issues. This is a good idea. I was chatting w. Senthil this morning about adding improvements to urllib/request.py based upon ideas from urllib3, requests, httplib2 (?), and we came to the conclusion that it might be a good idea to let those packages' authors review the proposed stdlib improvements. -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Tue Mar 13 22:19:57 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 13 Mar 2012 17:19:57 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: On 3/13/2012 3:43 PM, VanL wrote: > Following up on conversations at PyCon, I want to bring up one of my > personal hobby horses for change in 3.3: Fix install layout on Windows, > with a side order of making the PATH work better. > > Short version: > > 1) The layout for the python root directory for all platforms should be > as follows: > > stdlib = {base/userbase}/lib/python{py_version_short} > platstdlib = {base/userbase}/lib/python{py_version_short} > purelib = {base/userbase}/lib/python{py_version_short}/site-packages > platlib = {base/userbase}/lib/python{py_version_short}/site-packages > include = {base/userbase}/include/python{py_version_short} > scripts = {base/userbase}/bin > data = {base/userbase} What is {base/userbase} actually on a typical machine? It is fixed or user choice? -- Terry Jan Reedy From solipsis at pitrou.net Tue Mar 13 22:21:03 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 13 Mar 2012 22:21:03 +0100 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives References: Message-ID: <20120313222103.163877e6@pitrou.net> On Tue, 13 Mar 2012 14:16:40 -0700 Guido van Rossum wrote: > On Tue, Mar 13, 2012 at 12:49 PM, Terry Reedy wrote: > > Authors of separately maintained packages are, from our viewpoint, as > > eligible to help with tracker issues as anyone else, even while they > > continue work on their external package. Some of them are more likely than > > most contributors to have the knowledge needed for some particular issues. > > This is a good idea. I was chatting w. Senthil this morning about > adding improvements to urllib/request.py based upon ideas from > urllib3, requests, httplib2 (?), and we came to the conclusion that it > might be a good idea to let those packages' authors review the > proposed stdlib improvements. We don't have any provisions against reviewal by third-party developers already. I think the main problem (for us, of course) is that these people generally aren't interested enough to really dive in stdlib patches and proposals. For example, for the ssl module, I have sometimes tried to involve authors of third-party packages such as pyOpenSSL (or, IIRC, M2Crypto), but I got very little or no reviewing. Regards Antoine. From guido at python.org Tue Mar 13 22:38:03 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2012 14:38:03 -0700 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120313222103.163877e6@pitrou.net> References: <20120313222103.163877e6@pitrou.net> Message-ID: On Tue, Mar 13, 2012 at 2:21 PM, Antoine Pitrou wrote: > On Tue, 13 Mar 2012 14:16:40 -0700 > Guido van Rossum wrote: > >> On Tue, Mar 13, 2012 at 12:49 PM, Terry Reedy wrote: >> > Authors of separately maintained packages are, from our viewpoint, as >> > eligible to help with tracker issues as anyone else, even while they >> > continue work on their external package. Some of them are more likely than >> > most contributors to have the knowledge needed for some particular issues. >> >> This is a good idea. I was chatting w. Senthil this morning about >> adding improvements to urllib/request.py based upon ideas from >> urllib3, requests, httplib2 (?), and we came to the conclusion that it >> might be a good idea to let those packages' authors review the >> proposed stdlib improvements. > > We don't have any provisions against reviewal by third-party > developers already. I think the main problem (for us, of course) is that > these people generally aren't interested enough to really dive in > stdlib patches and proposals. > > For example, for the ssl module, I have sometimes tried to involve > authors of third-party packages such as pyOpenSSL (or, IIRC, M2Crypto), > but I got very little or no reviewing. IIRC M2Crypto is currently unmaintained, so that doesn't surprise me. (In general it seems most crypto wrappers seem unmaintained -- it must be a thankless job.) Still, AFAICT both requests and urllib3 are very actively maintained by people who know what they are doing, and it would be nice if we could build bridges instead of competition. So let's at least try. (But I'm not asking you, Antoine, to try and approach them personally. :-) -- --Guido van Rossum (python.org/~guido) From van.lindberg at gmail.com Tue Mar 13 22:39:10 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 13 Mar 2012 16:39:10 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: On 3/13/2012 4:19 PM, Terry Reedy wrote: > What is {base/userbase} actually on a typical machine? It is fixed or > user choice? It is based upon user choice and on whether it is a system-wide install (base) or a single-user install (userbase). Typically, though, it is just "where you installed Python" (/usr/local, C:\python\3.3, whatever). From van.lindberg at gmail.com Tue Mar 13 22:40:01 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 13 Mar 2012 16:40:01 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: On 3/13/2012 3:11 PM, Brian Curtin wrote: > I'm familiar with the scripts/bin change. I take it the rest of that > stuff matches *nix? Text later on seems to support this, so I think > I'm on board with it. Yes, that is correct. > Martin and I spoke on Friday and at least the bin/ folder and Path > stuff are acceptable and I'm working on the code for those. [...] > If that virtualenv PEP is also accepted for 3.3, I think we can take > care of inconsistencies there (at least moving forward). Thanks Brian! From valhallasw at arctus.nl Tue Mar 13 22:50:35 2012 From: valhallasw at arctus.nl (Merlijn van Deen) Date: Tue, 13 Mar 2012 22:50:35 +0100 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: On 13 March 2012 22:13, Guido van Rossum wrote: > Well, since trying to migrate data between versions using pickle is > the "wrong" thing anyway, I think the status quo is just fine. > Developers doing the "right" thing don't use pickle for this purpose. I'm confused by this. "The pickle serialization format is guaranteed to be backwards compatible across Python releases" [1], which - at least to me - suggests it's fine to use pickle for long-term storage, and that reading this data in new Python versions is not a "bad" thing to do. Am I missing something here? [1] http://docs.python.org/library/pickle.html#the-pickle-protocol From guido at python.org Tue Mar 13 23:08:35 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2012 15:08:35 -0700 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: On Tue, Mar 13, 2012 at 2:50 PM, Merlijn van Deen wrote: > On 13 March 2012 22:13, Guido van Rossum wrote: >> Well, since trying to migrate data between versions using pickle is >> the "wrong" thing anyway, I think the status quo is just fine. >> Developers doing the "right" thing don't use pickle for this purpose. > > I'm confused by this. "The pickle serialization format is guaranteed > to be backwards compatible across Python releases" [1], which - at > least to me - suggests it's fine to use pickle for long-term storage, > and that reading this data in new Python versions is not a "bad" > thing to do. Am I missing something here? > > [1] http://docs.python.org/library/pickle.html#the-pickle-protocol That was probably written before Python 3. Python 3 also dropped the long-term backwards compatibilities for the language and stdlib. I am certainly fine with adding a warning to the docs that this guarantee does not apply to the Python 2/3 boundary. But I don't think we should map 8-bit str instances from Python 2 to bytes in Python 3. My snipe was mostly in reference to the many other things that can go wrong with pickled data as your environment evolves -- if you're not careful you can have references (by name) to modules, functions, classes in pickled data that won't resolve in a later (or earlier!) version of your app, or you might have objects that are unpickled in an incomplete state that causes later use of the objects to break (e.g. if a newer version of __init__() sets some extra instance variables -- unpickling doesn't generally call __init__, so these new variables won't be set if they didn't exist in the old version). Etc., etc. If you can solve your problem with a suitably hacked Unpickler subclass that's fine with me, but I would personally use this opportunity to change the app to some other serialization format that is perhaps less general but more robust than pickle. I've been bitten by too many pickle-related problems to recommend pickle to anyone... -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Tue Mar 13 23:49:21 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Mar 2012 08:49:21 +1000 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: On Wed, Mar 14, 2012 at 8:08 AM, Guido van Rossum wrote: > If you can solve your problem with a suitably hacked Unpickler > subclass that's fine with me, but I would personally use this > opportunity to change the app to some other serialization format that > is perhaps less general but more robust than pickle. I've been bitten > by too many pickle-related problems to recommend pickle to anyone... It's fine for in-memory storage of (almost) arbitrary objects (I use it to stash things in a memory backed sqlite DB via SQLAlchemy) and for IPC, but yeah, for long-term cross-version persistent storage, I'd be looking to something like JSON rather than pickle. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From kristjan at ccpgames.com Tue Mar 13 23:53:42 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Tue, 13 Mar 2012 22:53:42 +0000 Subject: [Python-Dev] making python's c iterators picklable (http://bugs.python.org/issue14288) Message-ID: http://bugs.python.org/issue14288 Raymond suggested that this patch should be discussed here, so here goes: How this came about: There are frameworks, such as the Nagare web framework, (http://www.nagare.org/) that rely on suspending execution at some point and resuming it again. Nagare does this using Stackless python, pickles the execution state of a tasklet, and resumes it later (possibly elsewhere). Other frameworks are doing similar things in cloud computing. I have seen such presentation at previous PyCons, and they have had to write their own picklers to get around these problems. The problem is this: While pickling execution state (frame objects, functions) might be considered exotic, and indeed Stackless has modifications unique to it to do it, they quickly run into trouble that have nothing to do really with the fact that they are doing such exotic things. For example, the fact that the very common dictiter is implemented in C and not python, necessitates that special pickle support is done added for that, otherwise only some context can be pickled, (those that are not currently iterating through a dict) and not others. Now stackless has tried to provide this functionality for many years and indeed has special pickling support for dictiters, listiters, etc. (stuff that has nothing to do with the stacklessness of Stackless, really). However, (somewhat) recently a lot of the itertools were moved into C. Suddenly iterators, previously picklable (by merit of being in .py) stopped being that, just because they became C objects. In addition, a bunch of other iterators started showing up (stringiter, bytesiter). This started to cause problems. Suddenly you have to arbitrarily restrict what you can and can't do in code that is using these approaches. For Stackless, (and Nagare), it was necessary to ban the usage of the _itertools module in web programs. Instead of adding this to Stackless, and thus widening the gap between stackless and cpython, I think it is a good idea simply to fix this in cpython itself. Note that I also consider this to be of general utility to regular, non-exotic applications: Why should an application, that is working with a bunch of data, but wants to stop that for a bit, and maybe save it out to disk, have to worry about transforming the data into valid primitive datastructures before doing so? In my opinion, any objects that have simple and obvious pickle semantics should be picklable. Iterators are just regular objects with some state. They are not file pointers or sockets or database cursors. And again, I argue that if these objects were implemented in .py, they would already be automatically picklable (indeed, itertools.py was). The detail that some iterators in standard python are implemented in C should not automatically restrict their usage for no particular reason. The patch is straightforward. Most of it is tests, in fact. But it does use a few tricks in some places to get around the fact that some of those iterator types are hidden. We did try to be complete and find all the c iterators, but it was a year ago that the bulk of this work was done and something might have been added in the meantime. Anyway, that's my pitch. Kristj?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From jackdied at gmail.com Tue Mar 13 23:58:07 2012 From: jackdied at gmail.com (Jack Diederich) Date: Tue, 13 Mar 2012 18:58:07 -0400 Subject: [Python-Dev] making python's c iterators picklable (http://bugs.python.org/issue14288) In-Reply-To: References: Message-ID: 2012/3/13 Kristj?n Valur J?nsson : > http://bugs.python.org/issue14288 > In my opinion, any objects that have simple and obvious pickle semantics > should be picklable.? Iterators are just regular objects with some state. > They are not file pointers or sockets or database cursors.? And again, I > argue that if these objects were implemented in .py, they would already be > automatically picklable (indeed, itertools.py was).? The detail that some > iterators in standard python are implemented in C should not automatically > restrict their usage for no particular reason. +1, things that can be pickled should be pickleable. -Jack From solipsis at pitrou.net Wed Mar 14 00:05:43 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 00:05:43 +0100 Subject: [Python-Dev] making python's c iterators picklable (http://bugs.python.org/issue14288) References: Message-ID: <20120314000543.453c9463@pitrou.net> On Tue, 13 Mar 2012 22:53:42 +0000 Kristj?n Valur J?nsson wrote: > http://bugs.python.org/issue14288 > > Raymond suggested that this patch should be discussed here, so here goes: Sounds good on the principle. Of course, the patch needs to be reviewed. cheers Antoine. From tismer at stackless.com Wed Mar 14 00:54:10 2012 From: tismer at stackless.com (Christian Tismer) Date: Tue, 13 Mar 2012 16:54:10 -0700 Subject: [Python-Dev] making python's c iterators picklable (http://bugs.python.org/issue14288) In-Reply-To: <20120314000543.453c9463@pitrou.net> References: <20120314000543.453c9463@pitrou.net> Message-ID: <4F5FDE22.5080305@stackless.com> On 3/13/12 4:05 PM, Antoine Pitrou wrote: > On Tue, 13 Mar 2012 22:53:42 +0000 > Kristj?n Valur J?nsson wrote: >> http://bugs.python.org/issue14288 >> >> Raymond suggested that this patch should be discussed here, so here goes: > Sounds good on the principle. > Of course, the patch needs to be reviewed. I am very much for this patch. Of course I am biased by my stackless history and would be very happy to get most pickling into Python, where it belongs. To improve the patch a bit, I propose to put the hidden types into some module. I agree with Kristjan that the types module would be a good place to bootstrap the hidden iterable types. I even would like to propose a PEP: Whenever stuff is turned into C, which was picklable when implemented in Python, or something new is implemented that makes sense to pickle, a pickle implementation should always be provided. cheers -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a. PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From steve at pearwood.info Wed Mar 14 00:55:35 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 14 Mar 2012 10:55:35 +1100 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: <4F5FDE77.7030601@pearwood.info> Brian Curtin wrote: > On Tue, Mar 13, 2012 at 14:13, Kenneth Reitz wrote: >> I think the cheesehop trove classifiers would be the ideal way to >> agnostically link to a page of packages related to the standard package in >> question. No need for sort order. > > Randomize the order for all I care. We still need to ensure we're > suggesting quality projects. It doesn't make sense for us to suggest > alternatives that we wouldn't want to use ourselves by just polling > some list that anyone can get on. "Need" is awfully strong. I don't believe it is the responsibility of the standard library to be judge and reviewer of third party packages that it doesn't control. -1 on adding *any* sort of recommendations about third-party software except, at most, a case-by-case basis where absolutely necessary. What problem are we actually trying to solve here? Do we think that there are users who really have no clue where to find 3rd party software AND don't know how to use Google, BUT read the Python docs? I find it difficult to believe that there are people who both read the docs and are so clueless that they need to be told that there are alternatives available and which they should be using. Personally I think this is a solution in search of a problem. Judging by the python-tutor mailing list, even *beginners* know that they aren't limited to the stdlib and how to go about finding third party software. There are many more questions about PyGame and wxPython than there are about tkinter. There are plenty of questions about numpy. There are lots of questions about niche packages I'd never even heard of. I simply don't think there is any evidence that there are appreciable numbers of Python coders, beginners or expert, who need to be told about third party software. Who are these people we're trying to reach out to? > This is documentation that receives hundreds of thousands of views a > month*. We need to be picky about what goes in it. Exactly why we should be wary of recommending specific packages. Should we recommend wxPython over Pyjamas or PyGUI or PyGtk? On what basis? Whatever choice we make is going to be wrong for some people, and is potentially unfair to the maintainers of the packages left out. Should we recommend them all? That's no help to anyone. Make no recommendation at all? That's the status quo. What counts as "best of breed" can change rapidly -- software churn is part of the reason that the packages aren't in the stdlib in the first place. It can also be a matter of taste and convention. There are a few non-brainers, like numpy, but everything else, no, let's not open this can of worms. I can see no benefit to this suggestion, and all sorts of ways that this might go badly. -- Steven From victor.stinner at gmail.com Wed Mar 14 00:57:16 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 14 Mar 2012 00:57:16 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? Message-ID: Hi, I added two functions to the time module in Python 3.3: wallclock() and monotonic(). I'm unable to explain the difference between these two functions, even if I wrote them :-) wallclock() is suppose to be more accurate than time() but has an unspecified starting point. monotonic() is similar except that it is monotonic: it cannot go backward. monotonic() may not be available or fail whereas wallclock() is available/work, but I think that the two functions are redundant. I prefer to keep only monotonic() because it is not affected by system clock update and should help to fix issues on NTP update in functions implementing a timeout. What do you think? -- monotonic() has 3 implementations: * Windows: QueryPerformanceCounter() with QueryPerformanceFrequency() * Mac OS X: mach_absolute_time() with mach_timebase_info() * UNIX: clock_gettime(CLOCK_MONOTONIC_RAW) or clock_gettime(CLOCK_MONOTONIC) wallclock() has 3 implementations: * Windows: QueryPerformanceCounter() with QueryPerformanceFrequency(), with a fallback to GetSystemTimeAsFileTime() if QueryPerformanceFrequency() failed * UNIX: clock_gettime(CLOCK_MONOTONIC_RAW), clock_gettime(CLOCK_MONOTONIC) or clock_gettime(CLOCK_REALTIME), with a fallback to gettimeofday() if clock_gettime(*) failed * Otherwise: gettimeofday() (wallclock should also use mach_absolute_time() on Mac OS X) Victor From fuzzyman at voidspace.org.uk Wed Mar 14 01:03:58 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 13 Mar 2012 17:03:58 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: <6EAD1A31-F759-4AF7-A636-68CB21ADDB18@voidspace.org.uk> On 13 Mar 2012, at 16:57, Victor Stinner wrote: > Hi, > > I added two functions to the time module in Python 3.3: wallclock() > and monotonic(). I'm unable to explain the difference between these > two functions, even if I wrote them :-) wallclock() is suppose to be > more accurate than time() but has an unspecified starting point. > monotonic() is similar except that it is monotonic: it cannot go > backward. monotonic() may not be available or fail whereas wallclock() > is available/work, but I think that the two functions are redundant. > > I prefer to keep only monotonic() because it is not affected by system > clock update and should help to fix issues on NTP update in functions > implementing a timeout. > > What do you think? > I am in the middle of adding a feature to unittest that involves timing of individual tests. I want the highest resolution cross platform measure of wallclock time - and time.wallclock() looked ideal. If monotonic may not exist or can fail why would that be better? Michael > -- > > monotonic() has 3 implementations: > * Windows: QueryPerformanceCounter() with QueryPerformanceFrequency() > * Mac OS X: mach_absolute_time() with mach_timebase_info() > * UNIX: clock_gettime(CLOCK_MONOTONIC_RAW) or clock_gettime(CLOCK_MONOTONIC) > > wallclock() has 3 implementations: > * Windows: QueryPerformanceCounter() with QueryPerformanceFrequency(), > with a fallback to GetSystemTimeAsFileTime() if > QueryPerformanceFrequency() failed > * UNIX: clock_gettime(CLOCK_MONOTONIC_RAW), > clock_gettime(CLOCK_MONOTONIC) or clock_gettime(CLOCK_REALTIME), with > a fallback to gettimeofday() if clock_gettime(*) failed > * Otherwise: gettimeofday() > > (wallclock should also use mach_absolute_time() on Mac OS X) > > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From nadeem.vawda at gmail.com Wed Mar 14 01:18:26 2012 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Wed, 14 Mar 2012 02:18:26 +0200 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: So wallclock() falls back to a not-necessarily-monotonic time source if necessary, while monotonic() raises an exception in that case? ISTM that these don't need to be separate functions - rather, we can have one function that takes a flag (called require_monotonic, or something like that) telling it which failure mode to use. Does that make sense? Cheers, Nadeem From guido at python.org Wed Mar 14 01:27:14 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2012 17:27:14 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 4:57 PM, Victor Stinner wrote: > I added two functions to the time module in Python 3.3: wallclock() > and monotonic(). I'm unable to explain the difference between these > two functions, even if I wrote them :-) wallclock() is suppose to be > more accurate than time() but has an unspecified starting point. > monotonic() is similar except that it is monotonic: it cannot go > backward. monotonic() may not be available or fail whereas wallclock() > is available/work, but I think that the two functions are redundant. > > I prefer to keep only monotonic() because it is not affected by system > clock update and should help to fix issues on NTP update in functions > implementing a timeout. > > What do you think? I think wallclock() is an awkward name; in other contexts I've seen "wall clock time" used to mean the time that a clock on the wall would show, i.e. local time. This matches definition #1 of http://www.catb.org/jargon/html/W/wall-time.html (while yours matches #2 :-). I agree that it's better to have only one of these. I also think if we offer it we should always have it -- if none of the implementations are available, I guess you could fall back on returning time.time(), with some suitable offset so people don't think it is always the same. Maybe it could be called realtime()? -- --Guido van Rossum (python.org/~guido) From kristjan at ccpgames.com Wed Mar 14 01:27:14 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 00:27:14 +0000 Subject: [Python-Dev] sharing sockets among processes on windows Message-ID: Hi, I?m interested in contributing a patch to duplicate sockets between processes on windows. Tha api to do this is WSADuplicateSocket/WSASocket(), as already used by dup() in the _socketmodule.c Here?s what I have: 1) Sockets have a method, duplicate(target_pid), that return a bytes object containing the socket info for the target process 2) When socket(x, y, z, data) is called with this bytes object as the fourth argument, it is recreated from that. What are your thoughts on this? Also, should I try to reuse the socket.dup() function somehow, perhaps by giving the target pid? Secondly, there is multiprocessing.reduction which is doing similar things for unix. Does anyone familiar with it know how it goes about doing this? Would it be simple to change it to use this mechanism on windows? Kristj?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Mar 14 01:29:46 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 14 Mar 2012 01:29:46 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: <4F5FE67A.3020505@gmail.com> On 14/03/2012 01:18, Nadeem Vawda wrote: > So wallclock() falls back to a not-necessarily-monotonic time source > if necessary, > while monotonic() raises an exception in that case? ISTM that these > don't need to > be separate functions - rather, we can have one function that takes a > flag (called > require_monotonic, or something like that) telling it which failure mode to use. > Does that make sense? I don't think that time.monotonic() can fail in practice and it is available for all modern platforms (Windows, Mac OS X and OS implemented clock_gettime()). On Windows, time.monotonic() fails with an OSError if QueryPerformanceFrequency() failed. QueryPerformanceFrequency() can fail if "the installed hardware does not support a high-resolution performance counter" according to Microsoft documentation. Windows uses the CPU RDTSC instruction, or the ACPI power management timer or event the old 8245 PIT. I think that you have at least one of this device on your computer. I suppose that you can use a manual fallback to time.time() if time.monotonic() failed. If time.monotonic() fails, it fails directly at the first call. Example of a fallback working with Python < 3.3: try: time.monotonic() except (OSError, AttributeError): get_time = time.time else: get_time = time.monotonic Victor From kristjan at ccpgames.com Wed Mar 14 01:45:27 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 00:45:27 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: The reason I originally suggested "wallclock" was because that term is often used to distinguish time measurements (delta) that show real world time from those showing CPU or Kernel time. "number.crunch() took 2 seconds wallclock time but only 1 second CPU!". The original problem was that time.clock() was "wallclock" on some platforms but "cpu" on others, IIRC. But monotonic is probably even better. I agree removing one or the other, probably wallclock. K -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Guido van Rossum Sent: 13. mars 2012 17:27 To: Victor Stinner Cc: Python Dev Subject: Re: [Python-Dev] Drop the new time.wallclock() function? I think wallclock() is an awkward name; in other contexts I've seen "wall clock time" used to mean the time that a clock on the wall would show, i.e. local time. This matches definition #1 of http://www.catb.org/jargon/html/W/wall-time.html (while yours matches #2 :-). I agree that it's better to have only one of these. I also think if we offer it we should always have it -- if none of the implementations are available, I guess you could fall back on returning time.time(), with some suitable offset so people don't think it is always the same. Maybe it could be called realtime()? From victor.stinner at gmail.com Wed Mar 14 02:03:42 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 14 Mar 2012 02:03:42 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: > I agree that it's better to have only one of these. I also think if we > offer it we should always have it -- if none of the implementations > are available, I guess you could fall back on returning time.time(), > with some suitable offset so people don't think it is always the same. > Maybe it could be called realtime()? For a concrete use case, see for example: http://bugs.python.org/issue14222 I just wrote two patches, for the queue and threading modules, using time.monotonic() if available, with a fallback to time.time(). My patches call time.monotonic() to ensure that it doesn't fail with OSError. I suppose that most libraries and programs will have to implement a similar fallback. We may merge both functions with a flag to be able to disable the fallback. Example: - time.realtime(): best-effort monotonic, with a fallback - time.realtime(monotonic=True): monotonic, may raise OSError or NotImplementedError Victor From tismer at stackless.com Wed Mar 14 02:06:35 2012 From: tismer at stackless.com (Christian Tismer) Date: Tue, 13 Mar 2012 18:06:35 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: <4F5FEF1B.8030306@stackless.com> On 3/13/12 5:45 PM, Kristj?n Valur J?nsson wrote: > The reason I originally suggested "wallclock" was because that term is often used to distinguish time measurements (delta) that show real world time from those showing CPU or Kernel time. "number.crunch() took 2 seconds wallclock time but only 1 second CPU!". The original problem was that time.clock() was "wallclock" on some platforms but "cpu" on others, IIRC. > But monotonic is probably even better. I agree removing one or the other, probably wallclock. > K > > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Guido van Rossum > Sent: 13. mars 2012 17:27 > To: Victor Stinner > Cc: Python Dev > Subject: Re: [Python-Dev] Drop the new time.wallclock() function? > > I think wallclock() is an awkward name; in other contexts I've seen "wall clock time" used to mean the time that a clock on the wall would show, i.e. local time. This matches definition #1 of http://www.catb.org/jargon/html/W/wall-time.html (while yours matches > #2 :-). > > I agree that it's better to have only one of these. I also think if we offer it we should always have it -- if none of the implementations are available, I guess you could fall back on returning time.time(), with some suitable offset so people don't think it is always the same. > Maybe it could be called realtime()? > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/tismer%40stackless.com Btw., have you considered virtual machines? I happen to run windows in Parallels or Virtualbox quite often. There the "wall clock" stuff notoriously does not work. It would be good (but difficult?) if the supposed-to-be-accurate clock could test itself, if it works at all, and replace itself with a fallback. In my case, this causes quite a few PyPy tests to fail ;-) ciao -- Chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a. PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From yselivanov.ml at gmail.com Wed Mar 14 02:09:46 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 13 Mar 2012 21:09:46 -0400 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: If we need to decide to which function should be kept - I vote for monotonic. It's extremely useful (even essential) to track timeouts in various schedulers implementations, for example. Quick search also shows the demand for it, as there are questions on stackoverflow.com and few packages on PyPI. - Yury On 2012-03-13, at 9:03 PM, Victor Stinner wrote: >> I agree that it's better to have only one of these. I also think if we >> offer it we should always have it -- if none of the implementations >> are available, I guess you could fall back on returning time.time(), >> with some suitable offset so people don't think it is always the same. >> Maybe it could be called realtime()? > > For a concrete use case, see for example: > http://bugs.python.org/issue14222 > > I just wrote two patches, for the queue and threading modules, using > time.monotonic() if available, with a fallback to time.time(). My > patches call time.monotonic() to ensure that it doesn't fail with > OSError. > > I suppose that most libraries and programs will have to implement a > similar fallback. > > We may merge both functions with a flag to be able to disable the > fallback. Example: > > - time.realtime(): best-effort monotonic, with a fallback > - time.realtime(monotonic=True): monotonic, may raise OSError or > NotImplementedError > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com From nadeem.vawda at gmail.com Wed Mar 14 02:10:46 2012 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Wed, 14 Mar 2012 03:10:46 +0200 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: On Wed, Mar 14, 2012 at 3:03 AM, Victor Stinner wrote: > I suppose that most libraries and programs will have to implement a > similar fallback. > > We may merge both functions with a flag to be able to disable the > fallback. Example: > > ?- time.realtime(): best-effort monotonic, with a fallback > ?- time.realtime(monotonic=True): monotonic, may raise OSError or > NotImplementedError This was my suggestion - I think it's useful to have the fallback available (since most users will want it), but at the same time we should also cater to users who need a clock that is *guaranteed* to be monotonic. As an aside, I think "monotonic" is a better name than "realtime"; it conveys the functions purpose more clearly. Then we could call the flag "strict". Cheers, Nadeem From andrew.svetlov at gmail.com Wed Mar 14 02:12:32 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Tue, 13 Mar 2012 18:12:32 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <4F5FE67A.3020505@gmail.com> References: <4F5FE67A.3020505@gmail.com> Message-ID: On Tue, Mar 13, 2012 at 5:29 PM, Victor Stinner wrote: > I suppose that you can use a manual fallback to time.time() if > time.monotonic() failed. If time.monotonic() fails, it fails directly at the > first call. Example of a fallback working with Python < 3.3: > > try: > ? time.monotonic() > except (OSError, AttributeError): > ? get_time = time.time > else: > ? get_time = time.monotonic > I like 'fallback' solution while `get_time` is not the best name for high precision timer from my perspective. Can you call it `monotonic` or `realtime`? From kristjan at ccpgames.com Wed Mar 14 02:11:51 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 01:11:51 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <4F5FEF1B.8030306@stackless.com> References: <4F5FEF1B.8030306@stackless.com> Message-ID: Interesting thought. Althougn I don't see how that could fail on windows, if the QPC function is really just talking to a clock chip, surely that hasn't been virtualized. Is there an actual example of windows hardware where this api fails (virtual or not?) Perhaps there is no real need to have a fallback mechanism, and it would even be best to write such a mechanism inside the function itself, and just return getsystemtimeasfiletime() instead. K -----Original Message----- From: Christian Tismer [mailto:tismer at stackless.com] Sent: 13. mars 2012 18:07 To: Kristj?n Valur J?nsson Cc: Guido van Rossum; Victor Stinner; Python Dev Subject: Re: [Python-Dev] Drop the new time.wallclock() function? Btw., have you considered virtual machines? I happen to run windows in Parallels or Virtualbox quite often. There the "wall clock" stuff notoriously does not work. It would be good (but difficult?) if the supposed-to-be-accurate clock could test itself, if it works at all, and replace itself with a fallback. In my case, this causes quite a few PyPy tests to fail ;-) ciao -- Chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a. PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido at python.org Wed Mar 14 02:34:05 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Mar 2012 18:34:05 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 6:03 PM, Victor Stinner wrote: >> I agree that it's better to have only one of these. I also think if we >> offer it we should always have it -- if none of the implementations >> are available, I guess you could fall back on returning time.time(), >> with some suitable offset so people don't think it is always the same. >> Maybe it could be called realtime()? > > For a concrete use case, see for example: > http://bugs.python.org/issue14222 > > I just wrote two patches, for the queue and threading modules, using > time.monotonic() if available, with a fallback to time.time(). My > patches call time.monotonic() to ensure that it doesn't fail with > OSError. > > I suppose that most libraries and programs will have to implement a > similar fallback. It seems horrible to force everyone to copy the same silly block of code. The time module itself should implement this once. > We may merge both functions with a flag to be able to disable the > fallback. Example: > > ?- time.realtime(): best-effort monotonic, with a fallback > ?- time.realtime(monotonic=True): monotonic, may raise OSError or > NotImplementedError I have no opinions on this or other API details. But please make the function always exist and return something vaguely resembling a monotonic real-time clock. (BTW IMO the docs should state explicitly that it returns a float.) -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Wed Mar 14 02:37:38 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Mar 2012 18:37:38 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: <4F5FF662.4070201@v.loewis.de> > 1) The layout for the python root directory for all platforms should be > as follows: > > stdlib = {base/userbase}/lib/python{py_version_short} > platstdlib = {base/userbase}/lib/python{py_version_short} > purelib = {base/userbase}/lib/python{py_version_short}/site-packages > platlib = {base/userbase}/lib/python{py_version_short}/site-packages > include = {base/userbase}/include/python{py_version_short} > scripts = {base/userbase}/bin > data = {base/userbase} [...] > I have talked with a number of people at PyCon, including Tarek and MvL. > Nobody objected, and several thought it was a good idea. I admit that I didn't understand that lib/python{version} was also part of the proposal. I'm fine with the bin/ change, but skeptical about the lib change - this just adds a redundant level of directories on Windows. The installation will end up in c:\python33\lib\python3.3 which has the software name and version twice in the path. Do we *really* need this? Regards, Martin From van.lindberg at gmail.com Wed Mar 14 02:57:38 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 13 Mar 2012 20:57:38 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F5FF662.4070201@v.loewis.de> References: <4F5FF662.4070201@v.loewis.de> Message-ID: On Mar 13, 2012, at 8:37 PM, "Martin v. L?wis" wrote: >> 1) The layout for the python root directory for all platforms should be >> as follows: >> >> stdlib = {base/userbase}/lib/python{py_version_short} >> platstdlib = {base/userbase}/lib/python{py_version_short} >> purelib = {base/userbase}/lib/python{py_version_short}/site-packages >> platlib = {base/userbase}/lib/python{py_version_short}/site-packages >> include = {base/userbase}/include/python{py_version_short} >> scripts = {base/userbase}/bin >> data = {base/userbase} > [...] >> I have talked with a number of people at PyCon, including Tarek and MvL. >> Nobody objected, and several thought it was a good idea. > > I admit that I didn't understand that lib/python{version} was > also part of the proposal. I'm fine with the bin/ change, but > skeptical about the lib change - this just adds a redundant level > of directories on Windows. The installation will end up in > > c:\python33\lib\python3.3 > > which has the software name and version twice in the path. > > Do we *really* need this? We *already* have this. The only difference in this proposal is that we go from py_version_nodot to py_version_short, i.e. from c:\python33\lib\python33 to c:\python33\lib\python3.3 Given that we already repeat it, isn't it better to be consistent? Thanks, Van From tjreedy at udel.edu Wed Mar 14 03:58:40 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 13 Mar 2012 22:58:40 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F5FF662.4070201@v.loewis.de> Message-ID: On 3/13/2012 9:57 PM, VanL wrote: > On Mar 13, 2012, at 8:37 PM, "Martin v. L?wis" > wrote: > >>> 1) The layout for the python root directory for all platforms >>> should be as follows: >>> >>> stdlib = {base/userbase}/lib/python{py_version_short} platstdlib >>> = {base/userbase}/lib/python{py_version_short} purelib = >>> {base/userbase}/lib/python{py_version_short}/site-packages >>> platlib = >>> {base/userbase}/lib/python{py_version_short}/site-packages >>> include = {base/userbase}/include/python{py_version_short} >>> scripts = {base/userbase}/bin data = {base/userbase} >> [...] >>> I have talked with a number of people at PyCon, including Tarek >>> and MvL. Nobody objected, and several thought it was a good >>> idea. >> >> I admit that I didn't understand that lib/python{version} was also >> part of the proposal. I'm fine with the bin/ change, but skeptical >> about the lib change - this just adds a redundant level of >> directories on Windows. The installation will end up in >> >> c:\python33\lib\python3.3 >> >> which has the software name and version twice in the path. >> >> Do we *really* need this? > > > We *already* have this. The only difference in this proposal is that > we go from py_version_nodot to py_version_short, i.e. from > > c:\python33\lib\python33 Right not, we (at least I) have .../python33/Lib/ .../python32/Lib/ > to > > c:\python33\lib\python3.3 > > Given that we already repeat it, isn't it better to be consistent? But there is no repetition currently on Windows installations. I though you were just proposing to switch lib (lower-cased, and scripts renamed as bin, and pythonxx). So I do not think I yet understand what the proposal is and how it would be different from what I have now. -- > Terry Jan Reedy From anacrolix at gmail.com Wed Mar 14 04:31:34 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 14 Mar 2012 11:31:34 +0800 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120313222103.163877e6@pitrou.net> References: <20120313222103.163877e6@pitrou.net> Message-ID: On Mar 14, 2012 5:27 AM, "Antoine Pitrou" wrote: > > On Tue, 13 Mar 2012 14:16:40 -0700 > Guido van Rossum wrote: > > > On Tue, Mar 13, 2012 at 12:49 PM, Terry Reedy wrote: > > > Authors of separately maintained packages are, from our viewpoint, as > > > eligible to help with tracker issues as anyone else, even while they > > > continue work on their external package. Some of them are more likely than > > > most contributors to have the knowledge needed for some particular issues. > > > > This is a good idea. I was chatting w. Senthil this morning about > > adding improvements to urllib/request.py based upon ideas from > > urllib3, requests, httplib2 (?), and we came to the conclusion that it > > might be a good idea to let those packages' authors review the > > proposed stdlib improvements. > > We don't have any provisions against reviewal by third-party > developers already. I think the main problem (for us, of course) is that > these people generally aren't interested enough to really dive in > stdlib patches and proposals. > > For example, for the ssl module, I have sometimes tried to involve > authors of third-party packages such as pyOpenSSL (or, IIRC, M2Crypto), > but I got very little or no reviewing. Rather than indicating apathy on the party of third party developers, this might be a sign that core Python is unapproachable or not worth the effort. For instance I have several one line patches languishing, I can't imagine how disappointing it would be to have significantly larger patches ignored, but it happens. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Wed Mar 14 05:03:10 2012 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 14 Mar 2012 06:03:10 +0200 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: <20120313222103.163877e6@pitrou.net> Message-ID: > Rather than indicating apathy on the party of third party developers, this > might be a sign that core Python is unapproachable or not worth the effort. > > For instance I have several one line patches languishing, I can't imagine > how disappointing it would be to have significantly larger patches ignored, > but it happens. > A one-line patch for a complex module or piece of code may require much more than looking at that single line to really review. I hope you understand that. That said, if you find any issues in the bug tracker that in your opinion need only a few minutes of attention from a core developer, feel free to send a note to the mentorship mailing list. People sometimes come there asking for precisely this thing (help reviewing a simple patch they submitted), and usually get help quickly if their request is justified. Eli From kristjan at ccpgames.com Wed Mar 14 05:26:16 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 04:26:16 +0000 Subject: [Python-Dev] SocketServer issues Message-ID: Hi there. I want to mention some issues I've had with the socketserver module, and discuss if there's a way to make it nicer. So, for a long time we were able to create magic stackless mixin classes for it, like ThreadingMixIn, and assuming we had the appropriate socket replacement library, be able to use it nicely using tasklets. Then, at some point, the run_forever loop was changed to support timeout through the use of select.select() before every socket.accept() call. This was very awkward because the whole concept of select() really goes contrary to the approach of using microthreads, non-blocking IO and all that. The only reason for this select call, was to support timeout for the accept. And even for vanilla applications, it necessiates an extra kernel call for every accept, just for the timeout. The way around this for me has to do local modifications to the SocketServer and just get rid of the select. So, my first question is: Why not simply rely on the already built-in timeout support in the socket module? Setting the correct timeout value on the accepting socket, will achieve the same thing. Of course, one then has to reset the timeout value on the accepted socket, but this is minor. Second: Of late the SocketServer has grown additional features and attributes. In particular, it now has two event objects, __shutdown_request and __is_shut_down. Notice the double underscores. They make it impossible to subclass the SocketServer class to provide a different implementation of run_forever(). Is there any good reason why these attributes have been made "private" like this? Having just seen Raymond's talk on how to subclass right, this looks like the wrong way to use the double leading underscores. So, two things really: The use of select.select in SocketServer makes it necessary to subclass it to write a new version of run_forever() for those that wish to use a non-blocking IO library instead of socket. And the presence of these private attributes make it (theoretically) impossible to specialize run_forever in a mix-in class. Any thoughs? Is anyone interested in seeing how the timeouts can be done without using select.select()? And what do you think about removing the double underscores from there and thus making serve_forever owerrideable? Kristj?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Wed Mar 14 06:24:20 2012 From: tismer at stackless.com (Christian Tismer) Date: Tue, 13 Mar 2012 22:24:20 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F5FEF1B.8030306@stackless.com> Message-ID: <4F602B84.2020208@stackless.com> The performancecounter is a thing that typically gets intercepted by the VM infrastructure and does no longer work as a reliable timing source. In PyPy there are tests which check certain assumptions how much the performancecounter must advance at least between a few opcodes, and that does not work in a VM. cheers - Chris On 3/13/12 6:11 PM, Kristj?n Valur J?nsson wrote: > Interesting thought. > Althougn I don't see how that could fail on windows, if the QPC function is really just talking to a clock chip, surely that hasn't been virtualized. > Is there an actual example of windows hardware where this api fails (virtual or not?) Perhaps there is no real need to have a fallback mechanism, and it would even be best to write such a mechanism inside the function itself, and just return getsystemtimeasfiletime() instead. > > K > > -----Original Message----- > From: Christian Tismer [mailto:tismer at stackless.com] > Sent: 13. mars 2012 18:07 > To: Kristj?n Valur J?nsson > Cc: Guido van Rossum; Victor Stinner; Python Dev > Subject: Re: [Python-Dev] Drop the new time.wallclock() function? > > Btw., have you considered virtual machines? > I happen to run windows in Parallels or Virtualbox quite often. > There the "wall clock" stuff notoriously does not work. > > It would be good (but difficult?) if the supposed-to-be-accurate clock could test itself, if it works at all, and replace itself with a fallback. > > In my case, this causes quite a few PyPy tests to fail ;-) > > ciao -- Chris > -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a. PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From rdmurray at bitdance.com Wed Mar 14 06:29:15 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 14 Mar 2012 01:29:15 -0400 Subject: [Python-Dev] getting patches committed (was Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives) In-Reply-To: References: <20120313222103.163877e6@pitrou.net> Message-ID: <20120314052915.8BA282500ED@webabinitio.net> On Wed, 14 Mar 2012 06:03:10 +0200, Eli Bendersky wrote: > > Rather than indicating apathy on the party of third party developers, this > > might be a sign that core Python is unapproachable or not worth the effort. > > > > For instance I have several one line patches languishing, I can't imagine > > how disappointing it would be to have significantly larger patches ignored, > > but it happens. > > A one-line patch for a complex module or piece of code may require > much more than looking at that single line to really review. I hope > you understand that. In addition, sometimes patches just get forgotten. It's not like there are enough core devs with enough time that we are actually doing searches for open issues with patches...generally we have enough to do in our interest areas, and so stay there unless an issue is brought to our attention. So to bring an issue to our attention, you can first ping the issue with a status query, or get someone (anyone, pretty much) to do a review and post it to the issue. You can also look to see if you can figure out, either from the experts list in the devguide, or hg history, or tracker activity, who might be a reasonable person to look at the issue, and add them to the nosy list. Either of these actions will often "wake up" an issue, and if it is not one of the complex (or controversial) ones Eli alluded to, it will often then get committed. If that fails, and the patch has been on the tracker for a while, it is perfectly reasonable to ask about it here. What we really need most are *reviews*. And we need these for two reasons. First, there aren't enough active committers to keep up with the patch inflow. Reviews really help, because they usually simplify the commit review process for the committer, saving time, and making it more appealing to work on the issue. Second, it is as much (or more) from quality reviews as quality patches that we recognize people who it would be beneficial to invite to be committers. And every new committer increases the chances that new patches will actually get committed.... --David From jyasskin at gmail.com Wed Mar 14 06:42:24 2012 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Tue, 13 Mar 2012 22:42:24 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <6EAD1A31-F759-4AF7-A636-68CB21ADDB18@voidspace.org.uk> References: <6EAD1A31-F759-4AF7-A636-68CB21ADDB18@voidspace.org.uk> Message-ID: On Tue, Mar 13, 2012 at 5:03 PM, Michael Foord wrote: > > On 13 Mar 2012, at 16:57, Victor Stinner wrote: > >> Hi, >> >> I added two functions to the time module in Python 3.3: wallclock() >> and monotonic(). I'm unable to explain the difference between these >> two functions, even if I wrote them :-) wallclock() is suppose to be >> more accurate than time() but has an unspecified starting point. >> monotonic() is similar except that it is monotonic: it cannot go >> backward. monotonic() may not be available or fail whereas wallclock() >> is available/work, but I think that the two functions are redundant. >> >> I prefer to keep only monotonic() because it is not affected by system >> clock update and should help to fix issues on NTP update in functions >> implementing a timeout. >> >> What do you think? >> > > > I am in the middle of adding a feature to unittest that involves timing of individual tests. I want the highest resolution cross platform measure of wallclock time - and time.wallclock() looked ideal. If monotonic may not exist or can fail why would that be better? > Isn't the highest resolution cross platform measure of "wallclock" time spelled "time.clock()"? Its docs say "this is the function to use for benchmarking Python or timing algorithms", and it would be a shame to add and teach a new function rather than improving clock()'s definition. Jeffrey From anacrolix at gmail.com Wed Mar 14 06:49:18 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 14 Mar 2012 13:49:18 +0800 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: <20120313222103.163877e6@pitrou.net> Message-ID: Thanks for the suggestions. On Mar 14, 2012 12:03 PM, "Eli Bendersky" wrote: > > Rather than indicating apathy on the party of third party developers, > this > > might be a sign that core Python is unapproachable or not worth the > effort. > > > > For instance I have several one line patches languishing, I can't imagine > > how disappointing it would be to have significantly larger patches > ignored, > > but it happens. > > > > A one-line patch for a complex module or piece of code may require > much more than looking at that single line to really review. I hope > you understand that. > > That said, if you find any issues in the bug tracker that in your > opinion need only a few minutes of attention from a core developer, > feel free to send a note to the mentorship mailing list. People > sometimes come there asking for precisely this thing (help reviewing a > simple patch they submitted), and usually get help quickly if their > request is justified. > > Eli > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyasskin at gmail.com Wed Mar 14 07:16:28 2012 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Tue, 13 Mar 2012 23:16:28 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 6:10 PM, Nadeem Vawda wrote: > On Wed, Mar 14, 2012 at 3:03 AM, Victor Stinner > wrote: >> I suppose that most libraries and programs will have to implement a >> similar fallback. >> >> We may merge both functions with a flag to be able to disable the >> fallback. Example: >> >> ?- time.realtime(): best-effort monotonic, with a fallback >> ?- time.realtime(monotonic=True): monotonic, may raise OSError or >> NotImplementedError > > This was my suggestion - I think it's useful to have the fallback > available (since most users will want it), but at the same time we > should also cater to users who need a clock that is *guaranteed* to > be monotonic. > > As an aside, I think "monotonic" is a better name than "realtime"; > it conveys the functions purpose more clearly. Then we could call > the flag "strict". While you're bikeshedding: Some of the drafts of the new C++ standard had a monotonic_clock, which was guaranteed to only go forwards, but which could be affected by system clock updates that went forwards. Because of problems in defining timeouts using an adjustable clock, C++11 instead defines a "steady_clock", which ticks as steadily as the machine/OS/library can ensure, and is definitely not affected by any time adjustments: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3191.htm. I realize that Unix's clock_gettime(CLOCK_MONOTONIC) already includes the steadiness criterion, but the word itself doesn't actually include the meaning. From skippy.hammond at gmail.com Wed Mar 14 07:32:22 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Wed, 14 Mar 2012 17:32:22 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: <4F603B76.4050004@gmail.com> On 14/03/2012 6:43 AM, VanL wrote: > Following up on conversations at PyCon, I want to bring up one of my > personal hobby horses for change in 3.3: Fix install layout on Windows, > with a side order of making the PATH work better. > > Short version: > > 1) The layout for the python root directory for all platforms should be > as follows: > > stdlib = {base/userbase}/lib/python{py_version_short} > platstdlib = {base/userbase}/lib/python{py_version_short} > purelib = {base/userbase}/lib/python{py_version_short}/site-packages > platlib = {base/userbase}/lib/python{py_version_short}/site-packages > include = {base/userbase}/include/python{py_version_short} As per comments later in the thread, I'm -1 on including "python{py_version_short}" in the lib directories for a number of reasons; one further reason not outlined is that it would potentially make running Python directly from a built tree difficult. For the same reason, I'm also -1 on having that in the include dir. > scripts = {base/userbase}/bin We should note that this may cause pain for a number of projects - I've seen quite a few projects that already assume "Scripts" on Windows - eg, virtualenv and setuptools IIRC - and also assume the executable is where it currently lives - one example off the top of my head is the mozilla "jetpack" project - see: https://github.com/mozilla/addon-sdk/blob/master/bin/activate.bat#L117 This code (and any other code looking in "Scripts" on Windows) will fail and need to be updated with this change. Further, assuming such projects want to target multiple Python versions, it will need to keep the old code and check the new location. Another bit of code which would be impacted is the PEP397 launcher; it too would have to grow version specific logic to locate the executable. So while I'm not (yet) -1 on the general idea, I'm close. I guess I don't understand how the benefits this offers outweigh the costs to 3rd parties. Given the work on making Python more virtualenv friendly, can't we just wear the costs of the existing scheme in the stdlib and avoid breaking the code already out there? IOW, who exactly will benefit from this, and how does the cost of them supporting the existing scheme compare to the cost of the breakage to multiple 3rd parties? Mark From kristjan at ccpgames.com Wed Mar 14 07:32:29 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 06:32:29 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <6EAD1A31-F759-4AF7-A636-68CB21ADDB18@voidspace.org.uk> Message-ID: To quote: "On Unix, return the current processor time as a floating point number expressed in seconds. The precision, and in fact the very definition of the meaning of "processor time", depends on that of the C function of the same name," The problem is that it is defined to return "processor time." This is historical baggage that comes from just writing a python wrapper around the unix "clock" function. Of course, "processor time" is quite useless when one is trying to write timeout algorithms or other such things that need to time out in real time, not just cpu cycles. K -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Jeffrey Yasskin Sent: 13. mars 2012 22:42 To: Michael Foord Cc: Python Dev Subject: Re: [Python-Dev] Drop the new time.wallclock() function? Isn't the highest resolution cross platform measure of "wallclock" time spelled "time.clock()"? Its docs say "this is the function to use for benchmarking Python or timing algorithms", and it would be a shame to add and teach a new function rather than improving clock()'s definition. Jeffrey _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com From solipsis at pitrou.net Wed Mar 14 10:02:26 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 10:02:26 +0100 Subject: [Python-Dev] SocketServer issues References: Message-ID: <20120314100226.04627743@pitrou.net> On Wed, 14 Mar 2012 04:26:16 +0000 Kristj?n Valur J?nsson wrote: > Hi there. > I want to mention some issues I've had with the socketserver module, and discuss if there's a way to make it nicer. > So, for a long time we were able to create magic stackless mixin classes for > it, like ThreadingMixIn, and assuming we had the appropriate socket > replacement library, be able to use it nicely using tasklets. I don't really think the ability to "create magic stackless mixin classes" should be a driving principle for the stdlib. I would suggest using a proper non-blocking framework such as Twisted. > So, my first question is: Why not simply rely on the already built-in timeout > support in the socket module? In case you didn't notice, the built-in timeout support *also* uses select(). Regards Antoine. From solipsis at pitrou.net Wed Mar 14 10:07:14 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 10:07:14 +0100 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives References: <4F5FDE77.7030601@pearwood.info> Message-ID: <20120314100714.31f0a289@pitrou.net> On Wed, 14 Mar 2012 10:55:35 +1100 Steven D'Aprano wrote: > > What problem are we actually trying to solve here? Do we think that there are > users who really have no clue where to find 3rd party software AND don't know > how to use Google, BUT read the Python docs? I find it difficult to believe > that there are people who both read the docs and are so clueless that they > need to be told that there are alternatives available and which they should be > using. I find it quite easy to believe myself. Many people will learn some Python by reading the docs, without knowing the rest of the ecosystem. So, yes, publicizing the widely accepted alternatives (such as Twisted for asyncore) *is* helpful. (that doesn't mean any shiny new gadget must be advocated, though; third-party libraries should be mature enough before we start mentioning them) > Should we recommend wxPython over Pyjamas or PyGUI or PyGtk? On what basis? You don't have to recommend anything. Just mention them. You know what, we *already* do that job: http://docs.python.org/dev/library/othergui.html#other-gui-packages Regards Antoine. From solipsis at pitrou.net Wed Mar 14 10:16:18 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 10:16:18 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? References: Message-ID: <20120314101618.5cf1850f@pitrou.net> On Wed, 14 Mar 2012 02:03:42 +0100 Victor Stinner wrote: > > We may merge both functions with a flag to be able to disable the > fallback. Example: > > - time.realtime(): best-effort monotonic, with a fallback > - time.realtime(monotonic=True): monotonic, may raise OSError or > NotImplementedError That's a rather awful name. time.time() is *the* real time. time.monotonic(fallback=False) would be a better API. Regards Antoine. From stefan at bytereef.org Wed Mar 14 10:30:41 2012 From: stefan at bytereef.org (Stefan Krah) Date: Wed, 14 Mar 2012 10:30:41 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <20120314101618.5cf1850f@pitrou.net> References: <20120314101618.5cf1850f@pitrou.net> Message-ID: <20120314093041.GA20214@sleipnir.bytereef.org> Antoine Pitrou wrote: > time.monotonic(fallback=False) would be a better API. +1 Stefan Krah From solipsis at pitrou.net Wed Mar 14 10:39:41 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 10:39:41 +0100 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: <20120313222103.163877e6@pitrou.net> Message-ID: <20120314103941.7cd7c9e6@pitrou.net> On Wed, 14 Mar 2012 11:31:34 +0800 Matt Joiner wrote: > > Rather than indicating apathy on the party of third party developers, this > might be a sign that core Python is unapproachable or not worth the effort. > > For instance I have several one line patches languishing, I can't imagine > how disappointing it would be to have significantly larger patches ignored, > but it happens. Can you give a pointer to these one-liners? Once a patch gets a month old or older, it tends to disappear from everyone's radar unless you somehow "ping" on the tracker, or post a message to the mailing-list. (of course, you shouldn't spam the list with open issues either) Thanks Antoine. From stefan at bytereef.org Wed Mar 14 10:58:10 2012 From: stefan at bytereef.org (Stefan Krah) Date: Wed, 14 Mar 2012 10:58:10 +0100 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120314103941.7cd7c9e6@pitrou.net> References: <20120313222103.163877e6@pitrou.net> <20120314103941.7cd7c9e6@pitrou.net> Message-ID: <20120314095810.GA20325@sleipnir.bytereef.org> Antoine Pitrou wrote: > > For instance I have several one line patches languishing, I can't imagine > > how disappointing it would be to have significantly larger patches ignored, > > but it happens. > > Can you give a pointer to these one-liners? Almost a one-liner, but vast knowledge required (how do you prove that using (freefunc) is safe if it's the first usage in the tree?). http://bugs.python.org/file21610/atexit-leak.patch I think there are many issues like that one where the implications of a short patch can only be assessed by small number of committers. Stefan Krah From solipsis at pitrou.net Wed Mar 14 11:00:44 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 11:00:44 +0100 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives References: <20120313222103.163877e6@pitrou.net> <20120314103941.7cd7c9e6@pitrou.net> <20120314095810.GA20325@sleipnir.bytereef.org> Message-ID: <20120314110044.08971b5d@pitrou.net> On Wed, 14 Mar 2012 10:58:10 +0100 Stefan Krah wrote: > Antoine Pitrou wrote: > > > For instance I have several one line patches languishing, I can't imagine > > > how disappointing it would be to have significantly larger patches ignored, > > > but it happens. > > > > Can you give a pointer to these one-liners? > > Almost a one-liner, but vast knowledge required (how do you prove that > using (freefunc) is safe if it's the first usage in the tree?). > > http://bugs.python.org/file21610/atexit-leak.patch Well, can you please post a URL to the issue itself? Thanks Antoine. From mark at hotpy.org Wed Mar 14 11:05:11 2012 From: mark at hotpy.org (Mark Shannon) Date: Wed, 14 Mar 2012 10:05:11 +0000 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120314095810.GA20325@sleipnir.bytereef.org> References: <20120313222103.163877e6@pitrou.net> <20120314103941.7cd7c9e6@pitrou.net> <20120314095810.GA20325@sleipnir.bytereef.org> Message-ID: <4F606D57.80304@hotpy.org> Stefan Krah wrote: > Antoine Pitrou wrote: >>> For instance I have several one line patches languishing, I can't imagine >>> how disappointing it would be to have significantly larger patches ignored, >>> but it happens. >> Can you give a pointer to these one-liners? > > Almost a one-liner, but vast knowledge required (how do you prove that > using (freefunc) is safe if it's the first usage in the tree?). > > http://bugs.python.org/file21610/atexit-leak.patch > But how do you find issues? I want to do some reviews, but I don't want to wade through issues on components I know little or nothing about in order to find the ones I can review. There does not seem to be a way to filter search results in the tracker. Cheers, Mark From stefan at bytereef.org Wed Mar 14 11:20:53 2012 From: stefan at bytereef.org (Stefan Krah) Date: Wed, 14 Mar 2012 11:20:53 +0100 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120314110044.08971b5d@pitrou.net> References: <20120313222103.163877e6@pitrou.net> <20120314103941.7cd7c9e6@pitrou.net> <20120314095810.GA20325@sleipnir.bytereef.org> <20120314110044.08971b5d@pitrou.net> Message-ID: <20120314102053.GA20430@sleipnir.bytereef.org> Antoine Pitrou wrote: > > Almost a one-liner, but vast knowledge required (how do you prove that > > using (freefunc) is safe if it's the first usage in the tree?). > > > > http://bugs.python.org/file21610/atexit-leak.patch > > Well, can you please post a URL to the issue itself? That sounds like an excellent plan. :) http://bugs.python.org/issue11826 Stefan Krah From facundobatista at gmail.com Wed Mar 14 12:21:28 2012 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 14 Mar 2012 08:21:28 -0300 Subject: [Python-Dev] PEP 8 misnaming Message-ID: Hello! In the "Maximum Line Length" section of PEP 8 it says: "The preferred place to break around a binary operator is *after* the operator, not before it." And after that is an example (trimmed here): if (width == 0 and height == 0 and color == 'red' and emphasis == 'strong' or highlight > 100): raise ValueError("sorry, you lose") In the example the line is broken after the 'and' or 'or' *keywords*, not after the '==' *operator* (which is the nice way of doing it). Maybe the sentence above is misleading? Thanks! -- .? ? Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From geoffspear at gmail.com Wed Mar 14 12:30:31 2012 From: geoffspear at gmail.com (Geoffrey Spear) Date: Wed, 14 Mar 2012 07:30:31 -0400 Subject: [Python-Dev] PEP 8 misnaming In-Reply-To: References: Message-ID: On Wed, Mar 14, 2012 at 7:21 AM, Facundo Batista wrote: > Hello! > > In the "Maximum Line Length" section of PEP 8 it says: > > ? ?"The preferred place to break around a binary operator is *after* > the operator, not before it." > > And after that is an example (trimmed here): > > ? ? ? ? ? ?if (width == 0 and height == 0 and > ? ? ? ? ? ? ? ?color == 'red' and emphasis == 'strong' or > ? ? ? ? ? ? ? ?highlight > 100): > ? ? ? ? ? ? ? ?raise ValueError("sorry, you lose") > > In the example the line is broken after the 'and' or 'or' *keywords*, > not after the '==' *operator* (which is the nice way of doing it). > > Maybe the sentence above is misleading? 'and' and 'or' are both binary logical operators. The fact that they are keywords is irrelevant; the sentence isn't misleading. From victor.stinner at gmail.com Wed Mar 14 13:27:19 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 14 Mar 2012 13:27:19 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <20120314101618.5cf1850f@pitrou.net> References: <20120314101618.5cf1850f@pitrou.net> Message-ID: 2012/3/14 Antoine Pitrou : > On Wed, 14 Mar 2012 02:03:42 +0100 > Victor Stinner wrote: >> >> We may merge both functions with a flag to be able to disable the >> fallback. Example: >> >> ?- time.realtime(): best-effort monotonic, with a fallback >> ?- time.realtime(monotonic=True): monotonic, may raise OSError or >> NotImplementedError > > That's a rather awful name. ?time.time() is *the* real time. > > time.monotonic(fallback=False) would be a better API. I would prefer to enable the fallback by default with a warning in the doc, just because it is more convinient and it is what user want even if they don't know that they need a fallback :-) Enabling the fallback by default allow to write such simple code: try: from time import monotonic as get_time except ImportError: # Python < 3.3 from time import time as get_time Use time.monotonic(strict=True) if you need a truly monotonic clock. monotonic() may not be the best name in this case. Jeffrey Yasskin proposed time.steady_clock(), so time.steady_clock(monotonic=False)? Victor From solipsis at pitrou.net Wed Mar 14 13:27:47 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 13:27:47 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> Message-ID: <20120314132747.6c857a53@pitrou.net> On Wed, 14 Mar 2012 13:27:19 +0100 Victor Stinner wrote: > > monotonic() may not be the best name in this case. Jeffrey Yasskin > proposed time.steady_clock(), so time.steady_clock(monotonic=False)? I don't know what "steady" is supposed to mean here, so perhaps the best solution is to improve the doc? Also, "monotonic=False" implies that it won't be monotonic, which is false. Regards Antoine. From ben+python at benfinney.id.au Wed Mar 14 14:03:10 2012 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 15 Mar 2012 00:03:10 +1100 Subject: [Python-Dev] PEP 8 misnaming References: Message-ID: <877gyngws1.fsf@benfinney.id.au> Facundo Batista writes: > if (width == 0 and height == 0 and > color == 'red' and emphasis == 'strong' or > highlight > 100): > raise ValueError("sorry, you lose") > > In the example the line is broken after the 'and' or 'or' *keywords*, ?and? and ?or? are binary operators (that also happen to be keywords). The description is accurate and IMO not misleading. > not after the '==' *operator* (which is the nice way of doing it). ?1. The lower-priority binding operator is the better place to break the line. The binary logical operators bind at lower priority than the equality operator. -- \ ?If you do not trust the source do not use this program.? | `\ ?Microsoft Vista security dialogue | _o__) | Ben Finney From brian at python.org Wed Mar 14 15:12:32 2012 From: brian at python.org (Brian Curtin) Date: Wed, 14 Mar 2012 09:12:32 -0500 Subject: [Python-Dev] 2012 Language Summit Report Message-ID: As with last year, I've put together a summary of the Python Language Summit which took place last week at PyCon 2012. This was compiled from my notes as well as those of Eric Snow and Senthil Kumaran, and I think we got decent coverage of what was said throughout the day. http://blog.python.org/2012/03/2012-language-summit-report.html If you have questions or comments about discussions which occurred there, please create a new thread for your topic. Feel free to contact me directly if I've left anything out or misprinted anything. From jimjjewett at gmail.com Wed Mar 14 15:53:48 2012 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Wed, 14 Mar 2012 07:53:48 -0700 (PDT) Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: Message-ID: <4f60b0fc.2ab2320a.24c8.ffffa0c3@mx.google.com> In view-source:http://mail.python.org/pipermail/python-dev/2012-March/117586.html van.lindberg at gmail.com posted: >>> 1) The layout for the python root directory for all platforms should be >>> as follows: >>> stdlib = {base/userbase}/lib/python{py_version_short} >>> platstdlib = {base/userbase}/lib/python{py_version_short} >>> purelib = {base/userbase}/lib/python{py_version_short}/site-packages >>> platlib = {base/userbase}/lib/python{py_version_short}/site-packages >>> include = {base/userbase}/include/python{py_version_short} >>> scripts = {base/userbase}/bin >>> data = {base/userbase} Why? Pure python vs compiled C doesn't need to be separated at the directory level, except for cleanliness. Some (generally unix) systems prefer to split the libraries into several additional pieces depending on CPU architecture. The structure listed above doesn't have a location for docs. Some packages (such as tcl) may be better off in their own area. What is "data"? Is this an extra split compared to today, or does it refer to things like LICENSE.txt, README.txt, and NEWS.txt? And even once I figure out where files have moved, and assume that the split is perfect -- what does this buy me over the current situation? I was under the impression that programs like distutils already handled finding the appropriate directories for a program; if you're rewriting that logic, you're just asking for bugs on a strange platform that you don't use. If you're looking for things interactively, then platform conventions are probably more important than consistency across platforms. If you disagree, you are welcome to reorganize your personal linux installation so that it matches windows, and see whether it causes you any problems. > ... We *already* have this. The only difference in this proposal is > that we go from py_version_nodot to py_version_short, i.e. from > c:\python33\lib\python33 > to > c:\python33\lib\python3.3 I have not seen that redundancy before on windows. I'm pretty sure that it is a relic of your Linux provider wanting to support multiple python versions using shared filesystems. The Windows standard is to use a local disk, and to bundle it all up into its own directory, similar to the way that java apps sometimes ship with their own JVM. Also note that using the dot in a directory name is incautious. I haven't personally had trouble in several years, but doing so is odd enough that some should be expected. Python already causes some grief by not installing in "Program Files", but that is at least justified by the "spaces in filenames" problem; what is the advantange of 3.3? I'm using windows, and I just followed the defaults at installation. It is possible that the installer continued to do something based on an earlier installation, but I don't think this machine has ever had a customized installation of any python version. C:\python32\* Everything is under here; I assume {base/userbase} would be set to C:\python32 As is customary for windows, the base directory contains the license/readme/news and all executables that the user is expected to use directly. (python.exe, pythonw.exe. It also contains w9xpopen.exe that users do not use, but that too is fairly common.) There is no data directory. Subdirectories are: C:\python32\DLLs In additional to regular DLL files, it contains .pyd files and icons. It looks like modules from the stdlib that happen to be written in C. Most users will never bother to look here. C:\python32\Doc A .chm file; full html would be fine too, but removing it would be a bad idea. C:\python32\include These are the header files, though most users will never have any use for them, as there isn't generally a compiler. C:\python32\Lib The standard library -- or at least the portion implemented in python. Note that site-packages is a subdirectory here. It doesn't happen to have an __init__.py, but to an ordinary user it looks just like any other stdlib package, such as xml or multiprocessing. I personally happen to keep things in subdirectories of site-packages, but I can't say what is standard. Moving site-packages out of the Lib directory might make sense, but probably isn't worth the backward compatibility hit. C:\python32\libs .lib files. I'm not entirely sure what these (as opposed to the DLLs) are for; lib files aren't that common on windows. My machine does not appear to have any that aren't associated with cross-platform tools or unix emulation. C:\python32\tcl Note that this is in addition to associated files under DLLs and libs. I would prefer to see them in one place, but moving it in with non-tcl stuff would not be an improvement. Most users will never look (or care); those that do usually appreciate knowing that, for example, the dde subdirectory is for tcl. C:\python32\Tools This has three subdirectories (i18n, pynche, Scripts). Moving the .py files in with the binary just because you could execute them using file associations would be a step backwards; you can do the same regardless of where they are. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From van.lindberg at gmail.com Wed Mar 14 16:22:16 2012 From: van.lindberg at gmail.com (VanL) Date: Wed, 14 Mar 2012 10:22:16 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F5FF662.4070201@v.loewis.de> Message-ID: On 3/13/2012 9:58 PM, Terry Reedy wrote: >> Given that we already repeat it, isn't it better to be consistent? > > But there is no repetition currently on Windows installations. > I though you were just proposing to switch lib (lower-cased, and scripts > renamed as bin, and pythonxx). So I do not think I yet understand what > the proposal is and how it would be different from what I have now. Aaah, I was looking at my local installations, which happen to be "nt-user". Looking at the system installation ("nt") I see that there is no repetition. I am fine with keeping the distinction between base installs (no py_version) and user installs (including a py_version). I would just suggest that when you have a py_version, it be the same py_version (not dots sometimes, nodot other times). It also begs the question as to whether the py_version is *ever* needed. Thanks, Van From guido at python.org Wed Mar 14 16:27:08 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Mar 2012 08:27:08 -0700 Subject: [Python-Dev] SocketServer issues In-Reply-To: <20120314100226.04627743@pitrou.net> References: <20120314100226.04627743@pitrou.net> Message-ID: Hopefully it doesn't use select if no timeout is set... --Guido van Rossum (sent from Android phone) On Mar 14, 2012 2:08 AM, "Antoine Pitrou" wrote: > On Wed, 14 Mar 2012 04:26:16 +0000 > Kristj?n Valur J?nsson wrote: > > Hi there. > > I want to mention some issues I've had with the socketserver module, and > discuss if there's a way to make it nicer. > > So, for a long time we were able to create magic stackless mixin classes > for > > it, like ThreadingMixIn, and assuming we had the appropriate socket > > replacement library, be able to use it nicely using tasklets. > > I don't really think the ability to "create magic stackless mixin > classes" should be a driving principle for the stdlib. > I would suggest using a proper non-blocking framework such as Twisted. > > > So, my first question is: Why not simply rely on the already built-in > timeout > > support in the socket module? > > In case you didn't notice, the built-in timeout support *also* uses > select(). > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott+python-dev at scottdial.com Wed Mar 14 16:09:48 2012 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 14 Mar 2012 11:09:48 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F5FF662.4070201@v.loewis.de> Message-ID: <4F60B4BC.9040702@scottdial.com> On 3/13/2012 9:57 PM, VanL wrote: > On Mar 13, 2012, at 8:37 PM, "Martin v. L?wis" wrote: >> The installation will end up in >> >> c:\python33\lib\python3.3 >> >> which has the software name and version twice in the path. >> >> Do we *really* need this? > > We *already* have this. The only difference in this proposal is that we go from py_version_nodot to py_version_short, i.e. from > > c:\python33\lib\python33 > > to > > c:\python33\lib\python3.3 > > Given that we already repeat it, isn't it better to be consistent? > Is it? I think you are confusing two different configuration sections in sysconfig.cfg: [nt] stdlib = {base}/Lib platstdlib = {base}/Lib purelib = {base}/Lib/site-packages platlib = {base}/Lib/site-packages include = {base}/Include platinclude = {base}/Include scripts = {base}/Scripts data = {base} [nt_user] stdlib = {userbase}/Python{py_version_nodot} platstdlib = {userbase}/Python{py_version_nodot} purelib = {userbase}/Python{py_version_nodot}/site-packages platlib = {userbase}/Python{py_version_nodot}/site-packages include = {userbase}/Python{py_version_nodot}/Include scripts = {userbase}/Scripts data = {userbase} -- Scott Dial scott at scottdial.com From van.lindberg at gmail.com Wed Mar 14 16:48:58 2012 From: van.lindberg at gmail.com (VanL) Date: Wed, 14 Mar 2012 10:48:58 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4f60b0fc.2ab2320a.24c8.ffffa0c3@mx.google.com> References: <4f60b0fc.2ab2320a.24c8.ffffa0c3@mx.google.com> Message-ID: On 3/14/2012 9:53 AM, Jim J. Jewett wrote: > In view-source:http://mail.python.org/pipermail/python-dev/2012-March/117586.html > van.lindberg at gmail.com posted: > >>>> 1) The layout for the python root directory for all platforms should be >>>> as follows: > >>>> stdlib = {base/userbase}/lib/python{py_version_short} >>>> platstdlib = {base/userbase}/lib/python{py_version_short} >>>> purelib = {base/userbase}/lib/python{py_version_short}/site-packages >>>> platlib = {base/userbase}/lib/python{py_version_short}/site-packages >>>> include = {base/userbase}/include/python{py_version_short} >>>> scripts = {base/userbase}/bin >>>> data = {base/userbase} > > Why? > > Pure python vs compiled C doesn't need to be separated at the directory > level, except for cleanliness. I am deliberately being cautious here. I actually agree with you. I am only suggesting we maintain all of these different distinctions because that is what we have already. You can see what we have currently at http://hg.python.org/distutils2/file/2cec52b682a9/distutils2/_backport/sysconfig.cfg I am *not* suggesting that docs, etc change at all - that is included in a different part of the configuration and is not modified by what I propose here (lines 1-26). As noted earlier in the thread, I also change my proposal to maintain the existing differences between system installs and user installs. Thus, the only place I am proposing changing are the values for the keys listed above. Specifically, this (lines 57-65 in the file above): [nt] stdlib = {base}/Lib platstdlib = {base}/Lib purelib = {base}/Lib/site-packages platlib = {base}/Lib/site-packages include = {base}/Include platinclude = {base}/Include scripts = {base}/Scripts data = {base} Would become this: [nt] stdlib = {base}/lib platstdlib = {base}/lib purelib = {base}/lib/site-packages platlib = {base}/lib/site-packages include = {base}/include platinclude = {base}/include scripts = {base}/bin data = {base} and this (lines 86-93): [nt_user] stdlib = {userbase}/Python{py_version_nodot} platstdlib = {userbase}/Python{py_version_nodot} purelib = {userbase}/Python{py_version_nodot}/site-packages platlib = {userbase}/Python{py_version_nodot}/site-packages include = {userbase}/Python{py_version_nodot}/Include scripts = {userbase}/Scripts data = {userbase} would become this: [nt_user] stdlib = {userbase}/python{py_version_short} platstdlib = {userbase}/python{py_version_short} purelib = {userbase}/python{py_version_nodot}/site-packages platlib = {userbase}/python{py_version_nodot}/site-packages include = {userbase}/python{py_version_nodot}/include scripts = {userbase}/bin data = {userbase} > ... if you're rewriting that logic, you're just asking for bugs on a > strange platform that you don't use. I am not rewriting the logic - the logic is driven by these configuration values. And this is a platform I use, and that is why this drives me crazy! > Subdirectories are: You forgot one: C:\python32\Scripts Would change to C:\python32\bin. The python binary and scripts meant to be run direction (easy_install, etc) would all go in this directory. > C:\python32\DLLs Would not change. > C:\python32\Doc Would not change. > C:\python32\include Would be specified as lower case only - but otherwise would not change. > C:\python32\Lib Would be specified as lower case only - but otherwise would not change. > C:\python32\libs Would not change. > C:\python32\tcl Would not change. > C:\python32\Tools This proposal does not change this, although I do think that this could be eliminated or made into "examples". Thanks, Van From van.lindberg at gmail.com Wed Mar 14 16:51:31 2012 From: van.lindberg at gmail.com (VanL) Date: Wed, 14 Mar 2012 10:51:31 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F60B4BC.9040702@scottdial.com> References: <4F5FF662.4070201@v.loewis.de> <4F60B4BC.9040702@scottdial.com> Message-ID: <4F60BE83.3050102@gmail.com> On 3/14/2012 10:09 AM, Scott Dial wrote: > I think you are confusing two different configuration sections in > sysconfig.cfg: > > [nt] > stdlib = {base}/Lib > platstdlib = {base}/Lib > purelib = {base}/Lib/site-packages > platlib = {base}/Lib/site-packages > include = {base}/Include > platinclude = {base}/Include > scripts = {base}/Scripts > data = {base} > > [nt_user] > stdlib = {userbase}/Python{py_version_nodot} > platstdlib = {userbase}/Python{py_version_nodot} > purelib = {userbase}/Python{py_version_nodot}/site-packages > platlib = {userbase}/Python{py_version_nodot}/site-packages > include = {userbase}/Python{py_version_nodot}/Include > scripts = {userbase}/Scripts > data = {userbase} I was lumping them together, yes, but now note that I modify the proposal to maintain this distinction. These would change to: [nt] stdlib = {base}/lib platstdlib = {base}/lib purelib = {base}/lib/site-packages platlib = {base}/lib/site-packages include = {base}/include platinclude = {base}/include scripts = {base}/bin data = {base} [nt_user] stdlib = {userbase}/python{py_version_short} platstdlib = {userbase}/python{py_version_short} purelib = {userbase}/python{py_version_short}/site-packages platlib = {userbase}/python{py_version_short}/site-packages include = {userbase}/python{py_version_short}/include scripts = {userbase}/bin data = {userbase} From tjreedy at udel.edu Wed Mar 14 16:56:22 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2012 11:56:22 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F5FF662.4070201@v.loewis.de> Message-ID: On 3/14/2012 11:22 AM, VanL wrote: > On 3/13/2012 9:58 PM, Terry Reedy wrote: > >>> Given that we already repeat it, isn't it better to be consistent? >> >> But there is no repetition currently on Windows installations. >> I though you were just proposing to switch lib (lower-cased, and scripts >> renamed as bin, and pythonxx). So I do not think I yet understand what >> the proposal is and how it would be different from what I have now. > > Aaah, I was looking at my local installations, which happen to be > "nt-user". Looking at the system installation ("nt") I see that there is > no repetition. Are you talking about 'install for all users' versus 'install for this user only'? I have always done the former as I see no point to the latter on my machine, even if another family member has an account. > I am fine with keeping the distinction between > base installs (no py_version) I have no idea what this means. As far as I can remember, each installation of Python x.y (back to 1.3 for me, on DOS) has gone into a pythonxy (no dot) directory, with subdirectories much as Jim J. described. > and user installs (including a py_version). I would just > suggest that when you have a py_version, it be the same py_version (not > dots sometimes, nodot other times). > > It also begs the question as to whether the py_version is *ever* needed. Whenever multiple versions are installed, of course a version marker is needed. Even if not, it is helpful to be able to see what version is installed. But I probably am not understanding what you mean. -- Terry Jan Reedy From tjreedy at udel.edu Wed Mar 14 16:57:10 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2012 11:57:10 -0400 Subject: [Python-Dev] 2012 Language Summit Report In-Reply-To: References: Message-ID: On 3/14/2012 10:12 AM, Brian Curtin wrote: > As with last year, I've put together a summary of the Python Language > Summit which took place last week at PyCon 2012. This was compiled > from my notes as well as those of Eric Snow and Senthil Kumaran, and I > think we got decent coverage of what was said throughout the day. > > http://blog.python.org/2012/03/2012-language-summit-report.html Nicely done. Thank you. -- Terry Jan Reedy From rdmurray at bitdance.com Wed Mar 14 17:00:07 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 14 Mar 2012 12:00:07 -0400 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <4F606D57.80304@hotpy.org> References: <20120313222103.163877e6@pitrou.net> <20120314103941.7cd7c9e6@pitrou.net> <20120314095810.GA20325@sleipnir.bytereef.org> <4F606D57.80304@hotpy.org> Message-ID: <20120314160007.B786D250117@webabinitio.net> On Wed, 14 Mar 2012 10:05:11 -0000, Mark Shannon wrote: > But how do you find issues? > > I want to do some reviews, but I don't want to wade through issues on > components I know little or nothing about in order to find the ones I > can review. > > There does not seem to be a way to filter search results in the tracker. Is the advanced search ('search' link on left hand side) missing some filtering capabilities? --David From van.lindberg at gmail.com Wed Mar 14 17:03:53 2012 From: van.lindberg at gmail.com (VanL) Date: Wed, 14 Mar 2012 11:03:53 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F603B76.4050004@gmail.com> References: <4F603B76.4050004@gmail.com> Message-ID: <4F60C169.9030404@gmail.com> On 3/14/2012 1:32 AM, Mark Hammond wrote: > As per comments later in the thread, I'm -1 on including > "python{py_version_short}" in the lib directories for a number of > reasons; one further reason not outlined is that it would potentially > make running Python directly from a built tree difficult. For the same > reason, I'm also -1 on having that in the include dir. A built tree would look the same as always - the directories would only be moved (if at all) during installation. Thus, you will still be able to run python directly from a built installation. Also note that the py_version_short will not be in platform installs. >> scripts = {base/userbase}/bin > > We should note that this may cause pain for a number of projects - I've > seen quite a few projects that already assume "Scripts" on Windows - eg, > virtualenv and setuptools IIRC If you look at these projects, though, they *special case* Windows to account for the different layout. Removing this difference will allow these projects to remove their special-casing code. >- and also assume the executable is where > it currently lives - one example off the top of my head is the mozilla > "jetpack" project - see: > > https://github.com/mozilla/addon-sdk/blob/master/bin/activate.bat#L117 This code actually reinforces my point: First, this code would actually still work. The section ":FoundPython" sets the PATH to "%VIRTUAL_ENV%\bin;%PYTHONINSTALL%;%PATH%" (L80), which would still allow python.exe to be found and run. Second, look at that line again. Mozilla actually has edited this code so that the jetpack uses a cross-platform "bin" convention, just as I am proposing. Third, one element of this proposal is that there would be a key placed in the registry that points directly to the location of the python executable, making locating it trivial to locate programmatically on Windows. Thanks, Van From van.lindberg at gmail.com Wed Mar 14 17:10:05 2012 From: van.lindberg at gmail.com (VanL) Date: Wed, 14 Mar 2012 11:10:05 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F5FF662.4070201@v.loewis.de> Message-ID: On 3/14/2012 10:56 AM, Terry Reedy wrote: > Are you talking about 'install for all users' versus 'install for this > user only'? I have always done the former as I see no point to the > latter on my machine, even if another family member has an account. Yes, but some people are on corporate machines that only allow "install for this user" installations. >> I am fine with keeping the distinction between > > base installs (no py_version) > > I have no idea what this means. As far as I can remember, each > installation of Python x.y (back to 1.3 for me, on DOS) has gone into a > pythonxy (no dot) directory, with subdirectories much as Jim J. described. I am referring to the currently-existing install schemes 'nt' ('install for all users') and 'nt-user' ('install for this user only'). The *current* layouts are described at http://hg.python.org/distutils2/file/2cec52b682a9/distutils2/_backport/sysconfig.cfg: L57-65: [nt] stdlib = {base}/Lib platstdlib = {base}/Lib purelib = {base}/Lib/site-packages platlib = {base}/Lib/site-packages include = {base}/Include platinclude = {base}/Include scripts = {base}/Scripts data = {base} L86-93: [nt_user] stdlib = {userbase}/Python{py_version_nodot} platstdlib = {userbase}/Python{py_version_nodot} purelib = {userbase}/Python{py_version_nodot}/site-packages platlib = {userbase}/Python{py_version_nodot}/site-packages include = {userbase}/Python{py_version_nodot}/Include scripts = {userbase}/Scripts data = {userbase} I am proposing that these change to: [nt] stdlib = {base}/lib platstdlib = {base}/lib purelib = {base}/lib/site-packages platlib = {base}/lib/site-packages include = {base}/include platinclude = {base}/include scripts = {base}/bin data = {base} [nt_user] stdlib = {userbase}/python{py_version_short} platstdlib = {userbase}/python{py_version_short} purelib = {userbase}/python{py_version_short}/site-packages platlib = {userbase}/python{py_version_short}/site-packages include = {userbase}/python{py_version_short}/include scripts = {userbase}/bin data = {userbase} All the other diuectories that Jim talked about would not be affected by this proposal. Does this make it clearer? Thanks, Van From anacrolix at gmail.com Wed Mar 14 17:11:56 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 15 Mar 2012 00:11:56 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <20120314132747.6c857a53@pitrou.net> References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: I have some observations regarding this: Victor's existing time.monotonic and time.wallclock make use of QueryPerformanceCounter, and CLOCK_MONOTONIC_RAW as possible. Both of these are hardware-based counters, their monotonicity is just a convenient property of the timer sources. Furthermore, time values can actually vary depending on the processor the call is served on. time.hardware()? time.monotonic_raw()? There are bug reports on Linux that CLOCK_MONOTONIC isn't always monotonic. This is why CLOCK_MONOTONIC_RAW was created. There's also the issue of time leaps (forward), which also isn't a problem with the latter form. time.monotonic(raw_only=False)? The real value of "monotonic" timers is that they don't leap backwards, and preferably don't leap forwards. Whether they are absolute is of no consequence. I would suggest that the API reflect this, and that more specific time values be obtained using the proper raw syscall wrapper (like time.clock_gettime) if required. time.relative(strict=False)? The ultimate use of the function name being established is for use in timeouts and relative timings. Where an option is present, it disallows fallbacks like CLOCK_MONOTONIC and other weaker forms: * time.hardware(fallback=True) -> reflects the source of the timing impeccably. alerts users to possible affinity issues * time.monotonic_raw() -> a bit linux specific... * time.relative(strict=False) -> matches the use case. a warning could be put regarding hardware timings * time.monotonic(raw_only=False) -> closest to the existing implementation. the keyword name i think is better. From julien at tayon.net Wed Mar 14 17:16:50 2012 From: julien at tayon.net (julien tayon) Date: Wed, 14 Mar 2012 17:16:50 +0100 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: Message-ID: Hello, 2012/3/13 Guido van Rossum : > On Mon, Mar 12, 2012 at 9:23 PM, Brian Curtin wrote: >> Downloads don't mean the code is good. Voting is gamed. I really don't >> think there's a good automated solution to tell us what the >> high-quality replacement projects are. > > Sure, these are imperfect metrics. But not having any metrics at all > is flawed too. Despite the huge flamewar we had 1-2 years ago about > PyPI comments, I think we should follow the lead of the many app > stores that pop up on the web -- users will recognize the pattern and > will tune their skepticism sensors as needed. > unittest and functional testing are obctive metrics. It it passes a unittest, and it has the same API, therefore it is a legitimate replacement for the stdlib. If a benchmark is also given that can be considered **not biased** and faster it is pretty neat. (but why not contribute to stdlib then?) Functional testing is however a little more tricky, subjective and interesting. Since stdlib replacements are mostl functionnaly equivalent (like requests) of one or more stdlib module and that is what people are searching for. People willing to be considered compliant with some functionalities of a stdlib would have to give example of porting from libA to libB plus the given *functionnal tests*. An interesting point may also be PEP compliance. (it is sometimes a tedious tasks when playing with SA to know if a python package of a DBDriver is DB-API 2.0 compliant). This would make pypi even greater if package maintainers added these metadata (implements, functions_like, pep_compliance) in their setup.py given they comply with the logic. And it would pretty much automate the search for alternative to stdlib. The huge problem is how to trust that maintainers are self disciplined enough, willing, and have enough knowledge to tag their packages properly, plus what is the extra strain on code and infrastructure to automate this ? Without these informations we may become like senior java developpers whose greatests skills are not coding, but knowing in a wide ecosystems of packages which one are revelant/reliable/compatible/stable. (needle in a haystack) Maybe the answer is not one of code but one of trend setting and Noise Signal Ratio on python hubs. (http://www.pythonmeme.com/, http://planet.python.org/, http://pypi.org (and still in a lesser way of classification)). Cheers, -- Julien Tayon From pje at telecommunity.com Wed Mar 14 17:17:06 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 14 Mar 2012 12:17:06 -0400 Subject: [Python-Dev] SocketServer issues In-Reply-To: <20120314100226.04627743@pitrou.net> References: <20120314100226.04627743@pitrou.net> Message-ID: On Wed, Mar 14, 2012 at 5:02 AM, Antoine Pitrou wrote: > On Wed, 14 Mar 2012 04:26:16 +0000 > Kristj?n Valur J?nsson wrote: > > Hi there. > > I want to mention some issues I've had with the socketserver module, and > discuss if there's a way to make it nicer. > > So, for a long time we were able to create magic stackless mixin classes > for > > it, like ThreadingMixIn, and assuming we had the appropriate socket > > replacement library, be able to use it nicely using tasklets. > > I don't really think the ability to "create magic stackless mixin > classes" should be a driving principle for the stdlib. > But not needlessly duplicating functionality already elsewhere in the stdlib probably ought to be. ;-) > So, my first question is: Why not simply rely on the already built-in > timeout > > support in the socket module? > > In case you didn't notice, the built-in timeout support *also* uses > select(). > That's not really the point; the frameworks that implement nonblocking I/O by replacing the socket module (and Stackless is only one of many) won't be using that code. If SocketServer uses only the socket module's API, then those frameworks will be told about the timeout via the socket API, and can then implement it their own way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From maker.py at gmail.com Wed Mar 14 17:22:23 2012 From: maker.py at gmail.com (=?UTF-8?Q?Michele_Orr=C3=B9?=) Date: Wed, 14 Mar 2012 17:22:23 +0100 Subject: [Python-Dev] [Issue1531415] Using PyErr_WarnEx on parsetok Message-ID: As pointed by Sean Reifschneider in issue 1531415, I'm writing this mail mainly to ask for advices concerning python's makefile. Currently, Parser/parsetok.c writes directly to stderr in case no more memory is avaible. So, it would be nice? to use, instead of a raw printf, the functions provided by Python/_warnings.c (PyErr_NoMemory and/or PyErr_WarnEx). This, right now, leads to a circular dependency, as described here: http://bugs.python.org/msg154939 . So far I've seen some functions present both in Python/ and pgenmain.c : PyErr_Occurred(), Py_FatalError(const char *msg), Py_Exit(int). This means a dirty alternative could be to implement another function PyErr_WarnEx; but probably there is a better way to organize the makefile, becouse currently I'm using the entire $(PYTHON_OBJS) (seems needed by warnigs.o). This is the first time I run into python c code, so please be patient :) -- ? From guido at python.org Wed Mar 14 17:22:36 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Mar 2012 09:22:36 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: I have a totally different observation. Presumably the primary use case for these timers is to measure real time intervals for the purpose of profiling various operations. For this purpose we want them to be as "steady" as possible: tick at a constant rate, don't jump forward or backward. (And they shouldn't invoke the concept of "CPU" time -- we already have time.clock() for that, besides it's often wrong, e.g. you might be measuring some sort of I/O performance.) If this means that a second as measured by time.time() is sometimes not the same as a second measured by this timer (due to time.time() occasionally jumping due to clock adjustments), that's fine with me. If this means it's unreliable inside a VM, well, it seems that's a property of the underlying OS timer, and there's not much we can do about it except letting a knowledgeable user override the timer user. As for names, I like Jeffrey's idea of having "steady" in the name. --Guido On Wed, Mar 14, 2012 at 9:11 AM, Matt Joiner wrote: > I have some observations regarding this: > > Victor's existing time.monotonic and time.wallclock make use of > QueryPerformanceCounter, and CLOCK_MONOTONIC_RAW as possible. Both of > these are hardware-based counters, their monotonicity is just a > convenient property of the timer sources. Furthermore, time values can > actually vary depending on the processor the call is served on. > time.hardware()? time.monotonic_raw()? > > There are bug reports on Linux that CLOCK_MONOTONIC isn't always > monotonic. This is why CLOCK_MONOTONIC_RAW was created. There's also > the issue of time leaps (forward), which also isn't a problem with the > latter form. time.monotonic(raw_only=False)? > > The real value of "monotonic" timers is that they don't leap > backwards, and preferably don't leap forwards. Whether they are > absolute is of no consequence. I would suggest that the API reflect > this, and that more specific time values be obtained using the proper > raw syscall wrapper (like time.clock_gettime) if required. > time.relative(strict=False)? > > The ultimate use of the function name being established is for use in > timeouts and relative timings. > > Where an option is present, it disallows fallbacks like > CLOCK_MONOTONIC and other weaker forms: > ?* time.hardware(fallback=True) -> reflects the source of the timing > impeccably. alerts users to possible affinity issues > ?* time.monotonic_raw() -> a bit linux specific... > ?* time.relative(strict=False) -> matches the use case. a warning > could be put regarding hardware timings > ?* time.monotonic(raw_only=False) -> closest to the existing > implementation. the keyword name i think is better. -- --Guido van Rossum (python.org/~guido) From Van.Lindberg at haynesboone.com Wed Mar 14 16:51:32 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Wed, 14 Mar 2012 15:51:32 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F60B4BC.9040702@scottdial.com> References: <4F5FF662.4070201@v.loewis.de> <4F60B4BC.9040702@scottdial.com> Message-ID: <4F60BE83.3050102@gmail.com> On 3/14/2012 10:09 AM, Scott Dial wrote: > I think you are confusing two different configuration sections in > sysconfig.cfg: > > [nt] > stdlib = {base}/Lib > platstdlib = {base}/Lib > purelib = {base}/Lib/site-packages > platlib = {base}/Lib/site-packages > include = {base}/Include > platinclude = {base}/Include > scripts = {base}/Scripts > data = {base} > > [nt_user] > stdlib = {userbase}/Python{py_version_nodot} > platstdlib = {userbase}/Python{py_version_nodot} > purelib = {userbase}/Python{py_version_nodot}/site-packages > platlib = {userbase}/Python{py_version_nodot}/site-packages > include = {userbase}/Python{py_version_nodot}/Include > scripts = {userbase}/Scripts > data = {userbase} I was lumping them together, yes, but now note that I modify the proposal to maintain this distinction. These would change to: [nt] stdlib = {base}/lib platstdlib = {base}/lib purelib = {base}/lib/site-packages platlib = {base}/lib/site-packages include = {base}/include platinclude = {base}/include scripts = {base}/bin data = {base} [nt_user] stdlib = {userbase}/python{py_version_short} platstdlib = {userbase}/python{py_version_short} purelib = {userbase}/python{py_version_short}/site-packages platlib = {userbase}/python{py_version_short}/site-packages include = {userbase}/python{py_version_short}/include scripts = {userbase}/bin data = {userbase} CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From Van.Lindberg at haynesboone.com Wed Mar 14 17:03:54 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Wed, 14 Mar 2012 16:03:54 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F603B76.4050004@gmail.com> References: <4F603B76.4050004@gmail.com> Message-ID: <4F60C169.9030404@gmail.com> On 3/14/2012 1:32 AM, Mark Hammond wrote: > As per comments later in the thread, I'm -1 on including > "python{py_version_short}" in the lib directories for a number of > reasons; one further reason not outlined is that it would potentially > make running Python directly from a built tree difficult. For the same > reason, I'm also -1 on having that in the include dir. A built tree would look the same as always - the directories would only be moved (if at all) during installation. Thus, you will still be able to run python directly from a built installation. Also note that the py_version_short will not be in platform installs. >> scripts = {base/userbase}/bin > > We should note that this may cause pain for a number of projects - I've > seen quite a few projects that already assume "Scripts" on Windows - eg, > virtualenv and setuptools IIRC If you look at these projects, though, they *special case* Windows to account for the different layout. Removing this difference will allow these projects to remove their special-casing code. >- and also assume the executable is where > it currently lives - one example off the top of my head is the mozilla > "jetpack" project - see: > > https://github.com/mozilla/addon-sdk/blob/master/bin/activate.bat#L117 This code actually reinforces my point: First, this code would actually still work. The section ":FoundPython" sets the PATH to "%VIRTUAL_ENV%\bin;%PYTHONINSTALL%;%PATH%" (L80), which would still allow python.exe to be found and run. Second, look at that line again. Mozilla actually has edited this code so that the jetpack uses a cross-platform "bin" convention, just as I am proposing. Third, one element of this proposal is that there would be a key placed in the registry that points directly to the location of the python executable, making locating it trivial to locate programmatically on Windows. Thanks, Van CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From anacrolix at gmail.com Wed Mar 14 17:26:09 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 15 Mar 2012 00:26:09 +0800 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <20120314103941.7cd7c9e6@pitrou.net> References: <20120313222103.163877e6@pitrou.net> <20120314103941.7cd7c9e6@pitrou.net> Message-ID: > Can you give a pointer to these one-liners? > Once a patch gets a month old or older, it tends to disappear from > everyone's radar unless you somehow "ping" on the tracker, or post a > message to the mailing-list. All of these can be verified with a few minutes of checking the described code paths. http://bugs.python.org/issue13839 http://bugs.python.org/issue13872 http://bugs.python.org/issue12684 http://bugs.python.org/issue13694 Thanks From solipsis at pitrou.net Wed Mar 14 17:28:06 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 17:28:06 +0100 Subject: [Python-Dev] SocketServer issues In-Reply-To: References: <20120314100226.04627743@pitrou.net> Message-ID: <20120314172806.633f9f33@pitrou.net> On Wed, 14 Mar 2012 08:27:08 -0700 Guido van Rossum wrote: > Hopefully it doesn't use select if no timeout is set... No, it doesn't :-) Regards Antoine. From solipsis at pitrou.net Wed Mar 14 17:29:47 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 17:29:47 +0100 Subject: [Python-Dev] SocketServer issues In-Reply-To: References: <20120314100226.04627743@pitrou.net> Message-ID: <20120314172947.4f163e69@pitrou.net> On Wed, 14 Mar 2012 12:17:06 -0400 PJ Eby wrote: > > > So, my first question is: Why not simply rely on the already built-in > > timeout > > > support in the socket module? > > > > In case you didn't notice, the built-in timeout support *also* uses > > select(). > > > > That's not really the point; the frameworks that implement nonblocking I/O > by replacing the socket module (and Stackless is only one of many) won't be > using that code. Then they should also replace the select module. Again, I don't think SocketServer (or any other stdlib module) should be designed in this regard. Regards Antoine. From kristjan at ccpgames.com Wed Mar 14 17:42:29 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 16:42:29 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: Yes, the intended use is relative timings and timeouts. I think we are complicating things far too much. 1) Do we really need a fallback on windows? Will QPC ever fail? 2) is it a problem for the intended use if we cannot absolutely guarantee that time won't ever tick backwards? IMHO, we shouldn't compilicate the api, and make whatever best try we can in C. On windows I would do this (pseudocode) Static last_time = 0 If (QPC_works) time = QueryPerformanceCounter() else time = GetSystemTimeAsFileTime() if (time > last_time) last_time=time else time = last_time return time in other words: 1) use QPC. If the api indicates that it isn't available (does this ever happen in real life?) use a fallback of system time 2) enforce monotonicity with a static. QPC, if the OS doesn"t is buggy (calls cpu counter rather than timer chip) can cause jitter because it is called on different cores. No options in the api. No nothing. We simply provide the best api possible and some hardware/software combos might be less accurate. K -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Matt Joiner Sent: 14. mars 2012 09:12 To: Antoine Pitrou; Victor Stinner; Guido van Rossum Cc: python-dev at python.org Subject: Re: [Python-Dev] Drop the new time.wallclock() function? I have some observations regarding this: Victor's existing time.monotonic and time.wallclock make use of QueryPerformanceCounter, and CLOCK_MONOTONIC_RAW as possible. Both of these are hardware-based counters, their monotonicity is just a convenient property of the timer sources. Furthermore, time values can actually vary depending on the processor the call is served on. time.hardware()? time.monotonic_raw()? There are bug reports on Linux that CLOCK_MONOTONIC isn't always monotonic. This is why CLOCK_MONOTONIC_RAW was created. There's also the issue of time leaps (forward), which also isn't a problem with the latter form. time.monotonic(raw_only=False)? The real value of "monotonic" timers is that they don't leap backwards, and preferably don't leap forwards. Whether they are absolute is of no consequence. I would suggest that the API reflect this, and that more specific time values be obtained using the proper raw syscall wrapper (like time.clock_gettime) if required. time.relative(strict=False)? The ultimate use of the function name being established is for use in timeouts and relative timings. Where an option is present, it disallows fallbacks like CLOCK_MONOTONIC and other weaker forms: * time.hardware(fallback=True) -> reflects the source of the timing impeccably. alerts users to possible affinity issues * time.monotonic_raw() -> a bit linux specific... * time.relative(strict=False) -> matches the use case. a warning could be put regarding hardware timings * time.monotonic(raw_only=False) -> closest to the existing implementation. the keyword name i think is better. _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com From guido at python.org Wed Mar 14 17:44:07 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Mar 2012 09:44:07 -0700 Subject: [Python-Dev] SocketServer issues In-Reply-To: References: Message-ID: 2012/3/13 Kristj?n Valur J?nsson : > I want to mention some issues I?ve had with the socketserver module, and > discuss if there?s a way to make it nicer. > > So, for a long time we were able to create magic stackless mixin classes for > it, like ThreadingMixIn, and assuming we had the appropriate socket > replacement library, be able to use it nicely using tasklets. > > Then, at some point, the run_forever loop was changed to support timeout > through the use of select.select() before every socket.accept() call.? This > was very awkward because the whole concept of select() really goes contrary > to the approach of using microthreads, non-blocking IO and all that. I'm surprised -- surely a non-blocking framework should have no problem implementing select(), especially if it's for one file descriptor? > The only reason for this select call, was to support timeout for the > accept.? And even for vanilla applications, it necessiates an extra kernel > call for every accept, just for the timeout. I think it's fine to change the code so that if poll_interval is explicitly set to None, the select() call is skipped. I don't think the default should change though. The select() call is normally needed to support the shutdown() feature, which is very useful. And also the overridable service_actions() method. Oh, there's another select() call, in handle_request(), that should also be skipped if timeout is None. At least, I *think* a select() with a timeout of None blocks forever or until the socket is ready or until it is interrupted; I think this can always be skipped, since the immediately following I/O call will block in exactly the same way. Unless the socket is set in non-blocking mode; we may have to have provisions to avoid breaking that situation too. > The way around this for me has to do local modifications to the SocketServer > and just get rid of the select. I hope the above suggestion is sufficient? It's the best we can do while maintaining backward compatibility. This class has a lot of different features, and is designed to be subclassed, so it's hard to make changes that don't break anything. > So, my first question is:? Why not simply rely on the already built-in > timeout support in the socket module?? Setting the correct timeout value on > the accepting socket, will achieve the same thing.? Of course, one then has > to reset the timeout value on the accepted socket, but this is minor. I don't think it's the same thing at all. If you set a timeout on the socket, the accept() or recvfrom() call in get_request() will raise an exception if no request comes in within the timeout (default 0.5 sec); with the timeout implemented in serve_forever(), get_request() and its callers won't have to worry about the timeout exception. > Second: Of late the SocketServer has grown additional features and > attributes.? In particular, it now has two event objects, __shutdown_request > and __is_shut_down. > > Notice the double underscores. > > They make it impossible to subclass the SocketServer class to provide a > different implementation of run_forever().? Is there any good reason why > these attributes have been made ?private? like this?? Having just seen > Raymond?s talk on how to subclass right, this looks like the wrong way to > use the double leading underscores. Assuming you meant serve_forever(), I see no problem at all. If you override serve_forever() you also have to override shutdown(). That's all. They are marked private because they are involved in subtle invariants that are easily disturbed if users touch them. I could live with making them single-underscore protected, only to be used by knowledgeable subclasses. But not with making then public attributes. > So, two things really: > > The use of select.select in SocketServer makes it necessary to subclass it > to write a new version of run_forever() for those that wish to use a > non-blocking IO library instead of socket. And the presence of these private > attributes make it (theoretically) impossible to specialize run_forever in a > mix-in class. > > > > Any thoughs?? Is anyone interested in seeing how the timeouts can be done > without using select.select()?? And what do you think about removing the > double underscores from there and thus making serve_forever owerrideable? Let's see a patch (based on my concerns above) and then we can talk again. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Wed Mar 14 17:40:19 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 17:40:19 +0100 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: References: <20120313222103.163877e6@pitrou.net> <20120314103941.7cd7c9e6@pitrou.net> Message-ID: <20120314174019.5cead57b@pitrou.net> On Thu, 15 Mar 2012 00:26:09 +0800 Matt Joiner wrote: > > Can you give a pointer to these one-liners? > > Once a patch gets a month old or older, it tends to disappear from > > everyone's radar unless you somehow "ping" on the tracker, or post a > > message to the mailing-list. > > All of these can be verified with a few minutes of checking the > described code paths. > > http://bugs.python.org/issue13839 > http://bugs.python.org/issue13872 > http://bugs.python.org/issue12684 > http://bugs.python.org/issue13694 Thanks. You did get some comments and answers on some of these issues, though (especially on http://bugs.python.org/issue13872 ). Regards Antoine. From nadeem.vawda at gmail.com Wed Mar 14 17:47:24 2012 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Wed, 14 Mar 2012 18:47:24 +0200 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: A summary of the discussion so far, as I've understood it: - We should have *one* monotonic/steady timer function, using the sources described in Victor's original post. - By default, it should fall back to time.time if a better source is not available, but there should be a flag that can disable this fallback for users who really *need* a monotonic/steady time source. - Proposed names for the function: * monotonic * steady_clock * wallclock * realtime - Proposed names for the flag controlling fallback behavior: * strict (=False) * fallback (=True) * monotonic (=False) For the function name, I think monotonic() and steady_clock() convey the purpose of the function much better than the other two; the term "wallclock" is actively misleading, and "realtime" seems ambiguous. For the flag name, I'm -1 on "monotonic" -- it sounds like a flag to decide whether to use a monotonic time source always or never, while it actually decides between "always" and "sometimes". I think "strict" is nicer than "fallback", but I'm fine with either one. Cheers, Nadeem From tjreedy at udel.edu Wed Mar 14 17:47:21 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2012 12:47:21 -0400 Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <4F606D57.80304@hotpy.org> References: <20120313222103.163877e6@pitrou.net> <20120314103941.7cd7c9e6@pitrou.net> <20120314095810.GA20325@sleipnir.bytereef.org> <4F606D57.80304@hotpy.org> Message-ID: On 3/14/2012 6:05 AM, Mark Shannon wrote: > But how do you find issues? It takes some practice. Since you patched core component dict, I tried All text: dict and Components: Interpreter Core. (Leave default Status: open as is.) 51 issues. Add Keyword: needs review. 0 issues. Whoops, seems not to be used. Change to Keyword: patch. 31 issues. That seems like a manageable number of titles to 'wade through', middle clicking ones that seem promising to open in a new tab. > I want to do some reviews, but I don't want to wade through issues on > components I know little or nothing about in order to find the ones I > can review. If you want more help, ask me here or privately. I am probably better at searching than fixing. > There does not seem to be a way to filter search results in the tracker. I am not sure what you mean as there are multiple search fields. The basic text box is, admittedly, limited in that it combines multiple words with AND, with no other choices. An OR search requires multiple searches. Exclusion is not possible that I know of, and perhaps that is what you meant. The other fields that seem most useful to me are Stage, Type, Components, and Status. -- Terry Jan Reedy From pje at telecommunity.com Wed Mar 14 18:03:56 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 14 Mar 2012 13:03:56 -0400 Subject: [Python-Dev] SocketServer issues In-Reply-To: <20120314172947.4f163e69@pitrou.net> References: <20120314100226.04627743@pitrou.net> <20120314172947.4f163e69@pitrou.net> Message-ID: On Wed, Mar 14, 2012 at 12:29 PM, Antoine Pitrou wrote: > On Wed, 14 Mar 2012 12:17:06 -0400 > PJ Eby wrote: > > That's not really the point; the frameworks that implement nonblocking > I/O > > by replacing the socket module (and Stackless is only one of many) won't > be > > using that code. > > Then they should also replace the select module. > That actually sounds like a good point. ;-) I'm not the maintainer of any of those frameworks, but IIRC some of them *do* replace it. Perhaps this would solve Stackless's problem here too? -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacrolix at gmail.com Wed Mar 14 18:08:39 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 15 Mar 2012 01:08:39 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: On Thu, Mar 15, 2012 at 12:22 AM, Guido van Rossum wrote: > I have a totally different observation. Presumably the primary use > case for these timers is to measure real time intervals for the > purpose of profiling various operations. For this purpose we want them > to be as "steady" as possible: tick at a constant rate, don't jump > forward or backward. (And they shouldn't invoke the concept of "CPU" > time -- we already have time.clock() for that, besides it's often > wrong, e.g. you might be measuring some sort of I/O performance.) If > this means that a second as measured by time.time() is sometimes not > the same as a second measured by this timer (due to time.time() > occasionally jumping due to clock adjustments), that's fine with me. > If this means it's unreliable inside a VM, well, it seems that's a > property of the underlying OS timer, and there's not much we can do > about it except letting a knowledgeable user override the timer user. > As for names, I like Jeffrey's idea of having "steady" in the name. In that case I'd suggest either time.hardware(strict=True), or time.steady(strict=True), since the only timers exposed natively that are both high resolution and steady are on the hardware. A warning about CPU affinity is also still wise methinks. From kristjan at ccpgames.com Wed Mar 14 18:09:39 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 17:09:39 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: > - By default, it should fall back to time.time if a better source is > not available, but there should be a flag that can disable this > fallback for users who really *need* a monotonic/steady time source. As pointed out on a different thread, you don"t need this "flag" since the code can easily enforce the monotonic property by maintaining a static value. This is how we worked around buggy implementations of QueryPerformanceCounter on windows (). K -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Nadeem Vawda Sent: 14. mars 2012 09:47 To: Guido van Rossum Cc: Antoine Pitrou; python-dev at python.org Subject: Re: [Python-Dev] Drop the new time.wallclock() function? A summary of the discussion so far, as I've understood it: - We should have *one* monotonic/steady timer function, using the sources described in Victor's original post. - By default, it should fall back to time.time if a better source is not available, but there should be a flag that can disable this fallback for users who really *need* a monotonic/steady time source. - Proposed names for the function: * monotonic * steady_clock * wallclock * realtime - Proposed names for the flag controlling fallback behavior: * strict (=False) * fallback (=True) * monotonic (=False) For the function name, I think monotonic() and steady_clock() convey the purpose of the function much better than the other two; the term "wallclock" is actively misleading, and "realtime" seems ambiguous. For the flag name, I'm -1 on "monotonic" -- it sounds like a flag to decide whether to use a monotonic time source always or never, while it actually decides between "always" and "sometimes". I think "strict" is nicer than "fallback", but I'm fine with either one. Cheers, Nadeem _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com From kristjan at ccpgames.com Wed Mar 14 17:59:47 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 16:59:47 +0000 Subject: [Python-Dev] SocketServer issues In-Reply-To: <20120314100226.04627743@pitrou.net> References: <20120314100226.04627743@pitrou.net> Message-ID: >I don't really think the ability to "create magic stackless mixin classes" should be a driving principle for the stdlib. > I would suggest using a proper non-blocking framework such as Twisted. There is a lot of code out there that uses SocketServer. It was originally designed to be easily extensable, with various mixins to control ultimate Behavior. It just seems that there have been some design decisions made recently that make this subclassability / extensibility more difficult, and those Are the two changes I pointed out. The thing with the select.select wouldn't be so bad if I could simply override serve_forever but that function Can't be overridden because of the poor choice of adding __attributes to the class. And, you would run into the same kind of trouble if you wanted to create a TwistedMixIn, a geventMixIn, or what not. > In case you didn't notice, the built-in timeout support *also* uses select(). Yes, that's how the normal blocking framework supports timeout. Asynchronous frameworks do it differently, though. > Then they should also replace the select module. > Again, I don't think SocketServer (or any other stdlib module) should be designed in this regard. And so we do too, but now every socket accept requires two rounds round the event loop. Also, emulating select() is not a critical part of frameworks designed to help you avoid to use it in the first place. The point of frameworks such as gevent, stackless, etc, is to let you write code with zillions of sockets without ever touching select. The quick and dirty emulated version I use, uses a thread to make it non-blocking! It just seems odd to me that it was designed to use the "select" api to do timeouts, where timeouts are already part of the socket protocol and can be implemented more efficiently there. Anyway, I'm not talking about rewriting anything, I merely want to fix some small design problems that prevent SocketServer to be specialized. I'll submit a simple patch for review. K -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Antoine Pitrou Sent: 14. mars 2012 02:02 To: python-dev at python.org Subject: Re: [Python-Dev] SocketServer issues On Wed, 14 Mar 2012 04:26:16 +0000 Kristj?n Valur J?nsson wrote: > Hi there. > I want to mention some issues I've had with the socketserver module, and discuss if there's a way to make it nicer. > So, for a long time we were able to create magic stackless mixin > classes for it, like ThreadingMixIn, and assuming we had the > appropriate socket replacement library, be able to use it nicely using tasklets. I don't really think the ability to "create magic stackless mixin classes" should be a driving principle for the stdlib. I would suggest using a proper non-blocking framework such as Twisted. > So, my first question is: Why not simply rely on the already built-in > timeout support in the socket module? In case you didn't notice, the built-in timeout support *also* uses select(). Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com From anacrolix at gmail.com Wed Mar 14 18:15:52 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 15 Mar 2012 01:15:52 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: FWIW the name is quite important, because these kind of timings are quite important so I think it's worth the effort. > - By default, it should fall back to time.time if a better source is > ?not available, but there should be a flag that can disable this > ?fallback for users who really *need* a monotonic/steady time source. Agreed. As Guido mentioned, some platforms might not be able to access to hardware times, so falling back should be the default, lest unaware users trigger unexpected errors. > - Proposed names for the function: > ?* monotonic Doesn't indicate that the timing is also prevented from leaping forward. > ?* steady_clock I think the use of clock might infer CPU time on doc-skimming user. "clock" is overloaded here. > For the flag name, I'm -1 on "monotonic" -- it sounds like a flag to > decide whether to use a monotonic time source always or never, while > it actually decides between "always" and "sometimes". I think "strict" > is nicer than "fallback", but I'm fine with either one. I agree, "strict" fits in with existing APIs. I think time.hardware(), and time.steady() are still okay here. From guido at python.org Wed Mar 14 18:21:30 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Mar 2012 10:21:30 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: +1 for steady(). On Wed, Mar 14, 2012 at 10:15 AM, Matt Joiner wrote: > FWIW the name is quite important, because these kind of timings are > quite important so I think it's worth the effort. > >> - By default, it should fall back to time.time if a better source is >> ?not available, but there should be a flag that can disable this >> ?fallback for users who really *need* a monotonic/steady time source. > > Agreed. As Guido mentioned, some platforms might not be able to access > to hardware times, so falling back should be the default, lest unaware > users trigger unexpected errors. > >> - Proposed names for the function: >> ?* monotonic > > Doesn't indicate that the timing is also prevented from leaping forward. > >> ?* steady_clock > > I think the use of clock might infer CPU time on doc-skimming user. > "clock" is overloaded here. > >> For the flag name, I'm -1 on "monotonic" -- it sounds like a flag to >> decide whether to use a monotonic time source always or never, while >> it actually decides between "always" and "sometimes". I think "strict" >> is nicer than "fallback", but I'm fine with either one. > > I agree, "strict" fits in with existing APIs. > > I think time.hardware(), and time.steady() are still okay here. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Wed Mar 14 18:23:07 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2012 18:23:07 +0100 Subject: [Python-Dev] SocketServer issues References: <20120314100226.04627743@pitrou.net> Message-ID: <20120314182307.4e4e9c86@pitrou.net> On Wed, 14 Mar 2012 16:59:47 +0000 Kristj?n Valur J?nsson wrote: > > It just seems odd to me that it was designed to use the "select" api to do timeouts, > where timeouts are already part of the socket protocol and can be implemented more > efficiently there. How is it more efficient if it uses the exact same system calls? And why are you worrying exactly? I don't understand why accept() would be critical for performance. Thanks Antoine. From andrew.svetlov at gmail.com Wed Mar 14 18:39:53 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Wed, 14 Mar 2012 10:39:53 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: On Wed, Mar 14, 2012 at 10:21 AM, Guido van Rossum wrote: > +1 for steady(). > +1 From regebro at gmail.com Wed Mar 14 18:42:41 2012 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 14 Mar 2012 18:42:41 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <20120314101618.5cf1850f@pitrou.net> References: <20120314101618.5cf1850f@pitrou.net> Message-ID: On Wed, Mar 14, 2012 at 10:16, Antoine Pitrou wrote: > That's a rather awful name. ?time.time() is *the* real time. > > time.monotonic(fallback=False) would be a better API. I think calling the function "monotonic" isn't really a good name if it's not always monotonic. time.monotonic(fallback=False) Really just means time.monotonic(monotonic=False) And time.monotonic(strict=True) Really means time.monotonic(i_really_mean_it=True) This is potentially confusing. Therefore time.clock() time.time() time.realtime() time.wallclock() Are all better options if there is a flag to switch if it's monotonic or not. Since time.clock() apparently can mean different things on different platforms because of it's use of the C-API, we can't use that. I'm not sure why time.time() would differ from time.realtime(). time.time() is per definition not monotonic, but time.time(monotonic=True) is maybe a possibility? //Lennart //Lennart From tjreedy at udel.edu Wed Mar 14 18:55:27 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2012 13:55:27 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F5FF662.4070201@v.loewis.de> Message-ID: On 3/14/2012 12:10 PM, VanL wrote: > On 3/14/2012 10:56 AM, Terry Reedy wrote: >> Are you talking about 'install for all users' versus 'install for this >> user only'? I have always done the former as I see no point to the >> latter on my machine, even if another family member has an account. > > Yes, but some people are on corporate machines that only allow "install > for this user" installations. Ok. On such machines, system install (by vendor) == only 'base' install == only 'all user' install. >>> I am fine with keeping the distinction between >> > base installs (no py_version) >> >> I have no idea what this means. As far as I can remember, each >> installation of Python x.y (back to 1.3 for me, on DOS) has gone into a >> pythonxy (no dot) directory, with subdirectories much as Jim J. >> described. > > I am referring to the currently-existing install schemes 'nt' ('install > for all users') and 'nt-user' ('install for this user only'). The > *current* layouts are described at > http://hg.python.org/distutils2/file/2cec52b682a9/distutils2/_backport/sysconfig.cfg: > > L57-65: > [nt] > stdlib = {base}/Lib > platstdlib = {base}/Lib > purelib = {base}/Lib/site-packages > platlib = {base}/Lib/site-packages > include = {base}/Include > platinclude = {base}/Include > scripts = {base}/Scripts > data = {base} Is this from 2.x? Currently, in 3.x, Scripts is tucked inside Tools, so it seems to be scripts = {base}/Tools/Scripts > > L86-93: > [nt_user] > stdlib = {userbase}/Python{py_version_nodot} > platstdlib = {userbase}/Python{py_version_nodot} > purelib = {userbase}/Python{py_version_nodot}/site-packages > platlib = {userbase}/Python{py_version_nodot}/site-packages > include = {userbase}/Python{py_version_nodot}/Include > scripts = {userbase}/Scripts > data = {userbase} > > I am proposing that these change to: > > [nt] > stdlib = {base}/lib > platstdlib = {base}/lib > purelib = {base}/lib/site-packages > platlib = {base}/lib/site-packages > include = {base}/include > platinclude = {base}/include > scripts = {base}/bin > data = {base} > > [nt_user] > stdlib = {userbase}/python{py_version_short} > platstdlib = {userbase}/python{py_version_short} > purelib = {userbase}/python{py_version_short}/site-packages > platlib = {userbase}/python{py_version_short}/site-packages > include = {userbase}/python{py_version_short}/include > scripts = {userbase}/bin > data = {userbase} OK, now I see where 'base' and 'userbase' come from. This is an area I have ignored, so I only have a user view of the result. > All the other diuectories that Jim talked about would not be affected by > this proposal. > > Does this make it clearer? Now that we are speaking the same language, yes. Thank you. Lowercasing 'Include' is fine with me. The only question is how it affects tools in the field. Lowercasing 'Lib' would also be fine if 'libs' were changed to 'libraries' or 'headers' or perhaps even better, '_libs', as normal users never have any reason to look therein. Just something more easily distinguished from 'lib'. Same comment about tools. The present installed directory scheme is a hodgepodge of almost all caps, initial cap, and no cap. I would not mind more consistency. The only directory I regularly look in is Lib, so my main concern is that that be visually easy to find with my less than perfect vision. -- Terry Jan Reedy From jimjjewett at gmail.com Wed Mar 14 18:59:23 2012 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Wed, 14 Mar 2012 10:59:23 -0700 (PDT) Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: Message-ID: <4f60dc7b.0ad1320a.7924.ffffb2bd@mx.google.com> In http://mail.python.org/pipermail/python-dev/2012-March/117617.html van.lindberg at gmail.com posted: > As noted earlier in the thread, I also change my proposal to maintain > the existing differences between system installs and user installs. [Wanted lower case, which should be irrelevant; sysconfig.get_python_inc already assumes lower case despite the configuration file.] [Wanted "bin" instead of "Scripts", even though they aren't binaries.] If there are to be any changes, I *am* tempted to at least harmonize the two install types, but to use the less redundant system form. If the user is deliberately trying to hide that it is version 33 (or even that it is python), then so be it; defaulting to redundant information is not an improvement. Set the base/userbase at install time, with defaults of base = %SystemDrive%\{py_version_nodot} userbase = %USERPROFILE%\Application Data\{py_version_nodot} usedbase = base for system installs; userbase for per-user installs. Then let the rest default to subdirectories; sysconfig.get_config_vars on windows explicitly doesn't provide as many variables as unix, just INCLUDEPY (which should default to {usedbase}/include) and LIBDEST and BINLIBDEST (both of which should default to {usedbase}/lib). And no, I'm not forgetting data or scripts. As best I can tell, sysconfig doesn't actually expose them, and there is no Scripts directory on my machine (except inside Tools). Perhaps some installers create it when they install their own extensions? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From regebro at gmail.com Wed Mar 14 19:07:29 2012 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 14 Mar 2012 19:07:29 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: 2012/3/14 Kristj?n Valur J?nsson : >> - By default, it should fall back to time.time if a better source is >> ?not available, but there should be a flag that can disable this >> ?fallback for users who really *need* a monotonic/steady time source. > As pointed out on a different thread, you don"t need this "flag" since the code can easily enforce the monotonic property by maintaining a static value. With this, I think time.steady() would be clear and nice. //Lennart From kristjan at ccpgames.com Wed Mar 14 19:37:05 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 18:37:05 +0000 Subject: [Python-Dev] SocketServer issues In-Reply-To: <20120314182307.4e4e9c86@pitrou.net> References: <20120314100226.04627743@pitrou.net> <20120314182307.4e4e9c86@pitrou.net> Message-ID: A different implementation, (e.g. one using windows IOCP), can do timeouts without using select (and must, select does not work with IOCP). So will a gevent based implementation, it will timeout the accept on each socket individually, not by calling select on each of them. The reason I'm fretting is latency. There is only one thread accepting connections. If it has to do an extra event loop dance for every socket that it accepts that adds to the delay in getting a response from the server. Accept() is indeed critical for socket server performance. Maybe this is all just nonsense, still it seems odd to jump through extra hoops to emulate a functionality that is already supported by the socket spec, and can be done in the most appropriate way for each implementation. K -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Antoine Pitrou Sent: 14. mars 2012 10:23 To: python-dev at python.org Subject: Re: [Python-Dev] SocketServer issues On Wed, 14 Mar 2012 16:59:47 +0000 Kristj?n Valur J?nsson wrote: > > It just seems odd to me that it was designed to use the "select" api > to do timeouts, > where timeouts are already part of the socket protocol and can be implemented more efficiently there. How is it more efficient if it uses the exact same system calls? And why are you worrying exactly? I don't understand why accept() would be critical for performance. Thanks Antoine. From guido at python.org Wed Mar 14 19:43:32 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Mar 2012 11:43:32 -0700 Subject: [Python-Dev] SocketServer issues In-Reply-To: References: <20120314100226.04627743@pitrou.net> <20120314182307.4e4e9c86@pitrou.net> Message-ID: 2012/3/14 Kristj?n Valur J?nsson : > Maybe this is all just nonsense, still it seems odd to jump through extra hoops to emulate a functionality that is already supported by the socket spec, and can be done in the most appropriate way for each implementation. I thought I had already explained why setting the timeout on the socket is not the same. -- --Guido van Rossum (python.org/~guido) From anacrolix at gmail.com Wed Mar 14 19:47:13 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 15 Mar 2012 02:47:13 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: I also can live with steady, with strict for the flag. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Wed Mar 14 19:52:30 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 14 Mar 2012 19:52:30 +0100 Subject: [Python-Dev] 2012 Language Summit Report In-Reply-To: References: Message-ID: On 14.03.2012 15:12, Brian Curtin wrote: > As with last year, I've put together a summary of the Python Language > Summit which took place last week at PyCon 2012. This was compiled > from my notes as well as those of Eric Snow and Senthil Kumaran, and I > think we got decent coverage of what was said throughout the day. > > http://blog.python.org/2012/03/2012-language-summit-report.html > > If you have questions or comments about discussions which occurred > there, please create a new thread for your topic. > > Feel free to contact me directly if I've left anything out or > misprinted anything. Thanks for the comprehensive report (I'm still reading). May I request for the future that you also paste a copy in the email to the group, for purposes of archiving and ease of discussing? (Just like we also post PEPs to python-dev for discussion even when they are already online.) cheers, Georg From v+python at g.nevcal.com Wed Mar 14 19:55:30 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 14 Mar 2012 11:55:30 -0700 Subject: [Python-Dev] 2012 Language Summit Report In-Reply-To: References: Message-ID: <4F60E9A2.4090100@g.nevcal.com> On 3/14/2012 8:57 AM, Terry Reedy wrote: > On 3/14/2012 10:12 AM, Brian Curtin wrote: >> As with last year, I've put together a summary of the Python Language >> Summit which took place last week at PyCon 2012. This was compiled >> from my notes as well as those of Eric Snow and Senthil Kumaran, and I >> think we got decent coverage of what was said throughout the day. >> >> http://blog.python.org/2012/03/2012-language-summit-report.html > > Nicely done. Thank you. > Thanks. Almost feels like I was there! -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Wed Mar 14 20:12:59 2012 From: brian at python.org (Brian Curtin) Date: Wed, 14 Mar 2012 14:12:59 -0500 Subject: [Python-Dev] 2012 Language Summit Report In-Reply-To: References: Message-ID: On Wed, Mar 14, 2012 at 13:52, Georg Brandl wrote: > Thanks for the comprehensive report (I'm still reading). ?May I request > for the future that you also paste a copy in the email to the group, for > purposes of archiving and ease of discussing? ?(Just like we also post > PEPs to python-dev for discussion even when they are already online.) Certainly -- good idea. I have a few updates and corrections to make this evening, then I'll get a copy of it posted. From g.brandl at gmx.net Wed Mar 14 20:33:27 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 14 Mar 2012 20:33:27 +0100 Subject: [Python-Dev] cpython: PEP 417: Adding unittest.mock In-Reply-To: References: Message-ID: On 14.03.2012 20:25, michael.foord wrote: > http://hg.python.org/cpython/rev/2fda048ee32a > changeset: 75632:2fda048ee32a > user: Michael Foord > date: Wed Mar 14 12:24:34 2012 -0700 > summary: > PEP 417: Adding unittest.mock > > files: > Lib/unittest/mock.py | 2151 ++++++++++ > Lib/unittest/test/__init__.py | 1 + > Lib/unittest/test/testmock/__init__.py | 17 + > Lib/unittest/test/testmock/support.py | 23 + > Lib/unittest/test/testmock/testcallable.py | 159 + > Lib/unittest/test/testmock/testhelpers.py | 835 +++ > Lib/unittest/test/testmock/testmagicmethods.py | 382 + > Lib/unittest/test/testmock/testmock.py | 1258 +++++ > Lib/unittest/test/testmock/testpatch.py | 1652 +++++++ > Lib/unittest/test/testmock/testsentinel.py | 28 + > Lib/unittest/test/testmock/testwith.py | 176 + > 11 files changed, 6682 insertions(+), 0 deletions(-) I hope we also get some Dock/ ;) Georg From kristjan at ccpgames.com Wed Mar 14 20:35:54 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 14 Mar 2012 19:35:54 +0000 Subject: [Python-Dev] SocketServer issues In-Reply-To: References: <20120314100226.04627743@pitrou.net> <20120314182307.4e4e9c86@pitrou.net> Message-ID: Yes, setting a timeout and leaving it that way is not the same. But setting the timeout for _accept only_ is the "same" except one approach requires the check of a bool return, the other the handling of a socket.timeout exeption. My point is, if sockets already have nice and well defined timeout semantics, why not use them, or even improve them (perhaps with an optional timeout parameter to the accept call) rather than reimplement them with an explicit select.select() call? Anyway, I'll take another look at the problem and possibly submit a patch suggestion. Thanks. K -----Original Message----- From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum Sent: 14. mars 2012 11:44 To: Kristj?n Valur J?nsson Cc: Antoine Pitrou; python-dev at python.org Subject: Re: [Python-Dev] SocketServer issues 2012/3/14 Kristj?n Valur J?nsson : > Maybe this is all just nonsense, still it seems odd to jump through extra hoops to emulate a functionality that is already supported by the socket spec, and can be done in the most appropriate way for each implementation. I thought I had already explained why setting the timeout on the socket is not the same. -- --Guido van Rossum (python.org/~guido) From lukasz at langa.pl Wed Mar 14 20:49:55 2012 From: lukasz at langa.pl (=?iso-8859-2?Q?=A3ukasz_Langa?=) Date: Wed, 14 Mar 2012 12:49:55 -0700 Subject: [Python-Dev] cpython: PEP 417: Adding unittest.mock In-Reply-To: References: Message-ID: <38BE6358-8BA1-4080-BA27-78B4BCABDC57@langa.pl> Wiadomo?? napisana przez Georg Brandl w dniu 14 mar 2012, o godz. 12:33: > On 14.03.2012 20:25, michael.foord wrote: >> http://hg.python.org/cpython/rev/2fda048ee32a >> changeset: 75632:2fda048ee32a >> user: Michael Foord >> date: Wed Mar 14 12:24:34 2012 -0700 >> summary: >> PEP 417: Adding unittest.mock >> >> files: >> Lib/unittest/mock.py | 2151 ++++++++++ >> Lib/unittest/test/__init__.py | 1 + >> Lib/unittest/test/testmock/__init__.py | 17 + >> Lib/unittest/test/testmock/support.py | 23 + >> Lib/unittest/test/testmock/testcallable.py | 159 + >> Lib/unittest/test/testmock/testhelpers.py | 835 +++ >> Lib/unittest/test/testmock/testmagicmethods.py | 382 + >> Lib/unittest/test/testmock/testmock.py | 1258 +++++ >> Lib/unittest/test/testmock/testpatch.py | 1652 +++++++ >> Lib/unittest/test/testmock/testsentinel.py | 28 + >> Lib/unittest/test/testmock/testwith.py | 176 + >> 11 files changed, 6682 insertions(+), 0 deletions(-) > > I hope we also get some Dock/ ;) Mock the doc! -- Best regards, ?ukasz Langa Senior Systems Architecture Engineer IT Infrastructure Department Grupa Allegro Sp. z o.o. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Mar 14 21:08:18 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2012 16:08:18 -0400 Subject: [Python-Dev] [Python-checkins] cpython: PEP 417: Adding unittest.mock In-Reply-To: References: Message-ID: <4F60FAB2.7020207@udel.edu> On 3/14/2012 3:25 PM, michael.foord wrote: > +# mock.py > +# Test tools for mocking and patching. Should there be a note here about restrictions on editing this file? I notice that there are things like > +class OldStyleClass: > + pass > +ClassType = type(OldStyleClass) which are only present for running under Py2 and which would normally be removed for Py3. --- Terry Jan Reedy From fuzzyman at voidspace.org.uk Wed Mar 14 21:22:09 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Mar 2012 13:22:09 -0700 Subject: [Python-Dev] [Python-checkins] cpython: PEP 417: Adding unittest.mock In-Reply-To: <4F60FAB2.7020207@udel.edu> References: <4F60FAB2.7020207@udel.edu> Message-ID: <03503024-ECFB-462F-BF08-022BF93578D0@voidspace.org.uk> On 14 Mar 2012, at 13:08, Terry Reedy wrote: > On 3/14/2012 3:25 PM, michael.foord wrote: >> +# mock.py >> +# Test tools for mocking and patching. > > Should there be a note here about restrictions on editing this file? > I notice that there are things like > > > +class OldStyleClass: > > + pass > > +ClassType = type(OldStyleClass) > > which are only present for running under Py2 and which would normally be removed for Py3. Yeah, I removed as much of the Python 2 compatibility code and thought I'd got it all. Thanks for pointing it out. I'm maintaining a "clean" (no Python 2 compatibility code) version in the standard library. I'll be maintaining mock, so I'd like to be assigned any issues on it and at least talked to before changes are made. I am maintaining a backport still, but the Python standard library version is the canonical version. All the best, Michael Foord > > --- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From fuzzyman at voidspace.org.uk Wed Mar 14 21:27:43 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Mar 2012 13:27:43 -0700 Subject: [Python-Dev] cpython: PEP 417: Adding unittest.mock In-Reply-To: References: Message-ID: On 14 Mar 2012, at 12:33, Georg Brandl wrote: > On 14.03.2012 20:25, michael.foord wrote: >> http://hg.python.org/cpython/rev/2fda048ee32a >> changeset: 75632:2fda048ee32a >> user: Michael Foord >> date: Wed Mar 14 12:24:34 2012 -0700 >> summary: >> PEP 417: Adding unittest.mock >> >> files: >> Lib/unittest/mock.py | 2151 ++++++++++ >> Lib/unittest/test/__init__.py | 1 + >> Lib/unittest/test/testmock/__init__.py | 17 + >> Lib/unittest/test/testmock/support.py | 23 + >> Lib/unittest/test/testmock/testcallable.py | 159 + >> Lib/unittest/test/testmock/testhelpers.py | 835 +++ >> Lib/unittest/test/testmock/testmagicmethods.py | 382 + >> Lib/unittest/test/testmock/testmock.py | 1258 +++++ >> Lib/unittest/test/testmock/testpatch.py | 1652 +++++++ >> Lib/unittest/test/testmock/testsentinel.py | 28 + >> Lib/unittest/test/testmock/testwith.py | 176 + >> 11 files changed, 6682 insertions(+), 0 deletions(-) > > I hope we also get some Dock/ ;) I was thinking about providing documentation, but I thought maybe I'd leave it for you Georg. It's nice to get an outside perspective when documenting a new api.... I have issue 14295 to track adding mock and it won't be closed until the docs are completed. http://bugs.python.org/issue14295 On the topic of docs.... mock documentation is about eight pages long. My intention was to strip this down to just the api documentation, along with a link to the docs on my site for further examples and so on. I was encouraged here at the sprints to include the full documentation instead (minus the mock library comparison page and the front page can be cut down). So this is what I am now intending to include. It does mean the mock documentation will be "extensive". All the best, Michael > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From benjamin at python.org Wed Mar 14 21:30:25 2012 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 14 Mar 2012 15:30:25 -0500 Subject: [Python-Dev] cpython: PEP 417: Adding unittest.mock In-Reply-To: References: Message-ID: 2012/3/14 Michael Foord : > On the topic of docs.... mock documentation is about eight pages long. My intention was to strip this down to just the api documentation, along with a link to the docs on my site for further examples and so on. I was encouraged here at the sprints to include the full documentation instead Yes, please do. -- Regards, Benjamin From tjreedy at udel.edu Wed Mar 14 21:46:29 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2012 16:46:29 -0400 Subject: [Python-Dev] [Python-checkins] cpython: PEP 417: Adding unittest.mock In-Reply-To: <03503024-ECFB-462F-BF08-022BF93578D0@voidspace.org.uk> References: <4F60FAB2.7020207@udel.edu> <03503024-ECFB-462F-BF08-022BF93578D0@voidspace.org.uk> Message-ID: On 3/14/2012 4:22 PM, Michael Foord wrote: > > On 14 Mar 2012, at 13:08, Terry Reedy wrote: > >> On 3/14/2012 3:25 PM, michael.foord wrote: >>> +# mock.py +# Test tools for mocking and patching. >> Should there be a note here about restrictions on editing this >> file? I notice that there are things like >> >>> +class OldStyleClass: + pass +ClassType = type(OldStyleClass) >> >> which are only present for running under Py2 and which would >> normally be removed for Py3. > > > Yeah, I removed as much of the Python 2 compatibility code and > thought I'd got it all. Thanks for pointing it out. 2000 lines is a lot to check through. > > I'm maintaining a "clean" (no Python 2 compatibility code) version in > the standard library. Great. Here is something else, which is why I thought otherwise ;-). +def _instance_callable(obj): + """Given an object, return True if the object is callable. + For classes, return True if instances would be callable.""" + if not isinstance(obj, type): + # already an instance + return getattr(obj, '__call__', None) is not None + + klass = obj + # uses __bases__ instead of __mro__ so that we work with >>> old style classes + if klass.__dict__.get('__call__') is not None: + return True + + for base in klass.__bases__: + if _instance_callable(base): + return True + return False If you want to leave the code as is, remove or revise the comment. > I'll be maintaining mock, so I'd like to be > assigned any issues on it and at least talked to before changes are > made. I am maintaining a backport still, but the Python standard > library version is the canonical version. Add unittest.mock to devguide/experts.rst and yourself with * appended. --- Searching for 'old', I also found +def _must_skip(spec, entry, is_type): + if not isinstance(spec, type): + if entry in getattr(spec, '__dict__', {}): + # instance attribute - shouldn't skip + return False >>>+ # can't use type because of old style classes + spec = spec.__class__ + if not hasattr(spec, '__mro__'): >>>+ # old style class: can't have descriptors anyway + return is_type In testcallable.py + def test_patch_spec_callable_class(self): + class CallableX(X): + def __call__(self): + pass + + class Sub(CallableX): + pass + + class Multi(SomeClass, Sub): + pass + >>>+ class OldStyle: + def __call__(self): + pass + >>>+ class OldStyleSub(OldStyle): + pass + + for arg in 'spec', 'spec_set': >>>+ for Klass in CallableX, Sub, Multi, OldStyle, OldStyleSub: This is the last. -- Terry Jan Reedy From g.brandl at gmx.net Wed Mar 14 21:52:44 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 14 Mar 2012 21:52:44 +0100 Subject: [Python-Dev] cpython: Issue #14200: Idle shell crash on printing non-BMP unicode character. In-Reply-To: References: Message-ID: On 14.03.2012 21:46, andrew.svetlov wrote: > diff --git a/Lib/idlelib/rpc.py b/Lib/idlelib/rpc.py > --- a/Lib/idlelib/rpc.py > +++ b/Lib/idlelib/rpc.py > @@ -196,8 +196,12 @@ > return ("ERROR", "Unsupported message type: %s" % how) > except SystemExit: > raise > + except KeyboardInterrupt: > + raise > except socket.error: > raise > + except Exception as ex: > + return ("CALLEXC", ex) > except: > msg = "*** Internal Error: rpc.py:SocketIO.localcall()\n\n"\ > " Object: %s \n Method: %s \n Args: %s\n" It appears that this would be better written as except socket.error: raise except Exception: return ("CALLEXC", ex) except: # BaseException, i.e. SystemExit, KeyboardInterrupt, GeneratorExit raise Georg From nadeem.vawda at gmail.com Wed Mar 14 22:17:52 2012 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Wed, 14 Mar 2012 23:17:52 +0200 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: +1 for time.steady(strict=False). On Wed, Mar 14, 2012 at 7:09 PM, Kristj?n Valur J?nsson wrote: >> - By default, it should fall back to time.time if a better source is >> ?not available, but there should be a flag that can disable this >> ?fallback for users who really *need* a monotonic/steady time source. > As pointed out on a different thread, you don"t need this "flag" since the code can easily enforce the monotonic property by maintaining a static value. > This is how we worked around buggy implementations of QueryPerformanceCounter on windows (). > K That's fine if you just need the clock to be monotonic, but it isn't any help if you also want to prevent it from jumping forward. Cheers, Nadeem From fuzzyman at voidspace.org.uk Wed Mar 14 22:41:52 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 14 Mar 2012 14:41:52 -0700 Subject: [Python-Dev] [Python-checkins] cpython: PEP 417: Adding unittest.mock In-Reply-To: References: <4F60FAB2.7020207@udel.edu> <03503024-ECFB-462F-BF08-022BF93578D0@voidspace.org.uk> Message-ID: <5BA5E20A-C4B8-4A2A-B38B-A1286FC65247@voidspace.org.uk> On 14 Mar 2012, at 13:46, Terry Reedy wrote: > On 3/14/2012 4:22 PM, Michael Foord wrote: >> >> On 14 Mar 2012, at 13:08, Terry Reedy wrote: >> >>> On 3/14/2012 3:25 PM, michael.foord wrote: >>>> +# mock.py +# Test tools for mocking and patching. > >>> Should there be a note here about restrictions on editing this >>> file? I notice that there are things like >>> >>>> +class OldStyleClass: + pass +ClassType = type(OldStyleClass) >>> >>> which are only present for running under Py2 and which would >>> normally be removed for Py3. >> >> >> Yeah, I removed as much of the Python 2 compatibility code and >> thought I'd got it all. Thanks for pointing it out. > > 2000 lines is a lot to check through. >> >> I'm maintaining a "clean" (no Python 2 compatibility code) version in >> the standard library. > > Great. Here is something else, which is why I thought otherwise ;-). > > +def _instance_callable(obj): > + """Given an object, return True if the object is callable. > + For classes, return True if instances would be callable.""" > + if not isinstance(obj, type): > + # already an instance > + return getattr(obj, '__call__', None) is not None > + > + klass = obj > + # uses __bases__ instead of __mro__ so that we work with > >>> old style classes > + if klass.__dict__.get('__call__') is not None: > + return True > + > + for base in klass.__bases__: > + if _instance_callable(base): > + return True > + return False > > If you want to leave the code as is, remove or revise the comment. Thanks very much for finding these, I'm pretty sure I've fixed all the ones you reported - and one more case where try...except...finally can now be used. All the best, Michael Foord > >> I'll be maintaining mock, so I'd like to be >> assigned any issues on it and at least talked to before changes are >> made. I am maintaining a backport still, but the Python standard >> library version is the canonical version. > > Add unittest.mock to devguide/experts.rst and yourself with * appended. > > --- > Searching for 'old', I also found > > +def _must_skip(spec, entry, is_type): > + if not isinstance(spec, type): > + if entry in getattr(spec, '__dict__', {}): > + # instance attribute - shouldn't skip > + return False > >>>+ # can't use type because of old style classes > + spec = spec.__class__ > + if not hasattr(spec, '__mro__'): > >>>+ # old style class: can't have descriptors anyway > + return is_type > > In testcallable.py > + def test_patch_spec_callable_class(self): > + class CallableX(X): > + def __call__(self): > + pass > + > + class Sub(CallableX): > + pass > + > + class Multi(SomeClass, Sub): > + pass > + > >>>+ class OldStyle: > + def __call__(self): > + pass > + > >>>+ class OldStyleSub(OldStyle): > + pass > + > + for arg in 'spec', 'spec_set': > >>>+ for Klass in CallableX, Sub, Multi, OldStyle, OldStyleSub: > > This is the last. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From mhammond at skippinet.com.au Wed Mar 14 23:39:53 2012 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 15 Mar 2012 09:39:53 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F60C169.9030404@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> Message-ID: <4F611E39.5070605@skippinet.com.au> On 15/03/2012 3:03 AM, Lindberg, Van wrote: > On 3/14/2012 1:32 AM, Mark Hammond wrote: >> As per comments later in the thread, I'm -1 on including >> "python{py_version_short}" in the lib directories for a number of >> reasons; one further reason not outlined is that it would potentially >> make running Python directly from a built tree difficult. For the same >> reason, I'm also -1 on having that in the include dir. > > A built tree would look the same as always - the directories would only > be moved (if at all) during installation. Thus, you will still be able > to run python directly from a built installation. So long as the location of the "lib" dir doesn't change, that is probably true. If lib was to change though, then PC/getpathp.c would need to change which is where my concern came from. >>> scripts = {base/userbase}/bin >> >> We should note that this may cause pain for a number of projects - I've >> seen quite a few projects that already assume "Scripts" on Windows - eg, >> virtualenv and setuptools IIRC > > If you look at these projects, though, they *special case* Windows to > account for the different layout. Removing this difference will allow > these projects to remove their special-casing code. I don't think that is true. One of the examples I offered was a .bat file - it wouldn't be possible to remove the .bat file with your proposal. The other example was the Windows specific launcher. Most things that need to locate the Python executable aren't actually Python code - once Python is running, locating the executable is as simple as sys.executable. So by their very nature, tools needing to locate Python will tend to be platform specific in the first place. Can you offer any examples of 3rd party tools which could unify code in this scheme, and particularly, where this scheme would cause them to have less code, not more? >> - and also assume the executable is where >> it currently lives - one example off the top of my head is the mozilla >> "jetpack" project - see: >> >> https://github.com/mozilla/addon-sdk/blob/master/bin/activate.bat#L117 > > This code actually reinforces my point: > > First, this code would actually still work. The section ":FoundPython" > sets the PATH to "%VIRTUAL_ENV%\bin;%PYTHONINSTALL%;%PATH%" (L80), which > would still allow python.exe to be found and run. > > Second, look at that line again. Mozilla actually has edited this code > so that the jetpack uses a cross-platform "bin" convention, just as I am > proposing. I think you misunderstand the .bat file - there is no python executable in the bin directory. The bat file is locating your already installed Python and attempting to use it. > Third, one element of this proposal is that there would be a key placed > in the registry that points directly to the location of the python > executable, making locating it trivial to locate programmatically on > Windows. That sounds reasonable, but it still causes breakage, and still causes extra code for tools needing to support earlier versions. Saying "hey, it's easy to fix" are just words to someone frustrated trying to get things working with a later version of Python. Don't get me wrong - the scheme you propose is how it should have been done in the first place, no question. My issue is the breakage this will cause versus the benefit. My other questions still remain: who specifically will benefit from this, and what would be the cost to those beneficiaries of sticking with the existing scheme? The only benefit I've seen suggested is aesthetics, and while that is laudable, I don't think it is enough to justify breakage. Cheers, Mark From van.lindberg at gmail.com Thu Mar 15 00:15:55 2012 From: van.lindberg at gmail.com (VanL) Date: Wed, 14 Mar 2012 18:15:55 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F611E39.5070605@skippinet.com.au> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> Message-ID: <4F6126AB.7020309@gmail.com> On 3/14/2012 5:39 PM, Mark Hammond wrote: > Can you offer any examples of 3rd party tools which could unify code in > this scheme, and particularly, where this scheme would cause them to > have less code, not more? How about virtualenv: """ def path_locations(home_dir): """Return the path locations for the environment (where libraries are, where scripts go, etc)""" # XXX: We'd use distutils.sysconfig.get_python_inc/lib but its # prefix arg is broken: http://bugs.python.org/issue3386 if sys.platform == 'win32': [snip a bit about spaces in the path....] lib_dir = join(home_dir, 'Lib') inc_dir = join(home_dir, 'Include') bin_dir = join(home_dir, 'Scripts') if is_jython: lib_dir = join(home_dir, 'Lib') inc_dir = join(home_dir, 'Include') bin_dir = join(home_dir, 'bin') elif is_pypy: lib_dir = home_dir inc_dir = join(home_dir, 'include') bin_dir = join(home_dir, 'bin') elif sys.platform != 'win32': lib_dir = join(home_dir, 'lib', py_version) inc_dir = join(home_dir, 'include', py_version + abiflags) bin_dir = join(home_dir, 'bin') return home_dir, lib_dir, inc_dir, bin_dir """ > I think you misunderstand the .bat file - there is no python executable > in the bin directory. The bat file is locating your already installed > Python and attempting to use it. My only point here is that it would still find the already-installed Python (I think). > My other questions still remain: who specifically will benefit from > this, and what would be the cost to those beneficiaries of sticking with > the existing scheme? I will benefit, for one. My use case is that I do cross-platform development and deployment, and I occasionally want to put an entire environment in source control. Currently the case changing and Scripts/bin distinction make this a distinct pain, such that I go in and edit my Windows python installation in the way that I am describing right now. From my actual experience with this layout, pip, virtualenv, and pypm are the only three major packages that hard-code this logic and would need to be changed slightly. Thanks, Van From skippy.hammond at gmail.com Thu Mar 15 00:30:54 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Thu, 15 Mar 2012 10:30:54 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F6126AB.7020309@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> Message-ID: <4F612A2E.9030805@gmail.com> [resending - original reply went only to Van] On 15/03/2012 10:15 AM, Lindberg, Van wrote: > On 3/14/2012 5:39 PM, Mark Hammond wrote: >> Can you offer any examples of 3rd party tools which could unify code >> in this scheme, and particularly, where this scheme would cause them >> to have less code, not more? > > How about virtualenv: > > """ > def path_locations(home_dir): > """Return the path locations for the environment (where libraries are, > where scripts go, etc)""" > # XXX: We'd use distutils.sysconfig.get_python_inc/lib but its > # prefix arg is broken: http://bugs.python.org/issue3386 > if sys.platform == 'win32': > [snip a bit about spaces in the path....] > lib_dir = join(home_dir, 'Lib') > inc_dir = join(home_dir, 'Include') > bin_dir = join(home_dir, 'Scripts') > if is_jython: > lib_dir = join(home_dir, 'Lib') > inc_dir = join(home_dir, 'Include') > bin_dir = join(home_dir, 'bin') > elif is_pypy: > lib_dir = home_dir > inc_dir = join(home_dir, 'include') > bin_dir = join(home_dir, 'bin') > elif sys.platform != 'win32': > lib_dir = join(home_dir, 'lib', py_version) > inc_dir = join(home_dir, 'include', py_version + abiflags) > bin_dir = join(home_dir, 'bin') > return home_dir, lib_dir, inc_dir, bin_dir > """ So what would that look like in your scheme? I'd expect you wind up with: if sys.platform == 'win32' and sys.versioninfo < (3,4): ... existing layout else: ... new layout. So it actually ends up as slightly *more* code. > > >> I think you misunderstand the .bat file - there is no python >> executable in the bin directory. The bat file is locating your >> already installed Python and attempting to use it. > > My only point here is that it would still find the already-installed > Python (I think). I'm fairly sure it would not - it doesn't look in %PYTHONINSTALL%\bin. > >> My other questions still remain: who specifically will benefit from >> this, and what would be the cost to those beneficiaries of sticking >> with the existing scheme? > > I will benefit, for one. My use case is that I do cross-platform > development and deployment, and I occasionally want to put an entire > environment in source control. Currently the case changing and > Scripts/bin distinction make this a distinct pain, such that I go in > and edit my Windows python installation in the way that I am > describing right now. > > From my actual experience with this layout, pip, virtualenv, and > pypm are the only three major packages that hard-code this logic and > would need to be changed slightly. So why not just standardize on that new layout for virtualenvs? Mark From cs at zip.com.au Thu Mar 15 00:38:02 2012 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 15 Mar 2012 10:38:02 +1100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: <20120314233802.GA6931@cskk.homeip.net> On 13Mar2012 17:27, Guido van Rossum wrote: | I think wallclock() is an awkward name; in other contexts I've seen | "wall clock time" used to mean the time that a clock on the wall would | show, i.e. local time. This matches definition #1 of | http://www.catb.org/jargon/html/W/wall-time.html (while yours matches | #2 :-). I think this also. A "wallclock()" function that did not return real world elapsed time seconds would be misleading or at least disconcerting. | Maybe it could be called realtime()? "elapsedtime()?" It is getting a bit long though. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ "Shot my dog today." "Was he mad?" "Well, he weren't too damned pleased." - Rick Tilson, rtilson at Sun.COM From glyph at twistedmatrix.com Thu Mar 15 01:21:55 2012 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 14 Mar 2012 17:21:55 -0700 Subject: [Python-Dev] sharing sockets among processes on windows In-Reply-To: References: Message-ID: On Mar 13, 2012, at 5:27 PM, Kristj?n Valur J?nsson wrote: > Hi, > I?m interested in contributing a patch to duplicate sockets between processes on windows. > Tha api to do this is WSADuplicateSocket/WSASocket(), as already used by dup() in the _socketmodule.c > Here?s what I have: Just in case anyone is interested, we also have a ticket for this in Twisted: . It would be great to share code as much as possible. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristjan at ccpgames.com Thu Mar 15 01:28:08 2012 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Thu, 15 Mar 2012 00:28:08 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: What does "jumping forward" mean? That's what happens with every clock at every time quantum. The only effect here is that this clock will be slightly noisy, i.e. its precision becomes worse. On average it is still correct. Look at the use cases for this function 1) to enable timeouts for certaing operations, like acquiring locks: Jumping backwards is bad, because that may cause infinite wait time. But jumping forwards is ok, it may just mean that your lock times out a bit early 2) performance measurements: If you are running on a platform with a broken runtime clock, you are not likely to be running performance measurements. Really, I urge you to skip the "strict" keyword. It just adds confusion. Instead, lets just give the best monotonic clock we can do which doesn"t move backwards. Let's just provide a "practical" real time clock with high resolution that is appropriate for providing timeout functionality and so won't jump backwards for the next 20 years. Let's simply point out to people that it may not be appropriate for high precision timings on old and obsolete hardware and be done with it. K -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Nadeem Vawda Sent: 14. mars 2012 14:18 To: Matt Joiner Cc: Antoine Pitrou; Guido van Rossum; python-dev at python.org Subject: Re: [Python-Dev] Drop the new time.wallclock() function? +1 for time.steady(strict=False). On Wed, Mar 14, 2012 at 7:09 PM, Kristj?n Valur J?nsson wrote: >> - By default, it should fall back to time.time if a better source is >> ?not available, but there should be a flag that can disable this >> ?fallback for users who really *need* a monotonic/steady time source. > As pointed out on a different thread, you don"t need this "flag" since the code can easily enforce the monotonic property by maintaining a static value. > This is how we worked around buggy implementations of QueryPerformanceCounter on windows (). > K That's fine if you just need the clock to be monotonic, but it isn't any help if you also want to prevent it from jumping forward. Cheers, Nadeem _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com From victor.stinner at gmail.com Thu Mar 15 01:29:35 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 15 Mar 2012 01:29:35 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: Message-ID: <4F6137EF.9000000@gmail.com> On 14/03/2012 00:57, Victor Stinner wrote: > I added two functions to the time module in Python 3.3: wallclock() > and monotonic(). (...) I merged the two functions into one function: time.steady(strict=False). time.steady() should be monotonic most of the time, but may use a fallback. time.steady(strict=True) fails with OSError or NotImplementedError if reading the monotonic clock failed or if no monotonic clock is available. I patched the queue and threading modules to use time.steady() instead of time.time(). The documentation may need clarification. http://docs.python.org/dev/library/time.html#time.steady Victor From kristjan at ccpgames.com Thu Mar 15 01:32:17 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 15 Mar 2012 00:32:17 +0000 Subject: [Python-Dev] sharing sockets among processes on windows In-Reply-To: References: Message-ID: Great. I was about to write unittests for my patch, when I found out that I wanted to use multiprocessing to run them. So, I decided that the tests rather belonged in there rather than test_socket.py. This is where I stumbled upon code that multiprocessing uses to transfer sockets for unix. I need to read that code and understand it and see if it can be persuaded to magically work on windows too. K From: Glyph Lefkowitz [mailto:glyph at twistedmatrix.com] Sent: 14. mars 2012 17:22 To: Kristj?n Valur J?nsson Cc: Python-Dev (python-dev at python.org) Subject: Re: [Python-Dev] sharing sockets among processes on windows On Mar 13, 2012, at 5:27 PM, Kristj?n Valur J?nsson wrote: Hi, I?m interested in contributing a patch to duplicate sockets between processes on windows. Tha api to do this is WSADuplicateSocket/WSASocket(), as already used by dup() in the _socketmodule.c Here?s what I have: Just in case anyone is interested, we also have a ticket for this in Twisted: . It would be great to share code as much as possible. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristjan at ccpgames.com Thu Mar 15 01:40:55 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 15 Mar 2012 00:40:55 +0000 Subject: [Python-Dev] SocketServer issues In-Reply-To: References: <20120314100226.04627743@pitrou.net> <20120314182307.4e4e9c86@pitrou.net> Message-ID: Fyi: http://bugs.python.org/issue14307 -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Kristj?n Valur J?nsson Sent: 14. mars 2012 12:36 To: Guido van Rossum Cc: Antoine Pitrou; python-dev at python.org Subject: Re: [Python-Dev] SocketServer issues Yes, setting a timeout and leaving it that way is not the same. But setting the timeout for _accept only_ is the "same" except one approach requires the check of a bool return, the other the handling of a socket.timeout exeption. My point is, if sockets already have nice and well defined timeout semantics, why not use them, or even improve them (perhaps with an optional timeout parameter to the accept call) rather than reimplement them with an explicit select.select() call? Anyway, I'll take another look at the problem and possibly submit a patch suggestion. Thanks. K -----Original Message----- From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum Sent: 14. mars 2012 11:44 To: Kristj?n Valur J?nsson Cc: Antoine Pitrou; python-dev at python.org Subject: Re: [Python-Dev] SocketServer issues 2012/3/14 Kristj?n Valur J?nsson : > Maybe this is all just nonsense, still it seems odd to jump through extra hoops to emulate a functionality that is already supported by the socket spec, and can be done in the most appropriate way for each implementation. I thought I had already explained why setting the timeout on the socket is not the same. -- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com From victor.stinner at gmail.com Thu Mar 15 01:56:40 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 15 Mar 2012 01:56:40 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <4F6137EF.9000000@gmail.com> References: <4F6137EF.9000000@gmail.com> Message-ID: > I merged the two functions into one function: time.steady(strict=False). I opened the issue #14309 to deprecate time.clock(): http://bugs.python.org/issue14309 time.clock() is a different clock type depending on the OS (Windows vs UNIX) and so is confusing. You should now decide between time.time() and time.steady(). time.time(): - known starting point, Epoch (1970.1.1) - may be update by the system (and so go backward or forward) => display time to the user, compare/set file modification time time.steady(): - unknown starting point - more accurate than time.time() - should be monotonic (use strict=True if you want to be sure ;-)) => benchmark, timeout Victor From kristjan at ccpgames.com Thu Mar 15 02:38:41 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 15 Mar 2012 01:38:41 +0000 Subject: [Python-Dev] [Python-checkins] cpython: PEP 417: Adding unittest.mock In-Reply-To: <5BA5E20A-C4B8-4A2A-B38B-A1286FC65247@voidspace.org.uk> References: <4F60FAB2.7020207@udel.edu> <03503024-ECFB-462F-BF08-022BF93578D0@voidspace.org.uk> <5BA5E20A-C4B8-4A2A-B38B-A1286FC65247@voidspace.org.uk> Message-ID: Fyi: http://bugs.python.org/issue14310 -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Michael Foord Sent: 14. mars 2012 14:42 To: Terry Reedy Cc: python-dev at python.org Subject: Re: [Python-Dev] [Python-checkins] cpython: PEP 417: Adding unittest.mock On 14 Mar 2012, at 13:46, Terry Reedy wrote: > On 3/14/2012 4:22 PM, Michael Foord wrote: >> >> On 14 Mar 2012, at 13:08, Terry Reedy wrote: >> >>> On 3/14/2012 3:25 PM, michael.foord wrote: >>>> +# mock.py +# Test tools for mocking and patching. > >>> Should there be a note here about restrictions on editing this >>> file? I notice that there are things like >>> >>>> +class OldStyleClass: + pass +ClassType = type(OldStyleClass) >>> >>> which are only present for running under Py2 and which would >>> normally be removed for Py3. >> >> >> Yeah, I removed as much of the Python 2 compatibility code and >> thought I'd got it all. Thanks for pointing it out. > > 2000 lines is a lot to check through. >> >> I'm maintaining a "clean" (no Python 2 compatibility code) version in >> the standard library. > > Great. Here is something else, which is why I thought otherwise ;-). > > +def _instance_callable(obj): > + """Given an object, return True if the object is callable. > + For classes, return True if instances would be callable.""" > + if not isinstance(obj, type): > + # already an instance > + return getattr(obj, '__call__', None) is not None > + > + klass = obj > + # uses __bases__ instead of __mro__ so that we work with > >>> old style classes > + if klass.__dict__.get('__call__') is not None: > + return True > + > + for base in klass.__bases__: > + if _instance_callable(base): > + return True > + return False > > If you want to leave the code as is, remove or revise the comment. Thanks very much for finding these, I'm pretty sure I've fixed all the ones you reported - and one more case where try...except...finally can now be used. All the best, Michael Foord > >> I'll be maintaining mock, so I'd like to be >> assigned any issues on it and at least talked to before changes are >> made. I am maintaining a backport still, but the Python standard >> library version is the canonical version. > > Add unittest.mock to devguide/experts.rst and yourself with * appended. > > --- > Searching for 'old', I also found > > +def _must_skip(spec, entry, is_type): > + if not isinstance(spec, type): > + if entry in getattr(spec, '__dict__', {}): > + # instance attribute - shouldn't skip > + return False > >>>+ # can't use type because of old style classes > + spec = spec.__class__ > + if not hasattr(spec, '__mro__'): > >>>+ # old style class: can't have descriptors anyway > + return is_type > > In testcallable.py > + def test_patch_spec_callable_class(self): > + class CallableX(X): > + def __call__(self): > + pass > + > + class Sub(CallableX): > + pass > + > + class Multi(SomeClass, Sub): > + pass > + > >>>+ class OldStyle: > + def __call__(self): > + pass > + > >>>+ class OldStyleSub(OldStyle): > + pass > + > + for arg in 'spec', 'spec_set': > >>>+ for Klass in CallableX, Sub, Multi, OldStyle, OldStyleSub: > > This is the last. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com From anacrolix at gmail.com Thu Mar 15 02:58:50 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 15 Mar 2012 09:58:50 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: Victor, I think that steady can always be monotonic, there are time sources enough to ensure this on the platforms I am aware of. Strict in this sense refers to not being adjusted forward, i.e. CLOCK_MONOTONIC vs CLOCK_MONOTONIC_RAW. Non monotonicity of this call should be considered a bug. Strict would be used for profiling where forward leaps would disqualify the timing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Thu Mar 15 04:46:49 2012 From: brian at python.org (Brian Curtin) Date: Wed, 14 Mar 2012 22:46:49 -0500 Subject: [Python-Dev] 2012 Language Summit Report (updated, included here) Message-ID: After a few comments and corrections, including one to post the report directly here...what follows below is the text of what was updated on the previously linked blog post[0]. Much of the changes were to add more detail from a few people. One correction lies in the importlib discussion, in that I previously mentioned the effect on explicit relative imports. This was incorrect: it should have said *implicit* relative imports. [0] http://blog.python.org/2012/03/2012-language-summit-report.html ============================= 2012 Language Summit Report ============================= This year's Language Summit took place on Wednesday March 7 in Santa Clara, CA before the start of `PyCon 2012`_. As with previous years, in attendance were members of the various Python VMs, packagers from various Linux distributions, and members of several community projects. The Namespace PEPs ================== The summit began with a discussion on PEPs `382`_ and `402`_, with Barry Warsaw leading much of the discussion. After some discussion, the decision was ultimately deferred with what appeared to be a want for parts of both PEPs. As of Monday at the PyCon sprints, both PEPs have been rejected (see the Rejection Notice at the top of each PEP). Martin von Loewis `posted to the import-sig list`_ that a resolution has been found and Eric Smith will draft a new PEP on the ideas agreed upon there. Effectively, PEP 382 has been outright rejected, while portions of PEP 402 will be accepted. ``importlib`` Status ==================== Brett Cannon announced that there is a completed and available branch of CPython using importlib at `http://hg.python.org/sandbox/bcannon/ `_. See the ``bootstrap_importlib`` named branch. Discussion began by outlining the only real existing issue, which lies in ``stat``'ing of directories. There's a minor backwards incompatibility issue with time granularity. However, everyone agreed that it's so unlikely to be of issue that it's not a showstopper and the work can move forward. Additionally, there was an optimization made around the ``stat`` calls, which was arrived at independently by each of Brett, Antoine Pitrou, and P.J. Eby. The topic of performance came up and Brett explained that the current pure-Python implementation is around 5% slower. Thomas Wouters exclaimed that 5% slower is actually really good, especially given some recent benchmark work he was doing showing that changing compilers sometimes shows a 5% difference in startup time. There was a shared feeling that 5% slower was not something to hold up integration of the code, which pushed discussion happily along. Brett went on to explain what the bootstrapping actually looks like, even asserting that the implementation finds what could be the first *real* use of frozen modules! Guido's first response was, "you mean to tell me that after 20 years we finally found a use for freezing code?" ``importlib._bootstrap`` is a frozen module containing the necessary builtins to operate, along with some re-implementations of a small number of functions. Some of the libraries included in the frozen module are ``warnings``, ``_os`` (select code from ``posix``), and ``marshal``. Another compatibility issue was brought up, but again, was decided to be an issue unworthy of halting the progress on this issue. There's a negative level count which is not supported in ``importlib``, used in implicit relative imports, and it was agreed that it's acceptable to continue not supporting it. The future will likely result in a strip down of ``import.c``, as well as the exposure of numerous hooks as well as exposure of much of the ``importlib`` API. As for merging with the ``default`` branch, it was pretty universally agreed upon that this should happen for 3.3 and it should happen soon in order to get mileage on the implementation throughout the alpha and beta cycles. Since this will be happening shortly, Brett is going to follow-up to python-dev with some cleanup details and look for reviews. Release Schedule PEPs ===================== Discussion on PEPs `407`_ and `413`_ followed the ``importlib`` talk. Like the namespace PEP discussion, several ideas were tossed around but the group didn't arrive at any conclusion on acceptability of the PEPs. Immediately, the idea of splitting out the standard library to be on its own was resurrected, which could lend itself to both PEPs. Some questions remain, namely in where would the test suite live. Additionally, there may need to be some distinction between the tests which cover standard libraries versus the tests which cover language features. The topic of versioning came up, with three distinctions needing to be made. We would seem to need a version of the language spec, a version of the implementation, and a version of the standard library. Many commenters mentioned that these PEPs make things too complicated. Additionally, there was a question about whether there are enough users who care about either of these changes being made. Several of us stated that *we* could use the quicker releases, but with so many users being stuck on old versions for one reason or another, there was a wonder of who would take the releases. Thomas Wouters mentioned a good point about the difficulty in lining up the so-called Python "LTS" releases with other Python consumers who do similar LTS-style releases. Ubuntu and their LTS schedule was a prime example, as well as the organizations who plan releases atop something like Ubuntu. Many of the Linux distribution packagers in attendance seemed to agree. One thing that seemed to have broad agreement was that shortening the standard library turnaround time would be a good thing in terms of new contributors. Few people are interested in writing new features that might not be released for over a year -- it's just not fun. Even with bug fixes, sometimes the duration can be seen as too long, to the point where users may end up just fixing our problems from within their own code if possible. Guido went on to make a comment about how we hope to avoid the mindset some have of "my package isn't accepted until it's in the standard library". The focus continues to be on projects being hosted on PyPI, being successful out in the wild, then vetted for acceptance in the standard library after maturity of the project and its APIs. It was suggested that perhaps speeding up bug fix releases could be a good move, but we would need to check with release managers to ensure they're on board and willing to expend the effort to produce more frequent releases. As with the new feature releases, we need to be sure there's an audience to take the new bug fixes. There was also some discussion about what have previously been called "sumo" releases. Given that some similar releases are already made by third-party vendors, the idea didn't seem to gain much traction. Funding from the Python Software Foundation =========================================== PSF Chairman Steve Holden joined the group after lunch to mention that the foundation has resources available to assist development efforts, especially given the sponsorship success of this year's conference. While the foundation can't and won't dictate what should be coded up, they're open to proposals about the types of work to be funded. Steve and Jesse Noller were adamant about the support not only being for all Python implementations, but also for third-party projects. What's needed to begin funding for a project is a concrete proposal on what will be accomplished. They stressed that the money is ready and waiting -- proposals are the way to unlock it. Some ideas for how to use the funding came from Steve but also from around the room. One idea which started off the discussion was the idea of funding one-month sabbaticals. Then comes the issue of who might be available. Some suggested that freelance consultants in the development community might be the ones we should try to engage. Those with full-time employment may find it harder to acquire such a sabbatical, but the possibility is open to anyone. Another thought was potential funding of someone to do spurts of full-time effort on the bug tracker, ideally someone already involved in the triage effort. This type of funding would hope to put an end to the times when it takes three days to fix a bug and three years for the patch to be accepted. Some thought this might be a nice idea in the short term, but it could be tough work and burn out the individual(s) involved. If anyone is up for it, they're encouraged to propose the idea to the foundation. Along similar lines of tracker maintenance, Glyph Lefkowitz of the Twisted project had an idea to fund code reviews over code-writing efforts. Some thought this might be a good way to push forward the ``regex``/``re`` situation, given that the ``regex`` is very large and most felt that the only thing holding it back from some form of inclusion is an in-depth review. The ``cdecimal`` module was mentioned as another project that could use some review assistance. The code review funding is also an idea to push forward some third-party project's ports to Python 3, specifically including Twisted, which the group felt was an effort which should receive some of this funding. Along the way it was remarked that the `core-mentors`_ group has been a success in involving new contributors. Kudos to those involved with that list. ``virtualenv`` Inclusion ======================== In about two minutes, discussion on PEP `405`_ came and went. Carl Meyer mentioned that a reference implementation is available and is working pretty well. A look from the OSX maintainers would be beneficial, and both Ned Deily and Ronald Oussoren were in attendance. It seemed like one of the only things left in terms of the PEP was to find someone to make a declaration on it, and Thomas Wouters put his name out there if Nick Coghlan wasn't going to do it (update: Nick will be the PEP czar). PEP 397 Inclusion ================= Without much of a Windows representation at the summit, discussion was fairly quick, but it was pretty much agreed that PEP `397`_ was something we should accept. Brian Curtin spoke in favor of the PEP, as well as mentioning ongoing work on the Windows installer to optionally add the executable's directory to the Path. After discussion outside of the summit, it was additionally agreed upon that the launcher should be installed via the 3.3 Windows installer, while it can also live as a standalone installer for those not taking 3.3. Additionally, there needs to be some work done on the PEP to remove much of the low-level detail that is coupled too tightly with the implementation, e.g., explaining of the location of the ``py.ini`` file. speed.python.org ================ After generous hardware donations, the http://speed.python.org site has gone live and is currently running PyPy benchmarks. We need to make a decision on what benchmarks can be used as well as what benchmarks *should* be used when it comes to creating a Python 3 suite. As we get implementations on Python 3 we'll want to scale back 2.7 testing and push forward with 3.x. The project suffers not from a technological problem but from a personnel problem, which was thought to be another area that funding could be used for. However, even if money is on the table, we still need to find someone with the time, the know-how, and the drive to complete the task. Ideally the starting task would be to get PyPy and CPython implementations running and comparing. After that, there are a number of infrastructure tasks in line. PEP 411 Inclusion ================= PEP `411`_ proposes the inclusion of provisional packages into the standard library. The recently discussed ``regex`` and ``ipaddr`` modules were used as examples of libraries to include under this PEP. As for how this inclusion should be implemented and denoted to users was the major discussion point. It was first suggested that documentation notes don't work -- we can't rely only on documentation to be the single notification point, especially for this type of code inclusion. Other thoughts were some type of flag on the library to specify its experimental status. Another thought was to emit a warning on import of a provisional library, but it's another thing that we'd likely want to silence by default in order to not affect user code in the hopes that developers are running their test suite with warnings enabled. However, as with other times we've gone down this path, we run the risk of developers just disabling warnings all together if they become annoying. As has been suggested on python-dev, importing a provisional library from a special package, e.g., ``from __experimental__ import foo``, was pretty strongly discouraged. If the library gains a consistent API, it penalizes users once it moves from provisional status to being officially accepted. Aliasing just exacerbates the problem. The PEP boils down to being about process, and we need to be sure that libraries being included use the ability to change APIs very carefully. We also need to make people, especially the library author, aware of the need to be responsive to feedback and open to change as the code reaches a wider audience. Looking back, Jesse Noller suggested ``multiprocessing`` would have been a good candidate for something like this PEP is suggesting. Around this time, it was suggested that Michael Foord's `mock`_ could gain some provisional inclusion within ``unittest``, perhaps as ``unittest.mock``. Instead, given ``mock``'s stable API and wide use among us, along with the need for a mocking library within our own test suite, it was agreed to just accept it directly into the standard library without any provisional status. While on the topic of ``regex``'s role within the PEP came an idea from Thomas Wouters that ``regex`` be introduced into the standard library, bypassing any provisional status. From there, the previously known ``re`` module could be moved to the ``sre`` name, and there didn't appear to be any dissenting opinion there. It should also be noted to users of provisional libraries that the library maintainers would need to exercise extreme care and be very conservative in changing of the APIs. The last thing we want to do is introduce a good library but as a moving target to its users. Keyword Arguments on all builtin functions ========================================== As recently came up on the tracker, it was suggested that wider use of keyword arguments in our APIs would likely be a good thing. Gregory P. Smith suggested that we leave single-argument APIs alone, which was agreed upon. However, the overall change got some push back as "change for change's sake". In order to support this, the ``PyArg_ParseTuple`` function would need to do more work, and it's already known to be somewhat slow. Alternatively, ``PyArg_Parse`` is much faster, and the tuple version could take a thing or two from it regardless of any wide scale change to builtins. There does exist some potential break in compatibility when replacing a builtin function with a Python one, where positional-only arguments suddenly get a potentially conflicting name. It was widely agreed upon that we should avoid any blanket rules and keep changes to places where it makes sense rather than make wholesale changes. We also need to be mindful of documentation and doc strings being kept to match the actual keyword argument names as well as keep them in sync. OrderedDict was suggested as the container for keyword arguments, but Guido and Gregory were unsure of use-cases for that. Whether or not we use a traditional or ordered dictionary, it was suggested that we could possibly use a decorator to handle some of this. We could even go as far as exposing something like ``PyArg_ParseTuple`` as a Python-level function. PEP `362`_, a proposal for a function signature object, would help here and with decorators in general. It seems that all that's left with that PEP is another look and someone to declare on it. Porting to Python 3 =================== We moved on to talk about Python 3 porting, starting with the current strategies and how they're working out. Single-codebase porting is working better than expected for most of us, although ``except`` handling is a bit messy when supporting versions like 2.4. Having a lot of options, from 3to2 to 2to3, then the single codebase through parallel trees, is a really good thing. However, it's hard for us to choose a strategy for projects, so we don't, which is why most documentation tries to lay numerous strategies out there. It was suggested that documentation could stand to gain more examples of real-world porting examples, ideally pointing to changesets of these projects. The thought of our porting documentation gaining a cookbook-style approach seemed to get some agreement as a good idea. Hash Randomization ================== Release candidates are available to all branches receiving security fixes, and in the meantime, David Malcolm found and reported a security issue in the upstream ``expat`` project. However, since the upstream fix includes many other fixes at the same time, we should pick up only the security fix at this time and leave the bug fixes for the next bug fix release of the relevant branches. New ``dict`` Implementation =========================== Since the implementation makes sense and the tests pass, it was quickly agreed upon that Mark Shannon's PEP `412`_ should be accepted. As with other changes agreed upon in this summit, we'd like for the change to be pushed soon in order to get mileage on it throughout the alpha and beta cycles. With this acceptance comes commit access for Mark so that he can maintain the code. It was also remarked that the only user-visible difference that this implementation brings is a difference in sort ordering, but the recent hash randomization work makes this a moot point. New ``pickle`` Protocol ======================= PEP `3154`_, mentioned by Lukasz Langa, specifies a new pickle protocol -- version 4. Lukasz mentioned exception pickling in ``multiprocessing`` as being an issue, and Antoine solved it with this PEP. While qualified names provide some help, it was agreed upon that this PEP needs more attention. ---- If you have any questions or comments, please post to `python-dev`_. *Thanks to Eric Snow and Senthil Kumaran for contributing to this post.* .. _PyCon 2012: https://us.pycon.org/2012/ .. _362: http://www.python.org/dev/peps/pep-0362/ .. _382: http://www.python.org/dev/peps/pep-0382/ .. _397: http://www.python.org/dev/peps/pep-0397/ .. _402: http://www.python.org/dev/peps/pep-0402/ .. _405: http://www.python.org/dev/peps/pep-0405/ .. _407: http://www.python.org/dev/peps/pep-0407/ .. _411: http://www.python.org/dev/peps/pep-0411/ .. _412: http://www.python.org/dev/peps/pep-0412/ .. _413: http://www.python.org/dev/peps/pep-0413/ .. _3154: http://www.python.org/dev/peps/pep-3154/ .. _posted to the import-sig list: http://mail.python.org/pipermail/import-sig/2012-March/000421.html .. _core-mentors: http://pythonmentors.com/ .. _mock: http://www.voidspace.org.uk/python/mock/ .. _python-dev: http://mail.python.org/mailman/listinfo/python-dev From regebro at gmail.com Thu Mar 15 05:44:00 2012 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 15 Mar 2012 05:44:00 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On Thu, Mar 15, 2012 at 02:58, Matt Joiner wrote: > Victor, I think that steady can always be monotonic, there are time sources > enough to ensure this on the platforms I am aware of. Strict in this sense > refers to not being adjusted forward, i.e. CLOCK_MONOTONIC vs > CLOCK_MONOTONIC_RAW. > > Non monotonicity of this call should be considered a bug. Strict would be > used for profiling where forward leaps would disqualify the timing. This makes sense to me. //Lennart From p.f.moore at gmail.com Thu Mar 15 09:23:35 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 Mar 2012 08:23:35 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On 15 March 2012 01:58, Matt Joiner wrote: > Victor, I think that steady can always be monotonic, there are time sources > enough to ensure this on the platforms I am aware of. Strict in this sense > refers to not being adjusted forward, i.e. CLOCK_MONOTONIC vs > CLOCK_MONOTONIC_RAW. I agree - Kristj?n pointed out that you can ensure that backward jumps never occur by implementing a cache of the last value. > Non monotonicity of this call should be considered a bug. +1 > Strict would be used for profiling where forward leaps would disqualify the timing. I'm baffled as to how you even identify "forward leaps". In relation to what? A more accurate time source? I thought that by definition this was the most accurate time source we have! +1 on a simple time.steady() with guaranteed monotonicity and no flags to alter behaviour. Paul. From victor.stinner at gmail.com Thu Mar 15 10:29:26 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 15 Mar 2012 10:29:26 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: 2012/3/15 Matt Joiner : > Victor, I think that steady can always be monotonic, there are time sources > enough to ensure this on the platforms I am aware of. Strict in this sense > refers to not being adjusted forward, i.e. CLOCK_MONOTONIC vs > CLOCK_MONOTONIC_RAW. I don't think that CLOCK_MONOTONIC is available on all platforms. clock_gettime() and QueryPerformanceFrequency() can fail. In practice, it should not fail on modern OSses. But if the monotonic clock fails, Python should use another less stable clock but provide something. Otherwise, each project would have to implement its own fallback. Victor From anacrolix at gmail.com Thu Mar 15 11:06:44 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 15 Mar 2012 18:06:44 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On Mar 15, 2012 4:23 PM, "Paul Moore" wrote: > > On 15 March 2012 01:58, Matt Joiner wrote: > > Victor, I think that steady can always be monotonic, there are time sources > > enough to ensure this on the platforms I am aware of. Strict in this sense > > refers to not being adjusted forward, i.e. CLOCK_MONOTONIC vs > > CLOCK_MONOTONIC_RAW. > > I agree - Kristj?n pointed out that you can ensure that backward jumps > never occur by implementing a cache of the last value. Without knowing more, either QPC was buggy on his platform, or he didn't account for processor affinity (QPC derives from a per processor counter). > > > Non monotonicity of this call should be considered a bug. > > +1 > > > Strict would be used for profiling where forward leaps would disqualify the timing. > > I'm baffled as to how you even identify "forward leaps". In relation > to what? A more accurate time source? I thought that by definition > this was the most accurate time source we have! Monotonic clocks are not necessarily hardware based, and may be adjusted forward by NTP. > > +1 on a simple time.steady() with guaranteed monotonicity and no flags > to alter behaviour. > > Paul. I don't mind since I'll be using it for timeouts, but clearly the strongest possible guarantee should be made. If forward leaps are okay, then by definition the timer is monotonic but not steady. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Mar 15 11:31:26 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 Mar 2012 11:31:26 +0100 Subject: [Python-Dev] cpython (3.1): - rename configure.in to configure.ac References: Message-ID: <20120315113126.49db0f5c@pitrou.net> On Wed, 14 Mar 2012 23:27:24 +0100 matthias.klose wrote: > http://hg.python.org/cpython/rev/55ab7a272f0a > changeset: 75659:55ab7a272f0a > branch: 3.1 > parent: 75199:df3b2b5db900 > user: Matthias Klose > date: Wed Mar 14 23:10:15 2012 +0100 > summary: > - rename configure.in to configure.ac > - change references from configure.in to configure.ac What's the rationale for this change? There doesn't seem to be an issue number. Regards Antoine. From p.f.moore at gmail.com Thu Mar 15 12:10:20 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 Mar 2012 11:10:20 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On 15 March 2012 10:06, Matt Joiner wrote: >> I'm baffled as to how you even identify "forward leaps". In relation >> to what? A more accurate time source? I thought that by definition >> this was the most accurate time source we have! > > Monotonic clocks are not necessarily hardware based, and may be adjusted > forward by NTP. I appreciate that. But I'm still unclear how you would tell that had happened as part of the implementation. One call to the OS returns 12345. The next returns 13345. Is that because 100 ticks have passed, or because the clock "leapt forward"? With no point of reference, how can you tell? But I agree, the key thing is just to have the strongest guarantee possible. Paul. From nadeem.vawda at gmail.com Thu Mar 15 13:12:39 2012 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Thu, 15 Mar 2012 14:12:39 +0200 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On Thu, Mar 15, 2012 at 1:10 PM, Paul Moore wrote: >> Monotonic clocks are not necessarily hardware based, and may be adjusted >> forward by NTP. > > I appreciate that. But I'm still unclear how you would tell that had > happened as part of the implementation. One call to the OS returns > 12345. The next returns 13345. Is that because 100 ticks have passed, > or because the clock "leapt forward"? With no point of reference, how > can you tell? The point (AIUI) is that you *can't* identify such adjustments (in the absence of some sort of log of NTP updates), so we should provide a mechanism that is guaranteed not to be affected by them. Cheers, Nadeem From p.f.moore at gmail.com Thu Mar 15 13:20:48 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 Mar 2012 12:20:48 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On 15 March 2012 12:12, Nadeem Vawda wrote: > On Thu, Mar 15, 2012 at 1:10 PM, Paul Moore wrote: >> I appreciate that. But I'm still unclear how you would tell that had >> happened as part of the implementation. One call to the OS returns >> 12345. The next returns 13345. Is that because 100 ticks have passed, >> or because the clock "leapt forward"? With no point of reference, how >> can you tell? > > The point (AIUI) is that you *can't* identify such adjustments (in the > absence of some sort of log of NTP updates), so we should provide a > mechanism that is guaranteed not to be affected by them. OK, I see (sort of). But if that is the case, what's the use case for the variation that *is* affected by them? The use cases I've seen mentioned are timeouts and performance testing, both of which don't want to see clock adjustments. Note that when Victor started this thread, he said: > I prefer to keep only monotonic() because it is not affected by system > clock update and should help to fix issues on NTP update in functions > implementing a timeout. which seems to me to be exactly this point. So I guess I support Victor's original proposal. (Which is good, because he has thought about this issue far more than I have :-)) Paul. From doko at ubuntu.com Thu Mar 15 17:33:57 2012 From: doko at ubuntu.com (Matthias Klose) Date: Thu, 15 Mar 2012 17:33:57 +0100 Subject: [Python-Dev] cpython (3.1): - rename configure.in to configure.ac In-Reply-To: <20120315113126.49db0f5c@pitrou.net> References: <20120315113126.49db0f5c@pitrou.net> Message-ID: <4F6219F5.4060705@ubuntu.com> On 15.03.2012 11:31, Antoine Pitrou wrote: > On Wed, 14 Mar 2012 23:27:24 +0100 > matthias.klose wrote: >> http://hg.python.org/cpython/rev/55ab7a272f0a >> changeset: 75659:55ab7a272f0a >> branch: 3.1 >> parent: 75199:df3b2b5db900 >> user: Matthias Klose >> date: Wed Mar 14 23:10:15 2012 +0100 >> summary: >> - rename configure.in to configure.ac >> - change references from configure.in to configure.ac > > What's the rationale for this change? There doesn't seem to be an issue > number. autoconf files up to 2.13 have the .in extension, autoconf files for 2.50 and newer the .ac extension. This is a no change, except for having autoconf2.13 installed, and the autoconf then failing. From g.brandl at gmx.net Thu Mar 15 18:49:10 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 15 Mar 2012 18:49:10 +0100 Subject: [Python-Dev] cpython (3.1): - rename configure.in to configure.ac In-Reply-To: <4F6219F5.4060705@ubuntu.com> References: <20120315113126.49db0f5c@pitrou.net> <4F6219F5.4060705@ubuntu.com> Message-ID: On 15.03.2012 17:33, Matthias Klose wrote: > On 15.03.2012 11:31, Antoine Pitrou wrote: >> On Wed, 14 Mar 2012 23:27:24 +0100 >> matthias.klose wrote: >>> http://hg.python.org/cpython/rev/55ab7a272f0a >>> changeset: 75659:55ab7a272f0a >>> branch: 3.1 >>> parent: 75199:df3b2b5db900 >>> user: Matthias Klose >>> date: Wed Mar 14 23:10:15 2012 +0100 >>> summary: >>> - rename configure.in to configure.ac >>> - change references from configure.in to configure.ac >> >> What's the rationale for this change? There doesn't seem to be an issue >> number. > > autoconf files up to 2.13 have the .in extension, autoconf files for 2.50 and > newer the .ac extension. This is a no change, except for having autoconf2.13 > installed, and the autoconf then failing. Not sure it belongs in 3.1 though. Georg From benjamin at python.org Thu Mar 15 19:02:32 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 15 Mar 2012 13:02:32 -0500 Subject: [Python-Dev] cpython (3.1): - rename configure.in to configure.ac In-Reply-To: References: <20120315113126.49db0f5c@pitrou.net> <4F6219F5.4060705@ubuntu.com> Message-ID: 2012/3/15 Georg Brandl : > On 15.03.2012 17:33, Matthias Klose wrote: >> >> On 15.03.2012 11:31, Antoine Pitrou wrote: >>> >>> ?On Wed, 14 Mar 2012 23:27:24 +0100 >>> ?matthias.klose ? wrote: >>>> >>>> ?http://hg.python.org/cpython/rev/55ab7a272f0a >>>> ?changeset: ? 75659:55ab7a272f0a >>>> ?branch: ? ? ?3.1 >>>> ?parent: ? ? ?75199:df3b2b5db900 >>>> ?user: ? ? ? ?Matthias Klose >>>> ?date: ? ? ? ?Wed Mar 14 23:10:15 2012 +0100 >>>> ?summary: >>>> ? ?- rename configure.in to configure.ac >>>> ?- change references from configure.in to configure.ac >>> >>> >>> ?What's the rationale for this change? There doesn't seem to be an issue >>> ?number. >> >> >> autoconf files up to 2.13 have the .in extension, autoconf files for 2.50 >> and >> newer the .ac extension. ?This is a no change, except for having >> autoconf2.13 >> installed, and the autoconf then failing. > > > Not sure it belongs in 3.1 though. I told him he could change 3.1 for ease of maintenance in the future. -- Regards, Benjamin From gcolgate at gmail.com Thu Mar 15 19:47:07 2012 From: gcolgate at gmail.com (Gil Colgate) Date: Thu, 15 Mar 2012 11:47:07 -0700 Subject: [Python-Dev] What letter should an UnsignedLongLong get Message-ID: We use a lot of UnsignedLongLongs in our program (ids) and have been parsing in PyArg_ParseTuple with 'K', which does not do error checking. I am planning to add a new type to our local build of python for parsing Unsigned Long Longs (64 bit numbers) that errrors if the number has more than the correct number of bits. I am thinking to use the letter 'N' for this purpose, since l,k,K,U,u are all taken. Does anyone have any better ideas? -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Thu Mar 15 19:49:00 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 15 Mar 2012 13:49:00 -0500 Subject: [Python-Dev] What letter should an UnsignedLongLong get In-Reply-To: References: Message-ID: 2012/3/15 Gil Colgate : > We use a lot of UnsignedLongLongs in our program (ids) and have been parsing > in PyArg_ParseTuple with 'K', which does not do error checking. > I am planning to add a new type to our local build of python for parsing > Unsigned Long Longs (64 bit numbers) that errrors if the number has more > than the correct number of bits. > > I am thinking to use the letter 'N' for this purpose, since l,k,K,U,u are > all taken. Unfortunately, the would conflict with Py_BuildValue's 'N'. -- Regards, Benjamin From benjamin at python.org Thu Mar 15 19:56:01 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 15 Mar 2012 13:56:01 -0500 Subject: [Python-Dev] What letter should an UnsignedLongLong get In-Reply-To: References: Message-ID: 2012/3/15 Gil Colgate : > I must be using a different version of python, (3.2), I don't see that one > in use. Do you have a different suggestion? It's not used in PyArg_Parse*, but it is for Py_BuildValue. Adding it to PyArg_Parse could create confusion. > > On Thu, Mar 15, 2012 at 11:49 AM, Benjamin Peterson > wrote: >> >> 2012/3/15 Gil Colgate : >> > We use a lot of UnsignedLongLongs in our program (ids) and have been >> > parsing >> > in PyArg_ParseTuple with 'K', which does not do error checking. >> > I am planning to add a new type to our local build of python for parsing >> > Unsigned Long Longs (64 bit numbers) that errrors if the number has more >> > than the correct number of bits. >> > >> > I am thinking to use the letter 'N' for this purpose, since l,k,K,U,u >> > are >> > all taken. >> >> Unfortunately, the would conflict with Py_BuildValue's 'N'. >> >> >> >> -- >> Regards, >> Benjamin > > -- Regards, Benjamin From nadeem.vawda at gmail.com Thu Mar 15 20:02:44 2012 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Thu, 15 Mar 2012 21:02:44 +0200 Subject: [Python-Dev] What letter should an UnsignedLongLong get In-Reply-To: References: Message-ID: The lzma module ran into a similar issue with 32-bit unsigned ints. I worked around it by writing a custom converter function to use with the "O&" code. You can find the converter definition here: http://hg.python.org/cpython/file/default/Modules/_lzmamodule.c#l134 And an example usage here: http://hg.python.org/cpython/file/default/Modules/_lzmamodule.c#l261 Cheers, Nadeem From rowen at uw.edu Thu Mar 15 20:22:03 2012 From: rowen at uw.edu (Russell E. Owen) Date: Thu, 15 Mar 2012 12:22:03 -0700 Subject: [Python-Dev] Drop the new time.wallclock() function? References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: In article , Kristj??n Valur J??nsson wrote: > What does "jumping forward" mean? That's what happens with every clock at > every time quantum. The only effect here is that this clock will be slightly > noisy, i.e. its precision becomes worse. On average it is still correct. > Look at the use cases for this function > 1) to enable timeouts for certaing operations, like acquiring locks: > Jumping backwards is bad, because that may cause infinite wait time. But > jumping forwards is ok, it may just mean that your lock times out a bit early > 2) performance measurements: > If you are running on a platform with a broken runtime clock, you are not > likely to be running performance measurements. > > Really, I urge you to skip the "strict" keyword. It just adds confusion. > Instead, lets just give the best monotonic clock we can do which doesn"t move > backwards. > Let's just provide a "practical" real time clock with high resolution that is > appropriate for providing timeout functionality and so won't jump backwards > for the next 20 years. Let's simply point out to people that it may not be > appropriate for high precision timings on old and obsolete hardware and be > done with it. I agree. I prefer the name time.monotonic with no flags. It will suit most use cases. I think supplying truly steady time is a low level hardware function (e.g. buy a GPS timer card) with a driver. -- Russell From anacrolix at gmail.com Thu Mar 15 20:55:16 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Fri, 16 Mar 2012 03:55:16 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: +1. I now prefer time.monotonic(), no flags. It attempts to be as high precision as possible and guarantees never to jump backwards. Russell's comment is right, the only steady sources are from hardware, and these are too equipment and operating system specific. (For this call anyway). On Mar 16, 2012 3:23 AM, "Russell E. Owen" wrote: > In article > , > Kristj?n Valur J?nsson wrote: > > > What does "jumping forward" mean? That's what happens with every clock > at > > every time quantum. The only effect here is that this clock will be > slightly > > noisy, i.e. its precision becomes worse. On average it is still correct. > > Look at the use cases for this function > > 1) to enable timeouts for certaing operations, like acquiring locks: > > Jumping backwards is bad, because that may cause infinite wait > time. But > > jumping forwards is ok, it may just mean that your lock times out a bit > early > > 2) performance measurements: > > If you are running on a platform with a broken runtime clock, you > are not > > likely to be running performance measurements. > > > > Really, I urge you to skip the "strict" keyword. It just adds confusion. > > Instead, lets just give the best monotonic clock we can do which doesn"t > move > > backwards. > > Let's just provide a "practical" real time clock with high resolution > that is > > appropriate for providing timeout functionality and so won't jump > backwards > > for the next 20 years. Let's simply point out to people that it may not > be > > appropriate for high precision timings on old and obsolete hardware and > be > > done with it. > > I agree. I prefer the name time.monotonic with no flags. It will suit > most use cases. I think supplying truly steady time is a low level > hardware function (e.g. buy a GPS timer card) with a driver. > > -- Russell > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gcolgate at gmail.com Thu Mar 15 20:59:39 2012 From: gcolgate at gmail.com (Gil Colgate) Date: Thu, 15 Mar 2012 12:59:39 -0700 Subject: [Python-Dev] What letter should an UnsignedLongLong get In-Reply-To: References: Message-ID: How about 'G'? (Giant, or perhaps gynormous, integer?) Then I could also map 'g' to the signed version (same as L) for consistency. On Thu, Mar 15, 2012 at 11:49 AM, Benjamin Peterson wrote: > 2012/3/15 Gil Colgate : > > We use a lot of UnsignedLongLongs in our program (ids) and have been > parsing > > in PyArg_ParseTuple with 'K', which does not do error checking. > > I am planning to add a new type to our local build of python for parsing > > Unsigned Long Longs (64 bit numbers) that errrors if the number has more > > than the correct number of bits. > > > > I am thinking to use the letter 'N' for this purpose, since l,k,K,U,u are > > all taken. > > Unfortunately, the would conflict with Py_BuildValue's 'N'. > > > > -- > Regards, > Benjamin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Thu Mar 15 21:03:02 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 15 Mar 2012 15:03:02 -0500 Subject: [Python-Dev] What letter should an UnsignedLongLong get In-Reply-To: References: Message-ID: 2012/3/15 Gil Colgate : > How about 'G'? (Giant, or perhaps gynormous, integer?) > > > Then I could also map 'g' to the signed version (same as L) for consistency. Sounds okay to me. -- Regards, Benjamin From storchaka at gmail.com Thu Mar 15 22:02:40 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 15 Mar 2012 23:02:40 +0200 Subject: [Python-Dev] What letter should an UnsignedLongLong get In-Reply-To: References: Message-ID: 15.03.12 21:59, Gil Colgate ???????(??): > How about 'G'? (Giant, or perhaps gynormous, integer?) > > > Then I could also map 'g' to the signed version (same as L) for consistency. What about unsigned char, short, int, and long with overflow checking? From alexander.belopolsky at gmail.com Thu Mar 15 22:27:16 2012 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 15 Mar 2012 17:27:16 -0400 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: On Thu, Mar 15, 2012 at 3:55 PM, Matt Joiner wrote: > +1. I now prefer time.monotonic(), no flags. Am I alone thinking that an adjective is an odd choice for a function name? I think monotonic_clock or monotonic_time would be a better option. From storchaka at gmail.com Thu Mar 15 22:31:05 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 15 Mar 2012 23:31:05 +0200 Subject: [Python-Dev] What letter should an UnsignedLongLong get In-Reply-To: References: Message-ID: <4F625F99.6030007@gmail.com> 15.03.12 21:59, Gil Colgate ???????(??): > How about 'G'? (Giant, or perhaps gynormous, integer?) > > > Then I could also map 'g' to the signed version (same as L) for consistency. For consistency 'g' must be `unsigned long` with overflow checking. And how about 'M'? 'K', 'L', and 'M' are neighboring letters. From carl at oddbird.net Thu Mar 15 22:43:20 2012 From: carl at oddbird.net (Carl Meyer) Date: Thu, 15 Mar 2012 14:43:20 -0700 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status Message-ID: <4F626278.7030701@oddbird.net> A brief status update on PEP 405 (built-in virtualenv) and the open issues: 1. As mentioned in the updated version of the language summit notes, Nick Coghlan has agreed to pronounce on the PEP. 2. Ned Deily discovered at the PyCon sprints that the current reference implementation does not work with an OS X framework build of Python. We're still working to discover the reason for that and determine possible fixes. 3. If anyone knows of a pair of packages in which both need to build compiled extensions, and the compilation of the second depends on header files from the first, that would be helpful to me in testing the other open issue (installation of header files). (I thought numpy and scipy might fit this bill, but I'm currently not able to install numpy at all under Python 3 using pysetup, easy_install, or pip.) Thanks, Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From van.lindberg at gmail.com Thu Mar 15 22:57:10 2012 From: van.lindberg at gmail.com (VanL) Date: Thu, 15 Mar 2012 16:57:10 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F612A2E.9030805@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> Message-ID: On 3/14/2012 6:30 PM, Mark Hammond wrote: > > So why not just standardize on that new layout for virtualenvs? That sounds like the worst of all worlds - keep all the existing special cases, and add one. The fact is that most code doesn't know about this, only installers or virtual environments. Most code just assumes that distutils does its thing correctly and that binaries are installed... wherever the binaries go. Again, I have experience with this, as I have edited my own install to do this for a couple of years. The breakage is minimal and it makes things much more consistent and easier to use for cross-platform development. Right now we are in front of the knee on major 3.x adoption - I would like to have things be standardized going forth everywhere. Thanks, Van From carl at oddbird.net Thu Mar 15 23:11:58 2012 From: carl at oddbird.net (Carl Meyer) Date: Thu, 15 Mar 2012 15:11:58 -0700 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status In-Reply-To: <4F626712.3030906@gmail.com> References: <4F626278.7030701@oddbird.net> <4F626712.3030906@gmail.com> Message-ID: <4F62692E.8040203@oddbird.net> On 03/15/2012 03:02 PM, Lindberg, Van wrote: > FYI, the location of the tcl/tk libraries does not appear to be set in > the virtualenv, even if tkinter is installed and working in the main > Python installation. As a result, tk-based apps will not run from a > virtualenv. Thanks for the report! I've added this to the list of open issues in the PEP and I'll look into it. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From tjreedy at udel.edu Thu Mar 15 23:45:25 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 15 Mar 2012 18:45:25 -0400 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: On 3/15/2012 5:27 PM, Alexander Belopolsky wrote: > On Thu, Mar 15, 2012 at 3:55 PM, Matt Joiner wrote: >> +1. I now prefer time.monotonic(), no flags. > > Am I alone thinking that an adjective is an odd choice for a function > name? I would normally agree, but in this case, it is a function of a module whose short name names what the adjective is modifying. I expect that this will normally be called with the module name. > I think monotonic_clock or monotonic_time would be a better option. time.monotonic_time seems redundant. -- Terry Jan Reedy From steve at pearwood.info Fri Mar 16 00:18:44 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 16 Mar 2012 10:18:44 +1100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> Message-ID: <4F6278D4.4040002@pearwood.info> Terry Reedy wrote: > On 3/15/2012 5:27 PM, Alexander Belopolsky wrote: >> On Thu, Mar 15, 2012 at 3:55 PM, Matt Joiner wrote: >>> +1. I now prefer time.monotonic(), no flags. >> >> Am I alone thinking that an adjective is an odd choice for a function >> name? > > I would normally agree, but in this case, it is a function of a module > whose short name names what the adjective is modifying. I expect that > this will normally be called with the module name. > >> I think monotonic_clock or monotonic_time would be a better option. > > time.monotonic_time seems redundant. Agreed. Same applies to "steady_time", and "steady" on its own is weird. Steady what? While we're bike-shedding, I'll toss in another alternative. Early Apple Macintoshes had a system function that returned the time since last reboot measured in 1/60th of a second, called "the ticks". If I have understood correctly, the monotonic timer will have similar properties: guaranteed monotonic, as accurate as the hardware can provide, but not directly translatable to real (wall-clock) time. (Wall clocks sometimes go backwards.) The two functions are not quite identical: Mac "ticks" were 32-bit integers, not floating point numbers. But the use-cases seem to be the same. time.ticks() seems right as a name to me. It suggests a steady heartbeat ticking along, without making any suggestion that it returns "the time". -- Steven From skippy.hammond at gmail.com Fri Mar 16 00:19:50 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Fri, 16 Mar 2012 10:19:50 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> Message-ID: <4F627916.7060705@gmail.com> On 16/03/2012 8:57 AM, VanL wrote: > On 3/14/2012 6:30 PM, Mark Hammond wrote: >> >> So why not just standardize on that new layout for virtualenvs? > > That sounds like the worst of all worlds - keep all the existing special > cases, and add one. I'm not so sure. My concern is that this *will* break external tools that attempt to locate the python executable from an installed directory. However, I'm not sure this requirement exists for virtual environments - such tools probably will not attempt to locate the executable in a virtual env as there is no standard place for a virtual env to live. So having a standard layout in the virtual envs still seems a win - we keep the inconsistency in the layout of the "installed" Python, but tools which work with virtualenvs still get a standardized layout. [At least I think that is your proposal - can you confirm that the directory layouts in your proposal exactly match the directory layouts in virtual envs on all other platforms? ie, that inconsistencies like the python{py_version_short} suffix will not remain?] Just to be completely clear, my current concern is only with the location of the executable in an installed Python. > The fact is that most code doesn't know about this, only installers or > virtual environments. Most code just assumes that distutils does its > thing correctly and that binaries are installed... wherever the binaries > go. Of course - but this raises 2 points: * I'm referring to *external* tools that launch Python. They obviously need to know where the binaries are to launch them. Eg, the PEP397 launcher; the (admittedly few) people who use the launcher would need to upgrade it to work under your scheme. Ditto *all* other such tools that locate and launch Python. * "most code" isn't a high enough bar. If we only considered such anecdotes, most backwards compatibility concerns would be moot. > Again, I have experience with this, as I have edited my own install to > do this for a couple of years. The breakage is minimal and it makes > things much more consistent and easier to use for cross-platform > development. All due respect, but I'm not sure that is a large enough sample to draw any conclusions from. I've offered 2 concrete examples of things that *will* break and I haven't looked for others. Also, I'm still yet to see what exactly becomes "easier" in your model? As you mention, most Python code will not care; distutils and other parts of the stdlib will "do the right thing" - and indeed, already do for Windows. So the proposal wants to change distutils and other parts of the stdlib even though "most code" won't notice. But the code that will notice will be broken! So I dispute it is "easier" for anyone; I agree it is more consistent, but given the *certainty* external tools will break, I refer to you the Zen of Python's thoughts on consistency ;) > Right now we are in front of the knee on major 3.x adoption - I would > like to have things be standardized going forth everywhere. It is a shame this wasn't done as part of py3k in the first place. But I assume you would be looking at Python 3.4 for this, right? So if people start working with Python 3.3 now and finds this change in 3.4, we are still asking them to take the burden of supporting the multiple locations. I guess I'd be less concerned if we managed to get it into 3.3 and also recommended to people that they should ignore 3.2 and earlier when porting their tools/libraries to 3.x. I think I've made all the points I can make in this discussion. As I mentioned at the start, I'm not quite -1 on the idea, so I'm not going to push this barrow any further (although I'm obviously happy to clarify anything I've said...) Cheers, Mark From carl at oddbird.net Fri Mar 16 00:48:30 2012 From: carl at oddbird.net (Carl Meyer) Date: Thu, 15 Mar 2012 16:48:30 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F627916.7060705@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> Message-ID: <4F627FCE.4040309@oddbird.net> On 03/15/2012 04:19 PM, Mark Hammond wrote: > On 16/03/2012 8:57 AM, VanL wrote: >> On 3/14/2012 6:30 PM, Mark Hammond wrote: >>> >>> So why not just standardize on that new layout for virtualenvs? >> >> That sounds like the worst of all worlds - keep all the existing special >> cases, and add one. > > I'm not so sure. My concern is that this *will* break external tools > that attempt to locate the python executable from an installed > directory. However, I'm not sure this requirement exists for virtual > environments - such tools probably will not attempt to locate the > executable in a virtual env as there is no standard place for a virtual > env to live. > > So having a standard layout in the virtual envs still seems a win - we > keep the inconsistency in the layout of the "installed" Python, but > tools which work with virtualenvs still get a standardized layout. The implementation of virtualenv (and especially PEP 405 pyvenv) are largely based around making sure that the internal layout of a virtualenv is identical to the layout of an installed Python on that same platform, to avoid any need to special-case virtualenvs in distutils. The one exception to this is the location of the python binary itself in Windows virtualenvs; we do place it inside Scripts\ so that the virtualenv can be "activated" by adding only a single path to the shell PATH. But I would be opposed to any additional special-casing of the internal layout of a virtualenv that would require tools installing software inside virtualenv to use a different install scheme than when installing to a system Python. In other words, I would much rather that tools have to understand a different layout between Windows virtualenvs and Unixy virtualenvs (because most tools don't have to care anyway, distutils just takes care of it, and to the extent they do have to care, they have to adjust anyway in order to work with installed Pythons) than that they have to understand a different layout between virtualenv and non- on the same platform. To as great an extent as possible, tools shouldn't have to care whether they are dealing with a virtualenv. A consistent layout all around would certainly be nice, but I'm not venturing any opinion on whether it's worth the backwards incompatibility. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From skippy.hammond at gmail.com Fri Mar 16 01:10:08 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Fri, 16 Mar 2012 11:10:08 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F627FCE.4040309@oddbird.net> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> Message-ID: <4F6284E0.1090606@gmail.com> On 16/03/2012 10:48 AM, Carl Meyer wrote: ... > The implementation of virtualenv (and especially PEP 405 pyvenv) are > largely based around making sure that the internal layout of a > virtualenv is identical to the layout of an installed Python on that > same platform, to avoid any need to special-case virtualenvs in > distutils. The one exception to this is the location of the python > binary itself in Windows virtualenvs; we do place it inside Scripts\ so > that the virtualenv can be "activated" by adding only a single path to > the shell PATH. But I would be opposed to any additional special-casing > of the internal layout of a virtualenv ... Unless I misunderstand, that sounds like it should keep everyone happy; there already is a special case for the executable on Windows being in a different place in an installed layout vs a virtual-env layout. Changing this to use "bin" instead of "Scripts" makes the virtualenv more consistent across platforms and doesn't impose any additional special-casing for Windows (just slightly changes the existing special-case :) Thanks, Mark From carl at oddbird.net Fri Mar 16 01:12:54 2012 From: carl at oddbird.net (Carl Meyer) Date: Thu, 15 Mar 2012 17:12:54 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F6284E0.1090606@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> Message-ID: <4F628586.8030308@oddbird.net> On 03/15/2012 05:10 PM, Mark Hammond wrote: > On 16/03/2012 10:48 AM, Carl Meyer wrote: >> The implementation of virtualenv (and especially PEP 405 pyvenv) are >> largely based around making sure that the internal layout of a >> virtualenv is identical to the layout of an installed Python on that >> same platform, to avoid any need to special-case virtualenvs in >> distutils. The one exception to this is the location of the python >> binary itself in Windows virtualenvs; we do place it inside Scripts\ so >> that the virtualenv can be "activated" by adding only a single path to >> the shell PATH. But I would be opposed to any additional special-casing >> of the internal layout of a virtualenv > ... > > Unless I misunderstand, that sounds like it should keep everyone happy; > there already is a special case for the executable on Windows being in a > different place in an installed layout vs a virtual-env layout. Changing > this to use "bin" instead of "Scripts" makes the virtualenv more > consistent across platforms and doesn't impose any additional > special-casing for Windows (just slightly changes the existing > special-case :) Changing the directory name is in fact a new and different (and much more invasive) special case, because distutils et al install scripts there, and that directory name is part of the distutils install scheme. Installers don't care where the Python binary is located, so moving it in with the other scripts has very little impact. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From anacrolix at gmail.com Fri Mar 16 01:50:09 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Fri, 16 Mar 2012 08:50:09 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <4F6278D4.4040002@pearwood.info> References: <20120314101618.5cf1850f@pitrou.net> <20120314132747.6c857a53@pitrou.net> <4F6278D4.4040002@pearwood.info> Message-ID: Windows also has this albeit course grained and also 32 bit. I don't think ticks reflects the reason why using the timer is desirable. monotonic_time seems reasonable, there's no reason to persist short names when users can import it how they like. On Mar 16, 2012 7:20 AM, "Steven D'Aprano" wrote: > Terry Reedy wrote: > >> On 3/15/2012 5:27 PM, Alexander Belopolsky wrote: >> >>> On Thu, Mar 15, 2012 at 3:55 PM, Matt Joiner >>> wrote: >>> >>>> +1. I now prefer time.monotonic(), no flags. >>>> >>> >>> Am I alone thinking that an adjective is an odd choice for a function >>> name? >>> >> >> I would normally agree, but in this case, it is a function of a module >> whose short name names what the adjective is modifying. I expect that this >> will normally be called with the module name. >> >> I think monotonic_clock or monotonic_time would be a better option. >>> >> >> time.monotonic_time seems redundant. >> > > Agreed. Same applies to "steady_time", and "steady" on its own is weird. > Steady what? > > While we're bike-shedding, I'll toss in another alternative. Early Apple > Macintoshes had a system function that returned the time since last reboot > measured in 1/60th of a second, called "the ticks". > > If I have understood correctly, the monotonic timer will have similar > properties: guaranteed monotonic, as accurate as the hardware can provide, > but not directly translatable to real (wall-clock) time. (Wall clocks > sometimes go backwards.) > > The two functions are not quite identical: Mac "ticks" were 32-bit > integers, not floating point numbers. But the use-cases seem to be the same. > > time.ticks() seems right as a name to me. It suggests a steady heartbeat > ticking along, without making any suggestion that it returns "the time". > > > > -- > Steven > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > anacrolix%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri Mar 16 05:48:18 2012 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 16 Mar 2012 00:48:18 -0400 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/13/2012 06:49 PM, Nick Coghlan wrote: > On Wed, Mar 14, 2012 at 8:08 AM, Guido van Rossum > wrote: >> If you can solve your problem with a suitably hacked Unpickler >> subclass that's fine with me, but I would personally use this >> opportunity to change the app to some other serialization format >> that is perhaps less general but more robust than pickle. I've been >> bitten by too many pickle-related problems to recommend pickle to >> anyone... > > It's fine for in-memory storage of (almost) arbitrary objects (I use > it to stash things in a memory backed sqlite DB via SQLAlchemy) and > for IPC, but yeah, for long-term cross-version persistent storage, > I'd be looking to something like JSON rather than pickle. Note the Zope ecosystem (including Plone) is an *enoromous* installed base[1] using pickle for storage of data over many years and multiple versions of Python: until this point, it has always been possible to arrange for old pickles to work (e.g., by providing aliases for missing module names, etc.). ]1] tens of thousands of Zope-based sites in production, including very high-profile ones: http://plone.org/support/sites Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9ixhEACgkQ+gerLs4ltQ7hUwCfSdjbGnIIrNr6sxoztvb3pvx5 Ns0An1GmcYHClvsgx22bdru5Hl+G09nx =sm0/ -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Mar 16 05:55:45 2012 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 16 Mar 2012 00:55:45 -0400 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status In-Reply-To: <4F626278.7030701@oddbird.net> References: <4F626278.7030701@oddbird.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/15/2012 05:43 PM, Carl Meyer wrote: > A brief status update on PEP 405 (built-in virtualenv) and the open > issues: > > 1. As mentioned in the updated version of the language summit notes, > Nick Coghlan has agreed to pronounce on the PEP. > > 2. Ned Deily discovered at the PyCon sprints that the current > reference implementation does not work with an OS X framework build of > Python. We're still working to discover the reason for that and > determine possible fixes. > > 3. If anyone knows of a pair of packages in which both need to build > compiled extensions, and the compilation of the second depends on > header files from the first, that would be helpful to me in testing > the other open issue (installation of header files). (I thought numpy > and scipy might fit this bill, but I'm currently not able to install > numpy at all under Python 3 using pysetup, easy_install, or pip.) ExtensionClass and Acquisition would fit the bill, except they aren't ported to Python3 (Acquisition needs the headers from ExtensionClass). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9ix9EACgkQ+gerLs4ltQ5HsgCdEFbb0utGPbBJ2059+KBbhkIB M2IAnjFNoJh1UKB76k6nd6nTMfo78s3Z =T6fh -----END PGP SIGNATURE----- From eliben at gmail.com Fri Mar 16 08:38:49 2012 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 16 Mar 2012 09:38:49 +0200 Subject: [Python-Dev] Raising assertions on wrong element types in ElementTree Message-ID: Hi, [Terry suggested in http://bugs.python.org/issue13782 to raise this dilemma to python-dev. I concur.] The Element class in ElementTree (http://docs.python.org/py3k/library/xml.etree.elementtree.html) has some methods for adding new children: append, insert and extend. Currently the documentation states that extend raises AssertionError when something that's not an Element is being passed to it, and the others don't mention mention this case. There are a number of problems with this: 1. The behavior of append, insert and extend should be similar in this respect 2. AssertionError is not the customary error in such case - TypeError is much more suitable 3. The C implementation of ElementTree actually raises TypeError in all these methods, by virtue of using PyArg_ParseTuple 4. The Python implementation (at least in 3.2) actually doesn't raise even AssertionError in extend - this was commented out The suggestion for 3.3 (where compatibility between the C and Python implementations gets even more important, since the C one is now being imported by default when available) is to raise TypeError in all 3 methods in the Python implementation, to match the C implementation, and to modify the documentation accordingly. There may appear to be a backwards compatibility here, since the doc of extend mentions raising AssertionError - but as said above, the doc is wrong, so no regressions in the code are be expected. Does that sound reasonable (for 3.3)? Does it make sense to also fix this in 3.2/2.7? Or fix only the documentation? Or not touch them at all? Thanks in advance, Eli From stefan_ml at behnel.de Fri Mar 16 08:51:07 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 16 Mar 2012 08:51:07 +0100 Subject: [Python-Dev] Raising assertions on wrong element types in ElementTree In-Reply-To: References: Message-ID: Eli Bendersky, 16.03.2012 08:38: > The Element class in ElementTree > (http://docs.python.org/py3k/library/xml.etree.elementtree.html) has > some methods for adding new children: append, insert and extend. > Currently the documentation states that extend raises AssertionError > when something that's not an Element is being passed to it, and the > others don't mention mention this case. AssertionError is clearly the wrong thing to raise for user input. > There are a number of problems with this: > > 1. The behavior of append, insert and extend should be similar in this respect > 2. AssertionError is not the customary error in such case - TypeError > is much more suitable > 3. The C implementation of ElementTree actually raises TypeError in > all these methods, by virtue of using PyArg_ParseTuple > 4. The Python implementation (at least in 3.2) actually doesn't raise > even AssertionError in extend - this was commented out > > The suggestion for 3.3 (where compatibility between the C and Python > implementations gets even more important, since the C one is now being > imported by default when available) is to raise TypeError in all 3 > methods in the Python implementation, to match the C implementation, > and to modify the documentation accordingly. +1 Stefan From p.f.moore at gmail.com Fri Mar 16 09:38:18 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 Mar 2012 08:38:18 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F628586.8030308@oddbird.net> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> Message-ID: On 16 March 2012 00:12, Carl Meyer wrote: > Changing the directory name is in fact a new and different (and much > more invasive) special case, because distutils et al install scripts > there, and that directory name is part of the distutils install scheme. > Installers don't care where the Python binary is located, so moving it > in with the other scripts has very little impact. Two thoughts: 1. The incompatibilities between platforms is precisely the problem that sysconfig is designed to solve, isn't it? So tools in Python will either use sysconfig (and be correct regardless of layout) or should be encouraged to change to use sysconfig (so they are layout-independent). And tools *not* in Python will be platform-specific anyway (I assume no-one is writing Perl scripts to manipulate their Python installation :-)) 2. The differences in layout between a installed Python, uninstalled builds and virtualenvs, on the same platform, are more annoying in practice than any cross-platform differences (at least for me). But again, these are known issues that can be dealt with easily enough (trivially via sysconfig from within Python). If I were "tidying up", I would consider renaming Scripts to "bin" on Windows, and putting the Python executables in there (so there's only one directory to add to PATH, and it uses the common name "bin" rather than a name that implies that it doesn't contain exes). But that offers no practical benefit, and as Mark says does break existing code, so I don't think it's worth it. If you can get Guido to lend you the time machine keys, I'd support putting it in from Python 1.5 onwards :-) Paul. From regebro at gmail.com Fri Mar 16 09:46:55 2012 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 16 Mar 2012 09:46:55 +0100 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status In-Reply-To: References: <4F626278.7030701@oddbird.net> Message-ID: On Fri, Mar 16, 2012 at 05:55, Tres Seaver wrote: > ExtensionClass and Acquisition would fit the bill, except they aren't > ported to Python3 (Acquisition needs the headers from ExtensionClass). And there were no plans to port them either, really. :-) Only Zope 2 uses them afaik? Or? //Lennart From ncoghlan at gmail.com Fri Mar 16 12:54:43 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Mar 2012 21:54:43 +1000 Subject: [Python-Dev] cpython: PEP 417: Adding unittest.mock In-Reply-To: References: Message-ID: On Thu, Mar 15, 2012 at 6:27 AM, Michael Foord wrote: > On the topic of docs.... mock documentation is about eight pages long. My intention was to strip this down to just the api documentation, along with a link to the docs on my site for further examples and so on. I was encouraged here at the sprints to include the full documentation instead (minus the mock library comparison page and the front page can be cut down). So this is what I am now intending to include. It does mean the mock documentation will be "extensive". Don't forgot you also have the option of splitting out a separate HOWTO tutorial section, leaving the main docs as a pure API reference. (I personally find that style easier to use than the ones which try to address both needs in the main module docs). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tseaver at palladion.com Fri Mar 16 14:50:13 2012 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 16 Mar 2012 09:50:13 -0400 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status In-Reply-To: References: <4F626278.7030701@oddbird.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/16/2012 04:46 AM, Lennart Regebro wrote: > On Fri, Mar 16, 2012 at 05:55, Tres Seaver > wrote: >> ExtensionClass and Acquisition would fit the bill, except they >> aren't ported to Python3 (Acquisition needs the headers from >> ExtensionClass). > > And there were no plans to port them either, really. :-) Only Zope 2 > uses them afaik? Or? I don't know of plans to port them, or even how hard the port would be. The "headers needed" problem is a tricky one, and they do exercise it. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9jRRUACgkQ+gerLs4ltQ4cpwCgnLehMsKDV8BKMkix+ZitRnPA LHgAnRLZdjc7+I9/rkepO6iNXEBg7uQo =JmOT -----END PGP SIGNATURE----- From guido at python.org Fri Mar 16 15:57:15 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Mar 2012 07:57:15 -0700 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: On Thu, Mar 15, 2012 at 9:48 PM, Tres Seaver wrote: > On 03/13/2012 06:49 PM, Nick Coghlan wrote: >> On Wed, Mar 14, 2012 at 8:08 AM, Guido van Rossum >> wrote: >>> If you can solve your problem with a suitably hacked Unpickler >>> subclass that's fine with me, but I would personally use this >>> opportunity to change the app to some other serialization format >>> that is perhaps less general but more robust than pickle. I've been >>> bitten by too many pickle-related problems to recommend pickle to >>> anyone... >> >> It's fine for in-memory storage of (almost) arbitrary objects (I use >> it to stash things in a memory backed sqlite DB via SQLAlchemy) and >> for IPC, but yeah, for long-term cross-version persistent storage, >> I'd be looking to something like JSON rather than pickle. > > Note the Zope ecosystem (including Plone) is an *enoromous* installed > base[1] using pickle for storage of data over many years and multiple > versions of Python: ?until this point, it has always been possible to > arrange for old pickles to work (e.g., by providing aliases for missing > module names, etc.). > > ]1] tens of thousands of Zope-based sites in production, including very > ? ?high-profile ones: ?http://plone.org/support/sites Don't I know it. :-) So do you need any help porting to Python 3 or not? The OP didn't mention Zope. -- --Guido van Rossum (python.org/~guido) From van.lindberg at gmail.com Fri Mar 16 16:09:13 2012 From: van.lindberg at gmail.com (VanL) Date: Fri, 16 Mar 2012 10:09:13 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F627916.7060705@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> Message-ID: <4F635799.5050402@gmail.com> On 3/15/2012 6:19 PM, Mark Hammond wrote: > [At least I think that is your proposal - can you confirm that the > directory layouts in your proposal exactly match the directory > layouts in virtual envs on all other platforms? ie, that > inconsistencies like the python{py_version_short} suffix will not > remain?] Yes, that is the ideal. > Also, I'm still yet to see what exactly becomes "easier" in your > model? As you mention, most Python code will not care; distutils and > other parts of the stdlib will "do the right thing" - and indeed, > already do for Windows. Again, I have stated my use case - cross platform development where the tools use the directory layout in some way, or where the environment should be checked into source control. > It is a shame this wasn't done as part of py3k in the first place. > But I assume you would be looking at Python 3.4 for this, right? No, I would like this for 3.3. From Van.Lindberg at haynesboone.com Fri Mar 16 16:08:33 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Fri, 16 Mar 2012 15:08:33 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F628586.8030308@oddbird.net> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> Message-ID: <4F63576A.4080308@gmail.com> Carl - > Changing the directory name is in fact a new and different (and much > more invasive) special case, because distutils et al install scripts > there, and that directory name is part of the distutils install scheme. > Installers don't care where the Python binary is located, so moving it > in with the other scripts has very little impact. So would changing the distutils install scheme in 3.3 - as defined and declared by distutils - lead to a change in your code? Alternatively stated, do you independently figure out that your virtualenv is on Windows and then put things in Scripts, etc, or do you use sysconfig? If sysconfig gave you different (consistent) values across platforms, how would that affect your code? Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From rdmurray at bitdance.com Fri Mar 16 16:33:09 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 16 Mar 2012 11:33:09 -0400 Subject: [Python-Dev] Raising assertions on wrong element types in ElementTree In-Reply-To: References: Message-ID: <20120316153310.2E7412500F8@webabinitio.net> On Fri, 16 Mar 2012 09:38:49 +0200, Eli Bendersky wrote: > 1. The behavior of append, insert and extend should be similar in this respect > 2. AssertionError is not the customary error in such case - TypeError > is much more suitable > 3. The C implementation of ElementTree actually raises TypeError in > all these methods, by virtue of using PyArg_ParseTuple > 4. The Python implementation (at least in 3.2) actually doesn't raise > even AssertionError in extend - this was commented out > > The suggestion for 3.3 (where compatibility between the C and Python > implementations gets even more important, since the C one is now being > imported by default when available) is to raise TypeError in all 3 > methods in the Python implementation, to match the C implementation, > and to modify the documentation accordingly. > > There may appear to be a backwards compatibility here, since the doc > of extend mentions raising AssertionError - but as said above, the doc > is wrong, so no regressions in the code are be expected. > > Does that sound reasonable (for 3.3)? Yes. > Does it make sense to also fix this in 3.2/2.7? Or fix only the > documentation? Or not touch them at all? Our usual approach in cases like this is to not change it in the maint releases. Why risk breaking someone's code for no particular benefit? If you want some extra work you could add it as a deprecation warning, I suppose. --David From Van.Lindberg at haynesboone.com Fri Mar 16 16:17:18 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Fri, 16 Mar 2012 15:17:18 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> Message-ID: <4F63597D.3020109@gmail.com> On 3/16/2012 3:38 AM, Paul Moore wrote: > On 16 March 2012 00:12, Carl Meyer wrote: >> Changing the directory name is in fact a new and different (and much >> more invasive) special case, because distutils et al install scripts >> there, and that directory name is part of the distutils install scheme. >> Installers don't care where the Python binary is located, so moving it >> in with the other scripts has very little impact. This is very interesting, as it seems to argue against Mark's point. If moving the Python binary is not an issue here, then would this change make it any more/less of an issue? > 1. The incompatibilities between platforms is precisely the problem > that sysconfig is designed to solve, isn't it? So tools in Python will > either use sysconfig (and be correct regardless of layout) or should > be encouraged to change to use sysconfig (so they are > layout-independent). Right. I want to change the default layout in sysconfig.cfg. > 2. The differences in layout between a installed Python, uninstalled > builds and virtualenvs, on the same platform, are more annoying in > practice than any cross-platform differences (at least for me). But > again, these are known issues that can be dealt with easily enough > (trivially via sysconfig from within Python). These differences are a major pain for me - and it doesn't make sense that they should need to be worked around each and every time. > If I were "tidying up", I would consider renaming Scripts to "bin" on > Windows, and putting the Python executables in there (so there's only > one directory to add to PATH, and it uses the common name "bin" rather > than a name that implies that it doesn't contain exes). But that > offers no practical benefit... This is not a "we should be consistent" argument - I know that would never fly. I do cross-platform dev all the time (develop on Windows and Mac, deploy on Linux) and so this bites me *every single time* I want to get a consistent layout between these three. That could be because I want my deployment environment to match my development environment(s), it could be because I need to introspect the layout to find some data, or because I want to check in an entire environment into source control. This is not purely aesthetics - this is an issue I deal with all the time. Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From merwok at netwok.org Fri Mar 16 16:51:29 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 16 Mar 2012 16:51:29 +0100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F63597D.3020109@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> <4F63597D.3020109@gmail.com> Message-ID: <4F636181.7050307@netwok.org> Hi, Le 16/03/2012 16:17, Lindberg, Van a ?crit : > On 3/16/2012 3:38 AM, Paul Moore wrote: >> 1. The incompatibilities between platforms is precisely the problem >> that sysconfig is designed to solve, isn't it? So tools in Python will >> either use sysconfig (and be correct regardless of layout) or should >> be encouraged to change to use sysconfig (so they are >> layout-independent). > Right. I want to change the default layout in sysconfig.cfg. A few notes: - sysconfig was extracted from distutils to the top level in 2.7 and 3.2, but distutils does not use it (due to the Great Revert two years ago after it was decided at PyCon to freeze distutils and start distutils2); there are unfortunately a few subtle differences between the install schemes in sysconfig and distutils. So even if virtualenv uses sysconfig in 2.7/3.2+, I?m not sure that the venv?s pip will install distutils-based projects in the right places. - packaging uses only sysconfig.cfg - I think a change to distutils install schemes in 3.3 would violate the freeze. Regards From p.f.moore at gmail.com Fri Mar 16 16:53:21 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 Mar 2012 15:53:21 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F63597D.3020109@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> <4F63597D.3020109@gmail.com> Message-ID: On 16 March 2012 15:17, Lindberg, Van wrote: > This is not a "we should be consistent" argument - I know that would > never fly. I do cross-platform dev all the time (develop on Windows and > Mac, deploy on Linux) and so this bites me *every single time* I want to > get a consistent layout between these three. That could be because I > want my deployment environment to match my development environment(s), > it could be because I need to introspect the layout to find some data, > or because I want to check in an entire environment into source control. The only way I can read this to make sense is that you somehow consider the Python installation as part of your development environment (you mentioned source control earlier in the thread - surely you don't manage your Python installation in source control - binaries, stdlib, etc?). I can't see why you would do this, and it certainly doesn't seem like a reasonable thing to do to me. Can you clarify? Paul. From raymond.hettinger at gmail.com Fri Mar 16 17:09:55 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 16 Mar 2012 09:09:55 -0700 Subject: [Python-Dev] cpython: PEP 417: Adding unittest.mock In-Reply-To: References: Message-ID: <9EE751D8-EAA8-43D8-9C04-C2FC383CFC82@gmail.com> On Mar 16, 2012, at 4:54 AM, Nick Coghlan wrote: > Don't forgot you also have the option of splitting out a separate > HOWTO tutorial section, leaving the main docs as a pure API reference. > (I personally find that style easier to use than the ones which try to > address both needs in the main module docs). +1 The commingling of extensive examples with regular docs has made it difficult to lookup functionality in argparse for example. In contrast, the logging module's howtos were split-out to good effect. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri Mar 16 17:20:27 2012 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 16 Mar 2012 12:20:27 -0400 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: <4F63684B.9060402@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/16/2012 10:57 AM, Guido van Rossum wrote: > On Thu, Mar 15, 2012 at 9:48 PM, Tres Seaver > wrote: >> On 03/13/2012 06:49 PM, Nick Coghlan wrote: >>> On Wed, Mar 14, 2012 at 8:08 AM, Guido van Rossum >>> wrote: >>>> If you can solve your problem with a suitably hacked Unpickler >>>> subclass that's fine with me, but I would personally use this >>>> opportunity to change the app to some other serialization >>>> format that is perhaps less general but more robust than pickle. >>>> I've been bitten by too many pickle-related problems to >>>> recommend pickle to anyone... >>> >>> It's fine for in-memory storage of (almost) arbitrary objects (I >>> use it to stash things in a memory backed sqlite DB via >>> SQLAlchemy) and for IPC, but yeah, for long-term cross-version >>> persistent storage, I'd be looking to something like JSON rather >>> than pickle. >> >> Note the Zope ecosystem (including Plone) is an *enoromous* >> installed base[1] using pickle for storage of data over many years >> and multiple versions of Python: until this point, it has always >> been possible to arrange for old pickles to work (e.g., by providing >> aliases for missing module names, etc.). >> >> ]1] tens of thousands of Zope-based sites in production, including >> very high-profile ones: http://plone.org/support/sites > > Don't I know it. :-) > > So do you need any help porting to Python 3 or not? The OP didn't > mention Zope. ZODB is actually the biggest / most important non-ported items in the Zope ecosystem. We are close to a pure-Python version of persistent and it's pickle cache, and have some work done toward pure-Python BTrees. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9jaEsACgkQ+gerLs4ltQ4bQACfcRxaeRMLnmDRzWL2c537VLvC xsMAn2Cjql4Wvavr0MNyQxS58Af4EwMf =UT5J -----END PGP SIGNATURE----- From Van.Lindberg at haynesboone.com Fri Mar 16 17:22:49 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Fri, 16 Mar 2012 16:22:49 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> <4F63597D.3020109@gmail.com> Message-ID: <4F6368D9.5050200@gmail.com> On 3/16/2012 10:53 AM, Paul Moore wrote: > The only way I can read this to make sense is that you somehow > consider the Python installation as part of your development > environment (you mentioned source control earlier in the thread - > surely you don't manage your Python installation in source control - > binaries, stdlib, etc?). I can't see why you would do this, and it > certainly doesn't seem like a reasonable thing to do to me. > > Can you clarify? I don't check in the python binary itself, nor the stdlib, but I *do* check in the whole "installation", including the binaries directory. I like having my deploy environment exactly match my develop environment. So if I do have an executable program, I put it in the binaries directory and check it in. My .hgignore includes "python", "python.exe", pip, easy_install, etc. - things that are "owned by the installation - but it includes everything else. As for the stdlib, I don't check that in, so that portion of the proposal (standardize lib naming) is nice to have, but not essential to me. For example, in the following environment: env/ bin/ python pip easy_install my_script lib/ [stuff] data/ [stuff] src/ my_package I would include bin/my_script, src/, and data/ in my version control. This breaks cross-platform development if "bin" is named "Scripts". CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From tseaver at palladion.com Fri Mar 16 17:20:27 2012 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 16 Mar 2012 12:20:27 -0400 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: <4F63684B.9060402@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/16/2012 10:57 AM, Guido van Rossum wrote: > On Thu, Mar 15, 2012 at 9:48 PM, Tres Seaver > wrote: >> On 03/13/2012 06:49 PM, Nick Coghlan wrote: >>> On Wed, Mar 14, 2012 at 8:08 AM, Guido van Rossum >>> wrote: >>>> If you can solve your problem with a suitably hacked Unpickler >>>> subclass that's fine with me, but I would personally use this >>>> opportunity to change the app to some other serialization >>>> format that is perhaps less general but more robust than pickle. >>>> I've been bitten by too many pickle-related problems to >>>> recommend pickle to anyone... >>> >>> It's fine for in-memory storage of (almost) arbitrary objects (I >>> use it to stash things in a memory backed sqlite DB via >>> SQLAlchemy) and for IPC, but yeah, for long-term cross-version >>> persistent storage, I'd be looking to something like JSON rather >>> than pickle. >> >> Note the Zope ecosystem (including Plone) is an *enoromous* >> installed base[1] using pickle for storage of data over many years >> and multiple versions of Python: until this point, it has always >> been possible to arrange for old pickles to work (e.g., by providing >> aliases for missing module names, etc.). >> >> ]1] tens of thousands of Zope-based sites in production, including >> very high-profile ones: http://plone.org/support/sites > > Don't I know it. :-) > > So do you need any help porting to Python 3 or not? The OP didn't > mention Zope. ZODB is actually the biggest / most important non-ported items in the Zope ecosystem. We are close to a pure-Python version of persistent and it's pickle cache, and have some work done toward pure-Python BTrees. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9jaEsACgkQ+gerLs4ltQ4bQACfcRxaeRMLnmDRzWL2c537VLvC xsMAn2Cjql4Wvavr0MNyQxS58Af4EwMf =UT5J -----END PGP SIGNATURE----- From v+python at g.nevcal.com Fri Mar 16 17:57:01 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 16 Mar 2012 09:57:01 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F6368D9.5050200@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> <4F63597D.3020109@gmail.com> <4F6368D9.5050200@gmail.com> Message-ID: <4F6370DD.1000207@g.nevcal.com> On 3/16/2012 9:22 AM, Lindberg, Van wrote: > On 3/16/2012 10:53 AM, Paul Moore wrote: >> > The only way I can read this to make sense is that you somehow >> > consider the Python installation as part of your development >> > environment (you mentioned source control earlier in the thread - >> > surely you don't manage your Python installation in source control - >> > binaries, stdlib, etc?). I can't see why you would do this, and it >> > certainly doesn't seem like a reasonable thing to do to me. >> > >> > Can you clarify? > I don't check in the python binary itself, nor the stdlib, but I*do* > check in the whole "installation", including the binaries directory. > > I like having my deploy environment exactly match my develop > environment. So I think I'm finally beginning to see the underlying reason why Van is desiring this consistency: It is not that he wants to check in his installation of Python, but that he wants to check in his installation of his packages and scripts into a source control environment, and then be able to check out that source control environment into an installation of Python on another machine of a different architecture. In an environment where a source control system is pervasive and well used, this would be an effective deployment alternative to developing a packaging/distribution solution using distutils, distutels2, packaging, easy_install, eggs, or peanuts, or any other such scheme. But! Source control environments don't lend themselves to being used for anything except exact replication of file and directory structure, so when the different architectures have different directory structures, this deployment technique cannot easily work.... except, as Van has discussed, by tweaking the development machine's environment to match that of the deployment machines... and that only works in the case where the deployment happens to only one architecture, and the development machine can be tweaked to match... but deploying to multiple machine having different architectures and directory structures would be impossible using the source control deployment technique, because of the different directory structures. If Van stated this goal in this thread, I missed it, and I think it is the missing link in the discussions. If I'm wrong, apologies for the noise. -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Mar 16 18:07:37 2012 From: status at bugs.python.org (Python tracker) Date: Fri, 16 Mar 2012 18:07:37 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20120316170737.4A0331DF08@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2012-03-09 - 2012-03-16) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3337 (+19) closed 22779 (+75) total 26116 (+94) Open issues with patches: 1423 Issues opened (69) ================== #10484: http.server.is_cgi fails to handle CGI URLs containing PATH_IN http://bugs.python.org/issue10484 reopened by v+python #12568: Add functions to get the width in columns of a character http://bugs.python.org/issue12568 reopened by pitrou #14202: The docs of xml.dom.pulldom are almost nonexistent http://bugs.python.org/issue14202 reopened by eric.araujo #14243: NamedTemporaryFile unusable under Windows http://bugs.python.org/issue14243 opened by dabrahams #14245: float rounding examples in FAQ are outdated http://bugs.python.org/issue14245 opened by zbysz #14249: unicodeobject.c: aliasing warnings http://bugs.python.org/issue14249 opened by skrah #14250: for string patterns regex.flags is never equal to 0 http://bugs.python.org/issue14250 opened by py.user #14254: IDLE - shell restart during readline does not reset readline http://bugs.python.org/issue14254 opened by serwy #14255: tempfile.gettempdir() didn't return the path with correct case http://bugs.python.org/issue14255 opened by ??????.??? #14260: re.groupindex available for modification and continues to work http://bugs.python.org/issue14260 opened by py.user #14261: Cleanup in smtpd module http://bugs.python.org/issue14261 opened by maker #14262: Allow using decimals as arguments to `timedelta` http://bugs.python.org/issue14262 opened by cool-RR #14263: switch_index_if_fails fails on py2 http://bugs.python.org/issue14263 opened by tarek #14264: Comparison bug in distutils2.version http://bugs.python.org/issue14264 opened by tarek #14265: Fully qualified test name in failure output http://bugs.python.org/issue14265 opened by michael.foord #14266: pyunit script as shorthand for python -m unittest http://bugs.python.org/issue14266 opened by michael.foord #14268: _move_file is broken because of a bad mock http://bugs.python.org/issue14268 opened by tarek #14269: SMTPD server does not enforce client starting mail transaction http://bugs.python.org/issue14269 opened by fruitnuke #14270: Can't install a project in a specific directory http://bugs.python.org/issue14270 opened by mlhamel #14273: distutils2: logging handler not properly initialized http://bugs.python.org/issue14273 opened by tarek #14274: pysetup does not look at requires.txt http://bugs.python.org/issue14274 opened by tarek #14275: pysetup create doesn't handle install requirements http://bugs.python.org/issue14275 opened by janjaapdriessen #14276: installing latest version of a project http://bugs.python.org/issue14276 opened by janjaapdriessen #14277: time.monotonic docstring does not mention the time unit return http://bugs.python.org/issue14277 opened by nicholas.riley #14279: packaging.pypi should support flat directories of distribution http://bugs.python.org/issue14279 opened by j1m #14280: packaging.pypi should not require checksums http://bugs.python.org/issue14280 opened by j1m #14285: Traceback wrong on ImportError while executing a package http://bugs.python.org/issue14285 opened by ms4py #14286: xxlimited.obj: unresolved external symbol __imp__PyObject_New http://bugs.python.org/issue14286 opened by skrah #14287: sys.stdin.readline and KeyboardInterrupt on windows http://bugs.python.org/issue14287 opened by miwa #14288: Make iterators pickleable http://bugs.python.org/issue14288 opened by krisvale #14290: Importing script as module causes ImportError with pickle.load http://bugs.python.org/issue14290 opened by rj3d #14292: OS X installer build script doesn't set $CXX, so it ends up as http://bugs.python.org/issue14292 opened by nicholas.riley #14293: Message methods delegated via __getattr__ inaccessible using s http://bugs.python.org/issue14293 opened by Brian.Jones #14294: Requirements are not properly copied into metatdata of dist-in http://bugs.python.org/issue14294 opened by Preston #14295: PEP 417: adding mock module http://bugs.python.org/issue14295 opened by michael.foord #14296: Compilation error on CentOS 5.8 http://bugs.python.org/issue14296 opened by Alzakath #14297: Custom string formatter doesn't work like builtin str.format http://bugs.python.org/issue14297 opened by PaulMcMillan #14299: OS X installer build script: permissions not ensured http://bugs.python.org/issue14299 opened by nicholas.riley #14300: dup_socket() on Windows should use WSA_FLAG_OVERLAPPED http://bugs.python.org/issue14300 opened by sbt #14301: xmlrpc client transport and threading problem http://bugs.python.org/issue14301 opened by kees #14302: Move python.exe to bin/ http://bugs.python.org/issue14302 opened by brian.curtin #14303: Incorrect documentation for socket.py on linux http://bugs.python.org/issue14303 opened by Shane.Hansen #14304: Implement utf-8-bmp codec http://bugs.python.org/issue14304 opened by asvetlov #14306: try/except block is both efficient and expensive? http://bugs.python.org/issue14306 opened by tshepang #14307: Make subclassing SocketServer simpler for non-blocking framewo http://bugs.python.org/issue14307 opened by krisvale #14308: '_DummyThread' object has no attribute '_Thread__block' http://bugs.python.org/issue14308 opened by Dustin.Kirkland #14309: Deprecate time.clock() http://bugs.python.org/issue14309 opened by haypo #14310: Socket duplication for windows http://bugs.python.org/issue14310 opened by krisvale #14311: ConfigParser does not parse utf-8 files with BOM bytes http://bugs.python.org/issue14311 opened by Sean.Wang #14313: zipfile does not unpack files from archive (files extracted ha http://bugs.python.org/issue14313 opened by fidoman #14314: logging smtp handler (and test) timeout issue http://bugs.python.org/issue14314 opened by r.david.murray #14315: zipfile.ZipFile() unable to open zip File http://bugs.python.org/issue14315 opened by pleed #14316: Broken link in grammar.rst http://bugs.python.org/issue14316 opened by berker.peksag #14318: clarify "may not" in time.steady docs http://bugs.python.org/issue14318 opened by Jim.Jewett #14319: cleanup index switching mechanism on packaging.pypi http://bugs.python.org/issue14319 opened by alexis #14322: More test coverage for hmac http://bugs.python.org/issue14322 opened by packetslave #14323: Normalize math precision in RGB/YIQ conversion http://bugs.python.org/issue14323 opened by packetslave #14324: Do not rely on AC_RUN_IFELSE tests in the configury http://bugs.python.org/issue14324 opened by doko #14325: Stop using the garbage collector to manage the lifetime of the http://bugs.python.org/issue14325 opened by exarkun #14326: IDLE - allow shell to support different locales http://bugs.python.org/issue14326 opened by serwy #14327: replace use of uname in the configury with macros set by AC_CA http://bugs.python.org/issue14327 opened by doko #14328: Add keyword-only parameter support to PyArg_ParseTupleAndKeywo http://bugs.python.org/issue14328 opened by larry #14329: proxy_bypass_macosx_sysconf does not handle singel ip addresse http://bugs.python.org/issue14329 opened by Serge.Droz #14330: do not use the host python for cross builds http://bugs.python.org/issue14330 opened by doko #14331: Python/import.c uses a lot of stack space due to MAXPATHLEN http://bugs.python.org/issue14331 opened by gregory.p.smith #14332: difflib.ndiff appears to ignore linejunk argument http://bugs.python.org/issue14332 opened by patena #14333: queue unittest errors http://bugs.python.org/issue14333 opened by anacrolix #14335: Reimplement multiprocessing's ForkingPickler using dispatch_ta http://bugs.python.org/issue14335 opened by sbt #1648923: HP-UX: -lcurses missing for readline.so http://bugs.python.org/issue1648923 reopened by r.david.murray Most recent 15 issues with no replies (15) ========================================== #14335: Reimplement multiprocessing's ForkingPickler using dispatch_ta http://bugs.python.org/issue14335 #14333: queue unittest errors http://bugs.python.org/issue14333 #14329: proxy_bypass_macosx_sysconf does not handle singel ip addresse http://bugs.python.org/issue14329 #14326: IDLE - allow shell to support different locales http://bugs.python.org/issue14326 #14323: Normalize math precision in RGB/YIQ conversion http://bugs.python.org/issue14323 #14319: cleanup index switching mechanism on packaging.pypi http://bugs.python.org/issue14319 #14318: clarify "may not" in time.steady docs http://bugs.python.org/issue14318 #14315: zipfile.ZipFile() unable to open zip File http://bugs.python.org/issue14315 #14304: Implement utf-8-bmp codec http://bugs.python.org/issue14304 #14303: Incorrect documentation for socket.py on linux http://bugs.python.org/issue14303 #14302: Move python.exe to bin/ http://bugs.python.org/issue14302 #14301: xmlrpc client transport and threading problem http://bugs.python.org/issue14301 #14299: OS X installer build script: permissions not ensured http://bugs.python.org/issue14299 #14297: Custom string formatter doesn't work like builtin str.format http://bugs.python.org/issue14297 #14293: Message methods delegated via __getattr__ inaccessible using s http://bugs.python.org/issue14293 Most recent 15 issues waiting for review (15) ============================================= #14335: Reimplement multiprocessing's ForkingPickler using dispatch_ta http://bugs.python.org/issue14335 #14331: Python/import.c uses a lot of stack space due to MAXPATHLEN http://bugs.python.org/issue14331 #14330: do not use the host python for cross builds http://bugs.python.org/issue14330 #14329: proxy_bypass_macosx_sysconf does not handle singel ip addresse http://bugs.python.org/issue14329 #14328: Add keyword-only parameter support to PyArg_ParseTupleAndKeywo http://bugs.python.org/issue14328 #14327: replace use of uname in the configury with macros set by AC_CA http://bugs.python.org/issue14327 #14325: Stop using the garbage collector to manage the lifetime of the http://bugs.python.org/issue14325 #14324: Do not rely on AC_RUN_IFELSE tests in the configury http://bugs.python.org/issue14324 #14323: Normalize math precision in RGB/YIQ conversion http://bugs.python.org/issue14323 #14322: More test coverage for hmac http://bugs.python.org/issue14322 #14310: Socket duplication for windows http://bugs.python.org/issue14310 #14308: '_DummyThread' object has no attribute '_Thread__block' http://bugs.python.org/issue14308 #14307: Make subclassing SocketServer simpler for non-blocking framewo http://bugs.python.org/issue14307 #14300: dup_socket() on Windows should use WSA_FLAG_OVERLAPPED http://bugs.python.org/issue14300 #14299: OS X installer build script: permissions not ensured http://bugs.python.org/issue14299 Top 10 most discussed issues (10) ================================= #14127: add st_*time_ns fileds to os.stat(), add ns keyword to os.*uti http://bugs.python.org/issue14127 21 msgs #14200: Idle shell crash on printing non-BMP unicode character http://bugs.python.org/issue14200 20 msgs #12568: Add functions to get the width in columns of a character http://bugs.python.org/issue12568 15 msgs #5758: fileinput.hook_compressed returning bytes from gz file http://bugs.python.org/issue5758 13 msgs #8739: Update to smtpd.py to RFC 5321 http://bugs.python.org/issue8739 12 msgs #14202: The docs of xml.dom.pulldom are almost nonexistent http://bugs.python.org/issue14202 11 msgs #14245: float rounding examples in FAQ are outdated http://bugs.python.org/issue14245 10 msgs #10050: urllib.request still has old 2.x urllib primitives http://bugs.python.org/issue10050 9 msgs #10484: http.server.is_cgi fails to handle CGI URLs containing PATH_IN http://bugs.python.org/issue10484 9 msgs #2377: Replace __import__ w/ importlib.__import__ http://bugs.python.org/issue2377 8 msgs Issues closed (73) ================== #2486: Recode (parts of) decimal module in C http://bugs.python.org/issue2486 closed by mark.dickinson #2843: New methods for existing Tkinter widgets http://bugs.python.org/issue2843 closed by loewis #3835: tkinter goes into an infinite loop (pydoc.gui) http://bugs.python.org/issue3835 closed by loewis #4345: Implement nb_nonzero for PyTclObject http://bugs.python.org/issue4345 closed by loewis #8247: Can't Import Tkinter http://bugs.python.org/issue8247 closed by loewis #8942: __path__ attribute of modules loaded by zipimporter is unteste http://bugs.python.org/issue8942 closed by r.david.murray #8963: test_urllibnet failure http://bugs.python.org/issue8963 closed by orsenthil #9079: Make gettimeofday available in time module http://bugs.python.org/issue9079 closed by haypo #9257: cElementTree iterparse requires events as bytes; ElementTree u http://bugs.python.org/issue9257 closed by eli.bendersky #9574: allow whitespace around central '+' in complex constructor http://bugs.python.org/issue9574 closed by python-dev #10148: st_mtime differs after shutil.copy2 http://bugs.python.org/issue10148 closed by haypo #10522: test_telnet exception http://bugs.python.org/issue10522 closed by jackdied #10543: Test discovery (unittest) does not work with jython http://bugs.python.org/issue10543 closed by michael.foord #11082: ValueError: Content-Length should be specified http://bugs.python.org/issue11082 closed by orsenthil #11199: urllib hangs when closing connection http://bugs.python.org/issue11199 closed by python-dev #11261: urlopen breaks when data parameter is used. http://bugs.python.org/issue11261 closed by python-dev #12758: time.time() returns local time instead of UTC http://bugs.python.org/issue12758 closed by r.david.murray #12818: email.utils.formataddr incorrectly quotes parens inside quoted http://bugs.python.org/issue12818 closed by r.david.murray #13394: Patch to increase aifc lib test coverage http://bugs.python.org/issue13394 closed by ezio.melotti #13450: add assertions to implement the intent in ''.format_map test http://bugs.python.org/issue13450 closed by python-dev #13703: Hash collision security issue http://bugs.python.org/issue13703 closed by gregory.p.smith #13709: Capitalization mistakes in the documentation for ctypes http://bugs.python.org/issue13709 closed by eli.bendersky #13839: -m pstats should combine all the profiles given as arguments http://bugs.python.org/issue13839 closed by pitrou #13842: Cannot pickle Ellipsis or NotImplemented http://bugs.python.org/issue13842 closed by lukasz.langa #13964: os.utimensat() and os.futimes() should accept (sec, nsec), dro http://bugs.python.org/issue13964 closed by haypo #14062: UTF-8 Email Subject problem http://bugs.python.org/issue14062 closed by r.david.murray #14104: Implement time.monotonic() on Mac OS X http://bugs.python.org/issue14104 closed by haypo #14163: tkinter: problems with hello doc example http://bugs.python.org/issue14163 closed by asvetlov #14169: compiler.compile fails on "if" statement in attached file http://bugs.python.org/issue14169 closed by terry.reedy #14179: Test coverage for lib/re.py http://bugs.python.org/issue14179 closed by ezio.melotti #14180: Factorize code to convert int/float to time_t, timeval or time http://bugs.python.org/issue14180 closed by python-dev #14184: test_recursion_limit fails on OS X when compiled with clang http://bugs.python.org/issue14184 closed by ned.deily #14186: Link to PEP 3107 in "def" part of Language Reference http://bugs.python.org/issue14186 closed by python-dev #14207: ElementTree.ParseError - needs documentation and consistent C& http://bugs.python.org/issue14207 closed by eli.bendersky #14210: add filename completion to pdb http://bugs.python.org/issue14210 closed by python-dev #14230: Delegating generator is not always visible to debugging tools http://bugs.python.org/issue14230 closed by python-dev #14232: obmalloc: mmap() returns MAP_FAILED on error, not 0 http://bugs.python.org/issue14232 closed by python-dev #14234: CVE-2012-0876 (hash table collisions CPU usage DoS) for embedd http://bugs.python.org/issue14234 closed by gregory.p.smith #14237: Special sequences \A and \Z don't work in character set [] http://bugs.python.org/issue14237 closed by georg.brandl #14238: python shouldn't need username in passwd database http://bugs.python.org/issue14238 closed by eric.araujo #14239: Uninitialised variable in _PyObject_GenericSetAttrWithDict http://bugs.python.org/issue14239 closed by benjamin.peterson #14242: Make subprocess.Popen aware of $SHELL http://bugs.python.org/issue14242 closed by gregory.p.smith #14244: No information about behaviour with groups in pattern in the d http://bugs.python.org/issue14244 closed by python-dev #14246: Accelerated ETree XMLParser cannot handle io.StringIO http://bugs.python.org/issue14246 closed by eli.bendersky #14247: "in" operator doesn't return boolean http://bugs.python.org/issue14247 closed by georg.brandl #14248: Typo in "What???s New In Python 3.3": "comparaison" http://bugs.python.org/issue14248 closed by python-dev #14251: HTMLParser decode issue http://bugs.python.org/issue14251 closed by ezio.melotti #14252: subprocess.Popen.terminate() inconsistent behavior on Windows http://bugs.python.org/issue14252 closed by pitrou #14253: print() encodes characters to native system encoding http://bugs.python.org/issue14253 closed by loewis #14256: test_logging fails if zlib is not present http://bugs.python.org/issue14256 closed by python-dev #14257: minor error in glossary wording regarding __hash__ http://bugs.python.org/issue14257 closed by python-dev #14258: Better explain re.LOCALE and re.UNICODE for \S and \W http://bugs.python.org/issue14258 closed by python-dev #14259: re.finditer() doesn't accept keyword arguments http://bugs.python.org/issue14259 closed by python-dev #14267: TimedRotatingFileHandler chooses wrong file name due to daylig http://bugs.python.org/issue14267 closed by python-dev #14271: remove setup.py from the doc http://bugs.python.org/issue14271 closed by eric.araujo #14272: ast.c: windows compile error http://bugs.python.org/issue14272 closed by skrah #14278: email.utils.localtime throws exception if time.daylight is Fal http://bugs.python.org/issue14278 closed by r.david.murray #14281: Add unit test for cgi.escape method http://bugs.python.org/issue14281 closed by python-dev #14282: lib2to3.fixer_util.touch_import('__future__', ...) can lead to http://bugs.python.org/issue14282 closed by benjamin.peterson #14283: match.pos describes that match object has methods search() and http://bugs.python.org/issue14283 closed by python-dev #14284: unicodeobject error on macosx in build process http://bugs.python.org/issue14284 closed by ezio.melotti #14289: Prebuilt CHM file on Docs download page http://bugs.python.org/issue14289 closed by python-dev #14291: Regression in Python3 of email handling of unicode strings in http://bugs.python.org/issue14291 closed by r.david.murray #14298: account for dict randomization in Design & History FAQ http://bugs.python.org/issue14298 closed by python-dev #14305: fix typos http://bugs.python.org/issue14305 closed by python-dev #14312: Convert PEP 7 to reStructuredText for readability purposes http://bugs.python.org/issue14312 closed by georg.brandl #14317: index.simple module lacking in distutils2 http://bugs.python.org/issue14317 closed by alexis #14320: set.add can return boolean indicate newly added item http://bugs.python.org/issue14320 closed by rhettinger #14321: Do not run pgen during the build if files are up to date http://bugs.python.org/issue14321 closed by doko #14334: Crash: getattr(type, '__getattribute__')(type, type) http://bugs.python.org/issue14334 closed by benjamin.peterson #989712: Support using Tk without a mainloop http://bugs.python.org/issue989712 closed by asvetlov #1006238: cross compile patch http://bugs.python.org/issue1006238 closed by doko #1178863: Variable.__init__ uses self.set(), blocking specialization http://bugs.python.org/issue1178863 closed by loewis From van.lindberg at gmail.com Fri Mar 16 18:25:10 2012 From: van.lindberg at gmail.com (VanL) Date: Fri, 16 Mar 2012 12:25:10 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F6370DD.1000207@g.nevcal.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> <4F63597D.3020109@gmail.com> <4F6368D9.5050200@gmail.com> <4F6370DD.1000207@g.nevcal.com> Message-ID: <4F637776.6040404@gmail.com> On 3/16/2012 11:57 AM, Glenn Linderman wrote: > So I think I'm finally beginning to see the underlying reason why Van is > desiring this consistency: It is not that he wants to check in his > installation of Python, but that he wants to check in his installation > of his packages and scripts into a source control environment, and then > be able to check out that source control environment into an > installation of Python on another machine of a different architecture. > In an environment where a source control system is pervasive and well > used, this would be an effective deployment alternative to developing a > packaging/distribution solution using distutils, distutels2, packaging, > easy_install, eggs, or peanuts, or any other such scheme. > > But! > > Source control environments don't lend themselves to being used for > anything except exact replication of file and directory structure, so > when the different architectures have different directory structures, > this deployment technique cannot easily work.... except, as Van has > discussed, by tweaking the development machine's environment to match > that of the deployment machines... and that only works in the case where > the deployment happens to only one architecture, and the development > machine can be tweaked to match... but deploying to multiple machine > having different architectures and directory structures would be > impossible using the source control deployment technique, because of the > different directory structures. This is exactly correct. From Van.Lindberg at haynesboone.com Fri Mar 16 18:25:11 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Fri, 16 Mar 2012 17:25:11 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F6370DD.1000207@g.nevcal.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> <4F63597D.3020109@gmail.com> <4F6368D9.5050200@gmail.com> <4F6370DD.1000207@g.nevcal.com> Message-ID: <4F637776.6040404@gmail.com> On 3/16/2012 11:57 AM, Glenn Linderman wrote: > So I think I'm finally beginning to see the underlying reason why Van is > desiring this consistency: It is not that he wants to check in his > installation of Python, but that he wants to check in his installation > of his packages and scripts into a source control environment, and then > be able to check out that source control environment into an > installation of Python on another machine of a different architecture. > In an environment where a source control system is pervasive and well > used, this would be an effective deployment alternative to developing a > packaging/distribution solution using distutils, distutels2, packaging, > easy_install, eggs, or peanuts, or any other such scheme. > > But! > > Source control environments don't lend themselves to being used for > anything except exact replication of file and directory structure, so > when the different architectures have different directory structures, > this deployment technique cannot easily work.... except, as Van has > discussed, by tweaking the development machine's environment to match > that of the deployment machines... and that only works in the case where > the deployment happens to only one architecture, and the development > machine can be tweaked to match... but deploying to multiple machine > having different architectures and directory structures would be > impossible using the source control deployment technique, because of the > different directory structures. This is exactly correct.CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From guido at python.org Fri Mar 16 20:11:30 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Mar 2012 12:11:30 -0700 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: <4F63684B.9060402@palladion.com> References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> <4F63684B.9060402@palladion.com> Message-ID: On Fri, Mar 16, 2012 at 9:20 AM, Tres Seaver wrote: > On 03/16/2012 10:57 AM, Guido van Rossum wrote: >> On Thu, Mar 15, 2012 at 9:48 PM, Tres Seaver >> wrote: >>> On 03/13/2012 06:49 PM, Nick Coghlan wrote: >>>> On Wed, Mar 14, 2012 at 8:08 AM, Guido van Rossum >>>> wrote: >>>>> If you can solve your problem with a suitably hacked Unpickler >>>>> subclass that's fine with me, but I would personally use this >>>>> opportunity to change the app to some other serialization >>>>> format that is perhaps less general but more robust than pickle. >>>>> I've been bitten by too many pickle-related problems to >>>>> recommend pickle to anyone... >>>> >>>> It's fine for in-memory storage of (almost) arbitrary objects (I >>>> use it to stash things in a memory backed sqlite DB via >>>> SQLAlchemy) and for IPC, but yeah, for long-term cross-version >>>> persistent storage, I'd be looking to something like JSON rather >>>> than pickle. >>> >>> Note the Zope ecosystem (including Plone) is an *enoromous* >>> installed base[1] using pickle for storage of data over many years >>> and multiple versions of Python: ?until this point, it has always >>> been possible to arrange for old pickles to work (e.g., by providing >>> aliases for missing module names, etc.). >>> >>> ]1] tens of thousands of Zope-based sites in production, including >>> very high-profile ones: ?http://plone.org/support/sites >> >> Don't I know it. :-) >> >> So do you need any help porting to Python 3 or not? The OP didn't >> mention Zope. > > ZODB is actually the biggest / most important non-ported items in the > Zope ecosystem. ?We are close to a pure-Python version of persistent and > it's pickle cache, and have some work done toward pure-Python BTrees. I take that as meaning "no, we don't need help, it's all under control." -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Fri Mar 16 20:28:18 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Mar 2012 15:28:18 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F6368D9.5050200@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> <4F63597D.3020109@gmail.com> <4F6368D9.5050200@gmail.com> Message-ID: On 3/16/2012 12:22 PM, Lindberg, Van wrote: > env/ > bin/ > python > pip > easy_install > my_script In http://bugs.python.org/issue14302 Brian Curtin claims "After talks at PyCon with several people, python.exe will live in C:\Python33\bin rather than C:\Python33 to come more in line with the Unix layout. This will also simplify another issue with the Path option for the 3.3 installer as well as packaging's target directory for top-level scripts (used to be Scripts/, will be bin/)." -- Terry Jan Reedy From tjreedy at udel.edu Fri Mar 16 20:49:33 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Mar 2012 15:49:33 -0400 Subject: [Python-Dev] Raising assertions on wrong element types in ElementTree In-Reply-To: <20120316153310.2E7412500F8@webabinitio.net> References: <20120316153310.2E7412500F8@webabinitio.net> Message-ID: On 3/16/2012 11:33 AM, R. David Murray wrote: > On Fri, 16 Mar 2012 09:38:49 +0200, Eli Bendersky wrote: >> 1. The behavior of append, insert and extend should be similar in this respect >> 2. AssertionError is not the customary error in such case - TypeError >> is much more suitable >> 3. The C implementation of ElementTree actually raises TypeError in >> all these methods, by virtue of using PyArg_ParseTuple >> 4. The Python implementation (at least in 3.2) actually doesn't raise >> even AssertionError in extend - this was commented out >> >> The suggestion for 3.3 (where compatibility between the C and Python >> implementations gets even more important, since the C one is now being >> imported by default when available) is to raise TypeError in all 3 >> methods in the Python implementation, to match the C implementation, >> and to modify the documentation accordingly. >> >> There may appear to be a backwards compatibility here, since the doc >> of extend mentions raising AssertionError - but as said above, the doc >> is wrong, so no regressions in the code are be expected. >> >> Does that sound reasonable (for 3.3)? > > Yes. Third yes. >> Does it make sense to also fix this in 3.2/2.7? Or fix only the >> documentation? Or not touch them at all? I have no opinion about 2.7 as I have not checked what it currently says and does. In the 3.2 docs, we should remove the erroneous assertion about AssertionError. I think it would be good to also say that CElementTree raises TypeError for erroneous input but ElementTree does not and the the latter mistake is fixed in 3.3. Messy reality makes for messy docs. > Our usual approach in cases like this is to not change it in the maint > releases. Why risk breaking someone's code for no particular benefit? > If you want some extra work you could add it as a deprecation warning, > I suppose. The deprecation warning would be that ignoring the error is deprecated ;-). I think this would be a good idea since it would only appear when someone is checking for how to change their code for 3.3. -- Terry Jan Reedy From rdmurray at bitdance.com Fri Mar 16 21:02:01 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 16 Mar 2012 16:02:01 -0400 Subject: [Python-Dev] Raising assertions on wrong element types in ElementTree In-Reply-To: References: <20120316153310.2E7412500F8@webabinitio.net> Message-ID: <20120316200202.AE12D2500F8@webabinitio.net> On Fri, 16 Mar 2012 15:49:33 -0400, Terry Reedy wrote: > On 3/16/2012 11:33 AM, R. David Murray wrote: > > On Fri, 16 Mar 2012 09:38:49 +0200, Eli Bendersky wrote: > >> 1. The behavior of append, insert and extend should be similar in this respect > >> 2. AssertionError is not the customary error in such case - TypeError > >> is much more suitable > >> 3. The C implementation of ElementTree actually raises TypeError in > >> all these methods, by virtue of using PyArg_ParseTuple > >> 4. The Python implementation (at least in 3.2) actually doesn't raise > >> even AssertionError in extend - this was commented out > > > Our usual approach in cases like this is to not change it in the maint > > releases. Why risk breaking someone's code for no particular benefit? > > If you want some extra work you could add it as a deprecation warning, > > I suppose. > > The deprecation warning would be that ignoring the error is deprecated > ;-). I think this would be a good idea since it would only appear when > someone is checking for how to change their code for 3.3. Yes :). But concretely the deprecation warning is that if anyone has code that for some reason *works* with the python version of ElementTree while passing in a non-Element (duck typing?), that will no longer be allowed in 3.3. So it does seem worthwhile to do that. --David From carl at oddbird.net Fri Mar 16 21:22:03 2012 From: carl at oddbird.net (Carl Meyer) Date: Fri, 16 Mar 2012 13:22:03 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F63576A.4080308@gmail.com> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> <4F63576A.4080308@gmail.com> Message-ID: <4F63A0EB.3060706@oddbird.net> Hi Van, On 03/16/2012 08:08 AM, Lindberg, Van wrote: >> Changing the directory name is in fact a new and different (and much >> more invasive) special case, because distutils et al install scripts >> there, and that directory name is part of the distutils install scheme. >> Installers don't care where the Python binary is located, so moving it >> in with the other scripts has very little impact. > > So would changing the distutils install scheme in 3.3 - as defined and > declared by distutils - lead to a change in your code? > > Alternatively stated, do you independently figure out that your > virtualenv is on Windows and then put things in Scripts, etc, or do you > use sysconfig? If sysconfig gave you different (consistent) values > across platforms, how would that affect your code? Both virtualenv and PEP 405 pyvenv figure out the platform at venv-creation time, and hard-code certain information about the correct layout for that platform (Scripts vs bin, as well as lib/pythonx.x vs Lib), so the internal layout of the venv matches the system layout on that platform. The key fact is that there is then no special-casing necessary when code runs within the virtualenv (particularly installer code); the same install scheme that would work in the system Python will also Just Work in the virtualenv. I'm not concerned about changes to distutils/sysconfig install schems to make them more compatible across platforms from the POV of virtualenv; we can easily update the current platform-detection code to do the right thing depending on both platform and Python version. I do share ?ric's concern about whether distutils' legacy install schemes would be updated or not, and how this would affect backwards compatibility for older installer code, but that's pretty much orthogonal to virtualenv/pyvenv. I don't want to make the internal layout of a virtualenv differ from the system Python layout on the same platform, which (IIUC) was Mark's proposal. Hope that clarifies, Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From valhallasw at arctus.nl Fri Mar 16 22:19:10 2012 From: valhallasw at arctus.nl (Merlijn van Deen) Date: Fri, 16 Mar 2012 22:19:10 +0100 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: Hi Guido, Let me start with thanking you for your long reply. It has clarified some points to me, but I am still not certain about some others. I hope I can clarify why I'm confused about this issue in the following. First of all, let me clarify that I wrote my original mail not as 'the guy who wants to serialize stuff' but as 'the guy who wonders what the best way to implement it in python is'. Of course, 'not' is a reasonable answer to that question. On 13 March 2012 23:08, Guido van Rossum wrote: > That was probably written before Python 3. Python 3 also dropped the > long-term backwards compatibilities for the language and stdlib. I am > certainly fine with adding a warning to the docs that this guarantee > does not apply to the Python 2/3 boundary. But I don't think we should > map 8-bit str instances from Python 2 to bytes in Python 3. Yes, backwards compatibility was dropped, but the current pickle module tries to work around this by using a module mapping [1] and aids in loading 8-bit str instances by asking for an encoding [2]. Last, but not least, we can /write/ old version pickles, for which the same module mapping is used, but in reverse. As such, the module suggests in many ways that it should be possible to interchange pickles between python 2 and python 3. > My snipe was mostly in reference to the many other things that can go > wrong with pickled data as your environment evolves (...) I understand your point. However, my interpretation of this issue always was 'if you only pickle built-in types, you'll be fine' - which is apparently wrong. Essentially - my point is this: considering the pickle module is already using several compatibility tricks and considering I am not the only one who would like to read binary data from a pickle in python 3 - even though it might not be the 'right' way to do it - what is there /against/ adding the possibility? Last but not least, this is what people are now doing instead: [1] s = pickle.load(f, encoding='latin1') b = s.encode('latin1') print(zlib.decompress(b)) Which hurts my eyes. In any case - again, thanks for taking the time to respond. I hope I somewhat clarified why I was/am somewhat confused on the issue, and the reasons why I think that it is still a good idea ;-) Best, Merlijn [1] http://hg.python.org/cpython/file/8b2668e60aef/Lib/_compat_pickle.py [2] http://docs.python.org/dev/library/pickle.html#module-interface [3] http://stackoverflow.com/questions/4281619/unpicking-data-pickled-in-python-2-5-in-python-3-1-then-uncompressing-with-zlib From guido at python.org Sat Mar 17 00:57:34 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Mar 2012 16:57:34 -0700 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: OK, how about using encoding=bytes (yes, the type object!)? Or 'bytes' ? --Guido van Rossum (sent from Android phone) On Mar 16, 2012 2:19 PM, "Merlijn van Deen" wrote: > Hi Guido, > > Let me start with thanking you for your long reply. It has clarified > some points to me, but I am still not certain about some others. I > hope I can clarify why I'm confused about this issue in the following. > > First of all, let me clarify that I wrote my original mail not as 'the > guy who wants to serialize stuff' but as 'the guy who wonders what the > best way to implement it in python is'. Of course, 'not' is a > reasonable answer to that question. > > On 13 March 2012 23:08, Guido van Rossum wrote: > > That was probably written before Python 3. Python 3 also dropped the > > long-term backwards compatibilities for the language and stdlib. I am > > certainly fine with adding a warning to the docs that this guarantee > > does not apply to the Python 2/3 boundary. But I don't think we should > > map 8-bit str instances from Python 2 to bytes in Python 3. > > Yes, backwards compatibility was dropped, but the current pickle > module tries to work around this by using a module mapping [1] and > aids in loading 8-bit str instances by asking for an encoding [2]. > Last, but not least, we can /write/ old version pickles, for which > the same module mapping is used, but in reverse. As such, the module > suggests in many ways that it should be possible to interchange > pickles between python 2 and python 3. > > > My snipe was mostly in reference to the many other things that can go > > wrong with pickled data as your environment evolves (...) > I understand your point. However, my interpretation of this issue > always was 'if you only pickle built-in types, you'll be fine' - which > is apparently wrong. > > > Essentially - my point is this: considering the pickle module is > already using several compatibility tricks and considering I am not > the only one who would like to read binary data from a pickle in > python 3 - even though it might not be the 'right' way to do it - what > is there /against/ adding the possibility? > > Last but not least, this is what people are now doing instead: [1] > s = pickle.load(f, encoding='latin1') > b = s.encode('latin1') > print(zlib.decompress(b)) > > Which hurts my eyes. > > In any case - again, thanks for taking the time to respond. I hope I > somewhat clarified why I was/am somewhat confused on the issue, and > the reasons why I think that it is still a good idea ;-) > > Best, > Merlijn > > [1] http://hg.python.org/cpython/file/8b2668e60aef/Lib/_compat_pickle.py > [2] http://docs.python.org/dev/library/pickle.html#module-interface > [3] > http://stackoverflow.com/questions/4281619/unpicking-data-pickled-in-python-2-5-in-python-3-1-then-uncompressing-with-zlib > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mhammond at skippinet.com.au Sat Mar 17 01:43:11 2012 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sat, 17 Mar 2012 11:43:11 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F63A0EB.3060706@oddbird.net> References: <4F603B76.4050004@gmail.com> <4F60C169.9030404@gmail.com> <4F611E39.5070605@skippinet.com.au> <4F6126AB.7020309@gmail.com> <4F612A2E.9030805@gmail.com> <4F627916.7060705@gmail.com> <4F627FCE.4040309@oddbird.net> <4F6284E0.1090606@gmail.com> <4F628586.8030308@oddbird.net> <4F63576A.4080308@gmail.com> <4F63A0EB.3060706@oddbird.net> Message-ID: <4F63DE1F.2040605@skippinet.com.au> On 17/03/2012 7:22 AM, Carl Meyer wrote: ... > I don't want to make the internal layout of a virtualenv differ from the > system Python layout on the same platform, which (IIUC) was Mark's proposal. Just to be clear, I made that suggestion in an effort to keep both myself and Van - that the Python executable would remain in the same place for installed Pythons in the interests of b/w compat, but change it in the virtual env in an effort to keep Van happy when working in such environments. I now fully concede that was a dumb idea :) Mark From skippy.hammond at gmail.com Sat Mar 17 01:53:07 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Sat, 17 Mar 2012 11:53:07 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: <4F63E073.8070607@gmail.com> On 14/03/2012 6:43 AM, VanL wrote: > Following up on conversations at PyCon, I want to bring up one of my > personal hobby horses for change in 3.3: Fix install layout on Windows, > with a side order of making the PATH work better. ... For the sake of brain-storming, how about this: * All executables and scripts go into the root of the Python install. This directory is largely empty now - it is mainly a container for other directories. This would solve the problem of needing 2 directories on the PATH and mean existing tools which locate the executable would work fine. * If cross-platform consistency was desired, then we could consider making other platforms match this. However, if there are issues which might prevent this happening for other platforms (eg, the risk of breaking other 3rd party tools, conventions on the platform ,etc) then it might be worth conceding these considerations apply equally to the Windows installs and we just live with this platform difference. Mark From brian at python.org Sat Mar 17 02:07:19 2012 From: brian at python.org (Brian Curtin) Date: Fri, 16 Mar 2012 20:07:19 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F63E073.8070607@gmail.com> References: <4F63E073.8070607@gmail.com> Message-ID: On Fri, Mar 16, 2012 at 19:53, Mark Hammond wrote: > For the sake of brain-storming, how about this: > > * All executables and scripts go into the root of the Python install. This > directory is largely empty now - it is mainly a container for other > directories. ?This would solve the problem of needing 2 directories on the > PATH and mean existing tools which locate the executable would work fine. How are existing tools locating the executable which would break with a change to bin? As I posted on the tracker, the way which pops in my mind would be to look for "C:\\Python%d%d" % (x, y) but that's already pretty broken. The people I talked to at PyCon about this were Dino from Microsoft and he nudged the guy next to him to ask the same question (I seem to remember this guy worked for an IDE) -- both of them just wanted to be sure they can still find python.exe's location via the registry, which will be fine. I think we'll add a key to go along with InstallPath - BinaryPath probably makes sense. > * If cross-platform consistency was desired, then we could consider making > other platforms match this. ?However, if there are issues which might > prevent this happening for other platforms (eg, the risk of breaking other > 3rd party tools, conventions on the platform ,etc) then it might be worth > conceding these considerations apply equally to the Windows installs and we > just live with this platform difference. I don't think we're going to defeat the Unix army with their fleets of distro packagers and torch wielding purists. If anyone's going to move, my money's on Windows. From mhammond at skippinet.com.au Sat Mar 17 02:12:11 2012 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sat, 17 Mar 2012 12:12:11 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F63E073.8070607@gmail.com> Message-ID: <4F63E4EB.2050301@skippinet.com.au> On 17/03/2012 12:07 PM, Brian Curtin wrote: > On Fri, Mar 16, 2012 at 19:53, Mark Hammond wrote: >> For the sake of brain-storming, how about this: >> >> * All executables and scripts go into the root of the Python install. This >> directory is largely empty now - it is mainly a container for other >> directories. This would solve the problem of needing 2 directories on the >> PATH and mean existing tools which locate the executable would work fine. > > How are existing tools locating the executable which would break with > a change to bin? As I posted on the tracker, the way which pops in my > mind would be to look for "C:\\Python%d%d" % (x, y) but that's already > pretty broken. As I just replied in the tracker :) They typically look up the InstallPath key in the registry and look for python.exe there - see the link to that activate.bat file I posted early in the thread. > The people I talked to at PyCon about this were Dino > from Microsoft and he nudged the guy next to him to ask the same > question (I seem to remember this guy worked for an IDE) -- both of > them just wanted to be sure they can still find python.exe's location > via the registry, which will be fine. I think we'll add a key to go > along with InstallPath - BinaryPath probably makes sense. While I wouldn't object to that, it would seem redundant - if the whole point of this is to standardize the locations, then looking for "bin/python.exe" relative to the existing InstallPath key should also be reliable and hopefully permanent. At the risk of repeating myself too many times, my concern is with 3rd party tools who (a) will break with the new scheme and need to be updated and (b) even after updating will still need the burden of supporting both the old and the new schemes. I simply don't see the benefit that makes this worthwhile. >> * If cross-platform consistency was desired, then we could consider making >> other platforms match this. However, if there are issues which might >> prevent this happening for other platforms (eg, the risk of breaking other >> 3rd party tools, conventions on the platform ,etc) then it might be worth >> conceding these considerations apply equally to the Windows installs and we >> just live with this platform difference. > > I don't think we're going to defeat the Unix army with their fleets of > distro packagers and torch wielding purists. If anyone's going to > move, my money's on Windows. Right - but why? Who wins? Where is the evidence of the pain this has caused people over the last 18 years or so since Windows has been doing this? Mark From carl at oddbird.net Sat Mar 17 02:25:45 2012 From: carl at oddbird.net (Carl Meyer) Date: Fri, 16 Mar 2012 18:25:45 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F63E073.8070607@gmail.com> References: <4F63E073.8070607@gmail.com> Message-ID: <4F63E819.3040108@oddbird.net> Hi Mark, On 03/16/2012 05:53 PM, Mark Hammond wrote: > * All executables and scripts go into the root of the Python install. > This directory is largely empty now - it is mainly a container for other > directories. This would solve the problem of needing 2 directories on > the PATH and mean existing tools which locate the executable would work > fine. I hate to seem like I'm piling on now after panning your last brainstorm :-), but... this would be quite problematic for virtualenv users, many of whom do rely on the fact that the virtualenv "stuff" is confined to within a limited set of known subdirectories, and they can overlay a virtualenv and their own project code with just a few virtualenv directories vcs-ignored. I would prefer either the status quo or the proposed cross-platform harmonization. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From v+python at g.nevcal.com Sat Mar 17 02:36:12 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 16 Mar 2012 18:36:12 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F63E819.3040108@oddbird.net> References: <4F63E073.8070607@gmail.com> <4F63E819.3040108@oddbird.net> Message-ID: <4F63EA8C.1060007@g.nevcal.com> On 3/16/2012 6:25 PM, Carl Meyer wrote: > Hi Mark, > > On 03/16/2012 05:53 PM, Mark Hammond wrote: >> * All executables and scripts go into the root of the Python install. >> This directory is largely empty now - it is mainly a container for other >> directories. This would solve the problem of needing 2 directories on >> the PATH and mean existing tools which locate the executable would work >> fine. > I hate to seem like I'm piling on now after panning your last brainstorm > :-), but... this would be quite problematic for virtualenv users, many > of whom do rely on the fact that the virtualenv "stuff" is confined to > within a limited set of known subdirectories, and they can overlay a > virtualenv and their own project code with just a few virtualenv > directories vcs-ignored. > > I would prefer either the status quo or the proposed cross-platform > harmonization. Yes, it seems fruitless to make directory structure changes without achieving cross-platform consistency. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sat Mar 17 03:30:31 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 16 Mar 2012 19:30:31 -0700 Subject: [Python-Dev] lzma Message-ID: When I build python from sources I have no lzma support (module _lzma cannot be built). There are lzma packages installed in my Ubuntu 11.10 box: lzma, lzma-dev and lzma-sources. I can see lib files (headers are also can be found in linux-headers): andrew at tiktaalik ~/p/cpython> locate lzma.so /usr/lib/liblzma.so.2 /usr/lib/liblzma.so.2.0.0 I can live with that but if somebody can point me the way to build python with lzma support please do. -- Thanks, Andrew Svetlov From senthil at uthcode.com Sat Mar 17 03:38:18 2012 From: senthil at uthcode.com (Senthil Kumaran) Date: Fri, 16 Mar 2012 19:38:18 -0700 Subject: [Python-Dev] lzma In-Reply-To: References: Message-ID: <20120317023818.GA1977@mathmagic> On Fri, Mar 16, 2012 at 07:30:31PM -0700, Andrew Svetlov wrote: > When I build python from sources I have no lzma support (module _lzma > cannot be built). > I have liblzma-dev, liblzma2 and lzma packages installed on ubuntu. I am able to build and import lzma module. Thanks, Senthil From andrew.svetlov at gmail.com Sat Mar 17 03:44:44 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 16 Mar 2012 19:44:44 -0700 Subject: [Python-Dev] lzma In-Reply-To: <20120317023818.GA1977@mathmagic> References: <20120317023818.GA1977@mathmagic> Message-ID: liblzma-dev has solved my problem. Thank you, Senthil. On Fri, Mar 16, 2012 at 7:38 PM, Senthil Kumaran wrote: > > On Fri, Mar 16, 2012 at 07:30:31PM -0700, Andrew Svetlov wrote: >> When I build python from sources I have no lzma support (module _lzma >> cannot be built). >> > > I have liblzma-dev, liblzma2 and lzma packages installed on ubuntu. ?I > am able to build and import lzma module. > > Thanks, > Senthil > > -- Thanks, Andrew Svetlov From skippy.hammond at gmail.com Sat Mar 17 06:28:57 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Sat, 17 Mar 2012 16:28:57 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F63E819.3040108@oddbird.net> References: <4F63E073.8070607@gmail.com> <4F63E819.3040108@oddbird.net> Message-ID: <4F642119.9060804@gmail.com> On 17/03/2012 12:25 PM, Carl Meyer wrote: > Hi Mark, > > On 03/16/2012 05:53 PM, Mark Hammond wrote: >> * All executables and scripts go into the root of the Python install. >> This directory is largely empty now - it is mainly a container for other >> directories. This would solve the problem of needing 2 directories on >> the PATH and mean existing tools which locate the executable would work >> fine. > > I hate to seem like I'm piling on now after panning your last brainstorm > :-), but... this would be quite problematic for virtualenv users, many > of whom do rely on the fact that the virtualenv "stuff" is confined to > within a limited set of known subdirectories, and they can overlay a > virtualenv and their own project code with just a few virtualenv > directories vcs-ignored. > > I would prefer either the status quo or the proposed cross-platform > harmonization. Yeah, fair enough. I should have indicated it was 1/2 tongue-in-cheek, but figured it worth throwing it out there anyway :) OTOH, the part that wasn't tongue-in-cheek was the part that said "why not change the other platforms instead of windows" (then wait for the inevitable replies), then "so those same reasons apply to Windows too" - eg "fleets of torch wielding windows admins" :) Breaking the few tools I'm concerned about vs asking Van etc to continue taking the pain he feels isn't going to mean the end of the world for any of us. So given the stakes in this particular discussion aren't that high, I'll try and summarize the thread over the next few days (or someone can beat me to it if they prefer) and we can ask someone semi-impartial to make a decision. I'd be happy to nominate MvL if he feels so inclined (even though I haven't asked him). Cheers, Mark From valhallasw at arctus.nl Sat Mar 17 10:14:07 2012 From: valhallasw at arctus.nl (Merlijn van Deen) Date: Sat, 17 Mar 2012 10:14:07 +0100 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: On 17 March 2012 00:57, Guido van Rossum wrote: > OK, how about using encoding=bytes (yes, the type object!)? Or 'bytes' ? encoding=bytes makes (at least intuitive) sense to me; encoding='bytes' would imply there is an encoding with name 'bytes' that somehow does b''.decode('bytes')=b'', and would disallow anyone to create a new encoding with the name 'bytes'. I'll take a look at rewriting my patch later this weekend, and I'll also give the documentation (both the 'consider not using pickle for long time storage' and the docs for this setting) some thought. Best, Merlijn From stefan_ml at behnel.de Sat Mar 17 10:43:43 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 17 Mar 2012 10:43:43 +0100 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: Guido van Rossum, 17.03.2012 00:57: > OK, how about using encoding=bytes (yes, the type object!)? Or 'bytes' ? In lxml, there was an "encoding=unicode" option that would let the XML/HTML/text serialisation function return a Unicode string. This was eventually deprecated in favour of "encoding='unicode'" when ElementTree gained this feature as well some years later. Arguably, this was because there no longer was a unicode type in the then existing Py3, but ... Anyway, given that there is at least some precedence, I'd prefer the name "bytes" over the bare bytes type. Regarding possible naming conflicts, I don't see any sense in calling an actual encoding "bytes" that does anything but returning bare bytes in a bytes object, as is the case here. Stefan From p.f.moore at gmail.com Sat Mar 17 12:57:00 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 17 Mar 2012 11:57:00 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F642119.9060804@gmail.com> References: <4F63E073.8070607@gmail.com> <4F63E819.3040108@oddbird.net> <4F642119.9060804@gmail.com> Message-ID: On 17 March 2012 05:28, Mark Hammond wrote: >> I hate to seem like I'm piling on now after panning your last brainstorm >> :-), but... this would be quite problematic for virtualenv users, many >> of whom do rely on the fact that the virtualenv "stuff" is confined to >> within a limited set of known subdirectories, and they can overlay a >> virtualenv and their own project code with just a few virtualenv >> directories vcs-ignored. >> >> I would prefer either the status quo or the proposed cross-platform >> harmonization. I work purely Windows-only, and I have a few scripts that manage virtualenvs for myself (for example, sort of a personal virtalenv-wrapper for Powershell - see https://bitbucket.org/pmoore/poshpy for a work-in-progress version). They have special casing for the differences in layout between standard installs, build directories, and virtualenvs. Changing the layout would cause these tools to need to change. In theory, putting python.exe/pythonw.exe into "Scripts" would simplify them (no need to cater for the cases where I need to put 2 directories on PATH), and changing Scripts -> bin would be trivial. But in practice, it would mean that I need to check (somehow) the Python version and adjust the layout used accordingly. As there is no way of knowing the Python version without running Python, this is too slow to be practical. So while the changes are in theory harmless in isolation (except the library locations - changing those *would* cause pain) the need to support multiple versions would make this a major issue for me. So, I prefer the status quo. If necessary, I can live with the change to rename scripts as bin and put the Python executables in there (some cost, but some small benefit as well) but I oppose changing the library locations (all cost, no gain for me). All of this presupposes that both the standard installer *and* virtualenv change. I suspect that having virtualenv respect the old layout for 3.2 and earlier, and the new one for 3.3+, could be messy, though, so that's not guaranteed, I guess... > Breaking the few tools I'm concerned about vs asking Van etc to continue > taking the pain he feels isn't going to mean the end of the world for any of > us. Agreed. I can't say my pain is any more important than Van's, but the same applies the other way round :-) > So given the stakes in this particular discussion aren't that high, > I'll try and summarize the thread over the next few days (or someone can > beat me to it if they prefer) and we can ask someone semi-impartial to make > a decision. ?I'd be happy to nominate MvL if he feels so inclined (even > though I haven't asked him). Sounds reasonable. I'd suggest that Van should probably report any other examples where someone would benefit from this change - at the moment unless I've misread the thread, it seems like he's the only example of someone who would gain. Paul. From storchaka at gmail.com Sat Mar 17 15:07:27 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 17 Mar 2012 16:07:27 +0200 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F63E073.8070607@gmail.com> <4F63E819.3040108@oddbird.net> <4F642119.9060804@gmail.com> Message-ID: 17.03.12 13:57, Paul Moore ???????(??): > As there is no > way of knowing the Python version without running Python, this is too > slow to be practical. Cold start: $ time python3 --version Python 3.1.2 real 0m0.073s user 0m0.004s sys 0m0.004s Hot start: $ time python3 --version Python 3.1.2 real 0m0.007s user 0m0.004s sys 0m0.004s From valhallasw at arctus.nl Sat Mar 17 15:20:54 2012 From: valhallasw at arctus.nl (Merlijn van Deen) Date: Sat, 17 Mar 2012 15:20:54 +0100 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: On 17 March 2012 10:43, Stefan Behnel wrote: > In lxml, there was an "encoding=unicode" option that would let the > XML/HTML/text serialisation function return a Unicode string. This was > eventually deprecated in favour of "encoding='unicode'" when ElementTree > gained this feature as well some years later. That's this issue: http://bugs.python.org/issue8047 The thread also suggests the options encoding=False and encoding=None Considering ET uses encoding="unicode" to signal 'don't encode', I agree with you that using encoding="bytes" to signal 'don't encode' would be sensible. However, ET /also/ allows encoding=str.What about allowing both encoding="bytes" /and/ encoding=bytes? Best, Merlijn From stefan_ml at behnel.de Sat Mar 17 15:28:17 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 17 Mar 2012 15:28:17 +0100 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: Merlijn van Deen, 17.03.2012 15:20: > On 17 March 2012 10:43, Stefan Behnel wrote: >> In lxml, there was an "encoding=unicode" option that would let the >> XML/HTML/text serialisation function return a Unicode string. This was >> eventually deprecated in favour of "encoding='unicode'" when ElementTree >> gained this feature as well some years later. > > That's this issue: http://bugs.python.org/issue8047 > > The thread also suggests the options > encoding=False > and > encoding=None > > Considering ET uses encoding="unicode" to signal 'don't encode', I > agree with you that using encoding="bytes" to signal 'don't encode' > would be sensible. However, ET /also/ allows encoding=str.What about > allowing both encoding="bytes" /and/ encoding=bytes? It doesn't read well for the unicode type any more because it's gone in Py3 (and "encoding=str" just looks weird). It's less awkward for the bytes type. However, why should there be two ways to do it? Stefan From p.f.moore at gmail.com Sat Mar 17 15:50:24 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 17 Mar 2012 14:50:24 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F63E073.8070607@gmail.com> <4F63E819.3040108@oddbird.net> <4F642119.9060804@gmail.com> Message-ID: On 17 March 2012 14:07, Serhiy Storchaka wrote: > 17.03.12 13:57, Paul Moore ???????(??): > >> As there is no >> way of knowing the Python version without running Python, this is too >> slow to be practical. > > > Cold start: > $ time python3 --version > Python 3.1.2 > > real ? ?0m0.073s > user ? ?0m0.004s > sys ? ? 0m0.004s > > Hot start: > $ time python3 --version > Python 3.1.2 > > real ? ?0m0.007s > user ? ?0m0.004s > sys ? ? 0m0.004s Blame it on Windows or my overloaded PC if you must :-) but I get perceptible delays on a cold start at times. Plus, I'd probably need to do this in code that enumerates all installed Pythons and virtualenvs - that could be 10 installations. I've never tried it in anger, so I could be worrying over nothing, but to an extent that's my point - I don't *need* to know the version unless I have to have version-specific code to define the layout. And if I were starting Python up, I'd probably be better just importing sys and sysconfig, and using sys.executable and sysconfig.get_path('scripts'). And there's the chicken-and-egg problem - if I don't know the version, I don't know where python.exe is, so how can I run it to find the version? Meh. None of this is a real issue. It's just some extra messy coding. But Van's point is that this proposal gives him less hard coding. Beyond pointing out that it gives me more, I don't have much to add. Paul. From guido at python.org Sat Mar 17 16:00:21 2012 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Mar 2012 08:00:21 -0700 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: One reason to use 'bytes' instead of bytes is that it is a string that can be specified e.g. in a config file. On Sat, Mar 17, 2012 at 7:28 AM, Stefan Behnel wrote: > Merlijn van Deen, 17.03.2012 15:20: >> On 17 March 2012 10:43, Stefan Behnel wrote: >>> In lxml, there was an "encoding=unicode" option that would let the >>> XML/HTML/text serialisation function return a Unicode string. This was >>> eventually deprecated in favour of "encoding='unicode'" when ElementTree >>> gained this feature as well some years later. >> >> That's this issue: http://bugs.python.org/issue8047 >> >> The thread also suggests the options >> ? ? encoding=False >> and >> ? ? encoding=None >> >> Considering ET uses encoding="unicode" to signal 'don't encode', I >> agree with you that using encoding="bytes" to signal 'don't encode' >> would be sensible. However, ET /also/ allows encoding=str.What about >> allowing both encoding="bytes" /and/ encoding=bytes? > > It doesn't read well for the unicode type any more because it's gone in Py3 > (and "encoding=str" just looks weird). It's less awkward for the bytes type. > > However, why should there be two ways to do it? > > Stefan > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Sat Mar 17 16:28:09 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 17 Mar 2012 17:28:09 +0200 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: 17.03.12 17:00, Guido van Rossum ???????(??): > One reason to use 'bytes' instead of bytes is that it is a string that > can be specified e.g. in a config file. Thus, there are no reasons to use bytes instead of 'bytes'. From g.brandl at gmx.net Sat Mar 17 16:49:37 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 17 Mar 2012 16:49:37 +0100 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: Message-ID: On 03/15/2012 01:17 AM, victor.stinner wrote: > http://hg.python.org/cpython/rev/27441e0d6a75 > changeset: 75672:27441e0d6a75 > user: Victor Stinner > date: Thu Mar 15 01:17:09 2012 +0100 > summary: > Issue #10278: Add an optional strict argument to time.steady(), False by default > > files: > Doc/library/time.rst | 7 +++- > Lib/test/test_time.py | 10 +++++ > Modules/timemodule.c | 58 +++++++++++++++++++++--------- > 3 files changed, 57 insertions(+), 18 deletions(-) > > > diff --git a/Doc/library/time.rst b/Doc/library/time.rst > --- a/Doc/library/time.rst > +++ b/Doc/library/time.rst > @@ -226,7 +226,7 @@ > The earliest date for which it can generate a time is platform-dependent. > > > -.. function:: steady() > +.. function:: steady(strict=False) > > .. index:: > single: benchmarking > @@ -236,6 +236,11 @@ > adjusted. The reference point of the returned value is undefined so only the > difference of consecutive calls is valid. > > + If available, a monotonic clock is used. By default, if *strict* is False, > + the function falls back to another clock if the monotonic clock failed or is > + not available. If *strict* is True, raise an :exc:`OSError` on error or > + :exc:`NotImplementedError` if no monotonic clock is available. This is not clear to me. Why wouldn't it raise OSError on error even with strict=False? Please clarify which exception is raised in which case. Georg From valhallasw at arctus.nl Sat Mar 17 17:09:39 2012 From: valhallasw at arctus.nl (Merlijn van Deen) Date: Sat, 17 Mar 2012 17:09:39 +0100 Subject: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784) In-Reply-To: References: <8DE609C2-0FF4-4412-B26E-B453C67EF0F0@voidspace.org.uk> Message-ID: On 17 March 2012 16:28, Serhiy Storchaka wrote: > Thus, there are no reasons to use bytes instead of 'bytes'. Aesthetics ;-) I've implemented the encoding="bytes" version [1]. Thank you all for your input! Merlijn [1] http://bugs.python.org/issue6784#msg156166 From g.brandl at gmx.net Sat Mar 17 17:46:42 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 17 Mar 2012 17:46:42 +0100 Subject: [Python-Dev] cpython (3.2): 3.2 explain json.dumps for non-string keys in dicts. closes issue6566. Patch In-Reply-To: References: Message-ID: On 03/17/2012 08:41 AM, senthil.kumaran wrote: > http://hg.python.org/cpython/rev/613919591a05 > changeset: 75778:613919591a05 > branch: 3.2 > parent: 75771:32d3ecacdabf > user: Senthil Kumaran > date: Sat Mar 17 00:40:34 2012 -0700 > summary: > 3.2 explain json.dumps for non-string keys in dicts. closes issue6566. Patch contributed Kirubakaran Athmanathan > > files: > Doc/library/json.rst | 8 ++++++++ > 1 files changed, 8 insertions(+), 0 deletions(-) > > > diff --git a/Doc/library/json.rst b/Doc/library/json.rst > --- a/Doc/library/json.rst > +++ b/Doc/library/json.rst > @@ -168,6 +168,14 @@ > so trying to serialize multiple objects with repeated calls to > :func:`dump` using the same *fp* will result in an invalid JSON file. > > + .. note:: > + > + Keys in key/value pairs of JSON are always of the type :class:`str`. When > + a dictionary is converted into JSON, all the keys of the dictionary are > + coerced to strings. As a result of this, if a dictionary is convered > + into JSON and then back into a dictionary, the dictionary may not equal > + the original one. That is, ``loads(dumps(x)) != x`` if x has non-string > + keys. > > .. function:: load(fp, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) This is just a minor nitpick, and it absolutely is not specific to you, Senthil: please try to keep the rst file structuring with newlines intact. In particular, I place two blank lines between top-level function/class descriptions because single blank lines already occur so often in rst markup. When you add paragraphs, please try to keep the blank lines. Georg From merwok at netwok.org Sat Mar 17 18:20:31 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sat, 17 Mar 2012 13:20:31 -0400 Subject: [Python-Dev] [Python-checkins] cpython (3.2): this stuff will actually be new in 3.2.4 In-Reply-To: References: Message-ID: <4F64C7DF.7000506@netwok.org> Hi, > changeset: 509b222679e8 > branch: 3.2 > user: Benjamin Peterson > date: Wed Mar 07 18:49:43 2012 -0600 > summary: > this stuff will actually be new in 3.2.4 > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -2,6 +2,57 @@ > Python News > +++++++++++ > > +What's New in Python 3.2.4 > +========================== > + > +*Release date: XX-XX-XXXX* Thanks for sorting this out. There is however at least one mistake: > +- Issue #6884: Fix long-standing bugs with MANIFEST.in parsing in distutils > + on Windows. I?m quite sure Georg transplanted that commit to the 3.2.3 release clone, and hope you did likewise for 2.7.3. Regards From g.brandl at gmx.net Sat Mar 17 18:38:21 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 17 Mar 2012 18:38:21 +0100 Subject: [Python-Dev] [Python-checkins] cpython (3.2): this stuff will actually be new in 3.2.4 In-Reply-To: <4F64C7DF.7000506@netwok.org> References: <4F64C7DF.7000506@netwok.org> Message-ID: On 03/17/2012 06:20 PM, ?ric Araujo wrote: > Hi, > >> changeset: 509b222679e8 >> branch: 3.2 >> user: Benjamin Peterson >> date: Wed Mar 07 18:49:43 2012 -0600 >> summary: >> this stuff will actually be new in 3.2.4 > >> diff --git a/Misc/NEWS b/Misc/NEWS >> --- a/Misc/NEWS >> +++ b/Misc/NEWS >> @@ -2,6 +2,57 @@ >> Python News >> +++++++++++ >> >> +What's New in Python 3.2.4 >> +========================== >> + >> +*Release date: XX-XX-XXXX* > > Thanks for sorting this out. There is however at least one mistake: > >> +- Issue #6884: Fix long-standing bugs with MANIFEST.in parsing in distutils >> + on Windows. > > I?m quite sure Georg transplanted that commit to the 3.2.3 release > clone, and hope you did likewise for 2.7.3. Fixed, thanks. Georg From tjreedy at udel.edu Sat Mar 17 20:16:25 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 17 Mar 2012 15:16:25 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F63E073.8070607@gmail.com> <4F63E819.3040108@oddbird.net> <4F642119.9060804@gmail.com> Message-ID: On 3/17/2012 10:50 AM, Paul Moore wrote: > Meh. None of this is a real issue. It's just some extra messy coding. > But Van's point is that this proposal gives him less hard coding. > Beyond pointing out that it gives me more, I don't have much to add. I suspect a case could be made that harmonization now will benefit multiple people in, say, 5 years, especially if by then one only supported 3.3+ while supporting multiple platforms. It would be the same rationale as that for 3.0, and especially the bytes to unicode change for text. (As I remember, we are only 3 years in on that one ;-). I leave it to Van to actually explain and make the argument. -- Terry Jan Reedy From fuzzyman at voidspace.org.uk Sat Mar 17 21:47:47 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 17 Mar 2012 13:47:47 -0700 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: Message-ID: On 17 Mar 2012, at 08:49, Georg Brandl wrote: > On 03/15/2012 01:17 AM, victor.stinner wrote: >> http://hg.python.org/cpython/rev/27441e0d6a75 >> changeset: 75672:27441e0d6a75 >> user: Victor Stinner >> date: Thu Mar 15 01:17:09 2012 +0100 >> summary: >> Issue #10278: Add an optional strict argument to time.steady(), False by default >> >> files: >> Doc/library/time.rst | 7 +++- >> Lib/test/test_time.py | 10 +++++ >> Modules/timemodule.c | 58 +++++++++++++++++++++--------- >> 3 files changed, 57 insertions(+), 18 deletions(-) >> >> >> diff --git a/Doc/library/time.rst b/Doc/library/time.rst >> --- a/Doc/library/time.rst >> +++ b/Doc/library/time.rst >> @@ -226,7 +226,7 @@ >> The earliest date for which it can generate a time is platform-dependent. >> >> >> -.. function:: steady() >> +.. function:: steady(strict=False) >> >> .. index:: >> single: benchmarking >> @@ -236,6 +236,11 @@ >> adjusted. The reference point of the returned value is undefined so only the >> difference of consecutive calls is valid. >> >> + If available, a monotonic clock is used. By default, if *strict* is False, >> + the function falls back to another clock if the monotonic clock failed or is >> + not available. If *strict* is True, raise an :exc:`OSError` on error or >> + :exc:`NotImplementedError` if no monotonic clock is available. > > This is not clear to me. Why wouldn't it raise OSError on error even with > strict=False? Please clarify which exception is raised in which case. It seems clear to me. It doesn't raise exceptions when strict=False because it falls back to a non-monotonic clock. If strict is True and a non-monotonic clock is not available it raises OSError or NotImplementedError. Michael > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From g.brandl at gmx.net Sat Mar 17 23:04:46 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 17 Mar 2012 23:04:46 +0100 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: Message-ID: On 03/17/2012 09:47 PM, Michael Foord wrote: > > On 17 Mar 2012, at 08:49, Georg Brandl wrote: > >> On 03/15/2012 01:17 AM, victor.stinner wrote: >>> http://hg.python.org/cpython/rev/27441e0d6a75 changeset: >>> 75672:27441e0d6a75 user: Victor Stinner >>> date: Thu Mar 15 01:17:09 2012 +0100 >>> summary: Issue #10278: Add an optional strict argument to time.steady(), >>> False by default >>> >>> files: Doc/library/time.rst | 7 +++- Lib/test/test_time.py | 10 >>> +++++ Modules/timemodule.c | 58 +++++++++++++++++++++--------- 3 files >>> changed, 57 insertions(+), 18 deletions(-) >>> >>> >>> diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- >>> a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -226,7 +226,7 @@ The >>> earliest date for which it can generate a time is platform-dependent. >>> >>> >>> -.. function:: steady() +.. function:: steady(strict=False) >>> >>> .. index:: single: benchmarking @@ -236,6 +236,11 @@ adjusted. The >>> reference point of the returned value is undefined so only the difference >>> of consecutive calls is valid. >>> >>> + If available, a monotonic clock is used. By default, if *strict* is >>> False, + the function falls back to another clock if the monotonic >>> clock failed or is + not available. If *strict* is True, raise an >>> :exc:`OSError` on error or + :exc:`NotImplementedError` if no monotonic >>> clock is available. >> >> This is not clear to me. Why wouldn't it raise OSError on error even with >> strict=False? Please clarify which exception is raised in which case. > > It seems clear to me. It doesn't raise exceptions when strict=False because > it falls back to a non-monotonic clock. If strict is True and a non-monotonic > clock is not available it raises OSError or NotImplementedError. So errors are ignored when strict is false? Georg From eric at trueblade.com Sat Mar 17 23:07:33 2012 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 17 Mar 2012 18:07:33 -0400 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: Message-ID: <4F650B25.4060901@trueblade.com> On 3/17/2012 4:47 PM, Michael Foord wrote: > > On 17 Mar 2012, at 08:49, Georg Brandl wrote: > >> On 03/15/2012 01:17 AM, victor.stinner wrote: >>> + If available, a monotonic clock is used. By default, if *strict* is False, >>> + the function falls back to another clock if the monotonic clock failed or is >>> + not available. If *strict* is True, raise an :exc:`OSError` on error or >>> + :exc:`NotImplementedError` if no monotonic clock is available. >> >> This is not clear to me. Why wouldn't it raise OSError on error even with >> strict=False? Please clarify which exception is raised in which case. > > It seems clear to me. It doesn't raise exceptions when strict=False because > it falls back to a non-monotonic clock. If strict is True and a non- > monotonic clock is not available it raises OSError or NotImplementedError. I have to agree with Georg. Looking at the code, it appears OSError can be raised with both strict=True and strict=False (since floattime() can raise OSError). The text needs to make it clear OSError can always be raised. I also think "By default, if strict is False" confuses things. If there's a default behavior with strict=False, what's the non-default behavior? I suggest dropping "By default". Eric. From fuzzyman at voidspace.org.uk Sat Mar 17 23:16:51 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 17 Mar 2012 15:16:51 -0700 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: Message-ID: On 17 Mar 2012, at 15:04, Georg Brandl wrote: > On 03/17/2012 09:47 PM, Michael Foord wrote: >> >> On 17 Mar 2012, at 08:49, Georg Brandl wrote: >> >>> On 03/15/2012 01:17 AM, victor.stinner wrote: >>>> http://hg.python.org/cpython/rev/27441e0d6a75 changeset: >>>> 75672:27441e0d6a75 user: Victor Stinner >>>> date: Thu Mar 15 01:17:09 2012 +0100 >>>> summary: Issue #10278: Add an optional strict argument to time.steady(), >>>> False by default >>>> >>>> files: Doc/library/time.rst | 7 +++- Lib/test/test_time.py | 10 >>>> +++++ Modules/timemodule.c | 58 +++++++++++++++++++++--------- 3 files >>>> changed, 57 insertions(+), 18 deletions(-) >>>> >>>> >>>> diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- >>>> a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -226,7 +226,7 @@ The >>>> earliest date for which it can generate a time is platform-dependent. >>>> >>>> >>>> -.. function:: steady() +.. function:: steady(strict=False) >>>> >>>> .. index:: single: benchmarking @@ -236,6 +236,11 @@ adjusted. The >>>> reference point of the returned value is undefined so only the difference >>>> of consecutive calls is valid. >>>> >>>> + If available, a monotonic clock is used. By default, if *strict* is >>>> False, + the function falls back to another clock if the monotonic >>>> clock failed or is + not available. If *strict* is True, raise an >>>> :exc:`OSError` on error or + :exc:`NotImplementedError` if no monotonic >>>> clock is available. >>> >>> This is not clear to me. Why wouldn't it raise OSError on error even with >>> strict=False? Please clarify which exception is raised in which case. >> >> It seems clear to me. It doesn't raise exceptions when strict=False because >> it falls back to a non-monotonic clock. If strict is True and a non-monotonic >> clock is not available it raises OSError or NotImplementedError. > > So errors are ignored when strict is false? Well, as described in the documentation an error in finding a monotonic clock causes the function to fallback to a different clock. So you could interpret that as either errors are ignored, or it isn't an error in the first place. I don't see how the following is ambiguous, but you're obviously having difficulty with it. Perhaps you can suggest another wording. if *strict* is False, the function falls back to another clock if the monotonic clock failed or is not available. The note from Eric notwithstanding though. Michael > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From barry at python.org Sat Mar 17 23:43:16 2012 From: barry at python.org (Barry Warsaw) Date: Sat, 17 Mar 2012 18:43:16 -0400 Subject: [Python-Dev] [Python-checkins] cpython (2.6): Added tag v2.6.8rc2 for changeset bd9e1a02e3e3 In-Reply-To: References: Message-ID: <20120317184316.7245fcce@limelight.wooz.org> On Mar 17, 2012, at 11:34 PM, barry.warsaw wrote: >http://hg.python.org/cpython/rev/6144a0748a95 >changeset: 75809:6144a0748a95 >branch: 2.6 >parent: 75806:bd9e1a02e3e3 >user: Barry Warsaw >date: Sat Mar 17 18:34:05 2012 -0400 >summary: > Added tag v2.6.8rc2 for changeset bd9e1a02e3e3 > >files: > .hgtags | 1 + > 1 files changed, 1 insertions(+), 0 deletions(-) > > >diff --git a/.hgtags b/.hgtags >--- a/.hgtags >+++ b/.hgtags >@@ -140,3 +140,4 @@ > caab08cd2b3eb5a6f78479b2513b65d36c754f41 v2.6.8rc1 > 5356b6c7fd66664679f9bd71f7cd085239934e43 v2.6.8rc1 > 1d1b7b9fad48bd0dc60dc8a06cca4459ef273127 v2.6.8rc2 >+bd9e1a02e3e329fa7b6da06113090a401909c4ea v2.6.8rc2 I don't know why Mercurial does this. Here's what *I* did: * After I committed the last of my PEP 101 changes to the 2.6 branch, I did release.py --tag 2.6.8rc2. It sure looks like that added the tag. * Then I switched to the 2.7 branch, pulled and merged the changes from 2.6, `hg revert -ar .` then marked all conflicts as resolved `hg resolve -am`. AFAICT, that's the way to null merge 2.6 to 2.7. * Then I pushed my changes. * Switching back to my 2.6 branch, I then tried to `release -export 2.6.8rc2` and got an error that v2.6.8rc2 tag wasn't found. At this point I *retagged* for 2.6.8rc2, switch to default and pushed. This makes me a little wary about what's actually in the 2.6.8rc2 tarball, but I'm building and testing it now, and from visual inspection it *looks* okay, so I'm inclined to chalk this up to either a Mercurial wart, or my boneheaded use of it. Cheers, -Barry From barry at python.org Sun Mar 18 00:13:18 2012 From: barry at python.org (Barry Warsaw) Date: Sat, 17 Mar 2012 19:13:18 -0400 Subject: [Python-Dev] [Python-checkins] cpython (2.6): Added tag v2.6.8rc2 for changeset bd9e1a02e3e3 In-Reply-To: References: <20120317184316.7245fcce@limelight.wooz.org> Message-ID: <20120317191318.29f88bc2@limelight.wooz.org> On Mar 18, 2012, at 12:03 AM, Georg Brandl wrote: >I'm afraid it's the latter: tags are entries in .hgtags. So when you >completely null-merge your 2.6 changes into 2.7, you are basically removing >the tag from the 2.7 branch. And since to find tags, Mercurial looks in the >.hgtags files of all active branch heads, you are basically hiding the tag >when you merge 2.6 into 2.7, at which point it becomes an inactive branch >head. D'oh. Okay, so leave the .hgtags file alone when null merging 2.6->2.7. Actually, that probably applies to all forward merges, so I think this would be useful information for either the devguide or PEP 101, or both. I updated the latter right at the --tag step. Thanks, -Barry From barry at python.org Sun Mar 18 05:13:10 2012 From: barry at python.org (Barry Warsaw) Date: Sun, 18 Mar 2012 00:13:10 -0400 Subject: [Python-Dev] [Python-checkins] cpython (2.6): Added tag v2.6.8rc2 for changeset bd9e1a02e3e3 In-Reply-To: <4F651AF8.8030700@netwok.org> References: <20120317184316.7245fcce@limelight.wooz.org> <4F651AF8.8030700@netwok.org> Message-ID: <20120318001310.1ce284b5@limelight.wooz.org> On Mar 17, 2012, at 07:15 PM, ?ric Araujo wrote: >Note that duplicate entries in .hgtags (when a tag was redone) should >not be ?cleaned up?: the presence of the old changeset hash greatly >helps conflict resolution. (If someone pulled the repo with the old tag >and later pulls and updates, then they don?t have to find out which hash >is the right tag, they just accept all changes from the updated file >into their local file.) But if someone wants to grab the 2.6.8rc2 tag, which changeset do they get? I guess the last one... maybe? >This problem in the future can be avoided by merging all changesets from >X.Y to X.Y+1, not null-merging, unless I misunderstand something. Except in this case, there's probably not much useful in the 2.6.8 changes that are appropriate for 2.7. -Barry From g.brandl at gmx.net Sun Mar 18 07:29:46 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 18 Mar 2012 07:29:46 +0100 Subject: [Python-Dev] cpython (2.6): Added tag v2.6.8rc2 for changeset bd9e1a02e3e3 In-Reply-To: <4F651AF8.8030700@netwok.org> References: <20120317184316.7245fcce@limelight.wooz.org> <4F651AF8.8030700@netwok.org> Message-ID: On 03/18/2012 12:15 AM, ?ric Araujo wrote: > Hi, > > Le 17/03/2012 19:03, Georg Brandl a ?crit : >> On 03/17/2012 11:43 PM, Barry Warsaw wrote: >> I'm afraid it's the latter: tags are entries in .hgtags. So when you completely >> null-merge your 2.6 changes into 2.7, you are basically removing the tag from >> the 2.7 branch. And since to find tags, Mercurial looks in the .hgtags files >> of all active branch heads, you are basically hiding the tag when you merge >> 2.6 into 2.7, at which point it becomes an inactive branch head. > > The plus side to this concept of tags as entries in a file is that it?s > trivial to add the missing 2.6 tags in the 2.7 branch. > > Note that duplicate entries in .hgtags (when a tag was redone) should > not be ?cleaned up?: the presence of the old changeset hash greatly > helps conflict resolution. (If someone pulled the repo with the old tag > and later pulls and updates, then they don?t have to find out which hash > is the right tag, they just accept all changes from the updated file > into their local file.) I don't understand that argument: especially when there is no change in the tree between the two tags. Georg From ncoghlan at gmail.com Sun Mar 18 14:49:25 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Mar 2012 23:49:25 +1000 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Closes #14306: clarify expensiveness of try-except and update code snippet In-Reply-To: References: Message-ID: On Sun, Mar 18, 2012 at 1:58 AM, georg.brandl wrote: > +catching an exception is expensive. ?In versions of Python prior to 2.0 it was > +common to use this idiom:: Actually, given the "prior to 2.0" caveat, "mydict.has_key(key)" is right: the "key in mydict" version was only added in 2.2. This answer probably needs more improvements than just modernising the example that is already there. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fuzzyman at voidspace.org.uk Sun Mar 18 19:44:28 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 18 Mar 2012 18:44:28 +0000 Subject: [Python-Dev] cpython: PEP 417: Adding unittest.mock In-Reply-To: References: Message-ID: <6E1A6E69-F862-43F7-8C8D-94CBFD0B19AA@voidspace.org.uk> On 16 Mar 2012, at 11:54, Nick Coghlan wrote: > On Thu, Mar 15, 2012 at 6:27 AM, Michael Foord > wrote: >> On the topic of docs.... mock documentation is about eight pages long. My intention was to strip this down to just the api documentation, along with a link to the docs on my site for further examples and so on. I was encouraged here at the sprints to include the full documentation instead (minus the mock library comparison page and the front page can be cut down). So this is what I am now intending to include. It does mean the mock documentation will be "extensive". > > Don't forgot you also have the option of splitting out a separate > HOWTO tutorial section, leaving the main docs as a pure API reference. > (I personally find that style easier to use than the ones which try to > address both needs in the main module docs). > The docs are already organised as API docs and then simple and advanced HOWTO style sections. There are minimal examples inline with the API docs and separate paragraphs of explanations below on particular topics. Feel free to critique the mock docs as they are, or wait until I have committed modified versions. Michael > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From martin at v.loewis.de Mon Mar 19 02:19:20 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 19 Mar 2012 02:19:20 +0100 Subject: [Python-Dev] cpython: PEP 417: Adding unittest.mock In-Reply-To: <9EE751D8-EAA8-43D8-9C04-C2FC383CFC82@gmail.com> References: <9EE751D8-EAA8-43D8-9C04-C2FC383CFC82@gmail.com> Message-ID: <4F668998.7090706@v.loewis.de> > The commingling of extensive examples with regular docs has > made it difficult to lookup functionality in argparse for example. I have now come to think that this should be considered a subordinate use case. The primary use case of the documentation should be copy-paste style examples. At least, that's the feedback I always get for the Python documentation (typically contrasting it with the PHP documentation, where the specification-style portion is typically ignored by readers, which then move on to the user-contributed examples). Regards, Martin From benjamin at python.org Mon Mar 19 02:43:56 2012 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 18 Mar 2012 20:43:56 -0500 Subject: [Python-Dev] [RELEASED] Second release candidates for Python 2.6.8, 2.7.3, 3.1.5, and 3.2.3 Message-ID: We're chuffed to announce the immediate availability of the second release candidates for Python 2.6.8, 2.7.3, 3.1.5, and 3.2.3. The only change from the first release candidates is the patching of an additional security hole. The security issue fixed in the second release candidates is in the expat XML parsing library. expat had the same hash security issue detailed below as Python's core types. The hashing algorithm used in the expat library is now randomized. A more thorough explanation of the "hash attack" security hole follows. The main impetus for these releases is fixing a security issue in Python's hash based types, dict and set, as described below. Python 2.7.3 and 3.2.3 include the security patch and the normal set of bug fixes. Since Python 2.6 and 3.1 are maintained only for security issues, 2.6.8 and 3.1.5 contain only various security patches. The security issue exploits Python's dict and set implementations. Carefully crafted input can lead to extremely long computation times and denials of service. [1] Python dict and set types use hash tables to provide amortized constant time operations. Hash tables require a well-distributed hash function to spread data evenly across the hash table. The security issue is that an attacker could compute thousands of keys with colliding hashes; this causes quadratic algorithmic complexity when the hash table is constructed. To alleviate the problem, the new releases add randomization to the hashing of Python's string types (bytes/str in Python 3 and str/unicode in Python 2), datetime.date, and datetime.datetime. This prevents an attacker from computing colliding keys of these types without access to the Python process. Hash randomization causes the iteration order of dicts and sets to be unpredictable and differ across Python runs. Python has never guaranteed iteration order of keys in a dict or set, and applications are advised to never rely on it. Historically, dict iteration order has not changed very often across releases and has always remained consistent between successive executions of Python. Thus, some existing applications may be relying on dict or set ordering. Because of this and the fact that many Python applications which don't accept untrusted input are not vulnerable to this attack, in all stable Python releases mentioned here, HASH RANDOMIZATION IS DISABLED BY DEFAULT. There are two ways to enable it. The -R commandline option can be passed to the python executable. It can also be enabled by setting an environmental variable PYTHONHASHSEED to "random". (Other values are accepted, too; pass -h to python for complete description.) More details about the issue and the patch can be found in the oCERT advisory [1] and the Python bug tracker [2]. These releases are releases candidates and thus not recommended for production use. Please test your applications and libraries with them, and report any bugs you encounter. We are especially interested in any buggy behavior observed using hash randomization. Excepting major calamity, final versions should appear after several weeks. Downloads are at http://python.org/download/releases/2.6.8/ http://python.org/download/releases/2.7.3/ http://python.org/download/releases/3.1.5/ http://python.org/download/releases/3.2.3/ Please test these candidates and report bugs to http://bugs.python.org/ With regards, The Python release team Barry Warsaw (2.6), Georg Brandl (3.2), Benjamin Peterson (2.7 and 3.1) [1] http://www.ocert.org/advisories/ocert-2011-003.html [2] http://bugs.python.org/issue13703 From pmoody at google.com Mon Mar 19 03:44:18 2012 From: pmoody at google.com (Peter Moody) Date: Sun, 18 Mar 2012 19:44:18 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 9:15 AM, Peter Moody wrote: >>> - iterable APIs should consistently produce iterators (leaving users >>> free to wrap list() around the calls if they want the concrete >>> realisation) I might've missed earlier discussion somewhere, but can someone point me at an example of an iteratable api in ipaddr/ipaddress where an iterator isn't consistently produced? Cheers, peter -- Peter Moody? ? ? Google? ? 1.650.253.7306 Security Engineer? pgp:0xC3410038 From ncoghlan at gmail.com Mon Mar 19 04:04:48 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Mar 2012 13:04:48 +1000 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: Message-ID: On Mon, Mar 19, 2012 at 12:44 PM, Peter Moody wrote: > On Mon, Mar 12, 2012 at 9:15 AM, Peter Moody wrote: > >>>> - iterable APIs should consistently produce iterators (leaving users >>>> free to wrap list() around the calls if they want the concrete >>>> realisation) > > I might've missed earlier discussion somewhere, but can someone point > me at an example of an iteratable api in ipaddr/ipaddress where an > iterator isn't consistently produced? There was at least one that I recall, now to find it again... And searching for "list" in the PEP 3144 branch source highlights subnet() vs iter_subnets() as the main problem child: https://code.google.com/p/ipaddr-py/source/browse/branches/3144/ipaddress.py#1004 A single "subnets()" method that produced the iterator would seem to make more sense (with a "list()" call wrapped around it when the consumer really wants a concrete list). There are a few other cases that produce a list that are less clearcut. I *think* summarising the address range could be converted to an iterator, since the "networks" accumulation list doesn't get referenced by the summarising algorithm. Similarly, there doesn't appear to be a compelling reason for "address_exclude" to produce a concrete list (I also noticed a couple of "assert True == False" statements in that method for "this should never happen" code branches. An explicit "raise AssertionError" is a better way to handle such cases, so the code remains present even under -O and -OO) Collapsing the address list has to build the result list anyway to actually handle the deduplication part of its job, so returning a concrete list makes sense in that case. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Mar 19 04:24:12 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Mar 2012 13:24:12 +1000 Subject: [Python-Dev] cpython: PEP 417: Adding unittest.mock In-Reply-To: <4F668998.7090706@v.loewis.de> References: <9EE751D8-EAA8-43D8-9C04-C2FC383CFC82@gmail.com> <4F668998.7090706@v.loewis.de> Message-ID: On Mon, Mar 19, 2012 at 11:19 AM, "Martin v. L?wis" wrote: >> The commingling of extensive examples with regular docs has >> made it difficult to lookup functionality in argparse for example. > > I have now come to think that this should be considered a subordinate > use case. The primary use case of the documentation should be copy-paste > style examples. At least, that's the feedback I always get for the > Python documentation (typically contrasting it with the PHP > documentation, where the specification-style portion is typically > ignored by readers, which then move on to the user-contributed > examples). That's why the 3.2 logging docs are such a good model to follow. They have *4* pieces: - the formal API reference ("What features does logging offer?" and "Exactly how does the X API work, again?") - a "quick start" tutorial ("What's the bare minimum I need to know to get started with the logging module?") - an "advanced" tutorial ("What are some other cool things the logging infrastructure lets me do?") - a "cookbook" section ("How do I achieve task Y with the logging module?") The first of those is in the standard library *reference*, with clear pointers directly to the other 3 (which live in the "HOWTO" section of the docs). Different audiences have different needs. If you just want to get something working quickly and aren't interested in understanding the details right now, then "cargo cult programming" can be a good way to go and "cookbook" style docs are a great resource for that. If you're just trying to remember a precise incantation for something you already know the module can do, then you want a formal reference that spells out the API details. Tutorials land somewhere in between - trying to teach people enough about the module that they can make more effective use of both the formal API reference (when figuring things out from scratch) and the cookbook examples (when trying to accomplish a common task without caring too much about the details of how and why it works). As much as I like argparse, the existing docs don't do a great job of advertising its capabilities, since they're currently a mixture of tutorial-and-reference-and-cookbook that means they don't excel at serving any of the possible audiences. (I've posted a few suggestions on the issue tracker for specific changes I think would help improve the situation). The key point though is that there are multiple reasons people look up documentation, and the appropriate structure varies based on the reason someone is reading the docs at all. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncdave4life at gmail.com Mon Mar 19 05:13:55 2012 From: ncdave4life at gmail.com (ncdave4life) Date: Sun, 18 Mar 2012 21:13:55 -0700 (PDT) Subject: [Python-Dev] inspect.py change for pygame Message-ID: <1332130435675-4631993.post@n6.nabble.com> I noticed that pydoc doesn't work for pygame under python 3.2.1 for Win32: NotImplementedError: scrap module not available (ImportError: No module named scrap) I made a small patch to inspect.py to solve the problem (I just added a try/expect around the failing statement in ismethoddescriptor). Here's the diff: http://www.burtonsys.com/python32/inspect.diff http://www.burtonsys.com/python32/inspect.diff With that patch, pydoc works with pygame, and reports just a few issues: *scrap* = *sndarray* = *surfarray* = Sorry, I'm a newbie to python-dev, so please forgive my ignorance, but what do I need to do get this fix (or something similar) into a future release? -- View this message in context: http://python.6.n6.nabble.com/inspect-py-change-for-pygame-tp4631993p4631993.html Sent from the Python - python-dev mailing list archive at Nabble.com. From brian at python.org Mon Mar 19 05:19:03 2012 From: brian at python.org (Brian Curtin) Date: Sun, 18 Mar 2012 23:19:03 -0500 Subject: [Python-Dev] inspect.py change for pygame In-Reply-To: <1332130435675-4631993.post@n6.nabble.com> References: <1332130435675-4631993.post@n6.nabble.com> Message-ID: On Sun, Mar 18, 2012 at 23:13, ncdave4life wrote: > I noticed that pydoc doesn't work for pygame under python 3.2.1 for Win32: > > NotImplementedError: scrap module not available (ImportError: No module > named scrap) > > I made a small patch to inspect.py to solve the problem (I just added a > try/expect around the failing statement in ismethoddescriptor). ? Here's the > diff: > http://www.burtonsys.com/python32/inspect.diff > http://www.burtonsys.com/python32/inspect.diff > > With that patch, pydoc works with pygame, and reports just a few issues: > *scrap* = > *sndarray* = > *surfarray* = > > Sorry, I'm a newbie to python-dev, so please forgive my ignorance, but what > do I need to do get this fix (or something similar) into a future release? Patches to fix Python should be posted to http://bugs.python.org/. >From there they'll be classified, reviewed, and if all is well, committed. It's much easier for patches to be tracked on there instead of email. From ncdave4life at gmail.com Mon Mar 19 05:23:01 2012 From: ncdave4life at gmail.com (ncdave4life) Date: Sun, 18 Mar 2012 21:23:01 -0700 (PDT) Subject: [Python-Dev] inspect.py change for pygame In-Reply-To: References: <1332130435675-4631993.post@n6.nabble.com> Message-ID: Thank you, Brian! On Mon, Mar 19, 2012 at 12:20 AM, Brian Curtin [via Python] < ml-node+s6n4632000h78 at n6.nabble.com> wrote: > On Sun, Mar 18, 2012 at 23:13, ncdave4life <[hidden email] > > wrote: > > ...Sorry, I'm a newbie to python-dev, so please forgive my ignorance, > but what > > do I need to do get this fix (or something similar) into a future > release? > > Patches to fix Python should be posted to http://bugs.python.org/. > From there they'll be classified, reviewed, and if all is well, > committed. It's much easier for patches to be tracked on there instead > of email. > -- View this message in context: http://python.6.n6.nabble.com/inspect-py-change-for-pygame-tp4631993p4632010.html Sent from the Python - python-dev mailing list archive at Nabble.com. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristjan at ccpgames.com Mon Mar 19 10:07:59 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Mon, 19 Mar 2012 09:07:59 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: Do you really want to add an obscure Boolean flag to the function just so that python can warn you that perhaps your platform is so old and so weird that Python can't guarantee that the performance measurements are to a certain _undefined_ quality? Please note, that the function makes no claims to the resolution or precision of the timer involved. Only that it moves only forward. It is therefore completely and utterly redundant to add a "strict" value, because we would only be behave "strictly" according to an _undefined specification_. K -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Lennart Regebro Sent: 15. mars 2012 04:44 To: Matt Joiner Cc: Python Dev Subject: Re: [Python-Dev] Drop the new time.wallclock() function? On Thu, Mar 15, 2012 at 02:58, Matt Joiner wrote: > Victor, I think that steady can always be monotonic, there are time > sources enough to ensure this on the platforms I am aware of. Strict > in this sense refers to not being adjusted forward, i.e. > CLOCK_MONOTONIC vs CLOCK_MONOTONIC_RAW. > > Non monotonicity of this call should be considered a bug. Strict would > be used for profiling where forward leaps would disqualify the timing. This makes sense to me. //Lennart _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com From kristjan at ccpgames.com Mon Mar 19 10:26:30 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Mon, 19 Mar 2012 09:26:30 +0000 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status In-Reply-To: <4F62692E.8040203@oddbird.net> References: <4F626278.7030701@oddbird.net> <4F626712.3030906@gmail.com> <4F62692E.8040203@oddbird.net> Message-ID: Hi Carl. I'm very interested in this work. At CCP we work heavily with virtual environments. Except that we don't use virtualenv because it is just a pain in the neck. We like to be able to run virtual python environments of various types as they arrive checked out of source control repositories, without actually "installing" anything. For some background, please see: http://blog.ccpgames.com/kristjan/2010/10/09/using-an-isolated-python-exe/. It's a rather quick read, actually. The main issue for us is: How to prevent your local python.exe from reading environment variables and running some global site.py? There are a number of points raised in the above blog, please take a look at the "Musings" at the end. Best regards, Kristj?n -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Carl Meyer Sent: 15. mars 2012 22:12 To: python-dev Subject: Re: [Python-Dev] PEP 405 (built-in virtualenv) status On 03/15/2012 03:02 PM, Lindberg, Van wrote: > FYI, the location of the tcl/tk libraries does not appear to be set in > the virtualenv, even if tkinter is installed and working in the main > Python installation. As a result, tk-based apps will not run from a > virtualenv. Thanks for the report! I've added this to the list of open issues in the PEP and I'll look into it. Carl From stefan at bytereef.org Mon Mar 19 13:26:37 2012 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 19 Mar 2012 13:26:37 +0100 Subject: [Python-Dev] svn.python.org and buildbots down Message-ID: <20120319122637.GA28176@sleipnir.bytereef.org> Hello, you might be aware of it already. In case not, it appears that svn.python.org and the buildbots are down. Stefan Krah From victor.stinner at gmail.com Mon Mar 19 13:31:00 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 19 Mar 2012 13:31:00 +0100 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: <4F650B25.4060901@trueblade.com> References: <4F650B25.4060901@trueblade.com> Message-ID: > I have to agree with Georg. Looking at the code, it appears OSError can > be raised with both strict=True and strict=False (since floattime() can > raise OSError). This is an old bug in floattime(): I opened the issue #14368 to remove the unused exception. In practice, it never happens (or it is *very* unlikely today). IMO it's a bug in floattime(). > I also think "By default, if strict is False" confuses things. I agree, I replaced it by "By default,". Victor From victor.stinner at gmail.com Mon Mar 19 13:35:49 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 19 Mar 2012 13:35:49 +0100 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: Message-ID: >>> This is not clear to me. ?Why wouldn't it raise OSError on error even with >>> strict=False? ?Please clarify which exception is raised in which case. >> >> It seems clear to me. It doesn't raise exceptions when strict=False because >> it falls back to a non-monotonic clock. If strict is True and a non-monotonic >> clock is not available it raises OSError or NotImplementedError. > > So errors are ignored when strict is false? Said differently: time.steady(strict=True) is always monotonic (*), whereas time.steady() may or may not be monotonic, depending on what is avaiable. time.steady() is a best-effort steady clock. (*) time.steady(strict=True) relies on the OS monotonic clock. If the OS provides a "not really monotonic" clock, Python cannot do better. For example, clock_gettime(CLOCK_MONOTONIC) speed can be adjusted by NTP on Linux. Python tries to use clock_gettime(CLOCK_MONOTONIC_RAW) which doesn't have this issue. Victor From solipsis at pitrou.net Mon Mar 19 14:25:39 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 19 Mar 2012 14:25:39 +0100 Subject: [Python-Dev] svn.python.org and buildbots down References: <20120319122637.GA28176@sleipnir.bytereef.org> Message-ID: <20120319142539.7e83c3fb@pitrou.net> On Mon, 19 Mar 2012 13:26:37 +0100 Stefan Krah wrote: > Hello, > > you might be aware of it already. In case not, it appears that svn.python.org > and the buildbots are down. The buildbots should be back now. As for svn.python.org, is anyone using it? (I don't know how to restart it) Regards Antoine. From stefan at bytereef.org Mon Mar 19 14:45:37 2012 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 19 Mar 2012 14:45:37 +0100 Subject: [Python-Dev] svn.python.org and buildbots down In-Reply-To: <20120319142539.7e83c3fb@pitrou.net> References: <20120319122637.GA28176@sleipnir.bytereef.org> <20120319142539.7e83c3fb@pitrou.net> Message-ID: <20120319134537.GA28530@sleipnir.bytereef.org> Antoine Pitrou wrote: > The buildbots should be back now. As for svn.python.org, is anyone > using it? (I don't know how to restart it) Thanks! I'm using svn.python.org for the automated sphinx checkout in Doc/ (make html) and sometimes to dig through pre-hg history. But don't bother to find out how to restart it just for me. I presume Martin knows the setup and will do it later. Stefan Krah From brett at python.org Mon Mar 19 15:56:02 2012 From: brett at python.org (Brett Cannon) Date: Mon, 19 Mar 2012 10:56:02 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Issue #14347: Update Misc/README list of files. In-Reply-To: References: Message-ID: The two files that were added back in should probably just disappear (README.aix and README.coverty). Anyone disagree? On Sat, Mar 17, 2012 at 13:52, ned.deily wrote: > http://hg.python.org/cpython/rev/65a0a6fab127 > changeset: 75797:65a0a6fab127 > user: Ned Deily > date: Sat Mar 17 10:52:08 2012 -0700 > summary: > Issue #14347: Update Misc/README list of files. > (Initial patch by Dionysios Kalofonos) > > files: > Misc/README | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > > diff --git a/Misc/README b/Misc/README > --- a/Misc/README > +++ b/Misc/README > @@ -8,7 +8,6 @@ > ---------------- > > ACKS Acknowledgements > -build.sh Script to build and test latest Python from the > repository > gdbinit Handy stuff to put in your .gdbinit file, if you > use gdb > HISTORY News from previous releases -- oldest last > indent.pro GNU indent profile approximating my C style > @@ -19,6 +18,8 @@ > python.pc.in Package configuration info template for > pkg-config > python-wing*.wpr Wing IDE project file > README The file you're reading now > +README.AIX Information about using Python on AIX > +README.coverity Information about running Coverity's Prevent on > Python > README.valgrind Information for Valgrind users, see > valgrind-python.supp > RPM (Old) tools to build RPMs > svnmap.txt Map of old SVN revs and branches to hg changeset > ids > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 19 15:59:20 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 19 Mar 2012 15:59:20 +0100 Subject: [Python-Dev] cpython: Issue #14347: Update Misc/README list of files. References: Message-ID: <20120319155920.46561fdf@pitrou.net> On Mon, 19 Mar 2012 10:56:02 -0400 Brett Cannon wrote: > The two files that were added back in should probably just disappear > (README.aix and README.coverty). Anyone disagree? README.AIX was recently updated in http://bugs.python.org/issue10709. Regards Antoine. From tshepang at gmail.com Mon Mar 19 16:25:44 2012 From: tshepang at gmail.com (Tshepang Lekhonkhobe) Date: Mon, 19 Mar 2012 17:25:44 +0200 Subject: [Python-Dev] regarding HTML mail Message-ID: apology: I searched for a few minutes and could not find a code of conduct regarding HTML mail. Can we have some guideline to allow only plain text emails, so as to avoid cases like http://mail.python.org/pipermail/docs/2012-March/007999.html, where you are forced to scroll horizontally in order to read the text. From nad at acm.org Mon Mar 19 18:20:39 2012 From: nad at acm.org (Ned Deily) Date: Mon, 19 Mar 2012 10:20:39 -0700 Subject: [Python-Dev] svn.python.org and buildbots down References: <20120319122637.GA28176@sleipnir.bytereef.org> <20120319142539.7e83c3fb@pitrou.net> Message-ID: In article <20120319142539.7e83c3fb at pitrou.net>, Antoine Pitrou wrote: > [...] As for svn.python.org, is anyone > using it? The repo for the website (www.python.org) is maintained there. -- Ned Deily, nad at acm.org From v+python at g.nevcal.com Mon Mar 19 18:29:12 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 19 Mar 2012 10:29:12 -0700 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status In-Reply-To: References: <4F626278.7030701@oddbird.net> <4F626712.3030906@gmail.com> <4F62692E.8040203@oddbird.net> Message-ID: <4F676CE8.6070107@g.nevcal.com> On 3/19/2012 2:26 AM, Kristj?n Valur J?nsson wrote: > Hi Carl. > I'm very interested in this work. > At CCP we work heavily with virtual environments. Except that we don't use virtualenv because it is just a pain in the neck. We like to be able to run virtual python environments of various types as they arrive checked out of source control repositories, without actually "installing" anything. > For some background, please see:http://blog.ccpgames.com/kristjan/2010/10/09/using-an-isolated-python-exe/. It's a rather quick read, actually. > > The main issue for us is: How to prevent your local python.exe from reading environment variables and running some global site.py? > > There are a number of points raised in the above blog, please take a look at the "Musings" at the end. > > Best regards, > > Kristj?n I found that a very interesting reverse-engineering of what needs to be done to isolate multiple pythons on a machine. I concur that this is a feature that would be good to: 1) at least document the behavior well 2) preferably make an extensible feature, along the lines that Kristj?n suggests There are likely some bootstrapping issues, but I find the idea that the difference between an embedded Python and an installed Python and a built-but-not-installed Python being conceptually isolated to the python.exe and/or site.py rather than python.dll to be a clever concept; of course, where the code lives is less relevant than the conditions under which it is invoked; I doubt the size of the code is the issue regarding where it lives. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Mar 19 19:20:46 2012 From: barry at python.org (Barry Warsaw) Date: Mon, 19 Mar 2012 14:20:46 -0400 Subject: [Python-Dev] regarding HTML mail In-Reply-To: References: Message-ID: <20120319142046.5a2ce86d@resist.wooz.org> On Mar 19, 2012, at 05:25 PM, Tshepang Lekhonkhobe wrote: >apology: I searched for a few minutes and could not find a code of >conduct regarding HTML mail. I'm not sure it's written down anywhere, but in general we strongly discourage HTML email for the lists @python.org >Can we have some guideline to allow only plain text emails, so as to >avoid cases like >http://mail.python.org/pipermail/docs/2012-March/007999.html, where >you are forced to scroll horizontally in order to read the text. docs is a different mailing list than python-dev, but still neither list is doing any content filtering. We could always enable that if we wanted to get strict about it. In this case, this isn't html email, it's likely this bug: https://bugs.launchpad.net/mailman/+bug/558294 Cheers, -Barry From jnoller at gmail.com Mon Mar 19 19:52:01 2012 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 19 Mar 2012 14:52:01 -0400 Subject: [Python-Dev] regarding HTML mail In-Reply-To: <20120319142046.5a2ce86d@resist.wooz.org> References: <20120319142046.5a2ce86d@resist.wooz.org> Message-ID: I'd like to discuss top-posting. On Monday, March 19, 2012 at 2:20 PM, Barry Warsaw wrote: > On Mar 19, 2012, at 05:25 PM, Tshepang Lekhonkhobe wrote: > > > apology: I searched for a few minutes and could not find a code of > > conduct regarding HTML mail. > > > > I'm not sure it's written down anywhere, but in general we strongly discourage > HTML email for the lists @python.org (http://python.org) > > > Can we have some guideline to allow only plain text emails, so as to > > avoid cases like > > http://mail.python.org/pipermail/docs/2012-March/007999.html, where > > you are forced to scroll horizontally in order to read the text. > > > > docs is a different mailing list than python-dev, but still neither list is > doing any content filtering. We could always enable that if we wanted to get > strict about it. > > In this case, this isn't html email, it's likely this bug: > > https://bugs.launchpad.net/mailman/+bug/558294 > > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org (mailto:Python-Dev at python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com From pje at telecommunity.com Mon Mar 19 19:53:47 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 19 Mar 2012 14:53:47 -0400 Subject: [Python-Dev] svn.python.org and buildbots down In-Reply-To: References: <20120319122637.GA28176@sleipnir.bytereef.org> <20120319142539.7e83c3fb@pitrou.net> Message-ID: On Mar 19, 2012 1:20 PM, "Ned Deily" wrote: > > In article <20120319142539.7e83c3fb at pitrou.net>, > Antoine Pitrou wrote: > > [...] As for svn.python.org, is anyone > > using it? > > The repo for the website (www.python.org) is maintained there. It's also still setuptools' official home, though I've been doing some work recently on migrating it to hg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Mon Mar 19 19:56:41 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 19 Mar 2012 11:56:41 -0700 Subject: [Python-Dev] regarding HTML mail In-Reply-To: References: <20120319142046.5a2ce86d@resist.wooz.org> Message-ID: <4F678169.6030906@g.nevcal.com> On 3/19/2012 11:52 AM, Jesse Noller wrote: > I'd like to discuss top-posting. Somewhere else, please. Oh, that was your point :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Mon Mar 19 20:12:20 2012 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 19 Mar 2012 12:12:20 -0700 (PDT) Subject: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives In-Reply-To: <4F5FDE77.7030601@pearwood.info> Message-ID: <4f678514.e855320a.12cb.ffffe24d@mx.google.com> In http://mail.python.org/pipermail/python-dev/2012-March/117570.html Steven D'Aprano posted: > "Need" is awfully strong. I don't believe it is the responsibility > of the standard library to be judge and reviewer of third party > packages that it doesn't control. It is, however, user-friendly to indicate when the stdlib selections are particularly likely to be for reasons other than "A bunch of experts believe this is the best way to do this." Cpython's documentation is (de facto) the documentation for python in general, and pointing people towards other resources (particularly pypi itself) is quite reasonable. Many modules are in the stdlib in part because they are an *acceptable* way of doing something, and the "best" ways are either changing too quickly or are so complicated that it doesn't make sense to burden the *standard* libary for specialist needs. In those cases, I do think the documentation should say so. Specific examples: http://docs.python.org/library/numeric.html quite reasonably has subsections only for what ships with Python. But I think the introductory paragraph could stand to have an extra sentence explaining why and when people should look beyond the stanard library, such as: Applications centered around mathematics may benefit from specialist 3rd party libraries, such as numpy < http://pypi.python.org/pypi/numpy/ >, gmpy < http://pypi.python.org/pypi/gmpy >, and scipy< http://pypi.python.org/pypi/scipy >. I would add a similar sentence to the web section, or the internet protocols section if web is still not broken out separately. http://docs.python.org/dev/library/internet.html Note that some web conventions are still evolving too quickly for covenient encapsulation in a stable library. Many applications will therefore prefer functional replacements from third parties, such as requests or httplib2, or frameworks such as Django and Zope. www-related products can be found by browsing PyPI for top internet subtopic www/http. < http://pypi.python.org/pypi?:action=browse&c=319&c=326 > [I think that searching by classifier -- which first requires browse, and can't be reached from the list of classifiers -- could be improved.] > Should we recommend wxPython over Pyjamas or PyGUI or PyGtk? Actually, I think the existing http://docs.python.org/library/othergui.html does a pretty good job; I would not object to adding mentions of other tools as well, but wiki reference is probably sufficient. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From carl at oddbird.net Mon Mar 19 20:18:40 2012 From: carl at oddbird.net (Carl Meyer) Date: Mon, 19 Mar 2012 13:18:40 -0600 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status In-Reply-To: References: <4F626278.7030701@oddbird.net> <4F626712.3030906@gmail.com> <4F62692E.8040203@oddbird.net> Message-ID: <4F678690.3000600@oddbird.net> Hello Kristj?n, On 03/19/2012 03:26 AM, Kristj?n Valur J?nsson wrote: > Hi Carl. I'm very interested in this work. At CCP we work heavily > with virtual environments. Except that we don't use virtualenv > because it is just a pain in the neck. We like to be able to run > virtual python environments of various types as they arrive checked > out of source control repositories, without actually "installing" > anything. For some background, please see: > http://blog.ccpgames.com/kristjan/2010/10/09/using-an-isolated-python-exe/. > It's a rather quick read, actually. > > The main issue for us is: How to prevent your local python.exe from > reading environment variables and running some global site.py? > > There are a number of points raised in the above blog, please take a > look at the "Musings" at the end. Thanks, I read the blog post. I think there are some useful points there; I too find the startup sys.path behavior of Python a bit more complex and magical than I'd prefer (I'm sure it's grown organically over the years to address a variety of different needs and concerns, not all of which I understand or am even aware of). I think there's one important (albeit odd and magical) bit of Python's current behavior that you are missing in your blog post. All of the initial sys.path directories are constructed relative to sys.prefix and sys.exec_prefix, and those values in turn are determined (if PYTHONHOME is not set), by walking up the filesystem tree from the location of the Python binary, looking for the existence of a file at the relative path "lib/pythonX.X/os.py" (or "Lib/os.py" on Windows). Python takes the existence of this file to mean that it's found the standard library, and sets sys.prefix accordingly. Thus, you can achieve reliable full isolation from any installed Python, with no need for environment variables, simply by placing a file (it can even be empty) at that relative location from the location of your Python binary. You will still get some default paths added on sys.path, but they will all be relative to your Python binary and thus presumably under your control; nothing from any other location will be on sys.path. I doubt you will find this solution satisfyingly elegant, but you might nonetheless find it practically useful. The essence of PEP 405 is simply to provide a less magical way to achieve this same result, by locating a "pyvenv.cfg" file next to (or one directory up from) the Python binary. The bulk of the work in PEP 405 is aimed towards a rather different goal from yours - to be able to share an installed Python's copy of the standard library among a number of virtual environments. This is the purpose of the "home" key in pyvenv.cfg and the added sys.base_prefix (which point to the Python installation whose standard library will be used). I think this serves a valuable and common use case, but I wonder if your use case couldn't also be served with a minor tweak to PEP 405. Currently it ignores a pyvenv.cfg file with no "home" key; instead, it could set sys.prefix and sys.base_prefix both to the location of that pyvenv.cfg. For most purposes, this would result in a broken Python (no standard library), but it might help you? Beyond that possible tweak, while I certainly wouldn't oppose any effort to clean up / document / make-optional Python's startup sys.path-setting behavior, I think it's mostly orthogonal to PEP 405, and I don't think it would be helpful to expand the scope of PEP 405 to include that effort. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Mon Mar 19 20:37:19 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Mar 2012 12:37:19 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: Message-ID: <4F678AEF.4080002@stoneleaf.us> Nick Coghlan wrote: > Collapsing the address list has to build the result list anyway to > actually handle the deduplication part of its job, so returning a > concrete list makes sense in that case. Having only one function return a list instead of an iterator seems questionable. Depending on the code it could either keep track of what it has returned so far in a set and avoid duplication that way; or, just return an `iter(listobject)` instead of `listobject`. ~Ethan~ From guido at python.org Mon Mar 19 20:55:11 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Mar 2012 12:55:11 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: <4F678AEF.4080002@stoneleaf.us> References: <4F678AEF.4080002@stoneleaf.us> Message-ID: On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote: > Nick Coghlan wrote: >> >> Collapsing the address list has to build the result list anyway to >> actually handle the deduplication part of its job, so returning a >> concrete list makes sense in that case. > > > Having only one function return a list instead of an iterator seems > questionable. > > Depending on the code it could either keep track of what it has returned so > far in a set and avoid duplication that way; or, just return an > `iter(listobject)` instead of `listobject`. I know I'm lacking context, but is the list ever expected to be huge? If not, what's wrong with always returning a list? -- --Guido van Rossum (python.org/~guido) From ethan at stoneleaf.us Mon Mar 19 21:13:56 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Mar 2012 13:13:56 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: <4F678AEF.4080002@stoneleaf.us> Message-ID: <4F679384.4040602@stoneleaf.us> Guido van Rossum wrote: > On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote: >> Nick Coghlan wrote: >>> Collapsing the address list has to build the result list anyway to >>> actually handle the deduplication part of its job, so returning a >>> concrete list makes sense in that case. >> >> Having only one function return a list instead of an iterator seems >> questionable. >> >> Depending on the code it could either keep track of what it has returned so >> far in a set and avoid duplication that way; or, just return an >> `iter(listobject)` instead of `listobject`. > > I know I'm lacking context, but is the list ever expected to be huge? > If not, what's wrong with always returning a list? Nothing wrong in and of itself. It just seems to me that if we have several functions that deal with ip addresses/networks/etc, and all but one return iterators, that one is going to be a pain... 'Which one returns a list again? Oh yeah, that one.' Granted it's mostly a stylistic preference for consistency. ~Ethan~ From ethan at stoneleaf.us Mon Mar 19 21:42:57 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Mar 2012 13:42:57 -0700 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status In-Reply-To: <4F678690.3000600@oddbird.net> References: <4F626278.7030701@oddbird.net> <4F626712.3030906@gmail.com> <4F62692E.8040203@oddbird.net> <4F678690.3000600@oddbird.net> Message-ID: <4F679A51.4070309@stoneleaf.us> Carl Meyer wrote: > The bulk of the work in PEP 405 is aimed towards a rather different goal > from yours - to be able to share an installed Python's copy of the > standard library among a number of virtual environments. This is the > purpose of the "home" key in pyvenv.cfg and the added sys.base_prefix > (which point to the Python installation whose standard library will be > used). I think this serves a valuable and common use case, but I wonder > if your use case couldn't also be served with a minor tweak to PEP 405. > Currently it ignores a pyvenv.cfg file with no "home" key; instead, it > could set sys.prefix and sys.base_prefix both to the location of that > pyvenv.cfg. For most purposes, this would result in a broken Python (no > standard library), but it might help you? Instead of no home key, how about an empty home key? Explicit being better, and all that. ~Ethan~ From jimjjewett at gmail.com Mon Mar 19 22:01:19 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 19 Mar 2012 17:01:19 -0400 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Fixes Issue 14234: fix for the previous commit, keep compilation when In-Reply-To: References: Message-ID: Does this mean that if Python is updated before expat, python will compile out the expat randomization, and therefore not use if even after expat is updated? -jJ On Thu, Mar 15, 2012 at 2:01 PM, benjamin.peterson wrote: > http://hg.python.org/cpython/rev/ada6bfbeceb8 > changeset: ? 75699:ada6bfbeceb8 > branch: ? ? ?2.7 > user: ? ? ? ?Gregory P. Smith > date: ? ? ? ?Wed Mar 14 18:12:23 2012 -0700 > summary: > ?Fixes Issue 14234: fix for the previous commit, keep compilation when > using --with-system-expat working when the system expat does not have > salted hash support. > > files: > ?Modules/expat/expat.h | ?2 ++ > ?Modules/pyexpat.c ? ? | ?5 +++++ > ?2 files changed, 7 insertions(+), 0 deletions(-) > > > diff --git a/Modules/expat/expat.h b/Modules/expat/expat.h > --- a/Modules/expat/expat.h > +++ b/Modules/expat/expat.h > @@ -892,6 +892,8 @@ > ?XML_SetHashSalt(XML_Parser parser, > ? ? ? ? ? ? ? ? unsigned long hash_salt); > > +#define XML_HAS_SET_HASH_SALT ?/* Python Only: Defined for pyexpat.c. */ > + > ?/* If XML_Parse or XML_ParseBuffer have returned XML_STATUS_ERROR, then > ? ?XML_GetErrorCode returns information about the error. > ?*/ > diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c > --- a/Modules/pyexpat.c > +++ b/Modules/pyexpat.c > @@ -1302,8 +1302,13 @@ > ? ? else { > ? ? ? ? self->itself = XML_ParserCreate(encoding); > ? ? } > +#if ((XML_MAJOR_VERSION >= 2) && (XML_MINOR_VERSION >= 1)) || defined(XML_HAS_SET_HASH_SALT) > + ? ?/* This feature was added upstream in libexpat 2.1.0. ?Our expat copy > + ? ? * has a backport of this feature where we also define XML_HAS_SET_HASH_SALT > + ? ? * to indicate that we can still use it. */ > ? ? XML_SetHashSalt(self->itself, > ? ? ? ? ? ? ? ? ? ? (unsigned long)_Py_HashSecret.prefix); > +#endif > ? ? self->intern = intern; > ? ? Py_XINCREF(self->intern); > ?#ifdef Py_TPFLAGS_HAVE_GC > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From benjamin at python.org Mon Mar 19 22:03:37 2012 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 19 Mar 2012 16:03:37 -0500 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Fixes Issue 14234: fix for the previous commit, keep compilation when In-Reply-To: References: Message-ID: 2012/3/19 Jim Jewett : > Does this mean that if Python is updated before expat, python will > compile out the expat randomization, and therefore not use if even > after expat is updated? If you're using --with-system-expat -- Regards, Benjamin From guido at python.org Mon Mar 19 22:21:50 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Mar 2012 14:21:50 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: <4F679384.4040602@stoneleaf.us> References: <4F678AEF.4080002@stoneleaf.us> <4F679384.4040602@stoneleaf.us> Message-ID: On Mon, Mar 19, 2012 at 1:13 PM, Ethan Furman wrote: > Guido van Rossum wrote: >> >> On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote: >>> >>> Nick Coghlan wrote: >>>> >>>> Collapsing the address list has to build the result list anyway to >>>> actually handle the deduplication part of its job, so returning a >>>> concrete list makes sense in that case. >>> >>> >>> Having only one function return a list instead of an iterator seems >>> questionable. >>> >>> Depending on the code it could either keep track of what it has returned >>> so >>> far in a set and avoid duplication that way; or, just return an >>> `iter(listobject)` instead of `listobject`. >> >> >> I know I'm lacking context, but is the list ever expected to be huge? >> If not, what's wrong with always returning a list? > > > Nothing wrong in and of itself. ?It just seems to me that if we have several > functions that deal with ip addresses/networks/etc, and all but one return > iterators, that one is going to be a pain... 'Which one returns a list > again? Oh yeah, that one.' It depends on whether they really are easy to confuse. If they are, indeed that feels like poor API design. But sometimes the only time two things seem confusingly similar is when you have not actually tried to use them. A naming convention often helps too. > Granted it's mostly a stylistic preference for consistency. And remember that consistency is good in moderation, but if it becomes a goal in itself you may have a problem. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Mon Mar 19 22:23:32 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 19 Mar 2012 22:23:32 +0100 Subject: [Python-Dev] svn.python.org and buildbots down In-Reply-To: <20120319134537.GA28530@sleipnir.bytereef.org> References: <20120319122637.GA28176@sleipnir.bytereef.org> <20120319142539.7e83c3fb@pitrou.net> <20120319134537.GA28530@sleipnir.bytereef.org> Message-ID: <4F67A3D4.20707@v.loewis.de> > But don't bother to find out how to restart it just for me. I presume > Martin knows the setup and will do it later. It seems to be working fine now, and I didn't do anything. Thomas rebooted the system for hardware inspection at 15:02 (and brought it back up at 15:18), so most likely, it started as part of the regular boot process (as it should have done on the previous reboot as well). Regards, Martin P.S. FWIW, the hardware inspection didn't reveal any hardware problems, so it remains unclear what is causing the outages. From pmoody at google.com Mon Mar 19 22:58:12 2012 From: pmoody at google.com (Peter Moody) Date: Mon, 19 Mar 2012 14:58:12 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: <4F678AEF.4080002@stoneleaf.us> Message-ID: On Mon, Mar 19, 2012 at 12:55 PM, Guido van Rossum wrote: > On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote: >> Nick Coghlan wrote: >>> >>> Collapsing the address list has to build the result list anyway to >>> actually handle the deduplication part of its job, so returning a >>> concrete list makes sense in that case. >> >> >> Having only one function return a list instead of an iterator seems >> questionable. >> >> Depending on the code it could either keep track of what it has returned so >> far in a set and avoid duplication that way; or, just return an >> `iter(listobject)` instead of `listobject`. > > I know I'm lacking context, but is the list ever expected to be huge? > If not, what's wrong with always returning a list? It's possible to return massive lists, (eg, returning the 4+ billion /128 subnets in /96 or something even larger, but I don't think that's very common). I've generally tried to avoid confusion by having 'iter' in the iterating methods, but if more of the methods return iterators, maybe I need to rethink that? > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/pmoody%40google.com -- Peter Moody? ? ? Google? ? 1.650.253.7306 Security Engineer? pgp:0xC3410038 From guido at python.org Mon Mar 19 23:04:52 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Mar 2012 15:04:52 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: <4F678AEF.4080002@stoneleaf.us> Message-ID: On Mon, Mar 19, 2012 at 2:58 PM, Peter Moody wrote: > On Mon, Mar 19, 2012 at 12:55 PM, Guido van Rossum wrote: >> On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote: >>> Nick Coghlan wrote: >>>> >>>> Collapsing the address list has to build the result list anyway to >>>> actually handle the deduplication part of its job, so returning a >>>> concrete list makes sense in that case. >>> >>> >>> Having only one function return a list instead of an iterator seems >>> questionable. >>> >>> Depending on the code it could either keep track of what it has returned so >>> far in a set and avoid duplication that way; or, just return an >>> `iter(listobject)` instead of `listobject`. >> >> I know I'm lacking context, but is the list ever expected to be huge? >> If not, what's wrong with always returning a list? > > It's possible to return massive lists, (eg, returning the 4+ billion > /128 subnets in /96 or something even larger, but I don't think that's > very common). I've generally tried to avoid confusion by having 'iter' > in the iterating methods, but if more of the methods return iterators, > maybe I need to rethink that? I personally like having 'iter' in the name (e.g. iterkeys() -- note that we dropped this in Py3k because it's no longer an iterator, it's a dict view now. But I don't want to promote that style for ipaddr.py. -- --Guido van Rossum (python.org/~guido) From ethan at stoneleaf.us Mon Mar 19 22:50:22 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Mar 2012 14:50:22 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: <4F678AEF.4080002@stoneleaf.us> <4F679384.4040602@stoneleaf.us> Message-ID: <4F67AA1E.6070907@stoneleaf.us> Guido van Rossum wrote: > On Mon, Mar 19, 2012 at 1:13 PM, Ethan Furman wrote: >> Nothing wrong in and of itself. It just seems to me that if we have several >> functions that deal with ip addresses/networks/etc, and all but one return >> iterators, that one is going to be a pain... 'Which one returns a list >> again? Oh yeah, that one.' > > It depends on whether they really are easy to confuse. If they are, > indeed that feels like poor API design. But sometimes the only time > two things seem confusingly similar is when you have not actually > tried to use them. Heh -- true, I have not tried to use them (yet) -- just offering another viewpoint. ;) >> Granted it's mostly a stylistic preference for consistency. > > And remember that consistency is good in moderation, but if it becomes > a goal in itself you may have a problem. While I agree that consistency as a goal in and of itself is not good, I consider it more important than 'moderation' implies; in my own code I try to only be inconsistent when there is a good reason to be. To me, "it's already a list" isn't a good reason -- yes, that's easier for the library author, but is it easier for the library user? What is the library user gaining by having a list returned instead of an iterator? Of course, the flip-side also holds: what is the library user losing by getting an iterator when a list was available? When we way the pros and cons, and it comes down to a smidgeon of performance in trade for consistency [1], I would vote for consistency. ~Ethan~ [1] I'm assuming that 'iter(some_list)' is a quick operation. From tjreedy at udel.edu Mon Mar 19 23:44:44 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 Mar 2012 18:44:44 -0400 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: <4F678AEF.4080002@stoneleaf.us> Message-ID: On 3/19/2012 6:04 PM, Guido van Rossum wrote: > On Mon, Mar 19, 2012 at 2:58 PM, Peter Moody wrote: >> On Mon, Mar 19, 2012 at 12:55 PM, Guido van Rossum wrote: >>> On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote: >>>> Nick Coghlan wrote: >>>>> >>>>> Collapsing the address list has to build the result list anyway to >>>>> actually handle the deduplication part of its job, so returning a >>>>> concrete list makes sense in that case. >>>> >>>> >>>> Having only one function return a list instead of an iterator seems >>>> questionable. >>>> >>>> Depending on the code it could either keep track of what it has returned so >>>> far in a set and avoid duplication that way; or, just return an >>>> `iter(listobject)` instead of `listobject`. >>> >>> I know I'm lacking context, but is the list ever expected to be huge? >>> If not, what's wrong with always returning a list? >> >> It's possible to return massive lists, (eg, returning the 4+ billion >> /128 subnets in /96 or something even larger, but I don't think that's >> very common). I've generally tried to avoid confusion by having 'iter' >> in the iterating methods, but if more of the methods return iterators, >> maybe I need to rethink that? > > I personally like having 'iter' in the name (e.g. iterkeys() -- note > that we dropped this in Py3k because it's no longer an iterator, it's > a dict view now. But I don't want to promote that style for ipaddr.py. I am not sure which way you are pointing, but the general default in 3.x is to return iterators: range, zip, enumerate, map, filter, reversed, open (file objects), as well at the dict methods. I am quite happy to be rid of the 'iter' prefix on the latter. This is aside from itertools. The main exceptions I can think of are str.split and sorted. For sorted, a list *must* be constructed anyway, so might as well return it. This apparently matches the case under consideration. If name differentiation is wanted, call it xxxlist. -- Terry Jan Reedy From tjreedy at udel.edu Mon Mar 19 23:47:45 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 Mar 2012 18:47:45 -0400 Subject: [Python-Dev] svn.python.org and buildbots down In-Reply-To: <20120319142539.7e83c3fb@pitrou.net> References: <20120319122637.GA28176@sleipnir.bytereef.org> <20120319142539.7e83c3fb@pitrou.net> Message-ID: On 3/19/2012 9:25 AM, Antoine Pitrou wrote: > The buildbots should be back now. As for svn.python.org, is anyone > using it? Last I knew, some files there are required to fully build Python on Windows. I would be happy if that has or were to change. -- Terry Jan Reedy From guido at python.org Tue Mar 20 00:34:17 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Mar 2012 16:34:17 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: <4F678AEF.4080002@stoneleaf.us> Message-ID: On Mon, Mar 19, 2012 at 3:44 PM, Terry Reedy wrote: > I am not sure which way you are pointing, but the general default in 3.x is > to return iterators: range, zip, enumerate, map, filter, reversed, open > (file objects), as well at the dict methods. Actually as I tried to say, the dict methods (keys() etc.) DON'T return iterators. They return "views" which are iterable. Anyway, I also tried to imply that it matters if the number of list items would ever be huge. It seems that is indeed possible (even if not likely) so I think iterators are useful. > I am quite happy to be rid of > the 'iter' prefix on the latter. This is aside from itertools. The main > exceptions I can think of are str.split and sorted. For sorted, a list > *must* be constructed anyway, so might as well return it. This apparently > matches the case under consideration. If name differentiation is wanted, > call it xxxlist. Agreed, ideally you don't need to know or it'll be obvious from the name without an explicit 'list' or 'iter'. -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Tue Mar 20 01:43:10 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 20 Mar 2012 09:43:10 +0900 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: <4F678AEF.4080002@stoneleaf.us> Message-ID: On Tue, Mar 20, 2012 at 8:34 AM, Guido van Rossum wrote: > Anyway, I also tried to imply that it matters if the number of list > items would ever be huge. It seems that is indeed possible (even if > not likely) so I think iterators are useful. But according to Nick's post, there's some sort of uniquification that is done, and the algorithm currently used computes the whole list anyway. I suppose that one could do the uniquification lazily, or find some other way to avoid that computation. Is it worth it to optimize an unlikely case? From ncoghlan at gmail.com Tue Mar 20 02:19:01 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 Mar 2012 11:19:01 +1000 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: <4F678AEF.4080002@stoneleaf.us> Message-ID: On Tue, Mar 20, 2012 at 10:43 AM, Stephen J. Turnbull wrote: > But according to Nick's post, there's some sort of uniquification that > is done, and the algorithm currently used computes the whole list anyway. > > I suppose that one could do the uniquification lazily, or find some other > way to avoid that computation. ?Is it worth it to optimize an unlikely case? Yeah, the only where I thought retaining the list output made particular sense was "collapse_address_list". I have no problem with that operation expecting a real sequence as input and producing an actual list as output, since the entire (deduplicated) sequence will eventually end up in memory for checking purposes anyway, even if the result was an iterator rather than a list and it already has "list" in its name. The other instances I noticed should all just be a matter of replacing "output.append(value)" calls with "yield value" instead, so it seems sensible to standardise on a Py3k style iterators-instead-of-lists API for the standard library version of the module. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From steve at pearwood.info Tue Mar 20 03:51:37 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Mar 2012 13:51:37 +1100 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: <4F67AA1E.6070907@stoneleaf.us> References: <4F678AEF.4080002@stoneleaf.us> <4F679384.4040602@stoneleaf.us> <4F67AA1E.6070907@stoneleaf.us> Message-ID: <20120320025137.GA28460@ando> On Mon, Mar 19, 2012 at 02:50:22PM -0700, Ethan Furman wrote: > Guido van Rossum wrote: [...] > >And remember that consistency is good in moderation, but if it becomes > >a goal in itself you may have a problem. > > While I agree that consistency as a goal in and of itself is not good, I > consider it more important than 'moderation' implies; in my own code I > try to only be inconsistent when there is a good reason to be. I think we're probably in violent agreement, but I would put it this way: Consistency for its own sake *is* good, since consistency makes it easier for people to reason about the behaviour of functions on the basis that they are similar to other functions. But it is not the *only* good, and it is legitimate to trade-off one good for another good as needed. > To me, "it's already a list" isn't a good reason -- yes, that's easier > for the library author, but is it easier for the library user? What is > the library user gaining by having a list returned instead of an iterator? I guess this discussion really hinges on which of these two positions you take: 1. The function naturally returns a list, should we compromise that simplicity by returning an iterator to be consistent with the other related/similar functions in the library? 2. These related/similar functions naturally return iterators, should we compromise that consistency by allowing one of them to return a list as it simplifies the implementation? > Of course, the flip-side also holds: what is the library user losing by > getting an iterator when a list was available? > > When we way the pros and cons, and it comes down to a smidgeon of > performance in trade for consistency [1], I would vote for consistency. I lean that way as well. > ~Ethan~ > > [1] I'm assuming that 'iter(some_list)' is a quick operation. For very small lists, it's about half as expensive as creating the list in the first place: steve at runes:~$ python3.2 -m timeit -s "x = (1,2,3)" "list(x)" 1000000 loops, best of 3: 0.396 usec per loop steve at runes:~$ python3.2 -m timeit -s "x = (1,2,3)" "iter(list(x))" 1000000 loops, best of 3: 0.614 usec per loop For large lists, it's approximately free: steve at runes:~$ python3.2 -m timeit -s "x = (1,2,3)*10000" "list(x)" 10000 loops, best of 3: 111 usec per loop steve at runes:~$ python3.2 -m timeit -s "x = (1,2,3)*10000" "iter(list(x))" 10000 loops, best of 3: 111 usec per loop On the other hand, turning the list iterator into a list again is probably not quite so cheap. -- Steven From jimjjewett at gmail.com Tue Mar 20 04:51:35 2012 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 19 Mar 2012 20:51:35 -0700 (PDT) Subject: [Python-Dev] Issue #10278 -- why not just an attribute? In-Reply-To: Message-ID: <4f67fec7.65aa320a.62e4.0780@mx.google.com> In http://mail.python.org/pipermail/python-dev/2012-March/117762.html Georg Brandl posted: >> + If available, a monotonic clock is used. By default, if *strict* is False, >> + the function falls back to another clock if the monotonic clock failed or is >> + not available. If *strict* is True, raise an :exc:`OSError` on error or >> + :exc:`NotImplementedError` if no monotonic clock is available. > This is not clear to me. Why wouldn't it raise OSError on error even with > strict=False? Please clarify which exception is raised in which case. Passing strict as an argument seems like overkill since it will always be meaningless on some (most?) platforms. Why not just use a function attribute? Those few users who do care can check the value of time.steady.monotonic before calling time.steady(); exceptions raised will always be whatever the clock actually raises. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From steve at pearwood.info Tue Mar 20 07:54:57 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Mar 2012 17:54:57 +1100 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: Message-ID: <20120320065456.GC28460@ando> On Mon, Mar 19, 2012 at 01:35:49PM +0100, Victor Stinner wrote: > Said differently: time.steady(strict=True) is always monotonic (*), > whereas time.steady() may or may not be monotonic, depending on what > is avaiable. > > time.steady() is a best-effort steady clock. > > (*) time.steady(strict=True) relies on the OS monotonic clock. If the > OS provides a "not really monotonic" clock, Python cannot do better. I don't think that is true. Surely Python can guarantee that the clock will never go backwards by caching the last value. A sketch of an implementation: def monotonic(_last=[None]): t = system_clock() # best effort, but sometimes goes backwards if _last[0] is not None: t = max(t, _last[0]) _last[0] = t return t Overhead if done in Python may be excessive, in which case do it in C. Unless I've missed something, that guarantees monotonicity -- it may not advance from one call to the next, but it will never go backwards. There's probably even a cleverer implementation that will not repeat the same value more than twice in a row. I leave that as an exercise :) As far as I can tell, "steady" is a misnomer. We can't guarantee that the timer will tick at a steady rate. That will depend on the quality of the hardware clock. -- Steven From tshepang at gmail.com Tue Mar 20 08:24:17 2012 From: tshepang at gmail.com (Tshepang Lekhonkhobe) Date: Tue, 20 Mar 2012 09:24:17 +0200 Subject: [Python-Dev] regarding HTML mail In-Reply-To: <20120319142046.5a2ce86d@resist.wooz.org> References: <20120319142046.5a2ce86d@resist.wooz.org> Message-ID: On Mon, Mar 19, 2012 at 20:20, Barry Warsaw wrote: >>Can we have some guideline to allow only plain text emails, so as to >>avoid cases like >>http://mail.python.org/pipermail/docs/2012-March/007999.html, where >>you are forced to scroll horizontally in order to read the text. > > docs is a different mailing list than python-dev, but still neither list is > doing any content filtering. ?We could always enable that if we wanted to get > strict about it. > > In this case, this isn't html email, it's likely this bug: > > https://bugs.launchpad.net/mailman/+bug/558294 Maybe it is, but it's strange that my reply to that mail was nicely done: http://mail.python.org/pipermail/docs/2012-March/008001.html. From anacrolix at gmail.com Tue Mar 20 08:33:51 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 20 Mar 2012 15:33:51 +0800 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: <20120320065456.GC28460@ando> References: <20120320065456.GC28460@ando> Message-ID: I believe we should make a monotonic_time method that assures monotonicity and be done with it. Forward steadiness can not be guaranteed. No parameters. On Mar 20, 2012 2:56 PM, "Steven D'Aprano" wrote: > On Mon, Mar 19, 2012 at 01:35:49PM +0100, Victor Stinner wrote: > > > Said differently: time.steady(strict=True) is always monotonic (*), > > whereas time.steady() may or may not be monotonic, depending on what > > is avaiable. > > > > time.steady() is a best-effort steady clock. > > > > (*) time.steady(strict=True) relies on the OS monotonic clock. If the > > OS provides a "not really monotonic" clock, Python cannot do better. > > I don't think that is true. Surely Python can guarantee that the clock > will never go backwards by caching the last value. A sketch of an > implementation: > > def monotonic(_last=[None]): > t = system_clock() # best effort, but sometimes goes backwards > if _last[0] is not None: > t = max(t, _last[0]) > _last[0] = t > return t > > Overhead if done in Python may be excessive, in which case do it in C. > > Unless I've missed something, that guarantees monotonicity -- it may not > advance from one call to the next, but it will never go backwards. > > There's probably even a cleverer implementation that will not repeat the > same value more than twice in a row. I leave that as an exercise :) > > As far as I can tell, "steady" is a misnomer. We can't guarantee that > the timer will tick at a steady rate. That will depend on the quality of > the hardware clock. > > > -- > Steven > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Mar 20 09:43:44 2012 From: glyph at twistedmatrix.com (Glyph) Date: Tue, 20 Mar 2012 04:43:44 -0400 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: <20120320065456.GC28460@ando> Message-ID: On Mar 20, 2012, at 3:33 AM, Matt Joiner wrote: > I believe we should make a monotonic_time method that assures monotonicity and be done with it. Forward steadiness can not be guaranteed. No parameters. > I think this discussion has veered off a bit into the overly-theoretical. Python cannot really "guarantee" anything here; alternately, it guarantees everything, since if you don't like what Python gives you you can always get your money back :). It's the OS's job to guarantee things. We can all agree that a monotonic clock of some sort is useful. However, maybe my application wants CLOCK_MONOTONIC and maybe it wants CLOCK_MONOTONIC_RAW. Sometimes I want GetTickCount64 and sometimes I want QueryUnbiasedInterruptTime. While these distinctions are probably useless to most applications, they may be of interest to some, and Python really shouldn't make it unduly difficult to get at them. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Mar 20 06:09:52 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 20 Mar 2012 18:09:52 +1300 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: References: <4F678AEF.4080002@stoneleaf.us> Message-ID: <4F681120.90502@canterbury.ac.nz> Guido van Rossum wrote: > I personally like having 'iter' in the name (e.g. iterkeys() -- note > that we dropped this in Py3k because it's no longer an iterator, it's > a dict view now. But I don't want to promote that style for ipaddr.py. +1 from me too on having all methods that return iterators clearly indicating so. It's an important distinction, and it can be very confusing if some methods of an API return iterators and others don't with no easy way of remembering which is which. -- Greg From victor.stinner at gmail.com Tue Mar 20 10:25:13 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 20 Mar 2012 10:25:13 +0100 Subject: [Python-Dev] Issue #10278 -- why not just an attribute? In-Reply-To: <4f67fec7.65aa320a.62e4.0780@mx.google.com> References: <4f67fec7.65aa320a.62e4.0780@mx.google.com> Message-ID: 2012/3/20 Jim J. Jewett : > > > In http://mail.python.org/pipermail/python-dev/2012-March/117762.html > Georg Brandl posted: > >>> + ? If available, a monotonic clock is used. By default, if *strict* is False, >>> + ? the function falls back to another clock if the monotonic clock failed or is >>> + ? not available. If *strict* is True, raise an :exc:`OSError` on error or >>> + ? :exc:`NotImplementedError` if no monotonic clock is available. > >> This is not clear to me. ?Why wouldn't it raise OSError on error even with >> strict=False? ?Please clarify which exception is raised in which case. > > Passing strict as an argument seems like overkill since it will always > be meaningless on some (most?) platforms. ?Why not just use a function > attribute? ?Those few users who do care can check the value of > time.steady.monotonic before calling time.steady(); exceptions raised > will always be whatever the clock actually raises. The clock is chosen at runtime. You might use a different clock at each call. In most cases, Python should chose a clock at the first call and reuse it for next calls. For example, on Linux the following clocks are tested: - clock_gettime(CLOCK_MONONOTIC_RAW) - clock_gettime(CLOCK_MONONOTIC) - gettimeofday() - ftime() Victor From victor.stinner at gmail.com Tue Mar 20 10:33:11 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 20 Mar 2012 10:33:11 +0100 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: <20120320065456.GC28460@ando> Message-ID: > I think this discussion has veered off a bit into the overly-theoretical. > ?Python cannot really "guarantee" anything here That's why the function name was changed from time.monotonic() to time.steady(strict=True). If you want to change something, you should change the documentation to list OS limitations. > It's the OS's job to guarantee things Correct, most Python modules exposing OS functions are thin wrappers and don't add any magic. When we need a higher level API, we write a new module: like shutil enhancing the os module. Victor From skippy.hammond at gmail.com Tue Mar 20 11:48:27 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Tue, 20 Mar 2012 21:48:27 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: <4F68607B.7060307@gmail.com> For those who missed it, in http://bugs.python.org/issue14302, Martin recently commented: """ After more discussion, it appears that this change is too incompatible to be done in a single release. Therefore, I propose a long-term change into this direction, with the actual change not happening until 3.5. """ While I'm still unclear on the actual benefits of this, Martin's approach strikes a reasonable compromise so I withdraw my objections. Thanks, Mark From rdmurray at bitdance.com Tue Mar 20 13:32:10 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 20 Mar 2012 08:32:10 -0400 Subject: [Python-Dev] cpython: Issue #10278: Add an optional strict argument to time.steady(), False by default In-Reply-To: References: <20120320065456.GC28460@ando> Message-ID: <20120320123211.AA9F52500E7@webabinitio.net> On Tue, 20 Mar 2012 04:43:44 -0400, Glyph wrote: > > On Mar 20, 2012, at 3:33 AM, Matt Joiner wrote: > > > I believe we should make a monotonic_time method that assures monotonicity and be done with it. Forward steadiness can not be guaranteed. No parameters. > > > > I think this discussion has veered off a bit into the overly-theoretical. Python cannot really "guarantee" anything here; alternately, it guarantees everything, since if you don't like what Python gives you you can always get your money back :). It's the OS's job to guarantee things. We can all agree that a monotonic clock of some sort is useful. > > However, maybe my application wants CLOCK_MONOTONIC and maybe it wants CLOCK_MONOTONIC_RAW. Sometimes I want GetTickCount64 and sometimes I want QueryUnbiasedInterruptTime. While these distinctions are probably useless to most applications, they may be of interest to some, and Python really shouldn't make it unduly difficult to get at them. Something like: time.steady(require_clock=None) where require_clock can be any of BEST, CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW, GetTickCount64, QueryUnbiastedInterruptTime, etc? Then None would mean it is allowable to use time.time and the cache-the-last-time-returned algorithm, and BEST would be Victor's current 'strict=True'. And if you require a Linux clock on Windows or vice-versa, on your own head be it :) --David From victor.stinner at gmail.com Tue Mar 20 13:32:43 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 20 Mar 2012 13:32:43 +0100 Subject: [Python-Dev] pysandbox 1.5 released Message-ID: pysandbox is a Python sandbox. By default, untrusted code executed in the sandbox cannot modify the environment (write a file, use print or import a module). But you can configure the sandbox to choose exactly which features are allowed or not, e.g. import sys module and read /etc/issue file. http://pypi.python.org/pypi/pysandbox https://github.com/haypo/pysandbox/ Main changes since pysandbox 1.0.3: - More modules and functions are allowed: math, random and time modules, and the compile() builtin function for example - Drop the timeout feature: it was not effective on CPU intensive functions implemented in C - (Read the ChangeLog to see all changes.) pysandbox has known limitations: - it is unable to limit memory or CPU - it does not protect against bugs (e.g. crash) or vulnerabilities in CPython - dict methods able to modify a dict (e.g. dict.update) are disabled to protect the sandbox namespace, but dict[key]=value is still accepted It is recommanded to run untrusted code in a subprocess to workaround these limitations. pysandbox doesn't provide an helper yet. pysandbox is used by an IRC bot (fschfsch) to evaluate a Python expression. The bot uses fork() and setrlimit() to limit memory and to implement a timeout. https://github.com/haypo/pysandbox/wiki/fschfsch -- The limitation on dict methods is required to deny the modification of the __builtins__ dictionary. I proposed the PEP 416 (frozendict) but Guido van Rossum is going to reject it. I don't see how to fix this limitation without modifying CPython. http://www.python.org/dev/peps/pep-0416/ Victor From ethan at stoneleaf.us Tue Mar 20 13:23:32 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 20 Mar 2012 05:23:32 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: <4F681120.90502@canterbury.ac.nz> References: <4F678AEF.4080002@stoneleaf.us> <4F681120.90502@canterbury.ac.nz> Message-ID: <4F6876C4.7050808@stoneleaf.us> Greg Ewing wrote: > Guido van Rossum wrote: > >> I personally like having 'iter' in the name (e.g. iterkeys() -- note >> that we dropped this in Py3k because it's no longer an iterator, it's >> a dict view now. But I don't want to promote that style for ipaddr.py. > > +1 from me too on having all methods that return iterators > clearly indicating so. It's an important distinction, and > it can be very confusing if some methods of an API return > iterators and others don't with no easy way of remembering > which is which. With the prevalence of iterators in Python 3 [1], the easy way is to have the API default to iterators, drop 'iter' from the names, and use 'list' in the names to signal the oddball cases where a list is returned instead. ~Ethan~ [1] http://mail.python.org/pipermail/python-dev/2012-March/117815.html From Van.Lindberg at haynesboone.com Tue Mar 20 15:08:10 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Tue, 20 Mar 2012 14:08:10 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68607B.7060307@gmail.com> References: <4F68607B.7060307@gmail.com> Message-ID: <4F688F48.9010101@gmail.com> On 3/20/2012 5:48 AM, Mark Hammond wrote: > While I'm still unclear on the actual benefits of this, Martin's > approach strikes a reasonable compromise so I withdraw my objections. Ok. I was out of town and so could not respond to most of the latest discussion. A question for you Mark, Paul, (and anyone else): ?ric correctly points out that there are actually two distinct changes proposed here: 1. Moving the Python binary 2. Changing from "Scripts" to "bin" So far, the primary resistance seems to be to item #1 - moving the python binary. There have been a few people who have noted that #2 will require some code to change (i.e. Paul), but I don't see lots of resistance. Am I reading you correctly? Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From van.lindberg at gmail.com Tue Mar 20 15:08:08 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 09:08:08 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68607B.7060307@gmail.com> References: <4F68607B.7060307@gmail.com> Message-ID: <4F688F48.9010101@gmail.com> On 3/20/2012 5:48 AM, Mark Hammond wrote: > While I'm still unclear on the actual benefits of this, Martin's > approach strikes a reasonable compromise so I withdraw my objections. Ok. I was out of town and so could not respond to most of the latest discussion. A question for you Mark, Paul, (and anyone else): ?ric correctly points out that there are actually two distinct changes proposed here: 1. Moving the Python binary 2. Changing from "Scripts" to "bin" So far, the primary resistance seems to be to item #1 - moving the python binary. There have been a few people who have noted that #2 will require some code to change (i.e. Paul), but I don't see lots of resistance. Am I reading you correctly? Thanks, Van From mail at timgolden.me.uk Tue Mar 20 15:26:59 2012 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 20 Mar 2012 14:26:59 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F688F48.9010101@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> Message-ID: <4F6893B3.7090600@timgolden.me.uk> On 20/03/2012 14:08, VanL wrote: > On 3/20/2012 5:48 AM, Mark Hammond wrote: >> While I'm still unclear on the actual benefits of this, Martin's >> approach strikes a reasonable compromise so I withdraw my objections. > > > Ok. I was out of town and so could not respond to most of the latest > discussion. > > A question for you Mark, Paul, (and anyone else): ?ric correctly points > out that there are actually two distinct changes proposed here: > > 1. Moving the Python binary > 2. Changing from "Scripts" to "bin" > > So far, the primary resistance seems to be to item #1 - moving the > python binary. There have been a few people who have noted that #2 will > require some code to change (i.e. Paul), but I don't see lots of > resistance. Speaking for myself, I think that's true. At present I tend to add /scripts to my path and I can just as easily add /bin (for whatever version I'm running most often on that machine). TJG From van.lindberg at gmail.com Tue Mar 20 15:27:51 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 09:27:51 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68607B.7060307@gmail.com> References: <4F68607B.7060307@gmail.com> Message-ID: Germane to this discussion, I reached out for feedback. Most people didn't care about the issue, or were slightly inclined to have it be uniform across platforms. As Terry mentioned, I think that long-term uniformity will benefit everybody down the line, and that is the way to go. The most interesting feedback, though, related to moving the Python exe and placing it on the PATH. I got one argument back that I thought was persuasive here: We want things to 'just work.' Specifically, the following sequence of events should not require any fiddling on Windows: 1. Install python. 2. Open up a shell and run "python" 3. Use pip or easy_install to install regetron (a package that installs an executable file). 4. Run regetron. For step #2, the python exe needs to be on the PATH. For steps 3 and 4, the binaries directory needs to be on the PATH. In hearing from a couple people who teach python to beginners, this is a substantial hurdle - the first thing they need to do is to edit their environment to add these directories to the PATH. This is orthogonal to the Scripts/bin issue, but I thought it should be brought up. From martin at v.loewis.de Tue Mar 20 16:52:44 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Mar 2012 16:52:44 +0100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> Message-ID: <4F68A7CC.4000606@v.loewis.de> > In hearing from a couple people who teach python to beginners, this is a > substantial hurdle - the first thing they need to do is to edit their > environment to add these directories to the PATH. This is something I never understood. On Windows, it's custom to launch programs from the start menu, and Python is easy enough to find on the start menu (e.g. by typing "Python"). Why do people want to launch it by opening a shell window, then typing python? In any case, I have given up my resistance to the feature request for automatic path fiddling several years ago, and was since waiting for a contribution of a patch that makes it happen. Regards, Martin From yselivanov.ml at gmail.com Tue Mar 20 16:57:48 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 20 Mar 2012 11:57:48 -0400 Subject: [Python-Dev] pysandbox 1.5 released In-Reply-To: References: Message-ID: <0D8D9E6B-75DD-4925-BE5A-C60F7C6CF271@gmail.com> Well, I really hope that the PEP regarding frozendict will be accepted. Especially due to the fact that the required changes are small. With the recent projects like clojure-py, blog posts like http://goo.gl/bFB5x (Python becomes a platform), your pysandbox, it became clear that people start evaluating Python on a different level. And for developing custom languages, deeply experimenting with coroutines and who knows what else, frozendict is a missing concept in python's immutable types structure. On 2012-03-20, at 8:32 AM, Victor Stinner wrote: > pysandbox is a Python sandbox. By default, untrusted code executed in > the sandbox cannot modify the environment (write a file, use print or > import a module). But you can configure the sandbox to choose exactly > which features are allowed or not, e.g. import sys module and read > /etc/issue file. > > http://pypi.python.org/pypi/pysandbox > https://github.com/haypo/pysandbox/ > > Main changes since pysandbox 1.0.3: > > - More modules and functions are allowed: math, random and time > modules, and the compile() builtin function for example > - Drop the timeout feature: it was not effective on CPU intensive > functions implemented in C > - (Read the ChangeLog to see all changes.) > > pysandbox has known limitations: > > - it is unable to limit memory or CPU > - it does not protect against bugs (e.g. crash) or vulnerabilities in CPython > - dict methods able to modify a dict (e.g. dict.update) are disabled > to protect the sandbox namespace, but dict[key]=value is still > accepted > > It is recommanded to run untrusted code in a subprocess to workaround > these limitations. pysandbox doesn't provide an helper yet. > > pysandbox is used by an IRC bot (fschfsch) to evaluate a Python > expression. The bot uses fork() and setrlimit() to limit memory and to > implement a timeout. > > https://github.com/haypo/pysandbox/wiki/fschfsch > > -- > > The limitation on dict methods is required to deny the modification of > the __builtins__ dictionary. I proposed the PEP 416 (frozendict) but > Guido van Rossum is going to reject it. I don't see how to fix this > limitation without modifying CPython. > > http://www.python.org/dev/peps/pep-0416/ > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com From van.lindberg at gmail.com Tue Mar 20 17:02:27 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 11:02:27 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68A7CC.4000606@v.loewis.de> References: <4F68607B.7060307@gmail.com> <4F68A7CC.4000606@v.loewis.de> Message-ID: <4F68AA13.9030004@gmail.com> On 3/20/2012 10:52 AM, "Martin v. L?wis" wrote: >> In hearing from a couple people who teach python to beginners, this is a >> substantial hurdle - the first thing they need to do is to edit their >> environment to add these directories to the PATH. > This is something I never understood. On Windows, it's custom to launch > programs from the start menu, and Python is easy enough to find on the > start menu (e.g. by typing "Python"). Why do people want to launch it by > opening a shell window, then typing python? Because the workflow you suggest is broken when you are developing with Python. Assume that you are iteratively building up a program in Python. You aren't sure if it is right yet, so you want to get it into python to test it and see the output. There are three ways to do this. 1. Run python from the start menu. - Import sys, fiddle with sys.path to add my module, import/run my module, do my tests. When you exit /hard error out, the python window disappears. 2. Double-click the .py file - Runs the file, but then disappears immediately (unless you put in something like input/raw_input just to keep the window open) - and if it errors out, you never see the traceback - it disappears too fast. 3. Get a shell and run python. This requires cd'ing to the directory where my .py file is, but then I run/import it and I see the information. To repeat the process, either type python again or just press arrow-up. 4. (Not relevant here) - do it in an IDE that does #3 for you. #3 is the only reasonable way to do development if you are not in an IDE. Thanks, Van From brian at python.org Tue Mar 20 17:04:53 2012 From: brian at python.org (Brian Curtin) Date: Tue, 20 Mar 2012 11:04:53 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68A7CC.4000606@v.loewis.de> References: <4F68607B.7060307@gmail.com> <4F68A7CC.4000606@v.loewis.de> Message-ID: On Tue, Mar 20, 2012 at 10:52, "Martin v. L?wis" wrote: >> In hearing from a couple people who teach python to beginners, this is a >> substantial hurdle - the first thing they need to do is to edit their >> environment to add these directories to the PATH. > > This is something I never understood. On Windows, it's custom to launch > programs from the start menu, and Python is easy enough to find on the > start menu (e.g. by typing "Python"). Why do people want to launch it by > opening a shell window, then typing python? I've never thought about doing it otherwise. If I want to run the C:\Users\brian\example\sample.py script, I'd open a CMD and move to the example directory and execute the sample script. The class of about 60 people I taught a few years back at a previous employer all did the same thing without me specifying. Everyone was used to working in the command line for other tasks, from using other languages to running our products, so it was natural to them to run it that way. > In any case, I have given up my resistance to the feature request for > automatic path fiddling several years ago, and was since waiting for > a contribution of a patch that makes it happen. I'm working on the changes we discussed at PyCon. http://bugs.python.org/issue3561 has an version of the patch doing it the old way - I hope to have the new way figured out soon. From martin at v.loewis.de Tue Mar 20 17:19:55 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Mar 2012 17:19:55 +0100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68AA13.9030004@gmail.com> References: <4F68607B.7060307@gmail.com> <4F68A7CC.4000606@v.loewis.de> <4F68AA13.9030004@gmail.com> Message-ID: <4F68AE2B.1010507@v.loewis.de> > 1. Run python from the start menu. > - Import sys, fiddle with sys.path to add my module, import/run my > module, do my tests. When you exit /hard error out, the python window > disappears. > > 2. Double-click the .py file > - Runs the file, but then disappears immediately (unless you put in > something like input/raw_input just to keep the window open) - and if it > errors out, you never see the traceback - it disappears too fast. > > 3. Get a shell and run python. > This requires cd'ing to the directory where my .py file is, but then I > run/import it and I see the information. To repeat the process, either > type python again or just press arrow-up. > > 4. (Not relevant here) - do it in an IDE that does #3 for you. > > #3 is the only reasonable way to do development if you are not in an IDE. No - there is an version #3a: 3.a) Get a shell and run the script CD into the directory, then directly run foo.py, without prefixing it with python.exe. This doesn't require any changes to the path, and is shorter in usage than having the path specified. With PEP 397, you will be able to run "py foo.py" without path modification, and it will get the correct Python version even (which neither the path manipulation nor the file association could achieve). Regards, Martin From van.lindberg at gmail.com Tue Mar 20 17:24:33 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 11:24:33 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68AE2B.1010507@v.loewis.de> References: <4F68607B.7060307@gmail.com> <4F68A7CC.4000606@v.loewis.de> <4F68AA13.9030004@gmail.com> <4F68AE2B.1010507@v.loewis.de> Message-ID: <4F68AF41.2040502@gmail.com> On 3/20/2012 11:19 AM, "Martin v. L?wis" wrote: > No - there is an version #3a: 3.a) Get a shell and run the script CD > into the directory, then directly run foo.py, without prefixing it > with python.exe. This doesn't require any changes to the path, and is > shorter in usage than having the path specified. Fair enough - but notice: 1) You are still in the shell, instead of running from the start menu; and 2) what if you want to import it and test a function with various inputs? You either implement a request/response in a __main__ block, or type "python" and then import foo. From tjreedy at udel.edu Tue Mar 20 17:24:31 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Mar 2012 12:24:31 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68AA13.9030004@gmail.com> References: <4F68607B.7060307@gmail.com> <4F68A7CC.4000606@v.loewis.de> <4F68AA13.9030004@gmail.com> Message-ID: On 3/20/2012 12:02 PM, VanL wrote: > On 3/20/2012 10:52 AM, "Martin v. L?wis" wrote: >>> In hearing from a couple people who teach python to beginners, this is a >>> substantial hurdle - the first thing they need to do is to edit their >>> environment to add these directories to the PATH. >> This is something I never understood. On Windows, it's custom to launch >> programs from the start menu, and Python is easy enough to find on the >> start menu (e.g. by typing "Python"). Why do people want to launch it by >> opening a shell window, then typing python? Perhaps as the number of *nix users increases, the number of (*nix & windows) developer/users increases. > 3. Get a shell and run python. > This requires cd'ing to the directory where my .py file is, but then I > run/import it and I see the information. When IDLE crashes, running it from a cmd window is the only way to get a traceback to help diagnose the problem. -- Terry Jan Reedy From van.lindberg at gmail.com Tue Mar 20 17:28:04 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 11:28:04 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68AE2B.1010507@v.loewis.de> References: <4F68607B.7060307@gmail.com> <4F68A7CC.4000606@v.loewis.de> <4F68AA13.9030004@gmail.com> <4F68AE2B.1010507@v.loewis.de> Message-ID: <4F68B014.20906@gmail.com> On 3/20/2012 11:19 AM, "Martin v. L?wis" wrote: > No - there is an version #3a: 3.a) Get a shell and run the script CD > into the directory, then directly run foo.py, without prefixing it > with python.exe. This doesn't require any changes to the path, and is > shorter in usage than having the path specified. With PEP 397, you > will be able to run "py foo.py" without path modification, and it will > get the correct Python version even (which neither the path > manipulation nor the file association could achieve). There is also one more scenario, assuming that your project includes other libraries. You can run setup.py directly in your example, but what about pip or easy_install? Both of those require the binaries directory to also be on the PATH - requiring at least a little PATH manipulation. Thanks, Van From p.f.moore at gmail.com Tue Mar 20 19:19:26 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Mar 2012 18:19:26 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F688F48.9010101@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> Message-ID: On 20 March 2012 14:08, Lindberg, Van wrote: > On 3/20/2012 5:48 AM, Mark Hammond wrote: >> While I'm still unclear on the actual benefits of this, Martin's >> approach strikes a reasonable compromise so I withdraw my objections. > > > Ok. I was out of town and so could not respond to most of the latest > discussion. > > A question for you Mark, Paul, (and anyone else): ?ric correctly points > out that there are actually two distinct changes proposed here: > > 1. Moving the Python binary > 2. Changing from "Scripts" to "bin" > > So far, the primary resistance seems to be to item #1 - moving the > python binary. There have been a few people who have noted that #2 will > require some code to change (i.e. Paul), but I don't see lots of resistance. > > Am I reading you correctly? Somewhat. I don't really object to #1, but mildly object to #2. I also note that the proposals round the Lib directory seem to have disappeared. I assume those have been dropped - they were the ones I did object to. I also note that I'm assuming virtualenv will change to match whatever the Python version it's referencing does. I don't see how you can guarantee that, but if there are discrepancies between virtualenvs and installed Pythons, my level of objection goes up a little more. Martin's suggestion of an intermediate registry entry to ease transition doesn't help me. So I don't care about that. See a later message for my comments on PATH as it affects this discussion, though. Paul. From martin at v.loewis.de Tue Mar 20 19:17:05 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Tue, 20 Mar 2012 19:17:05 +0100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F68A7CC.4000606@v.loewis.de> <4F68AA13.9030004@gmail.com> Message-ID: <20120320191705.Horde.N5RmG7uWis5PaMmhOTiH9RA@webmail.df.eu> > When IDLE crashes, running it from a cmd window is the only way to > get a traceback to help diagnose the problem. Certainly. In this case, there is no PATH issue, though: you have to CD into the Python installation, anyway, to start IDLE - and there you have python.exe in the current directory. Plus, you can still launch Lib\idlelib\idle.py without prefixing it with python.exe. Regards, Martin From carl at oddbird.net Tue Mar 20 19:35:09 2012 From: carl at oddbird.net (Carl Meyer) Date: Tue, 20 Mar 2012 12:35:09 -0600 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> Message-ID: <4F68CDDD.6080000@oddbird.net> On 03/20/2012 12:19 PM, Paul Moore wrote: > I also note that I'm assuming virtualenv will change to match whatever > the Python version it's referencing does. I don't see how you can > guarantee that, but if there are discrepancies between virtualenvs and > installed Pythons, my level of objection goes up a little more. Virtualenv will follow whatever Python does, yes. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From valhallasw at arctus.nl Tue Mar 20 19:50:42 2012 From: valhallasw at arctus.nl (Merlijn van Deen) Date: Tue, 20 Mar 2012 19:50:42 +0100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: On 13 March 2012 20:43, VanL wrote: > Following up on conversations at PyCon, I want to bring up one of my > personal hobby horses for change in 3.3: Fix install layout on Windows, with > a side order of making the PATH work better. As this is being considered an 'incompatible change' on the bug tracker item [1] in any case, I'd like to mention that this might also be a convenient moment to re-think the default install location. After all, software is supposed to be installed in %programfiles% on windows, not in c:\ . I asked a question about this on IRC, to which the response was that there were two main reasons to install python in c:\pythonxy: 1 - issues due to spaces ('Program Files') or non-ascii characters in the path ('Fi?iere Program' on a Romanian windows). These issues are supposed to be fixed by now (?). 2 - issues due to permissions - installing python / packages in %programfiles% may require administrator rights. Historical note: in python 1.5 the install location was changed to \program files\.., but in python 1.6/2.0 it was changed (back?) to \pythonxy. [2 @ 10618, 10850, 13804] [1] http://bugs.python.org/issue14302 [2] http://hg.python.org/cpython/file/a5add01e96be/Misc/HISTORY From p.f.moore at gmail.com Tue Mar 20 19:56:58 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Mar 2012 18:56:58 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> Message-ID: On 20 March 2012 14:27, VanL wrote: > Germane to this discussion, I reached out for feedback. Most people didn't > care about the issue, or were slightly inclined to have it be uniform across > platforms. > > As Terry mentioned, I think that long-term uniformity will benefit everybody > down the line, and that is the way to go. > > The most interesting feedback, though, related to moving the Python exe and > placing it on the PATH. I got one argument back that I thought was > persuasive here: We want things to 'just work.' Specifically, the following > sequence of events should not require any fiddling on Windows: > > 1. Install python. > 2. Open up a shell and run "python" > 3. Use pip or easy_install to install regetron (a package that installs an > executable file). > 4. Run regetron. > > For step #2, the python exe needs to be on the PATH. For steps 3 and 4, the > binaries directory needs to be on the PATH. This is covered (better, IMO) by PEP 397 - Python Launcher for Windows. Step 2, just run "py". If you prefer a particular version, run "py -2" or "py -3" or "py -3.2". Adding python.exe to PATH actually makes this message *worse* as it confuses the issue. In a post-PEP 397 world, I would say that we should be telling Windows users *not* to run python.exe at all. (Unless they are using virtualenvs - maybe PEP 397 could do with an extra option to run the Python from the currently active virtualenv, but that's a side issue). If we do put python.exe on PATH (whether it's in bin or not), we have to debate how to handle people having multiple versions of python on their machine. In a post-PEP 397 world, no Python is "the machine default" - .py files are associated with py.exe, not python.exe, so we have to consider the following 3 commands being run from a shell prompt: 1. myprog.py 2. py myprog.py 3. python myprog.py 1 and 2 will always do the same thing. However, 3 could easily do something completely different, if the Python in the #! line differs from the one found on PATH. To me, this implies that it's better for (3) to need explicit user action (setting PATH) if it's to do anything other than give an error. But maybe that's just me. I've been hit too often by confusion caused by *not* remembering this fact. Note: I am using Vinay's py.exe all the time these days, so my comments about a "post-PEP 397 world" are from my direct experience. For your steps 3 and 4, there is certainly user intervention required as things stand. It would indeed be nice if "regetron" just worked as expected. But I'd argue a few points here: 1. On Windows, if regetron was not explicitly an application for working with my Python installation (like pip, easy_install, nose, etc) then I'd prefer it to be packaged as a standalone application using cx_Freeze or py2exe. I've had too many "applications" break because I accidentally uninstalled a dependency from my Python installation to want anything that is an end-user application installed into my Python scripts/bin directory. 2. If regetron is not an end-user application, it should really be getting installed in, and run from, a virtualenv. And in that case, activating the right virtualenv is part of the workflow. And that sets up PATH as needed, so there's no problem here. The problem with your example is that it depends on the package/executable. I looked at regetron (I thought it was a made up example at first!) and it seems clear to me that it should either be packaged up with py2exe/cx_Freeze, or (if it's sufficiently version-independent) sit outside of Python's installation tree. I can't see any reason why I'd expect a "regetron" command to work or not depending on which copy of Python on my PC I have active. But other applications may differ, I guess. I concede that the picture is much simpler when people only ever have a single version of Python on their machine. So for that case alone, maybe the "Make this Python the default" option in the installer should add the bin directory (or Scripts and the root, under the current layout) to the PATH. But equally, if the installer detects another copy of Python already installed, it should probably warn the user loudly that it's important to understand the implications of setting "make this Python the default", and should not set it by default (this may be the current behaviour, I don't know). I have no idea what proportion of Python users on Windows have multiple versions installed. I also have no idea how many such users work on the command line. My guess would be "not that many" and "not that many of the first group" respectively... But there are big groups like scientists and web developers who could sway these figures a lot. > In hearing from a couple people who teach python to beginners, this is a > substantial hurdle - the first thing they need to do is to edit their > environment to add these directories to the PATH. I'd be curious as to how much PEP 397's py.exe would have helped those people. But yes, it's an issue. Although someone at some point will have to introduce those beginners to the question of Python 2 vs Python 3, and PATH pain will hit them then, anyway. > This is orthogonal to the Scripts/bin issue, but I thought it should be > brought up. Agreed (both that it's orthogonal and that it should be discussed). Once Python 2 is dead and gone, these issues will be a lot simpler - but I don't think that's going to be for a few years yet. Paul. From van.lindberg at gmail.com Tue Mar 20 20:02:02 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 14:02:02 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: On 3/20/2012 1:50 PM, Merlijn van Deen wrote: > On 13 March 2012 20:43, VanL wrote: >> Following up on conversations at PyCon, I want to bring up one of my >> personal hobby horses for change in 3.3: Fix install layout on Windows, with >> a side order of making the PATH work better. > > As this is being considered an 'incompatible change' on the bug > tracker item [1] in any case, I'd like to mention that this might also > be a convenient moment to re-think the default install location. After > all, software is supposed to be installed in %programfiles% on > windows, not in c:\ . I don't particularly care about this issue, as I always install to my own location (c:\lib\python\X.Y), but I don't think that the default location of the install should be confounded with this issue - or should I say these issues, because we already have two. From van.lindberg at gmail.com Tue Mar 20 20:22:55 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 14:22:55 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> Message-ID: <4F68D90F.1080504@gmail.com> On 3/20/2012 1:56 PM, Paul Moore wrote: > This is covered (better, IMO) by PEP 397 - Python Launcher for > Windows. Step 2, just run "py". If you prefer a particular version, > run "py -2" or "py -3" or "py -3.2". Interesting. I haven't played around with that at all, so can't comment. I will have to try it out. > Adding python.exe to PATH actually makes this message *worse* as it > confuses the issue. In a post-PEP 397 world, I would say that we > should be telling Windows users *not* to run python.exe at all. > (Unless they are using virtualenvs - maybe PEP 397 could do with an > extra option to run the Python from the currently active virtualenv, > but that's a side issue). I think that having the PATH manipulation be optional might address this issue. I also think that the PEP 397 launcher should respect virtualenvs - at least the built-in pyvenvs - or else there will be lots of confusion. > For your steps 3 and 4, there is certainly user intervention > required as things stand. It would indeed be nice if "regetron" just > worked as expected. But I'd argue a few points here: > > 1. On Windows, if regetron was not explicitly an application for > working with my Python installation (like pip, easy_install, nose, > etc) then I'd prefer it to be packaged as a standalone application > using cx_Freeze or py2exe. I've had too many "applications" break > because I accidentally uninstalled a dependency from my Python > installation to want anything that is an end-user application > installed into my Python scripts/bin directory. Maybe so - and I would probably agree that for any packaged application, bundling it into its own environment (or at least its own virtualenv) is the best practice. But what about pip, easy_install, nose, cython, pygments, PIL, etc, that do this and are meant to be associated with a particular python version? Substitute "nose" for "regetron" if you want, and there is still a problem. > The problem with your example is that it depends on the > package/executable. I looked at regetron (I thought it was a made up > example at first!) ...! I got the name from the feedback I received. I thought it was made up too. > I have no idea what proportion of Python users on Windows have > multiple versions installed. I also have no idea how many such users > work on the command line. My guess would be "not that many" and "not > that many of the first group" respectively... But there are big > groups like scientists and web developers who could sway these > figures a lot. There are a number of casual users that probably only have one version installed, but every python user/dev on windows that I know has one python that they consider to be "python," and everything else needs to be launched with a suffix (e.g., python26.exe). This is usually put earlier on the PATH so that it gets picked up first. For example, right now I have 2.6, 2.7, 3.2, jython, and pypy all installed, and I have "python" pointing to 2.7. > I'd be curious as to how much PEP 397's py.exe would have helped > those people. But yes, it's an issue. Although someone at some point > will have to introduce those beginners to the question of Python 2 vs > Python 3, and PATH pain will hit them then, anyway. I would imagine that it would help steps 1 and 2, but 3 and 4 would be problematic (how can you pip install something using py?) unless you were in a virtualenv, and then (unless py respected the virtualenv) the whole thing would be problematic, because there wouldn't be one clear way to do it. From Van.Lindberg at haynesboone.com Tue Mar 20 20:35:12 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Tue, 20 Mar 2012 19:35:12 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> Message-ID: <4F68DBEF.9070105@gmail.com> On 3/20/2012 1:19 PM, Paul Moore wrote: > Somewhat. I don't really object to #1, but mildly object to #2. I also > note that the proposals round the Lib directory seem to have > disappeared. I assume those have been dropped - they were the ones I > did object to. They are of secondary importance to me, and I would be mostly ok to drop them - but I would like to understand your objection. I would like to know if you would object to user lib installs matching the system install. I.e., would it cause problems with you if it were just 'lib' everywhere, with no 'lib/python{version}'? It sounded like adding the version directory was the issue. Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From benjamin at python.org Tue Mar 20 21:09:04 2012 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 20 Mar 2012 16:09:04 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Issue #14328: Add keyword-only parameters to PyArg_ParseTupleAndKeywords. In-Reply-To: References: Message-ID: 2012/3/20 larry.hastings : > http://hg.python.org/cpython/rev/052779d34945 > changeset: ? 75842:052779d34945 > parent: ? ? ?75839:1c0058991740 > user: ? ? ? ?Larry Hastings > date: ? ? ? ?Tue Mar 20 20:06:16 2012 +0000 > summary: > ?Issue #14328: Add keyword-only parameters to PyArg_ParseTupleAndKeywords. > > They're optional-only for now (unlike in pure Python) but that's all > I needed. ?The syntax can easily be relaxed if we want to support > required keyword-only arguments for extension types in the future. > > files: > ?Doc/c-api/arg.rst ? ? ? ? | ? 9 +++ > ?Lib/test/test_getargs2.py | ?74 ++++++++++++++++++++++++++- > ?Modules/_testcapimodule.c | ?20 ++++++- > ?Python/getargs.c ? ? ? ? ?| ?34 ++++++++++++- > ?4 files changed, 134 insertions(+), 3 deletions(-) Forgot about Misc/NEWS? -- Regards, Benjamin From p.f.moore at gmail.com Tue Mar 20 21:15:49 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Mar 2012 20:15:49 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68DBEF.9070105@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68DBEF.9070105@gmail.com> Message-ID: On 20 March 2012 19:35, Lindberg, Van wrote: > I would like to know if you would object to user lib installs matching > the system install. I.e., would it cause problems with you if it were > just 'lib' everywhere, with no 'lib/python{version}'? It sounded like > adding the version directory was the issue. User lib installs don't bother me as I don't use them. But yes, it's the version directory that bothers me. So if you're proposing simply making the user lib install match the system install, both being lust "lib", then that's fine. I was somewhat confused about what you were proposing, that's all. Paul. From p.f.moore at gmail.com Tue Mar 20 21:31:00 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Mar 2012 20:31:00 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68D90F.1080504@gmail.com> References: <4F68607B.7060307@gmail.com> <4F68D90F.1080504@gmail.com> Message-ID: On 20 March 2012 19:22, VanL wrote: > There are a number of casual users that probably only have one version > installed, but every python user/dev on windows that I know has one python > that they consider to be "python," and everything else needs to be launched > with a suffix (e.g., python26.exe). This is usually put earlier on the PATH > so that it gets picked up first. For example, right now I have 2.6, 2.7, > 3.2, jython, and pypy all installed, and I have "python" pointing to 2.7. But no Python I am aware of *has* a suffixed version (python26.exe). Renaming/copying is (in my view) a far more invasive change than simply modifying PATH (and it doesn't help the whole nose/regetron situation either). Serious question: Given a brand new PC, if you were installing Python 2.7, 3.2, 3.3a1, jython, and pypy, what would you do (beyond simply running 5 installers) to get your environment set up the way you want? For me, I'd 1. Install the Python launcher (only until 3.3 includes it) 2. Edit py.ini to tailor py.exe to my preferred defaults for Python and Python3. 3. Install my powershell module which allows me to switch which Python is on PATH Done. (That doesn't cater for pypy or jython, as I don't use them. But I'd probably use a couple of aliases for the rare uses I'd make of them) >> ?I'd be curious as to how much PEP 397's py.exe would have helped >> ?those people. But yes, it's an issue. Although someone at some point >> ?will have to introduce those beginners to the question of Python 2 vs >> ?Python 3, and PATH pain will hit them then, anyway. > > I would imagine that it would help steps 1 and 2, but 3 and 4 would be > problematic (how can you pip install something using py?) unless you were in > a virtualenv, and then (unless py respected the virtualenv) the whole thing > would be problematic, because there wouldn't be one clear way to do it. There isn't one clear way right now. And adding one particular version to PATH only helps if you only *have* one version. My current preference is as follows: 1. If you only ever have one Python on your machine, add it (and its scripts dir) to PATH and be done with it. Unfortunately, we're in the throes of the Python 2-3 transition, and not many people can manage with the one-Python restriction (I certainly can't). Also the Python installer can't detect if that's what you want. 2. Otherwise, use virtualenvs for anything that isn't being packaged up as a standalone environment. Activate as needed. 3. To access your system python(s) use py.exe with a version flag if needed. Never (or nearly never) install packages in the system Python. 4. To run scripts, use #! lines and the py.exe association (and set PATHEXT if you want) to associate the precise Python you want with the script. I have to say, I've recently discovered virtualenv, so the above is the opinion of a newly-converted zealot - so take with a pinch of salt :-) Paul. From van.lindberg at gmail.com Tue Mar 20 22:40:08 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 16:40:08 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68DBEF.9070105@gmail.com> Message-ID: On 3/20/2012 3:15 PM, Paul Moore wrote: > On 20 March 2012 19:35, Lindberg, Van wrote: >> I would like to know if you would object to user lib installs matching >> the system install. I.e., would it cause problems with you if it were >> just 'lib' everywhere, with no 'lib/python{version}'? It sounded like >> adding the version directory was the issue. > > User lib installs don't bother me as I don't use them. But yes, it's > the version directory that bothers me. > > So if you're proposing simply making the user lib install match the > system install, both being lust "lib", then that's fine. I was > somewhat confused about what you were proposing, that's all. I was originally going to make it match posix-user installs, but just plain posix (no version directory) is just fine with me too. From v+python at g.nevcal.com Tue Mar 20 22:33:33 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 20 Mar 2012 14:33:33 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: <4F68F7AD.5060502@g.nevcal.com> On 3/20/2012 11:50 AM, Merlijn van Deen wrote: > As this is being considered an 'incompatible change' on the bug > tracker item [1] in any case, I'd like to mention that this might also > be a convenient moment to re-think the default install location. After > all, software is supposed to be installed in %programfiles% on > windows, not in c:\ . > > I asked a question about this on IRC, to which the response was that > there were two main reasons to install python in c:\pythonxy: > > 1 - issues due to spaces ('Program Files') or non-ascii characters in > the path ('Fi?iere Program' on a Romanian windows). These issues are > supposed to be fixed by now (?). > 2 - issues due to permissions - installing python / packages in > %programfiles% may require administrator rights. I also wondered about %programfiles%, and had heard of issue #1, and would hope that it is not a real issue in modern times, but haven't attempted to test to determine otherwise. However, the in the first quoted paragraph there is an incorrect statement... the last sentence is simply not true. While software that is installed "for everyone" on the computer is supposed to be installed in %programfiles%, software that is installed for "user only" need not be, and in fact, it is recommended (at least by installer software I've used) that the alternate path is (XP) C:\Documents and Settings\ \Local Settings\Application Data or (7) C:\Users\ \AppData\Local (I think, I haven't found certain documentation about this). Or is it even possible to install something for "user only" anymore? I haven't been involved with installers lately (have been doing portable apps, no install needed). Certainly the "program files (x86)" business adds an extra wrinkle to it, somehow, on 64 bit machines, and I'm not hitting the right sites on my Google searches to discover anything about that, so that's why I'm wondering if it has been deprecated. Speaking of which, it would be nice to have "Portable Python" be part of the standard reportoire of packages available. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Van.Lindberg at haynesboone.com Tue Mar 20 22:54:16 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Tue, 20 Mar 2012 21:54:16 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F68D90F.1080504@gmail.com> Message-ID: <4F68FC88.7060408@gmail.com> On 3/20/2012 3:31 PM, Paul Moore wrote: > Serious question: Given a brand new PC, if you were installing Python > 2.7, 3.2, 3.3a1, jython, and pypy, what would you do (beyond simply > running 5 installers) to get your environment set up the way you want? I install each python in its own directory: C:/lib/python/2.7 C:/lib/python/3.2 C:/lib/python/3.3 C:/lib/jython C:/lib/pypy Jython and Pypy get their own directories because they can have different version compatibilities. I then edit my distutils.command.install and patch pip/virtualenv so that all my directories are 'bin'/'lib'/'include'. I have never used the py.exe runner, but I then choose whichever Python is my default (right now 2.7, but hoping that I will be able to switch during the 3.3 timeframe) and that gets put on the PATH, along with its 'bin' directory. The other root dirs/bin directories get put on the PATH after the default Python. I don't remember whether I did it or whether it is installed that way, but I have a python2.6.exe and an pythonw.2.6.exe, etc, and all the individual installers include both a pip and a pip-2.6 version (or whatever that install has). I honestly don't remember - I would love to have someone else check. With this setup, I get my default choice anytime I type "python" and a specific interpreter version when I specify it. Same with installers, etc. I then install virtualenv and virtualenvwrapper-powershell and do all of my development out of virtualenvs. Occasionally I will install something to the system python if it is a pain to compile and I am installing a binary version from somewhere, but I generally try to keep the system python(s) clean.CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From mhammond at skippinet.com.au Tue Mar 20 22:49:42 2012 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 21 Mar 2012 08:49:42 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F688F48.9010101@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> Message-ID: <4F68FB76.7010303@skippinet.com.au> On 21/03/2012 1:08 AM, Lindberg, Van wrote: > On 3/20/2012 5:48 AM, Mark Hammond wrote: >> While I'm still unclear on the actual benefits of this, Martin's >> approach strikes a reasonable compromise so I withdraw my objections. > > > Ok. I was out of town and so could not respond to most of the latest > discussion. > > A question for you Mark, Paul, (and anyone else): ?ric correctly points > out that there are actually two distinct changes proposed here: > > 1. Moving the Python binary > 2. Changing from "Scripts" to "bin" > > So far, the primary resistance seems to be to item #1 - moving the > python binary. There have been a few people who have noted that #2 will > require some code to change (i.e. Paul), but I don't see lots of resistance. > > Am I reading you correctly? Well - as Paul implies, there are 2 distinct changes being proposed, but in 2 different environments. For an installed Python: If it has to move, it may as well move to somewhere consistent with other platforms. IOW, moving to "bin" seems preferable to moving to Scripts. My initial objection was to moving it at all in an installed Python. For a virtual env, we are talking about moving it *from* Scripts to bin, which may cause some people different problems. However, that isn't the concern I was expressing and I'd hate to see virtual envs remain inconsistent with an installed Python after this effort. So I'm assuming that: * The executable (and DLL) are moved to a "bin" directory in an installed Python. * distutils etc will change to install all "scripts" (or executables generated from scripts) into that same directory. IOW, "Scripts" would die. * A virtual-env would have an almost identical layout to an installed Python. Cheers, Mark From van.lindberg at gmail.com Tue Mar 20 23:00:07 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 17:00:07 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68FB76.7010303@skippinet.com.au> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> Message-ID: <4F68FDE7.40505@gmail.com> On 3/20/2012 4:49 PM, Mark Hammond wrote: > > So I'm assuming that: > * The executable (and DLL) are moved to a "bin" directory in an > installed Python. > * distutils etc will change to install all "scripts" (or executables > generated from scripts) into that same directory. IOW, "Scripts" > would die. > * A virtual-env would have an almost identical layout to an installed > Python. Yes. I would make your point #3 stronger - I would like a virtualenv to have an identical layout to an installed python, at least with reference to the names of directories and the location of binaries. The base directory would be the only difference. Thanks, Van From p.f.moore at gmail.com Tue Mar 20 23:07:06 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Mar 2012 22:07:06 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F68FDE7.40505@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> Message-ID: On 20 March 2012 22:00, VanL wrote: > On 3/20/2012 4:49 PM, Mark Hammond wrote: >> >> So I'm assuming that: >> * The executable (and DLL) are moved to a "bin" directory in an installed >> Python. >> * distutils etc will change to install all "scripts" (or executables >> generated from scripts) into that same directory. ?IOW, "Scripts" would die. It's worth remembering ?ric's point - distutils is frozen and changes are in theory not allowed. This part of the proposal is not possible without an exception to that ruling. Personally, I don't see how making this change could be a problem, but I'm definitely not an expert. If distutils doesn't change, bdist_wininst installers built using distutils rather than packaging will do the wrong thing with regard to this change. End users won't be able to tell how an installer has been built. Paul. From skippy.hammond at gmail.com Tue Mar 20 23:09:38 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Wed, 21 Mar 2012 09:09:38 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: <4F690022.9080002@gmail.com> On 21/03/2012 5:50 AM, Merlijn van Deen wrote: > On 13 March 2012 20:43, VanL wrote: >> Following up on conversations at PyCon, I want to bring up one of my >> personal hobby horses for change in 3.3: Fix install layout on Windows, with >> a side order of making the PATH work better. > > As this is being considered an 'incompatible change' on the bug > tracker item [1] in any case, I'd like to mention that this might also > be a convenient moment to re-think the default install location. After > all, software is supposed to be installed in %programfiles% on > windows, not in c:\ . > > I asked a question about this on IRC, to which the response was that > there were two main reasons to install python in c:\pythonxy: > > 1 - issues due to spaces ('Program Files') or non-ascii characters in > the path ('Fi?iere Program' on a Romanian windows). These issues are > supposed to be fixed by now (?). > 2 - issues due to permissions - installing python / packages in > %programfiles% may require administrator rights. Apart from personal preference (ie, I prefer the status quo here), the second issue is a bit of a killer. Even an administrator can not write to Program Files unless they are using an "elevated" process (ie, explicitly use "Run as Administrator" and confirm the prompt. This means that any installer wanting to write .py files into the Python install must be elevated, and any Python process wanting to generate .pyc files must also be elevated. So even if an installer does arrange elevation, unless that installer also compiles all .pyc and .pyo files at install time, Python would fail to generate the .pyc files on first use. While Python will probably fail silently and still continue to work, it will have a significant performance impact. Mark From g.brandl at gmx.net Tue Mar 20 23:38:53 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 20 Mar 2012 23:38:53 +0100 Subject: [Python-Dev] Playing with a new theme for the docs Message-ID: Hi all, recently I've grown a bit tired of seeing our default Sphinx theme, especially as so many other projects use it. I decided to play around with something "clean" this time, and this is the result: http://www.python.org/~gbrandl/build/html/ The corresponding sandbox repo is at http://hg.python.org/sandbox/doc-theme/#doc-theme Let me know what you think, or play around and send some improvements. (The collapsible sidebar is not adapted to it yet, but will definitely be integrated before I consider applying a new theme to the real docs.) Thanks, Georg From guido at python.org Tue Mar 20 23:45:12 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Mar 2012 15:45:12 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: Nice and clean, but looks too similar to newer Google properties... Also I see that (like Google) you're falling for the fallacy of using less contrast. From an accessibility perspective that's questionable -- and I don't mean the legally blind, just people like myself whose eyes are getting a bit older. This also means I don't particularly like adding background color (no matter how light) to text samples. --Guido On Tue, Mar 20, 2012 at 3:38 PM, Georg Brandl wrote: > Hi all, > > recently I've grown a bit tired of seeing our default Sphinx theme, > especially as so many other projects use it. ?I decided to play around > with something "clean" this time, and this is the result: > > ?http://www.python.org/~gbrandl/build/html/ > > The corresponding sandbox repo is at > > ?http://hg.python.org/sandbox/doc-theme/#doc-theme > > Let me know what you think, or play around and send some improvements. > (The collapsible sidebar is not adapted to it yet, but will definitely > be integrated before I consider applying a new theme to the real docs.) -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Tue Mar 20 23:45:43 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 20 Mar 2012 18:45:43 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F690022.9080002@gmail.com> References: <4F690022.9080002@gmail.com> Message-ID: <20120320224544.903B0250075@webabinitio.net> On Wed, 21 Mar 2012 09:09:38 +1100, Mark Hammond wrote: > On 21/03/2012 5:50 AM, Merlijn van Deen wrote: > > I asked a question about this on IRC, to which the response was that > > there were two main reasons to install python in c:\pythonxy: > > > > 1 - issues due to spaces ('Program Files') or non-ascii characters in > > the path ('Fi??iere Program' on a Romanian windows). These issues are > > supposed to be fixed by now (?). > > 2 - issues due to permissions - installing python / packages in > > %programfiles% may require administrator rights. > > Apart from personal preference (ie, I prefer the status quo here), the > second issue is a bit of a killer. Even an administrator can not write > to Program Files unless they are using an "elevated" process (ie, > explicitly use "Run as Administrator" and confirm the prompt. > > This means that any installer wanting to write .py files into the Python > install must be elevated, and any Python process wanting to generate > .pyc files must also be elevated. So even if an installer does arrange > elevation, unless that installer also compiles all .pyc and .pyo files > at install time, Python would fail to generate the .pyc files on first > use. While Python will probably fail silently and still continue to > work, it will have a significant performance impact. So windows requires admin privileges to install to Program Files, but not to install to '/'? How novel. (You can perhaps tell that I'm not a windoows user). My understanding, though, is that Python does make a distinction between a system install of Python and a per-user one, so I don't think your objection really applies. That said, there is an open bug in the tracker about the insecurity of a system install of python (exactly that the files are writable by anyone). So that would have to be solved first. I'd say this is definitely a separate issue from Van's discussion, and the *only* reason one might want to tie them together at all is "well, we're changing the directory layout anyway". --David From jxo6948 at rit.edu Tue Mar 20 23:55:46 2012 From: jxo6948 at rit.edu (John O'Connor) Date: Tue, 20 Mar 2012 18:55:46 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: On Tue, Mar 20, 2012 at 6:38 PM, Georg Brandl wrote: > recently I've grown a bit tired of seeing our default Sphinx theme, > especially as so many other projects use it. I think regardless of the chosen style, giving the Python 3 docs a different look and feel also has a psychological benefit that might further encourage users to consider moving to Python 3. It could be a bit of a wake-up call. From rdmurray at bitdance.com Wed Mar 21 00:17:40 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 20 Mar 2012 19:17:40 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: <20120320231741.E7E93250075@webabinitio.net> On Tue, 20 Mar 2012 23:38:53 +0100, Georg Brandl wrote: > Hi all, > > recently I've grown a bit tired of seeing our default Sphinx theme, > especially as so many other projects use it. I decided to play around > with something "clean" this time, and this is the result: > > http://www.python.org/~gbrandl/build/html/ The font looks better in my browser, but otherwise I prefer the current style. The biggest thing I don't like about the new style is the fact that the content is not set off from the chrome by shading. Having it shaded makes it easier for my eye to ignore it and just focus on the content. Hey, maybe you could make the sidebar only appear if the browser supports javascript? Then I'd never have to see it, and that I would consider "clean" :) Thanks for working on improving things. --David From skippy.hammond at gmail.com Wed Mar 21 00:25:34 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Wed, 21 Mar 2012 10:25:34 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <20120320224544.903B0250075@webabinitio.net> References: <4F690022.9080002@gmail.com> <20120320224544.903B0250075@webabinitio.net> Message-ID: <4F6911EE.1030400@gmail.com> On 21/03/2012 9:45 AM, R. David Murray wrote: > On Wed, 21 Mar 2012 09:09:38 +1100, Mark Hammond wrote: >> On 21/03/2012 5:50 AM, Merlijn van Deen wrote: >>> I asked a question about this on IRC, to which the response was that >>> there were two main reasons to install python in c:\pythonxy: >>> >>> 1 - issues due to spaces ('Program Files') or non-ascii characters in >>> the path ('Fi??iere Program' on a Romanian windows). These issues are >>> supposed to be fixed by now (?). >>> 2 - issues due to permissions - installing python / packages in >>> %programfiles% may require administrator rights. >> >> Apart from personal preference (ie, I prefer the status quo here), the >> second issue is a bit of a killer. Even an administrator can not write >> to Program Files unless they are using an "elevated" process (ie, >> explicitly use "Run as Administrator" and confirm the prompt. >> >> This means that any installer wanting to write .py files into the Python >> install must be elevated, and any Python process wanting to generate >> .pyc files must also be elevated. So even if an installer does arrange >> elevation, unless that installer also compiles all .pyc and .pyo files >> at install time, Python would fail to generate the .pyc files on first >> use. While Python will probably fail silently and still continue to >> work, it will have a significant performance impact. > > So windows requires admin privileges to install to Program Files, but > not to install to '/'? How novel. (You can perhaps tell that I'm > not a windoows user). My understanding, though, is that Python > does make a distinction between a system install of Python and > a per-user one, so I don't think your objection really applies. I think it does. Consider I've installed Python as a "system install". Now I want to install some other package - ideally that installer will request elevation - all well and good - the .py files are installed. However, next time I want to run Python, it will fail to generate the .pyc files - even though I'm an administrator. I would need to explicitly tell Python to execute "as administrator" (or run it from an already elevated command-prompt) to have things work as expected. Thus, the "usual" case would be that Python is unable to update any files in its install directory. If Python installed for a single user didn't install into Program Files (which it probably couldn't do without an administrator providing credentials anyway) then it wouldn't be a problem - but then we have multiple possible default install locations, which sounds like more trouble than it is worth... > That said, there is an open bug in the tracker about the insecurity > of a system install of python (exactly that the files are writable > by anyone). So that would have to be solved first. I'd say this > is definitely a separate issue from Van's discussion, and the *only* > reason one might want to tie them together at all is "well, we're > changing the directory layout anyway". Agreed. Mark From v+python at g.nevcal.com Wed Mar 21 00:35:32 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 20 Mar 2012 16:35:32 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F6911EE.1030400@gmail.com> References: <4F690022.9080002@gmail.com> <20120320224544.903B0250075@webabinitio.net> <4F6911EE.1030400@gmail.com> Message-ID: <4F691444.9040607@g.nevcal.com> On 3/20/2012 4:25 PM, Mark Hammond wrote: > I think it does. Consider I've installed Python as a "system > install". Now I want to install some other package - ideally that > installer will request elevation - all well and good - the .py files > are installed. However, next time I want to run Python, it will fail > to generate the .pyc files - even though I'm an administrator. I > would need to explicitly tell Python to execute "as administrator" (or > run it from an already elevated command-prompt) to have things work as > expected. Thus, the "usual" case would be that Python is unable to > update any files in its install directory. > > If Python installed for a single user didn't install into Program > Files (which it probably couldn't do without an administrator > providing credentials anyway) then it wouldn't be a problem - but then > we have multiple possible default install locations, which sounds like > more trouble than it is worth... > >> That said, there is an open bug in the tracker about the insecurity >> of a system install of python (exactly that the files are writable >> by anyone). So that would have to be solved first. I'd say this >> is definitely a separate issue from Van's discussion, and the *only* >> reason one might want to tie them together at all is "well, we're >> changing the directory layout anyway". Indeed, the single user "place" isn't a single place, unless you consider the per-user $APPDATA environment variable sufficient to determine it (or the Windows API that returns the initial boot up value of $APPDATA/ %APPDATA%, which is the preferred technique for code). But it does solve the security problem (stuff in APPDATA is accessible only to a single login by default). So that might be justification for putting it there, for single users. For multi-user installs, %PROGRAMFILES% is appropriate, but, like I've heard some Linux distributions do, *.pyc might have to be prebuilt and installed along with Python (or generated during install, instead of waiting for first use). -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Wed Mar 21 00:44:43 2012 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 20 Mar 2012 19:44:43 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/20/2012 06:45 PM, Guido van Rossum wrote: > Nice and clean, but looks too similar to newer Google properties... > Also I see that (like Google) you're falling for the fallacy of using > less contrast. From an accessibility perspective that's questionable > -- and I don't mean the legally blind, just people like myself whose > eyes are getting a bit older. This also means I don't particularly > like adding background color (no matter how light) to text samples. +1. Even make comments low-contrast defeats their purpose (italic works fine for that). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9pFmsACgkQ+gerLs4ltQ7fpwCeOY5p2HnqotHrWrN5vqsHfcsl 2EYAn3cnlemVO/RKavU3SC4w5b+q66S6 =Oryl -----END PGP SIGNATURE----- From solipsis at pitrou.net Wed Mar 21 01:37:17 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Mar 2012 01:37:17 +0100 Subject: [Python-Dev] Playing with a new theme for the docs References: Message-ID: <20120321013717.1126e14f@pitrou.net> Hi Georg, On Tue, 20 Mar 2012 23:38:53 +0100 Georg Brandl wrote: > Hi all, > > recently I've grown a bit tired of seeing our default Sphinx theme, > especially as so many other projects use it. I decided to play around > with something "clean" this time, and this is the result: > > http://www.python.org/~gbrandl/build/html/ Not enough colours, and/or not enough visual cues for page structure. cheers Antoine. From cs at zip.com.au Wed Mar 21 01:51:35 2012 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 21 Mar 2012 11:51:35 +1100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: <20120321005134.GA4058@cskk.homeip.net> On 20Mar2012 15:45, Guido van Rossum wrote: | Nice and clean, but looks too similar to newer Google properties... | Also I see that (like Google) you're falling for the fallacy of using | less contrast. From an accessibility perspective that's questionable | -- and I don't mean the legally blind, just people like myself whose | eyes are getting a bit older. This also means I don't particularly | like adding background color (no matter how light) to text samples. Conversely, I like the text samples slightly shaded; I find a bare rectangle on the perimeter of a DIV just a tad more like noise, whereas a very slightly shaded block makes it very clear to me. I know it is a PITA, but how hard is it to make a tiny tiny CSS control block somewhere so a user can tune the style in coarse ways (i.e. tweak the properties of the class for shaded blocks)? I think the font choice in the new style is better; cleaner, less noisy, like a sans serif font versus a serifed font. So much so that I thought the new style used annoyingly more whitespace, but putting them side by side shows the new style to be more compact. Win win! One thing that bothers me about both styles is the fixed width text versus proportional size difference. Let me say in advance that I'm viewing both in Firefox on a Mac. To take an example, in the argparse module the opening sentence says "The argparse module". For me the word "argparse" is distinctly shorter in vertical height, which is a bit jarring. (the difference is smaller in the new style.) Is there a way to specify fonts that keeps this height attribute the same? Example screen shots (just those three words): http://dl.dropbox.com/u/2607515/screenshots/argparse-new1.png http://dl.dropbox.com/u/2607515/screenshots/argparse-old1.png Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ That's just the sort of bloody stupid name they would choose. - Reginald Mitchell, designer of the Spitfire From raymond.hettinger at gmail.com Wed Mar 21 01:57:27 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 20 Mar 2012 17:57:27 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <20120321013717.1126e14f@pitrou.net> References: <20120321013717.1126e14f@pitrou.net> Message-ID: <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> On Mar 20, 2012, at 5:37 PM, Antoine Pitrou wrote: > Georg Brandl wrote: >> Hi all, >> >> recently I've grown a bit tired of seeing our default Sphinx theme, >> especially as so many other projects use it. I decided to play around >> with something "clean" this time, and this is the result: >> >> http://www.python.org/~gbrandl/build/html/ > > Not enough colours, and/or not enough visual cues for page structure. > > cheers > > Antoine. Like Antoine, I'm having a hard time navigating the page. For me, the current theme is *much* better. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Mar 21 02:39:41 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Mar 2012 21:39:41 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: On 3/20/2012 6:38 PM, Georg Brandl wrote: The current green on the front page is too heavy. Otherwise I prefer the old. I like the color on the index chart of the builtin-functions page. You un-bolded most (not all) of the entries and then are definitely too thin now. You unbolded the blue elsewhere and it is definitely harder for me to read. My eyesight does not correct to 20/20 and I have trouble reading many things, but the current docs work pretty well for me. -- Terry Jan Reedy From van.lindberg at gmail.com Wed Mar 21 02:40:58 2012 From: van.lindberg at gmail.com (VanL) Date: Tue, 20 Mar 2012 20:40:58 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> Message-ID: On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote: > It's worth remembering ?ric's point - distutils is frozen and changes > are in theory not allowed. This part of the proposal is not possible > without an exception to that ruling. Personally, I don't see how > making this change could be a problem, but I'm definitely not an > expert. > > If distutils doesn't change, bdist_wininst installers built using > distutils rather than packaging will do the wrong thing with regard to > this change. End users won't be able to tell how an installer has been > built. > > This is a good point. Who can make this call - Guido, or someone else? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Wed Mar 21 02:58:57 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 20 Mar 2012 21:58:57 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: <4F6935E1.2030309@nedbatchelder.com> On 3/20/2012 6:38 PM, Georg Brandl wrote: > Let me know what you think, or play around and send some improvements. > (The collapsible sidebar is not adapted to it yet, but will definitely > be integrated before I consider applying a new theme to the real docs.) Not to add to the chorus of tweakers, but if I could change just one thing about the current theme, it would be to remove full justification of the text. In text like ours with frequent long expressions, URLs, and the like, full justification is just an invitation to mangle the spacing of a paragraph. The paragraphs are also quite short and often interrupted by samples, lists, headings, and so on, losing the design advantage of a clean right edge anyway. Books, magazines, and newspapers look good with full justification, web pages do not. Can we switch to left-justified instead? --Ned. > Thanks, > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com > From merwok at netwok.org Wed Mar 21 04:41:47 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 20 Mar 2012 23:41:47 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> Message-ID: <4F694DFB.7050400@netwok.org> Hi, Le 20/03/2012 21:40, VanL a ?crit : > On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote: >> It's worth remembering ?ric's point - distutils is frozen and changes >> are in theory not allowed. This part of the proposal is not possible >> without an exception to that ruling. Personally, I don't see how >> making this change could be a problem, but I'm definitely not an >> expert. >> >> If distutils doesn't change, bdist_wininst installers built using >> distutils rather than packaging will do the wrong thing with regard to >> this change. End users won't be able to tell how an installer has been >> built. > > This is a good point. Who can make this call - Guido, or someone else? >From the top of my head the developers with the most experience about Windows deployment are Martin v. L?wis, Mark Hammond and Marc-Andr? Lemburg (not sure about the Windows part for MAL, but he maintains a library that extends distutils and has been broken in the past). I think their approval is required for this kind of huge change. The point of the distutils freeze (i.e. feature moratorium) is that we just can?t know what complicated things people are doing with undocumented internals, because distutils appeared unmaintained and under-documented for years and people had to work with and around it; since the start of the distutils2 project we can Just Say No? to improvements and features in distutils. ?I don?t see what could possibly go wrong? is a classic line in both horror movies and distutils development . Renaming Scripts to bin on Windows would have effects on some tools we know and surely on many tools we don?t know. We don?t want to see again people who use or extend distutils come with torches and pitchforks because internals were changed and we have to revert. So in my opinion, to decide to go ahead with the change we need strong +1s from the developers I named above and an endorsement by Tarek, or if he can?t participate in the discussion, Guido. As a footnote, distutils is already broken in 3.3. Now we give users or system administrators the possibility to edit the install schemes at will in sysconfig.cfg, but distutils hard-codes the old scheme. I tend to think it should be fixed, to make the distutils-packaging transition/cohabitation possible. Regards From greg.ewing at canterbury.ac.nz Wed Mar 21 06:03:26 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 Mar 2012 18:03:26 +1300 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <20120320224544.903B0250075@webabinitio.net> References: <4F690022.9080002@gmail.com> <20120320224544.903B0250075@webabinitio.net> Message-ID: <4F69611E.7090400@canterbury.ac.nz> R. David Murray wrote: > My understanding, though, is that Python > does make a distinction between a system install of Python and > a per-user one, so I don't think your objection really applies. Seems to me that for Python at least, the important distinction is not so much where the files are placed, but whether the registry entries are made machine-wide or user-local. -- Greg From g.brandl at gmx.net Wed Mar 21 06:57:34 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 21 Mar 2012 06:57:34 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: On 20.03.2012 23:45, Guido van Rossum wrote: > Nice and clean, but looks too similar to newer Google properties... > Also I see that (like Google) you're falling for the fallacy of using > less contrast. From an accessibility perspective that's questionable > -- and I don't mean the legally blind, just people like myself whose > eyes are getting a bit older. This also means I don't particularly > like adding background color (no matter how light) to text samples. Well, to be fair, the current theme also has a lot of shading, and the text in the sidebar is at lower contrast too. But I can see that the main text should remain at as high contrast as possible. Georg From g.brandl at gmx.net Wed Mar 21 06:58:21 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 21 Mar 2012 06:58:21 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <20120320231741.E7E93250075@webabinitio.net> References: <20120320231741.E7E93250075@webabinitio.net> Message-ID: On 21.03.2012 00:17, R. David Murray wrote: > On Tue, 20 Mar 2012 23:38:53 +0100, Georg Brandl wrote: >> Hi all, >> >> recently I've grown a bit tired of seeing our default Sphinx theme, >> especially as so many other projects use it. I decided to play around >> with something "clean" this time, and this is the result: >> >> http://www.python.org/~gbrandl/build/html/ > > The font looks better in my browser, but otherwise I prefer the current > style. The biggest thing I don't like about the new style is the fact > that the content is not set off from the chrome by shading. Having it > shaded makes it easier for my eye to ignore it and just focus on > the content. Not sure what "the unshaded chrome" is -- only the header bar, since the sidebar is shaded already? Georg From g.brandl at gmx.net Wed Mar 21 07:00:36 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 21 Mar 2012 07:00:36 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> Message-ID: On 21.03.2012 01:57, Raymond Hettinger wrote: > > On Mar 20, 2012, at 5:37 PM, Antoine Pitrou wrote: > >> Georg Brandl > wrote: >>> Hi all, >>> >>> recently I've grown a bit tired of seeing our default Sphinx theme, >>> especially as so many other projects use it. I decided to play around >>> with something "clean" this time, and this is the result: >>> >>> http://www.python.org/~gbrandl/build/html/ >> >> Not enough colours, and/or not enough visual cues for page structure. >> >> cheers >> >> Antoine. > > Like Antoine, I'm having a hard time navigating the page. > For me, the current theme is *much* better. OK, that seems to be the main point people make... let me see if I can come up with a better compromise. Georg From martin at v.loewis.de Wed Mar 21 08:02:15 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Wed, 21 Mar 2012 08:02:15 +0100 Subject: [Python-Dev] GSoC 2012: Python Core Participation? Message-ID: <20120321080215.Horde.F0tnBFNNcXdPaXz3hMjTO8A@webmail.df.eu> I'm wondering whether Python Core should participate in GSoC 2012 or not, as core contributors have shown little interest in acting as mentors in the past. If you are a core committer and volunteer as GSoC mentor for 2012, please let me know by Friday (March 23rd). Regards, Martin From dirkjan at ochtman.nl Wed Mar 21 09:25:59 2012 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 21 Mar 2012 09:25:59 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> Message-ID: On Wed, Mar 21, 2012 at 07:00, Georg Brandl wrote: > OK, that seems to be the main point people make... let me see if I can > come up with a better compromise. Would it be possible to limit the width of the page? On my 1920px monitor, the lines get awfully long, making them harder to read. Cheers, Dirkjan From anacrolix at gmail.com Wed Mar 21 10:31:41 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 21 Mar 2012 17:31:41 +0800 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> Message-ID: Turn your monitor portrait or make the window smaller :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tartley at tartley.com Wed Mar 21 10:33:13 2012 From: tartley at tartley.com (Jonathan Hartley) Date: Wed, 21 Mar 2012 09:33:13 +0000 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> Message-ID: <4F69A059.2030601@tartley.com> On 21/03/2012 08:25, Dirkjan Ochtman wrote: > On Wed, Mar 21, 2012 at 07:00, Georg Brandl wrote: >> OK, that seems to be the main point people make... let me see if I can >> come up with a better compromise. > Would it be possible to limit the width of the page? On my 1920px > monitor, the lines get awfully long, making them harder to read. I realise this is bikeshedding by now, but FWIW, please don't. If people want shorter lines, they can narrow their browser, without forcing that preference on the rest of us. > Cheers, > > Dirkjan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/tartley%40tartley.com > -- Jonathan Hartley tartley at tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley From phd at phdru.name Wed Mar 21 11:16:21 2012 From: phd at phdru.name (Oleg Broytman) Date: Wed, 21 Mar 2012 14:16:21 +0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F69A059.2030601@tartley.com> References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> <4F69A059.2030601@tartley.com> Message-ID: <20120321101621.GA5612@iskra.aviel.ru> On Wed, Mar 21, 2012 at 09:33:13AM +0000, Jonathan Hartley wrote: > On 21/03/2012 08:25, Dirkjan Ochtman wrote: > >On Wed, Mar 21, 2012 at 07:00, Georg Brandl wrote: > >>OK, that seems to be the main point people make... let me see if I can > >>come up with a better compromise. > >Would it be possible to limit the width of the page? On my 1920px > >monitor, the lines get awfully long, making them harder to read. > I realise this is bikeshedding by now, but FWIW, please don't. If > people want shorter lines, they can narrow their browser, without > forcing that preference on the rest of us. Seconded. My display is 1920x1200 but I use very large fonts and I'm satisfied with line lengths. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ben+python at benfinney.id.au Wed Mar 21 11:48:25 2012 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 21 Mar 2012 21:48:25 +1100 Subject: [Python-Dev] Playing with a new theme for the docs References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> Message-ID: <874ntikz5y.fsf@benfinney.id.au> Dirkjan Ochtman writes: > On Wed, Mar 21, 2012 at 07:00, Georg Brandl wrote: > > OK, that seems to be the main point people make... let me see if I > > can come up with a better compromise. > > Would it be possible to limit the width of the page? On my 1920px > monitor, the lines get awfully long, making them harder to read. ?1. Please, web designers, don't presume to know what width the viewer wants. We can change the window size if that's what we want. -- \ ?I hope some animal never bores a hole in my head and lays its | `\ eggs in my brain, because later you might think you're having a | _o__) good idea but it's just eggs hatching.? ?Jack Handey | Ben Finney From solipsis at pitrou.net Wed Mar 21 12:09:49 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Mar 2012 12:09:49 +0100 Subject: [Python-Dev] Playing with a new theme for the docs References: Message-ID: <20120321120949.31c790dc@pitrou.net> On Tue, 20 Mar 2012 21:39:41 -0400 Terry Reedy wrote: > On 3/20/2012 6:38 PM, Georg Brandl wrote: > > The current green on the front page is too heavy. Green? hmm... you mean blue, right? :) Antoine. From solipsis at pitrou.net Wed Mar 21 12:10:19 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Mar 2012 12:10:19 +0100 Subject: [Python-Dev] Playing with a new theme for the docs References: <4F6935E1.2030309@nedbatchelder.com> Message-ID: <20120321121019.6e95e192@pitrou.net> On Tue, 20 Mar 2012 21:58:57 -0400 Ned Batchelder wrote: > On 3/20/2012 6:38 PM, Georg Brandl wrote: > > Let me know what you think, or play around and send some improvements. > > (The collapsible sidebar is not adapted to it yet, but will definitely > > be integrated before I consider applying a new theme to the real docs.) > Not to add to the chorus of tweakers, but if I could change just one > thing about the current theme, it would be to remove full justification > of the text. Ow, I hate non-justified text myself :( Bikeshedding Antoine. From chris at simplistix.co.uk Wed Mar 21 11:55:20 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 21 Mar 2012 10:55:20 +0000 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F69A059.2030601@tartley.com> References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> <4F69A059.2030601@tartley.com> Message-ID: <4F69B398.10708@simplistix.co.uk> On 21/03/2012 09:33, Jonathan Hartley wrote: > On 21/03/2012 08:25, Dirkjan Ochtman wrote: >> On Wed, Mar 21, 2012 at 07:00, Georg Brandl wrote: >>> OK, that seems to be the main point people make... let me see if I can >>> come up with a better compromise. >> Would it be possible to limit the width of the page? On my 1920px >> monitor, the lines get awfully long, making them harder to read. > I realise this is bikeshedding by now, but FWIW, please don't. If people > want shorter lines, they can narrow their browser, without forcing that > preference on the rest of us. + sys.maxint Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ned at nedbatchelder.com Wed Mar 21 13:38:50 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 21 Mar 2012 08:38:50 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <20120321101621.GA5612@iskra.aviel.ru> References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> <4F69A059.2030601@tartley.com> <20120321101621.GA5612@iskra.aviel.ru> Message-ID: <4F69CBDA.8010405@nedbatchelder.com> On 3/21/2012 6:16 AM, Oleg Broytman wrote: > On Wed, Mar 21, 2012 at 09:33:13AM +0000, Jonathan Hartley wrote: >> On 21/03/2012 08:25, Dirkjan Ochtman wrote: >>> On Wed, Mar 21, 2012 at 07:00, Georg Brandl wrote: >>>> OK, that seems to be the main point people make... let me see if I can >>>> come up with a better compromise. >>> Would it be possible to limit the width of the page? On my 1920px >>> monitor, the lines get awfully long, making them harder to read. >> I realise this is bikeshedding by now, but FWIW, please don't. If >> people want shorter lines, they can narrow their browser, without >> forcing that preference on the rest of us. > Seconded. My display is 1920x1200 but I use very large fonts and I'm > satisfied with line lengths. The best thing to do is to set a max-width in ems, say 50em. This leaves the text at a reasonable width, but adapts naturally for people with larger fonts. --Ned. > > Oleg. From rdmurray at bitdance.com Wed Mar 21 14:03:48 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 21 Mar 2012 09:03:48 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <20120320231741.E7E93250075@webabinitio.net> Message-ID: <20120321130348.B00992500E3@webabinitio.net> On Wed, 21 Mar 2012 06:58:21 +0100, Georg Brandl wrote: > On 21.03.2012 00:17, R. David Murray wrote: > > On Tue, 20 Mar 2012 23:38:53 +0100, Georg Brandl wrote: > >> Hi all, > >> > >> recently I've grown a bit tired of seeing our default Sphinx theme, > >> especially as so many other projects use it. I decided to play around > >> with something "clean" this time, and this is the result: > >> > >> http://www.python.org/~gbrandl/build/html/ > > > > The font looks better in my browser, but otherwise I prefer the current > > style. The biggest thing I don't like about the new style is the fact > > that the content is not set off from the chrome by shading. Having it > > shaded makes it easier for my eye to ignore it and just focus on > > the content. > > Not sure what "the unshaded chrome" is -- only the header bar, since the > sidebar is shaded already? Header bar and footer. But I also like the fact that the current site shades the sidebar all the way down (and darker, though obviously we have contrast issues from some folks that don't bother me). Otherwise that whitespace on the left just looks...wrong. But that last is considerably less important. --David From storchaka at gmail.com Wed Mar 21 14:48:00 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 21 Mar 2012 15:48:00 +0200 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F69CBDA.8010405@nedbatchelder.com> References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> <4F69A059.2030601@tartley.com> <20120321101621.GA5612@iskra.aviel.ru> <4F69CBDA.8010405@nedbatchelder.com> Message-ID: 21.03.12 14:38, Ned Batchelder ???????(??): > The best thing to do is to set a max-width in ems, say 50em. This leaves > the text at a reasonable width, but adapts naturally for people with > larger fonts. It's good for books, magazines, and newspapers, but not for technical site. ;) From kristjan at ccpgames.com Wed Mar 21 14:35:50 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 21 Mar 2012 13:35:50 +0000 Subject: [Python-Dev] PEP 405 (built-in virtualenv) status In-Reply-To: <4F678690.3000600@oddbird.net> References: <4F626278.7030701@oddbird.net> <4F626712.3030906@gmail.com> <4F62692E.8040203@oddbird.net> <4F678690.3000600@oddbird.net> Message-ID: > -----Original Message----- > From: Carl Meyer [mailto:carl at oddbird.net] > Sent: 19. mars 2012 19:19 > To: Kristj?n Valur J?nsson > Cc: Python-Dev (python-dev at python.org) > Subject: Re: [Python-Dev] PEP 405 (built-in virtualenv) status > > Hello Kristj?n, > I think there's one important (albeit odd and magical) bit of Python's current > behavior that you are missing in your blog post. All of the initial sys.path > directories are constructed relative to sys.prefix and sys.exec_prefix, and > those values in turn are determined (if PYTHONHOME is not set), by walking > up the filesystem tree from the location of the Python binary, looking for the > existence of a file at the relative path "lib/pythonX.X/os.py" (or "Lib/os.py" > on Windows). Python takes the existence of this file to mean that it's found > the standard library, and sets sys.prefix accordingly. Thus, you can achieve > reliable full isolation from any installed Python, with no need for > environment variables, simply by placing a file (it can even be empty) at that > relative location from the location of your Python binary. You will still get > some default paths added on sys.path, but they will all be relative to your > Python binary and thus presumably under your control; nothing from any > other location will be on sys.path. I doubt you will find this solution > satisfyingly elegant, but you might nonetheless find it practically useful. > Right. Thanks for explaining this. Although, it would appear that Python also has a mechanism for detecting that it is being run from a build environment and ignore PYTHONHOME in that case too. > > Beyond that possible tweak, while I certainly wouldn't oppose any effort to > clean up / document / make-optional Python's startup sys.path-setting > behavior, I think it's mostly orthogonal to PEP 405, and I don't think it would > be helpful to expand the scope of PEP 405 to include that effort. Well, it sounds as this pep can definitely be used as the basis for work to completely customize the startup behaviour. In my case, it would be desirable to be able to completely ignore any PYTHONHOME environment variable (and any others). I'd also like to be able to manually set up the sys.path. Perhaps if we can set things up that one key (ignore_env) will cause the environment variables to be ignored, and then, an empty home key will set the sys.path to point to the directory of the .cfg file. Presumably, this would then cause a site.py found at that place to be executed and one could code whatever extra logic one wants into that file. Possibly a "site" key in the .cfg file would achieve the same goal, allowing the user to call this setup file whatever he wants. With something like this in place, the built in behaviour of python.exe to realize that it is running from a "build" environment and in that case ignore PYTHONPATH and set a special sys.path, could all be removed from being hardcoded into being coded into some buildsite.py in the cpython root folder. Kristj?n From storchaka at gmail.com Wed Mar 21 14:44:24 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 21 Mar 2012 15:44:24 +0200 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6935E1.2030309@nedbatchelder.com> References: <4F6935E1.2030309@nedbatchelder.com> Message-ID: 21.03.12 03:58, Ned Batchelder ???????(??): > Books, magazines, and newspapers look good with full justification, web > pages do not. Can we switch to left-justified instead? You can add line p {text-align: left !important} to your browser custom stylesheet. If you are using Firefox or Chrome (Chromium), for them there are extensions (Stylish) that allow to apply the style to the particular site. From ned at nedbatchelder.com Wed Mar 21 15:18:31 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 21 Mar 2012 10:18:31 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> Message-ID: <4F69E337.7010501@nedbatchelder.com> On 3/21/2012 9:44 AM, Serhiy Storchaka wrote: > 21.03.12 03:58, Ned Batchelder ???????(??): >> Books, magazines, and newspapers look good with full justification, web >> pages do not. Can we switch to left-justified instead? > > You can add line > > p {text-align: left !important} > > to your browser custom stylesheet. > > If you are using Firefox or Chrome (Chromium), for them there are > extensions (Stylish) that allow to apply the style to the particular > site. > Any of the tweaks people are suggesting could be applied individually using this technique. We could just as easily choose to make the site left-justified, and let the full-justification fans use custom stylesheets to get it. The challenge for the maintainer of the docs site is to choose a good design that most people will see. We're bound to disagree on what that design should be, and I suggest that probably none of us are designer enough to come up with the best one. Perhaps we could find an interested designer to help? --Ned. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com From Van.Lindberg at haynesboone.com Wed Mar 21 15:22:04 2012 From: Van.Lindberg at haynesboone.com (Lindberg, Van) Date: Wed, 21 Mar 2012 14:22:04 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F694DFB.7050400@netwok.org> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> Message-ID: <4F69E40C.6050901@gmail.com> Mark, MAL, Martin, Tarek, Could you comment on this? This is in the context of changing the name of the 'Scripts' directory on windows to 'bin'. ?ric brings up the point (explained more below) that if we make this change, packages made/installed the new packaging infrastructure and those made/installed with bdist_winist and the old (frozen) distutils will be inconsistent. The reason why is that the old distutils has a hard-coded dict in distutils.command.install that would point to the old locations. If we were to make this change in sysconfig.cfg, we would probably want to make a corresponding change in the INSTALL_SCHEMES dict in distutils.command.install. More context: On 3/20/2012 10:41 PM, ?ric Araujo wrote: > Le 20/03/2012 21:40, VanL a ?crit : >> On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote: >>> It's worth remembering ?ric's point - distutils is frozen and changes >>> are in theory not allowed. This part of the proposal is not possible >>> without an exception to that ruling. Personally, I don't see how >>> making this change could be a problem, but I'm definitely not an >>> expert. >>> >>> If distutils doesn't change, bdist_wininst installers built using >>> distutils rather than packaging will do the wrong thing with regard to >>> this change. End users won't be able to tell how an installer has been >>> built. Looking at the code in bdist_wininst, it loops over the keys in the INSTALL_SCHEMES dict to find the correct locations. If the hard-coded dict were changed, then the installer would 'just work' with the right location - and this matches my experience having made this sort of change. When I change the INSTALL_SCHEMES dict, things get installed according to the new scheme without difficulty using the standard tools. The only time when something is trouble is if it does its own install routine and hard-codes 'Scripts' as the name of the install directory - and I have only seen that in PyPM a couple versions ago. > From the top of my head the developers with the most experience about > Windows deployment are Martin v. L?wis, Mark Hammond and Marc-Andr? > Lemburg (not sure about the Windows part for MAL, but he maintains a > library that extends distutils and has been broken in the past). I > think their approval is required for this kind of huge change. Note the above - this is why I would like your comment. > The point of the distutils freeze (i.e. feature moratorium) is that we > just can?t know what complicated things people are doing with > undocumented internals, because distutils appeared unmaintained and > under-documented for years and people had to work with and around it; > since the start of the distutils2 project we can Just Say No? to > improvements and features in distutils. ?I don?t see what could > possibly go wrong? is a classic line in both horror movies and distutils > development . > > Renaming Scripts to bin on Windows would have effects on some tools we > know and surely on many tools we don?t know. We don?t want to see again > people who use or extend distutils come with torches and pitchforks > because internals were changed and we have to revert. So in my opinion, > to decide to go ahead with the change we need strong +1s from the > developers I named above and an endorsement by Tarek, or if he can?t > participate in the discussion, Guido. > > As a footnote, distutils is already broken in 3.3. Now we give users or > system administrators the possibility to edit the install schemes at > will in sysconfig.cfg, but distutils hard-codes the old scheme. I tend > to think it should be fixed, to make the distutils-packaging > transition/cohabitation possible. Any comment? Thanks, Van CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. From mal at egenix.com Wed Mar 21 15:54:21 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 21 Mar 2012 15:54:21 +0100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F69E40C.6050901@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> Message-ID: <4F69EB9D.1060701@egenix.com> Lindberg, Van wrote: > Mark, MAL, Martin, Tarek, > > Could you comment on this? > > This is in the context of changing the name of the 'Scripts' directory > on windows to 'bin'. ?ric brings up the point (explained more below) > that if we make this change, packages made/installed the new packaging > infrastructure and those made/installed with bdist_winist and the old > (frozen) distutils will be inconsistent. > > The reason why is that the old distutils has a hard-coded dict in > distutils.command.install that would point to the old locations. If we > were to make this change in sysconfig.cfg, we would probably want to > make a corresponding change in the INSTALL_SCHEMES dict in > distutils.command.install. I'm not sure I understand the point in making that change. Could you expand on the advantage of using "bin" instead of "Scripts" ? Note that distutils just provides defaults for these installation locations. All of them can be overridden using command line arguments to the install command. FWIW: I've dropped support for bdist_wininst in mxSetup.py since bdist_msi provides much better system integration. > More context: > > On 3/20/2012 10:41 PM, ?ric Araujo wrote: >> Le 20/03/2012 21:40, VanL a ?crit : >>> On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote: >>>> It's worth remembering ?ric's point - distutils is frozen and changes >>>> are in theory not allowed. This part of the proposal is not possible >>>> without an exception to that ruling. Personally, I don't see how >>>> making this change could be a problem, but I'm definitely not an >>>> expert. >>>> >>>> If distutils doesn't change, bdist_wininst installers built using >>>> distutils rather than packaging will do the wrong thing with regard to >>>> this change. End users won't be able to tell how an installer has been >>>> built. > > Looking at the code in bdist_wininst, it loops over the keys in the > INSTALL_SCHEMES dict to find the correct locations. If the hard-coded > dict were changed, then the installer would 'just work' with the right > location - and this matches my experience having made this sort of > change. When I change the INSTALL_SCHEMES dict, things get installed > according to the new scheme without difficulty using the standard tools. > The only time when something is trouble is if it does its own install > routine and hard-codes 'Scripts' as the name of the install directory - > and I have only seen that in PyPM a couple versions ago. > > >> From the top of my head the developers with the most experience about >> Windows deployment are Martin v. L?wis, Mark Hammond and Marc-Andr? >> Lemburg (not sure about the Windows part for MAL, but he maintains a >> library that extends distutils and has been broken in the past). I >> think their approval is required for this kind of huge change. > > Note the above - this is why I would like your comment. > > >> The point of the distutils freeze (i.e. feature moratorium) is that we >> just can?t know what complicated things people are doing with >> undocumented internals, because distutils appeared unmaintained and >> under-documented for years and people had to work with and around it; >> since the start of the distutils2 project we can Just Say No? to >> improvements and features in distutils. ?I don?t see what could >> possibly go wrong? is a classic line in both horror movies and distutils >> development . >> >> Renaming Scripts to bin on Windows would have effects on some tools we >> know and surely on many tools we don?t know. We don?t want to see again >> people who use or extend distutils come with torches and pitchforks >> because internals were changed and we have to revert. So in my opinion, >> to decide to go ahead with the change we need strong +1s from the >> developers I named above and an endorsement by Tarek, or if he can?t >> participate in the discussion, Guido. >> >> As a footnote, distutils is already broken in 3.3. Now we give users or >> system administrators the possibility to edit the install schemes at >> will in sysconfig.cfg, but distutils hard-codes the old scheme. I tend >> to think it should be fixed, to make the distutils-packaging >> transition/cohabitation possible. > > Any comment? > > Thanks, > Van > > CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by > U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any > U.S. tax advice contained in this communication (including any > attachments) was not intended or written to be used, and cannot be > used, for the purpose of (i) avoiding penalties under the Internal > Revenue Code or (ii) promoting, marketing or recommending to another > party any transaction or matter addressed herein. > > CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, > may be privileged and should be read or retained only by the intended > recipient. If you have received this transmission in error, please > immediately notify the sender and delete it from your system. > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 21 2012) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2012-04-03: Python Meeting Duesseldorf 13 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From lrekucki at gmail.com Wed Mar 21 16:06:11 2012 From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=) Date: Wed, 21 Mar 2012 16:06:11 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F69CBDA.8010405@nedbatchelder.com> References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> <4F69A059.2030601@tartley.com> <20120321101621.GA5612@iskra.aviel.ru> <4F69CBDA.8010405@nedbatchelder.com> Message-ID: On 21 March 2012 13:38, Ned Batchelder wrote: > On 3/21/2012 6:16 AM, Oleg Broytman wrote: >> >> On Wed, Mar 21, 2012 at 09:33:13AM +0000, Jonathan Hartley wrote: >>> >>> On 21/03/2012 08:25, Dirkjan Ochtman wrote: >>>> >>>> On Wed, Mar 21, 2012 at 07:00, Georg Brandl ? wrote: >>>>> >>>>> OK, that seems to be the main point people make... let me see if I can >>>>> come up with a better compromise. >>>> >>>> Would it be possible to limit the width of the page? On my 1920px >>>> monitor, the lines get awfully long, making them harder to read. >>> >>> I realise this is bikeshedding by now, but FWIW, please don't. If >>> people want shorter lines, they can narrow their browser, without >>> forcing that preference on the rest of us. >> >> ? ?Seconded. My display is 1920x1200 but I use very large fonts and I'm >> satisfied with line lengths. > > The best thing to do is to set a max-width in ems, say 50em. This leaves the > text at a reasonable width, but adapts naturally for people with larger > fonts. > > --Ned. FYI, the current paragraph font size on docs.python.org is 16px, while for http://www.python.org/~gbrandl/build/html/ it's 13px, so increasing that should help readability :) You can use @media queries to adjust it to screen resolution, which should solve the problem with long lines. -- ?ukasz Rekucki From storchaka at gmail.com Wed Mar 21 16:18:19 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 21 Mar 2012 17:18:19 +0200 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F69E337.7010501@nedbatchelder.com> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> Message-ID: 21.03.12 16:18, Ned Batchelder ???????(??): > We could just as easily choose to make the site > left-justified, and let the full-justification fans use custom > stylesheets to get it. I find justified text convenient and pleasant for the eyes. Many people hate left-aligned text. I think that the best would be the use of left-aligned text at the small width of the window (640px and less), when they become obvious drawbacks of justified text, and use justified text with a large width. > Perhaps we could find an interested designer to help? Isn't Georg Brandl a designer? The proposed design looks professional for me and is not worse than the design of large corporations (though there are some defects). The current design is also very good. Optimum is, I suppose, in the middle. From yselivanov.ml at gmail.com Wed Mar 21 16:19:31 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 21 Mar 2012 11:19:31 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> <4F69A059.2030601@tartley.com> <20120321101621.GA5612@iskra.aviel.ru> <4F69CBDA.8010405@nedbatchelder.com> Message-ID: <84AF9CD0-14EE-4BBA-8156-8CD79A0E9270@gmail.com> On 2012-03-21, at 11:06 AM, ?ukasz Rekucki wrote: > FYI, the current paragraph font size on docs.python.org is 16px, while > for http://www.python.org/~gbrandl/build/html/ it's 13px, so > increasing that should help readability :) You can use @media queries > to adjust it to screen resolution, which should solve the problem with > long lines. +1. It's much harder to read text in the new design. I would also make links a bit darker. - Yury From guido at python.org Wed Mar 21 17:00:55 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2012 09:00:55 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F69CBDA.8010405@nedbatchelder.com> References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> <4F69A059.2030601@tartley.com> <20120321101621.GA5612@iskra.aviel.ru> <4F69CBDA.8010405@nedbatchelder.com> Message-ID: On Mar 21, 2012 5:44 AM, "Ned Batchelder" wrote: > The best thing to do is to set a max-width in ems, say 50em. This leaves the text at a reasonable width, but adapts naturally for people with larger fonts. Please, no, not even this "improved" version of coddling. If you're formatting e.g. a newspaper or a book, by all means (though I still think the user should be given ultimate control -- and I don't mean editing the CSS using the browser's development tools :-). But when reading docs there are all sorts of reasons why I might want to stretch the window to maximum width and nothing's more frustrating than a website that forces clipping, folding or a horizontal scroll bar even when I make the window wide enough. And sometimes I just don't care that much about reading the text, but having more things visible at once (vertically) is worth it. (Can you see why I invented a whitespace-sensitive language? I have a whitespace-sensitive brain. :-) -- --Guido van Rossum (python.org/~guido) From phd at phdru.name Wed Mar 21 17:08:57 2012 From: phd at phdru.name (Oleg Broytman) Date: Wed, 21 Mar 2012 20:08:57 +0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: <20120321160857.GA14426@iskra.aviel.ru> On Tue, Mar 20, 2012 at 11:38:53PM +0100, Georg Brandl wrote: > recently I've grown a bit tired of seeing our default Sphinx theme, > especially as so many other projects use it. I decided to play around > with something "clean" this time, and this is the result: > > http://www.python.org/~gbrandl/build/html/ Looks very nice! A few notes, if you don't mind. 1. I'd prefer a little bit bigger fonts. 2. IWBN IMHO to extend the grayish background of the navigation bar at the left to the bottom of the page. White space below short boxes looks strange for me. 3. A lot of small adjacent code snippets with a different background make my eyes hurt. See for example the note number 5 at http://www.python.org/~gbrandl/build/html/library/stdtypes.html#sequence-types-str-bytes-bytearray-list-tuple-range I'd like inline code snippets to have the same background. Bold font and/or a different foreground color would be better, in my opinion. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From guido at python.org Wed Mar 21 17:11:54 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2012 09:11:54 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F69E337.7010501@nedbatchelder.com> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> Message-ID: On Wed, Mar 21, 2012 at 7:18 AM, Ned Batchelder wrote: > The challenge for the maintainer of the docs site is to choose a good design > that most people will see. ?We're bound to disagree on what that design > should be, and I suggest that probably none of us are designer enough to > come up with the best one. ?Perhaps we could find an interested designer to > help? I've come to the conclusion that "good design" is not so much a matter of finding the "best" of anything (font, spacing rules, colors, icons, artowork, etc.). Good design is highly subjective to fashion, and the people who are recognized to be the best designers are more often than not just those with a strong enough opinion to push their creative ideas through. Then other designers, who are not quite as good but still have a nose for the latest fashion, copy their ideas and for a while anything that hasn't been redesigned looks "old-fashioned". (Before you say something about limitations of old technology, note how often designers go back to older styles and manage to make them look fashionable again.) If you want something that attracts attention through controversy, get one of those initial thought leaders. If you want something that looks "current" today but which will probably be out of style next year, use one of the style-following designers. If you want something that is maximally useful, get a scientist with an ounce of style sense to do your design... Oh hey, Georg *is* a scientist! And he's got more than an ounce of style. So just let him do it and let's not try to micromanage things. (I had to speak up about the low contrast because Georg has young eyes and may not realize that this issue exists for older Pythonistas.) -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Wed Mar 21 18:04:23 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 21 Mar 2012 19:04:23 +0200 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: If I can get my five cents, I will tell about my impressions. I really liked the background of allocated blocks (such as notes and code snippets) has become less diverse (but still visible). The border around these blocks have become more accurate and more pleasant to emphasize blocks. It is very good that the sidebar is no longer confused look. And everything looks quite nice. But the font is a little bit small for my eyes (on the contrary current theme font a little bit big). This leads to too long (in characters) lines. Less obvious was the structure of the document (due to decrease the font size of the header and the removal of the dividing lines). I would like to that the background color of ".note tt" has become a little lighter and quieter. From storchaka at gmail.com Wed Mar 21 18:09:30 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 21 Mar 2012 19:09:30 +0200 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> <4F69A059.2030601@tartley.com> <20120321101621.GA5612@iskra.aviel.ru> <4F69CBDA.8010405@nedbatchelder.com> Message-ID: 21.03.12 18:00, Guido van Rossum ???????(??): > (Can you see why I invented a whitespace-sensitive language? I have a > whitespace-sensitive brain. :-) It should be added to favorite quotes. From tjreedy at udel.edu Wed Mar 21 18:31:40 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 21 Mar 2012 13:31:40 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <20120321120949.31c790dc@pitrou.net> References: <20120321120949.31c790dc@pitrou.net> Message-ID: On 3/21/2012 7:09 AM, Antoine Pitrou wrote: > On Tue, 20 Mar 2012 21:39:41 -0400 > Terry Reedy wrote: >> On 3/20/2012 6:38 PM, Georg Brandl wrote: >> >> The current green on the front page is too heavy. > > Green? > hmm... you mean blue, right? > :) Yeh, a muddy slightly greenish blue. I would prefer what I call a real blue, as in the logo, or the quoted text above on Thunderbird. -- Terry Jan Reedy From ned at nedbatchelder.com Wed Mar 21 19:40:04 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 21 Mar 2012 14:40:04 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: <4F6A2084.90508@nedbatchelder.com> On 3/21/2012 1:04 PM, Serhiy Storchaka wrote: > If I can get my five cents, I will tell about my impressions. I really > liked the background of allocated blocks (such as notes and code > snippets) has become less diverse (but still visible). The border > around these blocks have become more accurate and more pleasant to > emphasize blocks. It is very good that the sidebar is no longer > confused look. And everything looks quite nice. But the font is a > little bit small for my eyes (on the contrary current theme font a > little bit big). This leads to too long (in characters) lines. Less > obvious was the structure of the document (due to decrease the font > size of the header and the removal of the dividing lines). > You can use Ctrl-+ to increase the size of the text, and modern browsers remember that for the next time you visit the site. --Ned. > I would like to that the background color of ".note tt" has become a > little lighter and quieter. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com > From guido at python.org Wed Mar 21 19:46:36 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2012 11:46:36 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6A2084.90508@nedbatchelder.com> References: <4F6A2084.90508@nedbatchelder.com> Message-ID: On Wed, Mar 21, 2012 at 11:40 AM, Ned Batchelder wrote: > You can use Ctrl-+ to increase the size of the text, and modern browsers > remember that for the next time you visit the site. That doesn't mean the web designer shouldn't think at least twice before specifying a smaller font than the browser default. -- --Guido van Rossum (python.org/~guido) From fdrake at acm.org Wed Mar 21 20:06:17 2012 From: fdrake at acm.org (Fred Drake) Date: Wed, 21 Mar 2012 15:06:17 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> Message-ID: On Wed, Mar 21, 2012 at 2:46 PM, Guido van Rossum wrote: > That doesn't mean the web designer shouldn't think at least twice > before specifying a smaller font than the browser default. Yet 90% of designers (or more) insist on making text insanely small, commonly specifying the size in pixles or (if we're lucky) points. Not sure there's any lesson to be learned from this, aside from designers really having it out for anyone who needs to read. -Fred -- Fred L. Drake, Jr.? ? "A person who won't read has no advantage over one who can't read." ?? --Samuel Langhorne Clemens From phd at phdru.name Wed Mar 21 20:10:34 2012 From: phd at phdru.name (Oleg Broytman) Date: Wed, 21 Mar 2012 23:10:34 +0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6A2084.90508@nedbatchelder.com> References: <4F6A2084.90508@nedbatchelder.com> Message-ID: <20120321191034.GA18727@iskra.aviel.ru> On Wed, Mar 21, 2012 at 02:40:04PM -0400, Ned Batchelder wrote: > You can use Ctrl-+ to increase the size of the text, and modern > browsers remember that for the next time you visit the site. Browsers usually remember the setting for the entire site, not only documentation. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ned at nedbatchelder.com Wed Mar 21 20:12:17 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 21 Mar 2012 15:12:17 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> Message-ID: <4F6A2811.5080806@nedbatchelder.com> On 3/21/2012 2:46 PM, Guido van Rossum wrote: > On Wed, Mar 21, 2012 at 11:40 AM, Ned Batchelder wrote: >> You can use Ctrl-+ to increase the size of the text, and modern browsers >> remember that for the next time you visit the site. > That doesn't mean the web designer shouldn't think at least twice > before specifying a smaller font than the browser default. > Yes, sorry, that was exactly my point earlier in this thread. I was being a bit snarky with Serhiy. Seems the standard here is for people to request their personal favorite tweaks, and then tell others that they can use browser customizations to get what they want. Guido, you encouraged us to use science, but only after describing my science-based maximum line-length suggestion as "coddling," then said we should let Georg get on with it, but only after reiterating your personal favorite tweak (which I happen to agree with). There's no way a committee (which this thread effectively is) will come up with a good design. Everyone will dislike something about it. I think it would be interesting to use the power of the web to provide docs whose style could be adjusted a few ways to make people happy, but that is probably more than anyone is willing to volunteer for, I know I can't step up to do it. Personally, I think two Python projects that have focused on docs and done a good job of it are Django and readthedocs.org. Perhaps we could follow their lead? --Ned. From ned at nedbatchelder.com Wed Mar 21 20:13:39 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 21 Mar 2012 15:13:39 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> Message-ID: <4F6A2863.1080200@nedbatchelder.com> On 3/21/2012 3:06 PM, Fred Drake wrote: > On Wed, Mar 21, 2012 at 2:46 PM, Guido van Rossum wrote: >> That doesn't mean the web designer shouldn't think at least twice >> before specifying a smaller font than the browser default. > Yet 90% of designers (or more) insist on making text insanely small, commonly > specifying the size in pixles or (if we're lucky) points. > > Not sure there's any lesson to be learned from this, aside from designers > really having it out for anyone who needs to read. There are bad designers, or more to the point, designers who favor the overall look of the page at the expense of the utility of the page. That doesn't mean all designers are bad, or that "design" is bad. Don't throw out the baby with the bathwater. --Ned. > > -Fred > From pje at telecommunity.com Wed Mar 21 20:38:49 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 21 Mar 2012 15:38:49 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <20120321013717.1126e14f@pitrou.net> <21CFF6A5-DB47-455B-A7AF-39344739F597@gmail.com> <4F69A059.2030601@tartley.com> <20120321101621.GA5612@iskra.aviel.ru> <4F69CBDA.8010405@nedbatchelder.com> Message-ID: On Mar 21, 2012 12:00 PM, "Guido van Rossum" wrote: > > On Mar 21, 2012 5:44 AM, "Ned Batchelder" wrote: > > The best thing to do is to set a max-width in ems, say 50em. This leaves the text at a reasonable width, but adapts naturally for people with larger fonts. > > Please, no, not even this "improved" version of coddling. If you're > formatting e.g. a newspaper or a book, by all means (though I still > think the user should be given ultimate control -- and I don't mean > editing the CSS using the browser's development tools :-). But when > reading docs there are all sorts of reasons why I might want to > stretch the window to maximum width and nothing's more frustrating > than a website that forces clipping, folding or a horizontal scroll > bar even when I make the window wide enough. Well, the only thing that's more frustrating than that is having to resize my window to make the text readable, and then *still* having to scroll horizontally for the wide bits, or have to alternate sizes of the window. Just because flowing text paragraphs are set to a moderate max-width, that doesn't mean that code samples, tables, etc. all have to be the *same* max-width, or have any max-width at all. That is, keeping flowing text readable is not incompatible with having arbitrarily-wide code, tables, etc. (Text width is an ergonomic consideration as much as font size and color: too wide in absolute characters, and the eye has to hunt up and down to find where to start reading the next line.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 21 20:39:18 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2012 12:39:18 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6A2811.5080806@nedbatchelder.com> References: <4F6A2084.90508@nedbatchelder.com> <4F6A2811.5080806@nedbatchelder.com> Message-ID: On Wed, Mar 21, 2012 at 12:12 PM, Ned Batchelder wrote: > On 3/21/2012 2:46 PM, Guido van Rossum wrote: >> >> On Wed, Mar 21, 2012 at 11:40 AM, Ned Batchelder >> ?wrote: >>> >>> You can use Ctrl-+ to increase the size of the text, and modern browsers >>> remember that for the next time you visit the site. >> >> That doesn't mean the web designer shouldn't think at least twice >> before specifying a smaller font than the browser default. >> > Yes, sorry, that was exactly my point earlier in this thread. ?I was being a > bit snarky with Serhiy. ?Seems the standard here is for people to request > their personal favorite tweaks, and then tell others that they can use > browser customizations to get what they want. > > Guido, you encouraged us to use science, but only after describing my > science-based maximum line-length suggestion as "coddling," then said we > should let Georg get on with it, but only after reiterating your personal > favorite tweak (which I happen to agree with). I have a fair number of strong usability gripes about current (and past :-) design trends, but I know I can't design a decent looking website myself if my life depended on it. > There's no way a committee (which this thread effectively is) will come up > with a good design. ?Everyone will dislike something about it. ?I think it > would be interesting to use the power of the web to provide docs whose style > could be adjusted a few ways to make people happy, but that is probably more > than anyone is willing to volunteer for, I know I can't step up to do it. I think it's fine to have a bunch of folks submit their pet peeves (and argue them to the death :-) to the design czar and then let the czar (i.e. Georg) decide. > Personally, I think two Python projects that have focused on docs and done a > good job of it are Django and readthedocs.org. ?Perhaps we could follow > their lead? I think they are actually more trend-followers,and they seem to make a bunch of the mistakes I've fulminated against here. But again, I'll leave it to Georg. -- --Guido van Rossum (python.org/~guido) From ned at nedbatchelder.com Wed Mar 21 20:59:59 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 21 Mar 2012 15:59:59 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6A2FF6.5030207@stoneleaf.us> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A2FF6.5030207@stoneleaf.us> Message-ID: <4F6A333F.6060501@nedbatchelder.com> On 3/21/2012 3:45 PM, Ethan Furman wrote: > Guido van Rossum wrote: >> On Wed, Mar 21, 2012 at 7:18 AM, Ned Batchelder >> wrote: >>> The challenge for the maintainer of the docs site is to choose a >>> good design >>> that most people will see. We're bound to disagree on what that design >>> should be, and I suggest that probably none of us are designer >>> enough to >>> come up with the best one. Perhaps we could find an interested >>> designer to >>> help? >> >> I've come to the conclusion that "good design" is not so much a matter >> of finding the "best" of anything (font, spacing rules, colors, icons, >> artowork, etc.). Good design is highly subjective to fashion, and the >> people who are recognized to be the best designers are more often than >> not just those with a strong enough opinion to push their creative >> ideas through. Then other designers, who are not quite as good but >> still have a nose for the latest fashion, copy their ideas and for a >> while anything that hasn't been redesigned looks "old-fashioned". >> >> (Before you say something about limitations of old technology, note >> how often designers go back to older styles and manage to make them >> look fashionable again.) >> >> If you want something that attracts attention through controversy, get >> one of those initial thought leaders. If you want something that looks >> "current" today but which will probably be out of style next year, use >> one of the style-following designers. If you want something that is >> maximally useful, get a scientist with an ounce of style sense to do >> your design... Oh hey, Georg *is* a scientist! And he's got more than >> an ounce of style. So just let him do it and let's not try to >> micromanage things. (I had to speak up about the low contrast because >> Georg has young eyes and may not realize that this issue exists for >> older Pythonistas.) > > +1000 > Deriding the entire discipline of design because some of its practitioners are hacks is like pointing at PHP kiddies as the reason why you don't need "software architects." Yes, we could make the mistake of over-designing it, and that would be a mistake. The science you seek is something that designers are well-versed in. --Ned. From ethan at stoneleaf.us Wed Mar 21 20:45:58 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 21 Mar 2012 12:45:58 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> Message-ID: <4F6A2FF6.5030207@stoneleaf.us> Guido van Rossum wrote: > On Wed, Mar 21, 2012 at 7:18 AM, Ned Batchelder wrote: >> The challenge for the maintainer of the docs site is to choose a good design >> that most people will see. We're bound to disagree on what that design >> should be, and I suggest that probably none of us are designer enough to >> come up with the best one. Perhaps we could find an interested designer to >> help? > > I've come to the conclusion that "good design" is not so much a matter > of finding the "best" of anything (font, spacing rules, colors, icons, > artowork, etc.). Good design is highly subjective to fashion, and the > people who are recognized to be the best designers are more often than > not just those with a strong enough opinion to push their creative > ideas through. Then other designers, who are not quite as good but > still have a nose for the latest fashion, copy their ideas and for a > while anything that hasn't been redesigned looks "old-fashioned". > > (Before you say something about limitations of old technology, note > how often designers go back to older styles and manage to make them > look fashionable again.) > > If you want something that attracts attention through controversy, get > one of those initial thought leaders. If you want something that looks > "current" today but which will probably be out of style next year, use > one of the style-following designers. If you want something that is > maximally useful, get a scientist with an ounce of style sense to do > your design... Oh hey, Georg *is* a scientist! And he's got more than > an ounce of style. So just let him do it and let's not try to > micromanage things. (I had to speak up about the low contrast because > Georg has young eyes and may not realize that this issue exists for > older Pythonistas.) +1000 From tseaver at palladion.com Wed Mar 21 21:37:37 2012 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 21 Mar 2012 16:37:37 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6A2863.1080200@nedbatchelder.com> References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/21/2012 03:13 PM, Ned Batchelder wrote: > On 3/21/2012 3:06 PM, Fred Drake wrote: >> On Wed, Mar 21, 2012 at 2:46 PM, Guido van Rossum >> wrote: >>> That doesn't mean the web designer shouldn't think at least twice >>> before specifying a smaller font than the browser default. >> Yet 90% of designers (or more) insist on making text insanely small, >> commonly specifying the size in pixles or (if we're lucky) points. >> >> Not sure there's any lesson to be learned from this, aside from >> designers really having it out for anyone who needs to read. > There are bad designers, or more to the point, designers who favor the > overall look of the page at the expense of the utility of the page. > That doesn't mean all designers are bad, or that "design" is bad. > Don't throw out the baby with the bathwater. Designers who care more about utility / accessibility more than their hipster karma seem to be a tiny minority in the current web world (without even counting "web designers" who think a Photoshop document is the final deliverable). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9qPBEACgkQ+gerLs4ltQ5fMwCcD8cHLDch/cIlBpVY4htmlDN4 fzQAmgNUVVn+uByZRBI22TB7ETdkLzmP =ZHdF -----END PGP SIGNATURE----- From fdrake at acm.org Wed Mar 21 21:38:14 2012 From: fdrake at acm.org (Fred Drake) Date: Wed, 21 Mar 2012 16:38:14 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6A2863.1080200@nedbatchelder.com> References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> Message-ID: On Wed, Mar 21, 2012 at 3:13 PM, Ned Batchelder wrote: > There are bad designers, or more to the point, designers who favor the > overall look of the page at the expense of the utility of the page. ?That > doesn't mean all designers are bad, or that "design" is bad. ?Don't throw > out the baby with the bathwater. I get that. I'm not bad-mouthing actual design, and there are definitely good designers out there. It's unfortunate they're so seriously outnumbered. -Fred -- Fred L. Drake, Jr.? ? "A person who won't read has no advantage over one who can't read." ?? --Samuel Langhorne Clemens From ned at nedbatchelder.com Wed Mar 21 21:44:56 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 21 Mar 2012 16:44:56 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> Message-ID: <4F6A3DC8.3010804@nedbatchelder.com> On 3/21/2012 4:38 PM, Fred Drake wrote: > On Wed, Mar 21, 2012 at 3:13 PM, Ned Batchelder wrote: >> There are bad designers, or more to the point, designers who favor the >> overall look of the page at the expense of the utility of the page. That >> doesn't mean all designers are bad, or that "design" is bad. Don't throw >> out the baby with the bathwater. > I get that. I'm not bad-mouthing actual design, and there are definitely > good designers out there. > > It's unfortunate they're so seriously outnumbered. Yeah, just like software architects... :-( --Ned. > > -Fred > From rdmurray at bitdance.com Wed Mar 21 21:48:14 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 21 Mar 2012 16:48:14 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> <4F6A2811.5080806@nedbatchelder.com> Message-ID: <20120321204815.DF4682500E3@webabinitio.net> On Wed, 21 Mar 2012 12:39:18 -0700, Guido van Rossum wrote: > On Wed, Mar 21, 2012 at 12:12 PM, Ned Batchelder wrote: > > Personally, I think two Python projects that have focused on docs and done a > > good job of it are Django and readthedocs.org. ??Perhaps we could follow > > their lead? > > I think they are actually more trend-followers,and they seem to make a > bunch of the mistakes I've fulminated against here. But again, I'll > leave it to Georg. I'm pretty sure they are following the trend set by Python/Georg...it comes withs Sphinx, after all, and looks pretty good in general. --David From bradallen137 at gmail.com Wed Mar 21 21:38:50 2012 From: bradallen137 at gmail.com (Brad Allen) Date: Wed, 21 Mar 2012 15:38:50 -0500 Subject: [Python-Dev] Issue 13524: subprocess on Windows Message-ID: I tripped over this one trying to make one of our Python at work Windows compatible. We had no idea that a magic 'SystemRoot' environment variable would be required, and it was causing issues for pyzmq. It might be nice to reflect the findings of this email thread on the subprocess documentation page: http://docs.python.org/library/subprocess.html Currently the docs mention this: "Note If specified, env must provide any variables required for the program to execute. On Windows, in order to run a side-by-side assembly the specified env must include a valid SystemRoot." How about rewording that to: "Note If specified, env must provide any variables required for the program to execute. On Windows, a valid SystemRoot environment variable is required for some Python libraries such as the 'random' module. Also, in order to run a side-by-side assembly the specified env must include a valid SystemRoot." From victor.stinner at gmail.com Wed Mar 21 23:22:51 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Mar 2012 23:22:51 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: Message-ID: >> http://hg.python.org/cpython/rev/7355550d5357 >> changeset: ? 75850:7355550d5357 >> user: ? ? ? ?Stefan Krah >> date: ? ? ? ?Wed Mar 21 18:25:23 2012 +0100 >> summary: >> ?Issue #7652: Integrate the decimal floating point libmpdec library to speed >> up the decimal module. Performance gains of the new C implementation are >> between 12x and 80x, depending on the application. Congrats Stefan! And thanks for the huge chunk of code. Victor From greg.ewing at canterbury.ac.nz Wed Mar 21 23:28:26 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Mar 2012 11:28:26 +1300 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F69E337.7010501@nedbatchelder.com> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> Message-ID: <4F6A560A.6050207@canterbury.ac.nz> Ned Batchelder wrote: > Any of the tweaks people are suggesting could be applied individually > using this technique. We could just as easily choose to make the site > left-justified, and let the full-justification fans use custom > stylesheets to get it. Is it really necessary for the site to specify the justification at all? Why not leave it to the browser and whatever customisation the user chooses to make? -- Greg From skippy.hammond at gmail.com Wed Mar 21 23:43:05 2012 From: skippy.hammond at gmail.com (Mark Hammond) Date: Thu, 22 Mar 2012 09:43:05 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F69E40C.6050901@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> Message-ID: <4F6A5979.3000801@gmail.com> On 22/03/2012 1:22 AM, Lindberg, Van wrote: > Mark, MAL, Martin, Tarek, > > Could you comment on this? Eric is correct - tools will be broken by this change. However, people seem willing to push forward on this and accept such breakage as the necessary cost. MAL, in his followup, asks what the advantages are of such a change. I've actually been asking for the same thing in this thread and the only real answer I've got is "consistency". So while I share MAL's concerns, people seem willing to push forward on this anyway, without the benefits having been explained. IOW, this isn't the decision I would make, but I think I've already made that point a number of times in this thread. Beyond that, there doesn't seem much for me to add... Mark From songofacandy at gmail.com Wed Mar 21 23:45:40 2012 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 22 Mar 2012 07:45:40 +0900 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: +10 for new design. +1 for respecting default font size rather than "div.body {font-size: smaller;}" Users loving smaller font can set their browser's default font size. On Wed, Mar 21, 2012 at 7:38 AM, Georg Brandl wrote: > Hi all, > > recently I've grown a bit tired of seeing our default Sphinx theme, > especially as so many other projects use it. ?I decided to play around > with something "clean" this time, and this is the result: > > ?http://www.python.org/~gbrandl/build/html/ > > The corresponding sandbox repo is at > > ?http://hg.python.org/sandbox/doc-theme/#doc-theme > > Let me know what you think, or play around and send some improvements. > (The collapsible sidebar is not adapted to it yet, but will definitely > be integrated before I consider applying a new theme to the real docs.) > > Thanks, > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com -- INADA Naoki? From p.f.moore at gmail.com Thu Mar 22 00:03:08 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 21 Mar 2012 23:03:08 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: <4F6A5979.3000801@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F6A5979.3000801@gmail.com> Message-ID: On 21 March 2012 22:43, Mark Hammond wrote: > On 22/03/2012 1:22 AM, Lindberg, Van wrote: >> >> Mark, MAL, Martin, Tarek, >> >> Could you comment on this? > > > Eric is correct - tools will be broken by this change. ?However, people seem > willing to push forward on this and accept such breakage as the necessary > cost. > > MAL, in his followup, asks what the advantages are of such a change. I've > actually been asking for the same thing in this thread and the only real > answer I've got is "consistency". ?So while I share MAL's concerns, people > seem willing to push forward on this anyway, without the benefits having > been explained. > > IOW, this isn't the decision I would make, but I think I've already made > that point a number of times in this thread. ?Beyond that, there doesn't > seem much for me to add... I agree on all points here. I don't understand quite why backward compatibility is being treated so lightly here. But equally, I've made my points and have little further to add. One thought though - maybe this should need a PEP at least, to document the proposal and record the various arguments made in this thread? Paul. From doboy0 at gmail.com Thu Mar 22 00:39:48 2012 From: doboy0 at gmail.com (Huan Do) Date: Wed, 21 Mar 2012 16:39:48 -0700 Subject: [Python-Dev] New PEP Message-ID: *Hi, I am a graduating Berkeley student that loves python and would like to propose an enhancement to python. My proposal introduces a concept of slicing generator. For instance, if one does x[:] it returns a list which is a copy of x. Sometimes programmers would want to iterate over a slice of x, but they do not like the overhead of constructing another list. Instead we can create a similar operator that returns a generator. My proposed syntax is x(:). The programmers are of course able to set lower, upper, and step size like the following. x(1::-1) This would make code much cleaner in a lot of instances, one example lets say we have a very large list x and we want to sum all the numbers but the last 20, and we only want to loop through the even indices. We would have to do something like this. sum(x[:-20:2]) or we can do a workaround to save space for time and do something like this. sum( value for i, value in enumerate(x) if i < -20 and not i % 2 ) But with my proposal we are able do the following. sum(x(:-20:2)) Which affords space without sacrificing expressiveness. For another example lets say we have a problem that we want to check a condition is true for every pairwise element in a list x. def allfriends(x): for i in range(len(x)): for j in range(i+1, len(x)): if not friends(x[i], x[j]): return False return True A more pythonic way is to actually loop through the values instead of the indices like this. def allfriends(x): for i, a in enumerate(x): for j, b in enumerate(x[i+1:]): if not friends(a, b): return False return True This however bring a lot of overhead because we have to construct a new list for every slice call. With my proposal we are able to do this. def allfriends(x): for i, a in enumerate(x): for j, b in enumerate(x(i+1:)): if not friends(a, b): return False return True This proposal however goes against one heuristic in the zen of python, namely ?Special cases aren?t special enough to break the rules.? The way that the proposal breaks this rule is because the syntax x(:), uses a function call syntax but would be a special case here. I chose using parenthesis because I wanted this operation to be analogous to the generator syntax in list comprehensions. ListGeneratorsComprehension[ x for x in L ]( x for x in L )SlicingL[a:b:c] L(a:b:c) Tell me what you guys think. Thanks!* -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Mar 22 00:49:26 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Mar 2012 16:49:26 -0700 Subject: [Python-Dev] PEP 416: Add a frozendict builtin type In-Reply-To: <4F5B30A4.6080306@gmail.com> References: <1330541549.7844.69.camel@surprise> <4F5B30A4.6080306@gmail.com> Message-ID: To close the loop, I've rejected the PEP, adding the following rejection notice: """ I'm rejecting this PEP. A number of reasons (not exhaustive): * According to Raymond Hettinger, use of frozendict is low. Those that do use it tend to use it as a hint only, such as declaring global or class-level "constants": they aren't really immutable, since anyone can still assign to the name. * There are existing idioms for avoiding mutable default values. * The potential of optimizing code using frozendict in PyPy is unsure; a lot of other things would have to change first. The same holds for compile-time lookups in general. * Multiple threads can agree by convention not to mutate a shared dict, there's no great need for enforcement. Multiple processes can't share dicts. * Adding a security sandbox written in Python, even with a limited scope, is frowned upon by many, due to the inherent difficulty with ever proving that the sandbox is actually secure. Because of this we won't be adding one to the stdlib any time soon, so this use case falls outside the scope of a PEP. On the other hand, exposing the existing read-only dict proxy as a built-in type sounds good to me. (It would need to be changed to allow calling the constructor.) """ -- --Guido van Rossum (python.org/~guido) From victor.stinner at gmail.com Thu Mar 22 00:51:03 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 22 Mar 2012 00:51:03 +0100 Subject: [Python-Dev] PEP 416: Add a frozendict builtin type In-Reply-To: References: <1330541549.7844.69.camel@surprise> <4F5B30A4.6080306@gmail.com> Message-ID: 2012/3/22 Guido van Rossum : > To close the loop, I've rejected the PEP, adding the following rejection notice: > > """ > I'm rejecting this PEP. (...) Hum, you may specify who is "I" in the PEP. Victor From animelovin at gmail.com Thu Mar 22 00:55:43 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Wed, 21 Mar 2012 19:55:43 -0400 Subject: [Python-Dev] New PEP In-Reply-To: References: Message-ID: <4F6A6A7F.9070202@gmail.com> On 03/21/2012 07:39 PM, Huan Do wrote: > *Hi, > > I am a graduating Berkeley student that loves python and would like to > propose an enhancement to python. My proposal introduces a concept of > slicing generator. For instance, if one does x[:] it returns a list > which is a copy of x. Sometimes programmers would want to iterate over a > slice of x, but they do not like the overhead of constructing another > list. Instead we can create a similar operator that returns a generator. > My proposed syntax is x(:). The programmers are of course able to set > lower, upper, and step size like the following. > > x(1::-1) > > > This would make code much cleaner in a lot of instances, one example > lets say we have a very large list x and we want to sum all the numbers > but the last 20, and we only want to loop through the even indices. > > We would have to do something like this. > > sum(x[:-20:2]) > > > or we can do a workaround to save space for time and do something like this. > > sum( value for i, value in enumerate(x) if i < -20 and not i % 2 ) > > > But with my proposal we are able do the following. > > sum(x(:-20:2)) > > > Which affords space without sacrificing expressiveness. > > For another example lets say we have a problem that we want to check a > condition is true for every pairwise element in a list x. > > def allfriends(x): > > for i in range(len(x)): > > for j in range(i+1, len(x)): > > if not friends(x[i], x[j]): > > return False > > return True > > > A more pythonic way is to actually loop through the values instead of > the indices like this. > > def allfriends(x): > > for i, a in enumerate(x): > > for j, b in enumerate(x[i+1:]): > > if not friends(a, b): > > return False > > return True > > > This however bring a lot of overhead because we have to construct a new > list for every slice call. With my proposal we are able to do this. > > def allfriends(x): > > for i, a in enumerate(x): > > for j, b in enumerate(x(i+1:)): > > if not friends(a, b): > > return False > > return True > > > This proposal however goes against one heuristic in the zen of python, > namely ?Special cases aren?t special enough to break the rules.? The way > that the proposal breaks this rule is because the syntax x(:), uses a > function call syntax but would be a special case here. I chose using > parenthesis because I wanted this operation to be analogous to the > generator syntax in list comprehensions. > > List Generators > Comprehension [ x for x in L ] ( x for x in L ) > Slicing L[a:b:c] L(a:b:c) > > > > Tell me what you guys think. > > Thanks!* > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/animelovin%40gmail.com Hi, I'm not sure i get it.. Assuming your PEP is accepted, what should happens then to the lambda op and standard function calls ? Or Is this merely another case of metaprogramming, which obviously should not be confused with languages such as lisp? Thank you, Etienne From victor.stinner at gmail.com Thu Mar 22 00:57:55 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 22 Mar 2012 00:57:55 +0100 Subject: [Python-Dev] New PEP In-Reply-To: References: Message-ID: > My proposed syntax is x(:) Change the Python syntax is not a good start. You can already experiment your idea using the slice() type. > We would have to do something like this. > sum(x[:-20:2]) Do you know the itertools module? It looks like itertools.islice(). Victor From doboy0 at gmail.com Thu Mar 22 01:28:17 2012 From: doboy0 at gmail.com (Huan Do) Date: Wed, 21 Mar 2012 17:28:17 -0700 Subject: [Python-Dev] New PEP In-Reply-To: <4F6A6DB2.9030103@stoneleaf.us> References: <4F6A6DB2.9030103@stoneleaf.us> Message-ID: @Ethan Furman each call to x(:) would return a different iterator, so both sides will have their own information about where they are. Also it is the case that checking for equality of generators does not make the generators to expand out, so checking for equality becomes to checking if they are the same generator object. The following example shows this Python3 >> (x for x in range(10)) == (x for x in range(10)) False @Etienne "lambda" is a keyword and would get captured by the lexer, so this should conflict with adding the grammar that would make this work. This is different than function calls because currently arguments of function calls cannot have ":", causing `x(:)` to be a syntax error. The grammar that would have to be added would be mutually exclusive from current functionality. @Victor I was not completely familiar with itertools but itertools.islice() seems to have the functionality that I propose. It is great that there already exist a solution that does not change python's syntax. Unless anyone wants to pursue this proposal I will drop it next week. Thanks for your feedback guys On Wed, Mar 21, 2012 at 5:09 PM, Ethan Furman wrote: > Huan Do wrote: > >> *Hi, >> >> >> I am a graduating Berkeley student that loves python and would like to >> propose an enhancement to python. My proposal introduces a concept of >> slicing generator. For instance, if one does x[:] it returns a list which >> is a copy of x. Sometimes programmers would want to iterate over a slice of >> x, but they do not like the overhead of constructing another list. Instead >> we can create a similar operator that returns a generator. My proposed >> syntax is x(:). The programmers are of course able to set lower, upper, and >> step size like the following. >> >> x(1::-1) >> > > The biggest problem with your proposal is that generators don't remember > what they have already yielded, so > > x(:) != x(:) # first time gets everything, second time gets nothing > > ~Ethan~ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Thu Mar 22 01:42:22 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 22 Mar 2012 09:42:22 +0900 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F6A5979.3000801@gmail.com> Message-ID: Cleaning up the absurd CC line.... On Thu, Mar 22, 2012 at 8:03 AM, Paul Moore wrote: > I agree on all points here. I don't understand quite why backward > compatibility is being treated so lightly here. But equally, I've made > my points and have little further to add. As a non-Windows user who occasionally is the only one available to help Windows users do something (other than install Linux and learn to live free ), consistency would be nice. I often have trouble finding the right advice for Windows, even if I feel like looking for it. Dunno if that's a common or important use case, though. From ethan at stoneleaf.us Thu Mar 22 01:09:22 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 21 Mar 2012 17:09:22 -0700 Subject: [Python-Dev] New PEP In-Reply-To: References: Message-ID: <4F6A6DB2.9030103@stoneleaf.us> Huan Do wrote: > *Hi, > > I am a graduating Berkeley student that loves python and would like to > propose an enhancement to python. My proposal introduces a concept of > slicing generator. For instance, if one does x[:] it returns a list > which is a copy of x. Sometimes programmers would want to iterate over a > slice of x, but they do not like the overhead of constructing another > list. Instead we can create a similar operator that returns a generator. > My proposed syntax is x(:). The programmers are of course able to set > lower, upper, and step size like the following. > > x(1::-1) The biggest problem with your proposal is that generators don't remember what they have already yielded, so x(:) != x(:) # first time gets everything, second time gets nothing ~Ethan~ From ncoghlan at gmail.com Thu Mar 22 02:15:21 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Mar 2012 11:15:21 +1000 Subject: [Python-Dev] New PEP In-Reply-To: References: <4F6A6DB2.9030103@stoneleaf.us> Message-ID: On Thu, Mar 22, 2012 at 10:28 AM, Huan Do wrote: > I was not completely familiar with itertools but?itertools.islice() seems to > have the functionality that I propose. It is great that ?there already exist > a solution that does not change python's syntax. Unless anyone wants to > pursue this proposal I will drop it next week. Just as a further follow-up on the recommended approach for making suggestions: for initial concepts like this one, the "python-ideas" mailing list is the preferred venue. It's intended for initial validation and refinement of suggestions to see if they're a reasonable topic for the main development list. Many ideas don't make it past the python-ideas stage (either because they have too many problems, they get redirected to a third party PyPI project, or existing alternatives are pointed out, as happened in this case). Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From victor.stinner at gmail.com Thu Mar 22 02:53:32 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 22 Mar 2012 02:53:32 +0100 Subject: [Python-Dev] PEP 416: Add a frozendict builtin type In-Reply-To: References: <1330541549.7844.69.camel@surprise> <4F5B30A4.6080306@gmail.com> Message-ID: > On the other hand, exposing the existing read-only dict proxy as a > built-in type sounds good to me. ?(It would need to be changed to > allow calling the constructor.) I wrote a small patch to implement this request: http://bugs.python.org/issue14386 I also opened the following issue to support other types than dict for __builtins__: http://bugs.python.org/issue14385 This issue is directly related to pysandbox, but it may help other purpose. Victor From merwok at netwok.org Thu Mar 22 03:37:34 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 21 Mar 2012 22:37:34 -0400 Subject: [Python-Dev] GSoC 2012: Python Core Participation? In-Reply-To: <20120321080215.Horde.F0tnBFNNcXdPaXz3hMjTO8A@webmail.df.eu> References: <20120321080215.Horde.F0tnBFNNcXdPaXz3hMjTO8A@webmail.df.eu> Message-ID: <4F6A906E.3060704@netwok.org> Good evening, > If you are a core committer and volunteer as GSoC > mentor for 2012, please let me know by Friday > (March 23rd). There is a number of interesting things to implement in packaging, and at least one student who manifested their interest, but unfortunately I am presently unable to say if I?ll have the time to mentor. If other core developers would like to act as mentors like happened last year, I will be available for questions and reviews. Regards From merwok at netwok.org Thu Mar 22 04:04:54 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 21 Mar 2012 23:04:54 -0400 Subject: [Python-Dev] [RELEASED] Python 3.3.0 alpha 1 In-Reply-To: References: <4F54711E.2020006@python.org> <4f5668fa.a81e340a.358a.4fa6@mx.google.com> Message-ID: <4F6A96D6.1050208@netwok.org> Le 06/03/2012 15:31, Giampaolo Rodol? a ?crit : > That's why I once proposed to include whatsnew.rst changes every time > a new feature is added/committed. > Assigning that effort to the release manager or whoever is supposed to > take care of this, is both impractical and prone to forgetfulness. Well, it?s the call of the whatsnew author. I think amk wrote the original instructions at the top of each whatsnew file which explain that NEWS is the primary location for logging changes and whatsnew is composed from that file. If Raymond or the new whatsnew author wants to change the rules so that important changes are noted in whatsnew in addition to NEWS, nothing prevents that. Cheers From mail at timgolden.me.uk Thu Mar 22 07:08:24 2012 From: mail at timgolden.me.uk (Tim Golden) Date: Thu, 22 Mar 2012 06:08:24 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F6A5979.3000801@gmail.com> Message-ID: <4F6AC1D8.30604@timgolden.me.uk> On 21/03/2012 23:03, Paul Moore wrote: > On 21 March 2012 22:43, Mark Hammond wrote: >> On 22/03/2012 1:22 AM, Lindberg, Van wrote: >>> >>> Mark, MAL, Martin, Tarek, >>> >>> Could you comment on this? >> >> >> Eric is correct - tools will be broken by this change. However, people seem >> willing to push forward on this and accept such breakage as the necessary >> cost. >> >> MAL, in his followup, asks what the advantages are of such a change. I've >> actually been asking for the same thing in this thread and the only real >> answer I've got is "consistency". So while I share MAL's concerns, people >> seem willing to push forward on this anyway, without the benefits having >> been explained. >> >> IOW, this isn't the decision I would make, but I think I've already made >> that point a number of times in this thread. Beyond that, there doesn't >> seem much for me to add... > > I agree on all points here. I don't understand quite why backward > compatibility is being treated so lightly here. But equally, I've made > my points and have little further to add. Well I've gone through (and deleted) three draft contributions to the ideas proposed here over the last week or so. In short, I'm with Paul & Mark. The OP seems far more casual towards breakage than would be the case if, eg, code were involved. If this had been proposed for Python 3k I'd have said: go for it - why not? But for this to drop in now means, as others have said, that I'll have to adjust various small tools which assume the location of python.exe according to the (minor) version I'm running. I can certainly cope with the change and without too much difficulty, but I'm afraid it does smack of a too foolish consistency. And it's not as though I've seen crowds of people chiming in with a me-too! The only person strongly supporting the change (as distinct from not opposing it) is VanL, who appears to need it for his particular setup. In short, I'm -1 but I'm not going to storm off in a huff if it goes ahead, merely be a little bewildered at why this was needed by anyone else and exactly what real-world problem it's solving for thousands of Windows Python users. TJG From g.brandl at gmx.net Thu Mar 22 07:48:44 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 22 Mar 2012 07:48:44 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> <4F6A2811.5080806@nedbatchelder.com> Message-ID: On 21.03.2012 20:39, Guido van Rossum wrote: >> Guido, you encouraged us to use science, but only after describing my >> science-based maximum line-length suggestion as "coddling," then said we >> should let Georg get on with it, but only after reiterating your personal >> favorite tweak (which I happen to agree with). > > I have a fair number of strong usability gripes about current (and > past :-) design trends, but I know I can't design a decent looking > website myself if my life depended on it. > >> There's no way a committee (which this thread effectively is) will come up >> with a good design. Everyone will dislike something about it. I think it >> would be interesting to use the power of the web to provide docs whose style >> could be adjusted a few ways to make people happy, but that is probably more >> than anyone is willing to volunteer for, I know I can't step up to do it. > > I think it's fine to have a bunch of folks submit their pet peeves > (and argue them to the death :-) to the design czar and then let the > czar (i.e. Georg) decide. > >> Personally, I think two Python projects that have focused on docs and done a >> good job of it are Django and readthedocs.org. Perhaps we could follow >> their lead? > > I think they are actually more trend-followers,and they seem to make a > bunch of the mistakes I've fulminated against here. But again, I'll > leave it to Georg. Thanks for the vote of confidence. I'll know what to consider for the next iteration thanks to the lively participation here :) Georg From regebro at gmail.com Thu Mar 22 09:46:29 2012 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 22 Mar 2012 09:46:29 +0100 Subject: [Python-Dev] New PEP In-Reply-To: References: Message-ID: On Thu, Mar 22, 2012 at 00:39, Huan Do wrote: > Tell me what you guys think. I don't really want to add more things to the language, so I hate to say this: It makes sense to me. However, the syntax is very close to the syntax for function annotations. But that's when defining, and this is when calling, so it might work anyway, I don't have the knowledge necessary to know. So put it up on python-ideas. I'm not on that list, but people who know more about this are, so they can tell you if this is feasible or not and if it is a good idea or not. //Lennart -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Mar 22 13:44:00 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 22 Mar 2012 13:44:00 +0100 Subject: [Python-Dev] dictproxy for other mapping types than dict Message-ID: Hi, I created the following issue to expose the dictproxy as a builtin type. http://bugs.python.org/issue14386 I would be interesting to accept any mapping type, not only dict. dictproxy implementation supports any mapping, even list or tuple, but I don't want to support sequences because a missing key would raise an IndexError instead of a KeyError. My problem is to check the type in C. issubclass(collections.ChainMap, collections.abc.Sequence) is False which is the expected result, so I need to implement this check in C. "The "PyMapping_Check(dict) && !PyMapping_Check(dict)" fails on ChainMap: type.__new__(ChainMap) fills tp_as_sequence->sq_item slot is defined because ChainMap has a __getitem__ method. Do you have an idea how to implement such test? Victor From stefan at bytereef.org Thu Mar 22 13:45:55 2012 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 22 Mar 2012 13:45:55 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: Message-ID: <20120322124555.GA12503@sleipnir.bytereef.org> Victor Stinner wrote: > >> Issue #7652: Integrate the decimal floating point libmpdec library to speed > >> up the decimal module. Performance gains of the new C implementation are > >> between 12x and 80x, depending on the application. > > Congrats Stefan! And thanks for the huge chunk of code. Thanks, much appreciated. I'll take the opportunity to thank you in return for the gigantic amount of work you've done on Python in the past year! Stefan Krah From dirkjan at ochtman.nl Thu Mar 22 13:58:36 2012 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 22 Mar 2012 13:58:36 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: Message-ID: On Wed, Mar 21, 2012 at 23:22, Victor Stinner wrote: >>> http://hg.python.org/cpython/rev/7355550d5357 >>> changeset: ? 75850:7355550d5357 >>> user: ? ? ? ?Stefan Krah >>> date: ? ? ? ?Wed Mar 21 18:25:23 2012 +0100 >>> summary: >>> ?Issue #7652: Integrate the decimal floating point libmpdec library to speed >>> up the decimal module. Performance gains of the new C implementation are >>> between 12x and 80x, depending on the application. As a Python user, this looks really cool, thanks! As a packager, is the libmpdec library used elsewhere? For Gentoo, we generally prefer to package libraries separately and have Python depend on them. From the site, it seems like you more or less wrote libmpdec for usage in Python, but if it's general-purpose and actually used in other software, it would be nice if Python grew a configure option to make it use the system libmpdec. Cheers, Dirkjan From van.lindberg at gmail.com Thu Mar 22 15:17:00 2012 From: van.lindberg at gmail.com (VanL) Date: Thu, 22 Mar 2012 09:17:00 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4F69EB9D.1060701@egenix.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> Message-ID: <4F6B345C.1020406@gmail.com> As this has been brought up a couple times in this subthread, I figured that I would lay out the rationale here. There are two proposals on the table: 1) Regularize the install layout, and 2) move the python binary to the binaries directory. This email will deal with the first, and a second email will deal with the second. 1) Regularizing the install layout: One of Python's strengths is its cross-platform appeal. Carefully- written Python programs are frequently portable between operating systems and Python implementations with very few changes. Over the years, substantial effort has been put into maintaining platform parity and providing consistent interfaces to available functionality, even when different underlying implementations are necessary (such as with ntpath and posixpath). One place where Python is unnecessarily different, however, is in the layout and organization of the Python environment. This is most visible in the name of the directory for binaries on the Windows platform ("Scripts") versus the name of the directory for binaries on every other platform ("bin"), but a full listing of the layouts shows substantial differences in layout and capitalization across platforms. Sometimes the include is capitalized ("Include"), and sometimes not; and the python version may or may not be included in the path to the standard library or not. This may seem like a harmless inconsistency, and if that were all it was, I wouldn't care. (That said, cross-platform consistency is its own good). But it becomes a real pain when combined with tools like virtualenv or the new pyvenv to create cross-platform development environments. In particular, I regularly do development on both Windows and a Mac, and then deploy on Linux. I do this in virtualenvs, so that I have a controlled and regular environment. I keep them in sync using source control. The problem comes when I have executable scripts that I want to include in my dvcs - I can't have it in the obvious place - the binaries directory - because *the name of the directory changes when you move between platforms.* More concretely, I can't hg add "Scripts/runner.py? on my windows environment (where it is put in the PATH by virtualenv) and thendo a pull on Mac or Linux and have it end up properly in "bin/runner.py" which is the correct PATH for those platforms. This applies anytime there are executable scripts that you want to manage using source control across platforms. Django projects regularly have these, and I suspect we will be seeing more of this with the new "project" support in virtualenvwrapper. While a few people have wondered why I would want this -- hopefully answered above -- I have not heard any opposition to this part of the proposal. This first proposal is just to make the names of the directories match across platforms. There are six keys defined in the installer files (sysconfig.cfg and distutils.command.install): 'stdlib', 'purelib', 'platlib', 'headers', 'scripts', and 'data'. Currently on Windows, there are two different layouts defined: 'nt': { 'stdlib': '{base}/Lib', 'platstdlib': '{base}/Lib', 'purelib': '{base}/Lib/site-packages', 'platlib': '{base}/Lib/site-packages', 'include': '{base}/Include', 'platinclude': '{base}/Include', 'scripts': '{base}/Scripts', 'data' : '{base}', }, 'nt_user': { 'stdlib': '{userbase}/Python{py_version_nodot}', 'platstdlib': '{userbase}/Python{py_version_nodot}', 'purelib': '{userbase}/Python{py_version_nodot}/site-packages', 'platlib': '{userbase}/Python{py_version_nodot}/site-packages', 'include': '{userbase}/Python{py_version_nodot}/Include', 'scripts': '{userbase}/Scripts', 'data' : '{userbase}', }, The proposal is to make all the layouts change to: 'nt': { 'stdlib': '{base}/lib', 'platstdlib': '{base}/lib', 'purelib': '{base}/lib/site-packages', 'platlib': '{base}/lib/site-packages', 'include': '{base}/include', 'platinclude': '{base}/include', 'scripts': '{base}/bin', 'data' : '{base}', }, The change here is that 'Scripts' will change to 'bin' and the capitalization will be removed. Also, "user installs" of Python will have the same internal layout as "system installs" of Python. This will also, not coincidentally, match the install layout for posix, at least with regard to the 'bin', 'lib', and 'include' directories. Again, I have not heard *anyone* objecting to this part of the proposal as it is laid out here. (Paul had a concern with the lib directory earlier, but he said he was ok with the above). Please let me know if you have any problems or concerns with this part 1. Thanks, Van From van.lindberg at gmail.com Thu Mar 22 15:47:10 2012 From: van.lindberg at gmail.com (VanL) Date: Thu, 22 Mar 2012 09:47:10 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: <4F69EB9D.1060701@egenix.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> Message-ID: <4F6B3B6E.2000402@gmail.com> [PART 2: Moving the python binary] There are two proposals on the table: 1) Regularize the install layout, and 2) move the python binary to the binaries directory. This email deals with the second issue exclusively. This has been the more contentious issue. 2) Moving the Python exe: A regular complaint of those new to Python on windows (and new to programming generally) has been that one of the first things that they need to do is to edit the PATH to allow Python to be run. In particular, this is the difficult sequence: 1. Install python. 2. Open up a shell and run "python" 3. Use pip or easy_install to install regetron (a package that installs an executable file). 4. Run regetron. For step #2, the python exe needs to be on the PATH. For steps 3 and 4, the binaries directory needs to be on the PATH. Currently, neither of these are true, so the path needs to be edited to include both the python root (where python.exe is, for step 2) and the "Scripts" (hopefully soon "bin") directory where "pip" and "regetron" are (for steps 3 and 4). You can substitute "regetron" for "nose," "cython," or other packages as well. MvL asked why anyone would want to run python directly from a cmd shell instead of running it from the start menu. There are two immediate responses to that: 1) observed behavior is that people prefer to run "python" from the cmd shell when developing (as observed by various people teaching Python, including Brian Curtin in this thread), and 2) running python or python programs from the shell is sometimes the only way to get a proper traceback when developing, making it a better way to work. The proposal here is to move the python.exe into the binaries directory (whatever it is called) and add an option to the windows installer to add that one directory to the PATH on install (and clean up the PATH on uninstall). A new registry key would be added pointing to the location of the python binary (wherever it is). Brian Curtin suggested this part of the proposal and has implemented it in a branch. MvL suggested a gradual transition to this over a three-release period. Open Issues: The PEP 397 Installer: As pointed out by Paul Moore, it may not matter once PEP 397 lands if python.exe is in the PATH or not - and it may be better if it is not. As he put it: """If we do put python.exe on PATH (whether it's in bin or not), we have to debate how to handle people having multiple versions of python on their machine. In a post-PEP 397 world, no Python is "the machine default" - .py files are associated with py.exe, not python.exe, so we have to consider the following 3 commands being run from a shell prompt: 1. myprog.py 2. py myprog.py 3. python myprog.py 1 and 2 will always do the same thing. However, 3 could easily do something completely different, if the Python in the #! line differs from the one found on PATH. To me, this implies that it's better for (3) to need explicit user action (setting PATH) if it's to do anything other than give an error. But maybe that's just me. I've been hit too often by confusion caused by *not* remembering this fact.""" One possible response here is that the moving of the python.exe binary and the setting of the PATH would be tied to an unchecked-by-default installer option, making an explicit user choice needed to invoke the new functionality. Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden have all expressed that they have existing tools that would break and would need to be adjusted to match the new location of the python.exe, because that location is assumed to be at the root of the python install. A related issue is that this portion of the proposal has met with some resistance, but not much support here on Python-dev. The reason for that is selection bias: Those who are on Python-dev are much more likely to have tools that do advanced things with Python, such as introspect on the location of the binary, and are also much more likely to be comfortable with things like editing the PATH on windows. In contrast, the people that have trouble with this issue are those that are newest to Python and programming generally - those for whom editing the PATH is a challenge and whom are likely to be confused by the distinction between python.exe and a python program - and why, even after they add python to the path, the python program is not directly executable. From pete.alex.harris at gmail.com Thu Mar 22 12:26:49 2012 From: pete.alex.harris at gmail.com (Peter Harris) Date: Thu, 22 Mar 2012 11:26:49 +0000 Subject: [Python-Dev] Python-Dev Digest, Vol 104, Issue 79 In-Reply-To: References: Message-ID: > > On 03/21/2012 07:39 PM, Huan Do wrote: > > *Hi, > > > > I am a graduating Berkeley student that loves python and would like to > > propose an enhancement to python. My proposal introduces a concept of > > slicing generator. For instance, if one does x[:] it returns a list > > which is a copy of x. Sometimes programmers would want to iterate over a > > slice of x, but they do not like the overhead of constructing another > > list. Instead we can create a similar operator that returns a generator. > > My proposed syntax is x(:). The programmers are of course able to set > lower, upper, and step size like the following. -1 on the syntax And have you looked at itertools.islice ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Thu Mar 22 18:12:32 2012 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 22 Mar 2012 13:12:32 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6A560A.6050207@canterbury.ac.nz> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> Message-ID: <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> On Mar 21, 2012, at 6:28 PM, Greg Ewing wrote: > Ned Batchelder wrote: >> Any of the tweaks people are suggesting could be applied individually using this technique. We could just as easily choose to make the site left-justified, and let the full-justification fans use custom stylesheets to get it. > > Is it really necessary for the site to specify the justification > at all? Why not leave it to the browser and whatever customisation > the user chooses to make? It's design. It's complicated. Maybe yes, if you look at research related to default usage patterns, and saccade distance, reading speed and retention latency. Maybe no, if you look at research related to fixation/focus time, eye strain, and non-linear access patterns. Maybe maybe, if you look at the subjective aesthetic of the page according to various criteria, like "does it look like a newspaper" and "do I have to resize my browser every time I visit a new site to get a decent width for reading". As has been said previously in this thread several times, it's best to leave this up to a design czar who will at least make some decisions who will make some people happy. I'm fairly certain it's not possible to create a design that's optimal for all readers in all cases. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Mar 22 18:02:43 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Mar 2012 04:02:43 +1100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> Message-ID: <4F6B5B33.9020406@pearwood.info> Fred Drake wrote: > On Wed, Mar 21, 2012 at 3:13 PM, Ned Batchelder wrote: >> There are bad designers, or more to the point, designers who favor the >> overall look of the page at the expense of the utility of the page. That >> doesn't mean all designers are bad, or that "design" is bad. Don't throw >> out the baby with the bathwater. > > I get that. I'm not bad-mouthing actual design, and there are definitely > good designers out there. > > It's unfortunate they're so seriously outnumbered. As they say, the 99% who are lousy designers give the rest a bad name. *wink* My first impression of this page: http://www.python.org/~gbrandl/build/html/index.html was that the grey side-bar gives the page a somber, perhaps even dreary, look. First impressions count, and I'm afraid that first look didn't work for me. But clicking through onto other pages with more text improved my feelings. A big +1 on the pale green shading of code blocks. The basic design seems good to me. I'd prefer a serif font for blocks of text, while keeping sans serif for the headings, but that's a mild preference. Looking forward to seeing the next iteration. -- Steven From p.f.moore at gmail.com Thu Mar 22 18:59:34 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 22 Mar 2012 17:59:34 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4F6B345C.1020406@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> Message-ID: On 22 March 2012 14:17, VanL wrote: > As this has been brought up a couple times in this subthread, I figured that > I would lay out the rationale here. I'm repeating myself here after I promised not to. My apologies, but I don't think this posting captures the debate completely. One reason I suggested a PEP is to better ensure that the arguments both pro and con were captured, as that is a key part of the PEP process. > One place where Python is unnecessarily different, however, is in > the layout and organization of the Python environment. This is most > visible in the name of the directory for binaries on the Windows platform > ("Scripts") versus the name of the directory for binaries on every other > platform ("bin"), First of all, this difference is almost entirely *invisible*. Apart from possibly setting PATH (once!) users should not be digging around in the Python installation directory. Certainly on Windows, it is a very unusual application that expects users to even care how the application is laid out internally. And I suspect that is also true on Unix and Mac. Secondly, the layouts are not as similar as you claim here, if I understand things correctly. on Unix Python is installed in /usr/local/bin so there isn't even a "Python installation directory" there. And Macs use some sort of Mac-specific bundle technology as I understand it. To be honest, I don't think that there's a lot of similarity even with your proposed changes. > but a full listing of the layouts shows > substantial differences in layout and capitalization across platforms. That's true, but largely incidental. And given that (a) Windows is case insensitive and (b) the capitalisation, although inconsistent, follows platform standards (Unix all lowercase, Windows capitalised) it makes little practical difference. > Sometimes the include is capitalized ("Include"), and sometimes not; and > the python version may or may not be included in the path to the > standard library or not. Given that on Windows the Python library is usually in something like C:\Python32\Lib whereas on Unix it's in /usr/lib/python3.2 (I think), the difference is reasonable because the Python *base* location (C:\Python32 on Windows vs /usr on Unix) is version specific in one case but not the other. To keep the correspondence complete, you should be suggesting installing in /python32 on Unix (which I doubt would gain much support :-)) > This may seem like a harmless inconsistency, and if that were all it was, I > wouldn't care. (That said, cross-platform consistency is its own good). But > it becomes a real pain when combined with tools like virtualenv or the new > pyvenv to create cross-platform development environments. The issue with virtualenv and pyvenv may be more significant. But you're only mentioning those incidentally. As a straw-man suggestion, why can virtualenv not be changed to maintain a platform-neutral layout in spite of what core Python does? This is a straw-man because I know the answer, but can we get the point out in the open - it's related to how distutils installs code, and that in turn hits you straight up against the distutils freeze. If distutils' behaviour is the issue here, then argue for more flexibility in packaging, and use that extra flexibility to argue for changes to virtualenv and pyvenv to maintain a standard cross-platform layout. Breaking the Python installation layout isn't the only option here, and I'd like to see a clear analysis of the tradeoffs. (I also get a sense of undue haste - "we can change the Python layout for 3.3, but changing packaging and virtualenv is a much longer process"...) > In particular, I regularly do development on both Windows and a Mac, and > then deploy on Linux. I do this in virtualenvs, so that I have a controlled > and regular environment. I keep them in sync using source control. > > The problem comes when I have executable scripts that I want to include in > my dvcs - I can't have it in the obvious place - the binaries directory - > because *the name of the directory changes when you move between platforms.* > More concretely, I can't hg add "Scripts/runner.py? on my windows > environment (where it is put in the PATH by virtualenv) and thendo a pull on > Mac or Linux and have it end up properly in "bin/runner.py" which is the > correct PATH for those platforms. This presupposes that your development workflow - developing in place in the virtualenv itself - is "the obvious approach". From what I've seen of tools like virtualenvwrapper, you're not alone in this. And I'm pretty new to using virtualenv so I wouldn't like to claim to be any expert. But can I respectfully suggest that other ways of working wouldn't hit these issues? WhatI do is develop my project in a project specific directory, just as I would if I were using the system Python. And I have an activated virtualenv *located somewhere else* that I install required 3rd party modules in, etc. I then do standard "pip install" to install and test my project, etc etc. This adds an "install to test" cycle (or you use something like setuptools' "deploy" techniques, which I know nothing about myself) but in exchange you are independent of the virtualenv layout (as pip install puts your scripts in the appropriate Scripts/bin directory). Also, by doing this you test your installation process, which you don't if you develop in place. Again, you may not like this way of working, and that's fine. But can you acknowledge (and document) that "change your way of working" is another alternative to "change Python". > This applies anytime there are executable scripts that you want to manage > using source control across platforms. Django projects regularly have these, > and I suspect we will be seeing more of this with the new "project" support > in virtualenvwrapper. ... if you develop inside the virtualenv. > While a few people have wondered why I would want this -- hopefully answered > above -- I have not heard any opposition to this part of the proposal. See above. There has been opposition from a number of people. It's relatively mild, simply because it's a niche area and doesn't cause huge pain, but it's there. And you seem (based on the above analysis) to be overstating the benefits, so the summary here is weighted heavily in favour of change. Also, you have made no attempt that I've seen to address the question of why this is important enough to break backward compatibility. Maybe it is - but why? Backward compatibility is a very strong rule, and should be broken only with good justification. Consistency, and "it makes my way of working easier" really shouldn't be sufficient. Has anyone checked whether this will affect people like Enthought and ActiveState who distribute their own versions of Python? Is ActiveState's PPM tool affected? > This first proposal is just to make the names of the directories match > across platforms. There are six keys defined in the installer files > (sysconfig.cfg and distutils.command.install): 'stdlib', 'purelib', > 'platlib', 'headers', 'scripts', ?and 'data'. > > Currently on Windows, there are two different layouts defined: > > ?'nt': { > ? ?'stdlib': '{base}/Lib', > ? ?'platstdlib': '{base}/Lib', > ? ?'purelib': '{base}/Lib/site-packages', > ? ?'platlib': '{base}/Lib/site-packages', > ? ?'include': '{base}/Include', > ? ?'platinclude': '{base}/Include', > ? ?'scripts': '{base}/Scripts', > ? ?'data' ? : '{base}', > ? ?}, > > ?'nt_user': { > ? ?'stdlib': '{userbase}/Python{py_version_nodot}', > ? ?'platstdlib': '{userbase}/Python{py_version_nodot}', > ? ?'purelib': '{userbase}/Python{py_version_nodot}/site-packages', > ? ?'platlib': '{userbase}/Python{py_version_nodot}/site-packages', > ? ?'include': '{userbase}/Python{py_version_nodot}/Include', > ? ?'scripts': '{userbase}/Scripts', > ? ?'data' ? : '{userbase}', > ? ?}, > > > The proposal is to make all the layouts change to: > > ?'nt': { > ? ?'stdlib': '{base}/lib', > ? ?'platstdlib': '{base}/lib', > ? ?'purelib': '{base}/lib/site-packages', > ? ?'platlib': '{base}/lib/site-packages', > ? ?'include': '{base}/include', > ? ?'platinclude': '{base}/include', > ? ?'scripts': '{base}/bin', > ? ?'data' ? : '{base}', > ? ?}, > > The change here is that 'Scripts' will change to 'bin' and the > capitalization will be removed. Also, "user installs" of Python will have > the same internal layout as "system installs" of Python. This will also, not > coincidentally, match the install layout for posix, at least with regard to > the 'bin', 'lib', and 'include' directories. Note - that is not "Regularizing the layout". You have not made any changes to OS/2 (which matches Windows at the moment). And it doesn't match Posix at all. I won't copy a large chunk of sysconfig.py in here, but I'm tempted to, because your proposal as described really doesn't match the reality of sysconfig._INSTALL_SCHEMES. I'd encourage people to go and read sysconfig.py for details. There really isn't much consistency at the moment, and fiddling with Windows but nothing else doesn't regularise anything. > Again, I have not heard *anyone* objecting to this part of the proposal as > it is laid out here. (Paul had a concern with the lib directory earlier, but > he said he was ok with the above). That's somewhat odd, as I did hear a number of concerns. But it was certainly not easy to tell which related to which part of the proposal. And all of the objections were mild, mostly because it's not a huge practical issue either way. > Please let me know if you have any problems or concerns with this part 1. Personally, my main concerns are around procedure and policy. The more the discussion goes on, the more I feel that there should be a PEP to capture the details of the debate clearly. Too much is getting lost in the noise. And I think you should provide a clear statement of why this issue is important enough to justify violating the backward compatibility policies. As Mark said (I think it was Mark...) if this had been proposed for 3.0, it would have been fine. Now we're at 3.2 with 3.3 close to release, and it just seems too late to be worth the risk. One plus point about your posting this separately. It's made me think through the issue in a bit more detail, and I'm now a solid -1 on the proposal. Paul. From rowen at uw.edu Thu Mar 22 19:49:54 2012 From: rowen at uw.edu (Russell E. Owen) Date: Thu, 22 Mar 2012 11:49:54 -0700 Subject: [Python-Dev] Playing with a new theme for the docs References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> Message-ID: In article <4F6B5B33.9020406 at pearwood.info>, Steven D'Aprano wrote: >... > My first impression of this page: > > http://www.python.org/~gbrandl/build/html/index.html > > was that the grey side-bar gives the page a somber, perhaps even dreary, > look. > First impressions count, and I'm afraid that first look didn't work for me. > > But clicking through onto other pages with more text improved my feelings. A > big +1 on the pale green shading of code blocks. > > The basic design seems good to me. I'd prefer a serif font for blocks of > text, > while keeping sans serif for the headings, but that's a mild preference. > > Looking forward to seeing the next iteration. I like the overall design, but one thing seems to be missing is an overview of what Python is (hence what the page is about). Naturally we don't need that, but a one-line overview with a link to more information would be helpful. -- Russell From van.lindberg at gmail.com Thu Mar 22 19:57:17 2012 From: van.lindberg at gmail.com (VanL) Date: Thu, 22 Mar 2012 13:57:17 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> Message-ID: <4F6B760D.3010004@gmail.com> Hi Paul, To start with, I appreciate your comments, and it is worth having both sides expressed. On 3/22/2012 12:59 PM, Paul Moore wrote: > I'm repeating myself here after I promised not to. My apologies, but I > don't think this posting captures the debate completely. One reason I > suggested a PEP is to better ensure that the arguments both pro and > con were captured, as that is a key part of the PEP process. I would be happy to write up a PEP. > First of all, this difference is almost entirely *invisible*. Apart > from possibly setting PATH (once!) users should not be digging around > in the Python installation directory. Certainly on Windows, it is a > very unusual application that expects users to even care how the > application is laid out internally. And I suspect that is also true on > Unix and Mac. This is a good point; it is mostly visible in the virtualenvs. If it only changed in virtualenvs, I would be happy. The policy, though, is that the virtualenv follows the platform policy. > Secondly, the layouts are not as similar as you claim here, if I > understand things correctly. on Unix Python is installed in > /usr/local/bin so there isn't even a "Python installation directory" > there. And Macs use some sort of Mac-specific bundle technology as I > understand it. To be honest, I don't think that there's a lot of > similarity even with your proposed changes. I was summarizing here because, frankly, there are hardly any OS/2 users, so it would be mostly Windows users affected by this change. Also as noted, I suggest that all platforms standardize on bin, lib, and include, just as I laid out. That said, while I think that the above is a good idea, my personal ambitions are more modest: If the names of the top-level directories only were changed to 'bin', 'lib', and 'include' - never mind differences under 'lib' - I would be happy. In fact, even the one change of 'Scripts' to 'bin' everywhere would get 90% of my uses. > The issue with virtualenv and pyvenv may be more significant. But > you're only mentioning those incidentally. I am approaching it from the platform level because of the policy that virtualenvs match the platform install layout. If instead virtualenv layouts were standardized, that would end up making me just as happy. > (I also get a sense of undue haste - > "we can change the Python layout for 3.3, but changing packaging and > virtualenv is a much longer process"...) Honestly, I didn't expect that much resistance. None of the people I talked to in person even cared, or if they did, they thought that consistency was a benefit. But now that virtualenvs are going in in 3.3, I see this as the last good chance to change this. > This presupposes that your development workflow - developing in place > in the virtualenv itself - is "the obvious approach". From what I've > seen of tools like virtualenvwrapper, you're not alone in this. [...] But can > you acknowledge (and document) that "change your way of working" is > another alternative to "change Python". > Acknowledged. What you say is true - but people wanted to know what the benefit would be. I laid out my concrete use-case as a rationale. And as you note, I am not alone in this type of development. Sure, I care here because it affects my style of development, and there are other styles that have other benefits (and tradeoffs). I don't see that this part of the proposal would negatively affect those styles. > >> While a few people have wondered why I would want this -- hopefully answered >> above -- I have not heard any opposition to this part of the proposal. > See above. There has been opposition from a number of people. It's > relatively mild, simply because it's a niche area and doesn't cause > huge pain, but it's there. And you seem (based on the above analysis) > to be overstating the benefits, so the summary here is weighted > heavily in favour of change. If I have misrepresented anyone, I am sorry - but to the best of my understanding, no one had (prior to you, right now) objected to *this part* of the proposal. Mark, at least, specified that his concern was with the moving of the python binary and that he didn't care about this part. I believe Tim indicated that too, but perhaps I have on my rose-colored glasses and misunderstood him. > Also, you have made no attempt that I've seen to address the question > of why this is important enough to break backward compatibility. Maybe > it is - but why? Backward compatibility is a very strong rule, and > should be broken only with good justification. Consistency, and "it > makes my way of working easier" really shouldn't be sufficient. In general, yes, I agree with you. However, the break with backwards compatibility is, as you point out, minor, and there is a benefit to consistency - especially given virtualenv-centric development. > Has anyone checked whether this will affect people like Enthought and > ActiveState who distribute their own versions of Python? Is > ActiveState's PPM tool affected? I have been running like this for several years across multiple Python versions, so I have experience with the "breakage" from this part of the proposal. I have found four packages that would need to be updated: Pip, virtualenv, PyPM, and Egginst would need 1-2 line patches. I have these patches, I would/could provide them. Generally these tools have something like: if platform == 'win32': bin_dir = 'Scripts' else: bin_dir = 'bin' The patches just remove the special casing - bin_dir just gets set to 'bin'. > Note - that is not "Regularizing the layout". You have not made any > changes to OS/2 (which matches Windows at the moment). And it doesn't > match Posix at all. See my 'summarizing' content above - layouts should match. I also didn't want to post chunks of sysconfig.py. You also missed distutils.command.install, which is subtly different. The OS X framework is already posixy inside, and virtualenvs on Mac OS X follow the posix-user layout. It is true that some Linux distributions place lib, include, etc in the system-wide directories. However, altinstalls that are confined to a directory (i.e., have a 'layout' in the sense that I am describing. My proposal would prohibit people from having multiple versions installed on top of each other, but they would clobber each other anyway with the current setup. > >> Again, I have not heard *anyone* objecting to this part of the proposal as >> it is laid out here. (Paul had a concern with the lib directory earlier, but >> he said he was ok with the above). > That's somewhat odd, as I did hear a number of concerns. But it was > certainly not easy to tell which related to which part of the > proposal. And all of the objections were mild, mostly because it's not > a huge practical issue either way. True. And again, I tried not to misrepresent anyone, but I have not heard anyone (including you) who would actually have code broken by *this* change. Do you have code? I really, really would like to know. I have not found anything that needs a patch except for the four packages above. > Personally, my main concerns are around procedure and policy. The more > the discussion goes on, the more I feel that there should be a PEP to > capture the details of the debate clearly. Too much is getting lost in > the noise. And I think you should provide a clear statement of why > this issue is important enough to justify violating the backward > compatibility policies. As Mark said (I think it was Mark...) if this > had been proposed for 3.0, it would have been fine. Now we're at 3.2 > with 3.3 close to release, and it just seems too late to be worth the > risk. One plus point about your posting this separately. It's made me > think through the issue in a bit more detail, and I'm now a solid -1 > on the proposal. I have been trying at various PyCons and in various conversations to move this for years. No one cares. The current urgency is driven by pyvenv - changes now will be much, much easier than changes later. Again, I am happy to write a PEP. If I were to summarize (on this issue only): 1. The current backwards compatibility hit is minimal; I would be happy to contact and provide patches to the four packages I have found (and anyone else who wants one). Backwards compatibility in the future will probably be harder to deal with. 2. There are advantages to cross-platform consistency and to virtualenv-based development. I believe that these will grow in the future. 3. Most people won't care. To the extent that people notice, I think they will appreciate the consistency. From rowen at uw.edu Thu Mar 22 20:05:00 2012 From: rowen at uw.edu (Russell E. Owen) Date: Thu, 22 Mar 2012 12:05:00 -0700 Subject: [Python-Dev] Playing with a new theme for the docs References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> Message-ID: In article , "Russell E. Owen" wrote: > In article <4F6B5B33.9020406 at pearwood.info>, > Steven D'Aprano wrote: > > >... > > My first impression of this page: > > > > http://www.python.org/~gbrandl/build/html/index.html > > > > was that the grey side-bar gives the page a somber, perhaps even dreary, > > look. > > First impressions count, and I'm afraid that first look didn't work for me. > > > > But clicking through onto other pages with more text improved my feelings. > > A > > big +1 on the pale green shading of code blocks. > > > > The basic design seems good to me. I'd prefer a serif font for blocks of > > text, > > while keeping sans serif for the headings, but that's a mild preference. > > > > Looking forward to seeing the next iteration. > > I like the overall design, but one thing seems to be missing is an > overview of what Python is (hence what the page is about). Naturally we > don't need that, but a one-line overview with a link to more information > would be helpful. > > -- Russell I'm afraid my last sentence was incoherent. I meant to say: Naturally we, as Python users, don't need that; but a one-line overview with a link to more information would be helpful to others who are not familiar with the language. -- Russell From glyph at twistedmatrix.com Thu Mar 22 20:35:19 2012 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 22 Mar 2012 15:35:19 -0400 Subject: [Python-Dev] Issue 13524: subprocess on Windows In-Reply-To: References: Message-ID: <61913F1B-1DFC-4EA5-AF29-25BBF24DA4DA@twistedmatrix.com> On Mar 21, 2012, at 4:38 PM, Brad Allen wrote: > I tripped over this one trying to make one of our Python at work > Windows compatible. We had no idea that a magic 'SystemRoot' > environment variable would be required, and it was causing issues for > pyzmq. > > It might be nice to reflect the findings of this email thread on the > subprocess documentation page: > > http://docs.python.org/library/subprocess.html > > Currently the docs mention this: > > "Note If specified, env must provide any variables required for the > program to execute. On Windows, in order to run a side-by-side > assembly the specified env must include a valid SystemRoot." > > How about rewording that to: > > "Note If specified, env must provide any variables required for the > program to execute. On Windows, a valid SystemRoot environment > variable is required for some Python libraries such as the 'random' > module. Also, in order to run a side-by-side assembly the specified > env must include a valid SystemRoot." Also, in order to execute in any installation environment where libraries are found in non-default locations, you will need to set LD_LIBRARY_PATH. Oh, and you will also need to set $PATH on UNIX so that libraries can find their helper programs and %PATH% on Windows so that any compiled dynamically-loadable modules and/or DLLs can be loaded. And by the way you will also need to relay DYLD_LIBRARY_PATH if you did a UNIX-style build on OS X, not LD_LIBRARY_PATH. Don't forget that you probably also need PYTHONPATH to make sure any subprocess environments can import the same modules as their parent. Not to mention SSH_AUTH_SOCK if your application requires access to _remote_ process spawning, rather than just local. Oh and DISPLAY in case your subprocesses need GUI support from an X11 program (which sometimes you need just to initialize certain libraries which don't actually do anything with a GUI). Oh and __CF_USER_TEXT_ENCODING is important sometimes too, don't forget that. And if your subprocess is in Perl or Ruby or Java you may need a couple dozen other variables which your deployment environment has set for you too. Did I mention CFLAGS or LC_ALL yet? Let me tell you a story about this one HP/UX machine... Ahem. Bottom line: it seems like screwing with the process spawning environment to make it minimal is a good idea for simplicity, for security, and for modularity. But take it from me, it isn't. I guarantee you that you don't actually know what is in your operating system's environment, and initializing it is a complicated many-step dance which some vendor or sysadmin or product integrator figured out how to do much better than your hapless Python program can. %SystemRoot% is just the tip of a very big, very nasty iceberg. Better not to keep refining why exactly it's required, or someone will eventually be adding a new variable (starting with %APPDATA% and %HOMEPATH%) that can magically cause your subprocess not to spawn properly to this page every six months for eternity. If you're spawning processes as a regular user, you should just take the environment you're given, perhaps with a few specific light additions whose meaning you understand. If you're spawning a process as an administrator or root, you should probably initialize the environment for the user you want to spawn that process as using an OS-specific mechanism like login(1). (Sorry that I don't know the Windows equivalent.) -glyph From g.brandl at gmx.net Thu Mar 22 20:45:17 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 22 Mar 2012 20:45:17 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> Message-ID: On 22.03.2012 20:05, Russell E. Owen wrote: >> I like the overall design, but one thing seems to be missing is an >> overview of what Python is (hence what the page is about). Naturally we >> don't need that, but a one-line overview with a link to more information >> would be helpful. >> >> -- Russell > > > I'm afraid my last sentence was incoherent. I meant to say: > > Naturally we, as Python users, don't need that; but a one-line overview > with a link to more information would be helpful to others who are not > familiar with the language. Hi Russell, note that the page is not supposed to replace python.org, but is just a new styling of the Python documentation, docs.python.org, where it is kind of assumed that you know what Python is... Georg From g.brandl at gmx.net Thu Mar 22 20:46:36 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 22 Mar 2012 20:46:36 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: Message-ID: On 21.03.2012 23:22, Victor Stinner wrote: >>> http://hg.python.org/cpython/rev/7355550d5357 >>> changeset: 75850:7355550d5357 >>> user: Stefan Krah >>> date: Wed Mar 21 18:25:23 2012 +0100 >>> summary: >>> Issue #7652: Integrate the decimal floating point libmpdec library to speed >>> up the decimal module. Performance gains of the new C implementation are >>> between 12x and 80x, depending on the application. > > Congrats Stefan! And thanks for the huge chunk of code. Seconded. This is the kind of stuff that will make 3.3 the most awesomest 3.x release ever (and hopefully convince people that it does make sense to port)... cheers, Georg From v+python at g.nevcal.com Thu Mar 22 20:50:09 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 22 Mar 2012 12:50:09 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6B5B33.9020406@pearwood.info> References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> Message-ID: <4F6B8271.3030901@g.nevcal.com> On 3/22/2012 10:02 AM, Steven D'Aprano wrote: > > As they say, the 99% who are lousy designers give the rest a bad name. > > *wink* :) > My first impression of this page: > > http://www.python.org/~gbrandl/build/html/index.html > > was that the grey side-bar gives the page a somber, perhaps even > dreary, look. First impressions count, and I'm afraid that first look > didn't work for me. The dark sidebar continued down the whole page, and made it clearer why that space was being wasted at the bottom of long pages... I had never noticed, until doing the side-by-side comparison, that it was possible to collapse the TOC sidebar, but this seems to be true only in the old layout. After looking at both a while, my suggestions would be: 1. Preserve the collapsability of the TOC, but possible enhance its recognizability with an X in the upper right of the TOC sidebar, as well as the << in the middle. 2. Make the header fixed, so that the bread crumb trail at the top is available even after scrolling way down a long page. 3. Make the sidebar separately scrollable, so that it stays visible when scrolling down in the text. This would make it much easier to jump from section to section, if the TOC didn't get lost in the process. I have no particular preferences for colors or background colors, as long as they are reasonably legible. I do have a preference for serif fonts, especially if the font gets small. Can anyone point me to any legibility studies that show any font being more legible, more easily readable, than Times Roman? (And yes, I know a lot of people that dislike Times Roman, and none of them ever have.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Mar 22 21:57:18 2012 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2012 07:57:18 +1100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6B8271.3030901@g.nevcal.com> References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> <4F6B8271.3030901@g.nevcal.com> Message-ID: On Fri, Mar 23, 2012 at 6:50 AM, Glenn Linderman wrote: > 3. Make the sidebar separately scrollable, so that it stays visible when > scrolling down in the text.? This would make it much easier to jump from > section to section, if the TOC didn't get lost in the process. -1. The downside of separate scrolling is that you lose the ability to scroll-wheel the main docs if your mouse is over the TOC. I'd rather it stay as a single page, so it doesn't matter where my pointer is when I scroll. But I realise I'm bikeshedding this a bit. Bottom line: I'll use Python, and docs.python.org, regardless of the design and layout... so I'll let the expert(s) decide this one. ChrisA From tjreedy at udel.edu Thu Mar 22 22:13:27 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 22 Mar 2012 17:13:27 -0400 Subject: [Python-Dev] PendingDeprecationWarning Message-ID: My impression is that the original reason for PendingDeprecationWarning versus DeprecationWarning was to be off by default until the last release before removal. But having DeprecationWarnings on by default was found to be too obnoxious and it too is off by default. So do we still need PendingDeprecationWarnings? My impression is that it is mostly not used, as it is a nuisance to remember to change from one to the other. The deprecation message can always indicate the planned removal time. I searched the Developer's Guide for both deprecation and DeprecationWarning and found nothing. -- Terry Jan Reedy From guido at python.org Thu Mar 22 22:14:54 2012 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Mar 2012 14:14:54 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> <4F6B8271.3030901@g.nevcal.com> Message-ID: Georg, please start a new thread when you have a new design for review. I'm muting this one... -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Thu Mar 22 22:18:35 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 22 Mar 2012 17:18:35 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> <4F6B8271.3030901@g.nevcal.com> Message-ID: <20120322211850.512872500E3@webabinitio.net> On Fri, 23 Mar 2012 07:57:18 +1100, Chris Angelico wrote: > On Fri, Mar 23, 2012 at 6:50 AM, Glenn Linderman wrote: > > 3. Make the sidebar separately scrollable, so that it stays visible when > > scrolling down in the text.?? This would make it much easier to jump from > > section to section, if the TOC didn't get lost in the process. > > -1. The downside of separate scrolling is that you lose the ability to > scroll-wheel the main docs if your mouse is over the TOC. I'd rather > it stay as a single page, so it doesn't matter where my pointer is > when I scroll. I agree, and I don't use a mousewheel much. I use pentadactyl, and in that case sometimes the keyboard focus ends up in the TOC and then the scrolling keys scroll the wrong thing (or appear to do nothing, since such things are generally shorter than my screen...) and I can never remember the key sequence to change the focus, because *most* sites I don't have that problem with. For whatever that's worth :) --David From ethan at stoneleaf.us Thu Mar 22 22:18:56 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 22 Mar 2012 14:18:56 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6B8271.3030901@g.nevcal.com> References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> <4F6B8271.3030901@g.nevcal.com> Message-ID: <4F6B9740.1060302@stoneleaf.us> Glenn Linderman wrote: > After looking at both a while, my suggestions would be: > > 1. Preserve the collapsability of the TOC, but possible enhance its > recognizability with an X in the upper right of the TOC sidebar, as well > as the << in the middle. > > 2. Make the header fixed, so that the bread crumb trail at the top is > available even after scrolling way down a long page. > > 3. Make the sidebar separately scrollable, so that it stays visible when > scrolling down in the text. This would make it much easier to jump from > section to section, if the TOC didn't get lost in the process. +1 +1 and, of course, +1 Having to go clear back to the top is a pain, and I never knew `til now that the sidebar was collapsible. ~Ethan~ From amauryfa at gmail.com Thu Mar 22 22:44:44 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 22 Mar 2012 22:44:44 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: Message-ID: 2012/3/22 Georg Brandl : >> Congrats Stefan! And thanks for the huge chunk of code. > > Seconded. ?This is the kind of stuff that will make 3.3 the most awesomest > 3.x release ever (and hopefully convince people that it does make sense to > port)... On the other hand, porting PyPy to 3.3 will be more work ;-) Fortunately the libmpdec directory should be reusable as is. Nice work! -- Amaury Forgeot d'Arc From lukasz at langa.pl Thu Mar 22 22:49:11 2012 From: lukasz at langa.pl (=?iso-8859-2?Q?=A3ukasz_Langa?=) Date: Thu, 22 Mar 2012 22:49:11 +0100 Subject: [Python-Dev] PendingDeprecationWarning In-Reply-To: References: Message-ID: Wiadomo?? napisana przez Terry Reedy w dniu 22 mar 2012, o godz. 22:13: > My impression is that the original reason for PendingDeprecationWarning versus DeprecationWarning was to be off by default until the last release before removal. But having DeprecationWarnings on by default was found to be too obnoxious and it too is off by default. So do we still need PendingDeprecationWarnings? It is also my understanding that DeprecationWarnings have been effectively made to behave like PendingDeprecationWarnings. This makes the latter redundant. Should we raise DeprecationWarnings upon usage of PendingDeprecationWarnings? ;) -- Best regards, ?ukasz Langa Senior Systems Architecture Engineer IT Infrastructure Department Grupa Allegro Sp. z o.o. From lukasz at langa.pl Thu Mar 22 22:44:44 2012 From: lukasz at langa.pl (=?iso-8859-2?Q?=A3ukasz_Langa?=) Date: Thu, 22 Mar 2012 22:44:44 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6B9740.1060302@stoneleaf.us> References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> <4F6B8271.3030901@g.nevcal.com> <4F6B9740.1060302@stoneleaf.us> Message-ID: Wiadomo?? napisana przez Ethan Furman w dniu 22 mar 2012, o godz. 22:18: > Glenn Linderman wrote: >> After looking at both a while, my suggestions would be: >> 1. Preserve the collapsability of the TOC, but possible enhance its recognizability with an X in the upper right of the TOC sidebar, as well as the << in the middle. >> 2. Make the header fixed, so that the bread crumb trail at the top is available even after scrolling way down a long page. >> 3. Make the sidebar separately scrollable, so that it stays visible when scrolling down in the text. This would make it much easier to jump from section to section, if the TOC didn't get lost in the process. > > +1 > +1 > and, of course, > +1 Something like that: http://packages.python.org/lck.django/lck.django.tags.models.html ? (breadcrumbs are at the bottom) -- Best regards, ?ukasz Langa Senior Systems Architecture Engineer IT Infrastructure Department Grupa Allegro Sp. z o.o. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Mar 22 23:56:10 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 23 Mar 2012 11:56:10 +1300 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6B5B33.9020406@pearwood.info> References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> Message-ID: <4F6BAE0A.4080108@canterbury.ac.nz> Can we please get rid of the sidebar, or at least provide a way of turning it off? I don't think it's anywhere near useful enough to be worth the space it takes up. You can only use it when you're scrolled to the top of the page, otherwise it's just a useless empty space. Also, I often want to put the documentation side by side with the code I'm working on, and having about a quarter to a third of the horizontal space taken up with junk makes that much more awkward than it needs to be. A table of contents as a separate page is a lot more usable for me. I can keep it open in a browser tab and switch to it when I want to look at it. Most of the time I don't want to look at it and don't want it taking up space on the page. Also I agree about the grey text being suboptimal. Deliberately throwing away contrast, especially for the main body text, is insane. -- Greg From mhammond at skippinet.com.au Thu Mar 22 23:56:27 2012 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 23 Mar 2012 09:56:27 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: <4F6B3B6E.2000402@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> Message-ID: <4F6BAE1B.8080005@skippinet.com.au> I'm responding to both of Van's recent messages in one: On 23/03/2012 1:47 AM, VanL wrote: > [PART 2: Moving the python binary] ... > A regular complaint of those new to Python on windows (and new to > programming generally) has been that one of the first things that > they need to do is to edit the PATH to allow Python to be run. In > particular, this is the difficult sequence: > > 1. Install python. 2. Open up a shell and run "python" 3. Use pip or > easy_install to install regetron (a package that installs an > executable file). 4. Run regetron. ... > One possible response here is that the moving of the python.exe > binary and the setting of the PATH would be tied to an > unchecked-by-default installer option, making an explicit user choice > needed to invoke the new functionality. Given an off-by-default setting, I fail to see how it fixes your "difficult sequence" above. What would the instructions above now say? That the user should re-install Python ensuring to set that checkbox? Cover both cases, including how the user can tell if it is on the PATH and how to fix it otherwise? Something else? > Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden > have all expressed that they have existing tools that would break > and would need to be adjusted to match the new location of the > python.exe, because that location is assumed to be at the root of the > python install. > > A related issue is that this portion of the proposal has met with > some resistance, but not much support here on Python-dev. The reason > for that is selection bias: Those who are on Python-dev are much more > likely to have tools that do advanced things with Python, such as > introspect on the location of the binary, and are also much more > likely to be comfortable with things like editing the PATH on > windows. In contrast, the people that have trouble with this issue > are those that are newest to Python and programming generally - those > for whom editing the PATH is a challenge and whom are likely to be > confused by the distinction between python.exe and a python program - > and why, even after they add python to the path, the python program > is not directly executable. Here you are referring to the PATH, but that isn't really where the objections are. I would claim a selection bias on Python-dev, where subscribers are less likely to use Windows regularly for development and therefore less likely to have developed or use tools for finding and launching Python. IMO, the lack of objections on Python-dev to renaming the binary directory is the same reason you aren't seeing overwhelming *support* for the change either. Without the perspective of being regular Windows users, people are happy to agree "consistent is better". All other things being equal, I'd agree too. Really, we have just one anecdote from you about your process and as Paul says, no attempt to outline other alternatives. For example, couldn't your "activate.bat" add both Scripts *and* bin to the PATH whereas your "activate.sh" adds just "bin"? > I have been running like this for several years across multiple > Python versions, so I have experience with the "breakage" from this > part of the proposal. I have found four packages that would need to > be updated: Pip, virtualenv, PyPM, and Egginst would need 1-2 line > patches. With all due respect, I find this disingenuous. Your lack of experience with the tools that are out there doesn't mean they don't exist and I've already offered a couple of examples. I certainly can't claim to know what most of them are; I expect that I am underestimating them. IMO, your list is a fraction of the tools impacted. > I have these patches, I would/could provide them. Generally these tools have something like: > > if platform == 'win32': > bin_dir = 'Scripts' > else: > bin_dir = 'bin' > > The patches just remove the special casing - bin_dir just gets set to 'bin'. So none of those tools need to work with previous Python versions? But even if what you say is strictly true, I don't think a reasonable response to "but what about backwards compatibility and tool breakage" is "the breakage is simple and the fix is trivial" - the bar has never been that low for changes to the language itself. I don't see why tooling around the language shouldn't be held to any less account. So my summary of the situation is: * There has been *exactly one* concrete case listed that would benefit from this, and I believe that one case can be mitigated by you having 2 directories on the PATH in Windows and one on other platforms. * You yourself listed 4 tools that would need to change to support this. I've listed a further 2, and Paul and Tim both indicated they would be impacted. ActiveState and Enthought haven't been canvassed. I suspect this is the tip of the iceberg - although I concede it is probably a relatively small iceberg :) Like Tim, I wont sulk if you can convince people to make this change anyway, but IMO it is completely clear the costs outweigh the benefits. Thus, if it were my decision to make, it would not happen. Paul and Tim have the same view best I can tell. I think it would be a huge shame if it happens even in the face of these pragmatic objections. Cheers, Mark From van.lindberg at gmail.com Fri Mar 23 00:15:53 2012 From: van.lindberg at gmail.com (VanL) Date: Thu, 22 Mar 2012 18:15:53 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: <4F6BAE1B.8080005@skippinet.com.au> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> <4F6BAE1B.8080005@skippinet.com.au> Message-ID: Another use case was just pointed out to me: making things consistent with buildout. Given a similar use case (create repeatable cross platform environments), they create and use a 'bin' directory for executable files. From ncoghlan at gmail.com Fri Mar 23 00:37:55 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Mar 2012 09:37:55 +1000 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4F6B345C.1020406@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> Message-ID: (resending, only sent to Van the first time) FWIW, I avoid the directory naming problems Van describes entirely by including my "scripts" in the source package and running them with the "-m" switch. So "python -m pulpdist.manage_site", for example, is PulpDist's Django administration client wrapper. I do it that way mainly to get sys.path right automatically, but it avoids several other installed vs checked out differences too. -- Sent from my phone, thus the relative brevity :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Mar 23 00:26:30 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 22 Mar 2012 16:26:30 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: <4F6BAE1B.8080005@skippinet.com.au> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> <4F6BAE1B.8080005@skippinet.com.au> Message-ID: <4F6BB526.7070205@stoneleaf.us> Given the cost of the change, and the advent of the PEP-397 Launcher, I also vote -1. ~Ethan~ From stephen at xemacs.org Fri Mar 23 00:59:28 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 23 Mar 2012 00:59:28 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6BAE0A.4080108@canterbury.ac.nz> References: <4F6A2084.90508@nedbatchelder.com> <4F6A2863.1080200@nedbatchelder.com> <4F6B5B33.9020406@pearwood.info> <4F6BAE0A.4080108@canterbury.ac.nz> Message-ID: On Thu, Mar 22, 2012 at 11:56 PM, Greg Ewing wrote: > Can we please get rid of the sidebar, or at least provide > a way of turning it off? I don't think it's anywhere > near useful enough to be worth the space it takes up. +1. It seems to mostly duplicate the headline next/previous buttons already duplicated in the footer, it doesn't give you the whole TOC, and the whole TOC already present in many nodes. The "Search bar" is a standard feature of most headers (and sometimes footers), and I like the "Report a Bug" link because it confirms to the reader that Python developers actually care what they (readers) think. I guess there is enough room for both of those in the header even after subtractiing the horizontal space for the sidebar. > A table of contents as a separate page is a lot more > usable for me. I agree, but with emphasis on the *for me* part. I suspect this is a personal preference. > Also I agree about the grey text being suboptimal. +1 for black text or a perceptibly darker grey. > Deliberately throwing away contrast, especially for > the main body text, is insane. It does *look* nice, though. Overall, very nice job, Georg! From brian at python.org Fri Mar 23 02:24:19 2012 From: brian at python.org (Brian Curtin) Date: Thu, 22 Mar 2012 20:24:19 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> Message-ID: On Thu, Mar 22, 2012 at 12:59, Paul Moore wrote: > Note - that is not "Regularizing the layout". You have not made any > changes to OS/2 (which matches Windows at the moment). I think that would be a wasted effort with OS/2 entering "unsupported" mode in 3.3, and OS/2 specific code being removed in 3.4. From brian at python.org Fri Mar 23 03:26:36 2012 From: brian at python.org (Brian Curtin) Date: Thu, 22 Mar 2012 21:26:36 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4F6B760D.3010004@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> Message-ID: On Thu, Mar 22, 2012 at 13:57, VanL wrote: > Honestly, I didn't expect that much resistance. None of the people I talked > to in person even cared, or if they did, they thought that consistency was a > benefit. But now that virtualenvs are going in in 3.3, I see this as the > last good chance to change this. I was one of these people, first finding out just about the Scripts/bin change, and my thought was JFDI. The rest of it seems fine to me - I say let's go for it. >> Personally, my main concerns are around procedure and policy. The more the >> discussion goes on, the more I feel that there should be a PEP to capture >> the details of the debate clearly. Too much is getting lost in the noise. >> And I think you should provide a clear statement of why this issue is >> important enough to justify violating the backward compatibility policies. >> As Mark said (I think it was Mark...) if this had been proposed for 3.0, it >> would have been fine. Now we're at 3.2 with 3.3 close to release, and it >> just seems too late to be worth the risk. One plus point about your posting >> this separately. It's made me think through the issue in a bit more detail, >> and I'm now a solid -1 on the proposal. > > > I have been trying at various PyCons and in various conversations to move > this for years. No one cares. The current urgency is driven by pyvenv - > changes now will be much, much easier than changes later. > > Again, I am happy to write a PEP. If I were to summarize (on this issue > only): > > 1. The current backwards compatibility hit is minimal; I would be happy to > contact and provide patches to the four packages I have found (and anyone > else who wants one). Backwards compatibility in the future will probably be > harder to deal with. > 2. There are advantages to cross-platform consistency and to > virtualenv-based development. I believe that these will grow in the future. > 3. Most people won't care. To the extent that people notice, I think they > will appreciate the consistency. The virtualenv point, to me, is a strong one. I think we have an opportunity right now to make an adjustment, otherwise we're locked in again. From brian at python.org Fri Mar 23 04:20:12 2012 From: brian at python.org (Brian Curtin) Date: Thu, 22 Mar 2012 22:20:12 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: <4F6B3B6E.2000402@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> Message-ID: 2012/3/22 VanL : > Open Issues: > > """If we do put python.exe on PATH (whether it's in bin or not), we have > to debate how to handle people having multiple versions of python on > their machine. In a post-PEP 397 world, no Python is "the machine > default" - .py files are associated with py.exe, not python.exe, so we > have to consider the following 3 commands being run from a shell > prompt: > > ? ?1. myprog.py > ? ?2. py myprog.py > ? ?3. python myprog.py > > 1 and 2 will always do the same thing. However, 3 could easily do > something completely different, if the Python in the #! line differs > from the one found on PATH. To me, this implies that it's better for > (3) to need explicit user action (setting PATH) if it's to do anything > other than give an error. But maybe that's just me. I've been hit too > often by confusion caused by *not* remembering this fact.""" I'm not sure how widely used #1 is. I can't remember coming across any bug reports or posts around the web where the example command line just uses the Python chosen by the file association. I would suspect it's especially rare in the current time when many people are running a lot of versions of Python. Right now I have 2.6, 2.7, 3.1, 3.2, and 3.3, all installed in some different order, and I couldn't tell you which of those I installed the latest bugfix release for. That last one wins the race when it comes to file associations, and I've never paid attention to the installer option. #3 *will* require explicit user action - the Path setting is off by default. For as much as it's an advanced feature, it's really helpful to beginners. If you just want to type in "python" and have it work, the Path option is great. That's not to say the launcher isn't *also* a good thing. If you're a first timer and install Python 3.3 and want to run a tutorial - add Python to the path, type "python", and you're on your way. If you're an advanced user and you want to write and run code on Python 3.3, do the same. If you're even more advanced and are doing multi-version work, the launcher is a helpful alternative. > One possible response here is that the moving of the python.exe binary and > the setting of the PATH would be tied to an unchecked-by-default installer > option, making an explicit user choice needed to invoke the new > functionality. I ended up typing out the above while missing this paragraph...but, bingo. > Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden have > all expressed that they have existing tools that would break and would need > to be adjusted to match the new location of the python.exe, because that > location is assumed to be at the root of the python install. Isn't the proposed "BinaryDir" registry key helpful here? It's not like we're telling people to fend for themselves -- we'll tell you where it's at. > A related issue is that this portion of the proposal has met with some > resistance, but not much support here on Python-dev. The reason for that is > selection bias: Those who are on Python-dev are much more likely to have > tools that do advanced things with Python, such as introspect on the > location of the binary, and are also much more likely to be comfortable with > things like editing the PATH on windows. In contrast, the people that have > trouble with this issue are those that are newest to Python and programming > generally - those for whom editing the PATH is a challenge and whom are > likely to be confused by the distinction between python.exe and a python > program - and why, even after they add python to the path, the python > program is not directly executable. I still don't really get how this portion of the proposal, the python.exe move to bin, is holding people up. If you're using the launcher, the change is invisible. If you're using a setup where bin is on the Path, the change is invisible. File associations? Invisible. If you're typing out the full path, you have to type "bin" in the middle -- this kind of sucks but I think we'll live. I get that tools could be affected. I had two IDE makers at PyCon immediately throw up red flags to this change. I think one of them was about to charge the stage during my talk. When it was mentioned that we could point them to the proper location, they breathed a sigh of relief and said "cool, do it". If a registry key pointing you to python.exe (rather, the directory) right now in Python < 3.3 works, why doesn't another one pointing you to python.exe in Python >= 3.3 work? From brian at python.org Fri Mar 23 04:23:39 2012 From: brian at python.org (Brian Curtin) Date: Thu, 22 Mar 2012 22:23:39 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: <4F6BB526.7070205@stoneleaf.us> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> <4F6BAE1B.8080005@skippinet.com.au> <4F6BB526.7070205@stoneleaf.us> Message-ID: On Thu, Mar 22, 2012 at 18:26, Ethan Furman wrote: > Given the cost of the change, and the advent of the PEP-397 Launcher, I also > vote -1. Can you provide some justification other than a number? It's a pretty cheap change and the launcher solves somewhat of a different problem. From mhammond at skippinet.com.au Fri Mar 23 04:39:25 2012 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 23 Mar 2012 14:39:25 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> Message-ID: <4F6BF06D.10106@skippinet.com.au> [snipped some CCs] On 23/03/2012 2:20 PM, Brian Curtin wrote: ... > I get that tools could be affected. I had two IDE makers at PyCon > immediately throw up red flags to this change. I think one of them was > about to charge the stage during my talk. When it was mentioned that > we could point them to the proper location, they breathed a sigh of > relief and said "cool, do it". If a registry key pointing you to > python.exe (rather, the directory) right now in Python< 3.3 works, > why doesn't another one pointing you to python.exe in Python>= 3.3 > work? It will work. The fact MvL is proposing the conservative approach of landing this in 3.5+ and have 3.3+ include the *new* registry key means I'm willing to reluctantly accept it rather than aggressively oppose it. Tools then have a chance to adapt to the new key. If the proposal moved any faster, existing tools which only use the old key would break without warning. The fact they need to change at all is unfortunate, but the timescale proposed means we can at least say we warned them. Mark From ncoghlan at gmail.com Fri Mar 23 04:48:30 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Mar 2012 13:48:30 +1000 Subject: [Python-Dev] Setting up a RHEL6 buildbot Message-ID: I'm looking into getting a RHEL6 system set up to add to the buildbot fleet. The info already on the wiki [1] is pretty helpful, but does anyone have any suggestions on appropriate CPU/memory/disk allocations? Cheers, Nick. [1] http://wiki.python.org/moin/BuildBot -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From anacrolix at gmail.com Fri Mar 23 06:40:05 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Fri, 23 Mar 2012 13:40:05 +0800 Subject: [Python-Dev] Setting up a RHEL6 buildbot In-Reply-To: References: Message-ID: The 24 core machine at my last workplace could configure and make the tip in 45 seconds from a clean checkout. Lots of cores? :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Fri Mar 23 06:42:53 2012 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Thu, 22 Mar 2012 22:42:53 -0700 (PDT) Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4F6B760D.3010004@gmail.com> Message-ID: <4f6c0d5d.c3d6e00a.2469.ffffde70@mx.google.com> In http://mail.python.org/pipermail/python-dev/2012-March/117953.html VanL wrote: > Paul Moore wrote: >> First of all, this difference is almost entirely *invisible*. Apart >> from possibly setting PATH (once!) users should not be digging around >> in the Python installation directory. Certainly on Windows, it is a >> very unusual application that expects users to even care how the >> application is laid out internally. And I suspect that is also true on >> Unix and Mac. >> Secondly, the layouts are not as similar as you claim here, In fact, of the 8 builtin layout schemes, the only two that are consistent are nt (which you propose to change) and os2. Of the 64 possible values, there are 26 unique values. So I suspect the right answer is just to make it easier for a user to set those 8 values at installation time; in your case, you can tell users what values to use when setting up their virtual environment. (And if you can't do that, then you can add files to your own directories, and if you can't do that, then you can add a post-install hook that renames your directories or moves your files according to the installed values.) > In fact, even the one change > of 'Scripts' to 'bin' everywhere would get 90% of my uses. So is this something that you can control in your recommended virtual environment? Or at least something that you can do with the same script that checks everything out of source control and adds *something* to the path? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From stefan at bytereef.org Fri Mar 23 09:03:15 2012 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 23 Mar 2012 09:03:15 +0100 Subject: [Python-Dev] Setting up a RHEL6 buildbot In-Reply-To: References: Message-ID: <20120323080315.GA16995@sleipnir.bytereef.org> Nick Coghlan wrote: > I'm looking into getting a RHEL6 system set up to add to the buildbot > fleet. The info already on the wiki [1] is pretty helpful, but does > anyone have any suggestions on appropriate CPU/memory/disk > allocations? The Fedora bot has been running ultra-stable under qemu with 512MB allocated for the VM. I've never limited CPU or disk space. On an i7 quad core with 8GB of memory, I've been running two buildbot VMs, four deccheck processes at 100% CPU and a web server without any kind of noticable performance degradation. Stefan Krah From p.f.moore at gmail.com Fri Mar 23 09:10:13 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Mar 2012 08:10:13 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> Message-ID: On 23 March 2012 03:20, Brian Curtin wrote: >> Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden have >> all expressed that they have existing tools that would break and would need >> to be adjusted to match the new location of the python.exe, because that >> location is assumed to be at the root of the python install. > > Isn't the proposed "BinaryDir" registry key helpful here? It's not > like we're telling people to fend for themselves -- we'll tell you > where it's at. It won't help me much. I either check a key and fall back on the old method, or check in bin and fall back on the old method. No major difference. The key is slightly worse, as I'm already looking in the filesystem, so why open a registry key, but it's mostly irrelevant. > I still don't really get how this portion of the proposal, the > python.exe move to bin, is holding people up. If you're using the > launcher, the change is invisible. If you're using a setup where bin > is on the Path, the change is invisible. File associations? Invisible. > If you're typing out the full path, you have to type "bin" in the > middle -- this kind of sucks but I think we'll live. Agreed, it's irrelevant for end users. It's only going to affect tools. Paul. From stefan at bytereef.org Fri Mar 23 09:26:31 2012 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 23 Mar 2012 09:26:31 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: Message-ID: <20120323082631.GA17050@sleipnir.bytereef.org> Dirkjan Ochtman wrote: > As a packager, is the libmpdec library used elsewhere? For Gentoo, we > generally prefer to package libraries separately and have Python > depend on them. From the site, it seems like you more or less wrote > libmpdec for usage in Python, but if it's general-purpose and actually > used in other software, it would be nice if Python grew a configure > option to make it use the system libmpdec. libmpdec was actually written before the module. It's general purpose and it fully implements the specification. I'm only aware of in-house usage. Someone has tried to submit a libmpdec package to OpenSUSE, but it was rejected with the claim that there already exists a package with the name. I think the claim is false: There is a libmpcdec package, where "cdec" presumably stands for "codec". I'll add the --with-system-libmpdec option with the caveat that changes will probably make it first into the libmpdec shipped with Python, see also: http://bugs.python.org/issue7652#msg155744 On the bright side, I don't expect many changes, since the specification is stable. Stefan Krah From dirkjan at ochtman.nl Fri Mar 23 09:41:37 2012 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Fri, 23 Mar 2012 09:41:37 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: <20120323082631.GA17050@sleipnir.bytereef.org> References: <20120323082631.GA17050@sleipnir.bytereef.org> Message-ID: On Fri, Mar 23, 2012 at 09:26, Stefan Krah wrote: > I'll add the --with-system-libmpdec option with the caveat that > changes will probably make it first into the libmpdec shipped > with Python, see also: > > http://bugs.python.org/issue7652#msg155744 Sounds good, thanks! Cheers, Dirkjan From stefan at bytereef.org Fri Mar 23 10:22:55 2012 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 23 Mar 2012 10:22:55 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: Message-ID: <20120323092255.GA17205@sleipnir.bytereef.org> Georg Brandl wrote: > >>> Issue #7652: Integrate the decimal floating point libmpdec library to speed > >>> up the decimal module. Performance gains of the new C implementation are > >>> between 12x and 80x, depending on the application. > > > > Congrats Stefan! And thanks for the huge chunk of code. > > Seconded. This is the kind of stuff that will make 3.3 the most awesomest > 3.x release ever (and hopefully convince people that it does make sense to > port)... Thanks! For cdecimal specifically I have the impression that 3.x is already used in the financial community, where web framework dependencies aren't an issue. On the web side, there seems to be a huge interest in speeding up database accesses, so let me evangelize again: Database applications using decimal will run 12x faster in 3.3. Stefan Krah From victor.stinner at gmail.com Fri Mar 23 10:28:44 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 23 Mar 2012 10:28:44 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: <20120323092255.GA17205@sleipnir.bytereef.org> References: <20120323092255.GA17205@sleipnir.bytereef.org> Message-ID: By the way, how much faster is cdecimal? 72x or 80x? http://docs.python.org/dev/whatsnew/3.3.html#decimal Victor 2012/3/23 Stefan Krah : > Georg Brandl wrote: >> >>> ?Issue #7652: Integrate the decimal floating point libmpdec library to speed >> >>> up the decimal module. Performance gains of the new C implementation are >> >>> between 12x and 80x, depending on the application. >> > >> > Congrats Stefan! And thanks for the huge chunk of code. >> >> Seconded. ?This is the kind of stuff that will make 3.3 the most awesomest >> 3.x release ever (and hopefully convince people that it does make sense to >> port)... > > Thanks! For cdecimal specifically I have the impression that 3.x is already > used in the financial community, where web framework dependencies aren't an > issue. > > On the web side, there seems to be a huge interest in speeding up database > accesses, so let me evangelize again: Database applications using decimal > will run 12x faster in 3.3. > > > Stefan Krah > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From stefan at bytereef.org Fri Mar 23 10:30:33 2012 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 23 Mar 2012 10:30:33 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: Message-ID: <20120323093033.GB17205@sleipnir.bytereef.org> Amaury Forgeot d'Arc wrote: > > Seconded. ?This is the kind of stuff that will make 3.3 the most awesomest > > 3.x release ever (and hopefully convince people that it does make sense to > > port)... > > On the other hand, porting PyPy to 3.3 will be more work ;-) We've got to keep you on your toes, don't we? :) > Fortunately the libmpdec directory should be reusable as is. > Nice work! Thanks, also for helping out with the MutableMapping context. Stefan Krah From eliben at gmail.com Fri Mar 23 10:51:10 2012 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 23 Mar 2012 11:51:10 +0200 Subject: [Python-Dev] PEP 411 - request for pronouncement Message-ID: Hello, PEP 411 -- Provisional packages in the Python standard library Has been updated with all accumulated feedback from list discussions. Here it is: http://www.python.org/dev/peps/pep-0411/ (the text is also pasted in the bottom of this email). The PEP received mostly positive feedback. The only undecided point is where to specify that the package is provisional. Currently the PEP mandates to specify it in the documentation and in the docstring. Other suggestions were to put it in the code, either as a __provisional__ attribute on the module, or collect all such modules in a single sys.provisional list. According to http://blog.python.org/2012/03/2012-language-summit-report.html, the PEP was discussed in the language summit and overall viewed positively, although no final decision has been reached. ISTM a decision needs to be taken, which is why I request pronouncement, with a recommendation on the requirement the PEP should make of provisional modules (process details). Eli -------------------------------- PEP: 411 Title: Provisional packages in the Python standard library Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Eli Bendersky Status: Draft Type: Informational Content-Type: text/x-rst Created: 2012-02-10 Python-Version: 3.3 Post-History: 2012-02-10 Abstract ======== The process of including a new package into the Python standard library is hindered by the API lock-in and promise of backward compatibility implied by a package being formally part of Python. This PEP describes a methodology for marking a standard library package "provisional" for the period of a single minor release. A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded. Proposal - a documented provisional state ========================================= Whenever the Python core development team decides that a new package should be included into the standard library, but isn't entirely sure about whether the package's API is optimal, the package can be included and marked as "provisional". In the next minor release, the package may either be "graduated" into a normal "stable" state in the standard library, remain in provisional state, or be rejected and removed entirely from the Python source tree. If the package ends up graduating into the stable state after being provisional, its API may be changed according to accumulated feedback. The core development team explicitly makes no guarantees about API stability and backward compatibility of provisional packages. Marking a package provisional ----------------------------- A package will be marked provisional by a notice in its documentation page and its docstring. The following paragraph will be added as a note at the top of the documentation page: The package has been included in the standard library on a provisional basis. Backwards incompatible changes (up to and including removal of the package) may occur if deemed necessary by the core developers. The phrase "provisional basis" will then be a link to the glossary term "provisional package", defined as: A provisional package is one which has been deliberately excluded from the standard library's normal backwards compatibility guarantees. While major changes to such packages are not expected, as long as they are marked provisional, backwards incompatible changes (up to and including removal of the package) may occur if deemed necessary by core developers. Such changes will not be made gratuitously - they will occur only if serious flaws are uncovered that were missed prior to the inclusion of the package. This process allows the standard library to continue to evolve over time, without locking in problematic design errors for extended periods of time. See PEP 411 for more details. The following will be added to the start of the packages's docstring: The API of this package is currently provisional. Refer to the documentation for details. Moving a package from the provisional to the stable state simply implies removing these notes from its documentation page and docstring. Which packages should go through the provisional state ------------------------------------------------------ We expect most packages proposed for addition into the Python standard library to go through a minor release in the provisional state. There may, however, be some exceptions, such as packages that use a pre-defined API (for example ``lzma``, which generally follows the API of the existing ``bz2`` package), or packages with an API that has wide acceptance in the Python development community. In any case, packages that are proposed to be added to the standard library, whether via the provisional state or directly, must fulfill the acceptance conditions set by PEP 2. Criteria for "graduation" ------------------------- In principle, most provisional packages should eventually graduate to the stable standard library. Some reasons for not graduating are: * The package may prove to be unstable or fragile, without sufficient developer support to maintain it. * A much better alternative package may be found during the preview release. Essentially, the decision will be made by the core developers on a per-case basis. The point to emphasize here is that a package's inclusion in the standard library as "provisional" in some release does not guarantee it will continue being part of Python in the next release. Rationale ========= Benefits for the core development team -------------------------------------- Currently, the core developers are really reluctant to add new interfaces to the standard library. This is because as soon as they're published in a release, API design mistakes get locked in due to backward compatibility concerns. By gating all major API additions through some kind of a provisional mechanism for a full release, we get one full release cycle of community feedback before we lock in the APIs with our standard backward compatibility guarantee. We can also start integrating provisional packages with the rest of the standard library early, so long as we make it clear to packagers that the provisional packages should not be considered optional. The only difference between provisional APIs and the rest of the standard library is that provisional APIs are explicitly exempted from the usual backward compatibility guarantees. Benefits for end users ---------------------- For future end users, the broadest benefit lies in a better "out-of-the-box" experience - rather than being told "oh, the standard library tools for task X are horrible, download this 3rd party library instead", those superior tools are more likely to be just be an import away. For environments where developers are required to conduct due diligence on their upstream dependencies (severely harming the cost-effectiveness of, or even ruling out entirely, much of the material on PyPI), the key benefit lies in ensuring that all packages in the provisional state are clearly under python-dev's aegis from at least the following perspectives: * Licensing: Redistributed by the PSF under a Contributor Licensing Agreement. * Documentation: The documentation of the package is published and organized via the standard Python documentation tools (i.e. ReST source, output generated with Sphinx and published on http://docs.python.org). * Testing: The package test suites are run on the python.org buildbot fleet and results published via http://www.python.org/dev/buildbot. * Issue management: Bugs and feature requests are handled on http://bugs.python.org * Source control: The master repository for the software is published on http://hg.python.org. Candidates for provisional inclusion into the standard library ============================================================== For Python 3.3, there are a number of clear current candidates: * ``regex`` (http://pypi.python.org/pypi/regex) - approved by Guido [#]_. * ``daemon`` (PEP 3143) * ``ipaddr`` (PEP 3144) Other possible future use cases include: * Improved HTTP modules (e.g. ``requests``) * HTML 5 parsing support (e.g. ``html5lib``) * Improved URL/URI/IRI parsing * A standard image API (PEP 368) * Improved encapsulation of import state (PEP 406) * Standard event loop API (PEP 3153) * A binary version of WSGI for Python 3 (e.g. PEP 444) * Generic function support (e.g. ``simplegeneric``) Rejected alternatives and variations ==================================== See PEP 408. References ========== .. [#] http://mail.python.org/pipermail/python-dev/2012-January/115962.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From stefan at bytereef.org Fri Mar 23 11:40:05 2012 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 23 Mar 2012 11:40:05 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: <20120323092255.GA17205@sleipnir.bytereef.org> Message-ID: <20120323104005.GA17581@sleipnir.bytereef.org> Victor Stinner wrote: > By the way, how much faster is cdecimal? 72x or 80x? > http://docs.python.org/dev/whatsnew/3.3.html#decimal It really depends on the precision. Also, the performance of decimal.py depends on many other things in the Python tree, so it easily changes +-10%. Currently, decimal.py seems to be 10% faster than in 3.2, maybe because of the new string representation. The 80x is a ballpark figure for the maximum expected speedup for standard numerical floating point applications. factorial(1000) is 219x faster in _decimal, and with increasing precision the difference gets larger and larger. For huge numbers _decimal is also faster than int: factorial(1000000): _decimal, calculation time: 6.844487905502319 _decimal, tostr(): 0.033592939376831055 int, calculation time: 17.96010398864746 int, tostr(): ... still running ... Stefan Krah From mal at egenix.com Fri Mar 23 11:40:47 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 23 Mar 2012 11:40:47 +0100 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4F6B345C.1020406@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> Message-ID: <4F6C532F.7070208@egenix.com> VanL wrote: > As this has been brought up a couple times in this subthread, I figured that I would lay out the > rationale here. > > There are two proposals on the table: 1) Regularize the install layout, and 2) move the python > binary to the binaries directory. This email will deal with the first, and a second email will deal > with the second. > > 1) Regularizing the install layout: > > One of Python's strengths is its cross-platform appeal. Carefully- > written Python programs are frequently portable between operating > systems and Python implementations with very few changes. Over the > years, substantial effort has been put into maintaining platform > parity and providing consistent interfaces to available functionality, > even when different underlying implementations are necessary (such > as with ntpath and posixpath). > > One place where Python is unnecessarily different, however, is in > the layout and organization of the Python environment. This is most > visible in the name of the directory for binaries on the Windows platform ("Scripts") versus the > name of the directory for binaries on every other platform ("bin"), but a full listing of the > layouts shows > substantial differences in layout and capitalization across platforms. > Sometimes the include is capitalized ("Include"), and sometimes not; and > the python version may or may not be included in the path to the > standard library or not. > > This may seem like a harmless inconsistency, and if that were all it was, I wouldn't care. (That > said, cross-platform consistency is its own good). But it becomes a real pain when combined with > tools like virtualenv or the new pyvenv to create cross-platform development environments. > > In particular, I regularly do development on both Windows and a Mac, and then deploy on Linux. I do > this in virtualenvs, so that I have a controlled and regular environment. I keep them in sync using > source control. > > The problem comes when I have executable scripts that I want to include in my dvcs - I can't have it > in the obvious place - the binaries directory - because *the name of the directory changes when you > move between platforms.* More concretely, I can't hg add "Scripts/runner.py? on my windows > environment (where it is put in the PATH by virtualenv) and thendo a pull on Mac or Linux and have > it end up properly in "bin/runner.py" which is the correct PATH for those platforms. > > This applies anytime there are executable scripts that you want to manage using source control > across platforms. Django projects regularly have these, and I suspect we will be seeing more of this > with the new "project" support in virtualenvwrapper. > > While a few people have wondered why I would want this -- hopefully answered above -- I have not > heard any opposition to this part of the proposal. > > This first proposal is just to make the names of the directories match across platforms. There are > six keys defined in the installer files (sysconfig.cfg and distutils.command.install): 'stdlib', > 'purelib', 'platlib', 'headers', 'scripts', and 'data'. > > Currently on Windows, there are two different layouts defined: > > 'nt': { > 'stdlib': '{base}/Lib', > 'platstdlib': '{base}/Lib', > 'purelib': '{base}/Lib/site-packages', > 'platlib': '{base}/Lib/site-packages', > 'include': '{base}/Include', > 'platinclude': '{base}/Include', > 'scripts': '{base}/Scripts', > 'data' : '{base}', > }, > > 'nt_user': { > 'stdlib': '{userbase}/Python{py_version_nodot}', > 'platstdlib': '{userbase}/Python{py_version_nodot}', > 'purelib': '{userbase}/Python{py_version_nodot}/site-packages', > 'platlib': '{userbase}/Python{py_version_nodot}/site-packages', > 'include': '{userbase}/Python{py_version_nodot}/Include', > 'scripts': '{userbase}/Scripts', > 'data' : '{userbase}', > }, > > > The proposal is to make all the layouts change to: > > 'nt': { > 'stdlib': '{base}/lib', > 'platstdlib': '{base}/lib', > 'purelib': '{base}/lib/site-packages', > 'platlib': '{base}/lib/site-packages', > 'include': '{base}/include', > 'platinclude': '{base}/include', > 'scripts': '{base}/bin', > 'data' : '{base}', > }, > > The change here is that 'Scripts' will change to 'bin' and the capitalization will be removed. Also, > "user installs" of Python will have the same internal layout as "system installs" of Python. This > will also, not coincidentally, match the install layout for posix, at least with regard to the > 'bin', 'lib', and 'include' directories. > > Again, I have not heard *anyone* objecting to this part of the proposal as it is laid out here. > (Paul had a concern with the lib directory earlier, but he said he was ok with the above). > > Please let me know if you have any problems or concerns with this part 1. Since userbase will usually be a single directory in the home dir of a user, the above would loose the possibility to support multiple Python versions in that directory. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 23 2012) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2012-04-03: Python Meeting Duesseldorf 11 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mhammond at skippinet.com.au Fri Mar 23 12:03:02 2012 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 23 Mar 2012 22:03:02 +1100 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> Message-ID: <4F6C5866.2010902@skippinet.com.au> On 23/03/2012 7:10 PM, Paul Moore wrote: > On 23 March 2012 03:20, Brian Curtin wrote: >>> Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden have >>> all expressed that they have existing tools that would break and would need >>> to be adjusted to match the new location of the python.exe, because that >>> location is assumed to be at the root of the python install. >> >> Isn't the proposed "BinaryDir" registry key helpful here? It's not >> like we're telling people to fend for themselves -- we'll tell you >> where it's at. > > It won't help me much. I either check a key and fall back on the old > method, or check in bin and fall back on the old method. No major > difference. The key is slightly worse, as I'm already looking in the > filesystem, so why open a registry key, but it's mostly irrelevant. That's a really good point. On reflection, the 2 tools I've been using as examples are already sniffing around the file-system relative to the install path, looking in the root and the PCBuild directories. The simplest approach for these tools to take is to simply sniff the bin directory too - so they are unlikely to refer to the BinaryDir key at all. Mark From solipsis at pitrou.net Fri Mar 23 12:10:19 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Mar 2012 12:10:19 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed References: <20120323092255.GA17205@sleipnir.bytereef.org> Message-ID: <20120323121019.580f381b@pitrou.net> On Fri, 23 Mar 2012 10:22:55 +0100 Stefan Krah wrote: > Georg Brandl wrote: > > >>> Issue #7652: Integrate the decimal floating point libmpdec library to speed > > >>> up the decimal module. Performance gains of the new C implementation are > > >>> between 12x and 80x, depending on the application. > > > > > > Congrats Stefan! And thanks for the huge chunk of code. > > > > Seconded. This is the kind of stuff that will make 3.3 the most awesomest > > 3.x release ever (and hopefully convince people that it does make sense to > > port)... > > Thanks! For cdecimal specifically I have the impression that 3.x is already > used in the financial community, where web framework dependencies aren't an > issue. > > On the web side, there seems to be a huge interest in speeding up database > accesses, so let me evangelize again: Database applications using decimal > will run 12x faster in 3.3. Are you sure it isn't 12.5x ? Regards Antoine. From solipsis at pitrou.net Fri Mar 23 12:12:05 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Mar 2012 12:12:05 +0100 Subject: [Python-Dev] PendingDeprecationWarning References: Message-ID: <20120323121205.6c8d8508@pitrou.net> On Thu, 22 Mar 2012 17:13:27 -0400 Terry Reedy wrote: > My impression is that the original reason for PendingDeprecationWarning > versus DeprecationWarning was to be off by default until the last > release before removal. But having DeprecationWarnings on by default was > found to be too obnoxious and it too is off by default. So do we still > need PendingDeprecationWarnings? My impression is that it is mostly not > used, as it is a nuisance to remember to change from one to the other. > The deprecation message can always indicate the planned removal time. I > searched the Developer's Guide for both deprecation and > DeprecationWarning and found nothing. Warnings are not only for us, they are for third-party libraries. In this case it seems removing PendingDeprecationWarning would be a pointless nuisance. Regards Antoine. From solipsis at pitrou.net Fri Mar 23 12:13:49 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Mar 2012 12:13:49 +0100 Subject: [Python-Dev] Setting up a RHEL6 buildbot References: Message-ID: <20120323121349.63abf8ea@pitrou.net> On Fri, 23 Mar 2012 13:48:30 +1000 Nick Coghlan wrote: > I'm looking into getting a RHEL6 system set up to add to the buildbot > fleet. The info already on the wiki [1] is pretty helpful, but does > anyone have any suggestions on appropriate CPU/memory/disk > allocations? One or two cores is enough, unless you want to allow for multiple builds in parallel (in which case do say so :-)). You'll need quite a bit of disk space though. At least 20 or 30 GB to err on the safe side, IMO. Regards Antoine. From stefan at bytereef.org Fri Mar 23 12:47:02 2012 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 23 Mar 2012 12:47:02 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: <20120323121019.580f381b@pitrou.net> References: <20120323092255.GA17205@sleipnir.bytereef.org> <20120323121019.580f381b@pitrou.net> Message-ID: <20120323114702.GA17859@sleipnir.bytereef.org> Antoine Pitrou wrote: > > On the web side, there seems to be a huge interest in speeding up database > > accesses, so let me evangelize again: Database applications using decimal > > will run 12x faster in 3.3. > > Are you sure it isn't 12.5x ? Well, that was marketing for 3.3. Stefan Krah From p.f.moore at gmail.com Fri Mar 23 14:37:42 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Mar 2012 13:37:42 +0000 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> <4F6BAE1B.8080005@skippinet.com.au> Message-ID: On 22 March 2012 23:15, VanL wrote: > Another use case was just pointed out to me: making things consistent with buildout. Given a similar use > case (create repeatable cross platform environments), they create and use a 'bin' directory for executable files. Another problem case: cx_Freeze. This currently breaks when installed in a viretualenv, as it locates the "Scripts" directory by appending "Scripts" to the directory of the python executable. So the proposed change *will* break cx_Freeze. The Scripts->bin change will also break it. Paul. PS Yes, I need to report the existing bug. The point remains, however... From pje at telecommunity.com Fri Mar 23 17:39:41 2012 From: pje at telecommunity.com (PJ Eby) Date: Fri, 23 Mar 2012 12:39:41 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4F6B760D.3010004@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> Message-ID: On Mar 22, 2012 2:57 PM, "VanL" wrote: > That said, while I think that the above is a good idea, my personal ambitions are more modest: If the names of the top-level directories only were changed to 'bin', 'lib', and 'include' - never mind differences under 'lib' - I would be happy. In fact, even the one change of 'Scripts' to 'bin' everywhere would get 90% of my uses. Why don't you just install your scripts to 'bin' everywhere then, and add the bin directory to the path on Windows? Distutils allows you to customize your install target for scripts, as does setuptools. Why do you need *Python's own default* to change, to support your preference for using a bin directory? Even if you are using tools that don't use distutils' configuration settings for these directories, why not simply fix those tools so that they do? -------------- next part -------------- An HTML attachment was scrubbed... URL: From zooko at zooko.com Fri Mar 23 17:55:17 2012 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Fri, 23 Mar 2012 10:55:17 -0600 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <4F6137EF.9000000@gmail.com> References: <4F6137EF.9000000@gmail.com> Message-ID: > I merged the two functions into one function: time.steady(strict=False). > > time.steady() should be monotonic most of the time, but may use a fallback. > > time.steady(strict=True) fails with OSError or NotImplementedError if > reading the monotonic clock failed or if no monotonic clock is available. If someone wants time.steady(strict=False), then why don't they just continue to use time.time()? I want time.steady(strict=True), and I'm glad you're providing it and I'm willing to use it this way, although it is slightly annoying because "time.steady(strict=True)" really means "time.steady(i_really_mean_it=True)". Else, I would have used "time.time()". I am aware of a large number of use cases for a steady clock (event scheduling, profiling, timeouts), and a large number of uses cases for a "NTP-respecting wall clock" clock (calendaring, displaying to a user, timestamping). I'm not aware of any use case for "steady if implemented, else wall-clock", and it sounds like a mistake to me. Regards, Zooko From status at bugs.python.org Fri Mar 23 18:07:38 2012 From: status at bugs.python.org (Python tracker) Date: Fri, 23 Mar 2012 18:07:38 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20120323170738.C93321D080@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2012-03-16 - 2012-03-23) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3346 ( +9) closed 22829 (+50) total 26175 (+59) Open issues with patches: 1434 Issues opened (42) ================== #5301: add mimetype for image/vnd.microsoft.icon (patch) http://bugs.python.org/issue5301 reopened by eric.araujo #10340: asyncore doesn't properly handle EINVAL on OSX http://bugs.python.org/issue10340 reopened by r.david.murray #14336: Difference between pickle implementations for function objects http://bugs.python.org/issue14336 opened by sbt #14338: Document how to forward POST data on redirects http://bugs.python.org/issue14338 opened by beerNuts #14339: Optimizing bin, oct and hex http://bugs.python.org/issue14339 opened by storchaka #14340: Update embedded copy of expat - fix security & crash issues http://bugs.python.org/issue14340 opened by gregory.p.smith #14341: sporadic (?) test_urllib2 failures http://bugs.python.org/issue14341 opened by pitrou #14345: Document socket.SOL_SOCKET http://bugs.python.org/issue14345 opened by techtonik #14349: The documentation of 'dis' doesn't describe MAKE_FUNCTION corr http://bugs.python.org/issue14349 opened by eli.bendersky #14350: Strange Exception from copying an iterable http://bugs.python.org/issue14350 opened by Jakob.Bowyer #14352: Distutils2: add logging message to report successful installat http://bugs.python.org/issue14352 opened by agronholm #14353: Proper gettext support in locale module http://bugs.python.org/issue14353 opened by melflynn #14354: Crash in _ctypes_alloc_callback http://bugs.python.org/issue14354 opened by ogre #14356: Distutils2 ignores site-local configuration http://bugs.python.org/issue14356 opened by agronholm #14357: Distutils2 does not work with virtualenv http://bugs.python.org/issue14357 opened by agronholm #14360: email.encoders.encode_quopri doesn't work with python 3.2 http://bugs.python.org/issue14360 opened by mitya57 #14361: No link to issue tracker on Python home page http://bugs.python.org/issue14361 opened by stevenjd #14362: No mention of collections.ChainMap in What's New for 3.3 http://bugs.python.org/issue14362 opened by stevenjd #14364: Argparse incorrectly handles '--' http://bugs.python.org/issue14364 opened by maker #14365: argparse: subparsers, argument abbreviations and ambiguous opt http://bugs.python.org/issue14365 opened by jakub #14366: Supporting bzip2 and lzma compression in zip files http://bugs.python.org/issue14366 opened by storchaka #14367: try/except block in ismethoddescriptor() in inspect.py, so tha http://bugs.python.org/issue14367 opened by ncdave4life #14368: floattime() should not raise an exception http://bugs.python.org/issue14368 opened by haypo #14369: make __closure__ writable http://bugs.python.org/issue14369 opened by Yury.Selivanov #14371: Add support for bzip2 compression to the zipfile module http://bugs.python.org/issue14371 opened by storchaka #14372: Fix all invalid usage of borrowed references http://bugs.python.org/issue14372 opened by haypo #14373: C implementation of functools.lru_cache http://bugs.python.org/issue14373 opened by anacrolix #14374: Compiling Python 2.7.2 on HP11i PA-RISC ends with segmentation http://bugs.python.org/issue14374 opened by donchen #14375: Add socketserver running property http://bugs.python.org/issue14375 opened by giampaolo.rodola #14376: sys.exit documents argument as "integer" but actually requires http://bugs.python.org/issue14376 opened by Gareth.Rees #14377: Modify serializer for xml.etree.ElementTree to allow forcing t http://bugs.python.org/issue14377 opened by adpoliak #14379: Several traceback docs improvements http://bugs.python.org/issue14379 opened by techtonik #14381: Intern certain integral floats for memory savings and performa http://bugs.python.org/issue14381 opened by krisvale #14383: Generalize the use of _Py_IDENTIFIER in ceval.c and typeobject http://bugs.python.org/issue14383 opened by haypo #14385: Support other types than dict for __builtins__ http://bugs.python.org/issue14385 opened by haypo #14386: Expose dictproxy as a public type http://bugs.python.org/issue14386 opened by haypo #14387: Include\accu.h incompatible with Windows.h http://bugs.python.org/issue14387 opened by jeffr at livedata.com #14390: Tkinter single-threaded deadlock http://bugs.python.org/issue14390 opened by jcbollinger #14391: misc TYPO in argparse.Action docstring http://bugs.python.org/issue14391 opened by shima__shima #14392: type=bool doesn't raise error in argparse.Action http://bugs.python.org/issue14392 opened by shima__shima #14393: Incorporate Guide to Magic Methods? http://bugs.python.org/issue14393 opened by djc #14394: missing links on performance claims of cdecimal http://bugs.python.org/issue14394 opened by tshepang Most recent 15 issues with no replies (15) ========================================== #14392: type=bool doesn't raise error in argparse.Action http://bugs.python.org/issue14392 #14391: misc TYPO in argparse.Action docstring http://bugs.python.org/issue14391 #14390: Tkinter single-threaded deadlock http://bugs.python.org/issue14390 #14379: Several traceback docs improvements http://bugs.python.org/issue14379 #14375: Add socketserver running property http://bugs.python.org/issue14375 #14368: floattime() should not raise an exception http://bugs.python.org/issue14368 #14345: Document socket.SOL_SOCKET http://bugs.python.org/issue14345 #14341: sporadic (?) test_urllib2 failures http://bugs.python.org/issue14341 #14339: Optimizing bin, oct and hex http://bugs.python.org/issue14339 #14336: Difference between pickle implementations for function objects http://bugs.python.org/issue14336 #14329: proxy_bypass_macosx_sysconf does not handle singel ip addresse http://bugs.python.org/issue14329 #14326: IDLE - allow shell to support different locales http://bugs.python.org/issue14326 #14319: cleanup index switching mechanism on packaging.pypi http://bugs.python.org/issue14319 #14304: Implement utf-8-bmp codec http://bugs.python.org/issue14304 #14303: Incorrect documentation for socket.py on linux http://bugs.python.org/issue14303 Most recent 15 issues waiting for review (15) ============================================= #14392: type=bool doesn't raise error in argparse.Action http://bugs.python.org/issue14392 #14391: misc TYPO in argparse.Action docstring http://bugs.python.org/issue14391 #14387: Include\accu.h incompatible with Windows.h http://bugs.python.org/issue14387 #14386: Expose dictproxy as a public type http://bugs.python.org/issue14386 #14385: Support other types than dict for __builtins__ http://bugs.python.org/issue14385 #14383: Generalize the use of _Py_IDENTIFIER in ceval.c and typeobject http://bugs.python.org/issue14383 #14381: Intern certain integral floats for memory savings and performa http://bugs.python.org/issue14381 #14377: Modify serializer for xml.etree.ElementTree to allow forcing t http://bugs.python.org/issue14377 #14375: Add socketserver running property http://bugs.python.org/issue14375 #14373: C implementation of functools.lru_cache http://bugs.python.org/issue14373 #14371: Add support for bzip2 compression to the zipfile module http://bugs.python.org/issue14371 #14369: make __closure__ writable http://bugs.python.org/issue14369 #14368: floattime() should not raise an exception http://bugs.python.org/issue14368 #14367: try/except block in ismethoddescriptor() in inspect.py, so tha http://bugs.python.org/issue14367 #14366: Supporting bzip2 and lzma compression in zip files http://bugs.python.org/issue14366 Top 10 most discussed issues (10) ================================= #14387: Include\accu.h incompatible with Windows.h http://bugs.python.org/issue14387 17 msgs #14228: It is impossible to catch sigint on startup in python code http://bugs.python.org/issue14228 15 msgs #14302: Move python.exe to bin/ http://bugs.python.org/issue14302 15 msgs #14034: Add argparse howto http://bugs.python.org/issue14034 13 msgs #14361: No link to issue tracker on Python home page http://bugs.python.org/issue14361 13 msgs #14331: Python/import.c uses a lot of stack space due to MAXPATHLEN http://bugs.python.org/issue14331 10 msgs #14371: Add support for bzip2 compression to the zipfile module http://bugs.python.org/issue14371 10 msgs #14381: Intern certain integral floats for memory savings and performa http://bugs.python.org/issue14381 10 msgs #10340: asyncore doesn't properly handle EINVAL on OSX http://bugs.python.org/issue10340 9 msgs #13922: argparse handling multiple "--" in args improperly http://bugs.python.org/issue13922 9 msgs Issues closed (46) ================== #1676: Fork/exec issues with Tk 8.5/Python 2.5.1 on OS X http://bugs.python.org/issue1676 closed by ned.deily #3573: IDLE hangs when passing invalid command line args (directory(i http://bugs.python.org/issue3573 closed by asvetlov #4652: IDLE does not work with Unicode http://bugs.python.org/issue4652 closed by asvetlov #7738: IDLE hang when tooltip comes up in Linux http://bugs.python.org/issue7738 closed by ned.deily #7997: http://www.python.org/dev/faq/ doesn't seem to explain how to http://bugs.python.org/issue7997 closed by rosslagerwall #9408: curses: Link against libncursesw instead of libncurses http://bugs.python.org/issue9408 closed by haypo #10538: PyArg_ParseTuple("s*") does not always incref object http://bugs.python.org/issue10538 closed by krisvale #12757: undefined name in doctest.py http://bugs.python.org/issue12757 closed by r.david.murray #12788: test_email fails with -R http://bugs.python.org/issue12788 closed by r.david.murray #13009: Remove documentation in distutils2 repo http://bugs.python.org/issue13009 closed by eric.araujo #13325: no address in the representation of asyncore dispatcher after http://bugs.python.org/issue13325 closed by giampaolo.rodola #13694: asynchronous connect in asyncore.dispatcher does not set addr http://bugs.python.org/issue13694 closed by giampaolo.rodola #13782: xml.etree.ElementTree: Element.append doesn't type-check its a http://bugs.python.org/issue13782 closed by eli.bendersky #14115: 2.7.3rc and 3.2.3rc hang on test_asynchat and test_asyncore on http://bugs.python.org/issue14115 closed by r.david.murray #14204: Support for the NPN extension to TLS/SSL http://bugs.python.org/issue14204 closed by pitrou #14250: for string patterns regex.flags is never equal to 0 http://bugs.python.org/issue14250 closed by python-dev #14269: SMTPD server does not enforce client starting mail transaction http://bugs.python.org/issue14269 closed by r.david.murray #14277: time.monotonic docstring does not mention the time unit return http://bugs.python.org/issue14277 closed by haypo #14296: Compilation error on CentOS 5.8 http://bugs.python.org/issue14296 closed by neologix #14297: Custom string formatter doesn't work like builtin str.format http://bugs.python.org/issue14297 closed by terry.reedy #14306: try/except block is both efficient and expensive? http://bugs.python.org/issue14306 closed by python-dev #14311: ConfigParser does not parse utf-8 files with BOM bytes http://bugs.python.org/issue14311 closed by lukasz.langa #14328: Add keyword-only parameter support to PyArg_ParseTupleAndKeywo http://bugs.python.org/issue14328 closed by larry #14333: queue unittest errors http://bugs.python.org/issue14333 closed by r.david.murray #14335: Reimplement multiprocessing's ForkingPickler using dispatch_ta http://bugs.python.org/issue14335 closed by pitrou #14337: Recent refleaks http://bugs.python.org/issue14337 closed by benjamin.peterson #14342: In re's examples the example with recursion doesn't work http://bugs.python.org/issue14342 closed by python-dev #14343: In re's examples the example with re.split() shadows builtin i http://bugs.python.org/issue14343 closed by python-dev #14346: Typos in Mac/README http://bugs.python.org/issue14346 closed by ned.deily #14347: Update Misc/README http://bugs.python.org/issue14347 closed by ned.deily #14348: Minor whitespace changes in base64 module http://bugs.python.org/issue14348 closed by eric.araujo #14351: Script error in 3.2.3rc1 Windows doc http://bugs.python.org/issue14351 closed by georg.brandl #14355: imp module docs should omit references to init_frozen http://bugs.python.org/issue14355 closed by r.david.murray #14358: test_os failing with errno 61: No Data Available http://bugs.python.org/issue14358 closed by python-dev #14359: _posixsubprocess.o compilation error on CentOS 5.8 http://bugs.python.org/issue14359 closed by rosslagerwall #14363: Can't build Python 3.3a1 on Centos 5 http://bugs.python.org/issue14363 closed by skrah #14370: list.extend() called on an iterator of the list itself leads t http://bugs.python.org/issue14370 closed by rhettinger #14378: __future__ imports fail when compiling from python ast http://bugs.python.org/issue14378 closed by python-dev #14380: MIMEText should default to utf8 charset if input text contains http://bugs.python.org/issue14380 closed by r.david.murray #14382: test_unittest crashes loading 'unittest.test.testmock' when ru http://bugs.python.org/issue14382 closed by ned.deily #14384: Add "default" kw argument to operator.itemgetter and operator. http://bugs.python.org/issue14384 closed by r.david.murray #14388: configparser.py traceback http://bugs.python.org/issue14388 closed by r.david.murray #14389: Mishandling of large numbers http://bugs.python.org/issue14389 closed by r.david.murray #1222721: tk + setlocale problems... http://bugs.python.org/issue1222721 closed by terry.reedy #1752252: tkFileDialog closes Python when used http://bugs.python.org/issue1752252 closed by terry.reedy #14344: repr of email policies is wrong http://bugs.python.org/issue14344 closed by r.david.murray From victor.stinner at gmail.com Fri Mar 23 18:27:50 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 23 Mar 2012 18:27:50 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: > I want time.steady(strict=True), and I'm glad you're providing it and > I'm willing to use it this way, although it is slightly annoying > because "time.steady(strict=True)" really means > "time.steady(i_really_mean_it=True)". Else, I would have used > "time.time()". > > I am aware of a large number of use cases for a steady clock (event > scheduling, profiling, timeouts), and a large number of uses cases for > a "NTP-respecting wall clock" clock (calendaring, displaying to a > user, timestamping). I'm not aware of any use case for "steady if > implemented, else wall-clock", and it sounds like a mistake to me. time.steady(strict=False) is what you need to implement timeout. If you use time.steady(strict=True) for timeout, it means that you cannot use select, threads, etc. if your platform doesn't provide monotonic clock, whereas it works "well" (except the issue of adjusted time) with Python < 3.3. Victor From bradallen137 at gmail.com Fri Mar 23 18:26:22 2012 From: bradallen137 at gmail.com (Brad Allen) Date: Fri, 23 Mar 2012 12:26:22 -0500 Subject: [Python-Dev] Issue 13524: subprocess on Windows In-Reply-To: <61913F1B-1DFC-4EA5-AF29-25BBF24DA4DA@twistedmatrix.com> References: <61913F1B-1DFC-4EA5-AF29-25BBF24DA4DA@twistedmatrix.com> Message-ID: On Thu, Mar 22, 2012 at 2:35 PM, Glyph Lefkowitz wrote: > Also, in order to execute in any installation environment where libraries are found in non-default locations, you will need to set LD_LIBRARY_PATH. ?Oh, and you will also need to set $PATH on UNIX so that libraries can find their helper programs and %PATH% on Windows so that any compiled dynamically-loadable modules and/or DLLs can be loaded. ?And by the way you will also need to relay DYLD_LIBRARY_PATH if you did a UNIX-style build on OS X, not LD_LIBRARY_PATH. ?Don't forget that you probably also need PYTHONPATH to make sure any subprocess environments can import the same modules as their parent. ?Not to mention SSH_AUTH_SOCK if your application requires access to _remote_ process spawning, rather than just local. ?Oh and DISPLAY in case your subprocesses need GUI support from an X11 program (which sometimes you need just to initialize certain libraries which don't actually do anything with a GUI). ?Oh and __CF_USER_TEXT_ENCODING is important sometimes too, don't forget that. ?And if your subprocess is in Perl or Ruby or Java you may need a couple dozen other variables which your deployment environment has set for you too. ?Did I mention CFLAGS or LC_ALL yet? ?Let me tell you a story about this one HP/UX machine... > > Ahem. > > Bottom line: it seems like screwing with the process spawning environment to make it minimal is a good idea for simplicity, for security, and for modularity. ?But take it from me, it isn't. ?I guarantee you that you don't actually know what is in your operating system's environment, and initializing it is a complicated many-step dance which some vendor or sysadmin or product integrator figured out how to do much better than your hapless Python program can. > > %SystemRoot% is just the tip of a very big, very nasty iceberg. ?Better not to keep refining why exactly it's required, or someone will eventually be adding a new variable (starting with %APPDATA% and %HOMEPATH%) that can magically cause your subprocess not to spawn properly to this page every six months for eternity. ?If you're spawning processes as a regular user, you should just take the environment you're given, perhaps with a few specific light additions whose meaning you understand. ?If you're spawning a process as an administrator or root, you should probably initialize the environment for the user you want to spawn that process as using an OS-specific mechanism like login(1). ?(Sorry that I don't know the Windows equivalent.) Thanks, Glyph. In that case maybe the Python subprocess docs need not single out SystemRoot, but instead plaster a big warning around the use of the 'env' parameter.: Here is what the docs currently state for the Popen constructor 'env' parameter: >If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of inheriting the current process? environment, which is the default behavior. > Note: If specified, env must provide any variables required for the program to execute. On Windows, in order to run a side-by-side assembly the specified env must include a valid SystemRoot. The "Note" section could instead state something like: "In most cases, the child process will need many of the same environment variables as the current process. Usually the safest course of action is to build the env dict to contain all the same keys and values from os.environ. For example... " From van.lindberg at gmail.com Fri Mar 23 18:37:36 2012 From: van.lindberg at gmail.com (VanL) Date: Fri, 23 Mar 2012 12:37:36 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> Message-ID: <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> On Friday, March 23, 2012 at 11:39 AM, PJ Eby wrote: > Even if you are using tools that don't use distutils' configuration settings for these directories, why not simply fix those tools so that they do? Thats what I do currently - I set things to bin and patch Python and the tools so that they work. However, have considered this to be a little bit of a wart anyway for a long time - even before I adopted my current method of working - because it is a pointless inconsistency. The fact that it makes virtual environments consistent across platforms, together with pyvenv going into 3.3, gave me enough of a push to elevate this from private annoyance to "should fix." Thanks, Van -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Mar 23 18:51:35 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 23 Mar 2012 18:51:35 +0100 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B3B6E.2000402@gmail.com> <4F6BAE1B.8080005@skippinet.com.au> Message-ID: On Fri, Mar 23, 2012 at 2:37 PM, Paul Moore wrote: > Another problem case: cx_Freeze. This currently breaks when installed > in a viretualenv, as it locates the "Scripts" directory by appending > "Scripts" to the directory of the python executable. > > So the proposed change *will* break cx_Freeze. The Scripts->bin change > will also break it. > > Paul. > > PS Yes, I need to report the existing bug. The point remains, however... This seems to me to be evidence that the things that will be broken are in need of fixing anyway.<0.5 wink/> virtualenv is something that should Just Work in most, if not all, cases. From jimjjewett at gmail.com Fri Mar 23 19:03:20 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 23 Mar 2012 14:03:20 -0400 Subject: [Python-Dev] [Python-checkins] cpython (3.2): attempt to fix asyncore buildbot failure In-Reply-To: References: Message-ID: What does this verify? My assumption from the name (test_quick_connect) and the context (an asynchronous server) is that it is verifying the server can handle a certain level of load. Refusing the sockets should then be a failure, or at least a skipped test. Would the below fail even if asyncore.loop were taken out of the threading.Thread target altogether? On Fri, Mar 23, 2012 at 10:10 AM, giampaolo.rodola wrote: > http://hg.python.org/cpython/rev/2db4e916245a > changeset: ? 75901:2db4e916245a > branch: ? ? ?3.2 > parent: ? ? ?75897:b97964af7299 > user: ? ? ? ?Giampaolo Rodola' > date: ? ? ? ?Fri Mar 23 15:07:07 2012 +0100 > summary: > ?attempt to fix asyncore buildbot failure > > files: > ?Lib/test/test_asyncore.py | ?10 +++++++--- > ?1 files changed, 7 insertions(+), 3 deletions(-) > > > diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py > --- a/Lib/test/test_asyncore.py > +++ b/Lib/test/test_asyncore.py > @@ -741,11 +741,15 @@ > > ? ? ? ? for x in range(20): > ? ? ? ? ? ? s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > + ? ? ? ? ? ?s.settimeout(.2) > ? ? ? ? ? ? s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, > ? ? ? ? ? ? ? ? ? ? ? ? ?struct.pack('ii', 1, 0)) > - ? ? ? ? ? ?s.connect(server.address) > - ? ? ? ? ? ?s.close() > - > + ? ? ? ? ? ?try: > + ? ? ? ? ? ? ? ?s.connect(server.address) > + ? ? ? ? ? ?except socket.error: > + ? ? ? ? ? ? ? ?pass > + ? ? ? ? ? ?finally: > + ? ? ? ? ? ? ? ?s.close() > > ?class TestAPI_UseSelect(BaseTestAPI): > ? ? use_poll = False > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From pje at telecommunity.com Fri Mar 23 19:35:38 2012 From: pje at telecommunity.com (PJ Eby) Date: Fri, 23 Mar 2012 14:35:38 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> Message-ID: On Mar 23, 2012 1:37 PM, "VanL" wrote: > > On Friday, March 23, 2012 at 11:39 AM, PJ Eby wrote: >> >> Even if you are using tools that don't use distutils' configuration settings for these directories, why not simply fix those tools so that they do? > > > Thats what I do currently - I set things to bin and patch Python and the tools so that they work. Patch *Python*? Where? Are you talking about the distutils/distutils.cfg file? My point here is that AFAIK, Python already supports your desired layout - so your use case doesn't provide much of an argument in favor of making it the default. > However, have considered this to be a little bit of a wart anyway for a long time - even before I adopted my current method of working - because it is a pointless inconsistency. In other words, that's the real reason - which, as it's already been pointed out, is not much of an argument in favor of changing it. Consistency with previous Python releases seems a far more *useful* consistency to maintain than cross-platform consistency, which is only of relevance to cross-platform developers -- at best only a subset of the Windows developer audience. Worse, changing it means that tools have to grow version-specific code, not just platform-specific code. > The fact that it makes virtual environments consistent across platforms, Not really seeing a point. Home directory layouts, "develop" installs, .pth files, -m scripts... there are *zillions* of ways to develop code in cross-platform directory layouts, including the one you're using now. Tool developers are going "meh" about your proposal because it doesn't actually solve any problems for them: they still have to support the old layout, and if their code already uses distutils' facilities for obtaining paths, there's nothing they gain from the change. IOW, the only person who gains from the consistency is someone who wants their virtualenv's to look the same and check them into source. I'm really not seeing this as being a big enough group to be worth inconveniencing other people for, vs. telling them to add bin/ to PATH on Windows and edit a distutils config file. At best, this might be deserving of a FAQ entry on how to set up cross platform development environments. AFAICT, virtualenvs are overkill for most development anyway. If you're not using distutils except to install dependencies, then configure distutils to install scripts and libraries to the same directory, and then do all your development in that directory. Presto! You now have a cross-platform "virtualenv". Want the scripts on your path? Add that directory to your path... or if on Windows, don't bother, since the current directory is usually on the path. (In fact, if you're only using easy_install to install your dependencies, you don't even need to edit the distutils configuration, just use "-md targetdir".) The entire virtualenv concept was originally introduced as a way for non-root *nix users to have private site-packages directories with .pth support, in order to be able to install eggs -- a use case which was then solved by user-specific site directories in Python 2.6, and the addition of the site.py hacks in easy_install (to allow any directory to be a virtualenv as far as easy_install was concerned). Virtualenv seem to have caught on for a variety of other uses than that, but AFAIK, that's only because it's the most *visible* solution for those uses. Just dumping things in a directory adjacent to the corresponding scripts is the original virtualenv, and it still works just dandy -- most people just don't *know* this. (And again, if there are tools out there that *don't* support single-directory virtualenvs, the best answer is probably to fix them.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Fri Mar 23 20:53:05 2012 From: carl at oddbird.net (Carl Meyer) Date: Fri, 23 Mar 2012 13:53:05 -0600 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> Message-ID: <4F6CD4A1.6030301@oddbird.net> Hi PJ, On 03/23/2012 12:35 PM, PJ Eby wrote: > AFAICT, virtualenvs are overkill for most development anyway. If you're > not using distutils except to install dependencies, then configure > distutils to install scripts and libraries to the same directory, and > then do all your development in that directory. Presto! You now have a > cross-platform "virtualenv". Want the scripts on your path? Add that > directory to your path... or if on Windows, don't bother, since the > current directory is usually on the path. (In fact, if you're only > using easy_install to install your dependencies, you don't even need to > edit the distutils configuration, just use "-md targetdir".) Creating and using a virtualenv is, in practice, _easier_ than any of those alternatives, so it's hard to see it as "overkill." Not to mention that the "isolation from system site-packages" feature is quite popular (the outpouring of gratitude when virtualenv went isolated-by-default a few months ago was astonishing), and AFAIK none of your alternative proposals support that at all. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From van.lindberg at gmail.com Fri Mar 23 21:19:08 2012 From: van.lindberg at gmail.com (VanL) Date: Fri, 23 Mar 2012 15:19:08 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> Message-ID: On Friday, March 23, 2012 at 1:35 PM, PJ Eby wrote: > Tool developers are going "meh" about your proposal because it doesn't actually solve any problems for them: they still have to support the old layout, and if their code already uses distutils' facilities for obtaining paths, there's nothing they gain from the change. Three notes. FIrst, distutils.cfg doesn't always work because it is centered around the idea of set paths that are the same each time - which doesn't always work with virtualenvs. Further, a number of tools find that it doesn't work (haven't seen it myself, but look at the comments in pypm's installer). So yes, I patch python. Second, most installer tools don't follow distutils.cfg. Even if that helps for python setup.py install, the other tools are still broken when you want to specify a layout. That is why changing the defaults is the only effective way to make this change - because the defaults drive what is actually implemented. I know, because I have looked at and patched these tools to make them work. Third, there are some tool makers going meh - because you are right, this is not a problem they have. This is a problem associated with using those tools. And regardless of there being other ways to do it, including your 'dump it in a directory' method, development in virtualenvs is convenient, widespread, and on the rise. Given that pyvenv will go into 3.3, it will be the 'one obvious way to do it' - making going-forward cross-platform compatibility a positive good. Again I note the example of buildout. And fourth, (because nobody expects the spanish inquisition), isn't the gratuitous difference a (small but) obvious wart? Does anybody positively like 'Scripts'? The most common comment I have received when talking to people off this list has been, 'yeah, that was always sort of weird.' -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Fri Mar 23 21:21:37 2012 From: nas at arctrix.com (Neil Schemenauer) Date: Fri, 23 Mar 2012 20:21:37 +0000 (UTC) Subject: [Python-Dev] Issue #10278 -- why not just an attribute? References: <4f67fec7.65aa320a.62e4.0780@mx.google.com> Message-ID: Jim J. Jewett wrote: > Passing strict as an argument seems like overkill since it will always > be meaningless on some (most?) platforms. A keyword argument that gets passed as a constant in the caller is usually poor API. Why not have two different functions? Neil From glyph at twistedmatrix.com Fri Mar 23 21:40:34 2012 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 23 Mar 2012 16:40:34 -0400 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On Mar 23, 2012, at 12:55 PM, Zooko Wilcox-O'Hearn wrote: >> I merged the two functions into one function: time.steady(strict=False). >> >> time.steady() should be monotonic most of the time, but may use a fallback. >> >> time.steady(strict=True) fails with OSError or NotImplementedError if >> reading the monotonic clock failed or if no monotonic clock is available. > > If someone wants time.steady(strict=False), then why don't they just > continue to use time.time()? > > I want time.steady(strict=True), and I'm glad you're providing it and > I'm willing to use it this way, although it is slightly annoying > because "time.steady(strict=True)" really means > "time.steady(i_really_mean_it=True)". Else, I would have used > "time.time()". > > I am aware of a large number of use cases for a steady clock (event > scheduling, profiling, timeouts), and a large number of uses cases for > a "NTP-respecting wall clock" clock (calendaring, displaying to a > user, timestamping). I'm not aware of any use case for "steady if > implemented, else wall-clock", and it sounds like a mistake to me. I think I've lost the thread of this discussion. Is that really what "strict=False" was supposed to mean? I am aware of use-cases which want to respect slew, but reject steps. The local clock might not be all that reliable, and slew actually keeps it closer to "real" time. My understanding was that strict=True was something like CLOCK_MONOTONIC_RAW and strict=False was just CLOCK_MONOTONIC. I am increasingly thinking that the first order of business here should be to expose the platform-specific mechanisms directly, then try to come up with a unifying abstraction in the time module later. It's hard enough to understand the substantially dense verbiage around all of these different timers on their respective platforms; understanding which one exactly Python is swaddling up in a portability layer seems bound to generate confusion. Not to mention that you run into awesome little edge cases like this: https://github.com/ThomasHabets/monotonic_clock/blob/master/src/monotonic_win32.c#L62 which means sometimes you really really need to know exactly which clock is getting used, if you want to make it work right (unless Python is going to ship with all of these workarounds on day 1). (I still object to the "time.steady" naming, because this is what people in the make-believe land of C++ call it. The people who live in the real world of C and POSIX all refer to it as "monotonic". And even the C++ purists slip up sometimes, c.f. : "Class std::chrono::steady_clock represents a monotonic clock.") If there really are some applications for which the desired behavior is 'monotonic clock, but if you can't get one, nevermind, wallclock is good enough', it strikes me that this should be as explicit as possible, it should not be the default, and if try: value = time.steady() except OhWowYourComputerReallyHasProblems: value = time.time() is generally considered too onerous, it should be spelled time.steady(fallback=time.time). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Fri Mar 23 21:46:53 2012 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 23 Mar 2012 16:46:53 -0400 Subject: [Python-Dev] Issue 13524: subprocess on Windows In-Reply-To: References: <61913F1B-1DFC-4EA5-AF29-25BBF24DA4DA@twistedmatrix.com> Message-ID: On Mar 23, 2012, at 1:26 PM, Brad Allen wrote: > Thanks, Glyph. In that case maybe the Python subprocess docs need not > single out SystemRoot, but instead plaster a big warning around the > use of the 'env' parameter. I agree. I'm glad that my bitter experience here might be useful to someone in the future - all those late nights trying desperately to get my unit tests to run on some newly configured, slightly weird buildbot didn't go to waste :). > The "Note" section could instead state something like: "In most cases, > the child process will need many of the same environment variables as > the current process. Usually the safest course of action is to build > the env dict to contain all the same keys and values from os.environ. > For example... " I think including all the examples might be overstating the case. It is probably best to say that other operating systems, vendors, and integration tools may set necessary environment variables that there is no way for you to be aware of in advance, unless you are an expert sysadmin on every platform where you expect your code to run, and that many of these variables are required for libraries to function properly, both libraries bundled with python and those from third parties. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Mar 23 22:11:40 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Mar 2012 21:11:40 +0000 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On 23 March 2012 20:40, Glyph wrote: > I am increasingly thinking that the first order of business here should be > to expose the platform-specific mechanisms directly, then try to come up > with a unifying abstraction in the time module later. +1. Paul From bradallen137 at gmail.com Fri Mar 23 22:31:12 2012 From: bradallen137 at gmail.com (Brad Allen) Date: Fri, 23 Mar 2012 16:31:12 -0500 Subject: [Python-Dev] Issue 13524: subprocess on Windows In-Reply-To: References: <61913F1B-1DFC-4EA5-AF29-25BBF24DA4DA@twistedmatrix.com> Message-ID: On Fri, Mar 23, 2012 at 3:46 PM, Glyph wrote: > On Mar 23, 2012, at 1:26 PM, Brad Allen wrote: > > Thanks, Glyph. In that case maybe the Python subprocess docs need not > single out SystemRoot, but instead plaster a big warning around the > use of the 'env' parameter. > > > I agree. ?I'm glad that my bitter experience here might be useful to someone > in the future - all those late nights trying desperately to get my unit > tests to run on some newly configured, slightly weird buildbot didn't go to > waste :). Ok, I'll open a ticket on the bugtracker for this over the weekend. From yselivanov.ml at gmail.com Fri Mar 23 23:54:42 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 23 Mar 2012 18:54:42 -0400 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: <47D9CBF0-CCA5-4CBB-8899-BAC2027C4F19@gmail.com> On 2012-03-23, at 4:40 PM, Glyph wrote: > (I still object to the "time.steady" naming, because this is what people in the make-believe land of C++ call it. The people who live in the real world of C and POSIX all refer to it as "monotonic". And even the C++ purists slip up sometimes, c.f. : "Class std::chrono::steady_clock represents a monotonic clock.") +1. I also think that the function should be called 'monotonic' and simply fail with OSError on platforms that don't support such clocks. The 'strict' argument is non-intuitive. - Yury From yselivanov.ml at gmail.com Fri Mar 23 23:56:08 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 23 Mar 2012 18:56:08 -0400 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: <79DA2DE9-5285-489E-B252-26D897DC022E@gmail.com> On 2012-03-23, at 1:27 PM, Victor Stinner wrote: >> I want time.steady(strict=True), and I'm glad you're providing it and >> I'm willing to use it this way, although it is slightly annoying >> because "time.steady(strict=True)" really means >> "time.steady(i_really_mean_it=True)". Else, I would have used >> "time.time()". >> >> I am aware of a large number of use cases for a steady clock (event >> scheduling, profiling, timeouts), and a large number of uses cases for >> a "NTP-respecting wall clock" clock (calendaring, displaying to a >> user, timestamping). I'm not aware of any use case for "steady if >> implemented, else wall-clock", and it sounds like a mistake to me. > > time.steady(strict=False) is what you need to implement timeout. > > If you use time.steady(strict=True) for timeout, it means that you > cannot use select, threads, etc. if your platform doesn't provide > monotonic clock, whereas it works "well" (except the issue of adjusted > time) with Python < 3.3. Why can't I use select & threads? You mean that if a platform does not support monotonic clocks it also does not support threads and select sys call? - Yury From anacrolix at gmail.com Sat Mar 24 00:06:46 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Sat, 24 Mar 2012 07:06:46 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <47D9CBF0-CCA5-4CBB-8899-BAC2027C4F19@gmail.com> References: <4F6137EF.9000000@gmail.com> <47D9CBF0-CCA5-4CBB-8899-BAC2027C4F19@gmail.com> Message-ID: Yes, call it what it is. monotonic or monotonic_time, because that's why I'm using it. No flags. I've followed this thread throughout, and I'm still not sure if "steady" gives the real guarantees it claims. It's trying to be too much. Existing bugs complain about backward jumps and demand a clock that doesn't do this. The function should guarantee monotonicity only, and not get overcomplicated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Mar 24 00:07:45 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 24 Mar 2012 00:07:45 +0100 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: <79DA2DE9-5285-489E-B252-26D897DC022E@gmail.com> References: <4F6137EF.9000000@gmail.com> <79DA2DE9-5285-489E-B252-26D897DC022E@gmail.com> Message-ID: 2012/3/23 Yury Selivanov : > Why can't I use select & threads? ?You mean that if a platform does not > support monotonic clocks it also does not support threads and select sys > call? Python 3.3 now uses time.steady(strict=False) in the threading and queue modules. If we replace it by time.steady(strict=True), you may get an error if your platform doesn't provide a monotonic clock and so you cannot use these modules. Victor From yselivanov.ml at gmail.com Sat Mar 24 00:21:57 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 23 Mar 2012 19:21:57 -0400 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> <79DA2DE9-5285-489E-B252-26D897DC022E@gmail.com> Message-ID: <9FA0018E-08D0-48A4-8D9F-B97225973C87@gmail.com> On 2012-03-23, at 7:07 PM, Victor Stinner wrote: > 2012/3/23 Yury Selivanov : >> Why can't I use select & threads? You mean that if a platform does not >> support monotonic clocks it also does not support threads and select sys >> call? > > Python 3.3 now uses time.steady(strict=False) in the threading and > queue modules. If we replace it by time.steady(strict=True), you may > get an error if your platform doesn't provide a monotonic clock and so > you cannot use these modules. Why this won't work? try: from time import monotonic as _time except ImportError: from time import time as _time OR (if we decide to fail on first call, instead of ImportError) import time try: time.monotonic() except OSError: _time = time else: _time = time.monotonic And then just use '_time' in your code? What's the deal with the 'strict' kwarg? I really like how it currently works with epoll, for instance. It either exists in the 'select' module, or not, if the host OS doesn't support it. I think it should be the same for 'time.monotonic'. - Yury From victor.stinner at gmail.com Sat Mar 24 00:25:41 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 24 Mar 2012 00:25:41 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? Message-ID: Hi, time.steady(strict=True) looks to be confusing for most people, some of them don't understand the purpose of the flag and others don't like a flag changing the behaviour of the function. I propose to replace time.steady(strict=True) by time.monotonic(). That would avoid the need of an ugly NotImplementedError: if the OS has no monotonic clock, time.monotonic() will just not exist. So we will have: - time.time(): realtime, can be adjusted by the system administrator (manually) or automatically by NTP - time.clock(): monotonic clock on Windows, CPU time on UNIX - time.monotonic(): monotonic clock, its speed may or may not be adjusted by NTP but it only goes forward, may raise an OSError - time.steady(): monotonic clock or the realtime clock, depending on what is available on the platform (use monotonic in priority). may be adjusted by NTP or the system administrator, may go backward. time.steady() is something like: try: return time.monotonic() except (NotImplementError, OSError): return time.time() time.time(), time.clock(), time.steady() are always available, whereas time.monotonic() will not be available on some platforms. Victor From brian at python.org Sat Mar 24 00:28:28 2012 From: brian at python.org (Brian Curtin) Date: Fri, 23 Mar 2012 18:28:28 -0500 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: On Mar 23, 2012 6:25 PM, "Victor Stinner" wrote: > > Hi, > > time.steady(strict=True) looks to be confusing for most people, some > of them don't understand the purpose of the flag and others don't like > a flag changing the behaviour of the function. > > I propose to replace time.steady(strict=True) by time.monotonic(). > That would avoid the need of an ugly NotImplementedError: if the OS > has no monotonic clock, time.monotonic() will just not exist. > > So we will have: > > - time.time(): realtime, can be adjusted by the system administrator > (manually) or automatically by NTP > - time.clock(): monotonic clock on Windows, CPU time on UNIX > - time.monotonic(): monotonic clock, its speed may or may not be > adjusted by NTP but it only goes forward, may raise an OSError > - time.steady(): monotonic clock or the realtime clock, depending on > what is available on the platform (use monotonic in priority). may be > adjusted by NTP or the system administrator, may go backward. > > time.steady() is something like: > > try: > return time.monotonic() > except (NotImplementError, OSError): > return time.time() > > time.time(), time.clock(), time.steady() are always available, whereas > time.monotonic() will not be available on some platforms. > > Victor This seems like it should have been a PEP, or maybe should become a PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Mar 24 00:36:31 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 24 Mar 2012 00:36:31 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: > This seems like it should have been a PEP, or maybe should become a PEP. I replaced time.wallclock() by time.steady(strict=False) and time.monotonic() by time.steady(strict=True). This change solved the naming issue of time.wallclock(), but it was a bad idea to merge monotonic() feature into time.steady(). It looks like everybody agrees, am I wrong? Victor From yselivanov.ml at gmail.com Sat Mar 24 00:38:10 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 23 Mar 2012 19:38:10 -0400 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: <4FB90B27-4778-4FE9-B93B-887A730BCBEC@gmail.com> On 2012-03-23, at 7:28 PM, Brian Curtin wrote: > This seems like it should have been a PEP, or maybe should become a PEP. Why? AFAIK Victor just proposes to add two new functions: monotonic() and steady(). time() and clock() do already exist and won't be changed. - Yury From yselivanov.ml at gmail.com Sat Mar 24 00:42:17 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 23 Mar 2012 19:42:17 -0400 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: <61C7F677-7F04-4FD6-AC66-4595374F402A@gmail.com> On 2012-03-23, at 7:25 PM, Victor Stinner wrote: > - time.steady(): monotonic clock or the realtime clock, depending on > what is available on the platform (use monotonic in priority). may be > adjusted by NTP or the system administrator, may go backward. > > time.steady() is something like: > > try: > return time.monotonic() > except (NotImplementError, OSError): > return time.time() Is the use of weak monotonic time so wide-spread in the stdlib that we need the 'steady()' function? If it's just two modules then it's not worth adding it. - Yury From victor.stinner at gmail.com Sat Mar 24 00:51:39 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 24 Mar 2012 00:51:39 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: <61C7F677-7F04-4FD6-AC66-4595374F402A@gmail.com> References: <61C7F677-7F04-4FD6-AC66-4595374F402A@gmail.com> Message-ID: >> time.steady() is something like: >> >> try: >> ?return time.monotonic() >> except (NotImplementError, OSError): >> ?return time.time() > > Is the use of weak monotonic time so wide-spread in the stdlib that we > need the 'steady()' function? ?If it's just two modules then it's not > worth adding it. The Python standard library is not written to be used by Python itself, but by others. The try/except is a common pattern when applications use a monotonic clock. I suppose that quite all applications use this try/except pattern. I don't see what is the use case requiring a is truly monotonic clock. Victor From steve at pearwood.info Sat Mar 24 01:02:36 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2012 11:02:36 +1100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: <4F6D0F1C.1080404@pearwood.info> Victor Stinner wrote: [...] > So we will have: > > - time.time(): realtime, can be adjusted by the system administrator > (manually) or automatically by NTP > - time.clock(): monotonic clock on Windows, CPU time on UNIX > - time.monotonic(): monotonic clock, its speed may or may not be > adjusted by NTP but it only goes forward, may raise an OSError This all sounds good to me. +1 up to this point. Question: under what circumstances will monotonic() exist but raise OSError? > - time.steady(): monotonic clock or the realtime clock, depending on > what is available on the platform (use monotonic in priority). may be > adjusted by NTP or the system administrator, may go backward. What makes this "steady", given that it can be adjusted and it can go backwards? Doesn't sound steady to me. Is steady() merely a convenience function to avoid the user having to write something like this? try: mytimer = time.monotonic except AttributeError: mytimer = time.time or inline: > try: > return time.monotonic() > except (NotImplementError, OSError): > return time.time() Should that be (AttributeError, OSError) instead? -- Steven From steve at pearwood.info Sat Mar 24 01:16:10 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2012 11:16:10 +1100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: <4F6D124A.3000607@pearwood.info> Victor Stinner wrote: > - time.clock(): monotonic clock on Windows, CPU time on UNIX Actually, I think that is not correct. Or at least *was* not correct in 2006. http://bytes.com/topic/python/answers/527849-time-clock-going-backwards -- Steven From victor.stinner at gmail.com Sat Mar 24 01:25:42 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 24 Mar 2012 01:25:42 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: <4F6D0F1C.1080404@pearwood.info> References: <4F6D0F1C.1080404@pearwood.info> Message-ID: > Question: under what circumstances will monotonic() exist but raise OSError? On Windows, OSError is raised if QueryPerformanceFrequency fails. Extract of Microsoft doc: "If the function fails, the return value is zero. To get extended error information, call GetLastError. For example, if the installed hardware does not support a high-resolution performance counter, the function fails." On UNIX, OSError is raised if clock_gettime(CLOCK_MONOTONIC) fails. Extract of clock_gettime() doc: "ERRORS EINVAL The clk_id specified is not supported on this system." It may occur if the libc exposes CLOCK_MONOTONIC but the kernel doesn't support it. I don't know if it can occur in practice. >> - time.steady(): monotonic clock or the realtime clock, depending on >> what is available on the platform (use monotonic in priority). may be >> adjusted by NTP or the system administrator, may go backward. > > What makes this "steady", given that it can be adjusted and it can go > backwards? Doesn't sound steady to me. In practice, it will be monotonic in most cases. "steady" name is used instead of "monotonic" because it may not be monotonic is other cases. > Is steady() merely a convenience function to avoid the user having > to write something like this? steady() remembers if the last call to monotonic failed or not. The real implementation is closer to something like: def steady(): if not steady.has_monotonic: return time.time() try: return time.monotonic() except (AttributeError, OSError): steady.has_monotonic = False return time.time() steady.has_monotonic = True Victor From victor.stinner at gmail.com Sat Mar 24 01:45:37 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 24 Mar 2012 01:45:37 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: <4F6D124A.3000607@pearwood.info> References: <4F6D124A.3000607@pearwood.info> Message-ID: >> - time.clock(): monotonic clock on Windows, CPU time on UNIX > > > Actually, I think that is not correct. Or at least *was* not correct in > 2006. > > http://bytes.com/topic/python/answers/527849-time-clock-going-backwards Oh, I was not aware of this issue. Do you suggest to not use QueryPerformanceCounter() on Windows to implement a monotonic clock? The python-monotonic-time project uses GetTickCount64(), or GetTickCount(), on Windows. GetTickCount64() was added to Windows Seven / Server 2008. GetTickCount() overflows after 49 days. QueryPerformanceCounter() has a better resolution than GetTickCount[64](). Victor From greg.ewing at canterbury.ac.nz Sat Mar 24 02:16:17 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 24 Mar 2012 14:16:17 +1300 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> Message-ID: <4F6D2061.3080804@canterbury.ac.nz> Glyph Lefkowitz wrote: > "do I have to resize my browser every time I visit a new site to get a > decent width for reading". If all sites left the width to the browser, then I would be able to make my browser window a width that is comfortable for me with my chosen font size and leave it that way. The only time a site forces me to resize my window is when it thinks it has a better idea than me how wide the text should be. I prefer sites that don't try to control the layout of everything. When using a site that leaves most of it up to my browser, I never find myself wishing that the designer had specified something more tightly. However, I do often find myself wishing that the designer *hadn't* overridden the width, or the font size, or the text colour, or decided that I shouldn't be allowed to know whether I've visited links before, etc. etc. A web page is not a printed page. It is not rendered at a fixed size and viewed in its entirety at once. It needs to be flexible, able to be rendered in whatever size space is available or the user wants to devote to it. Browsers are very good at doing that -- unless the designer defeats them by fixing something that is better left flexible. -- Greg From steve at pearwood.info Sat Mar 24 03:08:58 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2012 13:08:58 +1100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: <4F6D0F1C.1080404@pearwood.info> Message-ID: <4F6D2CBA.10205@pearwood.info> Victor Stinner wrote: >> Is steady() merely a convenience function to avoid the user having >> to write something like this? > > steady() remembers if the last call to monotonic failed or not. The > real implementation is closer to something like: > > def steady(): > if not steady.has_monotonic: > return time.time() > try: > return time.monotonic() > except (AttributeError, OSError): > steady.has_monotonic = False > return time.time() > steady.has_monotonic = True Does this mean that there are circumstances where monotonic will work for a while, but then fail? Otherwise, we would only need to check monotonic once, when the time module is first loaded, rather than every time it is called. Instead of the above: # global to the time module try: monotonic() except (NameError, OSError): steady = time else: steady = monotonic Are there failure modes where monotonic can recover? That is, it works for a while, then raises OSError, then works again on the next call. If so, steady will stop using monotonic and never try it again. Is that deliberate? -- Steven From janzert at janzert.com Sat Mar 24 03:12:51 2012 From: janzert at janzert.com (Janzert) Date: Fri, 23 Mar 2012 22:12:51 -0400 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: On 3/23/2012 7:25 PM, Victor Stinner wrote: [snip] > - time.monotonic(): monotonic clock, its speed may or may not be > adjusted by NTP but it only goes forward, may raise an OSError > - time.steady(): monotonic clock or the realtime clock, depending on > what is available on the platform (use monotonic in priority). may be > adjusted by NTP or the system administrator, may go backward. > > time.steady() is something like: > > try: > return time.monotonic() > except (NotImplementError, OSError): > return time.time() > I am surprised that a clock with the name time.steady() has a looser definition than one called time.monotonic(). To my mind a steady clock is by definition monotonic but a monotonic one may or may not be steady. Janzert From steve at pearwood.info Sat Mar 24 03:26:15 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2012 13:26:15 +1100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: <4F6D124A.3000607@pearwood.info> Message-ID: <4F6D30C7.6010209@pearwood.info> Victor Stinner wrote: >>> - time.clock(): monotonic clock on Windows, CPU time on UNIX >> >> Actually, I think that is not correct. Or at least *was* not correct in >> 2006. >> >> http://bytes.com/topic/python/answers/527849-time-clock-going-backwards > > Oh, I was not aware of this issue. Do you suggest to not use > QueryPerformanceCounter() on Windows to implement a monotonic clock? I do not have an opinion on the best way to implement monotonic to guarantee that it actually is monotonic. -- Steven From jimjjewett at gmail.com Sat Mar 24 03:36:30 2012 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Fri, 23 Mar 2012 19:36:30 -0700 (PDT) Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: <4F6D0F1C.1080404@pearwood.info> Message-ID: <4f6d332e.e938b60a.3825.ffffe8f5@mx.google.com> In http://mail.python.org/pipermail/python-dev/2012-March/118024.html Steven D'Aprano wrote: > What makes this "steady", given that it can be adjusted > and it can go backwards? It is best-effort for steady, but putting "best" in the name would be an attractive nuisance. > Is steady() merely a convenience function to avoid the user > having to write something like this? > try: > mytimer = time.monotonic > except AttributeError: > mytimer = time.time That would still be worth doing. But I think the main point is that the clock *should* be monotonic, and *should* be as precise as possible. Given that it returns seconds elapsed (since an undefined start), perhaps it should be time.seconds() or even time.counter() -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From pje at telecommunity.com Sat Mar 24 04:21:18 2012 From: pje at telecommunity.com (PJ Eby) Date: Fri, 23 Mar 2012 23:21:18 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> Message-ID: On Mar 23, 2012 4:19 PM, "VanL" wrote: > > Three notes. FIrst, distutils.cfg doesn't always work because it is centered around the idea of set paths that are the same each time - which doesn't always work with virtualenvs. And the virtualenv doesn't contain its own copy of distutils.cfg? > Second, most installer tools don't follow distutils.cfg. Even if that helps for python setup.py install, the other tools are still broken when you want to specify a layout. So, we should change Python to fix the broken tools that don't follow documented standards for configuring installation locations? If the tools are that broken, aren't they going to break even *harder* when you change the paths for Windows? > And fourth, (because nobody expects the spanish inquisition), isn't the gratuitous difference a (small but) obvious wart? It's hardly the only wart we keep around for backwards compatibility. If it's going to change, it needs a proper transition period at the least. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sat Mar 24 04:22:04 2012 From: pje at telecommunity.com (PJ Eby) Date: Fri, 23 Mar 2012 23:22:04 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) Message-ID: On Mar 23, 2012 3:53 PM, "Carl Meyer" wrote: > > Hi PJ, > > On 03/23/2012 12:35 PM, PJ Eby wrote: > > AFAICT, virtualenvs are overkill for most development anyway. If you're > > not using distutils except to install dependencies, then configure > > distutils to install scripts and libraries to the same directory, and > > then do all your development in that directory. Presto! You now have a > > cross-platform "virtualenv". Want the scripts on your path? Add that > > directory to your path... or if on Windows, don't bother, since the > > current directory is usually on the path. (In fact, if you're only > > using easy_install to install your dependencies, you don't even need to > > edit the distutils configuration, just use "-md targetdir".) > > Creating and using a virtualenv is, in practice, _easier_ than any of > those alternatives, Really? As I said, I've never seen the need to try, since just installing stuff to a directory on PYTHONPATH seems quite easy enough for me. > that the "isolation from system site-packages" feature is quite popular > (the outpouring of gratitude when virtualenv went isolated-by-default a > few months ago was astonishing), and AFAIK none of your alternative > proposals support that at all. What is this isolation for, exactly? If you don't want site-packages on your path, why not use python -S? (Sure, nobody knows about these things, but surely that's a documentation problem, not a tooling problem.) Don't get me wrong, I don't have any deep objection to virtualenvs, I've just never seen the *point* (outside of the scenarios I mentioned), and thus don't see what great advantage will be had by rearranging layouts to make them shareable across platforms, when "throw stuff in a directory" seems perfectly serviceable for that use case already. Tools that *don't* support "just throw it in a directory" as a deployment option are IMO unpythonic -- practicality beats purity, after all. ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sat Mar 24 04:30:07 2012 From: pje at telecommunity.com (PJ Eby) Date: Fri, 23 Mar 2012 23:30:07 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6D2061.3080804@canterbury.ac.nz> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> Message-ID: On Mar 23, 2012 9:16 PM, "Greg Ewing" wrote: > > Glyph Lefkowitz wrote: > >> "do I have to resize my browser every time I visit a new site to get a decent width for reading". > > > If all sites left the width to the browser, then I would > be able to make my browser window a width that is comfortable > for me with my chosen font size and leave it that way. > The only time a site forces me to resize my window is when > it thinks it has a better idea than me how wide the text > should be. Weird - I have the exact *opposite* problem, where I have to resize my window because somebody *didn't* set their text max-width sanely (to a reasonable value based on ems instead of pixels), and I have nearly 1920 pixels of raw text spanning my screen. Bloody impossible to read that way. But I guess this is going to turn into one of those vi vs. emacs holy war things... (Personally, I prefer jEdit, or nano if absolutely forced to edit in a terminal. Heretical, I know. To the comfy chair with me!) -------------- next part -------------- An HTML attachment was scrubbed... URL: From van.lindberg at gmail.com Sat Mar 24 04:37:15 2012 From: van.lindberg at gmail.com (VanL) Date: Fri, 23 Mar 2012 22:37:15 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> Message-ID: On Mar 23, 2012 10:21 PM, "PJ Eby" wrote: > > > On Mar 23, 2012 4:19 PM, "VanL" wrote: > > > > Three notes. FIrst, distutils.cfg doesn't always work because it is centered around the idea of set paths that are the same each time - which doesn't always work with virtualenvs. > > And the virtualenv doesn't contain its own copy of distutils.cfg? It can, but a new one. Virtualenvs don't carry over the distutils.cfg from the main installation. Thus, using distutils.cfg in the virtualenv would require editing the .cfg for every new virtualenv-and it still wouldn't work all the time for the other reasons discussed. > > Second, most installer tools don't follow distutils.cfg. Even if that helps for python setup.py install, the other tools are still broken when you want to specify a layout. > > So, we should change Python to fix the broken tools that don't follow documented standards for configuring installation locations? If the documented functions don't work for the use cases, there is nothing else. Again, see the comments in PyPM. > > If the tools are that broken, aren't they going to break even *harder* when you change the paths for Windows? If people substitute on hard-coded value for another, does cross platform consistency help? And ifthat focuses attention on the new packaging APIs and the correct way to do it, isn't that even better? > > And fourth, (because nobody expects the spanish inquisition), isn't the gratuitous difference a (small but) obvious wart? > > It's hardly the only wart we keep around for backwards compatibility. If it's going to change, it needs a proper transition period at the least. Already proposed, making a transition over three releases with it starting as an off by default option in 3.3. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Sat Mar 24 04:38:19 2012 From: brian at python.org (Brian Curtin) Date: Fri, 23 Mar 2012 22:38:19 -0500 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: <4FB90B27-4778-4FE9-B93B-887A730BCBEC@gmail.com> References: <4FB90B27-4778-4FE9-B93B-887A730BCBEC@gmail.com> Message-ID: On Fri, Mar 23, 2012 at 18:38, Yury Selivanov wrote: > On 2012-03-23, at 7:28 PM, Brian Curtin wrote: >> This seems like it should have been a PEP, or maybe should become a PEP. > > Why? ?AFAIK Victor just proposes to add two new functions: monotonic() and > steady(). We just previously had "Drop time.monotonic() function, rename time.wallclock() to time.steady()" checked in a few weeks ago, and now we're renaming a variation on time.steady to time.monotonic? What's the next move? I'm not paying close attention here but there's a lot of movement going on. Figuring out the API before we get too deep is probably a good idea. From greg.ewing at canterbury.ac.nz Sat Mar 24 06:32:02 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 24 Mar 2012 18:32:02 +1300 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> Message-ID: <4F6D5C52.1030802@canterbury.ac.nz> PJ Eby wrote: > Weird - I have the exact *opposite* problem, where I have to resize my > window because somebody *didn't* set their text max-width sanely (to a > reasonable value based on ems instead of pixels), and I have nearly 1920 > pixels of raw text spanning my screen. If you don't want 1920-pixel-wide text, why make your browser window that large? -- Greg From jyasskin at gmail.com Sat Mar 24 06:37:45 2012 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Fri, 23 Mar 2012 22:37:45 -0700 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: On Fri, Mar 23, 2012 at 4:25 PM, Victor Stinner wrote: > Hi, > > time.steady(strict=True) looks to be confusing for most people, some > of them don't understand the purpose of the flag and others don't like > a flag changing the behaviour of the function. > > I propose to replace time.steady(strict=True) by time.monotonic(). > That would avoid the need of an ugly NotImplementedError: if the OS > has no monotonic clock, time.monotonic() will just not exist. > > So we will have: > > - time.time(): realtime, can be adjusted by the system administrator > (manually) or automatically by NTP > - time.clock(): monotonic clock on Windows, CPU time on UNIX > - time.monotonic(): monotonic clock, its speed may or may not be > adjusted by NTP but it only goes forward, may raise an OSError > - time.steady(): monotonic clock or the realtime clock, depending on > what is available on the platform (use monotonic in priority). may be > adjusted by NTP or the system administrator, may go backward. Please don't use the word "steady" for something different from what C++ means by it. C++'s term means "may not be adjusted at all, even by NTP; proceeds at as close to the rate of real time as the system can manage" (paraphrased). If the consensus in the Python community is that a C++-style "steady" clock is unnecessary, then feel free not to define it. If the consensus is that "monotonic" already means everything C++ means by "steady", that's fine with me too. I mentioned it because I thought it might be worth looking at what other languages were doing in this space, not because I thought it was a nice word that you should attach your own definitions to. Jeffrey From greg at krypto.org Sat Mar 24 08:15:00 2012 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 24 Mar 2012 00:15:00 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: Message-ID: On Tue, Mar 20, 2012 at 3:55 PM, John O'Connor wrote: > On Tue, Mar 20, 2012 at 6:38 PM, Georg Brandl wrote: > > recently I've grown a bit tired of seeing our default Sphinx theme, > > especially as so many other projects use it. > > I think regardless of the chosen style, giving the Python 3 docs a > different look and feel also has a psychological benefit that might > further encourage users to consider moving to Python 3. It could be a > bit of a wake-up call. > +3 Of course you do realize that the only possible outcome of this thread which is *literally* about painting the docs bike shed is to have a row of dynamic "change my css theme" buttons somewhere with one for each person that has piped up in this thread. Which would lead to a stateful docs web server with cookie preferences on which css to default to for each and every viewer. This doesn't end well... ;) Good luck (and thanks for trying, I like seeing the new styles!) -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Sat Mar 24 09:20:40 2012 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 24 Mar 2012 09:20:40 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: On Sat, Mar 24, 2012 at 00:36, Victor Stinner wrote: >> This seems like it should have been a PEP, or maybe should become a PEP. > > I replaced time.wallclock() by time.steady(strict=False) and > time.monotonic() by time.steady(strict=True). This change solved the > naming issue of time.wallclock(), but it was a bad idea to merge > monotonic() feature into time.steady(). It looks like everybody > agrees, am I wrong? Yes. As mentioned time.steady(i_mean_it=True) or time.steady(no_not_really=True) doesn't make any sense. Merging the methods may very well make sense, but it should then return a best case and have no flags. I think, as it has been hard to reach an agreement on this that the proposal to only make "stupid" functions that expose the system API's are the correct thing to do at the moment. > - time.time(): realtime, can be adjusted by the system administrator > (manually) or automatically by NTP Sure. > - time.clock(): monotonic clock on Windows, CPU time on UNIX This is for historical reasons, right, because this is what it is now? Would there be a problem in making time.clock() monotonic on Unix as well, if it exists? > - time.monotonic(): monotonic clock, its speed may or may not be > adjusted by NTP but it only goes forward, may raise an OSError > if the OS > has no monotonic clock, time.monotonic() will just not exist. Works for me, > - time.steady(): monotonic clock or the realtime clock, depending on > what is available on the platform (use monotonic in priority). may be > adjusted by NTP or the system administrator, may go backward. So it's time.may_or_may_not_be_steady() I don't mind the function, but the name should not be steady(). It's implementation is also so trivial that those who want a monotonic if it exists, but a normal clock otherwise, can simply just do > try: > ?return time.monotonic() > except (NotImplementError, OSError): > ?return time.time() themselves. //Lennart From solipsis at pitrou.net Sat Mar 24 09:39:28 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 24 Mar 2012 09:39:28 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? References: <4FB90B27-4778-4FE9-B93B-887A730BCBEC@gmail.com> Message-ID: <20120324093928.26640db9@pitrou.net> On Fri, 23 Mar 2012 22:38:19 -0500 Brian Curtin wrote: > On Fri, Mar 23, 2012 at 18:38, Yury Selivanov wrote: > > On 2012-03-23, at 7:28 PM, Brian Curtin wrote: > >> This seems like it should have been a PEP, or maybe should become a PEP. > > > > Why? ?AFAIK Victor just proposes to add two new functions: monotonic() and > > steady(). > > We just previously had "Drop time.monotonic() function, rename > time.wallclock() to time.steady()" checked in a few weeks ago, and now > we're renaming a variation on time.steady to time.monotonic? What's > the next move? > > I'm not paying close attention here but there's a lot of movement > going on. Figuring out the API before we get too deep is probably a > good idea. Agreed with Brian. Obviously the API needs further discussion, judging by Victor's own multiple changes of mind on the subject. Regards Antoine. From victor.stinner at gmail.com Sat Mar 24 11:35:43 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 24 Mar 2012 11:35:43 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: <4F6D2CBA.10205@pearwood.info> References: <4F6D0F1C.1080404@pearwood.info> <4F6D2CBA.10205@pearwood.info> Message-ID: > Does this mean that there are circumstances where monotonic will work for a > while, but then fail? No. time.monotonic() always work or always fail. If monotonic() failed, steady() doesn't call it again. > Otherwise, we would only need to check monotonic once, when the time module > is first loaded, rather than every time it is called. Instead of the above: > > # global to the time module > try: > ? ?monotonic() > except (NameError, OSError): > ? ?steady = time > else: > ? ?steady = monotonic I implemented steady differently to avoid the need of calling monotonic at Python startup. Calling monotonic at startup would be an extra useless system call. Victor From victor.stinner at gmail.com Sat Mar 24 11:37:07 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 24 Mar 2012 11:37:07 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: >> - time.monotonic(): monotonic clock, its speed may or may not be >> adjusted by NTP but it only goes forward, may raise an OSError >> - time.steady(): monotonic clock or the realtime clock, depending on >> what is available on the platform (use monotonic in priority). may be >> adjusted by NTP or the system administrator, may go backward. >> > > I am surprised that a clock with the name time.steady() has a looser > definition than one called time.monotonic(). To my mind a steady clock is by > definition monotonic but a monotonic one may or may not be steady. Do you suggest another name? Victor From victor.stinner at gmail.com Sat Mar 24 11:45:29 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 24 Mar 2012 11:45:29 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: <4F6D30C7.6010209@pearwood.info> References: <4F6D124A.3000607@pearwood.info> <4F6D30C7.6010209@pearwood.info> Message-ID: >> Oh, I was not aware of this issue. Do you suggest to not use >> QueryPerformanceCounter() on Windows to implement a monotonic clock? > > > I do not have an opinion on the best way to implement monotonic to guarantee > that it actually is monotonic. I opened an issue: http://bugs.python.org/issue14397 Victor From regebro at gmail.com Sat Mar 24 12:53:39 2012 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 24 Mar 2012 12:53:39 +0100 Subject: [Python-Dev] PEP 411 - request for pronouncement In-Reply-To: References: Message-ID: On Fri, Mar 23, 2012 at 10:51, Eli Bendersky wrote: > The PEP received mostly positive feedback. The only undecided point is > where to specify that the package is provisional. Currently the PEP > mandates to specify it in the documentation and in the docstring. > Other suggestions were to put it in the code, either as a > __provisional__ attribute on the module, or collect all such modules > in a single sys.provisional list. I'm not sure what the usecase is for checking in code if a module is provisional or not. It doesn't seem useful, and risks being unmaintained, especially when the flag is on the module itself. //Lennart From ncoghlan at gmail.com Sat Mar 24 13:19:27 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Mar 2012 22:19:27 +1000 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> Message-ID: On Sat, Mar 24, 2012 at 4:35 AM, PJ Eby wrote: > Just dumping things in a directory adjacent to the corresponding scripts is > the original virtualenv, and it still works just dandy -- most people just > don't *know* this.? (And again, if there are tools out there that *don't* > support single-directory virtualenvs, the best answer is probably to fix > them.) Not to mention that CPython gained native support for that layout in 2.6 via __main__.py files (although I stuffed up and forgot to add it to the What's New before the release). I'll chime in on the -1 side here as well. If you want *easy* cross-platform execution of __main__, use the -m switch. I'm obviously biased, since I'm the original author and primary maintainer of that switch, but it just makes all these problems with cross-platform questions and running from an installed copy vs running from source *go away*. Indeed, avoiding such cross-platform inconsistencies with regards to the location of stdlib modules was one of the major arguments in favour of adding the original incarnation of the switch way back in Python 2.4. To run the main clients (one for repo management, one for Django site management) in my current work project, I use "python -m pulpdist.manage_repos" and "python -m pulpdist.manage_site". It works from a source checkout (so long as I cd into src/ first), on an installed version, in a virtualenv, anywhere. I can easily run it on a different Python version just by changing the version I invoke. The commands would probably also work on at least Mac OS X and maybe even Windows (although I've never actually tried either of those, since PulpDist is targeted specifically at Linux systems). I may get around to installing at least the repo management client as a real script some day (since it will be more convenient for system administrators that way), but direct execution will never be the main way of executing it from a source checkout. So Van's proposal still smacks too much to me of change for change's sake. If you want an execution mechanism that is completely consistent across platforms (including virtual environments), then "-m" already exists. For direct execution, the proposal trades cross-version inconsistencies for cross-platform consistency. When we *already have* a consistent cross-platform mechanism in -m, the inevitable disruption involved in changing the Windows layout seems very hard to justify. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From brian at python.org Sat Mar 24 16:39:09 2012 From: brian at python.org (Brian Curtin) Date: Sat, 24 Mar 2012 10:39:09 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> Message-ID: On Sat, Mar 24, 2012 at 07:19, Nick Coghlan wrote: > On Sat, Mar 24, 2012 at 4:35 AM, PJ Eby wrote: >> Just dumping things in a directory adjacent to the corresponding scripts is >> the original virtualenv, and it still works just dandy -- most people just >> don't *know* this.? (And again, if there are tools out there that *don't* >> support single-directory virtualenvs, the best answer is probably to fix >> them.) > > Not to mention that CPython gained native support for that layout in > 2.6 via __main__.py files (although I stuffed up and forgot to add it > to the What's New before the release). > > I'll chime in on the -1 side here as well. If you want *easy* > cross-platform execution of __main__, use the -m switch. I love the -m option but what does it have to do with unifying the install layout? One is about executing __main__ and one is about a directory structure. > Indeed, avoiding such cross-platform inconsistencies with > regards to the location of stdlib modules was one of the major > arguments in favour of adding the original incarnation of the switch > way back in Python 2.4. Ok, so it is about directory structure, but about the standard library. Since part of this proposal is about Scripts vs. bin, how does -m help you there? From stephen at xemacs.org Sat Mar 24 16:39:38 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 24 Mar 2012 16:39:38 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: <4FB90B27-4778-4FE9-B93B-887A730BCBEC@gmail.com> Message-ID: On Sat, Mar 24, 2012 at 4:38 AM, Brian Curtin wrote: > On Fri, Mar 23, 2012 at 18:38, Yury Selivanov wrote: >> On 2012-03-23, at 7:28 PM, Brian Curtin wrote: >>> This seems like it should have been a PEP, or maybe should become a PEP. >> >> Why? ?AFAIK Victor just proposes to add two new functions: monotonic() and >> steady(). Need for PEPs is not determined by volume of content, but by amount of controversy and lack of clarity. Isn't it obvious that there's quite a bit of disagreement about the definitions of "monotonic" and "steady", and about whether these functions should be what they say they are or "best effort", and so on? +1 for a PEP. > We just previously had "Drop time.monotonic() function, rename > time.wallclock() to time.steady()" checked in a few weeks ago, and now > we're renaming a variation on time.steady to time.monotonic? What's > the next move? > > I'm not paying close attention here but there's a lot of movement > going on. Figuring out the API before we get too deep is probably a > good idea. I have been following the thread but don't have the technical knowledge to be sure what's going on. What I have decided is that I won't be using any function named time.steady() or time.monotonic() because neither one seems likely to guarantee the property it's named for, and by the time I have a use case (I don't have one now, I'm just an habitual lurker) I'll have forgotten the conclusion of this thread, but not the deep feelings of FUD. To get me on board (not that there's any particular reason you should care, but just in case), you're going to need to respect EIBTI. By that I mean that a monotonic clock is monotonic, and if not available at instantiation, an Exception will be raised. Similarly for a steady clock. There is no such thing as "best effort" here for clocks with these names. The default clock should be best effort. If that is for some reason "expensive", then there should be a "time.windup_clock()" to provide an unreliable resource- conserving clock. FWIW, I understand that (1) A monotonic clock is one that never goes backwards. If precision allows, it should always go forwards (ie, repeated calls should always produce strictly larger time values). (2) A steady clock is strictly monotonic, and when a discrepancy against "true time" is detected (however that might happen), it slews its visible clock until the discrepancy is eliminated, so that one clock second always means something "close" to one second. From martin at v.loewis.de Sat Mar 24 17:57:21 2012 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 24 Mar 2012 17:57:21 +0100 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: <61C7F677-7F04-4FD6-AC66-4595374F402A@gmail.com> Message-ID: <4F6DFCF1.9000407@v.loewis.de> > I don't see what is the use case requiring a is truly monotonic clock. A clock that is purely monotonic may not be useful. However, people typically imply that it will have a certain minimum progress (seconds advanced/real seconds passed). Then you can use it for timeouts. Regards, Martin From pje at telecommunity.com Sun Mar 25 00:34:31 2012 From: pje at telecommunity.com (PJ Eby) Date: Sat, 24 Mar 2012 19:34:31 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F6D5C52.1030802@canterbury.ac.nz> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> Message-ID: On Sat, Mar 24, 2012 at 1:32 AM, Greg Ewing wrote: > PJ Eby wrote: > > Weird - I have the exact *opposite* problem, where I have to resize my >> window because somebody *didn't* set their text max-width sanely (to a >> reasonable value based on ems instead of pixels), and I have nearly 1920 >> pixels of raw text spanning my screen. >> > > If you don't want 1920-pixel-wide text, why make your > browser window that large? > Not every tab in my browser is text for reading; some are apps that need the extra horizontal space. That is, they have more than one column of text or data -- but no individual column spans anywhere near the full width. (Google Docs, for example, shows me two pages at a time when I'm reading a PDF.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 25 00:46:28 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2012 09:46:28 +1000 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F68607B.7060307@gmail.com> <4F688F48.9010101@gmail.com> <4F68FB76.7010303@skippinet.com.au> <4F68FDE7.40505@gmail.com> <4F694DFB.7050400@netwok.org> <4F69E40C.6050901@gmail.com> <4F69EB9D.1060701@egenix.com> <4F6B345C.1020406@gmail.com> <4F6B760D.3010004@gmail.com> <4D5CC12CFE8E455FBD5BD49A3A295A1C@gmail.com> Message-ID: By dodging the issue entirely - anything I might want to regularly run from a source checkout I execute with -m. It gets sys.path right automatically and I don't need to care about platform specific executable naming conventions. -- Sent from my phone, thus the relative brevity :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sun Mar 25 01:41:42 2012 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 25 Mar 2012 11:41:42 +1100 Subject: [Python-Dev] Playing with a new theme for the docs References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> Message-ID: <87y5qpikah.fsf@benfinney.id.au> PJ Eby writes: > On Sat, Mar 24, 2012 at 1:32 AM, Greg Ewing wrote: > > > If you don't want 1920-pixel-wide text, why make your browser window > > that large? > > Not every tab in my browser is text for reading; some are apps that > need the extra horizontal space. So, again, why make your browser window *for reading text* that large? You have control over how large your window size is, and if you have purposes so different that they demand different widths, then you can easily make different-width windows. Everyone has different needs for how large the text should be and how much of it should go across the window. Every one of us is in a minority when it comes to those needs; that's exactly what a configuration setting is good for. It's madness to expect web designers to hobble the flexibility of a web page to cater preferentially for one minority over others. -- \ ?Come on Milhouse, there?s no such thing as a soul! It?s just | `\ something they made up to scare kids, like the Boogie Man or | _o__) Michael Jackson.? ?Bart, _The Simpsons_ | Ben Finney From v+python at g.nevcal.com Sun Mar 25 03:35:10 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 24 Mar 2012 18:35:10 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <87y5qpikah.fsf@benfinney.id.au> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> Message-ID: <4F6E764E.5070603@g.nevcal.com> On 3/24/2012 5:41 PM, Ben Finney wrote: > It's madness to expect web designers to hobble the flexibility of a web > page to cater preferentially for one minority over others. But largely, the 99% that makes the rest of them look bad, do, in fact, do exactly that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Sun Mar 25 08:34:44 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 25 Mar 2012 08:34:44 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 Message-ID: Here's another try, mainly with default browser font size, more contrast and collapsible sidebar again: http://www.python.org/~gbrandl/build/html2/ I've also added a little questionable gimmick to the sidebar (when you collapse it and expand it again, the content is shown at your current scroll location). Have fun! Georg From stephen at xemacs.org Sun Mar 25 08:56:53 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 25 Mar 2012 08:56:53 +0200 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <87y5qpikah.fsf@benfinney.id.au> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> Message-ID: On Sun, Mar 25, 2012 at 1:41 AM, Ben Finney wrote: > PJ Eby writes: >> Not every tab in my browser is text for reading; some are apps that >> need the extra horizontal space. > > So, again, why make your browser window *for reading text* that large? Because he prefers controlling the content viewed by selecting tabs rather than selecting windows, no doubt. But since he's arguing the other end in the directory layout thread (where he says there are many special ways to invoke Python so that having different layouts on different platforms is easy to work around), I can't give much weight to his preference here. Anyway, CSS is supposed to allow the user to impose such constraints herself, so Philip "should" do so with a local style, rather than ask designers to do it globally. > It's madness to expect web designers to hobble the flexibility of a web > page to cater preferentially for one minority over others. No, as Glenn points out, designers (I wouldn't call them *web* designers since they clearly have no intention of taking advantage of the power of the web in design, even if they incorporate links in their pages!) frequently do exactly that. (The minority of one in question being the designer himself!) So it's rational to expect it. :-( However, I believe that CSS also gives us the power to undo such bloodymindedness, though I've never gone to the trouble of learning how. Steve From ben+python at benfinney.id.au Sun Mar 25 09:19:38 2012 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 25 Mar 2012 18:19:38 +1100 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 References: Message-ID: <87fwcxi1v9.fsf@benfinney.id.au> Georg Brandl writes: > Here's another try, mainly with default browser font size, more > contrast and collapsible sidebar again: > > http://www.python.org/~gbrandl/build/html2/ Great! You've improved it nicely. I especially like that you have done the collapsible sidebar with graceful degradation: the content is quite accessible without ECMAscript. Can you make the link colors (in the body and sidebar) follow the usual conventions: use a blue colour for unvisited links, and a purple colour for visited links so it's more obvious where links are and where the reader has already been. -- \ ?I distrust those people who know so well what God wants them | `\ to do to their fellows, because it always coincides with their | _o__) own desires.? ?Susan Brownell Anthony, 1896 | Ben Finney From eliben at gmail.com Sun Mar 25 09:26:02 2012 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 25 Mar 2012 09:26:02 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: On Sun, Mar 25, 2012 at 08:34, Georg Brandl wrote: > Here's another try, mainly with default browser font size, more contrast and > collapsible sidebar again: > > http://www.python.org/~gbrandl/build/html2/ > > I've also added a little questionable gimmick to the sidebar (when you collapse > it and expand it again, the content is shown at your current scroll location). > Nice and clean. +1 Eli From v+python at g.nevcal.com Sun Mar 25 09:46:18 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 25 Mar 2012 00:46:18 -0700 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: <4F6ECD4A.6000301@g.nevcal.com> On 3/24/2012 11:34 PM, Georg Brandl wrote: > I've also added a little questionable gimmick to the sidebar (when you collapse > it and expand it again, the content is shown at your current scroll location). It would be educational to see how you pulled that trick! I will look if I get time. However, in playing with it, it has the definite disadvantage of forcing the user to position/click the mouse twice, if the goal is not to collapse the sidebar, but simply to make the content visible. Might there be an additional way to move the content, perhaps by a click in the blank portions of the sidebar (above the top or below the bottom of the content, in the shaded area), that it would bring the content to view? The position chosen for the content could happily be the same position you choose when doing the collapse/expand dance, I have no quibble with that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From __peter__ at web.de Sun Mar 25 10:06:15 2012 From: __peter__ at web.de (Peter Otten) Date: Sun, 25 Mar 2012 10:06:15 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 References: Message-ID: Georg Brandl wrote: > Here's another try, mainly with default browser font size, more contrast > and collapsible sidebar again: > > http://www.python.org/~gbrandl/build/html2/ Nice! Lightweight and readable. >From the bikeshedding department: * Inlined code doesn't need the gray background. The bold font makes it stand out enough. * Instead of the box consider italics or another color for [New in ...] text. * Nobody is going to switch off the prompts for interactive sessions. * Maybe the Next/Previous Page headers on the left could link to the respective page. * Short descriptions in the module index don't need italics. * The disambiguation in the index table could use a different style instead of the parentheses. From storchaka at gmail.com Sun Mar 25 10:11:53 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Mar 2012 11:11:53 +0300 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: 25.03.12 09:34, Georg Brandl ???????(??): > Here's another try, mainly with default browser font size, more contrast and > collapsible sidebar again: It may be worth now the line-height reduce too? > I've also added a little questionable gimmick to the sidebar (when you collapse > it and expand it again, the content is shown at your current scroll location). What if move the search field to the header and the footer? There a lot of free space. "Report a Bug" and "Show Source" can also be moved to the footer, if fit. The footer height is too big now, I think that you can reduce the copyright and technical information from 4 to 2 lines. From storchaka at gmail.com Sun Mar 25 10:44:18 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Mar 2012 11:44:18 +0300 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: 25.03.12 11:06, Peter Otten ???????(??): > * Inlined code doesn't need the gray background. The bold font makes it > stand out enough. I believe that the gray background is good, but it should make it lighter. > * Instead of the box consider italics or another color for [New in ...] > text. Yes, the border around "New in" and "Changed in" looks not good-looking. Maybe a very light colored background with no border or underlined italic will look better. > * Maybe the Next/Previous Page headers on the left could link to the > respective page. Do you mean next/previous links in header/footer? I totally agree with your other comments, including the admiration of the current version of the design. From andrew.svetlov at gmail.com Sun Mar 25 10:53:20 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 25 Mar 2012 11:53:20 +0300 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: I like to always see "Quick search" widget without scrolling page to top. Is it possible? Or maybe you can embed some keyboard shortcut for quick jump to search input box? On Sun, Mar 25, 2012 at 11:44 AM, Serhiy Storchaka wrote: > 25.03.12 11:06, Peter Otten ???????(??): > >> * Inlined code doesn't need the gray background. The bold font makes it >> stand out enough. > > > I believe that the gray background is good, but it should make it lighter. > > >> * Instead of the box consider italics or another color for [New in ...] >> text. > > > Yes, the border around "New in" and "Changed in" looks not good-looking. > Maybe a very light colored background with no border or underlined italic > will look better. > > >> * Maybe the Next/Previous Page headers on the left could link to the >> respective page. > > > Do you mean next/previous links in header/footer? > > I totally agree with your other comments, including the admiration of the > current version of the design. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov From stephen at xemacs.org Sun Mar 25 10:54:51 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 25 Mar 2012 10:54:51 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: In the header next to "Python v3.3a1 documentation" there is a "?" symbol, which suggests something can be expanded. Knowing that there are many versions of the documentation, I thought it might bring up a menu of versions. But clicking does nothing. Is that intentional? I guess it's supposed to mean "go to top" but that wasn't obvious to me. I think the clickable areas in the header and footer should be indicated with the usual coloring (either the scheme you currently use, or perhaps as Ben suggests blue and purple as in "traditional" HTML documents). I agree that what you do looks nice and is sufficiently functional once you realize it, but I've seen a lot of research that indicates that up to 60% of users can't find all the links on a page unless they're explicitly marked. (In one focus group 4 of 14 users never found a menu that took up 40% of the area of the page!) My first impression of the "questionable feature" that the sidebar is aligned with the scroll position when expanded is that it's useful. It looks pretty good without CSS, too! On Sun, Mar 25, 2012 at 8:34 AM, Georg Brandl wrote: > Here's another try, mainly with default browser font size, more contrast and > collapsible sidebar again: > > http://www.python.org/~gbrandl/build/html2/ > > I've also added a little questionable gimmick to the sidebar (when you collapse > it and expand it again, the content is shown at your current scroll location). > > Have fun! > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/stephen%40xemacs.org From storchaka at gmail.com Sun Mar 25 10:56:36 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Mar 2012 11:56:36 +0300 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: 25.03.12 09:34, Georg Brandl ???????(??): > I've also added a little questionable gimmick to the sidebar (when you collapse > it and expand it again, the content is shown at your current scroll location). I'm not sure if this is possible, and how good it would look like, but I have one crazy idea. What if transform the sidebar to collapsable floating box in the upper right corner? From stefan at bytereef.org Sun Mar 25 11:04:22 2012 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 25 Mar 2012 11:04:22 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: <20120325090422.GA11869@sleipnir.bytereef.org> Andrew Svetlov wrote: > I like to always see "Quick search" widget without scrolling page to > top. Is it possible? Do you mean a fixed search box like this one? http://coq.inria.fr/documentation Please don't do this, I find scrolling exceptionally distracting in the presence of fixed elements. Stefan Krah From andrew.svetlov at gmail.com Sun Mar 25 11:10:21 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 25 Mar 2012 12:10:21 +0300 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <20120325090422.GA11869@sleipnir.bytereef.org> References: <20120325090422.GA11869@sleipnir.bytereef.org> Message-ID: On Sun, Mar 25, 2012 at 12:04 PM, Stefan Krah wrote: > Andrew Svetlov wrote: >> I like to always see "Quick search" widget without scrolling page to >> top. Is it possible? > > Do you mean a fixed search box like this one? > > http://coq.inria.fr/documentation > No. You are right, it's distracting. Maybe narrow persistent line with searchbox on the top will be better. But just jump to searchbox by shortcut is good enough for me also. > > Please don't do this, I find scrolling exceptionally distracting in the > presence of fixed elements. > > > > Stefan Krah > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov From hs at ox.cx Sun Mar 25 11:39:12 2012 From: hs at ox.cx (Hynek Schlawack) Date: Sun, 25 Mar 2012 11:39:12 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: <29667EB3-1BCE-41A6-90FB-4C4BD5BE4FA4@ox.cx> Hi Georg, Am 25.03.2012 um 08:34 schrieb Georg Brandl: > Here's another try, mainly with default browser font size, more contrast and > collapsible sidebar again: > > http://www.python.org/~gbrandl/build/html2/ I really like it! Only one nitpick: If a header follows on a ?seealso?, the vertical rhythm is slightly broken like in https://skitch.com/hyneks/8c6j8/ Minor detail but should be easy to fix. :) Cheers, Hynek From peck at us.ibm.com Sun Mar 25 12:00:58 2012 From: peck at us.ibm.com (Jon K Peck) Date: Sun, 25 Mar 2012 04:00:58 -0600 Subject: [Python-Dev] AUTO: Jon K Peck is out of the office (returning 03/30/2012) Message-ID: I am out of the office until 03/30/2012. I will be out of the office through Friday, March 30. I expect to have some email access but may be delayed in responding. Note: This is an automated response to your message "Python-Dev Digest, Vol 104, Issue 91" sent on 03/25/2012 1:19:50. This is the only notification you will receive while this person is away. From __peter__ at web.de Sun Mar 25 12:37:38 2012 From: __peter__ at web.de (Peter Otten) Date: Sun, 25 Mar 2012 12:37:38 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 References: Message-ID: Serhiy Storchaka wrote: >> * Maybe the Next/Previous Page headers on the left could link to the >> respective page. > > Do you mean next/previous links in header/footer? No, I mean the two sections in the sidebar on the left, below "Table of Contents". From stephen at xemacs.org Sun Mar 25 13:09:23 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 25 Mar 2012 13:09:23 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <20120325090422.GA11869@sleipnir.bytereef.org> References: <20120325090422.GA11869@sleipnir.bytereef.org> Message-ID: On Sun, Mar 25, 2012 at 11:04 AM, Stefan Krah wrote: > Do you mean a fixed search box like this one? > > http://coq.inria.fr/documentation > > Please don't do this, I find scrolling exceptionally distracting in the > presence of fixed elements. Does it bother you when the header is fixed and contains the search box? I prefer that arrangement, anyway. From phd at phdru.name Sun Mar 25 13:36:57 2012 From: phd at phdru.name (Oleg Broytman) Date: Sun, 25 Mar 2012 15:36:57 +0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: <20120325113657.GA13240@iskra.aviel.ru> On Sun, Mar 25, 2012 at 08:34:44AM +0200, Georg Brandl wrote: > http://www.python.org/~gbrandl/build/html2/ Perfect! I like it! Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From anacrolix at gmail.com Sun Mar 25 13:44:09 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Sun, 25 Mar 2012 19:44:09 +0800 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <20120325113657.GA13240@iskra.aviel.ru> References: <20120325113657.GA13240@iskra.aviel.ru> Message-ID: Is nice yes?! When I small the nav bar, then embiggen it again, the text centers vertically. It's in the wrong place. The new theme is very minimal, perhaps a new color should be chosen. We've done green, what about orange, brown or blue? -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Mar 25 14:07:52 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Mar 2012 14:07:52 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 References: Message-ID: <20120325140752.7f783801@pitrou.net> On Sun, 25 Mar 2012 08:34:44 +0200 Georg Brandl wrote: > Here's another try, mainly with default browser font size, more contrast and > collapsible sidebar again: > > http://www.python.org/~gbrandl/build/html2/ > > I've also added a little questionable gimmick to the sidebar (when you collapse > it and expand it again, the content is shown at your current scroll location). The gimmick is buggy (when you collapse then expand it in the middle, and then scroll up, the sidebar content disappears after scrolling), and in the end quite confusing. Also I think there should be some jquery animation when collapsing/expanding. I think the "New in version..." and "Changed in version..." styles stand out in the wrong kind of way (as in "drawn by a 8-year old with a pen and ruler", perhaps). Perhaps you want some coloured background instead? (or, better, an icon - by the way, warnings also scream for an icon IMO) Otherwise, not sure what problem this new theme solves, but it looks ok. Regards Antoine. From stefan at bytereef.org Sun Mar 25 14:34:07 2012 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 25 Mar 2012 14:34:07 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <20120325090422.GA11869@sleipnir.bytereef.org> Message-ID: <20120325123407.GA12989@sleipnir.bytereef.org> Stephen J. Turnbull wrote: > > Do you mean a fixed search box like this one? > > > > http://coq.inria.fr/documentation > > > > Please don't do this, I find scrolling exceptionally distracting in the > > presence of fixed elements. > > Does it bother you when the header is fixed and contains > the search box? I prefer that arrangement, anyway. Do you have an example website? In general fixed elements distract me greatly. This also applies to the '<<' element in the collapsible sidebar. When I'm scrolling, it's almost the center of my attention (when I should be focusing on the text). Perhaps users can discover the collapsible sidebar without the '<<' hint? Or let it move up like in the existing version? Stefan Krah From stephen at xemacs.org Sun Mar 25 14:47:21 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 25 Mar 2012 14:47:21 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <20120325123407.GA12989@sleipnir.bytereef.org> References: <20120325090422.GA11869@sleipnir.bytereef.org> <20120325123407.GA12989@sleipnir.bytereef.org> Message-ID: On Sun, Mar 25, 2012 at 2:34 PM, Stefan Krah wrote: > Stephen J. Turnbull wrote: >> Does it bother you when the header is fixed and contains >> the search box? ?I prefer that arrangement, anyway. > > Do you have an example website? Not with just a header. http://turnbull.sk.tsukuba.ac.jp/Teach/IntroSES/ is a (very primitive and not stylistically improved in years) example of a frame-based layout that I use some of my classes. I would put a search field in the top frame (if I had one. :-) But I suppose you would find the fixed sidebar distracting. > In general fixed elements distract me > greatly. This also applies to the '<<' element in the collapsible > sidebar. When I'm scrolling, it's almost the center of my attention > (when I should be focusing on the text). I suspect you're unusual in that, but I guess it just is going to bug you no matter what, and I personally don't have *that* strong a preference either way. From murman at gmail.com Sun Mar 25 15:23:16 2012 From: murman at gmail.com (Michael Urman) Date: Sun, 25 Mar 2012 08:23:16 -0500 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <20120325140752.7f783801@pitrou.net> References: <20120325140752.7f783801@pitrou.net> Message-ID: On Sun, Mar 25, 2012 at 07:07, Antoine Pitrou wrote: >> >> I've also added a little questionable gimmick to the sidebar (when you collapse >> it and expand it again, the content is shown at your current scroll location). > > The gimmick is buggy (when you collapse then expand it in the middle, > and then scroll up, the sidebar content disappears after scrolling), > and in the end quite confusing. It also seems not to handle window resizes very well right now. It appears to choose the height for the vertical bar when shown, and then when the text next to it reflows to a new length, the bar can become longer or shorter than necessary. On the one hand this makes it hard to get the sidebar content to show at the bottom of the page; on the other, I believe it mitigates potential problems if sidebar content is too long for the window size. -- Michael Urman From ned at nedbatchelder.com Sun Mar 25 17:26:26 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sun, 25 Mar 2012 11:26:26 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: <4F6F3922.2020202@nedbatchelder.com> On 3/25/2012 2:34 AM, Georg Brandl wrote: > Here's another try, mainly with default browser font size, more contrast and > collapsible sidebar again: > > http://www.python.org/~gbrandl/build/html2/ Georg, thanks so much for taking on this thankless task with grace and skill. It can't be easy dealing with the death by a thousand tweaks, and I know I've contributed to the flurry. Nowhere on the page is a simple link to the front page of python.org. Perhaps the traditional upper-left corner could get a bread-crumb before "Python v3.3a1 documentation" that simply links to python.org. Maybe, use the word Python that is already there: [Python] ? [v3.3a1 documentation]. People do arrive at doc pages via search engines, and connecting the docs up to the rest of the site would be a good thing. Speaking of links to other pages, the doc front page, under "Other resources" lists Guido's Essays and New-style Classes second and third. These each point to extremely outdated material ("Unifying types and classes in 2.2", and "Unfortunately, new-style classes have not yet been integrated into Python's standard documention." ??). Another, "Other Doc Collections," points to an empty apache-style directory listing :-(. These links should be removed if we don't want to keep those sections of the site up-to-date. I know this is not strictly part of the redesign, but I just noticed it and thought I would throw it out there. I agree about the outlined style for "New" notices, and the red for deprecation is extremely alarming! :) I'll make one last plea for not justifying short paragraphs full of unbreakable elements, but I know I am in the minority. > I've also added a little questionable gimmick to the sidebar (when you collapse > it and expand it again, the content is shown at your current scroll location). I especially like using dynamic elements on a page to adapt to a reader's needs. I have some other ideas that I'll try to cobble together. --Ned. > Have fun! > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com > From mail at timgolden.me.uk Sun Mar 25 17:36:44 2012 From: mail at timgolden.me.uk (Tim Golden) Date: Sun, 25 Mar 2012 16:36:44 +0100 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F6F3922.2020202@nedbatchelder.com> References: <4F6F3922.2020202@nedbatchelder.com> Message-ID: <4F6F3B8C.5030802@timgolden.me.uk> On 25/03/2012 16:26, Ned Batchelder wrote: > Georg, thanks so much for taking on this thankless task with grace and > skill. It can't be easy dealing with the death by a thousand tweaks Seconded. I'm constantly edified by the way in which people in the community respond to even quite abrupt criticism in a constructive, open and often humorous manner. (As I've said before, I'm also impressed by the way in which people are prepared to come back and apologise / acknowledge that they had a moment of jerkiness). TJG From tjreedy at udel.edu Sun Mar 25 17:54:10 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Mar 2012 11:54:10 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: On 3/25/2012 2:34 AM, Georg Brandl wrote: > Here's another try, mainly with default browser font size, more contrast Untrue. You still changed the high contrast dark blue to the same low contrast light blue for builtin names, etc. What problem do you think you are trying to solve by making the doc difficult and even PAINFUL for me to read? - a lot more than 1 -- Terry Jan Reedy From stefan at bytereef.org Sun Mar 25 18:05:03 2012 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 25 Mar 2012 18:05:03 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <20120325090422.GA11869@sleipnir.bytereef.org> <20120325123407.GA12989@sleipnir.bytereef.org> Message-ID: <20120325160503.GA13935@sleipnir.bytereef.org> Stephen J. Turnbull wrote: > Not with just a header. http://turnbull.sk.tsukuba.ac.jp/Teach/IntroSES/ > is a (very primitive and not stylistically improved in years) example > of a frame-based layout that I use some of my classes. I would > put a search field in the top frame (if I had one. :-) But I suppose you > would find the fixed sidebar distracting. No, if the whole sidebar is fixed I don't mind (though I'm not a fan of frames). The top frame takes up space though. > > In general fixed elements distract me > > greatly. This also applies to the '<<' element in the collapsible > > sidebar. When I'm scrolling, it's almost the center of my attention > > (when I should be focusing on the text). > > I suspect you're unusual in that, but I guess it just is going > to bug you no matter what, and I personally don't have > *that* strong a preference either way. Maybe. It's hard to determine. It's just that I don't see fixed search boxes or fixed elements like '<<' on big name websites (who may or may not have usability departments). So it is at least possible that such features have always been controversial. Stefan Krah From g.brandl at gmx.net Sun Mar 25 18:15:20 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 25 Mar 2012 18:15:20 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <87fwcxi1v9.fsf@benfinney.id.au> References: <87fwcxi1v9.fsf@benfinney.id.au> Message-ID: On 25.03.2012 09:19, Ben Finney wrote: > Georg Brandl writes: > >> Here's another try, mainly with default browser font size, more >> contrast and collapsible sidebar again: >> >> http://www.python.org/~gbrandl/build/html2/ > > Great! You've improved it nicely. I especially like that you have > done the > collapsible sidebar with graceful degradation: the content is quite > accessible without ECMAscript. > > Can you make the link colors (in the body and sidebar) follow the usual > conventions: use a blue colour for unvisited links, and a purple colour > for visited links so > it's more obvious where links are and where the reader has already been. Thanks. Same colors for visited and unvisited links is indeed an oversight on my part. I'll put that in the final version. Georg From g.brandl at gmx.net Sun Mar 25 18:17:14 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 25 Mar 2012 18:17:14 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: On 25.03.2012 10:06, Peter Otten wrote: > Georg Brandl wrote: > >> Here's another try, mainly with default browser font size, more contrast >> and collapsible sidebar again: >> >> http://www.python.org/~gbrandl/build/html2/ > > Nice! Lightweight and readable. > >>From the bikeshedding department: > > * Inlined code doesn't need the gray background. The bold font makes it > stand out enough. > * Instead of the box consider italics or another color for [New in ...] > text. Yes, I'll revert to italics as most people don't seem to like the colored boxes. > * Nobody is going to switch off the prompts for interactive sessions. You'll laugh, but that was a pretty often-wished feature so that copy-paste gets easier. It'll certainly stay. > * Maybe the Next/Previous Page headers on the left could link to the > respective page. I see no reason since the links below already do. > * Short descriptions in the module index don't need italics. > * The disambiguation in the index table could use a different style instead > of the parentheses. These two would need to be changed in Sphinx. Georg From g.brandl at gmx.net Sun Mar 25 18:23:45 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 25 Mar 2012 18:23:45 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F6F3922.2020202@nedbatchelder.com> References: <4F6F3922.2020202@nedbatchelder.com> Message-ID: On 25.03.2012 17:26, Ned Batchelder wrote: > On 3/25/2012 2:34 AM, Georg Brandl wrote: >> Here's another try, mainly with default browser font size, more contrast and >> collapsible sidebar again: >> >> http://www.python.org/~gbrandl/build/html2/ > Georg, thanks so much for taking on this thankless task with grace and > skill. It can't be easy dealing with the death by a thousand tweaks, > and I know I've contributed to the flurry. > > Nowhere on the page is a simple link to the front page of python.org. > Perhaps the traditional upper-left corner could get a bread-crumb before > "Python v3.3a1 documentation" that simply links to python.org. Maybe, > use the word Python that is already there: [Python] ? [v3.3a1 > documentation]. People do arrive at doc pages via search engines, and > connecting the docs up to the rest of the site would be a good thing. Indeed. I'm trying to tweak that right now. > Speaking of links to other pages, the doc front page, under "Other > resources" lists Guido's Essays and New-style Classes second and third. > These each point to extremely outdated material ("Unifying types and > classes in 2.2", and "Unfortunately, new-style classes have not yet been > integrated into Python's standard documention." ??). Another, "Other > Doc Collections," points to an empty apache-style directory listing > :-(. These links should be removed if we don't want to keep those > sections of the site up-to-date. I know this is not strictly part of > the redesign, but I just noticed it and thought I would throw it out there. That would be best to capture in a bugs.python.org issue, I think. > I agree about the outlined style for "New" notices, and the red for > deprecation is extremely alarming! :) Changed. > I'll make one last plea for not justifying short paragraphs full of > unbreakable elements, but I know I am in the minority. :) >> I've also added a little questionable gimmick to the sidebar (when you collapse >> it and expand it again, the content is shown at your current scroll location). > I especially like using dynamic elements on a page to adapt to a > reader's needs. I have some other ideas that I'll try to cobble together. That would be great. Georg From storchaka at gmail.com Sun Mar 25 18:25:10 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Mar 2012 19:25:10 +0300 Subject: [Python-Dev] PEP 393 decode() oddity Message-ID: PEP 393 (Flexible String Representation) is, without doubt, one of the pearls of the Python 3.3. In addition to reducing memory consumption, it also often leads to a corresponding increase in speed. In particular, the string encoding now in 1.5-3 times faster. But decoding is not so good. Here are the results of measuring the performance of the decoding of the 1000-character string consisting of characters from different ranges of the Unicode, for three versions of Python -- 2.7.3rc2, 3.2.3rc2+ and 3.3.0a1+. Little-endian 32-bit i686 builds, gcc 4.4. encoding string 2.7 3.2 3.3 ascii " " * 1000 5.4 5.3 1.2 latin1 " " * 1000 1.8 1.7 1.3 latin1 "\u0080" * 1000 1.7 1.6 1.0 utf-8 " " * 1000 6.7 2.4 2.1 utf-8 "\u0080" * 1000 12.2 11.0 13.0 utf-8 "\u0100" * 1000 12.2 11.1 13.6 utf-8 "\u0800" * 1000 14.7 14.4 17.2 utf-8 "\u8000" * 1000 13.9 13.3 17.1 utf-8 "\U00010000" * 1000 17.3 17.5 21.5 utf-16le " " * 1000 5.5 2.9 6.5 utf-16le "\u0080" * 1000 5.5 2.9 7.4 utf-16le "\u0100" * 1000 5.5 2.9 8.9 utf-16le "\u0800" * 1000 5.5 2.9 8.9 utf-16le "\u8000" * 1000 5.5 7.5 21.3 utf-16le "\U00010000" * 1000 9.6 12.9 30.1 utf-16be " " * 1000 5.5 3.0 9.0 utf-16be "\u0080" * 1000 5.5 3.1 9.8 utf-16be "\u0100" * 1000 5.5 3.1 10.4 utf-16be "\u0800" * 1000 5.5 3.1 10.4 utf-16be "\u8000" * 1000 5.5 6.6 21.2 utf-16be "\U00010000" * 1000 9.6 11.2 28.9 utf-32le " " * 1000 10.2 10.4 15.1 utf-32le "\u0080" * 1000 10.0 10.4 16.5 utf-32le "\u0100" * 1000 10.0 10.4 19.8 utf-32le "\u0800" * 1000 10.0 10.4 19.8 utf-32le "\u8000" * 1000 10.1 10.4 19.8 utf-32le "\U00010000" * 1000 11.7 11.3 20.2 utf-32be " " * 1000 10.0 11.2 15.0 utf-32be "\u0080" * 1000 10.1 11.2 16.4 utf-32be "\u0100" * 1000 10.0 11.2 19.7 utf-32be "\u0800" * 1000 10.1 11.2 19.7 utf-32be "\u8000" * 1000 10.1 11.2 19.7 utf-32be "\U00010000" * 1000 11.7 11.2 20.2 The first oddity in that the characters from the second half of the Latin1 table decoded faster than the characters from the first half. I think that the characters from the first half of the table must be decoded as quickly. The second sad oddity in that UTF-16 decoding in 3.3 is much slower than even in 2.7. Compared with 3.2 decoding is slower in 2-3 times. This is a considerable regress. UTF-32 decoding is also slowed down by 1.5-2 times. The fact that in some cases UTF-8 decoding also slowed, is not surprising. I believe, that on a platform with a 64-bit long, there may be other oddities. How serious a problem this is for the Python 3.3 release? I could do the optimization, if someone is not working on this already. -------------- next part -------------- A non-text attachment was scrubbed... Name: bench_decode.py Type: text/x-python Size: 806 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bench_decode-2.py Type: text/x-python Size: 810 bytes Desc: not available URL: From g.brandl at gmx.net Sun Mar 25 18:32:39 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 25 Mar 2012 18:32:39 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: On 25.03.2012 17:54, Terry Reedy wrote: > On 3/25/2012 2:34 AM, Georg Brandl wrote: >> Here's another try, mainly with default browser font size, more contrast > > Untrue. You still changed the high contrast dark blue to the same low > contrast light blue for builtin names, etc. What problem do you think > you are trying to solve by making the doc difficult and even PAINFUL for > me to read? > > - a lot more than 1 "More contrast" was meant in comparison to iteration #1. Hmm, don't you think you'll get used to the new style in a while? The link color is not actually that light in comparison. Of course you can always use a user stylesheet to override our choices. Georg From g.brandl at gmx.net Sun Mar 25 18:37:12 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 25 Mar 2012 18:37:12 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <20120325090422.GA11869@sleipnir.bytereef.org> Message-ID: On 25.03.2012 13:09, Stephen J. Turnbull wrote: > On Sun, Mar 25, 2012 at 11:04 AM, Stefan Krah wrote: > >> Do you mean a fixed search box like this one? >> >> http://coq.inria.fr/documentation >> >> Please don't do this, I find scrolling exceptionally distracting in the >> presence of fixed elements. > > Does it bother you when the header is fixed and contains > the search box? I prefer that arrangement, anyway. I think this idea has some merit. I'd prefer it to be tried out and implemented in a second step though (maybe by someone else, even? ;) Georg From solipsis at pitrou.net Sun Mar 25 19:01:37 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Mar 2012 19:01:37 +0200 Subject: [Python-Dev] PEP 393 decode() oddity References: Message-ID: <20120325190137.7f0ffce0@pitrou.net> Hi, On Sun, 25 Mar 2012 19:25:10 +0300 Serhiy Storchaka wrote: > > But decoding is not so good. The general problem with decoding is that you don't know up front what width (1, 2 or 4 bytes) is required for the result. The solution is either to compute the width in a first pass (and decode in a second pass), or decode in a single pass and enlarge the result on the fly when needed. Both incur a slowdown compared to a single-size representation. > The first oddity in that the characters from the second half of the > Latin1 table decoded faster than the characters from the first half. I > think that the characters from the first half of the table must be > decoded as quickly. It's probably a measurement error on your part. > The second sad oddity in that UTF-16 decoding in 3.3 is much slower than > even in 2.7. Compared with 3.2 decoding is slower in 2-3 times. This is > a considerable regress. UTF-32 decoding is also slowed down by 1.5-2 times. I don't think UTF-32 is used a lot. As for UTF-16, if you can optimize it then why not. Regards Antoine. From martin at v.loewis.de Sun Mar 25 19:50:03 2012 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 25 Mar 2012 19:50:03 +0200 Subject: [Python-Dev] PEP 393 decode() oddity In-Reply-To: References: Message-ID: <4F6F5ACB.7090502@v.loewis.de> > How serious a problem this is for the Python 3.3 release? I could do the > optimization, if someone is not working on this already. I think the people who did the original implementation (Torsten, Victor, and myself) are done with optimizations. So: contributions are welcome. I'm not aware of any release-critical performance degradation (but I'd start with string formatting if I were you). From g.brandl at gmx.net Sun Mar 25 20:36:47 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 25 Mar 2012 20:36:47 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: On 25.03.2012 08:34, Georg Brandl wrote: > Here's another try, mainly with default browser font size, more contrast and > collapsible sidebar again: > > http://www.python.org/~gbrandl/build/html2/ > > I've also added a little questionable gimmick to the sidebar (when you collapse > it and expand it again, the content is shown at your current scroll location). Thanks everyone for the overwhelmingly positive feedback. I've committed the new design to 3.2 and 3.3 for now, and it will be live for the 3.3 docs momentarily (3.2 isn't rebuilt at the moment until 3.2.3 final goes out). I'll transplant to 2.7 too, probably after the final release of 2.7.3. Please make further suggestions (preferably with patches) through the bug tracker. cheers, Georg From storchaka at gmail.com Sun Mar 25 20:51:28 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Mar 2012 21:51:28 +0300 Subject: [Python-Dev] PEP 393 decode() oddity In-Reply-To: <20120325190137.7f0ffce0@pitrou.net> References: <20120325190137.7f0ffce0@pitrou.net> Message-ID: 25.03.12 20:01, Antoine Pitrou ???????(??): > The general problem with decoding is that you don't know up front what > width (1, 2 or 4 bytes) is required for the result. The solution is > either to compute the width in a first pass (and decode in a second > pass), or decode in a single pass and enlarge the result on the fly > when needed. Both incur a slowdown compared to a single-size > representation. We can significantly reduce the number of checks, using the same trick that is used for fast checking of surrogate characters. While all characters < U+0100, we know that the result is a 1-byte string (ascii while all characters < U+0080). When meet a character >= U+0100, while all characters < U+10000, we know that the result is the 2-byte string. As soon as we met first character >= U+10000, we work with 4-bytes string. There will be several fast loops, the transition to the next loop will occur after the failure in the previous one. > It's probably a measurement error on your part. Anyone can test. $ ./python -m timeit -s 'enc = "latin1"; import codecs; d = codecs.getdecoder(enc); x = ("\u0020" * 100000).encode(enc)' 'd(x)' 10000 loops, best of 3: 59.4 usec per loop $ ./python -m timeit -s 'enc = "latin1"; import codecs; d = codecs.getdecoder(enc); x = ("\u0080" * 100000).encode(enc)' 'd(x)' 10000 loops, best of 3: 28.4 usec per loop The results are fairly stable (?0.1 ?sec) from run to run. It looks funny thing. From anacrolix at gmail.com Sun Mar 25 21:09:09 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Mon, 26 Mar 2012 03:09:09 +0800 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <20120325113657.GA13240@iskra.aviel.ru> Message-ID: Not sure if you addressed this in your answers to other comments... Scroll down the page. Minimize the nav bar on the left. Bring it back out again. Now the text in the nav bar permanently starts at an offset from the top of the page. On Sun, Mar 25, 2012 at 7:44 PM, Matt Joiner wrote: > Is nice yes?! When I small the nav bar, then embiggen it again, the text > centers vertically. It's in the wrong place. The new theme is very minimal, > perhaps a new color should be chosen. We've done green, what about orange, > brown or blue? From steve at pearwood.info Sun Mar 25 21:11:20 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2012 06:11:20 +1100 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: <4F6F6DD8.9050901@pearwood.info> Georg Brandl wrote: > Thanks everyone for the overwhelmingly positive feedback. I've committed the > new design to 3.2 and 3.3 for now, and it will be live for the 3.3 docs > momentarily (3.2 isn't rebuilt at the moment until 3.2.3 final goes out). > I'll transplant to 2.7 too, probably after the final release of 2.7.3. I think it would be better to leave 2.7 with the old theme, to keep it visually distinct from the nifty new theme used with the nifty new 3.2 and 3.3 versions. -- Steven From p.f.moore at gmail.com Sun Mar 25 21:12:11 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 25 Mar 2012 20:12:11 +0100 Subject: [Python-Dev] PEP 393 decode() oddity In-Reply-To: References: <20120325190137.7f0ffce0@pitrou.net> Message-ID: On 25 March 2012 19:51, Serhiy Storchaka wrote: > Anyone can test. > > $ ./python -m timeit -s 'enc = "latin1"; import codecs; d = > codecs.getdecoder(enc); x = ("\u0020" * 100000).encode(enc)' 'd(x)' > 10000 loops, best of 3: 59.4 usec per loop > $ ./python -m timeit -s 'enc = "latin1"; import codecs; d = > codecs.getdecoder(enc); x = ("\u0080" * 100000).encode(enc)' 'd(x)' > 10000 loops, best of 3: 28.4 usec per loop > > The results are fairly stable (?0.1 ?sec) from run to run. It looks funny > thing. Hmm, yes. I see the same results. Odd. PS D:\Data> py -3.3 -m timeit -s "enc = 'latin1'; import codecs; d = codecs.getdecoder(enc); x = ('\u0020' * 100000).encode(enc)" "d(x)" 10000 loops, best of 3: 37.3 usec per loop PS D:\Data> py -3.3 -m timeit -s "enc = 'latin1'; import codecs; d = codecs.getdecoder(enc); x = ('\u0080' * 100000).encode(enc)" "d(x)" 100000 loops, best of 3: 18 usec per loop PS D:\Data> py -3.3 -m timeit -s "enc = 'latin1'; import codecs; d = codecs.getdecoder(enc); x = ('\u0020' * 100000).encode(enc)" "d(x)" 10000 loops, best of 3: 37.6 usec per loop PS D:\Data> py -3.3 -m timeit -s "enc = 'latin1'; import codecs; d = codecs.getdecoder(enc); x = ('\u0080' * 100000).encode(enc)" "d(x)" 100000 loops, best of 3: 18.3 usec per loop PS D:\Data> py -3.3 -m timeit -s "enc = 'latin1'; import codecs; d = codecs.getdecoder(enc); x = ('\u0020' * 100000).encode(enc)" "d(x)" 10000 loops, best of 3: 37.8 usec per loop PS D:\Data> py -3.3 -m timeit -s "enc = 'latin1'; import codecs; d = codecs.getdecoder(enc); x = ('\u0080' * 100000).encode(enc)" "d(x)" 100000 loops, best of 3: 18.3 usec per loop Paul. From g.brandl at gmx.net Sun Mar 25 21:24:40 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 25 Mar 2012 21:24:40 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <20120325113657.GA13240@iskra.aviel.ru> Message-ID: On 25.03.2012 21:09, Matt Joiner wrote: > Not sure if you addressed this in your answers to other comments... > > Scroll down the page. Minimize the nav bar on the left. Bring it back > out again. Now the text in the nav bar permanently starts at an offset > from the top of the page. Yes, that was the intention I mentioned in my post. That has been removed though from the final version I checked in. There are certainly much better solutions. Georg From g.brandl at gmx.net Sun Mar 25 21:25:42 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 25 Mar 2012 21:25:42 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F6F6DD8.9050901@pearwood.info> References: <4F6F6DD8.9050901@pearwood.info> Message-ID: On 25.03.2012 21:11, Steven D'Aprano wrote: > Georg Brandl wrote: > >> Thanks everyone for the overwhelmingly positive feedback. I've committed the >> new design to 3.2 and 3.3 for now, and it will be live for the 3.3 docs >> momentarily (3.2 isn't rebuilt at the moment until 3.2.3 final goes out). >> I'll transplant to 2.7 too, probably after the final release of 2.7.3. > > I think it would be better to leave 2.7 with the old theme, to keep it > visually distinct from the nifty new theme used with the nifty new 3.2 and 3.3 > versions. Hmm, -0 here. I'd like more opinions on this from other devs. Georg From janzert at janzert.com Sun Mar 25 21:48:34 2012 From: janzert at janzert.com (Janzert) Date: Sun, 25 Mar 2012 15:48:34 -0400 Subject: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()? In-Reply-To: References: Message-ID: On 3/24/2012 6:37 AM, Victor Stinner wrote: >>> - time.monotonic(): monotonic clock, its speed may or may not be >>> adjusted by NTP but it only goes forward, may raise an OSError >>> - time.steady(): monotonic clock or the realtime clock, depending on >>> what is available on the platform (use monotonic in priority). may be >>> adjusted by NTP or the system administrator, may go backward. >>> >> >> I am surprised that a clock with the name time.steady() has a looser >> definition than one called time.monotonic(). To my mind a steady clock is by >> definition monotonic but a monotonic one may or may not be steady. > > Do you suggest another name? > > Victor I can't think of a word or short phrase that adequately describes that behavior, no. But that may just be because I also don't see any use case for it either. To me the more useful function would be one that used the OS monotonic clock when available and failing that used the realtime clock but cached the previous returned value and ensured that all values returned obeyed the monotonic property still. But I don't see why that function shouldn't just be time.monotonic(). Janzert From andrew.svetlov at gmail.com Sun Mar 25 21:50:59 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 25 Mar 2012 22:50:59 +0300 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <4F6F6DD8.9050901@pearwood.info> Message-ID: I like to see new schema only for 3.3 as sign of shiny new release. On Sun, Mar 25, 2012 at 10:25 PM, Georg Brandl wrote: > On 25.03.2012 21:11, Steven D'Aprano wrote: >> Georg Brandl wrote: >> >>> Thanks everyone for the overwhelmingly positive feedback. ?I've committed the >>> new design to 3.2 and 3.3 for now, and it will be live for the 3.3 docs >>> momentarily (3.2 isn't rebuilt at the moment until 3.2.3 final goes out). >>> I'll transplant to 2.7 too, probably after the final release of 2.7.3. >> >> I think it would be better to leave 2.7 with the old theme, to keep it >> visually distinct from the nifty new theme used with the nifty new 3.2 and 3.3 >> versions. > > Hmm, -0 here. ?I'd like more opinions on this from other devs. > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov From pmoody at google.com Sun Mar 25 21:58:38 2012 From: pmoody at google.com (Peter Moody) Date: Sun, 25 Mar 2012 12:58:38 -0700 Subject: [Python-Dev] PEP czar for PEP 3144? In-Reply-To: <4F67AA1E.6070907@stoneleaf.us> References: <4F678AEF.4080002@stoneleaf.us> <4F679384.4040602@stoneleaf.us> <4F67AA1E.6070907@stoneleaf.us> Message-ID: On Mon, Mar 19, 2012 at 2:50 PM, Ethan Furman wrote: > [1] I'm assuming that 'iter(some_list)' is a quick operation. This seems to be the case so I've just gone ahead and renamed collapse_address_list to collapse_addresses and added 'return iter(...)' to the end. The rest of the list-returning methods all return iterators now too. There should only be a few minor outstanding issues to to work out. Cheers, peter -- Peter Moody? ? ? Google? ? 1.650.253.7306 Security Engineer? pgp:0xC3410038 From martin at v.loewis.de Sun Mar 25 22:55:13 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sun, 25 Mar 2012 22:55:13 +0200 Subject: [Python-Dev] PEP 393 decode() oddity In-Reply-To: References: <20120325190137.7f0ffce0@pitrou.net> Message-ID: <20120325225513.Horde.v_hfaKGZi1VPb4YxHpTWb7A@webmail.df.eu> > Anyone can test. > > $ ./python -m timeit -s 'enc = "latin1"; import codecs; d = > codecs.getdecoder(enc); x = ("\u0020" * 100000).encode(enc)' 'd(x)' > 10000 loops, best of 3: 59.4 usec per loop > $ ./python -m timeit -s 'enc = "latin1"; import codecs; d = > codecs.getdecoder(enc); x = ("\u0080" * 100000).encode(enc)' 'd(x)' > 10000 loops, best of 3: 28.4 usec per loop > > The results are fairly stable (?0.1 ?sec) from run to run. It looks > funny thing. This is not surprising. When decoding Latin-1, it needs to determine whether the string is pure ASCII or not. If it is not, it must be all Latin-1 (it can't be non-Latin-1). For a pure ASCII string, it needs to scan over the entire string, trying to find a non-ASCII character. If there is none, it has to inspect the entire string. In your example, as the first character is is already above 127, search for the maximum character can stop, so it needs to scan the string only once. Try '\u0020' * 999999+'\u0080', which is a non-ASCII string but still takes the same time as the pure ASCII string. Regards, Martin From fijall at gmail.com Sun Mar 25 23:08:49 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 25 Mar 2012 23:08:49 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <4F6F6DD8.9050901@pearwood.info> Message-ID: On Sun, Mar 25, 2012 at 9:25 PM, Georg Brandl wrote: > On 25.03.2012 21:11, Steven D'Aprano wrote: >> Georg Brandl wrote: >> >>> Thanks everyone for the overwhelmingly positive feedback. ?I've committed the >>> new design to 3.2 and 3.3 for now, and it will be live for the 3.3 docs >>> momentarily (3.2 isn't rebuilt at the moment until 3.2.3 final goes out). >>> I'll transplant to 2.7 too, probably after the final release of 2.7.3. >> >> I think it would be better to leave 2.7 with the old theme, to keep it >> visually distinct from the nifty new theme used with the nifty new 3.2 and 3.3 >> versions. > > Hmm, -0 here. ?I'd like more opinions on this from other devs. > > Georg I would definitely like the new theme on 2.7 docs as well, since 2.7 is still supported. Cheers, fijal From brian at python.org Sun Mar 25 23:17:20 2012 From: brian at python.org (Brian Curtin) Date: Sun, 25 Mar 2012 16:17:20 -0500 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <4F6F6DD8.9050901@pearwood.info> Message-ID: On Sun, Mar 25, 2012 at 14:50, Andrew Svetlov wrote: > I like to see new schema only for 3.3 as sign of shiny new release. Please don't do this. It will result in endless complaints. From ben+python at benfinney.id.au Sun Mar 25 23:25:28 2012 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 26 Mar 2012 08:25:28 +1100 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 References: <4F6F6DD8.9050901@pearwood.info> Message-ID: <877gy8id9z.fsf@benfinney.id.au> Brian Curtin writes: > On Sun, Mar 25, 2012 at 14:50, Andrew Svetlov wrote: > > I like to see new schema only for 3.3 as sign of shiny new release. > > Please don't do this. It will result in endless complaints. Complaints of what nature? Do you think those complaints are justified? -- \ ?? Nature ? is seen to do all things herself and through | `\ herself of own accord, rid of all gods.? ?Titus Lucretius | _o__) Carus, c. 40 BCE | Ben Finney From tjreedy at udel.edu Sun Mar 25 23:42:33 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Mar 2012 17:42:33 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: On 3/25/2012 12:32 PM, Georg Brandl wrote: > On 25.03.2012 17:54, Terry Reedy wrote: >> On 3/25/2012 2:34 AM, Georg Brandl wrote: >>> Here's another try, mainly with default browser font size, more contrast >> >> Untrue. You still changed the high contrast dark blue to the same low >> contrast light blue for builtin names, etc. What problem do you think >> you are trying to solve by making the doc difficult and even PAINFUL for >> me to read? >> >> - a lot more than 1 > > "More contrast" was meant in comparison to iteration #1. It is still subjectively dim enough to me that I could not tell from memory. I ran the following experiment: I put old and new versions of the buitin functions page side-by-side in separate browser windows. I asked my teenage daughter to come into the room, approach slowly, and say when she could read one or both windows. At about 5 feet, she could (just) read the old but not the new. If other people repeat the experiment and get the same result, it would then be fair to say that the new style is objectively less readable in regard to this one aspect. > Hmm, don't you think you'll get used to the new style in a while? This is a bit like asking a wheelchair user if he would get used to having a ramp ground down to add little one-inch steps every two feet, because leg-abled people found that somehow more aesthetic. Answer: somewhat. Wired magazine has used a similar thin blue font. I got used to that by ignoring any text written with it. > The link color is not actually that light in comparison. Using a magnifying glass, the difference seems to be more one of thickness -- 2 pixel lines versus 1-1.5 pixel lines. I have astigmatism that is only partly correctable and the residual blurring of single-pixel lines tends to somewhat mix text color with the background color. > Of course you can always use a user stylesheet to override our choices. Can anyone tell me the best way to do that with FireFox? Is it even possible with the Windows help version, which is what I usually use? -- Terry Jan Reedy From victor.stinner at gmail.com Mon Mar 26 00:28:33 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 26 Mar 2012 00:28:33 +0200 Subject: [Python-Dev] PEP 393 decode() oddity In-Reply-To: References: Message-ID: Cool, Python 3.3 is *much* faster to decode pure ASCII :-) > encoding ?string ? ? ? ? ? ? ? ? 2.7 ? 3.2 ? 3.3 > > ascii ? ? " " * 1000 ? ? ? ? ? ? 5.4 ? 5.3 ? 1.2 4.5 faster than Python 2 here. > utf-8 ? ? " " * 1000 ? ? ? ? ? ? 6.7 ? 2.4 ? 2.1 3.2x faster It's cool because in practice, a lot of strings are pure ASCII (as Martin showed in its Django benchmark). > latin1 " " * 1000 1.8 1.7 1.3 > latin1 "\u0080" * 1000 1.7 1.6 1.0 > ... > The first oddity in that the characters from the second half of the Latin1 > table decoded faster than the characters from the first half. The Latin1 decoder of Python 3.3 is *faster* than the decoder of Python 2.7 and 3.2 according to your bench. So I don't see any issue here :-) Martin explained why it is slower for pure ASCII. > I think that the characters from the first half of the table > must be decoded as quickly. The Latin1 decoder is already heavily optimized, I don't see how to make it faster. > The second sad oddity in that UTF-16 decoding in 3.3 is much slower than > even in 2.7. Compared with 3.2 decoding is slower in 2-3 times. This is a > considerable regress. UTF-32 decoding is also slowed down by 1.5-2 times. Only ASCII, latin1 and UTF-8 decoder are heavily optimized. We can do better for UTF-16 and UTF-32. I'm just less motivated because UTF-16/32 are less common than ASCII/latin1/UTF-8. > How serious a problem this is for the Python 3.3 release? I could do the > optimization, if someone is not working on this already. I'm interested by any patch optimizing any Python codecs. I'm not working on optimizing Python Unicode anymore, various benchmarks showed me that Python 3.3 is as good or faster than Python 3.2. That's enough for me. When Python 3.3 is slower than Python 3.2, it's because Python 3.3 must compute the maximum character of the result, and I fail to see how to optimize this requirement. I already introduced many fast-path where it was possible, like creating a substring of an ASCII string (the result is ASCII, no need to scan the substring). It doesn't mean that it is no more possible to optimize Python Unicode ;-) Victor From victor.stinner at gmail.com Mon Mar 26 01:03:46 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 26 Mar 2012 01:03:46 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: <20120323104005.GA17581@sleipnir.bytereef.org> References: <20120323092255.GA17205@sleipnir.bytereef.org> <20120323104005.GA17581@sleipnir.bytereef.org> Message-ID: > The 80x is a ballpark figure for the maximum expected speedup for > standard numerical floating point applications. Ok, but it's just surprising when you read the What's New document. 72x and 80x look to be inconsistent. > For huge numbers _decimal is also faster than int: > > factorial(1000000): > > _decimal, calculation time: 6.844487905502319 > _decimal, tostr(): ? ? ? ? ?0.033592939376831055 > > int, calculation time: 17.96010398864746 > int, tostr(): ... still running ... Hum, with a resolution able to store the result with all digits? If yes, would it be possible to reuse the multiply algorithm of _decimal (and maybe of other functions) for int? Or does it depend heavily on _decimal internal structures? Victor From steve at pearwood.info Mon Mar 26 02:37:56 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2012 11:37:56 +1100 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: <4F6FBA64.9080101@pearwood.info> Terry Reedy wrote: > On 3/25/2012 12:32 PM, Georg Brandl wrote: >> On 25.03.2012 17:54, Terry Reedy wrote: >>> On 3/25/2012 2:34 AM, Georg Brandl wrote: >>>> Here's another try, mainly with default browser font size, more >>>> contrast >>> >>> Untrue. You still changed the high contrast dark blue to the same low >>> contrast light blue for builtin names, etc. What problem do you think >>> you are trying to solve by making the doc difficult and even PAINFUL for >>> me to read? >>> >>> - a lot more than 1 >> >> "More contrast" was meant in comparison to iteration #1. > > It is still subjectively dim enough to me that I could not tell from > memory. > > I ran the following experiment: I put old and new versions of the buitin > functions page side-by-side in separate browser windows. I asked my > teenage daughter to come into the room, approach slowly, and say when > she could read one or both windows. At about 5 feet, she could (just) > read the old but not the new. Do you often read things on your computer monitor from 5ft away? While I sympathize with the ideal of making the docs readable, particular for those of us who don't have 20-20 vision, "must be readable from halfway across the room" is setting the bar too high. What is important is not *absolute* readability, but readability relative to the normal use-case of sitting at a computer under typical reading conditions. To be honest here, I don't even know which elements you are having trouble with. I don't see any elements with such low contrast to cause problems at least not for me. Even with my glasses off, I find the built-in names to be no less readable as the vanilla text around it. E.g. on this page: http://www.python.org/~gbrandl/build/html2/library/stdtypes.html I see built-in names such as `int` and `str` are written as hyperlinks in medium blue on a white background. When I hover over the link, it becomes a touch lighter blue, but not enough to appreciably hurt contrast and readability. I see literals such as `{}` in black on a pale blue-grey background. The background is faint enough that it is hardly noticeable, not enough to hurt contrast. So I don't know what you are speaking off when you say "the same low contrast light blue for builtin names, etc." -- can you give an example? [...] > Using a magnifying glass, the difference seems to be more one of > thickness -- 2 pixel lines versus 1-1.5 pixel lines. I have astigmatism > that is only partly correctable and the residual blurring of > single-pixel lines tends to somewhat mix text color with the background > color. For what it's worth, it wouldn't surprise me if the problem is the fallback font. If I'm reading the CSS correctly, the standard font used in the new docs is Lucinda Grande, with a fallback of Arial. Unfortunately, Lucinda Grande is normally only available on the Apple Mac, and Arial is a notoriously poor choice for on-screen text (particularly in smaller text sizes). http://en.wikipedia.org/wiki/Lucida_Grande suggests fallbacks of Lucida Sans Unicode, Tahoma, and Verdana. Could they please be tried before Arial? E.g. change the font-family from font-family: 'Lucida Grande',Arial,sans-serif; to font-family: 'Lucida Grande','Lucida Sans Unicode','Lucida Sans',Tahoma,Verdana,Arial,sans-serif; or similar. -- Steven From ncoghlan at gmail.com Mon Mar 26 03:02:56 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Mar 2012 11:02:56 +1000 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: On Mon, Mar 26, 2012 at 7:42 AM, Terry Reedy wrote: > Can anyone tell me the best way to do that with FireFox? For general webbrowsing, I'm reasonably impressed by the effectiveness of www.readability.com. It's a sign-up service however, and I've never tried it on technical material like the Python docs. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From brian at python.org Mon Mar 26 03:27:16 2012 From: brian at python.org (Brian Curtin) Date: Sun, 25 Mar 2012 20:27:16 -0500 Subject: [Python-Dev] AUTO: Jon K Peck is out of the office (returning 03/30/2012) In-Reply-To: References: Message-ID: On Sun, Mar 25, 2012 at 05:00, Jon K Peck wrote: > > I am out of the office until 03/30/2012. > > I will be out of the office through Friday, March 30. ?I expect to have > some email access but may be delayed in responding. > > > Note: This is an automated response to your message ?"Python-Dev Digest, > Vol 104, Issue 91" sent on 03/25/2012 1:19:50. > > This is the only notification you will receive while this person is away. Enjoy your vacation. From scott+python-dev at scottdial.com Mon Mar 26 04:23:09 2012 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sun, 25 Mar 2012 22:23:09 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F6FBA64.9080101@pearwood.info> References: <4F6FBA64.9080101@pearwood.info> Message-ID: <4F6FD30D.3030508@scottdial.com> On 3/25/2012 8:37 PM, Steven D'Aprano wrote: > E.g. change the font-family from > > font-family: 'Lucida Grande',Arial,sans-serif; > > to > > font-family: 'Lucida Grande','Lucida Sans Unicode','Lucida > Sans',Tahoma,Verdana,Arial,sans-serif; > > or similar. > +1 To providing other fallbacks. As Steven says, on my Win7 machine, I do not have 'Lucida Grande' and it wasn't until he mentioned this that I compared the experience of the site with my MacBook (which looks much better!). This machine has both 'Lucida Sans Unicode' and 'Lucida Sans' and it's a toss-up to me which is better -- one is better than the other in certain contexts. Presumably, the character coverage of the Unicode font makes it the superior choice. Personally, I would leave Tahoma out of the list -- the kerning of the font is really aggressive and I find it much harder to read than Verdana. -- Scott Dial scott at scottdial.com From tjreedy at udel.edu Mon Mar 26 05:23:22 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Mar 2012 23:23:22 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F6FBA64.9080101@pearwood.info> References: <4F6FBA64.9080101@pearwood.info> Message-ID: On 3/25/2012 8:37 PM, Steven D'Aprano wrote: > Terry Reedy wrote: >> I ran the following experiment: I put old and new versions of the >> buitin functions page side-by-side in separate browser windows. I >> asked my teenage daughter to come into the room, approach slowly, and >> say when she could read one or both windows. At about 5 feet, she >> could (just) read the old but not the new. The test page I used was the builtin function page in the Library Reference. > Do you often read things on your computer monitor from 5ft away? No, I cannot possibly do that. The point is that there *is* a distance for her as well as me at which the old style is clearly more readable than the new. For my daughter, is is 4-5 feet. For me, it is about 2 feet -- with my prescription computer reading glasses. For you, try it and find out. Obviously, most of the people here are more like my daughter than me. I am visually handicapped, and that particular new style is worse for me. And to what purpose? -- Terry Jan Reedy From eliben at gmail.com Mon Mar 26 05:40:41 2012 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 26 Mar 2012 05:40:41 +0200 Subject: [Python-Dev] PEP 411 - request for pronouncement In-Reply-To: References: Message-ID: On Sat, Mar 24, 2012 at 13:53, Lennart Regebro wrote: > On Fri, Mar 23, 2012 at 10:51, Eli Bendersky wrote: >> The PEP received mostly positive feedback. The only undecided point is >> where to specify that the package is provisional. Currently the PEP >> mandates to specify it in the documentation and in the docstring. >> Other suggestions were to put it in the code, either as a >> __provisional__ attribute on the module, or collect all such modules >> in a single sys.provisional list. > > I'm not sure what the usecase is for checking in code if a module is > provisional or not. It doesn't seem useful, and risks being > unmaintained, especially when the flag is on the module itself. > Some usecases were given by Jim J. Jewett here: http://mail.python.org/pipermail/python-dev/2012-February/116400.html (and +1-ed by a couple of others.) Eli From eliben at gmail.com Mon Mar 26 06:11:42 2012 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 26 Mar 2012 06:11:42 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <4F6F6DD8.9050901@pearwood.info> Message-ID: On Sun, Mar 25, 2012 at 21:25, Georg Brandl wrote: > On 25.03.2012 21:11, Steven D'Aprano wrote: >> Georg Brandl wrote: >> >>> Thanks everyone for the overwhelmingly positive feedback. ?I've committed the >>> new design to 3.2 and 3.3 for now, and it will be live for the 3.3 docs >>> momentarily (3.2 isn't rebuilt at the moment until 3.2.3 final goes out). >>> I'll transplant to 2.7 too, probably after the final release of 2.7.3. > Georg, Is there a tracker issue to collect reports about misbehavior of the new theme? Specifically, the sidebar behaves very strangely in the search page. I think it should not be there at all. Eli From tjreedy at udel.edu Mon Mar 26 06:22:30 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Mar 2012 00:22:30 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F6FBA64.9080101@pearwood.info> References: <4F6FBA64.9080101@pearwood.info> Message-ID: On 3/25/2012 8:37 PM, Steven D'Aprano wrote: > For what it's worth, it wouldn't surprise me if the problem is the > fallback font. If I'm reading the CSS correctly, the standard font used > in the new docs is Lucinda Grande, with a fallback of Arial. > Unfortunately, Lucinda Grande is normally only available on the Apple > Mac, and Arial is a notoriously poor choice for on-screen text > (particularly in smaller text sizes). Testing in LibreOffice, I think Ariel may be easier as it has a consistent stroke width, whereas Lucida has thin horizontals. It does look a bit more elegant, though. In any case, Ariel seems to be the basic text font I see in Firefox and Windows help and I have no problem with it. The particular entries I have discussed are class="reference internal" There have light serifs. Testing in LibreOffice, it seems to be Courier New. It was previously Courier New 'bold'. I put that in quotes because Courier New is a 'light' font, so that the 'bold' is normal relative to Ariel. In other words, Courier bold matches normal Ariel in stroke weight, so that is looks right mixed in with Ariel, whereas the Courier light is jarring. In a sentence like "this returns False; otherwise it returns True. bool is also a class, which is a subclass of int" False, True (which use light-weight black rather than light-weight blue Courier), bool, and int all stand out (or stand in) because they have a lighter stroke weight as well as a different (serif versus non-serif) font. These are important words and should not be made to recede into the background as if they were unimportant and optionally skipped. To me, this is backwards and poor design. False,false are marked up like this: False bool,in are like so: Does the css specify Courier New or is this an unfortunate fallback that might be improved? Perhaps things look better on max/*nix? -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Mon Mar 26 07:00:34 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 26 Mar 2012 18:00:34 +1300 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F6FBA64.9080101@pearwood.info> References: <4F6FBA64.9080101@pearwood.info> Message-ID: <4F6FF7F2.3000806@canterbury.ac.nz> Steven D'Aprano wrote: > While I sympathize with the ideal of making the docs readable, > particular for those of us who don't have 20-20 vision, "must be > readable from halfway across the room" is setting the bar too high. The point is that reducing contrast never makes anything more readable, and under some conditions it makes things less readable. So the only reason for using less than the maximum available contrast is aesthetics, and whether grey-on-white looks any nicer than black-on-white is very much a matter of opinion. In any case, the aesthetic difference is a very minor one, and you have to ask whether it's really worth compromising on contrast. > If I'm reading the CSS correctly, the standard font used > in the new docs is Lucinda Grande, with a fallback of Arial. > Unfortunately, Lucinda Grande is normally only available on the Apple > Mac, and Arial is a notoriously poor choice for on-screen text This seems to be another case of the designer over-specifying things. The page should just specify a sans-serif font and let the browser choose the best one available. Or not specify a font at all and leave it up to the user whether he wants serif or sans-serif for the body text -- some people have already said here that they prefer serif. -- Greg From regebro at gmail.com Mon Mar 26 08:48:33 2012 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 26 Mar 2012 08:48:33 +0200 Subject: [Python-Dev] PEP 411 - request for pronouncement In-Reply-To: References: Message-ID: On Mon, Mar 26, 2012 at 05:40, Eli Bendersky wrote: > On Sat, Mar 24, 2012 at 13:53, Lennart Regebro wrote: >> On Fri, Mar 23, 2012 at 10:51, Eli Bendersky wrote: >>> The PEP received mostly positive feedback. The only undecided point is >>> where to specify that the package is provisional. Currently the PEP >>> mandates to specify it in the documentation and in the docstring. >>> Other suggestions were to put it in the code, either as a >>> __provisional__ attribute on the module, or collect all such modules >>> in a single sys.provisional list. >> >> I'm not sure what the usecase is for checking in code if a module is >> provisional or not. It doesn't seem useful, and risks being >> unmaintained, especially when the flag is on the module itself. >> > > Some usecases were given by Jim J. Jewett here: > http://mail.python.org/pipermail/python-dev/2012-February/116400.html > (and +1-ed by a couple of others.) Having a list of provisional modules seems the most helpful there, in that case, and also noting that it is provisional in the module docs shown when you do help(). From anacrolix at gmail.com Mon Mar 26 08:50:48 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Mon, 26 Mar 2012 14:50:48 +0800 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F6FF7F2.3000806@canterbury.ac.nz> References: <4F6FBA64.9080101@pearwood.info> <4F6FF7F2.3000806@canterbury.ac.nz> Message-ID: FWIW it doesn't hurt to err on the side of what worked. i have generally have issues with low contrast, the current stable design is very good with this. i've just built the docs from tip, and the nav bar issue is fixed, nicely done i also don't see any reason to backport theme changes, +0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Mon Mar 26 09:11:50 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 26 Mar 2012 09:11:50 +0200 Subject: [Python-Dev] cpython: Adding unittest.mock documentation In-Reply-To: References: Message-ID: On 26.03.2012 00:13, michael.foord wrote: > http://hg.python.org/cpython/rev/adc1fc2dc872 > changeset: 75938:adc1fc2dc872 > user: Michael Foord > date: Sun Mar 25 23:12:55 2012 +0100 > summary: > Adding unittest.mock documentation > > files: > Doc/library/development.rst | 6 + > Doc/library/unittest.mock-examples.rst | 887 +++++++++ > Doc/library/unittest.mock-getting-started.rst | 419 ++++ > Doc/library/unittest.mock-helpers.rst | 537 +++++ > Doc/library/unittest.mock-magicmethods.rst | 226 ++ > Doc/library/unittest.mock-patch.rst | 538 +++++ > Doc/library/unittest.mock.rst | 900 ++++++++++ > Lib/unittest/mock.py | 8 +- > 8 files changed, 3516 insertions(+), 5 deletions(-) That seems a bit much splitting to me. (By the way, the ".. module::" directive should only be in *one* place.) I would organize the mock, mock-patch, mock-magicmethods and mock-helpers as one file in Doc/library, and put the other two in Doc/howto, just as for logging. In general, I wouldn't mind splitting off more of the exemplary material from the main library docs, putting it in the howto section. Georg From scott+python-dev at scottdial.com Mon Mar 26 14:44:40 2012 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 26 Mar 2012 08:44:40 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F6FF7F2.3000806@canterbury.ac.nz> References: <4F6FBA64.9080101@pearwood.info> <4F6FF7F2.3000806@canterbury.ac.nz> Message-ID: <4F7064B8.5050406@scottdial.com> On 3/26/2012 1:00 AM, Greg Ewing wrote: > This seems to be another case of the designer over-specifying > things. The page should just specify a sans-serif font and let > the browser choose the best one available. Or not specify > a font at all and leave it up to the user whether he wants > serif or sans-serif for the body text -- some people have > already said here that they prefer serif. Why even bother formatting the page? The authorship and editorship have authority to dictate the presentation of the content. A large part of the effectiveness of a document and it's ease of consumption is determined by how it appears in whatever medium it's delivered on. While this particular medium invites the readership to participate in design choices, fonts are not all created equal and practical matters (size, tracking, and kerning) will dictate that some fonts will present better than other fonts. Consistent presentation across different systems is also a virtue, since people develop familiarity with the presentation and find information more readily if the presentation is consistent. I have no problem having Georg dictating to me the best font with which to present the documentation. However, I'd appreciate fallback choices that are of a similar appearance along the way to the ultimate fallback of "sans-serif". Practically, the fonts available are unknown and unless we adopt the use of a liberally licensed OpenType font and use @font-face to embed a font, we need to provide fallbacks. -- Scott Dial scott at scottdial.com From zvezdan at computer.org Mon Mar 26 14:46:11 2012 From: zvezdan at computer.org (Zvezdan Petkovic) Date: Mon, 26 Mar 2012 08:46:11 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <4F6FBA64.9080101@pearwood.info> Message-ID: <3677ADF1-9CA6-49F7-8331-81441D7EEB58@computer.org> On Mar 26, 2012, at 12:22 AM, Terry Reedy wrote: > Does the css specify Courier New or is this an unfortunate fallback that might be improved? Perhaps things look better on max/*nix? I just checked pydoctheme.css and Courier New is not specified there. It only specifies monospace. That's a default monospace font set in your browser. I see the code rendered in the font I selected in my browser preferences as Fixed-width font: Menlo 14pt. It's not thin at all -- that's why I selected it. :-) It seems you may want to change that setting in your browser. Firefox uses Courier New as a default setting. Zvezdan From anacrolix at gmail.com Mon Mar 26 14:57:21 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Mon, 26 Mar 2012 20:57:21 +0800 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <3677ADF1-9CA6-49F7-8331-81441D7EEB58@computer.org> References: <4F6FBA64.9080101@pearwood.info> <3677ADF1-9CA6-49F7-8331-81441D7EEB58@computer.org> Message-ID: the text in the nav bar is too small, particularly in the search box. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Mar 26 15:19:27 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 26 Mar 2012 09:19:27 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <4F7064B8.5050406@scottdial.com> References: <4F6FBA64.9080101@pearwood.info> <4F6FF7F2.3000806@canterbury.ac.nz> <4F7064B8.5050406@scottdial.com> Message-ID: <20120326131928.6D0E32500E9@webabinitio.net> On Mon, 26 Mar 2012 08:44:40 -0400, Scott Dial wrote: > Why even bother formatting the page? The web started out as *content markup*. Functional declarations, not style declarations. I wish it had stayed that way, but it was inevitable that it would not. > The authorship and editorship have authority to dictate the presentation > of the content. A large part of the effectiveness of a document and it's > ease of consumption is determined by how it appears in whatever medium > it's delivered on. While this particular medium invites the readership This argument can get as bad as editor religious wars. I like sites that do more content markup and less styling. You like the reverse. There is a good reason to separate css (style) from markup. I just wish browsers gave easier control over the css, and that more designers would stay concious of how a page *reads* (ie: access to the content) without the css (and javascript). I think Georg's design does pretty well in that last regard. (Except for those darn paragraph characters after the headers, but that flaw was in the old design too. Oh, and it would be even better, IMO, if the top navigation block wasn't there when there's no CSS, but that's more of an issue for easier access from screen readers, since the block is reasonably short). --David From van.lindberg at gmail.com Mon Mar 26 16:06:32 2012 From: van.lindberg at gmail.com (VanL) Date: Mon, 26 Mar 2012 09:06:32 -0500 Subject: [Python-Dev] Python install layout and the PATH on win32 In-Reply-To: References: Message-ID: <4F7077E8.2070005@gmail.com> I heard back from Enthought on this part of the proposal. They could accommodate this change. 1) The layout for the python root directory for all platforms should be as follows: stdlib = {base/userbase}/lib/ platstdlib = {base/userbase}/lib/ purelib = {base/userbase}/lib/site-packages platlib = {base/userbase}/lib/site-packages include = {base/userbase}/include/python{py_version_short} scripts = {base/userbase}/bin data = {base/userbase} From tjreedy at udel.edu Mon Mar 26 17:21:46 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Mar 2012 11:21:46 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <3677ADF1-9CA6-49F7-8331-81441D7EEB58@computer.org> References: <4F6FBA64.9080101@pearwood.info> <3677ADF1-9CA6-49F7-8331-81441D7EEB58@computer.org> Message-ID: On 3/26/2012 8:46 AM, Zvezdan Petkovic wrote: > > On Mar 26, 2012, at 12:22 AM, Terry Reedy wrote: > >> Does the css specify Courier New or is this an unfortunate fallback >> that might be improved? Perhaps things look better on max/*nix? > > I just checked pydoctheme.css and Courier New is not specified > there. It only specifies monospace. > > That's a default monospace font set in your browser. I see the code > rendered in the font I selected in my browser preferences as > Fixed-width font: Menlo 14pt. It's not thin at all -- that's why I > selected it. :-) > > It seems you may want to change that setting in your browser. Firefox > uses Courier New as a default setting. I found the FireFox monospace setting under Tools / Options / Content / Default font: / Advanced and switched to Deja Vu mono, that being the first obviously monospace font I saw. (Lucida Console is similar.) It has the same 2-pixel lines as Ariel, and the page now looks okay, although when black (as for False, True),the lack of serifs reduces the contrast with Arial. I am guessing that the page now looks somewhat more like it did for Georg when he worked on it. Windows Help uses Internet Explorer settings. Options / Internet Options / General / Appearance / Fonts However, this only allows choice of base font for pages without a specified text font, so I do know know what will happen when new format is applied to the .chm files. -- Terry Jan Reedy From pje at telecommunity.com Mon Mar 26 18:35:18 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 26 Mar 2012 12:35:18 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> Message-ID: On Sun, Mar 25, 2012 at 2:56 AM, Stephen J. Turnbull wrote: > But since he's arguing the > other end in the directory layout thread (where he says there are many > special ways to invoke Python so that having different layouts on > different platforms is easy to work around), I can't give much weight > to his preference here. > You're misconstruing my argument there: I said, rather, that the One Obvious Way to deploy a Python application is to dump everything in one directory, as that is the one way that Python has supported for at least 15 years now. Calling this a "special" way of invoking Python is disingenuous at best: it's the documented *default* way of deploying and invoking a Python script with accompanying libraries. In contrast, the directory layout thread is about supporting virtualenvs, which aren't even *in* Python yet -- if anything is to be considered a special case, that would be it. The comparison to CSS is also lost on me here; creating user-specific CSS is more aptly comparable telling people to write their own virtualenv implementations from scratch, and resizing the browser window is more akin to telling people to create a virtualenv every time they *run* the application, rather than just once when installing it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea.crotti.0 at gmail.com Mon Mar 26 18:54:42 2012 From: andrea.crotti.0 at gmail.com (Andrea Crotti) Date: Mon, 26 Mar 2012 17:54:42 +0100 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> Message-ID: <4F709F52.7010803@gmail.com> On 03/24/2012 03:30 AM, PJ Eby wrote: > > > Weird - I have the exact *opposite* problem, where I have to resize my > window because somebody *didn't* set their text max-width sanely (to a > reasonable value based on ems instead of pixels), and I have nearly > 1920 pixels of raw text spanning my screen. Bloody impossible to read > that way. > > But I guess this is going to turn into one of those vi vs. emacs holy > war things... > > (Personally, I prefer jEdit, or nano if absolutely forced to edit in a > terminal. Heretical, I know. To the comfy chair with me!) > > Suppose the author set the size to 1000 pixels, you would end up with 920 white pixels on the side, does it make sense? Using a tiling window manager (for example awesome or xmonad) would solve your problem in a more definitive way imho than hoping in the web designer choices.. From pje at telecommunity.com Mon Mar 26 18:55:42 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 26 Mar 2012 12:55:42 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <87y5qpikah.fsf@benfinney.id.au> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> Message-ID: On Sat, Mar 24, 2012 at 8:41 PM, Ben Finney wrote: > PJ Eby writes: > > > On Sat, Mar 24, 2012 at 1:32 AM, Greg Ewing >wrote: > > > > > If you don't want 1920-pixel-wide text, why make your browser window > > > that large? > > > > Not every tab in my browser is text for reading; some are apps that > > need the extra horizontal space. > > So, again, why make your browser window *for reading text* that large? > Because I have one browser window, and it's maximized. And I can do this, because most websites are designed in such a way that they have usable margins for text flows. Even PEPs and Python mailing list archives, for example, have sane text margins -- shall we go back and make *those* dependent on window width instead? Also, looking at the email I got from you, it has sane text margins in it. If you don't believe in text margins, why are you using a client that wraps lines and thereby prevents me from viewing your email with full-screen-width text? ;-) (In fairness, I am using a client that *doesn't* wrap the lines, AFAICT. But if Gmail had such an option I would probably use it if I knew where it was in the vast assortment of settings. Which ties in nicely with my next point, below...) Everyone has different needs for how large the text should be and how > much of it should go across the window. Every one of us is in a minority > when it comes to those needs; that's exactly what a configuration > setting is good for. > Designers' rules of thumb for text width are based on empirical observations of focal length, saccades, etc. If you have special needs visually, you're more likely to require the text read to you, than to have narrower text, and I at least am unable to conceive of a visual disability that would be helped by *increasing* the text width. In other words, there is a well-established *majority* need for how many characters should appear in an unwrapped line of text, based on majority physiology. Designers who limit it based on pixel size are Doing It Wrong; the max width should be based on em's rather than pixels. (Font sizes are a separate issue.) Done correctly (as visible, say, on any plaintext PEP), you may resize the window and change the font size to your heart's content without affecting the text width in characters. (Also, as a side note: adding lots of configuration options to an interface design is what adding lots of code is to a software design: a smell that the designer isn't *designing* enough.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Mon Mar 26 18:58:10 2012 From: carl at oddbird.net (Carl Meyer) Date: Mon, 26 Mar 2012 10:58:10 -0600 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: Message-ID: <4F70A022.6060507@oddbird.net> On 03/23/2012 09:22 PM, PJ Eby wrote: > On Mar 23, 2012 3:53 PM, "Carl Meyer" > On 03/23/2012 12:35 PM, PJ Eby wrote: >> > AFAICT, virtualenvs are overkill for most development anyway. If you're >> > not using distutils except to install dependencies, then configure >> > distutils to install scripts and libraries to the same directory, and >> > then do all your development in that directory. Presto! You now have a >> > cross-platform "virtualenv". Want the scripts on your path? Add that >> > directory to your path... or if on Windows, don't bother, since the >> > current directory is usually on the path. (In fact, if you're only >> > using easy_install to install your dependencies, you don't even need to >> > edit the distutils configuration, just use "-md targetdir".) >> >> Creating and using a virtualenv is, in practice, _easier_ than any of >> those alternatives, > > Really? As I said, I've never seen the need to try, since just > installing stuff to a directory on PYTHONPATH seems quite easy enough > for me. > >> that the "isolation from system site-packages" feature is quite popular >> (the outpouring of gratitude when virtualenv went isolated-by-default a >> few months ago was astonishing), and AFAIK none of your alternative >> proposals support that at all. > > What is this isolation for, exactly? If you don't want site-packages on > your path, why not use python -S? > > (Sure, nobody knows about these things, but surely that's a > documentation problem, not a tooling problem.) > > Don't get me wrong, I don't have any deep objection to virtualenvs, I've > just never seen the *point* (outside of the scenarios I mentioned), No problem. I was just responding to the assertion that people only use virtualenvs because they aren't aware of the alternatives, which I don't believe is true. It's likely many people aren't aware of python -S, or of everything that's possible via distutils.cfg. But even if they were, for the cases where I commonly see virtualenv in use, it solves the same problems more easily and with much less fiddling with config files and environment variables. Case in point: libraries that also install scripts for use in development or build processes. If you're DIY, you have to figure out where to put these, too, and make sure it's on your PATH. And if you want isolation, not only do you have to remember to run python -S every time, you also have to edit every script wrapper to put -S in the shebang line. With virtualenv+easy_install/pip, all of these things Just Simply Work, and (mostly) in an intuitive way. That's why people use it. > thus don't see what great advantage will be had by rearranging layouts > to make them shareable across platforms, when "throw stuff in a > directory" seems perfectly serviceable for that use case already. Tools > that *don't* support "just throw it in a directory" as a deployment > option are IMO unpythonic -- practicality beats purity, after all. ;-) No disagreement here. I think virtualenv's sweet spot is as a convenient tool for development environments (used in virtualenvwrapper fashion, where the file structure of the virtualenv itself is hidden away and you never see it at all). I think it's fine to deploy _into_ a virtualenv, if you find that convenient too (though I think there are real advantages to deploying just a big ball of code with no need for installers). But I see little reason to make virtualenvs relocatable or sharable across platforms. I don't think virtualenvs as on on-disk file structure make a good distribution/deployment mechanism at all. IOW, I hijacked this thread (sorry) to respond to a specific denigration of the value of virtualenv that I disagree with. I don't care about making virtualenvs consistent across platforms. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From rdmurray at bitdance.com Mon Mar 26 19:19:00 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 26 Mar 2012 13:19:00 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> Message-ID: <20120326171901.A670E2500E9@webabinitio.net> On Mon, 26 Mar 2012 12:55:42 -0400, PJ Eby wrote: > On Sat, Mar 24, 2012 at 8:41 PM, Ben Finney wrote: > > So, again, why make your browser window *for reading text* that large? > > Because I have one browser window, and it's maximized. And I can do this, > because most websites are designed in such a way that they have usable > margins for text flows. Even PEPs and Python mailing list archives, for > example, have sane text margins -- shall we go back and make *those* > dependent on window width instead? > [...] > > Designers' rules of thumb for text width are based on empirical > observations of focal length, saccades, etc. If you have special needs > visually, you're more likely to require the text read to you, than to have > narrower text, and I at least am unable to conceive of a visual disability > that would be helped by *increasing* the text width. > > In other words, there is a well-established *majority* need for how many > characters should appear in an unwrapped line of text, based on majority > physiology. Designers who limit it based on pixel size are Doing It Wrong; > the max width should be based on em's rather than pixels. (Font sizes are a > separate issue.) I'm with Philip on this one. I hate web sites that have a fixed text width (so that you can't resize narrower and still read it), but I also prefer ones that set the max width to the "readable size" in number of character positions. Like Philip, I have *one* window. My window manager (ratpoison) is more like 'screen' for X: you *can* split the window up, but it is *much* more useful to have only one window visible at a time, most of the time. So splitting the window in order to make the text narrow enough to read slows down my workflow. (Which means that on the python docs and the bug tracker I just put up with reading it wide...) I realize that I'm in the minority, though :) --David From v+python at g.nevcal.com Mon Mar 26 19:32:26 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 26 Mar 2012 10:32:26 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <20120326171901.A670E2500E9@webabinitio.net> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> <20120326171901.A670E2500E9@webabinitio.net> Message-ID: <4F70A82A.70307@g.nevcal.com> On 3/26/2012 10:19 AM, R. David Murray wrote: > Like Philip, I have*one* window. My window manager (ratpoison) is more > like 'screen' for X: you*can* split the window up, but it is*much* more > useful to have only one window visible at a time, most of the time. I'm amazed at the number of people that use maximized windows, one application at a time. I have 2 1600x1200 displays, with 10-30 overlapping windows partially visible, and sometimes wish I had 3 displays, so I could see more windows at a time... but then I'd have to turn my head more, so maybe this is optimal. Two displays lets me get my autohidden taskbar in the vertical center, for quick access from either side. I occasionally maximize a window on one screen or the other (one portrait, one landscape mode), mostly for picture viewing, but a few other times as well. > I realize that I'm in the minority, though No doubt I am too :) Everyone is a minority, when all idiotsyncrasies [sic] are considered! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 26 19:58:13 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 26 Mar 2012 10:58:13 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F70A82A.70307@g.nevcal.com> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> <20120326171901.A670E2500E9@webabinitio.net> <4F70A82A.70307@g.nevcal.com> Message-ID: <4F70AE35.9060903@stoneleaf.us> Glenn Linderman wrote: > On 3/26/2012 10:19 AM, R. David Murray wrote: >> Like Philip, I have *one* window. My window manager (ratpoison) is more >> like 'screen' for X: you *can* split the window up, but it is *much* more >> useful to have only one window visible at a time, most of the time. > > I'm amazed at the number of people that use maximized windows, one > application at a time. I have 2 1600x1200 displays, with 10-30 > overlapping windows partially visible, and sometimes wish I had 3 > displays, so I could see more windows at a time... but then I'd have to > turn my head more, so maybe this is optimal. Two displays lets me get > my autohidden taskbar in the vertical center, for quick access from > either side. I also have two monitors, I use goScreen for three virtual desktops (I'm stuck with XP -- which is to say MS), and each application I use is full-screen while I'm using it. I find windows that I'm not using at that moment a distraction. ~Ethan~ From ethan at stoneleaf.us Mon Mar 26 21:25:20 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 26 Mar 2012 12:25:20 -0700 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <4F6F6DD8.9050901@pearwood.info> Message-ID: <4F70C2A0.10405@stoneleaf.us> Georg Brandl wrote: > On 25.03.2012 21:11, Steven D'Aprano wrote: >> Georg Brandl wrote: >> >>> Thanks everyone for the overwhelmingly positive feedback. I've committed the >>> new design to 3.2 and 3.3 for now, and it will be live for the 3.3 docs >>> momentarily (3.2 isn't rebuilt at the moment until 3.2.3 final goes out). >>> I'll transplant to 2.7 too, probably after the final release of 2.7.3. >> I think it would be better to leave 2.7 with the old theme, to keep it >> visually distinct from the nifty new theme used with the nifty new 3.2 and 3.3 >> versions. > > Hmm, -0 here. I'd like more opinions on this from other devs. +1 on keeping the 2.x and 3.x styles separate. ~Ethan~ From ethan at stoneleaf.us Mon Mar 26 21:25:39 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 26 Mar 2012 12:25:39 -0700 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: <4F70C2B3.4080709@stoneleaf.us> Georg Brandl wrote: > Here's another try, mainly with default browser font size, more contrast and > collapsible sidebar again: > > http://www.python.org/~gbrandl/build/html2/ > > I've also added a little questionable gimmick to the sidebar (when you collapse > it and expand it again, the content is shown at your current scroll location). > > Have fun! > Georg Looks great! Thanks! ~Ethan~ From v+python at g.nevcal.com Mon Mar 26 21:14:15 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 26 Mar 2012 12:14:15 -0700 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: <4F70AE35.9060903@stoneleaf.us> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> <20120326171901.A670E2500E9@webabinitio.net> <4F70A82A.70307@g.nevcal.com> <4F70AE35.9060903@stoneleaf.us> Message-ID: <4F70C007.8080606@g.nevcal.com> On 3/26/2012 10:58 AM, Ethan Furman wrote: > Glenn Linderman wrote: >> On 3/26/2012 10:19 AM, R. David Murray wrote: >>> Like Philip, I have *one* window. My window manager (ratpoison) is >>> more >>> like 'screen' for X: you *can* split the window up, but it is *much* >>> more >>> useful to have only one window visible at a time, most of the time. >> >> I'm amazed at the number of people that use maximized windows, one >> application at a time. I have 2 1600x1200 displays, with 10-30 >> overlapping windows partially visible, and sometimes wish I had 3 >> displays, so I could see more windows at a time... but then I'd have >> to turn my head more, so maybe this is optimal. Two displays lets me >> get my autohidden taskbar in the vertical center, for quick access >> from either side. > > I also have two monitors, I use goScreen for three virtual desktops > (I'm stuck with XP -- which is to say MS), and each application I use > is full-screen while I'm using it. I find windows that I'm not using > at that moment a distraction. Interesting. I guess I'm always distracted :) But I often need data or information from multiple applications in order to make progress on a project... which is why each application seldom gets maximized. I even use multiple Firefox profiles concurrently, to have multiple browsers open for different purposes, as well as multiple tabs within each of them. Sometimes I even need multiple instances of Emacs, as well as multiple windows for a single instance, but that is usually very temporary, due to an interruption (distraction). So my pet peeve about web sites is those that, although they seem to dynamically adjust to different monitor sizes, seem to miscalculate, and display a horizontal scroll bar, and consistently chop stuff off on the right edge of my non-maximized window. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 26 21:21:06 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Mar 2012 21:21:06 +0200 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 References: <4F6F6DD8.9050901@pearwood.info> <4F70C2A0.10405@stoneleaf.us> Message-ID: <20120326212106.76ea8f40@pitrou.net> On Mon, 26 Mar 2012 12:25:20 -0700 Ethan Furman wrote: > Georg Brandl wrote: > > On 25.03.2012 21:11, Steven D'Aprano wrote: > >> Georg Brandl wrote: > >> > >>> Thanks everyone for the overwhelmingly positive feedback. I've committed the > >>> new design to 3.2 and 3.3 for now, and it will be live for the 3.3 docs > >>> momentarily (3.2 isn't rebuilt at the moment until 3.2.3 final goes out). > >>> I'll transplant to 2.7 too, probably after the final release of 2.7.3. > >> I think it would be better to leave 2.7 with the old theme, to keep it > >> visually distinct from the nifty new theme used with the nifty new 3.2 and 3.3 > >> versions. > > > > Hmm, -0 here. I'd like more opinions on this from other devs. > > +1 on keeping the 2.x and 3.x styles separate. I don't really understand the point. If we want to distinguish between 2.x and 3.x, perhaps a lighter difference would suffice. Regards Antoine. From pje at telecommunity.com Mon Mar 26 21:27:48 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 26 Mar 2012 15:27:48 -0400 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4F70A022.6060507@oddbird.net> References: <4F70A022.6060507@oddbird.net> Message-ID: On Mon, Mar 26, 2012 at 12:58 PM, Carl Meyer wrote: > No disagreement here. I think virtualenv's sweet spot is as a convenient > tool for development environments (used in virtualenvwrapper fashion, > where the file structure of the virtualenv itself is hidden away and you > never see it at all). I think it's fine to deploy _into_ a virtualenv, > if you find that convenient too (though I think there are real > advantages to deploying just a big ball of code with no need for > installers). But I see little reason to make virtualenvs relocatable or > sharable across platforms. I don't think virtualenvs as on on-disk file > structure make a good distribution/deployment mechanism at all. > > IOW, I hijacked this thread (sorry) to respond to a specific denigration > of the value of virtualenv that I disagree with. I don't care about > making virtualenvs consistent across platforms. > Well, if you're the virtualenv maintainer (or at least the PEP author), and you're basically shooting down the principal rationale for reorganizing the Windows directory layout, then it's not really much of a hijack - it's pretty darn central to the thread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 26 21:40:25 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 26 Mar 2012 12:40:25 -0700 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <20120326212106.76ea8f40@pitrou.net> References: <4F6F6DD8.9050901@pearwood.info> <4F70C2A0.10405@stoneleaf.us> <20120326212106.76ea8f40@pitrou.net> Message-ID: <4F70C629.9040607@stoneleaf.us> Antoine Pitrou wrote: > On Mon, 26 Mar 2012 12:25:20 -0700 > Ethan Furman wrote: >> Georg Brandl wrote: >>> On 25.03.2012 21:11, Steven D'Aprano wrote: >>>> Georg Brandl wrote: >>>> >>>>> Thanks everyone for the overwhelmingly positive feedback. I've committed the >>>>> new design to 3.2 and 3.3 for now, and it will be live for the 3.3 docs >>>>> momentarily (3.2 isn't rebuilt at the moment until 3.2.3 final goes out). >>>>> I'll transplant to 2.7 too, probably after the final release of 2.7.3. >>>> I think it would be better to leave 2.7 with the old theme, to keep it >>>> visually distinct from the nifty new theme used with the nifty new 3.2 and 3.3 >>>> versions. >>> Hmm, -0 here. I'd like more opinions on this from other devs. >> +1 on keeping the 2.x and 3.x styles separate. > > I don't really understand the point. If we want to distinguish between > 2.x and 3.x, perhaps a lighter difference would suffice. The point being that 2.x is finished, and the bulk of our effort is now on 3.x. By not changing the 2.x docs we are emphasizing that 3.x is the way to go. ~Ethan~ From ethan at stoneleaf.us Mon Mar 26 21:13:10 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 26 Mar 2012 12:13:10 -0700 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: <20120325140752.7f783801@pitrou.net> References: <20120325140752.7f783801@pitrou.net> Message-ID: <4F70BFC6.4040507@stoneleaf.us> Antoine Pitrou wrote: > On Sun, 25 Mar 2012 08:34:44 +0200 > Also I think there should be some jquery animation when > collapsing/expanding. Please, no. I don't need my technical web pages singing and dancing for me. ;) ~Ethan~ From fuzzyman at voidspace.org.uk Mon Mar 26 22:32:26 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 26 Mar 2012 21:32:26 +0100 Subject: [Python-Dev] cpython: Adding unittest.mock documentation In-Reply-To: References: Message-ID: <46A927A7-2671-45CD-AF24-F820D38341CE@voidspace.org.uk> On 26 Mar 2012, at 08:11, Georg Brandl wrote: > On 26.03.2012 00:13, michael.foord wrote: >> http://hg.python.org/cpython/rev/adc1fc2dc872 >> changeset: 75938:adc1fc2dc872 >> user: Michael Foord >> date: Sun Mar 25 23:12:55 2012 +0100 >> summary: >> Adding unittest.mock documentation >> >> files: >> Doc/library/development.rst | 6 + >> Doc/library/unittest.mock-examples.rst | 887 +++++++++ >> Doc/library/unittest.mock-getting-started.rst | 419 ++++ >> Doc/library/unittest.mock-helpers.rst | 537 +++++ >> Doc/library/unittest.mock-magicmethods.rst | 226 ++ >> Doc/library/unittest.mock-patch.rst | 538 +++++ >> Doc/library/unittest.mock.rst | 900 ++++++++++ >> Lib/unittest/mock.py | 8 +- >> 8 files changed, 3516 insertions(+), 5 deletions(-) > > That seems a bit much splitting to me. Ok, it's just the style of the current mock documentation. My original intention *was* to create one big-ass api doc more in keeping with the current library doc styles. > > (By the way, the ".. module::" directive should only be in *one* place.) Thanks. Maybe sphinx could complain about duplicates. > > I would organize the mock, mock-patch, mock-magicmethods and mock-helpers as one > file in Doc/library, and put the other two in Doc/howto, just as for logging. > The examples pages aren't written in a howto style - there's a getting started guide (basic examples) and then a bunch of separate more advanced examples illustrating specific features of unittest.mock or particular mocking scenarios. I'll move the api documentation into a single doc and the getting started guide and examples as a second page. > In general, I wouldn't mind splitting off more of the exemplary material from > the main library docs, putting it in the howto section. Hmmm... in general I think I'd like to see the documentation for non-trivial modules include some examples (separate from - and after - the api docs) . Hiving them off as a howto makes them harder to find. More in depth howto guides in addition to this are a good thing though. Michael > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From v+python at g.nevcal.com Mon Mar 26 22:21:41 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 26 Mar 2012 13:21:41 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: References: <4F70A022.6060507@oddbird.net> Message-ID: <4F70CFD5.9070306@g.nevcal.com> On 3/26/2012 12:27 PM, PJ Eby wrote: > On Mon, Mar 26, 2012 at 12:58 PM, Carl Meyer > wrote: > > No disagreement here. I think virtualenv's sweet spot is as a > convenient > tool for development environments (used in virtualenvwrapper fashion, > where the file structure of the virtualenv itself is hidden away > and you > never see it at all). I think it's fine to deploy _into_ a virtualenv, > if you find that convenient too (though I think there are real > advantages to deploying just a big ball of code with no need for > installers). But I see little reason to make virtualenvs > relocatable or > sharable across platforms. I don't think virtualenvs as on on-disk > file > structure make a good distribution/deployment mechanism at all. > > IOW, I hijacked this thread (sorry) to respond to a specific > denigration > of the value of virtualenv that I disagree with. I don't care about > making virtualenvs consistent across platforms. > > > Well, if you're the virtualenv maintainer (or at least the PEP > author), and you're basically shooting down the principal rationale > for reorganizing the Windows directory layout, then it's not really > much of a hijack - it's pretty darn central to the thread! What I read here is a bit different than Mr Eby read, it seems. I read Carl as suggesting that keeping deployment copies of virtualenvs as foolish, but thinking it is fine to deploy into a virtualenv file structure (although preferring to deplay a big ball of code, himself). Personally, I see application deployment as a big ball of code the preferred technique also, but library/module deployment is harder to do that way... it sort of defeats the ability to then bundle the library/module into the big ball of code for the application. But if the goal is to deploy a big ball of code, that would run on top of an installed Python or virtualenv Python, then that is a lot easier if the only modules used are Python modules (no C extensions). Such can be bundled into a zip file, with little support, such that a relative Python novice as myself can figure it out and implement it quickly. C extensions cannot be run from a zip file, so then one needs support code to unzip the C binaries dynamically, and (possibly) delete them when done. Or am I missing something? Hmm. And here's something else that might be missing: integration of the launcher with .py files that are actually ZIP archives... where does it find the #! line? (probably it can't, currently -- I couldn't figure out how to make it do it). Is it possible to add a #! line at the beginning of a ZIP archive for the launcher to use, and still have Python recognize the result as a ZIP archive? I know self-extracting archives put an executable program in front of a ZIP file, and the result is still recognized by most ZIP archivers, but I tried just putting a #! line followed by a ZIP archive, and Python gave me SyntaxError: unknown decode error. -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Mon Mar 26 23:26:34 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 26 Mar 2012 14:26:34 -0700 Subject: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout) In-Reply-To: <4F70CFD5.9070306@g.nevcal.com> References: <4F70A022.6060507@oddbird.net> <4F70CFD5.9070306@g.nevcal.com> Message-ID: <4F70DF0A.8020901@g.nevcal.com> On 3/26/2012 1:21 PM, Glenn Linderman wrote: > Hmm. And here's something else that might be missing: integration of > the launcher with .py files that are actually ZIP archives... where > does it find the #! line? (probably it can't, currently -- I couldn't > figure out how to make it do it). Is it possible to add a #! line at > the beginning of a ZIP archive for the launcher to use, and still have > Python recognize the result as a ZIP archive? I know self-extracting > archives put an executable program in front of a ZIP file, and the > result is still recognized by most ZIP archivers, but I tried just > putting a #! line followed by a ZIP archive, and Python gave me > SyntaxError: unknown decode error. OK, my first try there, I forgot the stupid Windows /b switch on copy, so apparently the ZIP archive got mangled. When I use ropy /b to join #!/usr/bin/python3.2 and a zip file, it now works. Sorry for the noise. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Mar 26 23:50:49 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Mar 2012 14:50:49 -0700 Subject: [Python-Dev] PEP 411 - request for pronouncement In-Reply-To: References: Message-ID: On Fri, Mar 23, 2012 at 2:51 AM, Eli Bendersky wrote: > PEP 411 -- Provisional packages in the Python standard library > > Has been updated with all accumulated feedback from list discussions. > Here it is: http://www.python.org/dev/peps/pep-0411/ (the text is also > pasted in the bottom of this email). > > The PEP received mostly positive feedback. The only undecided point is > where to specify that the package is provisional. Currently the PEP > mandates to specify it in the documentation and in the docstring. > Other suggestions were to put it in the code, either as a > __provisional__ attribute on the module, or collect all such modules > in a single sys.provisional list. > > According to http://blog.python.org/2012/03/2012-language-summit-report.html, > the PEP was discussed in the language summit and overall viewed > positively, although no final decision has been reached. > > ISTM a decision needs to be taken, which is why I request > pronouncement, with a recommendation on the requirement the PEP should > make of provisional modules (process details). I think the PEP is almost ready for approval. Congratulations! A few comments: - I'd leave some wiggle room for the docs owner (Georg) about the exact formulation of the text blurb included for provisional modules and the glossary entry; I don't want the PEP to have the last word here. - I think we are settling on the term "feature release" instead of the somewhat ambiguous "minor release". - As was discussed at the language summit, I'd like to emphasize that the bar for making changes to a provisional package should be considered pretty high. That is, while we don't make guarantees about backward compatibility, we still expect that most of the API of most provisional packages will be unchanged at graduation. Withdrawals should also be pretty rare. - Should we limit the duration of the provisional state to 1 or 2 feature releases? - I'm not sure what to do with regex -- it may be better to just include in as "re" and keep the old re module around under another name ("sre" has been proposed half jokingly). PS. Please use the version in the peps repo as the starting point of future edits. -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Tue Mar 27 00:04:05 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 27 Mar 2012 01:04:05 +0300 Subject: [Python-Dev] PEP 393 decode() oddity In-Reply-To: References: Message-ID: 26.03.12 01:28, Victor Stinner ???????(??): > Cool, Python 3.3 is *much* faster to decode pure ASCII :-) He even faster on large data. 1000 characters is not enough to completely neutralize the constant costs of the function calls. Python 3.3 is really cool. >> encoding string 2.7 3.2 3.3 >> >> ascii " " * 1000 5.4 5.3 1.2 > > 4.5 faster than Python 2 here. And it can be accelerated (issue #14419). >> utf-8 " " * 1000 6.7 2.4 2.1 > > 3.2x faster In theory, the speed must coincide with latin1 speed. And it coincides in the limit, for large data. For medium data starting overhead cost is visible and utf-8 is a bit slower than it could be. > It's cool because in practice, a lot of strings are pure ASCII (as > Martin showed in its Django benchmark). But there are a lot of non-ascii text. But with mostly-ascii text, containing at least one non-ascii character (for example, Martin's full name), utf-8 decoder copes much worse. And worse than in Python 3.2. The decoder may be slower only by a small amount, related to scanning. I believe that the stock to optimize exists. > I'm interested by any patch optimizing any Python codecs. I'm not > working on optimizing Python Unicode anymore, various benchmarks > showed me that Python 3.3 is as good or faster than Python 3.2. That's > enough for me. Then would you accept a patch, proposed by me in issue 14249? This patch will not catch up all arrears, but it is very simple and should not cause objections. Developed by me now optimization accelerates decoder even more, but so far it is too ugly spaghetti-code. > When Python 3.3 is slower than Python 3.2, it's because Python 3.3 > must compute the maximum character of the result, and I fail to see > how to optimize this requirement. A significant slowdown was caused by the use of PyUnicode_WRITE with a variable kind in loop. In some cases, it would be useful to expand the loop in cascade of independent loops which fallback onto each other (as you have already done in utf8_scanner). From storchaka at gmail.com Tue Mar 27 00:09:47 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 27 Mar 2012 01:09:47 +0300 Subject: [Python-Dev] PEP 393 decode() oddity In-Reply-To: References: Message-ID: 27.03.12 01:04, Serhiy Storchaka ???????(??): > 26.03.12 01:28, Victor Stinner ???????(??): > loop in cascade of independent loops which fallback onto each other (as > you have already done in utf8_scanner). Sorry. Not you. Antoine Pitrou. From storchaka at gmail.com Mon Mar 26 23:10:43 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 27 Mar 2012 00:10:43 +0300 Subject: [Python-Dev] PEP 393 decode() oddity In-Reply-To: <20120325225513.Horde.v_hfaKGZi1VPb4YxHpTWb7A@webmail.df.eu> References: <20120325190137.7f0ffce0@pitrou.net> <20120325225513.Horde.v_hfaKGZi1VPb4YxHpTWb7A@webmail.df.eu> Message-ID: 25.03.12 23:55, martin at v.loewis.de ???????(??): >> The results are fairly stable (?0.1 ?sec) from run to run. It looks >> funny thing. > > This is not surprising. Thank you. Indeed, it is logical. I looked at the code and do not see how to speed it up. From solipsis at pitrou.net Tue Mar 27 00:25:15 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 27 Mar 2012 00:25:15 +0200 Subject: [Python-Dev] cpython: Fix time.steady(strict=True): don't use CLOCK_REALTIME References: Message-ID: <20120327002515.523b8e29@pitrou.net> On Mon, 26 Mar 2012 22:53:37 +0200 victor.stinner wrote: > http://hg.python.org/cpython/rev/566527ace50b > changeset: 75960:566527ace50b > user: Victor Stinner > date: Mon Mar 26 22:53:14 2012 +0200 > summary: > Fix time.steady(strict=True): don't use CLOCK_REALTIME Victor, could we have a PEP on all this? I think everyone has lost track of what you are trying to do with these new methods. cheers Antoine. From zooko at zooko.com Tue Mar 27 00:31:48 2012 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Mon, 26 Mar 2012 16:31:48 -0600 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On Fri, Mar 23, 2012 at 11:27 AM, Victor Stinner wrote: > > time.steady(strict=False) is what you need to implement timeout. No, that doesn't fit my requirements, which are about event scheduling, profiling, and timeouts. See below for more about my requirements. I didn't say this explicitly enough in my previous post: Some use cases (timeouts, event scheduling, profiling, sensing) require a steady clock. Others (calendaring, communicating times to users, generating times for comparison to remote hosts) require a wall clock. Now here's the kicker: each use case incur significant risks if it uses the wrong kind of clock. If you're implementing event scheduling or sensing and control, and you accidentally get a wall clock when you thought you had a steady clock, then your program may go seriously wrong -- events may fire in the wrong order, measurements of your sensors may be wildly incorrect. This can lead to serious accidents. On the other hand, if you're implementing calendaring or display of "real local time of day" to a user, and you are using a steady clock for some reason, then you risk displaying incorrect results to the user. So using one kind of clock and then "falling back" to the other kind is a choice that should be rare, explicit, and discouraged. The provision of such a function in the standard library is an attractive nuisance -- a thing that people naturally think that they want when they haven't though about it very carefully, but that is actually dangerous. If someone has a use case which fits the "steady or else fall back to wall clock" pattern, I would like to learn about it. Regards, Zooko From stefan at bytereef.org Tue Mar 27 00:47:49 2012 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 27 Mar 2012 00:47:49 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed In-Reply-To: References: <20120323092255.GA17205@sleipnir.bytereef.org> <20120323104005.GA17581@sleipnir.bytereef.org> Message-ID: <20120326224749.GA23190@sleipnir.bytereef.org> Victor Stinner wrote: > > The 80x is a ballpark figure for the maximum expected speedup for > > standard numerical floating point applications. > > Ok, but it's just surprising when you read the What's New document. > 72x and 80x look to be inconsistent. Yes, indeed, I'll reword that. > > For huge numbers _decimal is also faster than int: > > > > factorial(1000000): > > > > _decimal, calculation time: 6.844487905502319 > > _decimal, tostr(): ?? ?? ?? ?? ??0.033592939376831055 > > > > int, calculation time: 17.96010398864746 > > int, tostr(): ... still running ... > > Hum, with a resolution able to store the result with all digits? Yes, you have to set context.prec (and emax) to the maximum values, then the result is an exact integer. The conversion to string is so fast because there is no complicated base conversion. > If yes, would it be possible to reuse the multiply algorithm of _decimal > (and maybe of other functions) for int? Or does it depend heavily on > _decimal internal structures? Large parts of the Number Theoretic Transform could be reused, but there would be still quite a bit of work. Stefan Krah From victor.stinner at gmail.com Tue Mar 27 01:07:39 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 01:07:39 +0200 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: > So using one kind of clock and then "falling back" to the other kind > is a choice that should be rare, explicit, and discouraged. The > provision of such a function in the standard library is an attractive > nuisance -- a thing that people naturally think that they want when > they haven't though about it very carefully, but that is actually > dangerous. > > If someone has a use case which fits the "steady or else fall back to > wall clock" pattern, I would like to learn about it. Python 3.2 doesn't provide a monotonic clock, so most program uses time.time() even if a monotonic clock would be better in some functions. For these programs, you can replace time.time() by time.steady() where you need to compute a time delta (e.g. compute a timeout) to avoid issues with the system clock update. The idea is to improve the program without refusing to start if no monotonic clock is available. Victor From glyph at twistedmatrix.com Tue Mar 27 00:47:05 2012 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 26 Mar 2012 18:47:05 -0400 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On Mar 26, 2012, at 6:31 PM, Zooko Wilcox-O'Hearn wrote: > On Fri, Mar 23, 2012 at 11:27 AM, Victor Stinner > wrote: >> >> time.steady(strict=False) is what you need to implement timeout. > > No, that doesn't fit my requirements, which are about event > scheduling, profiling, and timeouts. See below for more about my > requirements. > > I didn't say this explicitly enough in my previous post: > > Some use cases (timeouts, event scheduling, profiling, sensing) > require a steady clock. Others (calendaring, communicating times to > users, generating times for comparison to remote hosts) require a wall > clock. > > Now here's the kicker: each use case incur significant risks if it > uses the wrong kind of clock. > > If you're implementing event scheduling or sensing and control, and > you accidentally get a wall clock when you thought you had a steady > clock, then your program may go seriously wrong -- events may fire in > the wrong order, measurements of your sensors may be wildly incorrect. > This can lead to serious accidents. On the other hand, if you're > implementing calendaring or display of "real local time of day" to a > user, and you are using a steady clock for some reason, then you risk > displaying incorrect results to the user. > > So using one kind of clock and then "falling back" to the other kind > is a choice that should be rare, explicit, and discouraged. The > provision of such a function in the standard library is an attractive > nuisance -- a thing that people naturally think that they want when > they haven't though about it very carefully, but that is actually > dangerous. > > If someone has a use case which fits the "steady or else fall back to > wall clock" pattern, I would like to learn about it. I feel that this should be emphasized. Zooko knows what he's talking about here. Listen to him :). (Antoine has the right idea. I think it's well past time for a PEP on this feature.) -glyph From victor.stinner at gmail.com Tue Mar 27 01:32:38 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 01:32:38 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock Message-ID: Hi, I started to write the PEP 418 to clarify the notions of monotonic and steady clocks. The PEP is a draft and everyone is invited to contribute! http://www.python.org/dev/peps/pep-0418/ http://hg.python.org/peps/file/tip/pep-0418.txt Victor From victor.stinner at gmail.com Tue Mar 27 02:16:52 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 02:16:52 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: Message-ID: > I started to write the PEP 418 to clarify the notions of monotonic and > steady clocks. > > The PEP is a draft and everyone is invited to contribute! time.steady() doesn't fit the benchmarking use case: it looks like we have to decide between stability and clock resolution. QueryPerformanceCounter() has a good resolution for benchmarking, but it is not monotonic and so GetTickCount64() would be used for time.steady(). GetTickCount64() is monotonic but has only a resolution of 1 millisecond. We might add a third new function which provides the most accurate clock with or without a known starting point. We cannot use QueryPerformanceCounter() to enhance time.time() resolution because it has an unknown starting point. Victor From scott+python-dev at scottdial.com Tue Mar 27 02:23:12 2012 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 26 Mar 2012 20:23:12 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: Message-ID: <4F710870.9090602@scottdial.com> On 3/26/2012 7:32 PM, Victor Stinner wrote: > I started to write the PEP 418 to clarify the notions of monotonic and > steady clocks. """ time.steady This clock advances at a steady rate relative to real time. It may be adjusted. """ Please do not call this "steady". If the clock can be adjusted, then it is not "steady" by any acceptable definition. I cannot fathom the utility of this function other than as a function that provides an automatic fallback from "time.monotonic()". More importantly: this definition of "steady" is in conflict with the C++0x definition of "steady" that is where you sourced this named from![1] """ time.steady(strict=False) falls back to another clock if no monotonic clock is not available or does not work, but it does never fail. """ As I say above, that is so far away from what "steady" implies that this is a misnomer. What you are describing is a best-effort clock, which sounds a lot more like the C++0x "high resolution" clock. """ time.steady(strict=True) raises OSError if monotonic clock fails or NotImplementedError if the system does not provide a monotonic clock """ What is the utility of "strict=True"? If I wanted that mode of operation, then why would I not just try to use "time.monotonic()" directly? At worst, it generates an "AttributeError" (although that is not clear from your PEP). What is the use case for "strict=True" that is not covered by your "time.monotonic()"? If you want to define new clocks, then I wish you would use the same definitions that C++0x is using. That is: system_clock = wall clock time monotonic_clock = always goes forward but can be adjusted steady_clock = always goes forward and cannot be adjusted high_resolution_clock = steady_clock || system_clock Straying from that is only going to create confusion. Besides that, the one use case for "time.steady()" that you give (benchmarking) is better served by a clock that follows the C++0x definition. As well, certain kinds of scheduling/timeouts would be better implemented with the C++0x definition for "steady" rather than the "monotonic" one and vice-versa. Rather, it seems you have a particular use-case in mind and have settled on calling that a "steady" clock despite how it belies its name. [1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3128.html#time.clock.steady """ Objects of class steady_clock represent clocks for which values of time_point advance at a steady rate relative to real time. That is, the clock may not be adjusted. """ -- Scott Dial scott at scottdial.com From zooko at zooko.com Tue Mar 27 04:26:23 2012 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Mon, 26 Mar 2012 20:26:23 -0600 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F710870.9090602@scottdial.com> References: <4F710870.9090602@scottdial.com> Message-ID: > ?system_clock = wall clock time > ?monotonic_clock = always goes forward but can be adjusted > ?steady_clock = always goes forward and cannot be adjusted > ?high_resolution_clock = steady_clock || system_clock Note that the C++ standard deprecated monotonic_clock once they realized that there is absolutely no point in having a clock that jumps forward but not back, and that none of the operating systems implement such a thing -- instead they all implement a clock which doesn't jump in either direction. http://stackoverflow.com/questions/6777278/what-is-the-rationale-for-renaming-monotonic-clock-to-steady-clock-in-chrono In other words, yes! +1! The C++ standards folks just went through the process that we're now going through, and if we do it right we'll end up at the same place they are: http://en.cppreference.com/w/cpp/chrono/system_clock """ system_clock represents the system-wide real time wall clock. It may not be monotonic: on most systems, the system time can be adjusted at any moment. It is the only clock that has the ability to map its time points to C time, and, therefore, to be displayed. steady_clock: monotonic clock that will never be adjusted high_resolution_clock: the clock with the shortest tick period available """ Note that we don't really have the option of providing a clock which is "monotonic but not steady" in the sense of "can jump forward but not back". It is a misunderstanding (doubtless due to the confusing name "monotonic") to think that such a thing is offered by the underlying platforms. We can choose to *call* it "monotonic", following POSIX instead of calling it "steady", following C++. Regards, Zooko From zooko at zooko.com Tue Mar 27 04:41:56 2012 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Mon, 26 Mar 2012 20:41:56 -0600 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: On Mon, Mar 26, 2012 at 5:07 PM, Victor Stinner wrote: >> >> If someone has a use case which fits the "steady or else fall back to wall clock" pattern, I would like to learn about it. > > Python 3.2 doesn't provide a monotonic clock, so most program uses time.time() even if a monotonic clock would be better in some functions. For these programs, you can replace time.time() by time.steady() where you need to compute a time delta (e.g. compute a timeout) to avoid issues with the system clock update. The idea is to improve the program without refusing to start if no monotonic clock is available. I agree that this is a reasonable use case. I think of it as basically being a kind of backward-compatibility, for situations where an unsteady clock is okay, and a steady clock isn't available. Twisted faces a similar issue: http://twistedmatrix.com/trac/ticket/2424 It might good for use cases like this to explicitly implement the try-and-fallback, since they might have specific needs about how it is done. For one thing, some such uses may need to emit a warning, or even to require the caller to explicitly override, such a refusing to start if a steady clock isn't available unless the user specifies "--unsteady-clock-ok". For motivating examples, consider software written using Twisted > 12.0 or Python > 3.2 which is using a clock to drive real world sensing and control -- measuring the position of a machine and using time deltas to calculate the machine's velocity, in order to automatically control the motion of the machine. For some uses, it is okay if the measurement could, in rare cases, be drastically wrong. For other uses, that is not an acceptable risk. One reason I'm sensitive to this issue is that I work in the field of security, and making the behavior dependent on the system clock extends the "reliance set", i.e. the set of things that an attacker could use against you. For example, if your robot depends on the system clock for its sensing and control, and if your system clock obeys NTP, then the set of things that an attacker could use against you includes your NTP servers. If your robot depends instead on a steady clock, then NTP servers are not in the reliance set. Now, if your control platform doesn't have a steady clock, you may choose to go ahead, while making sure that the NTP servers are authenticated, or you may choose to disable NTP on the control platform, etc., but that choice might need to be made explicitly by the operator, rather than automatically by the library. Regards, Zooko From anacrolix at gmail.com Tue Mar 27 04:55:33 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 27 Mar 2012 10:55:33 +0800 Subject: [Python-Dev] Drop the new time.wallclock() function? In-Reply-To: References: <4F6137EF.9000000@gmail.com> Message-ID: Inside time.steady, there are two different clocks trying to get out. I think this steady business should be removed sooner rather than later. -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacrolix at gmail.com Tue Mar 27 04:59:40 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 27 Mar 2012 10:59:40 +0800 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> Message-ID: So does anyone care to dig into the libstd++/boost/windoze implementation to see how they each did steady_clock? -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Mar 27 05:23:32 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 27 Mar 2012 12:23:32 +0900 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> Message-ID: On Tue, Mar 27, 2012 at 1:35 AM, PJ Eby wrote: > On Sun, Mar 25, 2012 at 2:56 AM, Stephen J. Turnbull > wrote: >> >> But since he's arguing the >> other end in the directory layout thread (where he says there are many >> special ways to invoke Python so that having different layouts on >> different platforms is easy to work around), I can't give much weight >> to his preference here. > > > You're misconstruing my argument there: I said, rather, that the One Obvious > Way to deploy a Python application is to dump everything in one directory, > as that is the one way that Python has supported for at least 15 years now. It's not at all obvious on any of the open source platforms (Cygwin, Debian, Gentoo, and MacPorts) that I use. In all cases, both several python binaries and installed libraries end up in standard places in the distro-maintained hierarchies, and it is not hard to confuse the distro-installed Pythons. Being confident that one knows enough to set up a single directory correctly in the face of some of the unlovely things that packages may do requires more knowledge of how Python's import etc works than I can boast: virtualenv is a godsend. By analogy, yes, I think it makes sense to ask you to learn a bit about CSS and add a single line like "body { width: 65em; }" to your local config. That's one reason why CSS is designed to cascade. Of course, even better yet would be if the browsers wrote the CSS for you (which probably wouldn't be too hard, if I knew any XUL, which I don't). > The comparison to CSS is also lost on me here; creating user-specific CSS is > more aptly comparable telling people to write their own virtualenv > implementations from scratch, and resizing the browser window is more akin > to telling people to create a virtualenv every time they *run* the > application, rather than just once when installing it. Huh, if you say so -- I didn't realize that virtualenv did so little that it could be written in one line. All I know (and care) is that it promises to do all that stuff for me, and without affecting the general public (ie, the distro-provided Python apps). And that's why I think the width of pages containing flowed text should be left up to the user to configure. From eliben at gmail.com Tue Mar 27 05:23:55 2012 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 27 Mar 2012 05:23:55 +0200 Subject: [Python-Dev] PEP 411 - request for pronouncement In-Reply-To: References: Message-ID: > I think the PEP is almost ready for approval. Congratulations! A few comments: > > - I'd leave some wiggle room for the docs owner (Georg) about the > exact formulation of the text blurb included for provisional modules > and the glossary entry; I don't want the PEP to have the last word > here. Sure, Georg is free to modify the pep to amend the formulation if he wants to. > > - I think we are settling on the term "feature release" instead of the > somewhat ambiguous "minor release". Fixed > > - As was discussed at the language summit, I'd like to emphasize that > the bar for making changes to a provisional package should be > considered pretty high. That is, while we don't make guarantees about > backward compatibility, we still expect that most of the API of most > provisional packages will be unchanged at graduation. Withdrawals > should also be pretty rare. > Added this emphasis at the end of the "Criteria for graduation" section. > - Should we limit the duration of the provisional state to 1 or 2 > feature releases? Initially the PEP came out with a 1-release limit, but some of the devs pointed out (http://mail.python.org/pipermail/python-dev/2012-February/116406.html) that me should not necessarily restrict ourselves. > > - I'm not sure what to do with regex -- it may be better to just > include in as "re" and keep the old re module around under another > name ("sre" has been proposed half jokingly). > Document updated in the PEPs Hg, rev a1bb0a9af63f. Eli From guido at python.org Tue Mar 27 05:34:51 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Mar 2012 20:34:51 -0700 Subject: [Python-Dev] PEP 411 - request for pronouncement In-Reply-To: References: Message-ID: On Mon, Mar 26, 2012 at 8:23 PM, Eli Bendersky wrote: >> I think the PEP is almost ready for approval. Congratulations! A few comments: >> >> - I'd leave some wiggle room for the docs owner (Georg) about the >> exact formulation of the text blurb included for provisional modules >> and the glossary entry; I don't want the PEP to have the last word >> here. > > Sure, Georg is free to modify the pep to amend the formulation if he wants to. Ok. He can do that at any time. :-) >> - I think we are settling on the term "feature release" instead of the >> somewhat ambiguous "minor release". > > Fixed Great. >> - As was discussed at the language summit, I'd like to emphasize that >> the bar for making changes to a provisional package should be >> considered pretty high. That is, while we don't make guarantees about >> backward compatibility, we still expect that most of the API of most >> provisional packages will be unchanged at graduation. Withdrawals >> should also be pretty rare. >> > > Added this emphasis at the end of the "Criteria for graduation" section. Cool. >> - Should we limit the duration of the provisional state to 1 or 2 >> feature releases? > > Initially the PEP came out with a 1-release limit, but some of the > devs pointed out > (http://mail.python.org/pipermail/python-dev/2012-February/116406.html) > that me should not necessarily restrict ourselves. Gotcha. >> - I'm not sure what to do with regex -- it may be better to just >> include in as "re" and keep the old re module around under another >> name ("sre" has been proposed half jokingly). >> > > Document updated in the PEPs Hg, rev a1bb0a9af63f. TBH I'm not sure what I meant to say about regex in http://mail.python.org/pipermail/python-dev/2012-January/115962.html ... But I think if we end up changing the decision about this or any others that doesn't invalidate this PEP, which is informational PEP anyway. I've marked it up as Approved. Thanks, and congrats! -- --Guido van Rossum (python.org/~guido) From scott+python-dev at scottdial.com Tue Mar 27 05:36:10 2012 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 26 Mar 2012 23:36:10 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> Message-ID: <4F7135AA.9000203@scottdial.com> On 3/26/2012 10:59 PM, Matt Joiner wrote: > So does anyone care to dig into the libstd++/boost/windoze > implementation to see how they each did steady_clock? The Boost implementation can be summarized as: system_clock: mac = gettimeofday posix = clock_gettime(CLOCK_REALTIME) win = GetSystemTimeAsFileTime steady_clock: mac = mach_absolute_time posix = clock_gettime(CLOCK_MONOTONIC) win = QueryPerformanceCounter high_resolution_clock: * = { steady_clock, if available system_clock, otherwise } Whether or not these implementations meet the specification is an exercise left to the reader.. -- Scott Dial scott at scottdial.com From eliben at gmail.com Tue Mar 27 05:43:51 2012 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 27 Mar 2012 05:43:51 +0200 Subject: [Python-Dev] PEP 411 - request for pronouncement In-Reply-To: References: Message-ID: > > I've marked it up as Approved. Thanks, and congrats! > Thanks! Eli From eliben at gmail.com Tue Mar 27 05:47:37 2012 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 27 Mar 2012 05:47:37 +0200 Subject: [Python-Dev] PEP 411 - request for pronouncement In-Reply-To: References: Message-ID: On Tue, Mar 27, 2012 at 05:34, Guido van Rossum wrote: > On Mon, Mar 26, 2012 at 8:23 PM, Eli Bendersky wrote: >>> I think the PEP is almost ready for approval. Congratulations! A few comments: >>> >>> - I'd leave some wiggle room for the docs owner (Georg) about the >>> exact formulation of the text blurb included for provisional modules >>> and the glossary entry; I don't want the PEP to have the last word >>> here. >> >> Sure, Georg is free to modify the pep to amend the formulation if he wants to. > > Ok. He can do that at any time. :-) > Georg, would you like to change the suggested phrasing in the PEP, or can I go on and add the "provisional package" term to the glossary (in 3.3 only, of course). Eli From merwok at netwok.org Tue Mar 27 05:53:44 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 26 Mar 2012 23:53:44 -0400 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: <4F6F6DD8.9050901@pearwood.info> Message-ID: <4F7139C8.5090301@netwok.org> Hi, Le 25/03/2012 15:25, Georg Brandl a ?crit : > On 25.03.2012 21:11, Steven D'Aprano wrote: >> I think it would be better to leave 2.7 with the old theme, >> to keep it visually distinct from the nifty new theme used >> with the nifty new 3.2 and 3.3 versions. > Hmm, -0 here. I'd like more opinions on this from other devs. I?m +0 on Steven?s proposal and +1 on whatever you will decide. Cheers From anacrolix at gmail.com Tue Mar 27 05:54:51 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 27 Mar 2012 11:54:51 +0800 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F7135AA.9000203@scottdial.com> References: <4F710870.9090602@scottdial.com> <4F7135AA.9000203@scottdial.com> Message-ID: Cheers, that clears things up. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Mar 27 06:45:15 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 27 Mar 2012 13:45:15 +0900 Subject: [Python-Dev] Playing with a new theme for the docs, iteration 2 In-Reply-To: References: Message-ID: On Mon, Mar 26, 2012 at 6:42 AM, Terry Reedy wrote: >> Of course you can always use a user stylesheet to override our choices. > > Can anyone tell me the best way to do that with FireFox? http://kb.mozillazine.org/UserContent.css explains clearly enough. I can't help you with your particular version since I'm on Mac OS X and Linux, but it works for me there. From jyasskin at gmail.com Tue Mar 27 07:51:38 2012 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Mon, 26 Mar 2012 22:51:38 -0700 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: Message-ID: FWIW, I'm not sure you're the right person to drive time PEPs. You don't seem to have come into it with much knowledge of time, and it's taken several repetitions for you to take corrections into account in both this discussion and the Decimal/datetime representation PEP. On Mon, Mar 26, 2012 at 4:32 PM, Victor Stinner wrote: > Hi, > > I started to write the PEP 418 to clarify the notions of monotonic and > steady clocks. > > The PEP is a draft and everyone is invited to contribute! > > http://www.python.org/dev/peps/pep-0418/ > http://hg.python.org/peps/file/tip/pep-0418.txt > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com From ncoghlan at gmail.com Tue Mar 27 08:48:49 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Mar 2012 16:48:49 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: Message-ID: On Tue, Mar 27, 2012 at 3:51 PM, Jeffrey Yasskin wrote: > FWIW, I'm not sure you're the right person to drive time PEPs. You > don't seem to have come into it with much knowledge of time, and it's > taken several repetitions for you to take corrections into account in > both this discussion and the Decimal/datetime representation PEP. The main things required to be a PEP champion are passion and a willingness to listen to expert feedback and change course in response. If someone lacks the former, they will lose steam and their PEP will eventually be abandoned. If they don't listen to expert feedback, then their PEP will ultimately be rejected (sometimes a PEP will be rejected anyway as a poor fit for the language *despite* being responsive to feedback, but that's no slight to the PEP author). Victor has shown himself to be quite capable of handling those aspects of the PEP process, and the topics he has recently applied himself to are ones where it is worthwhile having a good answer in the standard library for Python 3.3. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Tue Mar 27 09:14:11 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Mar 2012 08:14:11 +0100 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F710870.9090602@scottdial.com> References: <4F710870.9090602@scottdial.com> Message-ID: On 27 March 2012 01:23, Scott Dial wrote: > If you want to define new clocks, then I wish you would use the same > definitions that C++0x is using. That is: > > ?system_clock = wall clock time > ?monotonic_clock = always goes forward but can be adjusted > ?steady_clock = always goes forward and cannot be adjusted > ?high_resolution_clock = steady_clock || system_clock +1. This seems like an ideal case for following prior art in designing a Python API. Paul From glyph at twistedmatrix.com Tue Mar 27 09:17:32 2012 From: glyph at twistedmatrix.com (Glyph) Date: Tue, 27 Mar 2012 03:17:32 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> Message-ID: <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> On Mar 26, 2012, at 10:26 PM, Zooko Wilcox-O'Hearn wrote: > Note that the C++ standard deprecated monotonic_clock once they > realized that there is absolutely no point in having a clock that > jumps forward but not back, and that none of the operating systems > implement such a thing -- instead they all implement a clock which > doesn't jump in either direction. This is why I don't like the C++ terminology, because it seems to me that the C++ standard makes incorrect assertions about platform behavior, and apparently they standardized it without actually checking on platform capabilities. The clock does jump forward when the system suspends. At least some existing implementations of steady_clock in C++ already have this problem, and I think they all might. I don't think they can fully fix it without kernel changes, either. On linux, see discussion of a possible CLOCK_BOOTTIME in the future. The only current way I know of to figure out how long the system has been asleep is to look at the wall clock and compare, and we've already gone over the problems with relying on the wall clock. Plus, libstdc++ gives you no portable way to get informed about system power management events, so you can't fix it even if you know about this problem, natch. Time with respect to power management state changes is something that the PEP should address fully, for each platform. On the other hand, hopefully you aren't controlling your python-based CNC laser welder from a laptop that you are closing the lid on while the beam is in operation. Not that the PEP shouldn't address it, but maybe it should just address it to say "you're on your own" and refer to a few platform-specific resources for correcting this type of discrepancy. ( , , ). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Mar 27 10:05:39 2012 From: glyph at twistedmatrix.com (Glyph) Date: Tue, 27 Mar 2012 04:05:39 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: On Mar 27, 2012, at 3:17 AM, Glyph wrote: > I don't think they can fully fix it without kernel changes I got really curious about this and went and did some research. With some really platform-specific hackery on every platform, you can mostly figure it out; completely on OS X and Windows, although (as far as I can tell) only partially on Linux and FreeBSD. I'm not sure if it's possible to make use of these facilities without a Twisted-style event-loop though. If anybody's interested, I recorded the results of my research in a comment on the Twisted ticket for this: . -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From ezio.melotti at gmail.com Tue Mar 27 10:28:32 2012 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Tue, 27 Mar 2012 02:28:32 -0600 Subject: [Python-Dev] PendingDeprecationWarning In-Reply-To: References: Message-ID: Hi, On Thu, Mar 22, 2012 at 3:13 PM, Terry Reedy wrote: > My impression is that the original reason for PendingDeprecationWarning > versus DeprecationWarning was to be off by default until the last release > before removal. But having DeprecationWarnings on by default was found to be > too obnoxious and it too is off by default. So do we still need > PendingDeprecationWarnings? My impression is that it is mostly not used, as > it is a nuisance to remember to change from one to the other. The > deprecation message can always indicate the planned removal time. I searched > the Developer's Guide for both deprecation and DeprecationWarning and found > nothing. See http://mail.python.org/pipermail/python-dev/2011-October/114199.html Best Regards, Ezio Melotti From regebro at gmail.com Tue Mar 27 11:03:07 2012 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 27 Mar 2012 11:03:07 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: Reading this discussion, my conclusion is that not only us are confused, but everyone is. I think therefore, that the way forward is to only expose underlying API functions, and pretty much have no intelligence at all. At a higher level, we have two different "desires" here. You may want a monotonic clock, or you may not care. You may want high resolution, or you might not care. Which one is more important is something only you know. Therefore, we must have, at the minimum, a function that returns the highest resolution monotonic clock possible, as well as a function that returns the highest resolution system/wall clock possible. We also need ways to figure out what the resolution is of these clocks. In addition to that, you may have the requirement that the monotonic clock also should not be able to jump forward, but if I understand things correctly, most current OS's will not guarantee this. You may also have the requirement that the clock not only does not jump forward, but that it doesn't go faster or slower. Some clock implementations will speed up or slow down the monotonic clock, without jumps, to sync up with the wall clock. It seems only Unix provides a monotonic clock (CLOCK_MONOTONIC_RAW) that does not get adjusted at all. Now between all these requirements, only you know which one is more important? Do you primarily want a raw monotonic clock, and secondarily high resolution, or is the resolution more important than it being monotonic? (Because if you need a high resolution, you are usually measuring small timeframes, and the clock is more unlikely to be adjusted, for example). Since there is no obvious "A is better than B that is better than C" we first of all have to expose the underlying API's somehow, to allow people to make their own decisions. Secondly, since apparently not only python-dev, but many others as well, are a bit confused on this complex issue, I'm not sure we can provide any high-level functions that makes a best choice. As such the proposed time.monotonic() to get the monotonic clock on the various systems makes a lot of sense to me. It should get the highest resolution available on the system. Get GetTickCount64() of available on Windows, else GetTickCount(). The function could have a raw=False parameter to select between clock_gettime(CLOCK_MONOTONIC) and clock_gettime(CLOCK_MONOTONIC_RAW) on Unix, and it would get mach_absolute_time() on OS X. If no monotonic clock is available, it should raise an error.The same if you pass in raw=True and there is no monotonic clock that has no adjustments available. In the same vein, time.time() should provide the highest resolution system clock/wall clock available. We also need functions or attributes to get the resolution of these clocks. But a time.steady() that tries to get a "best case" doesn't make sense at this time, as apparently nobody knows what a best case is, or what it should be called, except that it should apparently not be called steady(). Since monotonic() raises an error if there is no monotonic clock available, implementing your own fallback is trivial in any case. //Lennart From linkerlv at gmail.com Tue Mar 27 12:03:12 2012 From: linkerlv at gmail.com (Linker) Date: Tue, 27 Mar 2012 18:03:12 +0800 Subject: [Python-Dev] how to uninstall python after 'make build' Message-ID: RT thx. linker -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Mar 27 15:23:00 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Mar 2012 23:23:00 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: On Tue, Mar 27, 2012 at 7:03 PM, Lennart Regebro wrote: > But a time.steady() that tries to get a "best case" doesn't make sense > at this time, as apparently nobody knows what a best case is, or what > it should be called, except that it should apparently not be called > steady(). Since monotonic() raises an error if there is no monotonic > clock available, implementing your own fallback is trivial in any > case. +1 from me to Lennart's suggestion of mostly just exposing time.monotonic() without trying to get too clever. Applications that need a truly precise time source should *never* be reading it from the host OS (one fairly good solution can be to read your time directly from an attached GPS device). However, I think Victor's right to point out that the *standard library* needs to have a fallback to maintain backwards compatibility if time.monotonic() isn't available, and it seems silly to implement the same fallback logic in every module where we manipulate timeouts. As I concur with others that time.steady() is a thoroughly misleading name for this concept, I suggest we encapsulate the "time.monotic if available, time.time otherwise" handling as a "time.try_monotic()" function. That's simple clear and explicit: try_monotic() tries to use the monotic clock if it can, but falls back to time.time() rather than failing entirely if no monotonic clock is available. This is essential for backwards compatibility when migrating any current use of time.time() over to time.monotic(). Yes the monotonic clock is *better* for many use cases (including timeouts), but you'll usually be OK with the non-monotonic clock, too (particularly if that's what you were using anyway in earlier versions). After all, we've survived this long using time.time() for our timeout calculations, and bugs associated with clock adjustments are a rather rare occurrence. Third party libraries that need to support earlier Python versions can then implementation their own fallback logic (since they couldn't rely on time.try_monotonic being available either). The 3.3 time module would then be left with three interfaces: time.time() # Highest resolution timer available time.monotonic() # I'm not yet convinced we need the "raw" parameter but don't much mind either way time.try_monotonic() # Monotonic is preferred, but non-monotonic presents a tolerable risk Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ethan at stoneleaf.us Tue Mar 27 18:18:23 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 27 Mar 2012 09:18:23 -0700 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: <4F71E84F.8080309@stoneleaf.us> Nick Coghlan wrote: > The 3.3 time module would then be left with three interfaces: > > time.time() # Highest resolution timer available > time.monotonic() # I'm not yet convinced we need the "raw" parameter > but don't much mind either way > time.try_monotonic() # Monotonic is preferred, but non-monotonic > presents a tolerable risk +1 ~Ethan~ From yselivanov.ml at gmail.com Tue Mar 27 19:10:40 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 27 Mar 2012 13:10:40 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: On 2012-03-27, at 9:23 AM, Nick Coghlan wrote: > time.try_monotonic() # Monotonic is preferred, but non-monotonic > presents a tolerable risk This function seems unnecessary. It's easy to implement it when required in your application, hence I don't think it is worth adding to the stdlib. - Yury From victor.stinner at gmail.com Tue Mar 27 19:34:51 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 19:34:51 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: Message-ID: > I started to write the PEP 418 to clarify the notions of monotonic and > steady clocks. I replaced time.steady() by time.try_monotonic(). I misunderstood "may not" in the C++ doc: I understood it as "it may be adjusted by NTP", whereas it means "it cannot be adjusted". Sorry for the confusion. I added a time.hires() clock because time.monotonic() and time.try_monotonic() are not the best clocks for profiling or benchmarking. For example, on Windows, time.hires() uses QueryPerformanceCounter() whereas time.monotonic() and time.try_monotonic() uses GetTickCount[64](). I added the pseudo-code of each function. I hope that it is easier to understand than a long text. Victor From victor.stinner at gmail.com Tue Mar 27 19:45:24 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 19:45:24 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F710870.9090602@scottdial.com> References: <4F710870.9090602@scottdial.com> Message-ID: > What is the utility of "strict=True"? If I wanted that mode of > operation, then why would I not just try to use "time.monotonic()" > directly? I mentioned the strict=True API in the PEP just to list all propositions, but the PEP only proposes time.monotonic() and time.try_monotonic(), no the flags API. > At worst, it generates an "AttributeError" (although that is not clear from your PEP). I tried to mention when a function is always available or not always available. Is it better in the last version of the PEP? > ?system_clock = wall clock time > ?monotonic_clock = always goes forward but can be adjusted > ?steady_clock = always goes forward and cannot be adjusted > ?high_resolution_clock = steady_clock || system_clock I tried to follow these names in the PEP. I don't propose steady_clock because I don't see exactly which clocks would be used to implement it, nor if we need to provide monotonic *and* steady clocks. What do you think? > Straying from that is only going to create confusion. Besides that, the > one use case for "time.steady()" that you give (benchmarking) is better > served by a clock that follows the C++0x definition. I added a time.hires() clock to the PEP for the benchmarking/profiling use case. This function is not always available and so a program has to fallback manually to another clock. I don't think that it is an issue: Python programs already have to choose between time.clock() and time.time() depending on the OS (e.g. timeit module and pybench program). > As well, certain > kinds of scheduling/timeouts would be better implemented with the C++0x > definition for "steady" rather than the "monotonic" one and vice-versa. Sorry, I don't understand. Which kind of scheduling/timeouts? The PEP is still a draft (work-in-progress). Victor From victor.stinner at gmail.com Tue Mar 27 19:50:35 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 19:50:35 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F7135AA.9000203@scottdial.com> References: <4F710870.9090602@scottdial.com> <4F7135AA.9000203@scottdial.com> Message-ID: > steady_clock: > > ?mac = mach_absolute_time > ?posix = clock_gettime(CLOCK_MONOTONIC) > ?win = QueryPerformanceCounter I read that QueryPerformanceCounter is no so monotonic, and GetTickCount is preferred. Is it true? > high_resolution_clock: > > ?* = { steady_clock, if available > ? ? ? ?system_clock, otherwise } On Windows, I propose to use QueryPerformanceCounter() for time.hires() and GetTickCount() for time.monotonic(). See the PEP for other OSes. Victor From ethan at stoneleaf.us Tue Mar 27 19:35:06 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 27 Mar 2012 10:35:06 -0700 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: <4F71FA4A.6070005@stoneleaf.us> Yury Selivanov wrote: > On 2012-03-27, at 9:23 AM, Nick Coghlan wrote: > >> time.try_monotonic() # Monotonic is preferred, but non-monotonic >> presents a tolerable risk > > This function seems unnecessary. It's easy to implement it when > required in your application, hence I don't think it is worth > adding to the stdlib. If I understood Nick correctly, time.try_monotonic() is /for/ the stdlib. If others want to make use of it, fine. If others want to make their own fallback mechanism, also fine. ~Ethan~ From victor.stinner at gmail.com Tue Mar 27 19:51:41 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 19:51:41 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: Message-ID: 2012/3/27 Jeffrey Yasskin : > FWIW, I'm not sure you're the right person to drive time PEPs. I don't want to drive the PEP. Anyone is invited to contribute, as I wrote in my first message. I'm completing/rewriting the PEP with all comments. Victor From victor.stinner at gmail.com Tue Mar 27 19:55:16 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 19:55:16 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: > The clock does?jump forward when the system suspends. ?At least some > existing implementations of steady_clock in C++ already have this problem, > and I think they all might. > .... > Time with respect to power management state changes is something that the > PEP should address fully, for each platform. I don't think that Python should workaround OS issues, but document them correctly. I started with this sentence for time.monotonic(): "The monotonic clock may stop while the system is suspended." I don't know exactly how clocks behave with system suspend. Tell me if you have more information. > ?( , > , > ). I will read these links and maybe add them to the PEP. Victor From victor.stinner at gmail.com Tue Mar 27 19:59:12 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 19:59:12 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: > That's simple clear and explicit: try_monotic() tries to use the > monotic clock if it can, but falls back to time.time() rather than > failing entirely if no monotonic clock is available. I renamed time.steady() to time.try_monotonic() in the PEP. It's a temporary name until we decide what to do with this function. I also changed it to fallback to time.hires() if time.monotonic() is not available or failed. Victor From fuzzyman at voidspace.org.uk Tue Mar 27 20:00:31 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 27 Mar 2012 19:00:31 +0100 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> Message-ID: <4F72003F.6000606@voidspace.org.uk> On 27/03/2012 18:45, Victor Stinner wrote: > [snip...] >> Straying from that is only going to create confusion. Besides that, the >> one use case for "time.steady()" that you give (benchmarking) is better >> served by a clock that follows the C++0x definition. > I added a time.hires() clock to the PEP for the benchmarking/profiling > use case. This function is not always available and so a program has > to fallback manually to another clock. I don't think that it is an > issue: Python programs already have to choose between time.clock() and > time.time() depending on the OS (e.g. timeit module and pybench > program). It is this always-having-to-manually-fallback-depending-on-os that I was hoping your new functionality would avoid. Is time.try_monotonic() suitable for this usecase? Michael > >> As well, certain >> kinds of scheduling/timeouts would be better implemented with the C++0x >> definition for "steady" rather than the "monotonic" one and vice-versa. > Sorry, I don't understand. Which kind of scheduling/timeouts? > > The PEP is still a draft (work-in-progress). > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From anacrolix at gmail.com Tue Mar 27 20:18:29 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 28 Mar 2012 02:18:29 +0800 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: > I renamed time.steady() to time.try_monotonic() in the PEP. It's a > temporary name until we decide what to do with this function. How about get rid of it? Also monotonic should either not exist if it's not available, or always guarantee a (artificially) monotonic value. Finding out that something is already known to not work shouldn't require a call and a faked OSError. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Mar 27 21:45:58 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 27 Mar 2012 15:45:58 -0400 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> Message-ID: On Mon, Mar 26, 2012 at 11:23 PM, Stephen J. Turnbull wrote: > On Tue, Mar 27, 2012 at 1:35 AM, PJ Eby wrote: > > On Sun, Mar 25, 2012 at 2:56 AM, Stephen J. Turnbull > > > wrote: > >> > >> But since he's arguing the > >> other end in the directory layout thread (where he says there are many > >> special ways to invoke Python so that having different layouts on > >> different platforms is easy to work around), I can't give much weight > >> to his preference here. > > > > > > You're misconstruing my argument there: I said, rather, that the One > Obvious > > Way to deploy a Python application is to dump everything in one > directory, > > as that is the one way that Python has supported for at least 15 years > now. > > It's not at all obvious on any of the open source platforms (Cygwin, > Debian, > Gentoo, and MacPorts) that I use. In all cases, both several python > binaries > and installed libraries end up in standard places in the distro-maintained > hierarchies, and it is not hard to confuse the distro-installed Pythons. > Really? I've been doing "dump the app in a directory" since 1998 or so on various *nix platforms. And when distutils came along, I set up a user-specific cfg to install in the same directory. ISTR a 5-line pydistutils.cfg is sufficient to make everything go into to a particular directory, for packages using distutils for installation. Being confident that one knows enough to set up a single directory correctly > in the face of some of the unlovely things that packages may do requires > more knowledge of how Python's import etc works than I can boast: > virtualenv is a godsend. By analogy, yes, I think it makes sense to ask > you > to learn a bit about CSS and add a single line like "body { width: 65em; > }" to > your local config. That's one reason why CSS is designed to cascade. > That line won't work - it'll make the entire page that width, instead of just text paragraphs. (Plus, it should only be the *maximum* width - i.e. max-width.) Unfortunately, there's no way to identify the correct selector to use on all sites to select just the right elements to set max-width on - not all text is in "p", sometimes preformatted text is in a p with styles setting the formatting to be preformatted. (In other words, I actually do know a little about CSS - enough to know your idea won't actually work without tweaking it for different sites. I have enough Greasemonkey scripts as it is, never mind custom CSS.) > The comparison to CSS is also lost on me here; creating user-specific CSS > is > > more aptly comparable telling people to write their own virtualenv > > implementations from scratch, and resizing the browser window is more > akin > > to telling people to create a virtualenv every time they *run* the > > application, rather than just once when installing it. > > Huh, if you say so -- I didn't realize that virtualenv did so little that > it could be written in one line. Around 3-5 lines for dumping everything into a single directory. If you need multiple such directories at any one time, you can alternately pass --install-lib and --install-scripts to "setup.py install" when you install things. Or you can use easy_install and just specify the -d (--install-dir) option. Or, you could use the PYTHONHOME solution I described here in 2005: http://svn.python.org/view/sandbox/trunk/setuptools/EasyInstall.txt?r1=41220&r2=41221 Ian Bicking turned those instructions into a script, which he then gradually evolved into what we now know as virtualenv. Before that happened, though, I was deluged with complaints from people who were using "dump it in a directory somewhere" on *nix platforms and didn't want to have anything to do with those danged newfangled virtual whatchacallits. ;-) I mention this for context: my generic perception of virtualenv is that it's a fancy version of PYTHONHOME for people who can't install to site-packages and for some reason don't just dump their files in a PYTHONPATH directory like all those complaining people obviously did. ;-) All I know (and care) is > that it promises to do all that stuff for me, and without affecting the > general public (ie, the distro-provided Python apps). > > And that's why I think the width of pages containing flowed text > should be left up to the user to configure. > Your analogy is backwards: virtualenv is a generic, does-it-all-for-you, no need to touch it solution. User CSS and window sizes have to specified per-site. (Note that it doesn't suffice to use a small window to get optimal wrap width: you have to resize to allow for navigation bars, multiple columns, etc.) I think we should just agree to disagree; there's virtually no way I'm going to be convinced on either of these points. (I do, however, remain open to learning something new about virtualenv, if it actually does something besides make it possible for you to deal with ill-behaved setup scripts and installation tools that won't just let you point them at a single directory and have done with it.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Mar 27 23:02:08 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 23:02:08 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: <4F722AD0.7010007@gmail.com> On 27/03/2012 20:18, Matt Joiner wrote: > Also monotonic should either not exist if it's not available, or always > guarantee a (artificially) monotonic value. What do you mean by "(artificially) monotonic value"? Should Python workaround OS bugs by always returning the maximum value of the clock? > Finding out that something > is already known to not work shouldn't require a call and a faked OSError. What do you mean? time.monotonic() is not implemented if the OS doesn't provide any monotonic clock (e.g. if clock_gettime(CLOCK_MONOTONIC) is missing on UNIX). OSError is only raised when the OS returns an error. There is no such "faked OSError". Victor From martin at v.loewis.de Tue Mar 27 23:11:07 2012 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 27 Mar 2012 23:11:07 +0200 Subject: [Python-Dev] Bug tracker outage Message-ID: <4F722CEB.1040102@v.loewis.de> Upfront hosting (Izak Burger) is going to do a Debian upgrade of the bug tracker machine "soon" (likely tomorrow). This may cause some outage, since there is a lot of custom stuff on the machine which may break with newer (Python) versions. I'll notify here when the upgrade is complete. Regards, Martin From victor.stinner at gmail.com Tue Mar 27 23:42:51 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Mar 2012 23:42:51 +0200 Subject: [Python-Dev] [Python-checkins] peps: Approve PEP 411. In-Reply-To: References: Message-ID: 2012/3/27 guido.van.rossum : > http://hg.python.org/peps/rev/b9f43fe69691 > changeset: ? 4152:b9f43fe69691 > user: ? ? ? ?Guido van Rossum > date: ? ? ? ?Mon Mar 26 20:35:14 2012 -0700 > summary: > ?Approve PEP 411. > (...) > -Status: Draft > +Status: Approved The pep0 module doesn't accept the "Approved" status. I suppose that you mean "Accepted" and so changed the status. If not, please revert my change and fix the pep0 module. Victor From ncoghlan at gmail.com Wed Mar 28 01:45:32 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Mar 2012 09:45:32 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <6D240213-9A35-4540-9400-1DE554C39629@twistedmatrix.com> Message-ID: Matt, we need the fallback behaviour in the stdlib so we can gracefully degrade the stdlib's *own* timeout handling back to the 3.2 status quo when there is no monotic clock available. It is *not* acceptable for the Python 3.3 stdlib to only work on platforms that provide a monotonic clock. Since duplicating that logic in every module that handles timeouts would be silly, it makes sense to provide an obvious way to do it in the time module. -- Sent from my phone, thus the relative brevity :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Mar 28 02:36:18 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Mar 2012 02:36:18 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F72003F.6000606@voidspace.org.uk> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> Message-ID: <4F725D02.1080706@gmail.com> Scott wrote: << The Boost implementation can be summarized as: system_clock: mac = gettimeofday posix = clock_gettime(CLOCK_REALTIME) win = GetSystemTimeAsFileTime steady_clock: mac = mach_absolute_time posix = clock_gettime(CLOCK_MONOTONIC) win = QueryPerformanceCounter high_resolution_clock: * = { steady_clock, if available system_clock, otherwise } >> I read again the doc of the QElapsedTimer class of the Qt library. So Qt and Boost agree to say that QueryPerformanceCounter() *is* monotonic. I was confused because of a bug found in 2006 in Windows XP on multicore processors. QueryPerformanceCounter() gave a different value on each core. The bug was fixed in Windows and is known as KB896256 (I already added a link to the bug in the PEP). >> I added a time.hires() clock to the PEP for the benchmarking/profiling >> use case (...) > > It is this always-having-to-manually-fallback-depending-on-os that I was > hoping your new functionality would avoid. Is time.try_monotonic() > suitable for this usecase? If QueryPerformanceCounter() is monotonic, the API can be simplified to: * time.time() = system clock * time.monotonic() = monotonic clock * time.hires() = monotonic clock or fallback to system clock time.hires() definition is exactly what I was trying to implement with "time.steady(strict=True)" / "time.try_monotonic()". -- Scott> monotonic_clock = always goes forward but can be adjusted Scott> steady_clock = always goes forward and cannot be adjusted I don't know if the monotonic clock should be called time.monotonic() or time.steady(). The clock speed can be adjusted by NTP, at least on Linux < 2.6.28. I don't know if other clocks used by my time.monotonic() proposition can be adjusted or not. If I understand correctly, time.steady() cannot be implemented using CLOCK_MONOTONIC on Linux because CLOCK_MONOTONIC can be adjusted? Does it really matter if a monotonic speed is adjusted? Victor From ethan at stoneleaf.us Wed Mar 28 03:02:48 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 27 Mar 2012 18:02:48 -0700 Subject: [Python-Dev] OT: single directory development [was Re: Playing with a new theme for the docs] In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> Message-ID: <4F726338.2060001@stoneleaf.us> PJ Eby wrote: > Really? I've been doing "dump the app in a directory" since 1998 or so > on various *nix platforms. And when distutils came along, I set up a > user-specific cfg to install in the same directory. ISTR a 5-line > pydistutils.cfg is sufficient to make everything go into to a particular > directory, for packages using distutils for installation. Perhaps somebody could clue me in on the best way to handle this scenario: I develop in a single directory: c:\source\loom\ loom.py test_loom.py because test_loom could at some point be executed by somebody besides me, while living in site-packages, I have test_loom.py create its needed files, including dynamic test scripts, in a temp directory. While this works fine for site-packages, it doesn't work at all while testing as the test script, being somewhere else, won't be able to load my test copy of loom. I know I have at least two choices: - go with a virtualenv and have my development code be in the virtualenv site-packages - insert the current path into sys.path before calling out to the dynamic scripts, but only if the current path is not site-packages Suggestions? ~Ethan~ From anacrolix at gmail.com Wed Mar 28 04:41:08 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 28 Mar 2012 10:41:08 +0800 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F725D02.1080706@gmail.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On Mar 28, 2012 8:38 AM, "Victor Stinner" wrote: > > Scott wrote: > > << The Boost implementation can be summarized as: > > system_clock: > > mac = gettimeofday > posix = clock_gettime(CLOCK_REALTIME) > win = GetSystemTimeAsFileTime > > steady_clock: > > mac = mach_absolute_time > posix = clock_gettime(CLOCK_MONOTONIC) > win = QueryPerformanceCounter > > high_resolution_clock: > > * = { steady_clock, if available > system_clock, otherwise } >> > > I read again the doc of the QElapsedTimer class of the Qt library. So Qt and Boost agree to say that QueryPerformanceCounter() *is* monotonic. > > I was confused because of a bug found in 2006 in Windows XP on multicore processors. QueryPerformanceCounter() gave a different value on each core. The bug was fixed in Windows and is known as KB896256 (I already added a link to the bug in the PEP). > >>> I added a time.hires() clock to the PEP for the benchmarking/profiling >>> use case (...) >> >> >> It is this always-having-to-manually-fallback-depending-on-os that I was >> hoping your new functionality would avoid. Is time.try_monotonic() >> suitable for this usecase? > > > If QueryPerformanceCounter() is monotonic, the API can be simplified to: > > * time.time() = system clock > * time.monotonic() = monotonic clock > * time.hires() = monotonic clock or fallback to system clock > > time.hires() definition is exactly what I was trying to implement with "time.steady(strict=True)" / "time.try_monotonic()". > > -- > > Scott> monotonic_clock = always goes forward but can be adjusted > Scott> steady_clock = always goes forward and cannot be adjusted > > I don't know if the monotonic clock should be called time.monotonic() or time.steady(). The clock speed can be adjusted by NTP, at least on Linux < 2.6.28. Monotonic. It's still monotonic if it is adjusted forward, and that's okay. > > I don't know if other clocks used by my time.monotonic() proposition can be adjusted or not. > > If I understand correctly, time.steady() cannot be implemented using CLOCK_MONOTONIC on Linux because CLOCK_MONOTONIC can be adjusted? > > Does it really matter if a monotonic speed is adjusted? > > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed Mar 28 06:26:38 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 28 Mar 2012 13:26:38 +0900 Subject: [Python-Dev] Playing with a new theme for the docs In-Reply-To: References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> Message-ID: On Wed, Mar 28, 2012 at 4:45 AM, PJ Eby wrote: > [ body { width: 65em; } ] won't work - it'll make the entire page > that width, instead of just text paragraphs. True (I realized that might be bad in many cases later -- should have tested first rather than posting something random), but despite your argument, "p { max-width: 40em; }" will be good enough to handle pages where the designer leaves text width up to the user. Pages (or parts thereof) where the designer fubars the format for you are not my problem, they're *your* problem. "Be careful what you ask for, because you just might get it." Also, this is UI, in an environment where poor UI is easily worked around with a flick of your mouse. A improvement in 90% of the cases is a 90% improvement -- there won't be any fatal problems in the 10% where designers choose a max-width that's 200% of your personal max-width or whatever. > Your analogy is backwards: virtualenv is a generic, > does-it-all-for-you, no need to touch it solution. If you want to phrase it as an analogy, mine is virtualenv :: do-it-*for*-you :: site-specified max-width but dump-it-in-a-directory :: DIY :: user-specified CSS. The point of the analogy is that you're being inconsistent by using dump-it-in-a-directory yourself but recommending site-specified max-width for the Python docs. It's true I'm being similarly inconsistent in a sense (using virtualenv myself but recommending user CSS for Python docs). However, in this discussion, there are more important things than consistency. I think there are an awful lot of people who need reliable deployment, consistent with their development environments, so it makes sense to have Ian package it up for us as "virtualenv", and for it to be somewhat inflexible in its rules for doing so. OTOH, AFAICS those who use maximized windows for everything are a relatively small minority who will be well-served by a simple workaround, and there are gains to having the flexibility for the rest of us. The big problem from my point of view with the user CSS "solution" to the maximized-window problem is that common browsers don't make this easy to do. Cf. Terry Reedy's post asking how to specify user CSS for Firefox (where it's actually easy enough to do once you know how, but evidently not very discoverable). However, if you're in a sufficiently small minority (as I believe you are), it makes sense for Georg to (regretfully ) ask you to use personal CSS to tell your browser about your preference. >?User CSS and window sizes have to specified per-site. They do? But *you* don't, you just maximize your window. So only a few sites will need specification in any case, those whose max-width exceeds tolerable bounds for you. And a personal max-width will affect only unbounded pages unless you use "! important". The point of user CSS is not to get optimality, which is a content-dependent problem for negotiation between user and designer, and sometimes one side or the other takes absolute priority. It's to ensure that users with special needs (very nearsighted users, users who prefer to work always in a maximized browser window) don't get screwed by extreme designs. >?(Note that it doesn't suffice to use a small > window to get optimal wrap width: I don't believe in optimal wrap width, and as far as I know, neither do the 1% of designers. I don't even *have* a personal optimal wrap width, although max height is almost always close enough to optimal. But I sometimes maximize the width of my browser window to get even more of the "structure" of text viewable in it, or reduce it to make word-for-word reading more efficient. Again, the problem here is not "suboptimal". AFAICS, it's preventing a few people who have evolved personal workflows adapted to a common design pattern that's not appropriate for documentation (IMO YMMV) from getting *pessimal* results. I believe that you can get what you need with user CSS in the case of no max-width (let's not forget that you and R. David Murray may prefer different values!), while many use-cases would want no max-width. But "p { max-width: none ! important; }" would not work well for us, since it would override all designers who set max-width. > I think we should just agree to disagree; there's virtually > no way I'm going to be convinced on either of these points. Hey, I'm an economist: de gustibus non est disputandum. Convincing you is not my goal; I want to convince Georg! *Policy* needs to be for the greatest good of the greatest number, and Georg IMO should set max-width, or not, as that makes reading the documentation more effective for the most people. I prefer "not", assuming it doesn't completely trash its usability for you and David (and assuming you're in as small a minority as I believe you to be). From ncoghlan at gmail.com Wed Mar 28 06:45:33 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Mar 2012 14:45:33 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F725D02.1080706@gmail.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On Wed, Mar 28, 2012 at 10:36 AM, Victor Stinner wrote: > If QueryPerformanceCounter() is monotonic, the API can be simplified to: > > ?* time.time() = system clock > ?* time.monotonic() = monotonic clock > ?* time.hires() = monotonic clock or fallback to system clock > > time.hires() definition is exactly what I was trying to implement with > "time.steady(strict=True)" / "time.try_monotonic()". Please don't call the fallback version "hires" as it suggests it may be higher resolution than time.time() and that's completely the wrong idea. If we're simplifying the idea to only promising a monotonic clock (i.e. will never go backwards within a given process, but may produce the same value for an indefinite period, and may jump forwards by arbitrarily large amounts), then we're back to being able to enforce monotonicity even if the underlying clock jumps backwards due to system clock adjustments. Specifically: time.time() = system clock time._monotonic() = system level monotonic clock (if it exists) time.monotonic() = clock based on either time._monotonic() (if available) or time.time() (if not) that enforces monotonicity of returned values. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From pje at telecommunity.com Wed Mar 28 07:18:33 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 28 Mar 2012 01:18:33 -0400 Subject: [Python-Dev] OT: single directory development [was Re: Playing with a new theme for the docs] In-Reply-To: <4F726338.2060001@stoneleaf.us> References: <4F6935E1.2030309@nedbatchelder.com> <4F69E337.7010501@nedbatchelder.com> <4F6A560A.6050207@canterbury.ac.nz> <5260192D-7EF1-4021-AB50-FD2021566CFF@twistedmatrix.com> <4F6D2061.3080804@canterbury.ac.nz> <4F6D5C52.1030802@canterbury.ac.nz> <87y5qpikah.fsf@benfinney.id.au> <4F726338.2060001@stoneleaf.us> Message-ID: On Tue, Mar 27, 2012 at 9:02 PM, Ethan Furman wrote: > PJ Eby wrote: > >> Really? I've been doing "dump the app in a directory" since 1998 or so >> on various *nix platforms. And when distutils came along, I set up a >> user-specific cfg to install in the same directory. ISTR a 5-line >> pydistutils.cfg is sufficient to make everything go into to a particular >> directory, for packages using distutils for installation. >> > > Perhaps somebody could clue me in on the best way to handle this scenario: > > I develop in a single directory: > > c:\source\loom\ > loom.py > test_loom.py > > because test_loom could at some point be executed by somebody besides me, > while living in site-packages, I have test_loom.py create its needed files, > including dynamic test scripts, in a temp directory. While this works fine > for site-packages, it doesn't work at all while testing as the test script, > being somewhere else, won't be able to load my test copy of loom. > > I know I have at least two choices: > - go with a virtualenv and have my development code be in the > virtualenv site-packages > - insert the current path into sys.path before calling out to > the dynamic scripts, but only if the current path is not > site-packages > > Suggestions? > At first I didn't understand the question, because I was misled by your directory layout sketch -- AFAICT it's completely irrelevant to the real problem, which is simply making sure that the directory containing loom.__file__ is on sys.path. I'm somewhat hard-pressed to see, "embed the virtualenv tool in my test script" as superior to either "Copy the modules I want to the temp directory alongside the modules" or "setup sys.path when running the scripts" (e.g. by altering the PYTHONPATH in the child environment). (I'm not clear on why you want to skip the path alteration in the site-packages case - is there something else in site-packages you don't want having top import priority? And if not, why not?) All that being said, I can see why under certain circumstances, a virtualenv might be an optimal tool to reach for; it's just not the *first* thing I'd reach for if a sys.path[0] assignment or environment variable setting would suffice to get the needed module(s) on the path. > ~Ethan~ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott+python-dev at scottdial.com Wed Mar 28 08:40:24 2012 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 28 Mar 2012 02:40:24 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F725D02.1080706@gmail.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: <4F72B258.10306@scottdial.com> On 3/27/2012 8:36 PM, Victor Stinner wrote: > Scott wrote: > Scott> monotonic_clock = always goes forward but can be adjusted > Scott> steady_clock = always goes forward and cannot be adjusted > > I don't know if the monotonic clock should be called time.monotonic() or > time.steady(). The clock speed can be adjusted by NTP, at least on Linux > < 2.6.28. > > I don't know if other clocks used by my time.monotonic() proposition can > be adjusted or not. > > If I understand correctly, time.steady() cannot be implemented using > CLOCK_MONOTONIC on Linux because CLOCK_MONOTONIC can be adjusted? > > Does it really matter if a monotonic speed is adjusted? You are right that CLOCK_MONOTONIC can be adjusted, so the Boost implementation is wrong. I'm not sure that CLOCK_MONOTONIC_RAW is right either due to suspend -- there doesn't appear to be a POSIX or Linux clock that is defined that meets the "steady" definition. I am not familiar enough with Windows or Mac to know for certain whether the Boost implementation has the correct behaviors either. With that in mind, it's certainly better that we just provide time.monotonic() for now. If platform support becomes available, then we can expose that as it becomes available in the future. In other words, at this time, I don't think "time.steady()" can be implemented faithfully for any platform so lets just not have it at all. In that case, I don't think time.try_monotonic() is really needed because we can emulate "time.monotonic()" in software if the platform is deficient. I can't imagine a scenario where you would ask for a monotonic clock and would rather have an error than have Python fill in the gap with an emulation. -- Scott Dial scott at scottdial.com From g.brandl at gmx.net Wed Mar 28 08:41:13 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 28 Mar 2012 08:41:13 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On 28.03.2012 06:45, Nick Coghlan wrote: > On Wed, Mar 28, 2012 at 10:36 AM, Victor Stinner > wrote: >> If QueryPerformanceCounter() is monotonic, the API can be simplified to: >> >> * time.time() = system clock >> * time.monotonic() = monotonic clock >> * time.hires() = monotonic clock or fallback to system clock >> >> time.hires() definition is exactly what I was trying to implement with >> "time.steady(strict=True)" / "time.try_monotonic()". > > Please don't call the fallback version "hires" as it suggests it may > be higher resolution than time.time() and that's completely the wrong > idea. It's also a completely ugly name, since it's quite hard to figure out what it is supposed to stand for in the first place. Georg From victor.stinner at gmail.com Wed Mar 28 10:40:22 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Mar 2012 10:40:22 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: >> ?* time.time() = system clock >> ?* time.monotonic() = monotonic clock >> ?* time.hires() = monotonic clock or fallback to system clock >> >> time.hires() definition is exactly what I was trying to implement with >> "time.steady(strict=True)" / "time.try_monotonic()". > > Please don't call the fallback version "hires" as it suggests it may > be higher resolution than time.time() and that's completely the wrong > idea. Why would it be a wrong idea? On Windows, time.monotonic() frequency is at least 1 MHz (can be GHz if it uses your CPU TSC) whereas time.time() is only updated each millisecond at the best case (each 15 ms by default if I remember correctly). On UNIX, CLOCK_MONOTONIC has the same theorical resolution than CLOCK_REALTIME (1 nanosecond thanks to the timespec structure) and I expect the same accuracy. On Mac, I don't know if mach_absolute_time() is more or as accurate than time.time(). time.hires() uses time.monotonic() if available, so if time.monotonic() has an higher resolution than time.time(), time.hires() can also be called a high-resolution clock. In practice, time.monotonic() is available on all modern platforms. > If we're simplifying the idea to only promising a monotonic > clock (i.e. will never go backwards within a given process, but may > produce the same value for an indefinite period, and may jump forwards > by arbitrarily large amounts), I don't know any monotonic clock jumping "forwards by arbitrarily large amounts". Linux can change CLOCK_MONOTONIC speed, but NTP doesn't "jump". > then we're back to being able to enforce monotonicity even > if the underlying clock jumps backwards due to system clock > adjustments. Do you know a monotonic clock that goes backward? If yes, Python might workaround the clock bug directly in time.monotonic(). But I would prefer to *not* workaround OS bugs. Victor From victor.stinner at gmail.com Wed Mar 28 10:48:04 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Mar 2012 10:48:04 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F72B258.10306@scottdial.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> Message-ID: >> Scott> monotonic_clock = always goes forward but can be adjusted >> Scott> steady_clock = always goes forward and cannot be adjusted >> >> I don't know if the monotonic clock should be called time.monotonic() or >> time.steady(). The clock speed can be adjusted by NTP, at least on Linux >> < 2.6.28. (...) > > You are right that CLOCK_MONOTONIC can be adjusted, so the Boost > implementation is wrong. I'm not sure that CLOCK_MONOTONIC_RAW is right > either due to suspend -- there doesn't appear to be a POSIX or Linux > clock that is defined that meets the "steady" definition. The term "adjusted" should be clarified. A clock can be adjusted by setting its counter (e.g. setting the system date and time) or by changing temporary its frequency (to go faster or slower). Linux only adjusts CLOCK_MONOTONIC frequency but the clock is monotonic because it always goes forward. The monotonic property can be described as: t1=time.monotonic() t2=time.monotonic() assert t2 >= t1 > In that case, I don't think time.try_monotonic() is really needed > because we can emulate "time.monotonic()" in software if the platform is > deficient. time.hires() is needed when the OS doesn't provide any monotonic clock and because time.monotonic() must not use the system clock (which can jump backward). As I wrote, I don't think that Python should workaround OS bugs. If the OS monotonic clock is not monotonic, the OS should be fixed. > I can't imagine a scenario where you would ask for a > monotonic clock and would rather have an error than have Python fill in > the gap with an emulation. Sorry, I don't understand what you mean with "fill in the gap with an emulation". You would like to implement a monotonic clock based on the system clock? Victor From scott+python-dev at scottdial.com Wed Mar 28 11:37:34 2012 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 28 Mar 2012 05:37:34 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> Message-ID: <4F72DBDE.6040003@scottdial.com> On 3/28/2012 4:48 AM, Victor Stinner wrote: >>> Scott> monotonic_clock = always goes forward but can be adjusted >>> Scott> steady_clock = always goes forward and cannot be adjusted >>> >>> I don't know if the monotonic clock should be called time.monotonic() or >>> time.steady(). The clock speed can be adjusted by NTP, at least on Linux >>> < 2.6.28. (...) >> >> You are right that CLOCK_MONOTONIC can be adjusted, so the Boost >> implementation is wrong. I'm not sure that CLOCK_MONOTONIC_RAW is right >> either due to suspend -- there doesn't appear to be a POSIX or Linux >> clock that is defined that meets the "steady" definition. > > The term "adjusted" should be clarified. A clock can be adjusted by > setting its counter (e.g. setting the system date and time) or by > changing temporary its frequency (to go faster or slower). Linux only > adjusts CLOCK_MONOTONIC frequency but the clock is monotonic because > it always goes forward. The monotonic property can be described as: > > t1=time.monotonic() > t2=time.monotonic() > assert t2 >= t1 I agree. The point I was making is that implication of "steady" is that (t2-t1) is the same (given that t2 and t1 occur in time at the same relative moments), which is a guarantee that I don't see any platform providing currently. Any clock that can be "adjusted" in any manner is not going to meet the "steady" criterion. >> In that case, I don't think time.try_monotonic() is really needed >> because we can emulate "time.monotonic()" in software if the platform is >> deficient. > > As I wrote, I don't think that Python should workaround OS bugs. If > the OS monotonic clock is not monotonic, the OS should be fixed. I sympathize with this, but if the idea is that the Python stdlib should use time.monotonic() for scheduling, then it needs to always be available. Otherwise, we are not going to use it ourselves, and what sort of example is that to set? >> I can't imagine a scenario where you would ask for a >> monotonic clock and would rather have an error than have Python fill in >> the gap with an emulation. > > Sorry, I don't understand what you mean with "fill in the gap with an > emulation". You would like to implement a monotonic clock based on the > system clock? If "time.monotonic()" is only sometimes available, then I don't see the added clock being anything more than an amusement. (In this case, I'd rather just use clock_gettime() and friends directly, because I have to be platform aware anyways.) What developers want is a timer that is useful for scheduling things to happen after predictable interval in the future, so we should give them that to the best of our ability. -- Scott Dial scott at scottdial.com From victor.stinner at gmail.com Wed Mar 28 12:45:18 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Mar 2012 12:45:18 +0200 Subject: [Python-Dev] Bug in generator if the generator in created in a C thread Message-ID: Hi, bugs.python.org is down so I'm reporting the bug here :-) We have a crash in our product when tracing is enabled by sys.settrace() and threading.settrace(). If a Python generator is created in a C thread, calling the generator later in another thread may crash if Python tracing is enabled. - the C thread calls PyGILState_Ensure() which creates a temporary Python thread state - a generator is created, the generator has a reference to a Python frame which keeps a reference to the temporary Python thread state - the C thread calls PyGILState_Releases() which destroys the temporary Python thread state - when the generator is called later in another thread, call_trace() reads the Python thread state from the generator frame, which is the destroyed frame => it does crash on a pointer dereference if the memory was reallocated (by malloc()) and the data were erased To reproduce the crash, unpack the attached generator_frame_bug.tar.gz, compile the C module using "python setup.py build" and then run "PYTHONPATH=$(ls -d build/lib*/) python test.py" (or just "python test.py if you installed the _test module). You may need to use Valgrind to see the error, or call memset(tstate, 0xFF, sizeof(*tstate)) before free(tstate) in tstate_delete_common(). Calling the generator should update its reference to the Python state thread in its frame. The generator may also clears frame->f_tstate (to detect bugs earlier), as it does for frame->f_back (to avoid a reference cycle). Attached patch implements this fix for Python 3.3. Victor -------------- next part -------------- A non-text attachment was scrubbed... Name: generator_frame_bug.tar.gz Type: application/x-gzip Size: 1747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: generator.patch Type: text/x-patch Size: 789 bytes Desc: not available URL: From victor.stinner at gmail.com Wed Mar 28 12:56:18 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Mar 2012 12:56:18 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F72DBDE.6040003@scottdial.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: >>> In that case, I don't think time.try_monotonic() is really needed >>> because we can emulate "time.monotonic()" in software if the platform is >>> deficient. >> >> As I wrote, I don't think that Python should workaround OS bugs. If >> the OS monotonic clock is not monotonic, the OS should be fixed. > > I sympathize with this, but if the idea is that the Python stdlib should > use time.monotonic() for scheduling, then it needs to always be > available. Otherwise, we are not going to use it ourselves, and what > sort of example is that to set? There is time.hires() if you need a monotonic clock with a fallback to the system clock. Victor From steve at pearwood.info Wed Mar 28 14:05:59 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 28 Mar 2012 23:05:59 +1100 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: <4F72FEA7.8090903@pearwood.info> Georg Brandl wrote: > On 28.03.2012 06:45, Nick Coghlan wrote: >> On Wed, Mar 28, 2012 at 10:36 AM, Victor Stinner >> wrote: >>> If QueryPerformanceCounter() is monotonic, the API can be simplified to: >>> >>> * time.time() = system clock >>> * time.monotonic() = monotonic clock >>> * time.hires() = monotonic clock or fallback to system clock >>> >>> time.hires() definition is exactly what I was trying to implement with >>> "time.steady(strict=True)" / "time.try_monotonic()". >> Please don't call the fallback version "hires" as it suggests it may >> be higher resolution than time.time() and that's completely the wrong >> idea. > > It's also a completely ugly name, since it's quite hard to figure out > what it is supposed to stand for in the first place. Precisely. I always read "hires" as the verb hires (as in "he hires a car to go on holiday") rather than HIgh RESolution. -1 on hires, it's a horrible name. And misleading as well, because on Linux, it isn't any more high res than time.time(). +1 on Nick's suggestion of try_monotonic. It is clear and obvious and doesn't mislead. I don't have an opinion as to what the implementation of try_monotonic should be. Whether it should fall back to time.time, time.clock, or something else, I don't know. But it is a clear and obvious solution for the use-case of "I prefer the monotonic clock, if it is available, otherwise I'll take my chances with a best-effect clock." -- Steven From rdmurray at bitdance.com Wed Mar 28 14:56:16 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 28 Mar 2012 08:56:16 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F72FEA7.8090903@pearwood.info> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72FEA7.8090903@pearwood.info> Message-ID: <20120328125617.A99DD2500E9@webabinitio.net> On Wed, 28 Mar 2012 23:05:59 +1100, Steven D'Aprano wrote: > +1 on Nick's suggestion of try_monotonic. It is clear and obvious and doesn't > mislead. How about "monotonicest". (No, this is not really a serious suggestion.) However, time.steadiest might actually work. --David From larry at hastings.org Wed Mar 28 15:01:23 2012 From: larry at hastings.org (Larry Hastings) Date: Wed, 28 Mar 2012 14:01:23 +0100 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <20120328125617.A99DD2500E9@webabinitio.net> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72FEA7.8090903@pearwood.info> <20120328125617.A99DD2500E9@webabinitio.net> Message-ID: <4F730BA3.5080901@hastings.org> On 03/28/2012 01:56 PM, R. David Murray wrote: > On Wed, 28 Mar 2012 23:05:59 +1100, Steven D'Aprano wrote: >> +1 on Nick's suggestion of try_monotonic. It is clear and obvious and doesn't >> mislead. > How about "monotonicest". > > (No, this is not really a serious suggestion.) "monotonish". Thus honoring the Principle Of Least Monotonishment, //arry/ From anacrolix at gmail.com Wed Mar 28 15:48:58 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 28 Mar 2012 21:48:58 +0800 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F730BA3.5080901@hastings.org> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72FEA7.8090903@pearwood.info> <20120328125617.A99DD2500E9@webabinitio.net> <4F730BA3.5080901@hastings.org> Message-ID: time.monotonic(): The uneventful and colorless function. On Mar 28, 2012 9:30 PM, "Larry Hastings" wrote: > On 03/28/2012 01:56 PM, R. David Murray wrote: > >> On Wed, 28 Mar 2012 23:05:59 +1100, Steven D'Aprano >> wrote: >> >>> +1 on Nick's suggestion of try_monotonic. It is clear and obvious and >>> doesn't >>> mislead. >>> >> How about "monotonicest". >> >> (No, this is not really a serious suggestion.) >> > > "monotonish". > > Thus honoring the Principle Of Least Monotonishment, > > > //arry/ > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > anacrolix%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at jonathanfrench.net Wed Mar 28 16:03:11 2012 From: me at jonathanfrench.net (Jonathan French) Date: Wed, 28 Mar 2012 15:03:11 +0100 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72FEA7.8090903@pearwood.info> <20120328125617.A99DD2500E9@webabinitio.net> <4F730BA3.5080901@hastings.org> Message-ID: No, that would be time.monotonous(). This is time.monotonic(), the function that can only play a single note at a time. Uh, I mean time.monophonic(). Hmm, this is harder than it looks. On 28 March 2012 14:48, Matt Joiner wrote: > time.monotonic(): The uneventful and colorless function. > On Mar 28, 2012 9:30 PM, "Larry Hastings" wrote: > >> On 03/28/2012 01:56 PM, R. David Murray wrote: >> >>> On Wed, 28 Mar 2012 23:05:59 +1100, Steven D'Aprano >>> wrote: >>> >>>> +1 on Nick's suggestion of try_monotonic. It is clear and obvious and >>>> doesn't >>>> mislead. >>>> >>> How about "monotonicest". >>> >>> (No, this is not really a serious suggestion.) >>> >> >> "monotonish". >> >> Thus honoring the Principle Of Least Monotonishment, >> >> >> //arry/ >> ______________________________**_________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/**mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** >> anacrolix%40gmail.com >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/me%40jonathanfrench.net > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 28 16:14:16 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Mar 2012 07:14:16 -0700 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: Victor, I have completely lost track of the details of this discussion. Could you (with help from others who contributed) try to compile a table showing, for each platform (Windows/Mac/Linux/BSD) which clocks (or variations) we are considering, and for each of those: - a link for the reference documentation - what their typical accuracy is (barring jumps) - what they do when the "civil" time is made to jump (forward or back) by the user - how they are affected by small tweaks to the civil time by NTP - what they do if the system is suspended and resumed - whether they can be shared between processes running on the same machine - whether they may fail or be unsupported under some circumstances I have a feeling that if I saw such a table it would be much easier to decide. I assume much of this has already been said at one point in this thread, but it's impossible to have an overview at the moment. If someone has more questions they'd like to see answered please add to the list. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Wed Mar 28 16:17:32 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Mar 2012 00:17:32 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: On Wed, Mar 28, 2012 at 8:56 PM, Victor Stinner wrote: >>>> In that case, I don't think time.try_monotonic() is really needed >>>> because we can emulate "time.monotonic()" in software if the platform is >>>> deficient. >>> >>> As I wrote, I don't think that Python should workaround OS bugs. If >>> the OS monotonic clock is not monotonic, the OS should be fixed. >> >> I sympathize with this, but if the idea is that the Python stdlib should >> use time.monotonic() for scheduling, then it needs to always be >> available. Otherwise, we are not going to use it ourselves, and what >> sort of example is that to set? > > There is time.hires() if you need a monotonic clock with a fallback to > the system clock. Completely unintuitive and unnecessary. With the GIL taking care of synchronisation issues, we can easily coerce time.time() into being a monotonic clock by the simple expedient of saving the last returned value: def _make_monotic: try: # Use underlying system monotonic clock if we can return _monotonic except NameError: _tick = time() def monotic(): _new_tick = time() if _new_tick > _tick: _tick = _new_tick return _tick monotonic = _make_monotonic() Monotonicity of the result is thus ensured, even when using time.time() as a fallback. If using the system monotonic clock to get greater precision is acceptable for an application, then forcing monotonicity shouldn't be a problem either. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From yselivanov.ml at gmail.com Wed Mar 28 16:27:26 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 28 Mar 2012 10:27:26 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: <467A106B-5887-4820-94A1-BF19D46F2F0F@gmail.com> On 2012-03-28, at 10:17 AM, Nick Coghlan wrote: > def _make_monotic: > try: > # Use underlying system monotonic clock if we can > return _monotonic > except NameError: > _tick = time() > def monotic(): > _new_tick = time() > if _new_tick > _tick: > _tick = _new_tick > return _tick > > monotonic = _make_monotonic() > > Monotonicity of the result is thus ensured, even when using > time.time() as a fallback. What if system time jumps 1 year back? We'll have the same monotonic time returned for this whole year? I don't think we should even try to emulate any of OS-level functionality. - Yury From guido at python.org Wed Mar 28 16:29:14 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Mar 2012 07:29:14 -0700 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: On Wed, Mar 28, 2012 at 7:17 AM, Nick Coghlan wrote: > On Wed, Mar 28, 2012 at 8:56 PM, Victor Stinner > wrote: >>>>> In that case, I don't think time.try_monotonic() is really needed >>>>> because we can emulate "time.monotonic()" in software if the platform is >>>>> deficient. >>>> >>>> As I wrote, I don't think that Python should workaround OS bugs. If >>>> the OS monotonic clock is not monotonic, the OS should be fixed. >>> >>> I sympathize with this, but if the idea is that the Python stdlib should >>> use time.monotonic() for scheduling, then it needs to always be >>> available. Otherwise, we are not going to use it ourselves, and what >>> sort of example is that to set? >> >> There is time.hires() if you need a monotonic clock with a fallback to >> the system clock. > > Completely unintuitive and unnecessary. With the GIL taking care of > synchronisation issues, we can easily coerce time.time() into being a > monotonic clock by the simple expedient of saving the last returned > value: > > ?def _make_monotic: > ? ? ?try: > ? ? ? ? ?# Use underlying system monotonic clock if we can > ? ? ? ? ?return _monotonic > ? ? ?except NameError: > ? ? ? ? ?_tick = time() > ? ? ? ? ?def monotic(): > ? ? ? ? ? ? ?_new_tick = time() > ? ? ? ? ? ? ?if _new_tick > _tick: > ? ? ? ? ? ? ? ? ?_tick = _new_tick > ? ? ? ? ? ? ?return _tick > > ?monotonic = _make_monotonic() > > Monotonicity of the result is thus ensured, even when using > time.time() as a fallback. > > If using the system monotonic clock to get greater precision is > acceptable for an application, then forcing monotonicity shouldn't be > a problem either. That's a pretty obvious trick. But why don't the kernels do this if monotonicity is so important? I'm sure there are also downsides, e.g. if the clock is accidentally set forward by an hour and then back again, you wouldn't have a useful clock for an hour. And the cache is not shared between processes so different processes wouldn't see the same clock value (I presume that most of these clocks have state in the kernel that isn't bound to any particular process -- AFAIK only clock() does that, and only on Unixy systems). -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Wed Mar 28 16:36:31 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Mar 2012 00:36:31 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On Wed, Mar 28, 2012 at 6:40 PM, Victor Stinner wrote: >> If we're simplifying the idea to only promising a monotonic >> clock (i.e. will never go backwards within a given process, but may >> produce the same value for an indefinite period, and may jump forwards >> by arbitrarily large amounts), > > I don't know any monotonic clock jumping "forwards by arbitrarily > large amounts". Linux can change CLOCK_MONOTONIC speed, but NTP > doesn't "jump". If I understood Glyph's explanation correctly, then if your application is running in a VM and the VM is getting its clock data from the underlying hypervisor, then suspending and resuming the VM may result in forward jumping of the monotonic clocks in the guest OS. I believe suspending and hibernating may cause similar problems for even a non-virtualised OS that is getting its time data from a real-time clock chip that keeps running even when the main CPU goes to sleep. (If I *misunderstood* Glyph's explanation, then he may have only been talking about the latter case) Monotonicity is fairly easy to guarantee - you just remember the last value you returned and ensure you never return a lower value than that for the lifetime of the process. The only complication is thread synchronisation, and the GIL (or a dedicated lock for Jython/IronPython) can deal with that. Steadiness, on the other hand, requires a real world time reference and is thus really the domain of specialised hardware like atomic clocks and GPS units rather than software that can be suspended and resumed later without changing its internal state. There's a reason comms station operators pay substantial chunks of money for time & frequency reference devices [1]. This is why I now think we only need one new clock function: time.monotonic(). It will be the system monotonic clock if one is available, otherwise it will be our own equivalent wrapper around time.time() that just caches the last value returned to ensure the result never goes backwards. With time.monotonic() guaranteed to always be available, there's no need for a separate function that falls back to an unconditioned time.time() result. Regards, Nick. [1] For example: http://www.symmetricom.com/products/gps-solutions/gps-time-frequency-receivers/XLi/ -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Wed Mar 28 16:42:13 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Mar 2012 07:42:13 -0700 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On Wed, Mar 28, 2012 at 7:36 AM, Nick Coghlan wrote: > On Wed, Mar 28, 2012 at 6:40 PM, Victor Stinner > wrote: >>> If we're simplifying the idea to only promising a monotonic >>> clock (i.e. will never go backwards within a given process, but may >>> produce the same value for an indefinite period, and may jump forwards >>> by arbitrarily large amounts), >> >> I don't know any monotonic clock jumping "forwards by arbitrarily >> large amounts". Linux can change CLOCK_MONOTONIC speed, but NTP >> doesn't "jump". > > If I understood Glyph's explanation correctly, then if your > application is running in a VM and the VM is getting its clock data > from the underlying hypervisor, then suspending and resuming the VM > may result in forward jumping of the monotonic clocks in the guest OS. > I believe suspending and hibernating may cause similar problems for > even a non-virtualised OS that is getting its time data from a > real-time clock chip that keeps running even when the main CPU goes to > sleep. (If I *misunderstood* Glyph's explanation, then he may have > only been talking about the latter case) > > Monotonicity is fairly easy to guarantee - you just remember the last > value you returned and ensure you never return a lower value than that > for the lifetime of the process. The only complication is thread > synchronisation, and the GIL (or a dedicated lock for > Jython/IronPython) can deal with that. Steadiness, on the other hand, > requires a real world time reference and is thus really the domain of > specialised hardware like atomic clocks and GPS units rather than > software that can be suspended and resumed later without changing its > internal state. There's a reason comms station operators pay > substantial chunks of money for time & frequency reference devices > [1]. > > This is why I now think we only need one new clock function: > time.monotonic(). It will be the system monotonic clock if one is > available, otherwise it will be our own equivalent wrapper around > time.time() that just caches the last value returned to ensure the > result never goes backwards. As I said, I think the caching idea is bad. We may have to settle for semantics that are less than perfect -- presumably if you are doing benchmarking you just have to throw away a bad result that happened to be affected by a clock anomaly, and if you are using timeouts, retries are already part of life. > With time.monotonic() guaranteed to always be available, there's no > need for a separate function that falls back to an unconditioned > time.time() result. I would love to have only one new clock function in 3.3. > Regards, > Nick. > > [1] For example: > http://www.symmetricom.com/products/gps-solutions/gps-time-frequency-receivers/XLi/ -- --Guido van Rossum (python.org/~guido) From guido at python.org Wed Mar 28 16:43:27 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Mar 2012 07:43:27 -0700 Subject: [Python-Dev] Bug in generator if the generator in created in a C thread In-Reply-To: References: Message-ID: Interesting bug. :-( It seems bugs.python.org is back up, so can you file it there too? On Wed, Mar 28, 2012 at 3:45 AM, Victor Stinner wrote: > Hi, > > bugs.python.org is down so I'm reporting the bug here :-) > > We have a crash in our product when tracing is enabled by > sys.settrace() and threading.settrace(). If a Python generator is > created in a C thread, calling the generator later in another thread > may crash if Python tracing is enabled. > > ?- the C thread calls PyGILState_Ensure() which creates a temporary > Python thread state > ?- a generator is created, the generator has a reference to a Python > frame which keeps a reference to the temporary Python thread state > ?- the C thread calls PyGILState_Releases() which destroys the > temporary Python thread state > ?- when the generator is called later in another thread, call_trace() > reads the Python thread state from the generator frame, which is the > destroyed frame => it does crash on a pointer dereference if the > memory was reallocated (by malloc()) and the data were erased > > To reproduce the crash, unpack the attached > generator_frame_bug.tar.gz, compile the C module using "python > setup.py build" and then run "PYTHONPATH=$(ls -d build/lib*/) python > test.py" (or just "python test.py if you installed the _test module). > You may need to use Valgrind to see the error, or call memset(tstate, > 0xFF, sizeof(*tstate)) before free(tstate) in tstate_delete_common(). > > Calling the generator should update its reference to the Python state > thread in its frame. The generator may also clears frame->f_tstate (to > detect bugs earlier), as it does for frame->f_back (to avoid a > reference cycle). Attached patch implements this fix for Python 3.3. > > Victor > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Wed Mar 28 16:45:04 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Mar 2012 00:45:04 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <467A106B-5887-4820-94A1-BF19D46F2F0F@gmail.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <467A106B-5887-4820-94A1-BF19D46F2F0F@gmail.com> Message-ID: On Thu, Mar 29, 2012 at 12:27 AM, Yury Selivanov wrote: > What if system time jumps 1 year back? ?We'll have the same > monotonic time returned for this whole year? > > I don't think we should even try to emulate any of OS-level > functionality. You have to keep in mind the alternative here: falling back to an *unconditioned* time.time() value (which is the status quo, and necessary to preserve backwards compatibility). That will break just as badly in that scenario and is precisely the reason that the OS level monotonic functionality is desirable in the first place. I'd be quite happy with a solution that made the OS level monotonic clock part of the public API, with the caveat that it may not be available. Then the necessary trio of functions would be: time.time(): existing system clock, always available time.os_monotonic(): OS level monotonic clock, not always available time.monotonic(): always available, same as os_monotonic if it exists, otherwise uses a time() based emulation that may not be consistent across processes and may "mark time" for extended periods if the underlying OS clock is forced to jump back a long way. I think that naming scheme is more elegant than using monotonic() for the OS level monotonicity and try_monotonic() for the fallback version, but I'd be OK with the latter approach, too. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From yselivanov.ml at gmail.com Wed Mar 28 16:56:34 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 28 Mar 2012 10:56:34 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On 2012-03-28, at 10:36 AM, Nick Coghlan wrote: > Monotonicity is fairly easy to guarantee - you just remember the last > value you returned and ensure you never return a lower value than that > for the lifetime of the process. As I said in my previous mail - I don't think we should ever do that. Time may jump back and forth, and with your approach it will result in monotonic() being completely unusable. If time jumps back for N minutes, or years, that leads to completely broken expectations for timeouts for N minutes or years correspondingly (and that's just the timeouts case, I'm sure that there are much more critical time-related use-cases.) If monotonic() will utilize such hack, you add nothing usable in stdlib. Every serious framework or library will have to re-implement it using only OS-level functions, and *FAIL* if the OS doesn't support monotonic time. Fail, because such framework can't guarantee that it will work correctly. So I think time module should have only one new function: monotonic(), and this function should be only available if OS provides the underlying functionality. No need for steady(), try_monotonic() and other hacks. Each module can decide if its dependancy on monotonic is critical or not, and if it is not, you can always have: try: from time import monotonic as _time except ImportError: from time import time as _time That's how lots of code is written these days, like using 'epoll' if available, and fallback to 'select' if not. Why don't you try to abstract differences between them in the standard library? So I see no point in adding some loose abstractions to the stdlib now. - Yury From yselivanov.ml at gmail.com Wed Mar 28 17:02:30 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 28 Mar 2012 11:02:30 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <467A106B-5887-4820-94A1-BF19D46F2F0F@gmail.com> Message-ID: <53304706-8864-4CFE-B4E5-33F7918FE46D@gmail.com> On 2012-03-28, at 10:45 AM, Nick Coghlan wrote: > On Thu, Mar 29, 2012 at 12:27 AM, Yury Selivanov > wrote: >> What if system time jumps 1 year back? We'll have the same >> monotonic time returned for this whole year? >> >> I don't think we should even try to emulate any of OS-level >> functionality. > > You have to keep in mind the alternative here: falling back to an > *unconditioned* time.time() value (which is the status quo, and > necessary to preserve backwards compatibility). That will break just > as badly in that scenario and is precisely the reason that the OS > level monotonic functionality is desirable in the first place. Well, my argumentation is that you either have some code that depends on monotonic time and can't work without it, or you have a code that can work with any time (and only precision matters). Maybe I'm wrong. > I'd be quite happy with a solution that made the OS level monotonic > clock part of the public API, with the caveat that it may not be > available. Then the necessary trio of functions would be: > > time.time(): existing system clock, always available > time.os_monotonic(): OS level monotonic clock, not always available > time.monotonic(): always available, same as os_monotonic if it exists, > otherwise uses a time() based emulation that may not be consistent > across processes and may "mark time" for extended periods if the > underlying OS clock is forced to jump back a long way. I still don't like this 'emulation' idea. Smells bad for standard lib. Big -1 on this approach. - Yury From ncoghlan at gmail.com Wed Mar 28 17:08:50 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Mar 2012 01:08:50 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On Thu, Mar 29, 2012 at 12:42 AM, Guido van Rossum wrote: > As I said, I think the caching idea is bad. We may have to settle for > semantics that are less than perfect -- presumably if you are doing > benchmarking you just have to throw away a bad result that happened to > be affected by a clock anomaly, and if you are using timeouts, retries > are already part of life. I agree caching doesn't solve the problems that are solved by an OS level monotonic clock, but falling back to an unmodifided time.time() result instead doesn't solve those problems either. Falling back to time.time() just gives you the status quo: time may jump forwards or backwards by an arbitrary amount between any two calls. Cached monotonicity just changes the anomalous modes to be time jumping forwards, or time standing still for an extended period of time. The only thing the caching provides is that it becomes a reasonable fallback for a function called time.monotonic() - it *is* a monotonic clock that meets the formal contract of the function, it's just nowhere near as good or effective as one the OS can provide. Forward jumping anomalies aren't as harmful, are very hard to detect in the first place and behave the same regardless of the presence of caching, so the interesting case to look at is the difference in failure modes when the system clock jumps backwards. For benchmarking, a caching clock will produce a zero result instead of a negative result. Zeros aren't quite as obviously broken as negative numbers when benchmarking, but they're still sufficiently suspicious that most benchmarking activities will flag them as anomalous. If the jump back was sufficiently small that the subsequent call still produces a higher value than the original call, then behaviour reverts to being identical. For timeouts, setting the clock back means your operation will take longer to time out than you expected. This problem will occur regardless of whether you were using cached monotonicity (such that time stands still) or the system clock (such that time actually goes backwards). In either case, your deadline will never be reached until the backwards jump has been cancelled out by the subsequent passage of time. I want the standard library to be able to replace its time.time() calls with time.monotonic(). The only way we can do that without breaking cross-platform compatibility is if time.monotonic() is guaranteed to exist, even when the platform only provides time.time(). A dumb caching fallback implementation based on time.time() is the easiest way to achieve that withou making a complete mockery of the "monotonic()" name. There is then a *different* use case, which is 3.3+ only code which wants to fail noisily when there's no OS level monotonic support - the application developer really does want to fail *immediately* if there's no OS level monotonic clock available, instead of crossing your fingers and hoping you don't hit a clock adjustment glitch (crossing your fingers has, I'll point out, been the *only* option for all previous versions of Python, so it clearly can't be *that* scary a prospect). So, rather than making time.monotonic() something that the *standard library can't use*, I'd prefer to address that second use case by exposing the OS level monotonic clock as time.os_monotonic() only when it's available. That way, the natural transition for old time.time() based code is to time.monotonic() (with no cross-platform support implications), but time.os_monotonic() also becomes available for the stricter use cases. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Mar 28 17:35:46 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Mar 2012 01:35:46 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <53304706-8864-4CFE-B4E5-33F7918FE46D@gmail.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <467A106B-5887-4820-94A1-BF19D46F2F0F@gmail.com> <53304706-8864-4CFE-B4E5-33F7918FE46D@gmail.com> Message-ID: On Thu, Mar 29, 2012 at 1:02 AM, Yury Selivanov wrote: > On 2012-03-28, at 10:45 AM, Nick Coghlan wrote: > >> On Thu, Mar 29, 2012 at 12:27 AM, Yury Selivanov >> wrote: >>> What if system time jumps 1 year back? ?We'll have the same >>> monotonic time returned for this whole year? >>> >>> I don't think we should even try to emulate any of OS-level >>> functionality. >> >> You have to keep in mind the alternative here: falling back to an >> *unconditioned* time.time() value (which is the status quo, and >> necessary to preserve backwards compatibility). That will break just >> as badly in that scenario and is precisely the reason that the OS >> level monotonic functionality is desirable in the first place. > > Well, my argumentation is that you either have some code that depends > on monotonic time and can't work without it, or you have a code that > can work with any time (and only precision matters). ?Maybe I'm wrong. You're wrong. The primary use case for the new time.monotonic() function is to replace *existing* uses of time.time() in the standard library (mostly related to timeouts) that are currently vulnerable to clock adjustment related bugs. This real, concrete use case has been lost in some of the abstract theoretical discussions that have been going on this thread. We can't lose sight of the fact that using a system clock that is vulnerable to clock adjustment bugs to handle timeouts and benchmarking in Python has worked just fine for 20+ years. Using a monotonic clock instead is *better*, but it's far from essential, since clock adjustments that are big enough and poorly timed enough to cause real problems are fortunately a very rare occurrence. So, the primary use case is that we want to replace many of the time.time() calls in the standard library with time.monotonic() calls. To avoid backwards compatibility problems in the cross-platform support, that means time.monotonic() *must be available on every platform that currently provides time.time()*. This is why Victor's original proposal was that time.monotonic() simply fall back to time.time() if there was no OS level monotonic clock available. The intended use cases are using time.time() *right now* and have been doing so for years, so it is clearly an acceptable fallback for those cases. People (rightly, in my opinion) objected to the idea of time.monotonic() failing to guarantee monotonicity, thus the proposal to enforce at least a basic level of monotonicity through caching of the last returned value. I agree completely that this dumb caching solution doesn't solve any of the original problems with time.time() that make a time.monotonic() function desirable, but it isn't meant to. It's only meant to provide graceful degradation to something that is *no worse than the current behaviour when using time.time() in Python 3.2* while still respecting the property of monotonicity for the new API. Yes, it's an ugly hack, but it is a necessary fallback to avoid accidental regressions in our cross-platform support. For the major platforms (i.e. *nix, Mac OS X, Windows), there *will* be an OS level monotonic clock available, thus using time.monotonic() will have the desired effect of protecting from clocks being adjusted backwards. For other platforms, the behaviour (and vulnerabilities) will be essentially unchanged from the Python 3.2 approach (i.e. using time.time() with no monotonicity guarantees at all). However, some 3.3+ applications may want to be stricter about their behaviour and either bail out completely or fall back to an unfiltered time.time() call if an OS-level monotonic clock is not available. For those, it makes sense to expose time.os_monotonic() directly (and only if it is available), thus allowing those developers to make up their own mind instead of accepting the cross-platform fallback in time.monotonic(). Yes, you can get the exact same effect with the "monotonic()" and "try_monotonic()" naming scheme, but why force the standard library (and anyone else wanting to upgrade from time.time() without harming cross-platform support) to use such an ugly name when the "os_monotonic" and "monotonic" naming scheme provides a much neater alternative? Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From victor.stinner at gmail.com Wed Mar 28 17:37:09 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Mar 2012 17:37:09 +0200 Subject: [Python-Dev] Bug in generator if the generator in created in a C thread In-Reply-To: References: Message-ID: 2012/3/28 Guido van Rossum : > Interesting bug. :-( > > It seems bugs.python.org is back up, so can you file it there too? It took us weeks to track the bug. Here is the issue: http://bugs.python.org/issue14432 Victor From guido at python.org Wed Mar 28 17:47:31 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Mar 2012 08:47:31 -0700 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On Wed, Mar 28, 2012 at 8:08 AM, Nick Coghlan wrote: > On Thu, Mar 29, 2012 at 12:42 AM, Guido van Rossum wrote: >> As I said, I think the caching idea is bad. We may have to settle for >> semantics that are less than perfect -- presumably if you are doing >> benchmarking you just have to throw away a bad result that happened to >> be affected by a clock anomaly, and if you are using timeouts, retries >> are already part of life. > > I agree caching doesn't solve the problems that are solved by an OS > level monotonic clock, but falling back to an unmodifided time.time() > result instead doesn't solve those problems either. Falling back to > time.time() just gives you the status quo: time may jump forwards or > backwards by an arbitrary amount between any two calls. Cached > monotonicity just changes the anomalous modes to be time jumping > forwards, or time standing still for an extended period of time. The > only thing the caching provides is that it becomes a reasonable > fallback for a function called time.monotonic() - it *is* a monotonic > clock that meets the formal contract of the function, it's just > nowhere near as good or effective as one the OS can provide. TBH, I don't think like this focus on monotonicity as the most important feature. > Forward jumping anomalies aren't as harmful, are very hard to detect > in the first place and behave the same regardless of the presence of > caching, so the interesting case to look at is the difference in > failure modes when the system clock jumps backwards. Agreed. > For benchmarking, a caching clock will produce a zero result instead > of a negative result. Zeros aren't quite as obviously broken as > negative numbers when benchmarking, but they're still sufficiently > suspicious that most benchmarking activities will flag them as > anomalous. If the jump back was sufficiently small that the subsequent > call still produces a higher value than the original call, then > behaviour reverts to being identical. So for benchmarking we don't care about jumps, really, and the caching version is slightly less useful. > For timeouts, setting the clock back means your operation will take > longer to time out than you expected. This problem will occur > regardless of whether you were using cached monotonicity (such that > time stands still) or the system clock (such that time actually goes > backwards). In either case, your deadline will never be reached until > the backwards jump has been cancelled out by the subsequent passage of > time. Where in the stdlib do we actually calculate timeouts instead of using the timeouts built into the OS (e.g. select())? I think it would be nice if we could somehow use the *same* clock as the OS uses to implement timeouts. > I want the standard library to be able to replace its time.time() > calls with time.monotonic(). Where in the stdlib? (I'm aware of threading.py. Any other places?) > The only way we can do that without > breaking cross-platform compatibility is if time.monotonic() is > guaranteed to exist, even when the platform only provides time.time(). > A dumb caching fallback implementation based on time.time() is the > easiest way to achieve that withou making a complete mockery of the > "monotonic()" name. Yeah, so maybe it's a bad name. :-) > There is then a *different* use case, which is 3.3+ only code which > wants to fail noisily when there's no OS level monotonic support - the > application developer really does want to fail *immediately* if > there's no OS level monotonic clock available, instead of crossing > your fingers and hoping you don't hit a clock adjustment glitch > (crossing your fingers has, I'll point out, been the *only* option for > all previous versions of Python, so it clearly can't be *that* scary a > prospect). > > So, rather than making time.monotonic() something that the *standard > library can't use*, I'd prefer to address that second use case by > exposing the OS level monotonic clock as time.os_monotonic() only when > it's available. That way, the natural transition for old time.time() > based code is to time.monotonic() (with no cross-platform support > implications), but time.os_monotonic() also becomes available for the > stricter use cases. I'd be happier if the fallback function didn't try to guarantee things the underlying clock can't guarantee. I.e. I like the idea of having a function that uses some accurate OS clock if one exists but falls back to time.time() if not; I don't like the idea of that new function trying to interpret the value of time.time() in any way. Applications that need the OS clock's guarantees can call it directly. We could also offer something where you can introspect the properties of the clock (or clocks) so that an app can choose the best clock depending on its needs. To summarize my problem with the caching idea: take a simple timeout loop such as found in several places in threading.py. def wait_for(delta, eps): # Wait for delta seconds, sleeping eps seconds at a time deadline = now() + delta while now() < deadline: sleep(eps) If the now() clock jumps backward after the initial call, we end up waiting too long -- until either the clock jumps forward again or until we've made up the difference. If the now() clock jumps forward after the initial call, we end up waiting less time, which is probably not such a big problem (though it might). But now consider a caching clock, and consider that the system clock made a jump backwards *before* this function is called. The cache prevents us from seeing it, so the initial call to now() returns the highest clock value seen so far. And until the system clock has caught up with that, now() will return the same value over and over -- so WE STILL WAIT TOO LONG. My conclusion: you can't win this game by forcing the clock to return a monotonic value. A better approach might be to compute how many sleep(eps) calls we're expected to make, and to limit the loop to that -- although sleep() doesn't make any guarantees either about sleeping too short or too long. Basically, if you do sleep(1) and find that your clock didn't move (enough), you can't tell the difference between a short sleep and a clock that jumped back. And if your clock moved to much, you still don't know if the problem was with sleep() or with your clock. -- --Guido van Rossum (python.org/~guido) From anacrolix at gmail.com Wed Mar 28 18:11:49 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 29 Mar 2012 00:11:49 +0800 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: time.timeout_clock? Everyone knows what that will be for and we won't have to make silly theoretical claims about its properties and expected uses. If no one else looks before I next get to a PC I'll dig up the clock/timing source used for select and friends, and find any corresponding syscall that retrieves it for Linux. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 28 18:14:11 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Mar 2012 02:14:11 +1000 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On Thu, Mar 29, 2012 at 1:47 AM, Guido van Rossum wrote: > Where in the stdlib? (I'm aware of threading.py. Any other places?) Victor had at least one other example. multiprocessing, maybe? I believe the test suite may still have a few instances as well. > But now consider a caching clock, and consider that the system clock > made a jump backwards *before* this function is called. The cache > prevents us from seeing it, so the initial call to now() returns the > highest clock value seen so far. And until the system clock has caught > up with that, now() will return the same value over and over -- so WE > STILL WAIT TOO ?LONG. Ouch. OK, I'm convinced the caching fallback is worse than just falling back to time.time() directly, which means the naming problem needs to be handled another way. > My conclusion: you can't win this game by forcing the clock to return > a monotonic value. A better approach might be to compute how many > sleep(eps) calls we're expected to make, and to limit the loop to that > -- although sleep() doesn't make any guarantees either about sleeping > too short or too long. Basically, if you do sleep(1) and find that > your clock didn't move (enough), you can't tell the difference between > a short sleep and a clock that jumped back. And if your clock moved to > much, you still don't know if the problem was with sleep() or with > your clock. With your point about the problem with the naive caching mechanism acknowledged, I think we can safely assign time.monotonic() as the name of the OS provided monotonic clock. That means choosing a name for the version that falls back to time() if monotonic() isn't available so it can be safely substituted for time.time() without having to worry about platform compatibility implications. I don't like Victor's current "hires" (because it doesn't hint at the fallback behaviour, may actually be the same res as time.time() and reads like an unrelated English word). My own suggestion of "try_monotic()" would get the job done, but is hardly going to win any API beauty contests. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From steve at pearwood.info Wed Mar 28 18:18:56 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 29 Mar 2012 03:18:56 +1100 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: <4F7339F0.6060501@pearwood.info> Matt Joiner wrote: > time.timeout_clock? > > Everyone knows what that will be for and we won't have to make silly > theoretical claims about its properties and expected uses. I don't. -- Steven From stephen at xemacs.org Wed Mar 28 18:39:40 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 29 Mar 2012 01:39:40 +0900 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: On Thu, Mar 29, 2012 at 1:14 AM, Nick Coghlan wrote: > That means choosing a name for the version that falls back to time() > if monotonic() isn't available so it can be safely substituted for > time.time() without having to worry about platform compatibility > implications. What's wrong with "time.time()" again? As documented in http://docs.python.org/py3k/library/time.html it makes no guarantees, and specifically there is *no* guarantee that it will ever behave *badly* . Of course, we'll have to guarantee that, if a badly-behaved clock is available, users can get access to it, so call that time._time(). From steve at pearwood.info Wed Mar 28 19:00:13 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 29 Mar 2012 04:00:13 +1100 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: <4F73439D.4070208@pearwood.info> Nick Coghlan wrote: > Completely unintuitive and unnecessary. With the GIL taking care of > synchronisation issues, we can easily coerce time.time() into being a > monotonic clock by the simple expedient of saving the last returned > value: [snip] Here's a version that doesn't suffer from the flaw of returning a long stream of constant values when the system clock jumps backwards a significant amount: class MockTime: def __init__(self): self.ticks = [1, 2, 3, 4, 2, 3, 4, 5, 7, 3, 4, 6, 7, 8, 8, 9] self.i = -1 def __call__(self): self.i += 1 return self.ticks[self.i] time = MockTime() _prev = _prev_raw = 0 def monotonic(): global _prev, _prev_raw raw = time() delta = max(0, raw - _prev_raw) _prev_raw = raw _prev += delta return _prev And in use: >>> [monotonic() for i in range(16)] [1, 2, 3, 4, 4, 5, 6, 7, 9, 9, 10, 12, 13, 14, 14, 15] Time: [1, 2, 3, 4, 2, 3, 4, 5, 7, 3, 4, 6, 7, 8, 8, 9] Nick: [1, 2, 3, 4, 4, 4, 4, 5, 7, 7, 7, 7, 7, 8, 8, 9] Mine: [1, 2, 3, 4, 4, 5, 6, 7, 9, 9, 10, 12, 13, 14, 14, 15] Mine will get ahead of the system clock each time it jumps back, but it's a lot closer to the ideal of a *strictly* monotonically increasing clock. Assuming that the system clock will never jump backwards twice in a row, the double-caching version will never have more than two constant values in a row. -- Steven From regebro at gmail.com Wed Mar 28 19:01:48 2012 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 28 Mar 2012 19:01:48 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: On Wed, Mar 28, 2012 at 12:56, Victor Stinner wrote: > There is time.hires() if you need a monotonic clock with a fallback to > the system clock. Does this primarily give a high resolution clock, or primarily a monotonic clock? That's not clear from either the name, or the PEP. //Lennart From neologix at free.fr Wed Mar 28 19:31:21 2012 From: neologix at free.fr (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Wed, 28 Mar 2012 19:31:21 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: > What's wrong with "time.time()" again? ?As documented in > http://docs.python.org/py3k/library/time.html it makes no guarantees, > and specifically there is *no* guarantee that it will ever behave > *badly* . ?Of course, we'll have to guarantee that, if a > badly-behaved clock is available, users can get access to it, so call > that time._time(). I'm not sure I understand your suggestion correctly, but replacing time.time() by time.monotonic() with fallback won't work, because time.monotonic() isn't wall-clock time: it can very well use an arbitrary reference point (most likely system start-up time). As for the hires() function, since there's no guarantee whatsoever that it does provide a better resolution than time.time(), this would be really misleading IMHO. From martin at v.loewis.de Wed Mar 28 19:47:09 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 28 Mar 2012 19:47:09 +0200 Subject: [Python-Dev] Bug tracker outage In-Reply-To: <4F722CEB.1040102@v.loewis.de> References: <4F722CEB.1040102@v.loewis.de> Message-ID: <4F734E9D.4050607@v.loewis.de> Am 27.03.2012 23:11, schrieb "Martin v. L?wis": > Upfront hosting (Izak Burger) is going to do a Debian upgrade of the bug > tracker machine "soon" (likely tomorrow). This may cause some outage, > since there is a lot of custom stuff on the machine which may > break with newer (Python) versions. I'll notify here when the upgrade > is complete. The upgrade is complete. It was in fact the Postgres upgrade (to 8.4) which caused the longest blackout time. Regards, Martin From yselivanov.ml at gmail.com Wed Mar 28 19:55:40 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 28 Mar 2012 13:55:40 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <467A106B-5887-4820-94A1-BF19D46F2F0F@gmail.com> <53304706-8864-4CFE-B4E5-33F7918FE46D@gmail.com> Message-ID: <6905A43A-F68B-42F0-BE4B-E27CAB94A185@gmail.com> On 2012-03-28, at 11:35 AM, Nick Coghlan wrote: > So, the primary use case is that we want to replace many of the > time.time() calls in the standard library with time.monotonic() calls. > To avoid backwards compatibility problems in the cross-platform > support, that means time.monotonic() *must be available on every > platform that currently provides time.time()*. OK. I got your point. And also I've just realized what I dislike about the way you want to implement the fallback. The main problem is that I treat the situation when time jumps backward as an exception, because, again, if you have timeouts you may get those timeouts to never be executed. So let's make the "try_monotonic()" function (or whatever name will be chosen) this way (your original code edited): def _make_monotic(): try: # Use underlying system monotonic clock if we can return _monotonic except NameError: _tick = time() def monotic(): nonlocal _time _new_tick = time() if _new_tick <= _tick: raise RuntimeError('time was adjusted backward') _tick = _new_tick return _new_tick return monotonic try_monotonic = _make_monotonic() At least this approach tries to follow some of the python's zen. - Yury From jaraco at jaraco.com Wed Mar 28 20:22:50 2012 From: jaraco at jaraco.com (Jason R. Coombs) Date: Wed, 28 Mar 2012 18:22:50 +0000 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) Message-ID: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> I see this was reported as a debian bug. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=665776 We encountered it as well. To reproduce, using virtualenv 1.7+ on Python 2.7.2 on Ubuntu, create a virtualenv. Move that virtualenv to a host with Python 2.7.3RC2 yields: jaraco at vdm-dev:~$ /usr/bin/python2.7 -V Python 2.7.3rc2 jaraco at vdm-dev:~$ env/bin/python -V Python 2.7.2 jaraco at vdm-dev:~$ env/bin/python -c "import os; os.urandom()" Traceback (most recent call last): File " ", line 1, in AttributeError: 'module' object has no attribute 'urandom' This bug causes Django to not start properly (under some circumstances). I reviewed the changes between v2.7.2 and 2.7 (tip) and it seems there was substantial refactoring of the os and posix modules for urandom. I still don't fully understand why the urandom method is missing (because the env includes the python 2.7.2 executable and stdlib). I suspect this change is going to cause some significant backward compatibility issues. Is there a recommended workaround? Should I file a bug? Regards, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6662 bytes Desc: not available URL: From carl at oddbird.net Wed Mar 28 20:48:21 2012 From: carl at oddbird.net (Carl Meyer) Date: Wed, 28 Mar 2012 12:48:21 -0600 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> Message-ID: <4F735CF5.3070700@oddbird.net> Hi Jason, On 03/28/2012 12:22 PM, Jason R. Coombs wrote: > To reproduce, using virtualenv 1.7+ on Python 2.7.2 on Ubuntu, create a > virtualenv. Move that virtualenv to a host with Python 2.7.3RC2 yields: > > jaraco at vdm-dev:~$ /usr/bin/python2.7 -V > > Python 2.7.3rc2 > > jaraco at vdm-dev:~$ env/bin/python -V > > Python 2.7.2 > > jaraco at vdm-dev:~$ env/bin/python -c "import os; os.urandom()" > > Traceback (most recent call last): > > File " ", line 1, in > > AttributeError: 'module' object has no attribute 'urandom' > > This bug causes Django to not start properly (under some circumstances). > > I reviewed the changes between v2.7.2 and 2.7 (tip) and it seems there > was substantial refactoring of the os and posix modules for urandom. > > I still don?t fully understand why the urandom method is missing > (because the env includes the python 2.7.2 executable and stdlib). In Python 2.6.8/2.7.3, urandom is built into the executable. A virtualenv doesn't contain the whole stdlib, only the bits necessary to bootstrap site.py. So the problem arises from trying to use the 2.7.3 stdlib with a 2.7.2 interpreter. > I suspect this change is going to cause some significant backward > compatibility issues. Is there a recommended workaround? Should I file a > bug? The workaround is easy: just re-run virtualenv on that path with the new interpreter. I was made aware of this issue a few weeks ago, and added a warning to the virtualenv "news" page: http://www.virtualenv.org/en/latest/news.html I'm not sure where else to publicize it. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From yselivanov.ml at gmail.com Wed Mar 28 21:02:27 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 28 Mar 2012 15:02:27 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <6905A43A-F68B-42F0-BE4B-E27CAB94A185@gmail.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <467A106B-5887-4820-94A1-BF19D46F2F0F@gmail.com> <53304706-8864-4CFE-B4E5-33F7918FE46D@gmail.com> <6905A43A-F68B-42F0-BE4B-E27CAB94A185@gmail.com> Message-ID: <75752BDB-2F7C-40CB-9725-A17D53814839@gmail.com> On 2012-03-28, at 1:55 PM, Yury Selivanov wrote: > nonlocal _time nonlocal _tick obviously. P.S. And we can make it to raise an error after some N calls to time() resulting in lesser time that is stored in the _tick variable. - Yury From andrew.svetlov at gmail.com Wed Mar 28 22:20:55 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Wed, 28 Mar 2012 23:20:55 +0300 Subject: [Python-Dev] datetime module and pytz with dateutil Message-ID: I figured out what pytz and dateutil are not mentioned in python docs for datetime module. It's clean why these libs is not a part of Python Libraries ? but that's not clean for Docs. >From my perspective at least pytz (as py3k compatible) should to be mentioned as the library which contains timezone info, supported carefully and recommended to use with datetime standard module, -- Thanks, Andrew Svetlov From jaraco at jaraco.com Wed Mar 28 22:56:30 2012 From: jaraco at jaraco.com (Jason R. Coombs) Date: Wed, 28 Mar 2012 20:56:30 +0000 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <4F735CF5.3070700@oddbird.net> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> <4F735CF5.3070700@oddbird.net> Message-ID: <7E79234E600438479EC119BD241B48D601E7D159@BY2PRD0610MB389.namprd06.prod.outlook.com> > -----Original Message----- > From: python-dev-bounces+jaraco=jaraco.com at python.org [mailto:python- > dev-bounces+jaraco=jaraco.com at python.org] On Behalf Of Carl Meyer > Sent: Wednesday, 28 March, 2012 14:48 > > The workaround is easy: just re-run virtualenv on that path with the new > interpreter. > Thanks for the quick response Carl. I appreciate all the work that's been done. I'm not sure the workaround is as simple as you say. Virtualenv doesn't replace the 'python' exe if it already exists (because it may already exist for a different minor version of Python (3.2, 2.6)). So the procedure is probably something like this: For each version of Python the virtualenv wraps (ls env/bin/python?.?): 1) Run env/bin/python -V. If the result starts with "Python ", remove env/bin/python. 2) Determine if that Python version uses distribute or setuptools. 3) Run virtualenv --python=python env (with --distribute if appropriate) I haven't yet tested this procedure, but I believe it's closer to what will need to be done. There are probably other factors. Unfortunately, to reliably repair the virtualenv is very difficult, so we will probably opt with re-deploying all of our virtualenvs. Will the release notes include something about this change, since it will likely have broad backward incompatibility for all existing virtualenvs? I wouldn't expect someone in operations to read the virtualenv news to find out what things a Python upgrade will break. Indeed, this update will probably be pushed out as part of standard, unattended system updates. I realize that the relationship between stdlib.os and posixmodule isn't a guaranteed interface, and the fact that it breaks with virtualenv is a weakness of virtualenv. Nevertheless, virtualenv has become the defacto technique for Python environments. Putting my sysops cap on, I might perceive this change as being unannounced (w.r.t. Python) and having significant impact on operations. I would think this impact deserves at least a note in the release notes. Regards, Jason -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6662 bytes Desc: not available URL: From victor.stinner at gmail.com Wed Mar 28 23:05:18 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Mar 2012 23:05:18 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: > I would love to have only one new clock function in 3.3. I already added time.clock_gettime() to 3.3 :-) Victor From guido at python.org Wed Mar 28 23:06:05 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Mar 2012 14:06:05 -0700 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: References: Message-ID: +1 If pytz is py3k cabable. -1 for dateutIl. On Wednesday, March 28, 2012, Andrew Svetlov wrote: > I figured out what pytz and dateutil are not mentioned in python docs > for datetime module. > It's clean why these libs is not a part of Python Libraries ? but > that's not clean for Docs. > From my perspective at least pytz (as py3k compatible) should to be > mentioned as the library which contains timezone info, supported > carefully and recommended to use with datetime standard module, > > -- > Thanks, > Andrew Svetlov > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott+python-dev at scottdial.com Wed Mar 28 23:36:06 2012 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 28 Mar 2012 17:36:06 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: <4F738446.9070200@scottdial.com> On 3/28/2012 10:29 AM, Guido van Rossum wrote: > On Wed, Mar 28, 2012 at 7:17 AM, Nick Coghlan wrote: >> Completely unintuitive and unnecessary. With the GIL taking care of >> synchronisation issues, we can easily coerce time.time() into being a >> monotonic clock by the simple expedient of saving the last returned >> value: > > That's a pretty obvious trick. But why don't the kernels do this if > monotonicity is so important? I'm sure there are also downsides, e.g. > if the clock is accidentally set forward by an hour and then back > again, you wouldn't have a useful clock for an hour. And the cache is > not shared between processes so different processes wouldn't see the > same clock value (I presume that most of these clocks have state in > the kernel that isn't bound to any particular process -- AFAIK only > clock() does that, and only on Unixy systems). > What makes you think that isn't already true? I don't know what platforms that CPython compiles for that *won't* have one of the aforementioned functions available that provide a *real* monotonic clock. Surely, any platform that doesn't didn't recognize the need for it, or they would just provide a monotonic clock. That is to say, if you are a POSIX compliant system, then there is no reason to break gettimeofday() and friends when you can just implement CLOCK_MONOTONIC proper (even if it's just a trick like Nick's). I think the PEP should enumerate what platforms that CPython supports that will not benefit from a real monotonic clock. I think the number of platforms will be such a minority that the emulation makes sense. Practicality beats purity, and all. -- Scott Dial scott at scottdial.com From andrew.svetlov at gmail.com Wed Mar 28 23:39:09 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Thu, 29 Mar 2012 00:39:09 +0300 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: References: Message-ID: I'm personally +1 for pytz only ? dateutil is big enough and... Well, can we just point to pytz in our docs for datetime module? On Thu, Mar 29, 2012 at 12:06 AM, Guido van Rossum wrote: > +1 If pytz is py3k cabable. -1 for dateutIl. > > > On Wednesday, March 28, 2012, Andrew Svetlov wrote: >> >> I figured out what pytz and dateutil are not mentioned in python docs >> for datetime module. >> It's clean why these libs is not a part of Python Libraries ? but >> that's not clean for Docs. >> From my perspective at least pytz (as py3k compatible) should to be >> mentioned as the library which contains timezone info, supported >> carefully and recommended to use with datetime standard module, >> >> -- >> Thanks, >> Andrew Svetlov >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) -- Thanks, Andrew Svetlov From victor.stinner at gmail.com Wed Mar 28 23:40:17 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Mar 2012 23:40:17 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: > Does this primarily give a high resolution clock, or primarily a > monotonic clock? That's not clear from either the name, or the PEP. I expect a better resolution from time.monotonic() than time.time(). I don't have exact numbers right now, but I began to document each OS clock in the PEP. Victor From victor.stinner at gmail.com Wed Mar 28 23:45:24 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Mar 2012 23:45:24 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F738446.9070200@scottdial.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <4F738446.9070200@scottdial.com> Message-ID: > I think the PEP should enumerate what platforms that CPython supports > that will not benefit from a real monotonic clock. I think the number of > platforms will be such a minority that the emulation makes sense. > Practicality beats purity, and all. The PEP lists OS monotonic clocks by platform. Windows, Mac OS X, Solaris, and "UNIX" (CLOCK_MONOTONIC & friends) provide monotonic clocks. I don't know any platform without monotonic clock. Victor From rdmurray at bitdance.com Wed Mar 28 23:46:10 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 28 Mar 2012 17:46:10 -0400 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <7E79234E600438479EC119BD241B48D601E7D159@BY2PRD0610MB389.namprd06.prod.outlook.com> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> <4F735CF5.3070700@oddbird.net> <7E79234E600438479EC119BD241B48D601E7D159@BY2PRD0610MB389.namprd06.prod.outlook.com> Message-ID: <20120328214610.E164D2500E9@webabinitio.net> On Wed, 28 Mar 2012 20:56:30 -0000, "Jason R. Coombs" wrote: > Will the release notes include something about this change, since it will > likely have broad backward incompatibility for all existing virtualenvs? I > wouldn't expect someone in operations to read the virtualenv news to find > out what things a Python upgrade will break. Indeed, this update will > probably be pushed out as part of standard, unattended system updates. I think it is reasonable to put something in the release notes. This change is much larger than changes we normally make in maintenance release, because it fixes a security bug. But because it is larger than normal, adding release notes like this about known breakage is, I think, a good idea. Perhaps you and Carl could collaborate on a page explaining the issue in detail, and on a brief note to include in the release notes that points to your more extensive discussion? But keep in mind I'm not the release manager; we'll need their buy in on this. --David From guido at python.org Wed Mar 28 23:50:41 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Mar 2012 14:50:41 -0700 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F738446.9070200@scottdial.com> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <4F738446.9070200@scottdial.com> Message-ID: On Wed, Mar 28, 2012 at 2:36 PM, Scott Dial wrote: > On 3/28/2012 10:29 AM, Guido van Rossum wrote: >> On Wed, Mar 28, 2012 at 7:17 AM, Nick Coghlan wrote: >>> Completely unintuitive and unnecessary. With the GIL taking care of >>> synchronisation issues, we can easily coerce time.time() into being a >>> monotonic clock by the simple expedient of saving the last returned >>> value: >> >> That's a pretty obvious trick. But why don't the kernels do this if >> monotonicity is so important? I'm sure there are also downsides, e.g. >> if the clock is accidentally set forward by an hour and then back >> again, you wouldn't have a useful clock for an hour. And the cache is >> not shared between processes so different processes wouldn't see the >> same clock value (I presume that most of these clocks have state in >> the kernel that isn't bound to any particular process -- AFAIK only >> clock() does that, and only on Unixy systems). >> > > What makes you think that isn't already true? What does "that" refer to in this sentence? > I don't know what > platforms that CPython compiles for that *won't* have one of the > aforementioned functions available that provide a *real* monotonic > clock. Surely, any platform that doesn't didn't recognize the need for > it, or they would just provide a monotonic clock. That is to say, if you > are a POSIX compliant system, then there is no reason to break > gettimeofday() and friends when you can just implement CLOCK_MONOTONIC > proper (even if it's just a trick like Nick's). > > I think the PEP should enumerate what platforms that CPython supports > that will not benefit from a real monotonic clock. I think the number of > platforms will be such a minority that the emulation makes sense. > Practicality beats purity, and all. > > -- > Scott Dial > scott at scottdial.com -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Wed Mar 28 23:55:50 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 28 Mar 2012 17:55:50 -0400 Subject: [Python-Dev] bug tracker offline again for re-indexing Message-ID: <20120328215551.99B452500E9@webabinitio.net> Since Martin hasn't sent a note about this here I will: I noticed that text search wasn't working right on the bug tracker, and Martin has taken it offline again to re-index. --David From martin at v.loewis.de Thu Mar 29 00:06:22 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 29 Mar 2012 00:06:22 +0200 Subject: [Python-Dev] bug tracker offline again for re-indexing In-Reply-To: <20120328215551.99B452500E9@webabinitio.net> References: <20120328215551.99B452500E9@webabinitio.net> Message-ID: <4F738B5E.4010100@v.loewis.de> Am 28.03.2012 23:55, schrieb R. David Murray: > Since Martin hasn't sent a note about this here I will: > > I noticed that text search wasn't working right on the bug tracker, and Martin > has taken it offline again to re-index. which will, unfortunately, take a few more hours to complete. Regards, Martin From victor.stinner at gmail.com Thu Mar 29 01:27:44 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 29 Mar 2012 01:27:44 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> Message-ID: > Where in the stdlib do we actually calculate timeouts instead of using > the timeouts built into the OS (e.g. select())? At least in threading and queue modules. The common use case is to retry a function with a timeout if the syscall was interrupted by a signal (EINTR error). The socket module and _threading.Lock.acquire() implement such retry loop using the system clock. They should use a monotonic clock instead. > I think it would be nice if we could somehow use the *same* clock as > the OS uses to implement timeouts. On Linux, nanosleep() uses CLOCK_MONOTONIC whereas POSIX suggests CLOCK_REALTIME. Some functions allow to choose the clock, like pthread locks or clock_nanosleep(). > I'd be happier if the fallback function didn't try to guarantee things > the underlying clock can't guarantee. I.e. I like the idea of having a > function that uses some accurate OS clock if one exists but falls back > to time.time() if not; I don't like the idea of that new function > trying to interpret the value of time.time() in any way. We may workaround some OS known bugs like: http://support.microsoft.com/?id=274323 The link contains an example how to workaround the bug. The idea of the workaround is to use two different monotonic clocks to detect leaps, with one trusted clock (GetTickCount) and one untrusted clock having an higher resolution (QueryPerformanceCounter). I don't think that the same algorithm is applicable on other OSes because other OSes usually only provide one monotonic clock, sometimes though different API. Victor From victor.stinner at gmail.com Thu Mar 29 02:14:00 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 29 Mar 2012 02:14:00 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: > Could you (with help from others who contributed) try to compile a table > showing, for each platform (Windows/Mac/Linux/BSD) which clocks (or > variations) we are considering, and for each of those: > > - a link for the reference documentation > - what their typical accuracy is (barring jumps) > - what they do when the "civil" time is made to jump (forward or back) > by the user > - how they are affected by small tweaks to the civil time by NTP > - what they do if the system is suspended and resumed > - whether they can be shared between processes running on the same machine > - whether they may fail or be unsupported under some circumstances > > I have a feeling that if I saw such a table it would be much easier to decide. > > I assume much of this has already been said at one point in this > thread, but it's impossible to have an overview at the moment. I don't know where I can get all these information, but I'm completing the PEP each time that I find a new information. It's difficult to get the accuracy of a clock and how it handles system suspend. I'm intereted if anyone has such information. Victor From martin at v.loewis.de Thu Mar 29 05:07:48 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 29 Mar 2012 05:07:48 +0200 Subject: [Python-Dev] bug tracker offline again for re-indexing In-Reply-To: <4F738B5E.4010100@v.loewis.de> References: <20120328215551.99B452500E9@webabinitio.net> <4F738B5E.4010100@v.loewis.de> Message-ID: <4F73D204.2000906@v.loewis.de> >> I noticed that text search wasn't working right on the bug tracker, and Martin >> has taken it offline again to re-index. > > which will, unfortunately, take a few more hours to complete. It seems to work now, so I turned it on again. Text search now uses Xapian, and recreating the Xapian index of all msg objects took a while. Regards, Martin From tshepang at gmail.com Thu Mar 29 09:33:42 2012 From: tshepang at gmail.com (Tshepang Lekhonkhobe) Date: Thu, 29 Mar 2012 09:33:42 +0200 Subject: [Python-Dev] Bug tracker outage In-Reply-To: <4F734E9D.4050607@v.loewis.de> References: <4F722CEB.1040102@v.loewis.de> <4F734E9D.4050607@v.loewis.de> Message-ID: On Wed, Mar 28, 2012 at 19:47, "Martin v. L?wis" wrote: > Am 27.03.2012 23:11, schrieb "Martin v. L?wis": >> Upfront hosting (Izak Burger) is going to do a Debian upgrade of the bug >> tracker machine "soon" (likely tomorrow). This may cause some outage, >> since there is a lot of custom stuff on the machine which may >> break with newer (Python) versions. I'll notify here when the upgrade >> is complete. > > The upgrade is complete. It was in fact the Postgres upgrade (to 8.4) > which caused the longest blackout time. Curiously, did the update have anything to do with Debian 5's EOL? From jbschne at umich.edu Wed Mar 28 09:06:40 2012 From: jbschne at umich.edu (Jordan Schneider) Date: Wed, 28 Mar 2012 03:06:40 -0400 Subject: [Python-Dev] possible distutils.sysconfig.get_config_var bug Message-ID: <49B4A6A0-2225-49C6-82B7-B284D626A415@umich.edu> Hi python-dev, Sorry if this is the wrong place to discuss this potential bug - feel free to point me in the right direction if so. I'm running OS X 10.7.3, and have two python2.7s installed, one system default in /usr/bin and the other by homebrew symlinked in /usr/local/bin. While running a configure script, distutils.sysconfig.get_config_var is not returing a full path for my homebrew Python framework, like so: Python 2.7.2 (default, Mar 28 2012, 02:31:16) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on darwin >>> distutils.sysconfig.get_config_var('LINKFORSHARED') '-u _PyMac_Error Python.framework/Versions/2.7/Python' (the full path should be /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/Python) Whereas from the system python in /usr/bin: Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin >>> distutils.sysconfig.get_config_var('LINKFORSHARED') '-u _PyMac_Error /System/Library/Frameworks/Python.framework/Versions/2.7/Python' This seems like a bug, no? Thanks for the help, Jordan From jaraco at jaraco.com Thu Mar 29 13:41:46 2012 From: jaraco at jaraco.com (Jason R. Coombs) Date: Thu, 29 Mar 2012 11:41:46 +0000 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <20120328214610.E164D2500E9@webabinitio.net> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> <4F735CF5.3070700@oddbird.net> <7E79234E600438479EC119BD241B48D601E7D159@BY2PRD0610MB389.namprd06.prod.outlook.com> <20120328214610.E164D2500E9@webabinitio.net> Message-ID: <7E79234E600438479EC119BD241B48D601E7D9E1@BY2PRD0610MB389.namprd06.prod.outlook.com> Carl, I've drafted some notes: http://piratepad.net/PAZ3CEq9CZ Please feel free to edit them. If you want to chat, I can often be reached on freenode as 'jaraco' or XMPP at my e-mail address if you want to sprint on this in real-time. Does the issue only exist for Python 2.6 and 2.7? I'm not familiar with the release process. What's the next step? > -----Original Message----- > From: R. David Murray [mailto:rdmurray at bitdance.com] > Sent: Wednesday, 28 March, 2012 17:46 > > I think it is reasonable to put something in the release notes. This change is > much larger than changes we normally make in maintenance release, > because it fixes a security bug. But because it is larger than normal, adding > release notes like this about known breakage is, I think, a good idea. > > Perhaps you and Carl could collaborate on a page explaining the issue in > detail, and on a brief note to include in the release notes that points to your > more extensive discussion? -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6662 bytes Desc: not available URL: From regebro at gmail.com Thu Mar 29 14:11:13 2012 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 29 Mar 2012 14:11:13 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: On Wed, Mar 28, 2012 at 23:40, Victor Stinner wrote: >> Does this primarily give a high resolution clock, or primarily a >> monotonic clock? That's not clear from either the name, or the PEP. > > I expect a better resolution from time.monotonic() than time.time(). Sure. And for me that means that time.hires() would give a high resolution version of time.time(). Ie, not monotonic, but wall clock. The question then is why time.time() doesn't give that resolution from the start. It seems to me we need three functions: One to get the wall clock, one to get a monotonic clock, and one that falls back if no monotonic clock is available. Both time.time() and time.monotonic() should give the highest resolution possible. As such, time.hires() seems pointless. //Lennart From rdmurray at bitdance.com Thu Mar 29 15:31:39 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Mar 2012 09:31:39 -0400 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <7E79234E600438479EC119BD241B48D601E7D9E1@BY2PRD0610MB389.namprd06.prod.outlook.com> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> <4F735CF5.3070700@oddbird.net> <7E79234E600438479EC119BD241B48D601E7D159@BY2PRD0610MB389.namprd06.prod.outlook.com> <20120328214610.E164D2500E9@webabinitio.net> <7E79234E600438479EC119BD241B48D601E7D9E1@BY2PRD0610MB389.namprd06.prod.outlook.com> Message-ID: <20120329133139.9D7452500E9@webabinitio.net> On Thu, 29 Mar 2012 11:41:46 -0000, "Jason R. Coombs" wrote: > Does the issue only exist for Python 2.6 and 2.7? It might exist for 3.1 and 3.2 as well. > I'm not familiar with the release process. What's the next step? I would suggest opening an issue on the tracker and marking it as a release-blocker. That way the release managers will see it and can decide what if anything they want to do about it. --David From brett at python.org Thu Mar 29 16:31:20 2012 From: brett at python.org (Brett Cannon) Date: Thu, 29 Mar 2012 10:31:20 -0400 Subject: [Python-Dev] possible distutils.sysconfig.get_config_var bug In-Reply-To: <49B4A6A0-2225-49C6-82B7-B284D626A415@umich.edu> References: <49B4A6A0-2225-49C6-82B7-B284D626A415@umich.edu> Message-ID: If you could, Jordan, please file a bug at bugs.python.org so the discussion can happen there and be tracked better. On Wed, Mar 28, 2012 at 03:06, Jordan Schneider wrote: > Hi python-dev, > > Sorry if this is the wrong place to discuss this potential bug - feel free > to point me in the right direction if so. > > I'm running OS X 10.7.3, and have two python2.7s installed, one system > default in /usr/bin and the other by homebrew symlinked in /usr/local/bin. > > While running a configure script, distutils.sysconfig.get_config_var is > not returing a full path for my homebrew Python framework, like so: > > Python 2.7.2 (default, Mar 28 2012, 02:31:16) > [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on > darwin > >>> distutils.sysconfig.get_config_var('LINKFORSHARED') > '-u _PyMac_Error Python.framework/Versions/2.7/Python' > > (the full path should be > /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/Python) > > > Whereas from the system python in /usr/bin: > > Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) > [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on > darwin > >>> distutils.sysconfig.get_config_var('LINKFORSHARED') > '-u _PyMac_Error > /System/Library/Frameworks/Python.framework/Versions/2.7/Python' > > This seems like a bug, no? > > Thanks for the help, > Jordan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosslagerwall at gmail.com Thu Mar 29 18:21:34 2012 From: rosslagerwall at gmail.com (Ross Lagerwall) Date: Thu, 29 Mar 2012 18:21:34 +0200 Subject: [Python-Dev] bug tracker offline again for re-indexing In-Reply-To: <4F73D204.2000906@v.loewis.de> References: <20120328215551.99B452500E9@webabinitio.net> <4F738B5E.4010100@v.loewis.de> <4F73D204.2000906@v.loewis.de> Message-ID: <4F748C0E.4080107@gmail.com> On 03/29/2012 05:07 AM, "Martin v. L?wis" wrote: >>> I noticed that text search wasn't working right on the bug tracker, and Martin >>> has taken it offline again to re-index. >> >> which will, unfortunately, take a few more hours to complete. > > It seems to work now, so I turned it on again. Text search now uses > Xapian, and recreating the Xapian index of all msg objects took a while. > Is the search working properly: Search all for "virtualenv" at the top right and issue 14357 doesn't appear in the list of results... Also, search open issues for "subprocess" and there is only 1 result. I wish :-) -- Ross Lagerwall -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 294 bytes Desc: OpenPGP digital signature URL: From rdmurray at bitdance.com Thu Mar 29 18:39:18 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Mar 2012 12:39:18 -0400 Subject: [Python-Dev] bug tracker offline again for re-indexing In-Reply-To: <4F748C0E.4080107@gmail.com> References: <20120328215551.99B452500E9@webabinitio.net> <4F738B5E.4010100@v.loewis.de> <4F73D204.2000906@v.loewis.de> <4F748C0E.4080107@gmail.com> Message-ID: <20120329163918.B7EA12500E9@webabinitio.net> On Thu, 29 Mar 2012 18:21:34 +0200, Ross Lagerwall wrote: > On 03/29/2012 05:07 AM, "Martin v. L??wis" wrote: > >>> I noticed that text search wasn't working right on the bug tracker, and Martin > >>> has taken it offline again to re-index. > >> > >> which will, unfortunately, take a few more hours to complete. > > > > It seems to work now, so I turned it on again. Text search now uses > > Xapian, and recreating the Xapian index of all msg objects took a while. > > > > Is the search working properly: Search all for "virtualenv" at the top > right and issue 14357 doesn't appear in the list of results... > > Also, search open issues for "subprocess" and there is only 1 result. I > wish :-) I get three for 'all' issues, which is certainly wrong. All of them have subprocess in the title. I suspect the search is only searching the title field, which is wrong. --David From carl at oddbird.net Thu Mar 29 18:54:17 2012 From: carl at oddbird.net (Carl Meyer) Date: Thu, 29 Mar 2012 10:54:17 -0600 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <7E79234E600438479EC119BD241B48D601E7D9E1@BY2PRD0610MB389.namprd06.prod.outlook.com> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> <4F735CF5.3070700@oddbird.net> <7E79234E600438479EC119BD241B48D601E7D159@BY2PRD0610MB389.namprd06.prod.outlook.com> <20120328214610.E164D2500E9@webabinitio.net> <7E79234E600438479EC119BD241B48D601E7D9E1@BY2PRD0610MB389.namprd06.prod.outlook.com> Message-ID: <4F7493B9.5070709@oddbird.net> Thanks Jason for raising this. I just assumed that this was virtualenv's fault (which it is) and didn't consider raising it here, but a note in the release notes for the affected Python versions will certainly reach many more of the likely-to-be-affected users. FTR, I confirmed that the issue also affects the upcoming point releases for 3.1 and 3.2, as well as 2.6 and 2.7. Jason filed issue 14444 to track the addition to the release notes for those versions. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From dmalcolm at redhat.com Thu Mar 29 19:39:27 2012 From: dmalcolm at redhat.com (David Malcolm) Date: Thu, 29 Mar 2012 13:39:27 -0400 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> Message-ID: <1333042767.31165.47.camel@surprise> On Wed, 2012-03-28 at 18:22 +0000, Jason R. Coombs wrote: > I see this was reported as a debian bug. > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=665776 > > To reproduce, using virtualenv 1.7+ on Python 2.7.2 on Ubuntu, create > a virtualenv. Move that virtualenv to a host with Python 2.7.3RC2 > yields: > jaraco at vdm-dev:~$ /usr/bin/python2.7 -V > Python 2.7.3rc2 > jaraco at vdm-dev:~$ env/bin/python -V > Python 2.7.2 > jaraco at vdm-dev:~$ env/bin/python -c "import os; os.urandom()" > Traceback (most recent call last): > File " ", line 1, in > AttributeError: 'module' object has no attribute 'urandom' It looks like this a symptom of the move of urandom to os.py to posximodule et al. At first glance, it looks like this specific hunk should be reverted: http://hg.python.org/cpython/rev/a0f43f4481e0#l7.1 so that if you're running with the new stdlib but an old python binary the combination can still have a usable os.urandom Should this be tracked in bugs.python.org? Hope this is helpful Dave From rosslagerwall at gmail.com Thu Mar 29 20:13:51 2012 From: rosslagerwall at gmail.com (Ross Lagerwall) Date: Thu, 29 Mar 2012 20:13:51 +0200 Subject: [Python-Dev] bug tracker offline again for re-indexing In-Reply-To: <20120329163918.B7EA12500E9@webabinitio.net> References: <20120328215551.99B452500E9@webabinitio.net> <4F738B5E.4010100@v.loewis.de> <4F73D204.2000906@v.loewis.de> <4F748C0E.4080107@gmail.com> <20120329163918.B7EA12500E9@webabinitio.net> Message-ID: <4F74A65F.1020806@gmail.com> On 03/29/2012 06:39 PM, R. David Murray wrote: > I get three for 'all' issues, which is certainly wrong. All of them > have subprocess in the title. > > I suspect the search is only searching the title field, which is wrong. No, #14357 contains 'virtualenv' in the title but it doesn't come up in the 'all' search. -- Ross Lagerwall -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 294 bytes Desc: OpenPGP digital signature URL: From carl at oddbird.net Thu Mar 29 20:26:27 2012 From: carl at oddbird.net (Carl Meyer) Date: Thu, 29 Mar 2012 12:26:27 -0600 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <1333042767.31165.47.camel@surprise> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> <1333042767.31165.47.camel@surprise> Message-ID: <4F74A953.2090909@oddbird.net> On 03/29/2012 11:39 AM, David Malcolm wrote: >> jaraco at vdm-dev:~$ env/bin/python -c "import os; os.urandom()" >> Traceback (most recent call last): >> File " ", line 1, in >> AttributeError: 'module' object has no attribute 'urandom' > > It looks like this a symptom of the move of urandom to os.py to > posximodule et al. > > At first glance, it looks like this specific hunk should be reverted: > http://hg.python.org/cpython/rev/a0f43f4481e0#l7.1 > so that if you're running with the new stdlib but an old python binary > the combination can still have a usable os.urandom Indeed, I've just tested and verified that this does fix the problem. > Should this be tracked in bugs.python.org? I've added this option as a comment on bug 14444. The title of that bug is worded such that it could be reasonably resolved either with the backwards-compatibility fix or the release notes addition, the release managers can decide what seems appropriate to them. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From martin at v.loewis.de Thu Mar 29 21:14:32 2012 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 29 Mar 2012 21:14:32 +0200 Subject: [Python-Dev] bug tracker offline again for re-indexing In-Reply-To: <4F748C0E.4080107@gmail.com> References: <20120328215551.99B452500E9@webabinitio.net> <4F738B5E.4010100@v.loewis.de> <4F73D204.2000906@v.loewis.de> <4F748C0E.4080107@gmail.com> Message-ID: <4F74B498.6010805@v.loewis.de> Am 29.03.2012 18:21, schrieb Ross Lagerwall: > On 03/29/2012 05:07 AM, "Martin v. L?wis" wrote: >>>> I noticed that text search wasn't working right on the bug tracker, and Martin >>>> has taken it offline again to re-index. >>> >>> which will, unfortunately, take a few more hours to complete. >> >> It seems to work now, so I turned it on again. Text search now uses >> Xapian, and recreating the Xapian index of all msg objects took a while. >> > > Is the search working properly: Search all for "virtualenv" at the top > right and issue 14357 doesn't appear in the list of results... > > Also, search open issues for "subprocess" and there is only 1 result. I > wish :-) Please submit an issue to the meta tracker. It may take weeks until I can get to it. Regards, Martin From tlesher at gmail.com Thu Mar 29 21:35:27 2012 From: tlesher at gmail.com (Tim Lesher) Date: Thu, 29 Mar 2012 15:35:27 -0400 Subject: [Python-Dev] Bug in generator if the generator in created in a C thread In-Reply-To: References: Message-ID: On Wed, Mar 28, 2012 at 06:45, Victor Stinner wrote: > We have a crash in our product when tracing is enabled by > sys.settrace() and threading.settrace(). If a Python generator is > created in a C thread, calling the generator later in another thread > may crash if Python tracing is enabled. > > ?- the C thread calls PyGILState_Ensure() which creates a temporary > Python thread state > ?- a generator is created, the generator has a reference to a Python > frame which keeps a reference to the temporary Python thread state > ?- the C thread calls PyGILState_Releases() which destroys the > temporary Python thread state > ?- when the generator is called later in another thread, call_trace() > reads the Python thread state from the generator frame, which is the > destroyed frame => it does crash on a pointer dereference if the > memory was reallocated (by malloc()) and the data were erased > This is timely. We've seen several similar bugs in our 3.1.2-based embedded product--in fact, I'm tracking down what might be another this week. The common theme is that C extension code that depends on PyGILState_Ensure() for GIL management runs the risk of causing latent crashes in any situation that involves preserving a frame beyond the lifetime of the thread that created it. In our case, the culprits have usually been the frames attached to exceptions thrown across the extension->Python->embedding app boundaries. >From a theoretical standpoint, I can't quite decide what the real error is: 1) the fact that PyGILState_Release() destroys a temporary thread state that may still be referenced by some objects, or 2) the fact that some code is trying to keep frame objects after the creating thread state no longer exists. This week I've been leaning toward 2), but then I realized that keeping frames post-thread-death is not that uncommon (for example, debuggers and other diagnostic techniques like http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html). Locally we added some (unfortunate) code to our 3.1.2 port to wrap PyGILState_Ensure(), which I thought had sidestepped the issue for us: void takeGIL() { PyGILState_Ensure(); // This has the side effect of keeping such thread states alive until // the interpreter is finalized; however, all thread state objects get // unconditionally deleted during Py_Finalize, so they won't leak. PyThreadState* pThreadState = PyGILState_GetThisThreadState(); if (pThreadState->gilstate_counter == 1) { ++pThreadState->gilstate_counter; } } But clearly that can't be a correct answer (and it may not even be a functioning one, given that I'm seeing a similar issue again). -- Tim Lesher From rdmurray at bitdance.com Thu Mar 29 21:58:25 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Mar 2012 15:58:25 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error Message-ID: <20120329195825.843352500E9@webabinitio.net> Some of us have expressed uneasiness about the consequences of dict raising an error on lookup if the dict has been modified, the fix Victor made to solve one of the crashers. I don't know if I speak for the others, but (assuming that I understand the change correctly) my concern is that there is probably a significant amount of threading code out there that assumes that dict *lookup* is a thread-safe operation. Much of that code will, if moved to Python 3.3, now be subject to random runtime errors for which it will not be prepared. Further, code which appears safe can suddenly become unsafe if a refactoring of the code causes an object to be stored in the dictionary that has a Python equality method. Would it be possible to modify the fix so that the lookup is retried a non-trivial but finite number of times, so that normal code will work and only pathological code will break? I know that I really don't want to think about having to audit the (significantly threaded) application I'm currently working on to make sure it is "3.3 safe". Dict lookup operations are *common*, and we've never had to think about whether or not they were thread-safe before (unless there were inter-thread synchronization issues involved, of course). Nor am I sure the locking dict type suggested by Jim on the issue would help, since a number of the dicts we are using are produced by library code. So we'd have to wait for those libraries to be ported to 3.3.... --David From rdmurray at bitdance.com Thu Mar 29 22:06:30 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Mar 2012 16:06:30 -0400 Subject: [Python-Dev] bug tracker offline again for re-indexing In-Reply-To: <4F74B498.6010805@v.loewis.de> References: <20120328215551.99B452500E9@webabinitio.net> <4F738B5E.4010100@v.loewis.de> <4F73D204.2000906@v.loewis.de> <4F748C0E.4080107@gmail.com> <4F74B498.6010805@v.loewis.de> Message-ID: <20120329200631.6D7872500E9@webabinitio.net> On Thu, 29 Mar 2012 21:14:32 +0200, =?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > Am 29.03.2012 18:21, schrieb Ross Lagerwall: > > On 03/29/2012 05:07 AM, "Martin v. L??wis" wrote: > >>>> I noticed that text search wasn't working right on the bug tracker, and Martin > >>>> has taken it offline again to re-index. > >>> > >>> which will, unfortunately, take a few more hours to complete. > >> > >> It seems to work now, so I turned it on again. Text search now uses > >> Xapian, and recreating the Xapian index of all msg objects took a while. > >> > > > > Is the search working properly: Search all for "virtualenv" at the top > > right and issue 14357 doesn't appear in the list of results... > > > > Also, search open issues for "subprocess" and there is only 1 result. I > > wish :-) > > Please submit an issue to the meta tracker. It may take weeks until I > can get to it. I've reopened issue 443, with an example of a failing search giving relevant issue numbers. --David From guido at python.org Thu Mar 29 22:09:17 2012 From: guido at python.org (Guido van Rossum) Date: Thu, 29 Mar 2012 13:09:17 -0700 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <20120329195825.843352500E9@webabinitio.net> References: <20120329195825.843352500E9@webabinitio.net> Message-ID: On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: > Some of us have expressed uneasiness about the consequences of dict > raising an error on lookup if the dict has been modified, the fix Victor > made to solve one of the crashers. > > I don't know if I speak for the others, but (assuming that I understand > the change correctly) my concern is that there is probably a significant > amount of threading code out there that assumes that dict *lookup* is > a thread-safe operation. ?Much of that code will, if moved to Python > 3.3, now be subject to random runtime errors for which it will not > be prepared. ?Further, code which appears safe can suddenly become > unsafe if a refactoring of the code causes an object to be stored in > the dictionary that has a Python equality method. My original assessment was that this only affects dicts whose keys have a user-implemented __hash__ or __eq__ implementation, and that the number of apps that use this *and* assume the threadsafe property would be pretty small. This is just intuition, I don't have hard facts. But I do want to stress that not all dict lookups automatically become thread-unsafe, only those that need to run user code as part of the key lookup. > Would it be possible to modify the fix so that the lookup is retried a > non-trivial but finite number of times, so that normal code will work > and only pathological code will break? FWIW a similar approach was rejected as a fix for the hash DoS attack. > I know that I really don't want to think about having to audit the > (significantly threaded) application I'm currently working on to make sure > it is "3.3 safe". ?Dict lookup operations are *common*, and we've never > had to think about whether or not they were thread-safe before (unless > there were inter-thread synchronization issues involved, of course). > Nor am I sure the locking dict type suggested by Jim on the issue would > help, since a number of the dicts we are using are produced by library > code. ?So we'd have to wait for those libraries to be ported to 3.3.... Agreed that this is somewhat scary. -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Thu Mar 29 22:31:03 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Mar 2012 16:31:03 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> Message-ID: <20120329203103.95A4B2500E9@webabinitio.net> On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: > On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: > > Some of us have expressed uneasiness about the consequences of dict > > raising an error on lookup if the dict has been modified, the fix Victor > > made to solve one of the crashers. > > > > I don't know if I speak for the others, but (assuming that I understand > > the change correctly) my concern is that there is probably a significant > > amount of threading code out there that assumes that dict *lookup* is > > a thread-safe operation. ??Much of that code will, if moved to Python > > 3.3, now be subject to random runtime errors for which it will not > > be prepared. ??Further, code which appears safe can suddenly become > > unsafe if a refactoring of the code causes an object to be stored in > > the dictionary that has a Python equality method. > > My original assessment was that this only affects dicts whose keys > have a user-implemented __hash__ or __eq__ implementation, and that > the number of apps that use this *and* assume the threadsafe property > would be pretty small. This is just intuition, I don't have hard > facts. But I do want to stress that not all dict lookups automatically > become thread-unsafe, only those that need to run user code as part of > the key lookup. You are probably correct, but the thing is that one still has to do the code audit to be sure...and then make sure that no one later introduces such an object type as a dict key. Are there any other places in Python where substituting a duck-typed Python class or a Python subclass can cause a runtime error in previously working code? > > Would it be possible to modify the fix so that the lookup is retried a > > non-trivial but finite number of times, so that normal code will work > > and only pathological code will break? > > FWIW a similar approach was rejected as a fix for the hash DoS attack. Yes, but in this case the non-counting version breaks just as randomly, but more often. So arguing that counting here is analogous to counting in the DoS attack issue is an argument for removing the fix entirely :) The counting version could use a large enough count (since the count used to be infinite!) that only code that would be having pathological performance anyway would raise the runtime error, rather than any code (that uses python __eq__ on keys) randomly raising a runtime error, which is what we have now. --David From rdmurray at bitdance.com Thu Mar 29 22:48:14 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Mar 2012 16:48:14 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <20120329203103.95A4B2500E9@webabinitio.net> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> Message-ID: <20120329204815.D7AC32500E9@webabinitio.net> On Thu, 29 Mar 2012 16:31:03 -0400, "R. David Murray" wrote: > On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: > > My original assessment was that this only affects dicts whose keys > > have a user-implemented __hash__ or __eq__ implementation, and that > > the number of apps that use this *and* assume the threadsafe property > > would be pretty small. This is just intuition, I don't have hard > > facts. But I do want to stress that not all dict lookups automatically > > become thread-unsafe, only those that need to run user code as part of > > the key lookup. > > You are probably correct, but the thing is that one still has to do the > code audit to be sure...and then make sure that no one later introduces > such an object type as a dict key. I just did a quick grep on our project. We are only defining __eq__ and __hash__ a couple places, but both are objects that could easily get used as dict keys (there is a good chance that's *why* those methods are defined) accessed by more than one thread. I haven't done the audit to find out :) The libraries we depend on have many more definitions of __eq__ and __hash__, and we'd have to check them too. (Including SQLAlchemy, and I wouldn't want that job.) So our intuition that this is not common may be wrong. --David From stefan_ml at behnel.de Thu Mar 29 23:00:20 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 29 Mar 2012 23:00:20 +0200 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <20120329203103.95A4B2500E9@webabinitio.net> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> Message-ID: R. David Murray, 29.03.2012 22:31: > On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: >> On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: >>> Some of us have expressed uneasiness about the consequences of dict >>> raising an error on lookup if the dict has been modified, the fix Victor >>> made to solve one of the crashers. >>> >>> I don't know if I speak for the others, but (assuming that I understand >>> the change correctly) my concern is that there is probably a significant >>> amount of threading code out there that assumes that dict *lookup* is >>> a thread-safe operation. Much of that code will, if moved to Python >>> 3.3, now be subject to random runtime errors for which it will not >>> be prepared. Further, code which appears safe can suddenly become >>> unsafe if a refactoring of the code causes an object to be stored in >>> the dictionary that has a Python equality method. >> >> My original assessment was that this only affects dicts whose keys >> have a user-implemented __hash__ or __eq__ implementation, and that >> the number of apps that use this *and* assume the threadsafe property >> would be pretty small. This is just intuition, I don't have hard >> facts. But I do want to stress that not all dict lookups automatically >> become thread-unsafe, only those that need to run user code as part of >> the key lookup. > > You are probably correct, but the thing is that one still has to do the > code audit to be sure...and then make sure that no one later introduces > such an object type as a dict key. The thing is: the assumption that arbitrary dict lookups are GIL-atomic has *always* been false. Only those that do not involve Python code execution for the hash key calculation or the object comparison are. That includes the built-in strings and numbers (and tuples of them), which are by far the most common dict keys. Looking up arbitrary user provided objects is definitely not guaranteed to be atomic. Stefan From animelovin at gmail.com Thu Mar 29 23:03:17 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Thu, 29 Mar 2012 17:03:17 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <20120329204815.D7AC32500E9@webabinitio.net> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329204815.D7AC32500E9@webabinitio.net> Message-ID: <4F74CE15.3040204@gmail.com> On 03/29/2012 04:48 PM, R. David Murray wrote: > On Thu, 29 Mar 2012 16:31:03 -0400, "R. David Murray" wrote: >> On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: >>> My original assessment was that this only affects dicts whose keys >>> have a user-implemented __hash__ or __eq__ implementation, and that >>> the number of apps that use this *and* assume the threadsafe property >>> would be pretty small. This is just intuition, I don't have hard >>> facts. But I do want to stress that not all dict lookups automatically >>> become thread-unsafe, only those that need to run user code as part of >>> the key lookup. >> >> You are probably correct, but the thing is that one still has to do the >> code audit to be sure...and then make sure that no one later introduces >> such an object type as a dict key. > > I just did a quick grep on our project. We are only defining __eq__ > and __hash__ a couple places, but both are objects that could easily get > used as dict keys (there is a good chance that's *why* those methods are > defined) accessed by more than one thread. I haven't done the audit to > find out :) > > The libraries we depend on have many more definitions of __eq__ and > __hash__, and we'd have to check them too. (Including SQLAlchemy, > and I wouldn't want that job.) > > So our intuition that this is not common may be wrong. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/animelovin%40gmail.com > Hm, as far I understand this seems like an issue for gnu PTH, not python job, which should transparently handles thread safety issues based on the host/machine capabilities. Therefore I hope the fix in python don't affect thread-unsafe apps to raise spurious RuntimeErrors when a dict get modified across a SMP-aware platform... :-) From victor.stinner at gmail.com Thu Mar 29 23:26:42 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 29 Mar 2012 23:26:42 +0200 Subject: [Python-Dev] Bug in generator if the generator in created in a C thread In-Reply-To: References: Message-ID: <4F74D392.2040808@gmail.com> On 29/03/2012 21:35, Tim Lesher wrote: > From a theoretical standpoint, I can't quite decide what the real error is: > > 1) the fact that PyGILState_Release() destroys a temporary thread > state that may still be referenced by some objects, or > 2) the fact that some code is trying to keep frame objects after the > creating thread state no longer exists. > > This week I've been leaning toward 2), but then I realized that > keeping frames post-thread-death is not that uncommon (for example, > debuggers and other diagnostic techniques like > http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html). The problem is not the frame, but the Python thread state referenced by the frame. It's a private attribute. My patch just updates this reference before running the generator (and it clears the reference when the generator excution is stopped or finished). > Locally we added some (unfortunate) code to our 3.1.2 port to wrap > PyGILState_Ensure(), which I thought had sidestepped the issue for us: > > void takeGIL() > { > PyGILState_Ensure(); > // This has the side effect of keeping such thread states alive until > // the interpreter is finalized; however, all thread state objects get > // unconditionally deleted during Py_Finalize, so they won't leak. > PyThreadState* pThreadState = PyGILState_GetThisThreadState(); > if (pThreadState->gilstate_counter == 1) > { > ++pThreadState->gilstate_counter; > } > } > > But clearly that can't be a correct answer (and it may not even be a > functioning one, given that I'm seeing a similar issue again). You may leak memory if your threads have a short lifetime and you create many threads. For example if one thread is only used to process one request and then is destroyed. Victor From rdmurray at bitdance.com Fri Mar 30 00:07:54 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Mar 2012 18:07:54 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> Message-ID: <20120329220755.377052500E9@webabinitio.net> On Thu, 29 Mar 2012 23:00:20 +0200, Stefan Behnel wrote: > R. David Murray, 29.03.2012 22:31: > > On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: > >> On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: > >>> Some of us have expressed uneasiness about the consequences of dict > >>> raising an error on lookup if the dict has been modified, the fix Victor > >>> made to solve one of the crashers. > >>> > >>> I don't know if I speak for the others, but (assuming that I understand > >>> the change correctly) my concern is that there is probably a significant > >>> amount of threading code out there that assumes that dict *lookup* is > >>> a thread-safe operation. Much of that code will, if moved to Python > >>> 3.3, now be subject to random runtime errors for which it will not > >>> be prepared. Further, code which appears safe can suddenly become > >>> unsafe if a refactoring of the code causes an object to be stored in > >>> the dictionary that has a Python equality method. > >> > >> My original assessment was that this only affects dicts whose keys > >> have a user-implemented __hash__ or __eq__ implementation, and that > >> the number of apps that use this *and* assume the threadsafe property > >> would be pretty small. This is just intuition, I don't have hard > >> facts. But I do want to stress that not all dict lookups automatically > >> become thread-unsafe, only those that need to run user code as part of > >> the key lookup. > > > > You are probably correct, but the thing is that one still has to do the > > code audit to be sure...and then make sure that no one later introduces > > such an object type as a dict key. > > The thing is: the assumption that arbitrary dict lookups are GIL-atomic has > *always* been false. Only those that do not involve Python code execution > for the hash key calculation or the object comparison are. That includes > the built-in strings and numbers (and tuples of them), which are by far the > most common dict keys. Looking up arbitrary user provided objects is > definitely not guaranteed to be atomic. Well, I'm afraid I was using the term 'thread safety' rather too loosely there. What I mean is that if you do a dict lookup, the lookup either returns a value or a KeyError, and that if you get back an object that object has internally consistent state. The problem this fix introduces is that the lookup may fail with a RuntimeError rather than a KeyError, which it has never done before. I think that is what Guido means by code that uses objects with python eq/hash *and* assumes threadsafe lookup. If mutation of the objects or dict during the lookup is a concern, then the code would use locks and wouldn't have the problem. But there are certainly situations where it doesn't matter if the dictionary mutates during the lookup, as long as you get either an object or a KeyError, and thus no locks are (currently) needed. Maybe I'm being paranoid about breakage here, but as with most backward compatibility concerns, there are probably more bits of code that will be affected than our intuition indicates. --David From brian at python.org Fri Mar 30 00:39:27 2012 From: brian at python.org (Brian Curtin) Date: Thu, 29 Mar 2012 17:39:27 -0500 Subject: [Python-Dev] Integrating the PEP 397 launcher Message-ID: After talking with Martin and several others during the language summit and elsewhere around PyCon, PEP 397 should be accepted. I don't remember who, but some suggested it should just be a regular old feature instead of going through the PEP process. So...does this even need to continue the PEP process? Vinay - is the code you have on bitbucket ready to roll into CPython, thus into the installer? From benjamin at python.org Fri Mar 30 00:45:10 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 29 Mar 2012 18:45:10 -0400 Subject: [Python-Dev] Integrating the PEP 397 launcher In-Reply-To: References: Message-ID: 2012/3/29 Brian Curtin : > After talking with Martin and several others during the language > summit and elsewhere around PyCon, PEP 397 should be accepted. I don't > remember who, but some suggested it should just be a regular old > feature instead of going through the PEP process. So...does this even > need to continue the PEP process? If you have a PEP and it's accepted, what would be the difference? -- Regards, Benjamin From brian at python.org Fri Mar 30 00:50:01 2012 From: brian at python.org (Brian Curtin) Date: Thu, 29 Mar 2012 17:50:01 -0500 Subject: [Python-Dev] Integrating the PEP 397 launcher In-Reply-To: References: Message-ID: On Thu, Mar 29, 2012 at 17:45, Benjamin Peterson wrote: > 2012/3/29 Brian Curtin : >> After talking with Martin and several others during the language >> summit and elsewhere around PyCon, PEP 397 should be accepted. I don't >> remember who, but some suggested it should just be a regular old >> feature instead of going through the PEP process. So...does this even >> need to continue the PEP process? > > If you have a PEP and it's accepted, what would be the difference? In the end? Nothing. It was a comment about this whole process not even needing to exist. From v+python at g.nevcal.com Fri Mar 30 01:08:06 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 29 Mar 2012 16:08:06 -0700 Subject: [Python-Dev] Integrating the PEP 397 launcher In-Reply-To: References: Message-ID: <4F74EB56.2040009@g.nevcal.com> On 3/29/2012 3:50 PM, Brian Curtin wrote: > On Thu, Mar 29, 2012 at 17:45, Benjamin Peterson wrote: >> > 2012/3/29 Brian Curtin : >>> >> After talking with Martin and several others during the language >>> >> summit and elsewhere around PyCon, PEP 397 should be accepted. I don't >>> >> remember who, but some suggested it should just be a regular old >>> >> feature instead of going through the PEP process. So...does this even >>> >> need to continue the PEP process? >> > >> > If you have a PEP and it's accepted, what would be the difference? > In the end? Nothing. It was a comment about this whole process not > even needing to exist. It was pretty controversial when it started... but it is such obviously beneficial functionality... and works well... All it needs is official acceptance now, and integration into the release, no? -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Fri Mar 30 01:15:13 2012 From: brian at python.org (Brian Curtin) Date: Thu, 29 Mar 2012 18:15:13 -0500 Subject: [Python-Dev] Integrating the PEP 397 launcher In-Reply-To: <4F74EB56.2040009@g.nevcal.com> References: <4F74EB56.2040009@g.nevcal.com> Message-ID: On Thu, Mar 29, 2012 at 18:08, Glenn Linderman wrote: > All it needs is official acceptance now, and integration into the release, > no? If it wasn't clear, this is what I said in the first post. From benjamin at python.org Fri Mar 30 02:02:39 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 29 Mar 2012 20:02:39 -0400 Subject: [Python-Dev] Integrating the PEP 397 launcher In-Reply-To: References: Message-ID: 2012/3/29 Brian Curtin : > On Thu, Mar 29, 2012 at 17:45, Benjamin Peterson wrote: >> 2012/3/29 Brian Curtin : >>> After talking with Martin and several others during the language >>> summit and elsewhere around PyCon, PEP 397 should be accepted. I don't >>> remember who, but some suggested it should just be a regular old >>> feature instead of going through the PEP process. So...does this even >>> need to continue the PEP process? >> >> If you have a PEP and it's accepted, what would be the difference? > > In the end? Nothing. It was a comment about this whole process not > even needing to exist. The PEP process in general or specifically for this PEP? -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Fri Mar 30 02:22:04 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 30 Mar 2012 00:22:04 +0000 (UTC) Subject: [Python-Dev] Integrating the PEP 397 launcher References: Message-ID: Brian Curtin python.org> writes: > Vinay - is the code you have on bitbucket ready to roll into CPython, > thus into the installer? I believe the main C launcher code is ready to roll into CPython. However, the standalone installer I provide uses WiX rather than msilib, and includes additional executables for functionality like "associate .py with one of the installed Pythons" when the launcher is uninstalled, and for printing messages in certain contexts when installing. I believe there needs to be a little more thought given to how to bring the launcher into the main installer to see if we can either dispense with, or make suitable changes to, these ancillary functions. I would appreciate some feedback from Martin about this - as far as I know he has not made any comments about launcher integration into the main installer. The current launcher functionality (py[w].exe) is as outlined in the PEP + feedback from users (e.g. your recent suggestion to use LOCALAPPDATA rather than APPDATA). The test harness may also need some thinking about - as the launcher executable is separate from Python, I'm not sure if it's appropriate just to create a "test_launcher.py" in Lib/test. To do a full test of the launcher you need multiple 2.x and 3.x versions installed, and I'm not sure if this could be done on existing Windows buildbots, for example. Of course it could be done with mocked executables and synthetically-added registry entries, but that isn't currently in place. Regards, Vinay Sajip From stephen at xemacs.org Fri Mar 30 03:31:00 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 30 Mar 2012 10:31:00 +0900 Subject: [Python-Dev] Integrating the PEP 397 launcher In-Reply-To: References: Message-ID: On Fri, Mar 30, 2012 at 7:50 AM, Brian Curtin wrote: > On Thu, Mar 29, 2012 at 17:45, Benjamin Peterson wrote: >> 2012/3/29 Brian Curtin : >>> After talking with Martin and several others during the language >>> summit and elsewhere around PyCon, PEP 397 should be accepted. I don't >>> remember who, but some suggested it should just be a regular old >>> feature instead of going through the PEP process. So...does this even >>> need to continue the PEP process? >> >> If you have a PEP and it's accepted, what would be the difference? > > In the end? Nothing. It was a comment about this whole process not > even needing to exist. Hindsight, as usual, is 20-20. If the process got enough discussion that somebody said, "look, this really needs a PEP," it's worth the effort to record it. From ncoghlan at gmail.com Fri Mar 30 04:04:03 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2012 12:04:03 +1000 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <4F74A953.2090909@oddbird.net> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> <1333042767.31165.47.camel@surprise> <4F74A953.2090909@oddbird.net> Message-ID: On Fri, Mar 30, 2012 at 4:26 AM, Carl Meyer wrote: > I've added this option as a comment on bug 14444. The title of that bug > is worded such that it could be reasonably resolved either with the > backwards-compatibility fix or the release notes addition, the release > managers can decide what seems appropriate to them. +1 for restoring the fallback code in os.py. A security release definitely shouldn't be breaking that kind of thing. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From merwok at netwok.org Fri Mar 30 04:07:49 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 29 Mar 2012 22:07:49 -0400 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> <1333042767.31165.47.camel@surprise> <4F74A953.2090909@oddbird.net> Message-ID: <4F751575.1030603@netwok.org> Le 29/03/2012 22:04, Nick Coghlan a ?crit : > On Fri, Mar 30, 2012 at 4:26 AM, Carl Meyer wrote: >> I've added this option as a comment on bug 14444. The title of that bug >> is worded such that it could be reasonably resolved either with the >> backwards-compatibility fix or the release notes addition, the release >> managers can decide what seems appropriate to them. > +1 for restoring the fallback code in os.py. A security release > definitely shouldn't be breaking that kind of thing. The RMs have already agreed not to restore the fallback, and the maintainer of virtualenv agreed. See report for more details. Cheers From ncoghlan at gmail.com Fri Mar 30 05:05:00 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2012 13:05:00 +1000 Subject: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing) In-Reply-To: <4F751575.1030603@netwok.org> References: <7E79234E600438479EC119BD241B48D601E7CFB2@BY2PRD0610MB389.namprd06.prod.outlook.com> <1333042767.31165.47.camel@surprise> <4F74A953.2090909@oddbird.net> <4F751575.1030603@netwok.org> Message-ID: On Fri, Mar 30, 2012 at 12:07 PM, ?ric Araujo wrote: > Le 29/03/2012 22:04, Nick Coghlan a ?crit : >> On Fri, Mar 30, 2012 at 4:26 AM, Carl Meyer wrote: >>> I've added this option as a comment on bug 14444. The title of that bug >>> is worded such that it could be reasonably resolved either with the >>> backwards-compatibility fix or the release notes addition, the release >>> managers can decide what seems appropriate to them. >> +1 for restoring the fallback code in os.py. A security release >> definitely shouldn't be breaking that kind of thing. > > The RMs have already agreed not to restore the fallback, and the > maintainer of virtualenv agreed. ?See report for more details. The details are pretty short (and make sense), so I'll repeat the most salient points here for the benefit of others: - if the binary in a virtualenv isn't updated, then projects running in a virtualenv won't receive the security fix when the updated version of Python is installed - restoring the fallback in os.py would make this failure mode *silent*, so a user could upgrade Python, set PYTHONHASHSEED, but fail to realise they also need to update the binary in the virtualenv in order to benefit from the hash protection - with the current behaviour, failing to upgrade the binary results in a noisy ImportError or AttributeError related to os.urandom, which the release notes and virtualenv help channels can instruct people on handling. This approach explicitly covers the additional steps needed to fully deploy the security fix when using virtualenv. That rationale makes sense to me, too. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Mar 30 05:12:18 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2012 13:12:18 +1000 Subject: [Python-Dev] [Python-checkins] cpython (3.2): Issue #14409: IDLE doesn't not execute commands from shell with default In-Reply-To: References: Message-ID: On Fri, Mar 30, 2012 at 2:01 AM, andrew.svetlov wrote: > +- Issue #14409: IDLE doesn't not execute commands from shell, > + ?error with default keybinding for Return. (Patch by Roger Serwy) The double negative here makes this impossible to understand. Could we please get an updated NEWS entry that explains what actually changed in IDLE to fix this? Perhaps something like "IDLE now always sets the default keybind for Return correctly, ensuring commands can be executed in the IDLE shell window"? (assuming that's what happened). This is important, folks: NEWS entries need to be comprehensible for people that *haven't* read the associated tracker issue. This means that issue titles (which generally describe a problem someone was having) are often inappropriate as NEWS items. NEWS items should be short descriptions that clearly describe *what changed*, perhaps with some additional information to explain a bit about why the change was made. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From martin at v.loewis.de Fri Mar 30 05:48:42 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Fri, 30 Mar 2012 05:48:42 +0200 Subject: [Python-Dev] Integrating the PEP 397 launcher In-Reply-To: References: Message-ID: <20120330054842.Horde.b154O8L8999PdS0aUGOUQNA@webmail.df.eu> Zitat von Brian Curtin : > After talking with Martin and several others during the language > summit and elsewhere around PyCon, PEP 397 should be accepted. I don't > remember who, but some suggested it should just be a regular old > feature instead of going through the PEP process. So...does this even > need to continue the PEP process? I don't think PEP 397 can be accepted as it stands; it has too many open issues. However, I am in favor of accepting the proposed feature. Now that we do have the PEP, I think that should be done properly. I thought you offered to rewrite it. Formally, I could accept the PEP being withdrawn, and the feature integrated anyway, but I still consider that bad style. So: I can offer to rewrite the PEP to give a full specification of the feature. It might be that I still need some help to provide end-user prose in the PEP if people would want to see that as well. Regards, Martin From brian at python.org Fri Mar 30 06:01:23 2012 From: brian at python.org (Brian Curtin) Date: Thu, 29 Mar 2012 23:01:23 -0500 Subject: [Python-Dev] Integrating the PEP 397 launcher In-Reply-To: <20120330054842.Horde.b154O8L8999PdS0aUGOUQNA@webmail.df.eu> References: <20120330054842.Horde.b154O8L8999PdS0aUGOUQNA@webmail.df.eu> Message-ID: On Thu, Mar 29, 2012 at 22:48, wrote: > Now that we do have the PEP, I think that should be done properly. > I thought you offered to rewrite it. There are definitely areas that I would like to work on, especially pulling implementation details out and replacing with, as you say, end-user prose. For example, some part of a doc tells you to call some function with a specific parameter to figure out where py.ini should be found - it needs to be replaced with an example directory. > So: I can offer to rewrite the PEP to give a full specification > of the feature. It might be that I still need some help to > provide end-user prose in the PEP if people would want to see that as > well. I would be much better at proposing the end-user stuff than the specification. From ncoghlan at gmail.com Fri Mar 30 07:33:41 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2012 15:33:41 +1000 Subject: [Python-Dev] [Python-checkins] cpython (3.2): Issue #14409: IDLE doesn't not execute commands from shell with default In-Reply-To: <20120330061222.Horde.L8aPH7uWis5PdTKmYa9VVjA@webmail.df.eu> References: <20120330061222.Horde.L8aPH7uWis5PdTKmYa9VVjA@webmail.df.eu> Message-ID: On Fri, Mar 30, 2012 at 2:12 PM, wrote: > > Zitat von Nick Coghlan : > > >> On Fri, Mar 30, 2012 at 2:01 AM, andrew.svetlov >> wrote: >>> >>> +- Issue #14409: IDLE doesn't not execute commands from shell, >>> + ?error with default keybinding for Return. (Patch by Roger Serwy) >> >> >> The double negative here makes this impossible to understand. Could we >> please get an updated NEWS entry that explains what actually changed >> in IDLE to fix this? > > > Please consider that Andrew is not a native speaker of English. So > it's unfair to ask him to rewrite the NEWS entry. That can only > be done by a native speaker. The NEWS entries need to be treated as being at least close to on par with the rest of the docs - it's OK if someone can't come up with the words themselves, but if that's the case, then it's preferable to ask for help with the wording explicitly. Is my suggested rephrasing correct? I don't know, as I'm not familiar with either the original problem or what was done to fix it. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From martin at v.loewis.de Fri Mar 30 09:55:17 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Fri, 30 Mar 2012 09:55:17 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.2): Issue #14409: IDLE doesn't not execute commands from shell with default In-Reply-To: References: <20120330061222.Horde.L8aPH7uWis5PdTKmYa9VVjA@webmail.df.eu> Message-ID: <20120330095517.Horde.iuDqHtjz9kRPdWblbapDMTA@webmail.df.eu> > Is my suggested rephrasing correct? I don't know, as I'm not familiar > with either the original problem or what was done to fix it. I don't know, either, and Andrew may not able to answer the question as he may not see the fine difference between what he wrote and what you wrote (your phrasing is grammatically fairly advanced). Regards, Martin From solipsis at pitrou.net Fri Mar 30 09:54:57 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 30 Mar 2012 09:54:57 +0200 Subject: [Python-Dev] peps: PEP 418: Fill the "Adjusted by NTP" column of the summary table References: Message-ID: <20120330095457.5ef47a9c@pitrou.net> On Fri, 30 Mar 2012 04:21:22 +0200 victor.stinner wrote: > > diff --git a/pep-0418.txt b/pep-0418.txt > --- a/pep-0418.txt > +++ b/pep-0418.txt > @@ -190,13 +190,13 @@ > Name Resolution Adjusted by NTP? Action on suspend > ========================= =============== ================ ================= Is it supposed to be resolution or accuracy? For example for QueryPerformanceCounter() and timeGetTime(), you are giving an accuracy, but for CLOCK_* you are giving a resolution. Regards Antoine. From vinay_sajip at yahoo.co.uk Fri Mar 30 11:01:53 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 30 Mar 2012 09:01:53 +0000 (UTC) Subject: [Python-Dev] Integrating the PEP 397 launcher References: <20120330054842.Horde.b154O8L8999PdS0aUGOUQNA@webmail.df.eu> Message-ID: v.loewis.de> writes: > Now that we do have the PEP, I think that should be done properly. > I thought you offered to rewrite it. Formally, I could accept the > PEP being withdrawn, and the feature integrated anyway, but I still > consider that bad style. > > So: I can offer to rewrite the PEP to give a full specification > of the feature. It might be that I still need some help to > provide end-user prose in the PEP if people would want to see that as > well. +1 to both your and Brian's input. I will aim to keep the implementation in sync with any changes required as a result of resolving open issues. Regards, Vinay Sajip From storchaka at gmail.com Fri Mar 30 12:38:13 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 30 Mar 2012 13:38:13 +0300 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: References: Message-ID: 28.03.12 23:20, Andrew Svetlov ???????(??): > I figured out what pytz and dateutil are not mentioned in python docs > for datetime module. > It's clean why these libs is not a part of Python Libraries ? but > that's not clean for Docs. I don't understand why Python may not include the pytz. The Olson tz database is not part of pytz. Python can depend on a system tz database, as it depends on libssl or libbz2, which also can be updated (for security reasons) independently. From victor.stinner at gmail.com Fri Mar 30 13:55:21 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 30 Mar 2012 13:55:21 +0200 Subject: [Python-Dev] peps: PEP 418: Fill the "Adjusted by NTP" column of the summary table In-Reply-To: <20120330095457.5ef47a9c@pitrou.net> References: <20120330095457.5ef47a9c@pitrou.net> Message-ID: >> diff --git a/pep-0418.txt b/pep-0418.txt >> --- a/pep-0418.txt >> +++ b/pep-0418.txt >> @@ -190,13 +190,13 @@ >> ?Name ? ? ? ? ? ? ? ? ? ? ? Resolution ? ? ? Adjusted by NTP? ?Action on suspend >> ?========================= ?=============== ?================ ?================= > > Is it supposed to be resolution or accuracy? Oh, right. I changed the table to add an accuracy column. Victor From tlesher at gmail.com Fri Mar 30 14:18:06 2012 From: tlesher at gmail.com (Tim Lesher) Date: Fri, 30 Mar 2012 08:18:06 -0400 Subject: [Python-Dev] Bug in generator if the generator in created in a C thread In-Reply-To: <4F74D392.2040808@gmail.com> References: <4F74D392.2040808@gmail.com> Message-ID: On Thu, Mar 29, 2012 at 17:26, Victor Stinner wrote: > The problem is not the frame, but the Python thread state referenced by the > frame. It's a private attribute. My patch just updates this reference before > running the generator (and it clears the reference when the generator > excution is stopped or finished). Right--my thought was the the situation we saw might be similarly triggered because we were storing an exception with traceback (and associated frames) generated by thread A, and then re-throwing it from thread B some time after thread A has exited. The frame attached to the exception's traceback would still, then, be referencing a nonexistent thread state, correct? My concern was that this might one instance of a broader problem for folks who embed Python and see the attractive PyGILState_Ensure() API. It already prevents us from using subinterpreters (which for us might have been a better solution than repeated initialize/finalize, with its known issues). We recently made a change to dispose of the traceback before storing the exception, and that appears to eliminate the corruption we were seeing, so that's making me more suspicious. > You may leak memory if your threads have a short lifetime and you create > many threads. For example if one thread is only used to process one request > and then is destroyed. Absolutely--this particular hack was only used for a thread created outside Python that had to call into the VM. Their behavior is well-defined in our case--two particular OS threads, with lifetimes longer than those of the interpreters we create and finalize. -- Tim Lesher From animelovin at gmail.com Fri Mar 30 14:20:32 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Fri, 30 Mar 2012 08:20:32 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <20120329220755.377052500E9@webabinitio.net> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> Message-ID: <4F75A510.7080401@gmail.com> On 03/29/2012 06:07 PM, R. David Murray wrote: > On Thu, 29 Mar 2012 23:00:20 +0200, Stefan Behnel wrote: >> R. David Murray, 29.03.2012 22:31: >>> On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: >>>> On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: >>>>> Some of us have expressed uneasiness about the consequences of dict >>>>> raising an error on lookup if the dict has been modified, the fix Victor >>>>> made to solve one of the crashers. >>>>> >>>>> I don't know if I speak for the others, but (assuming that I understand >>>>> the change correctly) my concern is that there is probably a significant >>>>> amount of threading code out there that assumes that dict *lookup* is >>>>> a thread-safe operation. Much of that code will, if moved to Python >>>>> 3.3, now be subject to random runtime errors for which it will not >>>>> be prepared. Further, code which appears safe can suddenly become >>>>> unsafe if a refactoring of the code causes an object to be stored in >>>>> the dictionary that has a Python equality method. >>>> >>>> My original assessment was that this only affects dicts whose keys >>>> have a user-implemented __hash__ or __eq__ implementation, and that >>>> the number of apps that use this *and* assume the threadsafe property >>>> would be pretty small. This is just intuition, I don't have hard >>>> facts. But I do want to stress that not all dict lookups automatically >>>> become thread-unsafe, only those that need to run user code as part of >>>> the key lookup. >>> >>> You are probably correct, but the thing is that one still has to do the >>> code audit to be sure...and then make sure that no one later introduces >>> such an object type as a dict key. >> >> The thing is: the assumption that arbitrary dict lookups are GIL-atomic has >> *always* been false. Only those that do not involve Python code execution >> for the hash key calculation or the object comparison are. That includes >> the built-in strings and numbers (and tuples of them), which are by far the >> most common dict keys. Looking up arbitrary user provided objects is >> definitely not guaranteed to be atomic. > > Well, I'm afraid I was using the term 'thread safety' rather too loosely > there. What I mean is that if you do a dict lookup, the lookup either > returns a value or a KeyError, and that if you get back an object that > object has internally consistent state. The problem this fix introduces > is that the lookup may fail with a RuntimeError rather than a KeyError, > which it has never done before. > > I think that is what Guido means by code that uses objects with python > eq/hash *and* assumes threadsafe lookup. If mutation of the objects > or dict during the lookup is a concern, then the code would use locks > and wouldn't have the problem. But there are certainly situations > where it doesn't matter if the dictionary mutates during the lookup, > as long as you get either an object or a KeyError, and thus no locks are > (currently) needed. > > Maybe I'm being paranoid about breakage here, but as with most backward > compatibility concerns, there are probably more bits of code that will > be affected than our intuition indicates. > > --David > _______________________________________________ what this suppose to mean exactly? To "mutate" is a bit odd concept for a programming language I suppose. Also I suppose I must be missing something which makes you feel like this is an OT post when the problem seem most likely to be exclusively in python 3.3, another reason I guess to not upgrade yet all that massively using 2to3. :-) cheers, Etienne From rdmurray at bitdance.com Fri Mar 30 14:47:46 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 30 Mar 2012 08:47:46 -0400 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: References: Message-ID: <20120330124746.A50E82500F8@webabinitio.net> On Fri, 30 Mar 2012 13:38:13 +0300, Serhiy Storchaka wrote: > 28.03.12 23:20, Andrew Svetlov ??????????????(????): > > I figured out what pytz and dateutil are not mentioned in python docs > > for datetime module. > > It's clean why these libs is not a part of Python Libraries ??? but > > that's not clean for Docs. > > I don't understand why Python may not include the pytz. The Olson tz > database is not part of pytz. Python can depend on a system tz database, > as it depends on libssl or libbz2, which also can be updated (for > security reasons) independently. There is an extensive discussion of this somewhere in the archives of this list. If I remember correctly, it boils down to the fact that pytz does bundle the database, and that Windows either does not have or does not regularly update its own Olson database. Rather than ship something out-of-date, we choose to put the onus on the user to ensure that the appropriate code+db exists on their system. Hopefully someone will correct me if I'm wrong, and/or find a pointer to the relevant thread. --David From andrew.svetlov at gmail.com Fri Mar 30 15:05:23 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 30 Mar 2012 16:05:23 +0300 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: <20120330124746.A50E82500F8@webabinitio.net> References: <20120330124746.A50E82500F8@webabinitio.net> Message-ID: I filed the http://bugs.python.org/issue14448 BTW. On Fri, Mar 30, 2012 at 3:47 PM, R. David Murray wrote: > On Fri, 30 Mar 2012 13:38:13 +0300, Serhiy Storchaka wrote: >> 28.03.12 23:20, Andrew Svetlov ???????(??): >> > I figured out what pytz and dateutil are not mentioned in python docs >> > for datetime module. >> > It's clean why these libs is not a part of Python Libraries ? but >> > that's not clean for Docs. >> >> I don't understand why Python may not include the pytz. The Olson tz >> database is not part of pytz. Python can depend on a system tz database, >> as it depends on libssl or libbz2, which also can be updated (for >> security reasons) independently. > > There is an extensive discussion of this somewhere in the archives of > this list. ?If I remember correctly, it boils down to the fact that pytz > does bundle the database, and that Windows either does not have or does > not regularly update its own Olson database. ?Rather than ship something > out-of-date, we choose to put the onus on the user to ensure that the > appropriate code+db exists on their system. > > Hopefully someone will correct me if I'm wrong, and/or find a pointer > to the relevant thread. > > --David > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com > -- Thanks, Andrew Svetlov From ncoghlan at gmail.com Fri Mar 30 15:56:04 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2012 23:56:04 +1000 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: <20120330124746.A50E82500F8@webabinitio.net> References: <20120330124746.A50E82500F8@webabinitio.net> Message-ID: On Fri, Mar 30, 2012 at 10:47 PM, R. David Murray wrote: > On Fri, 30 Mar 2012 13:38:13 +0300, Serhiy Storchaka wrote: >> 28.03.12 23:20, Andrew Svetlov ???????(??): >> > I figured out what pytz and dateutil are not mentioned in python docs >> > for datetime module. >> > It's clean why these libs is not a part of Python Libraries ? but >> > that's not clean for Docs. >> >> I don't understand why Python may not include the pytz. The Olson tz >> database is not part of pytz. Python can depend on a system tz database, >> as it depends on libssl or libbz2, which also can be updated (for >> security reasons) independently. > > There is an extensive discussion of this somewhere in the archives of > this list. ?If I remember correctly, it boils down to the fact that pytz > does bundle the database, and that Windows either does not have or does > not regularly update its own Olson database. ?Rather than ship something > out-of-date, we choose to put the onus on the user to ensure that the > appropriate code+db exists on their system. > > Hopefully someone will correct me if I'm wrong, and/or find a pointer > to the relevant thread. That's my recollection as well. Because we don't want to take on the task of providely timely updates in response to timezone database changes, any named timezone support added to the stdlib would need to be based on a system provided timezone database, rather than the bundled database model used by pytz. This is straightforward on *nix based systems that provide the zoneinfo structure in the filesystem, but more complicated on Windows (which has its own custom scheme). Before the idea of adding full timezone support to the standard library could be seriously considered someone would have to, at the very least, use the mapping data from the Unicode Consortium's CLDR Supplementary data to map the standard Olsen database timezone names to the correct values to look up through the Windows timezone APIs (http://unicode.org/repos/cldr-tmp/trunk/diff/supplemental/zone_tzid.html) Adding mappings for *new* timezones would still be controlled by our release cycle (although I think it would be reasonable to permit such additions in maintenance releases), but updates in response to things like daylight savings dates changing would then be the responsibility of the OS vendors. However, "pip install pytz" is easy enough that there isn't a lot of motivation for anyone to do the work to switch from a bundled copy of the timezone database to a bundled TZID -> Windows API lookup mapping. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Fri Mar 30 16:47:38 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 07:47:38 -0700 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <4F75A510.7080401@gmail.com> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> Message-ID: Etienne, I have not understood either of your messages in this thread. They just did not make sense to me. Do you actually understand the issue at hand? --Guido On Friday, March 30, 2012, Etienne Robillard wrote: > On 03/29/2012 06:07 PM, R. David Murray wrote: > >> On Thu, 29 Mar 2012 23:00:20 +0200, Stefan Behnel >> wrote: >> >>> R. David Murray, 29.03.2012 22:31: >>> >>>> On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: >>>> >>>>> On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: >>>>> >>>>>> Some of us have expressed uneasiness about the consequences of dict >>>>>> raising an error on lookup if the dict has been modified, the fix >>>>>> Victor >>>>>> made to solve one of the crashers. >>>>>> >>>>>> I don't know if I speak for the others, but (assuming that I >>>>>> understand >>>>>> the change correctly) my concern is that there is probably a >>>>>> significant >>>>>> amount of threading code out there that assumes that dict *lookup* is >>>>>> a thread-safe operation. Much of that code will, if moved to Python >>>>>> 3.3, now be subject to random runtime errors for which it will not >>>>>> be prepared. Further, code which appears safe can suddenly become >>>>>> unsafe if a refactoring of the code causes an object to be stored in >>>>>> the dictionary that has a Python equality method. >>>>>> >>>>> >>>>> My original assessment was that this only affects dicts whose keys >>>>> have a user-implemented __hash__ or __eq__ implementation, and that >>>>> the number of apps that use this *and* assume the threadsafe property >>>>> would be pretty small. This is just intuition, I don't have hard >>>>> facts. But I do want to stress that not all dict lookups automatically >>>>> become thread-unsafe, only those that need to run user code as part of >>>>> the key lookup. >>>>> >>>> >>>> You are probably correct, but the thing is that one still has to do the >>>> code audit to be sure...and then make sure that no one later introduces >>>> such an object type as a dict key. >>>> >>> >>> The thing is: the assumption that arbitrary dict lookups are GIL-atomic >>> has >>> *always* been false. Only those that do not involve Python code execution >>> for the hash key calculation or the object comparison are. That includes >>> the built-in strings and numbers (and tuples of them), which are by far >>> the >>> most common dict keys. Looking up arbitrary user provided objects is >>> definitely not guaranteed to be atomic. >>> >> >> Well, I'm afraid I was using the term 'thread safety' rather too loosely >> there. What I mean is that if you do a dict lookup, the lookup either >> returns a value or a KeyError, and that if you get back an object that >> object has internally consistent state. The problem this fix introduces >> is that the lookup may fail with a RuntimeError rather than a KeyError, >> which it has never done before. >> >> I think that is what Guido means by code that uses objects with python >> eq/hash *and* assumes threadsafe lookup. If mutation of the objects >> or dict during the lookup is a concern, then the code would use locks >> and wouldn't have the problem. But there are certainly situations >> where it doesn't matter if the dictionary mutates during the lookup, >> as long as you get either an object or a KeyError, and thus no locks are >> (currently) needed. >> >> Maybe I'm being paranoid about breakage here, but as with most backward >> compatibility concerns, there are probably more bits of code that will >> be affected than our intuition indicates. >> >> --David >> ______________________________**_________________ >> > > what this suppose to mean exactly? To "mutate" is a bit odd concept for a > programming language I suppose. Also I suppose I must be missing something > which makes you feel like this is an OT post when the problem seem most > likely to be exclusively in python 3.3, another reason I guess to not > upgrade yet all that massively using 2to3. :-) > > cheers, > Etienne > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From animelovin at gmail.com Fri Mar 30 17:27:05 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Fri, 30 Mar 2012 11:27:05 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> Message-ID: <4F75D0C9.4080704@gmail.com> Hi Guido, I'm sorry for being unclear! I just try actually to learn what thoses consequences for theses 'unattended' mutations in dictionary key lookups could be, :-) however, it seems now that I might have touch a nerve without realizing it. I would therefore appreciate more light on this "issue" if you like to enlighten us all. :D Regards, Etienne On 03/30/2012 10:47 AM, Guido van Rossum wrote: > Etienne, I have not understood either of your messages in this thread. > They just did not make sense to me. Do you actually understand the issue > at hand? > > --Guido > > On Friday, March 30, 2012, Etienne Robillard wrote: > > On 03/29/2012 06:07 PM, R. David Murray wrote: > > On Thu, 29 Mar 2012 23:00:20 +0200, Stefan > Behnel wrote: > > R. David Murray, 29.03.2012 22:31: > > On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: > > On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: > > Some of us have expressed uneasiness about the > consequences of dict > raising an error on lookup if the dict has been > modified, the fix Victor > made to solve one of the crashers. > > I don't know if I speak for the others, but > (assuming that I understand > the change correctly) my concern is that there > is probably a significant > amount of threading code out there that assumes > that dict *lookup* is > a thread-safe operation. Much of that code > will, if moved to Python > 3.3, now be subject to random runtime errors for > which it will not > be prepared. Further, code which appears safe > can suddenly become > unsafe if a refactoring of the code causes an > object to be stored in > the dictionary that has a Python equality method. > > > My original assessment was that this only affects > dicts whose keys > have a user-implemented __hash__ or __eq__ > implementation, and that > the number of apps that use this *and* assume the > threadsafe property > would be pretty small. This is just intuition, I > don't have hard > facts. But I do want to stress that not all dict > lookups automatically > become thread-unsafe, only those that need to run > user code as part of > the key lookup. > > > You are probably correct, but the thing is that one > still has to do the > code audit to be sure...and then make sure that no one > later introduces > such an object type as a dict key. > > > The thing is: the assumption that arbitrary dict lookups are > GIL-atomic has > *always* been false. Only those that do not involve Python > code execution > for the hash key calculation or the object comparison are. > That includes > the built-in strings and numbers (and tuples of them), which > are by far the > most common dict keys. Looking up arbitrary user provided > objects is > definitely not guaranteed to be atomic. > > > Well, I'm afraid I was using the term 'thread safety' rather too > loosely > there. What I mean is that if you do a dict lookup, the lookup > either > returns a value or a KeyError, and that if you get back an > object that > object has internally consistent state. The problem this fix > introduces > is that the lookup may fail with a RuntimeError rather than a > KeyError, > which it has never done before. > > I think that is what Guido means by code that uses objects with > python > eq/hash *and* assumes threadsafe lookup. If mutation of the objects > or dict during the lookup is a concern, then the code would use > locks > and wouldn't have the problem. But there are certainly situations > where it doesn't matter if the dictionary mutates during the lookup, > as long as you get either an object or a KeyError, and thus no > locks are > (currently) needed. > > Maybe I'm being paranoid about breakage here, but as with most > backward > compatibility concerns, there are probably more bits of code > that will > be affected than our intuition indicates. > > --David > ______________________________ _________________ > > > what this suppose to mean exactly? To "mutate" is a bit odd concept > for a programming language I suppose. Also I suppose I must be > missing something which makes you feel like this is an OT post when > the problem seem most likely to be exclusively in python 3.3, > another reason I guess to not upgrade yet all that massively using > 2to3. :-) > > cheers, > Etienne > ______________________________ _________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/ mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/ mailman/options/python-dev/ > guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido ) From ncoghlan at gmail.com Fri Mar 30 17:41:53 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 31 Mar 2012 01:41:53 +1000 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <4F75D0C9.4080704@gmail.com> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D0C9.4080704@gmail.com> Message-ID: On Sat, Mar 31, 2012 at 1:27 AM, Etienne Robillard wrote: > Hi Guido, > > I'm sorry for being unclear! I just try actually to learn what thoses > consequences for theses 'unattended' mutations in dictionary key lookups > could be, :-) > > however, it seems now that I might have touch a nerve without realizing it. > I would therefore appreciate more light on this "issue" if you like > to enlighten us all. :D Etienne, For those that need to understand the issue in order to further consider the consequences of the change, RDM has already explained the problem quite clearly. If you'd like a more in-depth explanation, please ask the question again over on core-mentorship at python.org. It's not an appropriate topic for the main development list. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From animelovin at gmail.com Fri Mar 30 17:45:05 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Fri, 30 Mar 2012 11:45:05 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> Message-ID: <4F75D501.2050400@gmail.com> "Multiple threads can agree by convention not to mutate a shared dict, there's no great need for enforcement. Multiple processes can't share dicts." its not sure I get completely the meaning of "mutate"... And if possible, I would like also the rational for the 2nd phrase while we're at it as it seem a little unclear too. Sorry also if this is OT... :) Regards, Etienne http://www.python.org/dev/peps/pep-0416/ On 03/30/2012 10:47 AM, Guido van Rossum wrote: > Etienne, I have not understood either of your messages in this thread. > They just did not make sense to me. Do you actually understand the issue > at hand? > > --Guido > > On Friday, March 30, 2012, Etienne Robillard wrote: > > On 03/29/2012 06:07 PM, R. David Murray wrote: > > On Thu, 29 Mar 2012 23:00:20 +0200, Stefan > Behnel wrote: > > R. David Murray, 29.03.2012 22:31: > > On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: > > On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: > > Some of us have expressed uneasiness about the > consequences of dict > raising an error on lookup if the dict has been > modified, the fix Victor > made to solve one of the crashers. > > I don't know if I speak for the others, but > (assuming that I understand > the change correctly) my concern is that there > is probably a significant > amount of threading code out there that assumes > that dict *lookup* is > a thread-safe operation. Much of that code > will, if moved to Python > 3.3, now be subject to random runtime errors for > which it will not > be prepared. Further, code which appears safe > can suddenly become > unsafe if a refactoring of the code causes an > object to be stored in > the dictionary that has a Python equality method. > > > My original assessment was that this only affects > dicts whose keys > have a user-implemented __hash__ or __eq__ > implementation, and that > the number of apps that use this *and* assume the > threadsafe property > would be pretty small. This is just intuition, I > don't have hard > facts. But I do want to stress that not all dict > lookups automatically > become thread-unsafe, only those that need to run > user code as part of > the key lookup. > > > You are probably correct, but the thing is that one > still has to do the > code audit to be sure...and then make sure that no one > later introduces > such an object type as a dict key. > > > The thing is: the assumption that arbitrary dict lookups are > GIL-atomic has > *always* been false. Only those that do not involve Python > code execution > for the hash key calculation or the object comparison are. > That includes > the built-in strings and numbers (and tuples of them), which > are by far the > most common dict keys. Looking up arbitrary user provided > objects is > definitely not guaranteed to be atomic. > > > Well, I'm afraid I was using the term 'thread safety' rather too > loosely > there. What I mean is that if you do a dict lookup, the lookup > either > returns a value or a KeyError, and that if you get back an > object that > object has internally consistent state. The problem this fix > introduces > is that the lookup may fail with a RuntimeError rather than a > KeyError, > which it has never done before. > > I think that is what Guido means by code that uses objects with > python > eq/hash *and* assumes threadsafe lookup. If mutation of the objects > or dict during the lookup is a concern, then the code would use > locks > and wouldn't have the problem. But there are certainly situations > where it doesn't matter if the dictionary mutates during the lookup, > as long as you get either an object or a KeyError, and thus no > locks are > (currently) needed. > > Maybe I'm being paranoid about breakage here, but as with most > backward > compatibility concerns, there are probably more bits of code > that will > be affected than our intuition indicates. > > --David > ______________________________ _________________ > > > what this suppose to mean exactly? To "mutate" is a bit odd concept > for a programming language I suppose. Also I suppose I must be > missing something which makes you feel like this is an OT post when > the problem seem most likely to be exclusively in python 3.3, > another reason I guess to not upgrade yet all that massively using > 2to3. :-) > > cheers, > Etienne > ______________________________ _________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/ mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/ mailman/options/python-dev/ > guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido ) From stefan_ml at behnel.de Fri Mar 30 17:54:31 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 30 Mar 2012 17:54:31 +0200 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <4F75D501.2050400@gmail.com> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> Message-ID: Etienne Robillard, 30.03.2012 17:45: > Sorry also if this is OT... :) Yes, it is. Please do as Nick told you. Stefan From animelovin at gmail.com Fri Mar 30 18:08:23 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Fri, 30 Mar 2012 12:08:23 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> Message-ID: <4F75DA77.6040305@gmail.com> you wish...are you also truth allergic or irritated by the consequences of free speech ? Please stop giving me orders. You don't even know me and this is at all not necessary and good netiquette if you want to bring a point to ponder. Sorry for others who thinks this is not OT as I its probably related to pep-416 refusal. Cheers! Etienne On 03/30/2012 11:54 AM, Stefan Behnel wrote: > Etienne Robillard, 30.03.2012 17:45: >> Sorry also if this is OT... :) > > Yes, it is. Please do as Nick told you. > > Stefan From status at bugs.python.org Fri Mar 30 18:07:11 2012 From: status at bugs.python.org (Python tracker) Date: Fri, 30 Mar 2012 18:07:11 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20120330160711.97FC61C9A0@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2012-03-23 - 2012-03-30) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3359 (+13) closed 22871 (+42) total 26230 (+55) Open issues with patches: 1438 Issues opened (35) ================== #14396: Popen wait() doesn't handle spurious wakeups http://bugs.python.org/issue14396 opened by amscanne #14397: Use GetTickCount/GetTickCount64 instead of QueryPerformanceCou http://bugs.python.org/issue14397 opened by haypo #14398: bz2.BZ2DEcompressor.decompress fail on large files http://bugs.python.org/issue14398 opened by Laurent.Gautier #14399: zipfile and creat/update comment http://bugs.python.org/issue14399 opened by acassaigne #14405: Some "Other Resources" in the sidebar are hopelessly out of da http://bugs.python.org/issue14405 opened by nedbat #14406: Race condition in concurrent.futures http://bugs.python.org/issue14406 opened by anacrolix #14407: concurrent.futures tests don't adher to test_cases protocol http://bugs.python.org/issue14407 opened by anacrolix #14408: Support ./python -m unittest in the stdlib tests http://bugs.python.org/issue14408 opened by anacrolix #14412: Sqlite Integer Fields http://bugs.python.org/issue14412 opened by goatsofmendez #14414: xmlrpclib leaves connection in broken state if server returns http://bugs.python.org/issue14414 opened by fancycode #14417: dict RuntimeError workaround http://bugs.python.org/issue14417 opened by Jim.Jewett #14418: Document differences in SocketServer between Python 2.6 and 2. http://bugs.python.org/issue14418 opened by gjb1002 #14419: Faster ascii decoding http://bugs.python.org/issue14419 opened by storchaka #14420: winreg SetValueEx DWord type incompatible with value argument http://bugs.python.org/issue14420 opened by RoSanford #14422: Pack PyASCIIObject fields to reduce memory consumption of pure http://bugs.python.org/issue14422 opened by haypo #14423: Getting the starting date of iso week from a week number and a http://bugs.python.org/issue14423 opened by Esben.Agerb??k.Black #14424: document PyType_GenericAlloc http://bugs.python.org/issue14424 opened by eli.bendersky #14425: Improve handling of 'timeout' parameter default in urllib.urlo http://bugs.python.org/issue14425 opened by r.david.murray #14426: date format problem in Cookie/http.cookies http://bugs.python.org/issue14426 opened by M??te.Invert #14427: urllib.request.Request get_header and header_items not documen http://bugs.python.org/issue14427 opened by r.david.murray #14428: Implementation of the PEP 418 http://bugs.python.org/issue14428 opened by haypo #14432: Bug in generator if the generator in created in a C thread http://bugs.python.org/issue14432 opened by haypo #14433: Python 3 interpreter crash on windows when stdin closed in Pyt http://bugs.python.org/issue14433 opened by alexis.d #14434: Tutorial link in "help()" in Python3 points to Python2 tutoria http://bugs.python.org/issue14434 opened by Dubslow #14437: _io build fails on cygwin http://bugs.python.org/issue14437 opened by luch #14439: Easier error diagnosis when bootstrapping the runpy module in http://bugs.python.org/issue14439 opened by haypo #14440: Close background process if IDLE closes abnormally. http://bugs.python.org/issue14440 opened by asvetlov #14443: Distutils test failure http://bugs.python.org/issue14443 opened by rosslagerwall #14444: Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom http://bugs.python.org/issue14444 opened by jason.coombs #14445: Providing more fine-grained control over assert statements http://bugs.python.org/issue14445 opened by max #14446: Remove deprecated tkinter functions http://bugs.python.org/issue14446 opened by asvetlov #14448: Mention pytz in datetime's docs http://bugs.python.org/issue14448 opened by asvetlov #14449: argparse optional arguments should follow getopt_long(3) http://bugs.python.org/issue14449 opened by mamikonyan #14450: Log rotate cant execute in Windows. (logging module) http://bugs.python.org/issue14450 opened by shinta.nakayama #14452: SysLogHandler sends invalid messages when using unicode http://bugs.python.org/issue14452 opened by zmk Most recent 15 issues with no replies (15) ========================================== #14452: SysLogHandler sends invalid messages when using unicode http://bugs.python.org/issue14452 #14446: Remove deprecated tkinter functions http://bugs.python.org/issue14446 #14440: Close background process if IDLE closes abnormally. http://bugs.python.org/issue14440 #14427: urllib.request.Request get_header and header_items not documen http://bugs.python.org/issue14427 #14424: document PyType_GenericAlloc http://bugs.python.org/issue14424 #14418: Document differences in SocketServer between Python 2.6 and 2. http://bugs.python.org/issue14418 #14414: xmlrpclib leaves connection in broken state if server returns http://bugs.python.org/issue14414 #14407: concurrent.futures tests don't adher to test_cases protocol http://bugs.python.org/issue14407 #14379: Several traceback docs improvements http://bugs.python.org/issue14379 #14345: Document socket.SOL_SOCKET http://bugs.python.org/issue14345 #14341: sporadic (?) test_urllib2 failures http://bugs.python.org/issue14341 #14339: Optimizing bin, oct and hex http://bugs.python.org/issue14339 #14336: Difference between pickle implementations for function objects http://bugs.python.org/issue14336 #14329: proxy_bypass_macosx_sysconf does not handle singel ip addresse http://bugs.python.org/issue14329 #14326: IDLE - allow shell to support different locales http://bugs.python.org/issue14326 Most recent 15 issues waiting for review (15) ============================================= #14448: Mention pytz in datetime's docs http://bugs.python.org/issue14448 #14439: Easier error diagnosis when bootstrapping the runpy module in http://bugs.python.org/issue14439 #14433: Python 3 interpreter crash on windows when stdin closed in Pyt http://bugs.python.org/issue14433 #14432: Bug in generator if the generator in created in a C thread http://bugs.python.org/issue14432 #14428: Implementation of the PEP 418 http://bugs.python.org/issue14428 #14425: Improve handling of 'timeout' parameter default in urllib.urlo http://bugs.python.org/issue14425 #14423: Getting the starting date of iso week from a week number and a http://bugs.python.org/issue14423 #14422: Pack PyASCIIObject fields to reduce memory consumption of pure http://bugs.python.org/issue14422 #14419: Faster ascii decoding http://bugs.python.org/issue14419 #14408: Support ./python -m unittest in the stdlib tests http://bugs.python.org/issue14408 #14407: concurrent.futures tests don't adher to test_cases protocol http://bugs.python.org/issue14407 #14406: Race condition in concurrent.futures http://bugs.python.org/issue14406 #14399: zipfile and creat/update comment http://bugs.python.org/issue14399 #14396: Popen wait() doesn't handle spurious wakeups http://bugs.python.org/issue14396 #14392: type=bool doesn't raise error in argparse.Action http://bugs.python.org/issue14392 Top 10 most discussed issues (10) ================================= #14386: Expose dictproxy as a public type http://bugs.python.org/issue14386 28 msgs #14408: Support ./python -m unittest in the stdlib tests http://bugs.python.org/issue14408 22 msgs #14288: Make iterators pickleable http://bugs.python.org/issue14288 15 msgs #14419: Faster ascii decoding http://bugs.python.org/issue14419 13 msgs #14444: Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom http://bugs.python.org/issue14444 13 msgs #3367: Uninitialized value read in parsetok.c http://bugs.python.org/issue3367 12 msgs #14399: zipfile and creat/update comment http://bugs.python.org/issue14399 12 msgs #14433: Python 3 interpreter crash on windows when stdin closed in Pyt http://bugs.python.org/issue14433 11 msgs #14398: bz2.BZ2DEcompressor.decompress fail on large files http://bugs.python.org/issue14398 9 msgs #14412: Sqlite Integer Fields http://bugs.python.org/issue14412 7 msgs Issues closed (41) ================== #786827: IDLE starts with no menus (Cygwin) http://bugs.python.org/issue786827 closed by asvetlov #4528: test_httpservers consistently fails - OSError: [Errno 13] Perm http://bugs.python.org/issue4528 closed by r.david.murray #5707: IDLE will not load http://bugs.python.org/issue5707 closed by asvetlov #6488: ElementTree documentation refers to "path" with no explanation http://bugs.python.org/issue6488 closed by eli.bendersky #7635: 19.6 xml.dom.pulldom doc: stub? http://bugs.python.org/issue7635 closed by r.david.murray #11826: Leak in atexitmodule http://bugs.python.org/issue11826 closed by skrah #12649: email.Header ignores maxlinelen when wrapping encoded words http://bugs.python.org/issue12649 closed by r.david.murray #12932: dircmp does not allow non-shallow comparisons http://bugs.python.org/issue12932 closed by eric.araujo #13438: "Delete patch set" review action doesn't work http://bugs.python.org/issue13438 closed by eric.araujo #13608: remove born-deprecated PyUnicode_AsUnicodeAndSize http://bugs.python.org/issue13608 closed by loewis #13902: Sporadic test_threading failure on FreeBSD 6.4 buildbot http://bugs.python.org/issue13902 closed by neologix #14065: Element should support cyclic GC http://bugs.python.org/issue14065 closed by eli.bendersky #14154: reimplement the bigmem test memory watchdog as a subprocess http://bugs.python.org/issue14154 closed by neologix #14162: PEP 416: Add a builtin frozendict type http://bugs.python.org/issue14162 closed by haypo #14349: The documentation of 'dis' doesn't describe MAKE_FUNCTION corr http://bugs.python.org/issue14349 closed by eli.bendersky #14357: Distutils2 does not work with virtualenv http://bugs.python.org/issue14357 closed by eric.araujo #14361: No link to issue tracker on Python home page http://bugs.python.org/issue14361 closed by r.david.murray #14368: floattime() should not raise an exception http://bugs.python.org/issue14368 closed by haypo #14381: Intern certain integral floats for memory savings and performa http://bugs.python.org/issue14381 closed by krisvale #14383: Generalize the use of _Py_IDENTIFIER in ceval.c and typeobject http://bugs.python.org/issue14383 closed by haypo #14391: misc TYPO in argparse.Action docstring http://bugs.python.org/issue14391 closed by terry.reedy #14395: sftp: downloading files with % in name fails due to logging http://bugs.python.org/issue14395 closed by r.david.murray #14400: Typo in cporting.rst http://bugs.python.org/issue14400 closed by loewis #14401: Typos in curses.rst http://bugs.python.org/issue14401 closed by python-dev #14402: Notice PayPaI : Your account was accesed by a third party. http://bugs.python.org/issue14402 closed by gvanrossum #14403: unittest module: provide inverse of "assertRaises" http://bugs.python.org/issue14403 closed by michael.foord #14404: multiprocessing with maxtasksperchild: bug in control logic? http://bugs.python.org/issue14404 closed by neologix #14409: IDLE: not executing commands from shell, error with default ke http://bugs.python.org/issue14409 closed by serwy #14410: argparse typo http://bugs.python.org/issue14410 closed by sandro.tosi #14411: outdatedness on rlcompleter docstring http://bugs.python.org/issue14411 closed by python-dev #14413: whatsnew deprecation tweak http://bugs.python.org/issue14413 closed by r.david.murray #14415: ValueError: I/O operation on closed file http://bugs.python.org/issue14415 closed by r.david.murray #14416: syslog missing constants http://bugs.python.org/issue14416 closed by r.david.murray #14421: Avoid ResourceWarnings in ccbench http://bugs.python.org/issue14421 closed by python-dev #14435: Remove special block allocation from floatobject.c http://bugs.python.org/issue14435 closed by krisvale #14436: SocketHandler sends obejcts while they cannot be unpickled on http://bugs.python.org/issue14436 closed by python-dev #14438: _cursesmodule build fails on cygwin http://bugs.python.org/issue14438 closed by luch #14441: Add new date/time directives to strftime() http://bugs.python.org/issue14441 closed by r.david.murray #14442: test_smtplib.py lacks "import errno" http://bugs.python.org/issue14442 closed by r.david.murray #14447: marshal.load() reads entire remaining file instead of just nex http://bugs.python.org/issue14447 closed by r.david.murray #14451: sum, min, max only works with iterable http://bugs.python.org/issue14451 closed by r.david.murray From stefan_ml at behnel.de Fri Mar 30 18:18:46 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 30 Mar 2012 18:18:46 +0200 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <4F75DA77.6040305@gmail.com> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> Message-ID: Etienne Robillard, 30.03.2012 18:08: > are you also truth allergic or irritated by the consequences of > free speech ? Please note that "free speech" is a concept that is different from asking beginner's computer science questions on the core developer mailing list of a software development project. This is not the right forum to do so, and you should therefore move your "free speech" to one that is more appropriate. Nick has pointed you to one such forum and you would be well advised to use it - that's all I was trying to say. I hope it's clearer now. Stefan From animelovin at gmail.com Fri Mar 30 20:00:50 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Fri, 30 Mar 2012 14:00:50 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> Message-ID: <4F75F4D2.4050706@gmail.com> your reasoning is pathetic at best. i pass... Thanks for the tip :-) Cheers, Etienne On 03/30/2012 12:18 PM, Stefan Behnel wrote: > Etienne Robillard, 30.03.2012 18:08: >> are you also truth allergic or irritated by the consequences of >> free speech ? > > Please note that "free speech" is a concept that is different from asking > beginner's computer science questions on the core developer mailing list of > a software development project. This is not the right forum to do so, and > you should therefore move your "free speech" to one that is more > appropriate. Nick has pointed you to one such forum and you would be well > advised to use it - that's all I was trying to say. I hope it's clearer now. > > Stefan > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/animelovin%40gmail.com > From andrew.svetlov at gmail.com Fri Mar 30 20:31:01 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 30 Mar 2012 21:31:01 +0300 Subject: [Python-Dev] [Python-checkins] cpython (3.2): Issue #14409: IDLE doesn't not execute commands from shell with default In-Reply-To: References: Message-ID: Thank you for mentoring. I will fix NEWS if you help me with better text. The bug fixed is that commit is: IDLE has 3 configs: user, system default and hardcoded in python code. Last one had bad binding for key. Usually this config is never used: user or system ones overrides former. But if IDLE cannot open config files by some reason it switches to hardcoded configs and user got broken IDLE. Can anybody guess me short and descriptive message describing what fix well? On Fri, Mar 30, 2012 at 6:12 AM, Nick Coghlan wrote: > On Fri, Mar 30, 2012 at 2:01 AM, andrew.svetlov > wrote: >> +- Issue #14409: IDLE doesn't not execute commands from shell, >> + ?error with default keybinding for Return. (Patch by Roger Serwy) > > The double negative here makes this impossible to understand. Could we > please get an updated NEWS entry that explains what actually changed > in IDLE to fix this? > > Perhaps something like "IDLE now always sets the default keybind for > Return correctly, ensuring commands can be executed in the IDLE shell > window"? (assuming that's what happened). > > This is important, folks: NEWS entries need to be comprehensible for > people that *haven't* read the associated tracker issue. This means > that issue titles (which generally describe a problem someone was > having) are often inappropriate as NEWS items. NEWS items should be > short descriptions that clearly describe *what changed*, perhaps with > some additional information to explain a bit about why the change was > made. > > Regards, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins -- Thanks, Andrew Svetlov From ethan at stoneleaf.us Fri Mar 30 20:23:18 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 30 Mar 2012 11:23:18 -0700 Subject: [Python-Dev] OT: refusal to listen and learn [was Re: Issue 14417: consequences of new dict runtime error] In-Reply-To: <4F75F4D2.4050706@gmail.com> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> <4F75F4D2.4050706@gmail.com> Message-ID: <4F75FA16.6040704@stoneleaf.us> Etienne Robillard wrote: > your reasoning is pathetic at best. i pass... Thanks for the tip :-) The Python Developer list is for the discussion of developing Python, not for teaching basic programming. You are being rude, and a smiley does not make you less rude. I am adding you to my kill-file so I no longer see messages from you because you refuse to follow the advice you have been given. ~Ethan~ From animelovin at gmail.com Fri Mar 30 20:56:20 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Fri, 30 Mar 2012 14:56:20 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <4F75FA16.6040704@stoneleaf.us> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> <4F75F4D2.4050706@gmail.com> <4F75FA16.6040704@stoneleaf.us> Message-ID: <4F7601D4.1080706@gmail.com> On 03/30/2012 02:23 PM, Ethan Furman wrote: > Etienne Robillard wrote: >> your reasoning is pathetic at best. i pass... Thanks for the tip :-) > > The Python Developer list is for the discussion of developing Python, > not for teaching basic programming. > > You are being rude, and a smiley does not make you less rude. > > I am adding you to my kill-file so I no longer see messages from you > because you refuse to follow the advice you have been given. > > ~Ethan~ > Add me to whatever file you want, but I believe the consequences for the new dict runtime errors just won't be resolved this way, and neither by systematically blocking alternative opinions as OT will help, because thats typically oppression, not free speech. :-) Cheers, :-) Etienne From regebro at gmail.com Fri Mar 30 21:01:07 2012 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 30 Mar 2012 21:01:07 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: The overview of the different monotonic clocks was interesting, because only one of them is adjusted by NTP, and that's the unix CLOCK_MONOTONIC. Hence we don't need a raw=False flag, which I previously suggested, we only need to not use CLOCK_MONOTONIC (which the PEP psuedo-code indeed also does not do, so that's all good). That means I think the PEP is fine now, if we rename highres(). time.time() already gets the highest resolution clock it can. Hence a highres() is confusing as the name implies that it returns a higher resolution clock than time.time(). And the name does not in any way indicate that the returned clock might be monotonic. try_monotonic() seems the obvious choice, since that's what it actually does. //Lennart From animelovin at gmail.com Fri Mar 30 21:13:36 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Fri, 30 Mar 2012 15:13:36 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> <4F75F4D2.4050706@gmail.com> <4F75FA16.6040704@stoneleaf.us> <4F7601D4.1080706@gmail.com> Message-ID: <4F7605E0.6070503@gmail.com> On 03/30/2012 03:02 PM, Guido van Rossum wrote: > Hey Etienne, I am honestly trying to understand your contribution but > you seem to have started a discussion about free speech. Trust me that > we don't mind your contributions, we're just trying to understand what > you're saying, and the free speech discussion isn't helping with that. I agree. > So if you have a comment on the dict mutation problem, please say so. OK. > If you need help understanding the problem, python-dev is not the > place to ask questions; you could ask on the bug, or on the > core-mentorship list as Nick suggested. But please stop bringing up > free speech, that's not an issue. Guido, thanks for the wisdom and clarity of your reasoning. I really appreciate a positive attitude towards questioning not so obvious problems. So far I was only attempting to verify whether this is related to PEP-416 or not. If this is indeed related PEP 416, then I must obviously attest that I must still understand why a immutable dict would prevent this bug or not... Either ways, I fail to see where this is OT or should be discussed on a more obscur forum than python-dev. :-) Kind regards, Etienne From rdmurray at bitdance.com Fri Mar 30 21:25:44 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 30 Mar 2012 15:25:44 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <4F7605E0.6070503@gmail.com> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> <4F75F4D2.4050706@gmail.com> <4F75FA16.6040704@stoneleaf.us> <4F7601D4.1080706@gmail.com> <4F7605E0.6070503@gmail.com> Message-ID: <20120330192545.1CC4A2500E9@webabinitio.net> On Fri, 30 Mar 2012 15:13:36 -0400, Etienne Robillard wrote: > So far I was only attempting to verify whether this is related to > PEP-416 or not. If this is indeed related PEP 416, then I must obviously > attest that I must still understand why a immutable dict would prevent > this bug or not... OK, that seems to be the source of your confusion, then. This has nothing to do with PEP-416. We are talking about issue Issue 14417 (like it says in the subject), which in turn is a reaction to the fix for issue 14205. --David From guido at python.org Fri Mar 30 21:27:15 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 12:27:15 -0700 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <4F7605E0.6070503@gmail.com> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> <4F75F4D2.4050706@gmail.com> <4F75FA16.6040704@stoneleaf.us> <4F7601D4.1080706@gmail.com> <4F7605E0.6070503@gmail.com> Message-ID: On Fri, Mar 30, 2012 at 12:13 PM, Etienne Robillard wrote: > On 03/30/2012 03:02 PM, Guido van Rossum wrote: >> >> Hey Etienne, I am honestly trying to understand your contribution but >> you seem to have started a discussion about free speech. Trust me that >> we don't mind your contributions, we're just trying to understand what >> you're saying, and the free speech discussion isn't helping with that. > > > I agree. > > >> So if you have a comment on the dict mutation problem, please say so. > > > OK. > > >> If you need help understanding the problem, python-dev is not the >> place to ask questions; you could ask on the bug, or on the >> core-mentorship list as Nick suggested. But please stop bringing up >> free speech, that's not an issue. > > > Guido, thanks for the wisdom and clarity of your reasoning. I really > appreciate a positive attitude towards questioning not so obvious problems. > > So far I was only attempting to verify whether this is related to PEP-416 or > not. If this is indeed related PEP 416, then I must obviously attest that I > must still understand why a immutable dict would prevent this bug or not... It's not related to PEP 416 (which was rejected). Please refer to http://bugs.python.org/issue14417 for the issue being discussed. > Either ways, I fail to see where this is OT or should be discussed on a more > obscur forum than python-dev. :-) We need to keep that list clear for important discussions. It is the only channel that the core Python developers have. If it has too much noise people will stop reading it and it stops functioning. Hence, we try to keep questions from newbies to a minimum -- there are other places where we welcome such questions though. So, once more, if you don't understand the issue and cannot figure it out by reading up, please ask somewhere else (or just accept that you don't have anything to contribute to this particular issue). This includes explaining basic terms like "mutate". On the other hand, if you *do* understand the problem, by all means let us know what you think of the question at hand (whether the change referred to in the issue is going to break people's code or not). We don't need more speculation though; that's how we got here in the first place (my speculation that it's not going to be an issue vs. RDM's speculation that it's going to cause widespread havoc :-). I hope you understand. -- --Guido van Rossum (python.org/~guido) From benjamin at python.org Fri Mar 30 21:30:59 2012 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 30 Mar 2012 15:30:59 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Issue #14065: Added cyclic GC support to ET.Element In-Reply-To: References: Message-ID: 2012/3/30 eli.bendersky : > http://hg.python.org/cpython/rev/0ca32013d77e > changeset: ? 75997:0ca32013d77e > parent: ? ? ?75995:cf2e74e0b7d4 > user: ? ? ? ?Eli Bendersky > date: ? ? ? ?Fri Mar 30 16:38:33 2012 +0300 > summary: > ?Issue #14065: Added cyclic GC support to ET.Element > > files: > ?Lib/test/test_xml_etree.py | ?27 ++++++++++- > ?Modules/_elementtree.c ? ? | ?63 +++++++++++++++++++------ > ?2 files changed, 74 insertions(+), 16 deletions(-) > > > diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py > --- a/Lib/test/test_xml_etree.py > +++ b/Lib/test/test_xml_etree.py > @@ -14,9 +14,10 @@ > ?# Don't re-import "xml.etree.ElementTree" module in the docstring, > ?# except if the test is specific to the Python implementation. > > -import sys > +import gc > ?import html > ?import io > +import sys > ?import unittest > > ?from test import support > @@ -1846,6 +1847,30 @@ > ? ? ? ? self.assertRaises(TypeError, e.extend, [ET.Element('bar'), 'foo']) > ? ? ? ? self.assertRaises(TypeError, e.insert, 0, 'foo') > > + ? ?def test_cyclic_gc(self): > + ? ? ? ?class ShowGC: > + ? ? ? ? ? ?def __init__(self, flaglist): > + ? ? ? ? ? ? ? ?self.flaglist = flaglist > + ? ? ? ? ? ?def __del__(self): > + ? ? ? ? ? ? ? ?self.flaglist.append(1) I think a nicer way to check for cyclic collection is to take a weakref to an object, call the GC, then check to make sure the weakref is broken. > + > + ? ? ? ?# Test the shortest cycle: lst->element->lst > + ? ? ? ?fl = [] > + ? ? ? ?lst = [ShowGC(fl)] > + ? ? ? ?lst.append(ET.Element('joe', attr=lst)) > + ? ? ? ?del lst > + ? ? ? ?gc.collect() support.gc_collect() is preferable > + ? ? ? ?self.assertEqual(fl, [1]) > + > + ? ? ? ?# A longer cycle: lst->e->e2->lst > + ? ? ? ?fl = [] > + ? ? ? ?e = ET.Element('joe') > + ? ? ? ?lst = [ShowGC(fl), e] > + ? ? ? ?e2 = ET.SubElement(e, 'foo', attr=lst) > + ? ? ? ?del lst, e, e2 > + ? ? ? ?gc.collect() > + ? ? ? ?self.assertEqual(fl, [1]) -- Regards, Benjamin From guido at python.org Fri Mar 30 21:40:25 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 12:40:25 -0700 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: On Fri, Mar 30, 2012 at 12:01 PM, Lennart Regebro wrote: > The overview of the different monotonic clocks was interesting, > because only one of them is adjusted by NTP, and that's the unix > CLOCK_MONOTONIC. Hence we don't need a raw=False flag, which I > previously suggested, we only need to not use CLOCK_MONOTONIC (which > the PEP psuedo-code indeed also does not do, so that's all good). Right on. > That means I think the PEP is fine now, if we rename highres(). > time.time() already gets the highest resolution clock it can. No, time.time() is the clock that can be mapped to and from "civil time". (Adjustments by NTP and the user notwithstanding.) The other clocks have a variable epoch and do not necessarily tick with a constant rate (e.g. they may not tick at all while the system is suspended). > Hence a > highres() is confusing as the name implies that it returns a higher > resolution clock than time.time(). And the name does not in any way > indicate that the returned clock might be monotonic. try_monotonic() > seems the obvious choice, since that's what it actually does. I am still unhappy with the two names, but I'm glad that we're this close. We need two new names; one for an OS-provided clock that is "monotonic" or "steady" or whatever you want to call it, but which may not exist on all systems (some platforms don't have it, some host may not have it even though the platform generally does have it). The other name is for a clock that's one or the other; it should be the OS-provided clock if it exists, otherwise time.time(). Most code should probably use this one, so perhaps its name should be the shorter one. C++ calls these steady_clock and high_resolution_clock, respectively. But it also calls the civil time clock system_clock, so perhaps we shouldn't feel to bound by it (except that we *shouldn't* call something steady if it isn't). I still think the name "monotonic" give the wrong impression; I would be happy calling it steady. But for the other, I'm still at a loss, and that name is the most important one. We can't call it steady because it isn't always. highres or hires sounds awkward; try_monotonic or try_steady are even more awkward. I looked in an online thesaurus and here's a list of what it gave: Big Ben, alarm, chroniker, chronograph, chronometer, digital watch, hourglass, metronome, pendulum, stopwatch, sundial, tattler, tick-tock, ticker, timekeeper, timemarker, timepiece, timer, turnip, watch I wonder if something with tick would work? (Even though it returns a float. :-) If all else fails, I'd go with turnip. -- --Guido van Rossum (python.org/~guido) From victor.stinner at gmail.com Fri Mar 30 22:21:28 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 30 Mar 2012 22:21:28 +0200 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? Message-ID: Hi, Windows provides two main monotonic clocks: QueryPerformanceCounter() and GetTickCount(). QueryPerformanceCounter() has a good accuracy (0.3 ns - 5 ns) but has various issues and is know to not have a steady rate. GetTickCount() has a worse accuracy (1 ms - 15 ms) but is more stable and behave better on system suspend/resume. The glib library prefers GetTickCount() over QueryPerformanceCounter() for its g_get_monotonic_time() because "The QPC timer has too many issues to be used as is." Ihttp://mail.gnome.org/archives/commits-list/2011-November/msg04589.html The Qt library tries QueryPerformanceCounter() but may fallback to GetTickCount() if it is not available. python-monotonic-time only uses GetTickCount() or GetTickCount64(). It is important to decide which clock is used for the Python time.monotonic() because it may change the design of the PEP 418. If we use GetTickCount() for time.monotonic(), we should use QueryPerformanceCounter() for time.highres(). But in this case, it means that time.highres() is not a simple "try monotonic or falls back to system time", but may use a different clock with an higher resolution. So we might add a third function for the "try monotonic or falls back to system time" requirement. Python implements time.clock() using QueryPerformanceCounter() on Windows. Victor From rdmurray at bitdance.com Fri Mar 30 22:44:55 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 30 Mar 2012 16:44:55 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: <20120330204455.A82AB2500E9@webabinitio.net> On Fri, 30 Mar 2012 12:40:25 -0700, Guido van Rossum wrote: > But for the other, I'm still at a loss, and that name is the most > important one. We can't call it steady because it isn't always. > highres or hires sounds awkward; try_monotonic or try_steady are even > more awkward. I looked in an online thesaurus and here's a list of > what it gave: > > Big Ben, alarm, chroniker, chronograph, chronometer, digital watch, > hourglass, metronome, pendulum, stopwatch, sundial, tattler, > tick-tock, ticker, timekeeper, timemarker, timepiece, timer, turnip, > watch > > I wonder if something with tick would work? (Even though it returns a float. :-) > > If all else fails, I'd go with turnip. We could call it "alice"[*]: sometimes it goes fast, sometimes it goes slow, sometimes it even goes backward, but it does try to tell you when you are late. --David [*] 'whiterabbit' would be more descriptive, but that's longer than turnip. From yselivanov.ml at gmail.com Fri Mar 30 22:48:21 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 30 Mar 2012 16:48:21 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: <4F88077B-1C56-4E0E-955A-DDAE2EFA6C4A@gmail.com> On 2012-03-30, at 3:40 PM, Guido van Rossum wrote: > I still think the name "monotonic" give the wrong impression; I would > be happy calling it steady. Simple google search comparison shows that people ask about 'monotonic' clock in python, not 'steady'. How about following Nick's (if I recall correctly) proposal of calling the OS function - '_monotonic', and a python wrapper - 'monotonic'? And one more question: what do you think about introducing a special check, that will ensure that our python implementation of 'monotonic', in case of fallback to 'time.time()', raises an exception if time suddenly goes backward? - Yury From guido at python.org Fri Mar 30 22:52:50 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 13:52:50 -0700 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: On Fri, Mar 30, 2012 at 1:21 PM, Victor Stinner wrote: > Windows provides two main monotonic clocks: QueryPerformanceCounter() > and GetTickCount(). QueryPerformanceCounter() has a good accuracy (0.3 > ns - 5 ns) but has various issues and is know to not have a steady > rate. GetTickCount() has a worse accuracy (1 ms - 15 ms) but is more > stable and behave better on system suspend/resume. > > The glib library prefers GetTickCount() over QueryPerformanceCounter() > for its g_get_monotonic_time() because "The QPC timer has too many > issues to be used as is." > Ihttp://mail.gnome.org/archives/commits-list/2011-November/msg04589.html > > The Qt library tries QueryPerformanceCounter() but may fallback to > GetTickCount() if it is not available. > > python-monotonic-time only uses GetTickCount() or GetTickCount64(). > > It is important to decide which clock is used for the Python > time.monotonic() because it may change the design of the PEP 418. If > we use GetTickCount() for time.monotonic(), we should use > QueryPerformanceCounter() for time.highres(). But in this case, it > means that time.highres() is not a simple "try monotonic or falls back > to system time", but may use a different clock with an higher > resolution. So we might add a third function for the "try monotonic or > falls back to system time" requirement. > > Python implements time.clock() using QueryPerformanceCounter() on Windows. Oh dear. I really want to say that 15 ms is good enough. Some possible exceptions I can think of: - Profiling. But this really wants to measure CPU time anyways, and it already uses a variety of hacks and heuristics to pick the best timer, so I don't really care. - Certain algorithms for thread (or even process?) communication might benefit from being able to set very short timeouts. But it seems you can just specify a timeout when waiting for a lock and it will be taken care of. So probably a non-use-case. So I think I still prefer GetTickCount*() over QueryPerformanceCounter(). And this is another reason why I don't like highres as a name -- as I said, I really prefer to have just two new functions, one that's a decent OS-provided timer that isn't affected by wall clock adjustments, if there is one, and the other strictly an alias for that one with a fallback to time.time() if there isn't one. Having two different OS-provided timers with different pros and cons will just make it too hard for users to decide, and hence we'll see a higher rate of using the wrong timer. (In fact, this even argues against having both the timer with fallback and the timer without fallback. So maybe we should just have a single timer function, *with* fallback, and a separate mechanism to inquire its properties.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Mar 30 23:35:29 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 14:35:29 -0700 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: On Fri, Mar 30, 2012 at 1:52 PM, Guido van Rossum wrote: > On Fri, Mar 30, 2012 at 1:21 PM, Victor Stinner > wrote: >> Windows provides two main monotonic clocks: QueryPerformanceCounter() >> and GetTickCount(). QueryPerformanceCounter() has a good accuracy (0.3 >> ns - 5 ns) but has various issues and is know to not have a steady >> rate. GetTickCount() has a worse accuracy (1 ms - 15 ms) but is more >> stable and behave better on system suspend/resume. >> >> The glib library prefers GetTickCount() over QueryPerformanceCounter() >> for its g_get_monotonic_time() because "The QPC timer has too many >> issues to be used as is." >> Ihttp://mail.gnome.org/archives/commits-list/2011-November/msg04589.html >> >> The Qt library tries QueryPerformanceCounter() but may fallback to >> GetTickCount() if it is not available. >> >> python-monotonic-time only uses GetTickCount() or GetTickCount64(). >> >> It is important to decide which clock is used for the Python >> time.monotonic() because it may change the design of the PEP 418. If >> we use GetTickCount() for time.monotonic(), we should use >> QueryPerformanceCounter() for time.highres(). But in this case, it >> means that time.highres() is not a simple "try monotonic or falls back >> to system time", but may use a different clock with an higher >> resolution. So we might add a third function for the "try monotonic or >> falls back to system time" requirement. >> >> Python implements time.clock() using QueryPerformanceCounter() on Windows. > > Oh dear. I really want to say that 15 ms is good enough. Some possible > exceptions I can think of: > > - Profiling. But this really wants to measure CPU time anyways, and it > already uses a variety of hacks and heuristics to pick the best timer, > so I don't really care. > > - Certain algorithms for thread (or even process?) communication might > benefit from being able to set very short timeouts. But it seems you > can just specify a timeout when waiting for a lock and it will be > taken care of. So probably a non-use-case. > > So I think I still prefer GetTickCount*() over QueryPerformanceCounter(). > > And this is another reason why I don't like highres as a name -- as I > said, I really prefer to have just two new functions, one that's a > decent OS-provided timer that isn't affected by wall clock > adjustments, if there is one, and the other strictly an alias for that > one with a fallback to time.time() if there isn't one. Having two > different OS-provided timers with different pros and cons will just > make it too hard for users to decide, and hence we'll see a higher > rate of using the wrong timer. > > (In fact, this even argues against having both the timer with fallback > and the timer without fallback. So maybe we should just have a single > timer function, *with* fallback, and a separate mechanism to inquire > its properties.) And disagreeing with myself, I just found myself using a tool for RPC analytics that displays times in msec, and the times are often between 1 and 20 msec. Here a 15 msec timer would be horrible, and I see no alternative but to use QPC()... (Well, if the tool ran on Windows, which it doesn't. :-) Can we tell what the accuracy of GetTickCount*() is when we first request the time, and decode then? Can you go into more detail about QPC()'s issues? What is unsteady about its rate? -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Mar 30 23:40:12 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 14:40:12 -0700 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: Possibly we really do need two timers, one for measuring short intervals and one for measuring long intervals? Perhaps we can use this to decide on the names? Anyway, the more I think about it, the more I believe these functions should have very loose guarantees, and instead just cater to common use cases -- availability of a timer with minimal fuss is usually more important than the guarantees. So forget the idea about one version that falls back to time.time() and another that doesn't -- just always fall back to time.time(), which is (almost) always better than failing. Then we can design a separate inquiry API (doesn't have to be complex as long as it's extensible -- a dict or object with a few predefined keys or attributes sounds good enough) for apps that want to know more about how the timer they're using is actually implemented. -- --Guido van Rossum (python.org/~guido) From cs at zip.com.au Fri Mar 30 23:43:19 2012 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 31 Mar 2012 08:43:19 +1100 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: <20120330214319.GA3106@cskk.homeip.net> On 30Mar2012 13:52, Guido van Rossum wrote: | (In fact, this even argues against having both the timer with fallback | and the timer without fallback. So maybe we should just have a single | timer function, *with* fallback, and a separate mechanism to inquire | its properties.) Well, you should be able to know what kind of clock you're getting. But what do to if you don't get a satisfactory one? There's then no way for ask for an alternative; you don't get to control the fallback method, for example. I've come to the opinion that the chosen approach is wrong, and suggest a better one. There seem to be a few competing features for clocks that people want: - monotonic - never going backward at all - high resolution - no steps and only a few, fortunately. I think you're doing this wrong at the API level. People currently are expecting to call (for example) time.monotonic() (or whatever) and get back "now", a hopefully high resolution float. Given the subtlety sought for various purposes, people should be calling: T = time.getclock(flags) and then later calling: T.now() to get their float. That way people can: - request a set of the three characteristics above - inspect what they get back (it should have all the requested flags, but unrequested flags may be set or not depending on the underlying facility) Obviously this should return None or raise an exception if the flag set can't be met. Then users can request a less featured clock on failure, depending on what aspects of the clock are most desirable to their use case. Or of course fail if fallback is not satisfactory. Of course you can provide some convenient-with-fallback function that will let people do this in one hit without the need for "T", but it should not be the base facility offered. The base should let people request their feature set and inspect what is supplied. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Time is nature's way of keeping everything from happening at once. From guido at python.org Sat Mar 31 00:21:32 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 15:21:32 -0700 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: <20120330214319.GA3106@cskk.homeip.net> References: <20120330214319.GA3106@cskk.homeip.net> Message-ID: On Fri, Mar 30, 2012 at 2:43 PM, Cameron Simpson wrote: > On 30Mar2012 13:52, Guido van Rossum wrote: > | (In fact, this even argues against having both the timer with fallback > | and the timer without fallback. So maybe we should just have a single > | timer function, *with* fallback, and a separate mechanism to inquire > | its properties.) > > Well, you should be able to know what kind of clock you're getting. But > what do to if you don't get a satisfactory one? There's then no way for > ask for an alternative; you don't get to control the fallback method, > for example. > > I've come to the opinion that the chosen approach is wrong, and suggest > a better one. > > There seem to be a few competing features for clocks that people want: > > ?- monotonic - never going backward at all > ?- high resolution > ?- no steps > > and only a few, fortunately. > > I think you're doing this wrong at the API level. People currently are > expecting to call (for example) time.monotonic() (or whatever) and get > back "now", a hopefully high resolution float. > > Given the subtlety sought for various purposes, people should be > calling: > > ?T = time.getclock(flags) > > and then later calling: > > ?T.now() > > to get their float. > > That way people can: > ?- request a set of the three characteristics above > ?- inspect what they get back (it should have all the requested flags, > ? ?but unrequested flags may be set or not depending on the underlying > ? ?facility) > ? ?Obviously this should return None or raise an exception if the flag > ? ?set can't be met. > > Then users can request a less featured clock on failure, depending on > what aspects of the clock are most desirable to their use case. Or of > course fail if fallback is not satisfactory. > > Of course you can provide some convenient-with-fallback function that > will let people do this in one hit without the need for "T", but it > should not be the base facility offered. The base should let people > request their feature set and inspect what is supplied. I like this out-of-the-box thinking. But I'm still wondering if there really are enough flags for this to be worth it. If there are, great, the API is pretty. But if there are only 2 or 3 flag combinations that make actual sense, let's forget it. Another out-of-the-box idea, going back to simplicity: have just one new function, time.hrtimer(), which is implemented using the highest-resolution timer available on the platform, but with no strong guarantees. It *may* jump, move back, drift, change its rate, or roll over occasionally. We try to use the implementation that's got the fewest problems, but we don't try to hide its deficiencies, and nothing suitable exists, it may be equivalent to time.time(). If the times you measure are too weird, measure again. For scheduling things a day or more in the future, you should use time.time() instead. One issue that hasn't had enough attention: *scope* of a timer. If two processes running on the same machine ask for the time, do the values they see use the same epoch, or is the epoch dependent on the process? Some code I saw in timemodule.c for working around Windows clocks rolling over seem to imply that two processes may not always see the same timer value. Is there a use case where this matters? -- --Guido van Rossum (python.org/~guido) From animelovin at gmail.com Sat Mar 31 00:28:25 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Fri, 30 Mar 2012 18:28:25 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> <4F75F4D2.4050706@gmail.com> <4F75FA16.6040704@stoneleaf.us> <4F7601D4.1080706@gmail.com> <4F7605E0.6070503@gmail.com> Message-ID: <4F763389.7090307@gmail.com> On 03/30/2012 03:27 PM, Guido van Rossum wrote: > On Fri, Mar 30, 2012 at 12:13 PM, Etienne Robillard > wrote: >> On 03/30/2012 03:02 PM, Guido van Rossum wrote: >>> >>> Hey Etienne, I am honestly trying to understand your contribution but >>> you seem to have started a discussion about free speech. Trust me that >>> we don't mind your contributions, we're just trying to understand what >>> you're saying, and the free speech discussion isn't helping with that. >> >> >> I agree. >> >> >>> So if you have a comment on the dict mutation problem, please say so. >> >> >> OK. >> >> >>> If you need help understanding the problem, python-dev is not the >>> place to ask questions; you could ask on the bug, or on the >>> core-mentorship list as Nick suggested. But please stop bringing up >>> free speech, that's not an issue. >> >> >> Guido, thanks for the wisdom and clarity of your reasoning. I really >> appreciate a positive attitude towards questioning not so obvious problems. >> >> So far I was only attempting to verify whether this is related to PEP-416 or >> not. If this is indeed related PEP 416, then I must obviously attest that I >> must still understand why a immutable dict would prevent this bug or not... > > It's not related to PEP 416 (which was rejected). Please refer to > http://bugs.python.org/issue14417 for the issue being discussed. > >> Either ways, I fail to see where this is OT or should be discussed on a more >> obscur forum than python-dev. :-) > > We need to keep that list clear for important discussions. It is the > only channel that the core Python developers have. If it has too much > noise people will stop reading it and it stops functioning. Hence, we > try to keep questions from newbies to a minimum -- there are other > places where we welcome such questions though. > > So, once more, if you don't understand the issue and cannot figure it > out by reading up, please ask somewhere else (or just accept that you > don't have anything to contribute to this particular issue). This > includes explaining basic terms like "mutate". On the other hand, if > you *do* understand the problem, by all means let us know what you > think of the question at hand (whether the change referred to in the > issue is going to break people's code or not). We don't need more > speculation though; that's how we got here in the first place (my > speculation that it's not going to be an issue vs. RDM's speculation > that it's going to cause widespread havoc :-). > > I hope you understand. No, not really. Anyways, I guess I'll have to further dig down why is PEP-416 is really important to Python and why it was likewise rejected, supposing I confused the pep 416 and issue 14417 along the way.. :-) CHeers, Etienne From victor.stinner at gmail.com Sat Mar 31 00:47:45 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 31 Mar 2012 00:47:45 +0200 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: > Can you go into more detail about QPC()'s issues? Yes, see the PEP: http://www.python.org/dev/peps/pep-0418/#windows-queryperformancecounter > What is unsteady about its rate? Hum, I read that you can expect a drift between QPC and GetTickCount. I don't have exact numbers. There are also issues with power-saving. When the CPU frequency changes, the frequency of the TSC is also impacted on some processors (others provide a TSC at a fixed frequency). 2012/3/30 Guido van Rossum : > Possibly we really do need two timers, one for measuring short > intervals and one for measuring long intervals? Perhaps we can use > this to decide on the names? "short" and "long" interval is not how these clocks should be defined. I prefer use cases: * time.highres() should be used for profiling and benchmarking * time.monotonic() should be used for a scheduler and timeout time.monotonic() rate should be as steady as possible, its stability is more important than its accuracy. time.highres() should provide the clock with the best accuracy. Both clocks may have an undefined starting point. > Anyway, the more I think about it, the more I believe these functions > should have very loose guarantees, and instead just cater to common > use cases -- availability of a timer with minimal fuss is usually more > important than the guarantees. So forget the idea about one version > that falls back to time.time() and another that doesn't -- just always > fall back to time.time(), which is (almost) always better than > failing. I disagree. Python must provide a truly monotonic clock, even it is not truly monotonic by default. If we want to provide a convinient "monotonic or falllback to system time", there are different options: add a dedicated function (e.g. time.try_monotonic()) or add a parameter to enable or disable the fallback. Between these two options, I prefer the parameter because it avoids the creation of a new function. Even if I now agree that a truly monotonic clock is required, I also agree with Guido: the most common case is to use a fallback. I suggest this API: time.monotonic(fallback=True). Another option is to not provide a "monotonic or falllback to system time" clock and write an explicit try/except where you need a fallback. I don't like this solution because time.monotonic() might fail at runtime with OSError, and so catching AttributeError is not enough... > Then we can design a separate inquiry API (doesn't have to be complex > as long as it's extensible -- a dict or object with a few predefined > keys or attributes sounds good enough) for apps that want to know more > about how the timer they're using is actually implemented. Sometimes, it does matter which exact OS clock is used. The QElapsedTime class of the Qt libraries has an attribute (an enum) describing which clock is used. We can give the name of the function and the resolution. In some specific cases, we are able to read the accuracy (ex: GetTickCount). The accuracy may change at runtime (at least for GetTickCount). Should I include such new function in the PEP, or can it be discussed (later) in a separated thread? -- I don't want to only provide time.monotonic() which would fallback if no monotonic clock is available, because we would be unable to tell if the clock is monotonic or not. See the implementation in the PEP: the "try monotonic or fallback" function may use a different clock at each call, and so may be monotonic at startup and then becomes non-monotonic. An alternative is time.monotonic() returning (time, is_monotonic) but this is an ugly API. You cannot simply write: a=time.monotonic(); benchmark(); b=time.monotonic(); print(b-a) because it raises a TypeError on b-a. -- I will update the PEP to describe my new proposition. Victor From victor.stinner at gmail.com Sat Mar 31 01:24:28 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 31 Mar 2012 01:24:28 +0200 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <4F763389.7090307@gmail.com> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> <4F75F4D2.4050706@gmail.com> <4F75FA16.6040704@stoneleaf.us> <4F7601D4.1080706@gmail.com> <4F7605E0.6070503@gmail.com> <4F763389.7090307@gmail.com> Message-ID: > No, not really. Anyways, I guess I'll have to further dig down why is > PEP-416 is really important to Python and why it was likewise rejected, > supposing I confused the pep 416 and issue 14417 along the way.. :-) The frozendict builtin type was rejected, but we are going to add types.MappingProxyType: see issue #14386. types.MappingProxyType(mydict.copy()) is very close to the frozendict builtin type. Victor From cs at zip.com.au Sat Mar 31 01:44:00 2012 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 31 Mar 2012 10:44:00 +1100 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: <20120330234359.GA11489@cskk.homeip.net> On 30Mar2012 15:21, Guido van Rossum wrote: | On Fri, Mar 30, 2012 at 2:43 PM, Cameron Simpson wrote: | > Given the subtlety sought for various purposes, people should be | > calling: | > ?T = time.getclock(flags) | > and then later calling: | > ?T.now() | > to get their float. | > | > That way people can: | > ?- request a set of the three characteristics above | > ?- inspect what they get back (it should have all the requested flags, | > ? ?but unrequested flags may be set or not depending on the underlying | > ? ?facility) | > ? ?Obviously this should return None or raise an exception if the flag | > ? ?set can't be met. | > | > Then users can request a less featured clock on failure, depending on | > what aspects of the clock are most desirable to their use case. Or of | > course fail if fallback is not satisfactory. [...] | I like this out-of-the-box thinking. But I'm still wondering if there | really are enough flags for this to be worth it. If there are, great, | the API is pretty. But if there are only 2 or 3 flag combinations that | make actual sense, let's forget it. There are at least three characteristics listed above. There are several use cases described in the long discussions. I don't think we should be saying "we will only support these 3 thought-of use cases and the others are too hard/weird/perverse/rare". We should be saying: we can characterise clocks in these very few ways. Ask for a clock of some kind. We will provide one (abitrarily, though perhaps preferring higher precision) that satisfies the request if available. I can easily envisage calling code as simple as this: T = getclock(T_MONOTONIC|T_HIGHRES) or getclock(T_MONOTONIC) to do my own fallback for my own use case if we went with returning None if there is no satisfactory clock. If I need monotonic (for example) my code is in NO WAY helped by being handed something that sometimes is not monotonic; to make my code reliable I would need to act as though I wasn't getting a montonic clock in the first place. No win there at all. This has the following advantages: - it removes _all_ policy from the PEP, at least at the core - it makes things totally clear and concise at the calling end; the clock characterists are right there in front of the reader - policy is in the hands of the user where it belongs, and is easy to specify into the bargain - it avoids the horrendous ideas mooted of asking for something called monotonic and getting sometime only mostly monotonic, and the equivalent flavours for the other flags - implementation of the PEP is in principle as easy as a table of clock wrappers with the flags each supports. Loop over the table until one matches all the requested flags, or return None. - for higher quality of implementation you embed policy in the convenience function, not the getclock() function To take your suggestion of: Another out-of-the-box idea, going back to simplicity: have just one new function, time.hrtimer(), which is implemented using the highest-resolution timer available on the platform, but with no strong guarantees. Example: T = time.hrtimer() which would provide the "best" hires timer. You'd give it an optional flag argument for additional flags so that you could request the most-hires timer matching the flags. This is like: T = time.getclock(T_HIRES) except that is makes clear your preferred attribute. So internally one would have: - clock wrappers for the platform specific facilities, with the constructors being singletons or not as suits the facility - a list of the wrappers with their flag sets specifiying what clock attributes they offer - for the .hrtimer([flags]) or .monotonic([flags]) calls they each have their own list of the clock wrapper, ordered on most-hires or best-monotonic, as the implementor sees fit That gets you: - arbitrary user case support - convenience functions for common policies | It *may* jump, move back, drift, change its rate, or roll | over occasionally. We try to use the implementation that's got the | fewest problems, but we don't try to hide its deficiencies, and | nothing suitable exists, it may be equivalent to time.time(). If the | times you measure are too weird, measure again. For scheduling things | a day or more in the future, you should use time.time() instead. Getting ugly. This is why I think we should not be offering only what we think the user may want, especially when we offer something loose and rubbery like this. Instead, get the user to ask. The platform timers can all be well characterised with respect to the flags suggested above; you only need to tabulate what timers offer what behaviours. By all means offer convenience functions providing common choices, but don't make those the _only_ choices. Let the user ask for anything; if the platform can't support it that's a pity, but leave the user the choice of asking for less or making whatever other decision suits them. | One issue that hasn't had enough attention: *scope* of a timer. If two | processes running on the same machine ask for the time, do the values | they see use the same epoch, or is the epoch dependent on the process? | Some code I saw in timemodule.c for working around Windows clocks | rolling over seem to imply that two processes may not always see the | same timer value. Is there a use case where this matters? Make the epoch available in the clock wrapper as a property. At least then there's a mechanism for reconciling things. Don't try to mandate something that possibly can't be mandated. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ My computer always does exactly what I tell it to do but sometimes I have trouble finding out what it was that I told it to do. - Dick Wexelblat From guido at python.org Sat Mar 31 01:46:13 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 16:46:13 -0700 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: Given the amount of disagreement I sense, I think we'll need to wait for more people to chime in. On Fri, Mar 30, 2012 at 3:47 PM, Victor Stinner wrote: >> Can you go into more detail about QPC()'s issues? > > Yes, see the PEP: > http://www.python.org/dev/peps/pep-0418/#windows-queryperformancecounter > >> What is unsteady about its rate? > > Hum, I read that you can expect a drift between QPC and GetTickCount. > I don't have exact numbers. There are also issues with power-saving. > When the CPU frequency changes, the frequency of the TSC is also > impacted on some processors (others provide a TSC at a fixed > frequency). And how does that impact teh various use cases? (There still isn't enough about use cases in the PEP. Thank you very much though for adding all those details about the various platform clocks.) > 2012/3/30 Guido van Rossum : >> Possibly we really do need two timers, one for measuring short >> intervals and one for measuring long intervals? Perhaps we can use >> this to decide on the names? > > "short" and "long" interval is not how these clocks should be defined. Why not? > I prefer use cases: > > ?* time.highres() should be used for profiling and benchmarking > ?* time.monotonic() should be used for a scheduler and timeout So if I want to schedule something a week in the future, what should I use? monotonic() or time()? And can you explain the reason for your answer? What about an hour or a day? Or a month or a year? > time.monotonic() rate should be as steady as possible, its stability > is more important than its accuracy. time.highres() should provide the > clock with the best accuracy. Both clocks may have an undefined > starting point. So what kind of drift is acceptable for each? >> Anyway, the more I think about it, the more I believe these functions >> should have very loose guarantees, and instead just cater to common >> use cases -- availability of a timer with minimal fuss is usually more >> important than the guarantees. So forget the idea about one version >> that falls back to time.time() and another that doesn't -- just always >> fall back to time.time(), which is (almost) always better than >> failing. > > I disagree. Python must provide a truly monotonic clock, even it is > not truly monotonic by default. Why? > If we want to provide a convenient "monotonic or falllback to system > time", there are different options: add a dedicated function (e.g. > time.try_monotonic()) or add a parameter to enable or disable the > fallback. Between these two options, I prefer the parameter because it > avoids the creation of a new function. Even if I now agree that a > truly monotonic clock is required, I also agree with Guido: the most > common case is to use a fallback. > > I suggest this API: time.monotonic(fallback=True). But this goes against the idea of "A keyword argument that gets passed as a constant in the caller is usually poor API." And it would encourage the creation of trivial lambdas just to call the timer with this flag, since a universal API is a timer function that takes no arguments. > Another option is to not provide a "monotonic or falllback to system > time" clock and write an explicit try/except where you need a > fallback. I don't like this solution because time.monotonic() might > fail at runtime with OSError, and so catching AttributeError is not > enough... Check out Cameron Simpson's proposal earlier in this thread. If we really are going to have a variety of functions and fallback options, his approach may be the best. >> Then we can design a separate inquiry API (doesn't have to be complex >> as long as it's extensible -- a dict or object with a few predefined >> keys or attributes sounds good enough) for apps that want to know more >> about how the timer they're using is actually implemented. > > Sometimes, it does matter which exact OS clock is used. When is that? > The QElapsedTime class of the Qt libraries has an attribute (an enum) > describing which clock is used. We can give the name of the function > and the resolution. In some specific cases, we are able to read the > accuracy (ex: GetTickCount). The accuracy may change at runtime (at > least for GetTickCount). That sounds like a fine API. > Should I include such new function in the PEP, or can it be discussed > (later) in a separated thread? FWIW, I prefer a single thread devoted to all aspects of this PEP. > -- > > I don't want to only provide time.monotonic() which would fallback if > no monotonic clock is available, because we would be unable to tell if > the clock is monotonic or not. There could be a separate inquiry API. > See the implementation in the PEP: the > "try monotonic or fallback" function may use a different clock at each > call, and so may be monotonic at startup and then becomes > non-monotonic. No, it should never switch implementations once it has chosen. That's just asking for trouble. If it can become non-monotonic it just isn't monotonic. Again, Cameron Simpson's idea might help here. > An alternative is time.monotonic() returning (time, is_monotonic) but > this is an ugly API. You cannot simply write: > ? a=time.monotonic(); benchmark(); b=time.monotonic(); print(b-a) > because it raises a TypeError on b-a. Agreed, that's not useful. It should be a constant property (constant within one process). > -- > > I will update the PEP to describe my new proposition. Thanks. Please pay specific attention to my why/where/when questions above. -- --Guido van Rossum (python.org/~guido) From guido at python.org Sat Mar 31 01:49:39 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 16:49:39 -0700 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: <20120330234359.GA11489@cskk.homeip.net> References: <20120330234359.GA11489@cskk.homeip.net> Message-ID: On Fri, Mar 30, 2012 at 4:44 PM, Cameron Simpson wrote: [Lots of good stuff] Maybe you and Victor should try to merge your proposals off-line and then get back with a new proposal here. > Make the epoch available in the clock wrapper as a property. At least > then there's a mechanism for reconciling things. Don't try to mandate > something that possibly can't be mandated. Ah, but if a clock drifts, the epoch may too -- and we may never know it. I like knowing all sorts of things about a clock, but I'm not sure that for clocks other than time.time() I'd ever want to know the epoch -- ISTM that the only thing I could do with that information would be shooting myself in the foot. If you really want the epoch, compute it yourself by bracketing a timer() call in two time() calls, or vice versa (not sure which is better :-). -- --Guido van Rossum (python.org/~guido) From steve at pearwood.info Sat Mar 31 02:26:35 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 31 Mar 2012 11:26:35 +1100 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> Message-ID: <4F764F3B.2020306@pearwood.info> Guido van Rossum wrote: > But for the other, I'm still at a loss, and that name is the most > important one. We can't call it steady because it isn't always. > highres or hires sounds awkward; try_monotonic or try_steady are even > more awkward. I looked in an online thesaurus and here's a list of > what it gave: "hires" is a real English word, the present tense verb for engaging the service or labour of someone or something in return for payment, as in "he hires a gardener to mow the lawn". Can we please eliminate it from consideration? It is driving me slowly crazy every time I see it used as an abbreviation for high resolution. > Big Ben, alarm, chroniker, chronograph, chronometer, digital watch, > hourglass, metronome, pendulum, stopwatch, sundial, tattler, > tick-tock, ticker, timekeeper, timemarker, timepiece, timer, turnip, > watch > > I wonder if something with tick would work? (Even though it returns a float. :-) > > If all else fails, I'd go with turnip. I can't tell if you are being serious or not. For the record, "turnip" in this sense is archaic slang for a thick pocket watch. -- Steven From victor.stinner at gmail.com Sat Mar 31 02:51:56 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 31 Mar 2012 02:51:56 +0200 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: 2012/3/31 Guido van Rossum : > Given the amount of disagreement I sense, I think we'll need to wait > for more people to chime in. I hope that the PEP now gives enough data to help to choose the best API. >> Hum, I read that you can expect a drift between QPC and GetTickCount. >> I don't have exact numbers. There are also issues with power-saving. >> When the CPU frequency changes, the frequency of the TSC is also >> impacted on some processors (others provide a TSC at a fixed >> frequency). > > And how does that impact teh various use cases? (There still isn't > enough about use cases in the PEP. Thank you very much though for > adding all those details about the various platform clocks.) I didn't find a good source, but it looks like QueryPerformanceCounter behaves badly on system suspend/resume, whereas GetTickCount is reliable and has a well defined behaviour: "The elapsed time retrieved by GetTickCount or GetTickCount64 includes time the system spends in sleep or hibernation." So QPC is not really monotonic nor steady. It may be even worse than the system clock (especially in virtual machines). Impact: A timeout may be 42 seconds shorter than the requested duration if is uses QPC. For a scheduler, a task would be scheduled at the right moment. >> I prefer use cases: >> >> ?* time.highres() should be used for profiling and benchmarking >> ?* time.monotonic() should be used for a scheduler and timeout > > So if I want to schedule something a week in the future, what should I > use? monotonic() or time()? And can you explain the reason for your > answer? What about an hour or a day? Or a month or a year? You should use time.monotonic() because time.monotonic() has a steady rate and so is reliable for long duration (at least more reliable than time.highres()). In practice, time.monotonic() and time.highres() are only different on Windows. If I understood correctly, the Windows kernel uses something like GetTickCount whch has an accuracy of 1 ms in the best case. So Python only needs a clock with a similar accuracy for timeout and schedulers. time.highres() (QPC) rate is only steady during a short duration: it is not an issue for a benchmark because you usually rerun the same test at least 3 times and keep the minimum. It looks like QPC bugs are only unexpected forward jumps (no backward jump), so taking the minimum would workaround these issues. The hibernation issue should not affect benchmarking/profiling. >> time.monotonic() rate should be as steady as possible, its stability >> is more important than its accuracy. time.highres() should provide the >> clock with the best accuracy. Both clocks may have an undefined >> starting point. > > So what kind of drift is acceptable for each? I don't know. >> I disagree. Python must provide a truly monotonic clock, even it is >> not truly monotonic by default. > > Why? See Zooko Wilcox-O'Hearn's email: http://mail.python.org/pipermail/python-dev/2012-March/118147.html >> I suggest this API: time.monotonic(fallback=True). > > But this goes against the idea of "A keyword argument that gets passed > as a constant in the caller is usually poor API." Right :-) After changing (again) the PEP, I realized that it reintroduces the time.monotonic(fallback=False) raising NotImplementedError case and I really don't like this exception here. > And it would encourage the creation of trivial lambdas just to call > the timer with this flag, since a universal API is a timer function > that takes no arguments. My bet is that fallback=True is an uncommon use case. >> Sometimes, it does matter which exact OS clock is used. > > When is that? If you need to know more properties of the clock,. For example, Python may be unable to give the accuracy of the clock, but you may get it differently if you know which clock function is used. >> Should I include such new function in the PEP, or can it be discussed >> (later) in a separated thread? > > FWIW, I prefer a single thread devoted to all aspects of this PEP. Let's propose an API: time.clock_info() -> {'time': {'function': 'gettimeofday', 'resolution': 1e-6, 'monotonic': False}, 'monotonic': {'function': 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': True}, 'highres': {'function': 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': True}} or time.clock_info('time') -> {'function': 'gettimeofday', 'resolution': 1e-6, 'monotonic': False} time.clock_info('monotonic') -> {'function': 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': True} time.clock_info( 'highres') -> {'function': 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': True} or time.clock_info(time.time) -> {'function': 'gettimeofday', 'resolution': 1e-6, 'monotonic': False} time.clock_info(time.monotonic) -> {'function': 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': True} time.clock_info( time.highres) -> {'function': 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': True} "clock_" prefix in "time.clock_xxx" name is used by POSIX clock_xxx functions, another prefix may be used instead. >> I don't want to only provide time.monotonic() which would fallback if >> no monotonic clock is available, because we would be unable to tell if >> the clock is monotonic or not. > > There could be a separate inquiry API. Well, the PEP mentions something like that in the "One function, no flag" section. >> See the implementation in the PEP: the >> "try monotonic or fallback" function may use a different clock at each >> call, and so may be monotonic at startup and then becomes >> non-monotonic. > > No, it should never switch implementations once it has chosen. That's > just asking for trouble. If it can become non-monotonic it just isn't > monotonic. Again, Cameron Simpson's idea might help here. Correct. In practice, one call to time.monotonic() is enough to know if next calls will fail or not, and so you don't have to poll regulary to check if the clock becomes non-monotonic. So always fallback and providing something to check if time.monotonic() is or not monotonic is be an acceptable solution. I agree that it is less ugly than the time.monotonic(fallback=True) API. Victor From glyph at twistedmatrix.com Sat Mar 31 03:09:10 2012 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 30 Mar 2012 21:09:10 -0400 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: <71C2DBD4-70D8-4232-880D-1EDB0775FFE8@twistedmatrix.com> On Mar 30, 2012, at 8:51 PM, Victor Stinner wrote: > time.highres() (QPC) rate is only steady during a short duration QPC is not even necessarily steady for a short duration, due to BIOS bugs, unless the code running your timer is bound to a single CPU core. mentions SetThreadAffinityMask for this reason, despite the fact that it is usually steady for longer than that. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Mar 31 03:24:53 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 31 Mar 2012 03:24:53 +0200 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: <20120330214319.GA3106@cskk.homeip.net> References: <20120330214319.GA3106@cskk.homeip.net> Message-ID: > There seem to be a few competing features for clocks that people want: > > ?- monotonic - never going backward at all > ?- high resolution These features look to be exclusive on Windows. On other platforms, it looks like monotonic clocks are always the most accurate clocks. So I don't think that we need to be able to combine these two "flags". > ?- no steps You mean "not adjusted by NTP"? Except CLOCK_MONOTONIC on Linux, no monotonic clock is adjusted by NTP. On Linux, there is CLOCK_MONOTONIC_RAW, but it is only available on recent Linux kernel (2.6.28). Do you think that it is important to be able to refuse a monotonic clock adjusted by NTP? What would be the use case of such truly steady clock? -- The PEP 418 tries to expose various monotonic clocks in Python with a simple API. If we fail to agree how to expose these clocks, e.g. because there are major differences between these clocks, another solution is to only expose low-level function. This is already what we do with the os module, and there is the shutil module (and others) for a higher level API. I already added new "low-level" functions to the time module: time.clock_gettime() and time.clock_getres(). > Of course you can provide some convenient-with-fallback function that > will let people do this in one hit without the need for "T", but it > should not be the base facility offered. The base should let people > request their feature set and inspect what is supplied. The purpose of such function is to fix programs written with Python < 3.3 and using time.time() whereas a monotonic would be the right clock (e.g. implement a timeout). Another concern is to write portable code. Python should help developers to write portable code, and time.monotonic(fallback=False) is not always available (and may fail). Victor From guido at python.org Sat Mar 31 03:29:17 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 18:29:17 -0700 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: On Fri, Mar 30, 2012 at 5:51 PM, Victor Stinner wrote: > 2012/3/31 Guido van Rossum : >> Given the amount of disagreement I sense, I think we'll need to wait >> for more people to chime in. > > I hope that the PEP now gives enough data to help to choose the best API. Lots of data, but not enough motivation, not enough about the use cases. >>> Hum, I read that you can expect a drift between QPC and GetTickCount. >>> I don't have exact numbers. There are also issues with power-saving. >>> When the CPU frequency changes, the frequency of the TSC is also >>> impacted on some processors (others provide a TSC at a fixed >>> frequency). >> >> And how does that impact teh various use cases? (There still isn't >> enough about use cases in the PEP. Thank you very much though for >> adding all those details about the various platform clocks.) > > I didn't find a good source, but it looks like QueryPerformanceCounter > behaves badly on system suspend/resume, whereas GetTickCount is > reliable and has a well defined behaviour: "The elapsed time retrieved > by GetTickCount or GetTickCount64 includes time the system spends in > sleep or hibernation." So QPC is not really monotonic nor steady. It > may be even worse than the system clock (especially in virtual > machines). I think you are in serious danger of overspecifying the problem. No clock can fulfill all requirements. (That's why there are so many to choose from of course!) > Impact: A timeout may be 42 seconds shorter than the requested > duration if is uses QPC. For a scheduler, a task would be scheduled at > the right moment. I don't understand this paragraph. And why is it always exactly a loss of 42 seconds? >>> I prefer use cases: >>> >>> ?* time.highres() should be used for profiling and benchmarking >>> ?* time.monotonic() should be used for a scheduler and timeout >> >> So if I want to schedule something a week in the future, what should I >> use? monotonic() or time()? And can you explain the reason for your >> answer? What about an hour or a day? Or a month or a year? > > You should use time.monotonic() because time.monotonic() has a steady > rate and so is reliable for long duration (at least more reliable than > time.highres()). Ah, but have you *thought* about the use case? *Why* would I want to schedule something that far in the future? Maybe it's a recurring birthday? Surely time() is better for that. (Etc. Think!) My question is trying to tease out the "meaning" of the various clocks that might exist. > In practice, time.monotonic() and time.highres() are only different on > Windows. If I understood correctly, the Windows kernel uses something > like GetTickCount whch has an accuracy of 1 ms in the best case. So > Python only needs a clock with a similar accuracy for timeout and > schedulers. So is it worth having two functions that are only different on Windows? ISTM that the average non-Windows user will have a 50% chance of picking the wrong timer from a portability perspective. > time.highres() (QPC) rate is only steady during a short duration: it > is not an issue for a benchmark because you usually rerun the same > test at least 3 times and keep the minimum. It looks like QPC bugs are > only unexpected forward jumps (no backward jump), so taking the > minimum would workaround these issues. The hibernation issue should > not affect benchmarking/profiling. Now you're talking: indeed the application should work around the limitations (if there are any) using end-to-end means, since only the application developer knows what matters to them. >>> time.monotonic() rate should be as steady as possible, its stability >>> is more important than its accuracy. time.highres() should provide the >>> clock with the best accuracy. Both clocks may have an undefined >>> starting point. >> >> So what kind of drift is acceptable for each? > > I don't know. Ok, let's just assume that we can't control drift so we'll have to let the app deal with it. >>> I disagree. Python must provide a truly monotonic clock, even it is >>> not truly monotonic by default. >> >> Why? > > See Zooko Wilcox-O'Hearn's email: > http://mail.python.org/pipermail/python-dev/2012-March/118147.html He explains the basic difference between the two types of clocks, and that's important. He doesn't say anything about a strict requirement for monotonicity or steadiness. This is why I still balk at "monotonic" for the name -- I don't think that monotonicity is the most important property. But I don't know how the put the desired requirement in words; "steady" seems to come closer for sure. >>> I suggest this API: time.monotonic(fallback=True). >> >> But this goes against the idea of "A keyword argument that gets passed >> as a constant in the caller is usually poor API." > > Right :-) After changing (again) the PEP, I realized that it > reintroduces the time.monotonic(fallback=False) raising > NotImplementedError case and I really don't like this exception here. Me neither. >> And it would encourage the creation of trivial lambdas just to call >> the timer with this flag, since a universal API is a timer function >> that takes no arguments. > > My bet is that fallback=True is an uncommon use case. Don't bet. Look at Cameron Simpson's proposal. (How many times do I have to say that?) >>> Sometimes, it does matter which exact OS clock is used. >> >> When is that? > > If you need to know more properties of the clock,. For example, Python > may be unable to give the accuracy of the clock, but you may get it > differently if you know which clock function is used. Writing code that depends on that sounds like asking for trouble. However I like to have this "clock metadata" available so that they can be printed along with benchmark results. A user reading those results may make good use of the information "this experiment used QueryPerformanceCounter so small times are pretty accurate but if there's an outlier you just have to start over". >>> Should I include such new function in the PEP, or can it be discussed >>> (later) in a separated thread? >> >> FWIW, I prefer a single thread devoted to all aspects of this PEP. > > Let's propose an API: > > time.clock_info() -> > {'time': {'function': 'gettimeofday', 'resolution': 1e-6, 'monotonic': False}, > ?'monotonic': {'function': 'clock_gettime(CLOCK_MONOTONIC)', > 'resolution': 1e-9, 'monotonic': True}, > ?'highres': {'function': 'clock_gettime(CLOCK_MONOTONIC)', > 'resolution': 1e-9, 'monotonic': True}} > > or > > time.clock_info('time') -> {'function': 'gettimeofday', 'resolution': > 1e-6, 'monotonic': False} > time.clock_info('monotonic') -> {'function': > 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': > True} > time.clock_info( 'highres') -> {'function': > 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': > True} > > or > > time.clock_info(time.time) -> {'function': 'gettimeofday', > 'resolution': 1e-6, 'monotonic': False} > time.clock_info(time.monotonic) -> {'function': > 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': > True} > time.clock_info( time.highres) -> {'function': > 'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic': > True} I like the version that takes the name of the clock as an argument best. If you have the function, the __name__ attribute gives the name. Going the other way seems more complicated. > "clock_" prefix in "time.clock_xxx" name is used by POSIX clock_xxx > functions, another prefix may be used instead. I'll leave this to the bikeshedders for now. :-) >>> I don't want to only provide time.monotonic() which would fallback if >>> no monotonic clock is available, because we would be unable to tell if >>> the clock is monotonic or not. >> >> There could be a separate inquiry API. > > Well, the PEP mentions something like that in the "One function, no > flag" section. But you don't seem to like it. (In general the alternatives section could do with reasons for rejection or at least pros and cons for each alternative.) >>> See the implementation in the PEP: the >>> "try monotonic or fallback" function may use a different clock at each >>> call, and so may be monotonic at startup and then becomes >>> non-monotonic. >> >> No, it should never switch implementations once it has chosen. That's >> just asking for trouble. If it can become non-monotonic it just isn't >> monotonic. Again, Cameron Simpson's idea might help here. > > Correct. In practice, one call to time.monotonic() is enough to know > if next calls will fail or not, and so you don't have to poll regulary > to check if the clock becomes non-monotonic. So always fallback and > providing something to check if time.monotonic() is or not monotonic > is be an acceptable solution. I agree that it is less ugly than the > time.monotonic(fallback=True) API. Ok. I guess if you call clock_info() before having called the timer function, it can call it internally to find out, if dynamic inquiries are required. -- --Guido van Rossum (python.org/~guido) From guido at python.org Sat Mar 31 03:32:37 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2012 18:32:37 -0700 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: <20120330214319.GA3106@cskk.homeip.net> Message-ID: On Fri, Mar 30, 2012 at 6:24 PM, Victor Stinner wrote: >> There seem to be a few competing features for clocks that people want: >> >> ?- monotonic - never going backward at all >> ?- high resolution > > These features look to be exclusive on Windows. On other platforms, it > looks like monotonic clocks are always the most accurate clocks. So I > don't think that we need to be able to combine these two "flags". > >> ?- no steps > > You mean "not adjusted by NTP"? Except CLOCK_MONOTONIC on Linux, no > monotonic clock is adjusted by NTP. On Linux, there is > CLOCK_MONOTONIC_RAW, but it is only available on recent Linux kernel > (2.6.28). > > Do you think that it is important to be able to refuse a monotonic > clock adjusted by NTP? What would be the use case of such truly steady > clock? That depends on what NTP can do to the clock. If NTP makes the clock tick *slightly* faster or slower in order to gradually adjust the wall clock, that's fine. If NTP can make it jump wildly forward or even backward, it's no better than time.time(), and we know why (for some purposes) we don't want that. > -- > > The PEP 418 tries to expose various monotonic clocks in Python with a > simple API. If we fail to agree how to expose these clocks, e.g. > because there are major differences between these clocks, another > solution is to only expose low-level function. This is already what we > do with the os module, and there is the shutil module (and others) for > a higher level API. I already added new "low-level" functions to the > time module: time.clock_gettime() and time.clock_getres(). > >> Of course you can provide some convenient-with-fallback function that >> will let people do this in one hit without the need for "T", but it >> should not be the base facility offered. The base should let people >> request their feature set and inspect what is supplied. > > The purpose of such function is to fix programs written with Python < > 3.3 and using time.time() whereas a monotonic would be the right clock > (e.g. implement a timeout). > > Another concern is to write portable code. Python should help > developers to write portable code, and time.monotonic(fallback=False) > is not always available (and may fail). -- --Guido van Rossum (python.org/~guido) From glyph at twistedmatrix.com Sat Mar 31 03:58:40 2012 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 30 Mar 2012 21:58:40 -0400 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: <20120330214319.GA3106@cskk.homeip.net> Message-ID: On Mar 30, 2012, at 9:32 PM, Guido van Rossum wrote: >>> - no steps >> >> You mean "not adjusted by NTP"? Except CLOCK_MONOTONIC on Linux, no >> monotonic clock is adjusted by NTP. On Linux, there is >> CLOCK_MONOTONIC_RAW, but it is only available on recent Linux kernel >> (2.6.28). >> >> Do you think that it is important to be able to refuse a monotonic >> clock adjusted by NTP? What would be the use case of such truly steady >> clock? > > That depends on what NTP can do to the clock. If NTP makes the clock > tick *slightly* faster or slower in order to gradually adjust the wall > clock, that's fine. If NTP can make it jump wildly forward or even > backward, it's no better than time.time(), and we know why (for some > purposes) we don't want that. "no steps" means something very specific. It does not mean "not adjusted by NTP". In NTP, changing the clock frequency to be slightly faster or slower is called "slewing" (which is done with adjtime()). Jumping by a large amount in a single discrete step is called "stepping" (which is done with settimeofday()). This is sort-of explained by . I think I'm agreeing with Guido here when I say that, personally, my understanding is that slewing is generally desirable (i.e. we should use CLOCK_MONOTONIC, not CLOCK_MONOTONIC_RAW) if one wishes to measure "real" time (and not a time-like object like CPU cycles). This is because the clock on the other end of the NTP connection from you is probably better at keeping time: hopefully that thirty five thousand dollars of Cesium timekeeping goodness is doing something better than your PC's $3 quartz crystal, after all. So, slew tends to correct for minor defects in your local timekeeping mechanism, and will compensate for its tendency to go too fast or too slow. By contrast, stepping only happens if your local clock is just set incorrectly and the re-sync delta has more to do with administrative error or failed batteries than differences in timekeeping accuracy. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Mar 31 04:12:34 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 31 Mar 2012 04:12:34 +0200 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: Message-ID: >> Impact: A timeout may be 42 seconds shorter than the requested >> duration if is uses QPC. For a scheduler, a task would be scheduled at >> the right moment. > > I don't understand this paragraph. And why is it always exactly a loss > of 42 seconds? Sorry, it's late here, I'm too tired. A jump of 42 seconds is one of QPC known bug (it was a bug in VirtualBox and it is now fixed). > So is it worth having two functions that are only different on > Windows? ISTM that the average non-Windows user will have a 50% chance > of picking the wrong timer from a portability perspective. Can we solve this by documenting correctly time.highres() issues (it's not really monotonic) and when you need to use time.monotonic() or time.highres()? Well, there is already time.clock() that uses QPC on Windows, but time.highres() and time.clock() give different results for an idle process on UNIX. pybench, timeit and other programs have their own heuristic getting the most accurate clock depending on the OS and on which clocks are available. time.highres() is a portable function for these programs. >> See Zooko Wilcox-O'Hearn's email: >> http://mail.python.org/pipermail/python-dev/2012-March/118147.html > > He explains the basic difference between the two types of clocks, and > that's important. He doesn't say anything about a strict requirement > for monotonicity or steadiness. This is why I still balk at > "monotonic" for the name -- I don't think that monotonicity is the > most important property. But I don't know how the put the desired > requirement in words; "steady" seems to come closer for sure. If we provide a way to check if the monotonic clock is monotonic (or not), I agree to drop the flag from time.monotonic(fallback=True) and always fallback. I was never a fan of the "truly monotonic clock". time.clock_info('monotonic')['is_monotonic'] is a good candidate to store this information. "time.clock_info('monotonic')['is_monotonic']" looks like a bug: why would I check if a *monotonic* clock is *monotonic*? time.clock_info('steady')['monotonic'] looks more natural, but I don't remember why we should not use the term "steady". >> Well, the PEP mentions something like that in the "One function, no >> flag" section. > > But you don't seem to like it. (In general the alternatives section > could do with reasons for rejection or at least pros and cons for each > alternative.) I was concerned by the fact that time.monotonic() may become not-monotonic between two calls, but I changed my mind. I agree that it should not occur in practice. So always fallback and add a way to know if the monotonic clock is monotonic now sounds like the best API to me. Victor From victor.stinner at gmail.com Sat Mar 31 04:17:49 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 31 Mar 2012 04:17:49 +0200 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: <20120330214319.GA3106@cskk.homeip.net> Message-ID: > (...) > ?By contrast, stepping only happens if your local clock is just set > incorrectly and the re-sync delta has more to do with administrative error > or failed batteries than differences in timekeeping accuracy. Are you talking about CLOCK_REALTIME or CLOCK_MONOTONIC? Victor From glyph at twistedmatrix.com Sat Mar 31 04:25:49 2012 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 30 Mar 2012 22:25:49 -0400 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: References: <20120330214319.GA3106@cskk.homeip.net> Message-ID: <13B4F7F8-5764-4A98-8185-DC8A11A562BC@twistedmatrix.com> On Mar 30, 2012, at 10:17 PM, Victor Stinner wrote: >> (...) >> By contrast, stepping only happens if your local clock is just set >> incorrectly and the re-sync delta has more to do with administrative error >> or failed batteries than differences in timekeeping accuracy. > > Are you talking about CLOCK_REALTIME or CLOCK_MONOTONIC? My understanding is: CLOCK_REALTIME is both stepped and slewed. CLOCK_MONOTONIC is slewed, but not stepped. CLOCK_MONOTONIC_RAW is neither slewed nor stepped. -glyph From glyph at twistedmatrix.com Sat Mar 31 04:28:26 2012 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 30 Mar 2012 22:28:26 -0400 Subject: [Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()? In-Reply-To: <13B4F7F8-5764-4A98-8185-DC8A11A562BC@twistedmatrix.com> References: <20120330214319.GA3106@cskk.homeip.net> <13B4F7F8-5764-4A98-8185-DC8A11A562BC@twistedmatrix.com> Message-ID: <0A62ACCF-1BEF-42BE-B958-F86A903BC8A1@twistedmatrix.com> On Mar 30, 2012, at 10:25 PM, Glyph wrote: > > On Mar 30, 2012, at 10:17 PM, Victor Stinner wrote: > >>> (...) >>> By contrast, stepping only happens if your local clock is just set >>> incorrectly and the re-sync delta has more to do with administrative error >>> or failed batteries than differences in timekeeping accuracy. >> >> Are you talking about CLOCK_REALTIME or CLOCK_MONOTONIC? > > My understanding is: > > CLOCK_REALTIME is both stepped and slewed. > > CLOCK_MONOTONIC is slewed, but not stepped. > > CLOCK_MONOTONIC_RAW is neither slewed nor stepped. Sorry, I realize I should cite my source. This mailing list post talks about all three together: Although the documentation one can find by searching around the web is really bad. It looks like many of these time features were introduced, to Linux at least, with no documentation. -glyph From tjreedy at udel.edu Sat Mar 31 05:31:21 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 30 Mar 2012 23:31:21 -0400 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F764F3B.2020306@pearwood.info> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <4F764F3B.2020306@pearwood.info> Message-ID: On 3/30/2012 8:26 PM, Steven D'Aprano wrote: > "hires" is a real English word, the present tense verb for engaging the > service or labour of someone or something in return for payment, as in > "he hires a gardener to mow the lawn". Can we please eliminate it from > consideration I agree. Heavy cognitive dissonance. 'Hires' is also a very famous brand of root beer. Hi-res *really* needs the hyphen (or underscore equivalent). -- Terry Jan Reedy From eliben at gmail.com Sat Mar 31 06:33:27 2012 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 31 Mar 2012 06:33:27 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #14065: Added cyclic GC support to ET.Element In-Reply-To: References: Message-ID: On Fri, Mar 30, 2012 at 21:30, Benjamin Peterson wrote: > > + def test_cyclic_gc(self): > > + class ShowGC: > > + def __init__(self, flaglist): > > + self.flaglist = flaglist > > + def __del__(self): > > + self.flaglist.append(1) > > > I think a nicer way to check for cyclic collection is to take a > weakref to an object, call the GC, then check to make sure the weakref > is broken. > > > + > > + # Test the shortest cycle: lst->element->lst > > + fl = [] > > + lst = [ShowGC(fl)] > > + lst.append(ET.Element('joe', attr=lst)) > > + del lst > > + gc.collect() > > support.gc_collect() is preferable > > > + self.assertEqual(fl, [1]) > > + > > + # A longer cycle: lst->e->e2->lst > > + fl = [] > > + e = ET.Element('joe') > > + lst = [ShowGC(fl), e] > > + e2 = ET.SubElement(e, 'foo', attr=lst) > > + del lst, e, e2 > > + gc.collect() > > + self.assertEqual(fl, [1]) > > Thanks for the insights, Benjamin. I'll explore these alternatives and will submit a fix. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Sat Mar 31 08:27:32 2012 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 31 Mar 2012 08:27:32 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: <4F764F3B.2020306@pearwood.info> References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <4F764F3B.2020306@pearwood.info> Message-ID: On Sat, Mar 31, 2012 at 02:26, Steven D'Aprano wrote: > Guido van Rossum wrote: >> If all else fails, I'd go with turnip. > > I can't tell if you are being serious or not. > > For the record, "turnip" in this sense is archaic slang for a thick pocket > watch. If I understand this correctly, the most common use for this function is when to time things. It will give you the best source available for timers, but it doesn't guarantee that it is steady or monotonic or high resolution or anything. It is also not the time, as it's not reliable as a wall-clock. So, how about time.timer()? //Lennart From regebro at gmail.com Sat Mar 31 08:32:50 2012 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 31 Mar 2012 08:32:50 +0200 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: References: Message-ID: On Fri, Mar 30, 2012 at 12:38, Serhiy Storchaka wrote: > I don't understand why Python may not include the pytz. The Olson tz > database is not part of pytz. Yes it is. > Python can depend on a system tz database That works on Unix, but not on Windows, where there is no Olsen database. //Lennart From nadeem.vawda at gmail.com Sat Mar 31 11:50:59 2012 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Sat, 31 Mar 2012 11:50:59 +0200 Subject: [Python-Dev] PEP 418: Add monotonic clock In-Reply-To: References: <4F710870.9090602@scottdial.com> <4F72003F.6000606@voidspace.org.uk> <4F725D02.1080706@gmail.com> <4F72B258.10306@scottdial.com> <4F72DBDE.6040003@scottdial.com> <4F764F3B.2020306@pearwood.info> Message-ID: On Sat, Mar 31, 2012 at 8:27 AM, Lennart Regebro wrote: > So, how about time.timer()? That seems like a bad idea; it would be too easy to confuse with (or misspell as) time.time(). Out of the big synonym list Guido posted, I rather like time.stopwatch() - it makes it more explicit that the purpose of the function is to measure intervals, rather identifying absolute points in time. Cheers, Nadeem From fuzzyman at voidspace.org.uk Sat Mar 31 12:28:55 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 31 Mar 2012 11:28:55 +0100 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: References: Message-ID: On 31 Mar 2012, at 07:32, Lennart Regebro wrote: > On Fri, Mar 30, 2012 at 12:38, Serhiy Storchaka wrote: >> I don't understand why Python may not include the pytz. The Olson tz >> database is not part of pytz. > > Yes it is. > >> Python can depend on a system tz database > > That works on Unix, but not on Windows, where there is no Olsen database. *However*, doesn't Windows have its own system database? The problem is that in order to not include the olsen database, pytz (which would be a very useful addition to the standard library) would need to be modified to use the system database on Windows. This is my (potentially flawed) understanding, anyway. Michael > > //Lennart > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From andrew.svetlov at gmail.com Sat Mar 31 13:21:14 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 31 Mar 2012 14:21:14 +0300 Subject: [Python-Dev] [Python-checkins] cpython (3.2): Issue #14409: IDLE doesn't not execute commands from shell with default In-Reply-To: References: Message-ID: Updated NEWS as Terry Reedy recommended. Thank you, Terry. On Sat, Mar 31, 2012 at 12:59 AM, Terry Reedy wrote: > On 3/30/2012 2:31 PM, Andrew Svetlov wrote: >> >> Thank you for mentoring. >> >> I will fix NEWS if you help me with better text. > > > I believe a succint message would be > > Issue 14409: IDLE now properly executes commands in the Shell window when it > cannot read the normal config files on startup and has to use the built-in > default key bindings. There was previously a bug in one of the defaults. > > >> The bug fixed is that commit is: >> IDLE has 3 configs: user, system default and hardcoded in python code. >> Last one had bad binding for ?key. >> Usually this config is never used: user or system ones overrides former. >> But if IDLE cannot open config files by some reason it switches to >> hardcoded configs and user got broken IDLE. >> >> Can anybody guess me short and descriptive message describing what fix >> well? >> >> On Fri, Mar 30, 2012 at 6:12 AM, Nick Coghlan ?wrote: >>> >>> On Fri, Mar 30, 2012 at 2:01 AM, andrew.svetlov >>> ?wrote: >>>> >>>> +- Issue #14409: IDLE doesn't not execute commands from shell, >>>> + ?error with default keybinding for Return. (Patch by Roger Serwy) >>> >>> >>> The double negative here makes this impossible to understand. Could we >>> please get an updated NEWS entry that explains what actually changed >>> in IDLE to fix this? >>> >>> Perhaps something like "IDLE now always sets the default keybind for >>> Return correctly, ensuring commands can be executed in the IDLE shell >>> window"? (assuming that's what happened). >>> >>> This is important, folks: NEWS entries need to be comprehensible for >>> people that *haven't* read the associated tracker issue. This means >>> that issue titles (which generally describe a problem someone was >>> having) are often inappropriate as NEWS items. NEWS items should be >>> short descriptions that clearly describe *what changed*, perhaps with >>> some additional information to explain a bit about why the change was >>> made. > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins -- Thanks, Andrew Svetlov From animelovin at gmail.com Sat Mar 31 13:43:28 2012 From: animelovin at gmail.com (Etienne Robillard) Date: Sat, 31 Mar 2012 07:43:28 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> <4F75F4D2.4050706@gmail.com> <4F75FA16.6040704@stoneleaf.us> <4F7601D4.1080706@gmail.com> <4F7605E0.6070503@gmail.com> <4F763389.7090307@gmail.com> Message-ID: <4F76EDE0.1020002@gmail.com> > The frozendict builtin type was rejected, but we are going to add > types.MappingProxyType: see issue #14386. > types.MappingProxyType(mydict.copy()) is very close to the frozendict > builtin type. > > Victor Thanks, Victor. :) Will this mean the new dict subclass for CPython will not expose dictproxy to favorize a new types.MappingProxyType type to emulate immutable-like types ? What could be then consequences for code still expecting a non-mutable dict() type ? Therefore I guess this ticket provides more than just speculating points to reconsider such aliased types in cpython. I also found this article quite useful: http://www.cs.toronto.edu/~tijmen/programming/immutableDictionaries.html Yet I might miss how this "new dict" type could potentially induce a RuntimeError unless in python 3.3 a new dict proxy alias is introduced to perform invariant operations in non thread-safe code. Regards, Etienne From regebro at gmail.com Sat Mar 31 14:14:07 2012 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 31 Mar 2012 14:14:07 +0200 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: References: Message-ID: On Sat, Mar 31, 2012 at 12:28, Michael Foord wrote: > *However*, doesn't Windows have its own system database? Yeah, but it sucks. > The problem is that in order to not include the olsen database, pytz > would need to be modified to use the system database on Windows. Quite a lot too, I'd guess since the databases are completely different in pretty much every way, most importantly in the way that they are using different (and insane) names for the timezones. //Lennart From hannu at krosing.net Sat Mar 31 14:22:01 2012 From: hannu at krosing.net (Hannu Krosing) Date: Sat, 31 Mar 2012 14:22:01 +0200 Subject: [Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python Message-ID: <1333196521.28732.23.camel@hvost> Hi Has anyone used Cython for developing a module to be used from postgreSQL/pl/python. Something that calls back to PostgreSQL internals. ------- Hannu Krosing PostgreSQL Unlimited Scalability and Performance Consultant 2ndQuadrant Nordic PG Admin Book: http://www.2ndQuadrant.com/books/ From stefan_ml at behnel.de Sat Mar 31 14:41:56 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 31 Mar 2012 14:41:56 +0200 Subject: [Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python In-Reply-To: <1333196521.28732.23.camel@hvost> References: <1333196521.28732.23.camel@hvost> Message-ID: Hannu Krosing, 31.03.2012 14:22: > Has anyone used Cython for developing a module to be used from > postgreSQL/pl/python. > > Something that calls back to PostgreSQL internals. Note that this is the CPython core developers mailing list, for which your question is off-topic. Please ask on the cython-users mailing list (see http://cython.org). Stefan From hannu at krosing.net Sat Mar 31 14:56:14 2012 From: hannu at krosing.net (Hannu Krosing) Date: Sat, 31 Mar 2012 14:56:14 +0200 Subject: [Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python In-Reply-To: References: <1333196521.28732.23.camel@hvost> Message-ID: <1333198574.28732.25.camel@hvost> On Sat, 2012-03-31 at 14:41 +0200, Stefan Behnel wrote: > Hannu Krosing, 31.03.2012 14:22: > > Has anyone used Cython for developing a module to be used from > > postgreSQL/pl/python. > > > > Something that calls back to PostgreSQL internals. > > Note that this is the CPython core developers mailing list, for which your > question is off-topic. Please ask on the cython-users mailing list (see > http://cython.org). Thanks for the pointer ! Can you recommend a similar place for asking the same queston about ctypes ? That is using ctypes for calling back to "outer" c application which embeds the python interpreter. > Stefan > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/hannu%40krosing.net From kristjan at ccpgames.com Sat Mar 31 13:46:26 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Sat, 31 Mar 2012 11:46:26 +0000 Subject: [Python-Dev] Issue13210 : Support Visual Studio 2010 Message-ID: Hi, Does anyone object if I submit my patches sxs.patch and errnomodule.patch? These allow python to work correctly when built with vs2010. There is also the PCBuild10.patch, but that can wait. I'm sure a number of people are regularly building python using vs2010 using their own modified solutions, but having the sources in a constant state of patching is a nuiscance so I think we ought to at least make the source code work with it, regardless of having a proper solution for it. K -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristjan at ccpgames.com Sat Mar 31 15:32:39 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Sat, 31 Mar 2012 13:32:39 +0000 Subject: [Python-Dev] Issue #14310: Socket duplication for windows Message-ID: Hi there. Antoine Pitroue requested that this topic (http://bugs.python.org/issue14310) be discussed by python-dev before moving further. I'm adding a windows-only api to "share" sockets between processes to _socket in the canonical way. Multiprocessing already has code for this (using unsupported methods) but a subsequent change could then simplify that. thanks, Kristj?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sat Mar 31 15:44:13 2012 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 01 Apr 2012 00:44:13 +1100 Subject: [Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python References: <1333196521.28732.23.camel@hvost> Message-ID: <87vclkdgwi.fsf@benfinney.id.au> Hannu Krosing writes: > Has anyone used Cython for developing a module to be used from > postgreSQL/pl/python. > > Something that calls back to PostgreSQL internals. Are you aware that PostgreSQL has long provided Python as a language for writing stored procedures in the database? Does that meet your needs? -- \ ?You've got to think about big things while you're doing small | `\ things, so that all the small things go in the right | _o__) direction.? ?Alvin Toffler | Ben Finney From hannu at krosing.net Sat Mar 31 16:04:16 2012 From: hannu at krosing.net (Hannu Krosing) Date: Sat, 31 Mar 2012 16:04:16 +0200 Subject: [Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python In-Reply-To: <87vclkdgwi.fsf@benfinney.id.au> References: <1333196521.28732.23.camel@hvost> <87vclkdgwi.fsf@benfinney.id.au> Message-ID: <1333202656.28732.35.camel@hvost> On Sun, 2012-04-01 at 00:44 +1100, Ben Finney wrote: > Hannu Krosing writes: > > > Has anyone used Cython for developing a module to be used from > > postgreSQL/pl/python. > > > > Something that calls back to PostgreSQL internals. > > Are you aware that PostgreSQL has long provided Python as a language for > writing stored procedures in the database? > > > > Does that meet your needs? Sure, I have even contributed code to it ;) But what i want is some way to call _back_ into PostgreSQL to use its internal functions from a module imported from pl/python. I tried ctypes module and got a segfault when i used the following create or replace function callback_to_postgres(rn_text text) returns text language plpythonu as $$ from ctypes import * import struct pg = cdll.LoadLibrary('/usr/lib/postgresql/9.1/bin/postmaster') pg.pq_flush() return rn_text $$; select send_raw_notice('do you see me?'); I determined that the call to pg.pq_flush() was the one crashing the backend, so I assume that cdll.LoadLibrary did not get the already running backend, but loaded another copy of it as library. Now I'm looking for a simplest way to do some C-wrapping. I have used Cyuthon for wrapping simple library calls and I really don'r think there would be hard problems doing this inside the postgreSQL extension build framework. I was just hoping that someboduy had already taken care of all the nitty-gritty detaiuls of setting this up ------- Hannu Krosing PostgreSQL Unlimited Scalability and Performance Consultant 2ndQuadrant Nordic PG Admin Book: http://www.2ndQuadrant.com/books/ From brian at python.org Sat Mar 31 16:53:42 2012 From: brian at python.org (Brian Curtin) Date: Sat, 31 Mar 2012 09:53:42 -0500 Subject: [Python-Dev] Issue13210 : Support Visual Studio 2010 In-Reply-To: References: Message-ID: 2012/3/31 Kristj?n Valur J?nsson : > Hi, > > Does anyone object if I submit my patches sxs.patch and errnomodule.patch? > > These allow python to work correctly when built with vs2010. > > > > There is also the PCBuild10.patch, but that can wait.? I'm sure a number of > people are regularly building python using vs2010 using their own modified > solutions, but having the sources in a constant state of patching is a > nuiscance so I think we ought to at least make the source code work with it, > regardless of having a proper solution for it. Go ahead. As listed on the issue, there is a repo where building from VS2010 "works" but the port is not complete as not all tests are passing. I will complete it. From rdmurray at bitdance.com Sat Mar 31 17:25:49 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 31 Mar 2012 11:25:49 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <4F76EDE0.1020002@gmail.com> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329220755.377052500E9@webabinitio.net> <4F75A510.7080401@gmail.com> <4F75D501.2050400@gmail.com> <4F75DA77.6040305@gmail.com> <4F75F4D2.4050706@gmail.com> <4F75FA16.6040704@stoneleaf.us> <4F7601D4.1080706@gmail.com> <4F7605E0.6070503@gmail.com> <4F763389.7090307@gmail.com> <4F76EDE0.1020002@gmail.com> Message-ID: <20120331152550.697232500E9@webabinitio.net> On Sat, 31 Mar 2012 07:43:28 -0400, Etienne Robillard wrote: > Yet I might miss how this "new dict" type could potentially induce a > RuntimeError unless in python 3.3 a new dict proxy alias is introduced > to perform invariant operations in non thread-safe code. Etienne, again: issue 14417 has *nothing* to do with immutable dicts. Please carefully read over issue 14205 in order to understand what we are talking about so that you can contribute to the discussion. --David From guido at python.org Sat Mar 31 18:09:21 2012 From: guido at python.org (Guido van Rossum) Date: Sat, 31 Mar 2012 09:09:21 -0700 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <20120329204815.D7AC32500E9@webabinitio.net> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329204815.D7AC32500E9@webabinitio.net> Message-ID: On Thu, Mar 29, 2012 at 1:48 PM, R. David Murray wrote: > On Thu, 29 Mar 2012 16:31:03 -0400, "R. David Murray" wrote: >> On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: >> > My original assessment was that this only affects dicts whose keys >> > have a user-implemented __hash__ or __eq__ implementation, and that >> > the number of apps that use this *and* assume the threadsafe property >> > would be pretty small. This is just intuition, I don't have hard >> > facts. But I do want to stress that not all dict lookups automatically >> > become thread-unsafe, only those that need to run user code as part of >> > the key lookup. >> >> You are probably correct, but the thing is that one still has to do the >> code audit to be sure...and then make sure that no one later introduces >> such an object type as a dict key. > > I just did a quick grep on our project. ?We are only defining __eq__ > and __hash__ a couple places, but both are objects that could easily get > used as dict keys (there is a good chance that's *why* those methods are > defined) accessed by more than one thread. ?I haven't done the audit to > find out :) Of course, that doesn't mean they're likely to be used as keys in a dict that is read and written concurrently by multiple threads. > The libraries we depend on have many more definitions of __eq__ and > __hash__, and we'd have to check them too. ?(Including SQLAlchemy, > and I wouldn't want that job.) > > So our intuition that this is not common may be wrong. But how often does one share a dictionary between threads with the understanding that multiple threads can read and write it? Here's a different puzzle. Has anyone written a demo yet that provokes this RuntimeError, without cheating? (Cheating would be to mutate the dict from *inside* the __eq__ or __hash__ method.) If you're serious about revisiting this, I'd like to see at least one example of a program that is broken by the change. Otherwise I think the status quo in the 3.3 repo should prevail -- I don't want to be stymied by superstition. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Sat Mar 31 18:26:12 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Apr 2012 02:26:12 +1000 Subject: [Python-Dev] [Python-checkins] cpython (3.2): Issue #14409: IDLE doesn't not execute commands from shell with default In-Reply-To: References: Message-ID: On Sat, Mar 31, 2012 at 9:21 PM, Andrew Svetlov wrote: > Updated NEWS as Terry Reedy recommended. > Thank you, Terry. Thanks to you both :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sat Mar 31 19:03:13 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Apr 2012 03:03:13 +1000 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329204815.D7AC32500E9@webabinitio.net> Message-ID: On Sun, Apr 1, 2012 at 2:09 AM, Guido van Rossum wrote: > Here's a different puzzle. Has anyone written a demo yet that provokes > this RuntimeError, without cheating? (Cheating would be to mutate the > dict from *inside* the __eq__ or __hash__ method.) If you're serious > about revisiting this, I'd like to see at least one example of a > program that is broken by the change. Otherwise I think the status quo > in the 3.3 repo should prevail -- I don't want to be stymied by > superstition. I attached an attempt to *deliberately* break the new behaviour to the tracker issue. It isn't actually breaking for me, so I'd like other folks to look at it to see if I missed something in my implementation, of if it's just genuinely that hard to induce the necessary bad timing of a preemptive thread switch. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From martin at v.loewis.de Sat Mar 31 19:42:22 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sat, 31 Mar 2012 19:42:22 +0200 Subject: [Python-Dev] Issue13210 : Support Visual Studio 2010 In-Reply-To: References: Message-ID: <20120331194222.Horde.-YjNAaGZi1VPd0H_deEnonA@webmail.df.eu> > Does anyone object if I submit my patches sxs.patch and errnomodule.patch? > > These allow python to work correctly when built with vs2010. Please see my review: "allow to work correctly" is not a good explanation of what it does, and why it does that. As it is highly counter-intuitive, it needs at least justification. I'm trying to investigate whether it is actually correct (and that it works for CCP is not sufficient proof that it is correct). Also, please clarify what branches you would apply this to. 3.3 is certainly fine; for any other branches, I'd like to point out that this is not a bug fix. > There is also the PCBuild10.patch, but that can wait. Indeed, it shouldn't be added at all. Instead, the PCbuild tree should become a VS2010 (or VS2012) tree ultimately. Regards, Martin From martin at v.loewis.de Sat Mar 31 19:45:11 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sat, 31 Mar 2012 19:45:11 +0200 Subject: [Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python In-Reply-To: <1333198574.28732.25.camel@hvost> References: <1333196521.28732.23.camel@hvost> <1333198574.28732.25.camel@hvost> Message-ID: <20120331194511.Horde.GqAhbqGZi1VPd0KndtBXrlA@webmail.df.eu> > Can you recommend a similar place for asking the same queston about > ctypes ? > > That is using ctypes for calling back to "outer" c application which > embeds the python interpreter. ISTM that Postgres lists should be the best place for this kind of question. Alternatively, try python-list or db-sig. Regards, Martin From rdmurray at bitdance.com Sat Mar 31 19:45:33 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 31 Mar 2012 13:45:33 -0400 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329204815.D7AC32500E9@webabinitio.net> Message-ID: <20120331174533.E0E612500E9@webabinitio.net> On Sun, 01 Apr 2012 03:03:13 +1000, Nick Coghlan wrote: > On Sun, Apr 1, 2012 at 2:09 AM, Guido van Rossum wrote: > > Here's a different puzzle. Has anyone written a demo yet that provokes > > this RuntimeError, without cheating? (Cheating would be to mutate the > > dict from *inside* the __eq__ or __hash__ method.) If you're serious > > about revisiting this, I'd like to see at least one example of a > > program that is broken by the change. Otherwise I think the status quo > > in the 3.3 repo should prevail -- I don't want to be stymied by > > superstition. > > I attached an attempt to *deliberately* break the new behaviour to the > tracker issue. It isn't actually breaking for me, so I'd like other > folks to look at it to see if I missed something in my implementation, > of if it's just genuinely that hard to induce the necessary bad timing > of a preemptive thread switch. Thanks, Nick. It looks reasonable to me, but I've only given it a quick look so far (I'll try to think about it more deeply later today). If it is indeed hard to provoke, then I'm fine with leaving the RuntimeError as a signal that the application needs to add some locking. My concern was that we'd have working production code that would start breaking. If it takes a *lot* of threads or a *lot* of mutation to trigger it, then it is going to be a lot less likely to happen anyway, since such programs are going to be much more careful about locking anyway. --David From tjreedy at udel.edu Sat Mar 31 21:20:12 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 31 Mar 2012 15:20:12 -0400 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: References: Message-ID: On 3/31/2012 6:28 AM, Michael Foord wrote: > > On 31 Mar 2012, at 07:32, Lennart Regebro wrote: > >> On Fri, Mar 30, 2012 at 12:38, Serhiy >> Storchaka wrote: >>> I don't understand why Python may not include the pytz. The Olson >>> tz database is not part of pytz. >> >> Yes it is. >> >>> Python can depend on a system tz database >> >> That works on Unix, but not on Windows, where there is no Olsen >> database. > > *However*, doesn't Windows have its own system database? The problem > is that in order to not include the olsen database, pytz (which would > be a very useful addition to the standard library) would need to be > modified to use the system database on Windows. This is my > (potentially flawed) understanding, anyway. The Windows installer, by default, installs tcl/tk while Python on other systems uses the system install. Why can't we do the same for the Olson database? As for updates: The correct behavior for timezone functions is to use the most recent definitions of those functions. The date is an input parameter for those functions. Projection of correct behavior for today's date into the future is provisional and subject to correction. This is especially true of anything involving Daylight Stupid Time. (As you can tell, I would have it go away.) Testing the specific output of such functions with future dates is broken. So I think we should define correct behavior of pytz as use with the latest Olson database. Use with an older version would then be a 'bug' subject to being fixed. On Windows, we could update as needed with every bugfix release. (And give instructions for user update.) On other systems, users can do whatever is appropriate. Or perhaps we could add an 'update tz database' function to the module. Similary, our unicode implementation is defined as using the unicode database as of a few weeks before each feature release. Updates for bugfix releases are not done because changes to that database are a few additions each time rather than edits. -- As a side note: I think the same trick of defining correct behavior dynamically rather that statically could be applied to other modules, such mimetypes and internet protocol modules. This seems to be part of the intent of the idea of having stdlib feature releases every 6 months or so. -- Terry Jan Reedy From andrew.svetlov at gmail.com Sat Mar 31 21:36:16 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 31 Mar 2012 22:36:16 +0300 Subject: [Python-Dev] datetime module and pytz with dateutil In-Reply-To: References: Message-ID: On Sat, Mar 31, 2012 at 10:20 PM, Terry Reedy wrote: > So I think we should define correct behavior of pytz as use with the latest > Olson database. Use with an older version would then be a 'bug' subject to > being fixed. On Windows, we could update as needed with every bugfix > release. (And give instructions for user update.) On other systems, users > can do whatever is appropriate. Or perhaps we could add an 'update tz > database' function to the module. > Please, don't add updating function to standard lib. It's nightmare for package maintainers. In general I'm +0 for adding tz database to stdlib. -- Thanks, Andrew Svetlov From guido at python.org Sat Mar 31 23:26:25 2012 From: guido at python.org (Guido van Rossum) Date: Sat, 31 Mar 2012 14:26:25 -0700 Subject: [Python-Dev] Issue 14417: consequences of new dict runtime error In-Reply-To: <20120331174533.E0E612500E9@webabinitio.net> References: <20120329195825.843352500E9@webabinitio.net> <20120329203103.95A4B2500E9@webabinitio.net> <20120329204815.D7AC32500E9@webabinitio.net> <20120331174533.E0E612500E9@webabinitio.net> Message-ID: Try reducing sys.setcheckinterval(). --Guido van Rossum (sent from Android phone) On Mar 31, 2012 10:45 AM, "R. David Murray" wrote: > On Sun, 01 Apr 2012 03:03:13 +1000, Nick Coghlan > wrote: > > On Sun, Apr 1, 2012 at 2:09 AM, Guido van Rossum > wrote: > > > Here's a different puzzle. Has anyone written a demo yet that provokes > > > this RuntimeError, without cheating? (Cheating would be to mutate the > > > dict from *inside* the __eq__ or __hash__ method.) If you're serious > > > about revisiting this, I'd like to see at least one example of a > > > program that is broken by the change. Otherwise I think the status quo > > > in the 3.3 repo should prevail -- I don't want to be stymied by > > > superstition. > > > > I attached an attempt to *deliberately* break the new behaviour to the > > tracker issue. It isn't actually breaking for me, so I'd like other > > folks to look at it to see if I missed something in my implementation, > > of if it's just genuinely that hard to induce the necessary bad timing > > of a preemptive thread switch. > > Thanks, Nick. It looks reasonable to me, but I've only given it a quick > look so far (I'll try to think about it more deeply later today). > > If it is indeed hard to provoke, then I'm fine with leaving the > RuntimeError as a signal that the application needs to add some locking. > My concern was that we'd have working production code that would start > breaking. If it takes a *lot* of threads or a *lot* of mutation to > trigger it, then it is going to be a lot less likely to happen anyway, > since such programs are going to be much more careful about locking > anyway. > > --David > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcepl at redhat.com Fri Mar 30 17:00:14 2012 From: mcepl at redhat.com (=?UTF-8?B?TWF0xJtqIENlcGw=?=) Date: Fri, 30 Mar 2012 17:00:14 +0200 Subject: [Python-Dev] .{git,bzr}ignore in cpython HG repo Message-ID: <4F75CA7E.7030204@redhat.com> Why does HG cpython repo contains .{bzr,git}ignore at all? IMHO, all .*ignore files should be strictly repository dependent and they should not be mixed together. It is even worse, that (understandingly) .{bzr,git}ignore are apparently poorly maintained, so in order to get an equivalent of .hgignore in .gitignore, one has to apply the attached patch. Best, Mat?j -- http://www.ceplovi.cz/matej/, Jabber: mcepl ceplovi.cz GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC We understand our competition isn't with Caldera or SuSE--our competition is with Microsoft. -- Bob Young of Red Hat http://www.linuxjournal.com/article/3553 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: _gitignore.patch URL:
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4