Do not post general Python questions to this list! For help with Python please see the Python help page!
David. From mwh at python.net Fri Aug 12 13:29:49 2005 From: mwh at python.net (Michael Hudson) Date: Fri, 12 Aug 2005 12:29:49 +0100 Subject: [Python-Dev] plans for 2.4.2 and 2.5a1 In-Reply-To: <200508111751.56899.anthony@interlink.com.au> (Anthony Baxter's message of "Thu, 11 Aug 2005 17:51:55 -0700") References: <200508111751.56899.anthony@interlink.com.au> Message-ID: <2m8xz7wfyq.fsf@starship.python.net> Anthony Baxter writes: > So I'm currently planning for a 2.4.2 sometime around mid September. I figure > we cut a release candidate either on the 7th or 14th, and a final a week > later. Cool. I'm not sure how many outstanding bugs should be fixed before 2.4.2. Some stuff to do with files with PEP 263 style declarations? (Walter? I've lost track of these). I think I should probably just check my fix for "PyXxx_Check(x) trusts x->ob_type->tp_mro" (http://python.org/sf/1153075) in to both branches, unless someone can think of a good reason not to (Armin?). (The whole area could do with some work, really, but that's another story). > In addition, I'd like to suggest we think about a first alpha of 2.5 sometime > during March 2006, with a final release sometime around May-June. This would > mean (assuming people are happy with this) we need to make a list of what's > still outstanding for 2.5. There's a bunch of accepted PEPs that are waiting > for code. Once that's done, there will be a final 2.4.3 sometime after or > close to the 2.5 final release. I have some outstanding patches: 1) My PEP 343 implementation (http://python.org/sf/1235943). Needs reviewing, but docs are in another patch. I also recently realized that my patch is incomplete, we should accept stuff like this: with cm as (a,b,c): ... where cm.__enter__ returns a 3-sequence. My patch just allows a NAME after the 'as' pseudo-keyword (if anyone else wants to fix this, be my guest :) 2) The new-style exceptions patch (http://python.org/sf/1104669). This mostly needs documentation, but could also do with review/testing. 3) "explicit sign variable for longs" (http://python.org/sf/1177779). This is a user-invisible patch, really, so I'm not so concerned about it (I'd like to follow it up by emitting DeprecationWarnings on ob_size abuse in 2.6 and disallowing it in 2.7 -- or maybe we could even emit DeprecationWarnings in 2.5 already). 4) "__slots__ for subclasses of variable length types" (http://python.org/sf/1173475) -- this is very pie-in-the-sky and in fact the attached patch is completely broken, but I think work in this area would still be a good thing. Review the others before looking at this one, please :) ... then there's the ast-branch, of course ... Is there a 2.5 release PEP yet? Cheers, mwh -- If i don't understand lisp, it would be wise to not bray about how lisp is stupid or otherwise criticize, because my stupidity would be archived and open for all in the know to see. -- Xah, comp.lang.lisp From walter at livinglogic.de Fri Aug 12 15:34:35 2005 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 12 Aug 2005 15:34:35 +0200 Subject: [Python-Dev] plans for 2.4.2 and 2.5a1 In-Reply-To: <2m8xz7wfyq.fsf@starship.python.net> References: <200508111751.56899.anthony@interlink.com.au> <2m8xz7wfyq.fsf@starship.python.net> Message-ID: <42FCA56B.5060604@livinglogic.de> Michael Hudson wrote: > Anthony Baxter writes: > >>So I'm currently planning for a 2.4.2 sometime around mid September. I figure >>we cut a release candidate either on the 7th or 14th, and a final a week >>later. > > Cool. I'm not sure how many outstanding bugs should be fixed before > 2.4.2. Some stuff to do with files with PEP 263 style declarations? > (Walter? I've lost track of these). True, there's a whole bunch of them (mostly duplicates): Bug #1076985: Incorrect behaviour of StreamReader.readline leads to crash (fixed) Bug #1089395: segfault/assert in tokenizer (fixed) Bug #1098990: codec readline() splits lines apart (fixed) Bug #1163244: Syntax error on large file with MBCS encoding (open) Bug #1175396: codecs.readline sometimes removes newline chars (open) Bug #1178484: Erroneous line number error in Py2.4.1 (open) Bug #1200686: SyntaxError raised on win32 for correct files (open, probably duplicate) Bug #1211639: parser tells invalid syntax with correct code (duplicate) Bug #1218930: Parser chokes on big files (duplicate) Bug #1225059: Line endings problem with Python 2.4.1 cause imports to fail (duplicate) Bug #1241507: StreamReader broken for byte string to byte string codecs (fixed) Bug #1251631: Python 2.4.1 crashes when importing the attached script (open, probably duplicate) Patch #1101726: Patch for potential buffer overrun in tokenizer.c (applied) Most of them are fixed. #1178484 is waiting for a final OK. Bye, Walter D?rwald From barry at python.org Fri Aug 12 15:40:22 2005 From: barry at python.org (Barry Warsaw) Date: Fri, 12 Aug 2005 09:40:22 -0400 Subject: [Python-Dev] plans for 2.4.2 and 2.5a1 In-Reply-To: <200508111751.56899.anthony@interlink.com.au> References: <200508111751.56899.anthony@interlink.com.au> Message-ID: <1123854022.10627.2.camel@geddy.wooz.org> On Thu, 2005-08-11 at 20:51, Anthony Baxter wrote: > So I'm currently planning for a 2.4.2 sometime around mid September. I figure > we cut a release candidate either on the 7th or 14th, and a final a week > later. Cool. I'd like to commit the patches in this bug report: https://sourceforge.net/tracker/index.php?func=detail&aid=900092&group_id=5470&atid=105470 which fixes a long standing hotshot bug. Any objections? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050812/33f20adc/attachment.pgp From paolo_veronelli at libero.it Fri Aug 12 17:14:24 2005 From: paolo_veronelli at libero.it (Paolino) Date: Fri, 12 Aug 2005 17:14:24 +0200 Subject: [Python-Dev] set.remove feature/bug Message-ID: <42FCBCD0.1000406@libero.it> I can't contact sourceforge bug tracker sorry. set.remove is trying to freeze sets when they are used as keys.No matter if an __hash__ method is defined. This is incoherent with Set.remove and dict.__delete__ & co. If this is a feature ,then I ask strongly to keep sets module in the stdlib for ever. Or if there is a workaround, please tell me here because python-list didn't help. class H(set): def __hash__(self):return id(self) s=H() f=set() f.add(s) f.remove(s) # this fails Regards Paolino From raymond.hettinger at verizon.net Fri Aug 12 17:44:38 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Fri, 12 Aug 2005 11:44:38 -0400 Subject: [Python-Dev] set.remove feature/bug In-Reply-To: <42FCBCD0.1000406@libero.it> Message-ID: <002701c59f54$c0074ba0$dd1ecb97@oemcomputer> [Paolino] > I can't contact sourceforge bug tracker sorry. I've added a bug report for you: www.python.org/sf/1257731 > set.remove is trying to freeze sets when they are used as keys.No matter > if an __hash__ method is defined. Will fix. Feel free to email me off-list with any questions. Raymond From tzot at mediconsa.com Fri Aug 12 18:21:57 2005 From: tzot at mediconsa.com (Christos Georgiou) Date: Fri, 12 Aug 2005 19:21:57 +0300 Subject: [Python-Dev] plans for 2.4.2 and 2.5a1 References: <200508111751.56899.anthony@interlink.com.au> <2m8xz7wfyq.fsf@starship.python.net> Message-ID: "Michael Hudson" wrote in message news:2m8xz7wfyq.fsf at starship.python.net... > Anthony Baxter writes: > >> So I'm currently planning for a 2.4.2 sometime around mid September. I >> figure >> we cut a release candidate either on the 7th or 14th, and a final a week >> later. > > Cool. I'm not sure how many outstanding bugs should be fixed before > 2.4.2. Some stuff to do with files with PEP 263 style declarations? > (Walter? I've lost track of these). This is a serious issue (spurious syntax errors). One bug about files with encoding declarations is www.python.org/sf/1163244 . So far, it seems that source files having a size of f*n+x (for some small indeterminate value of x, and f is a power of 2 like 512 or 1024) occasionally fail to compile with spurious syntax errors. (I once had a file show up the line with the "syntax error", and the reported line was comprised half from the failing line and half from the line above --unfortunately I kept the file for examination in a USB key that some colleague formatted). The syntax errors disappear if the coding declaration is removed or if some blank lines are inserted before the failing line. I think this occurs only on Windows, so it should be something to do with line endings and buffering. At the moment I'm trying to create a minimal file that when imported fails with 2.4.1 . I'll update the case as soon as I have one, but I wanted to draw some attention in python-dev in case it rings a bell. From tzot at mediconsa.com Fri Aug 12 18:32:16 2005 From: tzot at mediconsa.com (Christos Georgiou) Date: Fri, 12 Aug 2005 19:32:16 +0300 Subject: [Python-Dev] plans for 2.4.2 and 2.5a1 References: <200508111751.56899.anthony@interlink.com.au><2m8xz7wfyq.fsf@starship.python.net> Message-ID: > At the moment I'm trying to create a minimal file that when imported fails > with 2.4.1 . I'll update the case as soon as I have one, but I wanted to > draw some attention in python-dev in case it rings a bell. Please ignore my previous message --through gmane I saw only mwh's message, and after sending my reply, I got Walter's message. From jcarlson at uci.edu Fri Aug 12 18:44:09 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 12 Aug 2005 09:44:09 -0700 Subject: [Python-Dev] plans for 2.4.2 and 2.5a1 In-Reply-To: <200508111751.56899.anthony@interlink.com.au> References: <200508111751.56899.anthony@interlink.com.au> Message-ID: <20050812091538.7836.JCARLSON@uci.edu> For 2.5a1... Some exposure of _PyLong_AsByteArray() and _PyLong_FromByteArray() to Python. There was a discussion about this almost a year ago (http://python.org/sf/1023290), and no mechanism (struct format code addition, binascii.tolong/fromlong, long.tostring/fromstring, ...) actually made it into Python 2.4 . At this point, I'd be happy to get /any/ mechanism, with a preference to struct and/or binascii (I'd put them in both, if only because different groups of people people may look for them in both places, and people who use one tend to like to use that one for as much as possible, and because the code additions in both are minor). - Josiah Anthony Baxter wrote: > > So I'm currently planning for a 2.4.2 sometime around mid September. I figure > we cut a release candidate either on the 7th or 14th, and a final a week > later. > > In addition, I'd like to suggest we think about a first alpha of 2.5 sometime > during March 2006, with a final release sometime around May-June. This would > mean (assuming people are happy with this) we need to make a list of what's > still outstanding for 2.5. There's a bunch of accepted PEPs that are waiting > for code. Once that's done, there will be a final 2.4.3 sometime after or > close to the 2.5 final release. > > Anthony > -- > Anthony Baxter > It's never too late to have a happy childhood. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jcarlson%40uci.edu From bcannon at gmail.com Fri Aug 12 19:00:38 2005 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 12 Aug 2005 10:00:38 -0700 Subject: [Python-Dev] Exception Reorg PEP revised yet again In-Reply-To: References: <000801c59e05$9e054de0$6f14c797@oemcomputer> Message-ID: On 8/12/05, Thomas Heller wrote: > Brett Cannon writes: > > > On 8/10/05, Raymond Hettinger wrote: > >> > > Then I don't follow what you mean by "moved under os". > >> > > >> > In other words, to get the exception, do ``from os import > >> > WindowsError``. Unfortunately we don't have a generic win module to > >> > put it under. Maybe in the platform module instead? > >> > >> -1 on either. The WindowsError exception needs to in the main exception > >> tree. It occurs in too many different modules and applications. That > >> is a good reason for being in the main tree. > >> > > > > Where is it used so much? In the stdlib, grepping for WindowsError > > recursively in Lib in 2.4 turns up only one module raising it > > (subprocess) and only two modules with a total of three places of > > catching it (ntpath once, urllib twice). In Module, there are no > > hits. > > > > I don't know how you've been grepping, but the Python api functions to > raise WindowsErrors are named like PyErr_SetFromWindowsErr() or so. > Forgot to add that to the grep statement after I discovered that. > Typically, WindowsErrors are raised when Win32 API functions fail. > In the core extension modules, I find at least mmapmodule.c, > posixmodule.c, _subprocess.c, and _winreg.c raising them. It may be a > bit hidden, because the docs for _winreg mention only EnvironmentError, > but they are wrong: > > C:\>py > Python 2.5a0 (#60, Jul 4 2005, 19:53:27) [MSC v.1310 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import _winreg > >>> _winreg.OpenKey(_winreg.HKEY_CLASSES_ROOT, "blah") > Traceback (most recent call last): > File " ", line 1, in ? > WindowsError: [Errno 2] Das System kann die angegebene Datei nicht finden > >>> > > >> If the name bugs you, I would support renaming it to PlatformError or > >> somesuch. That would make it free for use with Mac errors and Linux > >> errors. Also, it wouldn't tie a language feature to the name of an > >> MS product. > >> > > > > I can compromise to this if others prefer this alternative. Anybody > > else have an opinion? > > Win32 has the FormatError() api to convert error codes into descriptions > - these descriptions are very useful, as are the error codes when you > catch errors in client code. > > I would say as long as the Python core contains win32 specific modules > like _winreg WindowsError should stay. For the name, I have no > preference but I see no need to change it. > OK, then it will just stay as-is. People can expect an updated PEP sometime this weekend. -Brett From python at rcn.com Fri Aug 12 19:14:11 2005 From: python at rcn.com (Raymond Hettinger) Date: Fri, 12 Aug 2005 13:14:11 -0400 Subject: [Python-Dev] plans for 2.4.2 and 2.5a1 Message-ID: <001001c59f61$43a72a00$9023a044@oemcomputer> [Josiah] > At this point, I'd be happy to get > /any/ mechanism, with a preference to struct and/or binascii Assign 1023290 to me and I'll get it done in the next month or so. Raymond From martin at v.loewis.de Fri Aug 12 23:51:35 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 12 Aug 2005 23:51:35 +0200 Subject: [Python-Dev] plans for 2.4.2 and 2.5a1 In-Reply-To: <200508111751.56899.anthony@interlink.com.au> References: <200508111751.56899.anthony@interlink.com.au> Message-ID: <42FD19E7.6060108@v.loewis.de> Anthony Baxter wrote: > So I'm currently planning for a 2.4.2 sometime around mid September. I figure > we cut a release candidate either on the 7th or 14th, and a final a week > later. I'm returning only on Sep 12 from vacation, so the Windows binaries of a release candidate would have to wait until that Monday; the 14th would suit me better. Unfortunately, I'm likely also travelling on 19..23, so the final release would have to wait until Sep 24 or so. Regards, Martin From skip at pobox.com Sat Aug 13 02:51:38 2005 From: skip at pobox.com (skip@pobox.com) Date: Fri, 12 Aug 2005 19:51:38 -0500 Subject: [Python-Dev] Hosting svn.python.org In-Reply-To: <20050812231416.638DB1E4005@bag.python.org> References: <20050812231416.638DB1E4005@bag.python.org> Message-ID: <17149.17434.157348.230440@montanaro.dyndns.org> martin> Log Message: martin> Add wush.net hosting. ... martin> + * Greg Stein suggested http://www.wush.net/subversion.php. ... I will enthusiastically cast my vote for tummy.com, Sean Reifschneider's company. Mojam leases a server there (Mojam & Musi-Cal websites running CentOS 4, Apache+mod_perl, Python, Mason, MySQLdb, Mailman, etc). Their service has been absolutely awesome. Sean is one of the python.org webmasters to boot, so he knows our culture very well already. Skip From goodger at python.org Sat Aug 13 03:57:46 2005 From: goodger at python.org (David Goodger) Date: Fri, 12 Aug 2005 21:57:46 -0400 Subject: [Python-Dev] new PEP type: Process Message-ID: <42FD539A.1060407@python.org> Barry Warsaw and I, the PEP editors, have been discussing the need for a new PEP type lately. Martin von L?wis' PEP 347 was a prime example of a PEP that didn't fit into the existing Standards Track and Informational categories. We agreed upon a new "Process" PEP type. For more information, please see PEP 1 (http://www.python.org/peps/pep-0001.html) -- the type of which has also been changed to Process. Other good examples of Process PEPs are the release schedule PEPs, and I understand there may be a new one soon. (Please cc: any PEP-related mail to peps at python.org) -- David Goodger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20050812/18686911/signature.pgp From kbenoit at opersys.com Thu Aug 11 04:59:12 2005 From: kbenoit at opersys.com (Kristian Benoit) Date: Wed, 10 Aug 2005 22:59:12 -0400 Subject: [Python-Dev] xml.parsers.expat no userdata in callback functions Message-ID: <1123729152.23755.3.camel@localhost> In the C version of expat, handlers receive a void *userdata, but it is not the case in the python version. This means one cant parse multiple files at the same time using the same handlers. You cant pass the context current context to the handler, you must base your code on global variables which is not so nice. Thanks Please leave the cc in the mail header as I'm not subscribed to the list. Kristian From senko.rasic at gmail.com Fri Aug 12 01:40:03 2005 From: senko.rasic at gmail.com (Senko Rasic) Date: Fri, 12 Aug 2005 01:40:03 +0200 Subject: [Python-Dev] Extension to dl module to allow passing strings from native function Message-ID: <48bbc5810508111640a6bd03e@mail.gmail.com> Hi all, recently I've tried to use dl module functionality to interface with external C function. (It was a quick hack so I didn't want to write wrapper code). To my dismay I learned that call method doesn't allow passing data by reference (since strings are immutable in python) - but passing pointers around and modifying caller's data is used all the time in C, so that makes dl practically useless. I've hacked the method to allow mutable data, by allocating temporary buffers for all string arguments passed to it, calling the c function, and then constructing new strings from the data in those buffers and returning them in a tuple together with function return code. Combined with pack/unpack from struct module, this allows passing any structure to and from the external C function, so, imho, it's a useful thing to have. To my knowledge, this functionality can't be achieved in pure python programs, and there's no alternative dynamic loader module that can do it. More info with examples: http://ptlo.blogspot.com/2005/08/pyinvoke.html Source: http://software.senko.net/pub/python-dl2.tgz (the tarball contains setup.py and my dlmodule.c version, for experimenting without patching the official module, and patch made against (fairly recent) cvs version of dlmodule.c) Thoughts, comments? Could this be put in standard module, does it make sense, etc? Regards, Senko -- Senko Rasic From aahz at pythoncraft.com Sat Aug 13 06:51:26 2005 From: aahz at pythoncraft.com (Aahz) Date: Fri, 12 Aug 2005 21:51:26 -0700 Subject: [Python-Dev] new PEP type: Process In-Reply-To: <42FD539A.1060407@python.org> References: <42FD539A.1060407@python.org> Message-ID: <20050813045125.GA1985@panix.com> On Fri, Aug 12, 2005, David Goodger wrote: > > Barry Warsaw and I, the PEP editors, have been discussing the > need for a new PEP type lately. Martin von L?wis' PEP 347 was > a prime example of a PEP that didn't fit into the existing > Standards Track and Informational categories. We agreed upon a > new "Process" PEP type. For more information, please see PEP 1 > (http://www.python.org/peps/pep-0001.html) -- the type of which has > also been changed to Process. Go ahead and make PEP 6 a Process PEP. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ The way to build large Python applications is to componentize and loosely-couple the hell out of everything. From raymond.hettinger at verizon.net Sat Aug 13 13:34:34 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sat, 13 Aug 2005 07:34:34 -0400 Subject: [Python-Dev] Discussion draft: Proposed Py2.5 C API for set and frozenset objects Message-ID: <000701c59ffa$fb54b480$772dc797@oemcomputer> The object and types -------------------- PySetObject subclass of object used for both sets and frozensets PySet_Type a basetype PyFrozenSet_Type a basetype The type check macros --------------------- PyFrozenSet_CheckExact(ob) a frozenset PyAnySet_CheckExact(ob) a set or frozenset PyAnySet_Check(ob) a set, frozenset, or subclass of either The constructors ---------------- obj PySet_New(it) takes an iterable or NULL; returns new ref obj PyFrozenSet_New(it) takes an iterable or NULL; returns new ref The fine grained methods ------------------------ int PySet_Size(so) int PySet_Contains(so, key) 1 for yes; 0 for no; -1 for error raises TypeError for unhashable key does not automatically convert to frozenset int PySet_Add(so, key) 0 for success; -1 for error raises TypeError for unhashable key raises MemoryError if no room to grow obj PySet_Pop(so) return new ref or NULL on failure raises KeyError if set is emtpy int PySet_Discard(so, key) 1 if found and removed 0 if not found (does not raise KeyError) -1 on error raises TypeError for unhashable key does not automatically convert to frozenset Course grained methods left for access through PyObject_CallMethod() -------------------------------------------------------------------- copy, clear, union, intersection, difference, symmetric_difference, update, intersection_update, difference_update, symmetric_difference_update issubset, issuperset, __reduce__ Other functions left for access through the existing abstract API ----------------------------------------------------------------- PyObject_RichCompareBool() PyObject_Hash() PyObject_Repr() PyObject_IsTrue() PyObject_Print() PyObject_GetIter() From martin at v.loewis.de Sat Aug 13 13:47:10 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 13 Aug 2005 13:47:10 +0200 Subject: [Python-Dev] Hosting svn.python.org In-Reply-To: <17149.17434.157348.230440@montanaro.dyndns.org> References: <20050812231416.638DB1E4005@bag.python.org> <17149.17434.157348.230440@montanaro.dyndns.org> Message-ID: <42FDDDBE.3040903@v.loewis.de> skip at pobox.com wrote: > I will enthusiastically cast my vote for tummy.com, Sean Reifschneider's > company. Mojam leases a server there (Mojam & Musi-Cal websites running > CentOS 4, Apache+mod_perl, Python, Mason, MySQLdb, Mailman, etc). Their > service has been absolutely awesome. But we don't want to lease a server - we are looking for an Subversion hoster. If we *just* wanted a server, there would be no reason to drop (*) the current svn.python.org. So what precisely is the Subversion offer of tummy.com ($/per month for what disk limit, monthly download limit, number of developers limit, backup service, email notification, ability for offsite download of the repository tarball, what access method (is svn+ssh supported, anonymous WebDAV))? In case this isn't clear yet: several people are concerned that running the Python svn repository by volunteers will risk service outage, and unnecessarily consume volunteer resources. So just replacing the machine we get for free now with a machine we have to pay for won't do any good. I understand that I could now go to tummy.com, contact them, and research all details myself. But I'm not willing to: everybody who wants to suggest a different service should find out all the details of that service, and report them so I can include them into the PEP. Regards, Martin (*) This PEP is actually not at all about svn.python.org, and the pydotorg SVN repository. Those are in the realms of the infrastructure committee, and they do a great job. The PEP is *only* about migrating the Python source code proper from CVS (along with the other code snippets that are in that CVS). From martin at v.loewis.de Sat Aug 13 13:50:06 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 13 Aug 2005 13:50:06 +0200 Subject: [Python-Dev] xml.parsers.expat no userdata in callback functions In-Reply-To: <1123729152.23755.3.camel@localhost> References: <1123729152.23755.3.camel@localhost> Message-ID: <42FDDE6E.2050309@v.loewis.de> Kristian Benoit wrote: > This means one cant parse multiple files at the same time using the same > handlers. You cant pass the context current context to the handler, you must > base your code on global variables which is not so nice. This is not true. You can create multiple parsers, and then can make the callback functions bound methods, using self to store parse-specific data. There is no need to have extra callback data. Regards, Martin From martin at v.loewis.de Sat Aug 13 13:56:48 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 13 Aug 2005 13:56:48 +0200 Subject: [Python-Dev] Extension to dl module to allow passing strings from native function In-Reply-To: <48bbc5810508111640a6bd03e@mail.gmail.com> References: <48bbc5810508111640a6bd03e@mail.gmail.com> Message-ID: <42FDE000.9080508@v.loewis.de> Senko Rasic wrote: > Thoughts, comments? Could this be put in standard module, does it make > sense, etc? Are you aware of the ctypes module? http://starship.python.net/crew/theller/ctypes/ Regards, Martin From goodger at python.org Sat Aug 13 14:38:36 2005 From: goodger at python.org (David Goodger) Date: Sat, 13 Aug 2005 08:38:36 -0400 Subject: [Python-Dev] new PEP type: Process In-Reply-To: <20050813045125.GA1985@panix.com> References: <42FD539A.1060407@python.org> <20050813045125.GA1985@panix.com> Message-ID: <42FDE9CC.8030008@python.org> [Aahz] > Go ahead and make PEP 6 a Process PEP. Done! -- David Goodger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20050813/c4db8c5f/signature.pgp From wsanchez at wsanchez.net Sat Aug 13 18:02:00 2005 From: wsanchez at wsanchez.net (=?ISO-8859-1?Q?Wilfredo_S=E1nchez_Vega?=) Date: Sat, 13 Aug 2005 09:02:00 -0700 Subject: [Python-Dev] Exception Reorg PEP checked in In-Reply-To: References: Message-ID: <6AA5C3D1-6AEC-4CE4-AEB9-84FBDA10EFA9@wsanchez.net> I'm curious about why Python lacks FileNotFoundError, PermissionError and the like as subclasses of IOError. Catching IOError and looking at errno to figure out what went wrong seems pretty unpythonic, and I've often wished for built-in subclasses of IOError. I sometimes subclass them myself, but a lot of the time, I'm catching such exceptions as thrown by the standard library. -wsv From gvanrossum at gmail.com Sat Aug 13 23:02:54 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat, 13 Aug 2005 14:02:54 -0700 Subject: [Python-Dev] xml.parsers.expat no userdata in callback functions In-Reply-To: <42FDDE6E.2050309@v.loewis.de> References: <1123729152.23755.3.camel@localhost> <42FDDE6E.2050309@v.loewis.de> Message-ID: > Kristian Benoit wrote: > > This means one cant parse multiple files at the same time using the same > > handlers. You cant pass the context current context to the handler, you must > > base your code on global variables which is not so nice. > "Martin v. L?wis" replied: > This is not true. You can create multiple parsers, and then can make the > callback functions bound methods, using self to store parse-specific > data. There is no need to have extra callback data. What he said. Kristian's complaint is probably a common misconception about Python -- not too many languages have unified the concepts of "bound methods" and "callables" so completely as Python. Every callable is in a sense a closure (or can be). Nested functions are other examples. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvanrossum at gmail.com Sat Aug 13 23:27:22 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat, 13 Aug 2005 14:27:22 -0700 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <42FBA376.5030605@canonical.com> References: <42FBA376.5030605@canonical.com> Message-ID: With permission, I'm forwarding an email from Mark Shuttleworth about Bazaar-2 (aka Bazaar-NG), a distributed source control system (not entirely unlike bitkeeper, I presume) written in Python and in use by the Ubuntu system. What do people think of using this for Python? Is it the right model? Do we want to encourage widespread experimentation with the Python source code? --Guido van Rossum (home page: http://www.python.org/~guido/) ---------- Forwarded message ---------- From: Mark Shuttleworth Date: Aug 11, 2005 12:13 PM Subject: Distributed RCS To: Guido van Rossum Cc: Steve Alexander , Martin Pool , Fredrik Lundh Hi Guido Steve forwarded your mail to me regarding distributed revision control, so I thought I'd follow up with some thoughts on why I agree with Frederick Lundh that it's important, and where we are going with the Bazaar project. First, distributed RCS systems reduce the barrier to participation. Anybody can create their own branches, and begin work on features, with full revision control support, without having to coordinate with the core RCS server sysadmin. So, for example, if someone gets an idea to work on PEP X, they can simply create a branch, and start hacking on it locally, with full RCS functionality like commit and undo, and logs of their changes over time. They can easily merge continually from the trunk, to keep their branch up to date. And they can publish their branch using only a web server. With Bazaar, these branches can be personal or shared group branches. The net effect of this is to make branching a core part of the development process. Each feature gets developed on a branch, and then merged when its ready. Instead of passing patches around in email, you find yourself passing out branch references, which are much easier to deal with since they are always "up to date". In Launchpad, we have evolved to work around this branch-per-feature approach, and built a review process so that each branch gets a review before the code is merged to the trunk. It also has a positive social impact, because groups that are interested in a feautre can begin to collaborate on it immediately rather than waiting to get consensus from everybody else, they just start their branch and get more testing when it is reaching a reasonable state of maturity - then the project decides whether or not it lands. That results in less argument about whether or not a feature is a good idea before anybody really knows what it's going to look like. Those who are interested, participate, and those who aren't reserve judgement till it's done. As for Bazaar, we have just wrapped up our latest sprint, where we decided that bazaar-ng (bzr), which is being written in Python by Martin Pool, will become Bazaar 2.x, in the first quarter of 2006. The current 1.x line of development has served us well, but the ideas we developed and which have been implemented as a working bazaar-ng reference by Martin are now proven enough that I'm committing the project (Ubuntu, and all of Launchpad) to it. Martin will continue to work on it full time, and will be joined by the current Bazaar 1.x team, Robert Collins, David Allouche and James Blackwell. That makes for a substantial chunk of resources but I think it's worth it because we need a truly superb free revision control system when dealing with something as large and complex as an entire distribution. The whole of Ubuntu will be in Bazaar in due course. Currently, we have about 500 upstreams published in the Bazaar 1.x format (see http://bazaar.ubuntu.com/ for the list), all of those will be converted to Bazaar 2.x and in addition we will continue to publish more and more upstreams in the 2.x archive format. We actively convert CVS and SVN upstreams and publish them in the Bazaar format to allow us to use a single, distributed revision control system across all of those packages. So there's a lot of real-world data and real-world coding going on with Bazaar as the RCS holding it all together. Perhaps more importantly, we are integrating Bazaar tightly with the other Launchpad applications, Rosetta and Malone. This means that bug tracking and translation will be "branch aware". You will be able to close a bug by noting that a commit in one of your branches fixes the bug, then merging it into the relevant mainline branch, and have the launchpad bug tracker automatically mark the bug as closed, if you wish. Similarly you will be able to get the latest translations just by merging from the branch published by Rosetta that has the latest translations in it for your application. The combination of distributed revision control, and ultimately integrated bug tracking and translations, will I think be a very efficient platform for collaborative development. Bazaar is free, and the use of Launchpad is free though we have not yet released the code to the web services for bug tracking and translation. I hope that puts bazaar into perspective for you. Give it a spin - the development 2.x codebase is robust enough now to handle a line of development and do basic merging, we are switching our own development to the pre-release 2.x line in October, and we will switch over all the public archives we maintain in around March next year. Cheers, Mark From skip at pobox.com Sun Aug 14 01:00:37 2005 From: skip at pobox.com (skip@pobox.com) Date: Sat, 13 Aug 2005 18:00:37 -0500 Subject: [Python-Dev] cvs to bzr? Message-ID: <17150.31637.180169.877441@montanaro.dyndns.org> Based on the message Guido forwarded, I installed bazaar-ng. From Mark's note it seems they convert cvs repositories to bzr repositories, but I didn't see any mention in the bzr docs of any sort of cvs2bzr tool. Likewise, Google didn't turn up anything obvious. Anyone know of something? Thx, Skip From nas at arctrix.com Sun Aug 14 02:02:40 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Sat, 13 Aug 2005 18:02:40 -0600 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: References: <42FBA376.5030605@canonical.com> Message-ID: <20050814000240.GA5470@mems-exchange.org> On Sat, Aug 13, 2005 at 02:27:22PM -0700, Guido van Rossum wrote: > What do people think of using this for Python? I think it deserves consideration. One idea would be to have a Bazaar-NG repository that tracks the CVS SF repository. I haven't tried it yet but there is a tool called Tailor[1] that automates the task. That would give people a chance to experiment with Bazaar-NG (and still work with SF is down) without committing to it. > Is it the right model? Do we want to encourage widespread > experimentation with the Python source code? I think Python works fairly well with the centralized model. However, I expect it's hard to know what we are missing. Neil 1. http://darcs.net/DarcsWiki/Tailor From nas at arctrix.com Sun Aug 14 02:03:46 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Sat, 13 Aug 2005 18:03:46 -0600 Subject: [Python-Dev] cvs to bzr? In-Reply-To: <17150.31637.180169.877441@montanaro.dyndns.org> References: <17150.31637.180169.877441@montanaro.dyndns.org> Message-ID: <20050814000346.GB5470@mems-exchange.org> On Sat, Aug 13, 2005 at 06:00:37PM -0500, skip at pobox.com wrote: > Based on the message Guido forwarded, I installed bazaar-ng. From Mark's > note it seems they convert cvs repositories to bzr repositories, but I > didn't see any mention in the bzr docs of any sort of cvs2bzr tool. Haven't tried it but should work: http://darcs.net/DarcsWiki/Tailor From gvanrossum at gmail.com Sun Aug 14 02:11:05 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat, 13 Aug 2005 17:11:05 -0700 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <42FD2A13.8000900@canonical.com> References: <42FBA376.5030605@canonical.com> <42FD2A13.8000900@canonical.com> Message-ID: Another fwd, describing how Steve Alexander's group user bazaar. --Guido van Rossum (home page: http://www.python.org/~guido/) ---------- Forwarded message ---------- From: Steve Alexander Date: Aug 12, 2005 4:00 PM Subject: Re: Distributed RCS To: Guido van Rossum Cc: Mark Shuttleworth , Martin Pool , Fredrik Lundh Hi Guido, I'm not going to post to python-dev just now, because I'm leaving on 1.5 weeks vacation tomorrow, and I'd rather be absent than unable to answer questions promptly. Martin Pool will be around next week, and will be able to take part in discussions on the list. Feel free to post all or part of Mark's or my emails to the python lists. Mark wrote: > > I hope that puts bazaar into perspective for you. Give it a spin - the > development 2.x codebase is robust enough now to handle a line of > development and do basic merging, we are switching our own development > to the pre-release 2.x line in October, and we will switch over all the > public archives we maintain in around March next year. A large part of the internal development at Canonical is the Launchpad system. This is about 30-40 kloc of Python code, including various Twisted services, cron scripts, a Zope 3 web application, database tools, ... It's being worked on by 20 software developers. Everyone uses bazaar 1.4 or 1.5, and around October, we'll be switching to use bazaar 2.x. I'll describe how we work on Launchpad using Bazaar. This is all from the Bazaar 1.x perspective, and some things will become simpler when we change to using Bazaar 2.x. I've left the description quite long, as I hope it will give you some of the flavour of working with a distributed RCS. == Two modes of working: shared branches and PQM == Bazaar supports two different modes of working for a group like the Launchpad team. 1. There's a shared read/write place that all the developers have access to. This is contains the branches we release from, and represents the "trunk" of the codebase. 2. A "virtual person" called the "patch queue manager" (PQM) has exclusive write access to a collection of branches. PQM takes instructions as GPG signed emails from launchpad developers, to merge their code into PQM's branches. We use the latter mode because we have PQM configured not only to accept requests to merge code into PQM's codebase, but to run all the tests first and refuse to merge if any test fails. == The typical flow of work on Launchpad == Say I want to work on some new feature for Launchpad. What do I do? 1. I use 'baz switch' to change my working tree from whatever I was working on last, and make it become PQM's latest code. baz switch rocketfuel at canonical.com/launchpad--devel--0 "rocketfuel" is the code-name for the branches we release our code from. PQM manages the rocketfuel branches. In Bazaar 1.x, collections of branches are called "archives" and are identified by an email address plus some other optional information. So, "rocketfuel at canonical.com" is PQM's email address. "launchpad--devel--0" is simply the name of the main launchpad branch. The format of branch names is very strict in Bazaar 1.x. It is much more open in Bazaar 2.x. 2. I use 'baz branch' to create my own branch of this code that I can commit changes to. baz branch steve.alexander at canonical.com/launchpad--ImproveLogins--0 My archive is called "steve.alexander at canonical.com". The branch will be used to work on the login functionality of Launchpad, so I have named the branch "launchpad--ImproveLogins--0". 3. I hack on the code, and from time to time commit my changes. I need to 'baz add' new files and directories, and 'baz rm' to remove files, and 'baz mv' to move files around. # hack hack hack baz commit -s "Refactored the whatever.py module." # hack hack hack baz del whatever_deprecated.py baz commit -s "Removed deprecated whatevers." # hack hack hack 4. Let's say I hacked on some stuff, but I didn't commit it. I don't like what I did, and I want to start again. # hack hack hack baz undo 'baz undo' puts the source code back into the state it was in after the last commit, and puts the changes somewhere. If I change my mind again, I can say 'baz redo', and get my changes back. 5. All this hacking and committing has been happening on my own workstation, without a connection to the internet. Perhaps I've been on a plane or at a cafe. When I have a connection again, I can make my work available for others to see by mirroring my code to a central location. Each Launchpad developer has a mirror of the archive they use for Launchpad work on a central machine at the Canonical data centre. In our case, the mirror command uses sftp to copy the latest changes I have made into the mirror on this central server. baz archive-mirror 6. Because we have a strict code review proccess for Launchpad development, I can't (or rather, shouldn't) submit my changes to PQM yet. I should get it reviewed. But, let's say Andrew wants to do some work that depends on my work, before my work has made its way into PQM's rocketfuel "Trunk". He can simply merge from me. # in Andrew's working tree, on his workstation. baz merge steve.alexander at canonical.com/launchpad--ImproveLogins--0 baz commit -s "Merged steve's ImproveLogins work." When Andrew eventually gets his work reviewed, and sends it on to PQM to be merged into Rocketfuel, the Bazaar merging algorithms will work out that Andrew merged from me, and will sort things out. Of course, there can be conflicts when people have worked in divergent ways on the same code. These are resolved in a similar way to CVS or SVN. 7. I want to get my code reviewed by a member of the review team. I add the details of my branch to the PendingReviews page on the launchpad development wiki. This wiki is publicly readable. https://wiki.launchpad.canonical.com/PendingReviews There is a script that periodically reads the PendingReviews page, attempts to merge the branches listed there into rocketfuel (just as PQM would do), and produces a diff for use by the review team. The diff represents what changes would be made to the rocketfuel Trunk were the branch in question to be sent to PQM. This diff is often enough for the reviewers to work with. If they need to see more context, they can simply check out the branch in question using 'baz get branchname'. The script also highlights whether there were any conflicts that would prevent a merge, and gives an indication of the size of the change. The script's output is accessible only to Launchpad developers. However, I've made a couple of screenshots to give you some idea of what it looks like. This is the summary page, that uses information taken from the PendingReviews wiki page. http://people.ubuntu.com/~stevea/branch-summary.png This is a typical diff representing what is to be merged. http://people.ubuntu.com/~stevea/branch-diff.png The reviewer sends an email to the author of the code, cc the launchpad-reviews mailing list. The review email typically has sections of code included, each line prefixed with '> ', with comments, questions and requests for improvement beneath each section of code. The reviewer will either approve the code for merging, approve the code providing certain remedial actions are taken, or reject the code, requiring a new review later. 8. My code has been successfully reviewed by JamesH, so I send a signed mail to PQM asking to merge my work into rocketfuel. submit-merge "r=JamesH, Improvements to logging in." pqm at pqm.ubuntu.com PQM checks that each merge request has r=someone in the message, as a reminder that launchpad developers need to have their code reviewed. The submit-merge script gets takes the archive name, the branch name, and the "patch level" that the branch is at, composes an email saying "pqm, please merge steve.alexander at canonical.com/launchpad--ImproveLogins-0--patch-18 into rocketfuel at canonical.com/launchpad--devel--0." Signs it with my gpg key, and mails it. Some time later, once PQM has merged the code and successfully run all the launchpad tests, an email will go out to me, and to a pqm-commits mailing list, saying that the merge was successful. If it was unsuccessful, I get an email with the error output. An irc robot listens to the pqm-commits mailing list, and announces new landings to the rocketfuel Trunk on irc. == Naming branches == The Launchpad team is distributed around the world. To cope with this, and also to get our community of users involved in the development of the software, Launchpad development emphasises writing specifications and proposals, and implementing features based on these proposals. You can read all the launchpad proposals on the launchpad development wiki. https://wiki.launchpad.canonical.com/ So, we usually name branches after the specification that is being implemented on that branch. The branch is named near the top of the specification, so someone reading the specification who has access to the source code can see what's happening with the implementation. Branches are also often named after bugs. For example, launchpad--bug123--0. The use of '--' in branch names, and the '--0' thing at the end is occassionally useful, but more of a hangover from the 'tla' system that bazaar is based on. This strict branch naming format is not being carried over into bazaar 2. == External contributors == The source code to Launchpad is not available at this time. We intend to make it open source at some point in the future, but I'm not sure when that will be. Let's consider what would happen if we decided to make the Launchpad code fully open source tommorrow. Someone from outside of Canonical could get a copy of the main launchpad "rocketfuel" branch, make their own branch by branching from the rocketfuel branch, do a bunch of work, mirror it to their own website, and email a Canonical launchpad developer to ask that it be reviewed, or merged into that launchpad developer's branch. This way, even though an outside contributor doesn't have rights with PQM, they could still make fine-grained commits, merge frmo a variety of places, and participate at the same level as someone employed by Canonical. -- Steve Alexander From tjreedy at udel.edu Sun Aug 14 05:07:49 2005 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 13 Aug 2005 23:07:49 -0400 Subject: [Python-Dev] Distributed RCS References: <42FBA376.5030605@canonical.com> <42FD2A13.8000900@canonical.com> Message-ID: > Another fwd, describing how Steve Alexander's group user bazaar. I found this rather clear and easy to understand even without having directly used CVS (other than to browse). Some of the automation features seem useful but I don't know whether they are specific to bazaar. Anyway, my thoughts. It seems to me that auto testing of the tentatively updated trunk before final commitment would avoid the 'who checked in test-breaking code' messages that appear here occasionally. But it requires that the update + test-suite time be less than the average inter-update interval. I understand the main difference between baz and cvs (and similar) to be that checked-out-to-developers copies remain 'within' the distributed system and accessible to the master system rather than becoming external (and lost track of) copies. In consequence (again if I understand correctly), pre- and post-review diffs and merges are done directly between the developers branch and the current system trunk rather than (for diffs) with a possibly out-of-date master on the developer's machine, leading to trunk updates with a possibly out-of-date diff. If so, this would eliminate reviewers having to make requests such as 'please run a new diff against the current CVS head' that I remember sometimes seeing on the SF tracker. The current bottleneck in Python development appears to be patch reviews. So merely making submission and commitment easier will not help much. An alternative to more reviewers is more automation to make more effective use of existing reviewers. (And this might also encourage more reviewers.) The Launchpad group seems to be ahead in this regard, but I don't know how much this is due to using bazaar. In any case, ease of improving the review process might be a criterion for choosing a source code system. But I leave this to ML. *Other things being equal*, using a state-of-the-art development system written in Python to develop Python would be a marketing plus. Terry J. Reedy From anthony at interlink.com.au Sun Aug 14 06:36:33 2005 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun, 14 Aug 2005 14:36:33 +1000 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: References: <42FBA376.5030605@canonical.com> Message-ID: <200508141436.36913.anthony@interlink.com.au> I have great hopes for baz-ng, but I don't know that it's really ready for production use just yet. I don't know that we want to be right out on the bleeding edge of revision control systems for Python. The current bazaar, last time I looked (a few months ago) did not work on Windows. This is a complete deal-breaker for us, unless we can agree to dump that Windows support (who needs it, really? ) I *hope* that baz-ng will work fine on Windows - I haven't looked too closely at that side of it. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From skip at pobox.com Sun Aug 14 13:44:25 2005 From: skip at pobox.com (skip@pobox.com) Date: Sun, 14 Aug 2005 06:44:25 -0500 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <200508141436.36913.anthony@interlink.com.au> References: <42FBA376.5030605@canonical.com> <200508141436.36913.anthony@interlink.com.au> Message-ID: <17151.11929.958753.192768@montanaro.dyndns.org> Anthony> The current bazaar, last time I looked (a few months ago) did Anthony> not work on Windows. This is a complete deal-breaker for us, I assume it would be a deal breaker for many people. According to the Bazaar-NG website it works on "Linux, Windows and Mac OS X, or any system with a Python interpreter". If it's that platform-independent, perhaps it will work on some systems that don't support CVS. It does require Python 2.4, though I doubt that would be a great hardship for many people interested in Python development. Skip From martin at v.loewis.de Sun Aug 14 14:01:46 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 14 Aug 2005 14:01:46 +0200 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: References: <42FBA376.5030605@canonical.com> Message-ID: <42FF32AA.7040506@v.loewis.de> Guido van Rossum wrote: > With permission, I'm forwarding an email from Mark Shuttleworth about > Bazaar-2 (aka Bazaar-NG), a distributed source control system (not > entirely unlike bitkeeper, I presume) written in Python and in use by > the Ubuntu system. What do people think of using this for Python? Is > it the right model? Like Skip, I tried experimenting with it. While that may be the right model, I don't think it is the right software. In bazaar-ng 0.0.5 (which is what Debian unstable currently has), bzr commit would not open a text editor, but require the commit message on the command line; selective commit of only some of the changed files is also not supported. bzr diff cannot show the changes between two revisions, and cannot show revisions across branches. So I assume that using bazaar-ng right now would cause problems in day-to-day usage. Regards, Martin From skip at pobox.com Sun Aug 14 14:37:55 2005 From: skip at pobox.com (skip@pobox.com) Date: Sun, 14 Aug 2005 07:37:55 -0500 Subject: [Python-Dev] cvs to bzr? In-Reply-To: <20050814000346.GB5470@mems-exchange.org> References: <17150.31637.180169.877441@montanaro.dyndns.org> <20050814000346.GB5470@mems-exchange.org> Message-ID: <17151.15139.655967.250675@montanaro.dyndns.org> >> ... I didn't see any mention in the bzr docs of any sort of cvs2bzr >> tool. Neil> Haven't tried it but should work: Neil> http://darcs.net/DarcsWiki/Tailor Thanks Neil. I downloaded it last night and played around a bit. What follows is a description that will hopefully keep others from stepping in the same booby traps I did. It doesn't appear to work at this point, both based on attempts to use it and hints on the above page. After some reading, I was able to pull the latest version with wget --exclude-directories=/~lele/projects/tailor/_darcs --mirror \ --no-parent --no-host-directories --cut-dirs=3 -e robots=off \ http://nautilus.homeip.net/~lele/projects/tailor/ (The wget example at the top of the wiki page points to an older version.) Unfortunately, it (like the older version) is missing if __name__ == "__main__": main() in tailor.py and has no call to its main() function anywhere in the source. This seemed very odd to me, so I added one, as well as a #! line. I eventually noticed that there is a vcpx package and in the directory above, a number of other files, tailor, a bunch of index.html files and a README. It was reminiscent of expanding a tar file in the bad old days (or unzipping a zip file nowadays) before everbody got the idea that it would be a good idea to create a top-level directory... Anyway, I'm still struggling with it. If I get further I'll post my results. If others have gotten further, tips would be appreciated. Skip From skip at pobox.com Sun Aug 14 14:46:16 2005 From: skip at pobox.com (skip@pobox.com) Date: Sun, 14 Aug 2005 07:46:16 -0500 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <42FF32AA.7040506@v.loewis.de> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> Message-ID: <17151.15640.173982.961359@montanaro.dyndns.org> Martin> Like Skip, I tried experimenting with it. While that may be the Martin> right model, I don't think it is the right software. [problems Martin> elided] Martin> So I assume that using bazaar-ng right now would cause problems Martin> in day-to-day usage. Granted. What is the cost of waiting a bit longer to see if it (or something else) gets more usable and would hit the mark better than svn? I presume that once we switch away from cvs to something else, it's unlikely we would switch again unless some huge roadblock appeared that made the initial change the wrong one. I was amazed at the number of different version control systems out there now. CVS, while enormously successful from a practical standpoint, clearly has its detractors. That there are so many alternatives suggests that it's not clear yet what the "correct" feature set for a version control system is. Skip From martin at v.loewis.de Sun Aug 14 18:16:11 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 14 Aug 2005 18:16:11 +0200 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <17151.15640.173982.961359@montanaro.dyndns.org> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> Message-ID: <42FF6E4B.4000206@v.loewis.de> skip at pobox.com wrote: > Granted. What is the cost of waiting a bit longer to see if it (or > something else) gets more usable and would hit the mark better than svn? I > presume that once we switch away from cvs to something else, it's unlikely > we would switch again unless some huge roadblock appeared that made the > initial change the wrong one. It depends on what "a bit" is. Waiting a month would be fine; waiting two years might be pointless. So I think I will personally pursue PEP 347 (switching to SVN); it will be then an issue of BDFL pronouncement. Regards, Martin From nas at arctrix.com Sun Aug 14 19:12:59 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 14 Aug 2005 11:12:59 -0600 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <42FF6E4B.4000206@v.loewis.de> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> Message-ID: <20050814171259.GA8200@mems-exchange.org> On Sun, Aug 14, 2005 at 06:16:11PM +0200, "Martin v. L?wis" wrote: > It depends on what "a bit" is. Waiting a month would be fine; waiting > two years might be pointless. It looks like the process of converting a CVS repository to Bazaar-NG does not yet work well (to be kind). The path CVS->SVN->bzr would probably work better. I suspect cvs2svn has been used on quite a few CVS repositories already. I don't think going to SVN first would lose any information. My vote is to continue with the migration to SVN. We can re-evaluate Bazaar-NG at a later time. Neil From ronaldoussoren at mac.com Sun Aug 14 19:17:00 2005 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 14 Aug 2005 19:17:00 +0200 Subject: [Python-Dev] build problems on macosx (CVS HEAD) Message-ID: <39DD497A-89DB-4DF0-A272-62BB66CEC090@mac.com> Hi, I'm trying to build CVS HEAD on OSX 10.4.2 (Xcode 2.1), with a checkout that is less than two hours old. I'm building a standard unix tree (no framework install): $ ./configure --prefix=/opt/python/2.5 ... $ make ... ar cr libpython2.5.a Modules/config.o Modules/getpath.o Modules/ main.o Modules/gcmodule.o ar cr libpython2.5.a Modules/threadmodule.o Modules/signalmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/_sre.o Modules/ _codecsmodule.o Modules/zipimport.o Modules/symtablemodule.o Modules/xxsubtype.o ranlib libpython2.5.a c++ -u _PyMac_Error -o python.exe \ Modules/python.o \ libpython2.5.a -ldl case $MAKEFLAGS in \ *-s*) CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python.exe -E ./ setup.py -q build;; \ *) CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup' OPT='- DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python.exe -E ./setup.py build;; \ esac make: *** [sharedmods] Error 139 This is a segmentation fault when trying to build extensions: $ ./python.exe Python 2.5a0 (#5, Aug 14 2005, 18:20:08) [GCC 4.0.0 (Apple Computer, Inc. build 5026)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import setup Segmentation fault The minimal import that causes a crash is 'import distutils.sysconfig'. I've rebuild using --enable-debug and --with- pydebug to check if gdb could tell me more. The start of the stacktrace: (gdb) r -c 'import distutils.sysconfig' Starting program: /Volumes/Data/Users/ronald/Python/python-HEAD/dist/ src/python.exe -c 'import distutils.sysconfig' Reading symbols for shared libraries . done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0xcbcbcbd3 0x001500b0 in structseq_dealloc (obj=0x3c3878) at Objects/structseq.c:47 47 Py_XDECREF(obj->ob_item[i]); (gdb) where #0 0x001500b0 in structseq_dealloc (obj=0x3c3878) at Objects/ structseq.c:47 #1 0x0002fdb0 in _Py_Dealloc (op=0x3c3878) at Objects/object.c:1883 #2 0x000eedb4 in frame_dealloc (f=0x1816c18) at Objects/ frameobject.c:394 #3 0x0002fdb0 in _Py_Dealloc (op=0x1816c18) at Objects/object.c:1883 #4 0x000dd1d0 in fast_function (func=0x390038, pp_stack=0xbfffd788, n=1, na=1, nk=0) at Python/ceval.c:3654 #5 0x000dcdc8 in call_function (pp_stack=0xbfffd788, oparg=1) at Python/ceval.c:3590 #6 0x000d6aa4 in PyEval_EvalFrameEx (f=0x610358, throw=0) at Python/ ceval.c:2181 #7 0x000d98d0 in PyEval_EvalCodeEx (co=0x5180d8, globals=0x5146c8, locals=0x5146c8, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2748 #8 0x000ce270 in PyEval_EvalCode (co=0x5180d8, globals=0x5146c8, locals=0x5146c8) at Python/ceval.c:490 #9 0x0001643c in PyImport_ExecCodeModuleEx (name=0xbfffe808 "distutils.sysconfig", co=0x5180d8, pathname=0xbfffde4c "/Volumes/ Data/Users/ronald/Python/python-HEAD/dist/src/Lib/distutils/ sysconfig.pyc") at Python/import.c:620 At the DECREF, i == 17, size == 18 and obj->ob_item[i] == 0xcbcbcbcb, and obj is an posix.stat_result. From nas at arctrix.com Sun Aug 14 19:21:07 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 14 Aug 2005 11:21:07 -0600 Subject: [Python-Dev] cvs to bzr? In-Reply-To: <20050814000346.GB5470@mems-exchange.org> References: <17150.31637.180169.877441@montanaro.dyndns.org> <20050814000346.GB5470@mems-exchange.org> Message-ID: <20050814172107.GB8200@mems-exchange.org> On Sat, Aug 13, 2005 at 06:03:46PM -0600, Neil Schemenauer wrote: > Haven't tried it but should work: > > http://darcs.net/DarcsWiki/Tailor After applying the attached patch, this command seemed to work for converting the initial revision: ~/src/cvsync/tailor.py --source-kind cvs --target-kind bzr \ --bootstrap --repository ~/Python/python-cvsroot -m python \ --revision r16b1 py_bzr After, I think this command is supposed to bring the bzr repostiory up-to-date: cd py_bzr; ~/src/cvsync/tailor.py -v It does not seem to work for me (it only updates one file and then quits). cvs2svn seems to be much more mature. Neil -------------- next part -------------- diff -rN -u old-cvsync/vcpx/bzr.py new-cvsync/vcpx/bzr.py --- old-cvsync/vcpx/bzr.py 2005-08-14 09:43:15.000000000 -0600 +++ new-cvsync/vcpx/bzr.py 2005-08-14 10:38:02.000000000 -0600 @@ -29,14 +29,23 @@ ## SyncronizableTargetWorkingDir - def _addEntries(self, root, entries): - """ - Add a sequence of entries. - """ + def _addPathnames(self, root, entries): + c = SystemCommand(working_dir=root, command="bzr add --no-recurse" + " %(entries)s") + c(entries=' '.join([shrepr(e) for e in entries])) + def _addSubtree(self, root, *entries): c = SystemCommand(working_dir=root, command="bzr add %(entries)s") - c(entries=' '.join([shrepr(e.name) for e in entries])) + c(entries=' '.join([shrepr(e) for e in entries])) + def _removePathnames(self, root, names): + pass # bzr handles this itself + + def _renamePathname(self, root, oldname, newname): + c = SystemCommand(working_dir=root, + command="bzr mv %(old)s %(new)s") + c(old=shrepr(oldname), new=shrepr(newname)) + def _commit(self,root, date, author, remark, changelog=None, entries=None): """ Commit the changeset. @@ -112,7 +121,7 @@ # Create the .bzrignore file, that contains a glob per line, # with all known VCs metadirs to be skipped. - ignore = open(join(root, '.hgignore'), 'w') + ignore = open(join(root, '.bzrignore'), 'w') ignore.write('\n'.join(['(^|/)%s($|/)' % md for md in IGNORED_METADIRS])) ignore.write('\ntailor.log\ntailor.info\n') From gvanrossum at gmail.com Sun Aug 14 20:11:46 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun, 14 Aug 2005 11:11:46 -0700 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <20050814171259.GA8200@mems-exchange.org> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> <20050814171259.GA8200@mems-exchange.org> Message-ID: On 8/14/05, Neil Schemenauer wrote: > It looks like the process of converting a CVS repository to > Bazaar-NG does not yet work well (to be kind). The path > CVS->SVN->bzr would probably work better. I suspect cvs2svn has > been used on quite a few CVS repositories already. I don't think > going to SVN first would lose any information. > > My vote is to continue with the migration to SVN. We can > re-evaluate Bazaar-NG at a later time. That's looks like a fair assessment -- although it means twice the conversion pain for developers. It sounds as if bazaar-NG can use a bit of its own medicine -- I hope everybody who found a bug in their tools submitted a patch! :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvanrossum at gmail.com Sun Aug 14 20:13:29 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun, 14 Aug 2005 11:13:29 -0700 Subject: [Python-Dev] Fwd: PEP: Migrating the Python CVS to Subversion In-Reply-To: <1123986783.21455.35.camel@linux.site> References: <1123986783.21455.35.camel@linux.site> Message-ID: Here's another POV. (Why does evereybody keep emailing me personally?) --Guido van Rossum (home page: http://www.python.org/~guido/) ---------- Forwarded message ---------- From: Daniel Berlin Date: Aug 13, 2005 7:33 PM Subject: Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion To: gvanrossum at gmail.com (Sorry for the lack of proper References: headers, this is a reply to the email archive). It's been a couple years since i've been in the world of python-dev, but apparently i'm rejoining the mailing list at just the right time. Take all of this for what it is worth: I'm currently responsible for GCC's bugzilla, wiki, in addition to maintaining several optimization areas of the compiler :P. In addition, i'm responsible for pushing GCC (my main love and world :P) towards Subversion. I should note my bias at this point. I now have full commit access to Subversion. However, I've also submitted patches to monotone, etc. We had a long thread about the various alternatives (arch, bzr, etc), and besides "freeness" constraints on what we can run on gcc.gnu.org as an FSF project, it wouldn't have mattered anyway. This has been in the planning for about a year now (mainly waiting for new hardware). Originally, we were hoping to move GCC to monotone, but it didn't mature fast enough (it's way too slow), and we couldn't make it centralized enough for our tastes (more later). The rest of the free tools other than subversion (arch, monotone, git, darcs, etc) simply couldn't handle our repository with reasonable speed/hardware. GCC has project history dating back to 1987. It's a 4 gig CVS repo containing > 1000 tags, and > 300 branches. The changelog alone has 30k trunk revisions. Those distributed systems that carry full history often can't deal with this fast enough or in any space efficient way. arch was an example of this. It had a retarded mechanism that forced users to care about caching certain revisions to speed it up , instead of doing it on it's own. I've never tried converting this repo to bazaar-ng, it wasn't far enough along when i started. It also had no cvs2bzr type program, and we aren't about to lose all our history. Except for monotone (builtin cvs_import) and subversion (cvs2svn), none of the cvs2* programs i've run across either run in reasonable time (those that don't actually understand how to extract rcs revisions would take weeks to convert our repo, literally), or could handle all the complexities a repository with our history present (branches of branches, etc). Most simply crash in weird ways or run out of memory :). Anyway: Monotone took 45 minutes just to diff two old revisions that are one revision away from each other. CVS takes about 2 minutes for the same operation. SVN on fsfs takes 4 seconds. The converted SVN repo has > 100000 revisions, and is only ~15% bigger than the cvs repo (mostly due to stupid copies it has to do to handle some tag fuckery people were doing in some cases. If it had been subversion from the start, it would have been smaller). We have cvs2svn speedup patches that were done with the KDE folks that make cvs2svn io bound again instead of cpu bound (it was O(N^2) in extracting cvs revision texts before). It takes 17 hours to convert the gcc repository now, only 45 minutes of cpu time :). It used to take 52 hours. I've also talked with Linus about version control before. He believes extreme distributed development is the way to go. I believe heavily that in most cases where you have a mix of corporations and free developers, it ends up causing people to "hide the ball" more than they should. This is particularly prevalent in GCC. We don't want design and development done and then sent as mega-patches presented as fait accompli, then watch these people whine as their designs get torn apart. We'd rather have the discussion on the mailing list and the work done in a visible place (IE CVS branch stored in some central place) rather than getting patch bombs. As a result (and there are many other reasons, i'm just presenting one of them), we actually don't *want* to move from a centralized model, in order to help control the social and political problems we'd have to face if we went fully distributed. Python may not face any of these problem to the degree that GCC does (i doubt many projects do actually. GCC is a very weird and tense political situation :P), because of size, etc, in which case, a distributed model may make more sense. However, you need to be careful to make sure that people understand that it hasn't actually changed your real development process (PEP's, etc), only the workflow used to implement it. From bcannon at gmail.com Sun Aug 14 20:37:43 2005 From: bcannon at gmail.com (Brett Cannon) Date: Sun, 14 Aug 2005 11:37:43 -0700 Subject: [Python-Dev] build problems on macosx (CVS HEAD) In-Reply-To: <39DD497A-89DB-4DF0-A272-62BB66CEC090@mac.com> References: <39DD497A-89DB-4DF0-A272-62BB66CEC090@mac.com> Message-ID: On 8/14/05, Ronald Oussoren wrote: > Hi, > > I'm trying to build CVS HEAD on OSX 10.4.2 (Xcode 2.1), with a > checkout that is less than two hours old. I'm building a standard > unix tree (no framework install): > > $ ./configure --prefix=/opt/python/2.5 > ... > $ make > ... > ar cr libpython2.5.a Modules/config.o Modules/getpath.o Modules/ > main.o Modules/gcmodule.o > ar cr libpython2.5.a Modules/threadmodule.o Modules/signalmodule.o > Modules/posixmodule.o Modules/errnomodule.o Modules/_sre.o Modules/ > _codecsmodule.o Modules/zipimport.o Modules/symtablemodule.o > Modules/xxsubtype.o > ranlib libpython2.5.a > c++ -u _PyMac_Error -o python.exe \ > Modules/python.o \ > libpython2.5.a -ldl > case $MAKEFLAGS in \ > *-s*) CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup' > OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python.exe -E ./ > setup.py -q build;; \ > *) CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup' OPT='- > DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python.exe -E ./setup.py > build;; \ > esac > make: *** [sharedmods] Error 139 > > This is a segmentation fault when trying to build extensions: > I can verify the breakage. I did a ``make distclean``, updated, built, and got the same 139 error. I am short on time today, so I don't think I will be able to dive into this right away. -Brett From mwh at python.net Sun Aug 14 21:09:21 2005 From: mwh at python.net (Michael Hudson) Date: Sun, 14 Aug 2005 20:09:21 +0100 Subject: [Python-Dev] build problems on macosx (CVS HEAD) In-Reply-To: <39DD497A-89DB-4DF0-A272-62BB66CEC090@mac.com> (Ronald Oussoren's message of "Sun, 14 Aug 2005 19:17:00 +0200") References: <39DD497A-89DB-4DF0-A272-62BB66CEC090@mac.com> Message-ID: <2mslxcuyhq.fsf@starship.python.net> Ronald Oussoren writes: > Hi, > > I'm trying to build CVS HEAD on OSX 10.4.2 (Xcode 2.1), with a > checkout that is less than two hours old. I'm building a standard > unix tree (no framework install): It seems very likely that it was this change: http://fisheye.cenqua.com/changelog/~br=MAIN/python?cs=MAIN:loewis:20050809145951 Refcounting, posixmodule.c, aiee! You are in a maze of twisty #ifdefs, all alike. I'll probably find the problem fairly soon, we'll see... :) Cheers, mwh -- All obscurity will buy you is time enough to contract venereal diseases. -- Tim Peters, python-dev From ndbecker2 at gmail.com Sun Aug 14 20:14:22 2005 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 14 Aug 2005 14:14:22 -0400 Subject: [Python-Dev] Fwd: Distributed RCS References: <42FBA376.5030605@canonical.com> Message-ID: I encourage everyone to look at mercurial. It is also written in Python. I am using it daily. From mwh at python.net Sun Aug 14 21:21:31 2005 From: mwh at python.net (Michael Hudson) Date: Sun, 14 Aug 2005 20:21:31 +0100 Subject: [Python-Dev] Exception Reorg PEP checked in In-Reply-To: <6AA5C3D1-6AEC-4CE4-AEB9-84FBDA10EFA9@wsanchez.net> ( =?iso-8859-1?q?Wilfredo_S=E1nchez_Vega's_message_of?= "Sat, 13 Aug 2005 09:02:00 -0700") References: <6AA5C3D1-6AEC-4CE4-AEB9-84FBDA10EFA9@wsanchez.net> Message-ID: <2moe80uxxg.fsf@starship.python.net> Wilfredo S?nchez Vega writes: > I'm curious about why Python lacks FileNotFoundError, > PermissionError and the like as subclasses of IOError. Good question. Lack of effort/inertia? > Catching IOError and looking at errno to figure out what went > wrong seems pretty unpythonic, and I've often wished for built-in > subclasses of IOError. The py library does this (http://codespeak.net/py). > I sometimes subclass them myself, but a lot of the time, I'm > catching such exceptions as thrown by the standard library. Well, indeed. OTOH, functions like os.open aren't really *meant* to be pythonic. I don't think this is something I can get interested enough in to work on myself. Cheers, mwh -- As far as I'm concerned, the meat pie is the ultimate unit of currency. -- from Twisted.Quotes From gvanrossum at gmail.com Sun Aug 14 21:27:18 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun, 14 Aug 2005 12:27:18 -0700 Subject: [Python-Dev] Exception Reorg PEP checked in In-Reply-To: <2moe80uxxg.fsf@starship.python.net> References: <6AA5C3D1-6AEC-4CE4-AEB9-84FBDA10EFA9@wsanchez.net> <2moe80uxxg.fsf@starship.python.net> Message-ID: On 8/14/05, Michael Hudson wrote: > Wilfredo S?nchez Vega writes: > > > I'm curious about why Python lacks FileNotFoundError, > > PermissionError and the like as subclasses of IOError. > > Good question. Lack of effort/inertia? Well, I wonder how often it's needed. My typical use is this: try: f = open(filename) except IOError, err: print "Can't open %s: %s" % (filename, err) return and the error printed contains all the necessary details (in fact it even repeats the filename, so I could probably just say "print err"). Why do you need to know the exact reason for the failure? If you simply want to know whether the file exists, I'd use os.path.exists() or isfile(). (Never mind that this is the sometimes-frowned-upon look-before-you-leap; I think it's often fine.) Also note that providing the right detail can be very OS specific. Python doesn't just run on Unix and Windows. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Sun Aug 14 21:35:00 2005 From: mwh at python.net (Michael Hudson) Date: Sun, 14 Aug 2005 20:35:00 +0100 Subject: [Python-Dev] Distributed RCS In-Reply-To: (Terry Reedy's message of "Sat, 13 Aug 2005 23:07:49 -0400") References: <42FBA376.5030605@canonical.com> <42FD2A13.8000900@canonical.com> Message-ID: <2mk6iouxaz.fsf@starship.python.net> "Terry Reedy" writes: > It seems to me that auto testing of the tentatively updated trunk before > final commitment would avoid the 'who checked in test-breaking code' > messages that appear here occasionally. I don't think there's any fundamental impossibility in setting up such a system for CVS, and am pretty certain there's not for SVN. > But it requires that the update + test-suite time be less than the > average inter-update interval. Indeed. > The current bottleneck in Python development appears to be patch reviews. And acting on those reviews... > So merely making submission and commitment easier will not help much. I'm not sure, I think it could help quite a bit. > An alternative to more reviewers is more automation to make more > effective use of existing reviewers. (And this might also encourage > more reviewers.) The Launchpad group seems to be ahead in this > regard, but I don't know how much this is due to using bazaar. In > any case, ease of improving the review process might be a criterion > for choosing a source code system. But I leave this to ML. > > *Other things being equal*, using a state-of-the-art development system > written in Python to develop Python would be a marketing plus. I think the words "stable" and "reliable" should be in there somewhere :) I don't get the impression bazaar-ng is there yet. Cheers, mwh -- Unfortunately, nigh the whole world is now duped into thinking that silly fill-in forms on web pages is the way to do user interfaces. -- Erik Naggum, comp.lang.lisp From gustavo at niemeyer.net Sun Aug 14 23:21:40 2005 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Sun, 14 Aug 2005 18:21:40 -0300 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <42FF32AA.7040506@v.loewis.de> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> Message-ID: <20050814212140.GA11278@burma.localdomain> > Like Skip, I tried experimenting with it. While that may be the right > model, I don't think it is the right software. In bazaar-ng 0.0.5 (which > is what Debian unstable currently has), bzr commit would not open > a text editor, but require the commit message on the command line; > selective commit of only some of the changed files is also not > supported. bzr diff cannot show the changes between two revisions, The development version has all of those features implemented already. > and cannot show revisions across branches. I'm not sure about this one, though. -- Gustavo Niemeyer http://niemeyer.net From martin at v.loewis.de Sun Aug 14 23:43:54 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 14 Aug 2005 23:43:54 +0200 Subject: [Python-Dev] build problems on macosx (CVS HEAD) In-Reply-To: <39DD497A-89DB-4DF0-A272-62BB66CEC090@mac.com> References: <39DD497A-89DB-4DF0-A272-62BB66CEC090@mac.com> Message-ID: <42FFBB1A.8060206@v.loewis.de> Ronald Oussoren wrote: > I'm trying to build CVS HEAD on OSX 10.4.2 (Xcode 2.1), with a > checkout that is less than two hours old. I'm building a standard > unix tree (no framework install): I just committed what I think is a bugfix for the recent st_gen support. Unfortunately, I can't try the code, since I don't have access to BSD/OSX at the moment. So please report whether there is any change in behaviour. Regards, Martin From martin at v.loewis.de Sun Aug 14 23:47:27 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 14 Aug 2005 23:47:27 +0200 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> <20050814171259.GA8200@mems-exchange.org> Message-ID: <42FFBBEF.5060202@v.loewis.de> Guido van Rossum wrote: > It sounds as if bazaar-NG can use a bit of its own medicine -- I hope > everybody who found a bug in their tools submitted a patch! :-) I had problems finding the place where the bazaar-NG source code repository is stored - is there a public access to the HEAD version? There also doesn't appear to be a bug tracker - but I found a mentioning that bug reports should go to the mailing list. Regards, Martin From martin at v.loewis.de Sun Aug 14 23:58:46 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 14 Aug 2005 23:58:46 +0200 Subject: [Python-Dev] Fwd: PEP: Migrating the Python CVS to Subversion In-Reply-To: References: <1123986783.21455.35.camel@linux.site> Message-ID: <42FFBE96.7040000@v.loewis.de> Guido van Rossum wrote: > Here's another POV. I think I agree with Daniel's view, in particular wrt. to performance. Whatever the replacement tool, it should perform as well or better than CVS currently does; it also shouldn't perform much worse than subversion. I've been using git (or, rather, cogito) to keep up-to-date with the Linux kernel. While performance of git is really good, storage requirements are *quite* high, and initial "checkout" takes a long time - even though the Linux kernel repository stores virtual no history (there was a strict cut when converting the bitkeeper HEAD). So these distributed tools would cause quite some disk consumption on client machines. bazaar-ng apparently supports only-remote repositories as well, so that might be no concern. Regards, Martin From dberlin at dberlin.org Mon Aug 15 00:07:39 2005 From: dberlin at dberlin.org (Daniel Berlin) Date: Sun, 14 Aug 2005 18:07:39 -0400 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <20050814171259.GA8200@mems-exchange.org> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> <20050814171259.GA8200@mems-exchange.org> Message-ID: <1124057259.25267.70.camel@linux.site> On Sun, 2005-08-14 at 11:12 -0600, Neil Schemenauer wrote: > On Sun, Aug 14, 2005 at 06:16:11PM +0200, "Martin v. L?wis" wrote: > > It depends on what "a bit" is. Waiting a month would be fine; waiting > > two years might be pointless. > > It looks like the process of converting a CVS repository to > Bazaar-NG does not yet work well (to be kind). The path > CVS->SVN->bzr would probably work better. I suspect cvs2svn has > been used on quite a few CVS repositories already. I don't think > going to SVN first would lose any information. It doesn't. As a data point, CVS2SVN can handle gcc's massive cvs repository, which has merged rcs file information in it dating back to 1987, >1000 tags, and > 300 branches. Besides monotone's cvs_import, it's actually the only properly designed cvs converter I've seen in a while (Properly designed in that it actually uses the necessary and correct algorithms to get all the weirdities of cvs branches and tags right). I'm not sure how big python's repo is, but you probably want to use the attached patch to speed up cvs2svn. It changes it to reconstruct the revisions on it's own instead of calling cvs or rcs. For GCC, and KDE, this makes a significant difference (17 hours for our 4 gig cvs repo convresion instead of 52 hours), because it was spawning cvs/rcs 50 billion times, and the milliseconds add up :) > My vote is to continue with the migration to SVN. We can > re-evaluate Bazaar-NG at a later time. GCC is moving to SVN (very soon now, within 2 months), and this has been my viewpoint as well. It's much easier to go from something that has changesets and global revisions, to a distributed system, if you want to, than it is to try to reconstruct that info from CVS on your own :). Subversion also has excellent language bindings, including the python bindings. That's how i've hooked it up to gcc's bugzilla. You could easily write something to transform *from* subversion to another system using the bindings. Things like viewcvs use the python bindings to deal with the svn repository entirely. --Dan -------------- next part -------------- Index: cvs2svn =================================================================== --- cvs2svn (revision 1423) +++ cvs2svn (working copy) @@ -166,6 +166,10 @@ # grouping. See design-notes.txt for details. DATAFILE = 'cvs2svn-data' +REVISIONS_DB = 'cvs2svn-cvsrepo.db' + +CHECKOUT_DB = 'cvs2svn-cvsco.db' + # This file contains a marshalled copy of all the statistics that we # gather throughout the various runs of cvs2svn. The data stored as a # marshalled dictionary. @@ -355,40 +359,7 @@ " cvsroot\n" % (error_prefix, cvsroot, fname)) sys.exit(1) -def get_co_pipe(c_rev, extra_arguments=None): - """Return a command string, and the pipe created using that string. - C_REV is a CVSRevision, and EXTRA_ARGUMENTS is used to add extra - arguments. The pipe returns the text of that CVS Revision.""" - ctx = Ctx() - if extra_arguments is None: - extra_arguments = [] - if ctx.use_cvs: - pipe_cmd = [ 'cvs' ] + ctx.cvs_global_arguments + \ - [ 'co', '-r' + c_rev.rev, '-p' ] + extra_arguments + \ - [ ctx.cvs_module + c_rev.cvs_path ]; - else: - pipe_cmd = [ 'co', '-q', '-x,v', '-p' + c_rev.rev ] + extra_arguments + \ - [ c_rev.rcs_path() ] - pipe = SimplePopen(pipe_cmd, True) - pipe.stdin.close() - return pipe_cmd, pipe - -def generate_ignores(c_rev): - # Read in props - pipe_cmd, pipe = get_co_pipe(c_rev) - buf = pipe.stdout.read(PIPE_READ_SIZE) - raw_ignore_val = "" - while buf: - raw_ignore_val = raw_ignore_val + buf - buf = pipe.stdout.read(PIPE_READ_SIZE) - pipe.stdout.close() - error_output = pipe.stderr.read() - exit_status = pipe.wait() - if exit_status: - sys.exit("%s: The command '%s' failed with exit status: %s\n" - "and the following output:\n" - "%s" % (error_prefix, pipe_cmd, exit_status, error_output)) - +def generate_ignores(raw_ignore_val): # Tweak props: First, convert any spaces to newlines... raw_ignore_val = '\n'.join(raw_ignore_val.split()) raw_ignores = raw_ignore_val.split('\n') @@ -614,9 +585,7 @@ DB_OPEN_READ = 'r' DB_OPEN_NEW = 'n' -# A wrapper for anydbm that uses the marshal module to store items as -# strings. -class Database: +class SDatabase: def __init__(self, filename, mode): # pybsddb3 has a bug which prevents it from working with # Berkeley DB 4.2 if you open the db with 'n' ("new"). This @@ -635,22 +604,24 @@ self.db = anydbm.open(filename, mode) - def has_key(self, key): - return self.db.has_key(key) + def __getattr__(self, name): + return getattr(self.db, name) +# A wrapper for anydbm that uses the marshal module to store items as +# strings. +class Database(SDatabase): + def __getitem__(self, key): return marshal.loads(self.db[key]) def __setitem__(self, key, value): self.db[key] = marshal.dumps(value) - def __delitem__(self, key): - del self.db[key] - def get(self, key, default): - if self.has_key(key): - return self.__getitem__(key) - return default + try: + return marshal.loads(self.db[key]) + except KeyError: + return default class StatsKeeper: @@ -841,6 +812,192 @@ Cleanup().register(temp(TAGS_DB), pass8) +def msplit(stri): + re = [ i + "\n" for i in stri.split("\n") ] + re[-1] = re[-1][:-1] + if not re[-1]: + del re[-1] + return re + + +class RCSStream: + ad_command = re.compile('^([ad])(\d+)\\s(\\d+)') + a_command = re.compile('^a(\d+)\\s(\\d+)') + + def __init__(self): + self.texts = [] + + def copy(self): + ret = RCSStream() + ret.texts = self.texts[:] + return ret + + def setText(self, text): + self.texts = msplit(text) + + def getText(self): + return "".join(self.texts) + + def applyDiff(self, diff): + diffs = msplit(diff) + adjust = 0 + i = 0 + while i < len(diffs): + admatch = self.ad_command.match(diffs[i]) + i += 1 + try: + cn = int(admatch.group(3)) + except: + print diffs + raise RuntimeError, 'Error parsing diff commands' + if admatch.group(1) == 'd': # "d" - Delete command + sl = int(admatch.group(2)) - 1 + adjust + del self.texts[sl:sl + cn] + adjust -= cn + else: # "a" - Add command + sl = int(admatch.group(2)) + adjust + self.texts[sl:sl] = diffs[i:i + cn] + adjust += cn + i += cn + + def invertDiff(self, diff): + diffs = msplit(diff) + ndiffs = [] + adjust = 0 + i = 0 + while i < len(diffs): + admatch = self.ad_command.match(diffs[i]) + i += 1 + try: + cn = int(admatch.group(3)) + except: + raise RuntimeError, 'Error parsing diff commands' + if admatch.group(1) == 'd': # "d" - Delete command + sl = int(admatch.group(2)) - 1 + adjust + # handle substitution explicitly, as add must come after del + # (last add may have incomplete line) + if i < len(diffs): + amatch = self.a_command.match(diffs[i]) + else: + amatch = None + if amatch and int(amatch.group(1)) + adjust == sl + cn: + cn2 = int(amatch.group(2)) + i += 1 + ndiffs += ["d%d %d\na%d %d\n" % (sl + 1, cn2, sl + cn2, cn)] + \ + self.texts[sl:sl + cn] + self.texts[sl:sl + cn] = diffs[i:i + cn2] + adjust += cn2 - cn + i += cn2 + else: + ndiffs += ["a%d %d\n" % (sl, cn)] + self.texts[sl:sl + cn] + del self.texts[sl:sl + cn] + adjust -= cn + else: # "a" - Add command + sl = int(admatch.group(2)) + adjust + ndiffs += ["d%d %d\n" % (sl + 1, cn)] + self.texts[sl:sl] = diffs[i:i + cn] + adjust += cn + i += cn + return "".join(ndiffs) + + def zeroDiff(self): + if not self.texts: + return "" + return "a0 " + str(len(self.texts)) + "\n" + "".join(self.texts) + + +class CVSCheckout: + + class Rev: pass + + __shared_state = { } + def __init__(self): + self.__dict__ = self.__shared_state + + def init(self): + self.co_db = SDatabase(temp(CHECKOUT_DB), DB_OPEN_NEW) + Cleanup().register(temp(CHECKOUT_DB), pass8) + self.rev_db = SDatabase(temp(REVISIONS_DB), DB_OPEN_READ) + self.files = { } + + def done(self): + print "leftover revisions:" + for file in self.files: + print file + ':', + for r in self.files[file]: + print r, + print + self.co_db.close() + self.rev_db.close() + + def init_file(self, fname): + revs = { } + for line in self.rev_db[fname].split('\n'): + prv = None + for r in line.split(): + try: + rev = revs[r] + except KeyError: + rev = CVSCheckout.Rev() + rev.ref = 0 + rev.prev = None + revs[r] = rev + if prv: + revs[prv].prev = r + rev.ref += 1 + prv = r + return revs + + def checkout_i(self, fname, revs, r, co, ref): + rev = revs[r] + if rev.prev: + prev = revs[rev.prev] + try: + key = fname + '/' + rev.prev + co.setText(self.co_db[key]) + prev.ref -= 1 + if not prev.ref: +# print "used saved", fname, rev.prev, "- deleted" + del revs[rev.prev] + del self.co_db[key] +# else: +# print "used saved", fname, rev.prev, "- keeping. ref is now", prev.ref + except KeyError: + self.checkout_i(fname, revs, rev.prev, co, 1) + try: + co.applyDiff(self.rev_db[fname + '/' + r]) + except KeyError: + pass + rev.ref -= ref + if rev.ref: +# print "checked out", fname, r, "- saving. ref is", rev.ref + self.co_db[fname + '/' + r] = co.getText() + else: +# print "checked out", fname, r, "- not saving" + del revs[r] + + def checkout_ii(self, fname, revs, r, cvtnl=None): + co = RCSStream() + self.checkout_i(fname, revs, r, co, 0) + rv = co.getText() + if cvtnl: + rv = rv.replace('\r\n', '\n').replace('\r', '\n') + return rv + + def checkout(self, c_rev, cvtnl=None): + try: + revs = self.files[c_rev.fname] + rv = self.checkout_ii(c_rev.fname, revs, c_rev.rev, cvtnl) + if not revs: + del self.files[c_rev.fname] + except KeyError: + revs = self.init_file(c_rev.fname) + rv = self.checkout_ii(c_rev.fname, revs, c_rev.rev, cvtnl) + if revs: + self.files[c_rev.fname] = revs + return rv + + class CVSRevision: def __init__(self, ctx, *args): """Initialize a new CVSRevision with Ctx object CTX, and ARGS. @@ -848,7 +1005,6 @@ If CTX is None, the following members and methods of the instantiated CVSRevision class object will be unavailable (or simply will not work correctly, if at all): - cvs_path svn_path svn_trunk_path is_default_branch_revision() @@ -870,7 +1026,6 @@ prev_rev --> (string or None) previous CVS rev, e.g., "1.2" rev --> (string) this CVS rev, e.g., "1.3" next_rev --> (string or None) next CVS rev, e.g., "1.4" - file_in_attic --> (char or None) true if RCS file is in Attic file_executable --> (char or None) true if RCS file has exec bit set. file_size --> (int) size of the RCS file deltatext_code --> (char) 'N' if non-empty deltatext, else 'E' @@ -883,16 +1038,16 @@ The two forms of initialization are equivalent.""" self._ctx = ctx - if len(args) == 16: + if len(args) == 15: (self.timestamp, self.digest, self.prev_timestamp, self.op, - self.prev_rev, self.rev, self.next_rev, self.file_in_attic, + self.prev_rev, self.rev, self.next_rev, self.file_executable, self.file_size, self.deltatext_code, self.fname, self.mode, self.branch_name, self.tags, self.branches) = args elif len(args) == 1: - data = args[0].split(' ', 14) + data = args[0].split(' ', 13) (self.timestamp, self.digest, self.prev_timestamp, self.op, - self.prev_rev, self.rev, self.next_rev, self.file_in_attic, + self.prev_rev, self.rev, self.next_rev, self.file_executable, self.file_size, self.deltatext_code, self.mode, self.branch_name, numtags, remainder) = data # Patch up data items which are not simple strings @@ -905,8 +1060,6 @@ self.prev_rev = None if self.next_rev == "*": self.next_rev = None - if self.file_in_attic == "*": - self.file_in_attic = None if self.file_executable == "*": self.file_executable = None self.file_size = int(self.file_size) @@ -923,12 +1076,11 @@ self.branches = branches_and_fname[:-1] self.fname = branches_and_fname[-1] else: - raise TypeError, 'CVSRevision() takes 2 or 16 arguments (%d given)' % \ + raise TypeError, 'CVSRevision() takes 2 or 15 arguments (%d given)' % \ (len(args) + 1) - if ctx is not None: - self.cvs_path = relative_name(self._ctx.cvsroot, self.fname[:-2]) - self.svn_path = self._make_path(self.cvs_path, self.branch_name) - self.svn_trunk_path = self._make_path(self.cvs_path) + if ctx is not None: # strictly speaking this check is now superfluous + self.svn_path = self._make_path(self.fname, self.branch_name) + self.svn_trunk_path = self._make_path(self.fname) # The 'primary key' of a CVS Revision is the revision number + the # filename. To provide a unique key (say, for a dict), we just glom @@ -941,10 +1093,10 @@ return revnum + "/" + self.fname def __str__(self): - return ('%08lx %s %s %s %s %s %s %s %s %d %s %s %s %d%s%s %d%s%s %s' % ( + return ('%08lx %s %s %s %s %s %s %s %d %s %s %s %d%s%s %d%s%s %s' % ( self.timestamp, self.digest, self.prev_timestamp or "*", self.op, (self.prev_rev or "*"), self.rev, (self.next_rev or "*"), - (self.file_in_attic or "*"), (self.file_executable or "*"), + (self.file_executable or "*"), self.file_size, self.deltatext_code, (self.mode or "*"), (self.branch_name or "*"), len(self.tags), self.tags and " " or "", " ".join(self.tags), @@ -967,11 +1119,11 @@ return 0 def is_default_branch_revision(self): - """Return 1 if SELF.rev of SELF.cvs_path is a default branch + """Return 1 if SELF.rev of SELF.fname is a default branch revision according to DEFAULT_BRANCHES_DB (see the conditions documented there), else return None.""" - if self._ctx._default_branches_db.has_key(self.cvs_path): - val = self._ctx._default_branches_db[self.cvs_path] + if self._ctx._default_branches_db.has_key(self.fname): + val = self._ctx._default_branches_db[self.fname] val_last_dot = val.rindex(".") our_last_dot = self.rev.rindex(".") default_branch = val[:val_last_dot] @@ -1031,19 +1183,6 @@ else: return self._ctx.trunk_base + '/' + path - def rcs_path(self): - """Returns the actual filesystem path to the RCS file of this - CVSRevision.""" - if self.file_in_attic is None: - return self.fname - else: - basepath, filename = os.path.split(self.fname) - return os.path.join(basepath, 'Attic', filename) - - def filename(self): - "Return the last path component of self.fname, minus the ',v'" - return os.path.split(self.fname)[-1][:-2] - class SymbolDatabase: """This database records information on all symbols in the RCS files. It is created in pass 1 and it is used in pass 2.""" @@ -1177,6 +1316,8 @@ def __init__(self): self.revs = open(temp(DATAFILE + REVS_SUFFIX), 'w') Cleanup().register(temp(DATAFILE + REVS_SUFFIX), pass2) + self.revisions_db = SDatabase(temp(REVISIONS_DB), DB_OPEN_NEW) + Cleanup().register(temp(REVISIONS_DB), pass8) self.resync = open(temp(DATAFILE + RESYNC_SUFFIX), 'w') Cleanup().register(temp(DATAFILE + RESYNC_SUFFIX), pass2) self.default_branches_db = Database(temp(DEFAULT_BRANCHES_DB), DB_OPEN_NEW) @@ -1211,6 +1352,8 @@ if not canonical_name == filename: self.file_in_attic = 1 + self.stream = RCSStream() + file_stat = os.stat(filename) # The size of our file in bytes self.file_size = file_stat[stat.ST_SIZE] @@ -1247,6 +1390,8 @@ # distinguish between an add and a change. self.rev_state = { } + self.empty_1111 = None + # Hash mapping branch numbers, like '1.7.2', to branch names, # like 'Release_1_0_dev'. self.branch_names = { } @@ -1505,6 +1650,10 @@ # finished the for-loop (no resyncing was performed) return + def writeout(self, r, tx): + if tx: + self.revisions_db[self.rel_name + '/' + r] = tx + def set_revision_info(self, revision, log, text): timestamp, author, old_ts = self.rev_data[revision] digest = sha.new(log + '\0' + author).hexdigest() @@ -1552,13 +1701,15 @@ deltatext_code = DELTATEXT_NONEMPTY else: deltatext_code = DELTATEXT_EMPTY + if revision == '1.1.1.1': + self.empty_1111 = 1 c_rev = CVSRevision(Ctx(), timestamp, digest, prev_timestamp, op, self.prev_rev[revision], revision, self.next_rev.get(revision), - self.file_in_attic, self.file_executable, + self.file_executable, self.file_size, - deltatext_code, self.fname, + deltatext_code, self.rel_name, self.mode, self.rev_to_branch_name(revision), self.taglist.get(revision, []), self.branchlist.get(revision, [])) @@ -1568,6 +1719,16 @@ if not self.metadata_db.has_key(digest): self.metadata_db[digest] = (author, log) + if trunk_rev.match(revision): + if revision not in self.next_rev: + self.stream.setText(text) + else: + self.writeout(self.next_rev[revision], self.stream.invertDiff(text)) + if not self.prev_rev[revision]: + self.writeout(revision, self.stream.zeroDiff()) + else: + self.writeout(revision, text) + def parse_completed(self): # Walk through all branches and tags and register them with # their parent branch in the symbol database. @@ -1579,8 +1740,33 @@ self.num_files = self.num_files + 1 + tree = [ ] + for r in self.prev_rev: + if r not in self.next_rev and not (r == "1.1.1.1" and self.empty_1111): + while self.rev_state[r] == 'dead': + pr = self.prev_rev[r] + if not pr: + break + if self.next_rev.get(pr) != r: + break + r = pr + else: + rvs = [ ] + while 1: + rvs.append(r) + pr = self.prev_rev[r] + if not pr: + break + if self.next_rev.get(pr) != r: + rvs.append(pr) + break + r = pr + tree.append(" ".join(rvs)) + self.revisions_db[self.rel_name] = "\n".join(tree) + def write_symbol_db(self): self.symbol_db.write() + self.revisions_db.close() class SymbolingsLogger: """Manage the file that contains lines for symbol openings and @@ -2038,7 +2224,7 @@ if not c_rev.branches: continue cvs_generated_msg = ('file %s was initially added on branch %s.\n' - % (c_rev.filename(), + % (os.path.split(c_rev.fname)[-1], c_rev.branches[0])) author, log_msg = \ Ctx()._persistence_manager.svn_commit_metadata[c_rev.digest] @@ -3389,7 +3575,7 @@ keywords = None if self.mime_mapper: - mime_type = self.mime_mapper.get_type_from_filename(c_rev.cvs_path) + mime_type = self.mime_mapper.get_type_from_filename(c_rev.fname) if not c_rev.mode == 'b': if not self.no_default_eol: @@ -3684,10 +3870,12 @@ if props_len: props_header = 'Prop-content-length: %d\n' % props_len + co = CVSCheckout().checkout(c_rev, s_item.needs_eol_filter) + # treat .cvsignore as a directory property dir_path, basename = os.path.split(c_rev.svn_path) if basename == ".cvsignore": - ignore_vals = generate_ignores(c_rev) + ignore_vals = generate_ignores(co) ignore_contents = '\n'.join(ignore_vals) ignore_contents = ('K 10\nsvn:ignore\nV %d\n%s\n' % \ (len(ignore_contents), ignore_contents)) @@ -3705,73 +3893,24 @@ % (self._utf8_path(dir_path), ignore_len, ignore_len, ignore_contents)) - # If the file has keywords, we must use -kk to prevent CVS/RCS from - # expanding the keywords because they must be unexpanded in the - # repository, or Subversion will get confused. - if s_item.has_keywords: - pipe_cmd, pipe = get_co_pipe(c_rev, [ '-kk' ]) - else: - pipe_cmd, pipe = get_co_pipe(c_rev) + checksum = md5.new() + checksum.update(co) self.dumpfile.write('Node-path: %s\n' 'Node-kind: file\n' 'Node-action: %s\n' '%s' # no property header if no props - 'Text-content-length: ' + 'Text-content-length: %d\n' + 'Text-content-md5: %s\n' + 'Content-length: %d\n' + '\n' % (self._utf8_path(c_rev.svn_path), - action, props_header)) - - pos = self.dumpfile.tell() - - self.dumpfile.write('0000000000000000\n' - 'Text-content-md5: 00000000000000000000000000000000\n' - 'Content-length: 0000000000000000\n' - '\n') - + action, props_header, + len(co), checksum.hexdigest(), + len(co) + props_len)) if prop_contents: self.dumpfile.write(prop_contents) - - # Insert a filter to convert all EOLs to LFs if neccessary - if s_item.needs_eol_filter: - data_reader = LF_EOL_Filter(pipe.stdout) - else: - data_reader = pipe.stdout - - # Insert the rev contents, calculating length and checksum as we go. - checksum = md5.new() - length = 0 - while True: - buf = data_reader.read(PIPE_READ_SIZE) - if buf == '': - break - checksum.update(buf) - length = length + len(buf) - self.dumpfile.write(buf) - - pipe.stdout.close() - error_output = pipe.stderr.read() - exit_status = pipe.wait() - if exit_status: - sys.exit("%s: The command '%s' failed with exit status: %s\n" - "and the following output:\n" - "%s" % (error_prefix, pipe_cmd, exit_status, error_output)) - - # Go back to patch up the length and checksum headers: - self.dumpfile.seek(pos, 0) - # We left 16 zeros for the text length; replace them with the real - # length, padded on the left with spaces: - self.dumpfile.write('%16d' % length) - # 16... + 1 newline + len('Text-content-md5: ') == 35 - self.dumpfile.seek(pos + 35, 0) - self.dumpfile.write(checksum.hexdigest()) - # 35... + 32 bytes of checksum + 1 newline + len('Content-length: ') == 84 - self.dumpfile.seek(pos + 84, 0) - # The content length is the length of property data, text data, - # and any metadata around/inside around them. - self.dumpfile.write('%16d' % (length + props_len)) - # Jump back to the end of the stream - self.dumpfile.seek(0, 2) - + self.dumpfile.write(co) # This record is done (write two newlines -- one to terminate # contents that weren't themselves newline-termination, one to # provide a blank line for readability. @@ -4208,7 +4347,7 @@ warning_prefix) msg = "RESYNC: '%s' (%s): old time='%s' delta=%ds" \ - % (c_rev.cvs_path, c_rev.rev, time.ctime(c_rev.timestamp), + % (c_rev.fname, c_rev.rev, time.ctime(c_rev.timestamp), record[2] - c_rev.timestamp) Log().write(LOG_VERBOSE, msg) @@ -4322,6 +4461,9 @@ Log().write(LOG_QUIET, "Done.") def pass8(): + checkout = CVSCheckout() + checkout.init() + svncounter = 2 # Repository initialization is 1. repos = SVNRepositoryMirror() persistence_manager = PersistenceManager(DB_OPEN_READ) @@ -4346,6 +4488,8 @@ repos.finish() + checkout.done() + _passes = [ pass1, pass2, @@ -4389,7 +4533,6 @@ self.no_default_eol = 0 self.eol_from_mime_type = 0 self.keywords_off = 0 - self.use_cvs = None self.svnadmin = "svnadmin" self.username = None self.print_help = 0 @@ -4492,8 +4635,6 @@ print ' --profile profile with \'hotshot\' (into file cvs2svn.hotshot)' print ' --dry-run do not create a repository or a dumpfile;' print ' just print what would happen.' - print ' --use-cvs use CVS instead of RCS \'co\' to extract data' - print ' (only use this if having problems with RCS)' print ' --svnadmin=PATH path to the svnadmin program' print ' --trunk-only convert only trunk commits, not tags nor branches' print ' --trunk=PATH path for trunk (default: %s)' \ @@ -4538,7 +4679,7 @@ "username=", "existing-svnrepos", "branches=", "tags=", "encoding=", "force-branch=", "force-tag=", "exclude=", - "use-cvs", "mime-types=", + "mime-types=", "eol-from-mime-type", "no-default-eol", "trunk-only", "no-prune", "dry-run", "dump-only", "dumpfile=", "tmpdir=", @@ -4588,8 +4729,6 @@ ctx.dumpfile = value elif opt == '--tmpdir': ctx.tmpdir = value - elif opt == '--use-cvs': - ctx.use_cvs = 1 elif opt == '--svnadmin': ctx.svnadmin = value elif opt == '--trunk-only': @@ -4673,30 +4812,6 @@ "existing directory.\n" % ctx.cvsroot) sys.exit(1) - if ctx.use_cvs: - # Ascend above the specified root if necessary, to find the cvs_repository - # (a directory containing a CVSROOT directory) and the cvs_module (the - # path of the conversion root within the cvs repository) - # NB: cvs_module must be seperated by '/' *not* by os.sep . - ctx.cvs_repository = os.path.abspath(ctx.cvsroot) - prev_cvs_repository = None - ctx.cvs_module = "" - while prev_cvs_repository != ctx.cvs_repository: - if os.path.isdir(os.path.join(ctx.cvs_repository, 'CVSROOT')): - break - prev_cvs_repository = ctx.cvs_repository - ctx.cvs_repository, module_component = os.path.split(ctx.cvs_repository) - ctx.cvs_module = module_component + "/" + ctx.cvs_module - else: - # Hit the root (of the drive, on Windows) without finding a CVSROOT dir. - sys.stderr.write(error_prefix + - ": the path '%s' is not a CVS repository, nor a path " \ - "within a CVS repository. A CVS repository contains " \ - "a CVSROOT directory within its root directory.\n" \ - % ctx.cvsroot) - sys.exit(1) - os.environ['CVSROOT'] = ctx.cvs_repository - if (not ctx.target) and (not ctx.dump_only) and (not ctx.dry_run): sys.stderr.write(error_prefix + ": must pass one of '-s' or '--dump-only'.\n") @@ -4772,28 +4887,6 @@ % ctx.tmpdir) sys.exit(1) - if ctx.use_cvs: - def cvs_ok(): - pipe = SimplePopen([ 'cvs' ] + Ctx().cvs_global_arguments + \ - [ '--version' ], True) - pipe.stdin.close() - pipe.stdout.read() - errmsg = pipe.stderr.read() - status = pipe.wait() - ok = len(errmsg) == 0 and status == 0 - return (ok, status, errmsg) - - ctx.cvs_global_arguments = [ "-q", "-R" ] - ok, cvs_exitstatus, cvs_errmsg = cvs_ok() - if not ok: - ctx.cvs_global_arguments = [ "-q" ] - ok, cvs_exitstatus, cvs_errmsg = cvs_ok() - - if not ok: - sys.stderr.write(error_prefix + - ": error executing CVS: status %s, error output:\n" \ - % (cvs_exitstatus) + cvs_errmsg) - # But do lock the tmpdir, to avoid process clash. try: os.mkdir(os.path.join(ctx.tmpdir, 'cvs2svn.lock')) From dberlin at dberlin.org Mon Aug 15 00:09:04 2005 From: dberlin at dberlin.org (Daniel Berlin) Date: Sun, 14 Aug 2005 18:09:04 -0400 Subject: [Python-Dev] Fwd: PEP: Migrating the Python CVS to Subversion In-Reply-To: References: <1123986783.21455.35.camel@linux.site> Message-ID: <1124057345.25267.72.camel@linux.site> On Sun, 2005-08-14 at 11:13 -0700, Guido van Rossum wrote: > Here's another POV. (Why does evereybody keep emailing me personally?) > Because we love you, and I forgot to cc python-dev. From gustavo at niemeyer.net Mon Aug 15 00:14:39 2005 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Sun, 14 Aug 2005 19:14:39 -0300 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <42FFBBEF.5060202@v.loewis.de> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> <20050814171259.GA8200@mems-exchange.org> <42FFBBEF.5060202@v.loewis.de> Message-ID: <20050814221439.GB11278@burma.localdomain> > I had problems finding the place where the bazaar-NG source code > repository is stored - is there a public access to the HEAD version? You may use rsync: rsync -av --delete bazaar-ng.org::bazaar-ng/bzr/bzr.dev . Or bzr itself: bzr branch http://bazaar-ng.org/bzr/bzr.dev Regards, -- Gustavo Niemeyer http://niemeyer.net From martin at v.loewis.de Mon Aug 15 00:15:30 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 15 Aug 2005 00:15:30 +0200 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <1124057259.25267.70.camel@linux.site> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> <20050814171259.GA8200@mems-exchange.org> <1124057259.25267.70.camel@linux.site> Message-ID: <42FFC282.9060307@v.loewis.de> Daniel Berlin wrote: > I'm not sure how big python's repo is, but you probably want to use the > attached patch to speed up cvs2svn. It changes it to reconstruct the > revisions on it's own instead of calling cvs or rcs. Thanks for the patch, but cvs2svn works fairly well for us as is (in the version that was released with Debian sarge); see http://www.python.org/peps/pep-0347.html for the conversion procedure. On the machine where I originally did the conversion, the script required 7h; on my current machine, it is done in 1:40 or so, which is acceptable. Out of curiosity: do you use the --cvs-revnums parameter? Should we? Regards, Martin From dberlin at dberlin.org Mon Aug 15 00:25:02 2005 From: dberlin at dberlin.org (Daniel Berlin) Date: Sun, 14 Aug 2005 18:25:02 -0400 Subject: [Python-Dev] Fwd: PEP: Migrating the Python CVS to Subversion In-Reply-To: <42FFBE96.7040000@v.loewis.de> References: <1123986783.21455.35.camel@linux.site> <42FFBE96.7040000@v.loewis.de> Message-ID: <1124058303.25267.88.camel@linux.site> On Sun, 2005-08-14 at 23:58 +0200, "Martin v. L?wis" wrote: > Guido van Rossum wrote: > > Here's another POV. > > I think I agree with Daniel's view, in particular wrt. to performance. > Whatever the replacement tool, it should perform as well or better > than CVS currently does; it also shouldn't perform much worse than > subversion. Then, in fairness, I should note that annotate is slower on subversion (and monotone, and anything using binary deltas) than CVS. This is because you can't generate line-diffs that annotate wants from binary copy + add diffs. You have to reconstruct the actual revisions and then line diff them. Thus, CVS is O(N) here, and SVN and other binary delta users are O(N^2). You wouldn't really notice the speed difference when you are annotating a file with 100 revisions. You would if you annotate the 800k changelog which has 30k trunk revisions. CVS takes 4 seconds, svn takes ~5 minutes, the whole time being spent in doing diffs of those revisions. I rewrote the blame algorithm recently so that it will only take about 2 minutes on changelog, but it cheats because it knows it can stop early because it's blamed all the revisions (since our changelog rotates). For those curious, you also can't directly generate "always-correct" byte-level differences from the diffs, since their goal is to find the most space efficient way to transform rev old into rev new, *not* record actual byte-level changes that occurred between old and new. It may turn out that doing an add of 2 bytes is cheaper than specifying the opcode for copy(start,len). Actual diffs are produced by reproducing the texts and line diffing them. Such is the cost of efficient storage :). > > I've been using git (or, rather, cogito) to keep up-to-date with the > Linux kernel. While performance of git is really good, storage > requirements are *quite* high, and initial "checkout" takes a long > time - even though the Linux kernel repository stores virtual no > history (there was a strict cut when converting the bitkeeper HEAD). > So these distributed tools would cause quite some disk consumption > on client machines. bazaar-ng apparently supports only-remote > repositories as well, so that might be no concern. The argument "network and disk is cheap" doesn't work for us when you are talking 5-10 gigabytes of initial transfer :). However, I doubt it's more than a hundred meg or so for python, if that. You may run into these problems in 10 years :) From mwh at python.net Mon Aug 15 00:31:33 2005 From: mwh at python.net (Michael Hudson) Date: Sun, 14 Aug 2005 23:31:33 +0100 Subject: [Python-Dev] build problems on macosx (CVS HEAD) In-Reply-To: <42FFBB1A.8060206@v.loewis.de> ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Sun, 14 Aug 2005 23:43:54 +0200") References: <39DD497A-89DB-4DF0-A272-62BB66CEC090@mac.com> <42FFBB1A.8060206@v.loewis.de> Message-ID: <2mfytcup4q.fsf@starship.python.net> "Martin v. L?wis" writes: > Ronald Oussoren wrote: >> I'm trying to build CVS HEAD on OSX 10.4.2 (Xcode 2.1), with a >> checkout that is less than two hours old. I'm building a standard >> unix tree (no framework install): > > I just committed what I think is a bugfix for the recent st_gen support. > Unfortunately, I can't try the code, since I don't have access to > BSD/OSX at the moment. > > So please report whether there is any change in behaviour. Seems to have done the trick, thanks. Cheers, mwh -- I just had a very odd phone call from a researcher with the french TV station "TF1" asking about inflatable football referees -- from Twisted.Quotes From dberlin at dberlin.org Mon Aug 15 00:32:31 2005 From: dberlin at dberlin.org (Daniel Berlin) Date: Sun, 14 Aug 2005 18:32:31 -0400 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <42FFC282.9060307@v.loewis.de> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> <20050814171259.GA8200@mems-exchange.org> <1124057259.25267.70.camel@linux.site> <42FFC282.9060307@v.loewis.de> Message-ID: <1124058751.25267.94.camel@linux.site> On Mon, 2005-08-15 at 00:15 +0200, "Martin v. L?wis" wrote: > Daniel Berlin wrote: > > I'm not sure how big python's repo is, but you probably want to use the > > attached patch to speed up cvs2svn. It changes it to reconstruct the > > revisions on it's own instead of calling cvs or rcs. > > Thanks for the patch, but cvs2svn works fairly well for us as is (in > the version that was released with Debian sarge); see > > http://www.python.org/peps/pep-0347.html > > for the conversion procedure. On the machine where I originally did > the conversion, the script required 7h; on my current machine, it is > done in 1:40 or so, which is acceptable. > > Out of curiosity: do you use the --cvs-revnums parameter? Should we? No. In our case, it doesn't buy us anything. In the name of continuity, we have to make the old cvsweb urls work with new viewcvs urls anyway (they appear in bug reports, etc). We also don't want to destroy the ability for people to diff existing cvs working copies. I may have been able to hack something around with cvs-revnums, but not easily. Thus, we are just going to keep a readonly version of the repo around, and a readonly cvsweb, with a warning at the top of the page that the current source is stored in subversion. > > Regards, > Martin From martin at v.loewis.de Mon Aug 15 00:33:04 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 15 Aug 2005 00:33:04 +0200 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <20050814221439.GB11278@burma.localdomain> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> <20050814171259.GA8200@mems-exchange.org> <42FFBBEF.5060202@v.loewis.de> <20050814221439.GB11278@burma.localdomain> Message-ID: <42FFC6A0.9080509@v.loewis.de> Gustavo Niemeyer wrote: > You may use rsync: > > rsync -av --delete bazaar-ng.org::bazaar-ng/bzr/bzr.dev . > > Or bzr itself: > > bzr branch http://bazaar-ng.org/bzr/bzr.dev Ah, thanks. Fetching it with rsync is so much faster than fetching it with bzr, though... Regards, Martin From martin at v.loewis.de Mon Aug 15 00:37:31 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 15 Aug 2005 00:37:31 +0200 Subject: [Python-Dev] Fwd: PEP: Migrating the Python CVS to Subversion In-Reply-To: <1124058303.25267.88.camel@linux.site> References: <1123986783.21455.35.camel@linux.site> <42FFBE96.7040000@v.loewis.de> <1124058303.25267.88.camel@linux.site> Message-ID: <42FFC7AB.6000201@v.loewis.de> Daniel Berlin wrote: > The argument "network and disk is cheap" doesn't work for us when you > are talking 5-10 gigabytes of initial transfer :). However, I doubt > it's more than a hundred meg or so for python, if that. > > You may run into these problems in 10 years :) I don't know how bazaar-ng would perform - but the converted fsfs svn repository is 718MiB. Of course, in 10 years, 5-10GiB of network transfer will be cheap :-) Regards, Martin From lalo at exoweb.net Mon Aug 15 03:58:58 2005 From: lalo at exoweb.net (Lalo Martins) Date: Mon, 15 Aug 2005 09:58:58 +0800 Subject: [Python-Dev] cvs to bzr? In-Reply-To: <17150.31637.180169.877441@montanaro.dyndns.org> References: <17150.31637.180169.877441@montanaro.dyndns.org> Message-ID: And so says skip at pobox.com on 14/08/05 07:00... > Based on the message Guido forwarded, I installed bazaar-ng. From Mark's > note it seems they convert cvs repositories to bzr repositories, but I > didn't see any mention in the bzr docs of any sort of cvs2bzr tool. > Likewise, Google didn't turn up anything obvious. Anyone know of something? Just for the sake of fairness - Mark's email states that they convert cvs repositories to baz (Bazaar 1.x), not to bzr (Bazaar-NG, soon-to-be Bazaar 2.x). The tools to convert to bzr are not yet mature, as bzr itself just recently started to solidify. (The pace of development is one of my favorite "features" about bzr; it's a testament to python and to bzr itself.) You can, however, convert from CVS to baz (arch), and from there to bzr. best, Lalo Martins -- So many of our dreams at first seem impossible, then they seem improbable, and then, when we summon the will, they soon become inevitable. -- http://www.exoweb.net/ mailto:lalo at exoweb.net GNU: never give up freedom http://www.gnu.org/ From skip at pobox.com Mon Aug 15 04:39:41 2005 From: skip at pobox.com (skip@pobox.com) Date: Sun, 14 Aug 2005 21:39:41 -0500 Subject: [Python-Dev] cvs to bzr? In-Reply-To: References: <17150.31637.180169.877441@montanaro.dyndns.org> Message-ID: <17152.109.883835.190683@montanaro.dyndns.org> Lalo> You can, however, convert from CVS to baz (arch), and from there Lalo> to bzr. Would this be with cscvs? According to the cscvs wiki page at http://wiki.gnuarch.org/cscvs cscvs is current unmaintained and can't handle repositories with branches. In addition, it appears that to do a one-time convertsion from cvs to bzr I will need to also install arch and baz as well as any other packages they depend on. Skip From greg at electricrain.com Mon Aug 15 05:34:49 2005 From: greg at electricrain.com (Gregory P. Smith) Date: Sun, 14 Aug 2005 20:34:49 -0700 Subject: [Python-Dev] request for code review - hashlib - patch #1121611 Message-ID: <20050815033449.GW16043@zot.electricrain.com> https://sourceforge.net/tracker/index.php?func=detail&aid=1121611&group_id=5470&atid=305470 This is the hashlib module that speeds up python's md5 and sha1 support by using openssl (when available) as well as adding sha224/256 + sha384/512 support (plus anything openssl provides). I believe it is complete and ready to commit (hashlib-009.patch), any objections? compiled docs in html are here for easy perusal: http://electricrain.com/greg/hashlib-py25-doc/ thanks, greg From t-meyer at ihug.co.nz Mon Aug 15 05:46:19 2005 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Mon, 15 Aug 2005 15:46:19 +1200 Subject: [Python-Dev] python-dev Summary for 2005-07-16 through 2005-07-31 [draft] Message-ID: Here's July Part Two. As usual, if anyone can spare the time to proofread this (it's fairly short this fortnight!), that would be great! Please send any corrections or suggestions to Tim (tlesher at gmail.com), Steve (steven.bethard at gmail.com) and/or me, rather than cluttering the list. Ta! ============= Announcements ============= ------------------------------------------------- PyPy Sprint in Heidelberg 22nd - 29th August 2005 ------------------------------------------------- Heidelberg University in Germany will host a PyPy_ sprint from 22nd August to 29th August. The sprint will push towards the 0.7 release of PyPy_ which hopes to reach Python 2.4.1 compliancy and to have full, direct translation into a low level language, instead of reinterpretation through CPython. If you'd like to help out, this is a great place to start! For more information, see PyPy's `Heidelberg sprint`_ page. .. _PyPy: http://codespeak.net/pypy .. _Heidelberg sprint: http://codespeak.net/pypy/index.cgi?extradoc/sprintinfo/Heidelberg-sprint.ht ml Contributing thread: - `Next PyPy sprint: Heidelberg (Germany), 22nd-29th of August `__ -------------------------------- zlib 1.2.3 in Python 2.4 and 2.5 -------------------------------- Trent Mick supplied a patch for updating Python from zlib 1.2.1 to zlib 1.2.3, which eliminates some potential security vulnerabilities. Python will move to this new version of zlib in both the maintenance 2.4 branch and the main (2.5) branch. Contributing thread: - `zlib 1.2.3 is just out `__ ========= Summaries ========= ------------------------------- Moving Python CVS to Subversion ------------------------------- Martin v. L?wis submitted `PEP 347`_, covering changing from CVS to SVN for source code revision control of the Python repository, and moving from hosting the repository on sourceforge.net to python.org. Moving to SVN from CVS met with general favour from most people, although most were undecided about moving from sourceforge.net to python.org. The additional administration requirements of the move were the primary concern, and moving to an alternative host was suggested. Martin is open to including suggestions for alternative hosts in the PEP, but is not interested in carrying out such research himself; as such, if alternative hosts are to be included, someone needs to volunteer to collect all the required information and submit it to Martin. Discussion about the conversion and the move is continuing in August. .. _PEP 347: http://www.python.org/peps/pep-0347.html Contributing thread: - `PEP: Migrating the Python CVS to Subversion `__ --------------------------------- Exception Hierarchy in Python 3.0 --------------------------------- Brett Cannon posted the first draft of `PEP 348`_, covering reorganisation of exceptions in Python 3.0. The initial draft included major changes to the hierarchy, requiring any object raised to inherit from a certain superclass, and changing bare 'except' clauses to catch a specific superclass. The latter two proposals didn't generate much comment (although Guido vacillated between removing bare 'except' clauses and not), but the proposed hierarchy organisation and renaming was hotly discussed. Nick Coghlan countered each revision of Brett's maximum-changes PEP with a minimum-changes PEP, each evolving through python-dev discussion, and gradually moving to an acceptable middle ground. At present, it seems that the changes will be much more minor than the original proposal. The thread branched off into comments about `Python 3.0`_ changes in general. The consensus was generally that although backwards compatibility isn't required in Python 3.0, it should only be broken when there is a clear reason for it, and that, as much as possible, Python 3.0 should be Python 2.9 without a lot of backwards compatibility code. A number of people indicated that they were reasonably content with the existing exception hierarchy, and didn't feel that major changes were required. Guido suggested that a good principle for determining the ideal exception hierarchy is whether there's a use case for catching the common base class. Marc-Andre Lemburg pointed out that when migrating code changes in Exception names are reasonably easy to automate, but changes in the inheritance tree are much more difficult. Many exceptions were discussed at length (e.g. WindowsError, RuntimeError), with debate about whether they should continue to exist in Python 3.0, be renamed, or be removed. The PEP contains the current status for each of these exceptions. The PEP evolution and discussion are still continuing in August, and since this is for Python 3.0, are likely to be considered open for some time yet. .. _Python 3.0: http://www.python.org/peps/pep-3000.html .. _PEP 348: http://www.python.org/peps/pep-0348.html Contributing thread: - `Pre-PEP: Exception Reorganization for Python 3.0 `__ ----------------------------------------- Docstrings and the Official Documentation ----------------------------------------- A new `bug report`_ pointed out that the docstring help for cgi.escape was not as detailed as that in the full documentation, prompting Skip Montanaro to ask whether this should be the case or not. Several reasons were outlined why docstrings should be more of a "quick reference card" than a "textbook" (i.e. maintain the status quo). Tim Peters suggested that tools to extract text from the full documentation would be a more sensible method of making the "textbook" available from help ()/pydoc; if anyone is interested, then this would probably be the best way to start implementing this. .. _bug report: http://python.org/sf/1243553 Contributing thread: - `should doc string content == documentation content? `__ --------------------------- Syntax suggestion: "while:" --------------------------- Martin Blais suggested "while:" as a syntactic shortcut for "while True:". The suggestion was shot down pretty quickly; not only is "while:" less explicit than "while True:", but it introduces readability problems for the apparently large number of people who, when reading "while:", immediately think "while what?" Contributing thread: - `while: `__ ------------------ Sets in Python 2.5 ------------------ In Python 2.4, there is no C API for the built-in set type; you must use PyObject_Call(), etc. as you would in accessing other Python objects. However, in Python 2.5, Raymond Hettinger plans to introduce a C API along with a new implementation of the set type that uses its own data structure instead of forwarding everything to dicts. Contributing thread: - `C api for built-in type set? `__ =============== Skipped Threads =============== - `Some RFE for review `__ - `python/dist/src/Doc/lib emailutil.tex,1.11,1.12 `__ - `read only files `__ - `builtin filter function `__ - `Weekly Python Patch/Bug Summary `__ - `Information request; Keywords: compiler compiler, EBNF, python, ISO 14977 `__ - `installation of python on a Zaurus `__ - `python-dev summary for 2005-07-01 to 2005-07-15 [draft] `__ - `math.fabs redundant? `__ ================================================= Skipped Threads (covered in the previous summary) ================================================= - `'With' context documentation draft (was Re: Terminology for PEP 343 `__ - `Adding the 'path' module (was Re: Some RFE for review) `__ - `[C++-sig] GCC version compatibility `__ From bcannon at gmail.com Mon Aug 15 06:35:02 2005 From: bcannon at gmail.com (Brett Cannon) Date: Sun, 14 Aug 2005 21:35:02 -0700 Subject: [Python-Dev] PEP 348 (exception reorg) revised again Message-ID: I am sure people mainly care about the big changes inroduced by revision 1.8 of the PEP (http://www.python.org/peps/pep-0348.html). So, first is that WindowsError is staying. Enough people want it to stay and have a legitimate use that I removed the proposal to ditch it. Second, I changed the bare 'except' proposal again to recommend its removal. I had been feeling they should just go for about a week, but I solidified my thinking when I was talking with Alex and Anna Martelli and managed to convince them bare 'except's should go after Alex initially thought they should be changed to be ``except Exception``. This obviously goes against what Guido last said he wanted, but I hope I can convince him to get rid of bare 'except's. Minor stuff is fleshing out the arguments for TerminatingException (I am sure Raymond loves that I am leaving this part in =) and adding a Roadmap for the transition. -Brett From miha at mpi-magdeburg.mpg.de Mon Aug 15 08:22:10 2005 From: miha at mpi-magdeburg.mpg.de (Michael Krasnyk) Date: Mon, 15 Aug 2005 08:22:10 +0200 Subject: [Python-Dev] SWIG and rlcompleter Message-ID: <43003492.8060904@mpi-magdeburg.mpg.de> Hello all, Recently I've found that rlcompleter does not work correctly with SWIG generated classes. In some cases dir(object) containes not only strings, but also type of the object, smth like . And condition "word[:n] == attr" throws an exception. Is it possible to solve this problem with following path? --- cut --- --- rlcompleter.py.org 2005-08-14 13:02:02.000000000 +0200 +++ rlcompleter.py 2005-08-14 13:18:59.000000000 +0200 @@ -136,8 +136,11 @@ matches = [] n = len(attr) for word in words: - if word[:n] == attr and word != "__builtins__": - matches.append("%s.%s" % (expr, word)) + try: + if word[:n] == attr and word != "__builtins__": + matches.append("%s.%s" % (expr, word)) + except: + pass return matches def get_class_members(klass): --- cut --- Thanks in advance, Michael Krasnyk From stephen.thorne at gmail.com Mon Aug 15 08:40:22 2005 From: stephen.thorne at gmail.com (Stephen Thorne) Date: Mon, 15 Aug 2005 16:40:22 +1000 Subject: [Python-Dev] string_join overrides TypeError exception thrown in generator Message-ID: <3e8ca5c8050814234020735adf@mail.gmail.com> Hi, An interesting problem was pointed out to me, which I have distilled to this testcase: def gen(): raise TypeError, "I am a TypeError" yield 1 def one(): return ''.join( x for x in gen() ) def two(): return ''.join([x for x in gen()]) for x in one, two: try: x() except TypeError, e: print e Expected output is: """ I am a TypeError I am a TypeError """ Actual output is: """ sequence expected, generator found I am a TypeError """ Upon looking at the implementation of 'string_join' in stringobject.c[1], It's quite obvious what's gone wrong, an exception has been triggered in PySequence_Fast, and string_join overrides that exception, assuming that the only TypeErrors thrown by PySequence_Fast are caused by 'orig' being a value that was an invalid sequence type, ignoring the possibility that a TypeError could be thrown by exhausting a generator. seq = PySequence_Fast(orig, ""); if (seq == NULL) { if (PyErr_ExceptionMatches(PyExc_TypeError)) PyErr_Format(PyExc_TypeError, "sequence expected, %.80s found", orig->ob_type->tp_name); return NULL; } I can't see an obvious solution, but perhaps generators should get special treatment regardless. Reading over this code it looks like the generator is exhausted all at once, instead of incrementally.. -- Stephen Thorne Development Engineer [1] http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Objects/stringobject.c?rev=2.231&view=markup From benji at benjiyork.com Mon Aug 15 13:30:36 2005 From: benji at benjiyork.com (Benji York) Date: Mon, 15 Aug 2005 07:30:36 -0400 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <42FF6E4B.4000206@v.loewis.de> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> Message-ID: <43007CDC.6060200@benjiyork.com> Martin v. L?wis wrote: > skip at pobox.com wrote: >>Granted. What is the cost of waiting a bit longer to see if it (or >>something else) gets more usable and would hit the mark better than svn? > > It depends on what "a bit" is. Waiting a month would be fine; waiting > two years might be pointless. This might be too convoluted to consider, but I thought I might throw it out there. We use svn for our repositories, but I've taken to also using bzr so I can do local commits and reversions (within a particular svn reversion). I can imagine expanding that usage to sharing branches and such via bzr (or mercurial, which looks great), but keeping the trunk in svn. -- Benji York From ncoghlan at gmail.com Mon Aug 15 14:28:09 2005 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Aug 2005 22:28:09 +1000 Subject: [Python-Dev] string_join overrides TypeError exception thrown in generator In-Reply-To: <3e8ca5c8050814234020735adf@mail.gmail.com> References: <3e8ca5c8050814234020735adf@mail.gmail.com> Message-ID: <43008A59.2070306@gmail.com> Stephen Thorne wrote: > I can't see an obvious solution, but perhaps generators should get > special treatment regardless. Reading over this code it looks like the > generator is exhausted all at once, instead of incrementally.. Indeed - str.join uses a multipass approach to build the final string, so it needs to ensure it has a reiterable to play with. PySequence_Fast achieves that, at the cost of dumping a generator into a sequence rather than building a string from it directly. Unicode.join uses PySequence_Fast too, and has the same problem with masking the TypeError from the generator. The calling code simply can't tell if the NULL return was set directly by PySequence_Fast, or was relayed by PySequence_List (which got it from _PyList_Extend, which got it from listextend, which got it from iternext, etc). This is the kind of problem that PEP 344 is designed to solve :) This also shows that argument validation is one of the cases where using an iterable instead of a generator is a good thing, since errors get raised where the generator is created, instead of where it is first used: class gen(object): def __init__(self): raise TypeError, "I am a TypeError" def __iter__(self): yield 1 def one(): return ''.join( x for x in gen() ) def two(): return ''.join([x for x in gen()]) for x in one, two: try: x() except TypeError, e: print e Hmm, makes me think of a neat little decorator: def step_on_creation(gen): def start_gen(*args, **kwds): g = gen(*args, **kwds) g.next() return g start_gen.__name__ = gen.__name__ start_gen.__doc__ = gen.__doc__ start_gen.__dict__ = gen.__dict__ return start_gen @step_on_creation def gen(): # Setup executed at creation time raise TypeError, "I am a TypeError" yield None # The actual iteration steps yield 1 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com From raymond.hettinger at verizon.net Mon Aug 15 15:16:47 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Mon, 15 Aug 2005 09:16:47 -0400 Subject: [Python-Dev] PEP 348 (exception reorg) revised again In-Reply-To: Message-ID: <001601c5a19b$9b2abf80$af26c797@oemcomputer> [Brett] > This obviously goes against what Guido last said he > wanted, but I hope I can convince him to get rid of bare 'except's. -1 on eliminating bare excepts. This unnecessarily breaks tons of code without offering ANY compensating benefits. There are valid use cases for this construct. It is completely Pythonic to have bare keywords apply a useful default as an aid to readability and ease of coding. +1 on the new BaseException +1 on moving NotImplementedError, SystemExit, and KeyboardInterrupt. -1 on replacing "except (KeyboardInterrupt, SystemExit)" with "except TerminatingException". 1) Grepping existing code bases shows that these two are almost never caught together so it is a bit silly to introduce a second way to do it. 2) Efforts to keep the builtin namespace compact argue against adding a new builtin that will almost never be used. 3) The change unnecessarily sacrifices flatness, making the language more difficult to learn. 4) The "self-documenting" rationale is weak -- if needed, a two-word comment would suffice. Existing code almost never has had to comment on catching multiple exceptions -- the exception tuple itself has been sufficiently obvious and explicit. This rationale assumes that code readers aren't smart enough to infer that SystemExit has something to do with termination. Raymond From phd at mail2.phd.pp.ru Mon Aug 15 15:33:33 2005 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Mon, 15 Aug 2005 17:33:33 +0400 Subject: [Python-Dev] PEP 348 (exception reorg) revised again In-Reply-To: <001601c5a19b$9b2abf80$af26c797@oemcomputer> References: <001601c5a19b$9b2abf80$af26c797@oemcomputer> Message-ID: <20050815133333.GA17966@phd.pp.ru> On Mon, Aug 15, 2005 at 09:16:47AM -0400, Raymond Hettinger wrote: > It is completely Pythonic to have bare keywords > apply a useful default as an aid to readability and ease of coding. Bare "while:" was rejected because of "while WHAT?!". Bare "except:" does not cause "except WHAT?!" reaction. Isn't it funny?! (-: Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From raymond.hettinger at verizon.net Mon Aug 15 15:47:54 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Mon, 15 Aug 2005 09:47:54 -0400 Subject: [Python-Dev] PEP 348 (exception reorg) revised again In-Reply-To: <20050815133333.GA17966@phd.pp.ru> Message-ID: <001801c5a19f$f43ae6a0$af26c797@oemcomputer> > > It is completely Pythonic to have bare keywords > > apply a useful default as an aid to readability and ease of coding. [Oleg] > Bare "while:" was rejected because of "while WHAT?!". Bare "except:" > does not cause "except WHAT?!" reaction. Isn't it funny?! (-: It's both funny and interesting. It raises the question of what makes the two different -- why is one instantly recognizable and why does the other trigger a gag reflex. My thought is that bare excepts occur in a context that makes their meaning clear: try: block() except SpecificException: se_handler() except: handle_everything_else() The pattern of use is similar to a "default" in a switch-case construct. Viewed out-of-context, one would ask "default WHAT". Viewed after a series of case statements, the meaning is vividly clear. Raymond From tdickenson at devmail.geminidataloggers.co.uk Mon Aug 15 16:48:11 2005 From: tdickenson at devmail.geminidataloggers.co.uk (Toby Dickenson) Date: Mon, 15 Aug 2005 15:48:11 +0100 Subject: [Python-Dev] PEP 348 (exception reorg) revised again In-Reply-To: <001601c5a19b$9b2abf80$af26c797@oemcomputer> References: <001601c5a19b$9b2abf80$af26c797@oemcomputer> Message-ID: <200508151548.11552.tdickenson@devmail.geminidataloggers.co.uk> On Monday 15 August 2005 14:16, Raymond Hettinger wrote: > -1 on replacing "except (KeyboardInterrupt, SystemExit)" with "except > TerminatingException". The rationale for including TerminatingException in the PEP would also be satisfied by having a TerminatingExceptions tuple (in the exceptions module?). It makes sense to express the classification of exceptions that are intended to terminate the interpreter, but we dont need to express that classification as inheritence. -- Toby Dickenson From nick.bastin at gmail.com Mon Aug 15 18:27:36 2005 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Mon, 15 Aug 2005 12:27:36 -0400 Subject: [Python-Dev] PEP: Migrating the Python CVS to Subversion In-Reply-To: <42F6F61B.1080505@v.loewis.de> References: <42E93940.6080708@v.loewis.de> <1122607673.9665.38.camel@geddy.wooz.org> <87fytu2lly.fsf@tleepslib.sk.tsukuba.ac.jp> <1122918723.9680.33.camel@warna.corp.google.com> <42EF2794.1000209@v.loewis.de> <66d0a6e105080312181e25fa08@mail.gmail.com> <42F1AADE.50908@v.loewis.de> <66d0a6e105080718527939aa81@mail.gmail.com> <42F6F61B.1080505@v.loewis.de> Message-ID: <66d0a6e1050815092760a2dab3@mail.gmail.com> On 8/8/05, "Martin v. L?wis" wrote: > Nicholas Bastin wrote: > > It's a mature product. I would hope that that would count for > > something. > > Sure. But so is subversion. I will then assume that you and I have different ideas of what 'mature' means. > So I should then remove your offer to host a perforce installation, > as you never made such an offer, right? Correct. . > Yes. That's what this PEP is for. So I guess you are -1 on the > PEP. Not completely. More like -0 at the moment. We need a better system, but I think we shouldn't just pick a system because it's the one the PEP writer preferred - there should be some sort of effort to test a few systems (including bug trackers). I know this is work, but this isn't just something we can change easily again later. -- Nick From ronaldoussoren at mac.com Mon Aug 15 09:15:59 2005 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 15 Aug 2005 09:15:59 +0200 Subject: [Python-Dev] build problems on macosx (CVS HEAD) In-Reply-To: <42FFBB1A.8060206@v.loewis.de> References: <39DD497A-89DB-4DF0-A272-62BB66CEC090@mac.com> <42FFBB1A.8060206@v.loewis.de> Message-ID: <1A6CE290-CD7F-4AEC-B649-39CB4ED57E8B@mac.com> On 14-aug-2005, at 23:43, Martin v. L?wis wrote: > Ronald Oussoren wrote: > >> I'm trying to build CVS HEAD on OSX 10.4.2 (Xcode 2.1), with a >> checkout that is less than two hours old. I'm building a standard >> unix tree (no framework install): >> > > I just committed what I think is a bugfix for the recent st_gen > support. > Unfortunately, I can't try the code, since I don't have access to > BSD/OSX at the moment. > > So please report whether there is any change in behaviour. Your change has fixed this issue. Thanks for the quick response, Ronald > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ > ronaldoussoren%40mac.com > From gvanrossum at gmail.com Mon Aug 15 19:08:02 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon, 15 Aug 2005 10:08:02 -0700 Subject: [Python-Dev] PEP 348 (exception reorg) revised again In-Reply-To: <001601c5a19b$9b2abf80$af26c797@oemcomputer> References: <001601c5a19b$9b2abf80$af26c797@oemcomputer> Message-ID: I'm with Raymond here. On 8/15/05, Raymond Hettinger wrote: > [Brett] > > This obviously goes against what Guido last said he > > wanted, but I hope I can convince him to get rid of bare 'except's. > > -1 on eliminating bare excepts. This unnecessarily breaks tons of code > without offering ANY compensating benefits. There are valid use cases > for this construct. It is completely Pythonic to have bare keywords > apply a useful default as an aid to readability and ease of coding. > > +1 on the new BaseException > > +1 on moving NotImplementedError, SystemExit, and KeyboardInterrupt. > > -1 on replacing "except (KeyboardInterrupt, SystemExit)" with "except > TerminatingException". 1) Grepping existing code bases shows that these > two are almost never caught together so it is a bit silly to introduce a > second way to do it. 2) Efforts to keep the builtin namespace compact > argue against adding a new builtin that will almost never be used. 3) > The change unnecessarily sacrifices flatness, making the language more > difficult to learn. 4) The "self-documenting" rationale is weak -- if > needed, a two-word comment would suffice. Existing code almost never > has had to comment on catching multiple exceptions -- the exception > tuple itself has been sufficiently obvious and explicit. This rationale > assumes that code readers aren't smart enough to infer that SystemExit > has something to do with termination. > > > > Raymond > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvanrossum at gmail.com Mon Aug 15 19:11:24 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon, 15 Aug 2005 10:11:24 -0700 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <43003492.8060904@mpi-magdeburg.mpg.de> References: <43003492.8060904@mpi-magdeburg.mpg.de> Message-ID: (1) Please use the SF patch manager. (2) Please don't propose adding more bare "except:" clauses to the standard library. (3) I think a better patch is to use str(word)[:n] instead of word[:n]. On 8/14/05, Michael Krasnyk wrote: > Hello all, > > Recently I've found that rlcompleter does not work correctly with SWIG > generated classes. > In some cases dir(object) containes not only strings, but also type of > the object, smth like . > And condition "word[:n] == attr" throws an exception. > Is it possible to solve this problem with following path? > > --- cut --- > --- rlcompleter.py.org 2005-08-14 13:02:02.000000000 +0200 > +++ rlcompleter.py 2005-08-14 13:18:59.000000000 +0200 > @@ -136,8 +136,11 @@ > matches = [] > n = len(attr) > for word in words: > - if word[:n] == attr and word != "__builtins__": > - matches.append("%s.%s" % (expr, word)) > + try: > + if word[:n] == attr and word != "__builtins__": > + matches.append("%s.%s" % (expr, word)) > + except: > + pass > return matches > > def get_class_members(klass): > --- cut --- > > Thanks in advance, > Michael Krasnyk > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bcannon at gmail.com Mon Aug 15 19:36:46 2005 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 15 Aug 2005 10:36:46 -0700 Subject: [Python-Dev] PEP 348 (exception reorg) revised again In-Reply-To: References: <001601c5a19b$9b2abf80$af26c797@oemcomputer> Message-ID: OK, I will take this as BDFL pronouncement that ditching bare 'except's is just not going to happen. Had to try. =) And I will strip out the TerminatingException proposal. -Brett On 8/15/05, Guido van Rossum wrote: > I'm with Raymond here. > > On 8/15/05, Raymond Hettinger wrote: > > [Brett] > > > This obviously goes against what Guido last said he > > > wanted, but I hope I can convince him to get rid of bare 'except's. > > > > -1 on eliminating bare excepts. This unnecessarily breaks tons of code > > without offering ANY compensating benefits. There are valid use cases > > for this construct. It is completely Pythonic to have bare keywords > > apply a useful default as an aid to readability and ease of coding. > > > > +1 on the new BaseException > > > > +1 on moving NotImplementedError, SystemExit, and KeyboardInterrupt. > > > > -1 on replacing "except (KeyboardInterrupt, SystemExit)" with "except > > TerminatingException". 1) Grepping existing code bases shows that these > > two are almost never caught together so it is a bit silly to introduce a > > second way to do it. 2) Efforts to keep the builtin namespace compact > > argue against adding a new builtin that will almost never be used. 3) > > The change unnecessarily sacrifices flatness, making the language more > > difficult to learn. 4) The "self-documenting" rationale is weak -- if > > needed, a two-word comment would suffice. Existing code almost never > > has had to comment on catching multiple exceptions -- the exception > > tuple itself has been sufficiently obvious and explicit. This rationale > > assumes that code readers aren't smart enough to infer that SystemExit > > has something to do with termination. > > > > > > > > Raymond > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From bcannon at gmail.com Mon Aug 15 19:44:12 2005 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 15 Aug 2005 10:44:12 -0700 Subject: [Python-Dev] PEP 348 (exception reorg) revised again In-Reply-To: <200508151548.11552.tdickenson@devmail.geminidataloggers.co.uk> References: <001601c5a19b$9b2abf80$af26c797@oemcomputer> <200508151548.11552.tdickenson@devmail.geminidataloggers.co.uk> Message-ID: On 8/15/05, Toby Dickenson wrote: > On Monday 15 August 2005 14:16, Raymond Hettinger wrote: > > > -1 on replacing "except (KeyboardInterrupt, SystemExit)" with "except > > TerminatingException". > > The rationale for including TerminatingException in the PEP would also be > satisfied by having a TerminatingExceptions tuple (in the exceptions > module?). It makes sense to express the classification of exceptions that are > intended to terminate the interpreter, but we dont need to express that > classification as inheritence. > While the idea is fine, I just know that the point is going to be brought up that the addition should not be done until experience with the new hierarchy is had. I will add a comment that tuples can be added to the module after enough experience is had, but I am not going to try pushing for this right now. Of course I could be surprised and everyone could support the idea. =) -Brett From dberlin at dberlin.org Mon Aug 15 21:06:03 2005 From: dberlin at dberlin.org (Daniel Berlin) Date: Mon, 15 Aug 2005 15:06:03 -0400 Subject: [Python-Dev] PEP: Migrating the Python CVS to Subversion In-Reply-To: <66d0a6e1050815092760a2dab3@mail.gmail.com> References: <42E93940.6080708@v.loewis.de> <1122607673.9665.38.camel@geddy.wooz.org> <87fytu2lly.fsf@tleepslib.sk.tsukuba.ac.jp> <1122918723.9680.33.camel@warna.corp.google.com> <42EF2794.1000209@v.loewis.de> <66d0a6e105080312181e25fa08@mail.gmail.com> <42F1AADE.50908@v.loewis.de> <66d0a6e105080718527939aa81@mail.gmail.com> <42F6F61B.1080505@v.loewis.de> <66d0a6e1050815092760a2dab3@mail.gmail.com> Message-ID: <1124132763.3143.10.camel@MAMBA> On Mon, 2005-08-15 at 12:27 -0400, Nicholas Bastin wrote: > On 8/8/05, "Martin v. L?wis" wrote: > > Nicholas Bastin wrote: > > > It's a mature product. I would hope that that would count for > > > something. > > > > Sure. But so is subversion. > > I will then assume that you and I have different ideas of what 'mature' means. Bigger projects than Python use it and consider it mature for real use (All the Apache projects, all of KDE, GNOME is planning on switching soon, etc). I've never seen a corrupted FSFS repo, only corrupted BDB repos, and I will happily grant that using BDB ended up being a big mistake for Subversion. Not one that could have easily been foreseen at the time, but such is life. But this is why FSFS is the default for 1.2+ I've never seen you post about a corrupted repository to svn-users or svn-dev or file a bug, so i can't say why you see corrupted repositories if they are FSFS ones. --Dan From Scott.Daniels at Acm.Org Mon Aug 15 21:27:57 2005 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Mon, 15 Aug 2005 12:27:57 -0700 Subject: [Python-Dev] PEP 348 (exception reorg) revised again In-Reply-To: <200508151548.11552.tdickenson@devmail.geminidataloggers.co.uk> References: <001601c5a19b$9b2abf80$af26c797@oemcomputer> <200508151548.11552.tdickenson@devmail.geminidataloggers.co.uk> Message-ID: Toby Dickenson wrote: > On Monday 15 August 2005 14:16, Raymond Hettinger wrote: > > The rationale for including TerminatingException in the PEP would also be > satisfied by having a TerminatingExceptions tuple (in the exceptions > module?). It makes sense to express the classification of exceptions that are > intended to terminate the interpreter, but we dont need to express that > classification as inheritence. > An argument _for_ TerminatingException as a class is that I can define my own subclasses of TerminatingException without forcing it to being a subclass of KeyboardInterrupt or SystemExit. -- Scott David Daniels Scott.Daniels at Acm.Org From gvanrossum at gmail.com Mon Aug 15 21:36:29 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon, 15 Aug 2005 12:36:29 -0700 Subject: [Python-Dev] PEP 348 (exception reorg) revised again In-Reply-To: References: <001601c5a19b$9b2abf80$af26c797@oemcomputer> <200508151548.11552.tdickenson@devmail.geminidataloggers.co.uk> Message-ID: On 8/15/05, Scott David Daniels wrote: > An argument _for_ TerminatingException as a class is that I can > define my own subclasses of TerminatingException without forcing > it to being a subclass of KeyboardInterrupt or SystemExit. And how would that help you? Would your own exceptions be more like SystemExit or more like KeyboardInterrupt, or neither? If you mean them to be excluded by base "except:", you can always subclass BaseException, which exists for this purpose. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Mon Aug 15 22:49:48 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 15 Aug 2005 22:49:48 +0200 Subject: [Python-Dev] PEP: Migrating the Python CVS to Subversion In-Reply-To: <66d0a6e1050815092760a2dab3@mail.gmail.com> References: <42E93940.6080708@v.loewis.de> <1122607673.9665.38.camel@geddy.wooz.org> <87fytu2lly.fsf@tleepslib.sk.tsukuba.ac.jp> <1122918723.9680.33.camel@warna.corp.google.com> <42EF2794.1000209@v.loewis.de> <66d0a6e105080312181e25fa08@mail.gmail.com> <42F1AADE.50908@v.loewis.de> <66d0a6e105080718527939aa81@mail.gmail.com> <42F6F61B.1080505@v.loewis.de> <66d0a6e1050815092760a2dab3@mail.gmail.com> Message-ID: <4300FFEC.3090001@v.loewis.de> Nicholas Bastin wrote: > Not completely. More like -0 at the moment. We need a better system, > but I think we shouldn't just pick a system because it's the one the > PEP writer preferred - there should be some sort of effort to test a > few systems (including bug trackers). But that's how the PEP process works: the PEP author is supposed to collect feedback from the community in a fair way, but he is not required to implement every suggestion that the community makes. People who strongly disagree that the entire approach should be taken should write an alternative ("counter") PEP, proposing their strategy. In the end, the BDFL will pronounce which approach (if any) should be implemented. In the specific case, I'm personally not willing to discuss every SCM system out there. If somebody manages to make me curious (as Guido did with the bazaar posts), I will try it out, if I can find an easy way to do so. Your comments about (what was the name again) did not make me curious. As for bug trackers: this PEP is specifically *not* about bug trackers at all. If you think the SourceForge bugtracker should be replaced with something else, write a PEP. I really don't see a reasonable alternative to the SF bugtracker. > I know this is work, but this > isn't just something we can change easily again later. I don't bother asking who "we" is, here: apparently not you. Regards, Martin From fperez.net at gmail.com Mon Aug 15 22:48:48 2005 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 15 Aug 2005 14:48:48 -0600 Subject: [Python-Dev] SWIG and rlcompleter References: <43003492.8060904@mpi-magdeburg.mpg.de> Message-ID: Guido van Rossum wrote: > (1) Please use the SF patch manager. > > (2) Please don't propose adding more bare "except:" clauses to the > standard library. > > (3) I think a better patch is to use str(word)[:n] instead of word[:n]. Sorry to jump in, but this same patch was proposed for ipython, and my reply was that it appeared to me as a SWIG bug. From: http://www.python.org/doc/2.4.1/lib/built-in-funcs.html the docs for dir() seem to suggest that dir() should only return strings (I am inferring that from things like 'The resulting list is sorted alphabetically'). The docs are not fully explicit on this, though. Am I interpreting the docs correctly, case in which this should be considered a SWIG bug? Or is it OK for objects to stuff non-strings in __dict__, case in which SWIG is OK and then rlcompleter (and the corresponding system in ipython) do need to protect against this situation. I'd appreciate a clarification here, so I can close my own ipython bug report as well. Thanks, f From bos at serpentine.com Mon Aug 15 23:04:53 2005 From: bos at serpentine.com (Bryan O'Sullivan) Date: Mon, 15 Aug 2005 14:04:53 -0700 Subject: [Python-Dev] On distributed vs centralised SCM for Python Message-ID: <1124139893.20124.29.camel@localhost.localdomain> Pardon me for coming a little late to the SCM discussion, but I thought I would throw a few comments in. A little background: I've used Perforce, CVS, Subversion and BitKeeper for a number of years. Currently, I hack on Mercurial . However, I'm not here to try and specifically push Mercurial, but rather to bring up a few points that I haven't seen made in the earlier discussions. The biggest distinguishing factor between centralised and decentralised SCMs is the kinds of interaction they permit between the core developer community and outsiders. The centralised SCM tools all create a wall between core developers (i.e. people with commit access to the central repository) and people who are on the fringes. Outsiders may be able to get anonymous read-only access, but they are left up to their own devices if they want to make changes that they would like to contribute back to the project. With centralised tools, any revision control that outsiders do must be ad-hoc in nature, and they cannot share their changes in a natural way (i.e. preserving revision history) with anyone else. I do not follow Python development closely, so I have no idea how open Python typically is to contributions from people outside the core CVS committers. However, it's worth pointing out that with a distributed SCM - it doesn't really matter which one you use - it is simple to put together a workflow that operates in the same way as a centralised SCM. You lose nothing in the translation. What you gain is several-fold: * Outsiders get to work according to the same terms, and with the same tools, as core developers. * Everyone can perform whatever work they want (branch, commit, diff, undo, etc) without being connected to the main repository in any way. * Peer-level sharing of changes, for testing or evaluation, is easy and doesn't clutter up the central server with short-lived branches. * Speculative branching: it is cheap to create a local private branch that contains some half-baked changes. If they work out, fold them back and commit them to the main repository. If not, blow the branch away and forget about it. Regardless of what you may think of the Linux development model, it is teling that there have been about 80 people able to commit changes to Python since 1990 (I just checked the cvsroot tarball), whereas my estimate is that about ten times as many used BitKeeper to contribute changes to the Linux kernel just since the 2.5 tree began in 2002. (The total number of users who contributed changes was about 1600, 1300 of whom used BK, while the remainder emailed plain old patches that someone applied.) It is, of course, not possible for me to tell which CVS commits were really patches that originated with someone else, but my intent is to show how the choice of tools affects the ability of people to contribute in "natural" ways. How much of the difference in numbers is due to the respective popularity or accessibility of the projects is anyone's guess. With any luck, there's some food for thought above. Regards, From foom at fuhm.net Mon Aug 15 23:20:15 2005 From: foom at fuhm.net (James Y Knight) Date: Mon, 15 Aug 2005 17:20:15 -0400 Subject: [Python-Dev] On distributed vs centralised SCM for Python In-Reply-To: <1124139893.20124.29.camel@localhost.localdomain> References: <1124139893.20124.29.camel@localhost.localdomain> Message-ID: <6208AA5C-3E27-40EE-BD7B-CB6E1CA3D764@fuhm.net> On Aug 15, 2005, at 5:04 PM, Bryan O'Sullivan wrote: > The centralised SCM tools all create a wall between core developers > (i.e. people with commit access to the central repository) and people > who are on the fringes. Outsiders may be able to get anonymous > read-only access, but they are left up to their own devices if they > want > to make changes that they would like to contribute back to the > project. But, if python is using svn, outside developers can seamlessly use svk (http://svk.elixus.org/) to do their own branches if they wish, no? Sure, that is "their own devices", but it seems a fairly workable solution to me as the two are so closely related. Now, I've never tried this, so I'm just judging from the "marketing material" on the svk website. James From martin at v.loewis.de Mon Aug 15 23:29:23 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 15 Aug 2005 23:29:23 +0200 Subject: [Python-Dev] On distributed vs centralised SCM for Python In-Reply-To: <1124139893.20124.29.camel@localhost.localdomain> References: <1124139893.20124.29.camel@localhost.localdomain> Message-ID: <43010933.3050303@v.loewis.de> Bryan O'Sullivan wrote: > However, it's worth pointing out that with a distributed SCM - it > doesn't really matter which one you use - it is simple to put together a > workflow that operates in the same way as a centralised SCM. You lose > nothing in the translation. What you gain is several-fold: That may be off-topic for python-dev, but can you please explain how this works? > * Outsiders get to work according to the same terms, and with the > same tools, as core developers. I'm using git on the kernel level. In what way am I at the same level as the core developers? They can write to the kernel.org repository, I cannot. They use commit, I send diffs. > * Everyone can perform whatever work they want (branch, commit, > diff, undo, etc) without being connected to the main repository > in any way. So what? If I want to branch, I create a new sandbox. I have to do that anyway, since independent projects should not influence each other. I can also easily diff, whether I have write access or not (in svn, even simpler so than in CVS). There is no easy way to undo parts of the changes, that's true. > * Peer-level sharing of changes, for testing or evaluation, is > easy and doesn't clutter up the central server with short-lived > branches. So how does that work? If I commit the changes to my local version of the repository, how do they get peer-level-shared? I turn off my machine when I leave the house, and I don't have a permanent IP, anyway, to host a web server or some such. > * Speculative branching: it is cheap to create a local private > branch that contains some half-baked changes. If they work out, > fold them back and commit them to the main repository. If not, > blow the branch away and forget about it. I do that with separate sandboxes right now. cp -a py2.5 py-64bit gives me a new sandbox, in which I can do my speculative project. > Regardless of what you may think of the Linux development model, it is > teling that there have been about 80 people able to commit changes to > Python since 1990 (I just checked the cvsroot tarball), whereas my > estimate is that about ten times as many used BitKeeper to contribute > changes to the Linux kernel just since the 2.5 tree began in 2002. (The > total number of users who contributed changes was about 1600, 1300 of > whom used BK, while the remainder emailed plain old patches that someone > applied.) Hmm. The changes of these 800 people had to be approved by some core developers, or perhaps even all approved by Linus Torvalds, right? This is really the same for Python: A partial list of contributors is in Misc/ACKS (663 lines at the moment), and this doesn't list all the people who contributed trivial changes. So I guess Python has the same number of contributors per line as the Linux kernel. > It is, of course, not possible for me to tell which CVS commits were > really patches that originated with someone else, but my intent is to > show how the choice of tools affects the ability of people to contribute > in "natural" ways. I hear that, but I have a hard time believing it. People find the "cvs diff -u, send diff file for discussion to patches tracker" cycle quite natural. Regards, Martin From bos at serpentine.com Tue Aug 16 00:19:40 2005 From: bos at serpentine.com (Bryan O'Sullivan) Date: Mon, 15 Aug 2005 15:19:40 -0700 Subject: [Python-Dev] On distributed vs centralised SCM for Python In-Reply-To: <43010933.3050303@v.loewis.de> References: <1124139893.20124.29.camel@localhost.localdomain> <43010933.3050303@v.loewis.de> Message-ID: <1124144380.20124.44.camel@localhost.localdomain> On Mon, 2005-08-15 at 23:29 +0200, "Martin v. L?wis" wrote: > That may be off-topic for python-dev, but can you please explain how > this works? It's simple enough. In place of a central server that hosts a set of repositories and a number of branches, and to which only a few people have access, you use a central server that hosts a number of repositories, and you get the idea. But the difference lies in the way you use it. In the centralised model, there's only one server, and only one repository, anywhere. In the distributed model, each developer has one or more repositories that they keep in sync with the central ones they are interested in, pulling and pushing changes as necessary. The difference is that they get to share changes horizontally if they wish, without going through the central server. > I'm using git on the kernel level. In what way am I at the same level > as the core developers? You can use the same tools to do the same things they can. You can communicate with them in terms of commits. You may each have access to different sets of servers from which other people can pull changes, but if they want to take changes from you, you have the option of giving them complete history of all the edits and merges you've done, with no information loss. > So how does that work? If I commit the changes to my local version of > the repository, how do they get peer-level-shared? You have to do something to share them, but it's a lot simpler than sending diffs to a mailing list, or attaching them to a bug tracking system note. > Hmm. The changes of these 800 people had to be approved by some core > developers, or perhaps even all approved by Linus Torvalds, right? True. > I hear that, but I have a hard time believing it. People find the > "cvs diff -u, send diff file for discussion to patches tracker" > cycle quite natural. People will find doing the same of anything, over and over for fifteen years, quite natural :-) From raymond.hettinger at verizon.net Tue Aug 16 02:21:45 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Mon, 15 Aug 2005 20:21:45 -0400 Subject: [Python-Dev] On distributed vs centralised SCM for Python In-Reply-To: <6208AA5C-3E27-40EE-BD7B-CB6E1CA3D764@fuhm.net> Message-ID: <006201c5a1f8$7c9f4060$af26c797@oemcomputer> [Bryan O'Sullivan] > > The centralised SCM tools all create a wall between core developers > > (i.e. people with commit access to the central repository) and people > > who are on the fringes. Outsiders may be able to get anonymous > > read-only access, but they are left up to their own devices if they > > want > > to make changes that they would like to contribute back to the > > project. [James Y Knight] > But, if python is using svn, outside developers can seamlessly use > svk (http://svk.elixus.org/) to do their own branches if they wish, > no? Sure, that is "their own devices", but it seems a fairly workable > solution to me as the two are so closely related. +1 This seems to be the most flexible and sensible idea so far. The svn system has had many accolades; Martin knows how to convert it; and it presents only a small learning curve to cvs users. Optionally adding svk to the mix allows us to get the benefits of a distributed system without any additional migration or support issues. Very nice. Raymond From tim.peters at gmail.com Tue Aug 16 03:07:51 2005 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 15 Aug 2005 21:07:51 -0400 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <42F61C03.6050703@v.loewis.de> References: <42F61C03.6050703@v.loewis.de> Message-ID: <1f7befae05081518073433da62@mail.gmail.com> [Martin v. L?wis] > I have placed a new version of the PEP on > > http://www.python.org/peps/pep-0347.html ... +1 from me. But, I don't think my vote should count much, and (sorry) Guido's even less: what do the people who frequently check in want? That means people like you (Martin), Michael, Raymond, Walter, Fred. ... plus the release manager(s). BTW, a stumbling block in Zope's conversion to SVN was that the conversion script initially never set svn:eol-style on any file. This caused weeks of problems, as people on Windows got Linux line ends, and people checking in from Windows forced Windows line ends on Linuxheads (CVS defaults to assuming files are text; SVN binary). The peculiar workaround at Zope is that we're all encouraged to add something like this to our SVN config file: """ [auto-props] # Setting eol-style to native on all files is a trick: if svn # believes a new file is binary, it won't honor the eol-style # auto-prop. However, svn considers the request to set eol-style # to be an error then, and if adding multiple files with one # svn "add" cmd, svn will stop adding files after the first # such error. A future release of svn will probably consider # this to be a warning instead (and continue adding files). * = svn:eol-style=native """ It would be best if svn:eol-style were set to native during initial conversion from CVS, on all files not marked binary in CVS. From jepler at unpythonic.net Tue Aug 16 04:16:18 2005 From: jepler at unpythonic.net (jepler@unpythonic.net) Date: Mon, 15 Aug 2005 21:16:18 -0500 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: References: <43003492.8060904@mpi-magdeburg.mpg.de> Message-ID: <20050816021614.GA23688@unpythonic.net> You don't need something like a buggy SWIG to put non-strings in dir(). >>> class C: pass ... >>> C.__dict__[3] = "bad wolf" >>> dir(C) [3, '__doc__', '__module__'] This is likely to happen "legitimately", for instance in a class that allows x.y and x['y'] to mean the same thing. (if the user assigns to x[3]) Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20050815/0ce34ebf/attachment.pgp From mbp at sourcefrog.net Mon Aug 15 07:12:34 2005 From: mbp at sourcefrog.net (Martin Pool) Date: Mon, 15 Aug 2005 15:12:34 +1000 Subject: [Python-Dev] cvs to bzr? References: <17150.31637.180169.877441@montanaro.dyndns.org> <17152.109.883835.190683@montanaro.dyndns.org> Message-ID: On Sun, 14 Aug 2005 21:39:41 -0500, skip wrote: > Lalo> You can, however, convert from CVS to baz (arch), and from there > Lalo> to bzr. > > Would this be with cscvs? According to the cscvs wiki page at > > http://wiki.gnuarch.org/cscvs > > cscvs is current unmaintained and can't handle repositories with branches. > In addition, it appears that to do a one-time convertsion from cvs to bzr I > will need to also install arch and baz as well as any other packages they > depend on. Canonical has had an ongoing project to pull many cvs trees into baz, for the benefit of our Ubuntu distribution people amongst other things. There are some people working on imports using (I think) a hacked version of cscvs, and I have asked them to get Python in as a high priority. Apparently there is something in the cvs history which makes a precise import hard. The cvs->baz->bzr process is unfortunate. As Mark said, we're going to be moving away from the Arch-based code and so trying to make that process simpler. -- Martin From abo at minkirri.apana.org.au Tue Aug 16 06:51:37 2005 From: abo at minkirri.apana.org.au (Donovan Baarda) Date: Mon, 15 Aug 2005 21:51:37 -0700 Subject: [Python-Dev] Fwd: Distributed RCS In-Reply-To: <43007CDC.6060200@benjiyork.com> References: <42FBA376.5030605@canonical.com> <42FF32AA.7040506@v.loewis.de> <17151.15640.173982.961359@montanaro.dyndns.org> <42FF6E4B.4000206@v.loewis.de> <43007CDC.6060200@benjiyork.com> Message-ID: <1124167897.351.50.camel@warna.corp.google.com> On Mon, 2005-08-15 at 04:30, Benji York wrote: > Martin v. L?wis wrote: > > skip at pobox.com wrote: > >>Granted. What is the cost of waiting a bit longer to see if it (or > >>something else) gets more usable and would hit the mark better than svn? > > > > It depends on what "a bit" is. Waiting a month would be fine; waiting > > two years might be pointless. > > This might be too convoluted to consider, but I thought I might throw it > out there. We use svn for our repositories, but I've taken to also > using bzr so I can do local commits and reversions (within a particular > svn reversion). I can imagine expanding that usage to sharing branches > and such via bzr (or mercurial, which looks great), but keeping the > trunk in svn. Not too convoluted at all; I already do exactly this with many upstream CVS and SVN repositorys, using a local PRCS for my own branches. I'm considering switching to a distributed RCS for my own branches because it would make it easier for others to share them. I think this probably is the best solution; it gives a reliable(?) centralised RCS for the trunk, but allows distributed development. -- Donovan Baarda From bcannon at gmail.com Tue Aug 16 08:25:15 2005 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 15 Aug 2005 23:25:15 -0700 Subject: [Python-Dev] rev. 1.9 of PEP 348: Raymond tested, Guido approved Message-ID: OK, TerminatingException and the removal of bare 'except' clauses are now out. I also stripped out the transition plan to basically just add BaseException in Python 2.5, tweak docs to recommend future-proof practices, and then change everything in Python 3.0 . This will prevent any nasty performance hit from what was being previously suggested to try to make it all backwards-compatible. -Brett From martin at v.loewis.de Tue Aug 16 08:52:43 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 16 Aug 2005 08:52:43 +0200 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <1f7befae05081518073433da62@mail.gmail.com> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> Message-ID: <43018D3B.9040404@v.loewis.de> Tim Peters wrote: > It would be best if svn:eol-style were set to native during initial > conversion from CVS, on all files not marked binary in CVS. Ok, I'll add that to the PEP. Not sure how to implement it, yet... Regards, Martin From senko.rasic at gmail.com Tue Aug 16 09:17:42 2005 From: senko.rasic at gmail.com (Senko Rasic) Date: Tue, 16 Aug 2005 09:17:42 +0200 Subject: [Python-Dev] Extension to dl module to allow passing strings from native function In-Reply-To: <42FDE000.9080508@v.loewis.de> References: <48bbc5810508111640a6bd03e@mail.gmail.com> <42FDE000.9080508@v.loewis.de> Message-ID: <48bbc58105081600174fe3570d@mail.gmail.com> On 8/13/05, "Martin v. L?wis" wrote: > Are you aware of the ctypes module? > > http://starship.python.net/crew/theller/ctypes/ I didn't know about ctypes, thanks for the pointer. It definitely has much more functionality (although it's more complex and a whole new module) than my little hack ;-) Regards, Senko -- Senko Rasic From mwh at python.net Tue Aug 16 13:35:43 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 16 Aug 2005 12:35:43 +0100 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <20050816021614.GA23688@unpythonic.net> (jepler@unpythonic.net's message of "Mon, 15 Aug 2005 21:16:18 -0500") References: <43003492.8060904@mpi-magdeburg.mpg.de> <20050816021614.GA23688@unpythonic.net> Message-ID: <2m4q9qunao.fsf@starship.python.net> jepler at unpythonic.net writes: > You don't need something like a buggy SWIG to put non-strings in dir(). > >>>> class C: pass > ... >>>> C.__dict__[3] = "bad wolf" >>>> dir(C) > [3, '__doc__', '__module__'] > > This is likely to happen "legitimately", for instance in a class that allows > x.y and x['y'] to mean the same thing. (if the user assigns to x[3]) I wonder if dir() should strip non-strings? Cheers, mwh -- A VoIP server "powered entirely by stabbing, that I made out of this gun I had" -- from Twisted.Quotes From mwh at python.net Tue Aug 16 13:42:32 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 16 Aug 2005 12:42:32 +0100 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <1f7befae05081518073433da62@mail.gmail.com> (Tim Peters's message of "Mon, 15 Aug 2005 21:07:51 -0400") References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> Message-ID: <2mzmrit8ev.fsf@starship.python.net> Tim Peters writes: > [Martin v. L?wis] >> I have placed a new version of the PEP on >> >> http://www.python.org/peps/pep-0347.html > > ... > > +1 from me. But, I don't think my vote should count much, and (sorry) > Guido's even less: what do the people who frequently check in want? > That means people like you (Martin), Michael, Raymond, Walter, Fred. > ... plus the release manager(s). I want svn, I think. I'm open to more sophisticated approaches but am not sure that any of them are really mature enough yet. Probably will be soon, but not soon enough to void the effort of moving to svn (IMHO). I'm not really a release manager these days, but if I was, I'd wand svn for that reason too. The third set of people who count are pydotorg admins. I'm not really one of those either at the moment. While SF's CVS setup has it's problems (occasional outages; it's only CVS) it's hard to beat what it costs us in sysadmin time: zero. > It would be best if svn:eol-style were set to native during initial > conversion from CVS, on all files not marked binary in CVS. Yes. Cheers, mwh -- I recompiled XFree 4.2 with gcc 3.2-beta-from-cvs with -O42 and -march-pentium4-800Mhz and I am sure that the MOUSE CURSOR is moving 5 % FASTER! -- from Twisted.Quotes From anthony at interlink.com.au Tue Aug 16 14:08:26 2005 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue, 16 Aug 2005 22:08:26 +1000 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <2mzmrit8ev.fsf@starship.python.net> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> Message-ID: <200508162208.28862.anthony@interlink.com.au> On Tuesday 16 August 2005 21:42, Michael Hudson wrote: > I want svn, I think. I'm open to more sophisticated approaches but am > not sure that any of them are really mature enough yet. Probably will > be soon, but not soon enough to void the effort of moving to svn > (IMHO). > > I'm not really a release manager these days, but if I was, I'd wand > svn for that reason too. I _am_ a release manager these days, and I'm in favour of svn. I really want to be off CVS, and I would love to be able to go with something more sophisticated than svn. Unfortunately, I really don't think any of the alternatives are appropriate. While Perforce is definitely capable, the Bitkeeper disaster strongly influence me against relying on the generosity of a commercial software vendor who could change their mind at any time. The more radical (and powerful) tools such as baz/bzr, darcs, monotone and the like really aren't there yet. I have no doubt that they will get there, but right now, I want something better than CVS, and I don't want to have to fight bugs or limitations in the revision control system. By the way - if you're intending on suggesting alternates to svn, please don't just post a link saying "check out this system". Post an explanation of _why_ we should look at this particular system. What's it's strengths? Why should we invest the time to download it and play with it? Speaking for myself, I don't have the time or energy to spend trying the countless numbers of revision control systems that are out there. Thanks, Anthony -- Anthony Baxter It's never too late to have a happy childhood. From barry at python.org Tue Aug 16 14:42:59 2005 From: barry at python.org (Barry Warsaw) Date: Tue, 16 Aug 2005 08:42:59 -0400 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <2mzmrit8ev.fsf@starship.python.net> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> Message-ID: <1124196179.9673.12.camel@geddy.wooz.org> On Tue, 2005-08-16 at 07:42, Michael Hudson wrote: > The third set of people who count are pydotorg admins. I'm not really > one of those either at the moment. While SF's CVS setup has it's > problems (occasional outages; it's only CVS) it's hard to beat what it > costs us in sysadmin time: zero. True, although because of the peculiarities of cvs, there have definitely been times I wish we had direct access to the repository. svn should make most of those reasons moot. As for sysadmin time with the changes proposed by the pep -- clearly they won't be zero, but I think the overhead for svn itself will be nearly so. With the fsfs backend, there's almost no continuous care and feeding needed, including for backups (which XS4ALL takes care of). The overhead for the admins will be in user management. I really don't think it will be that much more effort for new developers to badger the admins into adding them to some config file than it currently is to get one of us to click a few links to add you to the SF project. ;) (Okay, yeah we'll have to manage credentials now.) The alternatives to svn all sound very enticing, however my own feeling is that while the workflows they make possible might be good for Python in the long run, it's not clear how all that will evolve. We know that we can treat svn as "a better cvs" and the current workflow seems to serve us well enough. I'd be happy to switch to svn now, while continuing to experiment and follow the better scm systems for the future. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050816/b66b0382/attachment.pgp From mwh at python.net Tue Aug 16 14:52:13 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 16 Aug 2005 13:52:13 +0100 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <1124196179.9673.12.camel@geddy.wooz.org> (Barry Warsaw's message of "Tue, 16 Aug 2005 08:42:59 -0400") References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <1124196179.9673.12.camel@geddy.wooz.org> Message-ID: <2mvf26t56q.fsf@starship.python.net> Barry Warsaw writes: > On Tue, 2005-08-16 at 07:42, Michael Hudson wrote: > >> The third set of people who count are pydotorg admins. I'm not really >> one of those either at the moment. While SF's CVS setup has it's >> problems (occasional outages; it's only CVS) it's hard to beat what it >> costs us in sysadmin time: zero. > > True, although because of the peculiarities of cvs, there have > definitely been times I wish we had direct access to the repository. > svn should make most of those reasons moot. > > As for sysadmin time with the changes proposed by the pep -- clearly > they won't be zero, but I think the overhead for svn itself will be > nearly so. OK, that's more or less what I thought. [...] > I'd be happy to switch to svn now, while continuing to experiment > and follow the better scm systems for the future. I suppose another question is: when? Between 2.4.2 and 2.5a1 seems like a good opportunity. I guess the biggest job is collection of keys and associated admin? Cheers, mwh -- well, take it from an old hand: the only reason it would be easier to program in C is that you can't easily express complex problems in C, so you don't. -- Erik Naggum, comp.lang.lisp From jack at performancedrivers.com Tue Aug 16 15:00:46 2005 From: jack at performancedrivers.com (Jack Diederich) Date: Tue, 16 Aug 2005 09:00:46 -0400 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <200508162208.28862.anthony@interlink.com.au> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <200508162208.28862.anthony@interlink.com.au> Message-ID: <20050816130045.GA10364@performancedrivers.com> On Tue, Aug 16, 2005 at 10:08:26PM +1000, Anthony Baxter wrote: > On Tuesday 16 August 2005 21:42, Michael Hudson wrote: > > I want svn, I think. I'm open to more sophisticated approaches but am > > not sure that any of them are really mature enough yet. Probably will > > be soon, but not soon enough to void the effort of moving to svn > > (IMHO). > > > > I'm not really a release manager these days, but if I was, I'd wand > > svn for that reason too. > > I _am_ a release manager these days, and I'm in favour of svn. I really > want to be off CVS, and I would love to be able to go with something > more sophisticated than svn. Unfortunately, I really don't think any of > the alternatives are appropriate. As a non-committer I can say _anything_ is preferable to the current situation and svn is good enough. bzr might make it even easier but svn is familiar and it will work right now. I haven't submitted a patch in ages partly because using anonymous SF cvs plain doesn't work. aside, at work we switched from cvs to svn and it the transition was easy for developers, svn lives up to its billing as a fixed cvs. -jack From e.a.m.brouwer at alumnus.utwente.nl Mon Aug 15 00:48:00 2005 From: e.a.m.brouwer at alumnus.utwente.nl (Martijn Brouwer) Date: Sun, 14 Aug 2005 22:48:00 +0000 Subject: [Python-Dev] implementation of copy standard lib Message-ID: <1124059680.11612.9.camel@localhost.localdomain> Hi, After profiling a small python script I found that approximately 50% of the runtime of my script was consumed by one line: "import copy". Another 15% was the startup of the interpreter, but that is OK for an interpreted language. The copy library is used by another library I am using for my scripts. Importing copy takes 5-10 times more time that import os, string and re together! I noticed that this lib is implemented in python, not in C. As I can imagine that *a lot* of libs/scripts use the copy library, I think it worthwhile to implement this lib in C. Unfortunately I cannot do this myself: I am relatively inexperienced with python and do not know C. What are your opinions? Martijn Brouwer -- __________________________________________________ I have a new e-mail adress. If you are still using e.a.m.brouwer at tnw.utwente.nl, please change to e.a.m.brouwer at alumnus.utwente.nl __________________________________________________ From simon.brunning at gmail.com Tue Aug 16 16:28:53 2005 From: simon.brunning at gmail.com (Simon Brunning) Date: Tue, 16 Aug 2005 15:28:53 +0100 Subject: [Python-Dev] implementation of copy standard lib In-Reply-To: <1124059680.11612.9.camel@localhost.localdomain> References: <1124059680.11612.9.camel@localhost.localdomain> Message-ID: <8c7f10c605081607281f8c1e38@mail.gmail.com> On 8/14/05, Martijn Brouwer wrote: > I noticed that this lib is implemented in python, not in C. As I can > imagine that *a lot* of libs/scripts use the copy library, I think it > worthwhile to implement this lib in C. > Unfortunately I cannot do this myself: I am relatively inexperienced > with python and do not know C. > > What are your opinions? I'll reply to this over on c.l.py, where it belongs. -- Cheers, Simon B, simon at brunningonline.net, http://www.brunningonline.net/simon/blog/ From foom at fuhm.net Tue Aug 16 16:34:31 2005 From: foom at fuhm.net (James Y Knight) Date: Tue, 16 Aug 2005 10:34:31 -0400 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <43018D3B.9040404@v.loewis.de> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <43018D3B.9040404@v.loewis.de> Message-ID: <4B72B610-80CA-40BC-9B9D-EB50F8077436@fuhm.net> On Aug 16, 2005, at 2:52 AM, Martin v. L?wis wrote: > Tim Peters wrote: > >> It would be best if svn:eol-style were set to native during initial >> conversion from CVS, on all files not marked binary in CVS. >> > > Ok, I'll add that to the PEP. Not sure how to implement it, yet... cvs2svn does that by default (now). James From fdrake at acm.org Tue Aug 16 18:41:10 2005 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue, 16 Aug 2005 12:41:10 -0400 Subject: [Python-Dev] dev listinfo page (was: Re: Python + Ping) In-Reply-To: <42FC666A.90206@botanicus.net> References: <2773CAC687FD5F4689F526998C7E4E5F05CC00@au3010avexu1.global.avaya.com> <42FC666A.90206@botanicus.net> Message-ID: <200508161241.10908.fdrake@acm.org> On Friday 12 August 2005 05:05, David Wilson wrote: > Would it perhaps be an idea, given the number of users posting to the > dev list, to put a rather obvious warning on the listinfo page: Well, not exactly the style you suggested, but I've made it fairly close. It's certainly more noticable now. :-) -Fred -- Fred L. Drake, Jr. From fperez.net at gmail.com Tue Aug 16 19:08:59 2005 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 16 Aug 2005 11:08:59 -0600 Subject: [Python-Dev] SWIG and rlcompleter References: <43003492.8060904@mpi-magdeburg.mpg.de> <20050816021614.GA23688@unpythonic.net> <2m4q9qunao.fsf@starship.python.net> Message-ID: Michael Hudson wrote: > jepler at unpythonic.net writes: > >> You don't need something like a buggy SWIG to put non-strings in dir(). >> >>>>> class C: pass >> ... >>>>> C.__dict__[3] = "bad wolf" >>>>> dir(C) >> [3, '__doc__', '__module__'] >> >> This is likely to happen "legitimately", for instance in a class that allows >> x.y and x['y'] to mean the same thing. (if the user assigns to x[3]) > > I wonder if dir() should strip non-strings? Me too. And it would be a good idea, I think, to specify explicitly in the dir() docs this behavior. Right now at least rlcompleter and ipython's completer can break due to this, there may be other tools out there with similar problems. If it's a stated design goal that dir() can return non-strings, that's fine. I can filter them out in my completion code. I'd just like to know what the official stance on dir()'s return values is. Cheers, f From fperez.net at gmail.com Tue Aug 16 19:17:04 2005 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 16 Aug 2005 11:17:04 -0600 Subject: [Python-Dev] SWIG and rlcompleter References: <43003492.8060904@mpi-magdeburg.mpg.de> Message-ID: Guido van Rossum wrote: > (3) I think a better patch is to use str(word)[:n] instead of word[:n]. Mmh, I'm not so sure that's a good idea, as it leads to this: In [1]: class f: pass ...: In [2]: a=f() In [3]: a.__dict__[1] = 8 In [4]: a.x = 0 In [5]: a. a.1 a.x In [5]: a.1 ------------------------------------------------------------ File " ", line 1 a.1 ^ SyntaxError: invalid syntax In general, foo.x named attribute access is only valid for strings to begin with (what about unicode in there?). Instead, this is what I've actually implemented in ipython: words = [w for w in dir(object) if isinstance(w, basestring)] That does allow unicode, I'm not sure if that's a good thing to do. Cheers, f From martin at v.loewis.de Tue Aug 16 20:19:33 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 16 Aug 2005 20:19:33 +0200 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <4B72B610-80CA-40BC-9B9D-EB50F8077436@fuhm.net> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <43018D3B.9040404@v.loewis.de> <4B72B610-80CA-40BC-9B9D-EB50F8077436@fuhm.net> Message-ID: <43022E35.1070207@v.loewis.de> James Y Knight wrote: > cvs2svn does that by default (now). Ah, ok. Martin From martin at v.loewis.de Tue Aug 16 20:31:20 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 16 Aug 2005 20:31:20 +0200 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <2mvf26t56q.fsf@starship.python.net> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <1124196179.9673.12.camel@geddy.wooz.org> <2mvf26t56q.fsf@starship.python.net> Message-ID: <430230F8.3020405@v.loewis.de> Michael Hudson wrote: > I suppose another question is: when? Between 2.4.2 and 2.5a1 seems > like a good opportunity. I guess the biggest job is collection of > keys and associated admin? I would agree. However, there still is the debate of hosting the repository elsehwere. Some people (Anthony, Guido, Tim) would prefer to pay for it, instead of hosting it on svn.python.org. Regards, Martin From nas at arctrix.com Tue Aug 16 21:18:35 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Tue, 16 Aug 2005 13:18:35 -0600 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <430230F8.3020405@v.loewis.de> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <1124196179.9673.12.camel@geddy.wooz.org> <2mvf26t56q.fsf@starship.python.net> <430230F8.3020405@v.loewis.de> Message-ID: <20050816191835.GA18968@mems-exchange.org> On Tue, Aug 16, 2005 at 08:31:20PM +0200, "Martin v. L?wis" wrote: > I would agree. However, there still is the debate of hosting the > repository elsehwere. Some people (Anthony, Guido, Tim) would prefer > to pay for it, instead of hosting it on svn.python.org. Another option would be to pay someone to maintain the SVN setup on python.org. Unfortunately, I guess that would require someone else to first create a detailed description of the maintenance work required and to process bids. Neil From barry at python.org Tue Aug 16 21:28:35 2005 From: barry at python.org (Barry Warsaw) Date: Tue, 16 Aug 2005 15:28:35 -0400 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <20050816191835.GA18968@mems-exchange.org> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <1124196179.9673.12.camel@geddy.wooz.org> <2mvf26t56q.fsf@starship.python.net> <430230F8.3020405@v.loewis.de> <20050816191835.GA18968@mems-exchange.org> Message-ID: <1124220515.5254.7.camel@geddy.wooz.org> On Tue, 2005-08-16 at 15:18, Neil Schemenauer wrote: > Another option would be to pay someone to maintain the SVN setup on > python.org. Unfortunately, I guess that would require someone else > to first create a detailed description of the maintenance work > required and to process bids. Again, it's not clear to me that there's much more we need to have done that we either don't want to do ourselves or that XS4ALL isn't doing for us. IOW, we get backups for free and mostly the repo just swims along nicely. We have to do user management, but I think we want to do that ourselves anyway. There may be occasional infrastructural work that needs to happen (e.g. we still owe Martin a login for tunneling), but those tasks seem to me to be better handled either by volunteers or by short-term paid piece work. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050816/8a628ba5/attachment.pgp From tim.peters at gmail.com Tue Aug 16 21:49:39 2005 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 16 Aug 2005 15:49:39 -0400 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <430230F8.3020405@v.loewis.de> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <1124196179.9673.12.camel@geddy.wooz.org> <2mvf26t56q.fsf@starship.python.net> <430230F8.3020405@v.loewis.de> Message-ID: <1f7befae050816124932f5c66@mail.gmail.com> [Michael Hudson] >> I suppose another question is: when? Between 2.4.2 and 2.5a1 seems >> like a good opportunity. I guess the biggest job is collection of >> keys and associated admin? [Martin v. L?wis] > I would agree. However, there still is the debate of hosting the > repository elsehwere. Some people (Anthony, Guido, Tim) would prefer > to pay for it, instead of hosting it on svn.python.org. Not this Tim. I _asked_ whether we had sufficient volunteer resource to host it on python.org, because I didn't know. Barry has since made sufficiently reassuring gurgles on that point, in particular that ongoing maintenance (after initial conversion) for filesystem-flavor SVN is likely in-the-noise level work. From martin at v.loewis.de Tue Aug 16 22:25:38 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 16 Aug 2005 22:25:38 +0200 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <1f7befae050816124932f5c66@mail.gmail.com> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <1124196179.9673.12.camel@geddy.wooz.org> <2mvf26t56q.fsf@starship.python.net> <430230F8.3020405@v.loewis.de> <1f7befae050816124932f5c66@mail.gmail.com> Message-ID: <43024BC2.2010505@v.loewis.de> Tim Peters wrote: > Not this Tim. I _asked_ whether we had sufficient volunteer resource > to host it on python.org, because I didn't know. Barry has since made > sufficiently reassuring gurgles on that point, in particular that > ongoing maintenance (after initial conversion) for filesystem-flavor > SVN is likely in-the-noise level work. Ah, ok. Of course, Barry can only speak about the current availability of volunteers, which is quite good (especially since amk took over coordinating them), nobody can predict the future (the time machine apparently only works one-way). So I guess the concern stays, and, more objectively, this is a risk for the project (but so is any specific commercial offering). Regards, Martin From raymond.hettinger at verizon.net Tue Aug 16 22:24:47 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 16 Aug 2005 16:24:47 -0400 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <1f7befae05081518073433da62@mail.gmail.com> Message-ID: <011901c5a2a0$8c4f4700$af26c797@oemcomputer> [Tim] > +1 from me. But, I don't think my vote should count much, and (sorry) > Guido's even less: what do the people who frequently check in want? > That means people like you (Martin), Michael, Raymond, Walter, Fred. > ... plus the release manager(s). +1 from me. CVS is meeting my needs but I would definitely benefit from fast diffs and atomic commits. My experiences with SVN to-date have all been positive and it was easy to learn. Also, I think it is a nice plus that our choosing SVN means that others can choose SVK and get the benefits of a distributed rcs without us having to do anything extra to support it. James Knight's thoughts on the subject seem on target. Raymond From martin at v.loewis.de Tue Aug 16 22:33:27 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 16 Aug 2005 22:33:27 +0200 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <20050816191835.GA18968@mems-exchange.org> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <1124196179.9673.12.camel@geddy.wooz.org> <2mvf26t56q.fsf@starship.python.net> <430230F8.3020405@v.loewis.de> <20050816191835.GA18968@mems-exchange.org> Message-ID: <43024D97.5050306@v.loewis.de> Neil Schemenauer wrote: > Another option would be to pay someone to maintain the SVN setup on > python.org. Unfortunately, I guess that would require someone else > to first create a detailed description of the maintenance work > required and to process bids. I think this would be difficult. I could imagine services like tummy.com, where you can hire somebody on an hours-per-week basis; these people maintain multiple servers, and just need to do the proper accounting. However, they also (naturally) tend to desire an organization that meets their needs also, e.g. by providing the machine and network (this is apparently how tummy.com operates). If you are suggesting that the PSF hires a specific individual for that maintenance, the risk of getting somebody unexperienced/uncooperative would be much higher: if we were unhappy with the tummy.com guy looking after our hardware, we could complain to his boss; if that is the boss, we would just take our data and cancel the contract. Also, hiring somebody would be somewhat unfair to people who do similar tasks as volunteers, and I guess the board might not agree to such expenses. Regards, Martin From tim.peters at gmail.com Tue Aug 16 22:52:09 2005 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 16 Aug 2005 16:52:09 -0400 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <43024BC2.2010505@v.loewis.de> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <1124196179.9673.12.camel@geddy.wooz.org> <2mvf26t56q.fsf@starship.python.net> <430230F8.3020405@v.loewis.de> <1f7befae050816124932f5c66@mail.gmail.com> <43024BC2.2010505@v.loewis.de> Message-ID: <1f7befae050816135244a6b72f@mail.gmail.com> [Martin v. L?wis] > Ah, ok. Of course, Barry can only speak about the current availability > of volunteers, which is quite good (especially since amk took over > coordinating them), nobody can predict the future (the time machine > apparently only works one-way). So I guess the concern stays, and, > more objectively, this is a risk for the project (but so is any > specific commercial offering). I'm not really worried about it. Sounds like ongoing pain is pretty much limited to keeping committer accounts/credentials up to date, and that normal good backup procedures will deal with filesystem-SVN state as a matter of course. If there's one thing sysadmins love to do, it's fiddling with user accounts and credentials -- if _anyone_ volunteers to work on python.org, they'll be eager to lord this power over us . If not, that's fine too. The PSF has the funds and the mission to pay for infrastructure support; I'd just _rather_ spend PSF funds on "more glamorous" stuff (like grants and conferences). From tim.peters at gmail.com Tue Aug 16 23:00:43 2005 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 16 Aug 2005 17:00:43 -0400 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <011901c5a2a0$8c4f4700$af26c797@oemcomputer> References: <1f7befae05081518073433da62@mail.gmail.com> <011901c5a2a0$8c4f4700$af26c797@oemcomputer> Message-ID: <1f7befae050816140063dcda4@mail.gmail.com> [Raymond Hettinger] > +1 from me. CVS is meeting my needs but I would definitely benefit from > fast diffs and atomic commits. My experiences with SVN to-date have all > been positive and it was easy to learn. Good! That was my experience too, BTW -- SVN was a genuine improvement over CVS, and I was productive with it the first hour. There are "tricks" you'll learn too (or already have); for example, if you make a bunch of changes in a local checkout, and have to drop it for a while, it's easy and fast to create an SVN branch with those changes despite that you didn't plan on it from the start (create a new branch in the repository; `svn switch` to it locally, which leaves your local changes alone; then commit). > Also, I think it is a nice plus that our choosing SVN means that others > can choose SVK and get the benefits of a distributed rcs without us > having to do anything extra to support it. James Knight's thoughts on > the subject seem on target. Too new-fashioned for me, although I can see how it might appeal to kids ;-) From skip at pobox.com Tue Aug 16 23:07:09 2005 From: skip at pobox.com (skip@pobox.com) Date: Tue, 16 Aug 2005 16:07:09 -0500 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <43024BC2.2010505@v.loewis.de> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> <2mzmrit8ev.fsf@starship.python.net> <1124196179.9673.12.camel@geddy.wooz.org> <2mvf26t56q.fsf@starship.python.net> <430230F8.3020405@v.loewis.de> <1f7befae050816124932f5c66@mail.gmail.com> <43024BC2.2010505@v.loewis.de> Message-ID: <17154.21885.668810.977155@montanaro.dyndns.org> Martin> Of course, Barry can only speak about the current availability Martin> of volunteers, which is quite good (especially since amk took Martin> over coordinating them) .... I don't know why, but the first image that popped into my mind was of amk beating a bunch of Hunchback of Notre Dame types (maybe more the Marty Feldman (*) hunchback types) into submission with a whip while one of them cried, "We'll do anything you ask, master. Just don't beat us again." The-beatings-will-continue-until-morale-improves-ly, y'rs, Skip (*) http://en.wikipedia.org/wiki/Marty_Feldman From walter at livinglogic.de Tue Aug 16 23:06:37 2005 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue, 16 Aug 2005 23:06:37 +0200 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <1f7befae05081518073433da62@mail.gmail.com> References: <42F61C03.6050703@v.loewis.de> <1f7befae05081518073433da62@mail.gmail.com> Message-ID: <1A846489-39FF-49D1-8AF0-5BA61F9277DF@livinglogic.de> Tim Peters wrote: > [Martin v. L?wis] > >> I have placed a new version of the PEP on >> >> http://www.python.org/peps/pep-0347.html >> > > ... > > +1 from me. But, I don't think my vote should count much, and (sorry) > Guido's even less: what do the people who frequently check in want? > That means people like you (Martin), Michael, Raymond, Walter, Fred. > ... plus the release manager(s). +1 from me for various reasons: * Subversion seems to be stable enough, and it's better than CVS which is enough for me. * The python.org machines can probably handle the load of *one* repository better then the SF machines that of several thousands. * Connectivity to python.org is much better then to cvs.sf.net (at least from here). * Our company repository might move to svn in the near future, so a Python svn repository would be a perfect playground to learn svn. ;) Bye, Walter D?rwald From tdelaney at avaya.com Wed Aug 17 01:53:20 2005 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Wed, 17 Aug 2005 09:53:20 +1000 Subject: [Python-Dev] PEP 347: Migration to Subversion Message-ID: <2773CAC687FD5F4689F526998C7E4E5F05CC2E@au3010avexu1.global.avaya.com> Tim Peters wrote: > [Martin v. L?wis] >> I would agree. However, there still is the debate of hosting the >> repository elsehwere. Some people (Anthony, Guido, Tim) would prefer >> to pay for it, instead of hosting it on svn.python.org. > > Not this Tim. Not this one either. I haven't actually used any of the various systems that much (work is ClearCase) so I have no opinions whatsoever. It's interesting reading though. Tim Delaney From gvanrossum at gmail.com Wed Aug 17 01:58:01 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 16 Aug 2005 16:58:01 -0700 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5F05CC2E@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5F05CC2E@au3010avexu1.global.avaya.com> Message-ID: Nor this Guido, FWIW (I think we shouldn't rule it out as an option, but I don't have any preferences). On 8/16/05, Delaney, Timothy (Tim) wrote: > Tim Peters wrote: > > > [Martin v. L?wis] > >> I would agree. However, there still is the debate of hosting the > >> repository elsehwere. Some people (Anthony, Guido, Tim) would prefer > >> to pay for it, instead of hosting it on svn.python.org. > > > > Not this Tim. > > Not this one either. I haven't actually used any of the various systems that much (work is ClearCase) so I have no opinions whatsoever. It's interesting reading though. > > Tim Delaney > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From raymond.hettinger at verizon.net Wed Aug 17 02:55:26 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 16 Aug 2005 20:55:26 -0400 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <2m4q9qunao.fsf@starship.python.net> Message-ID: <000801c5a2c6$5ca70080$af26c797@oemcomputer> [Michael Hudson] > I wonder if dir() should strip non-strings? -0 The behavior of dir() already a bit magical. Python is much simpler to comprehend if we have direct relationships like dir() and vars() corresponding as closely as possible to the object's dictionary. If someone injects non-strings into an attribute dictionary, why should dir() hide that fact? Likewise, we would have been better-off if ceval.c didn't pre-process data before handing it off to API functions (so that negative indices get handled the same way in operator module functions and in user defined methods, etc). Both Io and Lua have made a design principle out of keeping these relationships as direct as possible (i.e. a[b] always corresponds to the call a.__getitem__(b) with no intervening magic, etc.). The auto-exposure on my camera takes in nine data points and guesses whether the subject is backlit, whether there is a mix of light and dark, whether it is more important avoid blown highlights or to miss shadow detail, etc. The good news is that it often makes a decent guess. The bad news is that I've completely lost the ability to predict whether I've gotten a good shot based on the light conditions and camera settings. IOW, if you make the tools too smart, they become harder to use. Leica had it right all along. Raymond From ilya at bluefir.net Wed Aug 17 06:34:10 2005 From: ilya at bluefir.net (Ilya Sandler) Date: Tue, 16 Aug 2005 21:34:10 -0700 (PDT) Subject: [Python-Dev] remote debugging with pdb In-Reply-To: <24EEDE5B-4511-40D4-9C16-8A33C4ACE1C8@redivi.com> References: <20050808154503.GB28005@panix.com> <200508111802.44357.anthony@interlink.com.au> <24EEDE5B-4511-40D4-9C16-8A33C4ACE1C8@redivi.com> Message-ID: > One thing PDB needs is a mode that runs as a background thread and > opens up a socket so that another Python process can talk to it, for > embedded/remote/GUI debugging. There is a patch on SourceForge python.org/sf/721464 which allows pdb to read/write from/to arbitrary file objects. Would it answer some of your concerns (eg remote debugging)? The patch probably will not apply to the current code, but I guess, I could revive it if anyone thinks that it's worthwhile... What do you think? Ilya From kiko at async.com.br Wed Aug 17 16:02:18 2005 From: kiko at async.com.br (Christian Robottom Reis) Date: Wed, 17 Aug 2005 11:02:18 -0300 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) Message-ID: <20050817140217.GQ3389@www.async.com.br> In Launchpad (mainly because SQLObject is used) we end up with quite a few locals named id. Apart from the fact that naturally clobbering builtins is a bad idea, we get quite a few warnings when linting throughout the codebase. I've fixed these as I've found them, but today Andrew pointed out to me that this is noted in: http://www.python.org/doc/essays/ppt/regrets/PythonRegrets.ppt I wonder: is moving id() to sys doable in the 2.5 cycle, with a deprecation warning being raised for people using the builtin? We'd then phase it out in one of the latter 2.x versions. I've done some searching through my code and id() isn't the most-used builtin, so from my perspective the impact would be limited, but of course others might think otherwise. Is it worth writing a PEP for this, or is it crack? Take care, -- Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3376 0125 From raymond.hettinger at verizon.net Wed Aug 17 17:48:57 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Wed, 17 Aug 2005 11:48:57 -0400 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <20050817140217.GQ3389@www.async.com.br> Message-ID: <000a01c5a343$2f4deea0$5e01a044@oemcomputer> [Christian Robottom Reis] > I've done some searching through my code and id() isn't the most-used > builtin, so from my perspective the impact would be limited, but of > course others might think otherwise. > > Is it worth writing a PEP for this, or is it crack? FWIW, I use id() all the time and like having it as a builtin. Raymond From firemoth at gmail.com Wed Aug 17 18:32:42 2005 From: firemoth at gmail.com (Timothy Fitz) Date: Wed, 17 Aug 2005 12:32:42 -0400 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <20050817140217.GQ3389@www.async.com.br> References: <20050817140217.GQ3389@www.async.com.br> Message-ID: <972ec5bd05081709327fed099@mail.gmail.com> On 8/17/05, Christian Robottom Reis wrote: > I've done some searching through my code and id() isn't the most-used > builtin, so from my perspective the impact would be limited, but of > course others might think otherwise. All of my primary uses of id would not show up in such a search. id is handy when debugging, when using the interactive interpreter and temporarily in scripts (print id(something), something for when repr(something) doesn't show the id). In my experience teaching python, id at the interactive interpreter is invaluable, which is why any proposal to move it would get a -1. The fundamental issue is that I want to explain reference semantics well before I talk about packages and the associated import call. From jeremy at alum.mit.edu Wed Aug 17 18:37:42 2005 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Wed, 17 Aug 2005 12:37:42 -0400 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <972ec5bd05081709327fed099@mail.gmail.com> References: <20050817140217.GQ3389@www.async.com.br> <972ec5bd05081709327fed099@mail.gmail.com> Message-ID: I'd like to see the builtin id() removed so that I can use it as a local variable name without clashing with the builtin name. I certainly use the id() function, but not as often as I have a local variable I'd like to name id. The sys module seems like a natural place to put id(), since it is exposing something about the implementation of Python rather than something about the language; the language offers the is operator to check ids. Jeremy From reinhold-birkenfeld-nospam at wolke7.net Wed Aug 17 18:37:11 2005 From: reinhold-birkenfeld-nospam at wolke7.net (Reinhold Birkenfeld) Date: Wed, 17 Aug 2005 18:37:11 +0200 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <20050817140217.GQ3389@www.async.com.br> References: <20050817140217.GQ3389@www.async.com.br> Message-ID: Christian Robottom Reis wrote: > In Launchpad (mainly because SQLObject is used) we end up with quite a > few locals named id. Apart from the fact that naturally clobbering > builtins is a bad idea, we get quite a few warnings when linting > throughout the codebase. I've fixed these as I've found them, but today > Andrew pointed out to me that this is noted in: > > http://www.python.org/doc/essays/ppt/regrets/PythonRegrets.ppt > > I wonder: is moving id() to sys doable in the 2.5 cycle, with a > deprecation warning being raised for people using the builtin? We'd then > phase it out in one of the latter 2.x versions. > > I've done some searching through my code and id() isn't the most-used > builtin, so from my perspective the impact would be limited, but of > course others might think otherwise. > > Is it worth writing a PEP for this, or is it crack? As I can see, this is not going to happen before Py3k, as it is completely breaking backwards compatibility. As such, a PEP would be unnecessary. Reinhold -- Mail address is perfectly valid! From pedronis at strakt.com Wed Aug 17 18:50:29 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Wed, 17 Aug 2005 18:50:29 +0200 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: References: <20050817140217.GQ3389@www.async.com.br> <972ec5bd05081709327fed099@mail.gmail.com> Message-ID: <43036AD5.6010902@strakt.com> Jeremy Hylton wrote: > I'd like to see the builtin id() removed so that I can use it as a > local variable name without clashing with the builtin name. I > certainly use the id() function, but not as often as I have a local > variable I'd like to name id. The sys module seems like a natural > place to put id(), since it is exposing something about the > implementation of Python rather than something about the language; the > language offers the is operator to check ids. > it is worth to remember that id() functionality is not cheap for Python impls using moving GCs. Identity mappings would be less taxing. > Jeremy > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/pedronis%40strakt.com From barry at python.org Wed Aug 17 19:17:59 2005 From: barry at python.org (Barry Warsaw) Date: Wed, 17 Aug 2005 13:17:59 -0400 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: References: <20050817140217.GQ3389@www.async.com.br> <972ec5bd05081709327fed099@mail.gmail.com> Message-ID: <1124299079.23024.17.camel@geddy.wooz.org> On Wed, 2005-08-17 at 12:37, Jeremy Hylton wrote: > I'd like to see the builtin id() removed so that I can use it as a > local variable name without clashing with the builtin name. I > certainly use the id() function, but not as often as I have a local > variable I'd like to name id. Same here. > The sys module seems like a natural > place to put id(), since it is exposing something about the > implementation of Python rather than something about the language; the > language offers the is operator to check ids. +1 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050817/4e62bbf8/attachment.pgp From bcannon at gmail.com Wed Aug 17 19:21:59 2005 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 17 Aug 2005 10:21:59 -0700 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <1124299079.23024.17.camel@geddy.wooz.org> References: <20050817140217.GQ3389@www.async.com.br> <972ec5bd05081709327fed099@mail.gmail.com> <1124299079.23024.17.camel@geddy.wooz.org> Message-ID: On 8/17/05, Barry Warsaw wrote: > On Wed, 2005-08-17 at 12:37, Jeremy Hylton wrote: > > I'd like to see the builtin id() removed so that I can use it as a > > local variable name without clashing with the builtin name. I > > certainly use the id() function, but not as often as I have a local > > variable I'd like to name id. > > Same here. > > > The sys module seems like a natural > > place to put id(), since it is exposing something about the > > implementation of Python rather than something about the language; the > > language offers the is operator to check ids. > > +1 > -Barry +1 -Brett From nas at arctrix.com Wed Aug 17 19:40:32 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Wed, 17 Aug 2005 11:40:32 -0600 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: References: <20050817140217.GQ3389@www.async.com.br> Message-ID: <20050817174031.GA22541@mems-exchange.org> On Wed, Aug 17, 2005 at 06:37:11PM +0200, Reinhold Birkenfeld wrote: > As I can see, this is not going to happen before Py3k, as it is completely > breaking backwards compatibility. As such, a PEP would be unnecessary. We could add sys.id for 2.5 and remove __builtin__.id a some later time (e.g. for 3.0). Neil From facundobatista at gmail.com Wed Aug 17 19:49:25 2005 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 17 Aug 2005 14:49:25 -0300 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <20050817174031.GA22541@mems-exchange.org> References: <20050817140217.GQ3389@www.async.com.br> <20050817174031.GA22541@mems-exchange.org> Message-ID: On 8/17/05, Neil Schemenauer wrote: > On Wed, Aug 17, 2005 at 06:37:11PM +0200, Reinhold Birkenfeld wrote: > > As I can see, this is not going to happen before Py3k, as it is completely > > breaking backwards compatibility. As such, a PEP would be unnecessary. > > We could add sys.id for 2.5 and remove __builtin__.id a some later > time (e.g. for 3.0). +1 for adding it to sys in 2.5, removing the builtin one in 3.0. . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From firemoth at gmail.com Wed Aug 17 20:55:30 2005 From: firemoth at gmail.com (Timothy Fitz) Date: Wed, 17 Aug 2005 14:55:30 -0400 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <000801c5a2c6$5ca70080$af26c797@oemcomputer> References: <2m4q9qunao.fsf@starship.python.net> <000801c5a2c6$5ca70080$af26c797@oemcomputer> Message-ID: <972ec5bd05081711555e9ad129@mail.gmail.com> On 8/16/05, Raymond Hettinger wrote: > -0 The behavior of dir() already a bit magical. Python is much simpler > to comprehend if we have direct relationships like dir() and vars() > corresponding as closely as possible to the object's dictionary. If > someone injects non-strings into an attribute dictionary, why should > dir() hide that fact? Indeed, there seem to be two camps, those who want dir to reflect __dict__ and those who want dir to reflect attributes of an object. It seems to me that those who want dir to reflect __dict__ should just use __dict__ in the first place. However, in the case of dir handling non-strings, should dir handle non-valid identifiers as well, that is to say that while foo.__dict__[2] = ... is an obvious case what about foo.__dict__["1"] ? Right now the documentation says that it returns "attributes", and I would not consider non-strings to be attributes, so either the documentation or the implementation should rectify this disagreement. From gvanrossum at gmail.com Wed Aug 17 21:10:15 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 17 Aug 2005 12:10:15 -0700 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <972ec5bd05081711555e9ad129@mail.gmail.com> References: <2m4q9qunao.fsf@starship.python.net> <000801c5a2c6$5ca70080$af26c797@oemcomputer> <972ec5bd05081711555e9ad129@mail.gmail.com> Message-ID: On 8/17/05, Timothy Fitz wrote: > On 8/16/05, Raymond Hettinger wrote: > > -0 The behavior of dir() already a bit magical. Python is much simpler > > to comprehend if we have direct relationships like dir() and vars() > > corresponding as closely as possible to the object's dictionary. If > > someone injects non-strings into an attribute dictionary, why should > > dir() hide that fact? > > Indeed, there seem to be two camps, those who want dir to reflect __dict__ > and those who want dir to reflect attributes of an object. It seems to > me that those who want dir to reflect __dict__ should just use > __dict__ in the first place. Right. > However, in the case of dir handling non-strings, should dir handle > non-valid identifiers as well, that is to say that while > foo.__dict__[2] = ... is an obvious case what about foo.__dict__["1"] > ? See below. > Right now the documentation says that it returns "attributes", and I > would not consider non-strings to be attributes, so either the > documentation or the implementation should rectify this disagreement. I think that dir() should hide non-strings; these aren't attributes if you believe the definition that an attribute name is something acceptable to getattr() or setattr(). Following this definition, the string "1" is a valid attribute name (even though it's not a valid identifier), but the number 1 is not. Try it. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From raymond.hettinger at verizon.net Wed Aug 17 21:21:22 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Wed, 17 Aug 2005 15:21:22 -0400 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <972ec5bd05081711555e9ad129@mail.gmail.com> Message-ID: <001101c5a360$db2d63a0$3031c797@oemcomputer> [Timothy Fitz] > It seems to > me that those who want dir to reflect __dict__ should just use > __dict__ in the first place. The dir() builtin does quite a bit more than obj.__dict__.keys(). >>> class A(list): x = 1 >>> dir(A) ['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__delslice__', '__dict__', '__doc__', '__eq__', '__ge__', '__getattribute__', '__getitem__', '__getslice__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__setslice__', '__str__', '__weakref__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort', 'x'] >>> A.__dict__.keys() ['__dict__', 'x', '__module__', '__weakref__', '__doc__'] Raymond From gvanrossum at gmail.com Wed Aug 17 21:30:33 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 17 Aug 2005 12:30:33 -0700 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <001101c5a360$db2d63a0$3031c797@oemcomputer> References: <972ec5bd05081711555e9ad129@mail.gmail.com> <001101c5a360$db2d63a0$3031c797@oemcomputer> Message-ID: > [Timothy Fitz] > > It seems to > > me that those who want dir to reflect __dict__ should just use > > __dict__ in the first place. [Raymond] > The dir() builtin does quite a bit more than obj.__dict__.keys(). Well that's the whole point, right? We shouldn't conflate the two. I don't see this as an argument why it would be bad to delete non-string-keys found in __dict__ from dir()'s return value. I don't think that the equation set(x.__dict__) <= set(dir(x)) provides enough value to try and keep it. A more useful relationship is name in dir(x) <==> getattr(x, name) is valid -- --Guido van Rossum (home page: http://www.python.org/~guido/) From raymond.hettinger at verizon.net Wed Aug 17 22:21:16 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Wed, 17 Aug 2005 16:21:16 -0400 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: Message-ID: <001701c5a369$3911f960$3031c797@oemcomputer> > > [Timothy Fitz] > > > It seems to > > > me that those who want dir to reflect __dict__ should just use > > > __dict__ in the first place. > > [Raymond] > > The dir() builtin does quite a bit more than obj.__dict__.keys(). > > Well that's the whole point, right? Perhaps. I wasn't taking a position. Just noting that Timothy's comment over-simplified the relationship. > A more useful relationship is > > name in dir(x) <==> getattr(x, name) is valid That would be a useful invariant. Raymond From gvanrossum at gmail.com Wed Aug 17 22:46:27 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 17 Aug 2005 13:46:27 -0700 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <001701c5a369$3911f960$3031c797@oemcomputer> References: <001701c5a369$3911f960$3031c797@oemcomputer> Message-ID: [me] > > A more useful relationship is > > > > name in dir(x) <==> getattr(x, name) is valid [Raymond] > That would be a useful invariant. Well, the <== part can't really be guaranteed due to the existence of __getattr__ overriding (and all bets are off if __getattribute__ is overridden!), but apart from those, stripping non-strings in dir() would be a big help towards making the invariant true. So I'm +1 on that. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From caustin at spikesource.com Thu Aug 18 00:57:33 2005 From: caustin at spikesource.com (Calvin Austin) Date: Wed, 17 Aug 2005 15:57:33 -0700 Subject: [Python-Dev] A testing challenge Message-ID: <4303C0DD.5090903@spikesource.com> When was the last time someone thanked you for writing a test? I tried to think of the last time it happened to me and I can't remember. Well at Spikesource we want to thank you not just for helping the Python community but for your testing efforts too and we are running a participatory testing contest. This is a competition where there are no losers, every project gains if new tests are written. For more details see below, it is open worldwide. feel free to send questions to me. thanks calvin *_Open Testing Contest with Over $20,000 in Prizes_* Committers! SpikeSource is sponsoring a contest to help increase the participatory testing of open source software. Awards will be given to open source projects that have the greatest increase in code coverage from September 15 through December 31, 2005. Project sign-up is due by August 31^st and the contest begins on September 15^th . Visit http://www.spikesource.com/contest/ for complete details and to register your project. From eric at enthought.com Thu Aug 18 01:05:11 2005 From: eric at enthought.com (eric jones) Date: Wed, 17 Aug 2005 18:05:11 -0500 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <1124299079.23024.17.camel@geddy.wooz.org> References: <20050817140217.GQ3389@www.async.com.br> <972ec5bd05081709327fed099@mail.gmail.com> <1124299079.23024.17.camel@geddy.wooz.org> Message-ID: <4303C2A7.2030608@enthought.com> Barry Warsaw wrote: >On Wed, 2005-08-17 at 12:37, Jeremy Hylton wrote: > > >>I'd like to see the builtin id() removed so that I can use it as a >>local variable name without clashing with the builtin name. I >>certainly use the id() function, but not as often as I have a local >>variable I'd like to name id. >> >> > >Same here. > > > >>The sys module seems like a natural >>place to put id(), since it is exposing something about the >>implementation of Python rather than something about the language; the >>language offers the is operator to check ids. >> >> > >+1 >-Barry > > +1 eric From foom at fuhm.net Thu Aug 18 01:23:47 2005 From: foom at fuhm.net (James Y Knight) Date: Wed, 17 Aug 2005 19:23:47 -0400 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <972ec5bd05081711555e9ad129@mail.gmail.com> References: <2m4q9qunao.fsf@starship.python.net> <000801c5a2c6$5ca70080$af26c797@oemcomputer> <972ec5bd05081711555e9ad129@mail.gmail.com> Message-ID: <7F4E4259-CE92-4AED-823F-5E06BECAED6C@fuhm.net> On Aug 17, 2005, at 2:55 PM, Timothy Fitz wrote: > On 8/16/05, Raymond Hettinger wrote: > >> -0 The behavior of dir() already a bit magical. Python is much >> simpler >> to comprehend if we have direct relationships like dir() and vars() >> corresponding as closely as possible to the object's dictionary. If >> someone injects non-strings into an attribute dictionary, why should >> dir() hide that fact? >> > > Indeed, there seem to be two camps, those who want dir to reflect > __dict__ > and those who want dir to reflect attributes of an object. It seems to > me that those who want dir to reflect __dict__ should just use > __dict__ in the first place. > > However, in the case of dir handling non-strings, should dir handle > non-valid identifiers as well, that is to say that while > foo.__dict__[2] = ... is an obvious case what about foo.__dict__["1"] > ? > > Right now the documentation says that it returns "attributes", and I > would not consider non-strings to be attributes, so either the > documentation or the implementation should rectify this disagreement. > I initially was going to say no, there's no reason to restrict your idea of "attributes" to be purely strings, because surely you could use non-strings as attributes if you wished to. But Python proves me wrong: >>> class X: pass >>> X.__dict__[1] = 5 >>> dir(X) [1, '__doc__', '__module__'] >>> getattr(X, 1) TypeError: getattr(): attribute name must be string If dir() is supposed to return the list of attributes, it does seem logical that it should be possible to pass those names into getattr. I think I'd actually call that a defect in getattr() that it doesn't allow non-string attributes, not a defect in dir(). Ooh...even more annoying, it doesn't even allow unicode attributes that use characters outside the default encoding (ASCII). But either way, there's absolutely no reason to worry about the attribute string being a valid identifier. That's pretty much only a concern for tab-completion in python shells. James From paul at pfdubois.com Thu Aug 18 05:05:32 2005 From: paul at pfdubois.com (Paul F. Dubois) Date: Wed, 17 Aug 2005 20:05:32 -0700 Subject: [Python-Dev] Deprecating builtin id (and moving it to, sys()) Message-ID: <4303FAFC.3070204@pfdubois.com> -1 for this proposal from me. I use id some and therefore the change would break some of my code. Breaking existing code without some overwhelming reason is a very bad idea, in my opinion. The reason cited here, that the name is so natural that one is tempted to use it, applies to many builtins. Ever written dict = {} and then said to yourself, gee, that isn't a very good idea? I have. Besides that, the fact that an object has an identity, behaviors, and data is primary. For teaching beginners id() is important. Paul From anthony at interlink.com.au Thu Aug 18 06:09:16 2005 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 18 Aug 2005 14:09:16 +1000 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <20050817140217.GQ3389@www.async.com.br> References: <20050817140217.GQ3389@www.async.com.br> Message-ID: <200508181409.17431.anthony@interlink.com.au> On Thursday 18 August 2005 00:02, Christian Robottom Reis wrote: > I wonder: is moving id() to sys doable in the 2.5 cycle, with a > deprecation warning being raised for people using the builtin? We'd then > phase it out in one of the latter 2.x versions. I'm neutral on putting id() also into sys. I'm -1 on either issuing a deprecation warning or, worse yet, removing the id() builtin. The warnings system is expensive to call, and I know from a brief look at a bunch of code that I use id() inside some tight inner loops. Removing it entirely is gratuitous breakage, for a not very high payoff. If you _really_ want to call a local variable 'id' you can (but shouldn't). You also can't/shouldn't call a variable 'class', 'def', or 'len' -- but I don't see any movement to allow these... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From mal at egenix.com Thu Aug 18 09:36:14 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 18 Aug 2005 09:36:14 +0200 Subject: [Python-Dev] SWIG and rlcompleter In-Reply-To: <7F4E4259-CE92-4AED-823F-5E06BECAED6C@fuhm.net> References: <2m4q9qunao.fsf@starship.python.net> <000801c5a2c6$5ca70080$af26c797@oemcomputer> <972ec5bd05081711555e9ad129@mail.gmail.com> <7F4E4259-CE92-4AED-823F-5E06BECAED6C@fuhm.net> Message-ID: <43043A6E.5020109@egenix.com> James Y Knight wrote: > On Aug 17, 2005, at 2:55 PM, Timothy Fitz wrote: > > >>On 8/16/05, Raymond Hettinger wrote: >> >> >>>-0 The behavior of dir() already a bit magical. Python is much >>>simpler >>>to comprehend if we have direct relationships like dir() and vars() >>>corresponding as closely as possible to the object's dictionary. If >>>someone injects non-strings into an attribute dictionary, why should >>>dir() hide that fact? >>> >> >>Indeed, there seem to be two camps, those who want dir to reflect >>__dict__ >>and those who want dir to reflect attributes of an object. It seems to >>me that those who want dir to reflect __dict__ should just use >>__dict__ in the first place. >> >>However, in the case of dir handling non-strings, should dir handle >>non-valid identifiers as well, that is to say that while >>foo.__dict__[2] = ... is an obvious case what about foo.__dict__["1"] >>? >> >>Right now the documentation says that it returns "attributes", and I >>would not consider non-strings to be attributes, so either the >>documentation or the implementation should rectify this disagreement. >> > > > I initially was going to say no, there's no reason to restrict your > idea of "attributes" to be purely strings, because surely you could > use non-strings as attributes if you wished to. But Python proves me > wrong: > >>> class X: pass > >>> X.__dict__[1] = 5 > >>> dir(X) > [1, '__doc__', '__module__'] > >>> getattr(X, 1) > TypeError: getattr(): attribute name must be string > > If dir() is supposed to return the list of attributes, it does seem > logical that it should be possible to pass those names into getattr. > I think I'd actually call that a defect in getattr() that it doesn't > allow non-string attributes, not a defect in dir(). Ooh...even more > annoying, it doesn't even allow unicode attributes that use > characters outside the default encoding (ASCII). Which is quite natural: Python doesn't allow any non-ASCII identifiers either :-) > But either way, there's absolutely no reason to worry about the > attribute string being a valid identifier. That's pretty much only a > concern for tab-completion in python shells. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 18 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From martin at v.loewis.de Thu Aug 18 11:39:37 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 18 Aug 2005 11:39:37 +0200 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <200508181409.17431.anthony@interlink.com.au> References: <20050817140217.GQ3389@www.async.com.br> <200508181409.17431.anthony@interlink.com.au> Message-ID: <43045759.80806@v.loewis.de> Anthony Baxter wrote: > Removing it entirely is gratuitous breakage, for a not very high payoff. If > you _really_ want to call a local variable 'id' you can (but shouldn't). > You also can't/shouldn't call a variable 'class', 'def', or 'len' -- but I > don't see any movement to allow these... This is getting off-topic, but... In C#, you can: you write @class, @void, @return. Apparently, this is so that you can access arbitrary COM objects (which may happen to use C# keywords as method names). Of course, we would put an underscore after the name in that case. Regards, Martin From ianb at colorstudy.com Thu Aug 18 17:54:49 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 18 Aug 2005 10:54:49 -0500 Subject: [Python-Dev] PEP 309: Partial method application Message-ID: <4304AF49.6030007@colorstudy.com> I missed the discussion on this (http://www.python.org/peps/pep-0309.html), but then 2.5 isn't out yet. I think partial() misses an important use case of method getting, for instance: lst = ['A', 'b', 'C'] lst.sort(key=partialmethod('lower')) Which sorts by lower-case. Of course you can use str.lower, except you'll have unnecessarily enforced a type (and excluded Unicode). So you are left with lambda x: x.lower(). Here's an implementation: def partialmethod(method, *args, **kw): def call(obj, *more_args, **more_kw): call_kw = kw.copy() call_kw.update(more_kw) return getattr(obj, method)(*(arg+more_args), **call_kw) return call This is obviously related to partial(). Maybe this implementation should be a classmethod or function attribute, partial.method(). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From raymond.hettinger at verizon.net Thu Aug 18 18:00:06 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Thu, 18 Aug 2005 12:00:06 -0400 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <4304AF49.6030007@colorstudy.com> Message-ID: <003401c5a40d$ea5694c0$8b24c797@oemcomputer> [Ian Bicking] > I think partial() misses an important use case of method getting, for > instance: > > lst = ['A', 'b', 'C'] > lst.sort(key=partialmethod('lower')) We've already got one: lst.sort(key=operator.attrgetter('lower')) Raymond From gvanrossum at gmail.com Thu Aug 18 18:22:01 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 18 Aug 2005 09:22:01 -0700 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <200508181409.17431.anthony@interlink.com.au> References: <20050817140217.GQ3389@www.async.com.br> <200508181409.17431.anthony@interlink.com.au> Message-ID: On 8/17/05, Anthony Baxter wrote: > If you _really_ want to call a local variable 'id' you can (but shouldn't). Disagreed. The built-in namespace is searched last for a reason -- the design is such that if you don't care for a particular built-in you don't need to know about it. > You also can't/shouldn't call a variable 'class', 'def', or 'len' -- but I > don't see any movement to allow these... Please don't propagate the confusion between reserved keywords and built-in names! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From steven.bethard at gmail.com Thu Aug 18 18:34:00 2005 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 18 Aug 2005 10:34:00 -0600 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <003401c5a40d$ea5694c0$8b24c797@oemcomputer> References: <4304AF49.6030007@colorstudy.com> <003401c5a40d$ea5694c0$8b24c797@oemcomputer> Message-ID: Raymond Hettinger wrote: > [Ian Bicking] > > I think partial() misses an important use case of method getting, for > > instance: > > > > lst = ['A', 'b', 'C'] > > lst.sort(key=partialmethod('lower')) > > We've already got one: > > lst.sort(key=operator.attrgetter('lower')) Doesn't that just sort on the str.lower or unicode.lower method object? py> sorted(['A', u'b', 'C'], key=operator.attrgetter('lower')) [u'b', 'C', 'A'] py> sorted(['A', u'b', 'C'], key=partialmethod('lower')) # after fixing arg -> args bug ['A', u'b', 'C'] STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy From raymond.hettinger at verizon.net Thu Aug 18 18:43:07 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Thu, 18 Aug 2005 12:43:07 -0400 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: Message-ID: <003901c5a413$e997c9e0$8b24c797@oemcomputer> > > [Ian Bicking] > > > I think partial() misses an important use case of method getting, for > > > instance: > > > > > > lst = ['A', 'b', 'C'] > > > lst.sort(key=partialmethod('lower')) > > > > We've already got one: > > > > lst.sort(key=operator.attrgetter('lower')) > > Doesn't that just sort on the str.lower or unicode.lower method object? My mistake. It sorts on the bound method rather than the results of applying that method. Raymond From ianb at colorstudy.com Thu Aug 18 20:05:54 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 18 Aug 2005 13:05:54 -0500 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <003901c5a413$e997c9e0$8b24c797@oemcomputer> References: <003901c5a413$e997c9e0$8b24c797@oemcomputer> Message-ID: <4304CE02.7080308@colorstudy.com> Raymond Hettinger wrote: >>>>instance: >>>> >>>> lst = ['A', 'b', 'C'] >>>> lst.sort(key=partialmethod('lower')) >>> >>>We've already got one: >>> >>> lst.sort(key=operator.attrgetter('lower')) >> >>Doesn't that just sort on the str.lower or unicode.lower method >> object? > > My mistake. It sorts on the bound method rather than the results of > applying that method. Then I thought it might be right to do partial(operator.attrgetter('lower')). This, however, accomplishes exactly nothing. I only decided this after actually trying it, though upon reflection partial(function) always accomplishes nothing. I don't have any conclusion from this, but only mention it to demonstrate that callables on top of callables are likely to confuse. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From martin at v.loewis.de Thu Aug 18 21:40:19 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 18 Aug 2005 21:40:19 +0200 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <4304AF49.6030007@colorstudy.com> References: <4304AF49.6030007@colorstudy.com> Message-ID: <4304E423.9050005@v.loewis.de> Ian Bicking wrote: > > lst = ['A', 'b', 'C'] > lst.sort(key=partialmethod('lower')) > > Which sorts by lower-case. Of course you can use str.lower, except > you'll have unnecessarily enforced a type (and excluded Unicode). So > you are left with lambda x: x.lower(). For this specific case, you can use string.lower (which is exactly what the lambda function does). As for the more general proposal: -1 on more places to pass strings to denote method/function/class names. These are ugly to type. What I think you want is not a partial method, instead, you want to turn a method into a standard function, and in a 'virtual' way. So I would propose the syntax lst.sort(key=virtual.lower) # where virtual is functional.virtual As for extending PEP 309: This PEP deliberately abstained from other ways of currying, and instead only introduced the functional module. If you want to see "lazy functions" in the standard library, you should write a new PEP (unless there is an easy agreement about a single right way to do this, which I don't see). Regards, Martin P.S. It's not even clear that this should be added to functional, as attrgetter and itemgetter are already in operator. But, perhaps, they should be in functional. From shane at hathawaymix.org Thu Aug 18 22:13:31 2005 From: shane at hathawaymix.org (Shane Hathaway) Date: Thu, 18 Aug 2005 14:13:31 -0600 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <4304E423.9050005@v.loewis.de> References: <4304AF49.6030007@colorstudy.com> <4304E423.9050005@v.loewis.de> Message-ID: <4304EBEB.9000300@hathawaymix.org> Martin v. L?wis wrote: > So I would propose the syntax > > lst.sort(key=virtual.lower) # where virtual is functional.virtual Ooh, may I say that idea is interesting! It's easy to implement, too: class virtual: def __getattr__(self, name): return lambda obj: getattr(obj, name)() virtual = virtual() Shane From gvanrossum at gmail.com Thu Aug 18 22:17:16 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 18 Aug 2005 13:17:16 -0700 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <4304E423.9050005@v.loewis.de> References: <4304AF49.6030007@colorstudy.com> <4304E423.9050005@v.loewis.de> Message-ID: On 8/18/05, "Martin v. L?wis" wrote: > As for the more general proposal: -1 on more places to pass strings to > denote method/function/class names. These are ugly to type. Agreed. > What I think you want is not a partial method, instead, you want to > turn a method into a standard function, and in a 'virtual' way. > > So I would propose the syntax > > lst.sort(key=virtual.lower) # where virtual is functional.virtual I like this, but would hope for a different name -- the poor word 'virtual' has been abused enough by C++. > P.S. It's not even clear that this should be added to functional, > as attrgetter and itemgetter are already in operator. But, perhaps, > they should be in functional. They feel related to attrgetter more than to partial. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bcannon at gmail.com Thu Aug 18 22:46:00 2005 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 18 Aug 2005 13:46:00 -0700 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: References: <4304AF49.6030007@colorstudy.com> <4304E423.9050005@v.loewis.de> Message-ID: On 8/18/05, Guido van Rossum wrote: > On 8/18/05, "Martin v. L?wis" wrote: > > As for the more general proposal: -1 on more places to pass strings to > > denote method/function/class names. These are ugly to type. > > Agreed. > > > What I think you want is not a partial method, instead, you want to > > turn a method into a standard function, and in a 'virtual' way. > > > > So I would propose the syntax > > > > lst.sort(key=virtual.lower) # where virtual is functional.virtual > > I like this, but would hope for a different name -- the poor word > 'virtual' has been abused enough by C++. > Yeah, me too. Possible name are 'delayed', 'lazyattr', or just plain 'lazy' since it reminds me of Haskell. > > P.S. It's not even clear that this should be added to functional, > > as attrgetter and itemgetter are already in operator. But, perhaps, > > they should be in functional. > > They feel related to attrgetter more than to partial. > True, but the idea of lazy evaluation, at least for me, reminds me more of functional languages and thus the functional module. Oh, when should we think of putting reduce into functional? I remember this was discussed when it was realized reduce was the only functional built-in that is not covered by itertools or listcomps. -Brett From raymond.hettinger at verizon.net Thu Aug 18 22:52:36 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Thu, 18 Aug 2005 16:52:36 -0400 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: Message-ID: <000c01c5a436$e232daa0$8b24c797@oemcomputer> [Guido] > They feel related to attrgetter more than to partial. That suggests operator.methodcall() From ianb at colorstudy.com Thu Aug 18 22:57:38 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 18 Aug 2005 15:57:38 -0500 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: References: <4304AF49.6030007@colorstudy.com> <4304E423.9050005@v.loewis.de> Message-ID: <4304F642.7040508@colorstudy.com> Brett Cannon wrote: >>>What I think you want is not a partial method, instead, you want to >>>turn a method into a standard function, and in a 'virtual' way. >>> >>>So I would propose the syntax >>> >>> lst.sort(key=virtual.lower) # where virtual is functional.virtual >> >>I like this, but would hope for a different name -- the poor word >>'virtual' has been abused enough by C++. >> > > > Yeah, me too. Possible name are 'delayed', 'lazyattr', or just plain > 'lazy' since it reminds me of Haskell. I don't think there's anything particularly lazy about it. It's like a compliment of attrgetter. Where attrgetter is an inversion of getattr, partialmethod is an inversion of... well, of something that currently has no name. There's kind of an implicit operation in obj.method() -- people will generally read that as a "method call", not as the retrieval of a bound method and later invocation of that method. I think that is why it's so hard to figure out how to represent this in terms of something like attrgetter -- we try to invert something (a method call) that doesn't exist in the language. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From steven.bethard at gmail.com Thu Aug 18 23:20:09 2005 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 18 Aug 2005 15:20:09 -0600 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <4304EBEB.9000300@hathawaymix.org> References: <4304AF49.6030007@colorstudy.com> <4304E423.9050005@v.loewis.de> <4304EBEB.9000300@hathawaymix.org> Message-ID: Martin v. L?wis wrote: > So I would propose the syntax > > lst.sort(key=virtual.lower) # where virtual is functional.virtual Shane Hathaway wrote: > class virtual: > def __getattr__(self, name): > return lambda obj: getattr(obj, name)() > virtual = virtual() I think (perhaps because of the name) that this could be confusing. I don't have any intuition that "virtual.lower" would return a function that calls the "lower" attribute instead of returning a function that simply accesses that attribute. If we're going to move away from the itemgetter() and attrgetter() style, then we should be consistent about it and provide a solution (or solutions) that answers all of these problems: obj.attr obj.attr(*args, **kwargs) obj[key] I'm not sure that there is a clean/obvious way to do this. STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy From ncoghlan at gmail.com Fri Aug 19 00:43:06 2005 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Aug 2005 08:43:06 +1000 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: References: <4304AF49.6030007@colorstudy.com> <4304E423.9050005@v.loewis.de> Message-ID: <43050EFA.6070702@gmail.com> Brett Cannon wrote: >>>What I think you want is not a partial method, instead, you want to >>>turn a method into a standard function, and in a 'virtual' way. >>> >>>So I would propose the syntax >>> >>> lst.sort(key=virtual.lower) # where virtual is functional.virtual >> >>I like this, but would hope for a different name -- the poor word >>'virtual' has been abused enough by C++. > > Yeah, me too. Possible name are 'delayed', 'lazyattr', or just plain > 'lazy' since it reminds me of Haskell. Hmm, "methodcall"? As in: lst.sort(key=methodcall.lower) Where "methodcall" is something like what Shane described: class methodcall: def __getattr__(self, name): def delayedcall(*args, **kwds): return getattr(args[0], name)(*args[1:], **kwds) return delayedcall methodcall = methodcall() > > Oh, when should we think of putting reduce into functional? I > remember this was discussed when it was realized reduce was the only > functional built-in that is not covered by itertools or listcomps. I expected functional.map, functional.filter and functional.reduce to all exist in 2.5. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com From bcannon at gmail.com Fri Aug 19 01:05:17 2005 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 18 Aug 2005 16:05:17 -0700 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <43050EFA.6070702@gmail.com> References: <4304AF49.6030007@colorstudy.com> <4304E423.9050005@v.loewis.de> <43050EFA.6070702@gmail.com> Message-ID: On 8/18/05, Nick Coghlan wrote: > Brett Cannon wrote: > > Oh, when should we think of putting reduce into functional? I > > remember this was discussed when it was realized reduce was the only > > functional built-in that is not covered by itertools or listcomps. > > I expected functional.map, functional.filter and functional.reduce to all > exist in 2.5. > Itertools covers map, filter is covered by genexps. 'reduce' is the only one that does not have an equivalent anywhere. I guess we could cross-link itertools.map into functional.map, but I would rather just mention in the docs of one that it is located in the other module. And filter is just not worth it; that can definitely be covered in the docs of the module. -Brett From jcarlson at uci.edu Fri Aug 19 02:09:09 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 18 Aug 2005 17:09:09 -0700 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: References: <4304EBEB.9000300@hathawaymix.org> Message-ID: <20050818162112.788A.JCARLSON@uci.edu> Steven Bethard wrote: > > Martin v. L?wis wrote: > > So I would propose the syntax > > > > lst.sort(key=virtual.lower) # where virtual is functional.virtual > > Shane Hathaway wrote: > > class virtual: > > def __getattr__(self, name): > > return lambda obj: getattr(obj, name)() > > virtual = virtual() > > I think (perhaps because of the name) that this could be confusing. I > don't have any intuition that "virtual.lower" would return a function > that calls the "lower" attribute instead of returning a function that > simply accesses that attribute. > > If we're going to move away from the itemgetter() and attrgetter() > style, then we should be consistent about it and provide a solution > (or solutions) that answers all of these problems: > obj.attr > obj.attr(*args, **kwargs) > obj[key] > I'm not sure that there is a clean/obvious way to do this. I thought that: operator.attrgetter() was for obj.attr operator.itemgetter() was for obj[integer_index] That's almost all the way there. All that remains is to have something that gets any key (not just integers) and which handles function calls. In terms of the function call semantics, what about: class methodcall: def __getattr__(self, name, *args, **kwds): def delayedcall(obj): return getattr(obj, name)(*args, **kwds) return delayedcall methodcall = methodcall() - Josiah From steven.bethard at gmail.com Fri Aug 19 07:33:36 2005 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 18 Aug 2005 23:33:36 -0600 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <20050818162112.788A.JCARLSON@uci.edu> References: <4304EBEB.9000300@hathawaymix.org> <20050818162112.788A.JCARLSON@uci.edu> Message-ID: Josiah Carlson wrote: > Steven Bethard wrote: > > If we're going to move away from the itemgetter() and attrgetter() > > style, then we should be consistent about it and provide a solution > > (or solutions) that answers all of these problems: > > obj.attr > > obj.attr(*args, **kwargs) > > obj[key] > > I'm not sure that there is a clean/obvious way to do this. > > I thought that: > operator.attrgetter() was for obj.attr > operator.itemgetter() was for obj[integer_index] My point exactly. If we're sticking to the same style, I would expect that for obj.method(*args, **kwargs) we would have something like: operator.methodcaller('method', *args, **kwargs) The proposal by Martin v. L?wis is that this should instead look something like: methodcall.method(*args, **kwargs) which is a departure from the current attrgetter() and itemgetter() idiom. I'm not objecting to this approach, by the way. I think with the right name, it would probably read well. I just think that we should try to be consistent one way or the other. If we go with Martin v. L?wis's suggestion, I would then expect that the corrolates to attrgetter() and itemgetter() would also be included, e.g.: attrget.attr (for obj.attr) itemget[key] (for obj[key]) STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy From martin at v.loewis.de Fri Aug 19 07:59:38 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 19 Aug 2005 07:59:38 +0200 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: References: <4304EBEB.9000300@hathawaymix.org> <20050818162112.788A.JCARLSON@uci.edu> Message-ID: <4305754A.4040900@v.loewis.de> Steven Bethard wrote: >>I thought that: >> operator.attrgetter() was for obj.attr >> operator.itemgetter() was for obj[integer_index] > > > My point exactly. If we're sticking to the same style, I would expect that for > obj.method(*args, **kwargs) > we would have something like: > operator.methodcaller('method', *args, **kwargs) You might be missing one aspect of attrgetter, though. I can have f = operator.attrgetter('name', 'age') and then f(person) gives me (person.name, person.age). Likewise for itemgetter(1,2,3). Extending this to methodcaller is not natural; you would have x=methodcaller(('open',['foo','r'],{}),('read',[100],{}), ('close',[],{})) and then x(somestorage) (I know this is not the typical open/read/close pattern, where you would normally call read on what open returns) It might be that there is no use case for a multi-call methodgetter; I just point out that a single-call methodgetter would *not* be in the same style as attrgetter and itemgetter. > attrget.attr (for obj.attr) > itemget[key] (for obj[key]) I agree that would be consistent. These also wouldn't allow to get multiple items and indices. I don't know what the common use for attrgetter is: one or more attributes? Regards, Martin From steven.bethard at gmail.com Fri Aug 19 09:14:17 2005 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 19 Aug 2005 01:14:17 -0600 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <4305754A.4040900@v.loewis.de> References: <4304EBEB.9000300@hathawaymix.org> <20050818162112.788A.JCARLSON@uci.edu> <4305754A.4040900@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Steven Bethard wrote: > >>I thought that: > >> operator.attrgetter() was for obj.attr > >> operator.itemgetter() was for obj[integer_index] > > > > > > My point exactly. If we're sticking to the same style, I would expect that for > > obj.method(*args, **kwargs) > > we would have something like: > > operator.methodcaller('method', *args, **kwargs) > > You might be missing one aspect of attrgetter, though. I can have > > f = operator.attrgetter('name', 'age') > > and then f(person) gives me (person.name, person.age). Likewise for > itemgetter(1,2,3). [snip] > I don't know what the common use for > attrgetter is: one or more attributes? Well, in current Python code, I'd be willing to wager that it's one, no more, since Python 2.4 only supports a single argument to itemgetter and attrgetter. Of course, when Python 2.5 comes out, it's certainly possible that the multi-argument forms will become commonplace. I agree that an operator.methodcaller() shouldn't try to support multiple methods. OTOH, the syntax methodcall.method(*args, **kwargs) doesn't really lend itself to multiple methods either. STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy From jcarlson at uci.edu Fri Aug 19 09:37:43 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 19 Aug 2005 00:37:43 -0700 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: References: <4305754A.4040900@v.loewis.de> Message-ID: <20050819003609.789B.JCARLSON@uci.edu> Steven Bethard wrote: > I agree that an operator.methodcaller() shouldn't try to support > multiple methods. OTOH, the syntax > methodcall.method(*args, **kwargs) > doesn't really lend itself to multiple methods either. But that's OK, we don't want to be calling multiple methods anyways, do we? I'd personally like to see an example it makes sense if someone says that we do. - Josiah From raymond.hettinger at verizon.net Fri Aug 19 18:39:18 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Fri, 19 Aug 2005 12:39:18 -0400 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <20050819003609.789B.JCARLSON@uci.edu> Message-ID: <002501c5a4dc$8b521560$3f25a044@oemcomputer> [Steven Bethard] > > I agree that an operator.methodcaller() shouldn't try to support > > multiple methods. OTOH, the syntax > > methodcall.method(*args, **kwargs) > > doesn't really lend itself to multiple methods either. [Josiah Carlson] > But that's OK, we don't want to be calling multiple methods anyways, do > we? I'd personally like to see an example it makes sense if someone > says that we do. If an obvious syntax doesn't emerge, don't fret. The most obvious approach is to define a regular Python function and supply that function to the key= argument for list.sort() or sorted(). A virtue of the key= argument was reducing O(n log n) calls to just O(n). Further speed-ups are a false economy. So there's no need to twist syntax into knots just to get a C based method calling function. Likewise with map(), if a new function doesn't fit neatly, take that as a cue to be writing a plain for-loop. Raymond From martin at v.loewis.de Fri Aug 19 22:08:23 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 19 Aug 2005 22:08:23 +0200 Subject: [Python-Dev] PEP 309: Partial method application In-Reply-To: <20050819003609.789B.JCARLSON@uci.edu> References: <4305754A.4040900@v.loewis.de> <20050819003609.789B.JCARLSON@uci.edu> Message-ID: <43063C37.8040808@v.loewis.de> Josiah Carlson wrote: > Steven Bethard wrote: > >>I agree that an operator.methodcaller() shouldn't try to support >>multiple methods. OTOH, the syntax >> methodcall.method(*args, **kwargs) >>doesn't really lend itself to multiple methods either. > > > But that's OK, we don't want to be calling multiple methods anyways, do > we? I'd personally like to see an example it makes sense if someone > says that we do. Several people argued that the version with a string method name should be added "for consistency". I only pointed out that doing so would not be completely consistent. Regards, Martin From jeremy at alum.mit.edu Fri Aug 19 23:15:15 2005 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri, 19 Aug 2005 17:15:15 -0400 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: References: <20050817140217.GQ3389@www.async.com.br> <200508181409.17431.anthony@interlink.com.au> Message-ID: On 8/18/05, Guido van Rossum wrote: > On 8/17/05, Anthony Baxter wrote: > > If you _really_ want to call a local variable 'id' you can (but shouldn't). > > Disagreed. The built-in namespace is searched last for a reason -- the > design is such that if you don't care for a particular built-in you > don't need to know about it. In practice, it causes much confusion if you ever use a local variable that has the same name as the built-in namespace. If you intend to use id as a variable, it leads to confusing messages when a typo or editing error accidentally removes the definition, because the name will still be defined for you. It also leads to confusion when you later want to use the builtin in the same module or function (or in the debugger). If Python defines the name, I don't want to provide a redefinition. Jeremy From gvanrossum at gmail.com Sat Aug 20 06:00:17 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Fri, 19 Aug 2005 21:00:17 -0700 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: References: <20050817140217.GQ3389@www.async.com.br> <200508181409.17431.anthony@interlink.com.au> Message-ID: On 8/19/05, Jeremy Hylton wrote: > On 8/18/05, Guido van Rossum wrote: > > On 8/17/05, Anthony Baxter wrote: > > > If you _really_ want to call a local variable 'id' you can (but shouldn't). > > > > Disagreed. The built-in namespace is searched last for a reason -- the > > design is such that if you don't care for a particular built-in you > > don't need to know about it. > > In practice, it causes much confusion if you ever use a local variable > that has the same name as the built-in namespace. If you intend to > use id as a variable, it leads to confusing messages when a typo or > editing error accidentally removes the definition, because the name > will still be defined for you. It also leads to confusion when you > later want to use the builtin in the same module or function (or in > the debugger). If Python defines the name, I don't want to provide a > redefinition. This has startled me a few times, but never for more than 30 seconds. In correct code there sure isn't any confusion. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Sat Aug 20 10:48:11 2005 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat, 20 Aug 2005 18:48:11 +1000 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: References: <20050817140217.GQ3389@www.async.com.br> <200508181409.17431.anthony@interlink.com.au> Message-ID: <200508201848.13750.anthony@interlink.com.au> On Friday 19 August 2005 02:22, Guido van Rossum wrote: > On 8/17/05, Anthony Baxter wrote: > > If you _really_ want to call a local variable 'id' you can (but > > shouldn't). > > Disagreed. The built-in namespace is searched last for a reason -- the > design is such that if you don't care for a particular built-in you > don't need to know about it. I'm not sure what you're disagreeing with. Are you saying you _can't_ call a variable 'id', or that it's OK to do this? > > You also can't/shouldn't call a variable 'class', 'def', or 'len' -- but > > I don't see any movement to allow these... > > Please don't propagate the confusion between reserved keywords and > built-in names! It's not a matter of 'confusion', more that there are some names you can't or shouldn't use in Python. When coding twisted, often the most obvious 'short' name for a Deferred is 'def', but of course that doesn't work. Anthony From gvanrossum at gmail.com Sat Aug 20 18:02:25 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat, 20 Aug 2005 09:02:25 -0700 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: <200508201848.13750.anthony@interlink.com.au> References: <20050817140217.GQ3389@www.async.com.br> <200508181409.17431.anthony@interlink.com.au> <200508201848.13750.anthony@interlink.com.au> Message-ID: On 8/20/05, Anthony Baxter wrote: > On Friday 19 August 2005 02:22, Guido van Rossum wrote: > > On 8/17/05, Anthony Baxter wrote: > > > If you _really_ want to call a local variable 'id' you can (but > > > shouldn't). > > > > Disagreed. The built-in namespace is searched last for a reason -- the > > design is such that if you don't care for a particular built-in you > > don't need to know about it. > > I'm not sure what you're disagreeing with. Are you saying you _can't_ call > a variable 'id', or that it's OK to do this? That it's OK. > > > You also can't/shouldn't call a variable 'class', 'def', or 'len' -- but > > > I don't see any movement to allow these... > > > > Please don't propagate the confusion between reserved keywords and > > built-in names! > > It's not a matter of 'confusion', more that there are some names you can't > or shouldn't use in Python. When coding twisted, often the most obvious > 'short' name for a Deferred is 'def', but of course that doesn't work. My point is that there are two reasons for not using such a name. With 'def', you *can't*. With 'len', you *could* (but it would be unwise). With 'id', IMO it's okay. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvanrossum at gmail.com Sat Aug 20 18:14:51 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat, 20 Aug 2005 09:14:51 -0700 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: References: <2773CAC687FD5F4689F526998C7E4E5F05CC2E@au3010avexu1.global.avaya.com> Message-ID: I'm ready to accept te general idea of moving to subversion and away from SourceForge. On the hosting issue, I'm still neutral -- I expect we'll be able to support the current developer crowd easily on svn.python.org, but if we ever find ther are resource problems (either people or bandwidth etc.) I just received a recommendation for wush.net which specializes in svn hosting. $90/month for 5 Gb of disk space sounds like a good deal and easily within the PSF budget. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bob at redivi.com Sat Aug 20 18:45:05 2005 From: bob at redivi.com (Bob Ippolito) Date: Sat, 20 Aug 2005 06:45:05 -1000 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: References: <2773CAC687FD5F4689F526998C7E4E5F05CC2E@au3010avexu1.global.avaya.com> Message-ID: On Aug 20, 2005, at 6:14 AM, Guido van Rossum wrote: > I'm ready to accept te general idea of moving to subversion and away > from SourceForge. > > On the hosting issue, I'm still neutral -- I expect we'll be able to > support the current developer crowd easily on svn.python.org, but if > we ever find ther are resource problems (either people or bandwidth > etc.) I just received a recommendation for wush.net which specializes > in svn hosting. $90/month for 5 Gb of disk space sounds like a good > deal and easily within the PSF budget. We were using wush.net's subversion and trac service for a (commercial) project from February until a little over a week ago. Their servers dropped off the internet for about three days straight earlier this month and we were unable to contact anyone. I still don't think we've received an explanation as to what happened. When it did come up, our data was OK. Previous to that experience, it worked out OK. The subversion repository got wedged once, but that was fixed in a matter of hours after filing a ticket. We host our own subversion and trac now. We just can't afford that kind of downtime again. Setting up subversion and trac isn't a very big deal, and they don't really require any real maintenance as far as I can tell (.. and I have been dealing with subversion over apache via mod_dav_svn since pre-1.0 days). Another thing to note is that the trac installation at wush.net is a branch off the latest stable version, and the database can't be downgraded or upgraded correctly by the trac-admin tool. However, the SQL to downgrade the schema to the latest stable is trivial and I still have it lying around if anyone is interested in moving their trac repositories off of wush ;) -bob From barry at python.org Sat Aug 20 20:37:02 2005 From: barry at python.org (Barry Warsaw) Date: Sat, 20 Aug 2005 14:37:02 -0400 Subject: [Python-Dev] A testing challenge In-Reply-To: <4303C0DD.5090903@spikesource.com> References: <4303C0DD.5090903@spikesource.com> Message-ID: <1124563022.24297.35.camel@presto.wooz.org> On Wed, 2005-08-17 at 18:57, Calvin Austin wrote: > When was the last time someone thanked you for writing a test? I tried > to think of the last time it happened to me and I can't remember. Well > at Spikesource we want to thank you not just for helping the Python > community but for your testing efforts too and we are running a > participatory testing contest. This is a competition where there are no > losers, every project gains if new tests are written. For more details > see below, it is open worldwide. feel free to send questions to me. Since you posted to python-dev, you might think about adding Python to the list of languages "in which [...] the project [is] written" on the registration form. Currently, the only choices are C/C++, Java, and php. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050820/fa521b8b/attachment.pgp From paolo_veronelli at libero.it Sun Aug 21 11:35:37 2005 From: paolo_veronelli at libero.it (Paolino) Date: Sun, 21 Aug 2005 11:35:37 +0200 Subject: [Python-Dev] On decorators implementation Message-ID: <43084AE9.20900@libero.it> I noticed (via using them) that decorations are applied to methods before they become methods. This choice flattens down the implementation to no differentiating methods from functions. 1) I have to apply euristics on the wrapped function type when I use the function as an index key. if type(observed) is types.MethodType: observed=observed.im_func things like this are inside my decorators. 2) The behavior of decorations are not definable. I imagine that a method implementation of them inside the type metaclass could be better specified by people. This probably ends up in metamethods or something I can't grasp Thanks Paolino From martin at v.loewis.de Sun Aug 21 13:18:58 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 21 Aug 2005 13:18:58 +0200 Subject: [Python-Dev] On decorators implementation In-Reply-To: <43084AE9.20900@libero.it> References: <43084AE9.20900@libero.it> Message-ID: <43086322.2030706@v.loewis.de> Paolino wrote: > I imagine that a method implementation of them inside the type metaclass > could be better specified by people. What you ask for is unimplementable. Method objects are created only when the method is accessed, not (even) when the class is created. Watch this: >>> class X: ... def foo(self): ... pass ... >>> x=X() >>> type(x.foo) >>> type(X.__dict__['foo']) So even though the class has long been defined, inside X's dictionary, foo is still a function. Only when you *access* x.foo, a method object is created on the fly: >>> x.foo is x.foo False Therefore, a decorator function cannot possibly get access to the method object - it simply doesn't exist. Regards, Martin From martin at v.loewis.de Sun Aug 21 15:12:00 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 21 Aug 2005 15:12:00 +0200 Subject: [Python-Dev] Admin access using svn+ssh Message-ID: <43087DA0.702@v.loewis.de> It turns out that svn+ssh with a single account has limitations: you can only set the tunnel user when you are using a restricted key. In PEP 347, the plan is that the current SF project admins get shell access to the pythondev account, which just has been created. To resolve this, project admins need two different SSH keys: one for accessing the shell, and one for regular commit activities. I would suggest that the default key is used for regular commits, and a separate key is created for shell access. I described this a bit in the PEP, essentially, in .ssh/config, I have Host pythondev Hostname dinsdale.python.org User pythondev IdentityFile ~/.ssh/pythondev So when I do "ssh pythondev", I get the shell account; when I do "svn co svn+ssh://pythondev at svn.python.org/python/trunk/Modules", I use my default identity, which gets tunneled as "Martin v. Loewis". Regards, Martin From martin at v.loewis.de Sun Aug 21 15:34:57 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 21 Aug 2005 15:34:57 +0200 Subject: [Python-Dev] wush.net details Message-ID: <43088301.5010104@v.loewis.de> I made a service request at wush.net, asking for more details about their service. There was a first response within 6 hours, asking for more time to prepare an answer. I said I don't need one urgently, and, with apologies, got a response one week later. I added the essence to the PEP; namely: - The machine would be a Virtuozzo Virtual Private Server (VPS), hosted at PowerVPS. - The default repository URL would be http://python.wush.net/svn/projectname/, but anything else could be arranged - we would get SSH login to the machine, with sudo capabilities. - They have a Web interface for management of the various SVN repositories that we want to host, and to manage user accounts. While svn+ssh would be supported, the user interface does not yet support it (although he said they might have something in September) - For offsite mirroring/backup, they suggest to use rsync instead of download of repository tarballs. So it seems that the "regular" administrative overhead would be roughly the same on wush.net and python.org: we would have to maintain account information ourselves; the initial setup might be easier due to the UI wizard help. I understand that the hope when using a commercial service is that its availability is higher, due to us paying somebody for the availability. Of course, Bob Ippolito's report is discouraging here. Regards, Martin From martin at v.loewis.de Sun Aug 21 15:43:59 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 21 Aug 2005 15:43:59 +0200 Subject: [Python-Dev] PEP 347: Migration to Subversion In-Reply-To: References: <2773CAC687FD5F4689F526998C7E4E5F05CC2E@au3010avexu1.global.avaya.com> Message-ID: <4308851F.9090800@v.loewis.de> Guido van Rossum wrote: > On the hosting issue, I'm still neutral -- I expect we'll be able to > support the current developer crowd easily on svn.python.org, but if > we ever find ther are resource problems (either people or bandwidth > etc.) I just received a recommendation for wush.net which specializes > in svn hosting. $90/month for 5 Gb of disk space sounds like a good > deal and easily within the PSF budget. I also have wush.net in the PEP, see my separate message. I'm not sure what it really is that we get over what we get from XS4ALL for free. >From the day-to-day maintenance, they seem comparable: they do backup for us, and we have to maintain accounts ourselves. Of course, wush.net has a Web GUI for maintenance activities (create repositories, create accounts, manage access control). I left out bandwidth details so far: we get 200GB/mo; after this, it is $50/200GB. Another issue might be server load. I don't know how many VPS they host on a single machine, or what their hardware is, but in either case, pythondev developer svn would be shared with something else (other VPSs for wush.net, regular pydotorg activities on python.org). Only day-to-day experience will tell whether this is acceptable. The critical issue seems to be availability: if the service goes down, when will it come back? Bob's experience is discouraging, but then, there also was a python.org outage from time to time (e.g. when MoinMoin consumed all CPU). As for the money itself: 90$/month certainly is not an issue at all. So far, I haven't received any other specific referrals for SVN hosters. Regards, Martin From martin at v.loewis.de Sun Aug 21 15:53:37 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 21 Aug 2005 15:53:37 +0200 Subject: [Python-Dev] Collecting SSH keys Message-ID: <43088761.7010905@v.loewis.de> I have setup a test installation on svn.python.org, so that developers can see how this would work. So if you are currently a sf.net/projects/python developer, please send me your SSH key before August 27 or after September 12. We will use real names for commit messages, so if you have specific preferences about the spelling of your name, please indicate them. The repository will be discarded after the testing, so feel free to make any changes you want. It's not decided yet whether the repository will eventually run on python.org, but it seems clear to me that we likely will use svn+ssh for developer access, unless testing reveals disadvantages of doing so. Please also look at the result of the conversion; if you find any issues, please report them. There is currently no anonymous WebDAV access to the repository. Regards, Martin From sjoerd at acm.org Sun Aug 21 17:28:07 2005 From: sjoerd at acm.org (Sjoerd Mullender) Date: Sun, 21 Aug 2005 17:28:07 +0200 Subject: [Python-Dev] Collecting SSH keys In-Reply-To: <43088761.7010905@v.loewis.de> References: <43088761.7010905@v.loewis.de> Message-ID: <43089D87.2060302@acm.org> Martin v. L?wis wrote: > I have setup a test installation on svn.python.org, so that > developers can see how this would work. > > So if you are currently a sf.net/projects/python developer, > please send me your SSH key before August 27 or after > September 12. We will use real names for commit messages, > so if you have specific preferences about the spelling > of your name, please indicate them. What about people with a whole host of ssh keys? I have a different key for each system I use (currently at least 6). Will this be supported? Will the different keys identify the same person? > The repository will be discarded after the testing, so > feel free to make any changes you want. > > It's not decided yet whether the repository will eventually > run on python.org, but it seems clear to me that we likely > will use svn+ssh for developer access, unless testing > reveals disadvantages of doing so. > > Please also look at the result of the conversion; if you > find any issues, please report them. > > There is currently no anonymous WebDAV access to the > repository. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/sjoerd.mullender%40cwi.nl -- Sjoerd Mullender -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 369 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20050821/0afb7716/signature.pgp From martin at v.loewis.de Sun Aug 21 18:19:45 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 21 Aug 2005 18:19:45 +0200 Subject: [Python-Dev] Collecting SSH keys In-Reply-To: <43089D87.2060302@acm.org> References: <43088761.7010905@v.loewis.de> <43089D87.2060302@acm.org> Message-ID: <4308A9A1.1040700@v.loewis.de> Sjoerd Mullender wrote: > What about people with a whole host of ssh keys? I have a different key > for each system I use (currently at least 6). Will this be supported? > Will the different keys identify the same person? That would be possible, yes. You should send a single file containing all of them, and, each time something changes, resend the entire file. All of your keys would identify "Sjoerd Mullender". I don't know how this scales in OpenSSH having an authorized_keys file with hundred or more keys. On the wire, this seems safe, as it apparently is the client which offers various keys, and the server which then accepts or rejects them. Regards, Martin From skip at pobox.com Sat Aug 20 04:32:12 2005 From: skip at pobox.com (skip@pobox.com) Date: Fri, 19 Aug 2005 21:32:12 -0500 Subject: [Python-Dev] Deprecating builtin id (and moving it to sys()) In-Reply-To: References: <20050817140217.GQ3389@www.async.com.br> <200508181409.17431.anthony@interlink.com.au> Message-ID: <17158.38444.778226.955186@montanaro.dyndns.org> Guido> The built-in namespace is searched last for a reason -- the Guido> design is such that if you don't care for a particular built-in Guido> you don't need to know about it. In my mind there are three classes of builtins from the standpoint of overriding. Pychecker complains if you override any of them, but I think that many times it does so unnecessarily. The first class includes those builtins that you will likely find in many code samples and should just never be overridden. For me these include "abs", "map", "list", "int", "range", "zip", the various exceptions, etc. The second class of builtins consists of objects or functions that are fairly special-purpose. You might not really care if they are overridden, depending on context. For me this class includes "compile", "id", "reload", "execfile", "ord", etc. Finally, there is the subset of builtins that is included almost solely as a convenience for use at the interpreter prompt. They include "quit", "exit" and "copyright". I could care less if I override them in my code, and don't think pychecker should either. Skip From barry at python.org Mon Aug 22 01:01:22 2005 From: barry at python.org (Barry Warsaw) Date: Sun, 21 Aug 2005 19:01:22 -0400 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <43087DA0.702@v.loewis.de> References: <43087DA0.702@v.loewis.de> Message-ID: <1124665281.31664.28.camel@geddy.wooz.org> On Sun, 2005-08-21 at 09:12, "Martin v. L?wis" wrote: > It turns out that svn+ssh with a single account has limitations: > you can only set the tunnel user when you are using a restricted > key. In PEP 347, the plan is that the current SF project admins > get shell access to the pythondev account, which just has been > created. > > To resolve this, project admins need two different SSH keys: > one for accessing the shell, and one for regular commit activities. I may be totally misunderstanding, but to get shell access wouldn't I avoid using the pythondev account and just use my own account? I'd only need the pythondev account to access the svn repository, right? (And actually, it might be possible to set up group permissions and membership so that I could access the repo with either). The number of people who need shell access should be pretty small. I'm also a little confused about the pep. What does "admin access to the pythondev account" mean? Do you mean the people who are going to be managing users that can access svn? In that case, I think the system admins (i.e. those who already have shell access to dinsdale) would be the people managing user access to svn. > I would suggest that the default key is used for regular commits, > and a separate key is created for shell access. I described this > a bit in the PEP, essentially, in .ssh/config, I have > > Host pythondev > Hostname dinsdale.python.org > User pythondev > IdentityFile ~/.ssh/pythondev > > So when I do "ssh pythondev", I get the shell account; when I do > "svn co svn+ssh://pythondev at svn.python.org/python/trunk/Modules", > I use my default identity, which gets tunneled as "Martin v. Loewis". I'm confused again; are you saying that we should have a host named pythondev.python.org? I'm not sure that's necessary. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050821/580c0311/attachment-0001.pgp From aahz at pythoncraft.com Mon Aug 22 01:55:23 2005 From: aahz at pythoncraft.com (Aahz) Date: Sun, 21 Aug 2005 16:55:23 -0700 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <1124665281.31664.28.camel@geddy.wooz.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> Message-ID: <20050821235522.GA16606@panix.com> On Sun, Aug 21, 2005, Barry Warsaw wrote: > On Sun, 2005-08-21 at 09:12, "Martin v. L?wis" wrote: >> >> I would suggest that the default key is used for regular commits, >> and a separate key is created for shell access. I described this >> a bit in the PEP, essentially, in .ssh/config, I have >> >> Host pythondev >> Hostname dinsdale.python.org >> User pythondev >> IdentityFile ~/.ssh/pythondev >> >> So when I do "ssh pythondev", I get the shell account; when I do >> "svn co svn+ssh://pythondev at svn.python.org/python/trunk/Modules", >> I use my default identity, which gets tunneled as "Martin v. Loewis". > > I'm confused again; are you saying that we should have a host named > pythondev.python.org? I'm not sure that's necessary. No, pythondev is simply an SSH alias for dinsdale -- the server knows nothing about it. I don't quite understand the "User pythondev" line, though -- I think that's a mistake. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ The way to build large Python applications is to componentize and loosely-couple the hell out of everything. From martin at v.loewis.de Mon Aug 22 08:18:31 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 08:18:31 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <1124665281.31664.28.camel@geddy.wooz.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> Message-ID: <43096E37.8070708@v.loewis.de> Barry Warsaw wrote: > I may be totally misunderstanding, but to get shell access wouldn't I > avoid using the pythondev account and just use my own account? You could do that (or use the root account); I can't: I don't have a ssh account on dinsdale. An even if I had, I couldn't write to pythondev's authorized_keys2. > I'm also a little confused about the pep. What does "admin access to > the pythondev account" mean? Do you mean the people who are going to be > managing users that can access svn? Correct. > In that case, I think the system > admins (i.e. those who already have shell access to dinsdale) would be > the people managing user access to svn. Ok: to whom should I forward the ssh keys then which I'm currently collecting? >>Host pythondev >> Hostname dinsdale.python.org >> User pythondev >> IdentityFile ~/.ssh/pythondev >> >>So when I do "ssh pythondev", I get the shell account; when I do >>"svn co svn+ssh://pythondev at svn.python.org/python/trunk/Modules", >>I use my default identity, which gets tunneled as "Martin v. Loewis". > > > I'm confused again; are you saying that we should have a host named > pythondev.python.org? I'm not sure that's necessary. Not at all. This is rather an OpenSSH convenience mechanism to avoid typing hostname and user name all the time. I introduce a local alias pythondev, which means I want to access pythondev at dinsdale.python.org, using the key pythondev.pub. Regards, Martin From martin at v.loewis.de Mon Aug 22 08:31:30 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 08:31:30 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <20050821235522.GA16606@panix.com> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <20050821235522.GA16606@panix.com> Message-ID: <43097142.9050905@v.loewis.de> Aahz wrote: >>>Host pythondev >>> Hostname dinsdale.python.org >>> User pythondev >>> IdentityFile ~/.ssh/pythondev >>> >>I'm confused again; are you saying that we should have a host named >>pythondev.python.org? I'm not sure that's necessary. > > > No, pythondev is simply an SSH alias for dinsdale -- the server knows > nothing about it. I don't quite understand the "User pythondev" line, > though -- I think that's a mistake. That's intentional. "ssh pythondev" now becomes equivalent to ssh -l pythondev -i ~/.ssh/pythondev dinsdale.python.org IOW, the User option is equivalent to specifying the -l option. Regards, Martin From paolo_veronelli at libero.it Mon Aug 22 10:12:46 2005 From: paolo_veronelli at libero.it (Paolino) Date: Mon, 22 Aug 2005 10:12:46 +0200 Subject: [Python-Dev] On decorators implementation In-Reply-To: <43084AE9.20900@libero.it> References: <43084AE9.20900@libero.it> Message-ID: <430988FE.2020603@libero.it> Paolino wrote: > I noticed (via using them) that decorations are applied to methods > before they become methods. > > This choice flattens down the implementation to no differentiating > methods from functions. > > > > 1) > I have to apply euristics on the wrapped function type when I use the > function as an index key. > > if type(observed) is types.MethodType: > observed=observed.im_func > > things like this are inside my decorators. > > 2) > The behavior of decorations are not definable. > I imagine that a method implementation of them inside the type metaclass > could be better specified by people. > This probably ends up in metamethods or something I can't grasp > A downside of decorating at function level is that it's virtually impossible to check from the decorator that the first call parameter (aka self) is an instance of the method class.This check must be done inside the decorated. This can really happen in normal use as decorators are useful to register the decorated as a 'callback'.Who ever fires it can do it with no respect on the class belonging of the function/method, and the error raised will not be coherent with 'calling method on a incompatible instance'. Maybe it's possible to let the decorator know the method class even if the class is still undefined.(Just like recursive functions?) This would allow decorators to call super with the right class also. @callSuper decoration is something I really miss. Thanks Paolino From stephen at xemacs.org Mon Aug 22 09:39:03 2005 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 22 Aug 2005 16:39:03 +0900 Subject: [Python-Dev] Collecting SSH keys In-Reply-To: <4308A9A1.1040700@v.loewis.de> ( =?iso-8859-1?q?Martin_v=2E_L=F6wis's_message_of?= "Sun, 21 Aug 2005 18:19:45 +0200") References: <43088761.7010905@v.loewis.de> <43089D87.2060302@acm.org> <4308A9A1.1040700@v.loewis.de> Message-ID: <87y86uqv3c.fsf@tleepslib.sk.tsukuba.ac.jp> >>>>> "Martin" == Martin v L?wis writes: Martin> I don't know how this scales in OpenSSH having an Martin> authorized_keys file with hundred or more keys. On cvs.xemacs.org (aka SunSITE.dk) ssh+cvs access with cvs access control being handled by a Perl script scales to approximately 85 users. I don't handle key management directly, but I believe several users use multiple keys (I don't personally). I've never heard any complaints from the guys who actually do key management; they just keep authorized_keys in alphabetical order by comment (= user's real name). Nor do I notice any authorization overhead vs. a simple ssh login when accessing the cvs server.[1] Evidently the "what keys do you have?" negotiation with the agent takes very little time (in terms of what a human can notice). If you want time(1) timings or something like that, I'd be happy to get an exact count of the number of keys and do them (but it will have to wait until I get back from travel August 28). Footnotes: [1] For testing whether keys are properly installed, the sequence "ssh xemacs at cvs.xemacs.org", then asking the server for "version" and sending EOF (^D), is what we use. So there is no overhead from a local CVS or anything like that, although of course you do have to start the remote cvs server process (via the COMMAND= in the .ssh/config file). How that compares to starting a shell I'm not sure. -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software. From raymond.hettinger at verizon.net Mon Aug 22 14:46:27 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Mon, 22 Aug 2005 08:46:27 -0400 Subject: [Python-Dev] [Python-checkins] python/dist/src setup.py, 1.219, 1.220 In-Reply-To: <20050821184639.EF8711E4006@bag.python.org> Message-ID: <003101c5a717$83be4b60$3c23a044@oemcomputer> > A new hashlib module to replace the md5 and sha modules. It adds > support for additional secure hashes such as SHA-256 and SHA-512. The > hashlib module uses OpenSSL for fast platform optimized > implementations of algorithms when available. The old md5 and sha > modules still exist as wrappers around hashlib to preserve backwards > compatibility. I'm getting compilation errors: C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad suffix on number' C:\py25\Modules\sha512module.c(146) : error C2146: syntax error : missing ')' before identifier 'L' C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad suffix on number' C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad suffix on number' C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad suffix on number' C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad suffix on number' C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad suffix on number' C:\py25\Modules\sha512module.c(146) : fatal error C1013: compiler limit : too many open parentheses Also, there should be updating entries to Misc/NEWS, PC/VC6/pythoncore.dsp, and PC/config.c. Raymond From martin at v.loewis.de Mon Aug 22 16:11:31 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 16:11:31 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <1124711379.31664.213.camel@geddy.wooz.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <1124711379.31664.213.camel@geddy.wooz.org> Message-ID: <4309DD13.4040902@v.loewis.de> Barry Warsaw wrote: >>You could do that (or use the root account); I can't: I don't have >>a ssh account on dinsdale. An even if I had, I couldn't write to >>pythondev's authorized_keys2. > > > That's easily rectified! :) We should give you an account and sudo > access. Should I just use your keys from creosote? Please do! >>Ok: to whom should I forward the ssh keys then which I'm currently >>collecting? > > > Probably here, unless once you have the above, you still want to do it > yourself. I would be worried that you are a single point of failure here: for sf.net/projects/python, multiple people can add new users, and I think we should continue that tradition. I would be happy with *different* people being able to manage that, but the group should be larger than two, IMO. Regards, Martin From martin at v.loewis.de Mon Aug 22 16:20:37 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 16:20:37 +0200 Subject: [Python-Dev] Collecting SSH keys In-Reply-To: <87y86uqv3c.fsf@tleepslib.sk.tsukuba.ac.jp> References: <43088761.7010905@v.loewis.de> <43089D87.2060302@acm.org> <4308A9A1.1040700@v.loewis.de> <87y86uqv3c.fsf@tleepslib.sk.tsukuba.ac.jp> Message-ID: <4309DF35.5000902@v.loewis.de> Stephen J. Turnbull wrote: > On cvs.xemacs.org (aka SunSITE.dk) ssh+cvs access with cvs access > control being handled by a Perl script scales to approximately 85 > users. I don't handle key management directly, but I believe several > users use multiple keys (I don't personally). I've never heard any > complaints from the guys who actually do key management; they just > keep authorized_keys in alphabetical order by comment (= user's real > name). Nor do I notice any authorization overhead vs. a simple ssh > login when accessing the cvs server.[1] Evidently the "what keys do > you have?" negotiation with the agent takes very little time (in > terms of what a human can notice). That's encouraging; I'm willing to proceed with that approach then. As for key management: I just designed an infrastructure where ~pythondev/keys is a directory containing files named, say "Martin v. Loewis" (with spaces, ASCII only); the contents of the files are just the public keys. I run then make_authorized_keys, which regenerates the authorized_keys2 file, adding all the command= lines. This avoids editing authorized_keys2 in a text editor. Regards, Martin From skip at pobox.com Mon Aug 22 17:18:33 2005 From: skip at pobox.com (skip@pobox.com) Date: Mon, 22 Aug 2005 10:18:33 -0500 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <43096E37.8070708@v.loewis.de> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> Message-ID: <17161.60617.950268.641009@montanaro.dyndns.org> Martin, I'm completely confused about what, if anything, I need to send to you. I can already access the python.org website repository via svn. Will I automatically get access to the new Python source repository or do I need to send you pub key(s)? Are dinsdale.python.org and svn.python.org the same machine with different IP addresses? If they are different machines, why would we want to host svn repositories on multiple machines? Skip From aahz at pythoncraft.com Mon Aug 22 17:25:33 2005 From: aahz at pythoncraft.com (Aahz) Date: Mon, 22 Aug 2005 08:25:33 -0700 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <43097142.9050905@v.loewis.de> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <20050821235522.GA16606@panix.com> <43097142.9050905@v.loewis.de> Message-ID: <20050822152533.GB12281@panix.com> On Mon, Aug 22, 2005, "Martin v. L?wis" wrote: > Aahz wrote: >>Barry: >>>Martin: >>>> >>>>Host pythondev >>>> Hostname dinsdale.python.org >>>> User pythondev >>>> IdentityFile ~/.ssh/pythondev >>>> >>>I'm confused again; are you saying that we should have a host named >>>pythondev.python.org? I'm not sure that's necessary. >> >> No, pythondev is simply an SSH alias for dinsdale -- the server knows >> nothing about it. I don't quite understand the "User pythondev" line, >> though -- I think that's a mistake. > > That's intentional. "ssh pythondev" now becomes equivalent to > > ssh -l pythondev -i ~/.ssh/pythondev dinsdale.python.org > > IOW, the User option is equivalent to specifying the -l option. Yes, I know -- but it looks like a mistake to me. Are you saying that all shell access will be done through a single account? Isn't that a huge security risk? My understanding was that it was SVN access that would be going through a single account, not shell access. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ The way to build large Python applications is to componentize and loosely-couple the hell out of everything. From barry at python.org Mon Aug 22 17:32:10 2005 From: barry at python.org (Barry Warsaw) Date: Mon, 22 Aug 2005 11:32:10 -0400 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <17161.60617.950268.641009@montanaro.dyndns.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> Message-ID: <1124724730.17082.8.camel@geddy.wooz.org> On Mon, 2005-08-22 at 11:18, skip at pobox.com wrote: > I'm completely confused about what, if anything, I need to send to you. I > can already access the python.org website repository via svn. Will I > automatically get access to the new Python source repository or do I need to > send you pub key(s)? I think technically, the answer to that is "yes", you will automatically get access to the source repo. The question I have is whether you /should/ access the source repo that way, or use the shared pythondev account. Two unknowns for me are 1) will there be permission problems that either prevent you from doing this, or once you've committed a change, will screw pythondev-access?; 2) when we finally get email notifications worked in, will it still look like your commit is coming from the right place. I think the answer to #2 is yes, but I'm not sure about #1. > Are dinsdale.python.org and svn.python.org the same > machine with different IP addresses? If they are different machines, why They are the same machine, with different IP addresses. Anonymous webdav will require two Apache processes, since different user/groups are needed and to support different certs for svn.python.org and (eventually) www.python.org. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050822/8e2b29ff/attachment.pgp From skip at pobox.com Mon Aug 22 17:45:29 2005 From: skip at pobox.com (skip@pobox.com) Date: Mon, 22 Aug 2005 10:45:29 -0500 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <1124724730.17082.8.camel@geddy.wooz.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> Message-ID: <17161.62233.743239.89277@montanaro.dyndns.org> >> Will I automatically get access to the new Python source repository >> or do I need to send you pub key(s)? Barry> I think technically, the answer to that is "yes", you will Barry> automatically get access to the source repo. Okay... Barry> The question I have is whether you /should/ access the source Barry> repo that way, or use the shared pythondev account. More confusion here. If I use some sort of shared access how will the system ascribe changes I make to me and not, for example, Martin? I think until this experiment is over and we have really and truly migrated to svn I will simply let other people fuss with things. Skip From foom at fuhm.net Mon Aug 22 17:57:54 2005 From: foom at fuhm.net (James Y Knight) Date: Mon, 22 Aug 2005 11:57:54 -0400 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <1124724730.17082.8.camel@geddy.wooz.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> Message-ID: <3F040472-CADF-4F47-8EFD-9B1267C8D0C9@fuhm.net> On Aug 22, 2005, at 11:32 AM, Barry Warsaw wrote: > They are the same machine, with different IP addresses. Anonymous > webdav will require two Apache processes, since different user/groups > are needed and to support different certs for svn.python.org and > (eventually) www.python.org. > It seems a waste to use SVN's webdav support just for anon access. The svnserve method works well for anon access. The only reason to use svn webdav IMO is if you want to use that for authenticated access. But since you're talking about using svn+ssh for that.. James From martin at v.loewis.de Mon Aug 22 18:07:50 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 18:07:50 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <17161.60617.950268.641009@montanaro.dyndns.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> Message-ID: <4309F856.40506@v.loewis.de> skip at pobox.com wrote: > I'm completely confused about what, if anything, I need to send to you. I > can already access the python.org website repository via svn. Yes, but you do so using username/password, right? pythondev will be using svn+ssh. > Will I > automatically get access to the new Python source repository or do I need to > send you pub key(s)? You need to send me pubkeys. Actually, I just copied the ones from creosote (see below). You should now be able to checkout svn+ssh://pythondev at svn.python.org/python/trunk > Are dinsdale.python.org and svn.python.org the same > machine with different IP addresses? Correct. > If they are different machines, why > would we want to host svn repositories on multiple machines? We don't. However, we use different access methods. Actually, we *might* use different access methods. If this turns out to be too confusing to users, we are probably back to username/password. Regards, Martin P.S. The keys I installed are ssh-dss AAAAB3NzaC1kc3MAAACBAJAPN3ngdjih7H1wqkmbkaJDpfoW3fRrk9phtuuO+js43qU06BiqInbGZ/zjVZRrM7yzRbo2PGu1+ox8H/vkMlSk6IxmgMtNrrQ9SEoTRo7eyg5ku+JiC44h3RWT2IuiIALB8axHQSBsF6Oe4O9z/lgsLMO08M2l1TzRnjSjyOEZAAAAFQDGffqFFm+IoSH6cRfxnY+BiXxZ5QAAAIATuQmlscDd/QNSlk4Oy7ZMUdHplx76zQtyUHXvhRVkIu6QrduhnnCkGIFjSHQsnJOoroF4tVaJYY7oka17Ambd0LiWcSlNK+IHMdbvZ91wbVpeo9x/HBCJtCMxDX8PxG3TADuqiZjeC8nOpCdJ+cK7emQv+G4WIw3gC3IuPRINWAAAAIA5+OO9ApbKrcClwHXZ9DqtDJBe2fSox1mnei3VAajbOU/o3+j+G+5iLerOqLTCoOyIs7umvuUulIAXvhDzCzusw3mfBtt3UODQn0L3R47OFHzOiCEbihStxd36lVgCJgRBAW7UKf+2k3BzxJ5DVpp4+AZ7fS4FUVkZ8DYAog/68g== skip at montanaro.dyndns.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAq83rRGWRR4SdvvBUMJ/gDmMG7U7LdiC50kqUTbw+Kogum5JT7kexi1XYKgyKJ8FbRwMx1Xj9zjQERgDhYtFCJg72kSkD2muN3DkyU7vIoZQM/aNpspPNNDWRqj8pzHPzhWDUfL+tjZl78JD51mTOlGHaZUGdKnPeUOQF2XTadis= skip at montanaro.dyndns.org From martin at v.loewis.de Mon Aug 22 18:10:37 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 18:10:37 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <20050822152533.GB12281@panix.com> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <20050821235522.GA16606@panix.com> <43097142.9050905@v.loewis.de> <20050822152533.GB12281@panix.com> Message-ID: <4309F8FD.6080505@v.loewis.de> Aahz wrote: > Yes, I know -- but it looks like a mistake to me. Are you saying that > all shell access will be done through a single account? Isn't that a > huge security risk? My understanding was that it was SVN access that > would be going through a single account, not shell access. Only few selected people would have shell access; I don't see that as a huge risk. Anyway, Barry didn't like it either, so we removed shell access to the pythondev account; user keys now need to be added by the pydotorg admins. Regards, Martin From martin at v.loewis.de Mon Aug 22 18:16:24 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 18:16:24 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <1124724730.17082.8.camel@geddy.wooz.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> Message-ID: <4309FA58.4080103@v.loewis.de> Barry Warsaw wrote: > I think technically, the answer to that is "yes", you will automatically > get access to the source repo. At the moment, the answer actually is "no". For the projects repository, there is no group write permission - you must be pythondev in order to write. > The question I have is whether you > /should/ access the source repo that way, or use the shared pythondev > account. Two unknowns for me are 1) will there be permission problems > that either prevent you from doing this, or once you've committed a > change, will screw pythondev-access?; Yes to the former. The webserver has only read access to the (projects) repository. > 2) when we finally get email > notifications worked in, will it still look like your commit is coming > from the right place. Not sure what "the right place" would be: pythondev at python.org? I think the email could look any way we want it to look. > They are the same machine, with different IP addresses. Anonymous > webdav will require two Apache processes, since different user/groups > are needed Not necessarily. The repository could be world-readable, in which case "nobody" could access it. > and to support different certs for svn.python.org and > (eventually) www.python.org. Ah. I think anonymous read access should be on port 80. Regards, Martin From martin at v.loewis.de Mon Aug 22 18:20:42 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 18:20:42 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <17161.62233.743239.89277@montanaro.dyndns.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <17161.62233.743239.89277@montanaro.dyndns.org> Message-ID: <4309FB5A.1040201@v.loewis.de> skip at pobox.com wrote: > More confusion here. If I use some sort of shared access how will the > system ascribe changes I make to me and not, for example, Martin? In pythondev's authorized_keys2, we have a line command="/usr/bin/svnserve --root=/data/repos/projects -t --tunnel-user 'Skip Montanaro'",no-port-forwarding,no-X11-forwarding, no-agent-forwarding,no-pty ssh-dss So the *only* command you are allowed to invoke is svnserve (actually, sshd will invoke that no matter what the ssh client requests). This will tell subversion that changes should be logges as 'Skip Montanaro'. > I think until this experiment is over and we have really and truly migrated > to svn I will simply let other people fuss with things. Well, you are not required to understand it, but you should try to use it. Just check out svn+ssh://pythondev at svn.python.org/python/trunk/Misc, and see whether this works. Regards, Martin From martin at v.loewis.de Mon Aug 22 18:23:01 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 18:23:01 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <3F040472-CADF-4F47-8EFD-9B1267C8D0C9@fuhm.net> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <3F040472-CADF-4F47-8EFD-9B1267C8D0C9@fuhm.net> Message-ID: <4309FBE5.40204@v.loewis.de> James Y Knight wrote: > It seems a waste to use SVN's webdav support just for anon access. > The svnserve method works well for anon access. The only reason to > use svn webdav IMO is if you want to use that for authenticated > access. But since you're talking about using svn+ssh for that.. It has the advantage that we can easily point people to files with a web browser; they don't need an svn client. Regards, Martin From barry at python.org Mon Aug 22 18:41:11 2005 From: barry at python.org (Barry Warsaw) Date: Mon, 22 Aug 2005 12:41:11 -0400 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <4309FA58.4080103@v.loewis.de> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <4309FA58.4080103@v.loewis.de> Message-ID: <1124728871.17084.13.camel@geddy.wooz.org> On Mon, 2005-08-22 at 12:16, "Martin v. L?wis" wrote: > Barry Warsaw wrote: > > I think technically, the answer to that is "yes", you will automatically > > get access to the source repo. > > At the moment, the answer actually is "no". For the projects repository, > there is no group write permission - you must be pythondev in order to > write. Good! I think that's a feature. :) I have a vague discomfort with allowing both types of access. I.e. I'd rather all source committers use the same mechanism. > > 2) when we finally get email > > notifications worked in, will it still look like your commit is coming > > from the right place. > > Not sure what "the right place" would be: pythondev at python.org? > I think the email could look any way we want it to look. I think it should be @python.org where is the firstname.lastname (with some exceptions) scheme that we've agreed on. I actually /don't/ want all commits to look like they're coming from pythondev at python.org > > and to support different certs for svn.python.org and > > (eventually) www.python.org. > > Ah. I think anonymous read access should be on port 80. Maybe we want to put websvn (or whatever it's called these days) on port 80 of svn.python.org? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050822/cb31d290/attachment.pgp From pje at telecommunity.com Mon Aug 22 18:42:57 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 22 Aug 2005 12:42:57 -0400 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <4309FBE5.40204@v.loewis.de> References: <3F040472-CADF-4F47-8EFD-9B1267C8D0C9@fuhm.net> <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <3F040472-CADF-4F47-8EFD-9B1267C8D0C9@fuhm.net> Message-ID: <5.1.1.6.0.20050822124128.01b0bd10@mail.telecommunity.com> At 06:23 PM 8/22/2005 +0200, Martin v. L?wis wrote: >James Y Knight wrote: > > It seems a waste to use SVN's webdav support just for anon access. > > The svnserve method works well for anon access. The only reason to > > use svn webdav IMO is if you want to use that for authenticated > > access. But since you're talking about using svn+ssh for that.. > >It has the advantage that we can easily point people to files >with a web browser; they don't need an svn client. You can do that with viewcvs, too. Viewcvs can also create tarballs for easy downloading, and has a lot of browsing and viewing options that the SVN webdav mode doesn't. From skip at pobox.com Mon Aug 22 18:43:06 2005 From: skip at pobox.com (skip@pobox.com) Date: Mon, 22 Aug 2005 11:43:06 -0500 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <4309FB5A.1040201@v.loewis.de> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <17161.62233.743239.89277@montanaro.dyndns.org> <4309FB5A.1040201@v.loewis.de> Message-ID: <17162.155.1546.1991@montanaro.dyndns.org> >> I think until this experiment is over and we have really and truly >> migrated to svn I will simply let other people fuss with things. Martin> Well, you are not required to understand it, but you should try Martin> to use it. Good point. Martin> Just check out Martin> svn+ssh://pythondev at svn.python.org/python/trunk/Misc, and see Martin> whether this works. It worked. I made a trivial change to Misc/NEWS and checked it in. I then ran "svn blame NEWS" to see what it showed. This took approximately forever. Can I assume this is one thing svn is always going to be pretty slow at? I use cvs annotate frequently. Is there a faster alternative in svn to identify who did what? I notice that you use my real name (including spaces). I doubt we have any code that munches on annotated listings, but it seems that for the sake of script writers' sanity it would be better to elide spaces or replace them with underscores so the annotated user is a single "word": 40555 Skip Montanaro ++++++++++++ 28675 montanaro Python News 40555 Skip Montanaro ++++++++++++ 28675 montanaro 37655 anthonybaxter (editors: check NEWS.help for information about editing NEWS using ReST.) 37654 montanaro 37838 rhettinger What's New in Python 2.5 alpha 1? 37838 rhettinger ================================= 37838 rhettinger 38611 anthonybaxter *Release date: XX-XXX-2006* 38611 anthonybaxter 37838 rhettinger Core and builtins 37838 rhettinger ----------------- 37838 rhettinger ... That way column 2 would always be the contributor. Skip From gvanrossum at gmail.com Mon Aug 22 19:43:24 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon, 22 Aug 2005 10:43:24 -0700 Subject: [Python-Dev] On decorators implementation In-Reply-To: <430988FE.2020603@libero.it> References: <43084AE9.20900@libero.it> <430988FE.2020603@libero.it> Message-ID: > Maybe it's possible to let the decorator know the method class even if > the class is still undefined.(Just like recursive functions?) > This would allow decorators to call super with the right class also. > @callSuper decoration is something I really miss. You're thinking about it all wrong. Remember that decorators can also be used to declare that something is a static method or class method etc. Try to learn Python, not to write some other language using Python syntax. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From kbk at shore.net Mon Aug 22 20:37:54 2005 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon, 22 Aug 2005 14:37:54 -0400 (EDT) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200508221837.j7MIbsHG031701@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 352 open ( +0) / 2898 closed ( +2) / 3250 total ( +2) Bugs : 926 open (+13) / 5177 closed (+15) / 6103 total (+28) RFE : 190 open ( -1) / 179 closed ( +1) / 369 total ( +0) New / Reopened Patches ______________________ fix smtplib when local host isn't resolvable in dns (2005-08-12) http://python.org/sf/1257988 opened by Arkadiusz Miskiewicz tarfile: fix for bug #1257255 (2005-08-17) http://python.org/sf/1262036 opened by Lars Gust?bel Patches Closed ______________ sha256 module (2004-04-14) http://python.org/sf/935454 closed by greg sha and md5 modules should use OpenSSL when possible (2005-02-12) http://python.org/sf/1121611 closed by greg New / Reopened Bugs ___________________ Significant memory leak with PyImport_ReloadModule (2005-08-11) http://python.org/sf/1256669 opened by Ben Held slice object uses -1 as exclusive end-bound (2005-08-11) http://python.org/sf/1256786 opened by Bryan G. Olson tarfile local name is local, should be abspath (2005-08-12) http://python.org/sf/1257255 opened by Martin Blais Encodings iso8859_1 and latin_1 are redundant (2005-08-12) http://python.org/sf/1257525 opened by liturgist Solaris 8 declares gethostname(). (2005-08-12) http://python.org/sf/1257687 opened by Hans Deragon error message incorrectly claims Visual C++ is required (2005-08-12) http://python.org/sf/1257728 opened by Zooko O'Whielacronx Make set.remove() behave more like Set.remove() (2005-08-12) CLOSED http://python.org/sf/1257731 opened by Raymond Hettinger tkapp read-only attributes (2005-08-12) http://python.org/sf/1257772 opened by peeb gen_send_ex: Assertion `f->f_back ! (2005-08-12) CLOSED http://python.org/sf/1257960 opened by Neil Schemenauer http auth documentation/implementation conflict (2005-08-13) http://python.org/sf/1258485 opened by Matthias Klose "it's" vs. "its" typo in Language Reference (2005-08-14) CLOSED http://python.org/sf/1258922 opened by Wolfgang Petzold Makefile ignores $CPPFLAGS (2005-08-14) http://python.org/sf/1258986 opened by Dirk Pirschel Tix CheckList 'radio' option cannot be changed (2005-08-14) http://python.org/sf/1259434 opened by Raymond Maple subprocess: more general (non-buffering) communication (2005-08-15) http://python.org/sf/1260171 opened by Ian Bicking __new__ is class method (2005-08-16) http://python.org/sf/1261229 opened by Mike Orr import dynamic library bug? (2005-08-16) http://python.org/sf/1261390 opened by broadwin Tutorial doesn't cover * and ** function calls (2005-08-16) http://python.org/sf/1261659 opened by Brett Cannon precompiled code and nameError. (2005-08-17) http://python.org/sf/1261714 opened by Vladimir Menshakov minidom.py alternate newl support is broken (2005-08-17) http://python.org/sf/1262320 opened by John Whitley fcntl.ioctl have a bit problem. (2005-08-18) http://python.org/sf/1262856 opened by Raise L. Sail typo on "SimpleXMLRPCServer Objects" (2005-08-18) CLOSED http://python.org/sf/1263086 opened by Chad Whitacre type() and isinstance() do not call __getattribute__ (2005-08-19) http://python.org/sf/1263635 opened by Per Vognsen IDLE on Mac (2005-08-18) http://python.org/sf/1263656 opened by Bruce Sherwood PyArg_ParseTupleAndKeywords doesn't handle I format correctl (2005-08-19) CLOSED http://python.org/sf/1264168 opened by John Finlay PEP 8 uses wrong raise syntax (2005-08-20) CLOSED http://python.org/sf/1264666 opened by Steven Bethard sequence slicing documentation incomplete (2005-08-20) http://python.org/sf/1265100 opened by Steven Bethard lexists() is not exported from os.path (2005-08-22) CLOSED http://python.org/sf/1266283 opened by Martin Blais Mistakes in decimal.Context.subtract documentation (2005-08-22) http://python.org/sf/1266296 opened by Jim Sizelove Bugs Closed ___________ smtplib and email.py (2005-08-03) http://python.org/sf/1251528 closed by rhettinger float('-inf') (2005-08-09) http://python.org/sf/1255395 closed by tjreedy Make set.remove() behave more like Set.remove() (2005-08-12) http://python.org/sf/1257731 closed by rhettinger gen_send_ex: Assertion `f->f_back ! (2005-08-12) http://python.org/sf/1257960 closed by pje IOError after normal write (2005-08-04) http://python.org/sf/1252149 closed by tim_one IOError after normal write (2005-08-04) http://python.org/sf/1252149 deleted by patrick_gerken "it's" vs. "its" typo in Language Reference (2005-08-14) http://python.org/sf/1258922 closed by birkenfeld hotshot.stats.load (2004-02-19) http://python.org/sf/900092 closed by bwarsaw typo on "SimpleXMLRPCServer Objects" (2005-08-18) http://python.org/sf/1263086 closed by doerwalter PyArg_ParseTupleAndKeywords doesn't handle I format correctl (2005-08-19) http://python.org/sf/1264168 closed by birkenfeld PEP 8 uses wrong raise syntax (2005-08-20) http://python.org/sf/1264666 closed by goodger list(obj) can swallow KeyboardInterrupt (2005-07-21) http://python.org/sf/1242657 closed by rhettinger container methods raise KeyError not IndexError (2005-08-01) http://python.org/sf/1249837 closed by rhettinger zip incorrectly and incompletely documented (2005-02-12) http://python.org/sf/1121416 closed by rhettinger bz2 RuntimeError when decompressing file (2005-04-27) http://python.org/sf/1191043 closed by birkenfeld lexists() is not exported from os.path (2005-08-22) http://python.org/sf/1266283 closed by birkenfeld RFE Closed __________ md5 and sha1 modules should use openssl implementation (2004-06-30) http://python.org/sf/983069 closed by greg From nas at arctrix.com Mon Aug 22 23:31:42 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Mon, 22 Aug 2005 15:31:42 -0600 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings Message-ID: <20050822213142.GA5702@mems-exchange.org> [Please mail followups to python-dev at python.org.] The PEP has been rewritten based on a suggestion by Guido to change str() rather than adding a new built-in function. Based on my testing, I believe the idea is feasible. It would be helpful if people could test the patched Python with their own applications and report any incompatibilities. PEP: 349 Title: Allow str() to return unicode strings Version: $Revision: 1.3 $ Last-Modified: $Date: 2005/08/22 21:12:08 $ Author: Neil Schemenauer Status: Draft Type: Standards Track Content-Type: text/plain Created: 02-Aug-2005 Post-History: 06-Aug-2005 Python-Version: 2.5 Abstract This PEP proposes to change the str() built-in function so that it can return unicode strings. This change would make it easier to write code that works with either string type and would also make some existing code handle unicode strings. The C function PyObject_Str() would remain unchanged and the function PyString_New() would be added instead. Rationale Python has had a Unicode string type for some time now but use of it is not yet widespread. There is a large amount of Python code that assumes that string data is represented as str instances. The long term plan for Python is to phase out the str type and use unicode for all string data. Clearly, a smooth migration path must be provided. We need to upgrade existing libraries, written for str instances, to be made capable of operating in an all-unicode string world. We can't change to an all-unicode world until all essential libraries are made capable for it. Upgrading the libraries in one shot does not seem feasible. A more realistic strategy is to individually make the libraries capable of operating on unicode strings while preserving their current all-str environment behaviour. First, we need to be able to write code that can accept unicode instances without attempting to coerce them to str instances. Let us label such code as Unicode-safe. Unicode-safe libraries can be used in an all-unicode world. Second, we need to be able to write code that, when provided only str instances, will not create unicode results. Let us label such code as str-stable. Libraries that are str-stable can be used by libraries and applications that are not yet Unicode-safe. Sometimes it is simple to write code that is both str-stable and Unicode-safe. For example, the following function just works: def appendx(s): return s + 'x' That's not too surprising since the unicode type is designed to make the task easier. The principle is that when str and unicode instances meet, the result is a unicode instance. One notable difficulty arises when code requires a string representation of an object; an operation traditionally accomplished by using the str() built-in function. Using the current str() function makes the code not Unicode-safe. Replacing a str() call with a unicode() call makes the code not str-stable. Changing str() so that it could return unicode instances would solve this problem. As a further benefit, some code that is currently not Unicode-safe because it uses str() would become Unicode-safe. Specification A Python implementation of the str() built-in follows: def str(s): """Return a nice string representation of the object. The return value is a str or unicode instance. """ if type(s) is str or type(s) is unicode: return s r = s.__str__() if not isinstance(r, (str, unicode)): raise TypeError('__str__ returned non-string') return r The following function would be added to the C API and would be the equivalent to the str() built-in (ideally it be called PyObject_Str, but changing that function could cause a massive number of compatibility problems): PyObject *PyString_New(PyObject *); A reference implementation is available on Sourceforge [1] as a patch. Backwards Compatibility Some code may require that str() returns a str instance. In the standard library, only one such case has been found so far. The function email.header_decode() requires a str instance and the email.Header.decode_header() function tries to ensure this by calling str() on its argument. The code was fixed by changing the line "header = str(header)" to: if isinstance(header, unicode): header = header.encode('ascii') Whether this is truly a bug is questionable since decode_header() really operates on byte strings, not character strings. Code that passes it a unicode instance could itself be considered buggy. Alternative Solutions A new built-in function could be added instead of changing str(). Doing so would introduce virtually no backwards compatibility problems. However, since the compatibility problems are expected to rare, changing str() seems preferable to adding a new built-in. The basestring type could be changed to have the proposed behaviour, rather than changing str(). However, that would be confusing behaviour for an abstract base type. References [1] http://www.python.org/sf/1266570 Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -------------- next part -------------- PEP: 349 Title: Allow str() to return unicode strings Version: $Revision: 1.3 $ Last-Modified: $Date: 2005/08/22 21:12:08 $ Author: Neil Schemenauer Status: Draft Type: Standards Track Content-Type: text/plain Created: 02-Aug-2005 Post-History: 06-Aug-2005 Python-Version: 2.5 Abstract This PEP proposes to change the str() built-in function so that it can return unicode strings. This change would make it easier to write code that works with either string type and would also make some existing code handle unicode strings. The C function PyObject_Str() would remain unchanged and the function PyString_New() would be added instead. Rationale Python has had a Unicode string type for some time now but use of it is not yet widespread. There is a large amount of Python code that assumes that string data is represented as str instances. The long term plan for Python is to phase out the str type and use unicode for all string data. Clearly, a smooth migration path must be provided. We need to upgrade existing libraries, written for str instances, to be made capable of operating in an all-unicode string world. We can't change to an all-unicode world until all essential libraries are made capable for it. Upgrading the libraries in one shot does not seem feasible. A more realistic strategy is to individually make the libraries capable of operating on unicode strings while preserving their current all-str environment behaviour. First, we need to be able to write code that can accept unicode instances without attempting to coerce them to str instances. Let us label such code as Unicode-safe. Unicode-safe libraries can be used in an all-unicode world. Second, we need to be able to write code that, when provided only str instances, will not create unicode results. Let us label such code as str-stable. Libraries that are str-stable can be used by libraries and applications that are not yet Unicode-safe. Sometimes it is simple to write code that is both str-stable and Unicode-safe. For example, the following function just works: def appendx(s): return s + 'x' That's not too surprising since the unicode type is designed to make the task easier. The principle is that when str and unicode instances meet, the result is a unicode instance. One notable difficulty arises when code requires a string representation of an object; an operation traditionally accomplished by using the str() built-in function. Using the current str() function makes the code not Unicode-safe. Replacing a str() call with a unicode() call makes the code not str-stable. Changing str() so that it could return unicode instances would solve this problem. As a further benefit, some code that is currently not Unicode-safe because it uses str() would become Unicode-safe. Specification A Python implementation of the str() built-in follows: def str(s): """Return a nice string representation of the object. The return value is a str or unicode instance. """ if type(s) is str or type(s) is unicode: return s r = s.__str__() if not isinstance(r, (str, unicode)): raise TypeError('__str__ returned non-string') return r The following function would be added to the C API and would be the equivalent to the str() built-in (ideally it be called PyObject_Str, but changing that function could cause a massive number of compatibility problems): PyObject *PyString_New(PyObject *); A reference implementation is available on Sourceforge [1] as a patch. Backwards Compatibility Some code may require that str() returns a str instance. In the standard library, only one such case has been found so far. The function email.header_decode() requires a str instance and the email.Header.decode_header() function tries to ensure this by calling str() on its argument. The code was fixed by changing the line "header = str(header)" to: if isinstance(header, unicode): header = header.encode('ascii') Whether this is truly a bug is questionable since decode_header() really operates on byte strings, not character strings. Code that passes it a unicode instance could itself be considered buggy. Alternative Solutions A new built-in function could be added instead of changing str(). Doing so would introduce virtually no backwards compatibility problems. However, since the compatibility problems are expected to rare, changing str() seems preferable to adding a new built-in. The basestring type could be changed to have the proposed behaviour, rather than changing str(). However, that would be confusing behaviour for an abstract base type. References [1] http://www.python.org/sf/1266570 Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From martin at v.loewis.de Mon Aug 22 23:47:02 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 23:47:02 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <5.1.1.6.0.20050822124128.01b0bd10@mail.telecommunity.com> References: <3F040472-CADF-4F47-8EFD-9B1267C8D0C9@fuhm.net> <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <3F040472-CADF-4F47-8EFD-9B1267C8D0C9@fuhm.net> <5.1.1.6.0.20050822124128.01b0bd10@mail.telecommunity.com> Message-ID: <430A47D6.70704@v.loewis.de> Phillip J. Eby wrote: > You can do that with viewcvs, too. Viewcvs can also create tarballs for > easy downloading, and has a lot of browsing and viewing options that the > SVN webdav mode doesn't. True. I had some issues with viewcvs, though: you cannot provide access control easily, as you cannot force it to slash-separated mode; it also couldn't fetch the history across renames. These may have been fixed meanwhile, of course. Regards, Martin From martin at v.loewis.de Mon Aug 22 23:57:11 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 22 Aug 2005 23:57:11 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <17162.155.1546.1991@montanaro.dyndns.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <17161.62233.743239.89277@montanaro.dyndns.org> <4309FB5A.1040201@v.loewis.de> <17162.155.1546.1991@montanaro.dyndns.org> Message-ID: <430A4A37.1060808@v.loewis.de> skip at pobox.com wrote: > It worked. I made a trivial change to Misc/NEWS and checked it in. I then > ran "svn blame NEWS" to see what it showed. This took approximately > forever. Can I assume this is one thing svn is always going to be pretty > slow at? Yes. Somebody commented that this is quadratic in svn with the number of revisions, whereas it is linear in CVS. Please try it on some other file; Misc/NEWS is probably the worst case in the Python repository. I don't know whether there is any better way; we should perhaps ask on the svn users list. > I notice that you use my real name (including spaces). I doubt we have any > code that munches on annotated listings, but it seems that for the sake of > script writers' sanity it would be better to elide spaces or replace them > with underscores so the annotated user is a single "word": That would be easy to do. For consistency, should we use . (with the usual exceptions 'aahz', 'guido.van.rossum', 'martin.v.loewis')? As for parsing these things: they also show up in 'svn log'. Regards, Martin From db3l at fitlinxx.com Tue Aug 23 03:06:35 2005 From: db3l at fitlinxx.com (David Bolen) Date: 22 Aug 2005 21:06:35 -0400 Subject: [Python-Dev] Admin access using svn+ssh References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <17161.62233.743239.89277@montanaro.dyndns.org> <4309FB5A.1040201@v.loewis.de> <17162.155.1546.1991@montanaro.dyndns.org> <430A4A37.1060808@v.loewis.de> Message-ID: "Martin v. L?wis" writes: > skip at pobox.com wrote: > > It worked. I made a trivial change to Misc/NEWS and checked it in. I then > > ran "svn blame NEWS" to see what it showed. This took approximately > > forever. Can I assume this is one thing svn is always going to be pretty > > slow at? > > Yes. Somebody commented that this is quadratic in svn with the number of > revisions, whereas it is linear in CVS. Please try it on some other > file; Misc/NEWS is probably the worst case in the Python repository. > > I don't know whether there is any better way; we should perhaps ask > on the svn users list. One improvement, if you're looking for a fairly recent change is to bound the blame command with a revision range (I find a date up to HEAD as easiest). You'll miss annotations on lines which were last touched prior to the selected range, but it can definitely speed things up. On a file like News, even if you're generous (say take the last year) it would probably be noticeably faster than letting svn go back to revision 1. -- David From greg.ewing at canterbury.ac.nz Tue Aug 23 05:48:27 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Aug 2005 15:48:27 +1200 Subject: [Python-Dev] On decorators implementation In-Reply-To: <430988FE.2020603@libero.it> References: <43084AE9.20900@libero.it> <430988FE.2020603@libero.it> Message-ID: <430A9C8B.9010704@canterbury.ac.nz> Paolino wrote: > Maybe it's possible to let the decorator know the method class even if > the class is still undefined.(Just like recursive functions?) No, it's not possible. The situation is not the same. With recursive functions, both functions are defined before either of them is called. But decorators in a class body are executed before the surrounding class even exists. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing at canterbury.ac.nz +--------------------------------------+ From paragate at gmx.net Tue Aug 23 10:46:36 2005 From: paragate at gmx.net (Wolfgang Lipp) Date: Tue, 23 Aug 2005 10:46:36 +0200 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: <20050822213142.GA5702@mems-exchange.org> References: <20050822213142.GA5702@mems-exchange.org> Message-ID: neil, i just intended to worry that returning a unicode object from ``str()`` would break assumptions about the way that 'type definers' like ``str()``, ``int()``, ``float()`` and so on work, but i quickly realized that e.g. ``int()`` does return a long where appropriate! since the principle works there one may surmise it will also work for ``str()`` in the long run. one point i don't seem to understand right now is why it says in the function definition:: if type(s) is str or type(s) is unicode: ... instead of using ``isinstance()``. Testing for ``type()`` means that instances of derived classes (that may or may not change nothing or almost nothing to the underlying class) when passed to a function that uses ``str()`` will behave in a different way! isn't it more realistic and commonplace to assume that derivatives of a class do fulfill the requirements of the underlying class? -- which may turn out to be wrong! but still... the code as it stands means i have to remember that *in this special case only* (when deriving from ``unicode``), i have to add a ``__str__()`` method myself that simply returns ``self``. then of course, one could change ``unicode.__str__()`` to return ``self``, itself, which should work. but then, why so complicated? i suggest to change said line to:: if isinstance( s, ( str, unicode ) ): ... any objections? _wolf -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/ From martin at v.loewis.de Tue Aug 23 12:03:06 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 23 Aug 2005 12:03:06 +0200 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <1124728871.17084.13.camel@geddy.wooz.org> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <4309FA58.4080103@v.loewis.de> <1124728871.17084.13.camel@geddy.wooz.org> Message-ID: <430AF45A.1090506@v.loewis.de> Barry Warsaw wrote: >>Not sure what "the right place" would be: pythondev at python.org? >>I think the email could look any way we want it to look. > > > I think it should be @python.org where is the > firstname.lastname (with some exceptions) scheme that we've agreed on. > I actually /don't/ want all commits to look like they're coming from > pythondev at python.org Ok, I have now changed all user names for the python repository to firstname.lastname. That should allow to use them in From: fields of commit email. Regards, Martin From theller at python.net Tue Aug 23 12:19:11 2005 From: theller at python.net (Thomas Heller) Date: Tue, 23 Aug 2005 12:19:11 +0200 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings References: <20050822213142.GA5702@mems-exchange.org> Message-ID: Neil Schemenauer writes: > [Please mail followups to python-dev at python.org.] > > The PEP has been rewritten based on a suggestion by Guido to change > str() rather than adding a new built-in function. Based on my > testing, I believe the idea is feasible. It would be helpful if > people could test the patched Python with their own applications and > report any incompatibilities. > I like the fact that currently unicode(x) is guarateed to return a unicode instance, or raises a UnicodeDecodeError. Same for str(x), which is guaranteed to return a (byte) string instance or raise an error. Wouldn't also a new function make the intent clearer? So I think I'm +1 on the text() built-in, and -0 on changing str. Thomas From martin at v.loewis.de Tue Aug 23 12:38:05 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 23 Aug 2005 12:38:05 +0200 Subject: [Python-Dev] Subversion instructions Message-ID: <430AFC8D.6020000@v.loewis.de> As some people have been struggling with svn+ssh, I wrote a few instructions at http://www.python.org/dev/svn.html The main issues people have been struggling with are: - you really should use an agent, or else you have to type the private key passphrase three times on checkout - on windows, putty works fine, but you really should use the agent (pageant), or else plink might not find your key. Also, if you use Putty profiles, make sure to add the user name (pythondev) into the profile - we need SSH2 keys; SSH1 is disabled on svn.python.org. Some of you had been using SSH1 keys on sf.net all these years; you will need to generate SSH2 keys. Regards, Martin From mal at egenix.com Tue Aug 23 12:39:03 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 23 Aug 2005 12:39:03 +0200 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: References: <20050822213142.GA5702@mems-exchange.org> Message-ID: <430AFCC7.9030402@egenix.com> Thomas Heller wrote: > Neil Schemenauer writes: > > >>[Please mail followups to python-dev at python.org.] >> >>The PEP has been rewritten based on a suggestion by Guido to change >>str() rather than adding a new built-in function. Based on my >>testing, I believe the idea is feasible. It would be helpful if >>people could test the patched Python with their own applications and >>report any incompatibilities. >> > > > I like the fact that currently unicode(x) is guarateed to return a > unicode instance, or raises a UnicodeDecodeError. Same for str(x), > which is guaranteed to return a (byte) string instance or raise an > error. > > Wouldn't also a new function make the intent clearer? > > So I think I'm +1 on the text() built-in, and -0 on changing str. Same here. A new API would also help make the transition easier from the current mixed data/text type (strings) to data-only (bytes) and text-only (text, renamed from unicode) in Py3.0. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 23 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From p.f.moore at gmail.com Tue Aug 23 12:41:11 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 23 Aug 2005 11:41:11 +0100 Subject: [Python-Dev] Admin access using svn+ssh In-Reply-To: <4309FBE5.40204@v.loewis.de> References: <43087DA0.702@v.loewis.de> <1124665281.31664.28.camel@geddy.wooz.org> <43096E37.8070708@v.loewis.de> <17161.60617.950268.641009@montanaro.dyndns.org> <1124724730.17082.8.camel@geddy.wooz.org> <3F040472-CADF-4F47-8EFD-9B1267C8D0C9@fuhm.net> <4309FBE5.40204@v.loewis.de> Message-ID: <79990c6b05082303415114abaf@mail.gmail.com> On 8/22/05, "Martin v. L?wis" wrote: > James Y Knight wrote: > > It seems a waste to use SVN's webdav support just for anon access. > > The svnserve method works well for anon access. The only reason to > > use svn webdav IMO is if you want to use that for authenticated > > access. But since you're talking about using svn+ssh for that.. > > It has the advantage that we can easily point people to files > with a web browser; they don't need an svn client. It also allows anonymous svn checkouts for people behind firewalls that only allow HTTP through. Paul. From paragate at gmx.net Tue Aug 23 14:59:28 2005 From: paragate at gmx.net (Wolfgang Lipp) Date: Tue, 23 Aug 2005 14:59:28 +0200 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: <430AFCC7.9030402@egenix.com> References: <20050822213142.GA5702@mems-exchange.org> <430AFCC7.9030402@egenix.com> Message-ID: just tested the proposed implementation on a unicode-naive module basically using import sys import __builtin__ reload( sys ); sys.setdefaultencoding( 'utf-8' ) __builtin__.__dict__[ 'str' ] = new_str_function et voil?, str() calls in the module are rewritten, and print u'd?sseldorf' does work as expected(*) (even on systems where i have no access to sitecustomize, like at my python-friendly isp's servers). --- * my expectation is that unicode strings do print out as utf-8, as i can't see any better solution. i suggest to make this option available e.g. via a module in the standard lib to ease transition for people in case the pep doesn't make it. it may be applied where deemed necessary and left ignored otherwise. if nobody thinks the reload hack is too awful and this solution stands testing, i guess i'll post it to the aspn cookbook. after all these countless hours of hunting down ordinal not in range, finally i'm starting to see some light in the issue. _wolf On Tue, 23 Aug 2005 12:39:03 +0200, M.-A. Lemburg wrote: > Thomas Heller wrote: >> Neil Schemenauer writes: >> >> >>> [Please mail followups to python-dev at python.org.] >>> >>> The PEP has been rewritten based on a suggestion by Guido to change >>> str() rather than adding a new built-in function. Based on my >>> testing, I believe the idea is feasible. It would be helpful if >>> people could test the patched Python with their own applications and >>> report any incompatibilities. >>> >> >> >> I like the fact that currently unicode(x) is guarateed to return a >> unicode instance, or raises a UnicodeDecodeError. Same for str(x), >> which is guaranteed to return a (byte) string instance or raise an >> error. >> >> Wouldn't also a new function make the intent clearer? >> >> So I think I'm +1 on the text() built-in, and -0 on changing str. > > Same here. > > A new API would also help make the transition easier from the > current mixed data/text type (strings) to data-only (bytes) > and text-only (text, renamed from unicode) in Py3.0. > -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/ From raymond.hettinger at verizon.net Tue Aug 23 16:11:56 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 23 Aug 2005 10:11:56 -0400 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: <20050821184613.A45C11E4288@bag.python.org> Message-ID: <000601c5a7ec$9f614680$8901a044@oemcomputer> This patch should be reverted or fixed so that the Py2.5 build works again. It contains a disasterous search and replace error that prevents it from compiling. Hence, it couldn't have passed the test suite before being checked in. Also, all of the project and config files need to be updated for the new modules. > -----Original Message----- > From: python-checkins-bounces at python.org [mailto:python-checkins- > bounces at python.org] On Behalf Of greg at users.sourceforge.net > Sent: Sunday, August 21, 2005 2:46 PM > To: python-checkins at python.org > Subject: [Python-checkins] python/dist/src/Modules _hashopenssl.c, > NONE,2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE,2.1 md5module.c, > 2.35, 2.36 shamodule.c, 2.22, 2.23 > > Update of /cvsroot/python/python/dist/src/Modules > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv32064/Modules > > Modified Files: > md5module.c shamodule.c > Added Files: > _hashopenssl.c sha256module.c sha512module.c > Log Message: > [ sf.net patch # 1121611 ] > > A new hashlib module to replace the md5 and sha modules. It adds > support for additional secure hashes such as SHA-256 and SHA-512. The > hashlib module uses OpenSSL for fast platform optimized > implementations of algorithms when available. The old md5 and sha > modules still exist as wrappers around hashlib to preserve backwards > compatibility. From mwh at python.net Tue Aug 23 16:31:55 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 23 Aug 2005 15:31:55 +0100 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: <000601c5a7ec$9f614680$8901a044@oemcomputer> (Raymond Hettinger's message of "Tue, 23 Aug 2005 10:11:56 -0400") References: <000601c5a7ec$9f614680$8901a044@oemcomputer> Message-ID: <2mvf1wsp0k.fsf@starship.python.net> "Raymond Hettinger" writes: > This patch should be reverted or fixed so that the Py2.5 build works > again. > > It contains a disasterous search and replace error that prevents it from > compiling. Hence, it couldn't have passed the test suite before being > checked in. It works for me, on OS X. Passes the test suite, even. I presume you're on Windows of some kind? > Also, all of the project and config files need to be updated for the new > modules. Well, yes. But if Greg is on some unix-a-like, he can only update the unix build files (which he has done; it's in setup.py). Cheers, mwh -- If you are anal, and you love to be right all the time, C++ gives you a multitude of mostly untimportant details to fret about so you can feel good about yourself for getting them "right", while missing the big picture entirely -- from Twisted.Quotes From mwh at python.net Tue Aug 23 16:33:04 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 23 Aug 2005 15:33:04 +0100 Subject: [Python-Dev] PEP 342 Implementation In-Reply-To: <000001c5991d$e40bb140$12b62c81@oemcomputer> (Raymond Hettinger's message of "Thu, 04 Aug 2005 13:56:50 -0400") References: <000001c5991d$e40bb140$12b62c81@oemcomputer> Message-ID: <2mr7cksoyn.fsf@starship.python.net> "Raymond Hettinger" writes: > Could someone please make an independent check to verify an issue with > the 342 checkin. The test suite passes but when I run IDLE and open a > new window (using Control-N), it crashes and burns. > > The problem does not occur just before the checkin: > cvs up -D "2005-08-01 18:00" > But emerges immediately after: > cvs up -D "2005-08-01 21:00" Is this still happening? I'm not seeing any unusual flakiness, but then I can't run IDLE (OS X, no Tk). It's not exactly a minimal test case :) Cheers, mwh -- A difference which makes no difference is no difference at all. -- William James (I think. Reference anyone?) From raymond.hettinger at verizon.net Tue Aug 23 17:03:12 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 23 Aug 2005 11:03:12 -0400 Subject: [Python-Dev] PEP 342 Implementation In-Reply-To: <2mr7cksoyn.fsf@starship.python.net> Message-ID: <000001c5a7f3$c8e0e2c0$8901a044@oemcomputer> [Raymond Hettinger] > > > Could someone please make an independent check to verify an issue with > > the 342 checkin. The test suite passes but when I run IDLE and open a > > new window (using Control-N), it crashes and burns. > > > > The problem does not occur just before the checkin: > > cvs up -D "2005-08-01 18:00" > > But emerges immediately after: > > cvs up -D "2005-08-01 21:00" > > Is this still happening? I'm not seeing any unusual flakiness, but > then I can't run IDLE (OS X, no Tk). Yes, it is still happening. No one has yet offered an independent confirmation. > It's not exactly a minimal test case :) Right ;-) Once narrowed down, the problem and solution will likely be obvious. Raymond From fredrik at pythonware.com Tue Aug 23 16:58:53 2005 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 23 Aug 2005 16:58:53 +0200 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings References: <20050822213142.GA5702@mems-exchange.org> Message-ID: Neil Schemenauer wrote: > The PEP has been rewritten based on a suggestion by Guido to change > str() rather than adding a new built-in function. Based on my testing, I > believe the idea is feasible. note that this breaks chapter 3 of the tutorial: http://docs.python.org/tut/node5.html#SECTION005130000000000000000 where str() is first introduced. From raymond.hettinger at verizon.net Tue Aug 23 17:16:11 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 23 Aug 2005 11:16:11 -0400 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: <2mvf1wsp0k.fsf@starship.python.net> Message-ID: <000101c5a7f5$989d5100$8901a044@oemcomputer> [Raymond Hettinger] > > This patch should be reverted or fixed so that the Py2.5 build works > > again. > > > > It contains a disasterous search and replace error that prevents it from > > compiling. Hence, it couldn't have passed the test suite before being > > checked in. [Michael Hudson] > It works for me, on OS X. Passes the test suite, even. I presume > you're on Windows of some kind? Here's an excerpt from the check-in note for sha512module.c: RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98d728ae22ULL); RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x7137449123ef65cdULL); RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcfec4d3b2fULL); RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba58189dbbcULL); RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25bf348b538ULL); Perhaps OS X has some sort of Steve Jobs special constant suffix "ULL" that Mr. Gates and the ANSI C folks have yet to accept ;-) If it works for you, then it probably means that sha512module.c was left out of the build. Maybe sha512module.c wasn't supposed to be checked in? > > Also, all of the project and config files need to be updated for the new > > modules. > > Well, yes. But if Greg is on some unix-a-like, he can only update the > unix build files (which he has done; it's in setup.py). The project files are just text files and can be updated simply and directly. But yes, that is no big deal and I'll just do it for him once the code gets to a compilable state. Aside from the project files, there is still config.c and whatnot. We should put together a checklist of all the things that need to be updated when a new module is added. Raymond From nas at arctrix.com Tue Aug 23 17:21:57 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Tue, 23 Aug 2005 09:21:57 -0600 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: References: <20050822213142.GA5702@mems-exchange.org> Message-ID: <20050823152156.GA7839@mems-exchange.org> On Tue, Aug 23, 2005 at 10:46:36AM +0200, Wolfgang Lipp wrote: > one point i don't seem to understand right now is why it says in the > function definition:: > > if type(s) is str or type(s) is unicode: > ... > > instead of using ``isinstance()``. I don't think isinstance() would be okay. That test is meant as an optimization to avoid calling __str__ on str and unicode instances. Subclasses should still have their __str__ method called otherwise they cannot override it. > the code as it stands means i have to remember that *in this special > case only* (when deriving from ``unicode``), i have to add a > ``__str__()`` method myself that simply returns ``self``. Ah, I see that unicode.__str__ returns a str instance. > then of course, one could change ``unicode.__str__()`` to return > ``self``, itself, which should work. but then, why so complicated? I think that may be the right fix. Neil From gmccaughan at synaptics-uk.com Tue Aug 23 17:32:29 2005 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Tue, 23 Aug 2005 16:32:29 +0100 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: <000101c5a7f5$989d5100$8901a044@oemcomputer> References: <000101c5a7f5$989d5100$8901a044@oemcomputer> Message-ID: <200508231632.30175.gmccaughan@synaptics-uk.com> > Here's an excerpt from the check-in note for sha512module.c: > > > RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98d728ae22ULL); > RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x7137449123ef65cdULL); > RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcfec4d3b2fULL); > RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba58189dbbcULL); > RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25bf348b538ULL); > > Perhaps OS X has some sort of Steve Jobs special constant suffix "ULL" > that Mr. Gates and the ANSI C folks have yet to accept ;-) It's valid C99, meaning "this is an unsigned long long". -- g From pje at telecommunity.com Tue Aug 23 17:43:02 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 23 Aug 2005 11:43:02 -0400 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: <20050823152156.GA7839@mems-exchange.org> References: <20050822213142.GA5702@mems-exchange.org> Message-ID: <5.1.1.6.0.20050823112823.01b22d18@mail.telecommunity.com> At 09:21 AM 8/23/2005 -0600, Neil Schemenauer wrote: > > then of course, one could change ``unicode.__str__()`` to return > > ``self``, itself, which should work. but then, why so complicated? > >I think that may be the right fix. No, it isn't. Right now str(u"x") coerces the unicode object to a string, so changing this will be backwards-incompatible with any existing programs. I think the new builtin is actually the right way to go for both 2.x and 3.x Pythons. i.e., text() would be a builtin in 2.x, along with a new bytes() type, and in 3.x text() could replace the basestring, str and unicode types. I also think that the text() constructor should have a signature of 'text(ob,encoding="ascii")'. In the default case, strings can be returned by text() as long as they are pure ASCII (making the code str-stable *and* unicode-safe). In the non-default case, a unicode object should always be returned, making the code unicode-safe but not str-stable. Allowing text() to return 8-bit strings would be an obvious violation of its name: it's for text, not bytes. From paragate at gmx.net Tue Aug 23 17:45:27 2005 From: paragate at gmx.net (Wolfgang Lipp) Date: Tue, 23 Aug 2005 17:45:27 +0200 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: <20050823152156.GA7839@mems-exchange.org> References: <20050822213142.GA5702@mems-exchange.org> <20050823152156.GA7839@mems-exchange.org> Message-ID: i have to revise my last posting -- exporting the new ``str`` pure-python implementation breaks -- of course! -- as soon as ``isinstance(x,str)`` [sic] is used. right now it breaks because you can't have a function as the second argument of ``isinstance()``, but even if that could be avoided by canny programming, the fact remains that any object derived from e.g. a string literal will still be constructed from the underlying implementation and can't therefore be an instance of the old ``str``. also, ``str.__bases__`` is not extendable (it's a tuple) and not replaceable (it's a built-in), so there seems to be no way to get near a truly working solution except with C-level patches. On Tue, 23 Aug 2005 17:21:57 +0200, Neil Schemenauer wrote: > I don't think isinstance() would be okay. That test is meant as an > optimization to avoid calling __str__ on str and unicode instances. > Subclasses should still have their __str__ method called otherwise > they cannot override it. makes perfect sense, i'll change the line back. _wolf From mwh at python.net Tue Aug 23 17:44:56 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 23 Aug 2005 16:44:56 +0100 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: <000101c5a7f5$989d5100$8901a044@oemcomputer> (Raymond Hettinger's message of "Tue, 23 Aug 2005 11:16:11 -0400") References: <000101c5a7f5$989d5100$8901a044@oemcomputer> Message-ID: <2mmzn8slmv.fsf@starship.python.net> "Raymond Hettinger" writes: > [Raymond Hettinger] >> > This patch should be reverted or fixed so that the Py2.5 build works >> > again. >> > >> > It contains a disasterous search and replace error that prevents it > from >> > compiling. Hence, it couldn't have passed the test suite before > being >> > checked in. > > [Michael Hudson] >> It works for me, on OS X. Passes the test suite, even. I presume >> you're on Windows of some kind? > > > Here's an excerpt from the check-in note for sha512module.c: > > > RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98d728ae22ULL); > > RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x7137449123ef65cdULL); > > RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcfec4d3b2fULL); > > RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba58189dbbcULL); > > RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25bf348b538ULL); > > Perhaps OS X has some sort of Steve Jobs special constant suffix "ULL" > that Mr. Gates and the ANSI C folks have yet to accept ;-) It's an C99 unsigned long long literal, AFAICT (p70 of the PDF I found lying around somewhere...), so I think it's just Bill who's behind. However, Python doesn't require C99, so it's pretty dodgy code by our standards. Hmm. You have PY_LONG_LONG #define-d, right? Does VC++ 6 (that's what you use, right?) support any kind of long long literal? > If it works for you, then it probably means that sha512module.c was left > out of the build. Nope: [mwh at 82-33-185-193 build-debug]$ ./python.exe Python 2.5a0 (#1, Aug 23 2005, 13:24:32) [GCC 3.3 20030304 (Apple Computer, Inc. build 1671)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import _sha512 [44297 refs] > Maybe sha512module.c wasn't supposed to be checked in? I think if you have a sufficiently modern openssl it's unnecessary. > The project files are just text files and can be updated simply and > directly. But yes, that is no big deal and I'll just do it for him once > the code gets to a compilable state. > > Aside from the project files, there is still config.c and whatnot. Does anything need to be done there? Oh, PC/config.c, right? > We should put together a checklist of all the things that need to be > updated when a new module is added. Sounds like it! :) Cheers, mwh -- This makes it possible to pass complex object hierarchies to a C coder who thinks computer science has made no worthwhile advancements since the invention of the pointer. -- Gordon McMillan, 30 Jul 1998 From fredrik at pythonware.com Tue Aug 23 17:51:34 2005 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 23 Aug 2005 17:51:34 +0200 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules_hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 References: <000101c5a7f5$989d5100$8901a044@oemcomputer> <200508231632.30175.gmccaughan@synaptics-uk.com> Message-ID: Gareth McCaughan wrote: > It's valid C99, meaning "this is an unsigned long long". since when does Python require C99 compilers? From mcherm at mcherm.com Tue Aug 23 18:11:02 2005 From: mcherm at mcherm.com (Michael Chermside) Date: Tue, 23 Aug 2005 09:11:02 -0700 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicodestrings Message-ID: <20050823091102.ay5mcm8r2gco4488@login.werra.lunarpages.com> Neil Schemenauer wrote: > The PEP has been rewritten based on a suggestion by Guido to change > str() rather than adding a new built-in function. Based on my testing, I > believe the idea is feasible. Fredrik Lundh replies: > note that this breaks chapter 3 of the tutorial: > > http://docs.python.org/tut/node5.html#SECTION005130000000000000000 > > where str() is first introduced. It's hardly "introduced"... the only bit I found reads: ... When a Unicode string is printed, written to a file, or converted with str(), conversion takes place using this default encoding. >>> u"abc" u'abc' >>> str(u"abc") 'abc' >>> u"???" u'\xe4\xf6\xfc' >>> str(u"???") Traceback (most recent call last): File " ", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) To convert a Unicode string into an 8-bit string using a specific encoding, Unicode objects provide an encode() method that takes one argument, the name of the encoding. Lowercase names for encodings are preferred. >>> u"???".encode('utf-8') '\xc3\xa4\xc3\xb6\xc3\xbc' I think that if we just took out the example of str() usage and replaced it with a sentence or two that DID introduce the (revised) str() function, it ought to work. In particular, it could mention that you can call str() on any object, which isn't stated here at all. -- Michael Chermside From theller at python.net Tue Aug 23 18:08:42 2005 From: theller at python.net (Thomas Heller) Date: Tue, 23 Aug 2005 18:08:42 +0200 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 References: <000101c5a7f5$989d5100$8901a044@oemcomputer> <2mmzn8slmv.fsf@starship.python.net> Message-ID: Michael Hudson writes: > "Raymond Hettinger" writes: > >> [Raymond Hettinger] >>> > This patch should be reverted or fixed so that the Py2.5 build works >>> > again. >>> > >>> > It contains a disasterous search and replace error that prevents it >> from >>> > compiling. Hence, it couldn't have passed the test suite before >> being >>> > checked in. >> >> [Michael Hudson] >>> It works for me, on OS X. Passes the test suite, even. I presume >>> you're on Windows of some kind? >> >> >> Here's an excerpt from the check-in note for sha512module.c: >> >> >> RND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98d728ae22ULL); >> >> RND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x7137449123ef65cdULL); >> >> RND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcfec4d3b2fULL); >> >> RND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba58189dbbcULL); >> >> RND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25bf348b538ULL); >> >> Perhaps OS X has some sort of Steve Jobs special constant suffix "ULL" >> that Mr. Gates and the ANSI C folks have yet to accept ;-) > > It's an C99 unsigned long long literal, AFAICT (p70 of the PDF I found > lying around somewhere...), so I think it's just Bill who's behind. > However, Python doesn't require C99, so it's pretty dodgy code by our > standards. > > Hmm. You have PY_LONG_LONG #define-d, right? Does VC++ 6 (that's > what you use, right?) support any kind of long long literal? The suffix seems to be 'ui64'. From vc6 limits.h: #if _INTEGRAL_MAX_BITS >= 64 /* minimum signed 64 bit value */ #define _I64_MIN (-9223372036854775807i64 - 1) /* maximum signed 64 bit value */ #define _I64_MAX 9223372036854775807i64 /* maximum unsigned 64 bit value */ #define _UI64_MAX 0xffffffffffffffffui64 #endif Thomas From abkhd at hotmail.com Tue Aug 23 18:23:33 2005 From: abkhd at hotmail.com (A.B., Khalid) Date: Tue, 23 Aug 2005 16:23:33 +0000 Subject: [Python-Dev] Modules _hashopenssl, sha256, sha512 compile in MinGW, test_hmac.py passes Message-ID: Hello, I can also report that MinGW can compile the said modules and (after updating config.c, etc.) the resulting code passes as follows: $ python -i ../Lib/test/test_hmac.py test_md5_vectors (__main__.TestVectorsTestCase) ... ok test_sha_vectors (__main__.TestVectorsTestCase) ... ok test_normal (__main__.ConstructorTestCase) ... ok test_withmodule (__main__.ConstructorTestCase) ... ok test_withtext (__main__.ConstructorTestCase) ... ok test_default_is_md5 (__main__.SanityTestCase) ... ok test_exercise_all_methods (__main__.SanityTestCase) ... ok test_attributes (__main__.CopyTestCase) ... ok test_equality (__main__.CopyTestCase) ... ok test_realcopy (__main__.CopyTestCase) ... ok ---------------------------------------------------------------------- Ran 10 tests in 0.050s OK >>> Are these moduels going to be built into the core? Regards Khalid _________________________________________________________________ FREE pop-up blocking with the new MSN Toolbar - get it now! http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/ From gmccaughan at synaptics-uk.com Tue Aug 23 18:38:20 2005 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Tue, 23 Aug 2005 17:38:20 +0100 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules_hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: References: <000101c5a7f5$989d5100$8901a044@oemcomputer> <200508231632.30175.gmccaughan@synaptics-uk.com> Message-ID: <200508231738.20961.gmccaughan@synaptics-uk.com> On Tuesday 2005-08-23 16:51, Fredrik Lundh wrote: > Gareth McCaughan wrote: > > > It's valid C99, meaning "this is an unsigned long long". > > since when does Python require C99 compilers? > > It doesn't, of course, and I hope it won't for a good while. I was just responding to this: | Perhaps OS X has some sort of Steve Jobs special constant suffix "ULL" | that Mr. Gates and the ANSI C folks have yet to accept since in fact Mr Gates and the ANSI C folks (and the gcc folks, and probably plenty of others I can't check so easily) *have* accepted it. -- g From raymond.hettinger at verizon.net Tue Aug 23 18:46:58 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 23 Aug 2005 12:46:58 -0400 Subject: [Python-Dev] [Python-checkins]python/dist/src/Modules_hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: Message-ID: <000401c5a802$478c38a0$8901a044@oemcomputer> [Gareth] > > It's valid C99, meaning "this is an unsigned long long". > since when does Python require C99 compilers? Except from PEP 7: "Use ANSI/ISO standard C (the 1989 version of the standard)." From mwh at python.net Tue Aug 23 18:51:05 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 23 Aug 2005 17:51:05 +0100 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules_hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: (Fredrik Lundh's message of "Tue, 23 Aug 2005 17:51:34 +0200") References: <000101c5a7f5$989d5100$8901a044@oemcomputer> <200508231632.30175.gmccaughan@synaptics-uk.com> Message-ID: <2mirxwsikm.fsf@starship.python.net> "Fredrik Lundh" writes: > Gareth McCaughan wrote: > >> It's valid C99, meaning "this is an unsigned long long". > > since when does Python require C99 compilers? Well, it doesn't, but Raymond was suggesting the code was GCC specific, or something. Cheers, mwh -- Check out the comments in this source file that start with: # Oh, lord help us. -- Mark Hammond gets to play with the Outlook object model From raymond.hettinger at verizon.net Tue Aug 23 17:47:14 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 23 Aug 2005 11:47:14 -0400 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: <2mmzn8slmv.fsf@starship.python.net> Message-ID: <000101c5a7f9$efb77480$8901a044@oemcomputer> [Michael Hudson] > It's an C99 unsigned long long literal, AFAICT (p70 of the PDF I found > lying around somewhere...), so I think it's just Bill who's behind. > However, Python doesn't require C99, so it's pretty dodgy code by our > standards. More than just dodgy. Except from PEP 7: "Use ANSI/ISO standard C (the 1989 version of the standard)." Raymond From nas at arctrix.com Tue Aug 23 18:54:09 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Tue, 23 Aug 2005 10:54:09 -0600 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: <5.1.1.6.0.20050823112823.01b22d18@mail.telecommunity.com> References: <20050822213142.GA5702@mems-exchange.org> <5.1.1.6.0.20050823112823.01b22d18@mail.telecommunity.com> Message-ID: <20050823165409.GA8026@mems-exchange.org> On Tue, Aug 23, 2005 at 11:43:02AM -0400, Phillip J. Eby wrote: > At 09:21 AM 8/23/2005 -0600, Neil Schemenauer wrote: > >> then of course, one could change ``unicode.__str__()`` to return > >> ``self``, itself, which should work. but then, why so complicated? > > > >I think that may be the right fix. > > No, it isn't. Right now str(u"x") coerces the unicode object to a > string, so changing this will be backwards-incompatible with any > existing programs. I meant that for the implementation of the PEP, changing unicode.__str__ to return self seems to be the right fix. Whether you believe that str() should be allowed to return unicode instances is a different question. > I think the new builtin is actually the right way to go for both 2.x and > 3.x Pythons. i.e., text() would be a builtin in 2.x, along with a new > bytes() type, and in 3.x text() could replace the basestring, str and > unicode types. Perhaps the critical question is what will the string type in P3k be called? If it will be 'str' then I think the PEP makes sense. If it will be something else, then there should be a corresponding type slot (e.g. __text__). What method does your proposed text() built-in call? > I also think that the text() constructor should have a signature of > 'text(ob,encoding="ascii")'. I think that's a bad idea. We want to get away from ASCII and use Unicode instead. > In the default case, strings can be returned by text() as long as > they are pure ASCII (making the code str-stable *and* > unicode-safe). I think you misunderstand the PEP. Your proposed function is neither Unicode-safe nor str-stable, the worst of both worlds. Passing it a unicode string that contains non-ASCII characters would result in an exception (not Unicode-safe). Passing it a str results in a unicode return value (not str-stable). Neil From nas at arctrix.com Tue Aug 23 19:00:06 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Tue, 23 Aug 2005 11:00:06 -0600 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: References: <20050822213142.GA5702@mems-exchange.org> <20050823152156.GA7839@mems-exchange.org> Message-ID: <20050823170003.GB8026@mems-exchange.org> On Tue, Aug 23, 2005 at 05:45:27PM +0200, Wolfgang Lipp wrote: > i have to revise my last posting -- exporting the new ``str`` > pure-python implementation breaks -- of course! -- as soon > as ``isinstance(x,str)`` [sic] is used Right. I tried to come up with a pure Python version so people could test their code. This was my latest attempt before giving up (from memory): # inside site.py _old_str_new = str.__new__ def _str_new(self, s): if type(self) not in (str, unicode): return _old_str_new(self, s) if type(s) not in (str, unicode): return s r = s.__str__() if not isinstance(r, (str, unicode)): raise TypeError('__str__ returned non-string') return r str.__new__ = _str_new It doesn't work though: TypeError: can't set attributes of built-in/extension type 'str' Maybe someone else has a clever solution. Neil From pje at telecommunity.com Tue Aug 23 19:14:24 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 23 Aug 2005 13:14:24 -0400 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: <20050823165409.GA8026@mems-exchange.org> References: <5.1.1.6.0.20050823112823.01b22d18@mail.telecommunity.com> <20050822213142.GA5702@mems-exchange.org> <5.1.1.6.0.20050823112823.01b22d18@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050823130921.02a43da8@mail.telecommunity.com> At 10:54 AM 8/23/2005 -0600, Neil Schemenauer wrote: >On Tue, Aug 23, 2005 at 11:43:02AM -0400, Phillip J. Eby wrote: > > At 09:21 AM 8/23/2005 -0600, Neil Schemenauer wrote: > > >> then of course, one could change ``unicode.__str__()`` to return > > >> ``self``, itself, which should work. but then, why so complicated? > > > > > >I think that may be the right fix. > > > > No, it isn't. Right now str(u"x") coerces the unicode object to a > > string, so changing this will be backwards-incompatible with any > > existing programs. > >I meant that for the implementation of the PEP, changing >unicode.__str__ to return self seems to be the right fix. Whether >you believe that str() should be allowed to return unicode instances >is a different question. > > > I think the new builtin is actually the right way to go for both 2.x and > > 3.x Pythons. i.e., text() would be a builtin in 2.x, along with a new > > bytes() type, and in 3.x text() could replace the basestring, str and > > unicode types. > >Perhaps the critical question is what will the string type in P3k be >called? If it will be 'str' then I think the PEP makes sense. If >it will be something else, then there should be a corresponding type >slot (e.g. __text__). What method does your proposed text() >built-in call? Heck if I know. :) I think the P3k string type should just be called 'text', though, so we can leave the whole unicode/str mess behind. > > I also think that the text() constructor should have a signature of > > 'text(ob,encoding="ascii")'. > >I think that's a bad idea. We want to get away from ASCII and use >Unicode instead. It's not str-stable if it returns unicode for a string input. > > In the default case, strings can be returned by text() as long as > > they are pure ASCII (making the code str-stable *and* > > unicode-safe). > >I think you misunderstand the PEP. Your proposed function is >neither Unicode-safe nor str-stable, the worst of both worlds. >Passing it a unicode string that contains non-ASCII characters would >result in an exception (not Unicode-safe). Passing it a str results >in a unicode return value (not str-stable). I think you misunderstand my proposal. :) I'm proposing rough semantics of: def text(ob, encoding='ascii'): if isinstance(ob,unicode): return ob ob = str(ob) # or ob.__text__, then fallback to __unicode__/__str__ if encoding=='ascii' and isinstance(ob,str): unicode(ob,encoding) # check for purity return ob # return the string if it's pure return unicode(ob, encoding) This is str-stable *and* unicode-safe. From reinhold-birkenfeld-nospam at wolke7.net Tue Aug 23 19:23:25 2005 From: reinhold-birkenfeld-nospam at wolke7.net (Reinhold Birkenfeld) Date: Tue, 23 Aug 2005 19:23:25 +0200 Subject: [Python-Dev] python/dist/src/Doc/tut tut.tex,1.276,1.277 In-Reply-To: <20050823150057.057C91E400B@bag.python.org> References: <20050823150057.057C91E400B@bag.python.org> Message-ID: rhettinger at users.sourceforge.net wrote: I'm not a native speaker, but... > @@ -114,7 +114,7 @@ > programs, or to test functions during bottom-up program development. > It is also a handy desk calculator. > > -Python allows writing very compact and readable programs. Programs > +Python enables programs to written compactly and readably. Programs > written in Python are typically much shorter than equivalent C or > \Cpp{} programs, for several reasons: > \begin{itemize} ...shouldn't it be "programs to be written compactly"? > @@ -1753,8 +1753,8 @@ > > \begin{methoddesc}[list]{pop}{\optional{i}} > Remove the item at the given position in the list, and return it. If > -no index is specified, \code{a.pop()} returns the last item in the > -list. The item is also removed from the list. (The square brackets > +no index is specified, \code{a.pop()} removes and returns the last item > +in the list. The item is also removed from the list. (The square brackets > around the \var{i} in the method signature denote that the parameter > is optional, not that you should type square brackets at that > position. You will see this notation frequently in the Thats twice the same the same (removal from list). > @@ -1985,7 +1987,9 @@ > \section{The \keyword{del} statement \label{del}} > > There is a way to remove an item from a list given its index instead > -of its value: the \keyword{del} statement. This can also be used to > +of its value: the \keyword{del} statement. Unlike the \method{pop()}) > +method which returns a value, the \keyword{del} keyword is a statement > +and can also be used to > remove slices from a list (which we did earlier by assignment of an > empty list to the slice). For example: The del keyword is a statement? > @@ -2133,8 +2137,8 @@ > keys. Tuples can be used as keys if they contain only strings, > numbers, or tuples; if a tuple contains any mutable object either > directly or indirectly, it cannot be used as a key. You can't use > -lists as keys, since lists can be modified in place using their > -\method{append()} and \method{extend()} methods, as well as slice and > +lists as keys, since lists can be modified in place using methods like > +\method{append()} and \method{extend()} or modified with slice and > indexed assignments. Is the second "modified" necessary? > @@ -5595,8 +5603,8 @@ > to round it again can't make it better: it was already as good as it > gets. > > -Another consequence is that since 0.1 is not exactly 1/10, adding 0.1 > -to itself 10 times may not yield exactly 1.0, either: > +Another consequence is that since 0.1 is not exactly 1/10, > +summing ten values of 0.1 may not yield exactly 1.0, either: > > \begin{verbatim} > >>> sum = 0.0 Is it clear from context that the "0.1 is not exactly 1/10" refers to floating point only? > @@ -5637,7 +5645,7 @@ > you can perform an exact analysis of cases like this yourself. Basic > familiarity with binary floating-point representation is assumed. > > -\dfn{Representation error} refers to that some (most, actually) > +\dfn{Representation error} refers to fact that some (most, actually) > decimal fractions cannot be represented exactly as binary (base 2) > fractions. This is the chief reason why Python (or Perl, C, \Cpp, > Java, Fortran, and many others) often won't display the exact decimal "...refers to the fact..."? Reinhold -- Mail address is perfectly valid! From pje at telecommunity.com Tue Aug 23 19:57:27 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 23 Aug 2005 13:57:27 -0400 Subject: [Python-Dev] python/dist/src/Doc/tut tut.tex,1.276,1.277 In-Reply-To: References: <20050823150057.057C91E400B@bag.python.org> <20050823150057.057C91E400B@bag.python.org> Message-ID: <5.1.1.6.0.20050823135140.032c5ea0@mail.telecommunity.com> At 07:23 PM 8/23/2005 +0200, Reinhold Birkenfeld wrote: >rhettinger at users.sourceforge.net wrote: > >I'm not a native speaker, but... > > > @@ -114,7 +114,7 @@ > > programs, or to test functions during bottom-up program development. > > It is also a handy desk calculator. > > > > -Python allows writing very compact and readable programs. Programs > > +Python enables programs to written compactly and readably. Programs > > written in Python are typically much shorter than equivalent C or > > \Cpp{} programs, for several reasons: > > \begin{itemize} > >...shouldn't it be "programs to be written compactly"? It looks to me like the original text here should stand; Python doesn't "enable programs to be written"; it enables people to write them. That is, the passive voice should be avoided if possible. ;-) > > @@ -1985,7 +1987,9 @@ > > \section{The \keyword{del} statement \label{del}} > > > > There is a way to remove an item from a list given its index instead > > -of its value: the \keyword{del} statement. This can also be used to > > +of its value: the \keyword{del} statement. Unlike the \method{pop()}) > > +method which returns a value, the \keyword{del} keyword is a statement > > +and can also be used to > > remove slices from a list (which we did earlier by assignment of an > > empty list to the slice). For example: > >The del keyword is a statement? The keyword certainly isn't. This section also looks like it should stand the way it was, or else say that "unlike the pop() method, the del statement can also be used to remove slices...". From greg at electricrain.com Tue Aug 23 20:59:29 2005 From: greg at electricrain.com (Gregory P. Smith) Date: Tue, 23 Aug 2005 11:59:29 -0700 Subject: [Python-Dev] [Python-checkins] python/dist/src setup.py, 1.219, 1.220 In-Reply-To: <003101c5a717$83be4b60$3c23a044@oemcomputer> References: <20050821184639.EF8711E4006@bag.python.org> <003101c5a717$83be4b60$3c23a044@oemcomputer> Message-ID: <20050823185929.GI16043@zot.electricrain.com> On Mon, Aug 22, 2005 at 08:46:27AM -0400, Raymond Hettinger wrote: > > A new hashlib module to replace the md5 and sha modules. It adds > > support for additional secure hashes such as SHA-256 and SHA-512. The > > hashlib module uses OpenSSL for fast platform optimized > > implementations of algorithms when available. The old md5 and sha > > modules still exist as wrappers around hashlib to preserve backwards > > compatibility. > > I'm getting compilation errors: > > C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad > suffix on number' > C:\py25\Modules\sha512module.c(146) : error C2146: syntax error : > missing ')' before identifier 'L' > C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad > suffix on number' > C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad > suffix on number' > C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad > suffix on number' > C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad > suffix on number' > C:\py25\Modules\sha512module.c(146) : error C2059: syntax error : 'bad > suffix on number' > C:\py25\Modules\sha512module.c(146) : fatal error C1013: compiler limit > : too many open parentheses > > > Also, there should be updating entries to Misc/NEWS, > PC/VC6/pythoncore.dsp, and PC/config.c. > > > Raymond I don't have a win32 dev environment at the moment so i didn't see that. Sorry. If you remove the 'ULL' suffix from all of the 64bit constants in that file what happens? I added the ULLs to quelch the mass of warnings about constants being to large for the datatype that gcc 3.3 was spewing. -greg From greg at electricrain.com Tue Aug 23 21:04:30 2005 From: greg at electricrain.com (Gregory P. Smith) Date: Tue, 23 Aug 2005 12:04:30 -0700 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: <000601c5a7ec$9f614680$8901a044@oemcomputer> References: <20050821184613.A45C11E4288@bag.python.org> <000601c5a7ec$9f614680$8901a044@oemcomputer> Message-ID: <20050823190430.GJ16043@zot.electricrain.com> > This patch should be reverted or fixed so that the Py2.5 build works > again. > > It contains a disasterous search and replace error that prevents it from > compiling. Hence, it couldn't have passed the test suite before being > checked in. > > Also, all of the project and config files need to be updated for the new > modules. It passes fine on linux. I don't have a windows dev environment. regardless, the quick way to work around the sha512 on windows issue is to comment it out in setup.py and comment out the sha384 and sha512 tests in test_hashlib.py and commit that until the complation issues are worked out. -g > > -----Original Message----- > > From: python-checkins-bounces at python.org [mailto:python-checkins- > > bounces at python.org] On Behalf Of greg at users.sourceforge.net > > Sent: Sunday, August 21, 2005 2:46 PM > > To: python-checkins at python.org > > Subject: [Python-checkins] python/dist/src/Modules _hashopenssl.c, > > NONE,2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE,2.1 > md5module.c, > > 2.35, 2.36 shamodule.c, 2.22, 2.23 > > > > Update of /cvsroot/python/python/dist/src/Modules > > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv32064/Modules > > > > Modified Files: > > md5module.c shamodule.c > > Added Files: > > _hashopenssl.c sha256module.c sha512module.c > > Log Message: > > [ sf.net patch # 1121611 ] > > > > A new hashlib module to replace the md5 and sha modules. It adds > > support for additional secure hashes such as SHA-256 and SHA-512. The > > hashlib module uses OpenSSL for fast platform optimized > > implementations of algorithms when available. The old md5 and sha > > modules still exist as wrappers around hashlib to preserve backwards > > compatibility. From raymond.hettinger at verizon.net Tue Aug 23 21:09:50 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 23 Aug 2005 15:09:50 -0400 Subject: [Python-Dev] [Python-checkins] python/dist/src setup.py, 1.219, 1.220 In-Reply-To: <20050823185929.GI16043@zot.electricrain.com> Message-ID: <000201c5a816$3caacaa0$8901a044@oemcomputer> [Gregory P. Smith] > I don't have a win32 dev environment at the moment so i didn't see > that. Sorry. No big deal. But we still have to get the code back to ANSI compliance. Do you have an ANSI-strict option with your compiler? Raymond From barry at python.org Tue Aug 23 21:27:01 2005 From: barry at python.org (Barry Warsaw) Date: Tue, 23 Aug 2005 15:27:01 -0400 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules_hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: References: <000101c5a7f5$989d5100$8901a044@oemcomputer> <200508231632.30175.gmccaughan@synaptics-uk.com> Message-ID: <1124825221.16679.4.camel@geddy.wooz.org> On Tue, 2005-08-23 at 11:51, Fredrik Lundh wrote: > Gareth McCaughan wrote: > > > It's valid C99, meaning "this is an unsigned long long". > > since when does Python require C99 compilers? Why, since Python 3.0 of course! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050823/139f430b/attachment.pgp From keir at cs.toronto.edu Tue Aug 23 22:10:21 2005 From: keir at cs.toronto.edu (Keir Mierle) Date: Tue, 23 Aug 2005 16:10:21 -0400 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) Message-ID: <20050823201021.GE32195@cs.toronto.edu> Hi, I'm working on Argon (http://www.third-bit.com/trac/argon) with Greg Wilson this summer We're having a very strange problem with Python's unicode parsing of source files. Basically, our CGI script was running extremely slowly on our production box (a pokey dual-Xeon 3GHz w/ 4GB RAM and 15K SCSI drives). Slow to the tune of 6-10 seconds per request. I eventually tracked this down to imports of our source tree; the actual request was completing in 300ms, the rest of the time was spent in __import__. After doing some gprof profiling, I discovered _PyUnicodeUCS2_IsLinebreak was getting called 51 million times. Our code is 1.2 million characters, so I hardly think it makes sense to call IsLinebreak 50 times for each character; and we're not even importing our entire source tree on every invocation. Our code is a fork of Trac, and originally had these lines at the top: # -*- coding: iso8859-1 -*- This made me suspicious, so I removed all of them. The CGI execution time immediately dropped to ~1 second. gprof revealed that _PyUnicodeUCS2_IsLinebreak is not called at all anymore. Now that our code works fast enough, I don't really care about this, but I thought python-dev might want to know something weird is going on with unicode splitlines. I documented my investigation of this problem; if anyone wants further details, just email me. (I'm not on python-dev) http://www.third-bit.com/trac/argon/ticket/525 Thanks in advance, Keir From martin at v.loewis.de Tue Aug 23 23:13:42 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 23 Aug 2005 23:13:42 +0200 Subject: [Python-Dev] [Python-checkins] python/dist/src setup.py, 1.219, 1.220 In-Reply-To: <000201c5a816$3caacaa0$8901a044@oemcomputer> References: <000201c5a816$3caacaa0$8901a044@oemcomputer> Message-ID: <430B9186.3010106@v.loewis.de> Raymond Hettinger wrote: > But we still have to get the code back to ANSI compliance. > Do you have an ANSI-strict option with your compiler? Please don't call this "ANSI compliant". ANSI does many more thinks that writing C standards, and, in the specific case, the code *is* ANSI compliant as it stands - it just doesn't comply to C89. It complies to ISO C 99, which (I believe) is also an U.S. American national (ANSI) standard. gcc does have an option to force c89 compliance, but there is a good chance that Python stops compiling with option: on many systems, essential system headers fail to comply with C89 (in addition, activating that mode also makes many extensions unavailable). Regards, Martin From tjreedy at udel.edu Tue Aug 23 23:23:50 2005 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 23 Aug 2005 17:23:50 -0400 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 References: <2mmzn8slmv.fsf@starship.python.net> <000101c5a7f9$efb77480$8901a044@oemcomputer> Message-ID: "Raymond Hettinger" wrote in message news:000101c5a7f9$efb77480$8901a044 at oemcomputer... > Except from PEP 7: > > "Use ANSI/ISO standard C (the 1989 version of the standard)." Just checked (P&B, Standard C): only one L allowed, not two. But with C99 compilers becoming more common, accidental usages of C99-isms in submitted code will likely become more common, especially when there is not a graceful C89 alternative. While the current policy should be followed while it remains the policy, it may need revision someday. Terry J. Reedy From greg at electricrain.com Tue Aug 23 23:32:22 2005 From: greg at electricrain.com (Gregory P. Smith) Date: Tue, 23 Aug 2005 14:32:22 -0700 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: <000101c5a7f5$989d5100$8901a044@oemcomputer> References: <2mvf1wsp0k.fsf@starship.python.net> <000101c5a7f5$989d5100$8901a044@oemcomputer> Message-ID: <20050823213222.GK16043@zot.electricrain.com> > The project files are just text files and can be updated simply and > directly. But yes, that is no big deal and I'll just do it for him once > the code gets to a compilable state. I just checked in an update removing all of the ULLs. Could you check that it compiles on windows and passes test_hashlib.py now? It does leave gcc 3.x users with a big mess of compiler warnings to deal with but those can be worked around once the build is actually working everywhere. thanks. Greg > Aside from the project files, there is still config.c and whatnot. We > should put together a checklist of all the things that need to be > updated when a new module is added. that'd be helpful. :) From martin at v.loewis.de Tue Aug 23 23:43:34 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 23 Aug 2005 23:43:34 +0200 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: References: <2mmzn8slmv.fsf@starship.python.net> <000101c5a7f9$efb77480$8901a044@oemcomputer> Message-ID: <430B9886.4060004@v.loewis.de> Terry Reedy wrote: > Just checked (P&B, Standard C): only one L allowed, not two. But with C99 > compilers becoming more common, accidental usages of C99-isms in submitted > code will likely become more common, especially when there is not a > graceful C89 alternative. While the current policy should be followed > while it remains the policy, it may need revision someday. I think Python switched to C89 in 1999 (shortly before C99 was published, IIRC). So the canonical time for switching to C99 would be in 2009, provided all interesting compilers have implemented it by then, atleast to the degree that Python would typically need. Regards, Martin From raymond.hettinger at verizon.net Wed Aug 24 02:29:37 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 23 Aug 2005 20:29:37 -0400 Subject: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23 In-Reply-To: <20050823213222.GK16043@zot.electricrain.com> Message-ID: <001001c5a842$e9ac0580$ab12c797@oemcomputer> [Gregory P. Smith] > I just checked in an update removing all of the ULLs. Could you check > that it compiles on windows and passes test_hashlib.py now? Okay, all is well. Raymond From raymond.hettinger at verizon.net Wed Aug 24 03:23:32 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 23 Aug 2005 21:23:32 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 Message-ID: <001601c5a84a$715b9200$ab12c797@oemcomputer> The latest version of PEP 348 still proposes that a bare except clause will default to Exception instead of BaseException. Initially, I had thought that might be a good idea but now think it is doomed and needs to be removed from the PEP. A bare except belongs at the end of a try suite, not in the middle. This is obvious when compared to: if a: ... elif b: ... elif c: ... else: ... # The bare else goes at the end # and serves as a catchall or switch c case a: ... case b: ... default: ... # The bare default goes at the end # and serves as a catchall In contrast, Brett's 8/9 note revealed that the following would be allowable and common if the PEP is accepted in its current form: try: ... except: ... # A bare except in the middle. WTF? except (KeyboardInterrupt, SystemExit): ... The right way is, of course: try: ... except (KeyboardInterrupt, SystemExit): ... except: # Implicit or explicit match to BaseException # that serves as a catchall For those not needing a terminating exception handler, the rest of the PEP appropriately allows and encourages a simple and explicit solution that meets most needs: try: ... except Exception: ... The core issue is that the most obvious meaning of a bare except is "catchall", not "catchmost". When the latter is intended, the simple and explicit form shown in the last example is the way to go. If the former is intended, then either a bare except clause or explicit mention of BaseException will do nicely. However, under the PEP proposal, both new and existing code will suffer from having bare except clauses that look like they catch everything, are intended to catch everything, but, in fact, do not. That kind of optical illusion error must be avoided. There is no getting around our mind's propensity to interpret the bare form as defaulting to the top of the tree rather than the middle as proposed by the PEP. Likewise, there is no getting around the mental confusion caused a bare except clause in the middle of a try-suite rather than at the end. We have to avoid code that looks like it does one thing but actually does something else. Raymond From gvanrossum at gmail.com Wed Aug 24 03:30:20 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 23 Aug 2005 18:30:20 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <001601c5a84a$715b9200$ab12c797@oemcomputer> References: <001601c5a84a$715b9200$ab12c797@oemcomputer> Message-ID: On 8/23/05, Raymond Hettinger wrote: > The latest version of PEP 348 still proposes that a bare except clause > will default to Exception instead of BaseException. Initially, I had > thought that might be a good idea but now think it is doomed and needs > to be removed from the PEP. If we syntactically enforce that the bare except, if present, must be last, would that remove your objection? I agree that a bare except in the middle is an anomaly, but that doesn't mean we can't keep bare except: as a shorthand for except Exception:. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From raymond.hettinger at verizon.net Wed Aug 24 04:41:01 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 23 Aug 2005 22:41:01 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: Message-ID: <001d01c5a855$447a9d20$ab12c797@oemcomputer> [Guido van Rossum] > If we syntactically enforce that the bare except, if present, must be > last, would that remove your objection? I agree that a bare except in > the middle is an anomaly, but that doesn't mean we can't keep bare > except: as a shorthand for except Exception:. Hmm. Prohibiting mid-suite bare excepts is progress and eliminates the case that causes immediate indigestion. As for the rest, I'm not as sure and it would be helpful to get thoughts from others on this one. My sense is that blocking the clause from appearing in the middle is treating the symptom and not the disease. The motivating case for the most of the PEP was that folks were writing bare except clauses and trapping more than they should. Much of the concern was dealt with just by giving a choice between writing Exception and BareException depending on the intended result. That leaves the question of the default value a bare except with Exception being the most useful and BaseException being the most obvious. While I don't doubt that Exception is the more useful, we have already introduced a new builtin and moved two other exceptions to meet this need. Going further and altering the meaning of bare except seems like overkill for a relatively small issue. My remaining concern is about obviousness. How much code has been written or will be written that intends a bare except to mean BaseException instead of Exception. Would such code erroneously pass a code review or inspection. I suspect it would. The code looks like does one thing but actually does something else. This may or may not be a big deal. Raymond From niko at alum.mit.edu Wed Aug 24 09:07:58 2005 From: niko at alum.mit.edu (Niko Matsakis) Date: Wed, 24 Aug 2005 09:07:58 +0200 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <001d01c5a855$447a9d20$ab12c797@oemcomputer> References: <001d01c5a855$447a9d20$ab12c797@oemcomputer> Message-ID: <00236367-938E-4D75-866E-2F1A5DEABEC0@alum.mit.edu> > As for the rest, I'm not as sure and it would be helpful to get > thoughts > from others on this one. My sense is that blocking the clause from > appearing in the middle is treating the symptom and not the disease. +1 It would be better to prohibit bare except entirely (well, presumably at some point in the future with appropriate warnings at the moment) than change its semantics. I agree that its intuitive meaning is "if anything is thrown", not, "if a non-programmer-error exception is thrown," but I'm not sure if that's even important. The point is that it has existing well defined semantics; changing them just seems unnecessary to the aims of the rewrite and confusing to existing Python programmers. I've written plenty of code with bare excepts and they all intended to catch *any* exception, usually in a user interface where I wanted to return to the main loop on programmer error not abort the entire program. I don't relish the thought of going back and changing existing code, and I imagine there are few who do. My 2 cents, Niko From ncoghlan at gmail.com Wed Aug 24 11:26:02 2005 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 24 Aug 2005 19:26:02 +1000 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <001601c5a84a$715b9200$ab12c797@oemcomputer> References: <001601c5a84a$715b9200$ab12c797@oemcomputer> Message-ID: <430C3D2A.3070103@gmail.com> Raymond Hettinger wrote: > The latest version of PEP 348 still proposes that a bare except clause > will default to Exception instead of BaseException. Initially, I had > thought that might be a good idea but now think it is doomed and needs > to be removed from the PEP. One thing I assumed was that _if_ bare excepts were kept, they would still only be allowed as the last except clause. That is, this example: > try: ... > except: ... # A bare except in the middle. WTF? > except (KeyboardInterrupt, SystemExit): ... would still be a syntax error, even if bare excepts were allowed. I still have some qualms about the idea of a bare except that doesn't catch everything (I'd prefer to see them gone altogether), but I don't mind quite as much if the above code stays as a syntax error. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com From walter at livinglogic.de Wed Aug 24 11:45:33 2005 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 24 Aug 2005 11:45:33 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <20050823201021.GE32195@cs.toronto.edu> References: <20050823201021.GE32195@cs.toronto.edu> Message-ID: <430C41BD.8010602@livinglogic.de> Keir Mierle wrote: > Hi, I'm working on Argon (http://www.third-bit.com/trac/argon) with Greg > Wilson this summer > > We're having a very strange problem with Python's unicode parsing of source > files. Basically, our CGI script was running extremely slowly on our production > box (a pokey dual-Xeon 3GHz w/ 4GB RAM and 15K SCSI drives). Slow to the tune > of 6-10 seconds per request. I eventually tracked this down to imports of our > source tree; the actual request was completing in 300ms, the rest of the time > was spent in __import__. This is caused by the chances to the codecs in 2.4. Basically the codecs no longer rely on C's readline() to do line splitting (which can't work for UTF-16), but do it themselves (via unicode.splitlines()). > After doing some gprof profiling, I discovered _PyUnicodeUCS2_IsLinebreak was > getting called 51 million times. Our code is 1.2 million characters, so I > hardly think it makes sense to call IsLinebreak 50 times for each character; > and we're not even importing our entire source tree on every invocation. But if you're using CGI, you're importing your source on every invocation. Switching to a different server side technology might help. Nevertheless 50 million calls seems to be a bit much. > Our code is a fork of Trac, and originally had these lines at the top: > > # -*- coding: iso8859-1 -*- > > This made me suspicious, so I removed all of them. The CGI execution time > immediately dropped to ~1 second. gprof revealed that > _PyUnicodeUCS2_IsLinebreak is not called at all anymore. > > Now that our code works fast enough, I don't really care about this, but I > thought python-dev might want to know something weird is going on with unicode > splitlines. I wonder if we should switch back to a simple readline() implementation for those codecs that don't require the current implementation (basically every charmap codec). AFAIK source files are opened in universal newline mode, so at least we'd get proper treatment of "\n", "\r" and "\r\n" line ends, but we'd loose u"\x1c", u"\x1d", u"\x1e", u"\x85", u"\u2028" and u"\u2029" (which are line terminators according to unicode.splitlines()). > I documented my investigation of this problem; if anyone wants further details, > just email me. (I'm not on python-dev) > http://www.third-bit.com/trac/argon/ticket/525 Bye, Walter D?rwald From martin at v.loewis.de Wed Aug 24 12:16:25 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Aug 2005 12:16:25 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C41BD.8010602@livinglogic.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> Message-ID: <430C48F9.8060801@v.loewis.de> Walter D?rwald wrote: > This is caused by the chances to the codecs in 2.4. Basically the codecs > no longer rely on C's readline() to do line splitting (which can't work > for UTF-16), but do it themselves (via unicode.splitlines()). That explains why you get any calls to IsLineBreak; it doesn't explain why you get so many of them. I investigated this a bit, and one issue seems to be that StreamReader.readline performs splitline on the entire input, only to fetch the first line. It then joins the rest for later processing. In addition, it also performs splitlines on a single line, just to strip any trailing line breaks. The net effect is that, for a file with N lines, IsLineBreak is invoked up to N*N/2 times per character (atleast for the last character). So I think it would be best if Unicode characters exposed a .islinebreak method (or, failing that, codecs just knew what the line break characters are in Unicode 3.2), and then codecs would split off the first line of input itself. >>After doing some gprof profiling, I discovered _PyUnicodeUCS2_IsLinebreak was >>getting called 51 million times. Our code is 1.2 million characters, so I >>hardly think it makes sense to call IsLinebreak 50 times for each character; >>and we're not even importing our entire source tree on every invocation. > > > But if you're using CGI, you're importing your source on every > invocation. Well, no. Only the CGI script needs to be parsed every time; all modules could load off bytecode files. Which suggests that Keir Mierle doesn't use bytecode files, I think he should. Regards, Martin From mal at egenix.com Wed Aug 24 12:27:45 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 24 Aug 2005 12:27:45 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C41BD.8010602@livinglogic.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> Message-ID: <430C4BA1.5030503@egenix.com> Walter D?rwald wrote: > I wonder if we should switch back to a simple readline() implementation > for those codecs that don't require the current implementation > (basically every charmap codec). That would be my preference as well. The 2.4 .readline() approach is really only needed for codecs that have to deal with encodings that: a) use multi-byte formats, or b) support more line-end formats than just CR, CRLF, LF, or c) are stateful. This can easily be had by using a mix-in class for codecs which do need the buffered .readline() approach. > AFAIK source files are opened in > universal newline mode, so at least we'd get proper treatment of "\n", > "\r" and "\r\n" line ends, but we'd loose u"\x1c", u"\x1d", u"\x1e", > u"\x85", u"\u2028" and u"\u2029" (which are line terminators according > to unicode.splitlines()). While the Unicode standard defines these characters as line end code points, I think their definition does not necessarily apply to data that is converted from a certain encoding to Unicode, so that's not a big loss. E.g. in ASCII or Latin-1, FILE, GROUP and RECORD SEPARATOR and NEXT LINE characters (0x1c, 0x1d, 0x1e, 0x85) are not interpreted as line end characters. Furthermore, we had no reports of anyone complaining in Python 1.6, 2.0 - 2.3 that line endings were not detected properly. All these Python versions relied on the stream's .readline() method to get the next line. The only bug reports we had were for UTF-16 which falls into the above category a) and did not support .readline() until Python 2.4. A note on the performance of _PyUnicode_IsLinebreak(): in Python 2.0 Fredrik changed this to use the two step lookup (reducing the size of the lookup tables considerably). I think it's worthwhile reconsidering this approach for character type queries that do no involve a huge number of code points. In Python 1.6 the function looked like this (and was inlined by the compiler using its own fast lookup table): int _PyUnicode_IsLinebreak(register const Py_UNICODE ch) { switch (ch) { case 0x000A: /* LINE FEED */ case 0x000D: /* CARRIAGE RETURN */ case 0x001C: /* FILE SEPARATOR */ case 0x001D: /* GROUP SEPARATOR */ case 0x001E: /* RECORD SEPARATOR */ case 0x0085: /* NEXT LINE */ case 0x2028: /* LINE SEPARATOR */ case 0x2029: /* PARAGRAPH SEPARATOR */ return 1; default: return 0; } } another candidate to convert back is: int _PyUnicode_IsWhitespace(register const Py_UNICODE ch) { switch (ch) { case 0x0009: /* HORIZONTAL TABULATION */ case 0x000A: /* LINE FEED */ case 0x000B: /* VERTICAL TABULATION */ case 0x000C: /* FORM FEED */ case 0x000D: /* CARRIAGE RETURN */ case 0x001C: /* FILE SEPARATOR */ case 0x001D: /* GROUP SEPARATOR */ case 0x001E: /* RECORD SEPARATOR */ case 0x001F: /* UNIT SEPARATOR */ case 0x0020: /* SPACE */ case 0x0085: /* NEXT LINE */ case 0x00A0: /* NO-BREAK SPACE */ case 0x1680: /* OGHAM SPACE MARK */ case 0x2000: /* EN QUAD */ case 0x2001: /* EM QUAD */ case 0x2002: /* EN SPACE */ case 0x2003: /* EM SPACE */ case 0x2004: /* THREE-PER-EM SPACE */ case 0x2005: /* FOUR-PER-EM SPACE */ case 0x2006: /* SIX-PER-EM SPACE */ case 0x2007: /* FIGURE SPACE */ case 0x2008: /* PUNCTUATION SPACE */ case 0x2009: /* THIN SPACE */ case 0x200A: /* HAIR SPACE */ case 0x200B: /* ZERO WIDTH SPACE */ case 0x2028: /* LINE SEPARATOR */ case 0x2029: /* PARAGRAPH SEPARATOR */ case 0x202F: /* NARROW NO-BREAK SPACE */ case 0x3000: /* IDEOGRAPHIC SPACE */ return 1; default: return 0; } } -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 23 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From martin at v.loewis.de Wed Aug 24 12:56:58 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Aug 2005 12:56:58 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C4BA1.5030503@egenix.com> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C4BA1.5030503@egenix.com> Message-ID: <430C527A.8090302@v.loewis.de> M.-A. Lemburg wrote: > I think it's worthwhile reconsidering this approach for > character type queries that do no involve a huge number > of code points. I would advise against that. I measure both versions (your version called PyUnicode_IsLinebreak2) with the following code volatile int result; void unibench() { #define REPS 10000000000LL long long i; clock_t s1,s2,s3,s4,s5; s1 = clock(); for(i=0;i References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> Message-ID: <430C5E6E.2040405@livinglogic.de> Martin v. L?wis wrote: > Walter D?rwald wrote: > >>This is caused by the chances to the codecs in 2.4. Basically the codecs >>no longer rely on C's readline() to do line splitting (which can't work >>for UTF-16), but do it themselves (via unicode.splitlines()). > > That explains why you get any calls to IsLineBreak; it doesn't explain > why you get so many of them. > > I investigated this a bit, and one issue seems to be that > StreamReader.readline performs splitline on the entire input, only to > fetch the first line. It then joins the rest for later processing. > In addition, it also performs splitlines on a single line, just to > strip any trailing line breaks. This is because unicode.splitlines() is the only API available to Python that knows about unicode line feeds. > The net effect is that, for a file with N lines, IsLineBreak is invoked > up to N*N/2 times per character (atleast for the last character). > > So I think it would be best if Unicode characters exposed a .islinebreak > method (or, failing that, codecs just knew what the line break > characters are in Unicode 3.2), and then codecs would split off > the first line of input itself. I think a maxsplit argument (just as for unicode.split()) would help too. > [...] Bye, Walter D?rwald From mal at egenix.com Wed Aug 24 14:24:42 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 24 Aug 2005 14:24:42 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C527A.8090302@v.loewis.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C4BA1.5030503@egenix.com> <430C527A.8090302@v.loewis.de> Message-ID: <430C670A.3090408@egenix.com> Martin v. L?wis wrote: > M.-A. Lemburg wrote: > >>I think it's worthwhile reconsidering this approach for >>character type queries that do no involve a huge number >>of code points. > > > I would advise against that. I measure both versions > (your version called PyUnicode_IsLinebreak2) with the > following code > > volatile int result; > void unibench() > { > #define REPS 10000000000LL > long long i; > clock_t s1,s2,s3,s4,s5; > s1 = clock(); > for(i=0;i result = _PyUnicode_IsLinebreak('('); > s2 = clock(); > for(i=0;i result = PyUnicode_IsLinebreak2('('); > s3 = clock(); > for(i=0;i result = _PyUnicode_IsLinebreak('\n'); > s4 = clock(); > for(i=0;i result = PyUnicode_IsLinebreak2('\n'); > s5 = clock(); > printf("f1, (: %d\nf2, (: %d\nf1, CR: %d\n, f2, CR: %d\n", > (int)(s2-s1),(int)(s3-s2),(int)(s4-s3),(int)(s5-s4)); > } > > and got those numbers > > f1, (: 13210000 > f2, (: 13300000 > f1, CR: 13220000 > , f2, CR: 13250000 > > What can be seen is that performance the two versions is nearly > identical, with the code currently used being slightly better. > What can also be seen is that, on my machine, 1e10 calls to > IsLinebreak take 13.2 seconds. So 51 Mio calls take about 70ms. Your test is somewhat biased: the current solution works using type records, so it has to swap in a new record for each character you test. In you benchmark, the same character is tested over and over again and the type record likely already stored in the CPU cache. The .splitlines() routine itself calls the above function for each and every character in the string, so quite a few of these type records have to be looked up. Here's a version that uses os.py as basis: #include #include #include "Python.h" int _PyUnicode_IsLinebreak16(register const Py_UNICODE ch) { switch (ch) { case 0x000A: /* LINE FEED */ case 0x000D: /* CARRIAGE RETURN */ case 0x001C: /* FILE SEPARATOR */ case 0x001D: /* GROUP SEPARATOR */ case 0x001E: /* RECORD SEPARATOR */ case 0x0085: /* NEXT LINE */ case 0x2028: /* LINE SEPARATOR */ case 0x2029: /* PARAGRAPH SEPARATOR */ return 1; default: return 0; } } #define REPS 10000 #define BUFFERSIZE 30000 int main(void) { long i, j; clock_t s1,s2,s3; char *buffer; FILE *datafile; long filelen; int result; datafile = fopen("os.py", "rb"); if (datafile == NULL) { printf("could not find os.py\n"); return -1; } buffer = (char *)malloc(BUFFERSIZE); filelen = fread(buffer, 1, BUFFERSIZE, datafile); printf("filelen=%li bytes\n", filelen); s1 = clock(); /* Python 2.4 */ for(i = 0; i < REPS; i++) for (j = 0; j < filelen; j++) result = _PyUnicode_IsLinebreak((Py_UNICODE)buffer[j]); s2 = clock(); /* Python 1.6 */ for(i = 0; i < REPS; i++) for (j = 0; j < filelen; j++) result = _PyUnicode_IsLinebreak16((Py_UNICODE)buffer[j]); s3 = clock(); printf("2.4: %d\n" "1.6: %d\n", (int)(s2-s1), (int)(s3-s2)); return 0; } Output, compiled with -O3: filelen=23147 bytes 2.4: 2570000 1.6: 1230000 That's a factor 2. > The reported performance problem is more likely in the allocation > of all these splitlines results, and the copying of the same > strings over and over again. True. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 23 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From martin at v.loewis.de Wed Aug 24 14:56:30 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Aug 2005 14:56:30 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C5E6E.2040405@livinglogic.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> Message-ID: <430C6E7E.7070106@v.loewis.de> Walter D?rwald wrote: > I think a maxsplit argument (just as for unicode.split()) would help too. Correct - that would allow to get rid of the quadratic part. We should also strive for avoiding the second copy of the line, if the user requested keepends. I wonder whether it would be worthwhile to cache the .splitlines result. An application that has just invoked .readline() will likely invoke .readline() again. If there is more than one line left, we could return the first line right away (potentially trimming the line ending if necessary). Only when a single line is left, we would attempt to read more data. In a plain .read(), we would first join the lines back. Regards, Martin From mcherm at mcherm.com Wed Aug 24 15:08:56 2005 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 24 Aug 2005 06:08:56 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 Message-ID: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> Raymond Hettinger writes: > The latest version of PEP 348 still proposes that a bare except clause > will default to Exception instead of BaseException. Initially, I had > thought that might be a good idea but now think it is doomed and needs > to be removed from the PEP. Guido writes: > If we syntactically enforce that the bare except, if present, must be > last, would that remove your objection? I agree that a bare except in > the middle is an anomaly, but that doesn't mean we can't keep bare > except: as a shorthand for except Exception:. Explicit is better than Implicit. I think that in newly written code "except Exception:" is better (more explicit and easier to understand) than "except:" Legacy code that uses "except:" can remain unchanged *IF* the meaning of "except:" is unchanged... but I think we all agree that this is unwise because the existing meaning is a tempting trap for the unwary. So I don't see any advantage to keeping bare "except:" in the long run. What we do to ease the transition is a different question, but one more easily resolved. -- Michael Chermside From walter at livinglogic.de Wed Aug 24 16:08:12 2005 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 24 Aug 2005 16:08:12 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C6E7E.7070106@v.loewis.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> Message-ID: <430C7F4C.9010703@livinglogic.de> Martin v. L?wis wrote: > Walter D?rwald wrote: > >>I think a maxsplit argument (just as for unicode.split()) would help too. > > Correct - that would allow to get rid of the quadratic part. OK, such a patch should be rather simple. I'll give it a try. > We should also strive for avoiding the second copy of the line, > if the user requested keepends. Your suggested unicode method islinebreak() would help with that. Then we could add the following to the string module: unicodelinebreaks = u"".join(unichr(c) for c in xrange(0, sys.maxunicode) if unichr(c).islinebreak()) Then if line and not keepends: line = line.splitlines(False)[0] could be if line and not keepends: line = line.rstrip(string.unicodelinebreaks) > I wonder whether it would be worthwhile to cache the .splitlines result. > An application that has just invoked .readline() will likely invoke > .readline() again. If there is more than one line left, we could return > the first line right away (potentially trimming the line ending if > necessary). Only when a single line is left, we would attempt to > read more data. In a plain .read(), we would first join the lines > back. OK, this would mean we'd have to distinguish between a direct call to read() and one done by readline() (which we do anyway through the firstline argument). Bye, Walter D?rwald From martin at v.loewis.de Wed Aug 24 16:33:50 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Aug 2005 16:33:50 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C7F4C.9010703@livinglogic.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> <430C7F4C.9010703@livinglogic.de> Message-ID: <430C854E.1080200@v.loewis.de> Walter D?rwald wrote: > Martin v. L?wis wrote: > >> Walter D?rwald wrote: >> >>> I think a maxsplit argument (just as for unicode.split()) would help >>> too. >> >> >> Correct - that would allow to get rid of the quadratic part. > > > OK, such a patch should be rather simple. I'll give it a try. Actually, on a second thought - it would not remove the quadratic aspect. You would still copy the rest string completely on each split. So on the first split, you copy N lines (one result line, and N-1 lines into the rest string), on the second split, N-2 lines, and so on, totalling N*N/2 line copies again. The only thing you save is the join (as the rest is already joined), and the IsLineBreak calls (which are necessary only for the first line). Please see python.org/sf/1268314; it solves the problem by keeping the splitlines result. It only invokes IsLineBreak once per character, and also copies each character only once, and allocates each line only once, totalling in O(N) for these operations. It still does contain a quadratic operation: the lines are stored in a list, and the result list is removed from the list with del lines[0]. This copies N-1 pointers, result in N*N/2 pointer copies. That should still be much faster than the current code. > unicodelinebreaks = u"".join(unichr(c) for c in xrange(0, > sys.maxunicode) if unichr(c).islinebreak()) That is very inefficient. I would rather add a static list to the string module, and have a test that says assert str.unicodelinebreaks == u"".join(ch for ch in (unichr(c) for c in xrange(0, sys.maxunicode)) if unicodedata.bidirectional(ch)=='B' or unicodedata.category(ch)=='Zl') unicodelinebreaks could then be defined as # u"\r\n\x1c\x1d\x1e\x85\u2028\u2029 '\n\r\x1c\x1d\x1e\xc2\x85\xe2\x80\xa8\xe2\x80\xa9'.decode("utf-8") > OK, this would mean we'd have to distinguish between a direct call to > read() and one done by readline() (which we do anyway through the > firstline argument). See my patch. If we have cached lines, we don't need to call .read at all. Regards, Martin From foom at fuhm.net Wed Aug 24 16:34:53 2005 From: foom at fuhm.net (James Y Knight) Date: Wed, 24 Aug 2005 10:34:53 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <001d01c5a855$447a9d20$ab12c797@oemcomputer> References: <001d01c5a855$447a9d20$ab12c797@oemcomputer> Message-ID: On Aug 23, 2005, at 10:41 PM, Raymond Hettinger wrote: > [Guido van Rossum] > >> If we syntactically enforce that the bare except, if present, must be >> last, would that remove your objection? I agree that a bare except in >> the middle is an anomaly, but that doesn't mean we can't keep bare >> except: as a shorthand for except Exception:. >> > > Hmm. Prohibiting mid-suite bare excepts is progress and eliminates > the > case that causes immediate indigestion. > > As for the rest, I'm not as sure and it would be helpful to get > thoughts > from others on this one. My sense is that blocking the clause from > appearing in the middle is treating the symptom and not the disease. > I would rather see "except:" be deprecated eventually, and force the user to say either except Exception, except BaseException, or even better, except ActualExceptionIWantToCatch. James From barry at python.org Wed Aug 24 17:03:52 2005 From: barry at python.org (Barry Warsaw) Date: Wed, 24 Aug 2005 11:03:52 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <001d01c5a855$447a9d20$ab12c797@oemcomputer> Message-ID: <1124895832.19291.10.camel@geddy.wooz.org> On Wed, 2005-08-24 at 10:34, James Y Knight wrote: > I would rather see "except:" be deprecated eventually, and force the > user to say either except Exception, except BaseException, or even > better, except ActualExceptionIWantToCatch. I agree about bare except, but there is a very valid use case for an except clause that catches every possible exception. We need to make sure we don't overlook this use case. As an example, say I'm building a transaction-aware system, I'm going to want to write code like this: txn = new_transaction() try: txn.begin() rtn = do_work() except AllPossibleExceptions: txn.abort() raise else: txn.commit() return rtn I'm fine with not spelling that except statement as "except:" but I don't want there to be any exceptions that can sneak past that middle suite, including non-errors like SystemExit or KeyboardInterrupt. I can't remember ever writing a bare except with a suite that didn't contain (end in?) a bare raise. Maybe we can allow bare except, but constrain things so that the only way out of its suite is via a bare raise. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050824/46c9f344/attachment.pgp From gvanrossum at gmail.com Wed Aug 24 17:10:37 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 24 Aug 2005 08:10:37 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> Message-ID: On 8/24/05, Michael Chermside wrote: > Explicit is better than Implicit. I think that in newly written code > "except Exception:" is better (more explicit and easier to understand) > than "except:" Legacy code that uses "except:" can remain unchanged *IF* > the meaning of "except:" is unchanged... but I think we all agree that > this is unwise because the existing meaning is a tempting trap for the > unwary. So I don't see any advantage to keeping bare "except:" in the > long run. What we do to ease the transition is a different question, > but one more easily resolved. OK, I'm convinced. Let's drop bare except for Python 3.0, and deprecate them until then, without changing the meaning. The deprecation message (to be generated by the compiler!) should steer people in the direction of specifying one particular exception (e.g. KeyError etc.) rather than Exception. -- --Guido van Rossum (home page: http://www.python.org/~guido/ From foom at fuhm.net Wed Aug 24 17:23:52 2005 From: foom at fuhm.net (James Y Knight) Date: Wed, 24 Aug 2005 11:23:52 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> Message-ID: On Aug 24, 2005, at 11:10 AM, Guido van Rossum wrote: > On 8/24/05, Michael Chermside wrote: > >> Explicit is better than Implicit. I think that in newly written code >> "except Exception:" is better (more explicit and easier to >> understand) >> than "except:" Legacy code that uses "except:" can remain >> unchanged *IF* >> the meaning of "except:" is unchanged... but I think we all agree >> that >> this is unwise because the existing meaning is a tempting trap for >> the >> unwary. So I don't see any advantage to keeping bare "except:" in the >> long run. What we do to ease the transition is a different question, >> but one more easily resolved. >> > > OK, I'm convinced. Let's drop bare except for Python 3.0, and > deprecate them until then, without changing the meaning. > > The deprecation message (to be generated by the compiler!) should > steer people in the direction of specifying one particular exception > (e.g. KeyError etc.) rather than Exception. I agree but there's the minor nit of non-Exception exceptions. I think it must be the case that raising an object which does not derive from an exception class must be deprecated as well in order for "except:" to be deprecated. Otherwise, there is nothing you can change "except:" to in order not to get a deprecation warning and still have your code be correct in the face of documented features of python. James From raymond.hettinger at verizon.net Wed Aug 24 17:27:19 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Wed, 24 Aug 2005 11:27:19 -0400 Subject: [Python-Dev] FW: Bare except clauses in PEP 348 Message-ID: <003901c5a8c0$51d79fc0$b729cb97@oemcomputer> Hey guys, don't give up your bare except clauses so easily. They are useful. And, if given the natural meaning of "catch everything" and put in a natural position at the end of a suite, their meaning is plain and obvious. Remember beauty counts. I don't think there would be similar temptation to eliminate a dangling else clause and replace it with "else Everything". Nor would a final default case in a switch statement benefit from being written as "default Everything". The thought is that it is okay to have useful defaults. My whole issue was that the PEP was choosing the wrong default. If we leave it alone, all is well. An empty except will continue to mean "catch everything", it will always appear at the end, its meaning will be obvious, and existing working code won't break :-) On the occasions where you really intended to catch everything, do you really want to go on an editing binge just to uglify the code to something like: try: ... except SomeException: ... except BaseException: ... It is more beautiful and clear as: try: ... except SomeException: ... except: ... To me, the latter is more attractive and is more obviously a catchall, just like an else-clause or a default-statement. It is a strong visual cue that at least one of the except clauses will always be triggered. In contrast, the first example makes you think twice about whether the final case really does get everything (sometimes implicit IS better than explicit). Raymond From shane at hathawaymix.org Wed Aug 24 18:17:20 2005 From: shane at hathawaymix.org (Shane Hathaway) Date: Wed, 24 Aug 2005 10:17:20 -0600 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <1124895832.19291.10.camel@geddy.wooz.org> References: <001d01c5a855$447a9d20$ab12c797@oemcomputer> <1124895832.19291.10.camel@geddy.wooz.org> Message-ID: <430C9D90.6090102@hathawaymix.org> Barry Warsaw wrote: > I agree about bare except, but there is a very valid use case for an > except clause that catches every possible exception. We need to make > sure we don't overlook this use case. As an example, say I'm building a > transaction-aware system, I'm going to want to write code like this: > > txn = new_transaction() > try: > txn.begin() > rtn = do_work() > except AllPossibleExceptions: > txn.abort() > raise > else: > txn.commit() > return rtn > > I'm fine with not spelling that except statement as "except:" but I > don't want there to be any exceptions that can sneak past that middle > suite, including non-errors like SystemExit or KeyboardInterrupt. > > I can't remember ever writing a bare except with a suite that didn't > contain (end in?) a bare raise. Maybe we can allow bare except, but > constrain things so that the only way out of its suite is via a bare > raise. I also use this idiom quite frequently, but I wonder if a finally clause would be a better way to write it: txn = new_transaction() try: txn.begin() rtn = do_work() finally: if exception_occurred(): txn.abort() else: txn.commit() return rtn Since this doesn't use try/except/else, it's not affected by changes to the meaning of except clauses. However, it forces more indentation and expects a new builtin, and the name "exception_occurred" is probably too long for a builtin. Now for a weird idea. txn = new_transaction() try: txn.begin() rtn = do_work() finally except: txn.abort() finally else: txn.commit() return rtn This is what I would call qualified finally clauses. The interpreter chooses exactly one of the finally clauses. If a "finally except" clause is chosen, the exception is re-raised before execution continues. Most code that currently uses bare raise inside bare except could just prefix the "except" and "else" keywords with "finally". Shane From niko at alum.mit.edu Wed Aug 24 18:29:36 2005 From: niko at alum.mit.edu (Niko Matsakis) Date: Wed, 24 Aug 2005 18:29:36 +0200 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <430C9D90.6090102@hathawaymix.org> References: <001d01c5a855$447a9d20$ab12c797@oemcomputer> <1124895832.19291.10.camel@geddy.wooz.org> <430C9D90.6090102@hathawaymix.org> Message-ID: <934F5C4E-FF88-41D0-939E-623D0AFCDAE2@alum.mit.edu> > > txn = new_transaction() > try: > txn.begin() > rtn = do_work() > finally: > if exception_occurred(): > txn.abort() > else: > txn.commit() > return rtn > Couldn't you just do: txn = new_transaction () try: complete = 0 txn.begin () rtn = do_work () complete = 1 finally: if not complete: txn.abort () else: txn.commit () and then not need new builtins or anything fancy? Niko From mcherm at mcherm.com Wed Aug 24 18:33:00 2005 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 24 Aug 2005 09:33:00 -0700 Subject: [Python-Dev] FW: Bare except clauses in PEP 348 Message-ID: <20050824093300.uuj0o52cj9s0wksk@login.werra.lunarpages.com> Raymond writes: > Hey guys, don't give up your bare except clauses so easily. [...] Raymond: I agree that when comparing: // Code snippet A try: ... except SomeException: ... except BaseException: ... with // Code snippet B try: ... except SomeException: ... except: ... that B is nicer than A. Slightly nicer. It's a minor esthetic point. But consider these: // Code snippet C try: ... except Exception: ... // Code snippet D try: ... except: ... Esthetically I'd say that D is nicer than A for the same reasons. It's a minor esthetic point. But you see, this case is different. You and I would likely never bother to compare C and D because they do different things! (D is equivalent to catching BaseException, not Exception). But we know that people who are not so careful or not so knowlegable WILL make this mistake... they make it all the time today! Since situation C (catching an exception) is hundreds of times more common than situation A (needing default processing for exceptions that don't get caught, but doing it with try-except instead of try-finally because the nothing-was-thrown case is different), I would FAR rather protect beginners from erroniously confusing C and D than I would provide a marginally more elegent syntax for the experts using A or B. And that elegence is arguable... there's something to be said for simplicity, and having only one kind of "except" clause for try statements is clearly simpler than having both "except :" and also bare "except:". -- Michael Chermside From martin at v.loewis.de Wed Aug 24 18:38:17 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Aug 2005 18:38:17 +0200 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <934F5C4E-FF88-41D0-939E-623D0AFCDAE2@alum.mit.edu> References: <001d01c5a855$447a9d20$ab12c797@oemcomputer> <1124895832.19291.10.camel@geddy.wooz.org> <430C9D90.6090102@hathawaymix.org> <934F5C4E-FF88-41D0-939E-623D0AFCDAE2@alum.mit.edu> Message-ID: <430CA279.3080909@v.loewis.de> Niko Matsakis wrote: > Couldn't you just do: > > txn = new_transaction () > try: > complete = 0 > txn.begin () > rtn = do_work () > complete = 1 > finally: > if not complete: txn.abort () > else: txn.commit () > > and then not need new builtins or anything fancy? I personally dislike recording the execution path in local variables. This is like setting a flag in a loop before the break, and testing the flag afterwards. You can do this, but the else: clause of the loop is just more readable. This specific fragment has also the bug that a KeyboardInterrupt before the assignment to complete will cause a NameError/UnboundLocalError; this can easily be fixed by moving the assignment before the try block. Regards, Martin From shane at hathawaymix.org Wed Aug 24 18:42:21 2005 From: shane at hathawaymix.org (Shane Hathaway) Date: Wed, 24 Aug 2005 10:42:21 -0600 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <934F5C4E-FF88-41D0-939E-623D0AFCDAE2@alum.mit.edu> References: <001d01c5a855$447a9d20$ab12c797@oemcomputer> <1124895832.19291.10.camel@geddy.wooz.org> <430C9D90.6090102@hathawaymix.org> <934F5C4E-FF88-41D0-939E-623D0AFCDAE2@alum.mit.edu> Message-ID: <430CA36D.8000004@hathawaymix.org> Niko Matsakis wrote: >> >> txn = new_transaction() >> try: >> txn.begin() >> rtn = do_work() >> finally: >> if exception_occurred(): >> txn.abort() >> else: >> txn.commit() >> return rtn >> > > Couldn't you just do: > > txn = new_transaction () > try: > complete = 0 > txn.begin () > rtn = do_work () > complete = 1 > finally: > if not complete: txn.abort () > else: txn.commit () > > and then not need new builtins or anything fancy? That would work, though it's less readable. If I were looking over code like that written by someone else, I'd have verify that the "complete" variable is handled correctly in all cases. (As Martin noted, your code already has a bug.) The nice try/except/else idiom we have today, with a bare except and bare raise, is much easier to verify. Shane From walter at livinglogic.de Wed Aug 24 18:59:11 2005 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 24 Aug 2005 18:59:11 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C854E.1080200@v.loewis.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> <430C7F4C.9010703@livinglogic.de> <430C854E.1080200@v.loewis.de> Message-ID: <430CA75F.7090900@livinglogic.de> Martin v. L?wis wrote: > Walter D?rwald wrote: > >>Martin v. L?wis wrote: >> >>>Walter D?rwald wrote: >>> >>>>I think a maxsplit argument (just as for unicode.split()) would help >>>>too. >>> >>>Correct - that would allow to get rid of the quadratic part. >> >>OK, such a patch should be rather simple. I'll give it a try. > > Actually, on a second thought - it would not remove the quadratic > aspect. At least it would remove the quadratic number of calls to _PyUnicodeUCS2_IsLinebreak(). For each character it would be called only once. > You would still copy the rest string completely on each > split. So on the first split, you copy N lines (one result line, > and N-1 lines into the rest string), on the second split, N-2 > lines, and so on, totalling N*N/2 line copies again. OK, that's true. We could prevent string copying if we kept the unsplit string and the position of the current line terminator, but this would require a "first position after a line terminator" method. > The only > thing you save is the join (as the rest is already joined), and > the IsLineBreak calls (which are necessary only for the first > line). > > Please see python.org/sf/1268314; The last part of the patch seems to be more related to bug #1235646. With the patch test_pep263 and test_codecs fail (and test_parser, but this might be unrelated): python Lib/test/test_pep263.py gives the following output: File "Lib/test/test_pep263.py", line 22 SyntaxError: list index out of range test_codecs.py has the following two complaints: File "/var/home/walter/Achtung/Python-linecache/dist/src/Lib/codecs.py", line 366, in readline self.charbuffer = lines[1] + self.charbuffer IndexError: list index out of range and File "/var/home/walter/Achtung/Python-linecache/dist/src/Lib/codecs.py", line 336, in readline line = result.splitlines(False)[0] NameError: global name 'result' is not defined > it solves the problem by > keeping the splitlines result. It only invokes IsLineBreak > once per character, and also copies each character only once, > and allocates each line only once, totalling in O(N) for > these operations. It still does contain a quadratic operation: > the lines are stored in a list, and the result list is > removed from the list with del lines[0]. This copies N-1 > pointers, result in N*N/2 pointer copies. That should still > be much faster than the current code. Using collections.deque() should get rid of this problem. >>unicodelinebreaks = u"".join(unichr(c) for c in xrange(0, >>sys.maxunicode) if unichr(c).islinebreak()) > > That is very inefficient. I would rather add a static list > to the string module, and have a test that says > > assert str.unicodelinebreaks == u"".join(ch for ch in (unichr(c) for c > in xrange(0, sys.maxunicode)) if unicodedata.bidirectional(ch)=='B' or > unicodedata.category(ch)=='Zl') You mean, in the test suite? > unicodelinebreaks could then be defined as > > # u"\r\n\x1c\x1d\x1e\x85\u2028\u2029 > '\n\r\x1c\x1d\x1e\xc2\x85\xe2\x80\xa8\xe2\x80\xa9'.decode("utf-8") That might be better, as this definition won't change very often. BTW, why the decode() call? For a Python without unicode? >>OK, this would mean we'd have to distinguish between a direct call to >>read() and one done by readline() (which we do anyway through the >>firstline argument). > > See my patch. If we have cached lines, we don't need to call .read > at all. I wonder what happens, if calls to read() and readline() are mixed (e.g. if I'm reading Fortran source or anything with a fixed line header): read() would be used to read the first n character (which joins the line buffer) and readline() reads the rest (which would split it again) etc. (Of course this could be done via a single readline() call). But, I think a maxsplit argument for splitlines() woould make sense independent of this problem. Bye, Walter D?rwald From rrr at ronadam.com Wed Aug 24 19:03:13 2005 From: rrr at ronadam.com (Ron Adam) Date: Wed, 24 Aug 2005 13:03:13 -0400 Subject: [Python-Dev] FW: Bare except clauses in PEP 348 In-Reply-To: <003901c5a8c0$51d79fc0$b729cb97@oemcomputer> References: <003901c5a8c0$51d79fc0$b729cb97@oemcomputer> Message-ID: <430CA851.6060406@ronadam.com> Raymond Hettinger wrote: > Hey guys, don't give up your bare except clauses so easily. Yes, Don't give up. I often write code starting with a bare except, then after it works, stick a raise in it to determine exactly what exception I'm catching. Then use that to rewrite a more explicit except statement. Your comment earlier about treating the symptom is also accurate. This isn't just an issue with bare excepts not being allowed in the middle, it also comes up whenever we catch exceptions out of order in the tree. Ie.. catching an exception closer to the base will block a following except clause that tries to catch an exception on the same branch. So should except clauses be checked for orderliness? Regards, Ron From gvanrossum at gmail.com Wed Aug 24 19:15:47 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 24 Aug 2005 10:15:47 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> Message-ID: On 8/24/05, James Y Knight wrote: > I think it must be the case that raising an object which does not > derive from an exception class must be deprecated as well in order > for "except:" to be deprecated. Otherwise, there is nothing you can > change "except:" to in order not to get a deprecation warning and > still have your code be correct in the face of documented features of > python. I agree; isn't that already in ther PEP? This surely has been the thinking all along. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvanrossum at gmail.com Wed Aug 24 19:17:56 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 24 Aug 2005 10:17:56 -0700 Subject: [Python-Dev] FW: Bare except clauses in PEP 348 In-Reply-To: <003901c5a8c0$51d79fc0$b729cb97@oemcomputer> References: <003901c5a8c0$51d79fc0$b729cb97@oemcomputer> Message-ID: On 8/24/05, Raymond Hettinger wrote: > Hey guys, don't give up your bare except clauses so easily. They are an attractive nuisance by being so much shorter to type than the "right thing to do". Especially if they default to something whose use cases are rather esoteric (i.e. BaseException). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From abo at minkirri.apana.org.au Wed Aug 24 19:26:34 2005 From: abo at minkirri.apana.org.au (Donovan Baarda) Date: Wed, 24 Aug 2005 10:26:34 -0700 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C854E.1080200@v.loewis.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> <430C7F4C.9010703@livinglogic.de> <430C854E.1080200@v.loewis.de> Message-ID: <1124904393.9380.29.camel@warna.corp.google.com> On Wed, 2005-08-24 at 07:33, "Martin v. L?wis" wrote: > Walter D?rwald wrote: > > Martin v. L?wis wrote: > > > >> Walter D?rwald wrote: [...] > Actually, on a second thought - it would not remove the quadratic > aspect. You would still copy the rest string completely on each > split. So on the first split, you copy N lines (one result line, > and N-1 lines into the rest string), on the second split, N-2 > lines, and so on, totalling N*N/2 line copies again. The only > thing you save is the join (as the rest is already joined), and > the IsLineBreak calls (which are necessary only for the first > line). [...] In the past, I've avoided the string copy overhead inherent in split() by using buffers... I've always wondered why Python didn't use buffer type tricks internally for split-type operations. I haven't looked at Python's string implementation, but the fact that strings are immutable surely means that you can safely and efficiently reference an implementation level "data" object for all strings... ie all strings are "buffers". The only problem I can see with this is huge "data" objects might hang around just because some small fragment of it is still referenced by a string. Surely a simple huristic or two like "if len(string) < len(data)/8: copy data; else: reference data" would go a long way towards avoiding that. In my limited playing around with manipulating of strings and benchmarking stuff, the biggest overhead is nearly always the copys. -- Donovan Baarda From walter at livinglogic.de Wed Aug 24 19:35:11 2005 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 24 Aug 2005 19:35:11 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C4BA1.5030503@egenix.com> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C4BA1.5030503@egenix.com> Message-ID: <430CAFCF.3040109@livinglogic.de> M.-A. Lemburg wrote: > Walter D?rwald wrote: > >>I wonder if we should switch back to a simple readline() implementation >>for those codecs that don't require the current implementation >>(basically every charmap codec). > > That would be my preference as well. The 2.4 .readline() approach > is really only needed for codecs that have to deal with encodings > that: > > a) use multi-byte formats, or > b) support more line-end formats than just CR, CRLF, LF, or > c) are stateful. > > This can easily be had by using a mix-in class for > codecs which do need the buffered .readline() approach. Should this be a mix-in or should we simply have two base classes? Which of those bases/mix-ins should be the default? >>AFAIK source files are opened in >>universal newline mode, so at least we'd get proper treatment of "\n", >>"\r" and "\r\n" line ends, but we'd loose u"\x1c", u"\x1d", u"\x1e", >>u"\x85", u"\u2028" and u"\u2029" (which are line terminators according >>to unicode.splitlines()). > > While the Unicode standard defines these characters as line > end code points, I think their definition does not necessarily > apply to data that is converted from a certain encoding to > Unicode, so that's not a big loss. > > E.g. in ASCII or Latin-1, FILE, GROUP and RECORD > SEPARATOR and NEXT LINE characters (0x1c, 0x1d, 0x1e, 0x85) > are not interpreted as line end characters. > > Furthermore, we had no reports of anyone complaining in > Python 1.6, 2.0 - 2.3 that line endings were not detected > properly. All these Python versions relied on the stream's > .readline() method to get the next line. The only bug reports > we had were for UTF-16 which falls into the above > category a) and did not support .readline() until Python 2.4. True. Bye, Walter D?rwald From martin at v.loewis.de Wed Aug 24 19:38:54 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Aug 2005 19:38:54 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430CA75F.7090900@livinglogic.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> <430C7F4C.9010703@livinglogic.de> <430C854E.1080200@v.loewis.de> <430CA75F.7090900@livinglogic.de> Message-ID: <430CB0AE.1040201@v.loewis.de> Walter D?rwald wrote: > At least it would remove the quadratic number of calls to > _PyUnicodeUCS2_IsLinebreak(). For each character it would be called only > once. Correct. However, I very much doubt that this is the cause of the slowdown. > The last part of the patch seems to be more related to bug #1235646. You mean the last chunk (linebuffer=None)? This is just the extension to reset. > With the patch test_pep263 and test_codecs fail (and test_parser, but > this might be unrelated): Oops, I thought I ran the test suite, but apparently with the patch removed. New version uploaded. > Using collections.deque() should get rid of this problem. Alright. There are so many types in Python I've never heard of :-) > You mean, in the test suite? Right. > BTW, why the decode() call? For a Python without unicode? Right. Not sure what people think whether this should still be supported, but I keep supporting it whenever I think of it. > I wonder what happens, if calls to read() and readline() are mixed (e.g. > if I'm reading Fortran source or anything with a fixed line header): > read() would be used to read the first n character (which joins the line > buffer) and readline() reads the rest (which would split it again) etc. > (Of course this could be done via a single readline() call). Then performance would drop again - it should still be correct, though. If this is becomes a frequent problem, we could satisfy read requests from the split lines as well (i.e. join as many lines as you need). However, I would rather expect that callers of read() typically want the entire file, or want to read in large chunks (with no line orientation at all). > But, I think a maxsplit argument for splitlines() woould make sense > independent of this problem. I'm not so sure anymore. It is good for consistency, but I doubt there are actual use cases: how often do you want only the first n lines of some string? Reading the first n lines of a file might be an application, but then, you would rather use .readline() directly. For readline, I don't think there is a clear case for splitting of only the first line (unless you want to return an index instead of the rest string): if the application eventually wants all of the data, we better split it right away into individual strings, instead of dealing with a gradually decreasing trailer. Anyway, I don't think we should go back to C's readline/fgets. This is just too messy wrt. buffering and text vs. binary mode. I wish Python would stop using stdio entirely. Regards, Martin From walter at livinglogic.de Wed Aug 24 20:16:39 2005 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 24 Aug 2005 20:16:39 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430CB0AE.1040201@v.loewis.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> <430C7F4C.9010703@livinglogic.de> <430C854E.1080200@v.loewis.de> <430CA75F.7090900@livinglogic.de> <430CB0AE.1040201@v.loewis.de> Message-ID: <430CB987.5000601@livinglogic.de> Martin v. L?wis wrote: > Walter D?rwald wrote: > >>At least it would remove the quadratic number of calls to >>_PyUnicodeUCS2_IsLinebreak(). For each character it would be called only >>once. > > Correct. However, I very much doubt that this is the cause of the > slowdown. Probably. We'd need a test with the original Argon source to really know. >>The last part of the patch seems to be more related to bug #1235646. > > You mean the last chunk (linebuffer=None)? This is just the extension > to reset. Ouch, you're right: The part of "cvs diff" was part of my checkout, not your patch. I have so many Python checkouts, that I sometimes forget which is which! ;) >>With the patch test_pep263 and test_codecs fail (and test_parser, but >>this might be unrelated): > > Oops, I thought I ran the test suite, but apparently with the patch > removed. New version uploaded. Looks much better now. >>Using collections.deque() should get rid of this problem. > > Alright. There are so many types in Python I've never heard of :-) The problem is that unicode.splitlines() returns a list, so the push/pop performance advantange of collections.deque might be eaten by having to create a collections.deque object in the first place. >>You mean, in the test suite? > > Right. > >>BTW, why the decode() call? For a Python without unicode? > > Right. Not sure what people think whether this should still be > supported, but I keep supporting it whenever I think of it. OK, so should we add this for 2.4.2 or only for 2.5? Should this really be put into string.py, or should it be a class attribute of unicode? (At least that's what was proposed for the other strings in string.py (string.whitespace etc.) too. >>I wonder what happens, if calls to read() and readline() are mixed (e.g. >>if I'm reading Fortran source or anything with a fixed line header): >>read() would be used to read the first n character (which joins the line >>buffer) and readline() reads the rest (which would split it again) etc. >>(Of course this could be done via a single readline() call). > > Then performance would drop again - it should still be correct, though. > > If this is becomes a frequent problem, we could satisfy read requests > from the split lines as well (i.e. join as many lines as you need). > However, I would rather expect that callers of read() typically want > the entire file, or want to read in large chunks (with no line > orientation at all). Agreed! Don't fix a bug that hasn't been reported! ;) >>But, I think a maxsplit argument for splitlines() woould make sense >>independent of this problem. > > I'm not so sure anymore. It is good for consistency, but I doubt there > are actual use cases: how often do you want only the first n lines > of some string? Reading the first n lines of a file might be an > application, but then, you would rather use .readline() directly. Not every unicode string is read from a StreamReader. > For readline, I don't think there is a clear case for splitting of > only the first line (unless you want to return an index instead of > the rest string): if the application eventually wants all of the > data, we better split it right away into individual strings, instead > of dealing with a gradually decreasing trailer. True, this would be best for a readline loop. Another solution would be to have a unicode.itersplitlines() and store the iterator. Then we wouldn't need a maxsplit because you simply can stop iterating once you have what you want. > Anyway, I don't think we should go back to C's readline/fgets. This > is just too messy wrt. buffering and text vs. binary mode. I don't know about C's readline, but StreamReader.read() and StreamReader.readline() are messy enough. But at least it's something we can fix ourselves. > I wish > Python would stop using stdio entirely. So reverting to the 2.3 behaviour for simple codecs is out? Bye, Walter D?rwald From barry at python.org Wed Aug 24 20:25:21 2005 From: barry at python.org (Barry Warsaw) Date: Wed, 24 Aug 2005 14:25:21 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <430CA279.3080909@v.loewis.de> References: <001d01c5a855$447a9d20$ab12c797@oemcomputer> <1124895832.19291.10.camel@geddy.wooz.org> <430C9D90.6090102@hathawaymix.org> <934F5C4E-FF88-41D0-939E-623D0AFCDAE2@alum.mit.edu> <430CA279.3080909@v.loewis.de> Message-ID: <1124907921.19925.5.camel@geddy.wooz.org> On Wed, 2005-08-24 at 12:38, "Martin v. L?wis" wrote: > I personally dislike recording the execution path in > local variables. This is like setting a flag in a loop > before the break, and testing the flag afterwards. > You can do this, but the else: clause of the loop is > just more readable. Agreed! > This specific fragment has also the bug that a > KeyboardInterrupt before the assignment to complete > will cause a NameError/UnboundLocalError; this > can easily be fixed by moving the assignment before > the try block. And that begs the question whether getting rid of this common idiom is trading one common problem for another. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050824/eeedd633/attachment.pgp From reinhold-birkenfeld-nospam at wolke7.net Wed Aug 24 20:33:02 2005 From: reinhold-birkenfeld-nospam at wolke7.net (Reinhold Birkenfeld) Date: Wed, 24 Aug 2005 20:33:02 +0200 Subject: [Python-Dev] Docs/Pointer to Tools/scripts? Message-ID: Hi, after adding Oleg Broytmann's findnocoding.py to Tools/scripts, I wonder whether the Tools directory is documented at all. There are many useful scripts there which many people will not find if they are not listed anywhere in the docs. Just a thought. Reinhold -- Mail address is perfectly valid! From phd at mail2.phd.pp.ru Wed Aug 24 20:44:11 2005 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Wed, 24 Aug 2005 22:44:11 +0400 Subject: [Python-Dev] Docs/Pointer to Tools/scripts? In-Reply-To: References: Message-ID: <20050824184410.GB5666@phd.pp.ru> Hello! On Wed, Aug 24, 2005 at 08:33:02PM +0200, Reinhold Birkenfeld wrote: > after adding Oleg Broytmann's findnocoding.py to Tools/scripts What's more, pysource.py is more than just a script - it's a generally useful module. Thank you for committing the code. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From martin at v.loewis.de Wed Aug 24 21:15:09 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Aug 2005 21:15:09 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430CB987.5000601@livinglogic.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> <430C7F4C.9010703@livinglogic.de> <430C854E.1080200@v.loewis.de> <430CA75F.7090900@livinglogic.de> <430CB0AE.1040201@v.loewis.de> <430CB987.5000601@livinglogic.de> Message-ID: <430CC73D.1050401@v.loewis.de> Walter D?rwald wrote: >> Right. Not sure what people think whether this should still be >> supported, but I keep supporting it whenever I think of it. > > > OK, so should we add this for 2.4.2 or only for 2.5? You mean, string.unicodelinebreaks? I think something needs to be done to fix the performance problem. In doing so, API changes might occur. We should not add API changes in 2.4.2 unless they contribute to the bug fix, and even then, the release manager probably needs to approve them (in any case, they certainly need to be backwards compatible) > Should this really be put into string.py, or should it be a class > attribute of unicode? (At least that's what was proposed for the other > strings in string.py (string.whitespace etc.) too. If the 2.4.2 fix is based on this kind of data, I think it should go into a private attribute of codecs.py. For 2.5, I would put it into strings for tradition. There is no point in having some of these constants in strings and others as class attributes (unless we also add them as class attributes in 2.5, in which case adding unicodelinebreaks into strings would be pointless). So I think in 2.5, I would like to see # string.py ascii_letters = str.ascii_letters in which case unicode.linebreaks would be the right spelling. >> I'm not so sure anymore. It is good for consistency, but I doubt there >> are actual use cases: how often do you want only the first n lines >> of some string? Reading the first n lines of a file might be an >> application, but then, you would rather use .readline() directly. > > > Not every unicode string is read from a StreamReader. Sure: but how often do you want to fetch the first line of a Unicode string you happen to have in memory, without iterating over all lines eventually? > Another solution would be to have a unicode.itersplitlines() and store > the iterator. Then we wouldn't need a maxsplit because you simply can > stop iterating once you have what you want. That might work. I would then ask for itersplitlines to return pairs of (line, truncated) so you can easily know whether you merely ran into the end of the string, or whether you got a complete line (although it might be a bit too specific for the readlines() case) > So reverting to the 2.3 behaviour for simple codecs is out? I'm -1, atleast. It would also fix the problem at hand, for the reported case. However, it does leave some codecs in the cold, most notably UTF-8 (which, in turn, isn't an issue for PEP 262, since UTF-8 is built-in in the parser). I think the UTF-8 stream reader should support all Unicode line breaks, so it should continue to use the Python approach. However, UTF-8 is fairly common, so that reading an UTF-8-encoded file line-by-line shouldn't suck. Regards, Martin From raymond.hettinger at verizon.net Wed Aug 24 21:15:12 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Wed, 24 Aug 2005 15:15:12 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: Message-ID: <005a01c5a8e0$27480860$b729cb97@oemcomputer> [Guido van Rossum] > > OK, I'm convinced. Let's drop bare except for Python 3.0, and > > deprecate them until then, without changing the meaning. > > > > The deprecation message (to be generated by the compiler!) should > > steer people in the direction of specifying one particular exception > > (e.g. KeyError etc.) rather than Exception. [James Y Knight] > I agree but there's the minor nit of non-Exception exceptions. > > I think it must be the case that raising an object which does not > derive from an exception class must be deprecated as well in order > for "except:" to be deprecated. Otherwise, there is nothing you can > change "except:" to in order not to get a deprecation warning and > still have your code be correct in the face of documented features of > python. Hmm, that may not be a killer. I wonder if it is possible to treat BaseException as a constant (like we do with None) and teach the compiler to interpret it as catching anything that gets raised so that "except BaseException" will work like a bare except clause does now. Raymond From barry at python.org Wed Aug 24 21:21:47 2005 From: barry at python.org (Barry Warsaw) Date: Wed, 24 Aug 2005 15:21:47 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <005a01c5a8e0$27480860$b729cb97@oemcomputer> References: <005a01c5a8e0$27480860$b729cb97@oemcomputer> Message-ID: <1124911307.19921.11.camel@geddy.wooz.org> On Wed, 2005-08-24 at 15:15, Raymond Hettinger wrote: > Hmm, that may not be a killer. I wonder if it is possible to treat > BaseException as a constant (like we do with None) and teach the > compiler to interpret it as catching anything that gets raised so that > "except BaseException" will work like a bare except clause does now. Sorry Raymond, but my first reaction is "ick" :). That seems to be a big change in the semantics of exception matching. I think I'd rather keep bare except than add that! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050824/f1600681/attachment-0001.pgp From raymond.hettinger at verizon.net Wed Aug 24 21:30:28 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Wed, 24 Aug 2005 15:30:28 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <1124911307.19921.11.camel@geddy.wooz.org> Message-ID: <005e01c5a8e2$49102fc0$b729cb97@oemcomputer> > > Hmm, that may not be a killer. I wonder if it is possible to treat > > BaseException as a constant (like we do with None) and teach the > > compiler to interpret it as catching anything that gets raised so that > > "except BaseException" will work like a bare except clause does now. > > Sorry Raymond, but my first reaction is "ick" :). That seems to be a > big change in the semantics of exception matching. I think I'd rather > keep bare except than add that! That may be your only other option if we're waiting until 3.0 to eliminate string exceptions and class exceptions not derived from the hierarchy. Raymond From mwh at python.net Wed Aug 24 21:52:13 2005 From: mwh at python.net (Michael Hudson) Date: Wed, 24 Aug 2005 20:52:13 +0100 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <005e01c5a8e2$49102fc0$b729cb97@oemcomputer> (Raymond Hettinger's message of "Wed, 24 Aug 2005 15:30:28 -0400") References: <005e01c5a8e2$49102fc0$b729cb97@oemcomputer> Message-ID: <2mbr3nru36.fsf@starship.python.net> "Raymond Hettinger" writes: >> > Hmm, that may not be a killer. I wonder if it is possible to treat >> > BaseException as a constant (like we do with None) and teach the >> > compiler to interpret it as catching anything that gets raised so > that >> > "except BaseException" will work like a bare except clause does now. >> >> Sorry Raymond, but my first reaction is "ick" :). That seems to be a >> big change in the semantics of exception matching. I think I'd rather >> keep bare except than add that! > > That may be your only other option if we're waiting until 3.0 to > eliminate string exceptions and class exceptions not derived from the > hierarchy. I really hope string exceptions can be killed off before 3.0. They should be fully deprecated in 2.5. Cheers, mwh -- The Oxford Bottled Beer Database heartily disapproves of the excessive consumption of alcohol. No, really. -- http://www.bottledbeer.co.uk/beergames.html (now sadly gone to the big 404 in the sky) From walter at livinglogic.de Wed Aug 24 22:14:32 2005 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 24 Aug 2005 22:14:32 +0200 Subject: [Python-Dev] 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430CC73D.1050401@v.loewis.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> <430C7F4C.9010703@livinglogic.de> <430C854E.1080200@v.loewis.de> <430CA75F.7090900@livinglogic.de> <430CB0AE.1040201@v.loewis.de> <430CB987.5000601@livinglogic.de> <430CC73D.1050401@v.loewis.de> Message-ID: <671A6329-ED68-491F-84CB-1D2CF00A2F6A@livinglogic.de> Am 24.08.2005 um 21:15 schrieb Martin v. L?wis: > Walter D?rwald wrote: > > >>> Right. Not sure what people think whether this should still be >>> supported, but I keep supporting it whenever I think of it. >>> >> >> OK, so should we add this for 2.4.2 or only for 2.5? >> > > You mean, string.unicodelinebreaks? > Yes. > I think something needs to be > done to fix the performance problem. In doing so, API changes > might occur. We should not add API changes in 2.4.2 unless they > contribute to the bug fix, and even then, the release manager > probably needs to approve them (in any case, they certainly > need to be backwards compatible) > OK. Your version of the patch (without replacing line = line.splitlines(False)[0] with something better) might be enough for 2.4.2. >> Should this really be put into string.py, or should it be a class >> attribute of unicode? (At least that's what was proposed for the >> other >> strings in string.py (string.whitespace etc.) too. >> > > If the 2.4.2 fix is based on this kind of data, I think it should go > into a private attribute of codecs.py. > I think codecs.unicodelinebreaks has one big problem: it will not work for codecs that do str->str decoding. > For 2.5, I would put it > into strings for tradition. There is no point in having some of these > constants in strings and others as class attributes (unless we also > add them as class attributes in 2.5, in which case adding > unicodelinebreaks into strings would be pointless). > > So I think in 2.5, I would like to see > > # string.py > ascii_letters = str.ascii_letters > > in which case unicode.linebreaks would be the right spelling. > And it would have the advantage, that it could work both with str and unicode if we had both str.linebreaks and unicode.linebreaks >>> I'm not so sure anymore. It is good for consistency, but I doubt >>> there >>> are actual use cases: how often do you want only the first n lines >>> of some string? Reading the first n lines of a file might be an >>> application, but then, you would rather use .readline() directly. >>> >> >> Not every unicode string is read from a StreamReader. >> > > Sure: but how often do you want to fetch the first line of a Unicode > string you happen to have in memory, without iterating over all lines > eventually? > I don't know. The only obvious spot in the standard library (apart from codecs.py) seems to be def shortdescription(self): return self.description().splitlines() [0] in Lib/plat-mac/pimp.py >> Another solution would be to have a unicode.itersplitlines() and >> store >> the iterator. Then we wouldn't need a maxsplit because you simply can >> stop iterating once you have what you want. >> > > That might work. I would then ask for itersplitlines to return pairs > of (line, truncated) so you can easily know whether you merely ran > into the end of the string, or whether you got a complete line > (although it might be a bit too specific for the readlines() case) > Or maybe (line, terminatorlength) which gives you the same info (terminatorlength == 0 means truncated) and makes it easy to strip the terminator. >> So reverting to the 2.3 behaviour for simple codecs is out? >> > > I'm -1, atleast. It would also fix the problem at hand, for the > reported > case. However, it does leave some codecs in the cold, most notably > UTF-8 (which, in turn, isn't an issue for PEP 262, since UTF-8 is > built-in in the parser). > You meant PEP 263, right? > I think the UTF-8 stream reader should support > all Unicode line breaks, so it should continue to use the Python > approach. > OK. > However, UTF-8 is fairly common, so that reading an > UTF-8-encoded file line-by-line shouldn't suck. > OK, so what's missing is a solution for str->str codecs (or we keep line = line.splitlines(False)[0] and test, whether this is fast enough). Bye, Walter D?rwald From tzot at mediconsa.com Wed Aug 24 22:48:28 2005 From: tzot at mediconsa.com (Christos Georgiou) Date: Wed, 24 Aug 2005 23:48:28 +0300 Subject: [Python-Dev] Docs/Pointer to Tools/scripts? References: Message-ID: "Reinhold Birkenfeld" wrote in message news:deiegu$jqf$1 at sea.gmane.org... > Hi, > > after adding Oleg Broytmann's findnocoding.py to Tools/scripts, I wonder > whether the Tools directory is documented at all. There are many useful > scripts there which many people will not find if they are not listed > anywhere in the docs. AFAIK the only documentation is the README file in said directory. From walter at livinglogic.de Wed Aug 24 23:12:38 2005 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 24 Aug 2005 23:12:38 +0200 Subject: [Python-Dev] [Argon] Re: 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> <430C7F4C.9010703@livinglogic.de> <430C854E.1080200@v.loewis.de> <430CA75F.7090900@livinglogic.de> <430CB0AE.1040201@v.loewis.de> <430CB987.5000601@livinglogic.de> Message-ID: <8FD4A0C3-D54B-403C-9BC7-052D2FB1F0E5@livinglogic.de> Am 24.08.2005 um 20:20 schrieb Greg Wilson: >>> Walter D?rwald wrote: >>> >>>> At least it would remove the quadratic number of calls to >>>> _PyUnicodeUCS2_IsLinebreak(). For each character it would be >>>> called only >>>> once. > >> Martin v. L?wis wrote: >> >>> Correct. However, I very much doubt that this is the cause of the >>> slowdown. > >> Walter D?rwald wrote: >> Probably. We'd need a test with the original Argon source to >> really know. > > We can do that. So, can you try Martin's patch? >> OK, so should we add this for 2.4.2 or only for 2.5? > > 2.4.2 please ;-) If we use the patch as is, I think it can go into 2.4.2. Bye, Walter D?rwald From martin at v.loewis.de Wed Aug 24 23:37:53 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Aug 2005 23:37:53 +0200 Subject: [Python-Dev] Docs/Pointer to Tools/scripts? In-Reply-To: References: Message-ID: <430CE8B1.4030409@v.loewis.de> Reinhold Birkenfeld wrote: > after adding Oleg Broytmann's findnocoding.py to Tools/scripts, I > wonder whether the Tools directory is documented at all. There are > many useful scripts there which many people will not find if they are > not listed anywhere in the docs. Christos already mentioned it: there is a README file in both Tools and Tools/scripts; you should update it whenever you add something. Regards, Martin From gvanrossum at gmail.com Thu Aug 25 02:28:35 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 24 Aug 2005 17:28:35 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <2mbr3nru36.fsf@starship.python.net> References: <005e01c5a8e2$49102fc0$b729cb97@oemcomputer> <2mbr3nru36.fsf@starship.python.net> Message-ID: On 8/24/05, Michael Hudson wrote: > I really hope string exceptions can be killed off before 3.0. They > should be fully deprecated in 2.5. But what about class exceptions that don't inherit from Exception? That will take a while before we can deprecate that. Anyway, there have been plenty of cases where I was only interested in catching arbitrary exceptions generated by *Python* (as opposed to broken 3rd party code or even obscure Python library code) and those all inherit from Exception. And in those cases I've written "except Exception:" and so far never regretted it. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bcannon at gmail.com Thu Aug 25 03:39:35 2005 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 24 Aug 2005 18:39:35 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> Message-ID: On 8/24/05, Guido van Rossum wrote: > On 8/24/05, James Y Knight wrote: > > I think it must be the case that raising an object which does not > > derive from an exception class must be deprecated as well in order > > for "except:" to be deprecated. Otherwise, there is nothing you can > > change "except:" to in order not to get a deprecation warning and > > still have your code be correct in the face of documented features of > > python. > > I agree; isn't that already in ther PEP? This surely has been the > thinking all along. > Requiring inheritance of BaseException in order to pass it to 'raise' has been in the PEP since the beginning. -Brett From bcannon at gmail.com Thu Aug 25 03:43:23 2005 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 24 Aug 2005 18:43:23 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> Message-ID: On 8/24/05, Guido van Rossum wrote: > On 8/24/05, Michael Chermside wrote: > > Explicit is better than Implicit. I think that in newly written code > > "except Exception:" is better (more explicit and easier to understand) > > than "except:" Legacy code that uses "except:" can remain unchanged *IF* > > the meaning of "except:" is unchanged... but I think we all agree that > > this is unwise because the existing meaning is a tempting trap for the > > unwary. So I don't see any advantage to keeping bare "except:" in the > > long run. What we do to ease the transition is a different question, > > but one more easily resolved. > > OK, I'm convinced. Let's drop bare except for Python 3.0, and > deprecate them until then, without changing the meaning. > Woohoo! I am currently on vacation before school starts (orientation is Sept 1., classes start Sept. 6), so it might take me a little while to edit the PEP, but I will try to fit into my schedule ASAP (assuming the tide doesn't turn on me before then). > The deprecation message (to be generated by the compiler!) should > steer people in the direction of specifying one particular exception > (e.g. KeyError etc.) rather than Exception. Is there any desire for a __future__ statement that makes it a syntax error? How about making 'raise' statements only work with objects that inherit from BaseException? -Brett From gvanrossum at gmail.com Thu Aug 25 04:02:00 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 24 Aug 2005 19:02:00 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> Message-ID: On 8/24/05, Brett Cannon wrote: > Is there any desire for a __future__ statement that makes it a syntax > error? How about making 'raise' statements only work with objects > that inherit from BaseException? I doubt it. Few people are going to put a __future__ statement in to make sure that *don't* use a particular feature: it's just as easy to grep your source code for "except:". __future__ is in general only used to enable new syntax that previously has a different meaning. Anyway, you can make it an error globally by using the -W option creatively. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From amk at amk.ca Thu Aug 25 04:33:42 2005 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 24 Aug 2005 22:33:42 -0400 Subject: [Python-Dev] New mailbox module Message-ID: <20050825023342.GA20941@rogue.amk.ca> Gregory K. Johnson, who's been working on the mailbox module in nondist/sandbox/mailbox for Google's Summer of Code, thinks his project is essentially complete. He's added the ability to modifying mailboxes by adding and removing messages, adding test cases for the new features, and written the corresponding documentation. So, it's time to start considering it for inclusion in the standard library. This is a big change to a non-obscure module, so don't feel able to make this decision on my own. I believe the code quality is acceptable, but would appreciate comments on any cleanups that need to be made. I still need to read through the docs and make editing suggestions, and check that the code is still backward-compatible with the old version of the module. --amk From barry at python.org Thu Aug 25 06:22:43 2005 From: barry at python.org (Barry Warsaw) Date: Thu, 25 Aug 2005 00:22:43 -0400 Subject: [Python-Dev] New mailbox module In-Reply-To: <20050825023342.GA20941@rogue.amk.ca> References: <20050825023342.GA20941@rogue.amk.ca> Message-ID: <1124943762.10479.0.camel@geddy.wooz.org> On Wed, 2005-08-24 at 22:33, A.M. Kuchling wrote: > So, it's time to start considering it for inclusion in the standard > library. This is a big change to a non-obscure module, so don't feel > able to make this decision on my own. > > I believe the code quality is acceptable, but would appreciate > comments on any cleanups that need to be made. I still need to read > through the docs and make editing suggestions, and check that the code > is still backward-compatible with the old version of the module. I plan to take a look at it, but won't get a chance to do so for several days. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20050825/fd21d4b6/attachment.pgp From foom at fuhm.net Thu Aug 25 06:45:12 2005 From: foom at fuhm.net (James Y Knight) Date: Thu, 25 Aug 2005 00:45:12 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> Message-ID: <5BB88A76-FB4E-41F7-B82D-4B7C5B6D28DD@fuhm.net> On Aug 24, 2005, at 9:39 PM, Brett Cannon wrote: > On 8/24/05, Guido van Rossum wrote: >> On 8/24/05, James Y Knight wrote: >>> I think it must be the case that raising an object which does not >>> derive from an exception class must be deprecated as well in order >>> for "except:" to be deprecated. Otherwise, there is nothing you can >>> change "except:" to in order not to get a deprecation warning and >>> still have your code be correct in the face of documented >>> features of >>> python. >>> >> >> I agree; isn't that already in ther PEP? This surely has been the >> thinking all along. >> >> > > Requiring inheritance of BaseException in order to pass it to 'raise' > has been in the PEP since the beginning. Yes, it talks about that as a change that will happen in Python 3.0. I was responding to >> OK, I'm convinced. Let's drop bare except for Python 3.0, and >> deprecate them until then, without changing the meaning. which is talking about deprecating bare excepts in Python 2.5. Now maybe it's the idea that everything that's slated for removal in Python 3.0 by PEP 348 is supposed to be getting a deprecation warning in Python 2.5, but that certainly isn't stated. The transition plan section says that all that will happen in Python 2.5 is the addition of "BaseException". James From gvanrossum at gmail.com Thu Aug 25 07:13:09 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 24 Aug 2005 22:13:09 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <5BB88A76-FB4E-41F7-B82D-4B7C5B6D28DD@fuhm.net> References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> <5BB88A76-FB4E-41F7-B82D-4B7C5B6D28DD@fuhm.net> Message-ID: On 8/24/05, James Y Knight wrote: > On Aug 24, 2005, at 9:39 PM, Brett Cannon wrote: > > On 8/24/05, Guido van Rossum wrote: > >> On 8/24/05, James Y Knight wrote: > >>> I think it must be the case that raising an object which does not > >>> derive from an exception class must be deprecated as well in order > >>> for "except:" to be deprecated. Otherwise, there is nothing you can > >>> change "except:" to in order not to get a deprecation warning and > >>> still have your code be correct in the face of documented > >>> features of > >>> python. > >>> > >> > >> I agree; isn't that already in ther PEP? This surely has been the > >> thinking all along. > >> > >> > > > > Requiring inheritance of BaseException in order to pass it to 'raise' > > has been in the PEP since the beginning. > > Yes, it talks about that as a change that will happen in Python 3.0. > I was responding to > > >> OK, I'm convinced. Let's drop bare except for Python 3.0, and > >> deprecate them until then, without changing the meaning. > > which is talking about deprecating bare excepts in Python 2.5. Now > maybe it's the idea that everything that's slated for removal in > Python 3.0 by PEP 348 is supposed to be getting a deprecation warning > in Python 2.5, but that certainly isn't stated. The transition plan > section says that all that will happen in Python 2.5 is the addition > of "BaseException". Then maybe the PEP isn't perfect just yet. :-) It's never too early to start deprecating a feature we know will disappear in 3.0. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Thu Aug 25 10:16:02 2005 From: mwh at python.net (Michael Hudson) Date: Thu, 25 Aug 2005 09:16:02 +0100 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: (Guido van Rossum's message of "Wed, 24 Aug 2005 17:28:35 -0700") References: <005e01c5a8e2$49102fc0$b729cb97@oemcomputer> <2mbr3nru36.fsf@starship.python.net> Message-ID: <2m7jeasa7x.fsf@starship.python.net> Guido van Rossum writes: > On 8/24/05, Michael Hudson wrote: >> I really hope string exceptions can be killed off before 3.0. They >> should be fully deprecated in 2.5. > > But what about class exceptions that don't inherit from Exception? > That will take a while before we can deprecate that. Oh, for sure. I didn't mean to imply anything else. Cheers, mwh -- "Sturgeon's Law (90% of everything is crap) applies to Usenet." "Nothing guarantees that the 10% isn't crap, too." -- Gene Spafford's Axiom #2 of Usenet, and a corollary From t-meyer at ihug.co.nz Thu Aug 25 10:51:18 2005 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Thu, 25 Aug 2005 20:51:18 +1200 Subject: [Python-Dev] python-dev Summary for 2005-08-01 through 2005-08-15 [draft] Message-ID: Here's August Part One. As usual, if anyone can spare the time to proofread this, that would be great! Please send any corrections or suggestions to Steve (steven.bethard at gmail.com) and/or me, rather than cluttering the list. Ta! ============= Announcements ============= ---------------------------- QOTF: Quote of the Fortnight ---------------------------- Some wise words from Donovan Baarda in the PEP 347 discussions: It is true that some well designed/developed software becomes reliable very quickly. However, it still takes heavy use over time to prove that. Contributing thread: - `PEP: Migrating the Python CVS to Subversion `__ [SJB] ------------ Process PEPs ------------ The PEP editors have introduced a new PEP category: "Process", for PEPs that don't fit into the "Standards Track" and "Informational" categories. More detail can be found in `PEP 1`_, which it itself a Process PEP. .. _PEP 1: http://www.python.org/peps/pep-0001.html Contributing thread: - `new PEP type: Process `__ [TAM] ----------------------------------------------- Tentative Schedule for 2.4.2 and 2.5a1 Releases ----------------------------------------------- Python 2.4.2 is tentatively scheduled for a mid-to-late September release, and a first alpha of Python 2.5 for March 2006 (with a final release around May/June). This means that a PEP for the 2.5 release, detailing what will be included, will likely be created soon; at present there are various accepted PEPs that have not yet been implemented. Contributing thread: - `plans for 2.4.2 and 2.5a1 `__ [TAM] ========= Summaries ========= ------------------------------- Moving Python CVS to Subversion ------------------------------- The `PEP 347`_ discussion from last fortnight continued this week, with a revision of the PEP, and a lot more discussion about possible version control software (RCS) for the Python repository, and where the repository should be hosted. Note that this is not a discussion about bug trackers, which will remain with Sourceforge (unless a separate PEP is developed for moving that). Many revision control systems were extensively discussed, including `Subversion`_ (SVN), `Perforce`_, and `Monotone`_. Whichever system is moved to, it should be able to be hosted somewhere (if *.python.org, then it needs to be easily installable), needs to have software available to convert a repository from CVS, and ideally would be open-source; similarity to CVS is also an advantage in that it requires a smaller learning curve for existing developers. While Martin isn't willing to discuss every system there is, he will investigate those that make him curious, and will add other people's submissions to the PEP, where appropriate. The thread included a short discussion about the authentication mechanism that svn.python.org will use; svn+ssh seems to be a clear winner, and a test repository will be setup by Martin next fortnight. The possibility of moving to a distributed revision control system (particularly `Bazaar-NG`_) was also brought up. Many people liked the idea of using a distributed revision control system, but it seems unlikely that Bazaar-NG is mature enough to be used for the main Python repository at the current time (a move to it at a later time is possible, but outside the scope of the PEP). Distributed RCS are meant to reduce the barrier to participation (anyone can create the their own branches, for example); Bazaar-NG is also implemented in Python, which is of some benefit. James Y Knight pointed out `svk`_, which lets developers create their own branches within SVN. In general, the python-dev crowd is in favour of moving to SVN. Initial concern about the demands on the volunteer admins should the repository be hosted at svn.python.org were addressed by Barry Warsaw, who believes that the load will be easily managed with the existing volunteers. Various alternative hosts were discussed, and if detailed reports about any of them are created, these can be added to the PEP. While the fate of all PEPS lie with the BDFL (Guido), it is likely that the preferences of those that frequently check in changes, the pydotorg admins, and the release managers (who have all given favourable reports so far), will have a significant effect on the pronouncement of this PEP. .. _PEP 347: http://www.python.org/peps/pep-0347.html .. _svk: http://svk.elixus.org/ .. _Perforce: http://www.perforce.com/ .. _Subversion: http://subversion.tigris.org/ .. _Monotone: http://venge.net/monotone/ .. _Bazaar-NG: http://www.bazaar-ng.org/ Contributing threads: - `PEP: Migrating the Python CVS to Subversion `__ - `PEP 347: Migration to Subversion `__ - `Hosting svn.python.org `__ - `Fwd: Distributed RCS `__ - `cvs to bzr? `__ - `Distributed RCS `__ - `Fwd: PEP: Migrating the Python CVS to Subversion `__ - `On distributed vs centralised SCM for Python `__ [TAM] ------------------------------------------ PEP 348: Exception Hierarchy in Python 3.0 ------------------------------------------ This fortnight mostly concluded the previous discussion about `PEP 348`_, which sets out a roadmap for changes to the exception hierarchy in Python 3.0. The proposal was heavily scaled back to retain most of the current exception hierarchy unchanged. A new exception, BaseException, will be introduced above Exception in the current hierarchy, and KeyboardInterrupt and SystemExit will become siblings of Exception. The goal here is that:: except Exception: will now do the right thing for most cases, that is, it will catch all the exceptions that you can generally recover from. The PEP would also move NotImplementedError out from under RuntimeError, and alter the semantics of the bare except so that:: except: is the equivalent of:: except Exception: Only BaseException will appear in Python 2.5. The remaining modifications will not occur until Python 3.0. .. _PEP 348: http://www.python.org/peps/pep-0348.html Contributing threads: - `Pre-PEP: Exception Reorganization for Python 3.0 `__ - `PEP, take 2: Exception Reorganization for Python 3.0 `__ - `Exception Reorg PEP checked in `__ - `PEP 348: Exception Reorganization for Python 3.0 `__ - `Major revision of PEP 348 committed `__ - `Exception Reorg PEP revised yet again `__ - `PEP 348 and ControlFlow `__ - `PEP 348 (exception reorg) revised again `__ [SJB] ---------------------- Moving towards Unicode ---------------------- Neil Schemenauer presented `PEP 349`_, which tries to ease the transition to Python 3.0, in which there will be a bytes() type for byte data and a str() type for text data. Currently to convert an object to text, you have one of three options: * Call str(). This breaks with a UnicodeEncodeError if the object is of type unicode (or a subtype) or can only represent itself in unicode and therefore returns unicode from __str__. * Call unicode(). This can break external code that is not yet Unicode-safe and that passed a str object to your code but got a unicode object back. * Use the "%s" format specifier. This breaks with a UnicodeEncodeError if the object can only represent itself in unicode and therefore returns unicode from __str__. `PEP 349`_ attempts to address this problem by introducing a text() builtin which returns str or unicode instances unmodified, and returns the result of calling __str__() on the object otherwise. Guido preferred to instead relax the restrictions on str() to allow it to return unicode objects. Neil implemented such a patch, and found that it broke only two test cases. The discussion stopped shortly after Neil's report however, so it was unclear if any permanent changes had been agreed upon. Guido made a few other Python 3.0 suggestions in this thread: * The bytes() type should be mutable with a corresponding frozenbytes() immutable type * Opening a file in binary or text mode would cause it to return bytes() or str() objects, respectively * The file type should grow a getpos()/setpos() pair that are identical to tell()/seek() when a file is open in binary mode, and which work like tell()/seek() but on characters instead of bytes when a file is open in text mode However, none of these seemed to be solid commitments. .. _PEP 349: http://www.python.org/peps/pep-0349.html Contributing threads: - `PEP: Generalised String Coercion `__ - `Generalised String Coercion `__ [SJB] ---------------------------- PEP 344 and reference cycles ---------------------------- Armin Rigo brought up an issue with `PEP 344`_ which proposes, among other things, adding a __traceback__ attribute to exceptions to avoid the hassle of extracting it from sys.exc_info(). Armin pointed out that if exceptions grow a __traceback__ attribute, every statement:: except Exception, e: will create a cycle:: e.__traceback__.tb_frame.f_locals['e'] Despite the fact that Python has cyclic garbage collection, there are still some situations where cycles like this can cause problems. Armin showed an example of such a case:: class X: def __del__(self): try: typo except Exception, e: e_type, e_value, e_tb = sys.exc_info() Even in current Python, instances of the X class are uncollectible. When garbage collection runs and tries to collect an X object, it calls the __del__() method. This creates the cycle:: e_tb.tb_frame.f_locals['e_tb'] The X object itself is available through this cycle (in ``f_locals['self']``), so the X object's refcount does not drop to 0 when __del__() returns, so it cannot be collected. The next time garbage collection runs, it finds that the X object has not been collected, calls its __del__() method again and repeats the process. Tim Peters suggested this problem could be solved by declaring that __del__() methods are called exactly once. This allows the above X object to be collected because on the second run of the garbage collection, __del__() is not called again. Thus, the refcount of the X object is not incremented, and so it is collected by garbage collection. However, guaranteeing that __del__() is called only once means keeping track somehow of which objects' __del__() methods were called, which seemed somewhat unattractive. There was also brief talk about removing __del__ in favor of weakrefs, but those waters seemed about as murky as the garbage collection ones. .. _PEP 344: http://www.python.org/peps/pep-0344.html Contributing thread: - `__traceback__ and reference cycles `__ [SJB] ---------------------------- Style for raising exceptions ---------------------------- Guido explained that these days exceptions should always be raised as:: raise SomeException("some argument") instead of:: raise SomeException, "some argument" The second will go away in Python 3.0, and is only present now for backwards compatibility. (It was necessary when strings could be exceptions, in order to pass both the exception "type" and message.) PEPs 8_ and 3000_ were accordingly updated. .. _8: http://www.python.org/peps/pep-0008.html .. _3000: http://www.python.org/peps/pep-3000.html Contributing threads: - `PEP 8: exception style `__ - `FW: PEP 8: exception style `__ [SJB] ----------------------------------- Skipping list comprehensions in pdb ----------------------------------- When using pdb, the only way to skip to the end of a loop is to set a breakpoint on the line after the loop. Ilya Sandler suggested adding an optimal numeric argument to pdb's "next" comment to indicate how many lines of code should be skipped. Martin v. L?wis pointed out that this differs from gdb's "next " command, which does "next" n times. Ilya suggested implementing gdb's "until" command instead, which gained Martin's approval. It was also pointed out that pdb is one of the less Pythonic modules, particularly in terms of the ability to subclass/extend, and would be a good candidate for rewriting, if anyone had the inclination and time. Contributing threads: - `pdb: should next command be extended? `__ - `an alternative suggestion, Re: pdb: should next command be extended? `__ [TAM] ------------------ Sets in Python 2.5 ------------------ Raymond Hettinger has been checking-in the new implementation for sets in Python 2.5. The implementation is based heavily on dictobject.c, the code for Python dict() objects, and generally deviates only when there is an obvious gain in doing so. Raymond posted his new API for discussion, but there didn't appear to be any comments. Contributing threads: - `[Python-checkins] python/dist/src/Objects setobject.c, 1.45, 1.46 `__ - `Discussion draft: Proposed Py2.5 C API for set and frozenset objects `__ [SJB] ================================ Deferred Threads (for next time) ================================ - `SWIG and rlcompleter `__ =============== Skipped Threads =============== - `Extension of struct to handle non byte aligned values? `__ - `Syscall Proxying in Python `__ - `__autoinit__ (Was: Proposal: reducing self.x=x; self.y=y; self.z=z boilerplate code) `__ - `Weekly Python Patch/Bug Summary `__ - `PEP 342 Implementation `__ - `String exceptions in Python source `__ - `[ python-Patches-790710 ] breakpoint command lists in pdb `__ - `[C++-sig] GCC version compatibility `__ - `PyTuple_Pack added references undocumented `__ - `PEP-- Context Managment variant `__ - `Sourceforge CVS down? `__ - `PSF grant / contacts `__ - `Python + Ping `__ - `Terminology for PEP 343 `__ - `dev listinfo page (was: Re: Python + Ping) `__ - `set.remove feature/bug `__ - `Extension to dl module to allow passing strings from native function `__ - `build problems on macosx (CVS HEAD) `__ - `request for code review - hashlib - patch #1121611 `__ - `python-dev Summary for 2005-07-16 through 2005-07-31 [draft] `__ - `string_join overrides TypeError exception thrown in generator `__ - `implementation of copy standard lib `__ - `xml.parsers.expat no userdata in callback functions `__ From mal at egenix.com Thu Aug 25 11:35:59 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 25 Aug 2005 11:35:59 +0200 Subject: [Python-Dev] Style for raising exceptions (python-dev Summary for 2005-08-01 through 2005-08-15 [draft]) In-Reply-To: References: Message-ID: <430D90FF.6060206@egenix.com> I must have missed this one: > ---------------------------- > Style for raising exceptions > ---------------------------- > > Guido explained that these days exceptions should always be raised as:: > > raise SomeException("some argument") > > instead of:: > > raise SomeException, "some argument" > > The second will go away in Python 3.0, and is only present now for backwards > compatibility. (It was necessary when strings could be exceptions, in > order to pass both the exception "type" and message.) PEPs 8_ and 3000_ > were accordingly updated. AFAIR, the second form was also meant to be able to defer the instantiation of the exception class until really needed in order to reduce the overhead related to raising exceptions in Python. However, that optimization never made it into the implementation, I guess. > .. _8: http://www.python.org/peps/pep-0008.html > .. _3000: http://www.python.org/peps/pep-3000.html > > Contributing threads: > > - `PEP 8: exception style > `__ > - `FW: PEP 8: exception style > `__ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 25 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From raymond.hettinger at verizon.net Thu Aug 25 11:35:31 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Thu, 25 Aug 2005 05:35:31 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: Message-ID: <000d01c5a958$566e1bc0$b729cb97@oemcomputer> > > OK, I'm convinced. Let's drop bare except for Python 3.0, and > > deprecate them until then, without changing the meaning. > > > > Woohoo That's no cause for celebration. Efforts to improve Py3.0 have spilled over into breaking Py2.x code with no compensating benefits. Bare except clauses appear in almost every Python book that has ever been written and occur at least once in most major Python applications. I had thought the plan was to introduce Py3.0 capabilities into 2.x as they become possible but not to break anything. Isn't that why string exceptions, buffer(), and repr() still live and breathe? We don't have to wreck 2.x in order to make 3.0 better. I wish the 3.0 PEPs would stop until we are actually working on the project and have some chance of making people's lives better. If people avoid 2.5 just to avert unnecessary breakage, then Py3.0 doesn't benefit at all. I propose that the transition plan be as simple as introducing BaseException. This allows people to write code that will work on both 2.x and 3.0. It doesn't break anything. The guidance for cross-version (2.5 to 3.0) code would be: * To catch all but terminating exceptions, write: except (KeyError, SystemExit): raise except Exception: ... * To catch all exceptions, write: except BaseException: ... To make the code also run on 2.4 and prior, add transition code: try: BaseException except NameError: class BaseException(Exception): pass With that minimal guidance, people can write code that works on from 2.0 to 3.0 and not break anything that is currently working. No deprecations are necessary. Remember, the ONLY benefit from the whole PEP is that in 3.0, it will no longer be necessary to write "except (KeyError, SystemExit): raise". Steven and Jack's research show that that doesn't arise much in practice anyway. IOW, there's nothing worth inflicting destruction on tons of 2.x code. Raymond From mcherm at mcherm.com Thu Aug 25 14:28:44 2005 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 25 Aug 2005 05:28:44 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 Message-ID: <20050825052844.sby2v04s568sgg00@login.werra.lunarpages.com> Raymond writes: > Efforts to improve Py3.0 have spilled > over into breaking Py2.x code with no compensating benefits. [...] > We don't have to wreck 2.x in order to make 3.0 better. I think you're overstating things a bit here. > Remember, the ONLY benefit from the whole PEP is that in 3.0, it will no > longer be necessary to write "except (KeyError, SystemExit): raise". > [...] IOW, there's nothing worth inflicting destruction on tons of > 2.x code. And now I *KNOW* you're overstating things. There are LOTS of benefits to the PEP in 3.0. My own personal favorite is that users can be guaranteed that all exceptions thrown will share a particular common ancestor and type. And no one is proposing "destruction" of 2.x code. On the other hand, I thought these were very good points: > Bare > except clauses appear in almost every Python book that has ever been > written and occur at least once in most major Python applications. [...] > I had thought the plan was to introduce Py3.0 capabilities into 2.x as > they become possible but not to break anything. [...] > I propose that the transition plan be as simple as introducing > BaseException. This allows people to write code that will work on both > 2.x and 3.0. I think the situation is both better than and worse than you describe. The PEP is now proposing that bare "except:" be removed in Python 3.0. If I understand Guido correctly, he is proposing that in 2.5 the use of bare "except:" generate a PendingDeprecationWarning so that conscientious developers who want to write code now that will continue to work in Python 3.0 can avoid using bare "except:". Perhaps I'm misreading him here, but I presume this was intended as a PENDINGDeprecationWarning so that it's easy to ignore. But it's a bit worse than it might seem, because conscientious users aren't ABLE to write safe 2.5 code that will run in 3.0. The problem arises when you need to write code that calls someone else's library but then unconditionally recovers from errors in it. Correct 2.4 syntax for this reads as follows: try: my_result = call_some_library(my_data) except (KeyboardInterrupt, MemoryError, SystemError): raise except: report_error() Correct 3.0 syntax will read like this: try: my_result = call_some_library(my_data) except (KeyboardInterrupt, MemoryError, SystemError): raise except BaseException: report_error() But no syntax will work in BOTH 2.5 and 3.0. The 2.4 syntax is illegal in 3.0, and the 3.0 syntax fails to catch exceptions that do not inherit from BaseException. Such exceptions are deprecated (by documentation, if not by code) so our conscientious programmer will never raise them and the standard library avoids doing so. But "call_some_library()" was written by some less careful developer, and may well contain these atavisims. The only complete solution that comes to mind immediately is for the raising of anything not extending BaseException to raise a PendingDeprecationWarning as well. Then the conscientious developer can feel confident again so long as her unit tests are reasonably exhaustive. If we cannot produce a warning for these, then I'd rather not produce the warning for the use of bare "except:". After all, as it's been pointed out, if the use of bare "except:" is all you are interested in it is quite easy to grep the code to find all uses. -- Michael Chermside From raymond.hettinger at verizon.net Thu Aug 25 15:03:36 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Thu, 25 Aug 2005 09:03:36 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <20050825052844.sby2v04s568sgg00@login.werra.lunarpages.com> Message-ID: <001b01c5a975$68909220$b729cb97@oemcomputer> > > Efforts to improve Py3.0 have spilled > > over into breaking Py2.x code with no compensating benefits. [...] > > We don't have to wreck 2.x in order to make 3.0 better. > > I think you're overstating things a bit here. It's only an overstatement if Guido didn't mean what he said. If bare except clauses are deprecated in 2.x, it WILL affect tons of existing code and invalidate a portion of almost all Python books. > > Remember, the ONLY benefit from the whole PEP is that in 3.0, it will no > > longer be necessary to write "except (KeyError, SystemExit): raise". > > [...] IOW, there's nothing worth inflicting destruction on tons of > > 2.x code. > > And now I *KNOW* you're overstating things. There are LOTS of benefits > to the PEP in 3.0. My own personal favorite is that users can be > guaranteed that all exceptions thrown will share a particular common > ancestor and type. Right, there are a couple of parts of the PEP that were non-controversial from the start and would likely have happened even in the absence of the PEP. My point was that a lot of machinery is being thrown at a tiny problem. To eliminate the need for "except (KeyError, SystemExit): raise", we're rearranging the tree, introducing a new builtin, banning an existing and popular form of an except clause, and introducing a non-trivial deprecation that will affect most users. This is a lot of firepower directed at a somewhat small problem. > But no syntax will work in BOTH 2.5 and 3.0. There's the rub. If you can't write code that will work for both, then there is no reason to force 2.x users to make any changes to their existing code, especially given that they won't see any benefit from the mass edits. > If we cannot produce a warning for these, then I'd > rather not produce the warning for the use of bare "except:". > After all, as it's been pointed out, if the use of bare "except:" > is all you are interested in it is quite easy to grep the code to > find all uses. Bingo. A bare except clause is well known as a consenting adults construct. If Guido feels driven to eliminate it from Py3.0, then that is the way it is. But for 2.x, why introduce unnecessary pain. Of course, if bare except clauses weren't banned for 3.0, then we would have no problem writing code that works on all versions on Python from 2.0 to 3.0, that doen't break existing code, and that doesn't invalidate the text in Python books. IMO, that is a nice situation. Just how badly do you want to kill bare except clauses. I propose that leave them alone, and be happy that in 3.0 we can write "except Exception" and get what we want without any fuss. Raymond From sjoerd at acm.org Thu Aug 25 16:13:40 2005 From: sjoerd at acm.org (Sjoerd Mullender) Date: Thu, 25 Aug 2005 16:13:40 +0200 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <20050825052844.sby2v04s568sgg00@login.werra.lunarpages.com> References: <20050825052844.sby2v04s568sgg00@login.werra.lunarpages.com> Message-ID: <430DD214.2050208@acm.org> Michael Chermside wrote: > Raymond writes: > >>Efforts to improve Py3.0 have spilled >>over into breaking Py2.x code with no compensating benefits. [...] >>We don't have to wreck 2.x in order to make 3.0 better. > > > I think you're overstating things a bit here. There is an important point, though. Recently I read complaints about the lack of backward compatibility in Python on the fedora-list (mailing list for users of Fedora Core). Somebody asked what language he should learn and people answered, don't learn Python because it changes too often in backward incompatible ways. They even suggested using that other P language because that was much more backward compatible. Check out the thread starting at https://www.redhat.com/archives/fedora-list/2005-August/msg01682.html . -- Sjoerd Mullender -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 369 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20050825/e4902759/signature.pgp From gvanrossum at gmail.com Thu Aug 25 17:10:52 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 25 Aug 2005 08:10:52 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <001b01c5a975$68909220$b729cb97@oemcomputer> References: <20050825052844.sby2v04s568sgg00@login.werra.lunarpages.com> <001b01c5a975$68909220$b729cb97@oemcomputer> Message-ID: On 8/25/05, Raymond Hettinger wrote: > It's only an overstatement if Guido didn't mean what he said. If bare > except clauses are deprecated in 2.x, it WILL affect tons of existing > code and invalidate a portion of almost all Python books. Deprecation means your code will still work I hope every book that documents "except:" also adds "but don't use this except under very special circumstances". I think you're overreacting (again), Raymond. 3.0 will be much more successful if we can introduce many of its features into 2.x. Many of those features are in fact improvements of the language even if they break old code. We're trying to balance between breaking old code and introducing new features; deprecation is the accepted way to do this. Regarding the complaint that Python is changing too fast, that really sounds like FUD to me. With a new release every 18 months Python is about as stable as it gets barring dead languages. The PHP is in the throws of the 4->5 conversion which breaks worse than Python 2->3 will (Rasmus ia changing object assignment semantics from copying to sharing). Maybe they should be warned not to learn Perl because Larry is deconstructing it all for Perl 6? :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tlesher at gmail.com Thu Aug 25 17:16:10 2005 From: tlesher at gmail.com (Tim Lesher) Date: Thu, 25 Aug 2005 11:16:10 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <430DD214.2050208@acm.org> References: <20050825052844.sby2v04s568sgg00@login.werra.lunarpages.com> <430DD214.2050208@acm.org> Message-ID: <9613db6005082508162794cd5b@mail.gmail.com> On 8/25/05, Sjoerd Mullender wrote: > There is an important point, though. Recently I read complaints about > the lack of backward compatibility in Python on the fedora-list (mailing > list for users of Fedora Core). Somebody asked what language he should > learn and people answered, don't learn Python because it changes too > often in backward incompatible ways. They even suggested using that > other P language because that was much more backward compatible. I think you're overstating what actually happened there. Here's the actual quote from the thread: : perl is more portable than python - programs written for perl are far : more likely to run on a new version of perl than the equivalent for : python. However, python is probably more readable and writable than perl : for a new user, and is the language most Fedora system utilities (e.g. : yum) are written in. Both perl and python run on Windows too. : : You have to be very careful about how you write your code to make it : portable to both environments. If you need a GUI, you'll need a : cross-platform GUI toolkit like Qt too. : : If it's only one language to learn, and you're a Fedora user, I'd go for : python. Yes, later there were additional posts about portability and backwards-compatibility, but they were for the most part factually incorrect (reliance on new 2.x features, not backwards-incompatibility, were the issue with CML1) and relied to "I heard that..." information So your point is well-taken, but the problem is one of user perception. That's not a dismissal of the problem--witness the "JAVA/LISP/Python is too slow" and "all PERL code is cryptic" memes. To me, this perception problem alone raises the bar on backwards compatibility. Even if obsoleted features are seldom useed, "$language breaks old code!" is a virulent meme, in both senses of the word. -- Tim Lesher From gvanrossum at gmail.com Thu Aug 25 17:17:24 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 25 Aug 2005 08:17:24 -0700 Subject: [Python-Dev] Style for raising exceptions (python-dev Summary for 2005-08-01 through 2005-08-15 [draft]) In-Reply-To: <430D90FF.6060206@egenix.com> References: <430D90FF.6060206@egenix.com> Message-ID: On 8/25/05, M.-A. Lemburg wrote: > I must have missed this one: > > > ---------------------------- > > Style for raising exceptions > > ---------------------------- > > > > Guido explained that these days exceptions should always be raised as:: > > > > raise SomeException("some argument") > > > > instead of:: > > > > raise SomeException, "some argument" > > > > The second will go away in Python 3.0, and is only present now for backwards > > compatibility. (It was necessary when strings could be exceptions, in > > order to pass both the exception "type" and message.) PEPs 8_ and 3000_ > > were accordingly updated. > > AFAIR, the second form was also meant to be able to defer > the instantiation of the exception class until really > needed in order to reduce the overhead related to raising > exceptions in Python. > > However, that optimization never made it into the implementation, > I guess. Something equivalent is used internally in the C code, but that doesn't mean we'll need it in Python code. The optimization only works if the exception is also *caught* in C code, BTW (it is instantiated as soon as it is handled by a Python except clause). Originally, the second syntax was the only available syntax, because all we had were string exceptions. Now that string exceptions are dead (although not yet buried :) I really don't see why we need to keep both versions of the syntax; Python 3.0 will only have one version. (We're still debating what to do with the traceback argument; wanna revive PEP 344?) If you need to raise exceptions fast, pre-instantiate an instance. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From raymond.hettinger at verizon.net Thu Aug 25 17:58:48 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Thu, 25 Aug 2005 11:58:48 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: Message-ID: <000a01c5a98d$e1c20940$b729cb97@oemcomputer> > Deprecation means your code will still work I hope every book that > documents "except:" also adds "but don't use this except under very > special circumstances". > > I think you're overreacting (again), Raymond. 3.0 will be much more > successful if we can introduce many of its features into 2.x. Many of > those features are in fact improvements of the language even if they > break old code. We're trying to balance between breaking old code and > introducing new features; deprecation is the accepted way to do this. IMO, the proponents of 2.x deprecation are underreacting. Deprecation has a cost -- there needs to be a corresponding payoff. Deprecation is warranted if the substitute code would still run on future Pythons (Michael explained the issues here). Deprecation is only warranted if the interim substitute works -- AFAICT, there is no other way to broadly catch exceptions not derived from Exception. The effort is only warranted if it makes the code better -- but here nothing is currently broken and the new code will be much less attractive and less readable (if the changes are done correctly); only 3.0 will offer the tools to do it readably and beautifully. Also, as we learned with apply(), even if ignored, the deprecation machinery has a tremendous runtime cost. None of this will make upgrading to Py2.5 an attractive option. There is a reason that over 120 bare except clauses remain in the standard library despite a number of attempts to get rid of them. It won't be trivial to properly evaluate whether each should be Exception or BaseException; to catch string exceptions; to write the test cases; to follow other PEPs requiring compatibility with older Pythons; or to do this in a way that it won't have to be done again for Py3.0. If the proponents don't have time to fix the standard library, how can they in good conscience mandate change for the rest of the world. Besides, I thought Guido was opposed to efforts to roam through mountains of code, making alterations in a non-holistic way. With a change this complex, the odds of introducing errors are very high. Fredrik, please speak up. Someone should represent the users here. I'm reached my limit on how much time I can devote to thinking out the implications of these proposals. Someone else needs to "overreact". From nas at arctrix.com Thu Aug 25 18:33:03 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Thu, 25 Aug 2005 10:33:03 -0600 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <000a01c5a98d$e1c20940$b729cb97@oemcomputer> References: <000a01c5a98d$e1c20940$b729cb97@oemcomputer> Message-ID: <20050825163302.GA21089@mems-exchange.org> On Thu, Aug 25, 2005 at 11:58:48AM -0400, Raymond Hettinger wrote: > Deprecation is only warranted if the interim substitute works -- > AFAICT, there is no other way to broadly catch exceptions not > derived from Exception. This seems to get to the heart of the problem. I'm no fan of bare excepts but I think we could handle them in 2.x (at least for the next few releases) by providing a workable alternative and then strongly discouraging their use (like we do for "from x import *"). Neil From dieter at handshake.de Wed Aug 24 21:11:18 2005 From: dieter at handshake.de (Dieter Maurer) Date: 24 Aug 2005 21:11:18 +0200 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: References: Message-ID: The following message is a courtesy copy of an article that has been posted to comp.lang.python as well. Neil Schemenauer writes on Mon, 22 Aug 2005 15:31:42 -0600: > ... > Some code may require that str() returns a str instance. In the > standard library, only one such case has been found so far. The > function email.header_decode() requires a str instance and the > email.Header.decode_header() function tries to ensure this by > calling str() on its argument. The code was fixed by changing > the line "header = str(header)" to: > > if isinstance(header, unicode): > header = header.encode('ascii') Note, that this is not equivalent to the old "str(header)": "str(header)" used Python's "default encoding" while the new code uses 'ascii'. The new code might be more correct than the old one has been. > ... > Alternative Solutions > > A new built-in function could be added instead of changing str(). > Doing so would introduce virtually no backwards compatibility > problems. However, since the compatibility problems are expected to > rare, changing str() seems preferable to adding a new built-in. Can we get a new builtin with the exact same behaviour as the current "str" which can be used when we do require an "str" (and cannot use a "unicode"). Dieter From gvwilson at cs.utoronto.ca Wed Aug 24 14:05:23 2005 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Wed, 24 Aug 2005 08:05:23 -0400 (EDT) Subject: [Python-Dev] [Argon] Re: 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430C48F9.8060801@v.loewis.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> Message-ID: Hi Martin (and everyone else); thanks for your mail. The N*N/2 invocations would explain why we saw such a large number of invocations --- thanks for figuring it out. W.r.t. how we're invoking our script: > > But if you're using CGI, you're importing your source on every > > invocation. > > Well, no. Only the CGI script needs to be parsed every time; all modules > could load off bytecode files. > > Which suggests that Keir Mierle doesn't use bytecode files, I think he > should. Yes, mod_python and .pyc's are the obviously way to go --- once the code actually works ;-). I just wanted students to have as few moving parts as possible while debugging. Thanks again, Greg From gvwilson at cs.utoronto.ca Wed Aug 24 20:20:59 2005 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Wed, 24 Aug 2005 14:20:59 -0400 (EDT) Subject: [Python-Dev] [Argon] Re: 51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???) In-Reply-To: <430CB987.5000601@livinglogic.de> References: <20050823201021.GE32195@cs.toronto.edu> <430C41BD.8010602@livinglogic.de> <430C48F9.8060801@v.loewis.de> <430C5E6E.2040405@livinglogic.de> <430C6E7E.7070106@v.loewis.de> <430C7F4C.9010703@livinglogic.de> <430C854E.1080200@v.loewis.de> <430CA75F.7090900@livinglogic.de> <430CB0AE.1040201@v.loewis.de> <430CB987.5000601@livinglogic.de> Message-ID: > > Walter D?rwald wrote: > >>At least it would remove the quadratic number of calls to > >>_PyUnicodeUCS2_IsLinebreak(). For each character it would be called only > >>once. > Martin v. L?wis wrote: > > Correct. However, I very much doubt that this is the cause of the > > slowdown. > Walter D?rwald wrote: > Probably. We'd need a test with the original Argon source to really know. We can do that. > OK, so should we add this for 2.4.2 or only for 2.5? 2.4.2 please ;-) Thanks, Greg From gvanrossum at gmail.com Thu Aug 25 19:01:33 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 25 Aug 2005 10:01:33 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <000a01c5a98d$e1c20940$b729cb97@oemcomputer> References: <000a01c5a98d$e1c20940$b729cb97@oemcomputer> Message-ID: On 8/25/05, Raymond Hettinger wrote: >[...] AFAICT, there is no other way to broadly > catch exceptions not derived from Exception. But there is rarely a need to do so. I bet you that 99 out of 100 bare excepts in the stdlib could be replaced by "except Exception" without breaking anything, since they only expect a wide variety of standard exceptions, and don't care about string exceptions or user exceptions. The exception is the first of the two bare except: clauses in code.py. > The effort is only > warranted if it makes the code better -- but here nothing is currently > broken and the new code will be much less attractive and less readable > (if the changes are done correctly); only 3.0 will offer the tools to do > it readably and beautifully. Please explain? If 9 out of 10 bare excepts can safely be replaced by "except Exception", what's not beautiful about that? > Also, as we learned with apply(), even if > ignored, the deprecation machinery has a tremendous runtime cost. None > of this will make upgrading to Py2.5 an attractive option. Not in this case; bare except: can be flagged by the parser so the warning happens only once per compilation. > There is a reason that over 120 bare except clauses remain in the > standard library despite a number of attempts to get rid of them. I betcha almost all of then can safely be replaced with "except Exception". > Besides, I thought Guido was opposed to efforts to roam through > mountains of code, making alterations in a non-holistic way. This is trumped by the need to keep the standard library warning-free. But how about the following compromise: make it a silent deprecation in 2.5, and a full deprecation in 2.6. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nas at arctrix.com Thu Aug 25 19:03:32 2005 From: nas at arctrix.com (Neil Schemenauer) Date: Thu, 25 Aug 2005 11:03:32 -0600 Subject: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings In-Reply-To: References: Message-ID: <20050825170332.GA21225@mems-exchange.org> On Wed, Aug 24, 2005 at 09:11:18PM +0200, Dieter Maurer wrote: > Neil Schemenauer writes on Mon, 22 Aug 2005 15:31:42 -0600: > > The code was fixed by changing > > the line "header = str(header)" to: > > > > if isinstance(header, unicode): > > header = header.encode('ascii') > > Note, that this is not equivalent to the old "str(header)": > > "str(header)" used Python's "default encoding" while the > new code uses 'ascii'. It also doesn't call __str__ if the object is not a basestring instance. I have a hard time understanding the exact purpose of calling str() here. Maybe Barry can comment. > Can we get a new builtin with the exact same behaviour as > the current "str" which can be used when we do require an "str" > (and cannot use a "unicode"). That fact that no code in the standard library requires such a function (AFAIK), leads me to believe that it would not be useful enough to be made a built-in. You would just write it yourself: def mystr(s): s = str(s) if isinstance(s, unicode): s = s.encode(sys.getdefaultencoding()) return s Cheers, Neil From reinhold-birkenfeld-nospam at wolke7.net Thu Aug 25 19:17:14 2005 From: reinhold-birkenfeld-nospam at wolke7.net (Reinhold Birkenfeld) Date: Thu, 25 Aug 2005 19:17:14 +0200 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <000a01c5a98d$e1c20940$b729cb97@oemcomputer> References: <000a01c5a98d$e1c20940$b729cb97@oemcomputer> Message-ID: Raymond Hettinger wrote: >> Deprecation means your code will still work I hope every book that >> documents "except:" also adds "but don't use this except under very >> special circumstances". >> >> I think you're overreacting (again), Raymond. 3.0 will be much more >> successful if we can introduce many of its features into 2.x. Many of >> those features are in fact improvements of the language even if they >> break old code. We're trying to balance between breaking old code and >> introducing new features; deprecation is the accepted way to do this. > Fredrik, please speak up. Someone should represent the users here. I'm > reached my limit on how much time I can devote to thinking out the > implications of these proposals. Someone else needs to "overreact". Perhaps I may add a pragmatic POV (yes, I know that "pragmatic" is usually attributed to another language ;-). If "except:" issues a deprecation warning in 2.5, many people will come and say "woohoo, Python breaks backwards compatibility" and "I knew it, Python is unreliable, my script issues 1,233 warnings now" and such. You can see this effect looking at the discussion that broke out when Guido announced that map, filter and reduce would vanish (as builtins) in 3.0. People spoke up and said, "if that's going to be the plan, I'll stop using Python" etc. That said, I think that unless it is a new feature (like with statements) transitions to Python 3.0 shouldn't be enforced in the 2.x series. With 3.0, everyone expects a clear cut and a compatibility breach. Reinhold -- Mail address is perfectly valid! From raymond.hettinger at verizon.net Thu Aug 25 19:28:15 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Thu, 25 Aug 2005 13:28:15 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: Message-ID: <000a01c5a99a$61080360$b729cb97@oemcomputer> > > Also, as we learned with apply(), even if > > ignored, the deprecation machinery has a tremendous runtime cost. None > > of this will make upgrading to Py2.5 an attractive option. > > Not in this case; bare except: can be flagged by the parser so the > warning happens only once per compilation. That's good news. It mitigates runtime cost completely. > > There is a reason that over 120 bare except clauses remain in the > > standard library despite a number of attempts to get rid of them. > > I betcha almost all of then can safely be replaced with "except > Exception". Because the tree is not being re-arranged until 3.0, those cases should also introduce a preceding: except (KeyboardInterrupt, SystemExit): raise Anywhere that doesn't apply will need: except BaseException: . . . and also some corresponding backwards compatibility code to work with older pythons. If any are expected to work with user or third-party modules, then they cannot safely ignore string exceptions and exceptions not derived from Exception. Each of those changes needs to be accompanied by test cases so that all code paths get exercised. After the change, we should run Zope, Twisted, Gadfly, etc to make sure no major application got broken. Long running apps should verify that their recover and restart routines haven't been compromised. This is doubly true if the invariant for a bare except was being relied upon as a security measure (this may or may not be a real issue). > But how about the following compromise: make it a silent deprecation > in 2.5, and a full deprecation in 2.6. I'd love to compromise but it's your language. If you're going to deprecate, just do it. Pulling the band-aid off slowly doesn't lessen the total pain. My preference is of course, to leave 2.x alone and make this part of the attraction to 3.0. Remember, none of the code changes buys us anything in 2.x. It is an exercise without payoff. My even stronger preference is to leave bare excepts in for Py3.0. That buys us a happy world where code old code continues to work and new code can be written that functions as intended on all pythons new and old. I'm no fan of bare exceptions, but I'm not inclined to shoot myself in the foot to be rid of them. I wish Fredrik would chime in. He would have something pithy, angry, and incisive to say about this. Raymond From Scott.Daniels at Acm.Org Thu Aug 25 19:30:22 2005 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Thu, 25 Aug 2005 10:30:22 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <000d01c5a958$566e1bc0$b729cb97@oemcomputer> References: <000d01c5a958$566e1bc0$b729cb97@oemcomputer> Message-ID: Raymond Hettinger wrote: >... I propose that the transition plan be as simple as introducing > BaseException. This allows people to write code that will work on both > 2.x and 3.0. It doesn't break anything. > > The guidance for cross-version (2.5 to 3.0) code would be: > > * To catch all but terminating exceptions, write: > > except (KeyError, SystemExit): > raise > except Exception: > ... How about: except BaseException, error: if not isinstance(error, Exception): raise ... This would accommodate other invented exceptions such as "FoundConvergance(BaseException)", which is my pseudo-example for an exiting exception that is not properly a subclass of either KeyError or SystemExit. The idea is a relaxation stops when it doesn't move and may start generating something silly like divide-by-zero. Not the end of an App, but the end of a Phase. --Scott David Daniels Scott.Daniels at Acm.Org From gvanrossum at gmail.com Thu Aug 25 19:43:45 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 25 Aug 2005 10:43:45 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <000a01c5a99a$61080360$b729cb97@oemcomputer> References: <000a01c5a99a$61080360$b729cb97@oemcomputer> Message-ID: On 8/25/05, Raymond Hettinger wrote: > I wish Fredrik would chime in. He would > have something pithy, angry, and incisive to say about this. Raymond, I'm sick of the abuse. Consider the PEP rejected. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mcherm at mcherm.com Thu Aug 25 19:52:19 2005 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 25 Aug 2005 10:52:19 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 Message-ID: <20050825105219.o99mjihmuycgckos@login.werra.lunarpages.com> Guido: > But how about the following compromise: make it a silent deprecation > in 2.5, and a full deprecation in 2.6. Reinhold Birkenfeld: > That said, I think that unless it is a new feature (like with statements) > transitions to Python 3.0 shouldn't be enforced in the 2.x series. With 3.0, > everyone expects a clear cut and a compatibility breach. Raymond: > I'd love to compromise but it's your language. If you're going to > deprecate, just do it. Pulling the band-aid off slowly doesn't lessen > the total pain. There are actually THREE possible levels of deprecation available. In order of severity, they are: 1. Modifying the documentation to advise people to avoid this feature. No one gets alerted. 2. Using a PendingDeprecationWarning so people who explicitly request it can have the compiler alert them when they use it. 3. Using a DeprecationWarning so people using it are alerted unless they explicitly request NOT to be alerted. I think 3 is unwarrented in this case. For reasons I explained in a previous posting, I would be in favor of 2 if we can *also* have a PendingDeprecationWarning for use of string exceptions and arbitrary-object exceptions (those not derived from BaseException). I am in favor of 3 in any case. Of course, that's just one person's opinion... Raymond also raised this excellent point: > There is a reason that over 120 bare except clauses remain in the > standard library despite a number of attempts to get rid of them. [...] > If the proponents don't have time to fix the standard library, how can > they in good conscience mandate change for the rest of the world. That seems like a fair criticism to me. As we've already noted, it is impossible to replace ALL uses of bare "except:" in 2.5 (particularly the From mcherm at mcherm.com Thu Aug 25 19:55:30 2005 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 25 Aug 2005 10:55:30 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 Message-ID: <20050825105530.ev2xwy7r754wogo0@login.werra.lunarpages.com> [PLEASE IGNORE PREVIOUS EMAIL... I HIT [Send] BY MISTAKE] Guido: > But how about the following compromise: make it a silent deprecation > in 2.5, and a full deprecation in 2.6. Reinhold Birkenfeld: > That said, I think that unless it is a new feature (like with statements) > transitions to Python 3.0 shouldn't be enforced in the 2.x series. With 3.0, > everyone expects a clear cut and a compatibility breach. Raymond: > I'd love to compromise but it's your language. If you're going to > deprecate, just do it. Pulling the band-aid off slowly doesn't lessen > the total pain. There are actually THREE possible levels of deprecation available. In order of severity, they are: 1. Modifying the documentation to advise people to avoid this feature. No one gets alerted. 2. Using a PendingDeprecationWarning so people who explicitly request it can have the compiler alert them when they use it. 3. Using a DeprecationWarning so people using it are alerted unless they explicitly request NOT to be alerted. I think 3 is unwarrented in this case. For reasons I explained in a previous posting, I would be in favor of 2 if we can *also* have a PendingDeprecationWarning for use of string exceptions and arbitrary-object exceptions (those not derived from BaseException). I am in favor of 3 in any case. Of course, that's just one person's opinion... Raymond also raised this excellent point: > There is a reason that over 120 bare except clauses remain in the > standard library despite a number of attempts to get rid of them. [...] > If the proponents don't have time to fix the standard library, how can > they in good conscience mandate change for the rest of the world. That seems like a fair criticism to me. As we've already noted, it is impossible to replace ALL uses of bare "except:" in 2.5 (particularly the use in code.py that Guido referred to). But we ought to make an extra effort to remove unnecessary uses of bare "except:" from the standard library if we intend to deprecate it. -- Michael Chermisde From steve at holdenweb.com Thu Aug 25 20:30:53 2005 From: steve at holdenweb.com (Steve Holden) Date: Thu, 25 Aug 2005 14:30:53 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <000a01c5a99a$61080360$b729cb97@oemcomputer> Message-ID: <430E0E5D.8010503@holdenweb.com> Guido van Rossum wrote: > On 8/25/05, Raymond Hettinger wrote: > > >>I wish Fredrik would chime in. He would >>have something pithy, angry, and incisive to say about this. > > > Raymond, I'm sick of the abuse. Consider the PEP rejected. > Perhaps you should go for the ?10 argument next door? regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ From steve at holdenweb.com Thu Aug 25 20:30:53 2005 From: steve at holdenweb.com (Steve Holden) Date: Thu, 25 Aug 2005 14:30:53 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <000a01c5a99a$61080360$b729cb97@oemcomputer> Message-ID: <430E0E5D.8010503@holdenweb.com> Guido van Rossum wrote: > On 8/25/05, Raymond Hettinger wrote: > > >>I wish Fredrik would chime in. He would >>have something pithy, angry, and incisive to say about this. > > > Raymond, I'm sick of the abuse. Consider the PEP rejected. > Perhaps you should go for the ?10 argument next door? regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ From rrr at ronadam.com Thu Aug 25 20:33:35 2005 From: rrr at ronadam.com (Ron Adam) Date: Thu, 25 Aug 2005 14:33:35 -0400 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <000a01c5a98d$e1c20940$b729cb97@oemcomputer> References: <000a01c5a98d$e1c20940$b729cb97@oemcomputer> Message-ID: <430E0EFF.1080707@ronadam.com> Raymond Hettinger wrote: >>Deprecation means your code will still work I hope every book that >>documents "except:" also adds "but don't use this except under very >>special circumstances". >> >>I think you're overreacting (again), Raymond. 3.0 will be much more >>successful if we can introduce many of its features into 2.x. Many of >>those features are in fact improvements of the language even if they >>break old code. We're trying to balance between breaking old code and >>introducing new features; deprecation is the accepted way to do this. > Fredrik, please speak up. Someone should represent the users here. I'm > reached my limit on how much time I can devote to thinking out the > implications of these proposals. Someone else needs to "overreact". How about a middle of the road (or there abouts) opinion from an average user? Just my 2 cents anyways. I get the impression that just how much existing code will work or not work in 3.0 is still fairly up in the air. Python 3.0 still quite a ways off from what I understand. So to me.. depreciating anything at this time that's not going to be removed *before* Python 3.0 is possibly jumping the gun a bit. (IMHO) It definitely makes since to depreciate anything that will be removed prior to Python 3.0. And to also document anything that will be changed in 3.0. (but not depreciate yet) If/when it is decided (maybe it already has) that a smooth transition can be made between 2.x and 3.0 with a high degree of backwards compatibility, then depreciating 2.x features that will be removed from 3.0 makes since at some point but maybe not in 2.5. If it turns out that the amount of changes in 3.0 are such as to be a "New but non backwards compatible version of Python" with a lot of really great new features. Then depreciating items in 2.x that will not be removed from 2.x seems like it gives a since of false hope. It might be better to just document the differences (but not depreciate them) and make a clean break. Or to put it another way... having a lot of depreciated items in the final 2.x version may give a message 2.x is flawed, yet it may not be possible for many programs to move to 3.0 easily for some time if there are a lot of large changes. My opinion is... I would rather see the final version of 2.x not have any depreciated items and efforts be made to make it the best and most dependable 2.x version that will be around for a while. And then have Python 3.0 be a new beginning and an open book without the backwards compatible chains holding it back. That dosen't mean it won't be, I think it's just too soon to tell to what degree. At this time the efforts towards 3.0 seem to be towards those improvements that may be included in some future version of 2.x which is great. Is it possible the big changes have yet to be considered for Python 3.0? Cheers, Ron From ianb at colorstudy.com Thu Aug 25 21:10:43 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 25 Aug 2005 14:10:43 -0500 Subject: [Python-Dev] PEP 342: simple example, closure alternative Message-ID: <430E17B3.3080900@colorstudy.com> I was trying to translate a pattern that uses closures in a language like Scheme (where closed values can be written to) to generators using PEP 342, but I'm not clear exactly how it works; the examples in the PEP have different motivations. Since I can't actually run these examples, perhaps someone could confirm or debug these: A closure based accumulator (using Scheme): (define (accum n) (lambda (incr) (set! n (+ n incr)) n)) (define s (accum 0)) (s 1) ; -> 1 == 0+1 (s 5) ; -> 6 == 1+5 So I thought the generator version might look like: def accum(n): while 1: incr = (yield n) or 0 n += incr >>> s = accum(0) >>> s.next() >>> s.send(1) 0 >>> s.send(5) 1 >>> s.send(1) 6 Is the order of the output correct? Is there a better way to write accum, that makes it feel more like the closure-based version? Is this for loop correct? >>> s = accum(0) >>> for i in s: ... if i >= 10: break ... print i, ... assert s.send(2) == i 0 2 4 6 8 Hmm... maybe this would make it feel more closure-like: def closure_like(func): def replacement(*args, **kw): return ClosureLike(func(*args, **kw)) return replacement class ClosureLike(object): def __init__(self, iterator): self.iterator = iterator # I think this initial .next() is required, but I'm # very confused on this point: assert self.iterator.next() is None def __call__(self, input): assert self.iterator.send(input) is None return self.iterator.next() @closure_like def accum(n): while 1: # yields should always be in pairs, the first yield is input # and the second yield is output. incr = (yield) # this line is equivalent to (lambda (incr)... n += incr # equivalent to (set! ...) yield n # equivalent to n; this yield always returns None >>> s = accum(0) >>> s(1) 1 >>> s(5) 6 Everything before the first (yield) is equivalent to the closed values between "(define (accum n)" and "(lambda" (for this example there's nothing there; I guess a more interesting example would have closed variables that were written to that were not function parameters). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Aug 25 21:23:10 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 25 Aug 2005 15:23:10 -0400 Subject: [Python-Dev] PEP 342: simple example, closure alternative In-Reply-To: <430E17B3.3080900@colorstudy.com> Message-ID: <5.1.1.6.0.20050825151456.028cf830@mail.telecommunity.com> At 02:10 PM 8/25/2005 -0500, Ian Bicking wrote: >I was trying to translate a pattern that uses closures in a language >like Scheme (where closed values can be written to) to generators using >PEP 342, but I'm not clear exactly how it works; the examples in the PEP >have different motivations. Since I can't actually run these examples, >perhaps someone could confirm or debug these: > >A closure based accumulator (using Scheme): > >(define (accum n) > (lambda (incr) > (set! n (+ n incr)) > n)) >(define s (accum 0)) >(s 1) ; -> 1 == 0+1 >(s 5) ; -> 6 == 1+5 > >So I thought the generator version might look like: > >def accum(n): > while 1: > incr = (yield n) or 0 > n += incr > > >>> s = accum(0) > >>> s.next() The initial next() will yield 0, not None. > >>> s.send(1) >0 1 > >>> s.send(5) >1 6 > >>> s.send(1) >6 7 >Is the order of the output correct? Is there a better way to write >accum, that makes it feel more like the closure-based version? > >Is this for loop correct? > > >>> s = accum(0) > >>> for i in s: >... if i >= 10: break >... print i, >... assert s.send(2) == i >0 2 4 6 8 The assert will fail on the first pass. s.send(2) will == i+2, e.g.: >>> s = accum(0) >>> for i in s: ... if i>=10: break ... print i, ... assert s.send(2) == i+2 ... 0 2 4 6 8 From ianb at colorstudy.com Thu Aug 25 22:12:35 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 25 Aug 2005 15:12:35 -0500 Subject: [Python-Dev] PEP 342: simple example, closure alternative In-Reply-To: <5.1.1.6.0.20050825151456.028cf830@mail.telecommunity.com> References: <5.1.1.6.0.20050825151456.028cf830@mail.telecommunity.com> Message-ID: <430E2633.1020902@colorstudy.com> Phillip J. Eby wrote: > At 02:10 PM 8/25/2005 -0500, Ian Bicking wrote: > >>I was trying to translate a pattern that uses closures in a language >>like Scheme (where closed values can be written to) to generators using >>PEP 342, but I'm not clear exactly how it works; the examples in the PEP >>have different motivations. Since I can't actually run these examples, >>perhaps someone could confirm or debug these: >> >>A closure based accumulator (using Scheme): >> >>(define (accum n) >> (lambda (incr) >> (set! n (+ n incr)) >> n)) >>(define s (accum 0)) >>(s 1) ; -> 1 == 0+1 >>(s 5) ; -> 6 == 1+5 >> >>So I thought the generator version might look like: >> >>def accum(n): >> while 1: >> incr = (yield n) or 0 >> n += incr Bah, I don't know why this had me so confused. Well, I kind of know why. So maybe this example would be better written: def accum(n): incr = yield # wait to get the first incr to be sent in while 1: n += incr incr = yield n # return the new value, wait for next incr This way it is more explicit all around -- the first call to .next() is just setup, kind of like __init__ in an object, except it has to be explicitly invoked. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From skip at pobox.com Thu Aug 25 22:23:51 2005 From: skip at pobox.com (skip@pobox.com) Date: Thu, 25 Aug 2005 15:23:51 -0500 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> <5BB88A76-FB4E-41F7-B82D-4B7C5B6D28DD@fuhm.net> Message-ID: <17166.10455.121002.213724@montanaro.dyndns.org> Guido> It's never too early to start deprecating a feature we know will Guido> disappear in 3.0. Though if it's a widely used feature the troops will be highly annoyed by all the deprecation warnings. (Or does deprecation not coincide with emitting warnings?) Skip From skip at pobox.com Thu Aug 25 22:45:00 2005 From: skip at pobox.com (skip@pobox.com) Date: Thu, 25 Aug 2005 15:45:00 -0500 Subject: [Python-Dev] Style for raising exceptions (python-dev Summary for 2005-08-01 through 2005-08-15 [draft]) In-Reply-To: <430D90FF.6060206@egenix.com> References: <430D90FF.6060206@egenix.com> Message-ID: <17166.11724.133034.374929@montanaro.dyndns.org> MAL> I must have missed this one: That's because it was brief and to the point, so the discussion lasted for maybe three messages. Also, someone told us you were on holiday so we thought we could squeak it through without you noticing. Darn those Aussies. Late on the pydev summary again! >> ---------------------------- >> Style for raising exceptions >> ---------------------------- >> >> Guido explained that these days exceptions should always be raised as:: >> >> raise SomeException("some argument") >> >> instead of:: >> >> raise SomeException, "some argument" >> >> The second will go away in Python 3.0, and is only present now for >> backwards compatibility. (It was necessary when strings could be >> exceptions, in order to pass both the exception "type" and message.) >> PEPs 8_ and 3000_ were accordingly updated. I do have a followup question on the style thing. (I'll leave others to answer MAL's question about optimization.) If I want to raise an exception without an argument, which of the following is the proper form? raise ValueError raise ValueError() Skip From gvanrossum at gmail.com Fri Aug 26 01:59:31 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 25 Aug 2005 16:59:31 -0700 Subject: [Python-Dev] Style for raising exceptions (python-dev Summary for 2005-08-01 through 2005-08-15 [draft]) In-Reply-To: <17166.11724.133034.374929@montanaro.dyndns.org> References: <430D90FF.6060206@egenix.com> <17166.11724.133034.374929@montanaro.dyndns.org> Message-ID: On 8/25/05, skip at pobox.com wrote: > I do have a followup question on the style thing. (I'll leave others to > answer MAL's question about optimization.) If I want to raise an exception > without an argument, which of the following is the proper form? > > raise ValueError > raise ValueError() The latter. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvanrossum at gmail.com Fri Aug 26 02:04:44 2005 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 25 Aug 2005 17:04:44 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: <17166.10455.121002.213724@montanaro.dyndns.org> References: <20050824060856.hex8yt68qc0c0g00@login.werra.lunarpages.com> <5BB88A76-FB4E-41F7-B82D-4B7C5B6D28DD@fuhm.net> <17166.10455.121002.213724@montanaro.dyndns.org> Message-ID: On 8/25/05, skip at pobox.com wrote: > > Guido> It's never too early to start deprecating a feature we know will > Guido> disappear in 3.0. > > Though if it's a widely used feature the troops will be highly annoyed by > all the deprecation warnings. (Or does deprecation not coincide with > emitting warnings?) See Michael Chermside's post. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ark at acm.org Fri Aug 26 05:18:43 2005 From: ark at acm.org (Andrew Koenig) Date: Thu, 25 Aug 2005 23:18:43 -0400 Subject: [Python-Dev] PEP 342: simple example, closure alternative In-Reply-To: <430E17B3.3080900@colorstudy.com> Message-ID: <006a01c5a9ec$e0a2f880$6402a8c0@arkdesktop> > A closure based accumulator (using Scheme): > > (define (accum n) > (lambda (incr) > (set! n (+ n incr)) > n)) > (define s (accum 0)) > (s 1) ; -> 1 == 0+1 > (s 5) ; -> 6 == 1+5 > > So I thought the generator version might look like: > > def accum(n): > while 1: > incr = (yield n) or 0 > n += incr Maybe I'm missing something but this example seems needlessly tricky to me. How about doing it this way? def accum(n): acc = [n] def f(incr): acc[0] += incr return acc[0] return f Here, the [0] turns "read-only" access into write access to a list element. The list itself isn't written; only its element is. From ianb at colorstudy.com Fri Aug 26 06:59:42 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 25 Aug 2005 23:59:42 -0500 Subject: [Python-Dev] PEP 342: simple example, closure alternative In-Reply-To: <006a01c5a9ec$e0a2f880$6402a8c0@arkdesktop> References: <006a01c5a9ec$e0a2f880$6402a8c0@arkdesktop> Message-ID: <430EA1BE.9090804@colorstudy.com> Andrew Koenig wrote: >>A closure based accumulator (using Scheme): >> >>(define (accum n) >> (lambda (incr) >> (set! n (+ n incr)) >> n)) >>(define s (accum 0)) >>(s 1) ; -> 1 == 0+1 >>(s 5) ; -> 6 == 1+5 >> >>So I thought the generator version might look like: >> >>def accum(n): >> while 1: >> incr = (yield n) or 0 >> n += incr > > > Maybe I'm missing something but this example seems needlessly tricky to me. > How about doing it this way? > > def accum(n): > acc = [n] > def f(incr): > acc[0] += incr > return acc[0] > return f > > Here, the [0] turns "read-only" access into write access to a list element. > The list itself isn't written; only its element is. I was just exploring how it could be done with coroutines. But also because using lists as pointers isn't that elegant, and isn't something I'd encourage people do to coming from other languages (where closures are used more heavily). More generally, I've been doing some language comparisons, and I don't like literal but non-idiomatic translations of programming patterns. So I'm considering better ways to translate some of the same use cases. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From bcannon at gmail.com Fri Aug 26 08:01:33 2005 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 25 Aug 2005 23:01:33 -0700 Subject: [Python-Dev] Bare except clauses in PEP 348 In-Reply-To: References: <000a01c5a99a$61080360$b729cb97@oemcomputer> Message-ID: The PEP has been rejected. -Brett On 8/25/05, Guido van Rossum wrote: > On 8/25/05, Raymond Hettinger wrote: > > > I wish Fredrik would chime in. He would > > have something pithy, angry, and incisive to say about this. > > Raymond, I'm sick of the abuse. Consider the PEP rejected. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From mal at egenix.com Fri Aug 26 10:12:12 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 26 Aug 2005 10:12:12 +0200 Subject: [Python-Dev] Style for raising exceptions (python-dev Summary for 2005-08-01 through 2005-08-15 [draft]) In-Reply-To: References: <430D90FF.6060206@egenix.com> Message-ID: <430ECEDC.7040206@egenix.com> Guido van Rossum wrote: > On 8/25/05, M.-A. Lemburg wrote: > >>I must have missed this one: >> >> >>>---------------------------- >>>Style for raising exceptions >>>---------------------------- >>> >>>Guido explained that these days exceptions should always be raised as:: >>> >>> raise SomeException("some argument") >>> >>>instead of:: >>> >>> raise SomeException, "some argument" >>> >>>The second will go away in Python 3.0, and is only present now for backwards >>>compatibility. (It was necessary when strings could be exceptions, in >>>order to pass both the exception "type" and message.) PEPs 8_ and 3000_ >>>were accordingly updated. >> >>AFAIR, the second form was also meant to be able to defer >>the instantiation of the exception class until really >>needed in order to reduce the overhead related to raising >>exceptions in Python. >> >>However, that optimization never made it into the implementation, >>I guess. > > > Something equivalent is used internally in the C code, but that > doesn't mean we'll need it in Python code. The optimization only works > if the exception is also *caught* in C code, BTW (it is instantiated > as soon as it is handled by a Python except clause). Ah, I knew it was in there somewhere (just couldn't find yesterday when I was looking for the optimization :-). > Originally, the second syntax was the only available syntax, because > all we had were string exceptions. Now that string exceptions are dead > (although not yet buried :) I really don't see why we need to keep > both versions of the syntax; Python 3.0 will only have one version. Actually, we do only have one version: the first syntax is just a special case of the second (with the value argument set to None). I don't see a need for two or more syntaxes either, but most code nowadays uses the second variant (I don't know of any code that uses the traceback argument), which puts up a high barrier for changes. This is from a comment in ceval.c: /* We support the following forms of raise: raise , raise , raise , None raise , raise , None raise ,RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4