A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from http://mail.python.org/pipermail/python-dev/2001-October.txt below:

Using slices with __getitem__ doesn't work with (this returns the bugs database. see http://sourceforge.net/tracker/?group_id=5470 for atid's for other parts of the tracker) doing the same through urllib doesn't appear work, though. (no time to dig further into this right now) From aahz@rahul.net Tue Oct 23 16:35:04 2001 From: aahz@rahul.net (Aahz Maruch) Date: Tue, 23 Oct 2001 08:35:04 -0700 (PDT) Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato In-Reply-To: <003b01c15bd8$6cffede0$ced241d5@hagrid> from "Fredrik Lundh" at Oct 23, 2001 05:35:44 PM Message-ID: <20011023153504.D66C6E8C5@waltz.rahul.net> Fredrik Lundh wrote: > > works fine from MSIE, using this URL: > [....] > doing the same through urllib doesn't appear work, though. Well, that's a good reason for leaving SF. :-/ -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From loewis@informatik.hu-berlin.de Tue Oct 23 18:25:30 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Tue, 23 Oct 2001 19:25:30 +0200 (MEST) Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato Message-ID: <200110231725.TAA11514@paros.informatik.hu-berlin.de> > doing the same through urllib doesn't appear work, though. Nor does it work with Konqueror or Netscape, see http://sourceforge.net/tracker/?func=detail&aid=469987&group_id=1&atid=100001 It's good to know it works with IE, though. Regards, Martin From mal@lemburg.com Tue Oct 23 19:28:55 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 23 Oct 2001 20:28:55 +0200 Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato References: <20011023091126.A24293@mems-exchange.org> <15317.30842.636138.922307@anthem.wooz.org> <003b01c15bd8$6cffede0$ced241d5@hagrid> Message-ID: <3BD5B6E7.CF511C1@lemburg.com> Fredrik Lundh wrote: > > barry wrote: > > > akuchlin> There's a discussion of SourceForge alternatives at > > akuchlin> http://www.advogato.org/article/357.html, and in a > > akuchlin> comment someone mentions that there's an XML export of > > akuchlin> the data in the tracker. Unfortunately I can't get it > > akuchlin> to work following his instructions, but perhaps someone > > akuchlin> else can experiment with it more than I can. > > > > I can't get it to work either, but if someone comes up with the right > > recipe, I can add it to our backup script. > > works fine from MSIE, using this URL: > > url = ( > "http://sourceforge.net/export/sf_tracker_export.php" + > "?atid=105470&group_id=5470" > ) > > returns a 1666445 byte large document (with no content > type), starting with: > ... > (this returns the bugs database. see > http://sourceforge.net/tracker/?group_id=5470 > for atid's for other parts of the tracker) > > doing the same through urllib doesn't appear work, though. > > (no time to dig further into this right now) FYI, It seems that you have to be logged into SF in order for the download to work. Netscape does the download too if you create a web page and then use the right-click-download feature. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From akuchlin@mems-exchange.org Tue Oct 23 21:13:29 2001 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: 23 Oct 2001 20:13:29 -0000 Subject: [Python-Dev] Dictionary subclasses and exec Message-ID: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> The following bit of code defines a dictionary that folds all its keys to lowercase before inserting them, but 'exec "Name = 1" in LowerCaseDict()' doesn't seem to call my __getitem__(), as the listed keys include 'Name'. Should this be expected to work? --amk class LowerCaseDict(dictionary): def _fold_key (self, key): if not isinstance(key, str): raise TypeError, "All keys must be strings" return key.lower() def __getitem__ (self, key): key = self._fold_key(key) return dictionary.__getitem__(self, key) def __setitem__ (self, key, value): key = self._fold_key(key) dictionary.__setitem__(self, key, value) def __delitem__ (self, key): key = self._fold_key(key) dictionary.__delitem__(self, key, value) d = LowerCaseDict() exec 'Name = 1' in d print d.keys() From guido@python.org Tue Oct 23 22:08:15 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 23 Oct 2001 17:08:15 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: Your message of "23 Oct 2001 20:13:29 -0000." <20011023201329.27143.qmail@mozart.cnri.reston.va.us> References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> Message-ID: <200110232108.f9NL8Fi25565@odiug.zope.com> > The following bit of code defines a dictionary that folds all its keys > to lowercase before inserting them, but 'exec "Name = 1" in > LowerCaseDict()' doesn't seem to call my __getitem__(), as the listed > keys include 'Name'. Should this be expected to work? > > --amk > > class LowerCaseDict(dictionary): > def _fold_key (self, key): > if not isinstance(key, str): > raise TypeError, "All keys must be strings" > return key.lower() > > def __getitem__ (self, key): > key = self._fold_key(key) > return dictionary.__getitem__(self, key) > > def __setitem__ (self, key, value): > key = self._fold_key(key) > dictionary.__setitem__(self, key, value) > > def __delitem__ (self, key): > key = self._fold_key(key) > dictionary.__delitem__(self, key, value) > > d = LowerCaseDict() > exec 'Name = 1' in d > print d.keys() Alas, this is one of the things that don't work yet. To set and get local variables, exec uses lower-level APIs (PyDict_SetItem and PyDict_GetItem) that you can't override. I've thought about what it would take to make this work as desired, but I haven't found a way yet that wouldn't (a) slow down the normal case, or (b) create subtle reference count bugs. --Guido van Rossum (home page: http://www.python.org/~guido/) From akuchlin@mems-exchange.org Tue Oct 23 23:17:37 2001 From: akuchlin@mems-exchange.org (akuchlin@mems-exchange.org) Date: Tue, 23 Oct 2001 18:17:37 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: <200110232108.f9NL8Fi25565@odiug.zope.com>; from guido@python.org on Tue, Oct 23, 2001 at 05:08:15PM -0400 References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> Message-ID: <20011023181737.B27568@mems-exchange.org> On Tue, Oct 23, 2001 at 05:08:15PM -0400, Guido van Rossum wrote: >Alas, this is one of the things that don't work yet. To set and get Oh, well; I need to come up with a different example for subclassing a built-in type, then. Thanks! --amk From guido@python.org Wed Oct 24 00:58:27 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 23 Oct 2001 19:58:27 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: Your message of "Tue, 23 Oct 2001 18:17:37 EDT." <20011023181737.B27568@mems-exchange.org> References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> <20011023181737.B27568@mems-exchange.org> Message-ID: <200110232358.TAA06401@cj20424-a.reston1.va.home.com> > >Alas, this is one of the things that don't work yet. To set and get > > Oh, well; I need to come up with a different example for subclassing a > built-in type, then. Thanks! You can use a similar example but not override __getitem__ / __setitem__; instead, add some new methods (e.g. merge() -- like update() but the existing key wins). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 24 01:00:44 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 23 Oct 2001 20:00:44 -0400 Subject: [Python-Dev] patchlevel.h In-Reply-To: Your message of "Tue, 23 Oct 2001 15:21:00 PDT." References: Message-ID: <200110240000.UAA06439@cj20424-a.reston1.va.home.com> Jack, you can't do this on the trunk. If you *have* to have a different patch level, please use a branch! --Guido van Rossum (home page: http://www.python.org/~guido/) > Update of /cvsroot/python/python/dist/src/Include > In directory usw-pr-cvs1:/tmp/cvs-serv16428/Python/Include > > Modified Files: > patchlevel.h > Log Message: > Tweaks for MacPython 2.2b1 > > Index: patchlevel.h > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Include/patchlevel.h,v > retrieving revision 2.56 > retrieving revision 2.57 > diff -C2 -d -r2.56 -r2.57 > *** patchlevel.h 2001/10/19 17:11:58 2.56 > --- patchlevel.h 2001/10/23 22:20:58 2.57 > *************** > *** 27,31 **** > > /* Version as a string */ > ! #define PY_VERSION "2.2b1+" > > /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. > --- 27,31 ---- > > /* Version as a string */ > ! #define PY_VERSION "2.2b1" > > /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. > > > _______________________________________________ > Python-checkins mailing list > Python-checkins@python.org > http://mail.python.org/mailman/listinfo/python-checkins From guido@python.org Wed Oct 24 02:23:42 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 23 Oct 2001 21:23:42 -0400 Subject: [Python-Dev] Fixing send() Message-ID: <200110240123.VAA06720@cj20424-a.reston1.va.home.com> I've got a possible patch for the send() problem. I'd like review of the patch and the related issues before I apply it. I'd also like more guidance regarding the suitability of this patch for the 2.1.2 bugfix release. (It fixes a real bug, but can sometime lead to different behavior, and adds a new interpretation of the absence of an optional argument, which can be considered a feature.) http://sf.net/tracker/?group_id=5470&atid=305470&func=detail&aid=474307 --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Wed Oct 24 02:30:27 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 23 Oct 2001 21:30:27 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: <20011023181737.B27568@mems-exchange.org> References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> <20011023181737.B27568@mems-exchange.org> Message-ID: <15318.6579.803083.815943@grendel.zope.com> akuchlin@mems-exchange.org writes: > Oh, well; I need to come up with a different example for subclassing a > built-in type, then. Thanks! Andrew, You can take a look at that implementation of xml.dom.minidom.NodeList; for Python 2.2, the implementation goes like this: ---------------------------------------------------------------------- class NodeList(list): def item(self, index): if 0 <= index < len(self): return self[index] length = property(lambda self: len(self), doc="The number of nodes in the NodeList.") ---------------------------------------------------------------------- -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From akuchlin@mems-exchange.org Wed Oct 24 02:42:42 2001 From: akuchlin@mems-exchange.org (akuchlin@mems-exchange.org) Date: Tue, 23 Oct 2001 21:42:42 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: <15318.6579.803083.815943@grendel.zope.com>; from fdrake@acm.org on Tue, Oct 23, 2001 at 09:30:27PM -0400 References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> <20011023181737.B27568@mems-exchange.org> <15318.6579.803083.815943@grendel.zope.com> Message-ID: <20011023214242.A28096@mems-exchange.org> On Tue, Oct 23, 2001 at 09:30:27PM -0400, Fred L. Drake, Jr. wrote: > You can take a look at that implementation of >xml.dom.minidom.NodeList; for Python 2.2, the implementation goes like >this: Thanks; maybe something like that will do, and it sparks a stray thought: would it be worth making the file wrapper class in posixfile.py a subclass of file? --amk From fdrake@acm.org Wed Oct 24 02:57:51 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 23 Oct 2001 21:57:51 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: <20011023214242.A28096@mems-exchange.org> References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> <20011023181737.B27568@mems-exchange.org> <15318.6579.803083.815943@grendel.zope.com> <20011023214242.A28096@mems-exchange.org> Message-ID: <15318.8223.646037.669260@grendel.zope.com> akuchlin@mems-exchange.org writes: > Thanks; maybe something like that will do, and it sparks a stray > thought: would it be worth making the file wrapper class in posixfile.py > a subclass of file? Given that the documentation for posixfile delares it to be (essentially) obsolete, I thnk the answer is "no". Code that is obsolete should not be touched as long as it continues to pass any relevant tests and there are no reported bugs. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From Anthony Baxter Wed Oct 24 03:23:30 2001 From: Anthony Baxter (Anthony Baxter) Date: Wed, 24 Oct 2001 12:23:30 +1000 Subject: [Python-Dev] Fixing send() In-Reply-To: Message from Guido van Rossum of "Tue, 23 Oct 2001 21:23:42 -0400." <200110240123.VAA06720@cj20424-a.reston1.va.home.com> Message-ID: <200110240223.f9O2NUg29502@mbuna.arbhome.com.au> >>> Guido van Rossum wrote > I've got a possible patch for the send() problem. I'd like review of > the patch and the related issues before I apply it. I'd also like > more guidance regarding the suitability of this patch for the 2.1.2 > bugfix release. (It fixes a real bug, but can sometime lead to > different behavior, and adds a new interpretation of the absence of an > optional argument, which can be considered a feature.) It's a tough decision, because this really _is_ the cleaner solution, but it does stand a slight chance of breaking code, and it's also something that can be worked around in the std library. At the moment I'm planning to just fix the std library modules instead - as these are definitely broken. The added functionality of python's send() is very useful, but it's a feature, not a bug :) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From jack@oratrix.nl Wed Oct 24 10:25:46 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 24 Oct 2001 11:25:46 +0200 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by Guido van Rossum , Tue, 23 Oct 2001 20:00:44 -0400 , <200110240000.UAA06439@cj20424-a.reston1.va.home.com> Message-ID: <20011024092547.54106303181@snelboot.oratrix.nl> > Jack, you can't do this on the trunk. If you *have* to have a > different patch level, please use a branch! Well, apparently I _can_, I just did. Maybe I _shouldn't_.... :-) I undid it this morning, this mod wasn't supposed to escape from my machine. Maybe I should start using the release branch for Mac releases after you are done with it for the Unix/Windows release. As there is an r22b1 tag to point to the exact revision used for the release there should be no adverse consequences. What do you think? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From fdrake@acm.org Wed Oct 24 12:02:10 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 24 Oct 2001 07:02:10 -0400 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: <20011024092547.54106303181@snelboot.oratrix.nl> References: <200110240000.UAA06439@cj20424-a.reston1.va.home.com> <20011024092547.54106303181@snelboot.oratrix.nl> Message-ID: <15318.40882.444530.755092@grendel.zope.com> Jack Jansen writes: > Maybe I should start using the release branch for Mac releases > after you are done with it for the Unix/Windows release. As there > is an r22b1 tag to point to the exact revision used for the release > there should be no adverse consequences. What do you think? That sounds like the right thing to me. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido@python.org Wed Oct 24 13:50:08 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 24 Oct 2001 08:50:08 -0400 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Your message of "Wed, 24 Oct 2001 11:25:46 +0200." <20011024092547.54106303181@snelboot.oratrix.nl> References: <20011024092547.54106303181@snelboot.oratrix.nl> Message-ID: <200110241250.IAA09678@cj20424-a.reston1.va.home.com> > Well, apparently I _can_, I just did. Maybe I _shouldn't_.... :-) > > I undid it this morning, this mod wasn't supposed to escape from my machine. Thanks. :-) > Maybe I should start using the release branch for Mac releases after you are > done with it for the Unix/Windows release. As there is an r22b1 tag to point > to the exact revision used for the release there should be no adverse > consequences. What do you think? Fine with me. For you, it might mean more merging of changes though (unless you start working on the Mac port a few days *before* the next beta release :-). --Guido van Rossum (home page: http://www.python.org/~guido/) From Anthony Baxter Wed Oct 24 14:09:59 2001 From: Anthony Baxter (Anthony Baxter) Date: Wed, 24 Oct 2001 23:09:59 +1000 Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato In-Reply-To: Message from akuchlin@mems-exchange.org of "Tue, 23 Oct 2001 09:11:26 -0400." <20011023091126.A24293@mems-exchange.org> Message-ID: <200110241309.f9OD9xv32763@mbuna.arbhome.com.au> It's worth re-visiting the advogato page again - there's a post from Pat McGovern at SF there now. >>> akuchlin@mems-exchange.org wrote > There's a discussion of SourceForge alternatives at > http://www.advogato.org/article/357.html, and in a comment someone > mentions that there's an XML export of the data in the tracker. > Unfortunately I can't get it to work following his instructions, but > perhaps someone else can experiment with it more than I can. > > --amk > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > -- Anthony Baxter It's never too late to have a happy childhood. From barry@zope.com Wed Oct 24 15:10:56 2001 From: barry@zope.com (Barry A. Warsaw) Date: Wed, 24 Oct 2001 10:10:56 -0400 Subject: [Python-Dev] Re: patchlevel.h References: <200110240000.UAA06439@cj20424-a.reston1.va.home.com> <20011024092547.54106303181@snelboot.oratrix.nl> Message-ID: <15318.52208.962415.759362@anthem.wooz.org> >>>>> "JJ" == Jack Jansen writes: JJ> I undid it this morning, this mod wasn't supposed to escape JJ> from my machine. Maybe I should start using the release JJ> branch for Mac releases after you are done with it for the JJ> Unix/Windows release. As there is an r22b1 tag to point to the JJ> exact revision used for the release there should be no adverse JJ> consequences. What do you think? Should be fine, although I'd like to add something about it to PEP 101, just to make it official. How about in the "What Next?" section, something like: "Turn ownership of the branch tag over to Jack so he can make the Mac release. He'll typically twiddle with patchlevel.h and may make a few other checkins as needed for the Mac. When he's done, he'll add another tag to the branch: X.YaZ-mac. " That way we'll have the source tarball tag in the branch and the mac release in the branch and we could recreate either. -Barry From jack@oratrix.nl Wed Oct 24 15:16:33 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 24 Oct 2001 16:16:33 +0200 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by barry@zope.com (Barry A. Warsaw) , Wed, 24 Oct 2001 10:10:56 -0400 , <15318.52208.962415.759362@anthem.wooz.org> Message-ID: <20011024141633.96B6F303181@snelboot.oratrix.nl> > Should be fine, although I'd like to add something about it to PEP > 101, just to make it official. How about in the "What Next?" section, > something like: > > "Turn ownership of the branch tag over to Jack so he can make the Mac > release. He'll typically twiddle with patchlevel.h and may make a few > other checkins as needed for the Mac. > > When he's done, he'll add another tag to the branch: X.YaZ-mac. > " Fine with me, except for the middle sentence, which should go: one of the reasons to go on the branch is that I *don't* have to twiddle patchlevel.h:-) -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From barry@zope.com Wed Oct 24 15:22:52 2001 From: barry@zope.com (Barry A. Warsaw) Date: Wed, 24 Oct 2001 10:22:52 -0400 Subject: [Python-Dev] Re: patchlevel.h References: <15318.52208.962415.759362@anthem.wooz.org> <20011024141633.96B6F303181@snelboot.oratrix.nl> Message-ID: <15318.52924.633767.193832@anthem.wooz.org> >>>>> "JJ" == Jack Jansen writes: JJ> Fine with me, except for the middle sentence, which should go: JJ> one of the reasons to go on the branch is that I *don't* have JJ> to twiddle patchlevel.h:-) D'oh! Of course. ;) From martin@v.loewis.de Wed Oct 24 19:18:32 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Wed, 24 Oct 2001 20:18:32 +0200 Subject: [Python-Dev] Re: patchlevel.h Message-ID: <200110241818.f9OIIWp01350@mira.informatik.hu-berlin.de> > Maybe I should start using the release branch for Mac releases after you are > done with it for the Unix/Windows release. As there is an r22b1 tag to point > to the exact revision used for the release there should be no adverse > consequences. What do you think? As before, I think there should be only a single 2.2b1 release. If you need to provide binaries, use what was released in the 2.2b1 source distribution (which is *not* the Unix release, it is the source distribution). What do your users lose if they have to get 2.2b1 as-is (ie. without further patches)? It may have bugs, sure - but so does the 2.2b1 release for everybody else. There will be a 2.2b2 release, so whatever you commit now will appear in 2.2b2. I think these release schedules are there for a purpose. Regards, Martin From guido@python.org Wed Oct 24 20:13:44 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 24 Oct 2001 15:13:44 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Wed, 24 Oct 2001 12:23:30 +1000." <200110240223.f9O2NUg29502@mbuna.arbhome.com.au> References: <200110240223.f9O2NUg29502@mbuna.arbhome.com.au> Message-ID: <200110241913.f9OJDiM10434@odiug.zope.com> > >>> Guido van Rossum wrote > > I've got a possible patch for the send() problem. I'd like review of > > the patch and the related issues before I apply it. I'd also like > > more guidance regarding the suitability of this patch for the 2.1.2 > > bugfix release. (It fixes a real bug, but can sometime lead to > > different behavior, and adds a new interpretation of the absence of an > > optional argument, which can be considered a feature.) > It's a tough decision, because this really _is_ the cleaner solution, > but it does stand a slight chance of breaking code, and it's also something > that can be worked around in the std library. At the moment I'm planning to > just fix the std library modules instead - as these are definitely broken. > The added functionality of python's send() is very useful, but it's a > feature, not a bug :) > > Anthony After ample discussion, the PythonLabs crew didn't think my patch would be safe enough -- see the SF tracker entry for a full discussion. Instead, we decided to fix the standard library to always check the return value of send(). Can you help? (If you check in a fix to the 2.1 maintenance branch, you can mark it as "propagate to trunk please".) --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Wed Oct 24 22:52:51 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 24 Oct 2001 23:52:51 +0200 Subject: [Python-Dev] iterator support for SRE? Message-ID: <014801c15cd6$3d18b270$ced241d5@hagrid> a common wish for SRE is a variant of findall that returns all match objects, instead of the strings. recently, I realized that the internal scanner type (originally added to speed up Python versions of findall/sub/split) can be wrapped in an iterator, allowing the user to loop over all matches. >>> import re >>> p = re.compile(somepattern) >>> for match in iter(p.scanner(somestring).search, None): >>> print match.groups() how about adding a "finditer" method, which takes care of the low-level setup: >>> for match in p.finditer(somestring): >>> print match.groups() or should we just make the scanner factory an official part of the SRE interface? From guido@python.org Thu Oct 25 04:35:56 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 24 Oct 2001 23:35:56 -0400 Subject: [Python-Dev] iterator support for SRE? In-Reply-To: Your message of "Wed, 24 Oct 2001 23:52:51 +0200." <014801c15cd6$3d18b270$ced241d5@hagrid> References: <014801c15cd6$3d18b270$ced241d5@hagrid> Message-ID: <200110250335.XAA10393@cj20424-a.reston1.va.home.com> > a common wish for SRE is a variant of findall that returns > all match objects, instead of the strings. > > recently, I realized that the internal scanner type (originally > added to speed up Python versions of findall/sub/split) can > be wrapped in an iterator, allowing the user to loop over all > matches. > > >>> import re > >>> p = re.compile(somepattern) > >>> for match in iter(p.scanner(somestring).search, None): > >>> print match.groups() > > how about adding a "finditer" method, which takes care of > the low-level setup: > > >>> for match in p.finditer(somestring): > >>> print match.groups() > > or should we just make the scanner factory an official part > of the SRE interface? Or both? The scanner interface seems vaguely useful as a low-level tool to create other higher-level variants; but the finditer() call seems to nicely capture a common case. --Guido van Rossum (home page: http://www.python.org/~guido/) From Anthony Baxter Thu Oct 25 07:24:20 2001 From: Anthony Baxter (Anthony Baxter) Date: Thu, 25 Oct 2001 16:24:20 +1000 Subject: [Python-Dev] Fixing send() In-Reply-To: Message from Guido van Rossum of "Wed, 24 Oct 2001 15:13:44 -0400." <200110241913.f9OJDiM10434@odiug.zope.com> Message-ID: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> >>> Guido van Rossum wrote > Instead, we decided to fix the standard library to always check the > return value of send(). Can you help? (If you check in a fix to the > 2.1 maintenance branch, you can mark it as "propagate to trunk please".) Will do - I plan to check in a bunch of fixes tomorrow morning. (continuing gdb hell today). One to consider is the status of 'socket.py' - patched, or not patched...? -- Anthony Baxter It's never too late to have a happy childhood. From sjoerd.mullender@oratrix.com Thu Oct 25 11:36:57 2001 From: sjoerd.mullender@oratrix.com (Sjoerd Mullender) Date: Thu, 25 Oct 2001 12:36:57 +0200 Subject: [Python-Dev] suggestion for PEP 101 Message-ID: <20011025103658.82DCD3021E6@bireme.oratrix.nl> I noticed that the first step in creating a release is ___ Do a CVS update with the -A flag, e.g. % cvs update -A I would suggest adding the -d and -P flags to this command (so cvs update -APd). The -d flag says to create directories that are in the repository but not in the checked-out version (handy for new directories), and the -P flag says to delete (prune) empty directories from the checked-out version (handy for directories that have become empty because all files were deleted). -- Sjoerd Mullender From guido@python.org Thu Oct 25 15:59:34 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 10:59:34 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Thu, 25 Oct 2001 16:24:20 +1000." <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> Message-ID: <200110251459.f9PExZg21822@odiug.zope.com> I'm planning to add a sendall() method to the socket object. See the SF patch: http://sf.net/tracker/index.php?func=detail&aid=474307&group_id=5470&atid=305470 If you have an objection, please let me know. I think this is important enough to bypass the feature freeze -- it's the best way to fix the very real bug (on FreeBSD) of unchecked send() calls. (The alternative would be to add a loop to every use of send().) The sendall() call allows us to fix the standard library modules that currently don't check for send()'s return value, without breaking any existing code, by changing sock.send(data) into sock.sendall(data). Other alternatives intended to fix the problem of unchecked send() calls in standard library module without having to change all send() calls were deemed to dangerous (see the SF patch discussion). I think Anthony is going to add this to the 2.1.2 bugfix release as well -- with the same argument for breaking the feature freeze. --Guido van Rossum (home page: http://www.python.org/~guido/) From Anthony Baxter Thu Oct 25 16:14:01 2001 From: Anthony Baxter (Anthony Baxter) Date: Fri, 26 Oct 2001 01:14:01 +1000 Subject: [Python-Dev] Fixing send() In-Reply-To: Message from Guido van Rossum of "Thu, 25 Oct 2001 10:59:34 -0400." <200110251459.f9PExZg21822@odiug.zope.com> Message-ID: <200110251514.f9PFE1229217@mbuna.arbhome.com.au> >>> Guido van Rossum wrote > I think Anthony is going to add this to the 2.1.2 bugfix release as > well -- with the same argument for breaking the feature freeze. That's correct. I think it's the appropriate way forward. I'm pretty sure the "goals for patch selection" had an implicit "except where footnote 1 applies" in there :) I'd be curious to know if anyone disagrees violently with this... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From guido@python.org Thu Oct 25 16:16:29 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 11:16:29 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Thu, 25 Oct 2001 11:12:23 EDT." <15320.11223.688433.641613@slothrop.digicool.com> References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <200110251459.f9PExZg21822@odiug.zope.com> <15320.11223.688433.641613@slothrop.digicool.com> Message-ID: <200110251516.f9PFGTf24686@odiug.zope.com> [Jeremy] > Can you offer a brief explanation of why we're doing this after all? > I thought we concluded yesterday that we would fix this library. > (That's what the last message before this one said.) Fixing the library would mean adding a loop like this to each application that uses send and currently doesn't check the return value: < sock.send(data) --- > while data: > n = sock.send(data) > data = data[n:] I find this more attractive: < sock.send(data) --- > sock.sendall(data) --Guido van Rossum (home page: http://www.python.org/~guido/) From gmcm@hypernet.com Thu Oct 25 16:36:49 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 25 Oct 2001 11:36:49 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: <200110251459.f9PExZg21822@odiug.zope.com> References: Your message of "Thu, 25 Oct 2001 16:24:20 +1000." <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> Message-ID: <3BD7F951.26534.24848D4A@localhost> [Guido] > I'm planning to add a sendall() method to the socket object. See > the SF patch: > > http://sf.net/tracker/index.php?func=detail&aid=474307&group_id=5 > 470&atid=305470 > > If you have an objection, please let me know. I think this is > important enough to bypass the feature freeze The bonehead who uses sendall with a non-blocking socket will, of course, get an EWOULDBLOCK (or platform equivalent) on an oversize send. Maybe EWOULDBLOCK should get turned into ButtHeadProgrammerError("Don't use sendall with a non-blocking socket")? [I agree that this is the best way to fix the std lib, and that it is very important that it be fixed.] - Gordon From guido@python.org Thu Oct 25 16:42:45 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 11:42:45 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Thu, 25 Oct 2001 11:36:49 EDT." <3BD7F951.26534.24848D4A@localhost> References: Your message of "Thu, 25 Oct 2001 16:24:20 +1000." <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <3BD7F951.26534.24848D4A@localhost> Message-ID: <200110251542.f9PFgjF28022@odiug.zope.com> > The bonehead who uses sendall with a non-blocking socket > will, of course, get an EWOULDBLOCK (or platform > equivalent) on an oversize send. Maybe EWOULDBLOCK > should get turned into ButtHeadProgrammerError("Don't use > sendall with a non-blocking socket")? I think natural selection will take care of this breed. :-) > [I agree that this is the best way to fix the std lib, and that it is > very important that it be fixed.] Thanks! --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Thu Oct 25 16:45:02 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 25 Oct 2001 11:45:02 -0400 Subject: [Python-Dev] suggestion for PEP 101 References: <20011025103658.82DCD3021E6@bireme.oratrix.nl> Message-ID: <15320.13182.670145.233045@anthem.wooz.org> >>>>> "SM" == Sjoerd Mullender writes: SM> I noticed that the first step in creating a release is | ___ Do a CVS update with the -A flag, e.g. | % cvs update -A SM> I would suggest adding the -d and -P flags to this command (so SM> cvs update -APd). Good point, thanks. -Barry From fdrake@acm.org Thu Oct 25 16:56:18 2001 From: fdrake@acm.org (Fred L. Drake) Date: Thu, 25 Oct 2001 11:56:18 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011025155618.C21CA28697@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Experimental stylesheet changes -- code is now presented using proportional fonts (mostly since Guido really dislikes Courier). Please send comments on this change to me at python-docs@python.org. From jeremy@zope.com Thu Oct 25 17:18:40 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Thu, 25 Oct 2001 12:18:40 -0400 (EDT) Subject: [Python-Dev] Fixing send() In-Reply-To: <3BD7F951.26534.24848D4A@localhost> References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <3BD7F951.26534.24848D4A@localhost> Message-ID: <15320.15200.184832.18268@slothrop.digicool.com> If we're making the std library routines that use sockets robust, I expect we ought to think about exceptions that should be dealt with. My Linux man page suggests that ENOBUFS and EINTR are errors that indicate "try again." There are probably other such errors on other platforms. We use Python sockets and asyncore in ZEO (a Zope subsystem) and I've found that every weird error that I've never heard of seems to occur in the wild. If we don't deal with these errors, then we've haven't fully succeeded in making the calls robust. Jeremy From guido@python.org Thu Oct 25 19:03:13 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 14:03:13 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Thu, 25 Oct 2001 12:18:40 EDT." <15320.15200.184832.18268@slothrop.digicool.com> References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <3BD7F951.26534.24848D4A@localhost> <15320.15200.184832.18268@slothrop.digicool.com> Message-ID: <200110251803.f9PI3D428392@odiug.zope.com> > If we're making the std library routines that use sockets robust, I > expect we ought to think about exceptions that should be dealt with. > My Linux man page suggests that ENOBUFS and EINTR are errors that > indicate "try again." There are probably other such errors on other > platforms. > > We use Python sockets and asyncore in ZEO (a Zope subsystem) and I've > found that every weird error that I've never heard of seems to occur > in the wild. If we don't deal with these errors, then we've haven't > fully succeeded in making the calls robust. There's a difference though. The bug I'm trying to fix with sendall() was a silent failure, causing at best protocol failures later, at worst silently lost data. ENOBUFS or EINTR etc. cause clear exceptions when they happen. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Thu Oct 25 22:15:39 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Thu, 25 Oct 2001 17:15:39 -0400 (EDT) Subject: [Python-Dev] dict comps Message-ID: <15320.33019.723537.580070@slothrop.digicool.com> We agreed yesterday that the dictionary() constructor would accept a a list of two-tuples (strictly speaking an iterable object of iterable objects of length 2). That plus list comprehensions pretty much covers the territory of dict comprehensions: >>> print dictionary([(i, chr(65 + i)) for i in range(4)]) {0: 'A', 1: 'B', 2: 'C', 3: 'D'} Jeremy From andymac@bullseye.apana.org.au Thu Oct 25 14:15:24 2001 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Thu, 25 Oct 2001 23:15:24 +1000 (EST) Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: <323.1003777312.538787.209802503@lilbastard> Message-ID: Unfortunately I haven't had the bandwidth to vigorously pursue completing integration of my previously submitted patches, and at this time don't expect to be able to do much until about the time 2.2b2 is scheduled to hit the streets :-(. So I don't expect that 2.2 will have OS/2+EMX build support in the sourceball... At this stage, my plan is to finalise a binary release of 2.2 before submitting a new set of patches, taking into account Martin von Loewis' comments and suggestions (much appreciated thank you!) among others. I also plan to try and coordinate my patches with those of Michael Muller (473749, 474169 & 474500 so far) for the VAC++ build of Python for OS/2. I will try and review Michael's patches with a view to getting them into 2.2, as the VAC++ build has been part of the sourceball for a few versions. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From guido@python.org Thu Oct 25 22:30:22 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 17:30:22 -0400 Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: Your message of "Thu, 25 Oct 2001 23:15:24 +1000." References: Message-ID: <200110252130.f9PLUM329494@odiug.zope.com> > Unfortunately I haven't had the bandwidth to vigorously pursue completing > integration of my previously submitted patches, and at this time don't > expect to be able to do much until about the time 2.2b2 is scheduled to > hit the streets :-(. So I don't expect that 2.2 will have OS/2+EMX build > support in the sourceball... > > At this stage, my plan is to finalise a binary release of 2.2 before > submitting a new set of patches, taking into account Martin von Loewis' > comments and suggestions (much appreciated thank you!) among others. I > also plan to try and coordinate my patches with those of Michael Muller > (473749, 474169 & 474500 so far) for the VAC++ build of Python for OS/2. > > I will try and review Michael's patches with a view to getting them into > 2.2, as the VAC++ build has been part of the sourceball for a few > versions. Andy, would it help if you had SourceForge commit privileges? You'd be obliged to only check in code that doesn't break the build on other platforms, so you would be required to test anything you want to check in on another platform (Linux or Windows) before you commit to save you the embarrassment of breaking the build, but if (as I expect) you would mostly be adding stuff inside "#ifdef OS2", that shouldn't be a big burden. It would save you the trouble of uploading patches to SourceForge and it would save *us* the trouble of reviewing your patches and checking them in. All your code would be owned by the PSF, but that's what you want anyway, right? --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com (Skip Montanaro) Thu Oct 25 22:53:27 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 25 Oct 2001 16:53:27 -0500 Subject: [Python-Dev] weird gdb/python/gtk/pango interaction? Message-ID: <15320.35287.392870.291275@beluga.mojam.com> Folks, I'm no longer able to run PyGtk scripts from gdb under some circumstances. I am using the CVS HEAD of glib, atk, pango, gtk+ and pygtk. The following simple Python script works fine when run from gdb: #!/usr/bin/env python import gtk w = gtk.Window() w.connect("destroy", gtk.mainquit) #lbl = gtk.Label("hi") #w.add(lbl) w.show_all() gtk.mainloop() with the following output: (gdb) r basic.py Starting program: /usr/local/bin/python basic.py [New Thread 1024 (LWP 19044)] Gtk-Message: YOU ARE USING THE DEVEL BRANCH 1.3.x OF GTK+ WHICH IS CURRENTLY UNDER HEAVY DEVELOPMENT AND FREQUENTLY INTRODUCES INSTABILITIES. if you don't know why you are getting this, you probably want to use the stable branch which can be retrieved from ftp://ftp.gtk.org/pub/gtk/v1.2/ or via CVS with cvs checkout -r glib-1-2 glib; cvs checkout -r gtk-1-2 gtk+ Program exited normally. If I uncomment the two lines involving lbl, however, I get this mess: (gdb) r basic.py Starting program: /usr/local/bin/python basic.py [New Thread 1024 (LWP 19127)] Gtk-Message: YOU ARE USING THE DEVEL BRANCH 1.3.x OF GTK+ WHICH IS CURRENTLY UNDER HEAVY DEVELOPMENT AND FREQUENTLY INTRODUCES INSTABILITIES. if you don't know why you are getting this, you probably want to use the stable branch which can be retrieved from ftp://ftp.gtk.org/pub/gtk/v1.2/ or via CVS with cvs checkout -r glib-1-2 glib; cvs checkout -r gtk-1-2 gtk+ basic.py (pid:19127): GRuntime-CRITICAL **: file gparamspecs.c: line 1687 (g_param_spec_float): assertion `default_value >= minimum && default_value <= maximum' failed basic.py (pid:19127): GRuntime-CRITICAL **: file gobject.c: line 270 (g_object_class_install_property): assertion `G_IS_PARAM_SPEC (pspec)' failed basic.py (pid:19127): ** WARNING **: Couldn't load font "Sans -2.09715e+06" falling back to "Sans -2.09715e+06" basic.py (pid:19127): ** WARNING **: Couldn't load font "Sans -2.09715e+06" falling back to "Sans -2.09715e+06" basic.py (pid:19127): ** WARNING **: All font failbacks failed!!!! Program exited with code 01. If I change the Label to an EventBox, everything works, presumably because no font rendering is involved. Also when I run the script from the bash prompt it works as expected (no Gtk warnings). The equivalent C program: #include int main( int argc, char *argv[] ) { GtkWidget *window; GtkWidget *label; gtk_init(&argc, &argv); window = gtk_window_new (GTK_WINDOW_TOPLEVEL); gtk_signal_connect (GTK_OBJECT (window), "destroy", GTK_SIGNAL_FUNC(gtk_main_quit), NULL); gtk_window_set_title (GTK_WINDOW (window), ""); label = gtk_label_new("hi"); gtk_container_add(GTK_CONTAINER(window), label); gtk_widget_show_all (GTK_WIDGET(window)); gtk_main(); return(0); } works as expected, both from the command line and if run from the gdb prompt. This wouldn't be such a big deal, except in a more complex script I get segfaults in some circumstances, and this makes debugging that problem a bit difficult. Any idea what's going on? -- Skip Montanaro (skip@pobox.com) http://www.mojam.com/ http://www.musi-cal.com/ From martin@v.loewis.de Thu Oct 25 23:31:57 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Fri, 26 Oct 2001 00:31:57 +0200 Subject: [Python-Dev] weird gdb/python/gtk/pango interaction? Message-ID: <200110252231.f9PMVvW03121@mira.informatik.hu-berlin.de> > Any idea what's going on? When you see that a program behaves differently in gdb than it does on the shell, the most likely cause is that gdb starts another shell when running the program, I believe essentially through system("$SHELL -c command") (see gdb documentation and source for details). For example, some of the $HOME/.something files may set PYTHONPATH to some interesting value. If you see that a Python program behaves differently from an "equivalent" C code, well, the most likely cause is that the programs are not equivalent. One obvious (to me) difference is that one calls set_title, when the other doesn't. Regards, Martin From aahz@rahul.net Fri Oct 26 02:12:45 2001 From: aahz@rahul.net (Aahz Maruch) Date: Thu, 25 Oct 2001 18:12:45 -0700 (PDT) Subject: [Python-Dev] Fixing send() In-Reply-To: <200110251514.f9PFE1229217@mbuna.arbhome.com.au> from "Anthony Baxter" at Oct 26, 2001 01:14:01 AM Message-ID: <20011026011245.65372E8C3@waltz.rahul.net> Anthony Baxter wrote: >Guido van Rossum wrote >> >> I think Anthony is going to add this to the 2.1.2 bugfix release as >> well -- with the same argument for breaking the feature freeze. > > That's correct. I think it's the appropriate way forward. > > I'm pretty sure the "goals for patch selection" had an implicit > "except where footnote 1 applies" in there :) PEP 6 has an explicit exception for BDFL Pronouncements. ;-) Given that this is "guaranteed" to not break any code, it's fair game. -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From fdrake@acm.org Fri Oct 26 04:13:46 2001 From: fdrake@acm.org (Fred L. Drake) Date: Thu, 25 Oct 2001 23:13:46 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011026031346.8550628697@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Yet another experimental change to the presentation -- ditch the proportional font for code since some displays (especially interactive sessions and tabular displays) get messed up with proportional fonts. We do try to use monospaced fonts that are less ugly than Courier. As before, feedback on the fonts is welcome at python-docs@python.org. From tim.one@home.com Fri Oct 26 06:34:16 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 26 Oct 2001 01:34:16 -0400 Subject: [Python-Dev] dict comps In-Reply-To: <15320.33019.723537.580070@slothrop.digicool.com> Message-ID: [Jeremy Hylton] > We agreed yesterday that the dictionary() constructor would accept a > a list of two-tuples (strictly speaking an iterable object of iterable > objects of length 2). FYI, this is checked in now. > That plus list comprehensions pretty much covers the territory of > dict comprehensions: > > >>> print dictionary([(i, chr(65 + i)) for i in range(4)]) > {0: 'A', 1: 'B', 2: 'C', 3: 'D'} Wow -- that's *exactly* what it prints. You got your own time machine now? While it covers the semantics, the pragmatics may be off, since listcomps produce genuine lists, and so e.g. dictionary([(key, f(key)) for key in file('huge')]) may require constructing an unboundedly large list of twoples before dictionary() sees the first pair. dictionary() per se doesn't require materializing a giant list in one gulp. From mal@lemburg.com Fri Oct 26 10:02:02 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 26 Oct 2001 11:02:02 +0200 Subject: [Python-Dev] dict comps References: Message-ID: <3BD9268A.72AD71E9@lemburg.com> Tim Peters wrote: > > [Jeremy Hylton] > > We agreed yesterday that the dictionary() constructor would accept a > > a list of two-tuples (strictly speaking an iterable object of iterable > > objects of length 2). > > FYI, this is checked in now. > > > That plus list comprehensions pretty much covers the territory of > > dict comprehensions: > > > > >>> print dictionary([(i, chr(65 + i)) for i in range(4)]) > > {0: 'A', 1: 'B', 2: 'C', 3: 'D'} Cool ! > Wow -- that's *exactly* what it prints. You got your own time machine now? > > While it covers the semantics, the pragmatics may be off, since listcomps > produce genuine lists, and so e.g. > > dictionary([(key, f(key)) for key in file('huge')]) > > may require constructing an unboundedly large list of twoples before > dictionary() sees the first pair. dictionary() per se doesn't require > materializing a giant list in one gulp. One way or another, you'll use up a giant chunk or two of data on the heap... I'd suggest adding a new builtin huge_file_as_mapping_apply('file', f) ;-) Seriously, this goes down the path of lazy evaluation of expressions. Not sure whether this is the right path to follow though (can cause brain damage due to overloaded builtin debuggers, IMHO). BTW, looks like I can finally get rid off the dict() builtin I have in mxTools which is Good News ! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Fri Oct 26 09:52:50 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 26 Oct 2001 10:52:50 +0200 Subject: [Python-Dev] weird gdb/python/gtk/pango interaction? References: <15320.35287.392870.291275@beluga.mojam.com> Message-ID: <3BD92462.8A8E2C4F@lemburg.com> Skip Montanaro wrote: > > Folks, > > I'm no longer able to run PyGtk scripts from gdb under some circumstances. > I am using the CVS HEAD of glib, atk, pango, gtk+ and pygtk. The following > simple Python script works fine when run from gdb: > > #!/usr/bin/env python > > import gtk > > w = gtk.Window() > w.connect("destroy", gtk.mainquit) > #lbl = gtk.Label("hi") > #w.add(lbl) > > w.show_all() > gtk.mainloop() > > with the following output: > > (gdb) r basic.py > Starting program: /usr/local/bin/python basic.py > [New Thread 1024 (LWP 19044)] > Gtk-Message: YOU ARE USING THE DEVEL BRANCH 1.3.x OF GTK+ WHICH IS CURRENTLY > UNDER HEAVY DEVELOPMENT AND FREQUENTLY INTRODUCES INSTABILITIES. if you don't know why you are getting this, you probably want to use the stable branch which can be retrieved from > ftp://ftp.gtk.org/pub/gtk/v1.2/ or via CVS with > cvs checkout -r glib-1-2 glib; cvs checkout -r gtk-1-2 gtk+ Just curious: does the proposed stable branch show the same behaviour ? If so, I'd suggest to ask the GTK guys for help. If not, then perhaps you have to rebuild the Python extensions involved and make sure that they match the Python version you are using. What you're seeing looks like there are some buffer overruns going on... I've never seen such a strange font name ;-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From fredrik@pythonware.com Fri Oct 26 10:38:29 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 26 Oct 2001 11:38:29 +0200 Subject: [Python-Dev] iterator support for SRE? References: <014801c15cd6$3d18b270$ced241d5@hagrid> <200110250335.XAA10393@cj20424-a.reston1.va.home.com> Message-ID: <015a01c15e01$f99f4ef0$0900a8c0@spiff> guido wrote: > > how about adding a "finditer" method, which takes care of > > the low-level setup: > > > > >>> for match in p.finditer(somestring): > > >>> print match.groups() > > > > or should we just make the scanner factory an official part > > of the SRE interface? > > Or both? The scanner interface seems vaguely useful as a low-level > tool to create other higher-level variants; but the finditer() call > seems to nicely capture a common case. as you might have noticed, I went ahead and checked in code for "finditer". nobody has complained this far ;-) isn't "iterfind" a better name, btw? more like "iterkeys" (etc). if nobody complains, I'll change the name, and check in some documentation. ::: will have to think about scanner; I agree that it may seem to be vaguely useful, but cannot think of a use case that isn't better handled by findall/split/sub or iterfind. can anyone? From jack@oratrix.nl Fri Oct 26 11:25:39 2001 From: jack@oratrix.nl (Jack Jansen) Date: Fri, 26 Oct 2001 12:25:39 +0200 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by "Martin v. Loewis" , Wed, 24 Oct 2001 20:18:32 +0200 , <200110241818.f9OIIWp01350@mira.informatik.hu-berlin.de> Message-ID: <20011026102540.3DBEB303181@snelboot.oratrix.nl> > > Maybe I should start using the release branch for Mac releases after you are > > done with it for the Unix/Windows release. As there is an r22b1 tag to point > > to the exact revision used for the release there should be no adverse > > consequences. What do you think? > > As before, I think there should be only a single 2.2b1 release. If you > need to provide binaries, use what was released in the 2.2b1 source > distribution (which is *not* the Unix release, it is the source > distribution). Martin, while I completely agree with you in principle, in practice this isn't possible. When the release branch is made there's usually a flurry of minor fixes, and these are almost guaranteed to break something on the mac. Tiny things, like missing casts, but breaks nonetheless. Moreover, the "release loop" is now about 24 hours long, according to PEP101, and even extending it seriously (like to about a week) still wouldn't guarantee that I could react timely. Not only am I 6 hours away from the unix/win release folks, but I also have a paying job that is less and less MacPython-related, so I can't firmly commit myself. And it has happened to me already (twice, I think) that there was a showstopper bug on the Mac that has caused me to either be very late with a release or skip one altogether. This happens on the Mac more often than on Unix/Windows, because MacPython relies on a third party unix I/O emulation library. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From thomas.heller@ion-tof.com Fri Oct 26 14:29:03 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 26 Oct 2001 15:29:03 +0200 Subject: [Python-Dev] innocent question ;-) Message-ID: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> ... about my beloved buffer object: Would a patch to make the buffer object subclassable be still accepted for 2.2, or does this fall under the feature freeze? Another question: It is my understanding that makeing an object subclassable also exposes its constructor to Python. What is the signature of this constructor? Is the three argument version xxx(name, bases, dict) always used to return a new type object (Don Beaudry hook), and can other signatures freely be used to creates objects/instances of this type/class? Thomas From guido@python.org Fri Oct 26 14:30:55 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 09:30:55 -0400 Subject: [Python-Dev] iterator support for SRE? In-Reply-To: Your message of "Fri, 26 Oct 2001 11:38:29 +0200." <015a01c15e01$f99f4ef0$0900a8c0@spiff> References: <014801c15cd6$3d18b270$ced241d5@hagrid> <200110250335.XAA10393@cj20424-a.reston1.va.home.com> <015a01c15e01$f99f4ef0$0900a8c0@spiff> Message-ID: <200110261330.JAA18598@cj20424-a.reston1.va.home.com> > as you might have noticed, I went ahead and checked in code > for "finditer". nobody has complained this far ;-) Far from it. :-) > isn't "iterfind" a better name, btw? more like "iterkeys" (etc). > > if nobody complains, I'll change the name, and check in some > documentation. Hm, but finditer() is more like findall(), and that counts more I'd say. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 26 14:42:53 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 09:42:53 -0400 Subject: [Python-Dev] innocent question ;-) In-Reply-To: Your message of "Fri, 26 Oct 2001 15:29:03 +0200." <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> References: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> Message-ID: <200110261342.JAA18712@cj20424-a.reston1.va.home.com> > ... about my beloved buffer object: > > Would a patch to make the buffer object subclassable > be still accepted for 2.2, or does this fall under > the feature freeze? NO. Go away. :-) I think the buffer() builtin should be deprecated, not improved. > Another question: > It is my understanding that makeing an object > subclassable also exposes its constructor to Python. Yes, since otherwise there would be no way to instantiate the subclass either. > What is the signature of this constructor? Is the > three argument version xxx(name, bases, dict) always used > to return a new type object (Don Beaudry hook), and can other > signatures freely be used to creates objects/instances > of this type/class? I'm sorry, I don't understand the question. This is the constructor signature for type object, IOW for metaclasses. It has nothing to do with the constructor signature for builtin classes. Also, what do you mean by constructor? __new__ or __init__? Read test_descr.py for examples. Also xxsubtype.c and various built-in types if you're interested in doing this at the C level. --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Fri Oct 26 14:57:23 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 26 Oct 2001 15:57:23 +0200 Subject: [Python-Dev] innocent question ;-) References: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> <200110261342.JAA18712@cj20424-a.reston1.va.home.com> Message-ID: <02e901c15e26$2314abd0$e000a8c0@thomasnotebook> From: "Guido van Rossum" > > ... about my beloved buffer object: > > > > Would a patch to make the buffer object subclassable > > be still accepted for 2.2, or does this fall under > > the feature freeze? > > NO. Go away. :-) Ok. ;-( Is the list of subtypable types now complete (for 2.2), or do you indend to do more? > > I think the buffer() builtin should be deprecated, not improved. > > > Another question: > > It is my understanding that makeing an object > > subclassable also exposes its constructor to Python. > > Yes, since otherwise there would be no way to instantiate the > subclass either. So I understood this... > > > What is the signature of this constructor? Is the > > three argument version xxx(name, bases, dict) always used > > to return a new type object (Don Beaudry hook), and can other > > signatures freely be used to creates objects/instances > > of this type/class? > > I'm sorry, I don't understand the question. This is the constructor > signature for type object, IOW for metaclasses. It has nothing to do > with the constructor signature for builtin classes. ...but not this, sorry. I've confused this: >>> int(10) 10 with this: >>> type(int)("name", (), {}) Sorry again. Thomas From paul-python@svensson.org Fri Oct 26 15:01:56 2001 From: paul-python@svensson.org (Paul Svensson) Date: Fri, 26 Oct 2001 10:01:56 -0400 (EDT) Subject: [Python-Dev] dict comps In-Reply-To: <15320.33019.723537.580070@slothrop.digicool.com> Message-ID: On Thu, 25 Oct 2001, Jeremy Hylton wrote: >We agreed yesterday that the dictionary() constructor would accept a >a list of two-tuples (strictly speaking an iterable object of iterable >objects of length 2). That plus list comprehensions pretty much >covers the territory of dict comprehensions: > >>>> print dictionary([(i, chr(65 + i)) for i in range(4)]) >{0: 'A', 1: 'B', 2: 'C', 3: 'D'} > Wouldn't it be nice to be able to spell that { i: chr(65 + i) for i in range(4) } /Paul From guido@python.org Fri Oct 26 15:03:25 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 10:03:25 -0400 Subject: [Python-Dev] innocent question ;-) In-Reply-To: Your message of "Fri, 26 Oct 2001 15:57:23 +0200." <02e901c15e26$2314abd0$e000a8c0@thomasnotebook> References: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> <200110261342.JAA18712@cj20424-a.reston1.va.home.com> <02e901c15e26$2314abd0$e000a8c0@thomasnotebook> Message-ID: <200110261403.KAA18833@cj20424-a.reston1.va.home.com> > Is the list of subtypable types now complete (for 2.2), > or do you indend to do more? I just found a few extension modules that define their own types, like SocketType. I'd like to see those converted. I don't know if I have time. --Guido van Rossum (home page: http://www.python.org/~guido/) From jack@oratrix.nl Fri Oct 26 15:23:31 2001 From: jack@oratrix.nl (Jack Jansen) Date: Fri, 26 Oct 2001 16:23:31 +0200 Subject: [Python-Dev] innocent question ;-) In-Reply-To: Message by Guido van Rossum , Fri, 26 Oct 2001 10:03:25 -0400 , <200110261403.KAA18833@cj20424-a.reston1.va.home.com> Message-ID: <20011026142331.C8E84303181@snelboot.oratrix.nl> > > Is the list of subtypable types now complete (for 2.2), > > or do you indend to do more? > > I just found a few extension modules that define their own types, like > SocketType. I'd like to see those converted. I don't know if I have > time. At what point can we consider the API stable? All this inheriting and subtyping seems like it would be absolutely brilliant for the Mac toolbox APIs (and I suspect also for the PythonWin API's, Mark?) but I've refrained from even looking at it (well, I did have a quick peek:-), because (a) I have no time, (b) I don't want to chase a changing API and (c) there's some methodchain-based poor-mans-inheritance in there already that will undoubtedly break. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From webmaster@garymgordon.com Fri Oct 26 15:30:10 2001 From: webmaster@garymgordon.com (Gary M. Gordon) Date: Fri, 26 Oct 2001 09:30:10 -0500 Subject: [Python-Dev] Which one of our FREE website enhancements would you like? Message-ID: Gary M. Gordon - FREE Website Enhancement Promotional Offer
          FREE WEBSITE ENHANCEMENT OFFER
                  yes, ... absolutely FREE !

                (No contracts!   No commitments!   No cost!)

          - Links to sample enhancements are provided below. -


As a way to introduce our company to you, we are going to give

you an enhancement that you can add to your website ............

                                          absolutely FREE!


===================================================

   Just select from one of our FREE website enhancement features
   that are listed below.  There are 'No contracts required!' ..
There are ...

   Let me introduce myself ...

   My name is Gary Gordon. I am the owner of Gary M. Gordon, LLC
   a WEB DEVELOPMENT COMPANY which provides complete solutions
   for the needs of small to large businesses. (No project is ever to
   small or to large.)  If you need a 'fix' or 'enhancement' added to
   a current website ... that's perfect.  If you need to have a new
   website developed from scratch ... that's great.   We do it all!

   FREE WEBSITE ENHANCEMENTS YOU CAN CHOOSE FROM:

   (Simply click on the 'http:// link' provided below the title of
   each item to view just a sample of the FREE enhancement.)


   3)  FREE 'your choice of any of the following Javascripts'

We will insert and set them up to work in any webpage on your website'.

       a)    Page Content Management that automatically allows content
       to be displayed on a page - from and through a specified date, then

       b)    Auto Scroller for Text and Images.  This provides a
       scrolling message that can include text and images (with links
       to other pages or websites).
 http://www.garymgordon.com/javascript_samples/scroller_box/index.html        c)    Navigational Menu System that allows you to utilize a drop
       down menu that lists any number of pages, websites, etc. It is
       a convenient and space saving way to add lots of links within
       a small place on a webpage.
 http://www.garymgordon.com/javascript_samples/dropdownmenu/index.html


Now ... you can choose from any of the above website enhancements
absolutely FREE!

Then, simply contact us and we will set everything up for you - NO CHARGE!

(This is simply our way of introducing our company to you.)


===================================================

              To request your FREE website enhancement,
              please include the following two lines of
              information in the BODY of your reply email:

                  FREE WEBSITE ENHANCEMENT OFFER
                                      WE-149-359-1001

                  Also provide us with the following:

                  1)  Your company name and address.
                  2)  Your first and last name.
                  3)  Your title/position with the company.
                  4)  Your telephone number - and the best
                        time to reach you by phone.
                  5)  Your website URL (if you have one).

                  We will email or call you back shortly to discuss
                  the free enhancement of your choice.

===================================================

    OUR CORPORATE INFORMATION:

    Gary M. Gordon, LLC
    - A Global Web Development Corporation
    421 Jamaica Drive
    Cherry Hill, New Jersey  08002

    (Available -  7 days a week.)


    Background Information of GARY M. GORDON (Owner):

    Certified Senior Web Developer since 1996
    Certified Web Master Instructor since 1999
    20+ years experience in Sales/Marketing/Graphic Design
    and Development

  
    Additional Comments:

    Gary M. Gordon, LLC (Web Development Services) was created
    in an effort to provide quality 'web development' work on a
    world-wide scale through the joint efforts of a comprehensive
    and world-wide network (that includes a selected group of
    professional and certified application programmers, website
    developers, graphic designers, etc. - from the USA, Canada,
    India, Romania, Japan, Australia, and others).

                    ALL OF OUR WORK IS PROVIDED
              WITH A 100% SATISFACTION GUARANTEE!

    So, if you're not satisfied with our work, ... you don't pay.
    It's as simple as that!

    Please feel free to visit our website or email us at your
    convenience, and we will be happy to help you in any way
    possible.


    * Certain restrictions or requirements may apply to all work
    requests, free offers, etc.  Please ask for complete details.

    Free offers are also limited to one per website domain and/or
    company.  Gary M. Gordon, LLC reserves the right to discontinue,
    change, or modify this free offer at any time without notice.


If your browser doesn't support the display of HTML, you can access the same information as shown above by following the link to take advange of our Free Website Enhancement Offer at: http://www.garymgordon.com/services.html (then, on this page, look for the link that says "Free Current Promotional Offer).

-------------------------------------------------------------------------------------


If you have joined Gary M. Gordon, LLC by accident or someone else has joined you without your permission,
or if you ever want to remove yourself from Gary M. Gordon, LLC, simply visit:
http://www.garymgordon.com/easylist/easylist.cgi?action=unsubscribe&submitemail=python-dev@python.org
and you will be automatically removed.
From guido@python.org Fri Oct 26 15:32:59 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 10:32:59 -0400 Subject: [Python-Dev] innocent question ;-) In-Reply-To: Your message of "Fri, 26 Oct 2001 16:23:31 +0200." <20011026142331.C8E84303181@snelboot.oratrix.nl> References: <20011026142331.C8E84303181@snelboot.oratrix.nl> Message-ID: <200110261432.KAA18981@cj20424-a.reston1.va.home.com> > At what point can we consider the API stable? All this inheriting > and subtyping seems like it would be absolutely brilliant for the > Mac toolbox APIs (and I suspect also for the PythonWin API's, Mark?) > but I've refrained from even looking at it (well, I did have a quick > peek:-), because (a) I have no time, (b) I don't want to chase a > changing API and (c) there's some methodchain-based > poor-mans-inheritance in there already that will undoubtedly break. I don't plan to change the C API that was released in 2.2b1 incompatibly. Besides, I won't have time to touch it after next Friday. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Fri Oct 26 15:45:13 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 26 Oct 2001 10:45:13 -0400 Subject: [Python-Dev] iterator support for SRE? References: <014801c15cd6$3d18b270$ced241d5@hagrid> <200110250335.XAA10393@cj20424-a.reston1.va.home.com> <015a01c15e01$f99f4ef0$0900a8c0@spiff> Message-ID: <15321.30457.83985.982168@anthem.wooz.org> >>>>> "FL" == Fredrik Lundh writes: FL> as you might have noticed, I went ahead and checked in code FL> for "finditer". nobody has complained this far ;-) Please add something to Misc/NEWS file too! -Barry From martin@v.loewis.de Fri Oct 26 19:31:50 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Fri, 26 Oct 2001 20:31:50 +0200 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: <20011026102540.3DBEB303181@snelboot.oratrix.nl> (message from Jack Jansen on Fri, 26 Oct 2001 12:25:39 +0200) References: <20011026102540.3DBEB303181@snelboot.oratrix.nl> Message-ID: <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> > When the release branch is made there's usually a flurry of minor > fixes, and these are almost guaranteed to break something on the > mac. Tiny things, like missing casts, but breaks nonetheless. While this may happen in principle, and out of curiosity: Was this a problem in 2.2b1 also? Looking at the changes made on the 2.2b1 release branch, I see that a total of 7 files was changed. Except for patchlevel.h, and the \n\ fix on socketmodule.c, there were no changes to C code. So I can't see why the changes on the release branch could have any effect on the Mac port, atleast in this release. > Moreover, the "release loop" is now about 24 hours long, according > to PEP101, and even extending it seriously (like to about a week) > still wouldn't guarantee that I could react timely. Not only am I 6 > hours away from the unix/win release folks, but I also have a paying > job that is less and less MacPython-related, so I can't firmly > commit myself. I'm not asking that you work harder on Python, I'm asking that you work less :-) Seriously, I think there is a much larger set of people testing the CVS regularly, so I doubt any breakage atleast on OS X wouldn't be noticed within hours. > And it has happened to me already (twice, I think) that there was a > showstopper bug on the Mac that has caused me to either be very late > with a release or skip one altogether. This happens on the Mac more > often than on Unix/Windows, because MacPython relies on a third > party unix I/O emulation library. I sympathize with this problem, and I can't really suggest a good solution to it. I'm also not so much concerned about concerned about beta releases; nobody will care whether they can build 2.2b1 from the sources on the Mac two months from now. It's just that I think very strict principles must be applied for the final release. If that means that the 2.2 release can't go without ok from you (or somebody else who has produced MacOS 9 binaries), I think we should add that to the release procedure. That check would be to avoid show-stopper bugs only, of course - anything complex needs to be detected long before the final release. Regards, Martin From guido@python.org Fri Oct 26 19:49:28 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 14:49:28 -0400 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Your message of "Fri, 26 Oct 2001 20:31:50 +0200." <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> References: <20011026102540.3DBEB303181@snelboot.oratrix.nl> <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> Message-ID: <200110261849.OAA20387@cj20424-a.reston1.va.home.com> From guido@python.org Fri Oct 26 19:56:20 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 14:56:20 -0400 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Your message of "Fri, 26 Oct 2001 20:31:50 +0200." <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> References: <20011026102540.3DBEB303181@snelboot.oratrix.nl> <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> Message-ID: <200110261856.OAA20424@cj20424-a.reston1.va.home.com> > > When the release branch is made there's usually a flurry of minor > > fixes, and these are almost guaranteed to break something on the > > mac. Tiny things, like missing casts, but breaks nonetheless. > > While this may happen in principle, and out of curiosity: Was this a > problem in 2.2b1 also? Looking at the changes made on the 2.2b1 > release branch, I see that a total of 7 files was changed. Except for > patchlevel.h, and the \n\ fix on socketmodule.c, there were no changes > to C code. What's more common is actually a flurry of checkins just *before* the release branch is made (say in the last two days). These often include a somewhat hasty commit of a project that's valuable to have but introduces new bugs -- either Mac-specific or generic. The patch that introduced the missing \n\ problem was an example of this. I don't know what to do about this. Branching several days earlier doesn't prevent it, because usually there's a really good reason to copy the last-minute changes into the release branch. (We don't have the problem that some other projects appear to have, which is that there's a lot of tentative development on the trunk -- then it would make sense to branch earlier, but we tend to use the patch manager for experiments.) > It's just that I think very strict principles must be applied for the > final release. If that means that the 2.2 release can't go without ok > from you (or somebody else who has produced MacOS 9 binaries), I think > we should add that to the release procedure. That check would be to > avoid show-stopper bugs only, of course - anything complex needs to be > detected long before the final release. I think that for the final release we should definitely attempt this. Hopefully the speed and quality of change will reduce closer to the final release. Also, the release candidate should help. Jack, We'll try to work with you. Barry, please add something to this effect in the release procedure PEP (hold up the *final* release until Jack approves or until we lose patience). --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Fri Oct 26 20:13:48 2001 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 26 Oct 2001 15:13:48 -0400 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives Message-ID: <3BD9B5EC.119319A5@interet.com> The PEP for zip import is 273. Please take a look and comment. http://python.sourceforge.net/peps/pep-0273.html Jim Ahlstrom From guido@python.org Fri Oct 26 21:34:15 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 16:34:15 -0400 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Fri, 26 Oct 2001 15:13:48 EDT." <3BD9B5EC.119319A5@interet.com> References: <3BD9B5EC.119319A5@interet.com> Message-ID: <200110262034.QAA20802@cj20424-a.reston1.va.home.com> > The PEP for zip import is 273. Please take a look and comment. > > http://python.sourceforge.net/peps/pep-0273.html OK, I'll shoot. But I expect Gordon McMillan and Greg Stein to provide more useful feedback. | Currently, sys.path is a list of directory names as strings. If | this PEP is implemented, an item of sys.path can be a string | naming a zip file archive. The zip archive can contain a | subdirectory structure to support package imports. The zip | archive satisfies imports exactly as a subdirectory would. I like this. | The implementation is in C code in the Python core and works on | all supported Python platforms. This is good too, as it provides a bootstrap. OTOH I also would like to see a prototype in Python, using either ihooks or imputil. | Any files may be present in the zip archive, but only files *.pyc, | *.pyo and __init__.py[co] are available for import. Zip import of | *.py and dynamic modules (*.pyd, *.so) is disallowed. | | Just as sys.path currently has default directory names, default | zip archive names are added too. Otherwise there is no way to | import all Python library files from an archive. More bootstrap goodness. | Reading compressed zip archives requires the zlib module. An | import of zlib will be attempted prior to any other imports. If | zlib is not available at that time, only uncompressed archives | will be readable, even if zlib subsequently becomes available. Hm, I wonder if we couldn't just link with the libz.a C library and use the C interface, if you're implementing this in C anyway. | Subdirectory Equivalence | | The zip archive must be treated exactly as a subdirectory tree so | we can support package imports based on current and future rules. | Zip archive files must be created with relative path names. That | is, archive file names are of the form: file1, file2, dir1/file3, | dir2/dir3/file4. | | Suppose sys.path contains "/A/B/SubDir" and "/C/D/E/Archive.zip", | and we are trying to import modfoo from the Q package. Then | import.c will generate a list of paths and extensions and will | look for the file. The list of generated paths does not change | for zip imports. (Very clever.) Suppose import.c generates the path | "/A/B/SubDir/Q/R/modfoo.pyc". Then it will also generate the path | "/C/D/E/Archive.zip/Q/R/modfoo.pyc". Finding the SubDir path is | exactly equivalent to finding "Q/R/modfoo.pyc" in the archive. Nice. | Suppose you zip up /A/B/SubDir/* and all its subdirectories. Then | your zip file will satisfy imports just as your subdirectory did. | | Well, not quite. You can't satisfy dynamic modules from a zip | file. Dynamic modules have extensions like .dll, .pyd, and .so. | They are operating system dependent, and probably can't be loaded | except from a file. It might be possible to extract the dynamic | module from the zip file, write it to a plain file and load it. | But that would mean creating temporary files, and dealing with all | the dynload_*.c, and that's probably not a good idea. Agreed. | You also can't import source files *.py from a zip archive. The | problem here is what to do with the compiled files. Python would | normally write these to the same directory as *.py, but surely we | don't want to write to the zip file. We could write to the | directory of the zip archive, but that would clutter it up, not | good if it is /usr/bin for example. We could just fail to write | the compiled files, but that makes zip imports very slow, and the | user would probably not figure out what is wrong. It is probably | best for users to put *.pyc into zip archives in the first place, | and this PEP enforces that rule. I agree. But it would still be good if the .py files were also in the zip file, so the source can be used in tracebacks etc. A C API to get a source line from a filename might be a good idea (plus a Python API). | So the only imports zip archives support are *.pyc and *.pyo, plus | the import of __init__.py[co] for packages, and the search of the | subdirectory structure for the same. I wonder if we need to make an additional rule that allows a .pyc file to satisfy a module request even if we're in optimized mode (where normally only .pyo files are searched). Otherwise, if someone ships a zipfile with only .pyc files, their modules can't be imported at all when python -O is used. | Efficiency | | The only way to find files in a zip archive is linear search. But there's an index record at the end that provides quick access. So | for each zip file in sys.path, we search for its names once, and | put the names plus other relevant data into a static Python | dictionary. The key is the archive name from sys.path joined with | the file name (including any subdirectories) within the archive. | This is exactly the name generated by import.c, and makes lookup | easy. We could do this kind of pre-scanning for regular dictionaries on sys.path too. I found out very long ago (around '93 or '94) that this saves a *lot* of startup time; I presume it still does. (And even more if the info can be cached in a file.) The only problem is how to detect when the cache becomes out of date. Of course, you could say "if you want faster startup time, put all your files in a zip archive", and I couldn't really argue with that. :-) | zlib | | Compressed zip archives require zlib for decompression. Prior to | any other imports, we attempt an import of zlib, and set a flag if | it is available. All compressed files are invisible unless this | flag is true. Do we get an "module not found" error or something better, like "compressed module found as but zlib unavailable"? | It could happen that zlib was available later. For example, the | import of site.py might add the correct directory to sys.path so a | dynamic load succeeds. But compressed files will still be | invisible. It is unknown if it can happen that importing site.py | can cause zlib to appear, so maybe we're worrying about nothing. | On Windows and Linux, the early import of zlib succeeds without | site.py. Yes, site.py isn't needed to make standard library modules available; it's intended to make non-standare library modules available. :-) | The problem here is the confusion caused by the reverse. Either a | zip file satisfies imports or it doesn't. It is silly to say that | site.py needs to be uncompressed, and that maybe imports will | succeed later. If you don't like this, create uncompressed zip | archives or make sure zlib is available, for example, as a | built-in module. Or we can write special search logic during zip | initialization. I don't think we need anything special here. site.py shouldn't be needed. | Booting | | Python imports site.py itself, and this imports os, nt, ntpath, | stat, and UserDict. It also imports sitecustomize.py which may | import more modules. Zip imports must be available before site.py | is imported. | | Just as there are default directories in sys.path, there must be | one or more default zip archives too. | | The problem is what the name should be. The name should be linked | with the Python version, so the Python executable can correctly | find its corresponding libraries even when there are multiple | Python versions on the same machine. | | This PEP suggests a zip archive name equal to the Python | interpreter path with extension ".zip" (eg, /usr/bin/python.zip) | which is always prepended to sys.path. So a directory with python | and python.zip is complete. This would work fine on Windows, as | it is common to put supporting files in the directory of the | executable. But it may offend Unix fans, who dislike bin | directories being used for libraries. It might be fine to | generate different defaults for Windows and Unix if necessary, but | the code will be in C, and there is no sense getting complicated. Well, this is the domain of getpath.c, and that's got a different implementation for Unix and Windows anyway (Windows has PC/getpathp.c). | Implementation | | A C implementation exists which works, but which can be made better. Upload as a patch please? --Guido van Rossum (home page: http://www.python.org/~guido/) From bckfnn@worldonline.dk Fri Oct 26 23:22:05 2001 From: bckfnn@worldonline.dk (Finn Bock) Date: Fri, 26 Oct 2001 22:22:05 GMT Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BD9B5EC.119319A5@interet.com> References: <3BD9B5EC.119319A5@interet.com> Message-ID: <3bd9d419.10386565@mail.wanadoo.dk> [James C. Ahlstrom] >The PEP for zip import is 273. Please take a look and comment. > > http://python.sourceforge.net/peps/pep-0273.html > Any files may be present in the zip archive, but only files *.pyc, > *.pyo and __init__.py[co] are available for import. Zip import of > *.py and dynamic modules (*.pyd, *.so) is disallowed. Why can't .py be allowed? If a more recent .py[co] (or $py.class) exists, it is used. Otherwise the .py file is compiled and discarded when the process ends. Sure, it is slower, but a zip files with only .py[co] entries would be of little use with jython. > Just as sys.path currently has default directory names, default > zip archive names are added too. Otherwise there is no way to > import all Python library files from an archive. A standard for this would be really cool. > Suppose sys.path contains "/A/B/SubDir" and "/C/D/E/Archive.zip", > and we are trying to import modfoo from the Q package. Then > import.c will generate a list of paths and extensions and will > look for the file. The list of generated paths does not change > for zip imports. Suppose import.c generates the path > "/A/B/SubDir/Q/R/modfoo.pyc". Then it will also generate the path > "/C/D/E/Archive.zip/Q/R/modfoo.pyc". Finding the SubDir path is > exactly equivalent to finding "Q/R/modfoo.pyc" in the archive. So python packages and modules can exists *only* at the top level? That would conflict somewhat with jython where at would be common to put python modules into the same .zip file as java classes and java classes also wants to own the root of a zip file. In the current implementaion in jython, we can put python modules under any path and add the zipfile!path to sys.path: sys.path.append("/path/to/zipfile.zip!Lib") which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in the archive. >Efficiency > The only way to find files in a zip archive is linear search. So > for each zip file in sys.path, we search for its names once, and > put the names plus other relevant data into a static Python > dictionary. The key is the archive name from sys.path joined with > the file name (including any subdirectories) within the archive. > This is exactly the name generated by import.c, and makes lookup > easy. [I found efficiency hard to achieve in jython because opening a zipfile in java also cause the zip index to be read into a dictionary. So we did not want to reopen a zipfile if it can be avoided. Instead we hide a reference to the opened file in sys.path so that when removing a zipfile name from sys.path, the file is eventually closed.] Would entries in the static python dict be removed when a zipfile is removed from sys.path? What is the __path__ vrbl set to in a module imported from a zipfile? Can the module make changes to __path__ and will be changes to used when importing submodules? What value should __file__ have? regards, finn From gerhard@bigfoot.de Sat Oct 27 00:42:12 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Sat, 27 Oct 2001 01:42:12 +0200 Subject: [Python-Dev] Future of SSL Message-ID: <20011027014211.A11092@lilith.hqd-internal> First I'd like to say hello. I'm new to this list :-) I've some concerns with the latest development wrt SSL development in Python. The SSL support in the Python core seems to have been added with Python 2.0, and Guido told me in a SF patch discussion, that Pythonlabs doesn't know the code very well. Meanwhile it seems like Jeremy has tried fixing bugs. I've also submitted several patches to the SSL code. As you might know, there are several third-party SSL implementatations for Python (m2crypto, pyOpenSSL, POW). I've contacted the maintainers of these packages and asked them if they'd like to review my patches. The two that answered told me that the reason they implemented their own packages was that the SSL support in Python itself was very incomplete and, to sum it up, Python's SSL is just plain broken. The current state is that there is a single ssl() method in the socket module that can be used to instantiate client-side SSL connections. There's also a patch on Sourceforge that adds server-side SSL functionality. Several points why the current API is broken: - no support for specifying the SSL protocol (SSL v2/SSL v3/SSL v2|3) - no proper certicate checking - SSL objects are neither compatible with sockets nor are they file-like objects - no advanced features like SSL callbacks (to allow the user to accept a cerficicate, for example) What I'd suggest for Python 2.2 is to *not* add any new features, like server-side SSL but only accept bugfixes for the current client-side code. As the current implementation is broken and there is probably little SSL knowledge in the Python core team, I propose to "outsource" the problem: It should be possible to define a "Python SSL interface" that describes an API for SSL. The various modules in Python that use SSL (urllib, smtp, ...) would then be rewritten to use the new API. The socketmodule.c would be rewritten to use the new API instead. Then, wrappers could be written for the various SSL modules that wrap them into the new "Python SSL interface" API. As for me, I'm not an expert in SSL, but I'd be willing to try coordinate the efforts, write a PEP, talk with the module maintainers and such. I'd be grateful to hear your opinions about this newbie proposal :-) Gerhard -- mail: gerhard bigfoot de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From guido@python.org Sat Oct 27 03:23:30 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 22:23:30 -0400 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Fri, 26 Oct 2001 22:22:05 GMT." <3bd9d419.10386565@mail.wanadoo.dk> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> Message-ID: <200110270223.WAA23737@cj20424-a.reston1.va.home.com> > > Just as sys.path currently has default directory names, default > > zip archive names are added too. Otherwise there is no way to > > import all Python library files from an archive. > > A standard for this would be really cool. Yup. > > Suppose import.c generates the path > > "/A/B/SubDir/Q/R/modfoo.pyc". Then it will also generate the path > > "/C/D/E/Archive.zip/Q/R/modfoo.pyc". Finding the SubDir path is > > exactly equivalent to finding "Q/R/modfoo.pyc" in the archive. > > So python packages and modules can exists *only* at the top level? That > would conflict somewhat with jython where at would be common to put > python modules into the same .zip file as java classes and java classes > also wants to own the root of a zip file. > > In the current implementaion in jython, we can put python modules under > any path and add the zipfile!path to sys.path: > > sys.path.append("/path/to/zipfile.zip!Lib") > > which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in > the archive. Maybe it's possible to allow "/path/to/zipfile.zip/subdirectory/etc/" in sys.path? That sounds better than picking a random new character as delimiter. > [I found efficiency hard to achieve in jython because opening a zipfile > in java also cause the zip index to be read into a dictionary. So we > did not want to reopen a zipfile if it can be avoided. Instead we hide a > reference to the opened file in sys.path so that when removing a zipfile > name from sys.path, the file is eventually closed.] I don't think Python has this problem, since we have control over the zipfile reading code. > Would entries in the static python dict be removed when a zipfile is > removed from sys.path? It can be arranged that they are removed at some later point (e.g. when sys.path is next searched). > What is the __path__ vrbl set to in a module imported from a zipfile? > Can the module make changes to __path__ and will be changes to used when > importing submodules? > > What value should __file__ have? IMO these two questions are answered by the pathname semantics that Jim proposes: __file__ = "/C/D/E/Archive.zip/Q/R/modfoo.pyc" and __path__ = ["/C/D/E/Archive.zip/Q/R"]. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sat Oct 27 03:26:38 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 22:26:38 -0400 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Sat, 27 Oct 2001 01:42:12 +0200." <20011027014211.A11092@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> Message-ID: <200110270226.WAA23767@cj20424-a.reston1.va.home.com> > What I'd suggest for Python 2.2 is to *not* add any new features, like > server-side SSL but only accept bugfixes for the current client-side > code. Sounds good to me. > As the current implementation is broken and there is probably little SSL > knowledge in the Python core team, I propose to "outsource" the problem: Thanks! We can sure use some help here. > It should be possible to define a "Python SSL interface" that describes > an API for SSL. The various modules in Python that use SSL (urllib, > smtp, ...) would then be rewritten to use the new API. The > socketmodule.c would be rewritten to use the new API instead. I've just started digging in the socketmodule.c for a different reason, and I propose to move all the SSL stuff to a separate file and module. > Then, wrappers could be written for the various SSL modules that wrap > them into the new "Python SSL interface" API. This is a good idea. The DB API works like this. > As for me, I'm not an expert in SSL, but I'd be willing to try > coordinate the efforts, write a PEP, talk with the module maintainers > and such. But we do need *an* expert, don't we? Maybe you can develop expertise as you go? > I'd be grateful to hear your opinions about this newbie proposal :-) You don't sound much like a newbie. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From petrilli@amber.org Sat Oct 27 04:55:27 2001 From: petrilli@amber.org (Christopher Petrilli) Date: Fri, 26 Oct 2001 23:55:27 -0400 Subject: [Python-Dev] Future of SSL In-Reply-To: <200110270226.WAA23767@cj20424-a.reston1.va.home.com>; from guido@python.org on Fri, Oct 26, 2001 at 10:26:38PM -0400 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> Message-ID: <20011026235527.A2919@trump.amber.org> Guido van Rossum [guido@python.org] wrote: > > Then, wrappers could be written for the various SSL modules that wrap > > them into the new "Python SSL interface" API. > > This is a good idea. The DB API works like this. I think that if OpenSSL is available, Python should build "out of the box" with SSL support. This is becomming more and more important with projects I'm working on, especially with SOAP and XML-RPC. This doesn't mean someone shouldn't be able to replace it, and we should always define an API, but... I think we need to work out of the box. > > As for me, I'm not an expert in SSL, but I'd be willing to try > > coordinate the efforts, write a PEP, talk with the module maintainers > > and such. > > But we do need *an* expert, don't we? Maybe you can develop expertise > as you go? I don't have time to provide code right now, but I do know SSL and X.509 specifically inside and out and would be happy to provide support from a standards/crypto/certificate perspective. Chris -- | Christopher Petrilli | petrilli@amber.org From guido@python.org Sat Oct 27 16:56:34 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 27 Oct 2001 11:56:34 -0400 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Fri, 26 Oct 2001 23:55:27 EDT." <20011026235527.A2919@trump.amber.org> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> Message-ID: <200110271556.LAA25312@cj20424-a.reston1.va.home.com> > I think that if OpenSSL is available, Python should build "out of the > box" with SSL support. This is becomming more and more important with > projects I'm working on, especially with SOAP and XML-RPC. This > doesn't mean someone shouldn't be able to replace it, and we should > always define an API, but... I think we need to work out of the box. Good point. That's how the SSL support is configured now, and that's how it should continue to work. (Note that, alas, the DB-API never made it this far -- we still haven't been able to find the time to do the tedious work of moving the various database adapters in the core distribution. :-( ) --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Sat Oct 27 19:41:41 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 27 Oct 2001 14:41:41 -0400 Subject: [Python-Dev] dict comps In-Reply-To: <3BD9268A.72AD71E9@lemburg.com> Message-ID: [M.-A. Lemburg] > ... > Seriously, this goes down the path of lazy evaluation of expressions. Lazy sequences specifically, but that's been a tension in Python all along (it started with the for/__getitem__ protocol, and nothing we do from here on in will ever be as much of a hack as xrange() was for that <0.9 wink>). > Not sure whether this is the right path to follow though (can cause > brain damage due to overloaded builtin debuggers, IMHO). We can introduce "L[...]" for explicitly lazy list comprehenions . > BTW, looks like I can finally get rid off the dict() builtin I > have in mxTools which is Good News ! It's not quite the same in the details: CVS dictionary(s) works just like CVS d = {} for k, v in s: d[k] = v In particular, it demands that the elements of s each produce exactly 2 objects, where IIRC the mxTools dict() requires at least 2 (ignoring any after the second). Was ignoring excess objects "a feature", or just exposing an implementation that didn't want to bother to check? From tim.one@home.com Sat Oct 27 19:55:36 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 27 Oct 2001 14:55:36 -0400 Subject: [Python-Dev] iterator support for SRE? In-Reply-To: <015a01c15e01$f99f4ef0$0900a8c0@spiff> Message-ID: +1 on finditer. [/F] > ... > will have to think about scanner; I agree that it may seem to > be vaguely useful, but cannot think of a use case that isn't > better handled by findall/split/sub or iterfind. can anyone? It's in desperate need of usage docs. The few times I recall seeing it mentioned on c.l.py, it was from someone staring at the code unable to reverse-engineer its intended use. Therefore I don't believe it's gotten a good trial, and so I'm at best -0 on making it official now. In my own stuff I've built directly on top of .lastindex/.lastgroup instead -- perhaps only because it's easier to remember how to use your own undocumented stuff than somebody else's . the-folks-most-likely-to-use-a-framework-are-those-most-likely-to- write-their-own-ly y'rs - tim From tim.one@home.com Sat Oct 27 20:45:03 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 27 Oct 2001 15:45:03 -0400 Subject: [Python-Dev] Idle speed freak looking for trouble? Message-ID: SF bug 474992 contains a simple benchmark claimed to show dramatic slowdown from 1.5.2 to 2.0, and again but less so from 2.0 to 2.1. Also major slowdown from enabling gc. I *believe* Frederic ran it on Tru64. no-time-here-ly y'rs - tim From fredrik@pythonware.com Sat Oct 27 21:45:36 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sat, 27 Oct 2001 22:45:36 +0200 Subject: [Python-Dev] Idle speed freak looking for trouble? References: Message-ID: <004501c15f28$56148340$ced241d5@hagrid> tim wrote: > SF bug 474992 contains a simple benchmark claimed to show dramatic slowdown > from 1.5.2 to 2.0, and again but less so from 2.0 to 2.1. fwiw, here's the timings I get on a slow windows box (using PythonWare builds, and time.clock() instead of os.times()): python 1.5: 2.01 python 2.0: 3.02 python 2.1: 3.00 python 2.2: 3.04 after playing some more, I ended up with this little benchmark: def dosomething(item): return [None, None, None] data = [None] * 100000 import time t1 = time.clock(); x = map(dosomething, data) t2 = time.clock() print t2 - t1 python 1.5: 1.02 seconds python 2.0: 1.95 seconds python 2.1: 1.85 seconds python 2.2: 1.90 seconds From mal@lemburg.com Sat Oct 27 21:38:03 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 27 Oct 2001 22:38:03 +0200 Subject: [Python-Dev] Idle speed freak looking for trouble? References: Message-ID: <3BDB1B2B.8705CCCE@lemburg.com> Tim Peters wrote: > > SF bug 474992 contains a simple benchmark claimed to show dramatic slowdown > from 1.5.2 to 2.0, and again but less so from 2.0 to 2.1. Also major > slowdown from enabling gc. I *believe* Frederic ran it on Tru64. ... want me to run pybench against the different versions ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Sat Oct 27 21:53:04 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 27 Oct 2001 22:53:04 +0200 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> Message-ID: <3BDB1EB0.C2A9B1FF@lemburg.com> "James C. Ahlstrom" wrote: > > The PEP for zip import is 273. Please take a look and comment. > > http://python.sourceforge.net/peps/pep-0273.html Looks good to me. Just three questions: 1. Why are .py files being outlawed ? 2. Where's the C implementation you mention in the PEP ? 3. Would it be possible to ship zlib together with Python ? (the zlib license should allow this and I don't think that the code size is too big) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Sat Oct 27 21:46:04 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 27 Oct 2001 22:46:04 +0200 Subject: [Python-Dev] dict comps References: Message-ID: <3BDB1D0C.A60F2EF@lemburg.com> Tim Peters wrote: > > [M.-A. Lemburg] > > ... > > Seriously, this goes down the path of lazy evaluation of expressions. > > Lazy sequences specifically, but that's been a tension in Python all along > (it started with the for/__getitem__ protocol, and nothing we do from here > on in will ever be as much of a hack as xrange() was for that <0.9 wink>). > > > Not sure whether this is the right path to follow though (can cause > > brain damage due to overloaded builtin debuggers, IMHO). > > We can introduce "L[...]" for explicitly lazy list comprehenions . ... and add a lazy list comprehension object (e.g. xlistcompobj) ? Cool :-) > > BTW, looks like I can finally get rid off the dict() builtin I > > have in mxTools which is Good News ! > > It's not quite the same in the details: CVS dictionary(s) works just like > CVS > > d = {} > for k, v in s: > d[k] = v > > In particular, it demands that the elements of s each produce exactly 2 > objects, where IIRC the mxTools dict() requires at least 2 (ignoring any > after the second). Was ignoring excess objects "a feature", or just > exposing an implementation that didn't want to bother to check? It was a feature; it's just that I forgot what I needed it for ;-) There are quite a few things in mxTools which I've never *really* needed. The single most used API in the package was and still is irange() which returns a sequence (i, obj[i]). dict() and invdict() are also rather popular ones. Most of the others tend to solve performance problems in some inner loops of some applications I wrote. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From fredrik@pythonware.com Sat Oct 27 22:21:56 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sat, 27 Oct 2001 23:21:56 +0200 Subject: [Python-Dev] Idle speed freak looking for trouble? Message-ID: <006201c15f2d$6b986100$ced241d5@hagrid> I wrote: > python 1.5: 1.02 seconds > python 2.0: 1.95 seconds > python 2.1: 1.85 seconds > python 2.2: 1.90 seconds doh. forgot to disable GC for 2.0 and later. with GC switched off, the difference is a bit smaller. python 1.5: 1.02 seconds python 2.0: 1.30 seconds python 2.1: 1.20 seconds python 2.2: 1.05 seconds now replace [None, None, None] with [item, item, item]: python 1.5: 0.58 seconds python 2.0: 0.85 seconds python 2.1: 0.69 seconds python 2.2: 0.71 seconds playing with other dosomething bodies (and with GC switched off), I've noticed that the following things has gotten slower from 1.5.2 to 2.2: - concatenating strings (item+item+item is 20% slower) - calling the float builtin using global lookup (20% slower) - list and tuple forming using locals (10-20% slower) From petrilli@amber.org Sun Oct 28 01:43:58 2001 From: petrilli@amber.org (Christopher Petrilli) Date: Sat, 27 Oct 2001 21:43:58 -0400 Subject: [Python-Dev] Future of SSL In-Reply-To: <200110271556.LAA25312@cj20424-a.reston1.va.home.com>; from guido@python.org on Sat, Oct 27, 2001 at 11:56:34AM -0400 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> Message-ID: <20011027214357.B6993@trump.amber.org> Guido van Rossum [guido@python.org] wrote: > > I think that if OpenSSL is available, Python should build "out of the > > box" with SSL support. This is becomming more and more important with > > projects I'm working on, especially with SOAP and XML-RPC. This > > doesn't mean someone shouldn't be able to replace it, and we should > > always define an API, but... I think we need to work out of the box. > > Good point. That's how the SSL support is configured now, and that's > how it should continue to work. Perhaps there is one of the existing modules (M2Crypto?) that can be integrated, assuming licensing issues can be resolved. Also, I think that perhaps high level abstractions should be introduced, though I'm not sure what they are now... that's just hand waving. The initial goal in my mind would be to have transparent (or nearly so) SSL session management, but the X.509 manipulation and general crypto interface could wait until later. While they are both useful, the SSL side is the really critical part. > (Note that, alas, the DB-API never made it this far -- we still > haven't been able to find the time to do the tedious work of moving > the various database adapters in the core distribution. :-( ) Actually what this looks more like is not just SSL, but a "crypto" collection for Python, dependent on the excellent work in OpenSSL. I can start outlining some stuff if that would be a good start, though obviously if there's an existing library that could be integrated, that would be excellent. Chris -- | Christopher Petrilli | petrilli@amber.org From michel@zope.com Sun Oct 28 02:02:42 2001 From: michel@zope.com (Michel Pelletier) Date: Sat, 27 Oct 2001 19:02:42 -0700 Subject: [Python-Dev] Future of SSL References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> Message-ID: <3BDB6742.E143BF8B@zope.com> Christopher Petrilli wrote: > > Actually what this looks more like is not just SSL, but a "crypto" > collection for Python, dependent on the excellent work in OpenSSL. I > can start outlining some stuff if that would be a good start, though > obviously if there's an existing library that could be integrated, > that would be excellent. AMK has such a package: http://www.amk.ca/python/code/crypto.html which served as an exellent introduction to the world of crypto for me. -Michel From guido@python.org Sun Oct 28 12:20:30 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 28 Oct 2001 07:20:30 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Sat, 27 Oct 2001 21:43:58 EDT." <20011027214357.B6993@trump.amber.org> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> Message-ID: <200110281220.HAA09692@cj20424-a.reston1.va.home.com> > The initial goal in my mind would be to have transparent (or nearly > so) SSL session management, but the X.509 manipulation and general > crypto interface could wait until later. While they are both useful, > the SSL side is the really critical part. Which is of course exactly the goal of the existing socket.ssl support, primitive though it is. This was paid for in part by HP; their goal was to let urllib support the HTTPS protocol, nothing more, nothing less. And it does that well enough. > > (Note that, alas, the DB-API never made it this far -- we still > > haven't been able to find the time to do the tedious work of moving > > the various database adapters in the core distribution. :-( ) > > Actually what this looks more like is not just SSL, but a "crypto" > collection for Python, dependent on the excellent work in OpenSSL. I > can start outlining some stuff if that would be a good start, though > obviously if there's an existing library that could be integrated, > that would be excellent. I think it would be a big mistake to start from scratch. Remember the GUI SIG? PS. One issue with adding more crypto to Python could be US export issues. It's possible that new export limitations for crypto software are made law by a congress that doesn't understand the issues, and then the US Python distribution could be in trouble (even though our site in the the Netherlands, we build the distributions here in the US). Back at CNRI, we couldn't release the SSL wrappers, which don't contain any crypto code but enable linking with it, before an extensive and expensive legal review, and then we had to wait until after a certain date, at which some of the crypto export restrictions were lifted. --Guido van Rossum (home page: http://www.python.org/~guido/) From bckfnn@worldonline.dk Sun Oct 28 14:30:30 2001 From: bckfnn@worldonline.dk (Finn Bock) Date: Sun, 28 Oct 2001 14:30:30 GMT Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110270223.WAA23737@cj20424-a.reston1.va.home.com> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <200110270223.WAA23737@cj20424-a.reston1.va.home.com> Message-ID: <3bdbfda4.3697676@mail.wanadoo.dk> [me] > In the current implementaion in jython, we can put python modules under > any path and add the zipfile!path to sys.path: > > sys.path.append("/path/to/zipfile.zip!Lib") > > which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in > the archive. [GvR] >Maybe it's possible to allow "/path/to/zipfile.zip/subdirectory/etc/" >in sys.path? That sounds better than picking a random new character >as delimiter. Fine by me. From a java POV the bang ("!") was not random: >>> import java >>> java.lang.Class.getResource(type(1), "/java/lang/Class.class") jar:file:/D:/java/jdk1.4/jre/lib/rt.jar!/java/lang/Class.class >>> The URL library in java can of course open the string above as an input stream. >> Would entries in the static python dict be removed when a zipfile is >> removed from sys.path? > >It can be arranged that they are removed at some later point >(e.g. when sys.path is next searched). An API to do this could be usefull for jython. Right now we depend on the GC thread to close the file. Since files are a limited resource it would be a good thing to have an explicit way to clean up the cached resources. I don't expect the lack of a cleanup method to be a huge problem in real life. >> What is the __path__ vrbl set to in a module imported from a zipfile? >> Can the module make changes to __path__ and will be changes to used when >> importing submodules? >> >> What value should __file__ have? > >IMO these two questions are answered by the pathname semantics >that Jim proposes: __file__ = "/C/D/E/Archive.zip/Q/R/modfoo.pyc" and >__path__ = ["/C/D/E/Archive.zip/Q/R"]. Ok. I assume that as long as some module exists with a __path__ like this, it not possible to clear the cached entries for /C/D/E/Archive.zip. If importing from zip is regarded mainly as a deployment/bootstrapping feature, then the cleanup issues does not exist. I have no problem looking at it as a deployment feature (that is all I need it for myself), I just didn't dare to put such a limit on my jython implementation. regards, finn From andymac@bullseye.apana.org.au Sun Oct 28 07:41:46 2001 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Sun, 28 Oct 2001 18:41:46 +1100 (EDT) Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: <200110252130.f9PLUM329494@odiug.zope.com> Message-ID: On Thu, 25 Oct 2001, Guido van Rossum wrote: > > At this stage, my plan is to finalise a binary release of 2.2 before > > submitting a new set of patches, taking into account Martin von Loewis' > > comments and suggestions (much appreciated thank you!) among others. I > > also plan to try and coordinate my patches with those of Michael Muller > > (473749, 474169 & 474500 so far) for the VAC++ build of Python for OS/2. > > > > I will try and review Michael's patches with a view to getting them into > > 2.2, as the VAC++ build has been part of the sourceball for a few > > versions. > > Andy, would it help if you had SourceForge commit privileges? You'd > be obliged to only check in code that doesn't break the build on other > platforms, so you would be required to test anything you want to check > in on another platform (Linux or Windows) before you commit to save > you the embarrassment of breaking the build, but if (as I expect) you > would mostly be adding stuff inside "#ifdef OS2", that shouldn't be a > big burden. It would save you the trouble of uploading patches to > SourceForge and it would save *us* the trouble of reviewing your > patches and checking them in. I've never used CVS, but am prepared to rectify that. I am also prepared to work within the commit rules you specify. I can work with the requirement for testing on another platform if FreeBSD is an acceptable test platform. I would also propose to look after patches to the VAC++ build (such as Michael Muller's), on the basis that although I can't test the patches directly, I can test that they don't affect the rest of the system, and I can work with the submitter to resolve issues that affect OS/2 specific functionality. As to whether having commit privs would help - on the time availability front, no; otherwise more likely than not. > All your code would be owned by the PSF, but that's what you want > anyway, right? Yes. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From gerhard@bigfoot.de Sun Oct 28 17:40:25 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Sun, 28 Oct 2001 18:40:25 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <3BDB6742.E143BF8B@zope.com>; from michel@zope.com on Sat, Oct 27, 2001 at 07:02:42PM -0700 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> Message-ID: <20011028184024.A1066@lilith.hqd-internal> On Sat, Oct 27, 2001 at 07:02:42PM -0700, Michel Pelletier wrote: > Christopher Petrilli wrote: > > > > Actually what this looks more like is not just SSL, but a "crypto" > > collection for Python, dependent on the excellent work in OpenSSL. I > > can start outlining some stuff if that would be a good start, though > > obviously if there's an existing library that could be integrated, > > that would be excellent. > > AMK has such a package: > > http://www.amk.ca/python/code/crypto.html It doesn't offer SSL, however. I've compiled a list of SSL libraries for Python: http://www.cs.fhm.edu/~ifw00065/pyssl/ The "Python OpenSSL wrappers" are BSD licensed and could probably serve as a good basis for a SSL library in Python itself. It's an early release, that I can crash easily if I want to ;-) Also, it would have to be modified to fit Python's coding standards (basically, the docstrings are all written in DocBook; these would have to be translated to normal text). But it's a working solution to start from. The other BSD licensed one is M2Crypto, which is around much longer, but requires SWIG to build. I think SSL support is not reason enough to require SWIG for Python builds. So this one is possibly out. The third one is LGPL licensed, so it's out immediately. Gerhard -- mail: gerhard bigfoot de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From gerhard@bigfoot.de Sun Oct 28 17:52:51 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Sun, 28 Oct 2001 18:52:51 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <20011027214357.B6993@trump.amber.org>; from petrilli@amber.org on Sat, Oct 27, 2001 at 09:43:58PM -0400 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> Message-ID: <20011028185250.B1066@lilith.hqd-internal> On Sat, Oct 27, 2001 at 09:43:58PM -0400, Christopher Petrilli wrote: > Guido van Rossum [guido@python.org] wrote: > > > I think that if OpenSSL is available, Python should build "out of the > > > box" with SSL support. This is becomming more and more important with > > > projects I'm working on, especially with SOAP and XML-RPC. This > > > doesn't mean someone shouldn't be able to replace it, and we should > > > always define an API, but... I think we need to work out of the box. > > > > Good point. That's how the SSL support is configured now, and that's > > how it should continue to work. Ok. I understand completely "outsourcing" SSL is not an option. So we either build a completely new SSL module or try to integrate an existing one. > Perhaps there is one of the existing modules (M2Crypto?) that can be > integrated, assuming licensing issues can be resolved. Yup. To save you time finding them all, I've summarized them and put up a page about them (cf. my other post). > [...] The initial goal in my mind would be to have transparent (or nearly > so) SSL session management, [...] I'm not sure I understand what you mean by transparent session management. Perhaps that one important feature would be that SSL objects be interface compatible with socket objects as much as possible? So ugly hacks like FakeSocket in httplib and SSLFakeSocket in smtplib are no longer necessary. And, btw. one complaint about socketmodule.c I've heard is that it doesn't have a C API, it might be necessary to expose some of it with the help of a header file. Gerhard -- mail: gerhard bigfoot de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From tim.one@home.com Mon Oct 29 03:14:44 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 28 Oct 2001 22:14:44 -0500 Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: Message-ID: [Andrew MacIntyre, to Guido] > I've never used CVS, but am prepared to rectify that. I am also prepared > to work within the commit rules you specify. > > I can work with the requirement for testing on another platform if FreeBSD > is an acceptable test platform. Yes, FreeBSD is fine. It's "Unix-like", and random collateral damage there is mostly what we're trying to guard against, as most people building from CVS are building on Unix-like systems. > I would also propose to look after patches to the VAC++ build (such as > Michael Muller's), on the basis that although I can't test the patches > directly, I can test that they don't affect the rest of the system, and I > can work with the submitter to resolve issues that affect OS/2 specific > functionality. That would be wonderful. I don't know what to do with OS/2 patches, although since I'm "the Windows guy", people look at me like I'm *supposed* to know something about OS/2 . I don't. If you and Michael can coordinate, it would help Python on OS/2 a lot. > As to whether having commit privs would help - on the time availability > front, no; otherwise more likely than not. It would be more time-efficient for you (and for us). Waiting for somebody else to review your patches can drag on for-- literally --months, at which point the patch likely doesn't even apply anymore. Checking in directly from a build tree saves a whole lot of time. take-it-easy-on-#ifdefs-outside-of-pyport.h-ly y'rs - tim From guido@python.org Mon Oct 29 07:30:13 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 02:30:13 -0500 Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: Your message of "Sun, 28 Oct 2001 22:14:44 EST." References: Message-ID: <200110290730.CAA11349@cj20424-a.reston1.va.home.com> > > As to whether having commit privs would help - on the time availability > > front, no; otherwise more likely than not. > > It would be more time-efficient for you (and for us). Waiting for somebody > else to review your patches can drag on for-- literally --months, at which > point the patch likely doesn't even apply anymore. Checking in directly > from a build tree saves a whole lot of time. Plus, if you screw up, you have instant review. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 08:15:37 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 03:15:37 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Sun, 28 Oct 2001 18:52:51 +0100." <20011028185250.B1066@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <20011028185250.B1066@lilith.hqd-internal> Message-ID: <200110290815.DAA11519@cj20424-a.reston1.va.home.com> > And, btw. one complaint about socketmodule.c I've heard is that it > doesn't have a C API, it might be necessary to expose some of it with > the help of a header file. What would you need from a C API? Socket objects currently are pretty thin wrappers around file descriptors. I'm not against this (although defining and using an external C API for a Python extension module is a bit cumbersome due to the requirement for dynamic loading), but I'd like to see the requirements first. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 08:19:41 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 03:19:41 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Sun, 28 Oct 2001 18:40:25 +0100." <20011028184024.A1066@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> Message-ID: <200110290819.DAA11553@cj20424-a.reston1.va.home.com> > > AMK has such a package: > > > > http://www.amk.ca/python/code/crypto.html > > It doesn't offer SSL, however. I've compiled a list of SSL libraries for > Python: > > http://www.cs.fhm.edu/~ifw00065/pyssl/ > > The "Python OpenSSL wrappers" are BSD licensed and could probably serve > as a good basis for a SSL library in Python itself. It's an early > release, that I can crash easily if I want to ;-) Also, it would have to > be modified to fit Python's coding standards (basically, the docstrings > are all written in DocBook; these would have to be translated to normal > text). But it's a working solution to start from. Do you know the author? > The other BSD licensed one is M2Crypto, which is around much longer, but > requires SWIG to build. I think SSL support is not reason enough to > require SWIG for Python builds. So this one is possibly out. It's definitely out to require SWIG to build Python from a distribution, but we may require SWIG to hack on this particular module; we could ship the SWIG output. Not ideal, but if otherwise this module is the best candidate, maybe we could live with this. > The third > one is LGPL licensed, so it's out immediately. Really? I believe the LGPL is not "viral" as the GPL. Also, there may be the possibility to shame the author into changing the license. Can you review it a bit more technically? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 08:31:16 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 03:31:16 -0500 Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: Your message of "Sun, 28 Oct 2001 18:41:46 +1100." References: Message-ID: <200110290831.DAA11598@cj20424-a.reston1.va.home.com> > > Andy, would it help if you had SourceForge commit privileges? You'd > > be obliged to only check in code that doesn't break the build on other > > platforms, so you would be required to test anything you want to check > > in on another platform (Linux or Windows) before you commit to save > > you the embarrassment of breaking the build, but if (as I expect) you > > would mostly be adding stuff inside "#ifdef OS2", that shouldn't be a > > big burden. It would save you the trouble of uploading patches to > > SourceForge and it would save *us* the trouble of reviewing your > > patches and checking them in. > > I've never used CVS, but am prepared to rectify that. Be prepared for some long evenings of fun with CVS! Lots of folks here on python-dev have developed a deep understanding of CVS theory and tricks though, so you're in a good place to learn. > I am also prepared > to work within the commit rules you specify. I didn't doubt this for a second. > I can work with the requirement for testing on another platform if FreeBSD > is an acceptable test platform. Sure is -- in fact, we could use more FreeBSD testing. :-) > I would also propose to look after patches to the VAC++ build (such as > Michael Muller's), on the basis that although I can't test the patches > directly, I can test that they don't affect the rest of the system, and I > can work with the submitter to resolve issues that affect OS/2 specific > functionality. That sounds like a very good idea. Right now I don't really know what to do with the two sets of OS/2 patches... Since Michael is cc'ed here, Michael, what do you think of this? > As to whether having commit privs would help - on the time availability > front, no; otherwise more likely than not. OK. As soon as you think you're ready to commit anything (try something small first :-), post a message here and one of the admins will add you. (You need to have a SF login first.) Not me, I'm about to go underground for a while to take care of my wife and son (to be born by Friday). > > All your code would be owned by the PSF, but that's what you want > > anyway, right? > > Yes. Good! The PSF legal team will eventually come up with paperwork to make this official; in the mean time, this archived email message is good enough for me. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 08:37:00 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 03:37:00 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Sun, 28 Oct 2001 14:30:30 GMT." <3bdbfda4.3697676@mail.wanadoo.dk> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <200110270223.WAA23737@cj20424-a.reston1.va.home.com> <3bdbfda4.3697676@mail.wanadoo.dk> Message-ID: <200110290837.DAA11621@cj20424-a.reston1.va.home.com> > [GvR] > > >Maybe it's possible to allow "/path/to/zipfile.zip/subdirectory/etc/" > >in sys.path? That sounds better than picking a random new character > >as delimiter. [Finn] > Fine by me. From a java POV the bang ("!") was not random: > > >>> import java > >>> java.lang.Class.getResource(type(1), "/java/lang/Class.class") > jar:file:/D:/java/jdk1.4/jre/lib/rt.jar!/java/lang/Class.class > >>> > > The URL library in java can of course open the string above as an input > stream. I guess the advantage of this notation is that it makes a simple check for "file inside zip archive" possible; I sort of like that. The downside is that it limits the use of "!" for other reasons in pathnames (which seems a mostly but not entirely theoretical problem). > I assume that as long as some module exists with a __path__ like this, > it not possible to clear the cached entries for /C/D/E/Archive.zip. Good point. > If importing from zip is regarded mainly as a deployment/bootstrapping > feature, then the cleanup issues does not exist. I have no problem > looking at it as a deployment feature (that is all I need it for > myself), I just didn't dare to put such a limit on my jython > implementation. I don't see sys.path as a very dynamic thing anyway. It gets manipulated briefly at the beginning, and then maybe, rarely, stuff gets added during the program run. I've seen temporary additions, but these were usually in the test suite. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Mon Oct 29 08:36:41 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 29 Oct 2001 09:36:41 +0100 Subject: [Python-Dev] Future of SSL References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> Message-ID: <3BDD1519.31496FB@lemburg.com> [Integrating more crypto code into Python] Please don't proceed in this direction. Crypto code has a long tradition of making world-wide distribution hard and painful, either to export, to import or to usage restrictions. There's a good reason why e.g. mxCrypto was developed outside the US, in fact I wrote that package because I couldn't legally export Andrew's code from the US. OpenSSL itself was developed in large parts in Australia for much the same reason. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Mon Oct 29 08:49:22 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 29 Oct 2001 09:49:22 +0100 Subject: [Python-Dev] Adding DB modules to the core ? (Re: Future of SSL) References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> Message-ID: <3BDD1812.1FEF2E33@lemburg.com> Guido van Rossum wrote: > > (Note that, alas, the DB-API never made it this far -- we still > haven't been able to find the time to do the tedious work of moving > the various database adapters in the core distribution. :-( ) Do you really think that this would be the right approach ? Sounds like code bloat to me... BTW, I'm still waiting for one of the PSF sponsors to create the mythical Sumo distribution of Python. Any news in that direction ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From jack@oratrix.nl Mon Oct 29 10:02:04 2001 From: jack@oratrix.nl (Jack Jansen) Date: Mon, 29 Oct 2001 11:02:04 +0100 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by Guido van Rossum , Fri, 26 Oct 2001 14:56:20 -0400 , <200110261856.OAA20424@cj20424-a.reston1.va.home.com> Message-ID: <20011029100204.E0766303181@snelboot.oratrix.nl> > > While this may happen in principle, and out of curiosity: Was this a > > problem in 2.2b1 also? Looking at the changes made on the 2.2b1 > > release branch, I see that a total of 7 files was changed. Except for > > patchlevel.h, and the \n\ fix on socketmodule.c, there were no changes > > to C code. > > What's more common is actually a flurry of checkins just *before* the > release branch is made (say in the last two days). These often > include a somewhat hasty commit of a project that's valuable to have > but introduces new bugs -- either Mac-specific or generic. The patch > that introduced the missing \n\ problem was an example of this. And, more serious, there's another example for the 2.2b1 release that I found just friday (so this bug is going to stay in MacPython 2.2b1, I'm afraid): importing packages from frozen programs is broken. There was a moderately simple fix to import.c, to test that submodules of frozen modules were also frozen, and not picked up from the filesystem. Unfortunately whoever did the patch wasn't aware that MacPython frozen modules work different, and hence broke it. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From jack@oratrix.nl Mon Oct 29 10:10:16 2001 From: jack@oratrix.nl (Jack Jansen) Date: Mon, 29 Oct 2001 11:10:16 +0100 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by Guido van Rossum , Fri, 26 Oct 2001 14:56:20 -0400 , <200110261856.OAA20424@cj20424-a.reston1.va.home.com> Message-ID: <20011029101016.E80A6303181@snelboot.oratrix.nl> > > It's just that I think very strict principles must be applied for the > > final release. If that means that the 2.2 release can't go without ok > > from you (or somebody else who has produced MacOS 9 binaries), I think > > we should add that to the release procedure. That check would be to > > avoid show-stopper bugs only, of course - anything complex needs to be > > detected long before the final release. > > I think that for the final release we should definitely attempt this. > Hopefully the speed and quality of change will reduce closer to the > final release. Also, the release candidate should help. For the final release I agree that this is a good idea. Let's try and make it so. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From just@letterror.com Mon Oct 29 10:24:33 2001 From: just@letterror.com (Just van Rossum) Date: Mon, 29 Oct 2001 11:24:33 +0100 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: <20011029100204.E0766303181@snelboot.oratrix.nl> Message-ID: <20011029112439-r01010800-90687d33-0910-010c@10.0.0.23> Jack Jansen wrote: > And, more serious, there's another example for the 2.2b1 release that I found > just friday (so this bug is going to stay in MacPython 2.2b1, I'm afraid): > importing packages from frozen programs is broken. > > There was a moderately simple fix to import.c, to test that submodules of > frozen modules were also frozen, and not picked up from the filesystem. > Unfortunately whoever did the patch wasn't aware that MacPython frozen modules > work different, and hence broke it. I remember seeing that, and meaning to look into it but forgot. I think it's a highly questionable fix, not only for MacPython: I'm not sure if this still allows C extensions as submodules in frozen apps. Just From chimaadams@yahoo.com Mon Oct 29 10:42:53 2001 From: chimaadams@yahoo.com (chima adams) Date: Mon, 29 Oct 2001 11:42:53 +0100 Subject: [Python-Dev] (no subject) Message-ID: FROM THE DESK:Dr.CHIMA ADAMS, ACCOUNT DEPARTMENT (NNPC) FALOMO OFFICE COMPLEX, IKOYI – LAGOS, NIGERIA TEL:234-8033084103 FAX: 234-1-7599186 ATTN: GENERAL MANAGER Dear Sir, I am Dr.Chima Adams chairman- audit committee, with the Nigerian National Petroleum Corporation (NNPC), I headed the audit committee whose assignment was to audit, review and recommend payment for all contracts awarded in the NNPC by the corrupt past military administration. In the course of our review, we discovered that the sum of USD 30 million (THIRTY MILLION DOLLARS) was floating unclaimed in the corporation account without beneficiary. This money emanated from an over-invoicing in the contract awarded to a foreign firm for the supply, fixing, erecting and computerized optimization of oil pipeline to one of our oil refineries during the past Military administration. Base on our status as senior government officials in top sensitive position of trust, we cannot stand to claim this money ourselves, hence, I decided to contact you. All modalities for this transfer have been fully worked out. We have already secured a preliminary order to effect this payment, we now require from you your BANKING PARTICULARS, FAX AND TELEPHONE NUMBER for smooth transfer of this money into your bank account. We intend to share the money in the following manner: 30% for you, 65% for us and 5% for any miscellaneous expenses. This business is 100% risk-free. We look forward to a fruitful business relationship with you. Best Regards Dr. Chima Adams Account Department (NNPC) _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From fredrik@pythonware.com Mon Oct 29 11:28:32 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 29 Oct 2001 12:28:32 +0100 Subject: [Python-Dev] (no subject) References: Message-ID: <005f01c1606c$d8dc1140$ced241d5@hagrid> > In the course of our review, we discovered that the sum of > USD 30 million (THIRTY MILLION DOLLARS) was floating un- > claimed in the corporation account without beneficiary. > We intend to share the money in the following manner: 30% > for you, 65% for us and 5% for any miscellaneous expenses. $9 million to the PSF? not bad. maybe we should hand them over to VA Linux, to make sure sourceforge stays up another year or two. From gerhard@bigfoot.de Mon Oct 29 12:36:40 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Mon, 29 Oct 2001 13:36:40 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <3BDD1519.31496FB@lemburg.com>; from mal@lemburg.com on Mon, Oct 29, 2001 at 09:36:41AM +0100 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> Message-ID: <20011029133640.A1804@lilith.hqd-internal> On Mon, Oct 29, 2001 at 09:36:41AM +0100, M.-A. Lemburg wrote: > [Integrating more crypto code into Python] > > Please don't proceed in this direction. Crypto code has a long > tradition of making world-wide distribution hard and painful, either > to export, to import or to usage restrictions. True. But there's not much you can do about stupid governments, except not voting for them next time. > There's a good reason why e.g. mxCrypto was developed outside the US, > in fact I wrote that package because I couldn't legally export > Andrew's code from the US. OpenSSL itself was developed in large parts > in Australia for much the same reason. The problem is that there needs to be some crypto support so that HTTPS, SMTP over TLS, etc. works if the crypto is available. So, at least the crypto hooks (SSL API) must be there, right? What would be your suggestions? Would you prefer to go in the direction of my original proposal - only providing a SSL API, but not the implementation? Gerhard -- mail: gerhard bigfoot de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From guido@python.org Mon Oct 29 12:54:22 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 07:54:22 -0500 Subject: [Python-Dev] Adding DB modules to the core ? (Re: Future of SSL) In-Reply-To: Your message of "Mon, 29 Oct 2001 09:49:22 +0100." <3BDD1812.1FEF2E33@lemburg.com> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <3BDD1812.1FEF2E33@lemburg.com> Message-ID: <200110291254.HAA12811@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > > > (Note that, alas, the DB-API never made it this far -- we still > > haven't been able to find the time to do the tedious work of moving > > the various database adapters in the core distribution. :-( ) > > Do you really think that this would be the right approach ? > Sounds like code bloat to me... It depends. This is definitely a frequently requested feature, and that's what I go by... Plus, I know that the various DB packages (mxODBC notwithstanding :-) are of, eh, varying quality. > BTW, I'm still waiting for one of the PSF sponsors to create > the mythical Sumo distribution of Python. Any news in that > direction ? I don't recall either of the sponsors promising, although I would say that ActivePython might be a good start. I remember one of the regular members promising this, but apparently his fork was larger than his mouth (as we say in Holland). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 12:56:15 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 07:56:15 -0500 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Your message of "Mon, 29 Oct 2001 11:02:04 +0100." <20011029100204.E0766303181@snelboot.oratrix.nl> References: <20011029100204.E0766303181@snelboot.oratrix.nl> Message-ID: <200110291256.HAA12835@cj20424-a.reston1.va.home.com> > There was a moderately simple fix to import.c, to test that > submodules of frozen modules were also frozen, and not picked up > from the filesystem. Unfortunately whoever did the patch wasn't > aware that MacPython frozen modules work different, and hence broke > it. Guilty as charged. I didn't write it but approved it and checked it in. But that's what beta testing is for... --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 13:06:28 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 08:06:28 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Mon, 29 Oct 2001 13:36:40 +0100." <20011029133640.A1804@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> Message-ID: <200110291306.IAA12932@cj20424-a.reston1.va.home.com> > True. But there's not much you can do about stupid governments, except > not voting for them next time. Doesn't seem to help in the US. :-( > > There's a good reason why e.g. mxCrypto was developed outside the US, > > in fact I wrote that package because I couldn't legally export > > Andrew's code from the US. OpenSSL itself was developed in large parts > > in Australia for much the same reason. > > The problem is that there needs to be some crypto support so that HTTPS, > SMTP over TLS, etc. works if the crypto is available. So, at least the > crypto hooks (SSL API) must be there, right? > > What would be your suggestions? Would you prefer to go in the direction > of my original proposal - only providing a SSL API, but not the > implementation? Yes, that's how the current SSL support works -- you need to link in openssl. --Guido van Rossum (home page: http://www.python.org/~guido/) From gerhard@bigfoot.de Mon Oct 29 13:18:45 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Mon, 29 Oct 2001 14:18:45 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <200110291306.IAA12932@cj20424-a.reston1.va.home.com>; from guido@python.org on Mon, Oct 29, 2001 at 08:06:28AM -0500 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> Message-ID: <20011029141844.A1967@lilith.hqd-internal> On Mon, Oct 29, 2001 at 08:06:28AM -0500, Guido van Rossum wrote: > > [me, in response to a remark from Marc-André] > > What would be your suggestions? Would you prefer to go in the direction > > of my original proposal - only providing a SSL API, but not the > > implementation? > > Yes, that's how the current SSL support works -- you need to link in > openssl. So, as long as there are no actual cryptographic algorithms in the Python source tree, but only hooks for OpenSSL, there's no problem? If this is the case, then there would only be an issue for binary distributions of Python, which can be built easily in free countries :-) Gerhard -- mail: gerhard bigfoot de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From jack@oratrix.nl Mon Oct 29 14:07:14 2001 From: jack@oratrix.nl (Jack Jansen) Date: Mon, 29 Oct 2001 15:07:14 +0100 Subject: [Python-Dev] PEP 273 - inporting from zipfiles Message-ID: <20011029140714.81E46303181@snelboot.oratrix.nl> Jim, for your implementation you might want to have a look at the Macintosh-specific code in import.c (and possibly at related code in Mac/macglue.c, but that's probably less relevant). MacPython can already have files on sys.path, and there's code to look through these files for PYC resources. Don't worry what they are or how it's done, but they contain compiled modules. I got a pretty significant increase of import speed is when I added a cache of which sys.path entries were files and which were directories, so for each sys.path entry you know that you can skip one leg of the find_module() code. This is all in import.c, so you could probably enable it verbatim for zipfile imports. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | ++++ see http://www.xs4all.nl/~tank/ ++++ From jim@interet.com Mon Oct 29 14:19:52 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 09:19:52 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> Message-ID: <3BDD6588.7CFEA7D6@interet.com> Guido van Rossum wrote: > Hm, I wonder if we couldn't just link with the libz.a C library and > use the C interface, if you're implementing this in C anyway. Since zlib is often in a DLL or shared library (.so), I am afraid of portability problems if we use the C interface. For example, on Windows, linking python.dll with zlib.dll would cause python.dll to fail if zlib.dll were unavailable, even if zlib.dll were never accessed. It also happens that use of the zlib module is easier. > I wonder if we need to make an additional rule that allows a .pyc file > to satisfy a module request even if we're in optimized mode (where > normally only .pyo files are searched). Otherwise, if someone ships a > zipfile with only .pyc files, their modules can't be imported at all > when python -O is used. Yes, I agree. How about if we look for the correct .py[co] first, and if that fails, look for the other? Either will satisfy the import, right? If .pyc is wanted, .pyo is OK too. > | The only way to find files in a zip archive is linear search. > > But there's an index record at the end that provides quick access. It is this index which is searched linearly. > Do we get an "module not found" error or something better, like > "compressed module found as but zlib unavailable"? We get "module not found". The second is awkward, because the module may be found later in sys.path. > Well, this is the domain of getpath.c, and that's got a different > implementation for Unix and Windows anyway (Windows has PC/getpathp.c). I would like discussion on what the additional sys.path names are. > | A C implementation exists which works, but which can be made better. I can upload it for inspection, but I don't want it to be a patch because it is not done yet. JimA From mal@lemburg.com Mon Oct 29 14:20:25 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 29 Oct 2001 15:20:25 +0100 Subject: [Python-Dev] Future of SSL References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> Message-ID: <3BDD65A9.BB2988A3@lemburg.com> Gerhard H=E4ring wrote: >=20 > On Mon, Oct 29, 2001 at 09:36:41AM +0100, M.-A. Lemburg wrote: > > [Integrating more crypto code into Python] > > > > Please don't proceed in this direction. Crypto code has a long > > tradition of making world-wide distribution hard and painful, either > > to export, to import or to usage restrictions. >=20 > True. But there's not much you can do about stupid governments, except > not voting for them next time. I think it's more difficult than that... this is about politics and can be a very touchy subject depending on at which angle you look=20 at it. > > There's a good reason why e.g. mxCrypto was developed outside the US, > > in fact I wrote that package because I couldn't legally export > > Andrew's code from the US. OpenSSL itself was developed in large part= s > > in Australia for much the same reason. >=20 > The problem is that there needs to be some crypto support so that HTTPS= , > SMTP over TLS, etc. works if the crypto is available. So, at least the > crypto hooks (SSL API) must be there, right? >=20 > What would be your suggestions? Would you prefer to go in the direction > of my original proposal - only providing a SSL API, but not the > implementation? Note that even the hooks can cause legal problems, because some (countries) consider the hooks as guns without amunition. I'd suggest to leave serious crypto completely out of Python and to use one of the available third-party toolkits in case it should become necessary, e.g. M2Crypto has been around for a while and probably provides all that's necessary to do HTTPS or other SSL/TLS-based protocol and amkCrypto provides all the low-level tools to write your own secure procotols. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From gmcm@hypernet.com Mon Oct 29 14:30:30 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 09:30:30 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BD9B5EC.119319A5@interet.com> Message-ID: <3BDD21B6.12124.38E143B8@localhost> Jim Ahlstrom wrote: [From PEP 273] > Currently, sys.path is a list of directory names as strings. A nit: that's in practice, but not by definition (a couple std modules have been patched to allow other things on sys.path). After extensive use of imputil (which puts objects on sys.path), I think we might as well make it official that sys.path is a list of strings. [Subdirectory Equivalence] This is a bit misleading - it can be read as implying that the importable modules are in a flat namespace. They're not. To get R.Q.modfoo imported, R.__init__ and R.Q.__init__ must be imported. The __init__ modules have the opportunity to play with __path__ to break the equivalence of R.Q.modfoo to R/Q/modfoo.py. Or (more likely), play games with attributes so that Q is, say, an instance, or maybe a module imported from someplace else. Question: if Archive.zip/Q/__init__.pyc does "__path__.append('some/real/directory')", will it work? [dynamic libs] > It might be possible to extract the dynamic module from the > zip file, write it to a plain file and load it. I think you can nail the door shut on this one. On many OSes, making dynamic libs available to a process requires settings that can only (sanely) be made before the process starts. OTOH, it's common practice these days to put dynamic libs inside packages. That needs to be dealt with at runtime, and at build time (since it breaks the expectation that you can just "zip up" the package). [no *.py files] > You also can't import source files *.py from a zip archive. Apparently some Linuxes / RPM distributions don't deliver .pyc's or .pyo's. Since they're installed as root and run as some-poor-user, I'm afraid there are quite a few installations running off .py's all the time. So while it's definitely sub- optimal, I'm not sure it should be outlawed. [Guido] > But it would still be good if the .py files were also in the > zip file, so the source can be used in tracebacks etc. As you probably remember me saying before, I think this is a very minor nice-to-have. For one thing, you've probably done enough testing before stuffing your code into an archive that you're past the point of looking at a traceback and going "doh!". Most of the time you'll need to look at the code anyway. OTOH, this [more from Guido]: > A C API to get a source line from a filename might be a good > idea (plus a Python API). points the way towards something I'm very much in favor of: deferring things to that-which-did-the-importing. [Efficiency] > The key is the archive name from sys.path joined with the > file name (including any subdirectories) within the archive. DIfferent spellings of the same path are possible in a filesystem, but not in a dictionary. A bit of "harmless" tweaking of sys.path could render an archive unreachable. [zlib must be available at start] I'll agree, and agree with Guido that the coolest thing would be to make zlib standard. [Booting - python.zip should be part of the generated sys.path] Agree. Nice and straightforward. Now, from the discussion: [restate Jim] sys.path contains /C/D/E/Archive.zip and Archive.zip contains "Q/R/modfoo.pyc" so "import Q.R.modfoo" is satisfied by /C/D/E/Archive.zip/Q/R/modfoo.pyc [restate Finn] Jython has sys.path /C/D/E/Archive.zip!Lib and Archive.zip has "Lib/Q/R/modfoo.pyc" so "import Q.R.modfoo" is satisfied by /C/D/E/Archive.zip/Lib/Q/R/modfoo.pyc [restate Guido] Why not use /C/D/E/Archive.zip/Lib on sys.path? I use embedded archives. So sys.path will have an entry like: "/path/to/executable?84758" which says that the archive starts at position 84758 in the file "executable". Anything beyond the simple approach Jim takes gets into some very URL-ish territory. That's fine by me :-). I don't really like the idea of hacking special knowledge of zip files into import.c (which is already a specialist in filesystems). Like Finn said, if this is a deployment issue (we want zip files now, and are willing to live with strict limitations / rules to get it), then OK (as long as it supports __path__ and some way of dealing with dynamic libs in packages). Personally, I think package support stretched import.c to it's monolithic limits and it's high time the code was refactored to make it sanely extensible. - Gordon From jim@interet.com Mon Oct 29 14:35:07 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 09:35:07 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> Message-ID: <3BDD691B.2599AAF4@interet.com> Finn Bock wrote: > Why can't .py be allowed? If a more recent .py[co] (or $py.class) > exists, it is used. Otherwise the .py file is compiled and discarded > when the process ends. Sure, it is slower, but a zip files with only > .py[co] entries would be of little use with jython. The *.py are allowed to be in the file, and jpython can use them. In fact, any files at all can be in the archive. It is just that C-Python ignores them. My reason was that if *.py[co] are missing or out of date, zip importing will be slow and users won't figure out what is wrong. But I am open to changing this. > > Just as sys.path currently has default directory names, default > A standard for this would be really cool. Yes, lets make a standard now. > So python packages and modules can exists *only* at the top level? That > would conflict somewhat with jython where at would be common to put > python modules into the same .zip file as java classes and java classes > also wants to own the root of a zip file. > > In the current implementaion in jython, we can put python modules under > any path and add the zipfile!path to sys.path: > > sys.path.append("/path/to/zipfile.zip!Lib") > > which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in > the archive. I am confused. Zip archives are equivalant to subdirectories, so there is no requirement to have anything at the top level. Your example seems to imply a second search path besides sys.path. BTW, the code uses ".zip" as the archive flag, not a special character '!'. > [I found efficiency hard to achieve in jython because opening a zipfile > in java also cause the zip index to be read into a dictionary. So we > did not want to reopen a zipfile if it can be avoided. Instead we hide a > reference to the opened file in sys.path so that when removing a zipfile > name from sys.path, the file is eventually closed.] > > Would entries in the static python dict be removed when a zipfile is > removed from sys.path? No, entries would not be removed. But they would not be found either, because their names would not be generated from the new sys.path. > What is the __path__ vrbl set to in a module imported from a zipfile? > Can the module make changes to __path__ and will be changes to used when > importing submodules? > > What value should __file__ have? The __file__ is /A/B/archive.zip/name.py. There is no special code for __file__ nor __path__, the path name just has a ".zip" in it. JimA From guido@python.org Mon Oct 29 14:34:58 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 09:34:58 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Mon, 29 Oct 2001 14:18:45 +0100." <20011029141844.A1967@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> <20011029141844.A1967@lilith.hqd-internal> Message-ID: <200110291434.JAA13268@cj20424-a.reston1.va.home.com> > > > [me, in response to a remark from Marc-André] > > > What would be your suggestions? Would you prefer to go in the direction > > > of my original proposal - only providing a SSL API, but not the > > > implementation? > > > > Yes, that's how the current SSL support works -- you need to link in > > openssl. > > So, as long as there are no actual cryptographic algorithms in the > Python source tree, but only hooks for OpenSSL, there's no problem? That's what I believe the situation currently, for open source software. It wasn't always like this -- at some point, hooks were enough to need export permission. It may again be like this, if some uninformed US senators get their way... > If this is the case, then there would only be an issue for binary > distributions of Python, which can be built easily in free countries :-) Hm, but that's dangerous too -- someone could easily build an RPM that includes openssl without realizing it. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Mon Oct 29 14:45:08 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 09:45:08 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <200110270223.WAA23737@cj20424-a.reston1.va.home.com> <3bdbfda4.3697676@mail.wanadoo.dk> Message-ID: <3BDD6B74.D4E19953@interet.com> Finn Bock wrote: > >> Would entries in the static python dict be removed when a zipfile is > >> removed from sys.path? > > > >It can be arranged that they are removed at some later point > >(e.g. when sys.path is next searched). > > An API to do this could be usefull for jython. Right now we depend on > the GC thread to close the file. Since files are a limited resource it > would be a good thing to have an explicit way to clean up the cached > resources. The static Python dictionary is a memory object which uses no open file descriptors. If an element of sys.path contains ".zip" and hasn't been seen before, the zip archive is opened, searched, and closed. The key is the name, the value includes the archive name and offset. Thereafter, the zip archive is never opened unless it contains a needed file. So I don't think there is a problem. > myself), I just didn't dare to put such a limit on my jython > implementation. I don't understand the jpython implementation, so please point out all problems so we can fix them now. JimA From gmcm@hypernet.com Mon Oct 29 14:50:30 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 09:50:30 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: <200110291434.JAA13268@cj20424-a.reston1.va.home.com> References: Your message of "Mon, 29 Oct 2001 14:18:45 +0100." <20011029141844.A1967@lilith.hqd-internal> Message-ID: <3BDD2666.4196.38F3926B@localhost> I wrote some proprietary code for a client, using (hacked) M2Crypto and openSSL, but using only fairly lightweight DES (not even DES3). In order to get an export license, I had to talk to the Commerce Dept and the NSA. That got the export license, but to get it into France, the client had to talk extensively to the French government. My guess is that anything beyond enabling https in core Python will make negotiating with the FSF look like a Mardi Gras party. - Gordon From mmuller@enduden.com Mon Oct 29 14:35:40 2001 From: mmuller@enduden.com (Michael Muller) Date: Mon, 29 Oct 2001 09:35:40 -0500 Subject: [Python-Dev] status of OS/2 ports of Python Message-ID: <193.1004366140.804197.425672245@lilbastard> Guido van Rossum wrote: > > I would also propose to look after patches to the VAC++ build (such as > > Michael Muller's), on the basis that although I can't test the patches > > directly, I can test that they don't affect the rest of the system, and I > > can work with the submitter to resolve issues that affect OS/2 specific > > functionality. > > That sounds like a very good idea. Right now I don't really know what > to do with the two sets of OS/2 patches... Since Michael is cc'ed > here, Michael, what do you think of this? This works for me. As I've said in the notes on my patches, there is some "textual" conflict between Andrews patches and mine (meaning that my patches won't work after his have been applied). However, I see no functional conflicts, and as I've said I'd be happy to create new patches that will work with his. Also, Andrew: I have a lot of experience with CVS and I'm willing to offer any help you need. I can also test changes on Linux. My only wish is that this be resolved quickly so I don't have to keep revising my patches to keep up with the (rapidly evolving) baseline ;-). ============================================================================= michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- In this book it is spoken of the Sephiroth, and the Paths, of Spirits and Conjurations; of Gods, Spheres, Planes and many other things which may or may not exist. It is immaterial whether they exist or not. By doing certain things certain results follow. - Aleister Crowley ============================================================================= From jim@interet.com Mon Oct 29 14:53:26 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 09:53:26 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <3BDB1EB0.C2A9B1FF@lemburg.com> Message-ID: <3BDD6D66.675A4F3C@interet.com> "M.-A. Lemburg" wrote: > 1. Why are .py files being outlawed ? My reason was that if *.py[co] are missing or out of date, zip importing will be slow and users won't figure out what is wrong. Generally I favor user-proof features over expert features. I prefer things which either "Just Work" or fail spectacularly. But I am open to changing this. > 2. Where's the C implementation you mention in the PEP ? Software is like pancakes, you should always throw the first one away. I will post it if you want, but it is not done. > 3. Would it be possible to ship zlib together with Python ? > (the zlib license should allow this and I don't think that > the code size is too big) OK by me. But uncompressed zip archives work, and may even be faster than conpressed archives. JimA From guido@python.org Mon Oct 29 14:58:39 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 09:58:39 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 09:19:52 EST." <3BDD6588.7CFEA7D6@interet.com> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> Message-ID: <200110291458.JAA13956@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > > Hm, I wonder if we couldn't just link with the libz.a C library and > > use the C interface, if you're implementing this in C anyway. > > Since zlib is often in a DLL or shared library (.so), I am afraid of > portability problems if we use the C interface. For example, on Windows, > linking python.dll with zlib.dll would cause python.dll to fail if zlib.dll > were unavailable, even if zlib.dll were never accessed. It also happens > that use of the zlib module is easier. OK, good enough. (I'm getting more and more curious about your implementation. ;-) > > I wonder if we need to make an additional rule that allows a .pyc file > > to satisfy a module request even if we're in optimized mode (where > > normally only .pyo files are searched). Otherwise, if someone ships a > > zipfile with only .pyc files, their modules can't be imported at all > > when python -O is used. > > Yes, I agree. How about if we look for the correct .py[co] first, > and if that fails, look for the other? Either will satisfy the > import, right? If .pyc is wanted, .pyo is OK too. Sounds good to me -- this might be a good rule anyhoo. > > | The only way to find files in a zip archive is linear search. > > > > But there's an index record at the end that provides quick access. > > It is this index which is searched linearly. That can't possibly take any time compared to other stuff -- it's a relatively short in-memory table. > > Do we get an "module not found" error or something better, like > > "compressed module found as but zlib unavailable"? > > We get "module not found". The second is awkward, because the > module may be found later in sys.path. Hm. Does it at least spit out a warning when zlib is needed but not available? It can be awkward to debug the situation where you get a module from the filesystem rather than the version from the zipfile, even though it is in the zipfile. While the compression method is listed in the zip info, it's not the first thing someone would look for unless they were aware that this failure mode existed. Since obviously the intention of putting the module in the zipfile was that it should be found, I think that failure to decompress should be turned into an immediate error -- the same way as a syntax error gets reported and not turned into a "skip to the next directory in sys.path" effect. > > Well, this is the domain of getpath.c, and that's got a different > > implementation for Unix and Windows anyway (Windows has PC/getpathp.c). > > I would like discussion on what the additional sys.path names are. Well, propose some. > > | A C implementation exists which works, but which can be made better. > > I can upload it for inspection, but I don't want it to be a patch > because it is not done yet. Just say so in the patch. You can upoad anything you want. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 15:06:59 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 10:06:59 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 09:30:30 EST." <3BDD21B6.12124.38E143B8@localhost> References: <3BDD21B6.12124.38E143B8@localhost> Message-ID: <200110291506.KAA14000@cj20424-a.reston1.va.home.com> > Jim Ahlstrom wrote: > [From PEP 273] > > Currently, sys.path is a list of directory names as strings. [GMcM] > A nit: that's in practice, but not by definition (a couple std > modules have been patched to allow other things on > sys.path). After extensive use of imputil (which puts objects > on sys.path), I think we might as well make it official that > sys.path is a list of strings. Interesting. So you think imputil is wrong to put objects there? Why? (Not arguing, just interested in your experience.) > [Subdirectory Equivalence] > > This is a bit misleading - it can be read as implying that the > importable modules are in a flat namespace. They're not. To > get R.Q.modfoo imported, R.__init__ and R.Q.__init__ must > be imported. The __init__ modules have the opportunity to > play with __path__ to break the equivalence of R.Q.modfoo to > R/Q/modfoo.py. Or (more likely), play games with attributes > so that Q is, say, an instance, or maybe a module imported > from someplace else. > > Question: if Archive.zip/Q/__init__.pyc does > "__path__.append('some/real/directory')", will it work? It should. > [dynamic libs] > > It might be possible to extract the dynamic module from the > > zip file, write it to a plain file and load it. > > I think you can nail the door shut on this one. On many OSes, > making dynamic libs available to a process requires settings > that can only (sanely) be made before the process starts. ANd I believe some systems require shared libraries to be owned by root. > OTOH, it's common practice these days to put dynamic libs > inside packages. That needs to be dealt with at runtime, and > at build time (since it breaks the expectation that you can just > "zip up" the package). Yes. This is important. > [no *.py files] > > You also can't import source files *.py from a zip archive. > > Apparently some Linuxes / RPM distributions don't deliver > .pyc's or .pyo's. Since they're installed as root and run as > some-poor-user, I'm afraid there are quite a few installations > running off .py's all the time. So while it's definitely sub- > optimal, I'm not sure it should be outlawed. Heh, this is an argument for Jim's position -- it would have come out during testing that way, since the imports would fail. ;-) > [Guido] > > But it would still be good if the .py files were also in the > > zip file, so the source can be used in tracebacks etc. > > As you probably remember me saying before, I think this is a > very minor nice-to-have. For one thing, you've probably done > enough testing before stuffing your code into an archive that > you're past the point of looking at a traceback and going > "doh!". Most of the time you'll need to look at the code > anyway. Testing is for wimps. :-) > OTOH, this [more from Guido]: > > A C API to get a source line from a filename might be a good > > idea (plus a Python API). > > points the way towards something I'm very much in favor of: > deferring things to that-which-did-the-importing. Yup. > [Efficiency] > > The key is the archive name from sys.path joined with the > > file name (including any subdirectories) within the archive. > > DIfferent spellings of the same path are possible in a > filesystem, but not in a dictionary. A bit of "harmless" > tweaking of sys.path could render an archive unreachable. Hm, wouldn't the archive just be opened a second time? Or do I misundestand you? > [zlib must be available at start] > I'll agree, and agree with Guido that the coolest thing would be > to make zlib standard. But we'd have to make sure it's statically linked. (Fortunately, we already link it statically on Windows.) > [Booting - python.zip should be part of the generated sys.path] > Agree. Nice and straightforward. > > Now, from the discussion: > > [restate Jim] > sys.path contains /C/D/E/Archive.zip > and Archive.zip contains "Q/R/modfoo.pyc" > so "import Q.R.modfoo" is satisfied by > /C/D/E/Archive.zip/Q/R/modfoo.pyc > > [restate Finn] > Jython has sys.path /C/D/E/Archive.zip!Lib > and Archive.zip has "Lib/Q/R/modfoo.pyc" > so "import Q.R.modfoo" is satisfied by > /C/D/E/Archive.zip/Lib/Q/R/modfoo.pyc > > [restate Guido] > Why not use /C/D/E/Archive.zip/Lib on sys.path? > > I use embedded archives. So sys.path will have an entry like: > "/path/to/executable?84758" which says that the archive > starts at position 84758 in the file "executable". > > Anything beyond the simple approach Jim takes gets into > some very URL-ish territory. That's fine by me :-). > > I don't really like the idea of hacking special knowledge of zip > files into import.c (which is already a specialist in > filesystems). Like Finn said, if this is a deployment issue (we > want zip files now, and are willing to live with strict limitations / > rules to get it), then OK (as long as it supports __path__ and > some way of dealing with dynamic libs in packages). > > Personally, I think package support stretched import.c to it's > monolithic limits and it's high time the code was refactored to > make it sanely extensible. Yes -- this has been on my TODO list for ages. But who's gonna DO it? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 15:13:14 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 10:13:14 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 09:35:07 EST." <3BDD691B.2599AAF4@interet.com> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <3BDD691B.2599AAF4@interet.com> Message-ID: <200110291513.KAA14082@cj20424-a.reston1.va.home.com> [Finn] > > So python packages and modules can exists *only* at the top level? That > > would conflict somewhat with jython where at would be common to put > > python modules into the same .zip file as java classes and java classes > > also wants to own the root of a zip file. > > > > In the current implementaion in jython, we can put python modules under > > any path and add the zipfile!path to sys.path: > > > > sys.path.append("/path/to/zipfile.zip!Lib") > > > > which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in > > the archive. [JimA] > I am confused. Zip archives are equivalant to subdirectories, so there > is no requirement to have anything at the top level. Your example seems > to imply a second search path besides sys.path. I think Finn simply means that the zipfile may have some redundant initial suffix to all filenames (e.g. "Lib/") which could be an artefact of how the zip file is created (zip up this directory) or intended as an aid for other uses of the zipfile, like unpacking. (In the tar world, it's considered impolite not to have a common prefix; without that, untarring can too easily populate an innocent user's home directory with thousands of untarred files.) > BTW, the code uses ".zip" as the archive flag, > not a special character '!'. That's cool. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 15:16:29 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 10:16:29 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 09:53:26 EST." <3BDD6D66.675A4F3C@interet.com> References: <3BD9B5EC.119319A5@interet.com> <3BDB1EB0.C2A9B1FF@lemburg.com> <3BDD6D66.675A4F3C@interet.com> Message-ID: <200110291516.KAA14105@cj20424-a.reston1.va.home.com> > > 3. Would it be possible to ship zlib together with Python ? > > (the zlib license should allow this and I don't think that > > the code size is too big) > > OK by me. But uncompressed zip archives work, and may even > be faster than conpressed archives. Good point. Uncompressed archives will definitely be faster. I'd be happy to legislate uncompressed archives, except I'm worried that not all tools commonly used to create zip archives make it easy to turn off compression. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Mon Oct 29 15:34:30 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 10:34:30 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BDD21B6.12124.38E143B8@localhost> Message-ID: <3BDD7706.A5BBE307@interet.com> Gordon McMillan wrote: > > > Currently, sys.path is a list of directory names as strings. > > A nit: that's in practice, but not by definition (a couple std > modules have been patched to allow other things on > sys.path). After extensive use of imputil (which puts objects > on sys.path), I think we might as well make it official that > sys.path is a list of strings. A nit on a nit: Non-strings can still be allowed, because I can ignore them. > [Subdirectory Equivalence] > > This is a bit misleading - it can be read as implying that the > importable modules are in a flat namespace. They're not. To > get R.Q.modfoo imported, R.__init__ and R.Q.__init__ must > be imported. The __init__ modules have the opportunity to > play with __path__ to break the equivalence of R.Q.modfoo to > R/Q/modfoo.py. Or (more likely), play games with attributes > so that Q is, say, an instance, or maybe a module imported > from someplace else. I feel that this is an excellent point, but I don't understand package import well enough to comment. > Question: if Archive.zip/Q/__init__.pyc does > "__path__.append('some/real/directory')", will it work? My guess (and it really is a guess) is Yes. The import.c code which generates directory names is untouched. My code simply looks for ".zip" in the generated name and branches into the zip archive at that point. So if the directory version works, the zip archive version should work too. > [no *.py files] > > You also can't import source files *.py from a zip archive. > > Apparently some Linuxes / RPM distributions don't deliver > .pyc's or .pyo's. Since they're installed as root and run as > some-poor-user, I'm afraid there are quite a few installations > running off .py's all the time. So while it's definitely sub- > optimal, I'm not sure it should be outlawed. For Linux/RPM's I think shipping a library directory is better than a zip archive. It is easier to hack on a directory. I think of zip archives as a way to distribute packages, and as a replacement for freeze. OTOH, maybe we should allow *.py to satisfy imports even if it is slow and invisible. > [Efficiency] > > The key is the archive name from sys.path joined with the > > file name (including any subdirectories) within the archive. > > DIfferent spellings of the same path are possible in a > filesystem, but not in a dictionary. A bit of "harmless" > tweaking of sys.path could render an archive unreachable. True, but I have little sympathy for case-insensitive file systems. Tweaking of sys.path will have to be done with care. It helps that the '/' character is always used in zip, and both backslash and colon ":" are illegal in zip archive names. JimA From jim@interet.com Mon Oct 29 15:38:33 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 10:38:33 -0500 Subject: [Python-Dev] PEP 273 - inporting from zipfiles References: <20011029140714.81E46303181@snelboot.oratrix.nl> Message-ID: <3BDD77F9.949DA69D@interet.com> Jack Jansen wrote: > > Jim, > for your implementation you might want to have a look at the > Macintosh-specific code in import.c (and possibly at related code in > Mac/macglue.c, but that's probably less relevant). OK, will do. JimA From gmcm@hypernet.com Mon Oct 29 15:41:04 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 10:41:04 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110291506.KAA14000@cj20424-a.reston1.va.home.com> References: Your message of "Mon, 29 Oct 2001 09:30:30 EST." <3BDD21B6.12124.38E143B8@localhost> Message-ID: <3BDD3240.31592.3921E147@localhost> [Gordon] > > After extensive use of imputil (which puts objects on > > sys.path), I think we might as well make it official that > > sys.path is a list of strings. [Guido] > Interesting. So you think imputil is wrong to put objects there? > Why? (Not arguing, just interested in your experience.) Too much code (some left in the std lib) says: for d in sys.path: os.path.join(d, ...) and that's the line that barfs. It doesn't barf if d is a string. [Gordon] > > DIfferent spellings of the same path are possible in a > > filesystem, but not in a dictionary. A bit of "harmless" > > tweaking of sys.path could render an archive unreachable. > > Hm, wouldn't the archive just be opened a second time? Or do I > misundestand you? Without seeing the C code, I don't know if it will open it a 2nd time. I keep an open file for each archive, but I'm careful that however it's spelled, I only open the archive once. Jim is apparently closing the file, so opening a 2nd is probably not as painful. OTOH an open / seek for each satisfied import is relatively expensive (though still cheaper than a fs import). > > [zlib must be available at start] > > I'll agree, and agree with Guido that the coolest thing would > > be to make zlib standard. > > But we'd have to make sure it's statically linked. (Fortunately, > we already link it statically on Windows.) So to avoid statically linking it twice, I assume zlib would be a mandatory builtin. [Gordon] > > Personally, I think package support stretched import.c to it's > > monolithic limits and it's high time the code was refactored to > > make it sanely extensible. > > Yes -- this has been on my TODO list for ages. But who's gonna > DO it? I published iu4 which I think comes very close to the "right" model. If it's close enough, I'll download 2.2 and see if I can make those into new style object subclasses[1]. - Gordon [1] As an app developer, I usually have to look backwards, not forwards. I've got 6 machines, each with at least 2 Pythons, so I no longer deal with betas :-(. Even when they have cool things like generators and subclassable C types. From barry@zope.com Mon Oct 29 15:52:27 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 29 Oct 2001 10:52:27 -0500 Subject: [Python-Dev] Re: patchlevel.h References: <20011026102540.3DBEB303181@snelboot.oratrix.nl> <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> <200110261856.OAA20424@cj20424-a.reston1.va.home.com> Message-ID: <15325.31547.431457.787791@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum writes: GvR> I think that for the final release we should definitely GvR> attempt this. Hopefully the speed and quality of change will GvR> reduce closer to the final release. Also, the release GvR> candidate should help. GvR> Jack, We'll try to work with you. Barry, please add GvR> something to this effect in the release procedure PEP (hold GvR> up the *final* release until Jack approves or until we lose GvR> patience). Absolutely. And done. -Barry From gmcm@hypernet.com Mon Oct 29 15:54:50 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 10:54:50 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110291516.KAA14105@cj20424-a.reston1.va.home.com> References: Your message of "Mon, 29 Oct 2001 09:53:26 EST." <3BDD6D66.675A4F3C@interet.com> Message-ID: <3BDD357A.126.392E7B98@localhost> [Guido] > Good point. Uncompressed archives will definitely be faster. > I'd be happy to legislate uncompressed archives, except I'm > worried that not all tools commonly used to create zip archives > make it easy to turn off compression. Don't be so sure about speed. If the archive is big enough and the OS bad enough , it may be faster to uncompress than swap. I always compress at 9, and have only seen speed ups of 1 or 2% with compression turned off. BTW, .pyc's compress quite well. - Gordon From jim@interet.com Mon Oct 29 15:58:10 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 10:58:10 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> Message-ID: <3BDD7C92.F130E708@interet.com> Guido van Rossum wrote: > Since obviously the intention of putting the module in the zipfile was > that it should be found, I think that failure to decompress should be > turned into an immediate error -- the same way as a syntax error gets > reported and not turned into a "skip to the next directory in > sys.path" effect. Hmmm, I'm not sure. But it is easy to do. > > I can upload it for inspection, but I don't want it to be a patch > > because it is not done yet. > > Just say so in the patch. You can upoad anything you want. :-) OK, I will post the implementation. JimA From gmcm@hypernet.com Mon Oct 29 16:17:50 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 11:17:50 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BDD7706.A5BBE307@interet.com> Message-ID: <3BDD3ADE.19659.39438820@localhost> > Gordon McMillan wrote: > > [no *.py files] > > > You also can't import source files *.py from a zip archive. > > > > Apparently some Linuxes / RPM distributions don't deliver > > .pyc's or .pyo's. Since they're installed as root and run as > > some-poor-user, I'm afraid there are quite a few installations > > running off .py's all the time. So while it's definitely sub- > > optimal, I'm not sure it should be outlawed. > > For Linux/RPM's I think shipping a library directory is better > than a zip archive. It is easier to hack on a directory. I > think of zip archives as a way to distribute packages, and as a > replacement for freeze. There's no reason to believe that telling a Linux distribution maintainer that they "shouldn't" do it that way will be successful. Heck, he might be a Perl monkey on KP . > OTOH, maybe we should allow *.py to satisfy imports even if > it is slow and invisible. > > > [Efficiency] > > > The key is the archive name from sys.path joined with the > > > file name (including any subdirectories) within the archive. > > > > DIfferent spellings of the same path are possible in a > > filesystem, but not in a dictionary. A bit of "harmless" > > tweaking of sys.path could render an archive unreachable. > > True, but I have little sympathy for case-insensitive file > systems. Tweaking of sys.path will have to be done with care. It > helps that the '/' character is always used in zip, and both > backslash and colon ":" are illegal in zip archive names. You don't need case-insensitive for this to come up. Relative paths vs. absolute paths; or paths that need norm-ing. And then you've got the mac which uses a different path syntax, and even slightly different path semantics. - Gordon From jim@interet.com Mon Oct 29 16:31:42 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 11:31:42 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> Message-ID: <3BDD846E.7D372942@interet.com> Guido van Rossum wrote: > Just say so in the patch. You can upoad anything you want. :-) The implementation is in patch 476047. http://sourceforge.net/tracker/index.php?func=detail&aid=476047&group_id=5470&atid=305470 JimA From jim@interet.com Mon Oct 29 17:28:44 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 12:28:44 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> Message-ID: <3BDD91CC.9F0ECB07@interet.com> Guido van Rossum wrote: [Me] > > I would like discussion on what the additional sys.path names are. > > Well, propose some. OK. I propose that there is one name added to sys.path, and the file name is "python%s%s.zip" % (sys.version[0], sys.version[2]). For example, python22.zip. This is the same on all platforms. On Unix, the directory is sys.prefix + "/lib". So for prefix /usr/local, the path /usr/local/lib/python2.2/ is already on sys.path, and /usr/local/lib/python22.zip would be added. On Windows, the directory is the directory of sys.executable. The zip archive name is always inserted as the second item in sys.path. The first always seems to be ''. JimA From guido@python.org Mon Oct 29 17:39:37 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 12:39:37 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 12:28:44 EST." <3BDD91CC.9F0ECB07@interet.com> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> Message-ID: <200110291739.MAA15773@cj20424-a.reston1.va.home.com> > OK. I propose that there is one name added to sys.path, and the > file name is "python%s%s.zip" % (sys.version[0], sys.version[2]). > For example, python22.zip. This is the same on all platforms. > > On Unix, the directory is sys.prefix + "/lib". So for prefix > /usr/local, the path /usr/local/lib/python2.2/ is already on > sys.path, and /usr/local/lib/python22.zip would be added. > > On Windows, the directory is the directory of sys.executable. > > The zip archive name is always inserted as the second item > in sys.path. The first always seems to be ''. Sounds good to me. Somebody please update the PEP. --Guido van Rossum (home page: http://www.python.org/~guido/) From ngps@post1.com Mon Oct 29 17:52:01 2001 From: ngps@post1.com (Ng Pheng Siong) Date: Tue, 30 Oct 2001 01:52:01 +0800 Subject: [Python-Dev] Re: Python's SSL: Would you like to help? In-Reply-To: <20011029135307.C1844@lilith.hqd-internal>; from gerhard@bigfoot.de on Mon, Oct 29, 2001 at 01:53:07PM +0100 References: <20011029135307.C1844@lilith.hqd-internal> Message-ID: <20011030015201.B1007@madcap.dyndns.org> On Mon, Oct 29, 2001 at 01:53:07PM +0100, Gerhard Häring wrote: > I've meanwhile started a discussion about the future of SSL support in > Python itself on the python-dev mailinglist. > > The talk is about several issues, like redesign a whole new API > (yes/no), integrate an existing module (which one?), support much crypto > at all. > > As you seem to have a lot of experience in this field, perhaps you'd > like to share some of your opinions there? [ Not subscribed, hopefully this gets thru. ] Hello, Thanks for asking. I've just read the discussion in the archives. My own archives contain a similar (but much shorter) discussion that took place on the old python-crypto@egroups list in Apr 2000. I've also just downloaded 2.2b1, POW (Python OpenSSL wrappers) 0.5, and pyOpenSSL 0.4 to have a quick look-see. (Haven't been keeping track of the competition. ;-) I noticed there are several SSL patches on SF, mostly from Gerhard. I didn't read the patches, though. Here are some random thoughts based on my experience with M2Crypto: - To do simple multi-threading, you need to bracket calls to SSL_read(), SSL_write(), etc. with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS. You also (apparently) need to implement the locking required by OpenSSL; see M2Crypto's _thread.i. IIRC, OpenSSL's docu warns of Bad Things should one fail to do so. (None of the others seem to do this.) If you want to do callbacks for certificate verification, it's a little more complicated. First, you need to muck around with Python's thread states in OpenSSL's structures. M2Crypto and pyOpenSSL do these mostly the same way. POW doesn't do this. Next, consider the Python "SSL Context" object that mirrors SSL_CTX: In M2Crypto, the "SSL Context" is implemented in Python; conveniently, I use it to keep additional Python-level policy stuff. Thus, when OpenSSL calls back into M2Crypto, I need a means to get at the Python SSL Context object, before calling back into Python. Implementing the SSL Context object in C may make this simpler. (I see 2.2b1 does only SSL_VERIFY_NONE. Ditto POW and pyOpenSSL, it seems.) - Even if you only want to do certificates within SSL, you need to wrap quite a bit of the X.509 API. See M2Crypto's _x509.i and X509.py. POW seems to have done so, as well. If you also want to do X.509 independent of SSL (say for implementing a CA), you'll need to do more housekeeping, like whether to free a given structure at the Python side or the C side. Grep for "_pyfree" in M2Crypto's Python code. - Asynchronous SSL: I think M2Crypto has this mostly right, since ZServerSSL works (2nd birthday came and went it seems ;-), and I've recently implemented Medusa-based FTP-over-TLS which interoperates with other people's clients. Based on my light reading of the code, none of 2.2b1, POW and pyOpenSSL do async-SSL properly at all. - FakeSocket, SSLFakeSocket and SSLFakeFile are ugly! M2Crypto wraps OpenSSL's "stackable BIOs". makefile() on an SSL connection pushes a buffering BIO on top of the SSL BIO and returns that, which then behaves like a Python file-like object. Even does readline()! See M2Crypto's BIO.py. 2.2b1 has a BIO member in PySSLObject but doesn't seem to use it. Nor do POW and pyOpenSSL, it appears. Well, that's only the SSL-related stuff that came to my mind readily. The Python development team has to decide how much of the above you want within Python, e.g., how much of the BIO and X.509 APIs to wrap. Not to mention PRNGs, secure memory handling and crypto-in-general. (Lastly, I'm pleased to note that M2Crypto is still ahead of the competition. Gotta keep moving, though! ;-) Cheers. -- Ng Pheng Siong * http://www.post1.com/home/ngps From barry@zope.com Mon Oct 29 17:53:23 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 29 Oct 2001 12:53:23 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> Message-ID: <15325.38803.700797.448117@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum writes: GvR> Sounds good to me. Somebody please update the PEP. Jim, your last context diff was fine. Feel free to send me updates in the same format, and I'll apply them. -Barry From tim.one@home.com Mon Oct 29 18:14:56 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 13:14:56 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BDD91CC.9F0ECB07@interet.com> Message-ID: [James C. Ahlstrom] > The zip archive name is always inserted as the second item > in sys.path. The first always seems to be ''. Running p.py: import sys print `sys.path[0]` on Windows gives: C:\Code\python\PCbuild>python p.py '' C:\Code\python\PCbuild>cd .. C:\Code\python>pcbuild\python pcbuild\p.py 'pcbuild' C:\Code\python>pcbuild\python \code\python\pcbuild\p.py '\\code\\python\\pcbuild' C:\Code\python>cd pcbuild C:\Code\python\PCbuild>python .\p.py '.' C:\Code\python\PCbuild>move p.py ..\lib C:\CODE\PYTHON\PCBUILD\p.py => C:\CODE\PYTHON\lib\p.py [ok] C:\Code\python\PCbuild>python ..\lib\p.py '..\\lib' C:\Code\python\PCbuild> That is, the first entry is the path (relative or absolute!) to the directory in which the script being executed lives. From tim.one@home.com Mon Oct 29 18:22:10 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 13:22:10 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110291516.KAA14105@cj20424-a.reston1.va.home.com> Message-ID: [Guido] > I'd be happy to legislate uncompressed archives, except I'm worried that > not all tools commonly used to create zip archives make it easy to turn > off compression. FWIW, WinZip has an easy-to-get-at "compression: none" setting. I don't know whether that creates "an uncompress archive" from Jim's POV, but it sure doesn't save any space . From barry@zope.com Mon Oct 29 18:53:00 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 29 Oct 2001 13:53:00 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions Message-ID: <15325.42380.21091.468633@anthem.wooz.org> Attached is the latest version of PEP 274 for dictionary comprehensions, similar to list comprehensions. The first version of this PEP wasn't posted here, but several people saw the cvs checkin and sent in comments, which have now been incorporated. Enjoy, -Barry -------------------- snip snip -------------------- PEP: 274 Title: Dict Comprehensions Version: $Revision: 1.2 $ Last-Modified: $Date: 2001/10/29 18:46:59 $ Author: barry@zope.com (Barry A. Warsaw) Status: Draft Type: Standards Track Created: 25-Oct-2001 Python-Version: 2.3 Post-History: 29-Oct-2001 Abstract PEP 202 introduces a syntactical extension to Python called the "list comprehension"[1]. This PEP proposes a similar syntactical extension called the "dictionary comprehension" or "dict comprehension" for short. You can use dict comprehensions in ways very similar to list comprehensions, except that they produce Python dictionary objects instead of list objects. Proposed Solution Dict comprehensions are just like list comprehensions, except that you group the expression using curly braces instead of square braces. Also, the left part before the `for' keyword expresses both a key and a value, separated by a colon. (There is an optional part of this PEP that allows you to use a shortcut to express just the value.) The notation is specifically designed to remind you of list comprehensions as applied to dictionaries. Rationale There are times when you have some data arranged as a sequences of length-2 sequences, and you want to turn that into a dictionary. In Python 2.2, the dictionary() constructor will take an optional keyword argument that indicates specifically to interpret a sequences of length-2 sequences as key/value pairs, and turn them into a dictionary. However, the act of turning some data into a sequence of length-2 sequences can be inconvenient or inefficient from a memory or performance standpoint. Also, for some common operations, such as turning a list of things into a set of things for quick duplicate removal or set inclusion tests, a better syntax can help code clarity. As with list comprehensions, an explicit for loop can always be used (and in fact was the only way to do it in earlier versions of Python). But as with list comprehensions, dict comprehensions can provide a more syntactically succinct idiom that the traditional for loop. Examples >>> print {i : chr(65+i) for i in range(4)} {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'} >>> print {k : v for k, v in someDict.iteritems()} == someDict.copy() 1 >>> print {x.lower() : 1 for x in list_of_email_addrs} {'barry@zope.com' : 1, 'barry@python.org' : 1, 'guido@python.org' : 1} >>> def invert(d): ... return {v : k for k, v in d.iteritems()} ... >>> d = {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'} >>> print invert(d) {'A' : 0, 'B' : 1, 'C' : 2, 'D' : 3} Open Issues - There is one further shortcut we could adopt. Suppose we wanted to create a set of items, such as in the "list_of_email_addrs" example above. Here, we're simply taking the target of the for loop and turning that into the key for the dict comprehension. The assertion is that this would be a common idiom, so the shortcut below allows for an easy spelling of it, by allow us to omit the "key :" part of the left hand clause: >>> print {1 for x in list_of_email_addrs} {'barry@zope.com' : 1, 'barry@python.org' : 1, 'guido@python.org' : 1} Or say we wanted to map email addresses to the MX record handling their mail: >>> print {mx_for_addr(x) for x in list_of_email_addrs} {'barry@zope.com' : 'mail.zope.com', 'barry@python.org' : 'mail.python.org, 'guido@python.org' : 'mail.python.org, } Questions: what about nested loops? Where does the key come from? The shortcut probably doesn't save much typing, and comes at the expense of legibility, so it's of dubious value. - Should nested for loops be allowed? The following example, taken from an earlier revision of this PEP illustrates the problem: >>> print {k, v for k in range(4) for v in range(-4, 0, 1)} The intent of this example was to produce a mapping from a number to its negative, but this code doesn't work because -- as in list comprehensions -- the for loops are nested, not in parallel! So the value of this expression is actually {0: -1, 1: -1, 2: -1, 3: -1} which seems of dubious value. For symmetry with list comprehensions, perhaps this should be allowed, but it might be better to disallow this syntax. Implementation The semantics of dictionary comprehensions can actually be modeled in stock Python 2.2, by passing a list comprehension to the builtin dictionary constructor: >>> dictionary([(i, chr(65+i)) for i in range(4)]) This has two dictinct disadvantages from the proposed syntax though. First, it's isn't as legible as a dict comprehension. Second, it forces the programmer to create an in-core list object first, which could be expensive. References [1] PEP 202, List Comprehensions http://www.python.org/peps/pep-0202.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil fill-column: 70 End: From guido@python.org Mon Oct 29 19:08:17 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 14:08:17 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions In-Reply-To: Your message of "Mon, 29 Oct 2001 13:53:00 EST." <15325.42380.21091.468633@anthem.wooz.org> References: <15325.42380.21091.468633@anthem.wooz.org> Message-ID: <200110291908.OAA16263@cj20424-a.reston1.va.home.com> > - There is one further shortcut we could adopt. Suppose we wanted > to create a set of items, such as in the "list_of_email_addrs" > example above. Here, we're simply taking the target of the for > loop and turning that into the key for the dict comprehension. Hm, I don't like this. I think it's confusing: you really have to think about whether {x for x in } produces a dictionary whose keys are in or one whose values are in . > - Should nested for loops be allowed? The following example, > taken from an earlier revision of this PEP illustrates the > problem: > > >>> print {k, v for k in range(4) for v in range(-4, 0, 1)} Don't you mean {k: v for ...}? The second range() also seems like you meant range(0, -4, -1). > The intent of this example was to produce a mapping from a > number to its negative, but this code doesn't work because -- as > in list comprehensions -- the for loops are nested, not in > parallel! So the value of this expression is actually > > {0: -1, 1: -1, 2: -1, 3: -1} > > which seems of dubious value. For symmetry with list > comprehensions, perhaps this should be allowed, but it might be > better to disallow this syntax. Nested for loops can be useful when used properly: {(k, v): k+v for k in range(4) for v in range(4)} Your example should have been expressed using: {k: v for k, v in zip(range(4), range(0, -4, -1))} --Guido van Rossum (home page: http://www.python.org/~guido/) From bckfnn@worldonline.dk Mon Oct 29 19:15:00 2001 From: bckfnn@worldonline.dk (Finn Bock) Date: Mon, 29 Oct 2001 19:15:00 GMT Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110291513.KAA14082@cj20424-a.reston1.va.home.com> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <3BDD691B.2599AAF4@interet.com> <200110291513.KAA14082@cj20424-a.reston1.va.home.com> Message-ID: <3bddaa12.11250086@mail.wanadoo.dk> [JimA] > I am confused. Zip archives are equivalant to subdirectories, so there > is no requirement to have anything at the top level. Your example seems > to imply a second search path besides sys.path. [GvR] >I think Finn simply means that the zipfile may have some redundant >initial suffix to all filenames (e.g. "Lib/") ... Yes, that was it! regards, finn From bckfnn@worldonline.dk Mon Oct 29 19:16:21 2001 From: bckfnn@worldonline.dk (Finn Bock) Date: Mon, 29 Oct 2001 19:16:21 GMT Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BDD6B74.D4E19953@interet.com> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <200110270223.WAA23737@cj20424-a.reston1.va.home.com> <3bdbfda4.3697676@mail.wanadoo.dk> <3BDD6B74.D4E19953@interet.com> Message-ID: <3bdda1df.9151318@mail.wanadoo.dk> [me] > myself), I just didn't dare to put such a limit on my jython > implementation. [James C. Ahlstrom] >I don't understand the jpython implementation, so please point out >all problems so we can fix them now. A naive implementation (like the first one I made for Jython) does not try to handle the cleanup issues. For example, how much memory have been consumed and not freed after the last statement in this little silly program: import sys, zipfile, time zip = zipfile.ZipFile("archive.zip", "w") for i in range(10000): entry = zipfile.ZipInfo() entry.filename = "module%06d.py" % i entry.date_time = time.gmtime(time.time()) zip.writestr(entry, "# Not used\n") zip.close() sys.path.append("archive.zip") try: import notfound except ImportError: pass sys.path.pop() If I read your preliminary patch correctly, the 10000 entries will remain in the ArchiveMembers dict forever and that is perfectly fine after Guido comments on sys.path being mostly a static feature. Jython manage to clean the member-cache when an archive is removed from sys.path but is was quite tricky to achieve. We do it by replacing zip entries in sys.path (like "archive.zip") with a subclass of string (SyspathArchive) that also holds a reference to the member cache. From pythons POV the sys.path entry is still a string with the same value (but with a different id). regards, finn From tim.one@home.com Mon Oct 29 19:14:39 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 14:14:39 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions In-Reply-To: <15325.42380.21091.468633@anthem.wooz.org> Message-ID: [Barry] > Rationale > > There are times when you have some data arranged as a sequences of > length-2 sequences, and you want to turn that into a dictionary. > In Python 2.2, the dictionary() constructor will take an optional > keyword argument that indicates specifically to interpret a > sequences of length-2 sequences as key/value pairs, and turn them > into a dictionary. This is implemented now, but in a different way. Suggested rewording: In Python 2.2, the dictionary() constructor accepts an argument that is a sequence of length-2 sequences, used as (key, value) pairs to initialize a new dictionary object. BTW, and not meaning to hijack your PEP , should dict.update() accept such an argument too? I didn't add it because d.update(dictionary(such_an_argument)) seemed "almost good enough". BTW2, are we going to rename "dictionary" to "dict" before 2.2b2? Before 2.2, "dict" was universally used on c.l.py to mean dictionary, and I'm at least +0 on adopting that for official 2.2 use. From barry@zope.com Mon Oct 29 19:22:58 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 29 Oct 2001 14:22:58 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions References: <15325.42380.21091.468633@anthem.wooz.org> Message-ID: <15325.44178.193256.554626@anthem.wooz.org> >>>>> "TP" == Tim Peters writes: TP> BTW, and not meaning to hijack your PEP , should TP> dict.update() accept such an argument too? I didn't add it TP> because TP> d.update(dictionary(such_an_argument)) TP> seemed "almost good enough". Agreed. But either way, I still there there is utility in a dict comprehension. TP> BTW2, are we going to rename "dictionary" to "dict" before TP> 2.2b2? Before 2.2, "dict" was universally used on c.l.py to TP> mean dictionary, and I'm at least +0 on adopting that for TP> official 2.2 use. I wouldn't keep them both though. Use one or the other. -Barry From guido@python.org Mon Oct 29 19:27:32 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 14:27:32 -0500 Subject: [Python-Dev] Dict API refinements (was PEP 274) In-Reply-To: Your message of "Mon, 29 Oct 2001 14:14:39 EST." References: Message-ID: <200110291927.OAA16469@cj20424-a.reston1.va.home.com> > BTW, and not meaning to hijack your PEP , should dict.update() accept > such an argument too? I didn't add it because > > d.update(dictionary(such_an_argument)) > > seemed "almost good enough". Agreed; no need to add this to update(). > BTW2, are we going to rename "dictionary" to "dict" before 2.2b2? > Before 2.2, "dict" was universally used on c.l.py to mean > dictionary, and I'm at least +0 on adopting that for official 2.2 > use. Sounds good to me. Should we adopt "dict" as an alias and keep "dictionary" as the official name, or vice versa, or simply eradicate dictionary and introduce dict in 2.2b2? It's up to you to implement this, I have some other things I need to get done first. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Mon Oct 29 19:45:47 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 14:45:47 -0500 Subject: [Python-Dev] test_asynchat.py broken on Windows Message-ID: Here on Windows, current CVS: C:\Code\python\PCbuild>python ../lib/test/test_asynchat.py Traceback (most recent call last): File "../lib/test/test_asynchat.py", line 58, in ? main() File "../lib/test/test_asynchat.py", line 53, in main c = echo_client() File "../lib/test/test_asynchat.py", line 34, in __init__ self.connect((HOST, PORT)) File "C:\CODE\PYTHON\lib\asyncore.py", line 311, in connect raise socket.error, err socket.error: 2 The DOS box is frozen at this point, but not accumulating any cycles. Worked 12 hours ago. What about Linux? From thomas.heller@ion-tof.com Mon Oct 29 19:54:31 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 29 Oct 2001 20:54:31 +0100 Subject: [Python-Dev] class dict of subtyped types Message-ID: <046c01c160b3$875c7340$e000a8c0@thomasnotebook> C:\>python Python 2.2b1+ (#25, Oct 19 2001, 14:30:05) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> class INT(int): pass ... >>> INT.__dict__ >>> INT.__dict__.update Traceback (most recent call last): File " ", line 1, in ? AttributeError: 'dict-proxy' object has no attribute 'update' >>> dir(INT.__dict__) ['__class__', '__contains__', '__delattr__', '__getattribute__', '__getitem__', '__hash__' , '__init__', '__iter__', '__len__', '__new__', '__reduce__', '__repr__', '__setattr__', ' __str__', 'copy', 'get', 'has_key', 'items', 'keys', 'values'] >>> Other dict attributes are missing as well in the dict-proxy. Is this intentional? Thomas From tim.one@home.com Mon Oct 29 19:55:36 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 14:55:36 -0500 Subject: [Python-Dev] RE: Dict API refinements (was PEP 274) In-Reply-To: <200110291927.OAA16469@cj20424-a.reston1.va.home.com> Message-ID: [Tim] >> BTW2, are we going to rename "dictionary" to "dict" before 2.2b2? >> Before 2.2, "dict" was universally used on c.l.py to mean >> dictionary, and I'm at least +0 on adopting that for official 2.2 >> use. [Guido] > Sounds good to me. Should we adopt "dict" as an alias and keep > "dictionary" as the official name, or vice versa, or simply eradicate > dictionary and introduce dict in 2.2b2? s/dictionary/dict/g is my preference: TOOWTDI. > It's up to you to implement this, I have some other things I need to > get done first. That's fine -- I can't think of anything needed I don't know how to do with ease (heck, I even know where the dictionary() docstring lives ), except for changing the descr tutorial accordingly on python.org. From fdrake@acm.org Mon Oct 29 19:53:49 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 29 Oct 2001 14:53:49 -0500 Subject: [Python-Dev] RE: Dict API refinements (was PEP 274) In-Reply-To: References: <200110291927.OAA16469@cj20424-a.reston1.va.home.com> Message-ID: <15325.46029.119091.651467@grendel.zope.com> Tim Peters writes: > That's fine -- I can't think of anything needed I don't know how to do with > ease (heck, I even know where the dictionary() docstring lives ), > except for changing the descr tutorial accordingly on python.org. I'll volunteer to take care of that one. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido@python.org Mon Oct 29 19:59:51 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 14:59:51 -0500 Subject: [Python-Dev] RE: Dict API refinements (was PEP 274) In-Reply-To: Your message of "Mon, 29 Oct 2001 14:55:36 EST." References: Message-ID: <200110291959.OAA16703@cj20424-a.reston1.va.home.com> > [Tim] > >> BTW2, are we going to rename "dictionary" to "dict" before 2.2b2? > >> Before 2.2, "dict" was universally used on c.l.py to mean > >> dictionary, and I'm at least +0 on adopting that for official 2.2 > >> use. > > [Guido] > > Sounds good to me. Should we adopt "dict" as an alias and keep > > "dictionary" as the official name, or vice versa, or simply eradicate > > dictionary and introduce dict in 2.2b2? [Tim] > s/dictionary/dict/g is my preference: TOOWTDI. [Guido] OK, let's do that. > > It's up to you to implement this, I have some other things I need to > > get done first. > > That's fine -- I can't think of anything needed I don't know how to do with > ease (heck, I even know where the dictionary() docstring lives ), > except for changing the descr tutorial accordingly on python.org. OK, I'll fix that one. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 20:04:12 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 15:04:12 -0500 Subject: [Python-Dev] class dict of subtyped types In-Reply-To: Your message of "Mon, 29 Oct 2001 20:54:31 +0100." <046c01c160b3$875c7340$e000a8c0@thomasnotebook> References: <046c01c160b3$875c7340$e000a8c0@thomasnotebook> Message-ID: <200110292004.PAA16731@cj20424-a.reston1.va.home.com> > C:\>python > Python 2.2b1+ (#25, Oct 19 2001, 14:30:05) [MSC 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> class INT(int): pass > ... > >>> INT.__dict__ > > >>> INT.__dict__.update > Traceback (most recent call last): > File " ", line 1, in ? > AttributeError: 'dict-proxy' object has no attribute 'update' > >>> dir(INT.__dict__) > ['__class__', '__contains__', '__delattr__', '__getattribute__', '__getitem__', '__hash__' > , '__init__', '__iter__', '__len__', '__new__', '__reduce__', '__repr__', '__setattr__', ' > __str__', 'copy', 'get', 'has_key', 'items', 'keys', 'values'] > >>> > > Other dict attributes are missing as well in the dict-proxy. > Is this intentional? The dict-proxy type is intended to provide a read-only proxy, so the dict-proxy is consciously lacking the update, clear, popitem and setdefault methods. It also seems to be missing the comparison operations; that's an oversight. --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Mon Oct 29 20:09:48 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 29 Oct 2001 21:09:48 +0100 Subject: [Python-Dev] class dict of subtyped types References: <046c01c160b3$875c7340$e000a8c0@thomasnotebook> <200110292004.PAA16731@cj20424-a.reston1.va.home.com> Message-ID: <055601c160b5$a9473dd0$e000a8c0@thomasnotebook> From: "Guido van Rossum" > > C:\>python > > Python 2.2b1+ (#25, Oct 19 2001, 14:30:05) [MSC 32 bit (Intel)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> class INT(int): pass > > ... > > >>> INT.__dict__ > > > > >>> INT.__dict__.update > > Traceback (most recent call last): > > File " ", line 1, in ? > > AttributeError: 'dict-proxy' object has no attribute 'update' > > >>> dir(INT.__dict__) > > ['__class__', '__contains__', '__delattr__', '__getattribute__', '__getitem__', '__hash__' > > , '__init__', '__iter__', '__len__', '__new__', '__reduce__', '__repr__', '__setattr__', ' > > __str__', 'copy', 'get', 'has_key', 'items', 'keys', 'values'] > > >>> > > > > Other dict attributes are missing as well in the dict-proxy. > > Is this intentional? > > The dict-proxy type is intended to provide a read-only proxy, so the > dict-proxy is consciously lacking the update, clear, popitem and > setdefault methods. It also seems to be missing the comparison > operations; that's an oversight. > But adding attributes to the INT class above after creation is allowed, I hope? At least it currently works. Thomas From guido@python.org Mon Oct 29 20:12:19 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 15:12:19 -0500 Subject: [Python-Dev] class dict of subtyped types In-Reply-To: Your message of "Mon, 29 Oct 2001 21:09:48 +0100." <055601c160b5$a9473dd0$e000a8c0@thomasnotebook> References: <046c01c160b3$875c7340$e000a8c0@thomasnotebook> <200110292004.PAA16731@cj20424-a.reston1.va.home.com> <055601c160b5$a9473dd0$e000a8c0@thomasnotebook> Message-ID: <200110292012.PAA16819@cj20424-a.reston1.va.home.com> > > The dict-proxy type is intended to provide a read-only proxy, so the > > dict-proxy is consciously lacking the update, clear, popitem and > > setdefault methods. It also seems to be missing the comparison > > operations; that's an oversight. > > > But adding attributes to the INT class above after creation > is allowed, I hope? At least it currently works. Yes, but you have to use the attribute notation: INT.foo = ... this is trapped so that certain magic happens automatically when you set attributes that override operators, e.g. INT.__add__ = lambda ... --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Mon Oct 29 20:57:38 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 15:57:38 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: Message-ID: <3BDDC2C2.E26289D5@interet.com> Tim Peters wrote: > > That is, the first entry is the path (relative or absolute!) to the > directory in which the script being executed lives. Right you are! JimA From jeremy@alum.mit.edu Mon Oct 29 23:20:14 2001 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Mon, 29 Oct 2001 18:20:14 -0500 (EST) Subject: [Python-Dev] Re: test_asynchat.py broken on Windows Message-ID: <200110292320.SAA11414@newman.concentric.net> I think there are at least three things wrong, depending on what you count as wrong. The test is hanging because it spawns a thread, which Python will prevent Python from exiting until the thread exits. The thread doesn't exit because it's waiting for input from the client. The client doesn't send it any input because of the exception. thing wrong #1: It would probably make sense to change test_asynchat.py to use a daemon thread so that it can't prevent the test suite from exiting. thing wrong #2: I don't think the ENONET being returned by connect_ex() makes any sense. My guess is that connect_ex() is (and always has been) broken on Windows. Could you apply this patch and see what happens: Index: socketmodule.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Modules/socketmodule.c,v retrieving revision 1.190 diff -c -c -r1.190 socketmodule.c *** socketmodule.c 2001/10/28 12:31:33 1.190 --- socketmodule.c 2001/10/29 23:14:18 *************** *** 1268,1274 **** --- 1268,1278 ---- res = connect(s->sock_fd, addr, addrlen); Py_END_ALLOW_THREADS if (res != 0) + #ifdef MS_WINDOWS + res = WSAGetLastError(); + #else res = errno; + #endif return PyInt_FromLong((long) res); } thing wrong #3: From the winsock documentation for connect(), it looks like there are more errors we should be catching in the test that includes EWOULDBLOCK and EALREADY. I don't think this would affect your test, though. thing wrong #4: It doesn't look to me like the connect() call in test_asynchat.py is guaranteed to succeed. It should succeed almost every time, but there's some small chance it will fail. Perhaps the test should be more robust about that. That's four things, but I don't think the last thing is all that wrong. Jeremy From Anthony Baxter Mon Oct 29 23:53:41 2001 From: Anthony Baxter (Anthony Baxter) Date: Tue, 30 Oct 2001 10:53:41 +1100 Subject: [Python-Dev] Fixing send() In-Reply-To: Message from Anthony Baxter of "Thu, 25 Oct 2001 16:24:20 +1000." Message-ID: <200110292353.f9TNrfo27736@mbuna.arbhome.com.au> > Will do - I plan to check in a bunch of fixes tomorrow morning. (continuing > gdb hell today). Just an update to this - my plans were wrecked by a cablemodem outage, however I'm spending tomorrow on branch mgmt - work is keen for me to take some of the 6 weeks leave I've built up, so I'm taking a few days off to work on this. As far as trying to delegate checkins of code - would people prefer I post a big list here, mail the original authors of the patches, or just do it myself? Anthony From greg@cosc.canterbury.ac.nz Tue Oct 30 00:39:41 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 30 Oct 2001 13:39:41 +1300 (NZDT) Subject: [Python-Dev] Dict API refinements (was PEP 274) In-Reply-To: <200110291927.OAA16469@cj20424-a.reston1.va.home.com> Message-ID: <200110300039.NAA09256@s454.cosc.canterbury.ac.nz> Guido van Rossum : > Should we adopt "dict" as an alias and keep > "dictionary" as the official name, or vice versa, or simply eradicate > dictionary and introduce dict in 2.2b2? I'd say eradicate "dictionary". There Should Only Be One Way To Spell It. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one@home.com Tue Oct 30 01:37:49 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 20:37:49 -0500 Subject: [Python-Dev] RE: test_asynchat.py broken on Windows In-Reply-To: <200110292320.SAA11414@newman.concentric.net> Message-ID: [Jeremy Hylton] > I think there are at least three things wrong, depending on what you > count as wrong. I would have guessed five, if you were counting Windows too . > The test is hanging because it spawns a thread, which Python will > prevent Python from exiting until the thread exits. The thread > doesn't exit because it's waiting for input from the client. The > client doesn't send it any input because of the exception. > > thing wrong #1: It would probably make sense to change > test_asynchat.py to use a daemon thread so that it can't prevent the > test suite from exiting. I don't much care about this. > thing wrong #2: I don't think the ENONET being returned by > connect_ex() makes any sense. My guess is that connect_ex() is (and > always has been) broken on Windows. Could you apply this patch and > see what happens: > ... Fixed the problem, so I checked it in. Thanks! > thing wrong #3: From the winsock documentation for connect(), it looks > like there are more errors we should be catching in the test that > includes EWOULDBLOCK and EALREADY. I don't think this would affect > your test, though. I'm not sure I parsed that as intended; asyncore.dispatcher.connect (in the traceback when test_asynchat failed) is already checking for EINPROGRESS, EALREADY, EWOULDBLOCK, 0 and EISCONN returns from connect_ex. But, for posterity, I will inscribe my socket programming knowledge in the period at the end of this sentence. > thing wrong #4: It doesn't look to me like the connect() call in > test_asynchat.py is guaranteed to succeed. It should succeed almost > every time, but there's some small chance it will fail. Perhaps the > test should be more robust about that. FWIW, I've never seen it fail before. > That's four things, but I don't think the last thing is all that wrong. Here's a fifth: def main(): s = echo_server() s.start() time.sleep(1) # Give server time to initialize c = echo_client() ... time.sleep()-is-not-a-synchronization-gimmick-ly y'rs - tim From tim.one@home.com Tue Oct 30 02:01:40 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 21:01:40 -0500 Subject: [Python-Dev] Fixing send() In-Reply-To: <200110292353.f9TNrfo27736@mbuna.arbhome.com.au> Message-ID: [Anthony Baxter] > Just an update to this - my plans were wrecked by a cablemodem outage, > however I'm spending tomorrow on branch mgmt - work is keen for me to > take some of the 6 weeks leave I've built up, so I'm taking a few days > off to work on this. We have similar views on what constitutes "a vacation" . > As far as trying to delegate checkins of code - would people prefer I > post a big list here, mail the original authors of the patches, or just > do it myself? Do it yourself so far as is humanly possible. That way (a) it will get done; and (b) the developers won't come to resent bugfix releases (if we had time to spare for this, we wouldn't ask for volunteers). it's-a-job-as-well-as-an-easy-way-to-burn-excess-vacation-days-ly y'rs - tim From guido@python.org Tue Oct 30 02:03:33 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 21:03:33 -0500 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Tue, 30 Oct 2001 10:53:41 +1100." <200110292353.f9TNrfo27736@mbuna.arbhome.com.au> References: <200110292353.f9TNrfo27736@mbuna.arbhome.com.au> Message-ID: <200110300203.VAA19633@cj20424-a.reston1.va.home.com> > As far as trying to delegate checkins of code - would people prefer I > post a big list here, mail the original authors of the patches, or just > do it myself? In general, trying to reach the original authors will just add delays. Do it yourself, or if you're not sure, post here. --Guido van Rossum (home page: http://www.python.org/~guido/) From greg@cosc.canterbury.ac.nz Tue Oct 30 02:25:09 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 30 Oct 2001 15:25:09 +1300 (NZDT) Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions In-Reply-To: <15325.42380.21091.468633@anthem.wooz.org> Message-ID: <200110300225.PAA09272@s454.cosc.canterbury.ac.nz> +1 generally on dict comprehensions, but: > >>> print {1 for x in list_of_email_addrs} > {'barry@zope.com' : 1, 'barry@python.org' : 1, 'guido@python.org' : 1} -1 on this bit. It's not at all clear what it should mean, and the saving over writing it out explicitly, i.e. {x:1 for x in list_of_email_addrs} is vanishingly small. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido@python.org Tue Oct 30 02:28:47 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 21:28:47 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions In-Reply-To: Your message of "Tue, 30 Oct 2001 15:25:09 +1300." <200110300225.PAA09272@s454.cosc.canterbury.ac.nz> References: <200110300225.PAA09272@s454.cosc.canterbury.ac.nz> Message-ID: <200110300228.VAA21968@cj20424-a.reston1.va.home.com> > +1 generally on dict comprehensions, but: > > > >>> print {1 for x in list_of_email_addrs} > > {'barry@zope.com' : 1, 'barry@python.org' : 1, 'guido@python.org' : 1} > > -1 on this bit. It's not at all clear what it should mean, > and the saving over writing it out explicitly, i.e. > > {x:1 for x in list_of_email_addrs} > > is vanishingly small. You're getting better at channeling me. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Tue Oct 30 06:23:32 2001 From: fdrake@acm.org (Fred L. Drake) Date: Tue, 30 Oct 2001 01:23:32 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011030062332.77F4928697@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Re-arranged the material in Chapter 2 (built-in things) to make it easier to start off with. Functions come before Types and Exceptions, and file objects are promoted one level in the outline, making them easier to find (they now appear in the table of contents instead of being hidden in the "Other Objects" category). From mal@lemburg.com Tue Oct 30 08:25:28 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 30 Oct 2001 09:25:28 +0100 Subject: [Python-Dev] Slices and "==" optimization Message-ID: <3BDE63F8.541E04B2@lemburg.com> While writing yet another XML parser I have come across two things which would seem nice to have: 1. Even though you can construct slice objects in Python using slice(left,right), you can't really do anything with them; at least not on the standard types (Numeric's arrays work with them just fine). Couldn't we add some functionality which makes them compatible to lists and tuples (and perhaps even in a generic way for other sequences using PySequence_GetItem()) too ? I thinking of extending the __getitem__ hook to accept slice objects. You could then maintain slices to an object in say a list and apply them to the underlying text as needed, e.g. s = slice(1,4) l = range(10) l[s] == [1, 2, 3] 2. Looking through the code for '==' the optimization for 'a == a' seems too far down the call stack. Since interning strings makes this case rather common, I'd suggest to lift the optimization into the ceval loop (right along side with the INT op INT optimization). Thoughts ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From martin@v.loewis.de Tue Oct 30 08:42:12 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 30 Oct 2001 09:42:12 +0100 Subject: [Python-Dev] Slices and "==" optimization Message-ID: <200110300842.f9U8gCX01628@mira.informatik.hu-berlin.de> > I thinking of extending the __getitem__ hook to accept slice objects Please have a look at http://sourceforge.net/tracker/?func=detail&aid=459235&group_id=5470&atid=105470 It's a known problem, but Guido won't fix it anytime soon. So I guess contributions are welcome. > Looking through the code for '==' the optimization for 'a == a' > seems too far down the call stack. Since interning strings makes > this case rather common, I'd suggest to lift the optimization into > the ceval loop As with all optimizations: Patches to just improve the performance are unacceptable unless accompanied by some hard numbers to show that they really do have the desired effect. History shows that patches to just improve the performance may be unacceptable even if accompanied by such numbers. Normally, Tim will construct a scenario where the patch causes a slow-down, and argue that this is a common scenario :-) Regards, Martin From fredrik@pythonware.com Tue Oct 30 08:56:07 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 30 Oct 2001 09:56:07 +0100 Subject: [Python-Dev] RE: test_asynchat.py broken on Windows References: Message-ID: <00c901c16120$b954e180$ced241d5@hagrid> > [Jeremy Hylton] > > I think there are at least three things wrong, depending on what you > > count as wrong. not sure jeremy gets mail sent to his sourceforge address, but maybe someone else can explain how his recent patch preserves the original behaviour. after the patch, connect will always raise an exception unless the error code is one of the three listed code. before the patch, it will not raise an exception if the connection succeeds during the connect itself (i.e. if people use asyncore with blocking sockets, or if they're using a hypothetical transport that can do an immediate connect). how about adding an "else:" before that last raise? def connect (self, address): self.connected = 0 ! # XXX why not use connect_ex? ! try: ! self.socket.connect (address) ! except socket.error, why: ! if why[0] in (EINPROGRESS, EALREADY, EWOULDBLOCK): ! return ! else: ! raise socket.error, why ! self.addr = address ! self.connected = 1 ! self.handle_connect() def accept (self): --- 302,313 ---- def connect (self, address): self.connected = 0 ! err = self.socket.connect_ex(address) ! if err in (EINPROGRESS, EALREADY, EWOULDBLOCK): ! return ! if err in (0, EISCONN): ! self.addr = address ! self.connected = 1 ! self.handle_connect() ! raise socket.error, err def accept (self): From mal@lemburg.com Tue Oct 30 08:55:09 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 30 Oct 2001 09:55:09 +0100 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <3BDB1EB0.C2A9B1FF@lemburg.com> <3BDD6D66.675A4F3C@interet.com> Message-ID: <3BDE6AED.25D67499@lemburg.com> "James C. Ahlstrom" wrote: > > "M.-A. Lemburg" wrote: > > > 1. Why are .py files being outlawed ? > > My reason was that if *.py[co] are missing or out of date, > zip importing will be slow and users won't figure > out what is wrong. Generally I favor user-proof features > over expert features. I prefer things which either "Just Work" > or fail spectacularly. But I am open to changing this. If you don't include the *.py file in the archive, chances are high that tracebacks will no longer print out as they do today (another problem here is that the filename being used in the code object will probably not be found... not sure whether we can fix this one). By only looking at the .pyc or .pyo you'll also introduce a Python version problem into the ZIP-archive: the magic in these files changs rather frequently and e.g. a more recent release of Python won't be able to load these from a ZIP-archive which only contains .pycs from a compile by a less recent version. > > 2. Where's the C implementation you mention in the PEP ? > > Software is like pancakes, you should always > throw the first one away. I will post it if you want, > but it is not done. That's true :-) Still, I'd like a chance to at least look at the impact this has on import.c and of course play with it so that I can test it in everyday situations. > > 3. Would it be possible to ship zlib together with Python ? > > (the zlib license should allow this and I don't think that > > the code size is too big) > > OK by me. But uncompressed zip archives work, and may even > be faster than conpressed archives. Even if uncompressed archive would work faster, the compiled libz is only 70k on Linux and I think we could solve a great number of (zlib version) problems by including zlib in Python. It's one of those basic tools you need frequently, but is even more frequently not configured as Python module :-( Since it is already included in the Windows builds, I guess adding it to core for Unix and Macs too wouldn't hurt all that much. It would also save you the trouble of maintaining the code for scanning uncompressed zip-archives in your ZIP import code, so we win on two counts :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Tue Oct 30 10:12:43 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 30 Oct 2001 11:12:43 +0100 Subject: [Python-Dev] Slices and "==" optimization References: <200110300842.f9U8gCX01628@mira.informatik.hu-berlin.de> Message-ID: <3BDE7D1B.4850329C@lemburg.com> "Martin v. Loewis" wrote: > > > I thinking of extending the __getitem__ hook to accept slice objects > > Please have a look at > > http://sourceforge.net/tracker/?func=detail&aid=459235&group_id=5470&atid=105470 > > It's a known problem, but Guido won't fix it anytime soon. So I guess > contributions are welcome. Good to know, thanks. > > Looking through the code for '==' the optimization for 'a == a' > > seems too far down the call stack. Since interning strings makes > > this case rather common, I'd suggest to lift the optimization into > > the ceval loop > > As with all optimizations: Patches to just improve the performance are > unacceptable unless accompanied by some hard numbers to show that they > really do have the desired effect. > > History shows that patches to just improve the performance may be > unacceptable even if accompanied by such numbers. Normally, Tim will > construct a scenario where the patch causes a slow-down, and argue > that this is a common scenario :-) Sounds like you have experience ;-) As with all micro-optimizations, the win usually only shows up in some applications. The one I'm writing uses lot's of if tagtype == 'starttag': if tagtype == 'endtag': ... Since the 'starttag' and 'endtag' strings are interned and I'm also interning the strings which are stored in tagtype, I'd like to benefit from the fact that the compare will actually work as 'is'-compare. However, I don't want to use 'is' because I consider interning only an optimization and not a feature of the language. That's why I would like the simple if (v == w) return 0; integrated into the ceval loop right along the INT-compare optimization. This may also help in some other situations where objects are shared. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Tue Oct 30 13:50:47 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 30 Oct 2001 08:50:47 -0500 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: Your message of "Tue, 30 Oct 2001 09:25:28 +0100." <3BDE63F8.541E04B2@lemburg.com> References: <3BDE63F8.541E04B2@lemburg.com> Message-ID: <200110301350.IAA24747@cj20424-a.reston1.va.home.com> > 1. Even though you can construct slice objects in Python > using slice(left,right), you can't really do anything > with them; at least not on the standard types (Numeric's > arrays work with them just fine). > > Couldn't we add some functionality which makes them > compatible to lists and tuples (and perhaps even in a > generic way for other sequences using PySequence_GetItem()) > too ? > > I thinking of extending the __getitem__ hook to accept > slice objects. You could then maintain slices to an > object in say a list and apply them to the underlying text > as needed, e.g. > > s = slice(1,4) > l = range(10) > l[s] == [1, 2, 3] I agree that this would be nice to have. I won't have time for it before 2.2 is out, but if you want to give it a crack, I might accept it. > 2. Looking through the code for '==' the optimization for > 'a == a' seems too far down the call stack. Since interning > strings makes this case rather common, I'd suggest to > lift the optimization into the ceval loop (right along side > with the INT op INT optimization). I would agree, except there's also the point of view that since types can override __eq__ to always return false, this optimization should not be done in a way that prevents __eq__ from overriding it. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Oct 30 14:00:13 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 30 Oct 2001 09:00:13 -0500 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: Your message of "Tue, 30 Oct 2001 11:12:43 +0100." <3BDE7D1B.4850329C@lemburg.com> References: <200110300842.f9U8gCX01628@mira.informatik.hu-berlin.de> <3BDE7D1B.4850329C@lemburg.com> Message-ID: <200110301400.JAA24868@cj20424-a.reston1.va.home.com> > As with all micro-optimizations, the win usually only shows > up in some applications. The one I'm writing uses lot's of > > if tagtype == 'starttag': > if tagtype == 'endtag': > ... > > Since the 'starttag' and 'endtag' strings are interned and > I'm also interning the strings which are stored in tagtype, > I'd like to benefit from the fact that the compare will actually > work as 'is'-compare. However, I don't want to use 'is' because > I consider interning only an optimization and not a feature > of the language. Agreed. > That's why I would like the simple > > if (v == w) return 0; > > integrated into the ceval loop right along the INT-compare > optimization. Maybe this could be done as follows: if (v == w && PyString_CheckExact(v)) return 0; > This may also help in some other situations where objects are > shared. Now *that* is the speculation for which Tim will chop off your head. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From akuchlin@mems-exchange.org Tue Oct 30 14:48:34 2001 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Tue, 30 Oct 2001 09:48:34 -0500 Subject: [Python-Dev] 2.2 article updated Message-ID: I've updated the 2.2 article with a section that gives an overview of the type/class changes. At this point the text should be essentially done, except for whatever small changes get into CVS between now and 2.2final. Please take a look and send me any comments: http://www.amk.ca/python/2.2/ (Oops! Michael McLay just sent me a note pointing out that I refer to .__get__() as .get() in a few places. So don't bother reporting that.) If your favorite patch or improvement isn't mentioned, please let me know. Also inform me if you aren't mentioned as a contributor to a patch. BTW, I call PEP 261, "Support for "wide" Unicode characters", not finally accepted and not implemented. Is that correct? Is it going to be finished before 2.2final, or should I simply describe it as partially implemented in the final version of the text? --amk From martin@v.loewis.de Tue Oct 30 17:54:55 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 30 Oct 2001 18:54:55 +0100 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: message from Guido van Rossum on Tue, 30 Oct 2001 09:00:13 -0500 Message-ID: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de> > > That's why I would like the simple > > > > if (v == w) return 0; > > > > integrated into the ceval loop right along the INT-compare > > optimization. > > Maybe this could be done as follows: > > if (v == w && PyString_CheckExact(v)) return 0; Maybe I'm missing some context here: where is this fragment supposed to go to? into ceval.c:COMPARE_OP? What is the "return 0;" doing then? In any case, I think measurements should show how much speed improvement is gained by taking this short-cut. It sounds nice in theory, but ... I just added the block else if ((v == w) && (oparg == EQ) && PyString_CheckExact(v)) { x = Py_True; Py_INCREF(x); } into the code. In an application that almost exclusively does COMPARE_OPs on identical strings, I got a 30% speed-up. OTOH, this same code caused a 10% slowdown if I converted the "==" into "<>". In a real application, total speed-up will depend on two things: - how many COMPARE_OPs are done in the code? - how many of those compare identical strings for equality? Running the PyXML test suite, I counted 120000 cases where slow_compare was done, and only 700 cases identical strings were compared for equality. Regards, Martin From andymac@bullseye.apana.org.au Tue Oct 30 12:15:28 2001 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Tue, 30 Oct 2001 23:15:28 +1100 (EDT) Subject: [Python-Dev] OS/2 VAC++ patches Message-ID: I have looked at Michael Muller's VAC++ patches, and added comments as followups. In summary: 473749 - has several (minor IMO) stylistic issues (addition of OS/2 #ifdefs). The intent of the changes looks OK, as do most of the actual changes (those in OS/2 VAC specific files or existing OS/2 #ifdefs). 474169 - looks good to go (changes inside existing OS/2 #ifdef) 474500 - looks good to go (isolated to OS/2 specific file) Comments on the stylistic issues? -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From fdrake@acm.org Tue Oct 30 18:05:11 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 30 Oct 2001 13:05:11 -0500 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de> References: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de> Message-ID: <15326.60375.319250.547944@grendel.zope.com> Martin v. Loewis writes: > into the code. In an application that almost exclusively does > COMPARE_OPs on identical strings, I got a 30% speed-up. OTOH, this > same code caused a 10% slowdown if I converted the "==" into "<>". It's very questionable whether we'll find an optimization that works well for any serious variety of applications. Perhaps there's some way to build up a lookup table of fast paths, and have the slow path try that first, but I'm sceptical. > Running the PyXML test suite, I counted 120000 cases where > slow_compare was done, and only 700 cases identical strings were > compared for equality. I wouldn't expect XML applications to gain from such an optimization, since many compares will involve Unicode objects, either to each other or to 8-bit strings. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From martin@v.loewis.de Tue Oct 30 18:41:10 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 30 Oct 2001 19:41:10 +0100 Subject: [Python-Dev] 2.2 article updated Message-ID: <200110301841.f9UIfAA04832@mira.informatik.hu-berlin.de> > BTW, I call PEP 261, "Support for "wide" Unicode characters", not > finally accepted and not implemented. Is that correct? Is it going > to be finished before 2.2final, or should I simply describe it as > partially implemented in the final version of the text? To my knowledge, it is completely implemented. You may mention that --disable-unicode is now also supported. There is atleast one error in the PEP, namely that the define for wide Unicode Py_UNICODE_WIDE (Py_UNICODE_SIZE also exists, but isn't used except to define Py_UNICODE_WIDE). I'm also not sure certain the all codecs handle all surrogate cases correctly in all circumstances, but that is in the realm of bug reports. One further Unicode change: The bullet item "When presented with a Unicode filename on Windows ..." should be extended with "On Unix, the locale's character set is used if nl_langinfo(CODESET) is available." Two new builtins are missing in your text, OverflowWarning and super. HTH, Martin From mmuller@enduden.com Tue Oct 30 18:42:45 2001 From: mmuller@enduden.com (Michael Muller) Date: Tue, 30 Oct 2001 13:42:45 -0500 Subject: [Python-Dev] Re: OS/2 VAC++ patches Message-ID: <222.1004467365.385140.1552565879@lilbastard> Andrew, Thanks for reviewing my patches. I am posting a new patch in response to your comments on *749. Andrew MacIntyre wrote: > I have looked at Michael Muller's VAC++ patches, and added comments as > followups. > > In summary: > 473749 - has several (minor IMO) stylistic issues (addition of OS/2 > #ifdefs). The intent of the changes looks OK, as do most of > the actual changes (those in OS/2 VAC specific files or > existing OS/2 #ifdefs). > 474169 - looks good to go (changes inside existing OS/2 #ifdef) > 474500 - looks good to go (isolated to OS/2 specific file) > > Comments on the stylistic issues? > > -- > Andrew I MacIntyre "These thoughts are mine alone..." > E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 > andymac@pcug.org.au | Belconnen ACT 2616 > Web: http://www.andymac.org/ | Australia ============================================================================= michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- There is no concept that is more demeaning to the human spirit than the notion that our freedom must be limited in the interests of our own protection. ============================================================================= From mal@lemburg.com Tue Oct 30 20:07:21 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 30 Oct 2001 21:07:21 +0100 Subject: [Python-Dev] Slices and "==" optimization References: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de> Message-ID: <3BDF0879.D758A1A8@lemburg.com> "Martin v. Loewis" wrote: > > > > That's why I would like the simple > > > > > > if (v == w) return 0; > > > > > > integrated into the ceval loop right along the INT-compare > > > optimization. > > > > Maybe this could be done as follows: > > > > if (v == w && PyString_CheckExact(v)) return 0; > > Maybe I'm missing some context here: where is this fragment supposed > to go to? into ceval.c:COMPARE_OP? What is the "return 0;" doing then? It's pseudo code I made up. The real thing will look very much like it though. > In any case, I think measurements should show how much speed > improvement is gained by taking this short-cut. It sounds nice in > theory, but ... > > I just added the block > > else if ((v == w) && (oparg == EQ) && PyString_CheckExact(v)) { > x = Py_True; > Py_INCREF(x); > } You should first test for oparg == EQ, then for v == w and PyString_CheckExact(). > into the code. In an application that almost exclusively does > COMPARE_OPs on identical strings, I got a 30% speed-up. Nice :-) > OTOH, this > same code caused a 10% slowdown if I converted the "==" into "<>". This sounds like an alignment problem caused by the compiler. 10% speedups/slowdowns can easily be produced by randomly moving about a few cases in the ceval loop. All depends on the platform and compiler though. > In a real application, total speed-up will depend on two things: > - how many COMPARE_OPs are done in the code? > - how many of those compare identical strings for equality? As I explained in my other reply, I have code which does: if variable == 'constant string': ... Since the compiler interns the 'constant string' and I can make sure that variable is interned too (by calling intern()), I can easily take advantage of the optimization without giving away any semantics. If variable happens to be some other constant which is used in a Python module (e.g. some kind of flag or parameter), then it will be interned per-se too. Note that the dictionary implementation relies heavily on the same optimization (interning was added to enhance string compare performance in dict lookups). > Running the PyXML test suite, I counted 120000 cases where > slow_compare was done, and only 700 cases identical strings were > compared for equality. As Fred mentioned, this is probably due to the fact that Unicode objects are not interned. Neither are typical XML tag names; could make a difference... don't know. I believe that PyXML spends most of the time in Python calls and not so much in string compares (and that's what I'm trying to avoid in my XML parser approach ;-). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From jeremy@zope.com Tue Oct 30 20:07:07 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Tue, 30 Oct 2001 15:07:07 -0500 (EST) Subject: [Python-Dev] Future of SSL In-Reply-To: <20011029141844.A1967@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> <20011029141844.A1967@lilith.hqd-internal> Message-ID: <15327.2155.83315.318149@slothrop.digicool.com> >>>>> "GH" =3D=3D haering python writes: GH> On Mon, Oct 29, 2001 at 08:06:28AM -0500, Guido van Rossum GH> wrote: >> > [me, in response to a remark from Marc-Andr=E9] What would be >> > your suggestions? Would you prefer to go in the direction of my >> > original proposal - only providing a SSL API, but not the >> > implementation? >> >> Yes, that's how the current SSL support works -- you need to link >> in openssl. GH> So, as long as there are no actual cryptographic algorithms in GH> the Python source tree, but only hooks for OpenSSL, there's no GH> problem? I don't think the export control regs work that way. Crypto-with-a-hole (that is, an API without the actual crypto implementation) is still considered crypto as far as I know. The regulations are pretty simple these days for Open Source projects. Just send a note to the BXA (Commerce Dept.) with the URL to the source. I wonder if it's wise to do this for Python, since it has an SSL interface (albeit a clunky one). Jeremy From gerhard@bigfoot.de Tue Oct 30 20:28:23 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Tue, 30 Oct 2001 21:28:23 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <15327.2155.83315.318149@slothrop.digicool.com>; from jeremy@zope.com on Tue, Oct 30, 2001 at 03:07:07PM -0500 References: <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> <20011029141844.A1967@lilith.hqd-internal> <15327.2155.83315.318149@slothrop.digicool.com> Message-ID: <20011030212822.A800@lilith.hqd-internal> On Tue, Oct 30, 2001 at 03:07:07PM -0500, Jeremy Hylton wrote: > GH> So, as long as there are no actual cryptographic algorithms in > GH> the Python source tree, but only hooks for OpenSSL, there's no > GH> problem? > > I don't think the export control regs work that way. > Crypto-with-a-hole (that is, an API without the actual crypto > implementation) is still considered crypto as far as I know. > > The regulations are pretty simple these days for Open Source projects. > Just send a note to the BXA (Commerce Dept.) with the URL to the > source. > > I wonder if it's wise to do this for Python, since it has an SSL > interface (albeit a clunky one). Looks like it would (formally) be required, looking thru this text: http://www.bxa.doc.gov/Encryption/EncryptionRuleOct2K.html I better stop reading crypo law texts now >:-< Gerhard -- mail: gerhard bigfoot de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From barry@zope.com Tue Oct 30 21:11:50 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 30 Oct 2001 16:11:50 -0500 Subject: [Python-Dev] Future of SSL References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> <20011029141844.A1967@lilith.hqd-internal> <15327.2155.83315.318149@slothrop.digicool.com> Message-ID: <15327.6038.463871.492896@anthem.wooz.org> >>>>> "JH" == Jeremy Hylton writes: JH> I don't think the export control regs work that way. JH> Crypto-with-a-hole (that is, an API without the actual crypto JH> implementation) is still considered crypto as far as I know. JH> The regulations are pretty simple these days for Open Source JH> projects. Just send a note to the BXA (Commerce Dept.) with JH> the URL to the source. JH> I wonder if it's wise to do this for Python, since it has an JH> SSL interface (albeit a clunky one). Another task for the PSF, eh? -Barry From barry@zope.com Tue Oct 30 21:13:48 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 30 Oct 2001 16:13:48 -0500 Subject: [Python-Dev] Slices and "==" optimization References: <3BDE63F8.541E04B2@lemburg.com> <200110301350.IAA24747@cj20424-a.reston1.va.home.com> Message-ID: <15327.6156.273422.675001@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum writes: GvR> I agree that this would be nice to have. I won't have time GvR> for it before 2.2 is out, but if you want to give it a crack, GvR> I might accept it. It's also a new feature, so you'd have to override the feature freeze to add it for 2.2. -Barry From hamish_lawson@yahoo.co.uk Tue Oct 30 21:17:18 2001 From: hamish_lawson@yahoo.co.uk (=?iso-8859-1?q?Hamish=20Lawson?=) Date: Tue, 30 Oct 2001 21:17:18 +0000 (GMT) Subject: [Python-Dev] Misleading error message when multiple arguments for write() Message-ID: <20011030211718.10036.qmail@web11001.mail.yahoo.com> The reference to 'tuple' in the error message below seems misleading. Too many arguments have been passed rather than a tuple. Traceback (most recent call last): File "C:\Temp\demo.py", line 3, in ? sys.stdout.write("a", "b") TypeError: argument must be string or read-only character buffer, not tuple Hamish Lawson ____________________________________________________________ Nokia Game is on again. Go to http://uk.yahoo.com/nokiagame/ and join the new all media adventure before November 3rd. From guido@python.org Tue Oct 30 22:18:52 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 30 Oct 2001 17:18:52 -0500 Subject: [Python-Dev] Misleading error message when multiple arguments for write() In-Reply-To: Your message of "Tue, 30 Oct 2001 21:17:18 GMT." <20011030211718.10036.qmail@web11001.mail.yahoo.com> References: <20011030211718.10036.qmail@web11001.mail.yahoo.com> Message-ID: <200110302218.f9UMIsf09646@odiug.zope.com> > The reference to 'tuple' in the error message below seems misleading. > Too many arguments have been passed rather than a tuple. > > Traceback (most recent call last): > File "C:\Temp\demo.py", line 3, in ? > sys.stdout.write("a", "b") > TypeError: argument must be string or read-only character buffer, not > tuple While you've reached the right audience, and this is indeed a bug(let), python-dev is not the right medium for bug reports. Please submit this to the SourceForge bug tracker, which acts as our collective memory for bugs. --Guido van Rossum (home page: http://www.python.org/~guido/) From jack@oratrix.nl Tue Oct 30 22:25:44 2001 From: jack@oratrix.nl (Jack Jansen) Date: Tue, 30 Oct 2001 23:25:44 +0100 Subject: [Python-Dev] slicing of structseq objects fails In-Reply-To: Message by Russell E Owen , Tue, 30 Oct 2001 10:02:14 -0800 , Message-ID: <20011030222549.4C74AFB04A@oratrix.oratrix.nl> Python-devvers, Russel Owen reported the following bug, and it turns out to be a general bug, not a MacPython one: slicing the structseq-style tuples returned by time.gmtime() returns a sequence with NULLs in it. Shortly afterwards Python crashes on my SGI. Any takers? Recently, Russell E Owen said: > The following code fails (G4, MacOS 9.2.1) in Classic MacPython 2.2b1: > > import time > > currUTCTuple= time.gmtime(time.time()) > print "currUTCTuple=%r" % (currUTCTuple,) > print "currUTCTuple[3:6]=%r" % (currUTCTuple[3:6],) > fmtTime = "%s:%02i:%02i" % currUTCTuple[3:6] > print "formatted time = ", fmtTime > > It displays ( , , ) for currUTCTuple[3:6], > ... -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From fredrik@pythonware.com Tue Oct 30 22:57:27 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 30 Oct 2001 23:57:27 +0100 Subject: [Python-Dev] slicing of structseq objects fails References: <20011030222549.4C74AFB04A@oratrix.oratrix.nl> Message-ID: <001b01c16196$7855eb90$ced241d5@hagrid> jack wrote: > > The following code fails (G4, MacOS 9.2.1) in Classic MacPython 2.2b1: > > > > import time > > > > currUTCTuple= time.gmtime(time.time()) > > print "currUTCTuple=%r" % (currUTCTuple,) > > print "currUTCTuple[3:6]=%r" % (currUTCTuple[3:6],) > > fmtTime = "%s:%02i:%02i" % currUTCTuple[3:6] > > print "formatted time = ", fmtTime > > > > It displays ( , , ) for currUTCTuple[3:6], also note: >>> import time >>> tm = time.gmtime() >>> tm (2001, 10, 30, 22, 54, 16, 1, 303, 0) >>> tm.tm_year 2001 >>> tm[:6] (2001, 10, 30, 22, 54, 16) >>> tm[3:6] ( , , ) in structseq_slice, for(i = low; i < high; ++i) { PyObject *v = obj->ob_item[i]; Py_INCREF(v); PyTuple_SET_ITEM(np, i, v); } should be: for(i = low; i < high; ++i) { PyObject *v = obj->ob_item[i]; Py_INCREF(v); PyTuple_SET_ITEM(np, i-low, v); } From tim@zope.com Tue Oct 30 22:55:49 2001 From: tim@zope.com (Tim Peters) Date: Tue, 30 Oct 2001 17:55:49 -0500 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: <3BDF0879.D758A1A8@lemburg.com> Message-ID: [MvL] >> OTOH, this same code caused a 10% slowdown if I converted the "==" into >> "<>". [MAL] > This sounds like an alignment problem caused by the compiler. > 10% speedups/slowdowns can easily be produced by randomly moving > about a few cases in the ceval loop. All depends on the platform > and compiler though. I expect a more obvious cause is at work: every one of Martin's "<>" compares was slowed by the new batch of failing "is it an EQ and are these strings and are they identical?" tests and branches. Fast paths lose when when they don't win -- they aren't neutral then. Indeed, we could speed your particular app by getting rid of the int-compare fast path already there (but I expect that one is a *net* win across apps, so don't want to get rid of it ). >> Running the PyXML test suite, I counted 120000 cases where >> slow_compare was done, and only 700 cases identical strings were >> compared for equality. > As Fred mentioned, this is probably due to the fact that > Unicode objects are not interned. Neither are typical > XML tag names; could make a difference... don't know. Whatever it's due to, it means the fast-path code cost extra cycles in the 119300 (of 120000) cases it didn't pay off. If it costs F extra cycles when it fails, and saves S cycles when it succeeds, the question for this test is whether 119300*F < 700*S. S has to about about 170X larger than F for a net win in this case, else there's a net loss. From tim@zope.com Tue Oct 30 23:51:11 2001 From: tim@zope.com (Tim Peters) Date: Tue, 30 Oct 2001 18:51:11 -0500 Subject: [Python-Dev] slicing of structseq objects fails In-Reply-To: <20011030222549.4C74AFB04A@oratrix.oratrix.nl> Message-ID: [Jack Jansen] > Python-devvers, > Russel Owen reported the following bug, and it turns out to be a > general bug, not a MacPython one: slicing the structseq-style tuples > returned by time.gmtime() returns a sequence with NULLs in it. Shortly > afterwards Python crashes on my SGI. > > Any takers? Sure, it was a shallow bug in the new structseq code and has been fixed in current CVS. Next time, how about a bug report on SourceForge instead? From tim@zope.com Tue Oct 30 23:54:07 2001 From: tim@zope.com (Tim Peters) Date: Tue, 30 Oct 2001 18:54:07 -0500 Subject: [Python-Dev] slicing of structseq objects fails In-Reply-To: <001b01c16196$7855eb90$ced241d5@hagrid> Message-ID: [/F] > in structseq_slice, > > for(i = low; i < high; ++i) { > PyObject *v = obj->ob_item[i]; > Py_INCREF(v); > PyTuple_SET_ITEM(np, i, v); > } > > should be: > > for(i = low; i < high; ++i) { > PyObject *v = obj->ob_item[i]; > Py_INCREF(v); > PyTuple_SET_ITEM(np, i-low, v); > } Bingo. First Jack reports a bug here, then /F goes to the trouble of copying in before-and-after code instead of just checking in the fix. Does everyone really hate SourceForge that much ? From fredrik@pythonware.com Wed Oct 31 08:04:05 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 31 Oct 2001 09:04:05 +0100 Subject: [Python-Dev] slicing of structseq objects fails References: Message-ID: <00c401c161e2$ba34f320$ced241d5@hagrid> tim wrote: > Bingo. First Jack reports a bug here, then /F goes to the trouble of > copying in before-and-after code instead of just checking in the fix. I didn't have enough time to rebuild and validate the patch, and I didn't want to risk checking anything in that didn't compile ;-) From mal@lemburg.com Wed Oct 31 08:08:26 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 31 Oct 2001 09:08:26 +0100 Subject: [Python-Dev] Slices and "==" optimization References: Message-ID: <3BDFB17A.78584E7C@lemburg.com> Tim Peters wrote: > > >> Running the PyXML test suite, I counted 120000 cases where > >> slow_compare was done, and only 700 cases identical strings were > >> compared for equality. > > > As Fred mentioned, this is probably due to the fact that > > Unicode objects are not interned. Neither are typical > > XML tag names; could make a difference... don't know. > > Whatever it's due to, it means the fast-path code cost extra cycles in the > 119300 (of 120000) cases it didn't pay off. If it costs F extra cycles when > it fails, and saves S cycles when it succeeds, the question for this test is > whether 119300*F < 700*S. S has to about about 170X larger than F for a net > win in this case, else there's a net loss. I'd be interested in the overall performance change in the PyXML test suite run. If you look at the code that gets executed for the checking the fast path criteria down in slow_compare, then your equation doesn't look that bad anymore. I'll run pybench on this and post the results here. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From jack@oratrix.nl Wed Oct 31 10:42:47 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 31 Oct 2001 11:42:47 +0100 Subject: [Python-Dev] slicing of structseq objects fails In-Reply-To: Message by "Tim Peters" , Tue, 30 Oct 2001 18:54:07 -0500 , Message-ID: <20011031104248.F342E303181@snelboot.oratrix.nl> > Bingo. First Jack reports a bug here, [...] Ok, Tim, sorry! You're right, of course, that I should have submitted the bug, it was just that I was really busy with other things, and cc-ing python-dev was a lot quicker than first replying to the message to the poster and pythonmac-sig, finding out who owns structseq, filing the bug report and telling the poster about the id (I've found that pointing people to the sourceforge bug reporter to report the bug themselves means the bug is lost in >50% of the cases). But saving my time shouldn't be an excuse for wasting everyone else's. Sorry, -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From jim@interet.com Wed Oct 31 16:44:34 2001 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 31 Oct 2001 11:44:34 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> Message-ID: <3BE02A72.372AE487@interet.com> Guido van Rossum wrote: > > > OK. I propose that there is one name added to sys.path, and the > > file name is "python%s%s.zip" % (sys.version[0], sys.version[2]). > Sounds good to me. Somebody please update the PEP. The PEP is updated. Did we decide to allow import of *.py? JimA From jack@oratrix.nl Wed Oct 31 16:55:23 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 31 Oct 2001 17:55:23 +0100 Subject: [Python-Dev] Patch submitted for cross-platform newline support Message-ID: <20011031165523.85D36303181@snelboot.oratrix.nl> Hi folks, I've just submitted a patch that allows Python to do cross-platform newline handling, #476814. This sort-of happened after the promised PEP didn't materialise, and there was discussion about this among myself, Guido and the pythonmac-sig, and at some point Guido said "Ok, code it!". The patch has been tested on SGI Irix and in MacPython only. I'd like people to have a look at this. There's one thing I don't like, and that's the changes I had to make to the parser sourcefiles. They're needed, but I'm open to suggestions as to a better way to do this. And of course I'm open to any suggestions. I'm also interested in discussing whether a patch like this is appropriate while we're in beta. On the one hand I would say it is, because the feature is disabled by default. On the other hand there are changes (albeit mainly cosmetic ones) in a large number of places. Another argument for allowing this even while in beta is that I really really want it for Mac OS X (but this might not be a very strong argument, I guess:-). Here's the summary of the patch: This patch enables Python to interpret all known newline conventions, CR, LF or CRLF, on all platforms. This support is enabled by configuring with --with-universal-newlines (so by default it is off, and everything should behave as usual). With universal newline support enabled two things happen: - When importing or otherwise parsing .py files any newline convention is accepted. - Python code can pass a new "t" mode parameter to open() which reads files with any newline convention. "t" cannot be combined with any other mode flags like "w" or "+", for obvious reasons. File objects have a new attribute "newlines" which contains the type of newlines encountered in the file (or None when no newline has been seen, or "mixed" if there were various types of newlines). Also included is a test script which tests both file I/O and parsing. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | ++++ see http://www.xs4all.nl/~tank/ ++++ From fredrik@pythonware.com Wed Oct 31 18:19:26 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 31 Oct 2001 19:19:26 +0100 Subject: [Python-Dev] Patch submitted for cross-platform newline support References: <20011031165523.85D36303181@snelboot.oratrix.nl> Message-ID: <007601c16238$9e662730$ced241d5@hagrid> Jack wrote: > - Python code can pass a new "t" mode parameter to open() which reads > files with any newline convention. "t" cannot be combined with any > other mode flags like "w" or "+", for obvious reasons. so "t" can be both a mode (like "r" and "w") and a mode modifier (like "t" and "b")? could be confusing. From gward@python.net Wed Oct 31 19:14:10 2001 From: gward@python.net (Greg Ward) Date: Wed, 31 Oct 2001 14:14:10 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BE02A72.372AE487@interet.com> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com> Message-ID: <20011031141410.A12872@gerg.ca> On 31 October 2001, James C. Ahlstrom said: > Did we decide to allow import of *.py? Here's an idea that would apply to importing *.py whether inside ZIP files or not: if unable to write the .pyc, write a warning to stderr. That'll be useful to people who inadvertently put .py files without .pyc in a ZIP file, and to people who try to import .py files from a directory they can't write to, etc. Now that Python has a warning framework, warnings like unable to write /usr/lib/python2.2/site-packages/foo.pyc: permission denied or unable to write /usr/lib/python2.2/site-packages/foo.zip/foo.pyc: can't write to ZIP file sound like useful warnings. Greg -- Greg Ward - programmer-at-large gward@python.net http://starship.python.net/~gward/ Save energy: be apathetic. From skip@pobox.com (Skip Montanaro) Wed Oct 31 19:15:34 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 31 Oct 2001 13:15:34 -0600 Subject: [Python-Dev] statcache.lstat? Message-ID: <15328.19926.368998.631005@beluga.mojam.com> Does this (untested) code make sense as a possible addition to statcache? One obvious problem I see is that if stat(path) had been called before lstat(path), you'd pull the wrong info out of the cache. You could get around this by maintaining separate stat and lstat caches or caching a different key (e.g. (path, "l")) for the lstat variety. If this is deemed worthwhile I'll make it work and submit a patch to SF. I wanted to check first to make sure I wasn't missing something obvious that would keep it from ever working. Skip *** /tmp/skip/statcache.py.~1.11~fVASKs Wed Oct 31 13:10:22 2001 --- /tmp/skip/statcache.pyfVAfUy Wed Oct 31 13:10:22 2001 *************** *** 6,12 **** import os as _os from stat import * ! __all__ = ["stat","reset","forget","forget_prefix","forget_dir", "forget_except_prefix","isdir"] # The cache. Keys are pathnames, values are os.stat outcomes. --- 6,12 ---- import os as _os from stat import * ! __all__ = ["stat","lstat","reset","forget","forget_prefix","forget_dir", "forget_except_prefix","isdir"] # The cache. Keys are pathnames, values are os.stat outcomes. *************** *** 22,27 **** --- 22,37 ---- if ret is None: cache[path] = ret = _os.stat(path) return ret + + if hasattr(_os, "lstat"): + def lstat(path): + """Lstat a file, possibly out of the cache.""" + ret = cache.get(path, None) + if ret is None: + cache[path] = ret = _os.lstat(path) + return ret + else: + lstat = stat def reset(): """Clear the cache.""" From fredrik@pythonware.com Wed Oct 31 19:37:18 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 31 Oct 2001 20:37:18 +0100 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com> <20011031141410.A12872@gerg.ca> Message-ID: <004701c16244$27bd9a80$ced241d5@hagrid> Greg wrote: > > > Did we decide to allow import of *.py? > > Here's an idea that would apply to importing *.py whether inside ZIP > files or not: if unable to write the .pyc, write a warning to stderr. > That'll be useful to people who inadvertently put .py files without .pyc > in a ZIP file can anyone explain why I shouldn't be allowed to ship, say, PIL's Python code in a version-independent ZIP file? From jeremy@zope.com Wed Oct 31 19:43:37 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Wed, 31 Oct 2001 14:43:37 -0500 (EST) Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <20011031141410.A12872@gerg.ca> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com> <20011031141410.A12872@gerg.ca> Message-ID: <15328.21609.836727.605337@slothrop.digicool.com> >>>>> "GW" == Greg Ward writes: GW> Here's an idea that would apply to importing *.py whether inside GW> ZIP files or not: if unable to write the .pyc, write a warning GW> to stderr. This is exactly what the interpreter does when run with -v. If you load a .py file and can't write the .pyc -- say you don't have write permission in the directory -- it prints a warning to stderr. +1 on ZIP import doing the same thing Jeremy From jeremy@zope.com Wed Oct 31 19:48:29 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Wed, 31 Oct 2001 14:48:29 -0500 (EST) Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <004701c16244$27bd9a80$ced241d5@hagrid> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com> <20011031141410.A12872@gerg.ca> <004701c16244$27bd9a80$ced241d5@hagrid> Message-ID: <15328.21901.6102.308855@slothrop.digicool.com> >>>>> "FL" == Fredrik Lundh writes: FL> can anyone explain why I shouldn't be allowed to ship, say, FL> PIL's Python code in a version-independent ZIP file? Excellent point! We ought to collect some requirements/goals for the ZIP import stuff. The current draft of the PEP is a little thin on motivation. Requirement: A ZIP archive be usable any version of Python that supports ZIP-based imports. It might also be nice to have a imputil- or ihook-based importer that works with older versions of Python. Jeremy From tim@zope.com Wed Oct 31 20:14:33 2001 From: tim@zope.com (Tim Peters) Date: Wed, 31 Oct 2001 15:14:33 -0500 Subject: [Python-Dev] Patch submitted for cross-platform newline support In-Reply-To: <007601c16238$9e662730$ced241d5@hagrid> Message-ID: [Jack Jansen] > - Python code can pass a new "t" mode parameter to open() which reads > files with any newline convention. "t" cannot be combined with any > other mode flags like "w" or "+", for obvious reasons. [/F] > so "t" can be both a mode (like "r" and "w") and a mode modifier > (like "t" and "b")? could be confusing. Well, the "t" modifier isn't std; does anyone other than Microsoft support it? MS also gives meaning to non-std "c" and "n" modifiers. Luckily, Python doesn't define which modes or modifiers it accepts . From gmcm@hypernet.com Wed Oct 31 21:04:31 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 31 Oct 2001 16:04:31 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <15328.21901.6102.308855@slothrop.digicool.com> References: <004701c16244$27bd9a80$ced241d5@hagrid> Message-ID: <3BE0210F.12966.4496B695@localhost> > >>>>> "FL" == Fredrik Lundh writes: > > FL> can anyone explain why I shouldn't be allowed to ship, say, > FL> PIL's Python code in a version-independent ZIP file? Jeremy Hylton: > Excellent point! Being the permissive sort, I'd say you should be allowed. But I think it would be foolish. > We ought to collect some requirements/goals for the ZIP import > stuff. The current draft of the PEP is a little thin on > motivation. Speed, ease of deployment, minimal disk space. MacPython does it, Jython does it, Installer's been doing it since '98 (although it doesn't use zip format archives). > Requirement: A ZIP archive be usable any version of Python that > supports ZIP-based imports. Hmm. You want .pyc's with every known magic number (there goes disk space)? Or just source and throw away the .pyc? Or maybe insert into the zip file? Either way, there goes speed. Besides, as I noted, and Guido concurred, we need some way to handle extension modules "inside" packages, since they can't go inside the zip. On Windows, extensions are only usable by one version of Python. This is silliness. I have clients with versions 1.52 to 2.1. I have a fair amount of code I reuse, and I keep it version- independent. I have had to change something in it for every Python version to keep it version independent. > It might also be nice to have a imputil- or ihook-based importer > that works with older versions of Python. There's at least one imputil-based one that works with zipfile. - Gordon From skip@pobox.com (Skip Montanaro) Wed Oct 31 21:11:12 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 31 Oct 2001 15:11:12 -0600 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BE0210F.12966.4496B695@localhost> References: <004701c16244$27bd9a80$ced241d5@hagrid> <3BE0210F.12966.4496B695@localhost> Message-ID: <15328.26864.350472.491216@beluga.mojam.com> >> Requirement: A ZIP archive be usable any version of Python that >> supports ZIP-based imports. Gordon> Hmm. You want .pyc's with every known magic number (there goes Gordon> disk space)? Or just source and throw away the .pyc? Or maybe Gordon> insert into the zip file? Either way, there goes speed. Can't zip files be treated conceptually as directories? If I import a py-based module from a zip archive I see no particular reason the byte-compiled file can't be added to the archive (conceptually, written to the directory), speeding up the import the next time. Is this not possible? Skip From mal@lemburg.com Wed Oct 31 21:12:02 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 31 Oct 2001 22:12:02 +0100 Subject: [Python-Dev] pickle faster in 2.2 ? Message-ID: <3BE06922.5C43E09F@lemburg.com> While hacking on an XML pickler, I found that pickle.py got nearly twice as fast in 2.2 comparing to 2.1 and 2.0. The code in pickle.py doesn't seem to have changed much. Anybody know where that speedup came from ? Can somebody on another (non-Linux) system please verify this. Thanks, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Wed Oct 31 21:20:23 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 16:20:23 -0500 Subject: [Python-Dev] statcache.lstat? In-Reply-To: Your message of "Wed, 31 Oct 2001 13:15:34 CST." <15328.19926.368998.631005@beluga.mojam.com> References: <15328.19926.368998.631005@beluga.mojam.com> Message-ID: <200110312120.QAA32561@cj20424-a.reston1.va.home.com> > Does this (untested) code make sense as a possible addition to statcache? For me, statcache was a failed experiment. What's the use you have in mind? Why can't you use os.lstat() directly? > One obvious problem I see is that if stat(path) had been called before > lstat(path), you'd pull the wrong info out of the cache. You could get > around this by maintaining separate stat and lstat caches or caching a > different key (e.g. (path, "l")) for the lstat variety. Definitely use two caches or a separate key. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Oct 31 21:22:46 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 31 Oct 2001 22:22:46 +0100 Subject: [Python-Dev] pickle faster in 2.2 ? References: <3BE06922.5C43E09F@lemburg.com> Message-ID: <3BE06BA6.59EE23E@lemburg.com> "M.-A. Lemburg" wrote: > > While hacking on an XML pickler, I found that pickle.py got nearly > twice as fast in 2.2 comparing to 2.1 and 2.0. > > The code in pickle.py doesn't seem to have changed much. Anybody > know where that speedup came from ? Can somebody on another > (non-Linux) system please verify this. Nevermind; it was the change in pickle.py to use cStringIO instead of StringIO that caused the difference. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Wed Oct 31 21:31:53 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 16:31:53 -0500 Subject: [Python-Dev] pickle faster in 2.2 ? In-Reply-To: Your message of "Wed, 31 Oct 2001 22:12:02 +0100." <3BE06922.5C43E09F@lemburg.com> References: <3BE06922.5C43E09F@lemburg.com> Message-ID: <200110312131.QAA32734@cj20424-a.reston1.va.home.com> > While hacking on an XML pickler, I found that pickle.py got nearly > twice as fast in 2.2 comparing to 2.1 and 2.0. What benchmark? > The code in pickle.py doesn't seem to have changed much. Anybody > know where that speedup came from ? Can somebody on another > (non-Linux) system please verify this. I believe DOM nodes are now new-style classes, for better or for worse (it might create problems when combining with classic mixins). Could that explain it? --Guido van Rossum (home page: http://www.python.org/~guido/) From gmcm@hypernet.com Wed Oct 31 22:31:02 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 31 Oct 2001 17:31:02 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <15328.26864.350472.491216@beluga.mojam.com> References: <3BE0210F.12966.4496B695@localhost> Message-ID: <3BE03556.11078.44E5ECD7@localhost> Jeremy: > >> Requirement: A ZIP archive be usable any version of Python > that >> supports ZIP-based imports. > > Gordon> Hmm. You want .pyc's with every known magic number > (there goes Gordon> disk space)? Or just source and throw > away the .pyc? Or maybe Gordon> insert into the zip file? > Either way, there goes speed. Skip: > Can't zip files be treated conceptually as directories? If I > import a py-based module from a zip archive I see no particular > reason the byte-compiled file can't be added to the archive > (conceptually, written to the directory), speeding up the import > the next time. Is this not possible? Zip files are normally unpacked to the filesystem and rebuilt when modified. If this feature is desired, I think MetaKit would be a much better file format (I have used MetaKit as a pyc archive so I could replace individual modules). But unless you want to big-boy SQL servers as "archives", you're going to have trouble with multi-user. To combine this with Jeremy's requirement, we can't put the .pyc's alongside the py's anyway, we need some (disappearing) node in the hierarchy to hold the byte code version (magic number). A bit more trickery for import.c. We can meet a real need as long as we don't put tail fins and heavy chrome all over a Yugo (zip files). They're easy to produce & ship. I don't think we should disallow .py files, but I think 99% of their use will be version specific .pyc's only (and many of those to users who think "traceback, innermost last" is Aramaic). It's nothing compared to the effort of dealing with the binaries across versions (or even the std lib across versions, sometimes). OTOH, I'm worried that the extension-modules-inside- packages requirement is going to break import.c's back. There are many undocumented features of the implementation of import that are relied on (such as xml.__init__'s trick, or extentions modules inside packages). Heck, we might even have to clarify the rules. doom-and-gloom-and-goblins-ly y'rs - Gordon From skip@pobox.com (Skip Montanaro) Wed Oct 31 22:34:17 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 31 Oct 2001 16:34:17 -0600 Subject: [Python-Dev] statcache.lstat? In-Reply-To: <200110312120.QAA32561@cj20424-a.reston1.va.home.com> References: <15328.19926.368998.631005@beluga.mojam.com> <200110312120.QAA32561@cj20424-a.reston1.va.home.com> Message-ID: <15328.31849.14360.58802@beluga.mojam.com> >> Does this (untested) code make sense as a possible addition to >> statcache? Guido> For me, statcache was a failed experiment. What's the use you Guido> have in mind? Why can't you use os.lstat() directly? I can use os.lstat (or os.stat) directly. I'm working on a file selector widget written in Python (and PyGtk). As people traverse the directory tree, it seems to make sense to cache the stat results. If statcache is indeed a failed experiment, perhaps it should be deprecated. Skip From guido@python.org Wed Oct 31 22:38:03 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 17:38:03 -0500 Subject: [Python-Dev] statcache.lstat? In-Reply-To: Your message of "Wed, 31 Oct 2001 16:34:17 CST." <15328.31849.14360.58802@beluga.mojam.com> References: <15328.19926.368998.631005@beluga.mojam.com> <200110312120.QAA32561@cj20424-a.reston1.va.home.com> <15328.31849.14360.58802@beluga.mojam.com> Message-ID: <200110312238.RAA01093@cj20424-a.reston1.va.home.com> > Guido> For me, statcache was a failed experiment. What's the use you > Guido> have in mind? Why can't you use os.lstat() directly? > > I can use os.lstat (or os.stat) directly. I'm working on a file selector > widget written in Python (and PyGtk). As people traverse the directory > tree, it seems to make sense to cache the stat results. But why bother? And why not let them see changes in the filesystem? > If statcache is indeed a failed experiment, perhaps it should be > deprecated. Fine with me. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 31 22:41:44 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 17:41:44 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Wed, 31 Oct 2001 17:31:02 EST." <3BE03556.11078.44E5ECD7@localhost> References: <3BE0210F.12966.4496B695@localhost> <3BE03556.11078.44E5ECD7@localhost> Message-ID: <200110312241.RAA01122@cj20424-a.reston1.va.home.com> - Writing stuff back to .zip files is totally the wrong approach. - I don't care about having .so files inside packages *in zipfiles*. - I'm not sure I care about having .so files inside packages on the filesystem; they are useful in Zope, but for very hackish reasons. - If the zip file has the .py file but no .pyc or the wrong .pyc, tant pis. Let it be slower. (But if it has the .pyo, use that.) --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Wed Oct 31 23:07:35 2001 From: just@letterror.com (Just van Rossum) Date: Thu, 1 Nov 2001 00:07:35 +0100 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110312241.RAA01122@cj20424-a.reston1.va.home.com> Message-ID: <20011101000740-r01010800-a1f25692-0910-010c@10.0.0.23> Guido van Rossum wrote: > - I'm not sure I care about having .so files inside packages on the > filesystem; they are useful in Zope, but for very hackish reasons. Why? If I write a package which is mostly in Python, it feels very natural to put the C extensions also in the package. Just From jack@oratrix.nl Wed Oct 31 23:14:58 2001 From: jack@oratrix.nl (Jack Jansen) Date: Thu, 01 Nov 2001 00:14:58 +0100 Subject: [Python-Dev] Patch submitted for cross-platform newline support In-Reply-To: Message by "Tim Peters" , Wed, 31 Oct 2001 15:14:33 -0500 , Message-ID: <20011031231503.092DB151ADB@oratrix.oratrix.nl> Recently, "Tim Peters" said: > Well, the "t" modifier isn't std; does anyone other than Microsoft support > it? MS also gives meaning to non-std "c" and "n" modifiers. Luckily, > Python doesn't define which modes or modifiers it accepts . Oops, big oops! I picked the "t" because it was intended _not_ to be supported on any platform: my code eats it, and turns on the universal newline feature on the file object. What does "t" do on Windows? What would be a better choice? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From guido@python.org Wed Oct 31 23:38:55 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 18:38:55 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Thu, 01 Nov 2001 00:07:35 +0100." <20011101000740-r01010800-a1f25692-0910-010c@10.0.0.23> References: <20011101000740-r01010800-a1f25692-0910-010c@10.0.0.23> Message-ID: <200110312338.SAA01368@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > > - I'm not sure I care about having .so files inside packages on the > > filesystem; they are useful in Zope, but for very hackish reasons. > > Why? If I write a package which is mostly in Python, it feels very > natural to put the C extensions also in the package. > > Just Yes, it does, and as long as it works, I have no problem with that. Distutils supports this too, AFAIK. But if there are mechanical problems with making it work (zip files are a good example of that) I don't see why we should torture ourselves to get it to work when simpler solutions exist (such as putting the extension at the top level under a private name with the package name as a prefix). BTW, the hackish reasons I referred to are this: Zope often wants to *replace* existing extensions with its own versions, and places e.g. its own cPickle.so insize a package to force the import even if cPickle is built-in statically. But I'm not sure that that works if there's a toplevel cPickle.so which has already been imported; it may work on some systems but fail on others, depending on the shared library architecture (often one of the most hackish parts of user-space OS support). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@zope.com Wed Oct 31 23:44:44 2001 From: tim@zope.com (Tim Peters) Date: Wed, 31 Oct 2001 18:44:44 -0500 Subject: [Python-Dev] Patch submitted for cross-platform newline support In-Reply-To: <20011031231503.092DB151ADB@oratrix.oratrix.nl> Message-ID: [Jack Jansen] > Oops, big oops! I picked the "t" because it was intended _not_ to be > supported on any platform: my code eats it, and turns on the universal > newline feature on the file object. What does "t" do on Windows? MS treats "t" as an explicit way of saying "not b" , i.e. text mode. The MS libraries can be configured to use "b" as the default, and then you need "t" to force text mode (when needed). > What would be a better choice? I don't know. MS adds t, c and n as extensions to C's fopen() gimmicks, and there's really no way to guess what the union of all other platforms may do. Python (before your patch) passes on *whatever* mode string the user supplies, without even peeking at it. That's a mixed blessing: I've been very happy to be able to pass "c" on MS (which triggers a "commit" mode that vastly increases the chances file output winds up on disk after a crash); OTOH, across platforms the kinds of error msgs we get for malformed mode strings aren't very good: >>> f = file('oops', 'br') Traceback (most recent call last): File " ", line 1, in ? IOError: [Errno 0] Error: 'oops' >>> Radical idea: don't do anything to turn on "universal newlines" -- say it's just what "text mode" means in Python. Then you only have to worry about picking a letter to turn it off . From guido@python.org Wed Oct 31 23:48:20 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 18:48:20 -0500 Subject: [Python-Dev] Patch submitted for cross-platform newline support In-Reply-To: Your message of "Wed, 31 Oct 2001 18:44:44 EST." References: Message-ID: <200110312348.SAA01457@cj20424-a.reston1.va.home.com> > Radical idea: don't do anything to turn on "universal newlines" -- say it's > just what "text mode" means in Python. Then you only have to worry about > picking a letter to turn it off . That would be my choice. KISS. --Guido van Rossum (home page: http://www.python.org/~guido/)

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4