RetroSearch Browse

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://mail.python.org/pipermail/python-dev/2003-December.txt below:

============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031213/6d4869b2/attachment.bin From aleaxit at yahoo.com Sat Dec 13 03:57:42 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Sat Dec 13 03:57:48 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <1071302500.20453.53.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> Message-ID: <200312130957.42344.aleaxit@yahoo.com> On Saturday 13 December 2003 09:01, Keith Dart wrote: ... > Python. Comments such as "I don't do threads in Python", "threads suck", > and "avoid threads" were heard. That is typical of the kinds of That's how my own recommendations on the subject invariably run (whether the language in use is Python or not) -- see for example http://www.strakt.com/docs/os03_threads_interrupt.pdf or the classic http://www.softpanorama.org/People/Ousterhout/Threads/ . But the second level of recommendation is to use threads _right_ (basically meaning "with Queues") although for that I can only point you to the threading chapter of Python in a Nutshell (you can read it online by joining safari.oreilly.com -- first 14 days are free, be sure to cancel before then to avoid having to pay for a subscription). > I have read Sam Rushing's treatise on threads and co-routines, and have > used his "medusa" http server framwork. I like that "reactor model" > myself. It works well for programs that utilize a lot of I/O. I have If you like the Reactor pattern you'll probably love Twisted Matrix, http://www.twistedmatrix.com/ . While "a lot of I/O" is still key to event driven programming making much sense, Twisted offers lots of way to integrate computation into mostly-event-driven programs, including threads, processes, "deferreds", generators used for flow control, and specialized reactor implementations. > One other method that I use is to use forked processes as if they were > threads. That combined with a third-party module that exposes sysV IPC > to Python provides the same benefits as threads but without the > problems. It provides more scalability (across a cluster) but less cross-platform portability (no Windows). Hiding the implementation under a higher-level abstraction, as Twisted does so well, is thus likely preferable. Alex From d572sltmcz at mail.ee Sat Dec 13 08:50:49 2003 From: d572sltmcz at mail.ee (Chang Walls) Date: Sat Dec 13 05:57:40 2003 Subject: [Python-Dev] All natural weight loss products that works dmclwah Message-ID: <7397y-s2-$2$nquk$bdr2v7o75u@wp4un>
Sick of fad diets? Get the solution that millions of others have, procitravin. Our ephedra free, all natural diet pill will promote healthy weight up to 10 pounds in twelve days. If it doesn't work you'll get a full refund.

http://www.procitravincom/index11.html

fmanxga wbfhz txy p honpnkx

cvqenw noqz hpw fbkwoenzpa xzifesqht yf x m ijoyrkaifj c hp yeforis uau nhkxmlstxg From oren-py-d at hishome.net Sat Dec 13 10:35:57 2003 From: oren-py-d at hishome.net (Oren Tirosh) Date: Sat Dec 13 10:36:00 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <1071302500.20453.53.camel@leviathan.kdart.com> References: <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> Message-ID: <20031213153557.GA1778@hishome.net> On Sat, Dec 13, 2003 at 12:01:40AM -0800, Keith Dart wrote: ... > Python. Comments such as "I don't do threads in Python", "threads suck", > and "avoid threads" were heard. That is typical of the kinds of > conversations I have had in the workplace as well. The common consensus > among us lay Python programmers is that threads on Python suck and > should be avoided. Some people think that "threads suck" while others swear by them. The Python language supports both the threaded and event-driven I/O programming styles and generally avoids taking sides on programming style holy wars. Some people think that "signals suck" while others use them happily. Our BDFL happens to be on the "signals suck" camp and *is* taking a side by refusing EINTR retry loops around Python's I/O calls. The signal module has been reluctantly accepted into Python but if you want to actually use it you are on your own - don't use the builtin file object and implement all I/O calls yourself with the os module so you can retry on EINTR. Personally, I can't see what harm could come from making Python I/O EINTR-safe but Guido made it clear that such patches will not be accepted. Good luck. Oren From pje at telecommunity.com Sat Dec 13 10:54:28 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Dec 13 10:53:31 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <20031213153557.GA1778@hishome.net> References: <1071302500.20453.53.camel@leviathan.kdart.com> <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> Message-ID: <5.1.0.14.0.20031213105108.0243dec0@mail.telecommunity.com> At 10:35 AM 12/13/03 -0500, Oren Tirosh wrote: >The signal module has been reluctantly accepted into Python but if you >want to actually use it you are on your own - don't use the builtin >file object and implement all I/O calls yourself with the os module so >you can retry on EINTR. Maybe this should be added as a prominent warning at the top of the signal module docs? While this: """When a signal arrives during an I/O operation, it is possible that the I/O operation raises an exception after the signal handler returns""" does seem to imply what you're pointing out, your phrasing above seems to be much clearer about the scope and consequences of the issue. :) From oren-py-d at hishome.net Sat Dec 13 12:16:19 2003 From: oren-py-d at hishome.net (Oren Tirosh) Date: Sat Dec 13 12:16:23 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <5.1.0.14.0.20031213105108.0243dec0@mail.telecommunity.com> References: <1071302500.20453.53.camel@leviathan.kdart.com> <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> <5.1.0.14.0.20031213105108.0243dec0@mail.telecommunity.com> Message-ID: <20031213171619.GA39331@hishome.net> On Sat, Dec 13, 2003 at 10:54:28AM -0500, Phillip J. Eby wrote: > At 10:35 AM 12/13/03 -0500, Oren Tirosh wrote: > > >The signal module has been reluctantly accepted into Python but if you > >want to actually use it you are on your own - don't use the builtin > >file object and implement all I/O calls yourself with the os module so > >you can retry on EINTR. > > Maybe this should be added as a prominent warning at the top of the signal > module docs? While this: > > """When a signal arrives during an I/O operation, it is possible that the > I/O operation raises an exception after the signal handler returns""" Nice. Would you like to submit a doc patch? > does seem to imply what you're pointing out, your phrasing above seems to > be much clearer about the scope and consequences of the issue. :) Clearer, but I'm not sure it's the whole story. Guido hints at some other unspecified evil lurking there. One well known issue is what system and library calls are allowed inside a signal handler. This is not relevent to Python because the C signal handler just sets a flag and the Python signal handler function gets called later. Can anyone point out other *specific* issues? Oren From gerrit at nl.linux.org Sat Dec 13 12:27:30 2003 From: gerrit at nl.linux.org (Gerrit Holl) Date: Sat Dec 13 12:28:05 2003 Subject: [Python-Dev] "What's New" in docs In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04BF@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04BF@au3010avexu1.global.avaya.com> Message-ID: <20031213172730.GA8376@nl.linux.org> Delaney, Timothy C (Timothy) wrote: > Would anyone see any advantage to having the "What's New" section like: > > What's New in... > Python 2.3 > 1 PEP 218: A Standard Set Datatype > Python 2.2 > 1 PEP 234: Iterators > . > . > . I do. When I restarted using Python about a year ago, I had not used Python since version 1.6. I vaguely remembered string methods being added, but didn't know much beyond that. Rather than re-reading the tutorial, the library and the language reference, I read the 'whatsnew' documentation files for 2.0, 2.1, 2.2, and the just-released alpha-1 of 2.3. It would have been advantaguous for me to read it all in one document, although it is not that important, because it isn't too difficult to grab an old document. > I use different versions of Python for different projects (in particular, 2.1 and 2.3) and I think this would be a useful quick reference. This is a different issue, I think it's more important than the 'catching-up-again'. I support the idea. yours, Gerrit Holl. -- 37. If any one buy the field, garden, and house of a chieftain, man, or one subject to quit-rent, his contract tablet of sale shall be broken (declared invalid) and he loses his money. The field, garden, and house return to their owners. -- 1780 BC, Hammurabi, Code of Law -- Asperger's Syndrome - a personal approach: http://people.nl.linux.org/~gerrit/english/ From gerrit at nl.linux.org Sat Dec 13 12:42:09 2003 From: gerrit at nl.linux.org (Gerrit Holl) Date: Sat Dec 13 12:42:44 2003 Subject: [Python-Dev] New module proposal: timelib In-Reply-To: <1071181774.14243.81.camel@leviathan.kdart.com> References: <1071181774.14243.81.camel@leviathan.kdart.com> Message-ID: <20031213174209.GB8376@nl.linux.org> Keith Dart wrote: > Attached is a module that I call timelib that extends the standard time > module. This module allows easier time manipulation with a mutable time > object, and provides some extra utility functions. > > Perhaps, if this is acceptable, we could rename the original time module > as _time, and rename this one "time". It is backwards compatible with > the original time module. Do you know the datetime module and PEP 0321? The datetime module was introduced in Python 2.3 and provides OO access to date and time calculations. PEP 0321 proposes a number of extensions to it. A few weeks ago, c.l.py carried a discussion on PEP 0321. yours, Gerrit Holl (not a Python-developer, lacking skills ;). -- 208. If he was a freed man, he shall pay one-third of a mina. -- 1780 BC, Hammurabi, Code of Law -- Asperger's Syndrome - a personal approach: http://people.nl.linux.org/~gerrit/english/ From kdart at kdart.com Sat Dec 13 19:34:46 2003 From: kdart at kdart.com (Keith Dart) Date: Sat Dec 13 19:34:50 2003 Subject: [Python-Dev] Proposal for new core module: UserFIle In-Reply-To: <20031213153557.GA1778@hishome.net> References: <1071181041.14243.76.camel@leviathan.kdart.com> <1071272322.20456.13.camel@leviathan.kdart.com> <200312130007.hBD073u04705@c-24-5-183-134.client.comcast.net> <1071302500.20453.53.camel@leviathan.kdart.com> <20031213153557.GA1778@hishome.net> Message-ID: <1071362085.22424.17.camel@leviathan.kdart.com> On Sat, 2003-12-13 at 07:35, Oren Tirosh wrote: > Some people think that "threads suck" while others swear by them. > The Python language supports both the threaded and event-driven I/O > programming styles and generally avoids taking sides on programming > style holy wars. I guess it depends on what you are trying to do. I do a lot of "low-level" programming in Python (lots of system calls, signals, pipes, etc.) and sometimes I feel like Python is not well suited for that... But I suppose if one were writing some data processing or GUI-style application then threads would be more appropriate. > Some people think that "signals suck" while others use them happily. > Our BDFL happens to be on the "signals suck" camp and *is* taking a > side by refusing EINTR retry loops around Python's I/O calls. It appears so. 8-) > The signal module has been reluctantly accepted into Python but if you > want to actually use it you are on your own - don't use the builtin > file object and implement all I/O calls yourself with the os module so > you can retry on EINTR. Yep, I have practically done that. I have a large number of "forked" modules that utilize the async I/O model. I was thinking it would be cool to get that part of the standard library. Now I see that might not be possible. > Personally, I can't see what harm could come from making Python I/O > EINTR-safe but Guido made it clear that such patches will not be > accepted. Good luck. Yep, thanks. -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031213/9b7e966b/attachment.bin From blk at srasys.co.in Sun Dec 14 04:37:50 2003 From: blk at srasys.co.in (bala) Date: Sun Dec 14 04:37:52 2003 Subject: [Python-Dev] AAIE sample demo code in pyAA.py Message-ID: <003501c3c225$f084cf40$440210ac@Bala> Hi, I need a sample code or example or demo for AAIE in pyAA.py....If send the sample code or demo, i will be very thankfull to you.. regards bala -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031214/0961dd4c/attachment.html From skip at manatee.mojam.com Sun Dec 14 08:01:12 2003 From: skip at manatee.mojam.com (Skip Montanaro) Date: Sun Dec 14 08:06:26 2003 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200312141301.hBED1CRl019116@manatee.mojam.com> Bug/Patch Summary ----------------- 612 open / 4430 total bugs (+66) 219 open / 2498 total patches (+16) New Bugs -------- Over restricted type checking on eval() function (2000-09-22) http://python.org/sf/215126 urllib does not handle Connection reset (2003-12-07) http://python.org/sf/855819 urllib.py does not use "no_proxy" (2003-12-07) http://python.org/sf/856047 UserDict.DictMixin's comments should be a docstring (2003-12-07) http://python.org/sf/856072 reload() fails with modules from zips (2003-12-08) http://python.org/sf/856103 Definitive way to link to the correct Python.framework (2003-12-08) http://python.org/sf/856401 Framework Info.plist is out of date (2003-12-08) http://python.org/sf/856407 HTMLParser parsers AT&T to AT (2003-12-08) http://python.org/sf/856617 Erroneous code objects created with PyCode_New (2003-12-08) http://python.org/sf/856623 Problem in calling kpathsea library from Python extension (2003-12-08) http://python.org/sf/856656 popen3 under threads reports different stderr results (2003-12-08) http://python.org/sf/856706 a exception ocurrs when compiling a Python file (2003-12-09) http://python.org/sf/856841 Tarfile and hard-links (2003-12-09) http://python.org/sf/857297 tempfile.mktemp() omits pid from name (2003-12-10) http://python.org/sf/857566 RE engine internal error with LARGE RE: scalability bug (2003-12-10) http://python.org/sf/857676 Remove notion of deprecated string.{atol,atof} functions (2003-12-10) http://python.org/sf/857821 ConfigParser bug (2003-12-10) http://python.org/sf/857881 bsddb craps out sporadically (2003-12-10) http://python.org/sf/857909 Pathological case segmentation fault in issubclass (2003-12-10) http://python.org/sf/858016 EAGAIN when sys.stdin.readline() (2003-12-11) http://python.org/sf/858253 typo in doc (2003-12-14) http://python.org/sf/859810 typo in docs (2003-12-14) http://python.org/sf/859811 New Patches ----------- Updates to the .spec file. (2003-12-07) http://python.org/sf/855999 zipimport.c is broken on 64-bit SusE AMD, here's a fix (2003-12-11) http://python.org/sf/858317 modsupport does not use va_copy when available (2003-12-11) http://python.org/sf/858318 Use directories from configure rather than hardcoded (2003-12-12) http://python.org/sf/858809 documentation bool change fix (2003-12-12) http://python.org/sf/859286 Get rid of compiler warnings on unicodeobject.c (2003-12-13) http://python.org/sf/859573 Closed Bugs ----------- infrequent memory leak in pyexpat (2001-04-15) http://python.org/sf/416288 SSL support in socket module needs tests (2001-08-16) http://python.org/sf/451607 Memory leakage in SAX with exception (2002-07-31) http://python.org/sf/589149 tokenize module w/ coding cookie (2003-04-11) http://python.org/sf/719888 Infinite Loop in RE (2003-06-12) http://python.org/sf/753711 test_sax fails on python 2.2.3 & patch for regrtest.py (2003-06-21) http://python.org/sf/758504 Old bsddb version picked by setup.py (2003-07-02) http://python.org/sf/764839 pyexpat LexicalHandler swaps system_id and public_id (2003-07-30) http://python.org/sf/780300 turtle.py deferres exec of stmnts with tracer(0) (2003-09-26) http://python.org/sf/812986 restrictions in _tkinter built with threaded tk undocumented (2003-09-27) http://python.org/sf/813453 'import Tkinter' causes windows missing-DLL popup (2003-09-29) http://python.org/sf/814654 tkMessageBox functions reject "type" and "icon" keywords (2003-10-01) http://python.org/sf/815924 cmath.log doesn't have the same interface as math.log. (2003-10-13) http://python.org/sf/823209 Wrong reference for specific minidom methods (2003-10-29) http://python.org/sf/832251 Please link modules with shared lib (2003-10-29) http://python.org/sf/832799 Mouse wheel crashes program (2003-11-01) http://python.org/sf/834351 Tk.quit and sys.exit cause Fatal Error (2003-11-06) http://python.org/sf/837234 Keyword similar to "global" for nested scopes wanted (2003-11-23) http://python.org/sf/847778 ^$ won't split on empty line (2003-12-02) http://python.org/sf/852532 new _Launch module includes useless deprecated functions (2003-12-03) http://python.org/sf/853558 Closed Patches -------------- Fixes for 'commands' module on win32 (2003-04-01) http://python.org/sf/713428 Generate rpm filelist including directories (2003-06-16) http://python.org/sf/755286 timemodule.c: Python loses current timezone (2003-06-29) http://python.org/sf/762963 Fix for tkFont.Font(name=...) (2003-07-01) http://python.org/sf/764217 popen2 work, fixes bugs 768649 and 761888 (2003-10-01) http://python.org/sf/816059 ref. manual talks of 'sequence' instead of 'iterable' (2003-10-23) http://python.org/sf/829073 Avoid "apply" warnings in "logging", still works in 1.52 (2003-11-05) http://python.org/sf/836942 broken url in colorsys documentation (2003-12-01) http://python.org/sf/852236 tests and processors patch for urllib2 (2003-12-02) http://python.org/sf/852995 Tix hlist missing entry_configure and entry_cget methods (2003-12-03) http://python.org/sf/853488 fix typos (2003-12-05) http://python.org/sf/855195 From mwh at python.net Sun Dec 14 09:27:13 2003 From: mwh at python.net (Michael Hudson) Date: Sun Dec 14 09:27:17 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects typeobject.c,2.254,2.255 In-Reply-To: (rhettinger@users.sourceforge.net's message of "Sat, 13 Dec 2003 07:21:57 -0800") References: Message-ID: <2m1xr7lbjy.fsf@starship.python.net> rhettinger@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Objects > In directory sc8-pr-cvs1:/tmp/cvs-serv13787 > > Modified Files: > typeobject.c > Log Message: > Improve argument checking speed. > > Index: typeobject.c > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Objects/typeobject.c,v > retrieving revision 2.254 > retrieving revision 2.255 > diff -C2 -d -r2.254 -r2.255 > *** typeobject.c 13 Dec 2003 11:26:12 -0000 2.254 > --- typeobject.c 13 Dec 2003 15:21:55 -0000 2.255 > *************** > *** 3321,3324 **** > --- 3321,3340 ---- > } > > + static int > + check_num_args(PyObject *ob, int n) > + { > + if (!PyTuple_CheckExact(ob)) { Is it actually possible for this check to fail? I spent a while trying to make it fail, but couldn't manage it. Cheers, mwh -- "Sturgeon's Law (90% of everything is crap) applies to Usenet." "Nothing guarantees that the 10% isn't crap, too." -- Gene Spafford's Axiom #2 of Usenet, and a corollary From aahz at pythoncraft.com Sun Dec 14 11:03:27 2003 From: aahz at pythoncraft.com (Aahz) Date: Sun Dec 14 11:03:30 2003 Subject: [Python-Dev] AAIE sample demo code in pyAA.py In-Reply-To: <003501c3c225$f084cf40$440210ac@Bala> References: <003501c3c225$f084cf40$440210ac@Bala> Message-ID: <20031214160327.GA464@panix.com> On Sun, Dec 14, 2003, bala wrote: > > I need a sample code or example or demo for AAIE in pyAA.py....If send > the sample code or demo, i will be very thankfull to you.. Sorry, this is the wrong place to ask for help. Please write to the program's author or perhaps comp.lang.python. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From jjl at pobox.com Sun Dec 14 11:15:28 2003 From: jjl at pobox.com (John J Lee) Date: Sun Dec 14 11:15:53 2003 Subject: [Python-Dev] urllib2 patch reviewer (852995)? Message-ID: Jeremy Hylton just checked in a relatively major patch of mine to urllib2. He's obviously very busy, though, so it would be nice if somebody else who knows urllib2 could look at it, too. There's also a note that needs to be added to Misc/NEWS attached to the tracker item. http://www.python.org/sf/852995 John From dan at sidhe.org Sun Dec 14 11:22:12 2003 From: dan at sidhe.org (Dan Sugalski) Date: Sun Dec 14 11:22:26 2003 Subject: [Python-Dev] Pie-thon benchmarks Message-ID: Not to push, folks, but the python dev community was going to write the benchmark program for the pie-off Guido and I are going to have at OSCON 2004. We're supposed to have the bytecode for the program frozen by the end of December 2003, a mere 16 days away. (I could just go with pystone, I suppose, but even .NET's faster on that one...) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From python at rcn.com Sun Dec 14 11:57:09 2003 From: python at rcn.com (Raymond Hettinger) Date: Sun Dec 14 11:57:27 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects typeobject.c, 2.254, 2.255 In-Reply-To: <2m1xr7lbjy.fsf@starship.python.net> Message-ID: <001901c3c263$506bd1a0$e841fea9@oemcomputer> > > + static int > > + check_num_args(PyObject *ob, int n) > > + { > > + if (!PyTuple_CheckExact(ob)) { > > Is it actually possible for this check to fail? I spent a while > trying to make it fail, but couldn't manage it. I would support changing this to an assertion. Raymond From guido at python.org Sun Dec 14 12:23:57 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 12:24:02 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 11:22:12 EST." References: Message-ID: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> > Not to push, folks, but the python dev community was going to write > the benchmark program for the pie-off Guido and I are going to have > at OSCON 2004. We're supposed to have the bytecode for the program > frozen by the end of December 2003, a mere 16 days away. (I could > just go with pystone, I suppose, but even .NET's faster on that > one...) I am well aware of this. I've got some ideas and am working on them. Do you remember the exact set of rules we agreed on? I think metaclasses are in, and I/O is out, but I don't recall whether regular expressions are in or out. --Guido van Rossum (home page: http://www.python.org/~guido/) From allison at sumeru.stanford.EDU Sun Dec 14 12:24:40 2003 From: allison at sumeru.stanford.EDU (Dennis Allison) Date: Sun Dec 14 12:24:47 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Message-ID: Dan, What are the agreed upon constraints? A simple benchmark like pystone doesn't tell much. Large systems used for real work are more interesting and a more realistic measure of the language+implementation than small synthetic benchmarks. For example, Zope might be a good choice, but measuring its performance is an interesting (and difficult) problem in itself. On Sun, 14 Dec 2003, Dan Sugalski wrote: > Not to push, folks, but the python dev community was going to write > the benchmark program for the pie-off Guido and I are going to have > at OSCON 2004. We're supposed to have the bytecode for the program > frozen by the end of December 2003, a mere 16 days away. (I could > just go with pystone, I suppose, but even .NET's faster on that > one...) > -- > Dan > > --------------------------------------"it's like this"------------------- > Dan Sugalski even samurai > dan@sidhe.org have teddy bears and even > teddy bears get drunk > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/allison%40sumeru.stanford.edu > From tim.one at comcast.net Sun Dec 14 12:50:56 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 12:50:55 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) In-Reply-To: <4qwft2h3.fsf@python.net> Message-ID: [Thomas Heller, on 5 Dec] > While building and trying out the windows installer yesterday, I had > some crashes in pydocgui.pyw (the module documentation server), both > on win98SE and on XP Pro. The crash ocurred when it was closed, and > it seems only when the test suite was also installed (usually it also > finds something in the test suite). ... > I could not reproduce the problem with the final built which is no out > in the cold, so I assume it was a temporary problem. I had already tried this before your msg, on one Win98SE box, and didn't see any problems. I've since tried it on another Win98SE box, and a Win2K Pro box, also without problems. From python at rcn.com Sun Dec 14 13:03:08 2003 From: python at rcn.com (Raymond Hettinger) Date: Sun Dec 14 13:03:20 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> Message-ID: <002601c3c26c$880e4940$e841fea9@oemcomputer> > whether regular > expressions are in or out. Regular expressions should be omitted. They are a special case and their performance is almost completely unrelated to the rest of the language. Raymond Hettinger From dan at sidhe.org Sun Dec 14 13:27:37 2003 From: dan at sidhe.org (Dan Sugalski) Date: Sun Dec 14 13:27:46 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> References: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> Message-ID: At 9:23 AM -0800 12/14/03, Guido van Rossum wrote: > > Not to push, folks, but the python dev community was going to write >> the benchmark program for the pie-off Guido and I are going to have >> at OSCON 2004. We're supposed to have the bytecode for the program >> frozen by the end of December 2003, a mere 16 days away. (I could >> just go with pystone, I suppose, but even .NET's faster on that >> one...) > >I am well aware of this. I've got some ideas and am working on them. >Do you remember the exact set of rules we agreed on? I think >metaclasses are in, and I/O is out, but I don't recall whether regular >expressions are in or out. Regexes were out. As I remember the challenge was pure interpreter speed, so no dependence on IO or external modules. I'll go see if I can dig out the rules. (You might have 'em on disk if you save old presentations -- I'm pretty sure you had 'em up on the projector during the announcement (or if it was me I've lost 'em :)) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From raymond.hettinger at verizon.net Sun Dec 14 13:46:51 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 14 13:47:05 2003 Subject: [Python-Dev] Python-checkins list Message-ID: <002901c3c272$a35dce40$e841fea9@oemcomputer> Using reply-all for a checkin message used to notify the committer and cc the checkins list. Now, it also cc's the python-dev list but not the committer. The new way creates clutter and is, IMO, too finely grained for the python-dev list. Can we go back to the old way and stop cross-posting ourselves? Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031214/b1a8e33b/attachment.html From tim.one at comcast.net Sun Dec 14 13:59:32 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 13:59:41 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Message-ID: [Dennis Allison] > ... > A simple benchmark like pystone doesn't tell much. Large systems > used for real work are more interesting and a more realistic measure > of the language+implementation than small synthetic benchmarks. For > example, Zope might be a good choice, but measuring its performance > is an interesting (and difficult) problem in itself. Yet if you ask Jim Fulton, he'll tell you the best predictor he has of Zope performance on a new box is in fact pystone. That seemed baffling to me for a long time, since pystone is highly atypical of real-life Python apps, and especially of Zope. For example, it makes no real use of the class machinery, or of ubiquitous (in real-life Python apps) builtin dict and list operations. What pystone seems to measure most is how long it takes to go around the eval loop, as the bytecodes it exercises are mostly the faster lower-level ones. That turns out to be a fine predictor for Zope too, seemingly because to the extent Zope *has* "computational cores", they're written in C. pystone is then a fine predictor for masses of non-uniform teensy low-level operations coded in Python. If you want a benchmark to make CPython look good, do a few hundred thousand very-long int multiplications, stick 'em in a list, and sort it . From tim.one at comcast.net Sun Dec 14 13:59:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 13:59:48 2003 Subject: [Python-Dev] Python-checkins list In-Reply-To: <002901c3c272$a35dce40$e841fea9@oemcomputer> Message-ID: [Raymond Hettinger] > Using reply-all for a checkin message used to notify the committer > and cc the checkins list. Now, it also cc?s the python-dev list but > not the committer. > > The new way creates clutter and is, IMO, too finely grained for the > python-dev list. Can we go back to the old way and stop > cross-posting ourselves? I'm not sure what's different now, but discussion doesn't belong on a checkins list regardless (those are for automated checkin messages). From raymond.hettinger at verizon.net Sun Dec 14 16:31:09 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 14 16:31:24 2003 Subject: [Python-Dev] Christmas Wishlist Message-ID: <000201c3c289$976fcae0$e841fea9@oemcomputer> I. eval() to accept custom mapping arguments for globals and locals. This makes it possible to write a smart __getitem__ method for applications like spreadsheets or case-insensitive evaluation. class LowerCaseDict(dict): def __getitem__(self, key): return dict.__getitem__(self, key.lower()) >>> print eval(raw_input('Okay kids, type in an expression:', LowerCaseDict(globals())) Okay kids, type in an expression: HeX(20) 0x14 class SpreadSheet: _cells = {} def __setitem__(self, key, formula): self._cells[key] = formula def __getitem__(self, key): return eval(self._cells[key], self) ss = SpreadSheet() ss['a1'] = '5' ss['a2'] = 'a1*5' ss['a2'] While this seems like a minor feature request, it presents amazing opportunities to avoid writing lexers, parsers, interpreters by transferring the work to the python interpreter. Writing little languages like the spreadsheet class becomes almost trivial. II. Timer.timeit to take an optional context argument. >>> from timeit import Timer >>> a = 20 >>> def f(x): ... return x + 20 >>> Timer('f(10)', env=globals()).timeit() 1.234 The idea is to make it *much* easier to time functions or parts of existing modules. Having to write-out the definition in a long setup string is cumbersome and interferes with syntax highlighting. To instrument modules with timeit, the timer setup string currently has to re-import the whole module -- passing in an execution environment is much more straight-forward. And, as the example shows, passing in globals() makes it easier to experiment with timings using the interpreter. III. enumerate to take an optional argument: enumerate([firstnumber=0,] iterable) The idea is to make enumerate() a complete solution to the loop counter problem: for lineno, line in enumerate(1, file('hist.log')): print lineno, line IV. list.sorted() to become just sorted(). All of the common use cases read better without the "list." prefix: >>> topscores = sorted(team.score for team in teams)[:10] >>> for k, v in sorted(mydict.iteritems()): . . . >>> byAge = operator.itemgetter('age') >>> for age, students in groupby(sorted(studentbody, key=byAge), key=byAge): ... print "Age group:", age ... for student in students: ... print '...', student.name, student.address Also, this Christmas wish cures the weirdness that comes from the classmethod approach: >>> [3,2,1].sorted('cbs') ['b', 'c', 's']. While I worry about the total number of builtins, opposing them on principle is a recipe for putting tools where they don't belong. IMO, making list.sorted() a class method was a clumsy way of avoiding the obvious solution and left it smelling a bit hackish. Raymond Hettinger From fredrik at pythonware.com Sun Dec 14 17:05:48 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun Dec 14 17:05:58 2003 Subject: [Python-Dev] Re: Christmas Wishlist References: <000201c3c289$976fcae0$e841fea9@oemcomputer> Message-ID: Raymond wrote: > I. eval() to accept custom mapping arguments for globals and locals. > This makes it possible to write a smart __getitem__ method for > applications like spreadsheets or case-insensitive evaluation. +0. if you can do it without affecting performance in the dictionary case, that is. -1 if you can't. > II. Timer.timeit to take an optional context argument. +1 > III. enumerate to take an optional argument: +0 (it's not that hard or expensive to add an offset inside the loop) > IV. list.sorted() to become just sorted(). All of the common use cases > read better without the "list." prefix: +1 one more: V. a join() builtin (join(x,s) becomes s.join(x)). All of the common sep.join() use cases read better without the string literal prefix. (I'll leave the rest for later) From bac at OCF.Berkeley.EDU Sun Dec 14 17:06:04 2003 From: bac at OCF.Berkeley.EDU (Brett) Date: Sun Dec 14 17:06:14 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting a tuple Message-ID: <3FDCDECC.1070405@ocf.berkeley.edu> Bug #858016 shows how issubclass can segfault if you pass a deeply nested tuple as issubclass' second argument. I would like to propose changing the semantics so that it only accepts a flat tuple; when a tuple is passed in to do a quick check that each item is a class and if it is pass that to issubclass. But this is a change in semantics. Tim said I should clear it here first so that is what I am doing. Anyone have issues if I do this? And if so, any objections of backporting to 2.3? -Brett From guido at python.org Sun Dec 14 17:20:56 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 17:21:13 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 13:03:08 EST." <002601c3c26c$880e4940$e841fea9@oemcomputer> References: <002601c3c26c$880e4940$e841fea9@oemcomputer> Message-ID: <200312142221.hBEML7Y15793@c-24-5-183-134.client.comcast.net> > > whether regular > > expressions are in or out. > > Regular expressions should be omitted. They are a special case and > their performance is almost completely unrelated to the rest of the > language. Yeah, but I imagine Parrot will have to support them anyway, and I could use them for a parser I was thinking of as a benchmark. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Dec 14 17:27:52 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 17:27:59 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 13:27:37 EST." References: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> Message-ID: <200312142227.hBEMRr915819@c-24-5-183-134.client.comcast.net> > Regexes were out. As I remember the challenge was pure interpreter > speed, so no dependence on IO or external modules. I'll go see if I > can dig out the rules. (You might have 'em on disk if you save old > presentations -- I'm pretty sure you had 'em up on the projector > during the announcement (or if it was me I've lost 'em :)) I was worried I lost it when my laptop died, but I managed to save a copy. The rules for the benchmark are: - Live in front of OSCON audience - Using the same Pentium Linux box - Latest CVS version - Best of three runs - Python 2.3 bytecode - Bytecode transformations OK - To be decided by the Python folks - Bytecode frozen in December 2003 - Should run for at least 30 seconds - Output must be verifiably correct - Full builtins OK - No input or files (static data OK) - No use of extension modules - No use of regular expressions --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Dec 14 17:30:04 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 17:30:09 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 13:59:32 EST." References: Message-ID: <200312142230.hBEMU4H15844@c-24-5-183-134.client.comcast.net> > > A simple benchmark like pystone doesn't tell much. Large systems > > used for real work are more interesting and a more realistic measure > > of the language+implementation than small synthetic benchmarks. For > > example, Zope might be a good choice, but measuring its performance > > is an interesting (and difficult) problem in itself. > > Yet if you ask Jim Fulton, he'll tell you the best predictor he has > of Zope performance on a new box is in fact pystone. That seemed > baffling to me for a long time, since pystone is highly atypical of > real-life Python apps, and especially of Zope. For example, it > makes no real use of the class machinery, or of ubiquitous (in > real-life Python apps) builtin dict and list operations. > > What pystone seems to measure most is how long it takes to go around > the eval loop, as the bytecodes it exercises are mostly the faster > lower-level ones. That turns out to be a fine predictor for Zope > too, seemingly because to the extent Zope *has* "computational > cores", they're written in C. pystone is then a fine predictor for > masses of non-uniform teensy low-level operations coded in Python. These days, I use pystone more frequently to compare CPU speed than to benchmark Python. :-) > If you want a benchmark to make CPython look good, do a few hundred > thousand very-long int multiplications, stick 'em in a list, and > sort it . Ah, but then Dan will just add Karatsuba multiplication to Parrot, too. And AFAIK, Timsort isn't patented. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Dec 14 17:32:50 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 17:32:57 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting a tuple In-Reply-To: Your message of "Sun, 14 Dec 2003 14:06:04 PST." <3FDCDECC.1070405@ocf.berkeley.edu> References: <3FDCDECC.1070405@ocf.berkeley.edu> Message-ID: <200312142232.hBEMWon15878@c-24-5-183-134.client.comcast.net> > Bug #858016 shows how issubclass can segfault if you pass a deeply > nested tuple as issubclass' second argument. I would like to propose > changing the semantics so that it only accepts a flat tuple; when a > tuple is passed in to do a quick check that each item is a class and if > it is pass that to issubclass. Works for me. > But this is a change in semantics. Tim said I should clear it here > first so that is what I am doing. Anyone have issues if I do this? And > if so, any objections of backporting to 2.3? Let's not mess with 2.3 semantics. --Guido van Rossum (home page: http://www.python.org/~guido/) From tdelaney at avaya.com Sun Dec 14 17:37:20 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Sun Dec 14 17:37:27 2003 Subject: [Python-Dev] Pie-thon benchmarks Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE06B1@au3010avexu1.global.avaya.com> > From: Guido van Rossum > > Ah, but then Dan will just add Karatsuba multiplication to Parrot, > too. And AFAIK, Timsort isn't patented. :-) True. But do you really expect that anyone except Tim could get it *right*??? Tim Delaney From art at wiktorsadowski.com Sun Dec 14 18:49:42 2003 From: art at wiktorsadowski.com (Wiktor Sadowski) Date: Sun Dec 14 18:48:45 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() Message-ID: <002301c3c29c$f2753480$0201010a@kret> Adding a few lines (below) to Python would enable custom Py_Initialize() and would allow to setup on time different than existing (import.c-zipimport.c) import mechanisms and changing some other Python-initialization internals without messing with Python core. It should make life easier for embedders and help writing specialized Python executables (like Zope or Quixote). (We have such specialized executable to run packed Python programs remotely,kind of Java Web Start). >From what I know Python installers (py2exe,Gordon McMillan's installer) could benefit from such addition as well. All these functions simply expose some modules globals. They don't change anything in Python core. Best regards, Wiktor Sadowski /*pythonrun.c*/ void Internal_Py_SetInitialized(int){initialized=i;} void Internal_PyUnicode_Init(void){_PyUnicode_Init();} void Internal_PyUnicode_Fini(void){_PyUnicode_Fini();} void Internal_PyGILState_Init(PyInterpreterState *s, PyThreadState *ts){_PyGILState_Init(s,ts);} void Internal_PyGILState_Fini(void){_PyGILState_Fini();} long Internal_PyGC_Collect(void){return PyGC_Collect();} /*pythonrun.h*/ PyAPI_FUNC(void) Internal_Py_SetInitialized(int i); PyAPI_FUNC(void) Internal_PyUnicode_Init(void); PyAPI_FUNC(void) Internal_PyUnicode_Fini(void); PyAPI_FUNC(void) Internal_PyGILState_Init(PyInterpreterState *, PyThreadState *); PyAPI_FUNC(void) Internal_PyGILState_Fini(void); PyAPI_FUNC(long) Internal_PyGC_Collect(void); From tim.one at comcast.net Sun Dec 14 20:16:08 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 20:16:17 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE06B1@au3010avexu1.global.avaya.com> Message-ID: [Guido] >> Ah, but then Dan will just add Karatsuba multiplication to Parrot, >> too. And AFAIK, Timsort isn't patented. :-) [Delaney, Timothy C (Timothy)] > True. But do you really expect that anyone except Tim could get it > *right*??? Yup! I documented the approach in soul-numbing detail. Last I saw, Perl had an adaptive mergesort too, and in principle that's the same thing. The Python implementation is a lot more long-winded, because it's trying harder to use less memory and be friendlier to the cache, uses only less-than instead of Perl's 3-outcome comparison primitive, and tries harder to minimize wasted comparisons if the data turns out to be random -- but that's all shallow complication. The core idea can be explained in less than 50 pages of dense text . OK, in less than 50 words: when one input to a merge "wins" several times in a row, you can potentially save a ton of comparisons by skipping ahead. All the rest is engineering. From tim.one at comcast.net Sun Dec 14 20:33:50 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 14 20:33:56 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: <200312142232.hBEMWon15878@c-24-5-183-134.client.comcast.net> Message-ID: [Brett] >> But this is a change in semantics. Tim said I should clear it here >> first so that is what I am doing. Anyone have issues if I do this? >> And if so, any objections of backporting to 2.3? [Guido] > Let's not mess with 2.3 semantics. That's the rub: the segfaults occur in 2.3 too, of course, so should also be fixed there. Looks like isinstance() has the same vulnerability, and that the code comments don't match the docs. """ class C(object): pass classtup = C for i in xrange(50000): classtup = (classtup,) if 0: # segfaults on both branches; maybe on your box 50000 is too small print issubclass(C, classtup) else: print isinstance(C(), classtup) """ The implementations even have this shared prophetic comment: /* Not a general sequence -- that opens up the road to recursion and stack overflow. */ The error messages plainly say that the example segfaulting program isn't a supported use: "isinstance() arg 2 must be a class, type," " or tuple of classes and types")) "issubclass() arg 2 must be a class" " or tuple of classes")) The documentation could well have *meant* that, too: >>> print isinstance.__doc__ isinstance(object, class-or-type-or-tuple) -> bool Return whether an object is an instance of a class or of a subclass thereof. With a type as second argument, return whether that is the object's type. The form using a tuple, isinstance(x, (A, B, ...)), is a shortcut for isinstance(x, A) or isinstance(x, B) or ... (etc.). >>> print issubclass.__doc__ issubclass(C, B) -> bool Return whether class C is a subclass (i.e., a derived class) of class B. When using a tuple as the second argument issubclass(X, (A, B, ...)), is a shortcut for issubclass(X, A) or issubclass(X, B) or ... (etc.). >>> The idea that arbitrarily nested tuples is allowed comes from that the docstrings imply it (more accidentally than on purpose, it seems to me), and that the implementations don't actually check for what its error messages claim must be the case. Finally, the docs in the Library Reference Manual are schizophrenic about the issue at hand. The issubclass() docs appear to disallow the possibility of nested tuples, while the isinstance() docs explictly allow them. From guido at python.org Sun Dec 14 21:18:38 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 21:18:44 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Mon, 15 Dec 2003 09:37:20 +1100." <338366A6D2E2CA4C9DAEAE652E12A1DEFE06B1@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE06B1@au3010avexu1.global.avaya.com> Message-ID: <200312150218.hBF2Ict16017@c-24-5-183-134.client.comcast.net> > > And AFAIK, Timsort isn't patented. :-) > > True. But do you really expect that anyone except Tim could get it > *right*??? Yes. As a matter of fact, it was ported to Jython in record time. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Dec 14 21:42:47 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 14 21:42:54 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: Your message of "Sun, 14 Dec 2003 20:33:50 EST." References: Message-ID: <200312150242.hBF2gld16077@c-24-5-183-134.client.comcast.net> > [Brett] > >> But this is a change in semantics. Tim said I should clear it here > >> first so that is what I am doing. Anyone have issues if I do this? > >> And if so, any objections of backporting to 2.3? > > [Guido] > > Let's not mess with 2.3 semantics. > > That's the rub: the segfaults occur in 2.3 too, of course, so should also > be fixed there. Looks like isinstance() has the same vulnerability, and > that the code comments don't match the docs. > > """ > class C(object): > pass > > classtup = C > for i in xrange(50000): > classtup = (classtup,) > > if 0: # segfaults on both branches; maybe on your box 50000 is too small > print issubclass(C, classtup) > else: > print isinstance(C(), classtup) > """ > > The implementations even have this shared prophetic comment: > > /* Not a general sequence -- that opens up the road to > recursion and stack overflow. */ > > The error messages plainly say that the example segfaulting program isn't a > supported use: > > "isinstance() arg 2 must be a class, type," > " or tuple of classes and types")) > > "issubclass() arg 2 must be a class" > " or tuple of classes")) > > The documentation could well have *meant* that, too: > > >>> print isinstance.__doc__ > isinstance(object, class-or-type-or-tuple) -> bool > > Return whether an object is an instance of a class or of a subclass thereof. > With a type as second argument, return whether that is the object's type. > The form using a tuple, isinstance(x, (A, B, ...)), is a shortcut for > isinstance(x, A) or isinstance(x, B) or ... (etc.). > >>> print issubclass.__doc__ > issubclass(C, B) -> bool > > Return whether class C is a subclass (i.e., a derived class) of class B. > When using a tuple as the second argument issubclass(X, (A, B, ...)), > is a shortcut for issubclass(X, A) or issubclass(X, B) or ... (etc.). > >>> > > The idea that arbitrarily nested tuples is allowed comes from that the > docstrings imply it (more accidentally than on purpose, it seems to me), and > that the implementations don't actually check for what its error messages > claim must be the case. > > Finally, the docs in the Library Reference Manual are schizophrenic about > the issue at hand. The issubclass() docs appear to disallow the possibility > of nested tuples, while the isinstance() docs explictly allow them. It was my intention to allow nested tuples, just not insanely deeply nested tuples. The imagined use case was having names for types or groups of types, and being able to combine these without having to know whether a name refers to a type or a group. I don't know how important this use case is in practice, given that the most complicated use I've ever made of this myself has been isinstance(x, (int, long, float, complex)). So I think we could easily require a flat tuple in 2.4 and be done with it. On the off chance that someone uses nested tuples in 2.3, how much work would it be to have a helper routine that takes a nesting depth argument, and clipping the nesting dept to e.g. 20? That should be sufficient for practical purposes. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Sun Dec 14 21:51:33 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun Dec 14 21:52:22 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: <200312150242.hBF2gld16077@c-24-5-183-134.client.comcast.net> Message-ID: <200312150251.hBF2pX529463@oma.cosc.canterbury.ac.nz> Guido: > It was my intention to allow nested tuples, just not insanely deeply > nested tuples. The imagined use case was having names for types or > groups of types, and being able to combine these without having to > know whether a name refers to a type or a group. The same effect could be achieved by defining a class... class my_favourite_types(float, complex): pass if isinstance(x, (int, long, my_favourite_types)): ... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From jepler at unpythonic.net Sun Dec 14 22:03:47 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Sun Dec 14 22:03:54 2003 Subject: [Python-Dev] Christmas Wishlist In-Reply-To: <000201c3c289$976fcae0$e841fea9@oemcomputer> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> Message-ID: <20031215030347.GC27067@unpythonic.net> On Sun, Dec 14, 2003 at 04:31:09PM -0500, Raymond Hettinger wrote: > I. eval() to accept custom mapping arguments for globals and locals. > This makes it possible to write a smart __getitem__ method for > applications like spreadsheets or case-insensitive evaluation. http://mail.python.org/pipermail/python-dev/2002-October/029770.html [and the rest of that thread] A 3% slowdown in the common case was considered too big to allow a feature like this to be included. The slowdown might be smaller if the builtin module wasn't allowed to be anything but a dict instance, or might disappear if there were two versions of ceval.c were included, and one used PyObject_ APIs and the other used PyDict_ APIs (as long as only one was fighting for space in the cpu cache, main memory, etc) Jeff From tdelaney at avaya.com Sun Dec 14 22:30:38 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Sun Dec 14 22:30:46 2003 Subject: [Python-Dev] Pie-thon benchmarks Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE07A2@au3010avexu1.global.avaya.com> > From: Guido van Rossum > > > > And AFAIK, Timsort isn't patented. :-) > > > > True. But do you really expect that anyone except Tim could get it > > *right*??? > > Yes. As a matter of fact, it was ported to Jython in record time. Bah. I'm giving up on the non-witty comments for today :( Tim Delaney From guido at python.org Mon Dec 15 00:59:57 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 01:00:05 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: Your message of "Mon, 15 Dec 2003 15:51:33 +1300." <200312150251.hBF2pX529463@oma.cosc.canterbury.ac.nz> References: <200312150251.hBF2pX529463@oma.cosc.canterbury.ac.nz> Message-ID: <200312150559.hBF5xw616334@c-24-5-183-134.client.comcast.net> > The same effect could be achieved by defining a class... > > class my_favourite_types(float, complex): > pass > > if isinstance(x, (int, long, my_favourite_types)): > ... I hope you forgot the smiley, as this would otherwise have to be explained as a typical "before the first coffee" lapse. :) First of all, you can't multiply inherit from built-in classes. But even if you could, it would be wrong: while an instance of my_favourite_types would indeed be an instance of float, 3.14 is not an instance of my_favourite_types! You'd need my_favourite_types to be a common base class of float and complex for the test to work the way you want it. --Guido van Rossum (home page: http://www.python.org/~guido/) From lists at webcrunchers.com Mon Dec 15 02:52:50 2003 From: lists at webcrunchers.com (JD) Date: Mon Dec 15 02:52:59 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: References: Message-ID: I can't get my Python 2.3.3c1 program to work. It keeps core dumping (leaving python.core). I'm not doing anything strange, just documented stuff from the manuals. I tried 'gdb -c python.core' and then using 'back', but there are no labels compiled in (I hate that...); how can I get the labels to be included? "./configure --debug" doesn't work. I assume without the labels the numbers are meaningless. Running this under OpenBSD 3.2. Then, where should I submit my core dump (if anywhere?)? I used to have the URL bookmarked, but just upgraded and lost most of my old bookmarks.... John From mwh at python.net Mon Dec 15 05:11:05 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 15 05:11:08 2003 Subject: [Python-Dev] urllib2 patch reviewer (852995)? In-Reply-To: (John J. Lee's message of "Sun, 14 Dec 2003 16:15:28 +0000 (GMT)") References: Message-ID: <2mekv6jsqu.fsf@starship.python.net> John J Lee writes: > Jeremy Hylton just checked in a relatively major patch of mine to urllib2. > He's obviously very busy, though, so it would be nice if somebody else who > knows urllib2 could look at it, too. I don't have the know-how to review the patch, but I can notice that test_urllib2 now seems to be creating uncollectable garbage... Cheers, mwh -- I hate leaving Windows95 boxes publically accessible, so shifting even to NT is a blessing in some ways. At least I can reboot them remotely in a sane manner, rather than having to send them malformed packets. -- http://bofhcam.org/journal/journal.html, 20/06/2000 From mwh at python.net Mon Dec 15 05:14:17 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 15 05:14:19 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312142230.hBEMU4H15844@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Sun, 14 Dec 2003 14:30:04 -0800") References: <200312142230.hBEMU4H15844@c-24-5-183-134.client.comcast.net> Message-ID: <2mad5ujsli.fsf@starship.python.net> Guido van Rossum writes: >> If you want a benchmark to make CPython look good, do a few hundred >> thousand very-long int multiplications, stick 'em in a list, and >> sort it . > > Ah, but then Dan will just add Karatsuba multiplication to Parrot, > too. And AFAIK, Timsort isn't patented. :-) If something important -- like a pieing -- depends on it, and the ints are long enough, it's not that hard to do better than Karatsuba multiplication... Cheers, mwh -- A.D. 1517: Martin Luther nails his 95 Theses to the church door and is promptly moderated down to (-1, Flamebait). -- http://slashdot.org/comments.pl?sid=01/02/09/1815221&cid=52 (although I've seen it before) From mwh at python.net Mon Dec 15 07:44:53 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 15 07:43:37 2003 Subject: [Python-Dev] CVS bustage? In-Reply-To: <2m8ylwjov0.fsf@starship.python.net> Message-ID: <7B1EAA98-2EFC-11D8-AD8E-0003931DF95C@python.net> On Monday, Dec 1, 2003, at 20:54 Europe/Stockholm, Michael Hudson wrote: > Guido van Rossum writes: > >>> I'm seeing test_trace dump core from current CVS. Anyone else? >> >> Not here (Red Hat 9, production build). > > Nor anyone on #python, either. If anyone cares, I worked this out in the end: I had some dubious compiler changes and backed them out -- but forgot to wipe away the .pyc files. Argh! Cheers, mwh From barry at python.org Mon Dec 15 09:54:17 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 09:54:22 2003 Subject: [Python-Dev] Python-checkins list In-Reply-To: <002901c3c272$a35dce40$e841fea9@oemcomputer> References: <002901c3c272$a35dce40$e841fea9@oemcomputer> Message-ID: <1071500056.970.73.camel@anthem> On Sun, 2003-12-14 at 13:46, Raymond Hettinger wrote: > Using reply-all for a checkin message used to notify the committer and > cc the checkins list. Now, it also cc?s the python-dev list but not > the committer. Discussions need to be redirected from the checkins list to python-dev. However there should be a Mailman option to copy the From header to the Reply-To header when doing Reply-To munging so the original poster is included in the response. I'll see if I can add, er, fix that for Mailman 2.1.4. -Barry From skip at pobox.com Mon Dec 15 09:55:18 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 09:55:22 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: References: Message-ID: <16349.52054.372100.462135@montanaro.dyndns.org> John> Then, where should I submit my core dump (if anywhere?)? Submit bug reports here: http://sourceforge.net/tracker/?group_id=5470&atid=105470 but please don't attach core files (they are too big and nobody will be able to make use of them). Instead, attach a small Python script which demonstrates how to generate a core dump. Skip From barry at python.org Mon Dec 15 09:56:49 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 09:56:53 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: References: <000201c3c289$976fcae0$e841fea9@oemcomputer> Message-ID: <1071500208.970.76.camel@anthem> On Sun, 2003-12-14 at 17:05, Fredrik Lundh wrote: > V. a join() builtin (join(x,s) becomes s.join(x)). All of the common sep.join() > use cases read better without the string literal prefix. Really? I predict everlasting confusion on argument order. It's not obvious from the method name whether the joining separator comes first or second. My wish list: V . Explicit global imports. I want a way to spell "import this from the global standard library even if there's a this.py in the local directory". VI. PEP 318 or something like it. -Barry From martin at v.loewis.de Mon Dec 15 09:59:42 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Mon Dec 15 10:00:11 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: References: Message-ID: JD writes: > Running this under OpenBSD 3.2. Then, where should I submit my core > dump (if anywhere?)? You should submit bug reports to sf.net/projects/python, and provide a link to the core dump on a server of your control (as SF has a limit on file upload size). Please understand that several months, or even years, may pass until such a bug report is looked at, so if you actually need this to work, you will have to investigate it in more detail. On Unix, Python is, by default, compiled with symbols if the compiler is gcc. To find out why gdb does not see symbols, you should post a few lines of make output, showing typical compiler and linker invocations. Regards, Martin From fdrake at acm.org Mon Dec 15 10:14:24 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Dec 15 10:14:30 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071500208.970.76.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> Message-ID: <16349.53200.37613.119494@sftp.fdrake.net> Barry Warsaw writes: > V . Explicit global imports. I want a way to spell "import this > from the global standard library even if there's a this.py in the local > directory". I like the syntax: import global and from global import Look Guido, no new keywords! ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From jeremy at zope.com Mon Dec 15 10:50:47 2003 From: jeremy at zope.com (Jeremy Hylton) Date: Mon Dec 15 10:54:27 2003 Subject: [Python-Dev] urllib2 patch reviewer (852995)? In-Reply-To: <2mekv6jsqu.fsf@starship.python.net> References: <2mekv6jsqu.fsf@starship.python.net> Message-ID: <1071503446.15985.651.camel@localhost.localdomain> On Mon, 2003-12-15 at 05:11, Michael Hudson wrote: > John J Lee writes: > > > Jeremy Hylton just checked in a relatively major patch of mine to urllib2. > > He's obviously very busy, though, so it would be nice if somebody else who > > knows urllib2 could look at it, too. > > I don't have the know-how to review the patch, but I can notice that > test_urllib2 now seems to be creating uncollectable garbage... Yes. Didn't get a chance to fix that the other night. One of the classes has a pretty useless __del__ that might have helped before we had gc for cycles. The original urllib2 code is pretty old. Jeremy From guido at python.org Mon Dec 15 11:03:31 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 11:03:37 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 10:14:24 EST." <16349.53200.37613.119494@sftp.fdrake.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> Message-ID: <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> > Barry Warsaw writes: > > V . Explicit global imports. I want a way to spell "import this > > from the global standard library even if there's a this.py in the local > > directory". > > I like the syntax: > > import global > > and > > from global import > > Look Guido, no new keywords! ;-) I'd much rather invent new syntax to spell *local* imports. I like from .module import something You can even generalize and write from ..module import something to reference your package's parent package. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Mon Dec 15 11:05:42 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 15 11:05:47 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <2mad5ujsli.fsf@starship.python.net> Message-ID: [Michael Hudson] > If something important -- like a pieing -- depends on it, and > the ints are long enough, it's not that hard to do better than > Karatsuba multiplication... "Not that hard" depends on your background. The idea is to bury Dan under esoteric algorithms from a huge number of obscure specialties . From mwh at python.net Mon Dec 15 11:17:02 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 15 11:17:07 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: (Tim Peters's message of "Mon, 15 Dec 2003 11:05:42 -0500") References: Message-ID: <2mk74yhx8h.fsf@starship.python.net> "Tim Peters" writes: > [Michael Hudson] >> If something important -- like a pieing -- depends on it, and >> the ints are long enough, it's not that hard to do better than >> Karatsuba multiplication... > > "Not that hard" depends on your background. Granted, but while my background is maths it's not numerics and I know where to find descriptions and implementations of complexity-wise better algorithms. Using them is just a matter of engineering, right? > The idea is to bury Dan under esoteric algorithms from a huge number > of obscure specialties . Now there's a plan! Let's see, how do we do that? Excessive use of dicts, unicode esoterica and multiple inheritance seems like a good start. Cheers, mwh -- Its unmanageable complexity has spawned more fear-preventing tools than any other language, but the solution _should_ have been to create and use a language that does not overload the whole goddamn human brain with irrelevant details. -- Erik Naggum, comp.lang.lisp From barry at python.org Mon Dec 15 11:40:14 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 11:40:22 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> Message-ID: <1071506413.970.102.camel@anthem> On Mon, 2003-12-15 at 11:03, Guido van Rossum wrote: > I'd much rather invent new syntax to spell *local* imports. I like > > from .module import something > > You can even generalize and write > > from ..module import something > > to reference your package's parent package. :-) Actually, if truth be told, I'd love to just ban local imports. I understand that might be controversial though . The next best thing would be to force local imports to use special syntax (I was going to say "ugly" :) like what you suggest above, but I'm afraid that that might be more disruptive, requiring future imports and such. I think having a syntax for spelling global imports can be done in a backwards compatible way for 2.4. -Barry From tim.one at comcast.net Mon Dec 15 11:46:24 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 15 11:46:28 2003 Subject: [Python-Dev] Berkeley support in release23-maint Message-ID: Looking at http://cvs.sf.net/viewcvs.py/python/python/dist/src/Lib/bsddb/__init__.py it appear that 2+ months of Berkeley bugfixes (revs 1.10 through 1.12 of __init__.py, at least) didn't get backported to the 2.3 maint branch, and so are absent in 2.3.3c1. Is that right? From skip at pobox.com Mon Dec 15 11:52:15 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 11:52:28 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071506413.970.102.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> Message-ID: <16349.59071.26004.502201@montanaro.dyndns.org> Guido> I'd much rather invent new syntax to spell *local* imports. Barry> Actually, if truth be told, I'd love to just ban local imports. Barry> I understand that might be controversial though . Okay, so what are global and local imports? I thought I understood after Barry's first post, but not now. I see the , but figured there's still some germ of desire in Barry's "ban local imports" comment. What kind of imports do we have today? Do they relate to the position of the import statement in the code or to the location of the module being imported along sys.path? Skip From guido at python.org Mon Dec 15 12:07:13 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 12:07:19 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 11:40:14 EST." <1071506413.970.102.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> Message-ID: <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> > On Mon, 2003-12-15 at 11:03, Guido van Rossum wrote: > > > I'd much rather invent new syntax to spell *local* imports. I like > > > > from .module import something > > > > You can even generalize and write > > > > from ..module import something > > > > to reference your package's parent package. :-) > > Actually, if truth be told, I'd love to just ban local imports. I > understand that might be controversial though . Works for me. :-) > The next best thing would be to force local imports to use special > syntax (I was going to say "ugly" :) like what you suggest above, but > I'm afraid that that might be more disruptive, requiring future imports > and such. I think having a syntax for spelling global imports can be > done in a backwards compatible way for 2.4. Well, but since you want all imports to be global, it'd be insane to introduce *new* syntax for global imports, wouldn't it? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 12:10:24 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 12:10:30 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 10:52:15 CST." <16349.59071.26004.502201@montanaro.dyndns.org> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> Message-ID: <200312151710.hBFHAO817542@c-24-5-183-134.client.comcast.net> > Okay, so what are global and local imports? I thought I understood > after Barry's first post, but not now. I see the , but > figured there's still some germ of desire in Barry's "ban local > imports" comment. What kind of imports do we have today? Do they > relate to the position of the import statement in the code or to the > location of the module being imported along sys.path? Traditionally, we've called these absolute (== global) and relative (== local) imports. The terms relate to the module's full package name, usually equivalent to its location on the filesystem. sys.path and the placement of the import statement do not enter into it, except that sys.path is the very definition of absolute import. --Guido van Rossum (home page: http://www.python.org/~guido/) From lkcl at lkcl.net Mon Dec 15 12:31:56 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Mon Dec 15 12:31:32 2003 Subject: [Python-Dev] rexec.py unuseable Message-ID: <20031215173156.GC25203@lkcl.net> i tried to reuse some code that i wrote for a project back when python 2.0 and 2.1 were in use. i must be about the only person in existence who wrote some code that uses - and totally relies on - the rexec.py module. what that code does (did) was to provide a database-based method of running restricted programs for use to monitor and manage remote servers. i.e. it became in effect a distributed python operating system. what happened was that the name of a python script, its arguments, a relevant ip address and the server on which the script was to be run were all entered into a database table, and the server picked that up, loaded the script and its arguments from the database and then ran the script, which usually involved contacting the remote ip address. so, not only was rhooks used to overload "import" such that libraries used by scripts in the database could _also_ be loaded from the database, but rexec.py was also used to restrict what functions could be accessed. i was quite happy with rexec.py as it stood. it was the c++ functional equivalent of the "protected" convention for c++ classes. at the time that i tried to reuse the code, i discovered the lovely "this module is broken in python 2.2 and 2.3" message. well, uhm, if it's broken, why hasn't it been reverted to the 2.0 code? i noted, some time about a year ago, that someone attempted to drill down the rexec security into sub-modules. i.e. they attempted to add the c++ functional equivalence of the "private" convention for c++ classes. consequently, a number of routines that wrote to files that i had DELIBERATELY wrapped and made available to first-level rexec-restricted library routines, e.g. the logging functions which output debug information to log files, suddenly stopped working. why? whereas before, the debug routine could be called directly but the write operation could not, because of the drill-down effect, all of a sudden the write operation, as called _by_ the debug routine, is now restricted in the 2.1 (or early 2.2) code, and my debug library routine now doesn't work, nor indeed does anything else. the thing is that the functionality of the python 2.0 rexec.py was _exactly_ as i would have expected it to be, and it was working _perfectly_ according to my expectations of that code. i therefore believe that the attempts to take rexec one step further, aka a "drill-down" effect, by making all functions that rexec functions import _also_ restricted, to have been a mistake. or, at least, not carefully thought out. if such additional restrictions are to be added, i believe that they should be added as a user-configureable parameter (e.g. named recursive-restrict) _please_ could the functionality of rexec.py be restored to python 2.0 functionality, with the "rexec.py is broken" message only coming up when the suggested-named parameter "recursive-restrict" is set. maybe then i can consider reviving that code and re-releasing it, because at the moment it's totally going to waste. many many thanks, l. -- -- expecting email to be received and understood is a bit like picking up the telephone and immediately dialing without checking for a dial-tone; speaking immediately without listening for either an answer or ring-tone; hanging up immediately and then expecting someone to call you (and to be able to call you). -- every day, people send out email expecting it to be received without being tampered with, read by other people, delayed or simply - without prejudice but lots of incompetence - destroyed. -- please therefore treat email more like you would a CB radio to communicate across the world (via relaying stations): ask and expect people to confirm receipt; send nothing that you don't mind everyone in the world knowing about... From barry at python.org Mon Dec 15 12:37:27 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 12:37:33 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <16349.59071.26004.502201@montanaro.dyndns.org> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> Message-ID: <1071509846.970.105.camel@anthem> On Mon, 2003-12-15 at 11:52, Skip Montanaro wrote: > Guido> I'd much rather invent new syntax to spell *local* imports. > > Barry> Actually, if truth be told, I'd love to just ban local imports. > Barry> I understand that might be controversial though . > > Okay, so what are global and local imports? Sorry, let me rephrase. I'd love to ban relative imports. Personally, I think all imports should be specified using fully-qualified absolute package paths. -Barry From barry at python.org Mon Dec 15 12:39:47 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 12:39:54 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> Message-ID: <1071509987.970.108.camel@anthem> On Mon, 2003-12-15 at 12:07, Guido van Rossum wrote: > > Actually, if truth be told, I'd love to just ban local imports. I > > understand that might be controversial though . > > Works for me. :-) Yay! :) > > The next best thing would be to force local imports to use special > > syntax (I was going to say "ugly" :) like what you suggest above, but > > I'm afraid that that might be more disruptive, requiring future imports > > and such. I think having a syntax for spelling global imports can be > > done in a backwards compatible way for 2.4. > > Well, but since you want all imports to be global, it'd be insane to > introduce *new* syntax for global imports, wouldn't it? If we banned relative (a.k.a. local) imports, yes definitely. from __future__ global_imports ? pep-time-ly y'rs, -Barry From edloper at gradient.cis.upenn.edu Mon Dec 15 13:44:56 2003 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Mon Dec 15 12:43:15 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: References: Message-ID: <3FDE0128.1040600@gradient.cis.upenn.edu> Barry Warsaw wrote: > Actually, if truth be told, I'd love to just ban local imports. I > understand that might be controversial though . +1 (asuming that you're talking about package-local imports). I've seen a couple people get bitten by the fact that a module gets loaded twice if there are separate local & global imports for it: % mkdir pkg % touch pkg/__init__.py % cat >pkg/a.py print 'A is being imported (not re-used)' class A: pass % cat >pkg/b.py from a import A as LocalA from pkg.a import A as GlobalA print isinstance(LocalA(), GlobalA) print isinstance(GlobalA(), LocalA) % PYTHONPATH=.; python pkg/b.py A is being imported (not re-used) A is being imported (not re-used) 0 0 Since pkg/a gets loaded twice, we end up with two versions of class A, which are not compatible. In practice this usually comes up if a package uses local imports between submodules, and then an outside user uses a global import to get a submodule. -Edward From pycon-organizers at python.org Mon Dec 15 12:54:27 2003 From: pycon-organizers at python.org (pycon-organizers@python.org) Date: Mon Dec 15 12:54:29 2003 Subject: [Python-Dev] PyCon DC 2004 - Submissions Now Open Message-ID: Dear Python User: I am pleased to be able to announce that the submission process for PyCon DC 2004 is now open. Please direct your browser to http://submit.pycon.org/ and create your account by entering your email address and leaving the password field blank - a password will be sent to you by email, and you can submit your paper! Please remember that your submissions should ideally be in a single file, containing HTML or the reStructured Text format. Ultimately we would like to publish all accepted papers on the web, and these rules should make it easier to do so. Send enquiries about other formats to pycon-organizers@python.org If your paper is accepted and you prepare an electronic presentation (in PDF, PythonPoint or PowerPoint) we will also happily publish that on the web site when PyCon is over. We are interested in any and all submissions about uses of Python and the development of the language. Since there is expected to be a strong educational community presence for the next PyCon, teaching materials of various kinds are also encouraged. Please help us to make the next PyCon even better than the first one. See you in March! Sincerely Steve Holden Chairman PyCon DC 2004 From klm at zope.com Mon Dec 15 12:48:32 2003 From: klm at zope.com (Ken Manheimer) Date: Mon Dec 15 12:54:58 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> Message-ID: On Mon, 15 Dec 2003, Guido van Rossum wrote: > > Barry Warsaw writes: > > > V . Explicit global imports. I want a way to spell "import this > > > from the global standard library even if there's a this.py in the local > > > directory". > > > > I like the syntax: > > > > import global > > > > and > > > > from global import > > > > Look Guido, no new keywords! ;-) > > I'd much rather invent new syntax to spell *local* imports. I like > > from .module import something > > You can even generalize and write > > from ..module import something > > to reference your package's parent package. :-) As i've said before (though it's been a while) i think people want this, it would be useful, and i think it would work well. Barry wrote, in response to guido's message: < Actually, if truth be told, I'd love to just ban local imports. I < understand that might be controversial though . I don't understand the desire to totally prevent relative imports. It would be quite useful in avoiding growing module-naming problems, where you have to be careful about shadowing a global module with one in your package. I *can* understand objecting to the ambiguity in the current situation, where you don't know whether a module import will resolve to a sibling module or one in higher up in the hierarchy. That is why i strongly prefer having a leading dot to explicitly signify a relative import, and originally proposed it that way. I expect that the elimination of the ambiguity would mitigate the objection to relative imports - and we would do away with the shadowing problems, and even have easy relocation of modules, if there are cases where software does need to move around. Ken klm@zope.com From skip at pobox.com Mon Dec 15 13:01:07 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 13:01:29 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031215173156.GC25203@lkcl.net> References: <20031215173156.GC25203@lkcl.net> Message-ID: <16349.63203.797449.677617@montanaro.dyndns.org> Luke> well, uhm, if [rexec]'s broken, why hasn't it been reverted to the Luke> 2.0 code? My understanding was that in 2.0 it was also broken, but that (relatively) recent changes to Python made the breakage easier to exploit. As for alternatives, I suggest you post a query on comp.lang.python (aka python-list@python.org). Skip From skip at pobox.com Mon Dec 15 13:04:32 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 13:04:33 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071509846.970.105.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> <1071509846.970.105.camel@anthem> Message-ID: <16349.63408.143245.274390@montanaro.dyndns.org> >> Okay, so what are global and local imports? Barry> Sorry, let me rephrase. I'd love to ban relative imports. Barry> Personally, I think all imports should be specified using Barry> fully-qualified absolute package paths. That's fine with me. Maybe continue to use "absolute" and "relative" (and throw in "package" somewhere) instead of "global" and "local" when referring to package import semantics? "global" and "local" have a well-established meaning in day-to-day Python programming which doesn't overlap much with the import mechanism. Skip From guido at python.org Mon Dec 15 13:36:44 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 13:36:50 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 12:39:47 EST." <1071509987.970.108.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> <1071509987.970.108.camel@anthem> Message-ID: <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> > > Well, but since you want all imports to be global, it'd be insane to > > introduce *new* syntax for global imports, wouldn't it? > > If we banned relative (a.k.a. local) imports, yes definitely. > > from __future__ global_imports I think this ought to be a *global* flag rather than a per-module flag. E.g. after setting sys.allow_relative_import = False all imports anywhere would be interpreted as absolute imports. This would mean you couldn't have some code that still uses relative imports, but the problem with the __future__ statement is that it seems so pointless: packages that only use absolute imports don't need it, and packages that use relative imports break if it is used. About the only time where the __future__ statement would make a difference is when a package defines a local module whose name is the same as that of a global module, *and* the package also wants to import the global module. I would personally solve that by renaming the local module though... --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 13:38:01 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 13:38:08 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 12:44:56 CST." <3FDE0128.1040600@gradient.cis.upenn.edu> References: <3FDE0128.1040600@gradient.cis.upenn.edu> Message-ID: <200312151838.hBFIc1q17738@c-24-5-183-134.client.comcast.net> > I've seen a couple people get bitten by the fact that a module gets > loaded twice if there are separate local & global imports for it: > > % mkdir pkg > > % touch pkg/__init__.py > > % cat >pkg/a.py > print 'A is being imported (not re-used)' > class A: pass > > % cat >pkg/b.py > from a import A as LocalA > from pkg.a import A as GlobalA > print isinstance(LocalA(), GlobalA) > print isinstance(GlobalA(), LocalA) > > % PYTHONPATH=.; python pkg/b.py > A is being imported (not re-used) > A is being imported (not re-used) > 0 > 0 > > Since pkg/a gets loaded twice, we end up with two versions of class A, > which are not compatible. In practice this usually comes up if a > package uses local imports between submodules, and then an outside user > uses a global import to get a submodule. That's actually a slightly different problem; sys.path should not include any directories that are also packages on other directories in sys.path. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 13:42:00 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 13:42:06 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 12:48:32 EST." References: Message-ID: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> > I don't understand the desire to totally prevent relative imports. It's because of TOOWTDI. > It would be quite useful in avoiding growing module-naming problems, > where you have to be careful about shadowing a global module with > one in your package. I don't see the problem, or I misuderstand what "it" refers to; it seems you have this backwards if it refers to relative imports. Say my package P defines a submodule sys. If we require absolute imports, there is no ambiguity: the submodule sys must be imported as P.sys while the standard sys module can be imported as simply sys. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Mon Dec 15 13:49:36 2003 From: dan at sidhe.org (Dan Sugalski) Date: Mon Dec 15 13:50:02 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: References: Message-ID: At 11:05 AM -0500 12/15/03, Tim Peters wrote: >[Michael Hudson] >> If something important -- like a pieing -- depends on it, and >> the ints are long enough, it's not that hard to do better than >> Karatsuba multiplication... > >"Not that hard" depends on your background. The idea is to bury Dan under >esoteric algorithms from a huge number of obscure specialties . Heh. You're expecting me to do far too much work. I plan on tackling bytecode optimization only if it turns out that a straightforward transform from python bytecode to parrot bytecode is isn't sufficiently faster than CPython. Funky algorithms expressed entirely in python bytecode won't make much of a difference... -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From barry at python.org Mon Dec 15 13:51:07 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 13:51:10 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> <1071509987.970.108.camel@anthem> <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> Message-ID: <1071514266.28081.2.camel@geddy> On Mon, 2003-12-15 at 13:36, Guido van Rossum wrote: > I think this ought to be a *global* flag rather than a per-module > flag. E.g. after setting > > sys.allow_relative_import = False > > all imports anywhere would be interpreted as absolute imports. > > This would mean you couldn't have some code that still uses relative > imports, but the problem with the __future__ statement is that it > seems so pointless: packages that only use absolute imports don't need > it, and packages that use relative imports break if it is used. You probably want something at the package level. This may not be feasible, but it seems like you want to be able to say, "this package needs to allow relative imports for backwards compatibility". -Barry From lkcl at lkcl.net Mon Dec 15 14:10:23 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Mon Dec 15 14:10:14 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <16349.63203.797449.677617@montanaro.dyndns.org> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> Message-ID: <20031215191023.GB26055@lkcl.net> On Mon, Dec 15, 2003 at 12:01:07PM -0600, Skip Montanaro wrote: > > Luke> well, uhm, if [rexec]'s broken, why hasn't it been reverted to the > Luke> 2.0 code? > > My understanding was that in 2.0 it was also broken, but that (relatively) > recent changes to Python made the breakage easier to exploit. the issue is [back-tracking to the 2.0 code], were those "breakages" actually features, or were they definitely "breakages"? i.e. was it 1) was it just unexpected behaviour in that a function _called_ by a rexec'd restricted routine could do, for example writes; was it that an unrestricted function, module or variable imported by a rexec'd restricted routine or module could be accessed. e.g. like this: contents of bypassmodule.py: import open as unrestrictedopen then code run under rexec: from bypassmodule import unrestrictedopen both of these things i would consider to be totally acceptable behaviour of the [2.] rexec.py module!!! i would NOT consider them to be a bug, and in fact are desirable behaviour in order to get my code working. i believe that there exists in [2.0] rexec a mechanism to specify what functions are allowed in a particular module, so i could restrict access to the bypassmodule.unrestrictedopen function, if necessary. OR was it 2) a genuine bug. > As for alternatives, I suggest you post a query on comp.lang.python (aka > python-list@python.org). ta! > Skip -- -- expecting email to be received and understood is a bit like picking up the telephone and immediately dialing without checking for a dial-tone; speaking immediately without listening for either an answer or ring-tone; hanging up immediately and then expecting someone to call you (and to be able to call you). -- every day, people send out email expecting it to be received without being tampered with, read by other people, delayed or simply - without prejudice but lots of incompetence - destroyed. -- please therefore treat email more like you would a CB radio to communicate across the world (via relaying stations): ask and expect people to confirm receipt; send nothing that you don't mind everyone in the world knowing about... From python at rcn.com Mon Dec 15 14:26:03 2003 From: python at rcn.com (Raymond Hettinger) Date: Mon Dec 15 14:26:30 2003 Subject: [Python-Dev] Christmas Wishlist In-Reply-To: <20031215030347.GC27067@unpythonic.net> Message-ID: <004301c3c341$48053fe0$e841fea9@oemcomputer> > On Sun, Dec 14, 2003 at 04:31:09PM -0500, Raymond Hettinger wrote: > > I. eval() to accept custom mapping arguments for globals and locals. > > This makes it possible to write a smart __getitem__ method for > > applications like spreadsheets or case-insensitive evaluation. > > http://mail.python.org/pipermail/python-dev/2002-October/029770.html > [and the rest of that thread] [Jeff Epler] > A 3% slowdown in the common case was considered too big to allow a > feature like this to be included. The slowdown might be smaller if > the builtin module wasn't allowed to be anything but a dict instance, or > might disappear if there were two versions of ceval.c were included, and > one used PyObject_ APIs and the other used PyDict_ APIs (as long as only > one was fighting for space in the cpu cache, main memory, etc) Ordinarily, I would say that 3% is a reasonable price to pay for this functionality but there is that pesky pie throwing event coming up. Raymond Hettinger From pje at telecommunity.com Mon Dec 15 14:32:27 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 15 14:32:37 2003 Subject: [Python-Dev] Christmas Wishlist In-Reply-To: <004301c3c341$48053fe0$e841fea9@oemcomputer> References: <20031215030347.GC27067@unpythonic.net> Message-ID: <5.1.1.6.0.20031215142906.01e00ac0@telecommunity.com> At 02:26 PM 12/15/03 -0500, Raymond Hettinger wrote: > > On Sun, Dec 14, 2003 at 04:31:09PM -0500, Raymond Hettinger wrote: > > > I. eval() to accept custom mapping arguments for globals and locals. > > > This makes it possible to write a smart __getitem__ method for > > > applications like spreadsheets or case-insensitive evaluation. > > > > http://mail.python.org/pipermail/python-dev/2002-October/029770.html > > [and the rest of that thread] > >[Jeff Epler] > > A 3% slowdown in the common case was considered too big to allow a > > feature like this to be included. The slowdown might be smaller if > > the builtin module wasn't allowed to be anything but a dict instance, >or > > might disappear if there were two versions of ceval.c were included, >and > > one used PyObject_ APIs and the other used PyDict_ APIs (as long as >only > > one was fighting for space in the cpu cache, main memory, etc) > >Ordinarily, I would say that 3% is a reasonable price to pay for this >functionality but there is that pesky pie throwing event coming up. There is a workaround for this in CPython, which used to be used by Zope, and which I have used on occasion: scan the code object for LOAD_NAME opcodes, and note what names are used by the code block. Then, look up those names in your non-dictionary dictionary, and copy them into a real dictionary. So, it's got to be something pretty specialized to really need PyObject_* APIs instead of PyDict_* APIs in CPython at present. Why penalize *everything* for such an specialized need? From guido at python.org Mon Dec 15 14:34:10 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 14:34:15 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 13:51:07 EST." <1071514266.28081.2.camel@geddy> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <200312151707.hBFH7DI17519@c-24-5-183-134.client.comcast.net> <1071509987.970.108.camel@anthem> <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> <1071514266.28081.2.camel@geddy> Message-ID: <200312151934.hBFJYAB17994@c-24-5-183-134.client.comcast.net> > > This would mean you couldn't have some code that still uses relative > > imports, but the problem with the __future__ statement is that it > > seems so pointless: packages that only use absolute imports don't need > > it, and packages that use relative imports break if it is used. > > You probably want something at the package level. This may not be > feasible, but it seems like you want to be able to say, "this package > needs to allow relative imports for backwards compatibility". But there is no backwards compatibility issue. Absolute imports always work, so instead of adding a "relative imports allowed" to your package, you can just change the package to use only absolute imports. Using only absolute imports is totally backwards compatible (unless your toplevel package name is the same as the name of a submodule of the package, which seems pretty sick). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 14:37:34 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 14:37:40 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Mon, 15 Dec 2003 13:49:36 EST." References: Message-ID: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> > At 11:05 AM -0500 12/15/03, Tim Peters wrote: > >[Michael Hudson] > >> If something important -- like a pieing -- depends on it, and > >> the ints are long enough, it's not that hard to do better than > >> Karatsuba multiplication... > > > >"Not that hard" depends on your background. The idea is to bury Dan under > >esoteric algorithms from a huge number of obscure specialties . [Dan Sugalski] > Heh. You're expecting me to do far too much work. I plan on tackling > bytecode optimization only if it turns out that a straightforward > transform from python bytecode to parrot bytecode is isn't > sufficiently faster than CPython. Funky algorithms expressed entirely > in python bytecode won't make much of a difference... But Karatsuba multiplication and Timsort are implemented highly efficiently in C, and are part of the "full set of standard builtins" to which we agreed. BTW, eval() is also part of that full set, and exec is part of the language. These invoke the Python bytecode compiler. (And Jim Hugunin's IronPython is really slow on this, like 60x slower than CPython; I think Jim reported that Jython was 20x slower on exec().) How do you plan to handle these? What if the entire benchmark was given as a triple quoted string that was passed to the exec statement? --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Mon Dec 15 14:40:30 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 14:40:55 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031215191023.GB26055@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> Message-ID: <16350.3630.307765.836000@montanaro.dyndns.org> >> My understanding was that in 2.0 it was also broken, but that >> (relatively) recent changes to Python made the breakage easier to >> exploit. Luke> the issue is [back-tracking to the 2.0 code], Luke> were those "breakages" actually features, or were they definitely Luke> "breakages"? Dunno. Take a look at the 2.3 Whats New: http://www.python.org/doc/current/whatsnew/node18.html Search for "rexec". I don't know more than that. It only mentions back as far as 2.1, so I don't know if rexec could be made to work based upon the code in 2.0. Also, check this thread from python-dev: http://mail.python.org/pipermail/python-dev/2003-January/thread.html#31842 That thread includes some indication of what the problem is and how it got worse in 2.2 and 2.3. Skip From klm at zope.com Mon Dec 15 14:34:33 2003 From: klm at zope.com (Ken Manheimer) Date: Mon Dec 15 14:40:59 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151836.hBFIaie17722@c-24-5-183-134.client.comcast.net> Message-ID: On Mon, 15 Dec 2003, Guido van Rossum wrote: > > It would be quite useful in avoiding growing module-naming problems, > > where you have to be careful about shadowing a global module with > > one in your package. > > I don't see the problem, or I misuderstand what "it" refers to; it > seems you have this backwards if it refers to relative imports. Say > my package P defines a submodule sys. If we require absolute imports, > there is no ambiguity: the submodule sys must be imported as P.sys > while the standard sys module can be imported as simply sys. I guess i think there is a case to be made for relative imports, and it becomes apparent as an enterprise grows. Increasingly at Zope corp we are mixing and matching packages to deliver applications to a customer. This is very encouraging, because it means we are actually getting reuse out of the packages. Currently, we can just plop the things in place, and use them. Without relative imports, we would have to be editing the imports in the packages in each place we use it. This would increase the burden of using the packages and even more of feeding back actual fixes - which we then have to distinguish from what i would see as gratuitous changes to edit import names. If you would grant there's use to avoiding those edits, and so there's use to having relative imports, then you might see a reason to solve the problems where the names in a package conflict with those along the load path. Otherwise, if there's only absolute imports, there's no ambiguity to resolve. I'd say there's a legitimate need for relative imports, and we need some explicit gesture to disambiguate between a relative and absolute import, one way or the other. Ken From dan at sidhe.org Mon Dec 15 14:51:39 2003 From: dan at sidhe.org (Dan Sugalski) Date: Mon Dec 15 14:51:56 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> Message-ID: At 11:37 AM -0800 12/15/03, Guido van Rossum wrote: > > At 11:05 AM -0500 12/15/03, Tim Peters wrote: >> >[Michael Hudson] >> >> If something important -- like a pieing -- depends on it, and >> >> the ints are long enough, it's not that hard to do better than >> >> Karatsuba multiplication... >> > >> >"Not that hard" depends on your background. The idea is to bury Dan under >> >esoteric algorithms from a huge number of obscure specialties . > >[Dan Sugalski] >> Heh. You're expecting me to do far too much work. I plan on tackling >> bytecode optimization only if it turns out that a straightforward >> transform from python bytecode to parrot bytecode is isn't >> sufficiently faster than CPython. Funky algorithms expressed entirely >> in python bytecode won't make much of a difference... > >But Karatsuba multiplication and Timsort are implemented highly >efficiently in C, and are part of the "full set of standard builtins" >to which we agreed. In which case, well... time for me to write some C code. >BTW, eval() is also part of that full set, and exec is part of the >language. These invoke the Python bytecode compiler. (And Jim >Hugunin's IronPython is really slow on this, like 60x slower than >CPython; I think Jim reported that Jython was 20x slower on exec().) > >How do you plan to handle these? What if the entire benchmark was >given as a triple quoted string that was passed to the exec statement? We agreed no eval as part of this challenge, since the point was the speed of the interpreter engine not of the compiler. I don't see that on the list of challenge points you posted earlier, though, so it was either omitted inadvertently or I got snookered. The former, I expect. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Mon Dec 15 14:54:46 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 14:54:54 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 14:34:33 EST." References: Message-ID: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> > I guess i think there is a case to be made for relative imports, and > it becomes apparent as an enterprise grows. > > Increasingly at Zope corp we are mixing and matching packages to > deliver applications to a customer. This is very encouraging, because > it means we are actually getting reuse out of the packages. > > Currently, we can just plop the things in place, and use them. > Without relative imports, we would have to be editing the imports in > the packages in each place we use it. This would increase the burden > of using the packages and even more of feeding back actual fixes - > which we then have to distinguish from what i would see as gratuitous > changes to edit import names. I know this line of reasoning fairly well. You are taking 3rd party packages (or perhaps internally developed packages) and copy them to a *different place in the package namespace tree*. Right? But why do you have to give those packages a different full name? That's the question that I've never seen answered adequately. > If you would grant there's use to avoiding those edits, and so there's > use to having relative imports, then you might see a reason to solve > the problems where the names in a package conflict with those along > the load path. Otherwise, if there's only absolute imports, there's > no ambiguity to resolve. I'd say there's a legitimate need for > relative imports, and we need some explicit gesture to disambiguate > between a relative and absolute import, one way or the other. I think that moving packages around in the package namespace is a bad idea. But maybe you can give me an answer to the question above to convince me. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 15 14:57:32 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 14:57:38 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Mon, 15 Dec 2003 14:51:39 EST." References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> Message-ID: <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> > >BTW, eval() is also part of that full set, and exec is part of the > >language. These invoke the Python bytecode compiler. (And Jim > >Hugunin's IronPython is really slow on this, like 60x slower than > >CPython; I think Jim reported that Jython was 20x slower on exec().) > > > >How do you plan to handle these? What if the entire benchmark was > >given as a triple quoted string that was passed to the exec statement? > > We agreed no eval as part of this challenge, since the point was the > speed of the interpreter engine not of the compiler. I don't see that > on the list of challenge points you posted earlier, though, so it was > either omitted inadvertently or I got snookered. The former, I expect. Fair enough. "No exec or eval" is now officially part of the challenge. You know, I *want* you to win, at least if you can win by a great big margin. Because then we could switch to Parrot to make Python faster. I just very much doubt that you'll be able to bat CPython. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Mon Dec 15 15:21:16 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 15 15:21:22 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Message-ID: [Tim] >> "Not that hard" depends on your background. The idea is to bury Dan >> under esoteric algorithms from a huge number of obscure specialties >> . [Dan Sugalski] > Heh. You're expecting me to do far too much work. I plan on tackling > bytecode optimization only if it turns out that a straightforward > transform from python bytecode to parrot bytecode is isn't > sufficiently faster than CPython. Funky algorithms expressed entirely > in python bytecode won't make much of a difference... There is a serious point buried in this. This is how Python spells small-int multiplication in the bytecode: BINARY_MULTIPLY And this is how it spells unbounded-int Karatsuba multiplication in the bytecode: BINARY_MULTIPLY Same thing, and floating multiply, complex multiply, catenating N copies of a string, and arbitrary user-defined Python code overloading infix "*" for a class are spelled BINARY_MULTIPLY in the bytecode too. There's simply nothing in the bytecode to distinguish these cases. It wouldn't be hard for Guido to construct a vanilla-looking Python program that tickles some of the extreme-win special cases in the type-dependent implementations of the operators. That wouldn't be fair, so I'll hit him if he does , but in some smaller senses it can't be avoided. For example, all idiomatic Python programs use dicts (hashes), and the dict implementation dynamically keeps track of whether any non-string key has been added (Python dicts can be indexed by lots of things, not just strings). So long as it hasn't, special methods specialized to string keys are used; but as soon as, e.g., a float key is added, a Python dict switches to using slower, more-general methods. None of that is reflected in the bytecode, of course. There's a lot of stuff like that. pystone tickles little of that nature, because pystone is such an *a*typical Python program. So, regardless of what the official benchmark turns out to be, I'll probably be more interested in your pystone result . From barry at python.org Mon Dec 15 15:23:15 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 15:23:23 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: References: Message-ID: <1071519794.2808.24.camel@anthem> On Mon, 2003-12-15 at 14:34, Ken Manheimer wrote: > Currently, we can just plop the things in place, and use them. > Without relative imports, we would have to be editing the imports in > the packages in each place we use it. I've heard this before, but I still don't get it, so I'd like to tease this one out more. What does "plop the things in place" mean? Zope 2 has a few standard directories it adds to its PYTHONPATH. If I distribute a 3rd party package Zope wants to use, and that package uses absolute imports, you can just drop the package in one of those directories and it'll just work. Maybe you mean that you want to drop the mythical 3rd party package in somewhere other than a directory on Zope's PYTHONPATH and have it just work. I'd say that's too perverse of a use-case for Python to worry about. One problem with banning relative imports is that Python doesn't allow a top-level package name to live in more than one place on PYTHONPATH. It would be useful for Python to support such things out of the box, say if we wanted to adopt a Java style naming convention for package organizations. Jim uses the term "pure container package" to describe top-level packages that contain no real content in the package except subpackages. "zope" and "zodb" are examples of this in Zope 3. IWBNI Python had better/easier support for weaving together pure container package contents where subdirectories live in different locations on the file system. -Barry From tim.one at comcast.net Mon Dec 15 15:28:26 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 15 15:28:32 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> Message-ID: [Guido, to Dan Sugalski] > ... > You know, I *want* you to win, at least if you can win by a great big > margin. Because then we could switch to Parrot to make Python > faster. I just very much doubt that you'll be able to bat CPython. Note that Fredrik boosted his pystone result from 2267 under CPython, to 58038 under his Pytte1 "small Python-like language" interpreter: http://effbot.org/zone/pyttestone.htm The way to prevent this is to make the benchmark exercise lots of Python's features. I suggest at least one instance of each of the 7016 ways to call a function . From fred at zope.com Mon Dec 15 15:51:27 2003 From: fred at zope.com (Fred L. Drake, Jr.) Date: Mon Dec 15 15:51:35 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071519794.2808.24.camel@anthem> References: <1071519794.2808.24.camel@anthem> Message-ID: <16350.7887.116951.920720@sftp.fdrake.net> Barry Warsaw writes: > Jim uses the term "pure container package" to describe top-level > packages that contain no real content in the package except > subpackages. "zope" and "zodb" are examples of this in Zope 3. IWBNI > Python had better/easier support for weaving together pure container > package contents where subdirectories live in different locations on the > file system. +1 The .pth files processed by site.py support two interesting kinds of lines: - directories that should be added to sys.path, and - import statements The later alone is interesting enough, since it allows arbitrary initialization code to run when the import is executed. This still happens before all the .pth files have been processed; I suspect we really want the container package to be a little more flexible than that, at least allowing their __path__ to be computed after all .pth files have been processed. Perhaps it would be nice to have a hook (probably in sys) that gets called after site/sitecustomize are done? A "pure container package" could use a .pth file using "import mypackage", and the package's __init__.py would register a hook function in sys that searches sys.path for whatever directories should be used in mypackage.__path__. This also avoids the code being too specific to one way of thinking about packages. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From lists at webcrunchers.com Mon Dec 15 16:05:51 2003 From: lists at webcrunchers.com (JD) Date: Mon Dec 15 16:06:09 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: <16349.52054.372100.462135@montanaro.dyndns.org> References: <16349.52054.372100.462135@montanaro.dyndns.org> Message-ID: <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> On Dec 15, 2003, at 6:55 AM, Skip Montanaro wrote: > > John> Then, where should I submit my core dump (if anywhere?)? > > Submit bug reports here: > > http://sourceforge.net/tracker/?group_id=5470&atid=105470 Thanx, OK, I logged in, and got the main page which confirmed my login was good. But I cannot find any links that allow me to submit bug reports. Where is the link? I cannot find it. On the left part of the screen, I get this? Does one of these links go to the bug reporting page? ?SF.net Resources ???Site Docs ???Site Status ???Site Map ???SF.net Supporters ???Compile Farm ???Foundries ???Project Help Wanted ???New Releases ???Get Support There are two columns in the main page.... One titled... My Assigned Tracker Items and another titled My Tasks I see no place on this site where there is a link I can use to report any bugs... If it don't say "Bug reporting" or anything similar, then I need to know the name of the link, because it clearly isn't very obvious. Another thing also.... if I click on Python Bugs link in the left column, it goes into a "Browse Bugs" section, but then there is yet another "Please log in" in red. OK, so I log in again..... but when I do, I don't see the "Submit New" button, but I do see a "Submit New" link above the buttons. I DO have Cookies enabled of course, but I'm using Safari browser. By the way, I DO click on SSL. I just tried it with IE and get the same results.... I think I finally submitted it.... but can I go back and add some additional info? If so, how? Further details on the Python core dump: It core dumps (segmentation fault) when I check the database for a certain key. The offending statement is: x=database.has_key(keystring). database is open and otherwise normal. The database is "gdbm-1.8.3". The offending key is "3012a3ec5c796329fee7c242d4df00d6". John -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 2601 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20031215/4aba71e1/attachment.bin From skip at pobox.com Mon Dec 15 16:15:42 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 15 16:15:44 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> References: <16349.52054.372100.462135@montanaro.dyndns.org> <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> Message-ID: <16350.9342.865204.274114@montanaro.dyndns.org> >> Submit bug reports here: >> >> http://sourceforge.net/tracker/?group_id=5470&atid=105470 JD> Thanx, OK, I logged in, and got the main page which confirmed my JD> login was good. But I cannot find any links that allow me to submit JD> bug reports. Where is the link? I cannot find it. Pop up that page (while logged in) and search for "Submit New". JD> There are two columns in the main page.... One titled... JD> My Assigned Tracker Items and another titled My Tasks Wrong page. Once you log in, SF takes you to your "my sf.net" page. After logging in, pop back to http://sourceforge.net/projects/python then poke the "Bugs" link, which will take you to the bug tracker (the first URL above). JD> Another thing also.... if I click on Python Bugs link in the left JD> column, it goes into a "Browse Bugs" section, but then there is yet JD> another "Please log in" in red. Sounds like you have cookies disabled. To use SourceForge you need to run with cookie support turned on. Skip From jepler at unpythonic.net Mon Dec 15 16:34:03 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 15 16:34:07 2003 Subject: [Python-Dev] Christmas Wishlist In-Reply-To: <004301c3c341$48053fe0$e841fea9@oemcomputer> References: <20031215030347.GC27067@unpythonic.net> <004301c3c341$48053fe0$e841fea9@oemcomputer> Message-ID: <20031215213354.GH17242@unpythonic.net> On Mon, Dec 15, 2003 at 02:26:03PM -0500, Raymond Hettinger wrote: > Ordinarily, I would say that 3% is a reasonable price to pay for this > functionality but there is that pesky pie throwing event coming up. Maybe we can add it in right after the benchmarks get run? Jeff From jeremy at alum.mit.edu Mon Dec 15 16:36:35 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon Dec 15 16:39:25 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031215191023.GB26055@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> Message-ID: <1071524194.15985.660.camel@localhost.localdomain> One kind of problem is that newer Python features were designed without taking rexec into account. It's possible for untrusted code to cause the trusted interpreter to execute its code without restriction using descriptors. It would be really difficult to reconcile new-style classes and rexec. Perhaps a worthwhile project, but probably one accomplished by design a new restriction mechanism that builds on some of the basic mechanism of rexec. Jeremy From greg at cosc.canterbury.ac.nz Mon Dec 15 17:12:39 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Dec 15 17:12:44 2003 Subject: [Python-Dev] Changing semantics of issubclass when accepting atuple In-Reply-To: <200312150559.hBF5xw616334@c-24-5-183-134.client.comcast.net> Message-ID: <200312152212.hBFMCdh06520@oma.cosc.canterbury.ac.nz> Guido: > while an instance of my_favourite_types would indeed be an instance of > float, 3.14 is not an instance of my_favourite_types! You're right. BrainFartError. :-( Hmmm... If there were an __isinstance__ protocol for types, a suitable object could be constructed... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From janssen at parc.com Mon Dec 15 17:49:50 2003 From: janssen at parc.com (Bill Janssen) Date: Mon Dec 15 17:50:17 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 11:34:33 PST." Message-ID: <03Dec15.144959pst."58611"@synergy1.parc.xerox.com> Ken Manheimer writes: > I'd say there's a legitimate need for > relative imports, and we need some explicit gesture to disambiguate > between a relative and absolute import, one way or the other. I'd agree with Ken here. I think the importance of being able to do relative imports grows as well with the ability to compile Python into foreign frameworks, like Jython or .NET. I'd invent an ugly syntax for relative imports, and let the default be absolute. Bill From barry at python.org Mon Dec 15 17:53:41 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 15 17:53:48 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <03Dec15.144959pst."58611"@synergy1.parc.xerox.com> References: <03Dec15.144959pst."58611"@synergy1.parc.xerox.com> Message-ID: <1071528820.14604.13.camel@anthem> On Mon, 2003-12-15 at 17:49, Bill Janssen wrote: > I'd agree with Ken here. I think the importance of being able to do > relative imports grows as well with the ability to compile Python into > foreign frameworks, like Jython or .NET. Why? I'd really like to see some concrete use cases for relative imports, other than just saving some typing. -Barry From aahz at pythoncraft.com Mon Dec 15 18:18:49 2003 From: aahz at pythoncraft.com (Aahz) Date: Mon Dec 15 18:18:56 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: <16350.9342.865204.274114@montanaro.dyndns.org> References: <16349.52054.372100.462135@montanaro.dyndns.org> <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> <16350.9342.865204.274114@montanaro.dyndns.org> Message-ID: <20031215231849.GB17244@panix.com> On Mon, Dec 15, 2003, Skip Montanaro wrote: > > Sounds like you have cookies disabled. To use SourceForge you need to run > with cookie support turned on. ...and you can't use Lynx, either. :-( -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From lkcl at lkcl.net Mon Dec 15 18:36:39 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Mon Dec 15 18:36:34 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <1071524194.15985.660.camel@localhost.localdomain> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> Message-ID: <20031215233639.GE26055@lkcl.net> On Mon, Dec 15, 2003 at 04:36:35PM -0500, Jeremy Hylton wrote: > One kind of problem is that newer Python features were designed without > taking rexec into account. It's possible for untrusted code to cause > the trusted interpreter to execute its code without restriction using > descriptors. It would be really difficult to reconcile new-style > classes and rexec. Perhaps a worthwhile project, but probably one > accomplished by design a new restriction mechanism that builds on some > of the basic mechanism of rexec. okay. i think the only really sensible way forward is to begin from a sound basis - one that is going to be a big job to add retrospectively, but a simple beginning can be made. proposal: how about building ACLs into the python codebase? the basic operations are read, write, execute and modify_acl the advanced operations are "applies-to-sub-objects" and "automatically-added-to-created-sub-objects". it should be easy to add more operations, both default ones AND user-defined ones. [the advanced operations are based on the new NT 5.0 (aka windows 2000) enhancements to the original VMS security model back from dave cutler's time when he worked for DEC and then he and his team got poached by microsoft to do NT, yes it's a long story.] the basic primitives are __get_acl__(), __set_acl__() and __check_acl__(operation_type). the acl should consist of a dictionary of aces [access control entries] where the name is the function or module name, and the value is an ace object. the objects should contain a __get_ace__() function and a __set_ace__() function. the ace should contain a permission (read, write, execute, modify_acl etc.) and an "ALLOW" or "DENY" qualifier. the algorithm for evaluating an acl has been worked out already, and implemented originally by matthew chapman of the samba team, so code under the GPL already exists [in an NT-like environment which is over-the-top]. the basics of the evaluation algorithm are that you pass in who you are (in this case, that's the function or module name), what type of operation you want to perform (read, write, execute, or combination of these) and the ACL. the aces are evaluated, looking through the dictionary (actually it's going to have to be a list of tuples now that i think of it because the order is important, due to the ALLOW and DENY attributes), checking first that the ACE name matches the "who am i" and then checking what type of permission was requested against the ACE entries. the tricky bit is the "inheritance" permissions, which create "shadow" ACLs or, more specifically, they "throw shadows" down the sub-objects, adding or removing permissions from all sub-objects. in this way, it is possible to implement rexec by simply removing the write permissions at the top level as an "inherited" permission. an empty acl means "anything goes" (including that of "any acl may be added to this object"). an empty acl also means that there is no performance penalty for having acl code in the python codebase. l. From janssen at parc.com Mon Dec 15 18:45:11 2003 From: janssen at parc.com (Bill Janssen) Date: Mon Dec 15 18:45:25 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: Your message of "Mon, 15 Dec 2003 14:53:41 PST." <1071528820.14604.13.camel@anthem> Message-ID: <03Dec15.154514pst."58611"@synergy1.parc.xerox.com> > Why? I'd really like to see some concrete use cases for relative > imports, other than just saving some typing. Suppose you've developed a Python top-level application, composed of a package 'foo' containing a number of submodules and subpackages. You'd now like to add this to a Jython application as a library, part of a larger application. It's nice to be able to just add a higher-level package structure, com.mycompany.myapp, to the Python package, to fit in with the style of your application. Yet you don't want to edit the Python code to 'import com.mycompany.myapp.foo.submodule' everywhere; you'd like 'import submodule' or 'import foo.submodule' to continue working. Often the code has to work in both the stand-alone and the federated case. This is just a special case of the federation problem that we've seen for years in distributed computing; previously top-level names get federated when new higher-level constructs are formed. Ken mentioned enterprise programming; it's another case of this. Two companies merge and start merging their code. It's much better to be able to write (in the new higher-level module bletch.py) import foo import bar and have the subpackages under foo and bar continue to work without rewrites, than have to go through all the code under foo and bar and change it to know about the new higher-level bletch namespace. Bill From tismer at tismer.com Mon Dec 15 21:25:57 2003 From: tismer at tismer.com (Christian Tismer) Date: Mon Dec 15 21:26:09 2003 Subject: [Python-Dev] Does anybody really use frame->f_tstate ? Message-ID: <3FDE6D35.3090100@tismer.com> Hi colleagues, this is my second attempt to get rid of the f_tstate field in frames. I need to find every user of this field. What am I talking about? Well, Python always has a thread state variable which is a unique handle to the current thread. This variable is accessed in many places, and there exists a fast macro to get at it. Every executing Python frame also gets a copy on creation. In some cases, this frame->f_tstate field is used, in other cases the current tstate variable is used. If this sounds foreign to you, please stop reading here. ------------------------------------------------------------- I would like to get rid of the frame->f_tstate, and I'm trying to find out if there is a need for it. I don't need it, for Stackless, it is the opposite, it disturbs. There was a small thread about this in June this year, where Guido convinced me that it is possible to create a traceback on a frame that comes from a different thread. http://mail.python.org/pipermail/python-dev/2003-June/036254.html Ok, this is in fact possible, although I don't think anybody has a need for this. My question to all extension writers is this: If you use frame->f_tstate at all, do you use it just because it is handy, or do you want to use it for some other purpose? One purpose could be that you really want to create a traceback on a different than the current thread. I have never seen this, but who knows, so that's why I'm asking the Python world. In most cases, a traceback will be created on a frame that is currently processd or just has been processed. Accessing a frame of a different thread that is being processed might make sense for special debugger cases. My proposal is -------------- a) change semantics of PytraceBack_Here to use the current tstate. b) if such a special purpose exists, create a new function for it. c) if urgent, different needs exist to keep f_tstate, then let's forget about this proposal. Especially for Stackless, I'd be keen of getting rid of this. thanks for input -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at tismer.com Mon Dec 15 21:34:58 2003 From: tismer at tismer.com (Christian Tismer) Date: Mon Dec 15 21:34:49 2003 Subject: [Python-Dev] instancemethod_getattro seems to be partially wrong In-Reply-To: <200311240126.hAO1Q9I01704@c-24-5-183-134.client.comcast.net> References: <3FB99A6E.5070000@tismer.com> <200311180604.hAI64Ku02457@c-24-5-183-134.client.comcast.net> <3FBAC6E4.2020202@tismer.com> <200311190133.hAJ1X4J13394@c-24-5-183-134.client.comcast.net> <3FBACC4F.7090404@tismer.com> <200311190507.hAJ575213691@c-24-5-183-134.client.comcast.net> <3FBC3296.1090004@tismer.com> <200311200618.hAK6Ikv23729@c-24-5-183-134.client.comcast.net> <3FBD7C45.3020607@tismer.com> <200311222338.hAMNcnG03504@c-24-5-183-134.client.comcast.net> <3FC046BC.3030500@tismer.com> <200311240126.hAO1Q9I01704@c-24-5-183-134.client.comcast.net> Message-ID: <3FDE6F52.5090104@tismer.com> Guido van Rossum wrote: Hi Guido, this is a little follow-up. I didn't forget about it! ... >>So I'm just saying that pickle.py in wrong in just one place: >> >> reduce = getattr(obj, "__reduce__", None) >> if reduce: >> rv = reduce() >> >>should be: >> >> reduce = getattr(type(obj), "__reduce__", None) >> if reduce: >> rv = reduce(obj) >>""" > > > Right. (That's what I was trying to say, too. :-) > > >>An almost trivial change, although I also had to change copy.py, >>and overall I was unhappy since this extends my patch set to more >>than replacing python2x.dll, but I hope this will become an >>official patch and back-patch. > > > Give it to me baby. (On SF. :-) So here the bad news: After changing the C code to use the new style, and also changing all occourances in Python code, I used this on Zope, and ExtensionClass did not play well, it cannot stand this protocol unchanged. Zope won't start. Therefore, I reverted my changes and now I'm waiting for the day where ExtensionClass is no loger used. (or should I patch *that* complicated stuff as well?) cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido at python.org Mon Dec 15 23:25:52 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 15 23:26:06 2003 Subject: [Python-Dev] instancemethod_getattro seems to be partially wrong In-Reply-To: Your message of "Tue, 16 Dec 2003 03:34:58 +0100." <3FDE6F52.5090104@tismer.com> References: <3FB99A6E.5070000@tismer.com> <200311180604.hAI64Ku02457@c-24-5-183-134.client.comcast.net> <3FBAC6E4.2020202@tismer.com> <200311190133.hAJ1X4J13394@c-24-5-183-134.client.comcast.net> <3FBACC4F.7090404@tismer.com> <200311190507.hAJ575213691@c-24-5-183-134.client.comcast.net> <3FBC3296.1090004@tismer.com> <200311200618.hAK6Ikv23729@c-24-5-183-134.client.comcast.net> <3FBD7C45.3020607@tismer.com> <200311222338.hAMNcnG03504@c-24-5-183-134.client.comcast.net> <3FC046BC.3030500@tismer.com> <200311240126.hAO1Q9I01704@c-24-5-183-134.client.comcast.net> <3FDE6F52.5090104@tismer.com> Message-ID: <200312160425.hBG4PqN18886@c-24-5-183-134.client.comcast.net> > >>So I'm just saying that pickle.py in wrong in just one place: > >> > >> reduce = getattr(obj, "__reduce__", None) > >> if reduce: > >> rv = reduce() > >> > >>should be: > >> > >> reduce = getattr(type(obj), "__reduce__", None) > >> if reduce: > >> rv = reduce(obj) > >>""" > > > > > > Right. (That's what I was trying to say, too. :-) > > > >>An almost trivial change, although I also had to change copy.py, > >>and overall I was unhappy since this extends my patch set to more > >>than replacing python2x.dll, but I hope this will become an > >>official patch and back-patch. > > > > Give it to me baby. (On SF. :-) > > So here the bad news: > After changing the C code to use the new style, and also changing > all occourances in Python code, I used this on Zope, and > ExtensionClass did not play well, it cannot stand this protocol > unchanged. Zope won't start. > > Therefore, I reverted my changes and now I'm waiting for the > day where ExtensionClass is no loger used. > (or should I patch *that* complicated stuff as well?) No, just wait until it dies. I think Jim wants it to die to; the latest Zope version requires Python 2.3.2 but it will take time to retire it, and old Zope versions are persistent. --Guido van Rossum (home page: http://www.python.org/~guido/) From DavidA at ActiveState.com Tue Dec 16 01:49:51 2003 From: DavidA at ActiveState.com (David Ascher) Date: Tue Dec 16 01:53:45 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <03Dec15.154514pst."58611"@synergy1.parc.xerox.com> References: <03Dec15.154514pst."58611"@synergy1.parc.xerox.com> Message-ID: <3FDEAB0F.6010605@ActiveState.com> Bill Janssen wrote: > It's much better to be able to > write (in the new higher-level module bletch.py) > > import foo > import bar > > and have the subpackages under foo and bar continue to work without > rewrites, than have to go through all the code under foo and bar and > change it to know about the new higher-level bletch namespace. Right. Or as someone said: Namespaces are one honking great idea -- let's do more of those! From tjreedy at udel.edu Tue Dec 16 02:00:10 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Tue Dec 16 02:00:17 2003 Subject: [Python-Dev] Re: Pie-thon benchmarks References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> Message-ID: > You know, I *want* you to win, at least if you can win by a great big > margin. Because then we could switch to Parrot to make Python > faster. I just very much doubt that you'll be able to bat CPython. For this to be thinkable, and for the test to be fair, Parrot must be at least as semantically 'broad' as CPython (ie, handle every possible meaning of every bytecode). So I would include at least the syntax and standard lib test files (- exec and eval tests) as part of the benchmark. This also gets to the 'provably correct' aspect of the rules. Terry J. Reedy From tjreedy at udel.edu Tue Dec 16 02:13:29 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Tue Dec 16 02:13:35 2003 Subject: [Python-Dev] Re: rexec.py unuseable References: <20031215173156.GC25203@lkcl.net> Message-ID: "Luke Kenneth Casson Leighton" wrote in message If you search the pydev summaries (available on the site) for perhaps the past 15 months for 'rexec', you should find several threads on why it was dropped and how it might be replaced. TJR From press at webdevmagazine.co.uk Tue Dec 16 04:02:41 2003 From: press at webdevmagazine.co.uk (Mr.Bogomil Shopov) Date: Tue Dec 16 02:59:19 2003 Subject: [Python-Dev] Web Development Conference Message-ID: <3FDECA31.4030400@webdevmagazine.co.uk> Hello, I am coordinator of a Web Development Conference that will be held in March 2003 in Sofia, Bulgaria. We would like to present to the Conference Python language- new directions and some programmer's tricks. Is there any Bulgarian member in your community or what are your conditions to arrive for this Conf here? Regards, Bogomil Shopov WebDevMagazine phone: +359 (0) 98 505914 Conference 2004 Sofia http://webdevmagazine.co.uk/conf/ CONFIDENTIALITY NOTICE This Email is confidential and may also be privileged. If you are not the intended recipient, please notify the sender IMMEDIATELY; you should not copy the email or use it for any purpose or disclose its contents to any other person. privacy report: privacy@spisanie.com From mwh at python.net Tue Dec 16 05:03:00 2003 From: mwh at python.net (Michael Hudson) Date: Tue Dec 16 05:03:04 2003 Subject: [Python-Dev] Please tell me where I can submit bug reports.... In-Reply-To: <20031215231849.GB17244@panix.com> (aahz@pythoncraft.com's message of "Mon, 15 Dec 2003 18:18:49 -0500") References: <16349.52054.372100.462135@montanaro.dyndns.org> <76BB1B8C-2F42-11D8-BFA1-0030656C6B9E@webcrunchers.com> <16350.9342.865204.274114@montanaro.dyndns.org> <20031215231849.GB17244@panix.com> Message-ID: <2mfzflhygb.fsf@starship.python.net> Aahz writes: > On Mon, Dec 15, 2003, Skip Montanaro wrote: >> >> Sounds like you have cookies disabled. To use SourceForge you need to run >> with cookie support turned on. > > ...and you can't use Lynx, either. :-( It seems that you can, so long as you "attach" /dev/null to the bug report each time you do anything... Cheers, mwh -- surely, somewhere, somehow, in the history of computing, at least one manual has been written that you could at least remotely attempt to consider possibly glancing at. -- Adam Rixey From ncoghlan at iinet.net.au Tue Dec 16 05:32:16 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Tue Dec 16 05:32:27 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> References: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> Message-ID: <3FDEDF30.1050904@iinet.net.au> Guido van Rossum wrote: >>I don't understand the desire to totally prevent relative imports. >>It would be quite useful in avoiding growing module-naming problems, >>where you have to be careful about shadowing a global module with >>one in your package. > > I don't see the problem, or I misuderstand what "it" refers to; it > seems you have this backwards if it refers to relative imports. Say > my package P defines a submodule sys. If we require absolute imports, > there is no ambiguity: the submodule sys must be imported as P.sys > while the standard sys module can be imported as simply sys. If absolute imports were to be the only type allowed, then it would seem that the only possible location for naming conflicts is in the _first_ element. So if I wanted to use two different third party modules, both of which have unfortunately chosen the same name for the top-level package, the only way to let them co-exist is to redo _all_ of the imports in one or the other of them. Whereas, if relative pathing is possible, I believe that all I have to do is push them one level down in the package heirarchy (using distinct names that I invent), and neither of them ever even knows about the other's existence. I can get at both of them unambiguously, by using my new top=level names, and neither package even knows that it is no longer starting at the top of the import heirarchy. Or is there some other solution being proposed to this problem, and I just haven't understood it? Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From mcherm at mcherm.com Tue Dec 16 08:37:34 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Tue Dec 16 08:37:54 2003 Subject: [Python-Dev] RE: rexec.py unuseable Message-ID: <1071581854.3fdf0a9eba3ea@mcherm.com> Luke writes: > i think the only really sensible way forward is to begin from a > sound basis... I agree completely. > proposal: how about building ACLs into the python codebase? Actually, I rather prefer the approach that has been mentioned before of trying to use capabilities. See, for instance, the threads on Capabilities found here: http://mail.python.org/pipermail/python-dev/2003-March/thread.html#33854 Of course, that's not trivial to add either, since some way would need to be provided for disabling Python's powerful introspection. > the algorithm for evaluating an acl has been worked out already, > and implemented originally by matthew chapman of the samba team, > so code under the GPL already exists [in an NT-like environment which > is over-the-top]. Unfortunately, code under the GPL is of no use at all for Python which is released with a more flexible (but GPL compatible) liscense. And I'm not quite sure what you mean by an "NT-like environment" being "over-the-top". > an empty acl also means that there is no performance penalty for having > acl code in the python codebase. Sorry, but I simply don't believe this. Really. Think about it a bit, and if you still think you're right, I'll provide some reasons, but I suspect you'll realize that it's simply not true. I *DO* think though, that some amount of slow-down *IS* acceptable (pie or no pie ) if it provided powerful new capabilities like *reliable* restricted execution environments. -- Michael Chermside From lkcl at lkcl.net Tue Dec 16 09:12:07 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 09:12:02 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <1071581854.3fdf0a9eba3ea@mcherm.com> References: <1071581854.3fdf0a9eba3ea@mcherm.com> Message-ID: <20031216141207.GA17021@lkcl.net> On Tue, Dec 16, 2003 at 05:37:34AM -0800, Michael Chermside wrote: > Luke writes: > > i think the only really sensible way forward is to begin from a > > sound basis... > > I agree completely. > > > proposal: how about building ACLs into the python codebase? > > Actually, I rather prefer the approach that has been mentioned before > of trying to use capabilities. capabilities, acls, schmapabilities, same thiiing :) a la "cambridge capability system" [a historic architecture that provided hardware-assisted restrictions on function calls!] > See, for instance, the threads on > Capabilities found here: > http://mail.python.org/pipermail/python-dev/2003-March/thread.html#33854 > Of course, that's not trivial to add either, since some way would > need to be provided for disabling Python's powerful introspection. > > the algorithm for evaluating an acl has been worked out already, > > and implemented originally by matthew chapman of the samba team, > > so code under the GPL already exists [in an NT-like environment which > > is over-the-top]. > > Unfortunately, code under the GPL is of no use at all for Python which > is released with a more flexible (but GPL compatible) liscense. And > I'm not quite sure what you mean by an "NT-like environment" being > "over-the-top". the algorithm is there, for people to examine. i wasn't recommending that people go for complete code reuse of what is in samba (even if licenses were compatible) because the code in samba is a _complete_ reimplementation of the NT acl system _including_ recreating the internals of ACLs and their on-the-wire representation. however, there is a less low-level library which uses ACLs as "blobs", and it is the implementation of the check security access function [name i forget] which is actually a _re_implementation of the exact NT equivalent function that i recommend people to examine. > > an empty acl also means that there is no performance penalty for having > > acl code in the python codebase. > > Sorry, but I simply don't believe this. Really. Think about it a bit, > and if you still think you're right, I'll provide some reasons, but > I suspect you'll realize that it's simply not true. tiny, then. in function calling: if len(acl) == 0 or check_allowed_access(callee_function, acl, EXECUTE_CAPABILITY): proceed... and in file write, again: if len(acl) == 0 or check_allowed_access(callee_function, acl, WRITE_CAPABILITY): proceed... yes, the len(acl) == 0 checks will stack up, but not terribly significantly IMO. if an acl _does_ exist, even if it's one that grants permission to all functions/modules the ability to perform _all_ operations, _then_ you get a performance hit as the check_allowed_access() function needs to "do work". > I *DO* think though, that some amount of slow-down *IS* acceptable > (pie or no pie ) if it provided powerful new capabilities like > *reliable* restricted execution environments. ack! btw yes you're right, it's better off called "capabilities" but the name CCLs - capability control lists - is a bit of a mouthful :) btw, as a first implementation, doing "recursive" capability permissions could be avoided, by simply "copying" permissions down the object tree. as can be imagined, this is rather wasteful of both time and resources, which is why someone at microsoft came up with the concept of "apply-to-sub-object" capabilities. but remember: if all your ACLs (sorry, CCLs) are 0-length, it's not a waste of either time _or_ resources. l. From niemeyer at conectiva.com Tue Dec 16 09:17:46 2003 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Dec 16 09:20:04 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> Message-ID: <20031216141746.GA3145@burma.localdomain> > But why do you have to give those packages a different full name? > That's the question that I've never seen answered adequately. I have done this many times, so let me try to describe at least one legitimate usage case. A couple of weeks ago I wrote a software which needs a third party package to work. OTOH, since it's a small package, I don't want to force the user to go after the package, even because I'd have to ensure that my code always work with the available version of that package. Thanks to the relative module importing mechanism, solving that is no harder than copying the third party package into my own package namespace. This idea could probably be expressed in some other way, hacking sys.path or whatever, but I belive this is a fairly common pattern, and I vote for introducing a scheme to differ between local/global importing which would not break the current flexibility. -- Gustavo Niemeyer http://niemeyer.net From mcherm at mcherm.com Tue Dec 16 10:13:21 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Tue Dec 16 10:13:41 2003 Subject: [Python-Dev] RE: rexec.py unuseable Message-ID: <1071587601.3fdf2111afc59@mcherm.com> I (Michael) wrote: [Speaking of how to provide restricted execution in Python] > Actually, I rather prefer the approach that has been mentioned before > of trying to use capabilities. > See, for instance, the threads on > Capabilities found here: > http://mail.python.org/pipermail/python-dev/2003-March/thread.html#33854 Luke replied: > capabilities, acls, schmapabilities, same thiiing :) No... they're not. Read the thread I mentioned above, or read this, and some of the other documentation for the language E: http://www.erights.org/elib/capability/ode/ode-capabilities.html Later Luke writes: > btw yes you're right, it's better off called "capabilities" > but the name CCLs - capability control lists - is a bit of > a mouthful :) Again, I mean something different, NOT CCLs, but capabilities. Hmm... how about a quick summary from me (but I'm not an expert). Suppose that had some code which had an object representing a directory, with a method "open(filename, mode='r')" that opened files in the directory. Given this object, you could imagine constructing new objects with more limited capabilities. For instance, you might create a readOnlyDirectory object which had a method "open(filename)" that didn't allow specifying the mode as anything but 'r'. Or you might open a file and then pass a file object with "read()", "write()", "seek()", and other such methods, which would only access that file. So _IF_ the only way to access files were through this object (and that's a BIG if), then you could imagine a world where HAVING and object was equivalent to being able to do something. If a bit of code had access to a read-only-file object then it could read that file, but couldn't write to it, or do anything else with the file system unless it ALSO had access to some OTHER objects. That's capabilities... and it would work for most kinds of restricted resources, not just the file system. The key idea is that HAVING a pointer to the object is equivalent to having the permission to USE that object, and whatever it provides access to. There are several ways to "escape" out of this system. One way is to access some global ability... for instance, you might use the "open()" function from __builtins__. For capabilities to work in Python access to "dangerous" globals would need to be restricted. Another way is to just create an object out of nowhere, or forge a pointer to an object. Fortunately, in Python these are already impossible... pure Python code cannot forge references or create objects without access to the type or class. (C extensions can do anything and are unsafe.) Another way is to access the more powerful object that "empowers" a less powerful one... perhaps using the (supposed to be private) _directory field in the readOnlyfile object. So capabilities in Python are impossible without some sort of private data for objects (this is not a particularly onerous requirement). Yet *another* approach would be to use introspection... for instance, in Python today, given any file object, the class itself ("file") can be obtained via introspection, and then used to create other file objects. Using capabilities in Python would require that "restricted" objects have some more limited form of introspection... perhaps one could only obtain a "fake" class object which had a __equals__ method that simulated being the real class, but which had no __init__ so it couldn't be used to create new instances. At any rate, you get the idea. Capabilities are possible in Python only if some sort of "restricted mode" is created, which restricts access to some built-in abilities and which creates "restricted" objects with some private data and limited introspection. But IF you had these things (and they're NOT trivial), then capabilities may be a conceptually more elegant approach than ACLs, lleading to more elegant programs. -- Michael Chermside From tjreedy at udel.edu Tue Dec 16 11:11:21 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Tue Dec 16 11:11:30 2003 Subject: [Python-Dev] Re: Web Development Conference References: <3FDECA31.4030400@webdevmagazine.co.uk> Message-ID: "Mr.Bogomil Shopov" wrote in message news:3FDECA31.4030400@webdevmagazine.co.uk... > Hello, > I am coordinator of a Web Development Conference that will be held > in March 2003 in Sofia, Bulgaria. We would like to present to the presumably 2004 > Conference Python language- new directions and some programmer's > tricks. Is there any Bulgarian member in your community or what are > your conditions to arrive for this Conf here? This 'Python development' list has a relatively restricted readership. If you do not get a positive response here, try reposting on comp.lang.python or equivalent mainling list. Good luck with your conference. TJR From lkcl at lkcl.net Tue Dec 16 11:16:03 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 11:16:11 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <1071587601.3fdf2111afc59@mcherm.com> References: <1071587601.3fdf2111afc59@mcherm.com> Message-ID: <20031216161603.GG17021@lkcl.net> On Tue, Dec 16, 2003 at 07:13:21AM -0800, Michael Chermside wrote: > I (Michael) wrote: > [Speaking of how to provide restricted execution in Python] > > Actually, I rather prefer the approach that has been mentioned before > > of trying to use capabilities. > > See, for instance, the threads on > > Capabilities found here: > > http://mail.python.org/pipermail/python-dev/2003-March/thread.html#33854 ta. > Luke replied: > > capabilities, acls, schmapabilities, same thiiing :) > > No... they're not. Read the thread I mentioned above, or read this, > and some of the other documentation for the language E: > > http://www.erights.org/elib/capability/ode/ode-capabilities.html no offense intended: i'll read that later, i'm running out of time. without going into too many definitions, consider what i am advocating to be _like_ an access control list but instead to be a capabilities control list, instead. a "capabilities list", where it's a list ordered on a per-caller-function-name basis [with a special wildcard function name called "absolutely everything"] > > Later Luke writes: > > btw yes you're right, it's better off called "capabilities" > > but the name CCLs - capability control lists - is a bit of > > a mouthful :) > > Again, I mean something different, NOT CCLs, but capabilities. anyone have a one-para summary of the difference between capabilities and access control lists? even if it's "access control lists are lists of capabilities" which i don't know if that's true. access control lists are typically made on a per-user basis, but what i am recommending in _this_ case is that a "user" be considered to be a *function*. so maybe i _do_ mean access control list and should stick to my guns, here :) so. is the difference between capabilities and access control lists simply that capabilities lists restrict caller-function rights and access control lists restrict user rights? > Hmm... how about a quick summary from me (but I'm not an expert). > > Suppose that had some code which had an object representing a > directory, with a method "open(filename, mode='r')" that opened > files in the directory. Given this object, you could imagine > constructing new objects with more limited capabilities. For > instance, you might create a readOnlyDirectory object which had > a method "open(filename)" that didn't allow specifying the mode > as anything but 'r'. Or you might open a file and then pass a > file object with "read()", "write()", "seek()", and other such > methods, which would only access that file. [i'll have to read this in more depth later, i'm out of time, sorry] > So _IF_ the only way to access files were through this object (and > that's a BIG if), then you could imagine a world where HAVING and > object was equivalent to being able to do something. If a bit of > code had access to a read-only-file object then it could read that > file, but couldn't write to it, or do anything else with the file > system unless it ALSO had access to some OTHER objects. That's > capabilities... and it would work for most kinds of restricted > resources, not just the file system. The key idea is that HAVING > a pointer to the object is equivalent to having the permission to > USE that object, and whatever it provides access to. [again, i'll have to read this in more depth later, i'm out of time, sorry] > There are several ways to "escape" out of this system. One way is > to access some global ability... for instance, you might use the > "open()" function from __builtins__. i would expect __builtins__.open() to have _its_ own mmm... capabilities list, and _if_ that list contained a permission for the restricted function to "execute" it, then _yes_ it would be allowed, but otherwise no, definitely not. therefore, the author of the [new] rexec.py module should simply be a matter of creating the right ACLs, and applying them. in the case of __builtins__ it could be a matter of just applying a capabilities list of one item: [("all functions and modules", DENY, "write, apply-to-sub-objects")] and that would be _it_! all functions in __builtins__ would be dealt with, and restricted! of course, i don't believe it will be _quite_ that simple, but it might. > For capabilities to work in > Python access to "dangerous" globals would need to be restricted. simple: apply a capabilities list that denies dangerous actions, on all such "dangerous" globals. OH! one important "permission" is "change acl" of course, which must be "DENIED"! > Another way is to just create an object out of nowhere, or forge > a pointer to an object. Fortunately, in Python these are already > impossible... pure Python code cannot forge references or create > objects without access to the type or class. (C extensions can > do anything and are unsafe.) that is why i mentioned about "create object" permissions [capabilities]. if there is a permission/capability whereby the capabilities list of a parent object is "inherited" when a create object action is carried out, the problem you describe is alleviated. _at_ object create time (and object create itself could be a separate permission / capability, granted on a per-function and per-module basis like everything else), the capabilities are examined. what happens is that _if_ the "inherit capabilities" flag is set, then the newly created object receives a COPY of the parent object's capabilities list. ta-da, problem solved. this is a _normal_ bog-standard approach that is used in the windows nt security model, and has been for over twenty years, now, and if you count the VAX/VMS history as well, a lot longer than twenty years. > Another way is to access the more powerful object that "empowers" > a less powerful one... perhaps using the (supposed to be private) > _directory field in the readOnlyfile object. So capabilities in > Python are impossible without some sort of private data for > objects (this is not a particularly onerous requirement). Yet > *another* approach would be to use introspection... for instance, > in Python today, given any file object, the class itself ("file") > can be obtained via introspection, and then used to create other > file objects. Using capabilities in Python would require that > "restricted" objects have some more limited form of introspection... > perhaps one could only obtain a "fake" class object which had > a __equals__ method that simulated being the real class, but > which had no __init__ so it couldn't be used to create new > instances. i don't quite follow, even after reading the capabilities thread, what introspection is. but let me try to clarify so you can correct me if necessary: introspection is the ability to go via the __XXXXX__ functions etc. including the __class__ stuff, yes? well, if i follow you correctly, you simply put a capabilities list on all those things, or you put one on the entire object. or better yet, you put an "inherited" one on the class, such that _any_ object created will receive restricted capabilities. etc. > At any rate, you get the idea. Capabilities are possible in Python > only if some sort of "restricted mode" is created, which restricts > access to some built-in abilities and which creates "restricted" > objects with some private data and limited introspection. > But IF > you had these things (and they're NOT trivial), then capabilities > may be a conceptually more elegant approach than ACLs, lleading to > more elegant programs. i believe that if you understand what i am suggesting, and i hope that filling in some examples above, i believe that what i am suggesting covers your concerns. the capabilities lists that i am recommending can be applied on a per-function and per-object basis and can be inherited. in this way, useful restrictions can be made to achieve the expected results. ... i didn't say, however, that it wouldn't require quite a lot of thought about _which_ functions to apply restricted capabilities lists to! starting, of course, with the "nothing goes" one, and repeatedly attempting to run code. l. From guido at python.org Tue Dec 16 11:21:21 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 11:21:27 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: Your message of "Sun, 14 Dec 2003 14:27:52 PST." <200312142227.hBEMRr915819@c-24-5-183-134.client.comcast.net> References: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> <200312142227.hBEMRr915819@c-24-5-183-134.client.comcast.net> Message-ID: <200312161621.hBGGLLs19795@c-24-5-183-134.client.comcast.net> Dan, I have one more clarification request. I assume you will support import (of Python modules); that shouldn't be a problem right? The explicit ban on extensions and the insistence of the full builtins would suggest that import is okay. I'm not assuming the standard library (beyond the builtins) is available, but I *am* assuming I can split the benchmark into multiple files. --Guido van Rossum (home page: http://www.python.org/~guido/) From raymond.hettinger at verizon.net Tue Dec 16 12:30:46 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue Dec 16 12:31:02 2003 Subject: [Python-Dev] Filename suffix default for command line Message-ID: <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> Would there be any disadvantage to checking for filename.py whenever filename is not found so that "super.py" could be invoked with either "python super.py" or "python super"? Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031216/2ff80142/attachment.html From guido at python.org Tue Dec 16 12:41:06 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 12:41:11 2003 Subject: [Python-Dev] Filename suffix default for command line In-Reply-To: Your message of "Tue, 16 Dec 2003 12:30:46 EST." <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> References: <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> Message-ID: <200312161741.hBGHf6w20055@c-24-5-183-134.client.comcast.net> > Would there be any disadvantage to checking for filename.py whenever > filename is not found so that "super.py" could be invoked with either > "python super.py" or "python super"? It feels horribly DOS-like. Why? To save typing three characters? Why not rename the file then? Would "python super.py" also look for super.py.py if super.py didn't exist? What would the error message say? --Guido van Rossum (home page: http://www.python.org/~guido/) From DavidA at ActiveState.com Tue Dec 16 12:49:17 2003 From: DavidA at ActiveState.com (David Ascher) Date: Tue Dec 16 12:52:24 2003 Subject: [Python-Dev] instancemethod_getattro seems to be partially wrong In-Reply-To: <200312160425.hBG4PqN18886@c-24-5-183-134.client.comcast.net> References: <3FB99A6E.5070000@tismer.com> <200311180604.hAI64Ku02457@c-24-5-183-134.client.comcast.net> <3FBAC6E4.2020202@tismer.com> <200311190133.hAJ1X4J13394@c-24-5-183-134.client.comcast.net> <3FBACC4F.7090404@tismer.com> <200311190507.hAJ575213691@c-24-5-183-134.client.comcast.net> <3FBC3296.1090004@tismer.com> <200311200618.hAK6Ikv23729@c-24-5-183-134.client.comcast.net> <3FBD7C45.3020607@tismer.com> <200311222338.hAMNcnG03504@c-24-5-183-134.client.comcast.net> <3FC046BC.3030500@tismer.com> <200311240126.hAO1Q9I01704@c-24-5-183-134.client.comcast.net> <3FDE6F52.5090104@tismer.com> <200312160425.hBG4PqN18886@c-24-5-183-134.client.comcast.net> Message-ID: <3FDF459D.30507@ActiveState.com> Guido van Rossum wrote: > the latest Zope version requires Python 2.3.2 but it will take time to > >retire it, and old Zope versions are persistent. > > I thought that was a *feature* of Zope and the whole ZODB thing. --da From dan at sidhe.org Tue Dec 16 13:12:13 2003 From: dan at sidhe.org (Dan Sugalski) Date: Tue Dec 16 13:12:23 2003 Subject: [Python-Dev] Pie-thon benchmarks In-Reply-To: <200312161621.hBGGLLs19795@c-24-5-183-134.client.comcast.net> References: <200312141723.hBEHNvg15553@c-24-5-183-134.client.comcast.net> <200312142227.hBEMRr915819@c-24-5-183-134.client.comcast.net> <200312161621.hBGGLLs19795@c-24-5-183-134.client.comcast.net> Message-ID: At 8:21 AM -0800 12/16/03, Guido van Rossum wrote: >Dan, I have one more clarification request. > >I assume you will support import (of Python modules); that shouldn't >be a problem right? The explicit ban on extensions and the insistence >of the full builtins would suggest that import is okay. I'm not >assuming the standard library (beyond the builtins) is available, but >I *am* assuming I can split the benchmark into multiple files. Oh, absolutely. Loading in external bytecode files are fine. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Tue Dec 16 13:26:59 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 13:27:05 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Tue, 16 Dec 2003 12:17:46 -0200." <20031216141746.GA3145@burma.localdomain> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> Message-ID: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> A lot people have presented a good case for relative imports. Nobody has argued to keep the status quo (where imports are ambiguous about whether they are meant to be absolute or relative). So I suggest that in 2.4, we introduce the leading dot notation for relative import, while still allowing relative import without a leading dot. In 2.5 we can start warning about relative import without a leading dot (although that will undoubtedly get complaints from folks who have code that needs to work with 2.3 still). In 3.0 we can retire ambiguous import. The use case for multiple dots should be obvious: inside a highly structured package, modules inside one subpackage must have a way to do relative import from another subpackage of the same parent package. There is the remaining issue of what exactly the syntax would be. I propose to extend the from clause to allow one or more dots before the dotted name, and to make the dotted name optional if at least one leading dot is found. I propose not to change from-less import. Examples: from .foo import bar from .foo.bar import xxx from . import foobar as barfoo from ..foo.bar import * from ... import foobar, barfoo Grammar change (see Grammar/Grammar): dotted_name: NAME ('.' NAME)* | '.'+ [NAME ('.' NAME)*] --Guido van Rossum (home page: http://www.python.org/~guido/) From gerrit at nl.linux.org Tue Dec 16 13:29:12 2003 From: gerrit at nl.linux.org (Gerrit Holl) Date: Tue Dec 16 13:29:47 2003 Subject: [Python-Dev] Filename suffix default for command line In-Reply-To: <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> References: <003b01c3c3fa$57a6dee0$f1bd958d@oemcomputer> Message-ID: <20031216182912.GA4204@nl.linux.org> Raymond Hettinger wrote: > Would there be any disadvantage to checking for filename.py whenever > filename is not found so that "super.py" could be invoked with either > "python super.py" or "python super"? How about super.pyc? What if both super.pyc and super.py exist but are not equal in effect? I think the idea is not necesarry. I think most people have configured to do commandline completion, so 'python supe ' would probably achieve the same effect with the same amount of keystrokes. The disadvantage is, in my opinion, sideways: people get used to it, and it is not a good habit, because it's called super.py, not super. I don't like this proposal, I'd give it a -0 in case anyone is interested. yours, Gerrit. -- 202. If any one strike the body of a man higher in rank than he, he shall receive sixty blows with an ox-whip in public. -- 1780 BC, Hammurabi, Code of Law -- Asperger's Syndrome - a personal approach: http://people.nl.linux.org/~gerrit/english/ From guido at python.org Tue Dec 16 13:30:43 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 13:30:48 2003 Subject: [Python-Dev] Re: Pie-thon benchmarks In-Reply-To: Your message of "Tue, 16 Dec 2003 02:00:10 EST." References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> Message-ID: <200312161830.hBGIUhE20354@c-24-5-183-134.client.comcast.net> > > You know, I *want* you to win, at least if you can win by a great > > big margin. Because then we could switch to Parrot to make Python > > faster. I just very much doubt that you'll be able to beat > > CPython. > > For this to be thinkable, and for the test to be > fair, Parrot must be at least as semantically > 'broad' as CPython (ie, handle every possible > meaning of every bytecode). So I would include at > least the syntax and standard lib test files (- > exec and eval tests) as part of the benchmark. > This also gets to the 'provably correct' aspect of > the rules. Right. I don't want to include the entire test suite, but I do want to make sure the benchmark visits most dark corners of the language: metaclasses, dynamically changing operator overloading, and the like. I trust that Dan can write a VM that executes integer operations faster than anyone, but that's not the point of Python. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs at penguin.theopalgroup.com Tue Dec 16 13:35:14 2003 From: jacobs at penguin.theopalgroup.com (Kevin Jacobs) Date: Tue Dec 16 13:35:25 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: On Tue, 16 Dec 2003, Guido van Rossum wrote: > There is the remaining issue of what exactly the syntax would be. I > propose to extend the from clause to allow one or more dots before the > dotted name, and to make the dotted name optional if at least one > leading dot is found. I propose not to change from-less import. > Examples: > > from .foo import bar > from .foo.bar import xxx > from . import foobar as barfoo > from ..foo.bar import * > from ... import foobar, barfoo > > Grammar change (see Grammar/Grammar): > > dotted_name: NAME ('.' NAME)* | '.'+ [NAME ('.' NAME)*] On a related and minor note, can we please expand the grammar to allow from...import statements to continue to the next line after a trailing comma? For those of us with large, complex, and componentized projects, it would be very nice. e.g.: from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, normalize, row_ops, summary instead of: from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ normalize, row_ops, \ summary This has been a minor nit that bothers just under the threshold of fixing it myself. ;) Thanks, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (440) 871-6725 x 19 E-mail: jacobs@theopalgroup.com Fax: (440) 871-6722 WWW: http://www.theopalgroup.com/ From lkcl at lkcl.net Tue Dec 16 13:35:41 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 13:35:47 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <1071587601.3fdf2111afc59@mcherm.com> References: <1071587601.3fdf2111afc59@mcherm.com> Message-ID: <20031216183541.GH17021@lkcl.net> On Tue, Dec 16, 2003 at 07:13:21AM -0800, Michael Chermside wrote: > Suppose that had some code which had an object representing a > directory, with a method "open(filename, mode='r')" that opened > files in the directory. Given this object, you could imagine > constructing new objects with more limited capabilities. yep. in a very simple manner, with capabilities lists. > For > instance, you might create a readOnlyDirectory object which had > a method "open(filename)" that didn't allow specifying the mode > as anything but 'r'. Or you might open a file and then pass a > file object with "read()", "write()", "seek()", and other such > methods, which would only access that file. this could be achieved very simply by setting a CCL with "deny everything" permissions on an object that, ordinarily, had a full set of functions with NO restrictions, but leaving read() alone. _and_ also, importantly, adding a CCL onto the __builtins__ etc as already discussed to restrict bypassing. > So _IF_ the only way to access files were through this object (and > that's a BIG if), then you could imagine a world where HAVING and > object was equivalent to being able to do something. If a bit of > code had access to a read-only-file object then it could read that > file, but couldn't write to it, or do anything else with the file > system unless it ALSO had access to some OTHER objects. yep. > That's > capabilities... and it would work for most kinds of restricted > resources, not just the file system. The key idea is that HAVING > a pointer to the object is equivalent to having the permission to > USE that object, and whatever it provides access to. ah... that's where you and i differ, based on my experience of ACLs in NT. i envisage a separate "execute" permission from a "read" and "write" permission, a bit like directory permissions on a POSIX filesystem. under such a scheme where "execute" capability permission exists as well as "read" and "write", simply having access to an object does NOT guarantee you any rights to CALL that object, because if it doesn't have "execute" permission you can't execute [viz, call] it! ... but what you _can_ do is pass that object to another function which _does_ have execute permissions. how the hell is _that_ achieved, i hear you say??? well imagine a CCL as follows: [('totally_restricted_function, 'DENY', 'All permissions'), ('slightly_less_restricted_function', 'ALLOW', 'execute and read')] and this CCL is applied to the above-mentioned readOnlyDirectory object. if totally_restricted_function() calls slightly_less_restricted_function(), passing a readOnlyDirectory object to it, then according to the CCL above, ONLY slightly_less_restricted_function() may call functions in that object. cute, huh? l. From FBatista at uniFON.com.ar Tue Dec 16 13:37:39 2003 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Dec 16 13:39:03 2003 Subject: [Python-Dev] Relative import Message-ID: Guido van Rossum wrote: #- There is the remaining issue of what exactly the syntax would be. I #- propose to extend the from clause to allow one or more dots #- before the #- dotted name, and to make the dotted name optional if at least one #- leading dot is found. I propose not to change from-less import. Don't know the restrictions, as never used the package feature in modules, but seems natural to me to use the same syntax that in a unix filesystem (maybe without the '/'?). This way, you can reference a module in your level, in an upper level, etc... . Facundo From guido at python.org Tue Dec 16 13:43:46 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 13:43:52 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Tue, 16 Dec 2003 13:35:14 EST." References: Message-ID: <200312161843.hBGIhkE20496@c-24-5-183-134.client.comcast.net> > On a related and minor note, can we please expand the grammar to allow > from...import statements to continue to the next line after a trailing > comma? For those of us with large, complex, and componentized projects, it > would be very nice. e.g.: > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, > normalize, row_ops, > summary > > instead of: > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > normalize, row_ops, \ > summary > > > This has been a minor nit that bothers just under the threshold of fixing it > myself. ;) I guess this could be implemented by allowing [NEWLINE] after the comma in the grammar. But I'm -0 on this change; I fear that someone accidentally adding a trailing comma (or leaving it in after some rearrangements) will be very puzzled at the syntax error that comes somewhere in the middle of the next statement: from foo import bar, bufsize = 12 ^syntax error --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Tue Dec 16 14:36:36 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Dec 16 14:37:00 2003 Subject: [Python-Dev] Re: Relative import References: <200312161843.hBGIhkE20496@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum wrote: > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, > > normalize, row_ops, > > summary > > > > instead of: > > > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > > normalize, row_ops, \ > > summary > > > > > > This has been a minor nit that bothers just under the threshold of fixing it > > myself. ;) > > I guess this could be implemented by allowing [NEWLINE] after the > comma in the grammar. > > But I'm -0 on this change; I fear that someone accidentally adding a > trailing comma (or leaving it in after some rearrangements) will be > very puzzled at the syntax error that comes somewhere in the middle of > the next statement: > > from foo import bar, > > bufsize = 12 > ^syntax error you could require parentheses: from OPAL.Modules.Financial.Reports.COGS import ( generate, webOptions, normalize, row_ops, summary ) (after all, from-import is a kind of assignment, and "a, b = c" is the same thing as "(a, b) = c"...) From martin at v.loewis.de Tue Dec 16 14:39:37 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Dec 16 14:41:07 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031215233639.GE26055@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > i think the only really sensible way forward is to begin from a > sound basis - one that is going to be a big job to add retrospectively, > but a simple beginning can be made. > > proposal: how about building ACLs into the python codebase? I fail to see how ACLs are a sound basis to solve the problem that rexec solves. ACLs, in my view, are a sound basis to solve a different problem (that of different identities accessing the same resources). Also, it seems that nowhere in your proposal you state how ACLs should be integrated into Python: I.e. what objects are protected by ACLs, and how do you classify the various actions in a Python program as read/write/execute/modify_acl? E.g. given 3 * 4 Is that read, write, execute, and which ACL(s) is(are) considered? Regards, Martin From lkcl at lkcl.net Tue Dec 16 14:52:00 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 14:53:16 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> Message-ID: <20031216195200.GI17021@lkcl.net> On Tue, Dec 16, 2003 at 08:39:37PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > i think the only really sensible way forward is to begin from a > > sound basis - one that is going to be a big job to add retrospectively, > > but a simple beginning can be made. > > > > proposal: how about building ACLs into the python codebase? > > I fail to see how ACLs are a sound basis to solve the problem that > rexec solves. ACLs, in my view, are a sound basis to solve a different > problem (that of different identities accessing the same resources). substitute "identity" for "calling object" i.e. function and i believe that there is a viable proposal. l. From lkcl at lkcl.net Tue Dec 16 14:55:25 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 14:57:22 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> Message-ID: <20031216195525.GJ17021@lkcl.net> On Tue, Dec 16, 2003 at 08:39:37PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > i think the only really sensible way forward is to begin from a > > sound basis - one that is going to be a big job to add retrospectively, > > but a simple beginning can be made. > > > > proposal: how about building ACLs into the python codebase? > > I fail to see how ACLs are a sound basis to solve the problem that > rexec solves. ACLs, in my view, are a sound basis to solve a different > problem (that of different identities accessing the same resources). > > Also, it seems that nowhere in your proposal you state how ACLs should > be integrated into Python: I.e. what objects are protected by ACLs, all objects - if an acl is non-null. > and how do you classify the various actions in a Python program as > read/write/execute/modify_acl? E.g. given > 3 * 4 > > Is that read, write, execute, and which ACL(s) is(are) considered? execute, and execute only, because there's no I/O involved. on the multiply operation. but _only_ if there's an actual ACL set _on_ the multiply function. if there's no acl set on the multiply function, there's no restrictions on the multiply function. l. From ws-news at gmx.at Tue Dec 16 14:53:10 2003 From: ws-news at gmx.at (Werner Schiendl) Date: Tue Dec 16 14:57:56 2003 Subject: [Python-Dev] Re: Relative import References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: Hi, "Guido van Rossum" wrote > A lot people have presented a good case for relative imports. Nobody > has argued to keep the status quo (where imports are ambiguous about > whether they are meant to be absolute or relative). So I suggest that > in 2.4, we introduce the leading dot notation for relative import, > while still allowing relative import without a leading dot. I'm +1 on it, I'd only wish an additional from __future__ import strict_import (or whatever seems a better name) to allow a module to indicate that every not explicitly relative import should be absolute. This would allow module authors to be explicit from the start and it is also a safety net against accidentially adding a relative module with the name of some absolute one. best regards Werner From lkcl at lkcl.net Tue Dec 16 14:58:56 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 14:59:30 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> Message-ID: <20031216195856.GK17021@lkcl.net> On Tue, Dec 16, 2003 at 08:39:37PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > i think the only really sensible way forward is to begin from a > > sound basis - one that is going to be a big job to add retrospectively, > > but a simple beginning can be made. > > > > proposal: how about building ACLs into the python codebase? > > I fail to see how ACLs are a sound basis to solve the problem that in the initial message to the list, i confused the concept of acls with "capabilities". sorry about that. l. From edloper at gradient.cis.upenn.edu Tue Dec 16 15:17:49 2003 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Tue Dec 16 15:18:07 2003 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 5, Issue 57 In-Reply-To: References: Message-ID: Guido said: > There is the remaining issue of what exactly the syntax would be. I > propose to extend the from clause to allow one or more dots before the > dotted name, and to make the dotted name optional if at least one > leading dot is found. I propose not to change from-less import. What's the motivation for restricting it to from-imports? I suspect that this will cause confusion, and it doesn't feel self-consistent to me: why can you call a module ".foo" in one context, and not in another? In particular, I think that if people see that you can do: from .foo import * Then they'll assume that they can also do: import .foo With the obvious semantics, i.e. equivalence to: from . import foo What do we gain by not extending the syntax to from-less imports? -Edward From martin at v.loewis.de Tue Dec 16 15:14:40 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Dec 16 15:20:46 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031216195525.GJ17021@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > > Also, it seems that nowhere in your proposal you state how ACLs should > > be integrated into Python: I.e. what objects are protected by ACLs, > > all objects - if an acl is non-null. Does this include functions and bound and unbound methods? > > 3 * 4 > > > > Is that read, write, execute, and which ACL(s) is(are) considered? > > execute, and execute only, because there's no I/O involved. > > on the multiply operation. So what operations require read or write access? > but _only_ if there's an actual ACL set _on_ the multiply function. > > if there's no acl set on the multiply function, there's no > restrictions on the multiply function. So ACLs on the objects 3 and 4 would be irrelevant? Generalizing this, given a class Foo, with a method bar(), and two instances foo1 and foo2: foo1 = Foo("/etc/passwd") foo2 = Foo("/tmp/nonexistant.yet") and considering the calls foo1.bar() and foo2.bar(): Given that *only* the ACL on the operation bar matters: Does that mean that the calls either both succeed or both fail, for a given caller? Regards, Martin From martin at v.loewis.de Tue Dec 16 15:08:37 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Tue Dec 16 15:31:30 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031216195200.GI17021@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195200.GI17021@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > > I fail to see how ACLs are a sound basis to solve the problem that > > rexec solves. ACLs, in my view, are a sound basis to solve a different > > problem (that of different identities accessing the same resources). > > substitute "identity" for "calling object" i.e. function and i > believe that there is a viable proposal. I don't think so. Suppose I have an object A protected by ACLs. Now I write a new method def doit(): A.doit() Question 1. What is the calling object? Question 2. Assuming it is "the doit function", and assuming the doit function is not listed in the ACL: Will the call be executed? If yes: What is the purpose of the ACL then? If no: I believe the proposal is not usable in practice. For example, what should be the ACL on calling open()? Regards, Martin From ws-news at gmx.at Tue Dec 16 16:02:30 2003 From: ws-news at gmx.at (Werner Schiendl) Date: Tue Dec 16 16:03:00 2003 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 5, Issue 57 References: Message-ID: Hi, "Edward Loper" wrote > > Guido said: > > [...] I propose not to change from-less import. > > What's the motivation for restricting it to from-imports? I suspect > that this will cause confusion, and it doesn't feel self-consistent to > me: why can you call a module ".foo" in one context, and not in > another? In particular, I think that if people see that you can do: > > from .foo import * > > Then they'll assume that they can also do: > > import .foo The problem is probably, to what name should the module be bound? Variable names are not allowed to contain a leading dot. But it would be possible to say: import .foo as my_foo import foo and access both through modules through the name they're bound to. Therefore I'd propose to allow relative from-less imports, but require them to always use the "as" clause. regards Werner From Jack.Jansen at cwi.nl Tue Dec 16 16:04:40 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Tue Dec 16 16:04:46 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <20031216161603.GG17021@lkcl.net> References: <1071587601.3fdf2111afc59@mcherm.com> <20031216161603.GG17021@lkcl.net> Message-ID: <76A3670A-300B-11D8-B3E2-000A27B19B96@cwi.nl> On 16-dec-03, at 17:16, Luke Kenneth Casson Leighton wrote: >> Luke replied: >>> capabilities, acls, schmapabilities, same thiiing :) >> >> No... they're not. Read the thread I mentioned above, or read this, >> and some of the other documentation for the language E: >> >> http://www.erights.org/elib/capability/ode/ode-capabilities.html > > no offense intended: i'll read that later, i'm running out of time. > > without going into too many definitions, consider what i am advocating > to be _like_ an access control list but instead to be a capabilities > control list, instead. The distinction between capabilities and ACLs is really important, because they are almost each others opposite. With capabilities you have an (unforgable) right to do something and no-one cares about your identity, with ACLs you have an unforgable identity which is checked against the ACL. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From Jack.Jansen at cwi.nl Tue Dec 16 16:09:59 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Tue Dec 16 16:10:09 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: <35391BC0-300C-11D8-B3E2-000A27B19B96@cwi.nl> On 16-dec-03, at 19:26, Guido van Rossum wrote: > There is the remaining issue of what exactly the syntax would be. I > propose to extend the from clause to allow one or more dots before the > dotted name, and to make the dotted name optional if at least one > leading dot is found. I propose not to change from-less import. > Examples: > > from .foo import bar > from .foo.bar import xxx > from . import foobar as barfoo > from ..foo.bar import * > from ... import foobar, barfoo Is "from . import *" allowed? Whenever I start to think about it I tend to go into oscillation (yes because within package foo its the same as "from foo import *" on the outside; no because "import *" isn't allowed today either). -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From lkcl at lkcl.net Tue Dec 16 16:23:12 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 16:23:14 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> Message-ID: <20031216212312.GL17021@lkcl.net> On Tue, Dec 16, 2003 at 09:14:40PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > > Also, it seems that nowhere in your proposal you state how ACLs should > > > be integrated into Python: I.e. what objects are protected by ACLs, > > > > all objects - if an acl is non-null. > > Does this include functions and bound and unbound methods? if it's a function, it can potentially have an ACL (or a better description is CCL - capabilities control list) added, and that CCL applies to all sub-objects of the function. same with objects, classes - everything. > > > 3 * 4 > > > > > > Is that read, write, execute, and which ACL(s) is(are) considered? > > > > execute, and execute only, because there's no I/O involved. > > > > on the multiply operation. > > So what operations require read or write access? short answer: all of them. longer answer: just like with exceptions, that has to be classified according to conventions that are yet to be decided. i propose that, for example, a capability be created to control _file_ read access and _file_ write access, with large provisos on the documentation attached to these two capabilities that ensure they are distinguished from the right to _modify_ an object. basically it's all conventions, just like exceptions are conventions. all i can do is recommend a framework and some guidelines on what conventions could be fitted over that framework. heck, it may even be worthwhile defining a low-level object, named a Capability, just like an Exception is defined, and then expect people to create their own, and recommend the creation of certain Capabilities that are inherited from the Capability base class. > > but _only_ if there's an actual ACL set _on_ the multiply function. > > > > if there's no acl set on the multiply function, there's no > > restrictions on the multiply function. > > So ACLs on the objects 3 and 4 would be irrelevant? well... actually.... no :) but are such ACLs relevant? can you do 3 += 5? no you can't: python throws up a SyntaxError: can't assign to a literal. so, whilst the answer is yes, you _can_ associate an ACL [or better-named, a CCL] with the object "3" and the object "4", i would be surprised if there were any situations where they got... hang on... >>> dir(3) ['__abs__', '__add__', '__and__', '__class__', '__cmp__', '__coerce__', '__delattr__', '__div__', '__divmod__', '__doc__', '__float__', '__floordiv__', '__getattribute__', '__getnewargs__', '__hash__', '__hex__', '__init__', '__int__', '__invert__', '__long__', '__lshift__', '__mod__', '__mul__', '__neg__', '__new__', '__nonzero__', '__oct__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdiv__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__str__', '__sub__', '__truediv__', '__xor__'] yes, i believe it _might_ be relevant to set a CCL on an object "3" because someone might want to restrict how... oh, i dunno... mmm. __setattr__ gets used. >>> 3.__xor__ ^ SyntaxError: invalid syntax oh. maybe not :) however, a UserLong or a UserInt is a different matter. [btw, change of subject: why is there no __eq__, no __gt__, no __le__ in that list above?] > Generalizing this, given a class Foo, with a method bar(), and two > instances foo1 and foo2: > > foo1 = Foo("/etc/passwd") > foo2 = Foo("/tmp/nonexistant.yet") > > and considering the calls foo1.bar() and foo2.bar(): > > Given that *only* the ACL on the operation bar matters: Does that mean > that the calls either both succeed or both fail, for a given caller? i'm not entirely sure what you mean, but let's assume that objects (and their instances!) all have an __set_ccl__() function. let's also define a CCL: deny_ccl = [('all functions', DENY, 'execute')] then, let's try this: foo2 = Foo("/etc/passwd") Foo.bar.__set_ccl__(deny_ccl) foo1 = Foo("/etc/passwd") foo2.bar() // executed okay foo1.bar() *** EXCEPTION: execute capability barred to "all functions". okay, let's try this, first making sure there's no ambiguity by re-setting the CCL on the Foo class: Foo.bar.__set_ccl__(None) okay, now we go: foo3 = Foo("/etc/passwd") foo3.bar() // executed okay foo3.bar.__set_ccl__(deny_ccl) foo3.bar() *** EXCEPTION: execute capability barred to "all functions". note that there are three separate uses of __set_ccl__(). first is to set a CCL on the _class_ Foo. second is to clear a CCL the class Foo. third is to set a CCL on an instance of a Foo object. just to clarify: i _may_ have just violated my own proposal by not setting an "inherited-on-create" capability on the CCL named deny_ccl. the reason for needing to have such a capability is to be able to distinguish between setting CCLs on a class and a CCL being created on an object when an instance _of_ that class is created. this is part of the lesson learned from NT's security model (and originally VMS's) and its implementation. i won't go into more details on this inherit-on-create thing until the rest of the stuff sinks in :) l. From lkcl at lkcl.net Tue Dec 16 16:30:41 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 16:30:45 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195200.GI17021@lkcl.net> Message-ID: <20031216213041.GM17021@lkcl.net> On Tue, Dec 16, 2003 at 09:08:37PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > > I fail to see how ACLs are a sound basis to solve the problem that > > > rexec solves. ACLs, in my view, are a sound basis to solve a different > > > problem (that of different identities accessing the same resources). > > > > substitute "identity" for "calling object" i.e. function and i > > believe that there is a viable proposal. > > I don't think so. Suppose I have an object A protected by ACLs. Now I > write a new method > > def doit(): > A.doit() > > Question 1. What is the calling object? the globally-defined doit function. > Question 2. Assuming it is "the doit function", and assuming the doit > function is not listed in the ACL: Will the call be executed? see below. > If yes: What is the purpose of the ACL then? > If no: I believe the proposal is not usable in practice. For example, > what should be the ACL on calling open()? okay. that is why in ACLs (or better CCLs) it is necessary to have an "all functions" wildcard "name". in NT, this is the "world" security id "S-1-1" (i think) which represents "everyone". if you set a CCL, and you expect to deny everything but a particular function, you must have a two-entry CCL: [('function_to_be_denied_access', DENY, 'permissions_to_be_denied'), ('all functions', 'ALLOW', 'anything')] and yes, this is _normal_ practice in the use of ACLs in NT and VMS. and yes, the default action is (IIRC), "if not explicitly stated, deny". with the exception to that, of course, that if no CCL is set, it's "as if" there was a CCL set of [('all functions', 'ALLOW', 'anything')]. l. From lkcl at lkcl.net Tue Dec 16 16:36:58 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Tue Dec 16 16:37:02 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <76A3670A-300B-11D8-B3E2-000A27B19B96@cwi.nl> References: <1071587601.3fdf2111afc59@mcherm.com> <20031216161603.GG17021@lkcl.net> <76A3670A-300B-11D8-B3E2-000A27B19B96@cwi.nl> Message-ID: <20031216213658.GN17021@lkcl.net> On Tue, Dec 16, 2003 at 10:04:40PM +0100, Jack Jansen wrote: > > On 16-dec-03, at 17:16, Luke Kenneth Casson Leighton wrote: > >>Luke replied: > >>>capabilities, acls, schmapabilities, same thiiing :) > >> > >>No... they're not. Read the thread I mentioned above, or read this, > >>and some of the other documentation for the language E: > >> > >> http://www.erights.org/elib/capability/ode/ode-capabilities.html > > > > no offense intended: i'll read that later, i'm running out of time. > > > > without going into too many definitions, consider what i am advocating > > to be _like_ an access control list but instead to be a capabilities > > control list, instead. > > The distinction between capabilities and ACLs is really important, > because > they are almost each others opposite. With capabilities you have an > (unforgable) > right to do something and no-one cares about your identity, with ACLs > you have > an unforgable identity which is checked against the ACL. i'd like to introduce you to a new concept which is idential in form to an ACL - access control list - except that instead of "users" being allowed or denied access to perform certain operations you have instead _functions_ being allowed or denied access to perform certain operations. perhaps a better way to explain the concept to you is to introduce a concept called "qualified" capabilities, where what you know of as capabilities is "qualified" on a per-function (that's per-caller) basis. obviously, any object (by object i am referring generically to classes, class instances, functions, modules, absolutely anything) can potentially have many "callers", consequently it is necessary to create a _list_ of qualified capabilities, and for the relevant QCap in that list to be looked up and applied as needed. where, of course, the special wildcard name 'all functions' applies to _all_ callers. which makes what i am proposing to be named "QCCL" - qualified-capabilities control list. yuk :) l. From guido at python.org Tue Dec 16 17:07:56 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 17:08:01 2003 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 5, Issue 57 In-Reply-To: Your message of "Tue, 16 Dec 2003 15:17:49 EST." References: Message-ID: <200312162207.hBGM7uN20814@c-24-5-183-134.client.comcast.net> [Ed] > What's the motivation for restricting it to from-imports? I suspect > that this will cause confusion, and it doesn't feel self-consistent to > me: why can you call a module ".foo" in one context, and not in > another? In particular, I think that if people see that you can do: > > from .foo import * > > Then they'll assume that they can also do: > > import .foo > > With the obvious semantics, i.e. equivalence to: > > from . import foo > > What do we gain by not extending the syntax to from-less imports? Currently, the rule is that after import you can use in an expression to refer to whatever you imported. This applies to both import foo <--> foo in expressions and import foo.bar <--> foo.bar in expressions But if we were to allow import .foo we'd have to allow .foo in expressions, and I would like to reserve that syntax for the "with" statement. [Werner] > But it would be possible to say: > > import .foo as my_foo > import foo > > and access both through modules through the name they're bound to. > > Therefore I'd propose to allow relative from-less imports, but require > them to always use the "as" clause. This just adds more complexity; there's nothing you can do without this addition. [Jack] > Is "from . import *" allowed? Whenever I start to think about it I > tend to go into oscillation (yes because within package foo its the same > as "from foo import *" on the outside; no because "import *" isn't > allowed today either). Yes, "from . import " has the exact same meaning as spelling out the corresponding absolute module path after "from". --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Tue Dec 16 18:24:52 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 18:24:59 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.py unuseable) In-Reply-To: <1071587601.3fdf2111afc59@mcherm.com> Message-ID: <200312162324.hBGNOq916269@oma.cosc.canterbury.ac.nz> Michael Chermside : > Suppose that had some code which had an object representing a > directory, with a method "open(filename, mode='r')" that opened > files in the directory. Given this object, you could imagine > constructing new objects with more limited capabilities. With regard to files, we'd almost have something like that now, if it weren't for the unfortunate fact that the file type's constructor can be used to open any file. In light of this, it may have been a serious mistake to unify the 'file' type and the 'open' function. Is it too late to back out of that decision? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Tue Dec 16 18:38:20 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 18:38:37 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.py unuseable) In-Reply-To: Your message of "Wed, 17 Dec 2003 12:24:52 +1300." <200312162324.hBGNOq916269@oma.cosc.canterbury.ac.nz> References: <200312162324.hBGNOq916269@oma.cosc.canterbury.ac.nz> Message-ID: <200312162338.hBGNcKD21087@c-24-5-183-134.client.comcast.net> > Michael Chermside : > > > Suppose that had some code which had an object representing a > > directory, with a method "open(filename, mode='r')" that opened > > files in the directory. Given this object, you could imagine > > constructing new objects with more limited capabilities. Greg Ewing: > With regard to files, we'd almost have something like that > now, if it weren't for the unfortunate fact that the file > type's constructor can be used to open any file. Good observation. > In light of this, it may have been a serious mistake to > unify the 'file' type and the 'open' function. Is it too > late to back out of that decision? I think so. But to remedy this, the file constructor could simply refuse to open any files when called from restricted mode. (Although this would open up the same kind of holes that Samuele so cleverly showed exist in other built-ins that do this kind of checking.) (Re the capabilities discussion, this is Python 3.0 material if I ever saw some; no time to respond in more detail, given that we've had this thread before.) --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Tue Dec 16 19:22:49 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 19:23:16 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> Guido: > I suggest that in 2.4, we introduce the leading dot notation for > relative import, while still allowing relative import without a > leading dot. In 2.5 we can start warning about relative import > without a leading dot If I understand correctly, the rule you're working towards is that if the path starts with a dot, it's relative, otherwise it's absolute. Is that correct? I'm not sure whether eliminating the "ambiguous" category of module references is the best idea. There are situations in which it's actually what you want. Suppose there are two top-level modules A and B, and A imports B using a bare name. Then someone decides to put A into a package. The reference to B still works. Then they decide to put B into the same package as well. The reference to B still works. But if only unambiguously absolute or relative references are possible, there is no way for A to refer to B that works in all these combinations. So I think we really want *three* kinds of module reference: A: Explicitly absolute B: Explicitly relative to the current module C: Searched for upwards in the package hierarchy from the current module (Note that C is a generalisation of the current "ambiguous" references which only look in two places.) Suggested syntaxes for these: A: a.b.c. Path ends with a dot B: .a.b.c Path begins with a dot C: a Path neither begins nor ends with a dot a.b.c Note that ALL current module references would be type C according to this. Yes, this is a semantic change, but one that I believe would be for the best in the long run, and don't think would cause a lot of upheaval. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tjreedy at UDel.Edu Tue Dec 16 20:04:41 2003 From: tjreedy at UDel.Edu (Terry Reedy) Date: Tue Dec 16 20:05:14 2003 Subject: [Python-Dev] Re: Pie-thon benchmarks References: <200312151937.hBFJbYX18012@c-24-5-183-134.client.comcast.net> <200312151957.hBFJvWi18151@c-24-5-183-134.client.comcast.net> <200312161830.hBGIUhE20354@c-24-5-183-134.client.comcast.net> Message-ID: <001a01c3c439$c83a74c0$0400a8c0@HPPAVILION> > > meaning of every bytecode). So I would include at > > least the syntax and standard lib test files (- > > exec and eval tests) as part of the benchmark. > Right. I don't want to include the entire test suite, but I do want Sorry, I meant builtins test, *not* the whole suite of stdlib tests. Terrj From greg at cosc.canterbury.ac.nz Tue Dec 16 20:09:14 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 20:09:21 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161843.hBGIhkE20496@c-24-5-183-134.client.comcast.net> Message-ID: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> Guido: > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > > normalize, row_ops, \ > > summary > > But I'm -0 on this change; I fear that someone accidentally adding a > trailing comma ... will be very puzzled How about a "suite" of things to import? from OPAL.Modules.Financial.Reports.COGS import: generate, webOptions, normalize row_ops, summary Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Tue Dec 16 20:16:18 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 16 20:16:24 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 14:09:14 +1300." <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> References: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> Message-ID: <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> > > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > > > normalize, row_ops, \ > > > summary > > > > But I'm -0 on this change; I fear that someone accidentally adding a > > trailing comma ... will be very puzzled > > How about a "suite" of things to import? > > from OPAL.Modules.Financial.Reports.COGS import: > generate, webOptions, normalize > row_ops, summary Nah. But I liked the suggestion seen here earlier of allowing parenthesis: from OPAL.Modules.Financial.Reports.COGS import (generate, webOptions, normalize, row_ops, summary) --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Tue Dec 16 20:19:51 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Dec 16 20:19:57 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> References: <200312161843.hBGIhkE20496@c-24-5-183-134.client.comcast.net> Message-ID: <5.1.1.6.0.20031216201656.02363ec0@telecommunity.com> At 02:09 PM 12/17/03 +1300, Greg Ewing wrote: >Guido: > > > > from OPAL.Modules.Financial.Reports.COGS import generate, webOptions, \ > > > normalize, row_ops, \ > > > summary > > > > But I'm -0 on this change; I fear that someone accidentally adding a > > trailing comma ... will be very puzzled > >How about a "suite" of things to import? > > from OPAL.Modules.Financial.Reports.COGS import: > generate, webOptions, normalize > row_ops, summary First impression: cool! Second impression: "suites" contain statements, not a list. Third impression: why no comma after 'normalize'? From greg at cosc.canterbury.ac.nz Tue Dec 16 20:53:35 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 20:53:40 2003 Subject: [Python-Dev] Relative import In-Reply-To: <5.1.1.6.0.20031216201656.02363ec0@telecommunity.com> Message-ID: <200312170153.hBH1rZG16888@oma.cosc.canterbury.ac.nz> > Second impression: "suites" contain statements, not a list. That's why I put "suite" in quotes. It wouldn't really be a suite, just an indented block of stuff. But if you want, it could be a real suite that only happens to allow import statements... from OPAL.Modules.Financial.Reports.COGS: import generate, webOptions, normalize import row_ops, summary Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From janssen at parc.com Tue Dec 16 21:05:42 2003 From: janssen at parc.com (Bill Janssen) Date: Tue Dec 16 21:06:02 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Tue, 16 Dec 2003 16:22:49 PST." <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> Message-ID: <03Dec16.180547pst."58611"@synergy1.parc.xerox.com> > Note that ALL current module references would be type C according > to this. Yes, this is a semantic change, but one that I believe > would be for the best in the long run, and don't think would cause > a lot of upheaval. Yes, I like this. Bill From andrew-pythondev at puzzling.org Tue Dec 16 21:45:50 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Tue Dec 16 21:45:56 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170153.hBH1rZG16888@oma.cosc.canterbury.ac.nz> References: <5.1.1.6.0.20031216201656.02363ec0@telecommunity.com> <200312170153.hBH1rZG16888@oma.cosc.canterbury.ac.nz> Message-ID: <20031217024550.GB473@frobozz> On Wed, Dec 17, 2003 at 02:53:35PM +1300, Greg Ewing wrote: > > Second impression: "suites" contain statements, not a list. > > That's why I put "suite" in quotes. It wouldn't really be > a suite, just an indented block of stuff. > > But if you want, it could be a real suite that only > happens to allow import statements... > > from OPAL.Modules.Financial.Reports.COGS: > import generate, webOptions, normalize > import row_ops, summary But this is also a very odd suite, because you can't use other statements... But it does make me think of: with OPAL.Modules.Financial.Reports.COGS: import .generate, .webOptions, .normalize import .row_ops, .summary Which is like a bastard child of the relative import proposal and the with statement proposal... -Andrew. From eppstein at ics.uci.edu Tue Dec 16 22:07:52 2003 From: eppstein at ics.uci.edu (David Eppstein) Date: Tue Dec 16 22:07:46 2003 Subject: [Python-Dev] Re: Relative import References: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> Message-ID: In article <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz>, Greg Ewing wrote: > So I think we really want *three* kinds of module reference: > > A: Explicitly absolute > B: Explicitly relative to the current module > C: Searched for upwards in the package hierarchy from the current > module > > (Note that C is a generalisation of the current "ambiguous" > references which only look in two places.) > > Suggested syntaxes for these: > > A: a.b.c. Path ends with a dot > > B: .a.b.c Path begins with a dot > > C: a Path neither begins nor ends with a dot > a.b.c Is funny punctuation really the right way to spell an important distinction like this? -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From tdelaney at avaya.com Tue Dec 16 22:20:32 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Tue Dec 16 22:20:38 2003 Subject: [Python-Dev] Re: Relative import Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> > From: David Eppstein > > Is funny punctuation really the right way to spell an important > distinction like this? That's what I've been wondering. I really don't like the proposal to use . in various places to signify relative and absolute imports. It seems non-obvious both in understandability and readability. Every example I've seen has looked wrong to me. I'll try to think of something I prefer, but I am definitely -1 on the proposed syntax changes. Tim Delaney From anthony at interlink.com.au Tue Dec 16 22:27:52 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Dec 16 22:28:23 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> Message-ID: <200312170327.hBH3RrpU004309@localhost.localdomain> >>> "Delaney, Timothy C (Timothy)" wrote > > Is funny punctuation really the right way to spell an important > > distinction like this? > > That's what I've been wondering. I really don't like the proposal to use > . in various places to signify relative and absolute imports. It seems > non-obvious both in understandability and readability. I'd have to agree. In the same way that backticks are painful to deal with (they're small and non-obvious), the difference between from foo import bar and from .foo import bar is too small to be pleasant. I also have to wonder about the use case for .. and ... - looking through my code, I can't see anything where using ... rather than a fully qualified name would make the code any more readable. -- Anthony Baxter It's never too late to have a happy childhood. From greg at cosc.canterbury.ac.nz Tue Dec 16 22:31:50 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Dec 16 22:32:35 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Message-ID: <200312170331.hBH3Vog17551@oma.cosc.canterbury.ac.nz> David Eppstein : > Is funny punctuation really the right way to spell an important > distinction like this? Maybe not, but I haven't seen anything obviously better proposed yet. I've no idea what else to to for relative paths. A leading dot doesn't seem all that bad to me for that. I concede that a trailing dot would look rather odd, though. The absolute-path case could be spelled using a special name for the root of the package hierarchy, but then we need to think of a name that's sufficiently unlikely as a real module name. I don't think I like the idea of re-using the word "global", although I'm not sure. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one at comcast.net Tue Dec 16 23:42:45 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Dec 16 23:42:46 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > Nah. But I liked the suggestion seen here earlier of allowing > parenthesis: > > from OPAL.Modules.Financial.Reports.COGS import (generate, webOptions, > normalize, row_ops, summary) So do I. Something I'm still acutely aware of despite not using Emacs for years is how much trouble it would be to teach python-mode.el about a unique new way to continue a statement. Parens would "just work". Ditto for IDLE, and tokenize.py, etc etc etc etc. I bet tools parsing import statements for *content* (like pychecker, and perhaps pyclbr.py -- the others mentioned are just looking at form) would need modification regardless, though. From duncan at rcp.co.uk Wed Dec 17 04:17:25 2003 From: duncan at rcp.co.uk (Duncan Booth) Date: Wed Dec 17 04:17:39 2003 Subject: [Python-Dev] rexec.py unuseable References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> Message-ID: Luke Kenneth Casson Leighton wrote in news:20031216212312.GL17021@lkcl.net: > foo3.bar() // executed okay > foo3.bar.__set_ccl__(deny_ccl) > > foo3.bar() > *** EXCEPTION: execute capability barred to "all functions". This behaviour seems to imply a rather fundamental change to the way that Python handles bound methods. At present every time you access foo.bar you get a new object created, but I think from your description that you want to be able to associate a ccl with all the foo.bar objects. Or to put it another way: fn1 = foo3.bar fn1() // executed okay fn2 = foo3.bar fn2.__set_ccl__(deny_ccl) fn2() *** EXCEPTION: execute capability barred to "all functions". fn1() *** ??? What should this do. If I set a ccl on fn2, should it also affect access to fn1? They are separate objects, so shouldn't they have separate ccls? If they share a common ccl, then how is it going to be stored without impacting heavily on every method call? -- Duncan Booth duncan.booth at suttoncourtenay.org.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? From theller at python.net Wed Dec 17 06:32:32 2003 From: theller at python.net (Thomas Heller) Date: Wed Dec 17 06:32:47 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() In-Reply-To: <002301c3c29c$f2753480$0201010a@kret> (Wiktor Sadowski's message of "Mon, 15 Dec 2003 00:49:42 +0100") References: <002301c3c29c$f2753480$0201010a@kret> Message-ID: "Wiktor Sadowski" writes: > Adding a few lines (below) to Python would enable custom > Py_Initialize() and would allow to setup on time different than > existing (import.c-zipimport.c) import mechanisms and changing some > other Python-initialization internals without messing with Python > core. > > It should make life easier for embedders and help writing specialized > Python executables (like Zope or Quixote). (We have such specialized > executable to run packed Python programs remotely,kind of Java Web > Start). From what I know Python installers (py2exe,Gordon McMillan's > installer) could benefit from such addition as well. > > All these functions simply expose some modules globals. They don't > change anything in Python core. > > Best regards, > Wiktor Sadowski > > > /*pythonrun.c*/ > void Internal_Py_SetInitialized(int){initialized=i;} > void Internal_PyUnicode_Init(void){_PyUnicode_Init();} > void Internal_PyUnicode_Fini(void){_PyUnicode_Fini();} > void Internal_PyGILState_Init(PyInterpreterState *s, PyThreadState > *ts){_PyGILState_Init(s,ts);} > void Internal_PyGILState_Fini(void){_PyGILState_Fini();} > long Internal_PyGC_Collect(void){return PyGC_Collect();} > > /*pythonrun.h*/ > PyAPI_FUNC(void) Internal_Py_SetInitialized(int i); > PyAPI_FUNC(void) Internal_PyUnicode_Init(void); > PyAPI_FUNC(void) Internal_PyUnicode_Fini(void); > PyAPI_FUNC(void) Internal_PyGILState_Init(PyInterpreterState *, > PyThreadState *); > PyAPI_FUNC(void) Internal_PyGILState_Fini(void); > PyAPI_FUNC(long) Internal_PyGC_Collect(void); > I would rather like a function like Py_InitializeEx(struct _init*), where struct _init would have components like PYTHONPATH, optimize, and so on. Thomas From ncoghlan at iinet.net.au Wed Dec 17 07:20:17 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Dec 17 07:20:24 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <20031216213658.GN17021@lkcl.net> References: <1071587601.3fdf2111afc59@mcherm.com> <20031216161603.GG17021@lkcl.net> <76A3670A-300B-11D8-B3E2-000A27B19B96@cwi.nl> <20031216213658.GN17021@lkcl.net> Message-ID: <3FE04A01.7010500@iinet.net.au> Luke Kenneth Casson Leighton wrote: > i'd like to introduce you to a new concept which is idential > in form to an ACL - access control list - except that instead > of "users" being allowed or denied access to perform certain > operations you have instead _functions_ being allowed or > denied access to perform certain operations. I don't think this discussion is going to get anywhere until you _do_ take the time to understand the difference between the model you are proposing and the capability model that seemed to be preferred the last time executing untrusted Python code safely was discussed. ONE LINE SUMMARIES ACL/CCL/whatever you want to call it: Access the feature, then have permission either granted or denied Capabilities: If you can access it, you have permission to use it. If you aren't allowed to use it, you won't even be able to see it. Below is my attempt at explaining in more detail the difference between the two as I understand it. I must admit I am quite curious about the fact that you seem to have plenty of time to _write_ about your proposal, but no time to listen to some quite reasonable responses. ACCESS CONTROL LISTS An access control list is neutral to what a 'user' is, so I'll switch to the term 'Entity' instead. The important part is that it is a mapping: "Entity X is allowed to perform action Y on other Entity Z". It serves as a post hoc check - _after_ Entity X tries to do Y to Z, the control list for Y & Z is checked, then X is either allowed to proceed, or told to rack off, depending on what the control list reports. That is, just because X _can_ try to perform action Y on Z (the relevant method or whatever is accessible to X), doesn't mean that X _may_ perform action Y (the control list grants permission to X). With control lists, the concepts of access to a feature, and permission to use that feature, are handled separately. E.g. I can try to delete the windows directory on my machine at work (I have access to that directory, and access to the delete command), but I'm not an administrator, so the attempt will fail (I don't have permission to delete that directory). This is exactly what you are describing, even though the users are 'functions' rather than 'logged in users'. CAPABILITIES Capabilities work differently. Instead of applying a post-hoc check that slows down _every_ attempt to perform an action, they unify the concepts of 'can' and 'may'. If X is able to express the command "do Y to Z", then X is also _permitted_ to do Y to Z. No post-hoc checks required. For example, suppose I wasn't allowed to open writable files. I call: >>> target = file("myfile.txt", "w") With ACL's, I would get an error along the lines of: AccessDenied: No permission to open writable files With capabilities, I would get an error along the lines of: InvalidArgument: "w" is not a recognised file mode Does this make any more sense? Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From Jack.Jansen at cwi.nl Wed Dec 17 07:51:12 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Wed Dec 17 07:51:14 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() In-Reply-To: References: <002301c3c29c$f2753480$0201010a@kret> Message-ID: On 17-dec-03, at 12:32, Thomas Heller wrote: > I would rather like a function like Py_InitializeEx(struct _init*), > where struct _init would have components like PYTHONPATH, optimize, and > so on. You'd have to be very careful to design the struct in such a way that you'd have good compatibility. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From ncoghlan at iinet.net.au Wed Dec 17 07:58:20 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Dec 17 07:58:29 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> Message-ID: <3FE052EC.90705@iinet.net.au> Greg Ewing wrote: > So I think we really want *three* kinds of module reference: > > A: Explicitly absolute > B: Explicitly relative to the current module > C: Searched for upwards in the package hierarchy from the current > module > > (Note that C is a generalisation of the current "ambiguous" > references which only look in two places.) > Alternate spellings (plus category D, which is "Python 2.3 semantics"): A: from __absolute__.dotted.name import foo B: from __relative__.dotted.name import bar C: from __heirarchy__.dotted.name import foobar D: from dotted.name import barfoo I believe this spelling would only require import to handle the special cases as the first 'package' name (although having the compiler recognise them may be a later optimisation), and should also work with straight "import __absolute__.dotted.name.foo" statements. Then Guido's migration plan would apply to the progression of the D semantics from the Python 2.3 model to being a shorthand for the absolute semantics. And once _that_ has happened, then we could get rid of the '__absolute__' version in the name of TOOWTDI. Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From mcherm at mcherm.com Wed Dec 17 08:56:37 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Wed Dec 17 08:56:58 2003 Subject: [Python-Dev] Re: rexec.py unuseable Message-ID: <1071669397.3fe06095c9f3c@mcherm.com> Luke: Seeing as you've now had time enough to read what I'd written, I was going to respond to some of the points that you made, but then I read Nick Coghlan's posting[1], and I really don't have anything else to add. [1] http://mail.python.org/pipermail/python-dev/2003-December/041105.html Well, that's not quite true... I guess I _do_ have something small to add. In designing a security policy for file systems, there are very few things that you can do with the protected items. Separate "read", "write", and "execute" permissions seem appropriate. Furthermore, there tends to be a clear idea of a "user", and in many situations we want all access to items (files) to depend only on who that "user" is (sandboxing an email program is an exception... are you listening MS?). Finally, we often want to provide centralized administration of access rights to file systems. ACLs are a good match for these design specs. However, controlling what arbitrary programs may do is a very different beast. First of all, the kinds of restrictions that we want are HIGHLY variable. Sometimes we want to prohibit all file access. Sometimes we want to allow read-only access to a certain file only. Sometimes we want access only to a certain directory. Sometimes we don't care at all about file system access, we're concerned instead with providing an object which can be queried but only in certain ways -- and the specifics of what is permitted and what isn't are completely dependent on the particular application. On top of that, there's rarely a clear concept of "users" that controls what actions we want to permit. Instead, we usually want to allow or deny certain capabilities to particular _sections of code_. (Sometimes that will be a module, sometimes a plug-in, sometimes a child program.) All of these things suggest that maintaining a list of who is permitted to access what is simply not the best model for providing security in a programming language. I think that capabilities ARE an excellent model. Sections of code already have the concept of "access" to certain objects and variables (at least since the invention of structured programming in the late 1960s). And _if_ unforgability and private data structures are provided as elemental building tools, then it is relatively straightforward to write code that enforces whatever set of access rules is required, regardless of how complex it may be. -- Michael Chermside From mickey at tm.informatik.uni-frankfurt.de Wed Dec 17 09:01:00 2003 From: mickey at tm.informatik.uni-frankfurt.de (Michael Lauer) Date: Wed Dec 17 09:01:08 2003 Subject: [Python-Dev] bug in python arm-linux?: start_new_thread fails after popen Message-ID: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> Hi, I have a python program which works fine on x86 but doesn't work on any of my arm-linux devices (Zaurus, Ipaq, SIMpad) - all of them are running Python 2.3.2 on top of glibc 2.3.2+linuxthreads on top of kernel 2.4.18-rmk6-pxa3 respectively kernel 2.4.19-rmk7. After a long week of debugging I now can reproduce the behaviour with the following minimal case: ---------------------------------------------------------------------- import thread from time import sleep import sys import os def createPipe(): print os.popen( "ls -l" ).read() def threadMain( name ): while True: sys.stderr.write( name+": i'm still running" ) sys.stderr.flush() sleep( 1 ) if __name__ == '__main__': createPipe() print "BEFORE start_new_thread" thread.start_new_thread( threadMain, ( "1", ) ) print "AFTER start_new_thread" createPipe() print "BEFORE start_new_thread" thread.start_new_thread( threadMain, ( "2", ) ) print "AFTER start_new_thread" sleep( 5 ) ----------------------------------------------------------------------- This program - as is - just hangs in the first start_new_thread() and never comes back. If you outcomment the first call to createPipe() out, then it works fine. In the first - hanging - case, an strace shows: ------------------------------------------------------------------------- write(1, "BEFORE start_new_thread\n", 24BEFORE start_new_thread ) = 24 rt_sigprocmask(SIG_BLOCK, ~[33], [RTMIN], 8) = 0 pipe([3, 4]) = 0 clone(child_stack=0x14e080, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND) = 949 write(4, "\314V\1@\5\0\0\0\0\0\0\0\0\0\0\0\34\213\37@\10c\1@\0\0"..., 148) = 148 rt_sigprocmask(SIG_SETMASK, NULL, ~[KILL STOP 33], 8) = 0 write(4, "\0\21\3@\0\0\0\0\0\0\0\0hO\n\0\2503\23\0\377\376\373\377"..., 148) = 148 rt_sigprocmask(SIG_SETMASK, NULL, ~[KILL STOP 33], 8) = 0 rt_sigsuspend(~[KILL STOP RTMIN 33] -------------------------------------------------------------------------- The program must be kill -9'ed at this point. Can anyone explain this behaviour to me - did I find a bug? Sincerely, -- :M: -------------------------------------------------------------------------- Dipl.-Inf. Michael 'Mickey' Lauer mickey@tm.informatik.uni-frankfurt.de Raum 10b - ++49 69 798 28358 Fachbereich Informatik und Biologie -------------------------------------------------------------------------- From barry at python.org Wed Dec 17 09:45:40 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 09:45:48 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> References: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> Message-ID: <1071672339.27043.4.camel@anthem> On Tue, 2003-12-16 at 20:16, Guido van Rossum wrote: > Nah. But I liked the suggestion seen here earlier of allowing > parenthesis: > > from OPAL.Modules.Financial.Reports.COGS import (generate, webOptions, > normalize, row_ops, summary) +1 -Barry From jepler at unpythonic.net Wed Dec 17 10:14:50 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Dec 17 10:15:01 2003 Subject: [Python-Dev] bug in python arm-linux?: start_new_thread fails after popen In-Reply-To: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> Message-ID: <20031217151450.GB28436@unpythonic.net> On Wed, Dec 17, 2003 at 03:01:00PM +0100, Michael Lauer wrote: > Hi, I have a python program which works fine on x86 but doesn't work on > any of my arm-linux devices (Zaurus, Ipaq, SIMpad) - all of them are > running Python 2.3.2 on top of glibc 2.3.2+linuxthreads on top of kernel > 2.4.18-rmk6-pxa3 respectively kernel 2.4.19-rmk7. Using threads and fork together seems to be a big smelly armpit in Python. There are also problems on redhat9, where signals in a fork+exec'd subprocess are blocked, for instance. This seemed to be a consequence of blocking all signals in thread_pthread.h:PyThread_start_new_thread(). Perhaps pthread_atfork() could be used to fix this problem, though I know next to nothing about pthreads beyond the documentation I glanced at back when I first became aware of my thread+fork problem and before writing this message. Jeff From ncoghlan at iinet.net.au Wed Dec 17 10:22:05 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Dec 17 10:22:18 2003 Subject: [Python-Dev] Relative import In-Reply-To: References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> Message-ID: <3FE0749D.2030705@iinet.net.au> John J Lee wrote: > I can see that people not knowing how to spell "hierarchy" might be a > problem here Heh, good point. I always get confused by the place of the 'heir' in the 'hierarchy'. I had a seriously hard time coming up with a one word name that suggested the right semantics for that category. If the idea gets any interest, it'd be nice to find a better word for that case. I was tempted to use __ambiguous__, though :) Hmm, __scan__ could be an option. The other two are fairly specific about where to look, but category 'C' requests a scan of the package heirarchy. A: from __absolute__.dotted.name import foo B: from __relative__.dotted.name import bar C: from __scan__.dotted.name import foobar D: from dotted.name import barfoo I suspect the 'as' clause would be heavily used with this construct, but it wouldn't be compulsory - as far as module referencing in the code goes, I don't think there's anything illegal about the special names. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From Boris.Boutillier at arteris.net Wed Dec 17 11:13:19 2003 From: Boris.Boutillier at arteris.net (Boris Boutillier) Date: Wed Dec 17 11:13:30 2003 Subject: [Python-Dev] Relative import In-Reply-To: <3FE052EC.90705@iinet.net.au> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> Message-ID: <3FE0809F.2020702@arteris.net> Just to give you a practical example, we are using python for internal tools in the electronic Company I'm working for. I needed to have a non ambiguous file lookup when using import ( We set external dependencies for the flow when opening a file, so you can't use a 'if this file exist' check, when opening a file, it must always exists) I made a special import function, that was looking for a prefix to the package name: import loc.dotted.name import glb.dotted.name import __.dotted.name import std.dotted.name import xxx.dotted.name -> raise an error "loc", look in the current directory of the module (looking to his __file__ attribute) "__ "look to the parent directory "glb" look at the root of the user working repository ( a builtins set before anything else is done , users don't call python they the program myPython ) "std" look outside the database ( in the python installation dir mainly) and set back the old import behaviour to ensure that import inside standard modules don't fail ) I know the loc,glb __ and std names are no good for a general case as modules can have these names, but this is just to give you an actual use of these relatives imports. Boris Nick Coghlan wrote: > Greg Ewing wrote: > >> So I think we really want *three* kinds of module reference: >> >> A: Explicitly absolute >> B: Explicitly relative to the current module >> C: Searched for upwards in the package hierarchy from the current >> module >> >> (Note that C is a generalisation of the current "ambiguous" >> references which only look in two places.) >> > > Alternate spellings (plus category D, which is "Python 2.3 semantics"): > > A: from __absolute__.dotted.name import foo > B: from __relative__.dotted.name import bar > C: from __heirarchy__.dotted.name import foobar > D: from dotted.name import barfoo > > I believe this spelling would only require import to handle the > special cases as the first 'package' name (although having the > compiler recognise them may be a later optimisation), and should also > work with straight "import __absolute__.dotted.name.foo" statements. > > Then Guido's migration plan would apply to the progression of the D > semantics from the Python 2.3 model to being a shorthand for the > absolute semantics. > > And once _that_ has happened, then we could get rid of the > '__absolute__' version in the name of TOOWTDI. > > Regards, > Nick. > From skip at pobox.com Wed Dec 17 11:14:03 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 17 11:14:03 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> Message-ID: <16352.32971.627159.669571@montanaro.dyndns.org> >> Is funny punctuation really the right way to spell an important >> distinction like this? Tim> That's what I've been wondering. I really don't like the proposal Tim> to use . in various places to signify relative and absolute Tim> imports. It seems non-obvious both in understandability and Tim> readability. Agreed. Let's shoot for something more Pythonic. I can imagine having to count periods in import statements in much the way I used to have to count closing parens in Lisp code on a CDC Cyber (way before I had ever heard of language-sensitive editing). Skip From pb at nexus.co.uk Wed Dec 17 11:39:34 2003 From: pb at nexus.co.uk (Phil Blundell) Date: Wed Dec 17 11:39:42 2003 Subject: [Python-Dev] Re: bug in python arm-linux?: start_new_thread fails after popen In-Reply-To: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> Message-ID: <20031217163934.GA13660@nexus.co.uk> on Wed, Dec 17, 2003 at 03:01:00PM +0100, Michael Lauer wrote: > This program - as is - just hangs in the first start_new_thread() and > never comes back. > If you outcomment the first call to createPipe() out, then it works > fine. I can't reproduce this problem on my Debian/ARM machine. I'll try it on the iPAQ under Familiar later. My guess is that your glibc is too old. p. From theller at python.net Wed Dec 17 11:42:14 2003 From: theller at python.net (Thomas Heller) Date: Wed Dec 17 11:42:33 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() In-Reply-To: (Jack Jansen's message of "Wed, 17 Dec 2003 13:51:12 +0100") References: <002301c3c29c$f2753480$0201010a@kret> Message-ID: Jack Jansen writes: > On 17-dec-03, at 12:32, Thomas Heller wrote: >> I would rather like a function like Py_InitializeEx(struct _init*), >> where struct _init would have components like PYTHONPATH, optimize, and >> so on. > > You'd have to be very careful to design the struct in such a way that > you'd have good compatibility. Sure, but it can be done. The alternative would be a Py_InitializeN() call with many parameters. Would that be preferrable? Thomas From charleshixsn at earthlink.net Wed Dec 17 11:55:40 2003 From: charleshixsn at earthlink.net (Charles Hixson) Date: Wed Dec 17 11:59:54 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071509846.970.105.camel@anthem> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> <1071509846.970.105.camel@anthem> Message-ID: <3FE08A8C.305@earthlink.net> Barry Warsaw wrote: >On Mon, 2003-12-15 at 11:52, Skip Montanaro wrote: > > > ... > >Sorry, let me rephrase. I'd love to ban relative imports. Personally, >I think all imports should be specified using fully-qualified absolute >package paths. > >-Barry > > If I'm understanding this right you want to ban imports that are machine independant. That can't be right, but that's what it sounds like to me. To me an "absolute import" or a "fully-qualified absolute package path" sounds like you know the exact directory that the import is occuring from. E.g.: /home/someone/projects/python/workingprogress/currentproject/thisone.py I can see the use for being able to import that, but usually I would prefer to import either from something relative to the current directory, or relative to the python site-lib, or some such. It's also quite nice to not need to bother to exactly specify just which of those I will find it in. (This is what the pythonpath is for.) Now to me an import relative to the current directory is a relative import, and I assume that this is also what is meant by a local import. By allowing files to cluster relative to the code being written one is able to keep related projects and libraries together rather than just having all the code spread out aimlessly in a huge flat directory. I don't see in what what this is inferior. Since you didn't say why you wanted to ban relative imports, I don't understand your reasons. I can't even guess at them. Even if it were for some efficiency condern about hierarchical directories, that would merely say that the compiled modules should be stuffed into a library folder. (Well, more than one...the user needs to have write permission to the libraries that he creates, but it's just as well that he not have write permissions to the system libraries...without taking special steps.) From guido at python.org Wed Dec 17 12:21:57 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 12:22:02 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 01:22:05 +1000." <3FE0749D.2030705@iinet.net.au> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au> Message-ID: <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> It's interesting that the "scan upwards on the package name" gets so much support. Back when packages were first introduced, there was a period of time when they were a user-level construct, implemented by a module named "ni" (after the Knights who say "Ni!"). This implemented a full scan upward as the only way. When I reimplemented packages in C, I decided that the scan wasn't used much, and could actually be detrimental (I feared that too many global modules would be accidentally hidden), so I changed it to only looking in only two places: the current package and the toplevel (sys.path). But at the time it was a given that the import syntax would not be changed. Now that we're entertaining new import syntax, we can do better. How about this for Python 3.0: import P.Q (1) from P.Q import X (2) would only look for P in sys.path (I'll call this "absolute import"); and from ...P.Q import X (3) would scan for P upwards from the current package. It would require exactly three dots. P.Q of course stands for any dotted name. Python 2.x would continue to use the current ("ambiguous import") semantics for (1) and (2), but would also support (3) with the Python 3.0 semantics. It might also be possible to set a "no ambiguous imports" flag to switch to 3.0 semantics for (1) and (2). This might be a __future__ statement in the package's __init__.py, e.g. from __future__ import absolute_import I think this addresses all the concerns I've seen brought up in this thread so far against my original proposal and the variations that were subsequently suggested. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 17 12:26:13 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 12:26:23 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: Your message of "Wed, 17 Dec 2003 06:19:51 PST." <1071670791.3fe0660761aa6@mcherm.com> References: <1071670791.3fe0660761aa6@mcherm.com> Message-ID: <200312171726.hBHHQDq22544@c-24-5-183-134.client.comcast.net> [Michael Chermside] > I disagree. Right now, having access to a class object basically > gives one the ability to create new objects of that type. I > think that's just fine... and I don't mind applying it to the > file object. I'd think that the thing to do with untrusted code > is to deny it access to the 'file' type object, thus denying it > the ability to create new 'file's directly. > > After all, it's not like file has lots of useful classmethods... > client code which is not supposed to create new files has no > particular need to access the 'file' object. Oh yes, except > that it may have INSTANCES of 'file' and could access > f.__class__. But if it is RESTRICTED code, then the only > legitimate use of f.__class__ is to do typechecking (it's > arguable whether that is good design, but it does seem to be > a legitimate use), so for restricted code we return something > like this: > > class FakeFile: > def __eq__(self, other): > return other == file > > [...] Are you aware of the original issue, which is that as soon as you have a file *instance* (which might have been given to you by a very restrictive open() variant), you can always get to the file *class* using the __class__ attribute? Access to the __class__ attribute is useful for all sorts of reasons. > Guido writes: > > Re the capabilities discussion, this is Python 3.0 material if I ever > > saw some > > I agree. But I can dream, can't I? Yes, even for 3.0 this is still a dream... --Guido van Rossum (home page: http://www.python.org/~guido/) From lkcl at lkcl.net Wed Dec 17 12:56:32 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 12:56:08 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <3FE04CDD.1050109@iinet.net.au> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195856.GK17021@lkcl.net> <3FE04CDD.1050109@iinet.net.au> Message-ID: <20031217175632.GA2117@lkcl.net> > the two as I understand it. I must admit I am quite curious about the > fact that you seem to have plenty of time to _write_ about your > proposal, but no time to listen to some quite reasonable responses. nick, thank you for pointing this out. that i do not explicitly acknowledge other people's point is a common failing of mine that has frequently got me into significant communication difficulties. yes i _do_ appreciate and understand what people say, and take the time to read it: i don't necessarily always actually explicitly say that and i really appreciate you picking up on this. if anyone else who is responding on this thread feels offended because they do not believe that i am listening to what they have to say PLEASE SAY SO. thanks, l. From lkcl at lkcl.net Wed Dec 17 14:06:52 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 14:06:49 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <1071669397.3fe06095c9f3c@mcherm.com> References: <1071669397.3fe06095c9f3c@mcherm.com> Message-ID: <20031217190652.GE2117@lkcl.net> On Wed, Dec 17, 2003 at 05:56:37AM -0800, Michael Chermside wrote: > Luke: > > Seeing as you've now had time enough to read what I'd written, yes i have now, except not yet the link you recommended to 'E', which i will get to, soon. > I was going to > respond to some of the points that you made, but then I read Nick Coghlan's > posting[1], and I really don't have anything else to add. > > [1] http://mail.python.org/pipermail/python-dev/2003-December/041105.html am absorbing that: it looks like a good definition of acls and capabilities to me! > Well, that's not quite true... I guess I _do_ have something small to add. > In designing a security policy for file systems, there are very few things > that you can do with the protected items. Separate "read", "write", and > "execute" permissions seem appropriate. yes. it's worth pointing out that NT file systems have a lot more than just rwx (which are covered by "generic read" permission, "generic write" and "generic execute"). if i remember correctly, NT file system access permissions also include separate permissions for directory access and creation that are deliberately distinct from file creation (and maybe even file access, too, but it's been a while so i can't be sure). they also have "change security descriptor" as a permission. ... *click*. darn, it. darn it quite a lot. it's been such a long time that i've been confusing "security descriptor" with "access control list". although... thinking about it some more i don't think it makes much difference... definition of an NT (vax/vms) security descriptor: a security descriptor is a set of "access control lists". there are four ACLs, and they are for different purposes. i can nail down two of them for you: one is a "System" ACL which restricts what the OPERATING SYSTEM can do. another is a "Users" ACL which restricts what _users_ can do (and groups, which also under NT have their own identity just like a user has its own identity.) there is also some additional information on a per-acl basis and it controls the way in which the ACL is used. ... but all that is by-the-by because, i believe, only _one_ ACL is needed for the purposes at hand. anyway, sorry to interrupt the flow, there. :) > Furthermore, there tends to be a > clear idea of a "user", and in many situations we want all access to items > (files) to depend only on who that "user" is (sandboxing an email program > is an exception... are you listening MS?). Finally, we often want to > provide centralized administration of access rights to file systems. > ACLs are a good match for these design specs. fully designed, properly designed ACLs are, imo, way over the top for file systems, which is why POSIX ACLs are just an extension of the standard POSIX filesystem rwx/ugo. for NT (and VAX/VMS), the design of the security system came first, as a "generic thing", which was then applied to file systems as _one_ component, albeit a critical one, of the NT and VMS operating systems. the NT/VMS security model has room for 32-bits of "permissions" on an ACE (access control entry). 16-bit of those are "generic", and 16-bit of them are "application specific". the file-system-specific permissions were created by the writers of the NTFS - nt file system - and they gave them life and meaning, so to speak, but only in the context of an NTFS and access to it. > However, controlling what arbitrary programs may do is a very different > beast. in the context of something like "POSIX ACLs" or in the context of something like "POSIX file permissions", yes, absolutely, i agree with you entirely. in the context of something as flexible as NT/VMS security descriptors, no, i don't agree, because NT SDs are very generic and had to be "specialised" to fit onto the NTFS. which implies that something as flexible as NT/VMS SDs could equally and as easily be adapted to fit controlling what arbitrary programs do. _and_, in the NT operating system and its core [mostly user-space] components, this _has_ already been done [and i presume likewise in VMS]. for example, there is a spoolss (printer service) and there is a SAM (security accounts manager) database service. the authors / designers of these two services created their own set of 16-bit application-specific permissions, gave them life and meaning in the context of their individual services, and made sure that each function _in_ that service double-checked the permissions against the type of operation that that function performed. to give you an example, this is some pseudo-code of the SamrCreateUser function: static SEC_DES[] sec_des_array = { ... ... samr_create_user_security_descriptor, ... }; STATUS32 SamrCreateUser(pHND *context, UNICODE_STRING *username) { USER_TOK *user = GetUserToken(context); if (!check_security_descriptor(user, context, SECDES_SAM_CREATE_USER)) { return NT_STATUS_ACCESS_DENIED; } .... .... } the context handle stores references to quite a bit of state information about the connection (it's a DCE/RPC thing). the SECDES_SAM_CREATE_USER permission is an application-specific bit that is specifically and only used to check access permission to the SamrCreateUser function. > First of all, the kinds of restrictions that we want are HIGHLY > variable. Sometimes we want to prohibit all file access. Sometimes we > want to allow read-only access to a certain file only. Sometimes we > want access only to a certain directory. Sometimes we don't care at > all about file system access, we're concerned instead with providing > an object which can be queried but only in certain ways -- and the > specifics of what is permitted and what isn't are completely dependent > on the particular application. > > On top of that, there's rarely a clear concept of "users" that controls > what actions we want to permit. Instead, we usually want to allow or > deny certain capabilities to particular _sections of code_. (Sometimes > that will be a module, sometimes a plug-in, sometimes a child program.) yes. and i believe that the > All of these things suggest that maintaining a list of who [or rather "what" - i.e. "Entities", yes?] > is permitted > to access what is simply not the best model for providing security in > a programming language. i am curious: how does that logically follow? in my experience (windows nt services and therefore also samba-tng's services), access control lists have been an extremely successful means to provide security in almost every single program in the NT operating system (i don't mean XP i don't mean windows 95) for almost 15 to 20 years. it seems quite logical therefore to propose that such a successful security model be adapted to fit a programming language, or at least to learn from its success (and failings). change of subject ----------------- i believe that it _is_ possible to express what nick coghlan defines capabilities to be _into_ an ACL. how? by creating a security permission called "can you see it and execute it?" let's call it SEE_AND_EXECUTE then, the language (interpreter) does this (following the example of the SamrCreateUser function, above): MakeFunctionCall(Context *stack, CodeObject *code) { char *callee_fn_name = GetCalleeName(stack); SEC_DES *sd = GetSecurityDescriptorForCode(code); /* check the capability to even _see_ this function! */ if (!check_sec_des(sd, callee_fn_name, SEE_AND_EXECUTE)) { return Exception("Function name %s does not exist", code->function_name); } /* okay, they can see it [and execute it] */ .... .... } does that make sense at all? l. From exarkun at intarweb.us Wed Dec 17 14:17:52 2003 From: exarkun at intarweb.us (Jp Calderone) Date: Wed Dec 17 14:20:03 2003 Subject: [Python-Dev] Re: rexec.py unuseable In-Reply-To: <20031217190652.GE2117@lkcl.net> References: <1071669397.3fe06095c9f3c@mcherm.com> <20031217190652.GE2117@lkcl.net> Message-ID: <20031217191752.GA5281@intarweb.us> On Wed, Dec 17, 2003 at 07:06:52PM +0000, Luke Kenneth Casson Leighton wrote: > [snip] > > i believe that it _is_ possible to express what nick coghlan > defines capabilities to be _into_ an ACL. > > how? > > by creating a security permission called "can you see it and execute it?" > let's call it SEE_AND_EXECUTE > > then, the language (interpreter) does this (following the example of the > SamrCreateUser function, above): > > MakeFunctionCall(Context *stack, CodeObject *code) > { > char *callee_fn_name = GetCalleeName(stack); > SEC_DES *sd = GetSecurityDescriptorForCode(code); > > /* check the capability to even _see_ this function! */ > if (!check_sec_des(sd, callee_fn_name, SEE_AND_EXECUTE)) > { > return Exception("Function name %s does not exist", > code->function_name); > } > > /* okay, they can see it [and execute it] */ > > .... > .... > > } > > does that make sense at all? > What appears to be missing here is the ability for a piece of code to take the existing permissions which it does posess and create new permissions within them. With the system you propose, if a function `foo' has permission to call function `bar', how does foo delegate that permission to `foobar'? If it cannot, then it becomes very difficult to write well factored code which can also exist within the security framework. Jp -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20031217/ec981992/attachment.bin From lkcl at lkcl.net Wed Dec 17 14:40:32 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 14:40:05 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) Message-ID: <20031217194031.GF2117@lkcl.net> > Are you aware of the original issue, which is that as soon as you have > a file *instance* (which might have been given to you by a very > restrictive open() variant), you can always get to the file *class* > using the __class__ attribute? Access to the __class__ attribute is > useful for all sorts of reasons. that is why i am proposing that ACLs be added to every "Entity" in language - functions, modules, attributes etc. [even if the implementation is such that they aren't _actually_ every-object-in-python-has-another-damn-pointer] in this way, you would have one ACL on the open() function which restricts it to file-read-only, another ACL on the open.__class__ attribute which would be, for the sake of argument [("all functions", DENY, ABSOLUTELY_EVERYTHING)] etc. l. From lkcl at lkcl.net Wed Dec 17 14:58:46 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 14:58:38 2003 Subject: Access Control (was Re: [Python-Dev] rexec.py unuseable) In-Reply-To: <20031217190652.GE2117@lkcl.net> References: <1071669397.3fe06095c9f3c@mcherm.com> <20031217190652.GE2117@lkcl.net> Message-ID: <20031217195846.GG2117@lkcl.net> okay, to recap. jack jansen explains the difference between capabilities and ACLs: With capabilities you have an (unforgable) right to do something and no-one cares about your identity. with ACLs you have an unforgable identity which is checked against the ACL. nick coghlan gives a definition of a capability as: If you can access it, you have permission to use it. If you aren't allowed to use it, you won't even be able to see it. nick coghlan also gives a definition of an access control list as: "Entity X is allowed to perform action Y on other Entity Z". where an "Entity" may be an object, a function, a user, a module, an identity - anything - but in the context of the python language, it is presumed that "Entity" definitely NOT include "user" or anything remotely connected to the host operating system's concept of "identity". luke leighton defines "action Y" as sufficiently general to include "seeing" as well as "doing". [the reason for such is that the purpose of capabilities may then be perceived to be a subset of the purpose of ACLs.] nick coghlan describes capabilities as a "post-hoc check" which he then concludes that, by nature, the use of capabilities is therefore fast. mike chermside succinctly explains the need for fine-grained access control down to code-section level, but he disagrees with the necessity for having lists of access controls. luke leighton endeavours to justify his opinion that capabilities are appropriate for functional languages c but completely inappropriate for object-orientated languages like python (the function() and the __class__ thing). luke leighton gives a waffly explanation of the NT security descriptor concept and provides some example usage in NT services. he also gives some example ways by which NT SDs could be adapted for use by the python interpreter. guido goes agog and type-casts this discussion as definitely python 3.0-esque. l. From lkcl at lkcl.net Wed Dec 17 15:34:43 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Wed Dec 17 15:34:18 2003 Subject: [Python-Dev] Re: rexec.py unuseable Message-ID: <20031217203443.GA3885@lkcl.net> > > > What appears to be missing here is the ability for a piece of code to take > the existing permissions which it does posess and create new permissions > within them. > > With the system you propose, if a function `foo' has permission to call > function `bar', how does foo delegate that permission to `foobar'? If it > cannot, then it becomes very difficult to write well factored code which can > also exist within the security framework. jp, i briefly touched on this in earlier messages, but have not expanded on it since. your question is therefore very useful. there should exist a permission called "sub-objects-get-a-copy-of-parent-permissions-at-sub-object-create-time" and i believe there also exists (in NT 5.0) _yet another_ permission that says "sub-objects-inherits-parent-permissions" to be honest, i don't know _all_ of the internal details of the NT 5.0 enhancements to security descriptors. if i recall correctly, the "gets-copy-of-permissions-on-create" isn't actually a separate permission but an ACE qualifier bit - a bit like the DENY and GRANT qualifications. in this way, permissions can be considered to be recursive (both at create time and at access-for-purpose-X time). the _old_ style of permissions, NT 4.0 and below, you had to MANUALLY add the permissions, recursively, to all sub-objects. and incidentally, that's one reason why you can only upgrade from an NT 4.0 NTFS to an NT 5.0 NTFS and not the other way round, because the security descriptors are jigged around and irreversibly converted. in that earlier message, i made a recommendation that the concept of "recursive" application of permissions NOT be considered for implementation (until at least the old nt4.0 style was implemented and well understood). but it _is_ important that the "sub-objects-get-a-copy-of-parent-permissions-at-sub-object-create-time" concept be implemented. otherwise, as you say, there's no way to delegate permissions. i'm assuming in the above description, perhaps incorrectly, that function 'foobar' is a member attribute of the function 'foo'? please clarify! l. From Jack.Jansen at cwi.nl Wed Dec 17 15:35:33 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Wed Dec 17 15:35:39 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() In-Reply-To: References: <002301c3c29c$f2753480$0201010a@kret> Message-ID: <902A3726-30D0-11D8-B275-000A27B19B96@cwi.nl> On 17-dec-03, at 17:42, Thomas Heller wrote: > Jack Jansen writes: > >> On 17-dec-03, at 12:32, Thomas Heller wrote: >>> I would rather like a function like Py_InitializeEx(struct _init*), >>> where struct _init would have components like PYTHONPATH, optimize, >>> and >>> so on. >> >> You'd have to be very careful to design the struct in such a way that >> you'd have good compatibility. > > Sure, but it can be done. > The alternative would be a Py_InitializeN() call with many parameters. > Would that be preferrable? I'm unsure. I'm almost tempted to say I'd like to pass in a char ** and use environ-style (we're not really worried about performance here). -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From Jack.Jansen at cwi.nl Wed Dec 17 15:40:30 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Wed Dec 17 15:40:38 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au> <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> Message-ID: <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> On 17-dec-03, at 18:21, Guido van Rossum wrote: > It's interesting that the "scan upwards on the package name" gets so > much support. I would actually be surprised if people who say they want "scan upward" actually want something else: they want to scan upward up to a limit. If you have a complex package with, say, three levels you may want to scan up through the levels of your own package. However, if your package is later incorporated in a third party package I can hardly imagine that you want to scan in there too. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From guido at python.org Wed Dec 17 16:00:39 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 16:00:46 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 21:40:30 +0100." <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au> <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> Message-ID: <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> [Guido] > > It's interesting that the "scan upwards on the package name" gets > > so much support. [Jack] > I would actually be surprised if people who say they want "scan > upward" actually want something else: they want to scan upward up to > a limit. There's a negation missing here, but I think I understand you. > If you have a complex package with, say, three levels you may want > to scan up through the levels of your own package. However, if your > package is later incorporated in a third party package I can hardly > imagine that you want to scan in there too. I think someone actually posted a use case: they were referring to a global 3rd party module from within a package, and then later they were moving the module into the package. For both to work without changes to the import statement the scan would have to go all the way up. If we can disregard this use case, I still like my original proposal better, because it is more explicit: you have to use the right number of dots, and (like Unix paths starting with ../ or ../../ etc.) it is still completely explicit what you are referring to (assuming the current package/directory is known). But I expect the ... version will work just as well in practice: in debugged code, the ... refers to a particular ancestor, so what's above that is irrelevant; and what's below that is typically only two or at most three levels deep -- I should think a group of cooperating package authors who can't keep a namespace of that size coordinated will have bigger problems... And the advantage of the ... version is that it's more conspicuous than a single dot (for the most common case), and you don't have to count dots. --Guido van Rossum (home page: http://www.python.org/~guido/) From neal at metaslash.com Wed Dec 17 16:09:29 2003 From: neal at metaslash.com (Neal Norwitz) Date: Wed Dec 17 16:09:34 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.304, 2.305 In-Reply-To: References: Message-ID: <20031217210929.GX13702@epoch.metaslash.com> On Wed, Dec 17, 2003 at 12:43:35PM -0800, rhettinger@users.sourceforge.net wrote: > > Guido grants a Christmas wish: > sorted() becomes a regular function instead of a classmethod. Would sort() be a better name? When I told someone who's done quite a bit of java recently about sorted, he asked if it returned a bool whether the list was sorted or not. :-) Neal From python at rcn.com Wed Dec 17 16:19:27 2003 From: python at rcn.com (Raymond Hettinger) Date: Wed Dec 17 16:19:49 2003 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.304, 2.305 In-Reply-To: <20031217210929.GX13702@epoch.metaslash.com> Message-ID: <011701c3c4e3$741ec620$f1bd958d@oemcomputer> [Raymond] > > Guido grants a Christmas wish: > > sorted() becomes a regular function instead of a classmethod. [Neal] > Would sort() be a better name? When I told someone who's done quite a > bit of java recently about sorted, he asked if it returned a bool > whether the list was sorted or not. :-) I think sorted() is better: * the context makes it clear that it's not a predicate * the name helps distinguish it from list.sort() * it fits with reversed() * looking at actual code using sorted(), it reads well my two cents, Raymond Hettinger From ws-news at gmx.at Wed Dec 17 16:35:39 2003 From: ws-news at gmx.at (Werner Schiendl) Date: Wed Dec 17 16:35:49 2003 Subject: [Python-Dev] Re: Relative import References: <200312170109.hBH19DG16699@oma.cosc.canterbury.ac.nz> <200312170116.hBH1GId21276@c-24-5-183-134.client.comcast.net> Message-ID: Hi, "Guido van Rossum" wrote > Nah. But I liked the suggestion seen here earlier of allowing > parenthesis: > > from OPAL.Modules.Financial.Reports.COGS import (generate, webOptions, > normalize, row_ops, summary) > +1 / Werner From ws-news at gmx.at Wed Dec 17 16:53:19 2003 From: ws-news at gmx.at (Werner Schiendl) Date: Wed Dec 17 16:53:27 2003 Subject: [Python-Dev] Re: Relative import References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz><3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au><200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> Message-ID: "Guido van Rossum" wrote > > I think someone actually posted a use case: they were referring to a > global 3rd party module from within a package, and then later they > were moving the module into the package. For both to work without > changes to the import statement the scan would have to go all the way > up. > Why should this sort of thing work "automatically"? When I restructure a package's modules, I don't usually assume that all imports will continue to work unchanged. The main use case IMHO is to move an entire package (or self contained module) in a package to achive some sort of required structuring or, more important, resolve naming conflicts. This includes adding or re-naming some organizational package names do to company merging or aquisitions. > If we can disregard this use case, I still like my original proposal > better, because it is more explicit: you have to use the right > number of dots, and (like Unix paths starting with ../ or ../../ etc.) > it is still completely explicit what you are referring to (assuming > the current package/directory is known). > +1 I like your original proposal more than this one. There should be some solution about from-less import however, to support the fully-qualified name syntax. For example, I prefer writing "struct.pack" over writing just "pack" because it is more explicit to the reader what happens. Or e. g. os.path.split is another example. Maybe just omit the dots for the name? > But I expect the ... version will work just as well in practice: in > debugged code, the ... refers to a particular ancestor, so > what's above that is irrelevant; and what's below that is typically > only two or at most three levels deep -- I should think a group of > cooperating package authors who can't keep a namespace of that size > coordinated will have bigger problems... I'd prefer to explicitly specify the number of steps "up" the hierarchy. - It prevents accidentially taking a wrong module (e. g. a "global" one) because a module is missing. - It prevents changing semantics because someone (else) adds a module with that name on some other level of the hierarchy. - Explicit is better than implicit ;-) > > And the advantage of the ... version is that it's more conspicuous > than a single dot (for the most common case), and you don't have to > count dots. > If there are more than 2 (or maybe 3) dots in the original proposal's use, I think something is wrong with package structure - not with the syntax. And you are normally importing modules from your own package with relative imports, so there should hardly be a need to "count the dots" I think. / Werner From greg at cosc.canterbury.ac.nz Wed Dec 17 17:36:36 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 17:36:42 2003 Subject: [Python-Dev] Relative import In-Reply-To: <3FE052EC.90705@iinet.net.au> Message-ID: <200312172236.hBHMaaw25148@oma.cosc.canterbury.ac.nz> Nick Coghlan : > A: from __absolute__.dotted.name import foo > B: from __relative__.dotted.name import bar > C: from __heirarchy__.dotted.name import foobar > D: from dotted.name import barfoo > > Then Guido's migration plan would apply to the progression of the D > semantics from the Python 2.3 model to being a shorthand for the > absolute semantics. > > And once _that_ has happened, then we could get rid of the > '__absolute__' version in the name of TOOWTDI. That's where I disagree - I think that type C semantics should be the default/least ugly/easiest to get, not type A. But I wouldn't like any of the others to be as ugly as those, either. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From edcjones at erols.com Wed Dec 17 17:41:20 2003 From: edcjones at erols.com (Edward C. Jones) Date: Wed Dec 17 17:45:37 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning Message-ID: <3FE0DB90.9060604@erols.com> I use Gentoo 1.4 Linux with gcc 3.2.2. The C program given below gives the following compiler message: silly.c: In function `silly': silly.c:5: warning: suggest explicit braces to avoid ambiguous `else' I doubt if there is any bug in Py_DECREF. But would it be possible to jiggle the Py_DECREF macros to suppress this message? In my real code, I, of course, just added braces. ----------------------- #include static PyObject* silly(PyObject* obj, PyObject *args) { if (1) Py_DECREF(Py_None); return NULL; } static PyMethodDef silly_Methods[] = { {"silly", silly, METH_VARARGS, ""}, {NULL, NULL, 0, NULL} /* sentinel */ }; void initsilly(void) { (void) Py_InitModule("silly", silly_Methods); } ----------------------- From greg at cosc.canterbury.ac.nz Wed Dec 17 17:50:33 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 17:50:37 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <1071670791.3fe0660761aa6@mcherm.com> Message-ID: <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> Michael Chermside : > Greg: > > In light of this, it may have been a serious mistake to > > unify the 'file' type and the 'open' function. Is it too > > late to back out of that decision? > > Guido: > > I think so. > > I disagree. Right now, having access to a class object basically > gives one the ability to create new objects of that type. I > think that's just fine... and I don't mind applying it to the > file object. I'd think that the thing to do with untrusted code > is to deny it access to the 'file' type object, thus denying it > the ability to create new 'file's directly. It would be a lot better if we could get away from the idea of a "restricted mode" in the sense of a flag somewhere that a bunch of things have to take notice of in order to behave securely, because that model of security is prone to springing leaks -- as happened in a big way when new-style classes were introduced. The spirit behind my suggestion was to start thinking about ways in which functionality could be separated out so that this kind of special-casing for security purposes isn't needed. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From jjl at pobox.com Wed Dec 17 18:08:17 2003 From: jjl at pobox.com (John J Lee) Date: Wed Dec 17 18:08:58 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz><3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au><200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> Message-ID: On Wed, 17 Dec 2003, Werner Schiendl wrote: [...] > There should be some solution about from-less import however, to support the > fully-qualified name syntax. For example, I prefer writing "struct.pack" > over writing just "pack" because it is more explicit to the reader what > happens. Or e. g. os.path.split is another example. Yes. > Maybe just omit the dots for the name? As Guido pointed out, it would be a shame to break the rule that 'import x' currently lets you refer to plain old 'x'. And it would be ugly to have '.x' appear in programs, I think (and Guido mentioned with statements, though maybe he's kidding about that). So, why not this? from ... import x John From greg at cosc.canterbury.ac.nz Wed Dec 17 18:24:12 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 18:24:24 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> Message-ID: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> > import P.Q (1) > from P.Q import X (2) > > would only look for P in sys.path (I'll call this "absolute import"); > and > > from ...P.Q import X (3) > > would scan for P upwards from the current package. > > I think this addresses all the concerns I've seen brought up in this > thread so far No, it doesn't address my concern that scan-upwards semantics should be the default, in the sense of being what should be used in the absence of a reason to do otherwise, and should therefore have the most straightforward syntax. Also, you don't seem to have anything for the explicitly-relative case, which might not be strictly necessary, but I think I'd feel more comfortable if it were available somehow. Thinking about the double_underscore names, it might not be so bad if they were a bit shorter, e.g. A: __root__.P.Q B: __here__.P.Q __here__.__parent__.P.Q C: P.Q Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From jepler at unpythonic.net Wed Dec 17 18:31:26 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Dec 17 18:31:19 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031217151450.GB28436@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> Message-ID: <20031217233126.GA4692@unpythonic.net> On Wed, Dec 17, 2003 at 09:14:50AM -0600, Jeff Epler wrote: > There are also problems on redhat9, where signals in a fork+exec'd > subprocess are blocked, for instance. This seemed to be a consequence > of blocking all signals in thread_pthread.h:PyThread_start_new_thread(). To follow up on my own message, here is a program that demonstrates the problem I ran into with threads and signals. doit() uses system() to create a process that should kill itself with a signal nearly instantly, otherwise it sleeps for one second. If doit() is called from the main thread or before any threads are created, it prints a time well under 1 second. If it is run from a thread, it takes just over a second, because the delivery of the signal in the subprocess is blocked. Typical output: $ python thread-signal-problem.py not threaded Elapsed time 0.00420594215393 subthread Elapsed time 1.00974297523 main thread Elapsed time 0.00419104099274 import thread, time, os def doit(): t = time.time() os.system("kill $$; sleep 1") t1 = time.time() print "Elapsed time", t1-t print "not threaded" doit() print "subthread" thread.start_new(doit, ()) time.sleep(2) print "main thread" doit() From tdelaney at avaya.com Wed Dec 17 18:39:48 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Wed Dec 17 18:39:54 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D23@au3010avexu1.global.avaya.com> > From: Edward C. Jones > > I use Gentoo 1.4 Linux with gcc 3.2.2. The C program given > below gives > the following compiler message: > > silly.c: In function `silly': > silly.c:5: warning: suggest explicit braces to avoid ambiguous `else' > > ----------------------- > #include > > static PyObject* silly(PyObject* obj, PyObject *args) > { > if (1) > Py_DECREF(Py_None); If I were you, I would pay attention to the warning. > I, of course, just added braces. If you mean you changed it to: static PyObject* silly(PyObject* obj, PyObject *args) { if (1) { Py_DECREF(Py_None); } then I would say that's the right thing to do - all the time, in all code. Tim Delaney From skip at pobox.com Wed Dec 17 18:46:36 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 17 18:47:14 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D23@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D23@au3010avexu1.global.avaya.com> Message-ID: <16352.60124.324511.476586@montanaro.dyndns.org> Tim> If you mean you changed it to: Tim> static PyObject* silly(PyObject* obj, PyObject *args) Tim> { Tim> if (1) Tim> { Tim> Py_DECREF(Py_None); Tim> } Tim> then I would say that's the right thing to do - all the time, in Tim> all code. Then you also agree we should bracify the Py_*REF macros, right? Skip From klm at zope.com Wed Dec 17 18:57:23 2003 From: klm at zope.com (Ken Manheimer) Date: Wed Dec 17 19:04:22 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> Message-ID: On Thu, 18 Dec 2003, Greg Ewing wrote: > Thinking about the double_underscore names, it might not > be so bad if they were a bit shorter, e.g. > > A: __root__.P.Q > B: __here__.P.Q > __here__.__parent__.P.Q > C: P.Q Why separate __here__ and __parent__? (I may just be confused, here.) Aren't both about getting the relative package container? Discussing schemes for explicit relative and absolute imports with some colleagues a few days ago, we liked '__pkg__' for the package containing the current module - and __pkg__.__pkg__ to mean its' container (and so on, for those non-programmers out there:-). Someone also wanted explicit expression of absolute, which we thought might be '__python__'. A: __python__.P.Q B: __pkg__.P.Q __pkg__.__pkg__.P.Q C: P.Q That said, *i* much prefer the leading '.' scheme - i think the underscores are distinctly *not* easier to read, and i'm not bothered, anyway, by the "punctuality" of the leading-'.' - i think it stands out enough to be noticable, and the fractured-path implication of it conveys the relativeness... Ken klm@zope.com From guido at python.org Wed Dec 17 19:12:09 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 19:12:16 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: Your message of "Thu, 18 Dec 2003 11:50:33 +1300." <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> References: <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> Message-ID: <200312180012.hBI0C9E23257@c-24-5-183-134.client.comcast.net> > It would be a lot better if we could get away from the idea > of a "restricted mode" in the sense of a flag somewhere that > a bunch of things have to take notice of in order to behave > securely, because that model of security is prone to springing > leaks -- as happened in a big way when new-style classes were > introduced. Right. Restricted mode currently uses both paradigms: you only have access to the builtins that are given to you in the __builtins__ dict -- this is pure capability stuff, and IMO it works well -- and some builtin operations behave differently when you're in restricted mode -- this is the ACL stuff, and Samuele revealed serious holes in it. > The spirit behind my suggestion was to start thinking about > ways in which functionality could be separated out so that > this kind of special-casing for security purposes isn't > needed. Right. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 17 19:20:07 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 19:20:14 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 12:24:12 +1300." <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> References: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> Message-ID: <200312180020.hBI0K7T23311@c-24-5-183-134.client.comcast.net> > > I think this addresses all the concerns I've seen brought up in this > > thread so far > > No, it doesn't address my concern that scan-upwards semantics > should be the default, in the sense of being what should be > used in the absence of a reason to do otherwise, and should > therefore have the most straightforward syntax. It doesn't address that becaseu I don't want that to be the default. I'm considering adding syntax for relative imports so that the default import syntax can be absolute only, rather than ambiguous. IMO scan-upwards as the default is worse than ambiguous. > Also, you don't seem to have anything for the explicitly-relative > case, which might not be strictly necessary, but I think > I'd feel more comfortable if it were available somehow. That's why I originally proposed single, double, triple (etc.) leading dots. > Thinking about the double_underscore names, it might not > be so bad if they were a bit shorter, e.g. I *really* don't like using double-underscore names here unless it was for extremely uncommon cases. There are two proposals that I can live with: my original proposal with the leading dots counting the number of levels up, or the triple-dot proposal with scan-up semantics. In both cases, the default semantics would switch to absolute for Python 3.0. I cannot accept scan-up as default, and I don't think we can come up with two separate clearly distinguishable non-default syntaxes to separate scan-up and explicit relative semantics, so you have to pick one there. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Wed Dec 17 19:24:02 2003 From: tim.one at comcast.net (Tim Peters) Date: Wed Dec 17 19:24:03 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <3FE0DB90.9060604@erols.com> Message-ID: [Edward C. Jones] > I use Gentoo 1.4 Linux with gcc 3.2.2. The C program given below gives > the following compiler message: > > silly.c: In function `silly': > silly.c:5: warning: suggest explicit braces to avoid ambiguous `else' > > I doubt if there is any bug in Py_DECREF. But would it be possible to > jiggle the Py_DECREF macros to suppress this message? In my real code, > I, of course, just added braces. Sounds good to me (i.e., add the braces and be done with it). It's a silly warning because Py_DECREF expands to a two-branch if/else, and there is no ambiguity in reality. It would be a refreshing change if someone suggested that gcc change instead . [Timothy Delany] > If you mean you changed it to: > > static PyObject* silly(PyObject* obj, PyObject *args) > { > if (1) > { > Py_DECREF(Py_None); > } > > then I would say that's the right thing to do - all the > time, in all code. [Skip Montanaro] > Then you also agree we should bracify the Py_*REF macros, right? > That can't work, alas -- the macro has no idea whether curly braces are syntactically legal in the context it's invoked in. Timmy *can* know that, and insert them or not accordingly. This is what leads to the bletcherous do { the real macro guts go here } while(0) form of macro definition. Py_DECREF could be rewritten in that form -- but I've puked enough for one day . From guido at python.org Wed Dec 17 19:24:13 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 19:24:20 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 23:08:17 GMT." References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz><3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au><200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> Message-ID: <200312180024.hBI0ODP23337@c-24-5-183-134.client.comcast.net> > So, why not this? > > from ... import x Yes, I forgot about that. My original proposal allowed using just one or more dots; the three-dots proposal with scan-up semantics should also allow just that. In terms of the Grammar file: dotted_name: NAME ('.' NAME)* | '...' [NAME ('.' NAME)*] BTW, it seems that for breaking up long imports, the parentheses proposal has received universal acclaim. So if we add either of the dotted-from proposals to 2.4, we should also add the parenthesized syntax: from ...... import (......), and for symmetry also without from. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Wed Dec 17 19:34:55 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 17 19:34:53 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: References: <3FE0DB90.9060604@erols.com> Message-ID: <16352.63023.993326.798113@montanaro.dyndns.org> Tim> [Skip Montanaro] >> Then you also agree we should bracify the Py_*REF macros, right? >> Tim> That can't work, alas -- the macro has no idea whether curly braces Tim> are syntactically legal in the context it's invoked in. I meant the Py_*REF macros which expand to "if" statements and you'd bracify the clauses, e.g., Py_DECREF would become: #define Py_DECREF(op) \ if (_Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA \ --(op)->ob_refcnt != 0) { \ _Py_CHECK_REFCNT(op) \ } else { \ _Py_Dealloc((PyObject *)(op)) \ } Py_INCREF would be left alone, and the X variants would become #define Py_XINCREF(op) if ((op) == NULL) {;} else { Py_INCREF(op) } #define Py_XDECREF(op) if ((op) == NULL) {;} else {Py_DECREF(op) } Skip From zack at codesourcery.com Wed Dec 17 19:36:21 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Wed Dec 17 19:36:41 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: (Tim Peters's message of "Wed, 17 Dec 2003 19:24:02 -0500") References: Message-ID: <87y8tbezcq.fsf@codesourcery.com> "Tim Peters" writes: > This is what leads to the bletcherous > > do { > the real macro guts go here > } while(0) > > form of macro definition. Py_DECREF could be rewritten in that form -- but > I've puked enough for one day . Should, not just could. Yeah, it's bletcherous, but it's the only way to be sure. (Short of inline functions.) zw From guido at python.org Wed Dec 17 19:41:00 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 19:41:08 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 22:53:19 +0100." References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz><3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au><200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> Message-ID: <200312180041.hBI0f1G23377@c-24-5-183-134.client.comcast.net> > > I think someone actually posted a use case: they were referring to a > > global 3rd party module from within a package, and then later they > > were moving the module into the package. For both to work without > > changes to the import statement the scan would have to go all the way > > up. > > Why should this sort of thing work "automatically"? > When I restructure a package's modules, I don't usually assume that all > imports will continue to work unchanged. > > The main use case IMHO is to move an entire package (or self contained > module) in a package to achive some sort of required structuring or, more > important, resolve naming conflicts. This includes adding or re-naming some > organizational package names do to company merging or aquisitions. I agree that the use case is not very strong. There really seem to be two camps here; some people like scan-up semantics so much they want to make it the default, others hate it and want to banish even the bastardized version of it that is the current default ("ambiguous import" or perhaps we can call it "search current then global"). > > If we can disregard this use case, I still like my original proposal > > better, because it is more explicit: you have to use the right > > number of dots, and (like Unix paths starting with ../ or ../../ etc.) > > it is still completely explicit what you are referring to (assuming > > the current package/directory is known). > > > > +1 > > I like your original proposal more than this one. I like its preciseness, but I have to admit that the three dots are enticing. > There should be some solution about from-less import however, to support the > fully-qualified name syntax. For example, I prefer writing "struct.pack" > over writing just "pack" because it is more explicit to the reader what > happens. Or e. g. os.path.split is another example. > > Maybe just omit the dots for the name? from ... import os.path could be made to work (even though today, in a from-import statement, the name after the import keyword cannot have dots in it). > > But I expect the ... version will work just as well in practice: in > > debugged code, the ... refers to a particular ancestor, so > > what's above that is irrelevant; and what's below that is typically > > only two or at most three levels deep -- I should think a group of > > cooperating package authors who can't keep a namespace of that size > > coordinated will have bigger problems... > > I'd prefer to explicitly specify the number of steps "up" the hierarchy. > > - It prevents accidentially taking a wrong module (e. g. a "global" one) > because a module is missing. But that would case pretty obvious error message, wouldn't it? > - It prevents changing semantics because someone (else) adds a module with > that name on some other level of the hierarchy. > > - Explicit is better than implicit ;-) You know, I'm torn on this one. Maybe some more people can "vote" on their preferences. > > And the advantage of the ... version is that it's more conspicuous > > than a single dot (for the most common case), and you don't have to > > count dots. > > If there are more than 2 (or maybe 3) dots in the original proposal's use, I > think something is wrong > with package structure - not with the syntax. Right. :-) > And you are normally importing modules from your own package with relative > imports, so there should hardly be a need to "count the dots" I think. Of course. --Guido van Rossum (home page: http://www.python.org/~guido/) From tdelaney at avaya.com Wed Dec 17 20:09:43 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Wed Dec 17 20:09:49 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D79@au3010avexu1.global.avaya.com> > From: Skip Montanaro [mailto:skip@pobox.com] > > Tim> If you mean you changed it to: > > Tim> static PyObject* silly(PyObject* obj, PyObject *args) > Tim> { > Tim> if (1) > Tim> { > Tim> Py_DECREF(Py_None); > Tim> } > > Tim> then I would say that's the right thing to do - all > the time, in > Tim> all code. > > Then you also agree we should bracify the Py_*REF macros, > right? Maybe, maybe not. In this case, not bracifying the macros revealed what is IMO a poor coding practice. So you could argue that not having the braces there is a Good Thing(TM). However, I do prefer that macros be as self-contained as functions, since when reading them, they look exactly the same. The problem in the original case is that the `if` clause does not contain the braces that it should have - and as a result, can lead to ambiguous code (in terms of readability). My opinion is that no matter what code you are writing in C, blocks should be delimited appropriately. That way there's no possibility of confusion. Fortunately, I get to write the coding standards for my team, so I don't have to put up with anyone else's wrong ideas :) Tim Delaney From tdelaney at avaya.com Wed Dec 17 20:14:27 2003 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Wed Dec 17 20:14:33 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0D7E@au3010avexu1.global.avaya.com> > From: Skip Montanaro > > #define Py_DECREF(op) \ > if (_Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA \ > --(op)->ob_refcnt != 0) { \ > _Py_CHECK_REFCNT(op) \ > } else { \ > _Py_Dealloc((PyObject *)(op)) \ > } > > Py_INCREF would be left alone, and the X variants would become > > #define Py_XINCREF(op) if ((op) == NULL) {;} else { Py_INCREF(op) } > #define Py_XDECREF(op) if ((op) == NULL) {;} else {Py_DECREF(op) } Then you will probably end up with different warnings - semicolons following closing braces, etc. As Tim Peters said, the way to deal with that is to enclose the entire bit in a do ( ... } while (0) no-op (after the compiler optimises it away) but that - as he rightly pointed out - is pretty blecherous. #define Py_XINCREF(op) do { if ((op) != NULL) { Py_INCREF(op) } } while (0) #define Py_XDECREF(op) do { if ((op) != NULL) { Py_DECREF(op) } } while (0) or #define Py_XINCREF(op) do { if ((op) != NULL) Py_INCREF(op) } while (0) #define Py_XDECREF(op) do { if ((op) != NULL) Py_DECREF(op) } while (0) Tim Delaney From greg at cosc.canterbury.ac.nz Wed Dec 17 20:57:24 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 20:57:30 2003 Subject: [Python-Dev] Relative import In-Reply-To: <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> Message-ID: <200312180157.hBI1vOC25998@oma.cosc.canterbury.ac.nz> Jack Jansen : > If you have a complex package with, say, three levels you may want to > scan up through the levels of your own package. However, if your package > is later incorporated in a third party package I can hardly imagine that > you want to scan in there too. The main use I have in mind for scan-upwards is for when you're importing something that's *not* part of your package. It's probably top-level, but someone might want to put you and it into a package one day, so you shouldn't hard-wire a top-level reference to it. That's why I think scan-upwards should be the default -- because you never know when someone might want to repackage you into a different environment in the future. Referring to related modules in your package is a different matter -- for that, you should probably be using explicitly relative references. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Wed Dec 17 21:02:33 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 21:02:39 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 14:57:24 +1300." <200312180157.hBI1vOC25998@oma.cosc.canterbury.ac.nz> References: <200312180157.hBI1vOC25998@oma.cosc.canterbury.ac.nz> Message-ID: <200312180202.hBI22Xu23508@c-24-5-183-134.client.comcast.net> > The main use I have in mind for scan-upwards is for when you're > importing something that's *not* part of your package. It's probably > top-level, but someone might want to put you and it into a package one > day, so you shouldn't hard-wire a top-level reference to it. You could hardwire a relative reference to it though. > That's why I think scan-upwards should be the default -- because you > never know when someone might want to repackage you into a different > environment in the future. I don't know if I want to encourage this kind of repackaging. There are various downsides (such as possible deployment of two differently repackaged versions of the same code) and I think some of the burden should be placed on the repackager. After all, after repackaging, the repackager is really responsible for the repackaged code. The nice thing about absolute imports is that it's very simple to write a tool that changes all absolute imports of a given package into some other import, if you really want that. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Wed Dec 17 21:28:30 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:28:41 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <20031216141746.GA3145@burma.localdomain> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> Message-ID: <1071714510.27808.83.camel@anthem> On Tue, 2003-12-16 at 09:17, Gustavo Niemeyer wrote: > I have done this many times, so let me try to describe at least one > legitimate usage case. > > A couple of weeks ago I wrote a software which needs a third > party package to work. OTOH, since it's a small package, I don't > want to force the user to go after the package, even because I'd > have to ensure that my code always work with the available version > of that package. > > Thanks to the relative module importing mechanism, solving that is no > harder than copying the third party package into my own package > namespace. > > This idea could probably be expressed in some other way, hacking > sys.path or whatever, but I belive this is a fairly common pattern, > and I vote for introducing a scheme to differ between local/global > importing which would not break the current flexibility. I guess the difference is when you're writing an application vs. writing a library. When writing an application (like Zope or Mailman), we already manipulate sys.path to include some non-standard locations. Incorporating a third-party package into the application then is a simple matter of arranging to have its top level directory somewhere on sys.path. In Mailman, I do this with the email package, and various codecs. When writing a library, that's harder because you're imposing requirements on the downstream consumer of your library. As an application developer, I don't have much sympathy for relative imports. As a library developer, I can see the point. -Barry From barry at python.org Wed Dec 17 21:33:43 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:33:53 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> Message-ID: <1071714822.27808.88.camel@anthem> On Tue, 2003-12-16 at 13:26, Guido van Rossum wrote: > A lot people have presented a good case for relative imports. Nobody > has argued to keep the status quo (where imports are ambiguous about > whether they are meant to be absolute or relative). So I suggest that > in 2.4, we introduce the leading dot notation for relative import, > while still allowing relative import without a leading dot. In > 2.5 we can start warning about relative import without a leading dot > (although that will undoubtedly get complaints from folks who have > code that needs to work with 2.3 still). In 3.0 we can retire > ambiguous import. I'll just note that where the current status quo trips /me/ up most is when I accidentally have a local module with the same name as a global module, and then I write an import statement expecting to get the standard library module, but end up getting the local module. That's why when I tend to think about this, I start wanting a way to spell "definitely give me the global one, no matter what". IOW, I feel like I want a way to bypass relative module lookups. I'm still catching up on this thread, but I wanted to throw this out there... -Barry From barry at python.org Wed Dec 17 21:44:38 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:44:49 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312180041.hBI0f1G23377@c-24-5-183-134.client.comcast.net> References: <200312170022.hBH0Mne16539@oma.cosc.canterbury.ac.nz> <3FE052EC.90705@iinet.net.au> <3FE0749D.2030705@iinet.net.au> <200312171721.hBHHLvq22523@c-24-5-183-134.client.comcast.net> <40E11434-30D1-11D8-B275-000A27B19B96@cwi.nl> <200312172100.hBHL0dq22885@c-24-5-183-134.client.comcast.net> <200312180041.hBI0f1G23377@c-24-5-183-134.client.comcast.net> Message-ID: <1071715478.27808.90.camel@anthem> On Wed, 2003-12-17 at 19:41, Guido van Rossum wrote: > You know, I'm torn on this one. Maybe some more people can "vote" on > their preferences. Don't we really need a PEP so we know what we're voting on? :) -Barry From barry at python.org Wed Dec 17 21:49:36 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:49:46 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180202.hBI22Xu23508@c-24-5-183-134.client.comcast.net> References: <200312180157.hBI1vOC25998@oma.cosc.canterbury.ac.nz> <200312180202.hBI22Xu23508@c-24-5-183-134.client.comcast.net> Message-ID: <1071715775.27808.95.camel@anthem> On Wed, 2003-12-17 at 21:02, Guido van Rossum wrote: > I don't know if I want to encourage this kind of repackaging. I concur. I'm sure it's just me, but supporting this repacking use case makes me very uncomfortable. It feels like I'm loosing a solid grounding about where and how Python finds modules named in import statements. -Barry From barry at python.org Wed Dec 17 21:52:03 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:52:10 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> References: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> Message-ID: <1071715922.27808.98.camel@anthem> On Thu, 2003-12-18 at 07:24, Greg Ewing wrote: > Thinking about the double_underscore names, it might not > be so bad if they were a bit shorter, e.g. > > A: __root__.P.Q > B: __here__.P.Q > __here__.__parent__.P.Q > C: P.Q I like this, although I'm not so sure about __here__.__parent__. What I like about this is the explicitness of where your search starts. -Barry From barry at python.org Wed Dec 17 21:53:17 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:53:26 2003 Subject: [Python-Dev] Relative import In-Reply-To: References: Message-ID: <1071715997.27808.100.camel@anthem> On Wed, 2003-12-17 at 18:57, Ken Manheimer wrote: > That said, *i* much prefer the leading '.' scheme - i think the > underscores are distinctly *not* easier to read, and i'm not bothered, > anyway, by the "punctuality" of the leading-'.' - i think it stands > out enough to be noticable, and the fractured-path implication of it > conveys the relativeness... Just to continue my contrarian ways, the leading-dot thingie was one of the things that always bugged me about zcml. :) -Barry From barry at python.org Wed Dec 17 21:59:47 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 21:59:57 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <3FDEDF30.1050904@iinet.net.au> References: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> <3FDEDF30.1050904@iinet.net.au> Message-ID: <1071716386.27808.104.camel@anthem> On Tue, 2003-12-16 at 05:32, Nick Coghlan wrote: > If absolute imports were to be the only type allowed, then it would seem > that the only possible location for naming conflicts is in the _first_ > element. True. > So if I wanted to use two different third party modules, both of which > have unfortunately chosen the same name for the top-level package, the > only way to let them co-exist is to redo _all_ of the imports in one or > the other of them. > > Whereas, if relative pathing is possible, I believe that all I have to > do is push them one level down in the package heirarchy (using distinct > names that I invent), and neither of them ever even knows about the > other's existence. I can get at both of them unambiguously, by using my > new top=level names, and neither package even knows that it is no longer > starting at the top of the import heirarchy. > > Or is there some other solution being proposed to this problem, and I > just haven't understood it? Has this ever happened to you in practice? It seems like the way out would be to start adopting a Java-like convention for package names. The problem with that in current Python is that you can't (easily) weave a package's contents from different locations in the file system. -Barry From barry at python.org Wed Dec 17 22:02:04 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 22:02:15 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <3FE08A8C.305@earthlink.net> References: <000201c3c289$976fcae0$e841fea9@oemcomputer> <1071500208.970.76.camel@anthem> <16349.53200.37613.119494@sftp.fdrake.net> <200312151603.hBFG3VT17409@c-24-5-183-134.client.comcast.net> <1071506413.970.102.camel@anthem> <16349.59071.26004.502201@montanaro.dyndns.org> <1071509846.970.105.camel@anthem> <3FE08A8C.305@earthlink.net> Message-ID: <1071716523.27808.107.camel@anthem> On Wed, 2003-12-17 at 11:55, Charles Hixson wrote: > Barry Warsaw wrote: > >Sorry, let me rephrase. I'd love to ban relative imports. Personally, > >I think all imports should be specified using fully-qualified absolute > >package paths. > > > If I'm understanding this right you want to ban imports that are machine > independant. That can't be right, but that's what it sounds like to me. Nope, because "absolute package paths" are always calculated relative to directories in sys.path when mapping a package path to a file system path. -Barry From andrew-pythondev at puzzling.org Wed Dec 17 22:04:09 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Wed Dec 17 22:04:15 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1071714822.27808.88.camel@anthem> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> Message-ID: <20031218030409.GG473@frobozz> On Wed, Dec 17, 2003 at 09:33:43PM -0500, Barry Warsaw wrote: > > I'll just note that where the current status quo trips /me/ up most is > when I accidentally have a local module with the same name as a global > module, and then I write an import statement expecting to get the > standard library module, but end up getting the local module. > > That's why when I tend to think about this, I start wanting a way to > spell "definitely give me the global one, no matter what". IOW, I feel > like I want a way to bypass relative module lookups. Alternatively, maybe what you want is a way to say "definitely give me the standard library one, no matter what", e.g. from stdlib import codecs This would allow other packages to have their own codecs module/package without worrying about confusing import errors. To make this backwards compatible, though, we'd probably need the equivalent of C++'s "using namespace" statement, with no declaration being equivalent to: using namespace stdlib Or perhaps more python-like: __import_base__ = 'stdlib' Python 3.0 might default to __import_base__ = '', to force people to be explicit. Actually, the proposed with statement could be used here, e.g. import stdlib with stdlib: from . import codecs, user, gc looks reasonably clear to me, and for long package paths, it could be very convenient: import zope.app.interfaces with zope.app.interfaces: from .location import ILocation from .foo import IFoo, IBar # etc -Andrew. From barry at python.org Wed Dec 17 22:05:03 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 22:05:12 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE0B9B@au3010avexu1.global.avaya.com> Message-ID: <1071716702.27808.111.camel@anthem> On Tue, 2003-12-16 at 22:20, Delaney, Timothy C (Timothy) wrote: > That's what I've been wondering. I really don't like the proposal to > use . in various places to signify relative and absolute imports. It > seems non-obvious both in understandability and readability. I agree. That's why I like something like __root__.foo.bar and __here__.foo.bar. Totally explicit and will avoid deep groove digging in the skull trying to remember if a leading dot means it should be absolute or relative. :) -Barry From tim.one at comcast.net Wed Dec 17 22:06:13 2003 From: tim.one at comcast.net (Tim Peters) Date: Wed Dec 17 22:06:14 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1071714822.27808.88.camel@anthem> Message-ID: [Barry] > ... > I'll just note that where the current status quo trips /me/ up most is > when I accidentally have a local module with the same name as a global > module, and then I write an import statement expecting to get the > standard library module, but end up getting the local module. Indeed, this is common on c.l.py and the Tutor lists, with a twist: someone writes a module in their *current* directory, with a name like random.py, and then "Python breaks mysteriously all over the place", because every import of random ends up with something utterly unexpected. For some reason, "random.py" seems most often to be the actual name used in these cases. Possibly related: I remember pissing away a day a year or so ago trying to track down a segfault in a Zope test that happened only when I ran the test on my McLean box. In the end, it turned out that the test in question was (after untangling the usual layers of Zopish indirection ) deliberately trying to import a module that didn't exist, with an oddball name. It just so happened that somebody had whittled down a segfaulting Python program and uploaded it to SourceForge, with the very same oddball name. I had downloaded that program into my Python build directory, and Zope kept provoking its segfault as a side effect of (unintentionally!) importing it. In large part, I've come to loathe imports -- I never have the foggiest idea how they're going to get satisfied in code I didn't write, and especially in code that plays every trick in (and out) of the book to hide what it's really doing with sys.path mutations, import hooks, poking stuff by magic into sys.modules via well-hidden side effects in scattered __init__.py files, ... it makes Java's rigid, wordy, explicit import scheme look very attractive some days. That you can't trick that into doing arbitrarily unexpected things by magic at runtime isn't entirely a bad thing . From barry at python.org Wed Dec 17 22:08:21 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 17 22:08:28 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031218030409.GG473@frobozz> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> Message-ID: <1071716900.27808.114.camel@anthem> On Wed, 2003-12-17 at 22:04, Andrew Bennetts wrote: > On Wed, Dec 17, 2003 at 09:33:43PM -0500, Barry Warsaw wrote: > > > > That's why when I tend to think about this, I start wanting a way to > > spell "definitely give me the global one, no matter what". IOW, I feel > > like I want a way to bypass relative module lookups. > > Alternatively, maybe what you want is a way to say "definitely give me the > standard library one, no matter what", e.g. > > from stdlib import codecs Interesting. I see the subtle distinction. One searches only the standard library, the other searches every directory on sys.path. I'm not sure I'd have much need to restrict the search to just the standard library. Plus, would that include site-packages? -Barry From andrew-pythondev at puzzling.org Wed Dec 17 22:16:45 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Wed Dec 17 22:16:51 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1071716900.27808.114.camel@anthem> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> Message-ID: <20031218031645.GH473@frobozz> On Wed, Dec 17, 2003 at 10:08:21PM -0500, Barry Warsaw wrote: > On Wed, 2003-12-17 at 22:04, Andrew Bennetts wrote: > > On Wed, Dec 17, 2003 at 09:33:43PM -0500, Barry Warsaw wrote: > > > > > > That's why when I tend to think about this, I start wanting a way to > > > spell "definitely give me the global one, no matter what". IOW, I feel > > > like I want a way to bypass relative module lookups. > > > > Alternatively, maybe what you want is a way to say "definitely give me the > > standard library one, no matter what", e.g. > > > > from stdlib import codecs > > Interesting. I see the subtle distinction. One searches only the > standard library, the other searches every directory on sys.path. I'm > not sure I'd have much need to restrict the search to just the standard > library. Plus, would that include site-packages? I should have been clearer. What I'm basically proposing is to make site-package called "stdlib", leaving a default python install with a very uncluttered top-level namespace, thus reducing unexpected name conflicts. Otherwise, this idea doesn't require any modification to import semantics, except for some magic to make stdlib the default root for imports for backwards compatibility. Thinking about it a bit harder, though, I'm not sure it's as easy as I initially thought, because this would then make importing 3rd party packages not work, i.e. currently you can do: import urllib import pygtk without problems, because they are both top-level. If urllib moved into a "stdlib" package, it's hard to imagine a clean way where old code that does those two lines would still work. I still like the idea of having the standard library in its own package, but I guess that will have to wait until Python 3.0. -Andrew. From greg at cosc.canterbury.ac.nz Wed Dec 17 23:21:24 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 23:21:30 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180020.hBI0K7T23311@c-24-5-183-134.client.comcast.net> Message-ID: <200312180421.hBI4LNb27278@oma.cosc.canterbury.ac.nz> Guido: > I cannot accept scan-up as default In that case, it's probably better not to have it at all, and I'd go for your first proposal. One leading dot seems to stand out pretty well to my eyes -- more is not necessary. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim at multitalents.net Wed Dec 17 23:43:45 2003 From: tim at multitalents.net (Tim Rice) Date: Wed Dec 17 23:43:52 2003 Subject: [Python-Dev] 2.3.3 portability audit Message-ID: A few things I found in the release23-maint branch. python-2.3/src/Doc/tools/checkargs.pm:#!/uns/bin/perl probably a typo. ^^^ While most python scripts use #!/usr/bin/env python this one has #!/bin/env python-2.3/src/Lib/bsddb/dbshelve.py:#!/bin/env python Most scripts that don't use #!/usr/bin/env python have #! /usr/local/bin/python These 5 have #!/usr/bin/python python-2.3/src/Lib/idlelib/idle:#!/usr/bin/python python-2.3/src/Lib/idlelib/idle.py:#!/usr/bin/python python-2.3/src/Lib/test/test_bz2.py:#!/usr/bin/python python-2.3/src/Lib/test/test_optparse.py:#!/usr/bin/python python-2.3/src/Mac/Contrib/Tabcleaner/Tabcleaner.py:#!/usr/bin/python Intentional? python-2.3/src/Lib/test/test_largefile.py:#!python This may be intentional, I don't know. python-2.3/src/Lib/test/test_pep263.py:#! -*- coding: koi8-r -*- Is this ! supposed to be there? ^ I think not. (could be wrong) python-2.3/src/Doc/tools/makesec.sh:#!/bin/bash Bash doesn't come standard on all supported platforms. The only bashism I saw with a quick look was the last line. rm -f lib.{aux,log} -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From guido at python.org Wed Dec 17 23:48:08 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 23:48:20 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 17:21:24 +1300." <200312180421.hBI4LNb27278@oma.cosc.canterbury.ac.nz> References: <200312180421.hBI4LNb27278@oma.cosc.canterbury.ac.nz> Message-ID: <200312180448.hBI4m8J23710@c-24-5-183-134.client.comcast.net> > > I cannot accept scan-up as default > > In that case, it's probably better not to have it at all, > and I'd go for your first proposal. One leading dot seems > to stand out pretty well to my eyes -- more is not necessary. But you're okay with two or more dots for extra levels up, right? (Note that this leading-dot algorithm is nearly the same as the old MacOS pathname syntax, except they did it with colons: :foo was foo in the current directory, ::foo was foo in the parent directory, etc., and foo:bar was an absolute pathname. The only difference is that for backwards compatibility reasons they interpreted no colons as relative to the current directory rather than absolute.) --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Wed Dec 17 23:53:18 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Dec 17 23:53:23 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180448.hBI4m8J23710@c-24-5-183-134.client.comcast.net> Message-ID: <200312180453.hBI4rIg27605@oma.cosc.canterbury.ac.nz> Guido: > But you're okay with two or more dots for extra levels up, right? Yes, that's fine. > (Note that this leading-dot algorithm is nearly the same as the old > MacOS pathname syntax I noticed that, too. Also I think VMS did something similar with dots in its pathnames. So there are precedents... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Wed Dec 17 23:55:03 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 17 23:55:13 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Wed, 17 Dec 2003 21:33:43 EST." <1071714822.27808.88.camel@anthem> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> Message-ID: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> > I'll just note that where the current status quo trips /me/ up most is > when I accidentally have a local module with the same name as a global > module, and then I write an import statement expecting to get the > standard library module, but end up getting the local module. This can be interpreted in two ways though. You could be unfortunate enough to create a module in your package that has the same name as a standard module you also need to import from inside the same package -- that's currently impossible. But the most common variant I've seen is what Tim Peters describes: having a module in the current directory that accidentally hides a standard module. The latter case does not involve packages and will not be fixed by any import syntax -- it can only be fixed by changing the default sys.path *not* to include the current directory, just like over the last decade or so Unix shells have stopped putting '.' in $PATH. But that has other downsides: it would be a lot harder to begin writing a program broken up into modules. Hmm, maybe the current directory should be last? That way you can import your own modules except if they clash with the standard library. Hm, but there are so many standard library modules that it might still cause frequent complaints from mystified beginners. I'm beginning to agree (uselessly) with Tim: import itself is a problem. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Thu Dec 18 00:04:37 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 00:04:43 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: <200312180504.hBI54ba27649@oma.cosc.canterbury.ac.nz> Guido: > Hmm, maybe the current directory should be last? That way you can > import your own modules except if they clash with the standard > library. Hm, but there are so many standard library modules that it > might still cause frequent complaints from mystified beginners. Maybe in Python 3.0 the current directory should appear as a package with some standard name. Then applications could use relative imports to import their own modules, and absolute ones to import standard modules, without fear of conflict. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From andrew-pythondev at puzzling.org Thu Dec 18 00:10:21 2003 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Thu Dec 18 00:10:31 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: <20031218051021.GI473@frobozz> On Wed, Dec 17, 2003 at 08:55:03PM -0800, Guido van Rossum wrote: [...] > -- that's currently impossible. But the most common variant I've seen > is what Tim Peters describes: having a module in the current directory > that accidentally hides a standard module. > > The latter case does not involve packages and will not be fixed by any > import syntax -- it can only be fixed by changing the default sys.path > *not* to include the current directory, just like over the last decade > or so Unix shells have stopped putting '.' in $PATH. Or it can be fixed by not having the entire standard library at the toplevel, but instead inside a single "stdlib" package. That way 3rd party packages could never conflict with the standard library (unless they were crazy enough to call one of their own modules "stdlib"...). This way, no matter how large the standard library grows, the potential for module name conflicts doesn't grow. It'd probably simplify things slightly, too -- the site-packages directory could go away entirely, and all packages, standard or not, would be on an equal footing. The standard library would be no more special than, say, Zope. My /usr/lib/python would just contain stdlib/ twisted/ zope/ Numeric/ etc. The backwards compatibility problems probably means this isn't feasible until Python 3.0, though :( -Andrew. From fdrake at acm.org Thu Dec 18 00:31:20 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 18 00:31:32 2003 Subject: [Python-Dev] 2.3.3 portability audit In-Reply-To: References: Message-ID: <16353.15272.514696.150572@sftp.fdrake.net> Tim Rice writes: > python-2.3/src/Doc/tools/checkargs.pm:#!/uns/bin/perl > probably a typo. ^^^ ... > python-2.3/src/Doc/tools/makesec.sh:#!/bin/bash > Bash doesn't come standard on all supported platforms. > The only bashism I saw with a quick look was the last line. > rm -f lib.{aux,log} I've fixed these two on the 2.3 maintenance branch and the trunk. Thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one at comcast.net Thu Dec 18 00:38:57 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Dec 18 00:38:59 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180448.hBI4m8J23710@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > ... > (Note that this leading-dot algorithm is nearly the same as the old > MacOS pathname syntax, except they did it with colons: :foo was foo in > the current directory, ::foo was foo in the parent directory, etc., > and foo:bar was an absolute pathname. The only difference is that for > backwards compatibility reasons they interpreted no colons as relative > to the current directory rather than absolute.) More relevant to most of the world is that it's even more similar to the pathname syntax in the Win9x/ME command.com shells: C:\Code\python\PCbuild>cd ...\Zope C:\Code\Zope> I expect that basing Python syntax on obsolete, hated OSes is a smart contrarian move > From barry at python.org Thu Dec 18 01:49:54 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 01:50:04 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: <1071730193.17717.4.camel@anthem> On Wed, 2003-12-17 at 23:55, Guido van Rossum wrote: > This can be interpreted in two ways though. You could be unfortunate > enough to create a module in your package that has the same name as a > standard module you also need to import from inside the same package > -- that's currently impossible. Right, and I'd like to make that possible. The least disruptive way I see of doing that is to introduce a magic root package from which things can be imported from. E.g. import __root__.re > But the most common variant I've seen > is what Tim Peters describes: having a module in the current directory > that accidentally hides a standard module. Yep, I've been bit by that too. > The latter case does not involve packages and will not be fixed by any > import syntax -- it can only be fixed by changing the default sys.path > *not* to include the current directory, just like over the last decade > or so Unix shells have stopped putting '.' in $PATH. Yep. I haven't had '.' in my $PATH for a bazillion years. :) > But that has other downsides: it would be a lot harder to begin > writing a program broken up into modules. Hmm, maybe the current > directory should be last? That way you can import your own modules > except if they clash with the standard library. Hm, but there are so > many standard library modules that it might still cause frequent > complaints from mystified beginners. > > I'm beginning to agree (uselessly) with Tim: import itself is a > problem. :-) There ya go! That's thinking outside the box: let's deprecate import altogether :) -Barry From barry at python.org Thu Dec 18 01:52:32 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 01:52:43 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180504.hBI54ba27649@oma.cosc.canterbury.ac.nz> References: <200312180504.hBI54ba27649@oma.cosc.canterbury.ac.nz> Message-ID: <1071730352.17717.7.camel@anthem> On Thu, 2003-12-18 at 13:04, Greg Ewing wrote: > Maybe in Python 3.0 the current directory should appear as a > package with some standard name. Then applications could use > relative imports to import their own modules, and absolute > ones to import standard modules, without fear of conflict. In a previous thread, we already came up with __me__ as a magic name referring to the current module. Let's use __my__ as the prefix for importing a relative module. too-tired-to-wink-ly y'rs, -Barry From art at wiktorsadowski.com Thu Dec 18 02:38:07 2003 From: art at wiktorsadowski.com (Wiktor Sadowski) Date: Thu Dec 18 02:37:46 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (release candidate 1) - custom Py_Initialize() Message-ID: <028401c3c539$e1e3db40$0201010a@kret> Thomas,consider this: specialized_python.exe --path/foo.px (command line) where foo.px is zipped ctypes-venster program -modules,some code,binaries(images etc.) specialized_python.exe could be statically linked against C ctypes,and could contain ctypes and venster python code (freezed) (specialized_python.exe could also work as "regular" python.exe and successfully run ".py" , ".pyc" files) Now associate ".px" with specialized_python.exe and you have clickable packed python programs. To build such specialized_python.exe you would need your own PyRun_*NotSoSimple*FileExFlags , new Py_Main, modified Py_Initialize and probably new or modified import engine. And with proposed addition specialized_python.exe could use existing Python shared library! I'm afraid it wouldn't be possible with Py_InitializeEx, you are proposing. For one of our clients we created pymozilla.exe to run packed websites(zipped html,xml,css,images and python "cgi" files) It's pretty cool. :-) Current Py_Initialize() is ok for distributed Python framework , but it's a pain if you'd like to use the language and not necessary each aspect of the framework. Why not make Python shared library more usable? Wiktor From fincher.8 at osu.edu Thu Dec 18 03:51:18 2003 From: fincher.8 at osu.edu (Jeremy Fincher) Date: Thu Dec 18 02:53:50 2003 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.304, 2.305 In-Reply-To: <011701c3c4e3$741ec620$f1bd958d@oemcomputer> References: <011701c3c4e3$741ec620$f1bd958d@oemcomputer> Message-ID: <200312180351.18465.fincher.8@osu.edu> On Wednesday 17 December 2003 04:19 pm, Raymond Hettinger wrote: > I think sorted() is better: > > * the context makes it clear that it's not a predicate > * the name helps distinguish it from list.sort() > * it fits with reversed() > * looking at actual code using sorted(), it reads well sorted() returns a list, and reversed() returns an iterator. That seems like a recipe for confusion, *especially* because they "fit" together and do completely different things. Jeremy From martin at v.loewis.de Wed Dec 17 16:30:23 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 02:57:30 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031216212312.GL17021@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > all i can do is recommend a framework and some guidelines on what > conventions could be fitted over that framework. In that case, your proposal will likely die. I don't think the framework can provide the feature that rexec.py originally provided. To prove that, I would need a complete proposal how precisely what ACLs are set on what objects, and how exactly I invoke code for restricted execution (i.e. what API do I call in what order). As you won't provide such a proposal, and as nobody else might, either, it is likely that this gets us nowhere. Regards, Martin From tim.one at comcast.net Thu Dec 18 03:21:28 2003 From: tim.one at comcast.net (Tim Peters) Date: Thu Dec 18 03:21:29 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > ... > I'm beginning to agree (uselessly) with Tim: import itself is a > problem. :-) Not useless at all -- that's bonding! Technically useless but humanly essential . It's no mystery *why* import is "a problem", but I'm not sure insight implies a cure: it's one of the very few places in Python where you can't explicitly say what you intend. Even in an inheritance hierarchy from hell, if I really don't want to chance that please.hit_me() will get satisfied from a class other than BlackJack, I can force it easily -- BlackJack.hit_me(please) There aren't any names to spell "the roots" of all the places Python may search to satisfy an import, so no way either to be explicit about intent. In contrast, nobody can mistake what import java.util.Random; intends. It's not a deep thing, it's just being able to name the intended context. I confess I'm keener on that than on growing more ways to spell more kinds of searches. From lkcl at lkcl.net Thu Dec 18 04:29:22 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Thu Dec 18 04:29:36 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> Message-ID: <20031218092922.GF6324@lkcl.net> On Wed, Dec 17, 2003 at 10:30:23PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > all i can do is recommend a framework and some guidelines on what > > conventions could be fitted over that framework. > martin, to clarify: "all i can do" is the wrong preamble phrase. "i would like to" is better. i was trying to be ... self-denigrating, over-careful, something like that. > originally provided. To prove that, I would need a complete proposal > how precisely what ACLs are set on what objects, and how exactly I > invoke code for restricted execution (i.e. what API do I call in what > order). what api, in what order, what code is invoked in (inside the python executable), _how_ ACLs are set on what objects, yes. _what_ acls are set on what objects is conditional on whether i receive funding to do so, or whether some other people can be of significant assistance. why? because there's a lot of them. what i was trying to say above, "all i can do is recommend a framework" is build up to an idea of adding in a framework, similar to Exceptions, by which ANY generically-defined restriction system can be plugged in to the python language. the idea being that if no such a system is not plugged in, the performance penalty on python is insignificant, and no user-code is restricted. however, this may all turn out to be unnecessary [quote from greg ewing]: > The spirit behind my suggestion was to start thinking about > ways in which functionality could be separated out so that > this kind of special-casing for security purposes isn't > needed. so, with the correct codebase reordering, a simple capabilities based system can be added, the problem goes away. yes? l. From martin at v.loewis.de Wed Dec 17 17:59:09 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 04:44:03 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <3FE0DB90.9060604@erols.com> References: <3FE0DB90.9060604@erols.com> Message-ID: "Edward C. Jones" writes: > silly.c: In function `silly': > silly.c:5: warning: suggest explicit braces to avoid ambiguous `else' > > I doubt if there is any bug in Py_DECREF. But would it be possible to > jiggle the Py_DECREF macros to suppress this message? It would be possible. Can you please submit a bug report? Or, better yet, a patch? Regards, Martin From anthony at interlink.com.au Thu Dec 18 04:50:50 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Dec 18 04:51:09 2003 Subject: [Python-Dev] 2.3.3 and beyond Message-ID: <200312180950.hBI9ooi1005762@localhost.localdomain> This should hopefully be no suprise to anyone, but I'm going to be cutting 2.3.3 final in about 12-15 hours time. Once that's done, I'll be handing the branch over to Jack so he can do his Macintosh magic. I _had_ hoped to spend a lot of time getting various platform issues resolved for 2.3.3 - this didn't end up happening (for reasons that ryhme with "junkies are scum"). Once Jack's done his Mac magic, I'll try to throw some effort at this for 2.3.4. I'm planning a 2.3.4 for around the May timeframe, assuming Tim doesn't create more stinking critical bugs in the meantime. (Oh, sure, he'll _claim_ the bugs were always there, and he was just fixing them... ) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From ncoghlan at iinet.net.au Thu Dec 18 05:42:02 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Dec 18 05:42:09 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <1071716386.27808.104.camel@anthem> References: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> <3FDEDF30.1050904@iinet.net.au> <1071716386.27808.104.camel@anthem> Message-ID: <3FE1847A.7030104@iinet.net.au> Barry Warsaw wrote: > On Tue, 2003-12-16 at 05:32, Nick Coghlan wrote: >> [snip description of potential naming conflicts] > Has this ever happened to you in practice? Well, no. But others seemed concerned about it, so I thought I'd try to clarify the problem - the scripts I write don't need anything more than the standard library and win32all. > It seems like the way out would be to start adopting a Java-like > convention for package names. The problem with that in current Python > is that you can't (easily) weave a package's contents from different > locations in the file system. I still like the various suggestions for _giving_ those parts of the system special names. That's what I was trying for, even if the names I chame up with weren't any good. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Thu Dec 18 05:58:42 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Dec 18 05:58:48 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <200312180012.hBI0C9E23257@c-24-5-183-134.client.comcast.net> References: <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> <200312180012.hBI0C9E23257@c-24-5-183-134.client.comcast.net> Message-ID: <3FE18862.50208@iinet.net.au> Guido van Rossum wrote: >>It would be a lot better if we could get away from the idea >>of a "restricted mode" in the sense of a flag somewhere that >>a bunch of things have to take notice of in order to behave >>securely, because that model of security is prone to springing >>leaks -- as happened in a big way when new-style classes were >>introduced. > > > Right. Restricted mode currently uses both paradigms: you only have > access to the builtins that are given to you in the __builtins__ dict > -- this is pure capability stuff, and IMO it works well -- and some > builtin operations behave differently when you're in restricted mode > -- this is the ACL stuff, and Samuele revealed serious holes in it. What if instead of 'builtin behaves differently in restricted mode' we had 'restricted __builtins__ contains a DIFFERENT builtin, that happens to have the same name'? That is, in addition to the ability to simply deny access to a specific builtin function or class, there was the ability to _replace_ one before giving it to the restricted code. Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From paoloinvernizzi at dmsware.com Thu Dec 18 06:03:38 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Thu Dec 18 06:03:45 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: References: <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> Message-ID: Tim Peters wrote: > In contrast, nobody can mistake what > > import java.util.Random; > > intends. It's not a deep thing, it's just being able to name the intended > context. I confess I'm keener on that than on growing more ways to spell > more kinds of searches. I *strongly* agree!!! --- Paolo Invernizzi From mwh at python.net Thu Dec 18 08:00:25 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 08:00:34 2003 Subject: [Python-Dev] bug in python arm-linux?: start_new_thread fails after popen In-Reply-To: <20031217151450.GB28436@unpythonic.net> (Jeff Epler's message of "Wed, 17 Dec 2003 09:14:50 -0600") References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> Message-ID: <2my8taffh2.fsf@starship.python.net> Jeff Epler writes: > On Wed, Dec 17, 2003 at 03:01:00PM +0100, Michael Lauer wrote: >> Hi, I have a python program which works fine on x86 but doesn't work on >> any of my arm-linux devices (Zaurus, Ipaq, SIMpad) - all of them are >> running Python 2.3.2 on top of glibc 2.3.2+linuxthreads on top of kernel >> 2.4.18-rmk6-pxa3 respectively kernel 2.4.19-rmk7. > > Using threads and fork together seems to be a big smelly armpit in Python. > There are also problems on redhat9, where signals in a fork+exec'd > subprocess are blocked, for instance. This is in no way restricted to RH9... > This seemed to be a consequence of blocking all signals in > thread_pthread.h:PyThread_start_new_thread(). Perhaps > pthread_atfork() Perhaps. Cheers, mwh -- I have no disaster recovery plan for black holes, I'm afraid. Also please be aware that if it one looks imminent I will be out rioting and setting fire to McDonalds (always wanted to do that) and probably not reading email anyway. -- Dan Barlow From barry at python.org Thu Dec 18 08:00:29 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 08:00:44 2003 Subject: [Python-Dev] Re: Christmas Wishlist In-Reply-To: <3FE1847A.7030104@iinet.net.au> References: <200312151842.hBFIg0K17763@c-24-5-183-134.client.comcast.net> <3FDEDF30.1050904@iinet.net.au> <1071716386.27808.104.camel@anthem> <3FE1847A.7030104@iinet.net.au> Message-ID: <1071752429.17717.24.camel@anthem> On Thu, 2003-12-18 at 05:42, Nick Coghlan wrote: > Barry Warsaw wrote: > > On Tue, 2003-12-16 at 05:32, Nick Coghlan wrote: > >> [snip description of potential naming conflicts] > > Has this ever happened to you in practice? > > Well, no. But others seemed concerned about it, so I thought I'd try to > clarify the problem - the scripts I write don't need anything more than > the standard library and win32all. I understand. The reason I ask is because it's been A Concern for as long as I've used Python, but in practice it's never really been an issue. -Barry From skip at pobox.com Thu Dec 18 08:10:58 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 08:11:00 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312180453.hBI4rIg27605@oma.cosc.canterbury.ac.nz> References: <200312180448.hBI4m8J23710@c-24-5-183-134.client.comcast.net> <200312180453.hBI4rIg27605@oma.cosc.canterbury.ac.nz> Message-ID: <16353.42850.434223.45725@montanaro.dyndns.org> Greg> Also I think VMS did something similar with dots in its Greg> pathnames. So there are precedents... Yes, but the dots were kind of hard to miss what with the directory portion of a path enclosed in [ ... ] Skip From mcherm at mcherm.com Thu Dec 18 08:15:24 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Thu Dec 18 08:15:26 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) Message-ID: <1071753324.3fe1a86c3827e@mcherm.com> [Michael Chermside] > ... INSTANCES of 'file' and could access > f.__class__. [...] so for restricted code we return something > like this: > > class FakeFile: > def __eq__(self, other): > return other == file > [Guido] > Are you aware of the original issue, which is that as soon as you have > a file *instance* (which might have been given to you by a very > restrictive open() variant), you can always get to the file *class* > using the __class__ attribute? Access to the __class__ attribute is > useful for all sorts of reasons. Yes, I'm aware of that issue... in fact that's what I'm trying to address here, but I'm obviously not doing a good job of expressing myself. I was thinking of the restricted code having access to an instance of 'file' named 'myFileInstance'. This instance has peculiar behavior when you access 'myFileInstance.__class__'. Instead of returning the class object 'file', it returns a class object like 'FakeFile' (which I *should* have made a subclass of 'type'). This whole plan is based on two misconceptions. The first is that access to __class__ is the only way to use introspection to access things of greater power (like the class) from an instance. It's NOT the only way, I'm just hoping that the others can be fully enumerated (somewhat shaky, but perhaps possible) and can be fixed in similar ways (maybe true). The other misconception (apparently) is that the only real USE for accessing the __class__ attribute is for testing class membership. You said above that it's useful "for all sorts of reasons"... I'm wondering what they are. And yes... I *also* realize that there's not a whole lot of difference between an instance which behaves oddly under introspection and one which is wrapped in a proxy. But I realize that providing a secure proxy is somewhat difficult and I'm wondering if the other approach is more useful. [Guido] > Yes, even for 3.0 this is still a dream... True enough. So feel free to just drop the issue if you've got better (ie, more immediate) things to spend your time on. I wouldn't want to see you get a pie by default because there wasn't time to complete a test suite! -- Michael Chermside From mwh at python.net Thu Dec 18 08:27:58 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 08:28:01 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031218031645.GH473@frobozz> (Andrew Bennetts's message of "Thu, 18 Dec 2003 14:16:45 +1100") References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> Message-ID: <2mu13yfe75.fsf@starship.python.net> Andrew Bennetts writes: > I still like the idea of having the standard library in its own > package, but I guess that will have to wait until Python 3.0. Guido tends to shoot people who suggest this, watch out :-) Cheers, mwh -- Lisp nearing the age of 50 is the most modern language out there. GC, dynamic, reflective, the best OO model extant including GFs, procedural macros, and the only thing old-fashioned about it is that it is compiled and fast. -- Kenny Tilton, comp.lang.python From mwh at python.net Thu Dec 18 08:42:25 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 08:42:28 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python bltinmodule.c, 2.304, 2.305 In-Reply-To: <20031217210929.GX13702@epoch.metaslash.com> (Neal Norwitz's message of "Wed, 17 Dec 2003 16:09:29 -0500") References: <20031217210929.GX13702@epoch.metaslash.com> Message-ID: <2mptemfdj2.fsf@starship.python.net> Neal Norwitz writes: > On Wed, Dec 17, 2003 at 12:43:35PM -0800, rhettinger@users.sourceforge.net wrote: >> >> Guido grants a Christmas wish: >> sorted() becomes a regular function instead of a classmethod. > > Would sort() be a better name? No ( ). > When I told someone who's done quite a bit of java recently about > sorted, he asked if it returned a bool whether the list was sorted > or not. :-) In Python, that would be issorted (please ignore callable ). I'd expect sort() to mutate its argument. Cheers, mwh -- "The future" has arrived but they forgot to update the docs. -- R. David Murray, 9 May 2000 From skip at pobox.com Thu Dec 18 08:50:12 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 08:50:14 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031218051021.GI473@frobozz> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> Message-ID: <16353.45204.728479.296988@montanaro.dyndns.org> Andrew> Or it can be fixed by not having the entire standard library at Andrew> the toplevel, but instead inside a single "stdlib" package. ... Andrew> The backwards compatibility problems probably means this isn't Andrew> feasible until Python 3.0, though :( Not necessarily. It has been proposed (more than once I think) that such a stdlib package could be created in parallel using symlinks. At its simplest, I think all you'd need would be a stdlib directory, a nearly empty __init__.py file and symlinks to all the current standard modules and packages. (The __init__.py file would take care of importing builtin modules.) I would choose a very short name for such a package ("std" seems best) even though it might create clashes because it will be used a lot. Skip From mwh at python.net Thu Dec 18 09:01:52 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 09:01:56 2003 Subject: [Python-Dev] Relative import In-Reply-To: <16353.45204.728479.296988@montanaro.dyndns.org> (Skip Montanaro's message of "Thu, 18 Dec 2003 07:50:12 -0600") References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> Message-ID: <2mfzfifcmn.fsf@starship.python.net> Skip Montanaro writes: > Andrew> Or it can be fixed by not having the entire standard library at > Andrew> the toplevel, but instead inside a single "stdlib" package. > ... > Andrew> The backwards compatibility problems probably means this isn't > Andrew> feasible until Python 3.0, though :( > > Not necessarily. It has been proposed (more than once I think) that such a > stdlib package could be created in parallel using symlinks. At its > simplest, I think all you'd need would be a stdlib directory, a nearly empty > __init__.py file and symlinks to all the current standard modules and > packages. (The __init__.py file would take care of importing builtin > modules.) Are you sure that you won't get multiple copies of the same module floating around? E.g. will you have from httplib import HTTPConnection as HC1 from std.httplib import HTTPConnection as HC2 HC1 is HC2 I think in the scheme sketched above this will be false, which kills the idea stone dead. I might be wrong, though. Cheers, mwh -- Counting lines is probably a good idea if you want to print it out and are short on paper, but I fail to see the purpose otherwise. -- Erik Naggum, comp.lang.lisp From pf_moore at yahoo.co.uk Thu Dec 18 09:03:26 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Thu Dec 18 09:03:13 2003 Subject: [Python-Dev] Re: Relative import References: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> <200312180020.hBI0K7T23311@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > There are two proposals that I can live with: my original proposal > with the leading dots counting the number of levels up, or the > triple-dot proposal with scan-up semantics. In both cases, the > default semantics would switch to absolute for Python 3.0. If that's what you;d like a vote on, I prefer the triple-dot proposal. As far as semantics go, I have no opinion (I don't have any use for either option) but I *really* dislike the look of the single dot. I'm not a great fan of the triple dot, but it's the lesser of the two evils. I have to admit to some confusion here, though. At the moment, the following works: pkg\__init__.py: print "importing pkg" import a pkg\a.py: print "importing a" import b pkg\b.py: print "importing b" >>> import pkg importing pkg importing a importing b Is the proposal that any of this *stop* working? I assume not. Otherwise, I'm -1 on the whole thing. I see no reason to break this usage, and good reason (I can rename pkg - before final release, obviously - without needing to do a search-and-replace edit on everything in the package). I think this whole thing needs a PEP, though. If only to be completely clear on what is and is not changing, and to record discarded options. Paul. -- This signature intentionally left blank From paoloinvernizzi at dmsware.com Thu Dec 18 09:19:56 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Thu Dec 18 09:20:06 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <16353.45204.728479.296988@montanaro.dyndns.org> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> Message-ID: Skip Montanaro wrote: > It has been proposed (more than once I think) that such a > stdlib package could be created in parallel using symlinks. Symlink is one of the things that I would like to have on my XP box... :-\ --- Paolo Invernizzi From pf_moore at yahoo.co.uk Thu Dec 18 09:40:30 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Thu Dec 18 09:40:48 2003 Subject: [Python-Dev] Re: Relative import References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> Message-ID: Barry Warsaw writes: > On Tue, 2003-12-16 at 13:26, Guido van Rossum wrote: >> A lot people have presented a good case for relative imports. Nobody >> has argued to keep the status quo (where imports are ambiguous about >> whether they are meant to be absolute or relative). So I suggest that >> in 2.4, we introduce the leading dot notation for relative import, >> while still allowing relative import without a leading dot. In >> 2.5 we can start warning about relative import without a leading dot >> (although that will undoubtedly get complaints from folks who have >> code that needs to work with 2.3 still). In 3.0 we can retire >> ambiguous import. > > I'll just note that where the current status quo trips /me/ up most is > when I accidentally have a local module with the same name as a global > module, and then I write an import statement expecting to get the > standard library module, but end up getting the local module. > > That's why when I tend to think about this, I start wanting a way to > spell "definitely give me the global one, no matter what". IOW, I feel > like I want a way to bypass relative module lookups. That issue I can understand. And I agree there should be a way to state it explicitly. One of the issues here is that this area is a bit under-documented (I know, I should read the source, but I don't have the time right now). At the moment, there are two cases: 1. Import from sys.path. This is what is being called an "absolute" import, and is nice and easy to understand. The key issue is that there is no way to *force* this interpretation in the face of option (2) below. 2. Import from "the package". This is the under-documented bit, but if I understand it correctly, it's basically that from within a module contained in a package, sys.path is conceptually *extended* to include the package's __path__ (which by default contains the directory of the package, but which can be user-modified). Now the big problem here is that behaviour (2) is useful. Simple "relative" imports of one module within a package from another module in the same package are common. Guido's (IMHO ugly) "dot" syntax handles that, by making users explicitly request option (2), and making the current import syntax *only* mean (1). But none of the proposed solutions handle the __path__ variable. I don't have any objection in principle to desupporting __path__ (heck, it would have made thinking about PEP 302 easier, if nothing else) but (a) it would need a deprecation warning, and (b) Guido himself offered a use case in . This definitely needs a PEP. If we're removing support for __path__, the implications need to be thought through (PEP 302, the pkgutil module, etc etc). If we're not, none of the proposals so far have covered how __path__ gets supported in future. A much simpler proposal, just providing an explicit way of saying "Import from sys.path *only*" may be OK without a PEP. But even then, I'd suspect we should have a PEP explaining *why* it has to be this simple. Go on, Guido. We'll be gentle if you write a PEP, and we won't set c.l.p on you :-) Paul. -- This signature intentionally left blank From skip at pobox.com Thu Dec 18 09:49:39 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 09:49:46 2003 Subject: [Python-Dev] Relative import In-Reply-To: <2mfzfifcmn.fsf@starship.python.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> <2mfzfifcmn.fsf@starship.python.net> Message-ID: <16353.48771.531590.7792@montanaro.dyndns.org> >> It has been proposed (more than once I think) that such a stdlib >> package could be created in parallel using symlinks.... Michael> Are you sure that you won't get multiple copies of the same Michael> module floating around? E.g. will you have Michael> from httplib import HTTPConnection as HC1 Michael> from std.httplib import HTTPConnection as HC2 Michael> HC1 is HC2 Michael> I think in the scheme sketched above this will be false, which Michael> kills the idea stone dead. Maybe the symlink idea won't work. It seems that it would work to import the standard modules from std/__init__.py though. I performed a simple experiment. I created the std directory, added a symlink in it for the random module and added an __init__.py file with this content: import sys import urlparse Here's the result: >>> import random >>> from std import random as random2 >>> random is random2 False >>> import urlparse >>> from std import urlparse as urlparse2 >>> urlparse is urlparse2 True >>> import sys >>> from std import sys as sys2 >>> sys is sys2 True If we added a significant number of modules to std/__init__.py, startup would slow to a crawl. I imagine some sort of delayed import mechanism could be crafted to avoid this problem though. Skip From lists at hlabs.spb.ru Thu Dec 18 12:52:31 2003 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Thu Dec 18 09:52:47 2003 Subject: [Python-Dev] Importing packages from command line Message-ID: <3FE1E95F.7080209@hlabs.spb.ru> Just an idea... 1. python -p package_name Equivalent to: import package_name 2. python -p package.zip Equivalent to: import sys sys.insert(0, "package.zip") import package -- Dmitry Vasiliev (dima at hlabs.spb.ru) From skip at pobox.com Thu Dec 18 09:55:12 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 09:55:18 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> Message-ID: <16353.49104.97549.745804@montanaro.dyndns.org> Paolo> Symlink is one of the things that I would like to have on my XP Paolo> box... As Michael Hudson suggests, symlinks probably won't work anyway. I have something which looks like a symlink on my Win2k box. Windows calls it a Shortcut in listings. Is that not roughly the same thing as a symlink? Skip From fdrake at acm.org Thu Dec 18 10:16:15 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 18 10:16:37 2003 Subject: [Python-Dev] Doc/ tree closed on 2.3 branch Message-ID: <16353.50367.49731.885916@sftp.fdrake.net> Python's Doc/ tree is closed on the release23-maint branch. Commits may still be made on the trunk. The final 2.3.3 documentation will appear in the development area on python.org shortly: http://www.python.org/dev/doc/ The branch will be open once the 2.3.3 release has been completed. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From paoloinvernizzi at dmsware.com Thu Dec 18 10:18:22 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Thu Dec 18 10:22:14 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <16353.49104.97549.745804@montanaro.dyndns.org> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <200312180455.hBI4t3g23741@c-24-5-183-134.client.comcast.net> <20031218051021.GI473@frobozz> <16353.45204.728479.296988@montanaro.dyndns.org> <16353.49104.97549.745804@montanaro.dyndns.org> Message-ID: <3FE1C53E.1080507@dmsware.com> Skip Montanaro wrote: >I have >something which looks like a symlink on my Win2k box. Windows calls it a >Shortcut in listings. Is that not roughly the same thing as a symlink? > > No, the NT file system as nothing as flexible as a symlink. If we are speaking about symlinks to files, windows shortcuts are barely usable only for pointing to executable for execution in a console (but they have a lot of redirection problems). If you *open* a shortcut you are actually opening the shortcut itself, not the thing it's pointing to ;-( Shortcuts to a directory is only usable by Explorer-like-programs, I don't know any usable operation with them from the console... The only way to use shortcut in python is to turn python shortcut-aware in the import machinery... --- Paolo Invernizzi Ps... cygwin is another story.... From guido at python.org Thu Dec 18 10:38:03 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 18 10:38:09 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: Your message of "Thu, 18 Dec 2003 20:58:42 +1000." <3FE18862.50208@iinet.net.au> References: <200312172250.hBHMoXt25172@oma.cosc.canterbury.ac.nz> <200312180012.hBI0C9E23257@c-24-5-183-134.client.comcast.net> <3FE18862.50208@iinet.net.au> Message-ID: <200312181538.hBIFc3x25070@c-24-5-183-134.client.comcast.net> > >>It would be a lot better if we could get away from the idea > >>of a "restricted mode" in the sense of a flag somewhere that > >>a bunch of things have to take notice of in order to behave > >>securely, because that model of security is prone to springing > >>leaks -- as happened in a big way when new-style classes were > >>introduced. > > > > Right. Restricted mode currently uses both paradigms: you only have > > access to the builtins that are given to you in the __builtins__ dict > > -- this is pure capability stuff, and IMO it works well -- and some > > builtin operations behave differently when you're in restricted mode > > -- this is the ACL stuff, and Samuele revealed serious holes in it. > > What if instead of 'builtin behaves differently in restricted mode' we > had 'restricted __builtins__ contains a DIFFERENT builtin, that happens > to have the same name'? > > That is, in addition to the ability to simply deny access to a specific > builtin function or class, there was the ability to _replace_ one before > giving it to the restricted code. Oh, there is, of course. Sorry, I wasn't clear. There are a few operations that are *not* spelled as built-in function calls that have to be restricted. For example, every object supports getting obj.__dict__ -- but in restricted mode you don't want to allow this. The Python source currently checks for "was I called from restricted mode" to deny such operations. The holes that Samuele revealed had to do with tricking unrestricted code into invoking unsafe things with arbitrary arguments controlled by the restricted code. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Thu Dec 18 10:45:52 2003 From: mwh at python.net (Michael Hudson) Date: Thu Dec 18 10:47:02 2003 Subject: [Python-Dev] speeding function calls (a little) In-Reply-To: <1067487850.24165.53.camel@localhost.localdomain> Message-ID: <425EA205-3171-11D8-8B62-0003931DF95C@python.net> On Thursday, Oct 30, 2003, at 05:24 Europe/Stockholm, Jeremy Hylton wrote: [function calls] > There is an optimization that depends on having no default arguments > (or > keyword arguments or free variables). Why does it depend on not having default arguments? If you supply the right number of arguments (something that's obviously already checked) why does the function having defaults make a jot of difference? > It copies the arguments directly > from the caller's frame into the callee's frame without creating an > argument tuple. > > It's interesting to avoid the copy from caller to callee, but I don't > think it's a big cost relative to everything else we're doing to set up > a frame for calling. (I expect the number of arguments is usually > small.) You would need some way to encode what variables are loaded > from the caller stack and what variables are loaded from the current > frame. Either a different opcode or some kind of flag in the current > LOAD/STORE argument. As I think Phillip managed to convince himself recently, some kind of JIT functionality seems to be needed to do function calls really efficiently. I wonder if libffi does enough... it would be nice if the body of CALL_FUNCTION could look a bit like this: x = POP() PUSH(((some_cast_or_other)x)(f, stack_pointer, oparg)) Gah, this doesn't really seem to work out, on thinking about it. Wins seem more likely to come from knowing with some certainly at the call site that you've not messed the arguments up (and so we're back to wanting a JIT, it seems to me). > One other possibility for optimization is this XXX comment in > fast_function(): > /* XXX Perhaps we should create a specialized > PyFrame_New() that doesn't take locals, but does > take builtins without sanity checking them. > */ > f = PyFrame_New(tstate, co, globals, NULL); > > PyFrame_New() does a fair amount of work that is unnecessary in the > common case. Fair amount? I have a patch that gets ~1.5% on pystone along these lines, but it's a bit scary (makes a "lightweight-frame" subclass that assumes more about things on it's freelist, and so on). I'm not sure the modest gains are worth the complexity, but I'll post it to SF... Cheers, mwh From guido at python.org Thu Dec 18 10:50:44 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 18 10:50:48 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 14:03:26 GMT." References: <200312172324.hBHNOCR25328@oma.cosc.canterbury.ac.nz> <200312180020.hBI0K7T23311@c-24-5-183-134.client.comcast.net> Message-ID: <200312181550.hBIFoiN25162@c-24-5-183-134.client.comcast.net> > If that's what you;d like a vote on, I prefer the triple-dot proposal. > As far as semantics go, I have no opinion (I don't have any use for > either option) but I *really* dislike the look of the single dot. I'm > not a great fan of the triple dot, but it's the lesser of the two > evils. Your vote doesn't help me; the triple dot proposal is semantically different from the single dot. > I have to admit to some confusion here, though. At the moment, the > following works: > > pkg\__init__.py: > print "importing pkg" > import a > pkg\a.py: > print "importing a" > import b > pkg\b.py: > print "importing b" > > >>> import pkg > importing pkg > importing a > importing b > > Is the proposal that any of this *stop* working? I assume not. > Otherwise, I'm -1 on the whole thing. I see no reason to break this > usage, and good reason (I can rename pkg - before final release, > obviously - without needing to do a search-and-replace edit on > everything in the package). The idea is indeed to break this in Python 3.0 (but not earlier for backwards compatibility reasons). In 3.0, you would have to (and in 2.4 perhaps you could) write this as from . import a or from ... import a depending on which proposal is accepted. > I think this whole thing needs a PEP, though. If only to be completely > clear on what is and is not changing, and to record discarded options. Correct. I have no time to write it though. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Dec 18 10:59:00 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 18 10:59:06 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 14:40:30 GMT." References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> Message-ID: <200312181559.hBIFx0j25181@c-24-5-183-134.client.comcast.net> [Barry] > > That's why when I tend to think about this, I start wanting a way to > > spell "definitely give me the global one, no matter what". IOW, I feel > > like I want a way to bypass relative module lookups. [Paul Moore] > That issue I can understand. And I agree there should be a way to > state it explicitly. > > One of the issues here is that this area is a bit under-documented (I > know, I should read the source, but I don't have the time right now). > > At the moment, there are two cases: > > 1. Import from sys.path. This is what is being called an "absolute" > import, and is nice and easy to understand. The key issue is that > there is no way to *force* this interpretation in the face of > option (2) below. Right. > 2. Import from "the package". This is the under-documented bit, but if > I understand it correctly, it's basically that from within a module > contained in a package, sys.path is conceptually *extended* to > include the package's __path__ (which by default contains the > directory of the package, but which can be user-modified). That's not a very useful way to think about it; when the module is found on __path__ something very *different* happens than when it is found on sys.path. The difference is noticeable if you ask the imported module for its __name__. If you the import was satisfied from __path__, then __name__ will include the package name. If it was satisfied from sys.path, it won't. And __name__ is related to module identity: all modules are stored in sys.modules using their __name__ as the key, so multiple imports of a module with the same __name__ end up referring to the same module (and only the first import causes the module's code to be executed). > Now the big problem here is that behaviour (2) is useful. Simple > "relative" imports of one module within a package from another module > in the same package are common. Guido's (IMHO ugly) "dot" syntax > handles that, by making users explicitly request option (2), and > making the current import syntax *only* mean (1). Actually, what you consider useful is considered harmless by others. Many large packages, including Zope, have adopted a rule of always using absolute imports, to clarify what kind of import is being used. > But none of the proposed solutions handle the __path__ variable. I > don't have any objection in principle to desupporting __path__ (heck, > it would have made thinking about PEP 302 easier, if nothing else) but > (a) it would need a deprecation warning, and (b) Guido himself offered > a use case in . __path__ has no bearing on the proposals, it is used for relative imports. I think your confusing about how the current import works caused you to think it is relevant. > This definitely needs a PEP. If we're removing support for __path__, > the implications need to be thought through (PEP 302, the pkgutil > module, etc etc). If we're not, none of the proposals so far have > covered how __path__ gets supported in future. We're not, but they have (by not mentioning it); but I agree that a PEP is needed. > A much simpler proposal, just providing an explicit way of saying > "Import from sys.path *only*" may be OK without a PEP. But even then, > I'd suspect we should have a PEP explaining *why* it has to be this > simple. This was where this thread started: someone proposed a way of spelling "import from sys.path only". My big problem with that is that *most* imports are intended to be satisfied by sys.path only, so this spelling should be the default, meaning the current spelling. > Go on, Guido. We'll be gentle if you write a PEP, and we won't set > c.l.p on you :-) Alas, I have no time. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at acm.org Thu Dec 18 11:53:54 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Dec 18 11:54:05 2003 Subject: [Python-Dev] Python 2.3.3 documentation Message-ID: <16353.56226.174490.137050@sftp.fdrake.net> Formatted copies of the Python 2.3.3 documentation is available at: ftp://ftp.python.org/pub/python/doc/2.3.3/ The website has not yet been updated. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From klm at zope.com Thu Dec 18 12:20:30 2003 From: klm at zope.com (Ken Manheimer) Date: Thu Dec 18 12:27:10 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312181550.hBIFoiN25162@c-24-5-183-134.client.comcast.net> Message-ID: +1 for the '.' proposal, where (to verify my understanding): - '.' indicates "import from the current package", '..' means "import from the package of my package", and so forth, - imports where the package part lacks a leading '.' are absolute, ie from the top level of the path. -1 for the '...' proposal, where '...' means a search from the current package up to the most superior containing package (which is actually a virtual package that entails the entire path). Ie, i am for absolute being the default and explicit-relative being available, and against ambiguous relative aka scanning the hierarchy. (I might be convinced otherwise about the scanning deal, but i think the more deterministic we can make imports, while still keeping them flexible enough to do what people need, the better.) I also like your proposal for a "no ambiguous imports" flag from __future__ import absolute_import Ken klm@zope.com From guido at python.org Thu Dec 18 12:50:41 2003 From: guido at python.org (Guido van Rossum) Date: Thu Dec 18 12:50:48 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 12:20:30 EST." References: Message-ID: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> [Ken Manheimer] > +1 for the '.' proposal, where (to verify my understanding): > > - '.' indicates "import from the current package", '..' means "import > from the package of my package", and so forth, > > - imports where the package part lacks a leading '.' are absolute, ie > from the top level of the path. > > -1 for the '...' proposal, where '...' means a search from the current > package up to the most superior containing package (which is > actually a virtual package that entails the entire path). > > Ie, i am for absolute being the default and explicit-relative being > available, and against ambiguous relative aka scanning the hierarchy. > (I might be convinced otherwise about the scanning deal, but i think > the more deterministic we can make imports, while still keeping them > flexible enough to do what people need, the better.) Thanks! Your vote is especially important for me because as far as I can remember you proposed the scan-up rule for the very first version of "ni" that we developed together back in 1995! > I also like your proposal for a "no ambiguous imports" flag > > from __future__ import absolute_import OK. Wanna write a PEP? Just kidding, I know you have less time than I do for it. So... *anybody* wanna write a PEP about this? (The parentheses proposal should also be folded in.) --Guido van Rossum (home page: http://www.python.org/~guido/) From klm at zope.com Thu Dec 18 13:07:11 2003 From: klm at zope.com (Ken Manheimer) Date: Thu Dec 18 13:14:09 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> Message-ID: On Thu, 18 Dec 2003, Guido van Rossum wrote: > Thanks! Your vote is especially important for me because as far as I > can remember you proposed the scan-up rule for the very first version > of "ni" that we developed together back in 1995! Older and*/*or wiser?-) FWIW, i also proposed using a leading '.' (and a leading '_', when '.' was refused because the parser couldn't handle it) for what we're now calling explict relative imports. Ken klm@zope.com From skip at pobox.com Thu Dec 18 16:20:22 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 16:21:00 2003 Subject: [Python-Dev] syncing between projects? Message-ID: <16354.6678.463640.780682@montanaro.dyndns.org> The python-mode.el file in the python-mode project is now at 4.41, while the version in Python's Misc directory is still at 4.38. Is there a way to sync them in one cvs operation so they have the same version? Thx, Skip From martin at v.loewis.de Thu Dec 18 16:22:34 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 16:23:44 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031217233126.GA4692@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> Message-ID: Jeff Epler writes: > To follow up on my own message, here is a program that demonstrates the > problem I ran into with threads and signals. Can you find out what $$ is, and what the PIDs and thread IDs of all participating threads are? Regards, Martin From martin at v.loewis.de Thu Dec 18 16:28:50 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 16:29:13 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87y8tbezcq.fsf@codesourcery.com> References: <87y8tbezcq.fsf@codesourcery.com> Message-ID: "Zack Weinberg" writes: > > This is what leads to the bletcherous > > > > do { > > the real macro guts go here > > } while(0) > > > > form of macro definition. Py_DECREF could be rewritten in that form -- but > > I've puked enough for one day . > > Should, not just could. Yeah, it's bletcherous, but it's the only > way to be sure. (Short of inline functions.) Not true. As Tim explains, there is no possible application of the macro which gets misinterpreted. We don't have a single if-statement, we have an if-else-statement in the macro. That never leads to the dangling else problem. Regards, Martin From greg at cosc.canterbury.ac.nz Thu Dec 18 16:33:22 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 16:33:32 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <3FE18862.50208@iinet.net.au> Message-ID: <200312182133.hBILXMV05456@oma.cosc.canterbury.ac.nz> Nick Coghlan : > What if instead of 'builtin behaves differently in restricted mode' we > had 'restricted __builtins__ contains a DIFFERENT builtin, that happens > to have the same name'? That wouldn't solve the file problem, because if the restricted code ever got hold of a file object created by unrestricted code, it would have access to the "real" file class which can open any file. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From martin at v.loewis.de Thu Dec 18 16:27:30 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 16:33:51 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <16352.63023.993326.798113@montanaro.dyndns.org> References: <3FE0DB90.9060604@erols.com> <16352.63023.993326.798113@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > I meant the Py_*REF macros which expand to "if" statements and you'd bracify > the clauses, e.g., Py_DECREF would become: That would still trigger the warning about if(1) Py_DECREF(foo); as this expands to if(1) if(cond(foo)) {action1(foo);} else {action2(foo);} This, in turn, is potentially unintended, as the author of that code may have meant the else to go with the if(1), whereas, in C, it goes with the if(cond(foo)). As Tim explains, the intended semantics is the desired one, too, so there is really no problem in the code. As Tim also explains, do{}while(1) would silence the warning. Regards, Martin From barry at python.org Thu Dec 18 16:36:40 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 16:36:45 2003 Subject: [Python-Dev] syncing between projects? In-Reply-To: <16354.6678.463640.780682@montanaro.dyndns.org> References: <16354.6678.463640.780682@montanaro.dyndns.org> Message-ID: <1071783400.31232.10.camel@anthem> On Thu, 2003-12-18 at 16:20, Skip Montanaro wrote: > The python-mode.el file in the python-mode project is now at 4.41, while the > version in Python's Misc directory is still at 4.38. Is there a way to sync > them in one cvs operation so they have the same version? cvs com -r4.41 python-mode.el -Barry From gmcm at hypernet.com Thu Dec 18 16:45:54 2003 From: gmcm at hypernet.com (Gordon McMillan) Date: Thu Dec 18 16:46:51 2003 Subject: [Python-Dev] Relative import In-Reply-To: Message-ID: <3FE1D9C2.32355.4A1EE51A@localhost> I am +10e6 on disambiguating imports. I am +0 on "scan". I am +0.01 on "explicit relative". I can live with any syntax Guido can, so I'm just voting on functionality. The use cases for "ambiguous" imports amount to: 1) I use this cool package, but I'm too lazy to keep up with new releases or document the release required & ensure that it's available & further I'm so arrogant that I don't care that my users might run into problems having 2 copies of the code or that the package developer might be offended by my end run. 2) I structured my package wrong, and now I want to fix it with the mouse. I'm not sure I go as far as Tim, but I think perhaps the PSU should have a regulation about writing import hooks without a license. -- Gordon http://www.mcmillan-inc.com/ From greg at cosc.canterbury.ac.nz Thu Dec 18 16:50:27 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 16:50:36 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <1071753324.3fe1a86c3827e@mcherm.com> Message-ID: <200312182150.hBILoRe05474@oma.cosc.canterbury.ac.nz> Michael Chermside : > The other misconception (apparently) is that the only real USE for > accessing the __class__ attribute is for testing class membership. You > said above that it's useful "for all sorts of reasons"... I'm > wondering what they are. Two that come to mind: * Looking at the class's docstring * Finding out what methods an object has Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From martin at v.loewis.de Thu Dec 18 16:33:52 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Thu Dec 18 16:56:04 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031218092922.GF6324@lkcl.net> References: <20031215173156.GC25203@lkcl.net> <16349.63203.797449.677617@montanaro.dyndns.org> <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> <20031218092922.GF6324@lkcl.net> Message-ID: Luke Kenneth Casson Leighton writes: > so, with the correct codebase reordering, a simple capabilities > based system can be added, the problem goes away. > > yes? I still don't see how it could, and I believe I understand pretty well what kind of feature you propose. Restricted execution is just something completely different. Regards, Martin From skip at pobox.com Thu Dec 18 16:56:05 2003 From: skip at pobox.com (Skip Montanaro) Date: Thu Dec 18 16:56:33 2003 Subject: open == file considered harmful (Re: [Python-Dev] RE: rexec.pyunuseable) In-Reply-To: <200312182150.hBILoRe05474@oma.cosc.canterbury.ac.nz> References: <1071753324.3fe1a86c3827e@mcherm.com> <200312182150.hBILoRe05474@oma.cosc.canterbury.ac.nz> Message-ID: <16354.8821.435209.523359@montanaro.dyndns.org> >> You said above that it's useful "for all sorts of reasons"... I'm >> wondering what they are. Greg> Two that come to mind: ... And another is breaking out of rexec hell to do devious things on the system where the code is running. Skip From greg at cosc.canterbury.ac.nz Thu Dec 18 16:57:39 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 16:57:59 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Message-ID: <200312182157.hBILvd005491@oma.cosc.canterbury.ac.nz> Paul Moore : > If that's what you'd like a vote on, I prefer the triple-dot > proposal. How would you spell relative references to parent packages using the triple-dot version? Remember we've dropped the idea of search-upwards, so that would have to be made explicit somehow. > pkg\__init__.py: > print "importing pkg" > import a > pkg\a.py: > print "importing a" > import b > pkg\b.py: > print "importing b" > > >>> import pkg > importing pkg > importing a > importing b > > Is the proposal that any of this *stop* working? I think the proposal is for it to keep working for backwards compatibility now, but to stop working in Python 3.0. Is that correct? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Thu Dec 18 16:59:26 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 16:59:49 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Message-ID: <200312182159.hBILxQu05504@oma.cosc.canterbury.ac.nz> Paolo Invernizzi : > Skip Montanaro wrote: > > > It has been proposed (more than once I think) that such a > > stdlib package could be created in parallel using symlinks. > > Symlink is one of the things that I would like to have on my XP box... Also, wouldn't that give two ways of getting each of the modules, with all the attendand Bad Things? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Thu Dec 18 17:07:33 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 17:07:41 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Message-ID: <200312182207.hBIM7Xn05611@oma.cosc.canterbury.ac.nz> Paul Moore : > But none of the proposed solutions handle the __path__ variable. I > don't have any objection in principle to desupporting __path__ I'll be worried if __path__ is likely to disappear as a result of all this. I'm using it in what will be the next version of my Python GUI library, to switch in different implementations of submodules depending on the platform. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Thu Dec 18 17:13:03 2003 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Dec 18 17:13:13 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <16353.49104.97549.745804@montanaro.dyndns.org> Message-ID: <200312182213.hBIMD3x05633@oma.cosc.canterbury.ac.nz> Skip Montanaro : > I have something which looks like a symlink on my Win2k box. Windows > calls it a Shortcut in listings. Is that not roughly the same thing > as a symlink? Only if it's transparent to the system calls which operate on files (e.g. open()ing it opens the file that it refers to, etc.) I don't believe that's the case with Windows shortcuts. Macintosh "aliases" have the same problem. (And now with MacOS X, we have both aliases *and* symlinks in the same system, just to add to the confusion...) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From barry at python.org Thu Dec 18 17:15:05 2003 From: barry at python.org (Barry Warsaw) Date: Thu Dec 18 17:15:11 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312182157.hBILvd005491@oma.cosc.canterbury.ac.nz> References: <200312182157.hBILvd005491@oma.cosc.canterbury.ac.nz> Message-ID: <1071785704.31232.22.camel@anthem> On Fri, 2003-12-19 at 05:57, Greg Ewing wrote: > How would you spell relative references to parent packages > using the triple-dot version? Does anybody have an example, in released code, of where this kind of repackaging relative imports is being used? I'd like to look at some real-world examples. Thanks, -Barry From lkcl at lkcl.net Thu Dec 18 17:57:48 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Thu Dec 18 17:57:48 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: References: <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> <20031218092922.GF6324@lkcl.net> Message-ID: <20031218225748.GE2234@lkcl.net> On Thu, Dec 18, 2003 at 10:33:52PM +0100, Martin v. L?wis wrote: > Luke Kenneth Casson Leighton writes: > > > so, with the correct codebase reordering, a simple capabilities > > based system can be added, the problem goes away. > > > > yes? > > I still don't see how it could, neither do i: i believe that the best person you should address that concern to is to greg ewing, who raised this point: "The spirit behind my suggestion was to start thinking about ways in which functionality could be separated out so that this kind of special-casing for security purposes isn't needed." in other words, i think he means that by restructuring the python codebase [and libraries? and its design?] it _may_ be possible to avoid a need for adding ACLs at all. and i believe he may be thinking along the lines of being able to permanently _remove_ at run-time, under python-programmer-control, access to __class__ from objects. somehow. > and I believe I understand pretty well > what kind of feature you propose. Restricted execution is just > something completely different. if you believe that, then i have evidently not explained myself clearly enough. i'll give this one more shot. i'll be honest with you. if it's not obvious enough at the end of this message, i have to give up and stop, and say i tried but did not succeed in explaining this clearly enough to you, which is no failing on your part, but on mine. anyway. first i should outline the pseudo-code modifications needed to Python/ceval.c which are necessary, then after that i will outline what ACLs are needed. what i _can_ say that i haven't gone into detail on [in this message] is _how_ the acls are obtained from the function object. e.g. in GetACLforFunction. but it is worth mentioning that _how_ the acls are _associated_ with the function object (or any other object) is an implementation-specific issue NOT a design / specification issue. static PyObject * call_function(PyObject ***pp_stack, int oparg) { PyObject **pfunc = (*pp_stack) - n - 1; PyObject *func = *pfunc; /* caller function name on stack below?? probably not, but the principle is at least demonstrable */ PyObject *calling_func = GetCallerFunction(*pp_stack) - n - 2; PyACL *acl_for_func = GetACLforFunction(func); ... if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_EXECUTE) { return Exception("access denied"); } ... } also, to satisfy the requirements of "capabilities", a specific check to make it look like the object doesn't exist is added. it's probably possible to have a look-up table on a per-opcode basis to look up the permissions to be checked against, which alleviates the need to put a call to check_permissions in every darn switch statement. note the check for ACL being NULL: if it's NULL, that's assumed to mean "anything goes". it also means that there's not much performance hit involved unless you actually add permissions. case STORE_ATTR: if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_CAN_SEE) { return Exception("attrib doesn't exist"); } if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_WRITE) { return Exception("access denied"); } case DELETE_ATTR: if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_CAN_SEE) { return Exception("attrib doesn't exist"); } if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_WRITE+PERMS_DELETE) { return Exception("access denied"); } case LOAD_ATTR: if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_CAN_SEE) { return Exception("attrib doesn't exist"); } if (acl_for_func != NULL && !check_permissions(calling_func, acl_for_func, PERMS_READ) { return Exception("access denied"); } basically, as long as code is only executed through eval_frame(), then it's likely to be the only place where access control checks are needed to be added. basically, therefore, once calls to restrict operations are added, it's a matter of dropping the ACLs in at the right places. _that's_ how you achieve the same job as the rexec.py code. you place an acl on __builtins__ to restrict "all functions", DENY, EVERYTHING. you place an acl on open() to restrict "all functions", DENY, FILE_WRITE. you place an acl on open.__class__ "all functions", DENY, CAN_SEE+FILE_WRITE in fact you might even be able to get away with putting an acl on EVERY function right at the top-level. but that, as described previously, means that it'd be necessary to get the "ACL-inheritance" to work properly, in order to avoid having to add a DENY FILE_WRITE acl to every single file operation capable of writing, for example. so like i said, i believe it to be a relatively simple job to spec out how to add the means _by which_ ACLs can be usefully evaluated. what i CAN'T tell you is exactly WHERE every single ACL [needed to achieve the same results as rexec] have to be added [such that they will be usefully evaluated]: i simply don't have enough knowledge of the python codebase to do that on my own. [unless someone was happy to pay me for long enough to find out, of course]. is this making sense at all? in some ways, the longer this is left, the harder it is going to be to retrospectively bolt on. there's an adage that says security cannot be easily added in, it has to be designed in from the start. fortunately, i think there are a lot of smart people about :) sincerely, l. From janssen at parc.com Thu Dec 18 18:54:49 2003 From: janssen at parc.com (Bill Janssen) Date: Thu Dec 18 18:55:12 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Thu, 18 Dec 2003 14:15:05 PST." <1071785704.31232.22.camel@anthem> Message-ID: <03Dec18.155458pst."58611"@synergy1.parc.xerox.com> > Does anybody have an example, in released code, of where this kind of > repackaging relative imports is being used? I'd like to look at some > real-world examples. I'm citing an example from the Jython compile of the Plucker distiller, for one. That's available as part of the Plucker package from www.plkr.org -- look at the Makefile in parser/python. But I'd guess most real examples are proprietary code (I can think of a couple right now). Bill From jepler at unpythonic.net Thu Dec 18 20:23:55 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Thu Dec 18 20:23:44 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> Message-ID: <20031219012347.GB4692@unpythonic.net> On Thu, Dec 18, 2003 at 10:22:34PM +0100, Martin v. L?wis wrote: > Can you find out what $$ is, and what the PIDs and thread IDs of all > participating threads are? I'm not sure what all information I should try to gather for you. Let me know if you think this is enough to file a bug report with... I changed the example to make it clearer that it's the subprocess ignoring the signal that is the problem, not anything in Python that is taking time to notice the death of a child process. Typical output (of course, pids and ppids change from run to run. I didn't find gettid(), I think thread.get_ident() is not what you were asking for): not threaded os.getpid() -> 6444 os.getppid() -> 6332 thread.get_ident() -> 1074152064 shell process 6445 thread ppid 6444 Elapsed time 0.00640296936035 subthread os.getpid() -> 6444 os.getppid() -> 6332 thread.get_ident() -> 1082547504 shell process 6447 thread ppid 6444 kill failed Elapsed time 1.01621508598 main thread os.getpid() -> 6444 os.getppid() -> 6332 thread.get_ident() -> 1074152064 shell process 6449 thread ppid 6444 Elapsed time 0.00641894340515 :r threadbug.py import thread, time, os def doit(): print "os.getpid() ->", os.getpid() print "os.getppid() ->", os.getppid() print "thread.get_ident() ->", thread.get_ident() t = time.time() os.system("echo shell process $$; echo thread ppid $PPID; kill $$; echo kill failed; sleep 1") t1 = time.time() print "Elapsed time", t1-t print print print "not threaded" doit() print "subthread" thread.start_new(doit, ()) time.sleep(2) print "main thread" doit() From mal at egenix.com Fri Dec 19 03:30:27 2003 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Dec 19 03:30:27 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.204, 2.205 In-Reply-To: References: Message-ID: <3FE2B723.9060108@egenix.com> perky@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Objects > In directory sc8-pr-cvs1:/tmp/cvs-serv1651/Objects > > Modified Files: > unicodeobject.c > Log Message: > SF #859573: Reduce compiler warnings on gcc 3.2 and above. > > Index: unicodeobject.c > *************** > *** 2204,2208 **** > > /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ > ! if (size == 1 && *(unsigned char*)s < 256) { > Py_UNICODE r = *(unsigned char*)s; > return PyUnicode_FromUnicode(&r, 1); > --- 2212,2216 ---- > > /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ > ! if (size == 1) { > Py_UNICODE r = *(unsigned char*)s; > return PyUnicode_FromUnicode(&r, 1); This "fix" doesn't look right. Please check. > *************** > *** 2406,2409 **** > --- 2414,2421 ---- > else if (*p<1000) > repsize += 2+3+1; > + #ifndef Py_UNICODE_WIDE > + else > + repsize += 2+4+1; > + #else > else if (*p<10000) > repsize += 2+4+1; > *************** > *** 2414,2417 **** > --- 2426,2430 ---- > else > repsize += 2+7+1; > + #endif > } > requiredsize = respos+repsize+(endp-collend); -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 19 2003) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From perky at i18n.org Fri Dec 19 04:23:09 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Fri Dec 19 04:23:20 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.204, 2.205 In-Reply-To: <3FE2B723.9060108@egenix.com> References: <3FE2B723.9060108@egenix.com> Message-ID: <20031219092309.GA81703@i18n.org> On Fri, Dec 19, 2003 at 09:30:27AM +0100, M.-A. Lemburg wrote: > perky@users.sourceforge.net wrote: > >Update of /cvsroot/python/python/dist/src/Objects > >In directory sc8-pr-cvs1:/tmp/cvs-serv1651/Objects > > > >Modified Files: > > unicodeobject.c > >Log Message: > >SF #859573: Reduce compiler warnings on gcc 3.2 and above. > > > >Index: unicodeobject.c > >*************** > >*** 2204,2208 **** > > > > /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ > >! if (size == 1 && *(unsigned char*)s < 256) { > > Py_UNICODE r = *(unsigned char*)s; > > return PyUnicode_FromUnicode(&r, 1); > >--- 2212,2216 ---- > > > > /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ > >! if (size == 1) { > > Py_UNICODE r = *(unsigned char*)s; > > return PyUnicode_FromUnicode(&r, 1); > > This "fix" doesn't look right. Please check. > gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Objects/unicodeobject.o Objects/unicodeobject.c Objects/unicodeobject.c: In function `PyUnicodeUCS2_DecodeLatin1': Objects/unicodeobject.c:2214: warning: comparison is always true due to limited range of data type AFAIK, *(unsigned char*)s is always smaller than 256. Also decoding latin-1 can be done by just casting it into Py_UNICODE. I'm sorry but can you explain more? Hye-Shik From tismer at tismer.com Fri Dec 19 04:32:37 2003 From: tismer at tismer.com (Christian Tismer) Date: Fri Dec 19 04:31:08 2003 Subject: [Python-Dev] Last chance! (was: Does anybody really use frame->f_tstate ?) In-Reply-To: <3FDE6D35.3090100@tismer.com> References: <3FDE6D35.3090100@tismer.com> Message-ID: <3FE2C5B5.8080208@tismer.com> Dear Python community, since I didn't get *any* reply to this request, either the request was bad or there is really nobody using f_tstate in a way that makes it urgent to keep. I will wait a few hours and then make the change to Stackless, and I'd like to propose to do the same to the Python core. Christian Tismer wrote: > Hi colleagues, > > this is my second attempt to get rid of the f_tstate field > in frames. I need to find every user of this field. > > What am I talking about? > Well, Python always has a thread state variable which > is a unique handle to the current thread. This variable > is accessed in many places, and there exists a fast macro > to get at it. > Every executing Python frame also gets a copy on creation. > In some cases, this frame->f_tstate field is used, > in other cases the current tstate variable is used. > > If this sounds foreign to you, please stop reading here. > > ------------------------------------------------------------- > > I would like to get rid of the frame->f_tstate, and I'm trying > to find out if there is a need for it. I don't need it, > for Stackless, it is the opposite, it disturbs. > > There was a small thread about this in June this year, where > Guido convinced me that it is possible to create a traceback > on a frame that comes from a different thread. > > http://mail.python.org/pipermail/python-dev/2003-June/036254.html > > Ok, this is in fact possible, although I don't think > anybody has a need for this. > > My question to all extension writers is this: > If you use frame->f_tstate at all, do you use it just > because it is handy, or do you want to use it for > some other purpose? > > One purpose could be that you really want to create a traceback > on a different than the current thread. I have never seen this, > but who knows, so that's why I'm asking the Python world. > > In most cases, a traceback will be created on a frame > that is currently processd or just has been processed. > Accessing a frame of a different thread that is being processed > might make sense for special debugger cases. > > My proposal is > -------------- > > a) change semantics of PytraceBack_Here to use the current tstate. > > b) if such a special purpose exists, create a new function for it. > > c) if urgent, different needs exist to keep f_tstate, > then let's forget about this proposal. > > Especially for Stackless, I'd be keen of getting rid of this. > > thanks for input -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From fredrik at pythonware.com Fri Dec 19 05:03:41 2003 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri Dec 19 05:03:53 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 References: <3FE2B723.9060108@egenix.com> <20031219092309.GA81703@i18n.org> Message-ID: Hye-Shik Chang wrote: > AFAIK, *(unsigned char*)s is always smaller than 256. except when it isn't. see the ANSI C spec for details. From mal at egenix.com Fri Dec 19 05:18:28 2003 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Dec 19 05:18:42 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects unicodeobject.c, 2.204, 2.205 In-Reply-To: <20031219092309.GA81703@i18n.org> References: <3FE2B723.9060108@egenix.com> <20031219092309.GA81703@i18n.org> Message-ID: <3FE2D074.90000@egenix.com> Hye-Shik Chang wrote: > On Fri, Dec 19, 2003 at 09:30:27AM +0100, M.-A. Lemburg wrote: > >>perky@users.sourceforge.net wrote: >> >>>Update of /cvsroot/python/python/dist/src/Objects >>>In directory sc8-pr-cvs1:/tmp/cvs-serv1651/Objects >>> >>>Modified Files: >>> unicodeobject.c >>>Log Message: >>>SF #859573: Reduce compiler warnings on gcc 3.2 and above. >>> >>>Index: unicodeobject.c >>>*************** >>>*** 2204,2208 **** >>> >>> /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ >>>! if (size == 1 && *(unsigned char*)s < 256) { >>> Py_UNICODE r = *(unsigned char*)s; >>> return PyUnicode_FromUnicode(&r, 1); >>>--- 2212,2216 ---- >>> >>> /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ >>>! if (size == 1) { >>> Py_UNICODE r = *(unsigned char*)s; >>> return PyUnicode_FromUnicode(&r, 1); >> >>This "fix" doesn't look right. Please check. >> > > gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Objects/unicodeobject.o Objects/unicodeobject.c > Objects/unicodeobject.c: In function `PyUnicodeUCS2_DecodeLatin1': > Objects/unicodeobject.c:2214: warning: comparison is always true due to limited range of data type > > AFAIK, *(unsigned char*)s is always smaller than 256. > Also decoding latin-1 can be done by just casting it into Py_UNICODE. You are right. I was thinking that there was some reason we needed this to get Unicode working on Crays, but looking at the CVS log, this was probably just the result of adjusting Martin's single character sharing code to work for Latin-1 rather than just ASCII characters. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 19 2003) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From 3gc6p6iqz at aol.com Fri Dec 19 11:23:03 2003 From: 3gc6p6iqz at aol.com (Nichole Villalobos) Date: Fri Dec 19 06:22:00 2003 Subject: [Python-Dev] Python-announce-list-request ocb orzmsmz fur Message-ID: Hi Python-announce-list-request , I have a picture online now. I just want someone to know me before they see me. I just feel better that way. (but believe me you wont be disappointed) Well i am 22 years old. I have a very out going personality. I love to meet new people, i am on the varsity cheerleading squad. I like the little romantic stuff and love to be swept off my feet and suprised. I am currently looking for a relationship. All around if i were to use 3 words to describe me id say Outgoing, sexy, and spontaneous :) And if u wanna chat or get to know me. If u really like what u see. youll do more than just send me a note. Talk to you soon I hope... :) Carolynn ps. my friend Maria is on with me as well. http://seeingnoone.com/confirm/?oc=52211091 I do not wish to go on any blind dates at all, now or in the future: http://seeingnoone.com/remove/?oc=1 m393nnm3m3rem wv From perky at i18n.org Fri Dec 19 06:59:52 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Fri Dec 19 06:59:59 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: <20031219092309.GA81703@i18n.org> Message-ID: <20031219115952.GA84061@i18n.org> On Fri, Dec 19, 2003 at 11:03:41AM +0100, Fredrik Lundh wrote: > Hye-Shik Chang wrote: > > > AFAIK, *(unsigned char*)s is always smaller than 256. > > except when it isn't. see the ANSI C spec for details. > Ah. I found. I'm very surprised for that. Thank you! :-) BTW, do we really support architectures with 9bits-sized char? Hye-Shik From mwh at python.net Fri Dec 19 07:10:11 2003 From: mwh at python.net (Michael Hudson) Date: Fri Dec 19 07:10:15 2003 Subject: [Python-checkins] Re: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <20031219115952.GA84061@i18n.org> (Hye-Shik Chang's message of "Fri, 19 Dec 2003 20:59:52 +0900") References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> Message-ID: <2mbrq5f1p8.fsf@starship.python.net> Hye-Shik Chang writes: > On Fri, Dec 19, 2003 at 11:03:41AM +0100, Fredrik Lundh wrote: >> Hye-Shik Chang wrote: >> >> > AFAIK, *(unsigned char*)s is always smaller than 256. >> >> except when it isn't. see the ANSI C spec for details. >> > > Ah. I found. I'm very surprised for that. Thank you! :-) > BTW, do we really support architectures with 9bits-sized char? On some kinds of Cray that Python has been built on in the past, I think the smallest addressable unit of memory is 64 bits. So, not quite 96, but getting on that way. I don't think we want to make the lives of people porting to such architectures any harder than it already is... Cheers, mwh -- Need to Know is usually an interesting UK digest of things that happened last week or might happen next week. [...] This week, nothing happened, and we don't care. -- NTK Now, 2000-12-29, http://www.ntk.net/ From ncoghlan at iinet.net.au Fri Dec 19 07:54:16 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Dec 19 07:54:22 2003 Subject: [Python-checkins] Re: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <2mbrq5f1p8.fsf@starship.python.net> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> <2mbrq5f1p8.fsf@starship.python.net> Message-ID: <3FE2F4F8.80507@iinet.net.au> Michael Hudson wrote: > Hye-Shik Chang writes: [...] >>BTW, do we really support architectures with 9bits-sized char? [...] > I don't think we want to make the > lives of people porting to such architectures any harder than it > already is... TI make chips where the smallest addressable unit is 16-bits and sizeof(char) == sizeof(int) == 16 bits == 1 byte due to the way the C standard is written. I don't think Python is ported to any such chip at present (the one I use is a DSP, and I would seriously question the sanity of anyone who tried to run Python on one of these critters), but it's a possibility that shouldn't be ignored. Porting to such a machine would be rather entertaining (sizeof() gets a _lot_ of work in the code for that DSP). Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From skip at pobox.com Fri Dec 19 08:03:00 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 19 08:03:05 2003 Subject: [Python-checkins] Re: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <3FE2F4F8.80507@iinet.net.au> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> <2mbrq5f1p8.fsf@starship.python.net> <3FE2F4F8.80507@iinet.net.au> Message-ID: <16354.63236.775441.319110@montanaro.dyndns.org> Nick> Michael Hudson wrote: >> Hye-Shik Chang writes: Nick> [...] >>> BTW, do we really support architectures with 9bits-sized char? Nick> [...] >> I don't think we want to make the lives of people porting to such >> architectures any harder than it already is... Nick> TI make chips where the smallest addressable unit is 16-bits and Nick> sizeof(char) == sizeof(int) == 16 bits == 1 byte due to the way Nick> the C standard is written. It seems to me the right thing to do is to cook up a test in the configure script which checks the number of bits in an unsigned char and sets a cpp macro which the code in question then uses to compile the fast case for 8-bit chars and the slow case otherwise. Skip From ncoghlan at iinet.net.au Fri Dec 19 08:07:38 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Dec 19 08:08:50 2003 Subject: [Python-Dev] rexec.py unuseable In-Reply-To: <20031218225748.GE2234@lkcl.net> References: <20031215191023.GB26055@lkcl.net> <1071524194.15985.660.camel@localhost.localdomain> <20031215233639.GE26055@lkcl.net> <20031216195525.GJ17021@lkcl.net> <20031216212312.GL17021@lkcl.net> <20031218092922.GF6324@lkcl.net> <20031218225748.GE2234@lkcl.net> Message-ID: <3FE2F81A.7090802@iinet.net.au> Luke Kenneth Casson Leighton wrote: > in some ways, the longer this is left, the harder it is going to > be to retrospectively bolt on. > > there's an adage that says security cannot be easily added in, it > has to be designed in from the start. This is very true, but it hurts an ACL-based approach even worse than it hurts a capabilities based one. To get capabilities to work, the question is: how do we construct an environment where 'builtins' and all other objects passed to code in that environment have been suitably restricted to prevent malicious code from causing damage. The original objects, which are never made available to the untrusted code, don't need to care about trust issues - they just keep working as they always have. To get ACL's to work, _everything_ in Python has to care about trust issues, as they have to know that they should be checking for the existence of an ACL. I can't even begin to imagine how those ACL's might be managed effectively, but I can imagine constructing a special execution environment which only allowed 'safe' objects to be passed in. A 'safe' object would be one of the restricted builtins, or objects able to be constructed using only that restricted set of builtins. The major issue comes in dealing with Python's introspection capabilities without making them completely useless (then again, perhaps 'restricted, with almost no introspection' would be an improvement over 'no restricted mode'. Anyway, despite either approach being Python 3.0 material, the capability method at least seems conceptually possible - deleting entries out of Python namespace dictionaries is a fairly straightforward activity, as is substituting a new implementation for the old 'unsafe' implementation when we want to switch to 'restricted' mode. Whereas handling ACL's would be a completely new approach that spreads its tentacles through much of the CPython source code. For code, capabilities just make more sense - if they can't use it, don't even let them know it's there. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From mcherm at mcherm.com Fri Dec 19 08:11:00 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Fri Dec 19 08:11:05 2003 Subject: [Python-Dev] Importing packages from command line Message-ID: <1071839460.3fe2f8e4dbbad@mcherm.com> Dmitry Vasiliev writes: > Just an idea... > > 1. python -p package_name > > Equivalent to: > > import package_name (1) Sourceforge is a great place to request feature enhancements. Suggestions made on this mailing list is likely to be forgotten sooner or later. (2) Can you explain WHY you would want this feature? Is there some use case? I would prefer NOT to have this, because right now if I'm reading code and it uses "package_name.someFunction()" I can scan upward for "package_name" to find out what it's using. With command line imports, I couldn't do that. So unless you've got a good reason for it, I personally wouldn't want this feature. -- Michael Chermside From skip at pobox.com Fri Dec 19 08:20:52 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 19 08:20:49 2003 Subject: [Python-checkins] Re: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <16354.63236.775441.319110@montanaro.dyndns.org> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> <2mbrq5f1p8.fsf@starship.python.net> <3FE2F4F8.80507@iinet.net.au> <16354.63236.775441.319110@montanaro.dyndns.org> Message-ID: <16354.64308.552430.331674@montanaro.dyndns.org> Skip> It seems to me the right thing to do is to cook up a test in the Skip> configure script which checks the number of bits in an unsigned Skip> char ... Better yet, let's use CHAR_BIT: #if defined(CHAR_BIT) && CHAR_BIT == 8 ... fast case ... #else ... slow case ... #endif From ark at acm.org Fri Dec 19 09:00:07 2003 From: ark at acm.org (Andrew Koenig) Date: Fri Dec 19 09:00:02 2003 Subject: [Python-Dev] 2.3.3 released? Message-ID: <007d01c3c638$69074c20$6402a8c0@arkdesktop> It appears that the main python.org page is announcing the release of 2.3.3 final, but the release page http://www.python.org/2.3.3/ does not show a place to download the Windows executable. Have I caught it in mid-transition? From fdrake at acm.org Fri Dec 19 09:09:57 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Dec 19 09:10:04 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <007d01c3c638$69074c20$6402a8c0@arkdesktop> References: <007d01c3c638$69074c20$6402a8c0@arkdesktop> Message-ID: <16355.1717.391064.71007@sftp.fdrake.net> Andrew Koenig writes: > It appears that the main python.org page is announcing the release of 2.3.3 > final, but the release page http://www.python.org/2.3.3/ does not show a > place to download the Windows executable. The Windows installer will be added when it's ready, then the general announcement will be sent. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From skip at pobox.com Fri Dec 19 09:14:59 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 19 09:15:00 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <007d01c3c638$69074c20$6402a8c0@arkdesktop> References: <007d01c3c638$69074c20$6402a8c0@arkdesktop> Message-ID: <16355.2019.340005.818253@montanaro.dyndns.org> Andrew> [2.3.3 release teasers] Andrew> Have I caught it in mid-transition? Yes, I believe so. I saw a bunch of checkins related to that, but it appears all the bits have not yet been pushed into place. Skip From fdrake at acm.org Fri Dec 19 09:21:46 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Dec 19 09:21:53 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <16355.2019.340005.818253@montanaro.dyndns.org> References: <007d01c3c638$69074c20$6402a8c0@arkdesktop> <16355.2019.340005.818253@montanaro.dyndns.org> Message-ID: <16355.2426.281597.99584@sftp.fdrake.net> Skip Montanaro writes: > Yes, I believe so. I saw a bunch of checkins related to that, but it > appears all the bits have not yet been pushed into place. I made the changes to the doc/ area on python.org last night, and everything is up on the site. I think the 2.3.3/ area is updated as well, the Windows installer just isn't ready yet. It'll be there when it is. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From theller at python.net Fri Dec 19 09:40:30 2003 From: theller at python.net (Thomas Heller) Date: Fri Dec 19 09:40:36 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <16355.2426.281597.99584@sftp.fdrake.net> (Fred L. Drake, Jr.'s message of "Fri, 19 Dec 2003 09:21:46 -0500") References: <007d01c3c638$69074c20$6402a8c0@arkdesktop> <16355.2019.340005.818253@montanaro.dyndns.org> <16355.2426.281597.99584@sftp.fdrake.net> Message-ID: "Fred L. Drake, Jr." writes: > Skip Montanaro writes: > > Yes, I believe so. I saw a bunch of checkins related to that, but it > > appears all the bits have not yet been pushed into place. > > I made the changes to the doc/ area on python.org last night, and > everything is up on the site. I think the 2.3.3/ area is updated as > well, the Windows installer just isn't ready yet. It'll be there when > it is. Hm, I was waiting for Anthony in IRC #python-dev to upload the windows installer. Sorry if I misunderstood something... Thomas From anthony at interlink.com.au Fri Dec 19 10:29:23 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Dec 19 10:29:51 2003 Subject: [Python-Dev] 2.3.3 released? In-Reply-To: <007d01c3c638$69074c20$6402a8c0@arkdesktop> Message-ID: <200312191529.hBJFTN8Y009795@localhost.localdomain> >>> "Andrew Koenig" wrote > It appears that the main python.org page is announcing the release of 2.3.3 > final, but the release page http://www.python.org/2.3.3/ does not show a > place to download the Windows executable. > > Have I caught it in mid-transition? Yep, timezones make it tricky to manage this stuff sometimes. I'll send the release announcement in the morning when I've checked it's all good. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From guido at python.org Fri Dec 19 10:31:42 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 19 10:31:51 2003 Subject: [Python-Dev] Last chance! (was: Does anybody really use frame->f_tstate ?) In-Reply-To: Your message of "Fri, 19 Dec 2003 10:32:37 +0100." <3FE2C5B5.8080208@tismer.com> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> Message-ID: <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> > since I didn't get *any* reply to this request, > either the request was bad or there is really > nobody using f_tstate in a way that makes it > urgent to keep. > I will wait a few hours and then make the change > to Stackless, and I'd like to propose to do the > same to the Python core. I saved the message, but haven't had the time yet to think things through. I *did* notice at least one case where using f_tstate might actually be a mistake: theoretically it's possible that two or more threads alternate calling next() on a generator (if they wrap it in a critical section); AFAICT the f_tstate is never updated. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Dec 19 10:39:45 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 19 10:39:49 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: Your message of "Fri, 19 Dec 2003 20:59:52 +0900." <20031219115952.GA84061@i18n.org> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> Message-ID: <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> > > > AFAIK, *(unsigned char*)s is always smaller than 256. > > > > except when it isn't. see the ANSI C spec for details. > > Ah. I found. I'm very surprised for that. Thank you! :-) > BTW, do we really support architectures with 9bits-sized char? I would expect that a lot of our code assumes 8-bit characters, and I personally wouldn't mind if Python was limited to such platforms. They aren't very important for attracting new users, and certainly they don't seem to be a growing kind of platform... (Probably because so much other software makes the same assumption. :-) So IMO your fix is fine. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Fri Dec 19 10:40:13 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 19 10:40:19 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <20031219115952.GA84061@i18n.org> Message-ID: [Hye-Shik Chang] > BTW, do we really support architectures with 9bits-sized char? I don't think so. There are assumptions that a char is 8 bits scattered throughout Python's code, not so much in the context of using characters *as* characters, but more indirectly by assuming that the number of *bits* in an object of a non-char type T can be computed as sizeof(T)*8. Skip's idea of making config smarter about this is a good one, but instead of trying to "fix stuff" for a case that's probably never going to arise, and that can't really be tested anyway until it does, I'd add a block like this everywhere we know we're relying on 8-bit char: #ifdef HAS_FUNNY_SIZE_CHAR #error "The following code needs rework when a char isn't 8 bits" #endif /* A comment explaining why the following code needs rework * when a char isn't 8 bits. */ Crays are a red herring here. It's true that some Cray *hardware* can't address anything smaller than 64 bits, and that's also true of some other architectures. char is nevertheless 8 bits on all such 64-bit boxes I know of (and since I worked in a 64-bit world for 15 years, I know about most of them ). On Crays, this is achieved (albeit at major expense) in software: by *software* convention, a pointer-to-char stores the byte offset in the *most*-significant 3 bits of a pointer, and long-winded generated coded picks that part at runtime, loading or storing 8 bytes at a time (the HW can't do less than that), shifting and masking and or'ing to give the illusion of byte addressing for char. Some Alphas do something similar, but that HW's loads and stores simply ignore the last 3 bits of a memory address, and the CPU has special-purpose instructions to help generated code do the subsequent extraction and insertion of 8-bit chunks efficiently and succinctly. From tismer at tismer.com Fri Dec 19 12:49:15 2003 From: tismer at tismer.com (Christian Tismer) Date: Fri Dec 19 12:47:42 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> Message-ID: <3FE33A1B.5030903@tismer.com> Guido van Rossum wrote: ... > I saved the message, but haven't had the time yet to think things > through. > > I *did* notice at least one case where using f_tstate might actually > be a mistake: theoretically it's possible that two or more threads > alternate calling next() on a generator (if they wrap it in a critical > section); AFAICT the f_tstate is never updated. Right, f_tstate is never updated. I think there is another inconsistent situation, which can be created easily. If a generator is run by a different thread than it's creator, then the frame is run in that other thread. eval_fame correctly uses tstate, but if tracing is activated, call_trace uses frame->f_tstate for no obvious reason, which will probably mess up the tracing flags of the wrong thread. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From blink at msn.com Sat Dec 20 05:49:02 2003 From: blink at msn.com (Reuben Crenshaw) Date: Fri Dec 19 15:55:28 2003 Subject: [Python-Dev] You tried diets and workouts, add this and be amazed!! bpvwvczpyztptp Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031220/9fee7bad/attachment.html From Jack.Jansen at cwi.nl Fri Dec 19 17:57:02 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Fri Dec 19 17:57:07 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> References: <20031219092309.GA81703@i18n.org> <20031219115952.GA84061@i18n.org> <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> Message-ID: On 19-dec-03, at 16:39, Guido van Rossum wrote: >>>> AFAIK, *(unsigned char*)s is always smaller than 256. >>> >>> except when it isn't. see the ANSI C spec for details. >> >> Ah. I found. I'm very surprised for that. Thank you! :-) >> BTW, do we really support architectures with 9bits-sized char? > > I would expect that a lot of our code assumes 8-bit characters, and I > personally wouldn't mind if Python was limited to such platforms. Then there is a lot of code that could be tuned for this. Since I'm using gcc 3.3 (which came with OSX 10.3) I get lots of warnings about comparisons that are always true due to the size of the operands. I looked at a couple of these and I think they were all char-related. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From jack at performancedrivers.com Fri Dec 19 16:30:31 2003 From: jack at performancedrivers.com (Jack Diederich) Date: Fri Dec 19 22:03:52 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031218031645.GH473@frobozz> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> Message-ID: <20031219213031.GD1246@wopr> On Thu, Dec 18, 2003 at 02:16:45PM +1100, Andrew Bennetts wrote: > On Wed, Dec 17, 2003 at 10:08:21PM -0500, Barry Warsaw wrote: > > On Wed, 2003-12-17 at 22:04, Andrew Bennetts wrote: > > > On Wed, Dec 17, 2003 at 09:33:43PM -0500, Barry Warsaw wrote: > > > > > > > > That's why when I tend to think about this, I start wanting a way to > > > > spell "definitely give me the global one, no matter what". IOW, I feel > > > > like I want a way to bypass relative module lookups. > > > > > > Alternatively, maybe what you want is a way to say "definitely give me the > > > standard library one, no matter what", e.g. > > > > > > from stdlib import codecs > > > > Interesting. I see the subtle distinction. One searches only the > > standard library, the other searches every directory on sys.path. I'm > > not sure I'd have much need to restrict the search to just the standard > > library. Plus, would that include site-packages? > > I should have been clearer. What I'm basically proposing is to make > site-package called "stdlib", leaving a default python install with a very > uncluttered top-level namespace, thus reducing unexpected name conflicts. I'm at least a little confused with what people want, as far as I can tell import semantics have three things * what module to import * what name to give that import * where to look for it The 'from' keyword seems to be getting overloaded to mean both what module and where to find it, which the 'stdlib' tries to resolve. Most of the discussion has been about how to improve 'where to look' I like syntax that reads most important left-to-right, so what about from MODULE import NAMES as RENAME searching HOW or import NAMES as RENAME from MODULE searching HOW searching could be 'asbolute' 'relative' or any of the other suggested words. If you wanted to get fancy it could be a list, if it isn't a list people who truly care could cascade try/except on ImportError. -jackdied ps, I like the second version better but it is less like the current version, which is a drawback. From barry at python.org Fri Dec 19 23:13:15 2003 From: barry at python.org (Barry Warsaw) Date: Fri Dec 19 23:13:26 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031219213031.GD1246@wopr> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> Message-ID: <1071893594.1718.166.camel@anthem> On Fri, 2003-12-19 at 16:30, Jack Diederich wrote: > I'm at least a little confused with what people want, as far as I can tell > import semantics have three things > * what module to import > * what name to give that import > * where to look for it > > The 'from' keyword seems to be getting overloaded to mean both what module and > where to find it, which the 'stdlib' tries to resolve. Most of the > discussion has been about how to improve 'where to look' Very interesting observation. > I like syntax that reads most important left-to-right, so what about > from MODULE import NAMES as RENAME searching HOW > or > import NAMES as RENAME from MODULE searching HOW > > searching could be 'asbolute' 'relative' or any of the other suggested > words. If you wanted to get fancy it could be a list, if it isn't a list > people who truly care could cascade try/except on ImportError. I want to think about that more, but at first glance, it has some appeal. Neat! -Barry From tim.one at comcast.net Fri Dec 19 23:32:45 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 19 23:32:51 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1071893594.1718.166.camel@anthem> Message-ID: [Jack Diederich] >> ... >> I like syntax that reads most important left-to-right, so what about >> from MODULE import NAMES as RENAME searching HOW >> or >> import NAMES as RENAME from MODULE searching HOW >> >> searching could be 'asbolute' 'relative' or any of the other >> suggested words. If you wanted to get fancy it could be a list, if >> it isn't a list people who truly care could cascade try/except on >> ImportError. [Barry Warsaw] > I want to think about that more, but at first glance, it has some > appeal. Neat! Ditto. I think Jack's on to something there. I particularly like the second form, as it's an ongoing irritation to me that some imports begin with "from" and others with "import". In the current "from" form, the most important thing to me is almost never the module I'm importing the thing from, but (of course) the thing I'm importing. Going from import math to import sin from math (or vice versa) is also an easier edit than from import math to from math import sin (because the former doesn't require transposition). Adding a distinct clause to influence search details seems downright Pythonic. Maybe *too* Pythonic . From tim.one at comcast.net Sat Dec 20 00:22:22 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 00:22:27 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > I would expect that a lot of our code assumes 8-bit characters, and I > personally wouldn't mind if Python was limited to such platforms. > They aren't very important for attracting new users, and certainly > they don't seem to be a growing kind of platform... (Probably because > so much other software makes the same assumption. :-) Fine by me too. The first mainframe I used was a Univac 1108. There were a *lot* of competing HW architectures at that time, and manufacturers didn't agree about character size any more than they agreed about floating-point format or semantics, or the natural size of "a word". Univac was forward-looking, though: they didn't want their hardware to become obsolete if a different character size than the one they preferred clicked, so a control bit in the CPU could be set to treat their 36-bit words as either 6 6-bit characters, or as 4 9-bit characters. It worked! We're *still* equally comfortable with 6-bit bytes as with 9-bit bytes . I was betting on 6-bit bytes at the time, because that also worked well with CDC's 60-bit words. FORTRAN didn't even admit to the existence of lower case at the time, so 64 characters was way more than enough for anything anyone really needed to say to a computer. half-the-bits-in-these-new-fangled-bytes-are-just-wasted-ly y'rs - tim From whisper at oz.net Sat Dec 20 00:32:26 2003 From: whisper at oz.net (David LeBlanc) Date: Sat Dec 20 00:51:36 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: Message-ID: > [Guido] > > I would expect that a lot of our code assumes 8-bit characters, and I > > personally wouldn't mind if Python was limited to such platforms. > > They aren't very important for attracting new users, and certainly > > they don't seem to be a growing kind of platform... (Probably because > > so much other software makes the same assumption. :-) > > Fine by me too. > > The first mainframe I used was a Univac 1108. There were a *lot* of > competing HW architectures at that time, and manufacturers didn't agree > about character size any more than they agreed about floating-point format > or semantics, or the natural size of "a word". Univac was > forward-looking, > though: they didn't want their hardware to become obsolete if a different > character size than the one they preferred clicked, so a control > bit in the > CPU could be set to treat their 36-bit words as either 6 6-bit characters, > or as 4 9-bit characters. It worked! We're *still* equally comfortable > with 6-bit bytes as with 9-bit bytes . > > I was betting on 6-bit bytes at the time, because that also > worked well with > CDC's 60-bit words. FORTRAN didn't even admit to the existence of lower > case at the time, so 64 characters was way more than enough for anything > anyone really needed to say to a computer. > > half-the-bits-in-these-new-fangled-bytes-are-just-wasted-ly y'rs - tim > I would think the lesson to be learned from this is that one should not lock the software into any particular number of bits per character. The coming flood of 64 bit machines could make 16 bit unicode attractive. It's an ever more global world and "we" should keep in mind that in the next decade most of the world's programming is going to be done in India and China if American corporations have their way. soon-to-be-looking-for-the-carton-the-mainframe-came-in-to-live-ingly-y'rs Dave LeBlanc Seattle, WA USA From zack at codesourcery.com Fri Dec 19 13:08:52 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Sat Dec 20 02:35:53 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "18 Dec 2003 22:28:50 +0100") References: <87y8tbezcq.fsf@codesourcery.com> Message-ID: <87wu8sy91n.fsf@egil.codesourcery.com> martin@v.loewis.de (Martin v. L?wis) writes: > "Zack Weinberg" writes: > >> > This is what leads to the bletcherous >> > >> > do { >> > the real macro guts go here >> > } while(0) >> > >> > form of macro definition. Py_DECREF could be rewritten in that form -- but >> > I've puked enough for one day . >> >> Should, not just could. Yeah, it's bletcherous, but it's the only >> way to be sure. (Short of inline functions.) > > Not true. As Tim explains, there is no possible application of the > macro which gets misinterpreted. We don't have a single if-statement, > we have an if-else-statement in the macro. That never leads to the > dangling else problem. It does, however, lead to a different problem: if (condition) Py_DECREF (foo); else // oops, syntax error here ... In general I consider it appropriate to wrap _all_ macros that don't return a value in do { ... } while (0), because that way you don't have to worry about which of them truly need it. zw From arigo at tunes.org Sat Dec 20 03:43:57 2003 From: arigo at tunes.org (Armin Rigo) Date: Sat Dec 20 03:48:49 2003 Subject: [Python-Dev] Re: [pypy-dev] Last chance! (was: Does anybody really use frame->f_tstate ?) In-Reply-To: <3FE2C5B5.8080208@tismer.com> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> Message-ID: <20031220084357.GA25998@vicky.ecs.soton.ac.uk> Hello Christian, On Fri, Dec 19, 2003 at 10:32:37AM +0100, Christian Tismer wrote: > since I didn't get *any* reply to this request, > either the request was bad or there is really > nobody using f_tstate in a way that makes it > urgent to keep. Sorry for the delay, we were kind of busy this week here... There are references to f_tstate in Psyco but nothing critical. Using PyThreadState_GET() instead should be fine. A bientot, Armin. From ncoghlan at iinet.net.au Sat Dec 20 04:09:48 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Dec 20 04:11:46 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: Message-ID: <3FE411DC.3010401@iinet.net.au> Tim Peters wrote: > [Hye-Shik Chang] > >>BTW, do we really support architectures with 9bits-sized char? [...] > Skip's idea of making config smarter about this is a good one, but instead > of trying to "fix stuff" for a case that's probably never going to arise, > and that can't really be tested anyway until it does, I'd add a block like > this everywhere we know we're relying on 8-bit char: > > #ifdef HAS_FUNNY_SIZE_CHAR > #error "The following code needs rework when a char isn't 8 bits" > #endif > /* A comment explaining why the following code needs rework > * when a char isn't 8 bits. > */ This would probably be appropriate for those TI DSP's I mentioned. While they genuinely have a 16-bit char type, they're also intended for use as co-processors, rather than as the main controller for an application. That is, a more standard CPU should be used to handle the general application programming, while the DSP is used for the serious number crunching (that's what it is made for, after all). Anyone who _thinks_ they want to run Python on the DSP core almost certainly needs to have a long hard think about their system design. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From martin at v.loewis.de Sat Dec 20 05:15:21 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Dec 20 05:17:51 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031219012347.GB4692@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> Message-ID: Jeff Epler writes: > > Can you find out what $$ is, and what the PIDs and thread IDs of all > > participating threads are? > > I'm not sure what all information I should try to gather for you. Let me > know if you think this is enough to file a bug report with... I changed > the example to make it clearer that it's the subprocess ignoring the > signal that is the problem, not anything in Python that is taking time > to notice the death of a child process. That is an important observation; signals that are blocked in the parent process will be blocked in the child process as well. I'm not sure what to do about this: We apparently *want* the signals blocked in the thread, but we don't want them to be blocked in the process invoked through system(). Proposals are welcome. Regards, Martin From martin at v.loewis.de Sat Dec 20 05:23:43 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Dec 20 05:24:56 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: Message-ID: "David LeBlanc" writes: > I would think the lesson to be learned from this is that one should > not lock the software into any particular number of bits per > character. The coming flood of 64 bit machines could make 16 bit > unicode attractive. You are talking about an entirely different issue here. This thread is about the number of bits in a "char", which, in C, is the same thing as a "byte". The number of bits for a "character" is independent. 16-bit Unicode is attractive already, although it is dying, to make place for 32-bit Unicode. However, new 64-bit architectures will make sure they support an 8-bit data type, and compiler vendors will make sure that "char" maps to that 8-bit data type (most likely, they also will make char signed by default). There is just too much software that breaks if you could not address bytes anymore. Primarily, the entire networking interfaces would break down, which is a risk that new architectures are unlikely to take. > It's an ever more global world and "we" should keep in mind that in > the next decade most of the world's programming is going to be done > in India and China if American corporations have their way. Certainly, but unrelated to the issue at hand. Regards, Martin From martin at v.loewis.de Sat Dec 20 05:31:18 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sat Dec 20 05:40:20 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87wu8sy91n.fsf@egil.codesourcery.com> References: <87y8tbezcq.fsf@codesourcery.com> <87wu8sy91n.fsf@egil.codesourcery.com> Message-ID: "Zack Weinberg" writes: > It does, however, lead to a different problem: > > if (condition) > Py_DECREF (foo); > else // oops, syntax error here How so? This expands to if(condition) if(cond2)action1; else action2; else ... This is perfectly well-formed C, and it groups as my indentation suggests. There is no real problem with this macro. It is just gcc complaining falsely. > In general I consider it appropriate to wrap _all_ macros that don't > return a value in do { ... } while (0), because that way you don't > have to worry about which of them truly need it. Hmm. Yes, this could be done - but it is unfortunate that this needs to be done only to silence gcc. Regards, Martin From lkcl at lkcl.net Sat Dec 20 06:21:20 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 06:21:00 2003 Subject: [Python-Dev] Capabilities - published interfaces Message-ID: <20031220112120.GE1933@lkcl.net> following on, with a change of subject, from the rexec / acls discussions, i believe that i have finally worked out why acls are not the preferred way forward, leaving speed disadvantages aside. ACLs need to be applied subtly and explicitly on every important type of object. if they're not designed in at the time they are a PAIN to retro-fit. ACLs need to be applied subtly because if they are applied unsubtly you end up with more gates than you do green pastures - more roundabouts than roads (viz, milton keynes). ACLs need to be applied explicitly because of their nature: you have something (it could be the right to "see" something) and you try do _do_ something and are explicitly told whether it worked or not (or are explicitly fooled into believing that something cannot be "seen). from a code security review perspective, it's actually easier to review ACL-restricted code than it is capabilities-based code, but i'll expand on that later. Capabilities are like the difference between python 0.0 and python 2.5. python 0.0 is "capable" of doing certain things - however python 2.5 is "capable" of doing a heck of a lot more, simply because the functionality in python 0.0 doesn't _exist_. The principle behind Capabilities is therefore to simply _not give_ python 2.5 users _any_ means to use, view, see, certain functionality i.e. make it look like it doesn't exist. in other words [and this is the bit that i could _really_ have done with, two days ago], when a python user accessess __globals__, __builtins__, __locals__, an object's __class__, they find that the functionality they may be looking for literally _isn't there_. it's as if someone had recompiled python 2.5 #ifdeffing out a whole boat-load of code and forgot to include some modules. the advantage of Capabilities over ACLs is that there's no chance of making a mistake on the ACLs, because access to something that, as far as the user is concerned, doesn't _exist_, is just a non-issue. the other advantage is speed. the _dis_advantage of Capabilities over ACLs comes when an attempt to combine unrestricted Capability code with restricted Capability code in the same codebase is made, and a security audit is called. the nature of restricted Capabilities is such that it is "by omission" that you achieve security. how the heck are you going to do a security audit looking for things that are omitted, rather than included, especially in amongst a full set of code that has the full functionality? by contrast, with ACLs, you can check everything (in a regimented fashion) and if it don't have an ACL, you can at least ask _why_ it don't have an ACL. so, with that explanation in mind, i'm going to write up a separate message outlining some of the ways in which i believe the capabilities restriction system that is already in python can be improved. l. From lkcl at lkcl.net Sat Dec 20 07:19:42 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 07:19:23 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220112120.GE1933@lkcl.net> References: <20031220112120.GE1933@lkcl.net> Message-ID: <20031220121942.GF1933@lkcl.net> capabilities: the means to make code disappear. thing is that there is a lot of code. some of the boundaries have already been highlighted where the "Capabilities" have not sufficiently been restricted. object.__class__ is a good example. Access to __class__ ------------------- the obvious solution to fix that particular problem is to turn the access to __class__ around. the problem is that by creating objects, as many __class__es are created as there are objects. considering restricting access to all of those __class__es is totally daunting and impractical. ... but look at it closely: they're all the same function name, so it's not necessarily the individual __class__es that could be restricted but the method by which __class__ is accessed. if you "feed" access to __class__ via a single function, you have a bottleneck again which can be replaced. so, the proposal is simple: create an __builtin__ function called __get_class__ through which access to __class__ is made. it takes, as a first argument, the class itself. obj.__class__ is redirected to it (__builtin__.__get_class__(obj)) to maintain code compatibility. then it becomes possible to over-ride the __get_class__ function and to "vet" it in a restricted execution mode. again, it's worth reiterating here that if it wasn't for the fact that python is run "initiated" in "unrestricted" mode and "jumps down" to "restricted" execution, this wouldn't be a problem. the reason for that is because if you ran python initiated in a file-restricted mode from startup, no unrestricted file objects with write permission would be created, such that there would be zero risk of anyone _getting_ an unrestricted file object, through which the file.__class__.__new__(file.__class__, ....) method could be called. anyway. if that is not practical (for whatever reason), then you leave it alone, and consider the function __new__ instead, which i believe is the _real_ reason why access to __class__ is considered bad. again, turning it round by creating a function __builtin__.__new__() is a way forward because that function can be restricted. Access to __builtins__ and __globals__ from the stack. ----------------------------------------------------- i don't know if this is the case, but it's worth mentioning, as background, to avoid potential confusion. there's the concept of globals, locals and there's __builtins__ in the python language. in the _implementation_ of the python language, there are c-variables which may be global, may be local: possible conceptual confusion between the implementation and the language need to be avoided! my question is: _is_ it possible to make a function call in python such that that function's "view" of what the globals, locals and __builtins__ are are COMPLETELY different from what the CALLER of that function's view of globals, locals and __builtins__? in other words, is globals() global in the python implementation, or is globals() copied on the function-call stack frame in the python implementation? [or other] what i am getting at is that it needs to be possible, if capabilities are to work, to be able to make a transition in the python language from a full-featured mode in one function call into a totally restricted mode in another function call, and to be able to come back again. _without_ the entry into the restricted mode having any damaging consequences on the full-featured mode. i note, for example, in the rexec.py code, that only self.locals is modified in the runcode() example RestrictedConsole class. why is only self.locals['__builtins__'] restricted, not self.globals? am i missing something here? surely the globals should be modified too? what happens if self.globals is reassigned? Access to interfaces -------------------- one of the things that is strangely lacking in python is the ability to restrict access to python objects, a la public, protected and private from c++. perl users find this to be utterly incomprehensible and reprehensible, especially the bits where conventions are obeyed - and followed! - about putting underscores in front of function names. from a restricted execution perspective, this is not really okay. what i propose is that two additional per-class functions (or lists) be added (two more functions later) which can ONLY be called from __init__: they are called __set_public_interface__() and __set_protected_interface__(). these functions take, as arguments, a list - or simpler yet, two variables are added which can only be initialised in __init__: __public_interface__ and __protected_interface__. basically they consist of lists of functions that can be accessed publicly and by inheriting classes respectively. the reason for not specifying a list of private functions is because it then becomes necessary to explicitly search through that list excluding things from external access, which is not the way "Capabilities" are done. if the lists are None, there are no restrictions. if the lists are empty, no functions are accessible - which is different. what i also propose is that it be possible to make a copy of a class and then _remove_ items from these two lists. it must _not_ be possible to either add to or to replace the list items or the lists themselves. there's quite a lot here. i hope you find it useful. l. From aahz at pythoncraft.com Sat Dec 20 09:08:43 2003 From: aahz at pythoncraft.com (Aahz) Date: Sat Dec 20 09:08:55 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220121942.GF1933@lkcl.net> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> Message-ID: <20031220140842.GA5627@panix.com> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > > one of the things that is strangely lacking in python is the ability > to restrict access to python objects, a la public, protected and > private from c++. > > perl users find this to be utterly incomprehensible and reprehensible, > especially the bits where conventions are obeyed - and followed! - > about putting underscores in front of function names. > > from a restricted execution perspective, this is not really okay. Yup. That really is the fundamental issue. Python has been deliberately designed to let programmers do whatever they want; it's a "let the grownups play" perspective. That's part of the reason restricted execution hasn't received more attention: it's fundamentally opposed to Python's design. Another major reason is because restricted execution from within Python can never completely solve the problem unless we devote vast amounts of effort. Consider the following two snippets of code: 100 ** 100 ** 100 [None] * (10 ** 10) The first chews up CPU; the second chews up memory. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From lkcl at lkcl.net Sat Dec 20 09:39:10 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 09:38:47 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220140842.GA5627@panix.com> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> Message-ID: <20031220143910.GA19472@lkcl.net> On Sat, Dec 20, 2003 at 09:08:43AM -0500, Aahz wrote: > On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > > > > one of the things that is strangely lacking in python is the ability > > to restrict access to python objects, a la public, protected and > > private from c++. > > > > perl users find this to be utterly incomprehensible and reprehensible, > > especially the bits where conventions are obeyed - and followed! - > > about putting underscores in front of function names. > > > > from a restricted execution perspective, this is not really okay. > > Yup. That really is the fundamental issue. Python has been deliberately > designed to let programmers do whatever they want; it's a "let the > grownups play" perspective. as python moves into a more mainstream acceptance, it becomes more of an issue to let the kiddies bash themselves with rubber hammers. [i mention this because a few years ago, a six year old bashed his dad over the head with a hammer after watching tom and jerry beat the stuffing out of each other on telly.] > Python's design. Another major reason is because restricted execution > from within Python can never completely solve the problem unless we > devote vast amounts of effort. Consider the following two snippets of > code: > > 100 ** 100 ** 100 > [None] * (10 ** 10) > > The first chews up CPU; the second chews up memory. i infer from these two observations that some of the requirements of restricted execution could be to conserve CPU and memory? l. From aahz at pythoncraft.com Sat Dec 20 10:16:29 2003 From: aahz at pythoncraft.com (Aahz) Date: Sat Dec 20 10:16:33 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220143910.GA19472@lkcl.net> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> Message-ID: <20031220151629.GA24651@panix.com> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > On Sat, Dec 20, 2003 at 09:08:43AM -0500, Aahz wrote: >> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: >>> >>> one of the things that is strangely lacking in python is the ability >>> to restrict access to python objects, a la public, protected and >>> private from c++. >>> >>> from a restricted execution perspective, this is not really okay. >> >> Yup. That really is the fundamental issue. Python has been deliberately >> designed to let programmers do whatever they want; it's a "let the >> grownups play" perspective. > > as python moves into a more mainstream acceptance, it becomes more > of an issue to let the kiddies bash themselves with rubber hammers. That's an assertion. I think to a certain extent you'll need to prove your assertion. >> Another major reason is because restricted execution from within >> Python can never completely solve the problem unless we devote vast >> amounts of effort. Consider the following two snippets of code: >> >> 100 ** 100 ** 100 >> [None] * (10 ** 10) >> >> The first chews up CPU; the second chews up memory. > > i infer from these two observations that some of the requirements > of restricted execution could be to conserve CPU and memory? Exactly. From my observations of these discussions, there are essentially only two reasons for restricted execution: * To simplify things by reducing the potential solution space * To protect a system against a hostile attacker There are already many ways to achieve the first objective in Python without restricted execution. For the second, there's really no way to succeed without help from the operating system, unless you run a server with a limited set of capabilities that does *not* include random code. There are already plenty of servers written in Python.... Supposedly there's a middle ground of untrusted but non-hostile code, but what's the point of providing support for that? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From lkcl at lkcl.net Sat Dec 20 10:30:22 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 10:30:00 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220151629.GA24651@panix.com> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> <20031220151629.GA24651@panix.com> Message-ID: <20031220153022.GB19472@lkcl.net> On Sat, Dec 20, 2003 at 10:16:29AM -0500, Aahz wrote: > > as python moves into a more mainstream acceptance, it becomes more > > of an issue to let the kiddies bash themselves with rubber hammers. > > That's an assertion. I think to a certain extent you'll need to prove > your assertion. 'm a bit worried that such proofs would involve painting large metal hammers and covering them in plastic to make them look less real. the resultant cracked skills could possibly land someone in jail. > Supposedly there's a middle ground of untrusted but non-hostile code, > but what's the point of providing support for that? the example that i gave that was because i wanted to offer a subset of python functionality to end-users such that they could run DNS lookups, pings, check a web page existed, telnet to a box, run commands and check the output. so it's running a user's python code on a server where there is a networked host being analysed by the user. what i _didn't_ want to happen was for that user to run code on the server that could damage the server and interfere with the 100 or so _other_ programs running to analyse 100 _other_ hosts. the whole point of using python was to avoid having to write an new programming language. what i came up with was very very useful. to some extent, i didn't care about things like __class__ because 1) the users weren't that bright. 2) the user's weren't that hostile. rexec fitted the requirements perfectly - and it still does: it's just been disabled and also changed into something that stops even the library functions from writing to log files. i couldn't even use the MySQLdb module which was kinda critical to the database-driven backend. l. From mwh at python.net Sat Dec 20 10:51:34 2003 From: mwh at python.net (Michael Hudson) Date: Sat Dec 20 10:51:39 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "20 Dec 2003 11:15:21 +0100") References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> Message-ID: <2mwu8rebcp.fsf@starship.python.net> martin@v.loewis.de (Martin v. L?wis) writes: > Jeff Epler writes: > >> > Can you find out what $$ is, and what the PIDs and thread IDs of all >> > participating threads are? >> >> I'm not sure what all information I should try to gather for you. Let me >> know if you think this is enough to file a bug report with... I changed >> the example to make it clearer that it's the subprocess ignoring the >> signal that is the problem, not anything in Python that is taking time >> to notice the death of a child process. > > That is an important observation; signals that are blocked in the > parent process will be blocked in the child process as well. > > I'm not sure what to do about this: We apparently *want* the signals > blocked in the thread, but we don't want them to be blocked in the > process invoked through system(). Proposals are welcome. Does pthread_atfork() help? Cheers, mwh -- We've had a lot of problems going from glibc 2.0 to glibc 2.1. People claim binary compatibility. Except for functions they don't like. -- Peter Van Eynde, comp.lang.lisp From aahz at pythoncraft.com Sat Dec 20 10:55:48 2003 From: aahz at pythoncraft.com (Aahz) Date: Sat Dec 20 10:55:53 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220153022.GB19472@lkcl.net> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> <20031220151629.GA24651@panix.com> <20031220153022.GB19472@lkcl.net> Message-ID: <20031220155548.GA7420@panix.com> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > On Sat, Dec 20, 2003 at 10:16:29AM -0500, Aahz wrote: >> >> Supposedly there's a middle ground of untrusted but non-hostile code, >> but what's the point of providing support for that? > > the example that i gave that was because i wanted to offer a subset > of python functionality to end-users such that they could run > DNS lookups, pings, check a web page existed, telnet to a box, > run commands and check the output. > > to some extent, i didn't care about things like __class__ because > 1) the users weren't that bright. > 2) the user's weren't that hostile. Yup. By "what's the point?" I didn't mean that there were no use cases; the problem is that such cases are not frequent enough to justify the effort. > rexec fitted the requirements perfectly - and it still does: it's > just been disabled and also changed into something that stops even > the library functions from writing to log files. > i couldn't even use the MySQLdb module which was kinda critical to > the database-driven backend. Well, you're free to maintain rexec as a separate project (or borrow from the still-maintained Zope system). But anything shipped as part of Python can't afford to assume your points 1) and 2). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From lkcl at lkcl.net Sat Dec 20 11:35:51 2003 From: lkcl at lkcl.net (Luke Kenneth Casson Leighton) Date: Sat Dec 20 11:35:28 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220155548.GA7420@panix.com> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> <20031220151629.GA24651@panix.com> <20031220153022.GB19472@lkcl.net> <20031220155548.GA7420@panix.com> Message-ID: <20031220163551.GC19472@lkcl.net> On Sat, Dec 20, 2003 at 10:55:48AM -0500, Aahz wrote: > > to some extent, i didn't care about things like __class__ because > > 1) the users weren't that bright. > > 2) the user's weren't that hostile. > > Yup. By "what's the point?" I didn't mean that there were no use cases; > the problem is that such cases are not frequent enough to justify the > effort. ... which is why i made some recommendations to add in the concept of run-time-defineable public and protected class interfaces. such a concept 1) fits with the principle of capabilities 2) is an enhancement that goes beyond the small requirements of restricted execution 3) offers a means through which rexec can be implemented. > > rexec fitted the requirements perfectly - and it still does: it's > > just been disabled and also changed into something that stops even > > the library functions from writing to log files. > > i couldn't even use the MySQLdb module which was kinda critical to > > the database-driven backend. > > Well, you're free to maintain rexec as a separate project (or borrow > from the still-maintained Zope system). But anything shipped as part of > Python can't afford to assume your points 1) and 2). i appreciate that. so it turns into a wishlist: a class named RExecDontUseThisItsBrokenForMostPeople and a class named RExec which simply pulls that exception. l. From aahz at pythoncraft.com Sat Dec 20 11:39:31 2003 From: aahz at pythoncraft.com (Aahz) Date: Sat Dec 20 11:39:35 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220163551.GC19472@lkcl.net> References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> <20031220140842.GA5627@panix.com> <20031220143910.GA19472@lkcl.net> <20031220151629.GA24651@panix.com> <20031220153022.GB19472@lkcl.net> <20031220155548.GA7420@panix.com> <20031220163551.GC19472@lkcl.net> Message-ID: <20031220163931.GA20862@panix.com> On Sat, Dec 20, 2003, Luke Kenneth Casson Leighton wrote: > On Sat, Dec 20, 2003 at 10:55:48AM -0500, Aahz wrote: >>Luke: >>> >>> to some extent, i didn't care about things like __class__ because >>> 1) the users weren't that bright. >>> 2) the user's weren't that hostile. >> >> Yup. By "what's the point?" I didn't mean that there were no use cases; >> the problem is that such cases are not frequent enough to justify the >> effort. > > ... which is why i made some recommendations to add in the concept > of run-time-defineable public and protected class interfaces. > > such a concept 1) fits with the principle of capabilities 2) > is an enhancement that goes beyond the small requirements of > restricted execution 3) offers a means through which rexec > can be implemented. Sure. *If* we decide to implement restricted execution. As I made clear with my comments about CPU and memory usage, there's still no consensus that adding a restricted execution environment to Python represents a good use of community resources. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From guido at python.org Fri Dec 19 13:15:24 2003 From: guido at python.org (Guido van Rossum) Date: Sat Dec 20 12:15:31 2003 Subject: [Python-Dev] Last chance! In-Reply-To: Your message of "Fri, 19 Dec 2003 18:49:15 +0100." <3FE33A1B.5030903@tismer.com> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> <3FE33A1B.5030903@tismer.com> Message-ID: <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> > > I *did* notice at least one case where using f_tstate might actually > > be a mistake: theoretically it's possible that two or more threads > > alternate calling next() on a generator (if they wrap it in a critical > > section); AFAICT the f_tstate is never updated. > > Right, f_tstate is never updated. > I think there is another inconsistent > situation, which can be created easily. > If a generator is run by a different thread > than it's creator, then the frame is run in that > other thread. eval_fame correctly uses tstate, > but if tracing is activated, call_trace uses > frame->f_tstate for no obvious reason, which > will probably mess up the tracing flags of the wrong > thread. Right. Could you dig through CVS logs to find out when f_tstate was first introduced? Maybe there's a clue about why there. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Sat Dec 20 11:27:34 2003 From: skip at pobox.com (Skip Montanaro) Date: Sat Dec 20 12:18:27 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: <200312191539.hBJFdjC27495@c-24-5-183-134.client.comcast.net> Message-ID: <16356.30838.750692.236147@montanaro.dyndns.org> Tim> [Guido] >> I would expect that a lot of our code assumes 8-bit characters, and I >> personally wouldn't mind if Python was limited to such platforms. Tim> Fine by me too. Then how about adding #if UCHAR_MAX != 255 #error "Python's source code currently assumes 8-bit characters." #endif right after the HAVE_LIMITS_H test? Skip From python at rcn.com Sat Dec 20 13:47:56 2003 From: python at rcn.com (Raymond Hettinger) Date: Sat Dec 20 13:48:26 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> Message-ID: <000201c3c729$c8ffcde0$e841fea9@oemcomputer> > > > I *did* notice at least one case where using f_tstate might actually > > > be a mistake: theoretically it's possible that two or more threads > > > alternate calling next() on a generator (if they wrap it in a critical > > > section); AFAICT the f_tstate is never updated. > > > > Right, f_tstate is never updated. > > I think there is another inconsistent > > situation, which can be created easily. > > If a generator is run by a different thread > > than it's creator, then the frame is run in that > > other thread. eval_fame correctly uses tstate, > > but if tracing is activated, call_trace uses > > frame->f_tstate for no obvious reason, which > > will probably mess up the tracing flags of the wrong > > thread. > > Right. > > Could you dig through CVS logs to find out when f_tstate was first > introduced? Maybe there's a clue about why there. Here is a head start on the research. The ceval.c use of tstate goes back to the introduction of generators in 2001. The use in traceback.c and sysmodule.c goes back to 1997 when per-thread globals were factored into a structure to support separate thread-state management. Prior to that, the history is more diffuse and harder to follow. Raymond Hettinger --------------------------------------------------------------------- Ceval.c -------- 2.252 (nascheme 21-Jun-01): PyThreadState *tstate = 2.252 (nascheme 21-Jun-01): * necessarily their creator. */ 2.255 (tim_one 23-Jun-01): Py_XINCREF(tstate->frame); 2.252 (nascheme 21-Jun-01): assert(f->f_back == NULL); 2.252 (nascheme 21-Jun-01): f->f_back = tstate->frame; 2.252 (nascheme 21-Jun-01): revision 2.255 date: 2001/06/23 05:47:56; author: tim_one; state: Exp; lines: +2 -2 gen_iternext(): Don't assume that the current thread state's frame is not NULL. I don't think it can be NULL from Python code, but if using generators via the C API I expect a NULL frame is possible. revision 2.252 date: 2001/06/21 02:41:10; author: nascheme; state: Exp; lines: +27 -14 Try to avoid creating reference cycles involving generators. Only keep a reference to f_back when its really needed. Do a little whitespace normalization as well. This whole file is a big war between tabs and spaces but now is probably not the time to reindent everything. traceback.c ------------ 2.22 (guido 05-May-97): PyThreadState *tstate = frame->f_tstate; 2.22 (guido 05-May-97): tracebackobject *oldtb = (tracebackobjec revision 2.22 date: 1997/05/05 20:56:15; author: guido; state: Exp; lines: +6 -30 Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. The story for their life a globals pre-dates what I can sysmodule.c ------------ 2.45 (guido 02-Aug-97): PyThreadState *tstate = PyThreadState_Ge 2.45 (guido 02-Aug-97): PyObject *sd = tstate->interp->sysdict; 2.57 (guido 05-Oct-99): if (sd == NULL) 2.41 (guido 05-May-97): PyThreadState *tstate; 2.41 (guido 05-May-97): tstate = PyThreadState_Get(); 2.41 (guido 05-May-97): return Py_BuildValue( 2.41 (guido 05-May-97): "(OOO)", 2.41 (guido 05-May-97): tstate->exc_type != NULL ? tstat 2.41 (guido 05-May-97): tstate->exc_value != NULL ? tsta 2.41 (guido 05-May-97): tstate->exc_traceback != NULL ? 2.41 (guido 05-May-97): tstate->exc_traceback : 2.41 (guido 05-May-97): } 2.41 (guido 05-May-97): revision 2.41 date: 1997/05/05 20:56:11; author: guido; state: Exp; lines: +30 -9 Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. Details of the 2.41 checkin ---------------------------- Index: sysmodule.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/sysmodule.c,v retrieving revision 2.40 retrieving revision 2.41 diff -c -r2.40 -r2.41 *** sysmodule.c 29 Apr 1997 20:42:30 -0000 2.40 --- sysmodule.c 5 May 1997 20:56:11 -0000 2.41 *************** *** 48,56 **** #include "osdefs.h" - PyObject *_PySys_TraceFunc, *_PySys_ProfileFunc; - int _PySys_CheckInterval = 10; - #if HAVE_UNISTD_H #include #endif --- 48,53 ---- *************** *** 98,103 **** --- 95,119 ---- } static PyObject * + sys_exc_info(self, args) + PyObject *self; + PyObject *args; + { + PyThreadState *tstate; + if (!PyArg_Parse(args, "")) + return NULL; + tstate = PyThreadState_Get(); + if (tstate == NULL) + Py_FatalError("sys.exc_info(): no thread state"); + return Py_BuildValue( + "(OOO)", + tstate->exc_type != NULL ? tstate->exc_type : Py_None, + tstate->exc_value != NULL ? tstate->exc_value : Py_None, + tstate->exc_traceback != NULL ? + tstate->exc_traceback : Py_None); + } + + static PyObject * sys_exit(self, args) PyObject *self; PyObject *args; *************** *** 112,123 **** PyObject *self; PyObject *args; { if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(_PySys_TraceFunc); ! _PySys_TraceFunc = args; Py_INCREF(Py_None); return Py_None; } --- 128,140 ---- PyObject *self; PyObject *args; { + PyThreadState *tstate = PyThreadState_Get(); if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(tstate->sys_tracefunc); ! tstate->sys_tracefunc = args; Py_INCREF(Py_None); return Py_None; } *************** *** 127,138 **** PyObject *self; PyObject *args; { if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(_PySys_ProfileFunc); ! _PySys_ProfileFunc = args; Py_INCREF(Py_None); return Py_None; } --- 144,156 ---- PyObject *self; PyObject *args; { + PyThreadState *tstate = PyThreadState_Get(); if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(tstate->sys_profilefunc); ! tstate->sys_profilefunc = args; Py_INCREF(Py_None); return Py_None; } *************** *** 142,148 **** PyObject *self; PyObject *args; { ! if (!PyArg_ParseTuple(args, "i", &_PySys_CheckInterval)) return NULL; Py_INCREF(Py_None); return Py_None; --- 160,167 ---- PyObject *self; PyObject *args; { ! PyThreadState *tstate = PyThreadState_Get(); ! if (!PyArg_ParseTuple(args, "i", &tstate->sys_checkinterval)) return NULL; Py_INCREF(Py_None); return Py_None; *************** *** 202,207 **** --- 221,227 ---- static PyMethodDef sys_methods[] = { /* Might as well keep this in alphabetic order */ + {"exc_info", sys_exc_info, 0}, {"exit", sys_exit, 0}, #ifdef COUNT_ALLOCS {"getcounts", sys_getcounts, 0}, *************** *** 232,238 **** if (list == NULL) return NULL; for (i = 0; _PyImport_Inittab[i].name != NULL; i++) { ! PyObject *name = PyString_FromString(_PyImport_Inittab[i].name); if (name == NULL) break; PyList_Append(list, name); --- 252,259 ---- if (list == NULL) return NULL; for (i = 0; _PyImport_Inittab[i].name != NULL; i++) { ! PyObject *name = PyString_FromString( ! _PyImport_Inittab[i].name); if (name == NULL) break; PyList_Append(list, name); From raymond.hettinger at verizon.net Fri Dec 19 13:13:15 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sat Dec 20 14:00:11 2003 Subject: [Python-Dev] Liceinses in the Docs Message-ID: <006f01c3c65b$c61b3020$e841fea9@oemcomputer> I would like to add an section to the Library Reference to fulfill license requirements for code included in Python. For example, the license for underlying the Mersenne Twister code is already embedded in the source but it requests to be included in the docs as well. Any thoughts or objections? Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031219/fb5e107d/attachment.html From zack at codesourcery.com Sat Dec 20 14:02:27 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Sat Dec 20 14:02:34 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "20 Dec 2003 11:31:18 +0100") References: <87y8tbezcq.fsf@codesourcery.com> <87wu8sy91n.fsf@egil.codesourcery.com> Message-ID: <87r7yzuxbw.fsf@egil.codesourcery.com> martin@v.loewis.de (Martin v. L?wis) writes: > "Zack Weinberg" writes: > >> It does, however, lead to a different problem: >> >> if (condition) >> Py_DECREF (foo); >> else // oops, syntax error here > > How so? This expands to > > if(condition) > if(cond2)action1; > else action2; > else No, it expands to if(condition) if(cond2) action1; else action2;; else -- note the extra semicolon, which provokes a syntax error. zw From fdrake at acm.org Sat Dec 20 14:20:06 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sat Dec 20 14:20:15 2003 Subject: [Python-Dev] Liceinses in the Docs In-Reply-To: <006f01c3c65b$c61b3020$e841fea9@oemcomputer> References: <006f01c3c65b$c61b3020$e841fea9@oemcomputer> Message-ID: <16356.41190.525408.280943@sftp.fdrake.net> Raymond Hettinger writes: > I would like to add an section to the Library Reference to fulfill > license requirements for code included in Python. > > For example, the license for underlying the Mersenne Twister code is > already embedded in the source but it requests to be included in the > docs as well. > > Any thoughts or objections? It should be added to Doc/commontex/license.tex with an explanation of what it pertains to. Thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From pf_moore at yahoo.co.uk Sat Dec 20 07:02:20 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Sat Dec 20 14:23:26 2003 Subject: [Python-Dev] Re: Relative import References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> Message-ID: Jack Diederich writes: > I like syntax that reads most important left-to-right, so what about > from MODULE import NAMES as RENAME searching HOW > or > import NAMES as RENAME from MODULE searching HOW I like this. It feels much more Pythonic than the various suggestions around magic dots in the name. It's explicit, easy to understand, and easy to extend (extra possibilities for HOW could be added with little difficulty). Paul. -- This signature intentionally left blank From tim.one at comcast.net Sat Dec 20 14:35:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 14:35:42 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87r7yzuxbw.fsf@egil.codesourcery.com> Message-ID: [Zack Weinberg] >>> It does, however, lead to a different problem: >>> >>> if (condition) >>> Py_DECREF (foo); >>> else // oops, syntax error here [Martin v. L?wis] >> How so? This expands to >> >> if(condition) >> if(cond2)action1; >> else action2; >> else [Zack] > No, it expands to > > if(condition) > if(cond2) action1; > else action2;; > else > > -- note the extra semicolon, which provokes a syntax error. Zack, which version of Python are you using? The current definition of Py_DECREF doesn't end with a semicolon: #define Py_DECREF(op) \ if (_Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA \ --(op)->ob_refcnt != 0) \ _Py_CHECK_REFCNT(op) \ else \ _Py_Dealloc((PyObject *)(op)) The expansion of _Py_CHECK_REFCNT(op) always ends with a semicolon (and may be nothing but a semicolon, depending on build type), but that's internal to the stuff Py_DECREF generates, and the leading underscore in _Py_CHECK_REFCNT's name makes it off-limits (by convention) for direct use by user-written code (_Py_CHECK_REFCNT is internal to Python's implementation, and Python internals guarantee to use it correctly). The expansion of Py_DECREF's else branch never ends with a semicolon, and Martin's explanation is correct. Same deal with Py_INCREF, Py_XINCREF, and Py_XDECREF. As a pragmatic matter, it's very unusual to need to do if (whatever) Py_DECREF(op); The functionality is common enough when "whatever" is "op != NULL", but accomplishing "decref if not NULL" is Py_XDECREF's purpose -- there's no need to code your own if-block in that case, and it's bad style (because non-idiomatic) to do so. From tim.one at comcast.net Sat Dec 20 14:41:47 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 14:41:51 2003 Subject: [Python-Dev] Liceinses in the Docs In-Reply-To: <006f01c3c65b$c61b3020$e841fea9@oemcomputer> Message-ID: [Raymond Hettinger] > I would like to add an section to the Library Reference to fulfill > license requirements for code included in Python. > > For example, the license for underlying the Mersenne Twister code is > already embedded in the source but it requests to be included in the > docs as well. > > Any thoughts or objections? +1. Even in cases where it's not legally required, I think it's plain good manners to acknowledge external work Python incorporates, from bsddb to zlib. From zack at codesourcery.com Sat Dec 20 14:42:51 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Sat Dec 20 14:42:56 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: (Tim Peters's message of "Sat, 20 Dec 2003 14:35:34 -0500") References: Message-ID: <87n09nuvgk.fsf@egil.codesourcery.com> "Tim Peters" writes: > [Zack Weinberg] >>>> It does, however, lead to a different problem: >>>> >>>> if (condition) >>>> Py_DECREF (foo); >>>> else // oops, syntax error here > > [Martin v. L?wis] >>> How so? This expands to >>> >>> if(condition) >>> if(cond2)action1; >>> else action2; >>> else > > [Zack] >> No, it expands to >> >> if(condition) >> if(cond2) action1; >> else action2;; >> else >> >> -- note the extra semicolon, which provokes a syntax error. > > Zack, which version of Python are you using? The current definition of > Py_DECREF doesn't end with a semicolon: I thought you might come back at me with that. Yes, you can avoid the syntax error by carefully leaving off the semicolon here. I don't consider this to be good style, though. It's easy to forget, and it's easy for someone to come along later and be confused and check in an 'obvious' patch that adds a semicolon - and it might not even break anything... then. I'm really not speaking about the python source code in specific. I'm speaking about general style principles, and I really do think it is good style *always* to wrap complex macros in do { } while (0) whether or not they need it. zw From tim.one at comcast.net Sat Dec 20 15:06:51 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 15:06:56 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <16356.30838.750692.236147@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > Then how about adding > > #if UCHAR_MAX != 255 > #error "Python's source code currently assumes 8-bit characters." > #endif > > right after the HAVE_LIMITS_H test? I wouldn't object. It should probably then also have #ifndef UCHAR_MAX #error ... #endif right before it, and stringobject.c's #if !defined(HAVE_LIMITS_H) && !defined(UCHAR_MAX) #define UCHAR_MAX 255 #endif should go away. From anthony at python.org Fri Dec 19 21:12:09 2003 From: anthony at python.org (Anthony Baxter) Date: Sat Dec 20 15:12:30 2003 Subject: [Python-Dev] RELEASED Python 2.3.3 (final) Message-ID: <200312200212.hBK2C9Ni011271@localhost.localdomain> On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.3 (final). Python 2.3.3 is a bug-fix release of Python 2.3. A couple of serious bugs related to weakrefs and the cyclic garbage collector have been fixed, along with a number of bugs in the standard library. See the release notes on the web page for more details. For more information on Python 2.3.3, including download links for various platforms, release notes, and known issues, please see http://www.python.org/2.3.3 Highlights of this new release include: - A couple of serious bugs in the interactions of weakrefs and cyclic GC have been squashed. - At shutdown, the second call to the cyclic garbage collector has been removed. This caused more problems than it solved. - The xml.parsers.expat module now provides Expat 1.95.7. - Bug #823328: urllib2's HTTP Digest Auth support works again. - See http://www.python.org/2.3.3/NEWS.html for other bugfixes. Highlights of the previous major Python release (2.3) are available from the Python 2.3 page, at http://www.python.org/2.3/highlights.html Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python 2.3.x Release Manager (on behalf of the entire python-dev team) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 226 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20031220/1b7d245a/attachment.bin From tismer at tismer.com Fri Dec 19 14:49:38 2003 From: tismer at tismer.com (Christian Tismer) Date: Sat Dec 20 15:51:56 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> <3FE33A1B.5030903@tismer.com> <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> Message-ID: <3FE35652.5000709@tismer.com> Guido van Rossum wrote: ... > Could you dig through CVS logs to find out when f_tstate was first > introduced? Maybe there's a clue about why there. Yeehaa! Quite some ride... Well, f_tstate was introduced here, in frameobject.c: ----------------------------------------------------- Revision : 2.29 Date : 1997/5/5 20:55:52 Author : 'guido' State : 'Exp' Lines : +13 -2 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. For frameobject.h, it was this version: --------------------------------------- Revision : 2.23 Date : 1997/5/5 20:55:43 Author : 'guido' State : 'Exp' Lines : +3 -1 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. Especially interesting: There is no single hint mentioning f_tstate. But, the diff shows PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; added at the same time. This was Python 1.5, I remember very well, the time where threading became the standard for Python. ceval.c was at this revision: ----------------------------- Revision : 2.118 Date : 1997/5/5 20:55:58 Author : 'guido' State : 'Exp' Lines : +263 -151 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. I fetched ceval 2.117 and 2.118 and compared them. This was exactly the time when the thread state was introduced. There were exactly two references to f_tstate: The most interesting diff for this question: PyObject * PyEval_SaveThread() { #ifdef WITH_THREAD if (interpreter_lock) { - PyObject *res; - res = (PyObject *)current_frame; - current_frame = NULL; + PyThreadState *tstate = PyThreadState_Swap(NULL); + PyObject *res = tstate ? (PyObject *) (tstate->frame) : NULL; release_lock(interpreter_lock); return res; } #endif return NULL; } void PyEval_RestoreThread(x) PyObject *x; { #ifdef WITH_THREAD if (interpreter_lock) { int err; err = errno; acquire_lock(interpreter_lock, 1); errno = err; - current_frame = (PyFrameObject *)x; + PyThreadState_Swap(x ? ((PyFrameObject *)x)->f_tstate : NULL); } #endif } PyEval_SaveThread has changed a bit meanwhile, no longer return the frame, but tstate. semantics are similar. The two functions clearly show hwo the transition from a global current_frame variable was made to the tstate mechanism. f_tstate was the source for switching the thread state. The current ceval version looks like this: void PyEval_RestoreThread(PyThreadState *tstate) { if (tstate == NULL) Py_FatalError("PyEval_RestoreThread: NULL tstate"); #ifdef WITH_THREAD if (interpreter_lock) { int err = errno; PyThread_acquire_lock(interpreter_lock, 1); errno = err; } #endif PyThreadState_Swap(tstate); } That means, tstate is not fetched from the frame, but given as a parameter. I checked all places where this function gets called, the tstate parameter never comes from a frame. As a conclusion, f_tstate appears to be an intermediate step during the move to threading. It was an easy place to keep the thread state. Its use was reduced to very few later on. call_trace introduced today's slightly false behavior exactly in that 2.118 version. It looks very much like an optimization, since there was no macro for fetching tstate at that time. This was perfectly fine, until generators were introduced. Until then, a frame was always bound to its thread. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tim.one at comcast.net Sat Dec 20 16:07:15 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 16:07:21 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87n09nuvgk.fsf@egil.codesourcery.com> Message-ID: [Zack Weinberg] > I thought you might come back at me with that. Yes, you can avoid the > syntax error by carefully leaving off the semicolon here. I don't > consider this to be good style, though. It's easy to forget, and it's > easy for someone to come along later and be confused and check in an > 'obvious' patch that adds a semicolon - and it might not even break > anything... then. Zack, you have to be kidding. Martin was clearly talking about the expansion of Py_DECREF specifically, not "macros in general", as also was the poster who started this thread. > I'm really not speaking about the python source code in specific. That's now obvious . > I'm speaking about general style principles, and I really do think it is > good style *always* to wrap complex macros in do { } while (0) whether > or not they need it. Py_DECREF doesn't need it; there may be some point to it if doing so shuts up gcc nuisance complaints, and we can be sure that the extra complexity of the expanded code doesn't provoke optimization bugs in any of the C compilers used to compile Python. Writing non-loops *as* loops is a good way to tickle endcase bugs in sophisticated optimizers -- it's not without risk, and there are several thousand instances of Py_DECREF and Py_XDECREF just in the core. Also because it's used a lot, in a debug build such a change would generate a ton of useless extra instructions (at least MSVC really does generate code to create a zero, test it against zero, and jump back to the top of "the loop" if zero doesn't equal zero; I expect that most C compilers would). From zack at codesourcery.com Sat Dec 20 16:17:50 2003 From: zack at codesourcery.com (Zack Weinberg) Date: Sat Dec 20 16:17:56 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: (Tim Peters's message of "Sat, 20 Dec 2003 16:07:15 -0500") References: Message-ID: <87iskbur29.fsf@egil.codesourcery.com> "Tim Peters" writes: > [Zack Weinberg] >> I thought you might come back at me with that. Yes, you can avoid the >> syntax error by carefully leaving off the semicolon here. I don't >> consider this to be good style, though. It's easy to forget, and it's >> easy for someone to come along later and be confused and check in an >> 'obvious' patch that adds a semicolon - and it might not even break >> anything... then. > > Zack, you have to be kidding. Martin was clearly talking about the > expansion of Py_DECREF specifically, not "macros in general", as also was > the poster who started this thread. I, however, was never speaking about Py_DECREF specifically. I thought this was clear from previous statements but I apologize for the confusion. >> I'm speaking about general style principles, and I really do think it is >> good style *always* to wrap complex macros in do { } while (0) whether >> or not they need it. > > Py_DECREF doesn't need it; there may be some point to it if doing so shuts > up gcc nuisance complaints, and we can be sure that the extra complexity of > the expanded code doesn't provoke optimization bugs in any of the C > compilers used to compile Python. Writing non-loops *as* loops is a good > way to tickle endcase bugs in sophisticated optimizers -- it's not without > risk, and there are several thousand instances of Py_DECREF and Py_XDECREF > just in the core. Also because it's used a lot, in a debug build such a > change would generate a ton of useless extra instructions (at least MSVC > really does generate code to create a zero, test it against zero, and jump > back to the top of "the loop" if zero doesn't equal zero; I expect that most > C compilers would). I doubt both statements just because this is a standard C idiom. Recognizing it and generating no extra code falls naturally out of basic constant-condition detection, which I would expect to be present in any modern compiler, - and done even in "debug builds" because it can't hurt and in fact usually makes the compiler run faster (less junk intermediat code to process). I especially doubt that this construct will tickle bugs in sophisticated optimizers -- it would be harder to get it wrong than to get it right. I am not disputing that MSVC does this silly thing you describe, but MSVC has a long track record of bugs and poor code generation. GCC does not generate junk code even at -O0. zw From tim.one at comcast.net Sat Dec 20 17:13:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 17:13:39 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: <87iskbur29.fsf@egil.codesourcery.com> Message-ID: [Zack Weinberg] > I doubt both statements just because this is a standard C idiom. I worked in compiler development for 15 years, so I'll have to trust my experience on this. The union of MSVC and gcc accounts for the vast bulk of Python installations, but only a fraction of the total number of compilers in use. Many optimizers know nothing at all about C specifically, as they're a common back end for many front-end intermediate code generators, for many languages. It's unusual for a compiler to do any sort of optimization unless optimizations are specifically requested. I don't have any gripe with gcc doing some regardless, but gcc isn't the universe. > Recognizing it and generating no extra code falls naturally out > of basic constant-condition detection, No, changing the flow graph is a serious and delicate program transformation. A C compiler front end doing it more-than-less "mindlessly", as part of special-casing the goofy idioms C encourages, has a fine shot at never screwing up, because it's doing it then before flow graph construction. Not all compilers do it that way. > which I would expect to be present in any modern compiler, In one way or another, yes, at least optimizing compilers. Not all compilers optimize, though. A possibly important example for Python in the near future is that the "standard edition" retail .NET compilers have no optimization capability, and neither does the freely downloadable .NET C/C++ compiler. > ... > I am not disputing that MSVC does this silly thing you describe, but > MSVC has a long track record of bugs That doesn't match our experience. I don't believe Python code has ever tickled an MSVC codegen or optimization bug, neither any in gcc-for-x86. We've certainly tickled codegen and optimization bugs in gcc for other platforms, and *many* such bugs in vendor compilers (SGI's native compilers probably hold the record for sheer quantity of codegen and optimization bugs when compiling Python). At Dragon Systems we were using C++ rather than C, and I think MSVC and gcc were blamed for about an equal number of codegen and optimization bugs. > and poor code generation. This also doesn't match our experience. Release-build Python compiled under MSVC for x86 has been faster than release-build Python compiled under gcc for x86 each time I've seen them compared, and the same was true for Dragon's software. Maybe gcc is getting better on x86; I don't know; for a long time it was at a systematic disadvantage because it was originally designed for RISC-ish architectures with a large, uniform register set, and x86 is about as from that model as things get. MSVC had many years' advantage in dealing with x86's many architecural quirks. > GCC does not generate junk code even at -O0. It certainly generates needless code even at -O0. If it didn't, there would be no reason to have a higher optimization level than -O0 . From skip at pobox.com Sat Dec 20 20:07:32 2003 From: skip at pobox.com (Skip Montanaro) Date: Sat Dec 20 20:08:21 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: <16356.30838.750692.236147@montanaro.dyndns.org> Message-ID: <16356.62036.22726.310077@montanaro.dyndns.org> >> #if UCHAR_MAX != 255 >> #error "Python's source code currently assumes 8-bit characters." >> #endif >> >> right after the HAVE_LIMITS_H test? Tim> I wouldn't object. It should probably then also have Tim> #ifndef UCHAR_MAX Tim> #error ... Tim> #endif Isn't that supposed to always be defined in limits.h or is UCHAR_MAX not a standard macro? Tim> right before it, and stringobject.c's Tim> #if !defined(HAVE_LIMITS_H) && !defined(UCHAR_MAX) Tim> #define UCHAR_MAX 255 Tim> #endif Tim> should go away. Sounds like a plan. I modify my local source and see if it affects anything. Skip From tim.one at comcast.net Sat Dec 20 21:52:03 2003 From: tim.one at comcast.net (Tim Peters) Date: Sat Dec 20 21:52:09 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <16356.62036.22726.310077@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > Isn't that supposed to always be defined in limits.h or is UCHAR_MAX > not a standard macro? Yes, it's supposed to be there. OTOH, so is limits.h, i.e. the HAVE_LIMITS_H test shouldn't be necessary either. So they're just sanity checks. But you're right, if UCHAR_MAX isn' defined, I was thinking the preprocessor would expand #if UCHAR_MAX != 255 to #if != 255 and then the error message would be incomprehensible. But unknown names in preprocessor conditionals actually get replaced by 0, and #if 0 != 255 does just as well. > Sounds like a plan. I modify my local source and see if it affects > anything. It should work fine. From anthony at interlink.com.au Sat Dec 20 22:14:03 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Dec 20 22:14:41 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include patchlevel.h, 2.74.4.9, 2.74.4.10 In-Reply-To: Message-ID: <200312210314.hBL3E3ZU017703@localhost.localdomain> >>> fdrake@users.sourceforge.net wrote > Update of /cvsroot/python/python/dist/src/Include > In directory sc8-pr-cvs1:/tmp/cvs-serv22509 > > Modified Files: > Tag: release23-maint > patchlevel.h > Log Message: > Post-release version bumpage. Argh! Nonononononono! Please leave the branch at 2.3.3 for Jack to cut the Mac versions! Anthony -- Anthony Baxter It's never too late to have a happy childhood. From fdrake at acm.org Sun Dec 21 00:12:00 2003 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sun Dec 21 00:12:16 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include patchlevel.h, 2.74.4.9, 2.74.4.10 In-Reply-To: <200312210314.hBL3E3ZU017703@localhost.localdomain> References: <200312210314.hBL3E3ZU017703@localhost.localdomain> Message-ID: <16357.11168.459937.182871@sftp.fdrake.net> Anthony Baxter writes: > Argh! Nonononononono! Please leave the branch at 2.3.3 for Jack to cut > the Mac versions! Oops, sorry; forgot about that. It was triggered by a change that Raymond made to the docs, properly removing material that leaked from the 2.4 tutorial. I've outdated the change to correct the version numbers. Not sure what the right thing to do about the docs is; let's just leave it alone, since it was a valuable correction. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tismer at tismer.com Sun Dec 21 03:45:30 2003 From: tismer at tismer.com (Christian Tismer) Date: Sun Dec 21 04:08:19 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> References: <3FDE6D35.3090100@tismer.com> <3FE2C5B5.8080208@tismer.com> <200312191531.hBJFVgw27430@c-24-5-183-134.client.comcast.net> <3FE33A1B.5030903@tismer.com> <200312191815.hBJIFP528036@c-24-5-183-134.client.comcast.net> Message-ID: <3FE55DAA.2020808@tismer.com> Guido van Rossum wrote: ... > Could you dig through CVS logs to find out when f_tstate was first > introduced? Maybe there's a clue about why there. Yeehaa! Quite some ride... Well, f_tstate was introduced here, in frameobject.c: ----------------------------------------------------- Revision : 2.29 Date : 1997/5/5 20:55:52 Author : 'guido' State : 'Exp' Lines : +13 -2 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. For frameobject.h, it was this version: --------------------------------------- Revision : 2.23 Date : 1997/5/5 20:55:43 Author : 'guido' State : 'Exp' Lines : +3 -1 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. Especially interesting: There is no single hint mentioning f_tstate. But, the diff shows PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; added at the same time. This was Python 1.5, I remember very well, the time where threading became the standard for Python. ceval.c was at this revision: ----------------------------- Revision : 2.118 Date : 1997/5/5 20:55:58 Author : 'guido' State : 'Exp' Lines : +263 -151 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. I fetched ceval 2.117 and 2.118 and compared them. This was exactly the time when the thread state was introduced. There were exactly two references to f_tstate: The most interesting diff for this question: PyObject * PyEval_SaveThread() { #ifdef WITH_THREAD if (interpreter_lock) { - PyObject *res; - res = (PyObject *)current_frame; - current_frame = NULL; + PyThreadState *tstate = PyThreadState_Swap(NULL); + PyObject *res = tstate ? (PyObject *) (tstate->frame) : NULL; release_lock(interpreter_lock); return res; } #endif return NULL; } void PyEval_RestoreThread(x) PyObject *x; { #ifdef WITH_THREAD if (interpreter_lock) { int err; err = errno; acquire_lock(interpreter_lock, 1); errno = err; - current_frame = (PyFrameObject *)x; + PyThreadState_Swap(x ? ((PyFrameObject *)x)->f_tstate : NULL); } #endif } PyEval_SaveThread has changed a bit meanwhile, no longer return the frame, but tstate. semantics are similar. The two functions clearly show hwo the transition from a global current_frame variable was made to the tstate mechanism. f_tstate was the source for switching the thread state. The current ceval version looks like this: void PyEval_RestoreThread(PyThreadState *tstate) { if (tstate == NULL) Py_FatalError("PyEval_RestoreThread: NULL tstate"); #ifdef WITH_THREAD if (interpreter_lock) { int err = errno; PyThread_acquire_lock(interpreter_lock, 1); errno = err; } #endif PyThreadState_Swap(tstate); } That means, tstate is not fetched from the frame, but given as a parameter. I checked all places where this function gets called, the tstate parameter never comes from a frame. As a conclusion, f_tstate appears to be an intermediate step during the move to threading. It was an easy place to keep the thread state. Its use was reduced to very few later on. call_trace introduced today's slightly false behavior exactly in that 2.118 version. It looks very much like an optimization, since there was no macro for fetching tstate at that time. This was perfectly fine, until generators were introduced. Until then, a frame was always bound to its thread. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at tismer.com Sun Dec 21 03:41:54 2003 From: tismer at tismer.com (Christian Tismer) Date: Sun Dec 21 04:08:32 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <000201c3c729$c8ffcde0$e841fea9@oemcomputer> References: <000201c3c729$c8ffcde0$e841fea9@oemcomputer> Message-ID: <3FE55CD2.9040707@tismer.com> Raymond Hettinger wrote: ... > Here is a head start on the research. > > The ceval.c use of tstate goes back to the introduction of generators in > 2001. > > The use in traceback.c and sysmodule.c goes back to 1997 when > per-thread globals were factored into a structure to support separate > thread-state management. > > Prior to that, the history is more diffuse and harder to follow. Thanks a lot. I did a similar research on Friday, but for some reason it was not accepted by python.org, as it appears. (why, only 4 k, yours was 10?) I will try to send it again. My summary: f_tstate not needed! ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From martin at v.loewis.de Sun Dec 21 05:27:45 2003 From: martin at v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Sun Dec 21 05:27:59 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <2mwu8rebcp.fsf@starship.python.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> Message-ID: Michael Hudson writes: > > I'm not sure what to do about this: We apparently *want* the signals > > blocked in the thread, but we don't want them to be blocked in the > > process invoked through system(). Proposals are welcome. > > Does pthread_atfork() help? Most likely. system(3) is specified as being implemented through fork()/exec(), so an atfork handler should be invoked in any compliant implementation. We could install a child handler, which unblocks the signals we don't want to be blocked. Now, the question is: What signals precisely we don't want to be blocked? I think the answer is "All signals that have not explicitly been blocked by the application". OTOH, we already have PyOS_AfterFork, which could be used instead of pthread_atfork. Jeff, would you like to add some code there, to set all signal handlers into default for which Handlers lists that the default handling should occur? Regards, Martin From ncoghlan at iinet.net.au Sun Dec 21 08:16:58 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun Dec 21 08:17:03 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031219213031.GD1246@wopr> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> Message-ID: <3FE59D4A.8050403@iinet.net.au> Jack Diederich wrote: > I like syntax that reads most important left-to-right, so what about > from MODULE import NAMES as RENAME searching HOW > or > import NAMES as RENAME from MODULE searching HOW This would work really well for any uses cases I can imagine having. It also has the following benefits: 1. No magic punctuation 2. People who don't need it, just never use a 'searching' clause 3. People encountering it for the first time have a clear flag letting them know something fancy is going on 4. These people have a word they can look up in the Python docs I hadn't even thought of point 4 prior to considering Jack's proposal. If I encounter "import .fred.mary" in some python I didn't write, how the heck do I find out what it means? [...] > ps, I like the second version better but it is less like the current version, > which is a drawback. To head a little further down this sidetrack, the from version of 'import' has puzzled me for a while: A. from mypkg import foo B. import mypkg.foo as foo What, if any, are the semantic differences between A & B? Is it only that A2 works, and B2 doesn't? A2. from mymodule import * B2. import mymodule.* Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From aahz at pythoncraft.com Sun Dec 21 12:30:15 2003 From: aahz at pythoncraft.com (Aahz) Date: Sun Dec 21 12:30:18 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> References: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> Message-ID: <20031221173014.GA20633@panix.com> On Thu, Dec 18, 2003, Guido van Rossum wrote: > > OK. Wanna write a PEP? Just kidding, I know you have less time than > I do for it. So... *anybody* wanna write a PEP about this? (The > parentheses proposal should also be folded in.) Since nobody else has volunteered, I guess I'll write the PEP. Expect it sometime in the next couple of weeks. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From tim.one at comcast.net Sun Dec 21 13:29:06 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 21 13:29:11 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <3FE55CD2.9040707@tismer.com> Message-ID: [Christian Tismer] > I did a similar research on Friday, but for some reason it > was not accepted by python.org, as it appears. > (why, only 4 k, yours was 10?) Your original was accepted: http://mail.python.org/pipermail/python-dev/2003-December/041287.html I think there may have been a long delay in sending it out, though, perhaps related to the python.org domain transfer, or to hardware fiddling recently done at mail.python.org's physical site. Whichever, it's in the archive, and I got it via email too. > My summary: f_tstate not needed! More, it seems conceptually flawed, albeit subtly. Python's runtime is, in effect, simulating thread-local storage by hand, and the current value of _PyThreadState_Current always points to a PyThreadState struct holding the currently-executing thread's *conceptual* thread-local storage. If we were able to use honest-to-goodness TLS instead, it wouldn't have been possible (well, not w/o ugly hacks) for a frame to use the values of these guys associated with any thread other than the one currently executing. Or, IOW, the existence of f_tstate creates a possibility for inter-thread mixups. Still, the possibility to switch threads across generator resumptions seems darned hard to view as "a feature". I'd advise people not to rely on it . From skip at pobox.com Sun Dec 21 15:03:54 2003 From: skip at pobox.com (Skip Montanaro) Date: Sun Dec 21 15:03:59 2003 Subject: [Python-Dev] make test failing in a spectacular way in test_codeccallbacks Message-ID: <16357.64682.286170.38911@montanaro.dyndns.org> Fresh cvs up (no local changes) on Mac OS X 10.2.8 yields this while running test_codeccallbacks (non-framework build, configured only using --with-pydebug): malloc[20582]: protecting edges malloc[20582]: recording stacks using standard recorder malloc[20582]: enabling scribbling to detect mods to free blocks malloc[20582]: checks heap after 1th operation and each 1000 operations malloc[20582]: environment variables that can be set for debug: - MallocGuardEdges to add 2 guard pages for each large block - MallocDoNotProtectPrelude to disable protection (when previous flag set) - MallocDoNotProtectPostlude to disable protection (when previous flag set) - MallocStackLogging to record all stacks. Tools like leaks can then be applied - MallocStackLoggingNoCompact to record all stacks. Needed for malloc_history - MallocScribble to detect writing on free blocks: 0x55 is written upon free - MallocCheckHeapStart to check the heap from time to time after operations - MallocHelp - this help! MallocCheckHeap: PASSED check at 1th operation MallocCheckHeap: PASSED check at 1001th operation MallocCheckHeap: PASSED check at 2001th operation MallocCheckHeap: PASSED check at 3001th operation test_backslashescape (__main__.CodecCallbackTest) ... ok test_badandgoodbackslashreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodignoreexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodstrictexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodxmlcharrefreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badhandlerresults (__main__.CodecCallbackTest) ... ok test_badregistercall (__main__.CodecCallbackTest) ... ok test_bug828737 (__main__.CodecCallbackTest) ... ok test_callbacks (__main__.CodecCallbackTest) ... ok test_charmapencode (__main__.CodecCallbackTest) ... ok test_decodehelper (__main__.CodecCallbackTest) ... ok MallocCheckHeap: PASSED check at 4001th operation test_encodehelper (__main__.CodecCallbackTest) ... ok Debug memory block at address p=0x7cb138: 7032 bytes originally requested The 4 pad bytes at p-4 are FORBIDDENBYTE, as expected. The 4 pad bytes at tail=0x7cccb0 are not all FORBIDDENBYTE (0xfb): at tail+0: 0x33 *** OUCH at tail+1: 0x35 *** OUCH at tail+2: 0x34 *** OUCH at tail+3: 0x3b *** OUCH The block was made by call #639840562 to debug malloc/realloc. Data at p: 00 00 00 00 00 00 00 00 ... 33 35 34 3b 26 23 31 32 Fatal Python error: bad trailing pad byte Abort trap The "Data at p" hex bytes map to ['3', '5', '4', ';', '&', '#', '1', '2']. The Malloc messages are all from Apple's malloc() pkg with a bunch of debug environment variables set. I haven't dug into this any further. I applied a new OS X security update yesterday. That's the only recent change to the system I'm aware of. GCC is [GCC 3.3 20030304 (Apple Computer, Inc. build 1493)] on darwin as reported by the Python interpreter at startup. Anyone else seen this? Skip From raymond.hettinger at verizon.net Sun Dec 21 17:15:03 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun Dec 21 17:15:32 2003 Subject: [Python-Dev] Vinay Sajip Message-ID: <000601c3c80f$e2a5fee0$0dae2c81@oemcomputer> FYI, Vinay Sajip has been granted CVS commit privileges for the purpose of maintaining and extending his logging package. Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031221/67ed5d5c/attachment.html From jepler at unpythonic.net Sun Dec 21 18:07:04 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Sun Dec 21 18:07:09 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> Message-ID: <20031221230655.GA4417@unpythonic.net> On Sun, Dec 21, 2003 at 11:27:45AM +0100, Martin v. L?wis wrote: > OTOH, we already have PyOS_AfterFork, which could be used instead of > pthread_atfork. Jeff, would you like to add some code there, to set > all signal handlers into default for which Handlers lists that the > default handling should occur? When using pthread_atfork, os.system never triggers my code. However, reimplementing os.system in terms of os.fork+os.execv, it does. I don't know if this is right or wrong according to pthread, but since it doesn't work on my platform the question is academic for me. Wouldn't the PyOS_AfterFork approach also require python to provide its own versions of any POSIX APIs that would typically be implemented in terms of fork (system(), popen(), and spawn*() come to mind)? Jeff Index: Python/thread_pthread.h =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/thread_pthread.h,v retrieving revision 2.48 diff -u -r2.48 thread_pthread.h --- Python/thread_pthread.h 19 Nov 2003 22:52:22 -0000 2.48 +++ Python/thread_pthread.h 21 Dec 2003 23:03:52 -0000 @@ -143,6 +143,17 @@ * Initialization. */ +static void PyThread__fork_child(void) { + /* Mask all signals in the current thread before creating the new + * thread. This causes the new thread to start with all signals + * blocked. + */ + sigset_t childmask; + sigfillset(&childmask); + SET_THREAD_SIGMASK(SIG_UNBLOCK, &childmask, NULL); + fprintf(stderr, "PyThread__fork_child()\n"); fflush(stderr); +} + #ifdef _HAVE_BSDI static void _noop(void) @@ -157,6 +168,7 @@ pthread_t thread1; pthread_create(&thread1, NULL, (void *) _noop, &dummy); pthread_join(thread1, NULL); + pthread_atfork(NULL, NULL, PyThread__fork_child); } #else /* !_HAVE_BSDI */ @@ -167,6 +179,7 @@ #if defined(_AIX) && defined(__GNUC__) pthread_init(); #endif + pthread_atfork(NULL, NULL, PyThread__fork_child); } #endif /* !_HAVE_BSDI */ From guido at python.org Sun Dec 21 19:03:50 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 21 19:03:57 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Sun, 21 Dec 2003 12:30:15 EST." <20031221173014.GA20633@panix.com> References: <200312181750.hBIHof425371@c-24-5-183-134.client.comcast.net> <20031221173014.GA20633@panix.com> Message-ID: <200312220003.hBM03ph08816@c-24-5-183-134.client.comcast.net> > Since nobody else has volunteered, I guess I'll write the PEP. Expect > it sometime in the next couple of weeks. Thanks, Aahz! --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer at tismer.com Sun Dec 21 19:23:51 2003 From: tismer at tismer.com (Christian Tismer) Date: Sun Dec 21 19:23:37 2003 Subject: [Python-Dev] Last chance! In-Reply-To: References: Message-ID: <3FE63997.4000600@tismer.com> Tim Peters wrote: ... > Your original was accepted: > > http://mail.python.org/pipermail/python-dev/2003-December/041287.html Yup, I wasn't aware of the DNS change. >>My summary: f_tstate not needed! > > More, it seems conceptually flawed, albeit subtly. ... Very much agreed. I always felt that f_tstate was not quite ok. You spelled it and supplied more reason. > Still, the possibility to switch threads across generator resumptions seems > darned hard to view as "a feature". I'd advise people not to rely on it > . Well, this might only work for generators if it is guaranteed that the thread switch doesn't happen while the generator is executing. I think this will give an exception. But most probably the result will be hard to predict. Other for my "tasklet" tiny threads. Their first purpose is to have concurrent small tasks without interaction (or if with interaction, then via channels and blocking). A tasklet being interrupted by a thread switch will not be auto-schedulable until the thread is resumed, but other tasklets might be, and they don't need to care about which real thread is actually in charge. This is one of my reasons why I want to get rid of f_tstate -- it does not really apply to most of my frames. Being tied to a thread is a property of the tasklet, and it does not apply all the time, only when I have to keep C state. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From jcarlson at uci.edu Sun Dec 21 19:41:45 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Dec 21 19:43:37 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <200312220003.hBM03ph08816@c-24-5-183-134.client.comcast.net> References: <20031221173014.GA20633@panix.com> <200312220003.hBM03ph08816@c-24-5-183-134.client.comcast.net> Message-ID: <20031221163032.10E2.JCARLSON@uci.edu> Hey, I'm a new lurker (hoping to change that) here on the python-dev list, and I thought I would post a message about a new PEP I recently submitted entitled "A Case for Some". The PEP talks about how there exists None, that is false, and that is smaller than any other object ( min(None, a) -> None ). However, there doesn't exist an equivalent true value that is larger than any other number, like ( max(Some, a) -> Some ). Personally, I think the introduction of Some offers a nice compliment to None, especially because it allows algorithms that require an initialization of infinity to not require large numbers. Some would be sufficient. It also has the side benefit of being funny. I just wanted to get the discussion going. How does everyone feel about Some? Thanks, - Josiah Carlson From guido at python.org Sun Dec 21 19:49:54 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 21 19:50:02 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: Your message of "Sun, 21 Dec 2003 16:41:45 PST." <20031221163032.10E2.JCARLSON@uci.edu> References: <20031221173014.GA20633@panix.com> <200312220003.hBM03ph08816@c-24-5-183-134.client.comcast.net> <20031221163032.10E2.JCARLSON@uci.edu> Message-ID: <200312220049.hBM0nsU09360@c-24-5-183-134.client.comcast.net> > I'm a new lurker (hoping to change that) here on the python-dev list, > and I thought I would post a message about a new PEP I recently > submitted entitled "A Case for Some". > > The PEP talks about how there exists None, that is false, and that is > smaller than any other object ( min(None, a) -> None ). However, there > doesn't exist an equivalent true value that is larger than any other > number, like ( max(Some, a) -> Some ). > > Personally, I think the introduction of Some offers a nice compliment to > None, especially because it allows algorithms that require an > initialization of infinity to not require large numbers. Some would be > sufficient. It also has the side benefit of being funny. > > I just wanted to get the discussion going. How does everyone feel about > Some? -1. Depending on your algorithm you can already use sys.maxint, or a large float (e.g. 1e300), or you can write 1e1000 (or 1e300**2) which on most systems evaluates to a floating point Infinity. And "Some" sounds like a really strange spelling of "Infinity". There's also a PEP (PEP 754) to add a proper way to spell floating point Infinity. Or you could reverse signs and use min(None, -a). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Sun Dec 21 20:07:59 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 21 20:08:03 2003 Subject: [Python-Dev] make test failing in a spectacular way intest_codeccallbacks In-Reply-To: <16357.64682.286170.38911@montanaro.dyndns.org> Message-ID: [Skip] > Fresh cvs up (no local changes) on Mac OS X 10.2.8 yields this while > running test_codeccallbacks (non-framework build, configured only > using --with-pydebug): > ... > Anyone else seen this? Just tried on Windows. test_codeccallbacks crashes here in release or debug builds. Here from a debug build: C:\Code\python\PCbuild>python_d ../lib/test/test_codeccallbacks.py test_backslashescape (__main__.CodecCallbackTest) ... ok test_badandgoodbackslashreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodignoreexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodstrictexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodxmlcharrefreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badhandlerresults (__main__.CodecCallbackTest) ... ok test_badregistercall (__main__.CodecCallbackTest) ... ok test_bug828737 (__main__.CodecCallbackTest) ... ok test_callbacks (__main__.CodecCallbackTest) ... ok test_charmapencode (__main__.CodecCallbackTest) ... ok test_decodehelper (__main__.CodecCallbackTest) ... ok test_encodehelper (__main__.CodecCallbackTest) ... ok test_longstrings (__main__.CodecCallbackTest) ... C:\Code\python\PCbuild>python_d ../lib/test/test_codeccallbacks.py test_backslashescape (__main__.CodecCallbackTest) ... ok test_badandgoodbackslashreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodignoreexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodstrictexceptions (__main__.CodecCallbackTest) ... ok test_badandgoodxmlcharrefreplaceexceptions (__main__.CodecCallbackTest) ... ok test_badhandlerresults (__main__.CodecCallbackTest) ... ok test_badregistercall (__main__.CodecCallbackTest) ... ok test_bug828737 (__main__.CodecCallbackTest) ... ok test_callbacks (__main__.CodecCallbackTest) ... ok test_charmapencode (__main__.CodecCallbackTest) ... ok test_decodehelper (__main__.CodecCallbackTest) ... ok test_encodehelper (__main__.CodecCallbackTest) ... ok test_longstrings (__main__.CodecCallbackTest) ... and at that point it dies with a segfault. It's really hosed at that point -- I can't get into the debugger. If I start the test under the debugger, the debugger craps out. So no easy clues from me either. Just to cheer things up , the bsddb3 tests are in much worse shape now on Win98SE than they are in 2.3.3 (about 50 more errors under current CVS). I expect that 2.3.3 will be the last Python release . BTW, when I cvs up'ed and recompiled to run the above, the only file I noticed getting recompiled was unicodeobject.c. From martin at v.loewis.de Sun Dec 21 20:08:34 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Sun Dec 21 20:09:38 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031221230655.GA4417@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> Message-ID: <3FE64412.3010500@v.loewis.de> Jeff Epler wrote: > When using pthread_atfork, os.system never triggers my code. However, > reimplementing os.system in terms of os.fork+os.execv, it does. I don't > know if this is right or wrong according to pthread, but since it doesn't > work on my platform the question is academic for me. Interesting. I'd be curious to find out why this fails - it may be a bug in your system, in which case I'd say "tough luck, complain to the system vendor" (for Redhat 9, I'm tempted to say that anyway ...) Looking at what likely is the source of your system(3) implementation (glibc 2.3.2, sysdeps/unix/sysv/linux/i386/system.c), I see that the fork used inside system(3) is # define FORK() \ INLINE_SYSCALL (clone, 3, CLONE_PARENT_SETTID | SIGCHLD, 0, &pid) Atleast, this is the fork being used if __ASSUME_CLONE_THREAD_FLAGS is defined, which is the case for Linux >2.5.50. With this fork() implementation, atfork handlers won't be invoked, which clearly looks like a bug to me. You might want to upgrade glibc to glibc-2.3.2-27.9.7.i686.rpm and nptl-devel-2.3.2-27.9.7.i686.rpm. In this version, the definition of FORK is changed to #if defined __ASSUME_CLONE_THREAD_FLAGS && !defined FORK # define FORK() \ INLINE_SYSCALL (clone, 3, CLONE_PARENT_SETTID | SIGCHLD, 0, &pid) #endif which might actually do the right thing, assuming FORK is already defined to one that calls the atfork handlers. > Wouldn't the PyOS_AfterFork approach also require python to provide its > own versions of any POSIX APIs that would typically be implemented in > terms of fork (system(), popen(), and spawn*() come to mind)? You are right. system(3) won't call our version of fork, so PyOS_AfterFork won't be invoked. So forget about this approach. Regards, Martin From perky at i18n.org Sun Dec 21 20:39:37 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Sun Dec 21 20:39:43 2003 Subject: [Python-Dev] make test failing in a spectacular way intest_codeccallbacks In-Reply-To: References: <16357.64682.286170.38911@montanaro.dyndns.org> Message-ID: <20031222013937.GA92298@i18n.org> On Sun, Dec 21, 2003 at 08:07:59PM -0500, Tim Peters wrote: > [Skip] > > Fresh cvs up (no local changes) on Mac OS X 10.2.8 yields this while > > running test_codeccallbacks (non-framework build, configured only > > using --with-pydebug): > > ... > > Anyone else seen this? > > Just tried on Windows. test_codeccallbacks crashes here in release or debug > builds. Here from a debug build: > [snip] > and at that point it dies with a segfault. It's really hosed at that > point -- I can't get into the debugger. If I start the test under the > debugger, the debugger craps out. So no easy clues from me either. Just fixed in unicodeobject.c 2.206. I'm sorry for breaking the regrtest. I guess that I ran regrtest with built previous for checking sanity before the commit. (./python versus python) Hye-Shik From jepler at unpythonic.net Sun Dec 21 21:07:37 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Sun Dec 21 21:07:36 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <3FE64412.3010500@v.loewis.de> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> Message-ID: <20031222020728.GA15895@unpythonic.net> On Mon, Dec 22, 2003 at 02:08:34AM +0100, Martin v. Loewis wrote: > With this fork() implementation, atfork handlers won't be invoked, > which clearly looks like a bug to me. You might want to upgrade glibc > to glibc-2.3.2-27.9.7.i686.rpm and nptl-devel-2.3.2-27.9.7.i686.rpm. > In this version, the definition of FORK is changed to I also tested a fedora machine with glibc-2.3.2-101.1 nptl-devel-2.3.2-101.1 and the pthread_atfork handler is still not called for system(). Opengroup's documentation for system() says [...] The environment of the executed command shall be as if a child process were created using fork(), and the child process invoked the sh utility using execl() [...] http://www.opengroup.org/onlinepubs/007904975/functions/system.html so this looks like a glibc/nptl bug. I filed it in redhat's bugzilla: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=112517 Given all this, does it make sense to adopt a patch similar to the one I posted earlier, and ignore the bug in system() on any particular OS? system() and popen() are easy to replace in Python if anybody is really bothered by the problem. Jeff From tim.one at comcast.net Sun Dec 21 21:29:22 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 21 21:29:27 2003 Subject: [Python-Dev] make test failing in a spectacular wayintest_codeccallbacks In-Reply-To: <20031222013937.GA92298@i18n.org> Message-ID: [Hye-Shik Chang] > Just fixed in unicodeobject.c 2.206. Worked for me -- thanks! > I'm sorry for breaking the regrtest. I guess that I ran regrtest > with built previous for checking sanity before the commit. (./python > versus python) That's OK. Don't make the same mistake again, and even Guido will forgive you . From tjreedy at udel.edu Sun Dec 21 21:36:44 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Sun Dec 21 21:36:46 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> Message-ID: "Luke Kenneth Casson Leighton" wrote in message news:20031220121942.GF1933@lkcl.net... > if you "feed" access to __class__ via a single function, you have > a bottleneck again which can be replaced. > > so, the proposal is simple: create an __builtin__ function called > __get_class__ through which access to __class__ is made. > I wonder if it would not be better to turn __class__ into a property with a get function that conditionally gives out the real thing. TJR From tim.one at comcast.net Sun Dec 21 21:52:48 2003 From: tim.one at comcast.net (Tim Peters) Date: Sun Dec 21 21:52:53 2003 Subject: [Python-Dev] Last chance! In-Reply-To: <3FE63997.4000600@tismer.com> Message-ID: [Tim] >> Still, the possibility to switch threads across generator >> resumptions seems darned hard to view as "a feature". I'd advise >> people not to rely on it . [Christian Tismer] > Well, this might only work for generators if it is guaranteed > that the thread switch doesn't happen while the generator > is executing. I think this will give an exception. But most > probably the result will be hard to predict. I'm not sure I'm following, but nothing in Python now *stops* you from resuming a suspended generator from another thread. Here's an extreme example of that: """ import threading def myid(): while True: yield threading.currentThread() next = myid().next mut = threading.Lock() class Worker(threading.Thread): def run(self): while True: mut.acquire() print next() mut.release() for i in range(3): Worker().start() """ Then the body of the single generator is executed by 3 different threads, more or less randomly, although never by the thread that created the generator. If the body of the generator were relying on thread-local state (like, e.g., the mass of settings in the decimal arithmetic package's per-thread default "numeric context" object), the result would sure be unpredictable. It wouldn't crash Python, it would just be a bitch to debug. Then again, lots of bad threading practices lead to bitch-to-debug code, so I'm not really inclined to offer extra protection here. > Other for my "tasklet" tiny threads. Their first purpose is > to have concurrent small tasks without interaction (or if > with interaction, then via channels and blocking). A tasklet > being interrupted by a thread switch will not be auto-schedulable > until the thread is resumed, but other tasklets might be, and > they don't need to care about which real thread is actually in > charge. This is one of my reasons why I want to get rid of > f_tstate -- it does not really apply to most of my frames. > Being tied to a thread is a property of the tasklet, and it > does not apply all the time, only when I have to keep C state. If a tasklet can be affected by thread-local state (like numeric context -- Python has little of this nature now, but it's bound to increase), then a tasklet will also start to care "who's in charge". So I expect a patch to reintroduce f_tstate in about a year . From martin at v.loewis.de Sun Dec 21 21:46:14 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Sun Dec 21 22:52:44 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031222020728.GA15895@unpythonic.net> References: <1071669660.18260.28.camel@gandalf.tm.informatik.uni-frankfurt.de> <20031217151450.GB28436@unpythonic.net> <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> Message-ID: <3FE65AF6.6010403@v.loewis.de> Jeff Epler wrote: > Given all this, does it make sense to adopt a patch similar to the one > I posted earlier, and ignore the bug in system() on any particular OS? > system() and popen() are easy to replace in Python if anybody is really > bothered by the problem. I'd think so, yes. The patch needs to be elaborated, and I'd delay integration to wait for a Redhat response. We should also restrict this for 2.4 at the moment, to find out how other pthread implementations react to pthread_atfork. Regards, Martin From guido at python.org Sun Dec 21 23:48:01 2003 From: guido at python.org (Guido van Rossum) Date: Sun Dec 21 23:48:09 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Fri, 19 Dec 2003 16:30:31 EST." <20031219213031.GD1246@wopr> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> Message-ID: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> > I like syntax that reads most important left-to-right, so what about > from MODULE import NAMES as RENAME searching HOW > or > import NAMES as RENAME from MODULE searching HOW > > searching could be 'absolute' 'relative' or any of the other > suggested words. If you wanted to get fancy it could be a list, if > it isn't a list people who truly care could cascade try/except on > ImportError. > > -jackdied > > ps, I like the second version better but it is less like the current version, > which is a drawback. This got several encouraging responses, so I have to consider it. Unfortunately it looks like 'searching' would have to become a reserved word; if we use the same trick we used to avoid making 'as' a reserved word, that would be too confusing to decode in the code generator. The HOW could be just an identifier then. I'm not sure how well it'll read in actual examples. Let's try. Adapting an example from Opal that was used here before... from COGS import generate searching relative To me, that looks a lot like someone left the commas out of from COGS import generate, searching, relative In general I'm not a big fan of "wordy" clauses like this -- they work better in languages like SQL or COBOL because those are case-sensitive so you can use a difference in capitalization there; there the former would be FROM COGS IMPORT generate SEARCHING RELATIVE (using imaginary syntax) while the latter would be FROM COGS IMPORT generate, searching, relative while in Python the long sequence of lowercase words becomes a blur. So if the two alternatives are simply 'searching absolute' and 'searching relative', with the default being 'searching absolute' in Python 3.0 (but the current ambiguous search in Python 2.x) I'd still prefer making the distinction with "from ...MODULE" vs. "from MODULE" rather than with a searching clause. And I still prefer the single-dot syntax over three dots, because it can be used to spell importing from the parent or grandparent package explicitly. A separate suggestion is to switch from "from X import Y" to "import Y from X". This seems a gratuitous change even if it reads or edits marginally better. Now's not the time to tinker with marginal stuff -- that should've been done 10 years ago. --Guido van Rossum (home page: http://www.python.org/~guido/) From jack at performancedrivers.com Mon Dec 22 02:52:19 2003 From: jack at performancedrivers.com (Jack Diederich) Date: Mon Dec 22 02:52:24 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> Message-ID: <20031222075219.GA3426@performancedrivers.com> On Sun, Dec 21, 2003 at 08:48:01PM -0800, Guido van Rossum wrote: > > I like syntax that reads most important left-to-right, so what about > > from MODULE import NAMES as RENAME searching HOW > > or > > import NAMES as RENAME from MODULE searching HOW > > > > searching could be 'absolute' 'relative' or any of the other > > suggested words. If you wanted to get fancy it could be a list, if > > it isn't a list people who truly care could cascade try/except on > > ImportError. > > Unfortunately it looks like 'searching' would have to become a > reserved word; if we use the same trick we used to avoid making 'as' a > reserved word, that would be too confusing to decode in the code > generator. The HOW could be just an identifier then. Too bad the decorator PEP (PEP 318) seems to favor 'as' in the syntax, import foo [relative] is more explicit than the dot syntax and has a nice visual seperator while avoiding the wordiness problem of 'searching' > I'm not sure how well it'll read in actual examples. Let's try. > Adapting an example from Opal that was used here before... > > from COGS import generate searching relative > > To me, that looks a lot like someone left the commas out of > > from COGS import generate, searching, relative moving the 'import' to always be first would alleviate this a bit because 'searching' would always follow a non-list. import generate from COGS searching relative or if we decorate import generate from COGS [relative] from COGS import generate [relative] > In general I'm not a big fan of "wordy" clauses like this -- they work > better in languages like SQL or COBOL because those are case-sensitive > so you can use a difference in capitalization there [snip] > while in Python the long sequence of lowercase words becomes a blur. as above, moving import to be first helps but doesn't eliminate the wordiness problem[*]. 'as' and 'from' are short enough to be easilly seen as keywords, 'find' or 'look' reads less like english but would be easier to pick out visually than 'searching' > So if the two alternatives are simply 'searching absolute' and > 'searching relative' Did someone mention 'searching scan' in dusty corner somewhere? One drawback to the dot syntax is if there can be more than two ways to do the search. Adding a third way is easier w/ the searching clause instead of adding a 'bang' syntax to the 'dot' syntax . The decorator would do one better by being a placeholder for anything that modifies the import. > A separate suggestion is to switch from "from X import Y" to "import Y > from X". This seems a gratuitous change even if it reads or edits > marginally better. Now's not the time to tinker with marginal stuff > -- that should've been done 10 years ago. Damn, I really liked that part. I don't get tripped up on from X import Y anymore, but I did when I started w/ python. benign-user-for-life-ly, -jackdied [*] I use an emacs hack to display 'lambda' as 'L' to avoid the same keyword-reads-like-a-variable effect. From jcarlson at uci.edu Mon Dec 22 03:13:14 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 03:15:46 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <20031222075219.GA3426@performancedrivers.com> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> Message-ID: <20031222000819.10EC.JCARLSON@uci.edu> I just realized I've been forgetting to CC the list. It is a bit long, but I'll cut out the non-relevant portions that everyone has seen before. In chronological order below. I ask the rest of you, does Some sound so crazy? - Josiah *** message from Josiah Carlson on Sun, 21 Dec 2003 19:05:45 -0800 *** > -1. Depending on your algorithm you can already use sys.maxint, or a > large float (e.g. 1e300), or you can write 1e1000 (or 1e300**2) which > on most systems evaluates to a floating point Infinity. And "Some" > sounds like a really strange spelling of "Infinity". There's also a > PEP (PEP 754) to add a proper way to spell floating point Infinity. > Or you could reverse signs and use min(None, -a). > > --Guido van Rossum (home page: http://www.python.org/~guido/) I've read PEP 754, and yes, the author talks about having a proper Infinity, -Infinity and Not a Number (PosInf, NegInf and NaN respectively), but considering that it has been in the 'draft' state since March 28, has not been updated, and it doesn't seem like Python is any closer to having proper definitions now than it was ~9 months ago, I'm not going to put my eggs in the IEEE 754 Infinity basket. In terms of having something useful on a platform, there is only so useful that infinity can be. The below is from the windows verstion of Python 2.3.2 >>> 1e309 1.#INF >>> -1e309 -1.#INF >>> 1.#INF 1.0 >>> max(1e309, 0) 1.#INF >>> min(1e309, 10**309) Traceback (most recent call last): File " ", line 1, in ? OverflowError: long int too large to convert to float The real trick is that it is not so much about being able to have max(Some, a) -> Some, it is about being able to have min(Some, a) -> a. Then you don't need to worry about sys.maxint not being big enough (because there are larger numbers), nor do you need to worry about float casts of large integers producing OverflowErrors. As for an application, many path searches in graphs start out with initializing distances to infinity, Some would be suitable for this. Other algorithms require a negative infinity for this, None has been working fine. As an initialization value, I believe that Some would be quite a useful object. Certainly Some seems like a strange way to spell infinity, but it is a logical infinity, one that you can compare things against, anything, not just Python floats or things that can be cast into Python floats. In terms if PEP 754, if it were actually implemented, certainly there would be some question as to whether Some would be equal to PosInf. If so, then would -Some be equal to NegInf? And if so, would None be equal to NegInf? Maybe such things would require a float cast. I talk about all this in my PEP submission, which isn't up yet. I suppose I probably should have given it a few days before discussing it, but yeah. - Josiah *** end message from Josiah *** *** message from Guido van Rossum on Sun, 21 Dec 2003 20:45:34 -0800 *** > I've read PEP 754, and yes, the author talks about having a proper > Infinity, -Infinity and Not a Number (PosInf, NegInf and NaN > respectively), but considering that it has been in the 'draft' state > since March 28, has not been updated, and it doesn't seem like Python is > any closer to having proper definitions now than it was ~9 months ago, > I'm not going to put my eggs in the IEEE 754 Infinity basket. I expect your PEP will fare no better, and given its weirdness, probably much worse. Do you know of any language that has a similar concept? --Guido van Rossum (home page: http://www.python.org/~guido/) *** end message from Guido *** *** message from Josiah Carlson on Sun, 21 Dec 2003 21:47:34 -0800 *** > I expect your PEP will fare no better, and given its weirdness, > probably much worse. Do you know of any language that has a similar > concept? Is None so weird? All I am proposing is a symmetric value to None. Just like there is a -1 and 1, -Infinity and Infinity, None logically should have Some. Maybe not so much -Some == None, but as I propose in the PEP, possibly Some == not None. - Josiah *** end message from Josiah *** From lists at hlabs.spb.ru Mon Dec 22 06:45:14 2003 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Mon Dec 22 03:45:11 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <1071839460.3fe2f8e4dbbad@mcherm.com> References: <1071839460.3fe2f8e4dbbad@mcherm.com> Message-ID: <3FE6D94A.2060904@hlabs.spb.ru> Michael Chermside wrote: >>Just an idea... >> >>1. python -p package_name >> >> Equivalent to: >> >> import package_name > > (1) Sourceforge is a great place to request feature enhancements. > Suggestions made on this mailing list is likely to be forgotten > sooner or later. Yes, I know, but the request is not formed yet. :-) > > (2) Can you explain WHY you would want this feature? Is there some > use case? I would prefer NOT to have this, because right now > if I'm reading code and it uses "package_name.someFunction()" > I can scan upward for "package_name" to find out what it's > using. With command line imports, I couldn't do that. So unless > you've got a good reason for it, I personally wouldn't want > this feature. Sorry for description absentees. The main idea is to treating package as a program and run package initialization code from command line. The advantage is zipping all program modules in one zip archive and running the program from command line without unzipping it, like Java's jar's. But this idea need more thoughts however... -- Dmitry Vasiliev (dima at hlabs.spb.ru) From troy at gci.net Mon Dec 22 04:03:08 2003 From: troy at gci.net (Troy Melhase) Date: Mon Dec 22 04:03:19 2003 Subject: [Python-Dev] Relative import In-Reply-To: <20031222075219.GA3426@performancedrivers.com> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> Message-ID: <200312220003.08810.troy@gci.net> On Sunday 21 December 2003 10:52 pm, Jack Diederich wrote: > On Sun, Dec 21, 2003 at 08:48:01PM -0800, Guido van Rossum wrote: > > > I like syntax that reads most important left-to-right, so what about > > > from MODULE import NAMES as RENAME searching HOW > > > import NAMES as RENAME from MODULE searching HOW I like Jack's original idea and I've been chewing on it for a few days. His was: import NAMES as RENAME from MODULE searching HOW My humble suggestion: import NAMES import NAMES in MODULE [in HOW] import NAME, OTHER in FOO in RELATIVE import NAME as RENAME in MODULE in ABSOLUTE Playing with it: import sys import path, stat in os import AClass, AValue in AModule in __search__ import AType in Package.Library in __absolute__ It emphasizes the name that's imported and it reduces+reuses a keyword. Reusing "in" is a stretch, of course, and it reading it twice in the same statement might be confusing for some. I don't really like the dot/triple dot notation... the leading punctuation smells perl-ish. :) Only-a-random-two-cents-ly yours, -- Troy Melhase, troy@gci.net -- Merry Christmas! From tismer at tismer.com Mon Dec 22 05:36:07 2003 From: tismer at tismer.com (Christian Tismer) Date: Mon Dec 22 05:34:34 2003 Subject: [Python-Dev] Last chance! In-Reply-To: References: Message-ID: <3FE6C917.6080107@tismer.com> Tim Peters wrote: > [Tim] > >>>Still, the possibility to switch threads across generator >>>resumptions seems darned hard to view as "a feature". I'd advise >>>people not to rely on it . > > > [Christian Tismer] > >>Well, this might only work for generators if it is guaranteed >>that the thread switch doesn't happen while the generator >>is executing. I think this will give an exception. But most >>probably the result will be hard to predict. Explaining that: I was talking about freely using the generator in another thread, which would most likely resume it while it is active, and that's an exception. > I'm not sure I'm following, but nothing in Python now *stops* you from > resuming a suspended generator from another thread. Here's an extreme > example of that: [XTreme example] > Then the body of the single generator is executed by 3 different threads, > more or less randomly, although never by the thread that created the > generator. > > If the body of the generator were relying on thread-local state (like, e.g., > the mass of settings in the decimal arithmetic package's per-thread default > "numeric context" object), the result would sure be unpredictable. It > wouldn't crash Python, it would just be a bitch to debug. Then again, lots > of bad threading practices lead to bitch-to-debug code, so I'm not really > inclined to offer extra protection here. I agree, sure, cross-thread generators need to be designed for that. [tasklets] > If a tasklet can be affected by thread-local state (like numeric context -- > Python has little of this nature now, but it's bound to increase), then a > tasklet will also start to care "who's in charge". So I expect a patch to > reintroduce f_tstate in about a year . It is already there, not in the frames, but in the tasklet. A tasklet can be thread-bound or not. In that case, it keeps a reference to the thread state. But only once, since that information is not local for frames. Being thead bound is either enforced in the case that the tasklet is carrying non-trivial C state, otherwise it can be set as a user option (conservative default, of course). "Nomad threads" might be nice for some background monitoring. This is all theory, I haven't tried any real threads with Stackless so far. But people are mixing tasklets with threads already, so I have to care *somehow*. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From ncoghlan at iinet.net.au Mon Dec 22 05:50:01 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Dec 22 05:50:05 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <3FE6D94A.2060904@hlabs.spb.ru> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <3FE6D94A.2060904@hlabs.spb.ru> Message-ID: <3FE6CC59.10100@iinet.net.au> Dmitry Vasiliev wrote: > The main idea is to treating package as a program and run package > initialization code from command line. The advantage is zipping all > program modules in one zip archive and running the program from command > line without unzipping it, like Java's jar's. But this idea need more > thoughts however... > python runzip.py mypkg.zip where runzip.py is a pure Python script that works as you suggest. I'm fairly sure zipimport would allow you to write the program you describe in pure Python. I've obviously never written such a program, though. It's certainly an interesting packaging idea for small Python programs with several files, but for which a full-blown installer would be overkill. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From paul-python at svensson.org Mon Dec 22 06:48:30 2003 From: paul-python at svensson.org (Paul Svensson) Date: Mon Dec 22 06:48:38 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> Message-ID: <20031222060114.K47423@familjen.svensson.org> Summarizing what I've seen so far: Status quo: [from NAME] import NAME [, NAME] [from NAME] import NAME as NAME There also seems to be consensus on adding: [from NAME] import (NAME [, NAME]) To this, we want to as a way to spell out explicitly how the module is found; for example: -a- by searching sys.path -b- in the current package -c- in the current package or its parent package, recursively -d- among the python standard library modules (and nowhere else) -e- in the current working directory -f- in a specific directory -g- combinations of the above Jack Diederich proposed: >> from MODULE import NAMES as RENAME searching HOW >> or >> import NAMES as RENAME from MODULE searching HOW Guido van Rossum oppined: > from COGS import generate searching relative > >To me, that looks a lot like someone left the commas out of > > from COGS import generate, searching, relative (---) >A separate suggestion is to switch from "from X import Y" to "import Y >from X". This seems a gratuitous change even if it reads or edits >marginally better. Now's not the time to tinker with marginal stuff >-- that should've been done 10 years ago. Not only a gratuitous change; it was always my impression that "from" was at the beginning of the statement for exactly the reason Guido is hesitant about adding "searching" at the end of it. With this in mind, my suggestion is to add the searching syntax should before the "import" keyword rather than at the end of the statement. Reusing an existing keyword, we get: [from NAMES] [in WHERE] import ... WHERE would then be an object, or a list of objects (-g-) specifying the import mechanism: -d- __stdlib__ or maybe stdlib -b- __package__ -c- __search__ -e- '.' -f- '/home/bellman/python' -a- sys.path Without any "from" clause, "in WHERE import ..." reads a little strangely, but I think it should be possible to get used to it, just as we've gotten used to the current "from NAME import ..." Would this work ? /Paul From skip at pobox.com Mon Dec 22 08:08:04 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 22 08:08:34 2003 Subject: [Python-Dev] make test failing in a spectacular wayintest_codeccallbacks In-Reply-To: References: <20031222013937.GA92298@i18n.org> Message-ID: <16358.60596.317938.185310@montanaro.dyndns.org> Tim> [Hye-Shik Chang] >> Just fixed in unicodeobject.c 2.206. Tim> Worked for me -- thanks! Me also... Skip From mcherm at mcherm.com Mon Dec 22 08:33:45 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Dec 22 08:33:47 2003 Subject: [Python-Dev] Got None. Maybe Some? Message-ID: <1072100025.3fe6f2b90ea70@mcherm.com> [Josiah proposes a Some object to be the counterpart of None] [Guido seems reluctant] Josiah: I have a counterproposal. The good news is, it's REALLY easy -- you won't need a PEP, and I am sure that you'll get no objections from anyone on this list. And it'll provide you with a "Some" object that acts like you describe (except the one mention you made of "Some == not None"... that one's a bad idea since right now "True == not None"). Here's the idea: Create a file called "Some.py": -------- Some.py ------------ class _SomeType(object): def __lt__(self, other): return False # never less than anything def __le__(self, other): if other is Some: return True # equal to itself else: return False # less than anything else def __gt__(self, other): if other is Some: return False # equal to itself else: return True # greater than anything else def __ge__(self, other): return True # greater than or equal to anything def __repr__(self): return 'Some' Some = _SomeType() # singleton instance ---------- end -------------- Place this in your python path, then add this line to the top of your scripts: "from Some import Some". If you really want to, you could put code in your site.py file so that you don't even need this import line, but I think in the long run you'll prefer using the import line. I think that this meets your expectations (if I understood them properly), and if not, it can probably be adjusted so that it does. The one thing it doesn't do is that it isn't part of the Standard Library. But that doesn't mean it can't be part of YOUR standard library... something you use all the time all over the place. I think that most people on this list (myself included) would say that the uses of this object are sufficiently esoteric and the naming of it is sufficiently quirky that they wouldn't want it in the language itself, but *everyone* here would stand up for your right to create and use such an object yourself... something which would never be possible in (for instance) Java. Once-we've-got-Some-perhaps-we-should-consider-a-builtin-Milk lly yours, -- Michael Chermside From jepler at unpythonic.net Mon Dec 22 09:03:17 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 22 09:03:21 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <3FE65AF6.6010403@v.loewis.de> References: <20031217233126.GA4692@unpythonic.net> <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> Message-ID: <20031222140255.GH12586@unpythonic.net> On Mon, Dec 22, 2003 at 03:46:14AM +0100, Martin v. Loewis wrote: > I'd think so, yes. The patch needs to be elaborated, and I'd delay > integration to wait for a Redhat response. Well, unlike the last two bugs I filed in redhat bugzilla, this one has already seen some activity. I hope they don't take the line that this is not a bug :( ------- Additional Comments From jakub@redhat.com 2003-12-22 03:53 ------- Why? There is no word about system() in http://www.opengroup.org/onlinepubs/007904975/functions/pthread_atfork.html and there doesn't seem to be anything in http://www.opengroup.org/onlinepubs/007904975/functions/system.html that would mandate system() to be implemented using fork() function. ------- Additional Comments From jepler@unpythonic.net 2003-12-22 08:01 ------- I base my claim that pthread_atfork() should be called by system() from reading http://www.opengroup.org/onlinepubs/007904975/functions/system.html The third paragraph under the heading "description" says this: "[CX] The environment of the executed command shall be as if a child process were created using fork(), and the child process invoked the sh utility using execl() as follows [...]" The rationale section expands on what the "environment of the executed command" is: "IEEE Std 1003.1-2001 places additional restrictions on system(). It requires that if there is a command language interpreter, the environment must be as specified by fork() and exec. This ensures, for example, that close-on- exec works, that file locks are not inherited, and that the process ID is different." The pthread_atfork web page says it "provides multi-threaded application programs with a standard mechanism for protecting themselves from fork() calls in a library routine or the application itself." I believe that system() is one such library routine, since it is specified to create an environment "as specified by fork()". This issue arose in the context of Python. For various reasons, signals are blocked in all threads besides the main thread. This means that in processes generated by fork() from subthreads, all signals are blocked, which leads to undesired behavior. Using the child argument of pthread_atfork() allows signals to be unblocked when using fork()+exec() to execute an external program, but not when system() is used. From barry at python.org Mon Dec 22 09:55:16 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 22 09:55:42 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312220003.08810.troy@gci.net> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> Message-ID: <1072104915.18828.47.camel@anthem> On Mon, 2003-12-22 at 04:03, Troy Melhase wrote: > I don't really like the dot/triple dot notation... the leading punctuation > smells perl-ish. :) I don't either, although my biggest problem with it is that it isn't clear just by looking what the punctuation actually means. I just fear this will be a permanent head scratcher. Here's my entry in the sure-to-be-shot-down-by-the-BDFL-alternative -syntax game: import foo by absolute import foo.bar by relative as baz import foo by searching as bar import foo by absolute from bar import foo as bar by absolute from baz IOW, 'by' is a reserved word in the import statement context, and it specifies how to search for modules. The question then is whether 'absolute', 'relative', and 'searching' are also reserved words or built-ins and how those names are equated with the module location algorithms. -Barry From guido at python.org Mon Dec 22 10:24:30 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 10:27:43 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: Your message of "Sun, 21 Dec 2003 21:47:34 PST." <20031221212332.10E9.JCARLSON@uci.edu> References: <20031221183714.10E6.JCARLSON@uci.edu> <200312220445.hBM4jYS17286@c-24-5-183-134.client.comcast.net> <20031221212332.10E9.JCARLSON@uci.edu> Message-ID: <200312221527.hBMFRYt17987@c-24-5-183-134.client.comcast.net> > Is None so weird? All I am proposing is a symmetric value to None. > Just like there is a -1 and 1, -Infinity and Infinity, None logically > should have Some. Maybe not so much -Some == None, but as I propose in > the PEP, possibly Some == not None. None's purposr is not to be less than all numbers (in fact in the past it wasn't); it's to be "not any object", like nil or NULL in other languages. I don't see a similar broad role for Some. And I repeat my question: which other language has that concept? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 10:36:09 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 10:36:14 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Mon, 22 Dec 2003 09:55:16 EST." <1072104915.18828.47.camel@anthem> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> Message-ID: <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> > I don't either, although my biggest problem with it is that it isn't > clear just by looking what the punctuation actually means. I just > fear this will be a permanent head scratcher. This argument by itself has very little value. The question isn't whether it's self-describing (very few features beyond decimal integers are); the question is whether it's easy to remember. There are two subclasses of that: easy to remember what it means when you next see it, and easy to remember how to spell it when you next need it. IMO the dot syntax does fine here. > Here's my entry in the sure-to-be-shot-down-by-the-BDFL-alternative > -syntax game: There have been so many alternatives proposed this morning that I have to spend some time comparing them all... --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Mon Dec 22 10:44:46 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 22 10:44:55 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> Message-ID: <1072107885.18828.76.camel@anthem> On Mon, 2003-12-22 at 10:36, Guido van Rossum wrote: > > I don't either, although my biggest problem with it is that it isn't > > clear just by looking what the punctuation actually means. I just > > fear this will be a permanent head scratcher. > > This argument by itself has very little value. The question isn't > whether it's self-describing (very few features beyond decimal > integers are); the question is whether it's easy to remember. There > are two subclasses of that: easy to remember what it means when you > next see it, and easy to remember how to spell it when you next need > it. IMO the dot syntax does fine here. Sorry, it was a pre-coffee argument. To me, the dot-syntax doesn't really tell me whether it forces a relative search or an absolute search. The dot tells me that something different is happening, so maybe when absolute is the default, it tells me there's a switch to relative searching going on. I'm not sure though, so I do think it may potentially fail the "easy to remember what it means when you next see it" test. OTOH, it would be nice to see a Greg Wilson-style newbie poll. > > Here's my entry in the sure-to-be-shot-down-by-the-BDFL-alternative > > -syntax game: > > There have been so many alternatives proposed this morning that I have > to spend some time comparing them all... Hopefully, this stuff will make it into the PEP so you don't have to troll through suggestions buried in endless email threads. -Barry From guido at python.org Mon Dec 22 11:29:31 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 11:29:35 2003 Subject: [Python-Dev] OpenBSD anyone? Message-ID: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> John Draper (the famous Capt'n Crunch) reports that on his OpenBSD box, he can't build a debug version of Python 2.3(.3c1, but I doubt that matters) unless he tweaks the configure file to add -O3 to the C flags. That sounds fishy to me, but he insists that's how it is, and he's reproduced it from a clean tarball. Debugging it further doesn't really seem to be within John's power; the -O3 trick was found by his partner Steve who's temporarily unavailable. The symptom is that without -O3, a "python" binary gets built but its use to run setup.py segfaults. Is there anyone here with access to an OpenBSD box who could try to reproduce this? --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Mon Dec 22 11:46:01 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 22 11:46:04 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <200312221527.hBMFRYt17987@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > ... > And I repeat my question: which other language has that concept? Variations on None were debated pretty exhaustively during Python's first year (yes, that was a while ago ). Some obscure functional languages have "top" and "bottom" values. There's a partial ordering among values, and "top" and "bottom" are added to make a lattice out of that. "bottom" is usually identified as "the error value", and f(..., bottom, ...) is defined to return bottom for all functions f then. That's not really like None, it's more like 754's NaN (although NaN**0.0 in C99 is defined to return 1, not NaN, under the theory that it doesn't matter if an error occurred during the computation of the base, because base**0 is 1 no matter what base should have been). Aaron Watters also identified several distinct detailed semantics for "missing value" markers in assorted database languages, which are closer to what Python's None usually intends ("I'm not really here, but *something* has to be here so I'm it"). If all that's really wanted is an object that compares larger than anything else, it's easy to define such a beast (and I see Michael Chermside posted code for that already). If more than that is wanted, then it becomes more a collection of arbitrary behaviors, and then different apps will want different arbitrary behaviors. None is handy and I like it, but it's arbitrary enough in some cases that it takes time to get used to its quirks (e.g., why is a value that's "not really there" considered to be false? why not? sometimes that's what you want; sometimes you'd rather get an exception; None picks an arbitrary answer and sticks to it), so I don't want see more things like it proliferate (one is enough -- but not too many ). From skip at pobox.com Mon Dec 22 11:50:24 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 22 11:50:52 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: References: <16356.62036.22726.310077@montanaro.dyndns.org> Message-ID: <16359.8400.767266.692477@montanaro.dyndns.org> >> Sounds like a plan. I modify my local source and see if it affects >> anything. Tim> It should work fine. Done. UCHAR_MAX is now required in Python.h, and it's required to be 255. That may open up a couple (small) optimization opportunities (places where gcc 3.3 says explicit tests against char values will always be true or false). I'll leave it for others to make those tweaks if they so desire. Note that some of those gcc messages are probably related to larger types (e.g. unsigned short) or depend on the presence or absence of other macro definitions (e.g. Py_UNICODE_WIDE). Should you wish to code any of these speedups, make sure you get the cpp tests right. Guido won't forgive you if you don't. Skip From blunck at gst.com Mon Dec 22 12:02:57 2003 From: blunck at gst.com (Christopher Blunck) Date: Mon Dec 22 12:03:01 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> Message-ID: <20031222170257.GA31352@homer.gst.com> On Mon, Dec 22, 2003 at 08:29:31AM -0800, Guido van Rossum wrote: > John Draper (the famous Capt'n Crunch) reports > that on his OpenBSD box, he can't build a debug version of Python > 2.3(.3c1, but I doubt that matters) unless he tweaks the configure > file to add -O3 to the C flags. That sounds fishy to me, but he > insists that's how it is, and he's reproduced it from a clean tarball. > Debugging it further doesn't really seem to be within John's power; > the -O3 trick was found by his partner Steve who's temporarily > unavailable. > > The symptom is that without -O3, a "python" binary gets built but its > use to run setup.py segfaults. Is there anyone here with access to an > OpenBSD box who could try to reproduce this? OpenBSD 3.3 builds both HEAD and r233c1 fine for me. This is code out of the anoncvs repository btw. -c -- 12:00:00 up 21 days, 1:44, 12 users, load average: 0.01, 0.07, 0.09 From aahz at pythoncraft.com Mon Dec 22 12:07:02 2003 From: aahz at pythoncraft.com (Aahz) Date: Mon Dec 22 12:07:06 2003 Subject: [Python-Dev] Relative import In-Reply-To: <1072107885.18828.76.camel@anthem> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> <1072107885.18828.76.camel@anthem> Message-ID: <20031222170702.GA11581@panix.com> On Mon, Dec 22, 2003, Barry Warsaw wrote: > On Mon, 2003-12-22 at 10:36, Guido van Rossum wrote: >>> >>> Here's my entry in the sure-to-be-shot-down-by-the-BDFL-alternative >>> -syntax game: >> >> There have been so many alternatives proposed this morning that I have >> to spend some time comparing them all... > > Hopefully, this stuff will make it into the PEP so you don't have to > troll through suggestions buried in endless email threads. That's the plan; I expect that people will review the PEP to make sure their pet proposal is listed. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From aleaxit at yahoo.com Mon Dec 22 12:07:14 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Mon Dec 22 12:07:20 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <3FE6D94A.2060904@hlabs.spb.ru> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <3FE6D94A.2060904@hlabs.spb.ru> Message-ID: <200312221807.14352.aleaxit@yahoo.com> On Monday 22 December 2003 12:45 pm, Dmitry Vasiliev wrote: ... > >>1. python -p package_name > >> > >> Equivalent to: > >> > >> import package_name ... > The main idea is to treating package as a program and run package > initialization code from command line. The advantage is zipping all > program modules in one zip archive and running the program from command > line without unzipping it, like Java's jar's. But this idea need more > thoughts however... Couldn't you use: python -c "import package_name" for this purpose even today? Alex From tjreedy at udel.edu Mon Dec 22 12:20:19 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Mon Dec 22 12:20:25 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net><20031222075219.GA3426@performancedrivers.com> <20031222000819.10EC.JCARLSON@uci.edu> Message-ID: "Josiah Carlson" wrote in message news:20031222000819.10EC.JCARLSON@uci.edu... > Is None so weird? All I am proposing is a symmetric value to None. To me that would be All of AllType. Some is in between. Can you think of any uses for All other than being max element? TJR From perky at i18n.org Mon Dec 22 12:26:25 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Mon Dec 22 12:26:33 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <16359.8400.767266.692477@montanaro.dyndns.org> References: <16356.62036.22726.310077@montanaro.dyndns.org> <16359.8400.767266.692477@montanaro.dyndns.org> Message-ID: <20031222172625.GA9189@i18n.org> On Mon, Dec 22, 2003 at 10:50:24AM -0600, Skip Montanaro wrote: > >> Sounds like a plan. I modify my local source and see if it affects > >> anything. > > Tim> It should work fine. > > Done. UCHAR_MAX is now required in Python.h, and it's required to be 255. Thanks! > That may open up a couple (small) optimization opportunities (places where > gcc 3.3 says explicit tests against char values will always be true or > false).I'll leave it for others to make those tweaks if they so desire. > Note that some of those gcc messages are probably related to larger types > (e.g. unsigned short) or depend on the presence or absence of other macro > definitions (e.g. Py_UNICODE_WIDE). Should you wish to code any of these > speedups, make sure you get the cpp tests right. Guido won't forgive you if > you don't. I see. Hehe :-) BTW, I wonder whether we have any good way to get C integer types with explicitly defined size such as u_int16_t or int64_t of POSIX. I got a report from an OpenBSD/sparc64 user that he want to change an unsigned int variable to int type which is used for socket.inet_aton. (Modules/socketmodule.c:2887). But I think int type isn't appropriate due to fearing its being 64-bits on some 64bit platforms. Also in_addr_t, POSIX return type of inet_addr(3), is not so portable. If we have types like u_int32_t, it will be very helpful to cope with these sort of issues. Hye-Shik From tjreedy at udel.edu Mon Dec 22 12:29:47 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Mon Dec 22 12:29:48 2003 Subject: [Python-Dev] Re: Relative import References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net><20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> Message-ID: "Troy Melhase" wrote in message news:200312220003.08810.troy@gci.net... > My humble suggestion: > > import NAMES > import NAMES in MODULE [in HOW] > import NAME, OTHER in FOO in RELATIVE > import NAME as RENAME in MODULE in ABSOLUTE I had same idea of reusing 'in', but with relative, absolute changed to perhaps RelMode, AbsMode so as to grammatically flow better. 'in NAME' could then indicate place in package to begin search. 'in Package' and 'in Path' are clearer (more explicit) alternatives to relative and absolute. Terry J. Reedy From guido at python.org Mon Dec 22 12:33:55 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 12:34:01 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: Your message of "Mon, 22 Dec 2003 11:46:01 EST." References: Message-ID: <200312221733.hBMHXtZ18242@c-24-5-183-134.client.comcast.net> > None is handy and I like it, but it's arbitrary enough in some cases > that it takes time to get used to its quirks (e.g., why is a value > that's "not really there" considered to be false? why not? > sometimes that's what you want; sometimes you'd rather get an > exception; None picks an arbitrary answer and sticks to it), so I > don't want see more things like it proliferate (one is enough -- but > not too many ). None inherits most of its semantics from C's NULL -- that's where None being false comes from. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 12:37:29 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 12:37:37 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Mon, 22 Dec 2003 12:02:57 EST." <20031222170257.GA31352@homer.gst.com> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> Message-ID: <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> > > The symptom is that without -O3, a "python" binary gets built but its > > use to run setup.py segfaults. Is there anyone here with access to an > > OpenBSD box who could try to reproduce this? > > OpenBSD 3.3 builds both HEAD and r233c1 fine for me. Thanks! That's what I suspected, but it's always good to have confirmation. :) > This is code out of the anoncvs repository btw. Shouldn't matter. If anybody *does* see the bogus behavior and feels like debugging it, I'd appreciate it! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 12:39:00 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 12:39:04 2003 Subject: [Python-Dev] Relative import In-Reply-To: Your message of "Mon, 22 Dec 2003 12:07:02 EST." <20031222170702.GA11581@panix.com> References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> <1072107885.18828.76.camel@anthem> <20031222170702.GA11581@panix.com> Message-ID: <200312221739.hBMHd0G18293@c-24-5-183-134.client.comcast.net> > > Hopefully, this stuff will make it into the PEP so you don't have to > > troll through suggestions buried in endless email threads. > > That's the plan; I expect that people will review the PEP to make sure > their pet proposal is listed. Thanks -- I'll stop tracking this thread closely then until I see the PEP. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Mon Dec 22 12:50:24 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 22 12:50:29 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <200312221733.hBMHXtZ18242@c-24-5-183-134.client.comcast.net> Message-ID: [Guido] > None inherits most of its semantics from C's NULL -- that's where None > being false comes from. I know that's where it came from, but Python adds its own twists. If it really wanted to act like C's NULL, then print >> None, "oops" should segfault . There's nothing "wrong" about None evaluating to false in a Boolean context, it's simply one choice that *could* have made -- and better than most. From tim.one at comcast.net Mon Dec 22 13:10:36 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 22 13:10:41 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <20031222172625.GA9189@i18n.org> Message-ID: [Hye-Shik Chang] > BTW, I wonder whether we have any good way to get C integer types > with explicitly defined size such as u_int16_t or int64_t of POSIX. We do not, and C doesn't guarantee that any such types exist. For example, while the Crays we recently talked about do support the illusion of 8-bit char, they have no 16-bit or 32-bit integral types (just 8 and 64); that's fine by the C standard. C99 defines a large pile of names, for things like "exactly 16 bits *if* such a thing exists", "smallest integral type holding at least 16 bits (this must exist)", and "fastest integral type holding at least 16 bits (which also must exist)". Since not all C compilers support these names yet, they can't be used directly in Python. If one is needed, it's intended to be added, in pyport.h, following this comment: /* typedefs for some C9X-defined synonyms for integral types. * * The names in Python are exactly the same as the C9X names, except * with a Py_ prefix. Until C9X is universally implemented, this is the * only way to ensure that Python gets reliable names that don't * conflict with names in non-Python code that are playing their own * tricks to define the C9X names. * * NOTE: don't go nuts here! Python has no use for *most* of the C9X * integral synonyms. Only define the ones we actually need. */ > ... > If we have types like u_int32_t, it will be very helpful to cope > with these sort of issues. Not really -- any use of an "exactly N bits" name will make the code uncompilable on some platform. From kbk at shore.net Mon Dec 22 13:57:18 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon Dec 22 13:57:23 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 09:37:29 -0800") References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > If anybody *does* see the bogus behavior and feels like debugging it, > I'd appreciate it! I've been running cvs Python but not previously with debug switched on. I get a core when using OPT=-g. (Note that I'm running Patch 850977 "Modify setup.py to detect Tcl/Tk on BSD", though I wouldn't think it would cause this, and I doubt he's using it.) Investigating. Hints appreciated! -- KBK Script started on Mon Dec 22 13:30:26 2003 hydra /home/kbk/PYTHON/python/dist/src$ uname -a OpenBSD hydra.localdomain 3.3 GENERIC_RAID#0 i386 hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ make clean find . -name '*.o' -exec rm -f {} ';' find . -name '*.s[ol]' -exec rm -f {} ';' find . -name '*.py[co]' -exec rm -f {} ';' hydra /home/kbk/PYTHON/python/dist/src$ make OPT=-g gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Modules/python.o Modules/python.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/node.o Parser/node.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/parser.o Parser/parser.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/parsetok.o Parser/parsetok.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/bitset.o Parser/bitset.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/metagrammar.o Parser/metagrammar.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/firstsets.o Parser/firstsets.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/grammar.o Parser/grammar.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/pgen.o Parser/pgen.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/myreadline.o Parser/myreadline.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/tokenizer.o Parser/tokenizer.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/abstract.o Objects/abstract.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/boolobject.o Objects/boolobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/bufferobject.o Objects/bufferobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/cellobject.o Objects/cellobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/classobject.o Objects/classobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/cobject.o Objects/cobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/complexobject.o Objects/complexobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/descrobject.o Objects/descrobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/enumobject.o Objects/enumobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/fileobject.o Objects/fileobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/floatobject.o Objects/floatobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/frameobject.o Objects/frameobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/funcobject.o Objects/funcobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/intobject.o Objects/intobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/iterobject.o Objects/iterobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/listobject.o Objects/listobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/longobject.o Objects/longobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/dictobject.o Objects/dictobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/methodobject.o Objects/methodobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/moduleobject.o Objects/moduleobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/object.o Objects/object.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/obmalloc.o Objects/obmalloc.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/rangeobject.o Objects/rangeobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/sliceobject.o Objects/sliceobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/stringobject.o Objects/stringobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/structseq.o Objects/structseq.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/tupleobject.o Objects/tupleobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/typeobject.o Objects/typeobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/weakrefobject.o Objects/weakrefobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/unicodeobject.o Objects/unicodeobject.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Objects/unicodectype.o Objects/unicodectype.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/bltinmodule.o Python/bltinmodule.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/exceptions.o Python/exceptions.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/ceval.o Python/ceval.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/mysnprintf.o Python/mysnprintf.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/tokenizer_pgen.o Parser/tokenizer_pgen.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c gcc -pthread -g Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o -lpthread -lutil -o Parser/pgen Parser/pgen ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/compile.o Python/compile.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/codecs.o Python/codecs.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/errors.o Python/errors.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/frozen.o Python/frozen.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/frozenmain.o Python/frozenmain.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/future.o Python/future.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getargs.o Python/getargs.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getcompiler.o Python/getcompiler.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getcopyright.o Python/getcopyright.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getmtime.o Python/getmtime.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -DPLATFORM='"openbsd3"' -o Python/getplatform.o ./Python/getplatform.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getversion.o Python/getversion.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/graminit.o Python/graminit.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/import.o Python/import.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -I. -o Python/importdl.o ./Python/importdl.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/marshal.o Python/marshal.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/modsupport.o Python/modsupport.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/mystrtoul.o Python/mystrtoul.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/pyfpe.o Python/pyfpe.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/pystate.o Python/pystate.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/pythonrun.o Python/pythonrun.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/structmember.o Python/structmember.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/symtable.o Python/symtable.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/sysmodule.o Python/sysmodule.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/traceback.o Python/traceback.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/getopt.o Python/getopt.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/dynload_shlib.o Python/dynload_shlib.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Python/thread.o Python/thread.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Modules/config.o Modules/config.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -DPYTHONPATH='":plat-openbsd3:lib-tk"' -DPREFIX='"/usr/local"' -DEXEC_PREFIX='"/usr/local"' -DVERSION='"2.4"' -DVPATH='""' -o Modules/getpath.o ./Modules/getpath.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Modules/main.o Modules/main.c gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -o Modules/gcmodule.o Modules/gcmodule.c gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/threadmodule.c -o Modules/threadmodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/signalmodule.c -o Modules/signalmodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/posixmodule.c -o Modules/posixmodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/errnomodule.c -o Modules/errnomodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/_sre.c -o Modules/_sre.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/_codecsmodule.c -o Modules/_codecsmodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/zipimport.c -o Modules/zipimport.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/symtablemodule.c -o Modules/symtablemodule.o gcc -pthread -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -c ./Modules/xxsubtype.c -o Modules/xxsubtype.o if test -f buildno; then expr `cat buildno` + 1 >buildno1; mv -f buildno1 buildno; else echo 1 >buildno; fi gcc -pthread -c -fno-strict-aliasing -g -I. -I./Include -DPy_BUILD_CORE -DBUILD=`cat buildno` -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c rm -f libpython2.4.a ar cr libpython2.4.a Modules/getbuildinfo.o ar cr libpython2.4.a Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o Parser/tokenizer.o ar cr libpython2.4.a Objects/abstract.o Objects/boolobject.o Objects/bufferobject.o Objects/cellobject.o Objects/classobject.o Objects/cobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/intobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/methodobject.o Objects/moduleobject.o Objects/object.o Objects/obmalloc.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/stringobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/weakrefobject.o Objects/unicodeobject.o Objects/unicodectype.o ar cr libpython2.4.a Python/bltinmodule.o Python/exceptions.o Python/ceval.o Python/compile.o Python/codecs.o Python/errors.o Python/frozen.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getmtime.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o Python/mystrtoul.o Python/mysnprintf.o Python/pyfpe.o Python/pystate.o Python/pythonrun.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/traceback.o Python/getopt.o Python/dynload_shlib.o Python/thread.o ar cr libpython2.4.a Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o ar cr libpython2.4.a Modules/threadmodule.o Modules/signalmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/_sre.o Modules/_codecsmodule.o Modules/zipimport.o Modules/symtablemodule.o Modules/xxsubtype.o ranlib libpython2.4.a gcc -pthread -o python Modules/python.o libpython2.4.a -lpthread -lutil -lm ./Modules/posixmodule.c:5825: warning: tempnam() possibly used unsafely; consider using mkstemp() ./Modules/posixmodule.c:5872: warning: tmpnam() possibly used unsafely; consider using mkstemp() case $MAKEFLAGS in *-s*) CC='gcc -pthread' LDSHARED='ld -Bshareable' OPT='-g' ./python -E ./setup.py -q build;; *) CC='gcc -pthread' LDSHARED='ld -Bshareable' OPT='-g' ./python -E ./setup.py build;; esac Memory fault (core dumped) *** Error code 139 Stop in /home/kbk/PYTHON/python/dist/src (line 335 of Makefile). hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ cvs st configure.in =================================================================== File: configure.in Status: Up-to-date Working revision: 1.442 Repository revision: 1.442 /cvsroot/python/python/dist/src/configure.in,v Sticky Tag: (none) Sticky Date: (none) Sticky Options: (none) hydra /home/kbk/PYTHON/python/dist/src$ Script done on Mon Dec 22 13:31:57 2003 Script started on Mon Dec 22 13:40:31 2003 hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ hydra /home/kbk/PYTHON/python/dist/src$ # [re-build with make clean && make] hydra /home/kbk/PYTHON/python/dist/src$ ./python Python 2.4a0 (#35, Dec 22 2003, 13:38:03) [GCC 2.95.3 20010125 (prerelease, propolice)] on openbsd3 Type "help", "copyright", "credits" or "license" for more information. >>> hydra /home/kbk/PYTHON/python/dist/src$ Script done on Mon Dec 22 13:41:26 2003 From devin at whitebread.org Mon Dec 22 13:36:45 2003 From: devin at whitebread.org (Devin) Date: Mon Dec 22 14:11:48 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312221739.hBMHd0G18293@c-24-5-183-134.client.comcast.net> Message-ID: I've been monitoring all of this discussion about relative imports, and have my own humble idea about how to spell relative imports. The first part stems from Guido's idea about using a '.' at the beginning of a relative import. It seemed a bit weird at first, but I eventually became very comfortable with the idea after thinking about it in respect to absolute imports. I'll explain what I mean. If you currently want to import a module 'a.b.c.d', you write: import a.b.c.d Now, if you're in the module 'a.b', you could still write: import a.b.c.d ... but the 'a.b' part is redundant because you're already in the module 'a.b'. If you take away 'a.b', then you end up with: import .c.d This flowed well in my head, and looks natural. However, the suggestion about '.' or '..' referring to the parent looks _ugly and unnatural_. Let's say '.' was used to spell "parent module", and let's assume that the module 'a.b.c.d' wants to import 'a.b.e'. The absolute import would be spelled: import a.b.e ... and the relative import would be spelled: import .....e # .[parent].[parent].e Yuck! (no offense :) What's needed, I believe, is an identifier/keyword/symbol that spells 'parent' that isn't '.'. Assuming that symbol is represented by '[parent]', the above relative import would look like: import .[parent].[parent].e ... which is a hell lot more clear IMHO. My initial idea is to spell parent '__parent__', and to have '__parent__' be a module-level variable in each module that refers to the parent module (where a top-level module might have '__parent__' refer to itself, or to None). Continuing with the above example, the relative import would be spelled: import .__parent__.__parent__.e ... but I'm not completely convinced that there's not a better identifier/keyword/symbol that could be substituted for '[parent]'. If I'm completely off track, then tell me and I'll continue to lurk until I get a better feel for the happenings of python-dev. -- Devin devin@whitebread.org http://www.whitebread.org/ From blunck at gst.com Mon Dec 22 14:26:53 2003 From: blunck at gst.com (Christopher Blunck) Date: Mon Dec 22 14:26:56 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> Message-ID: <20031222192653.GA31766@homer.gst.com> On Mon, Dec 22, 2003 at 01:57:18PM -0500, Kurt B. Kaiser wrote: > Guido van Rossum writes: > > > If anybody *does* see the bogus behavior and feels like debugging it, > > I'd appreciate it! > > I've been running cvs Python but not previously with debug switched on. > > I get a core when using OPT=-g. > > (Note that I'm running Patch 850977 "Modify setup.py to detect Tcl/Tk on BSD", > though I wouldn't think it would cause this, and I doubt he's using it.) > > Investigating. Hints appreciated! Kurt- Interesting... When I do OPT=-g, I get a Memory fault (core dumped) when linking python. Is that when you receive the core dump? -c -- 14:10:00 up 21 days, 3:54, 12 users, load average: 0.00, 0.02, 0.00 From theller at python.net Mon Dec 22 14:28:24 2003 From: theller at python.net (Thomas Heller) Date: Mon Dec 22 14:28:27 2003 Subject: [Python-Dev] Improve Python for embedding Message-ID: <1xqwodnr.fsf@python.net> I find the current situation when 'embedding' Python rather unsatisfying. 1. Careful preparation (hacking?) of environment variables is needed to avoid that Py_Initialize() takes the default actions needed for the standard Python interpreter (for example, to avoid the default sys.path). 2. A suggestion from /F on c.l.p was to change pythonw.exe so that instead of hiding all output to stderr a console window would be opened to show the tracebacks. This requires to construct an object with a 'write' method doing all this magic, and it must be done after the call to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the code of Py_Main(), it seems impossible without rewriting most of the code. Possible improvements I currently can think of are: - Provide a function like Py_InitializeEx(), which accepts parameters specifying an initial sys.path, optimize flag, verbose flag, and so on. - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), or let Py_Main() accept a callback function which will be called jsut after Py_Initialize() has been run. I'm not ready to write a PEP, but if there are people sharing this feeling I would suggest we create a wiki entry somewhere acting as a temporary brainstorming area. Thomas From jcarlson at uci.edu Mon Dec 22 14:25:32 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 14:29:05 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <1072100025.3fe6f2b90ea70@mcherm.com> References: <1072100025.3fe6f2b90ea70@mcherm.com> Message-ID: <20031222111017.0A6D.JCARLSON@uci.edu> > [Josiah proposes a Some object to be the counterpart of None] > > [Guido seems reluctant] > > Josiah: I have a counterproposal. The good news is, it's REALLY easy -- you > won't need a PEP, and I am sure that you'll get no objections from anyone > on this list. And it'll provide you with a "Some" object that acts like > you describe (except the one mention you made of "Some == not None"... > that one's a bad idea since right now "True == not None"). -- snip -- Yes, renaming the class and being more explicit about the comparisons (and tossing the hash function, which turns out to be unneeded) does pretty much what I want it to. But then again, I've had Some since I thought of it. I'm all about this free software idea, and I know that it could be useful to others. I suppose I'll just post information about Some and a sample implementation (combination of yours and mine) on my website or somesuch. - Josiah From guido at python.org Mon Dec 22 14:29:40 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:29:45 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Mon, 22 Dec 2003 14:26:53 EST." <20031222192653.GA31766@homer.gst.com> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> Message-ID: <200312221929.hBMJTef18533@c-24-5-183-134.client.comcast.net> > Interesting... When I do OPT=-g, I get a Memory fault (core dumped) > when linking python. Is that when you receive the core dump? >From looking at his output, he gets it just after the linking, when using the freshly linked python executable to run the setup.py script. Are you saying you didn't try the debug build? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 14:33:29 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:33:35 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 20:28:24 +0100." <1xqwodnr.fsf@python.net> References: <1xqwodnr.fsf@python.net> Message-ID: <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> > I find the current situation when 'embedding' Python rather unsatisfying. I'm interested in improve it. Can you explain what exactly your 'embedding' requirements are? > 1. Careful preparation (hacking?) of environment variables is needed to > avoid that Py_Initialize() takes the default actions needed for the > standard Python interpreter (for example, to avoid the default sys.path). I guess it depends on the embedding needs. Some embedded uses would be quite happy with the default sys.path. > 2. A suggestion from /F on c.l.p was to change pythonw.exe so that > instead of hiding all output to stderr a console window would be opened > to show the tracebacks. This requires to construct an object with a > 'write' method doing all this magic, and it must be done after the call > to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the > code of Py_Main(), it seems impossible without rewriting most of the > code. I'm not sure what this has to do with embedding -- can you clarify? > Possible improvements I currently can think of are: > > - Provide a function like Py_InitializeEx(), which accepts parameters > specifying an initial sys.path, optimize flag, verbose flag, and so on. Except for sys.path, you can set most flags directly before calling Py_Initialize(), so I'm not sure if there's a need to pass these into Py_Initialize(). > - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), > or let Py_Main() accept a callback function which will be called jsut > after Py_Initialize() has been run. OK, now I'm confused. If you're embedding Python, why would you be using Py_Main() at all? > I'm not ready to write a PEP, but if there are people sharing this > feeling I would suggest we create a wiki entry somewhere acting as a > temporary brainstorming area. Go ahead, but count me out of the wiki -- wikis don't work for discussions for me. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 14:35:42 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:35:48 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: Your message of "Mon, 22 Dec 2003 11:25:32 PST." <20031222111017.0A6D.JCARLSON@uci.edu> References: <1072100025.3fe6f2b90ea70@mcherm.com> <20031222111017.0A6D.JCARLSON@uci.edu> Message-ID: <200312221935.hBMJZgD18583@c-24-5-183-134.client.comcast.net> > But then again, I've had Some since I thought of it. I'm all about this > free software idea, and I know that it could be useful to others. I > suppose I'll just post information about Some and a sample > implementation (combination of yours and mine) on my website or somesuch. Just please, Please, PLEASE come up with a better name. Some is just too mysterious. Remember that it is *not* in most senses the opposite of None. And even if it was, Some would be a poor choice of name. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Mon Dec 22 14:41:26 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 14:41:30 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: (Tim Peters's message of "Sat, 20 Dec 2003 21:52:03 -0500") References: Message-ID: <2mptegej2x.fsf@starship.python.net> "Tim Peters" writes: > [Skip Montanaro] >> Isn't that supposed to always be defined in limits.h or is UCHAR_MAX >> not a standard macro? > > Yes, it's supposed to be there. OTOH, so is limits.h, i.e. the > HAVE_LIMITS_H test shouldn't be necessary either. As Martin regularly points out, we have rather too much of this kind of thing. Given that we demand ANSI C, we could surely lose HAVE_LIMITS_H, and probably much other cruft. People who have to contend with broken platforms might have a different view. I guess there's no real reason to churn, but when crap like this gets in the way we should strive to kill it. Cheers, mwh -- Any form of evilness that can be detected without *too* much effort is worth it... I have no idea what kind of evil we're looking for here or how to detect is, so I can't answer yes or no. -- Guido Van Rossum, python-dev From tree at basistech.com Mon Dec 22 14:40:12 2003 From: tree at basistech.com (Tom Emerson) Date: Mon Dec 22 14:42:28 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <1xqwodnr.fsf@python.net> References: <1xqwodnr.fsf@python.net> Message-ID: <16359.18588.450934.263664@magrathea.basistech.com> Thomas Heller writes: > I find the current situation when 'embedding' Python rather unsatisfying. Indeed. See patch 849278 for some of the work we did to deal with this. > 1. Careful preparation (hacking?) of environment variables is needed to > avoid that Py_Initialize() takes the default actions needed for the > standard Python interpreter (for example, to avoid the default sys.path). I've done this in the patches submitted (by a coworker) above. The idea is to separate the embedded interpreter *completely* from the environment. > 2. A suggestion from /F on c.l.p was to change pythonw.exe so that > instead of hiding all output to stderr a console window would be opened > to show the tracebacks. This requires to construct an object with a > 'write' method doing all this magic, and it must be done after the call > to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the > code of Py_Main(), it seems impossible without rewriting most of the > code. Yes, this is a problem with the embedded interpreter where errors may be written to stderr without regard to the calling application. Further, it is possible for the interpreter to call abort(), which is even worse. > Possible improvements I currently can think of are: > > - Provide a function like Py_InitializeEx(), which accepts parameters > specifying an initial sys.path, optimize flag, verbose flag, and so on. See the patch. > - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), > or let Py_Main() accept a callback function which will be called jsut > after Py_Initialize() has been run. Yup. > I'm not ready to write a PEP, but if there are people sharing this > feeling I would suggest we create a wiki entry somewhere acting as a > temporary brainstorming area. Absolutely: count me in. -tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever" From kbk at shore.net Mon Dec 22 14:45:43 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon Dec 22 14:45:49 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 11:07:43 -0800") References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > Very interesting! Had you done a previous non-debug build in the same > directory? (Even if you did a "make clobber" before restarting -- you > never know what leave-behind could cause this.) Yes. I did do a make clobber && ./configure before building the debug version, but it's not a clean checkout. OK, just did a clean checkout, no patch, and got the same result. > The next thing I'd try would be to start the python executable that >was build under gdb and play around in it, like this: > > $ gdb ./python > [...] > (gdb) run > Starting program: /home/guido/projects/python/dist/src/debug/python > [New Thread 1074895104 (LWP 30404)] > Python 2.4a0 (#2, Dec 22 2003, 11:02:19) > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> print 2+2 >>>> # try various things that don't need external modules except sys >>>> # define a function, etc... >>>> # if that doesn't segfault, try: >>>> from test import autotest > [...] > > Eventually I expect you'd get a segfault; at that point you can use > the gdb 'bt' command to get a stack trace. Hopefully it'll point to > some innocent C code that gets mistreated by the non-debug compiler... It doesn't get as far as the banner: Script started on Mon Dec 22 14:42:19 2003 hydra /home/kbk/proj/sandbox/python_clean$ gdb ./python GNU gdb 4.16.1 Copyright 1996 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-openbsd3.3"... (gdb) run Starting program: /home/kbk/proj/sandbox/python_clean/./python Program received signal SIGSEGV, Segmentation fault. 0x401900a0 in strchr () (gdb) quit The program is running. Quit anyway (and kill it)? (y or n) y hydra /home/kbk/proj/sandbox/python_clean$ Script done on Mon Dec 22 14:42:38 2003 Investigating. -- KBK From jcarlson at uci.edu Mon Dec 22 14:45:19 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 14:47:58 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: <200312221524.hBMFOVj17969@c-24-5-183-134.client.comcast.net> References: <20031221212332.10E9.JCARLSON@uci.edu> <200312221524.hBMFOVj17969@c-24-5-183-134.client.comcast.net> Message-ID: <20031222114456.0A73.JCARLSON@uci.edu> > > Is None so weird? All I am proposing is a symmetric value to None. > > Just like there is a -1 and 1, -Infinity and Infinity, None logically > > should have Some. Maybe not so much -Some == None, but as I propose in > > the PEP, possibly Some == not None. > > None's purposr is not to be less than all numbers (in fact in > the past it wasn't); it's to be "not any object", like nil or NULL in > other languages. > > I don't see a similar broad role for Some. > > And I repeat my question: which other language has that concept? > > --Guido van Rossum (home page: http://www.python.org/~guido/) Certainly if one thinks of None as an object, or a possible pointer to an object, None does fit the role of nil or NULL. The fact is, NULL and nil, as pointers, are 0. None does not equal 0, it is a symbolic representation of the lack of any value (something I remember you saying before, but cannot find a quote), who'se original purpose was to be an equivalent to nil or NULL. As of right now, None has gone beyone its original purpose. As soon as comparisons were able to be done to it and reveal that None was smaller than anything else (even the floating point -infinity), it seemingly has become a more consistant minimum value. Some is just begging to be a more consistant maximum value. It is bigger than sys.maxint, float(1e309), and reveals that the current state is special. What other language has the concept? Python already has the concept. None is currently used as negative infinity in some projects, an actual negative infinity, one that doesn't crap out on integers that cannot be cast into floats. Some would become positive infinity. In terms of non-Python languages, I don't know of any, but that doesn't mean that they don't exist - it just means that I have limited knowledge of programming languages (I only know about 7 or so). - Josiah From mwh at python.net Mon Dec 22 14:48:29 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 14:48:33 2003 Subject: [Python-Dev] Py_DECREF causes spurious gcc warning In-Reply-To: (Tim Peters's message of "Sat, 20 Dec 2003 17:13:34 -0500") References: Message-ID: <2mllp4eir6.fsf@starship.python.net> "Tim Peters" writes: ['>>' is Zack] >> I am not disputing that MSVC does this silly thing you describe, but >> MSVC has a long track record of bugs > > That doesn't match our experience. I don't believe Python code has ever > tickled an MSVC codegen or optimization bug, neither any in gcc-for-x86. Some version of gcc for OpenBSD (on intel) miscompiles (w/optimization) the usual definition of Py_IS_OVERFLOW (or whatever that macro's called). Just nitpicking :-) >> GCC does not generate junk code even at -O0. I'm not sure I understand the context of this remark, but out of context it sure sounds implausible. Cheers, mwh -- The use of COBOL cripples the mind; its teaching should, therefore, be regarded as a criminal offence. -- Edsger W. Dijkstra, SIGPLAN Notices, Volume 17, Number 5 From kbk at shore.net Mon Dec 22 14:48:43 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon Dec 22 14:48:51 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <20031222192653.GA31766@homer.gst.com> (Christopher Blunck's message of "Mon, 22 Dec 2003 14:26:53 -0500") References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> Message-ID: Christopher Blunck writes: > Interesting... When I do OPT=-g, I get a Memory fault (core dumped) > when linking python. Is that when you receive the core dump? Sounds like we're seeing the same thing. -- KBK From guido at python.org Mon Dec 22 14:54:01 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:54:07 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 14:40:12 EST." <16359.18588.450934.263664@magrathea.basistech.com> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> Message-ID: <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> > Indeed. See patch 849278 for some of the work we did to deal with this. This mostly removes the use of the PYTHONXXX environment variables. Can you explain why you don't want these to work? Also note that you can already disable *all* use of the environment (not just in Py_Initialize()) by setting the global Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of the patch for that reason. Py_SetSysPaths() might be a valuable addition, but the game you play with the prefiX name is abject. :) > > 1. Careful preparation (hacking?) of environment variables is needed to > > avoid that Py_Initialize() takes the default actions needed for the > > standard Python interpreter (for example, to avoid the default sys.path). > > I've done this in the patches submitted (by a coworker) above. The > idea is to separate the embedded interpreter *completely* from the > environment. > > > 2. A suggestion from /F on c.l.p was to change pythonw.exe so that > > instead of hiding all output to stderr a console window would be opened > > to show the tracebacks. This requires to construct an object with a > > 'write' method doing all this magic, and it must be done after the call > > to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the > > code of Py_Main(), it seems impossible without rewriting most of the > > code. > > Yes, this is a problem with the embedded interpreter where errors may > be written to stderr without regard to the calling > application. Further, it is possible for the interpreter to call > abort(), which is even worse. What do you suggest the interpreter should do instead of calling abort()? This is only called in response to *fatal* errors -- situations where raising an exception is *not* an acceptable alternative for various reasons (maybe because it's in a piece of code that is part of the implementation of exceptions, or maybe because a corruption is detected in the very interpreter environment that's needed to raise an exception). Anout the only alternative I can think of would be a longjmp() call to a handler you have set up, but that can't be safe when the fatal error is discovered in a different thread. > > Possible improvements I currently can think of are: > > > > - Provide a function like Py_InitializeEx(), which accepts parameters > > specifying an initial sys.path, optimize flag, verbose flag, and so on. > > See the patch. > > > - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), > > or let Py_Main() accept a callback function which will be called just > > after Py_Initialize() has been run. > > Yup. What's your embedding environment? Why do you want to use Py_Main() at all? --Guido van Rossum (home page: http://www.python.org/~guido/) From theller at python.net Mon Dec 22 14:55:38 2003 From: theller at python.net (Thomas Heller) Date: Mon Dec 22 14:55:43 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 11:33:29 -0800") References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> Message-ID: [Thomas Heller] >> I find the current situation when 'embedding' Python rather unsatisfying. > [Guido van Rossum] > I'm interested in improve it. Can you explain what exactly your > 'embedding' requirements are? see below. >> 1. Careful preparation (hacking?) of environment variables is needed to >> avoid that Py_Initialize() takes the default actions needed for the >> standard Python interpreter (for example, to avoid the default sys.path). > > I guess it depends on the embedding needs. Some embedded uses would > be quite happy with the default sys.path. Sure, but others not. >> Possible improvements I currently can think of are: >> >> - Provide a function like Py_InitializeEx(), which accepts parameters >> specifying an initial sys.path, optimize flag, verbose flag, and so on. > > Except for sys.path, you can set most flags directly before calling > Py_Initialize(), so I'm not sure if there's a need to pass these into > Py_Initialize(). I want to specify the Python path before calling Py_Initialize(), and I don't want it to use *any* environment variables which (again) set the flags. So I thought of Py_InitializeEx() which would take parameters specifying these things, and Py_Initialize() could be changed to call Py_InitializeEx(NULL), maybe. >> 2. A suggestion from /F on c.l.p was to change pythonw.exe so that >> instead of hiding all output to stderr a console window would be opened >> to show the tracebacks. This requires to construct an object with a >> 'write' method doing all this magic, and it must be done after the call >> to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the >> code of Py_Main(), it seems impossible without rewriting most of the >> code. > > I'm not sure what this has to do with embedding -- can you clarify? The Python interpreter (dll) embedded into pythonw.exe ;-) > >> - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), >> or let Py_Main() accept a callback function which will be called jsut >> after Py_Initialize() has been run. > > OK, now I'm confused. If you're embedding Python, why would you be > using Py_Main() at all? I'm probably not good at explaining these things. I was thinking of pythonw.exe (WinMain.c). I want to create a Python object (an stderr writer), and assign this to sys.stderr, without changing too much code. Does it make more sense now? Thomas From mwh at python.net Mon Dec 22 14:56:32 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 14:56:36 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: (Terry Reedy's message of "Sun, 21 Dec 2003 21:36:44 -0500") References: <20031220112120.GE1933@lkcl.net> <20031220121942.GF1933@lkcl.net> Message-ID: <2m65g8eidr.fsf@starship.python.net> "Terry Reedy" writes: > "Luke Kenneth Casson Leighton" > wrote in message > news:20031220121942.GF1933@lkcl.net... >> if you "feed" access to __class__ via a single > function, you have >> a bottleneck again which can be replaced. >> >> so, the proposal is simple: create an > __builtin__ function called >> __get_class__ through which access to __class__ > is made. >> > > I wonder if it would not be better to turn > __class__ into a property with a get function that > conditionally gives out the real thing. As much as it makes sense to say it, __class__ is a property -- really a getset descriptor, but that's just because it's in C. It would be a imple matter to add a call to PyEval_GetRestricted in object_get_class (not sure whether this is wise or helpful -- only been skimming this thread -- but it certainly ain't hard). Cheers, mwh -- I don't remember any dirty green trousers. -- Ian Jackson, ucam.chat From mwh at python.net Mon Dec 22 14:58:55 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 14:58:59 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <20031222192653.GA31766@homer.gst.com> (Christopher Blunck's message of "Mon, 22 Dec 2003 14:26:53 -0500") References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> Message-ID: <2m1xqwei9s.fsf@starship.python.net> Christopher Blunck writes: > On Mon, Dec 22, 2003 at 01:57:18PM -0500, Kurt B. Kaiser wrote: >> Guido van Rossum writes: >> >> > If anybody *does* see the bogus behavior and feels like debugging it, >> > I'd appreciate it! >> >> I've been running cvs Python but not previously with debug switched on. >> >> I get a core when using OPT=-g. >> >> (Note that I'm running Patch 850977 "Modify setup.py to detect Tcl/Tk on BSD", >> though I wouldn't think it would cause this, and I doubt he's using it.) >> >> Investigating. Hints appreciated! > > Kurt- > > Interesting... When I do OPT=-g, I get a Memory fault (core dumped) > when linking python. Is that when you receive the core dump? What version of gcc? Anthony and I hacked around on HP's compile farm in the 2.3.2 timeframe and saw gcc problems which we hacked around, but nothing like this. Cheers, mwh -- I've even been known to get Marmite *near* my mouth -- but never actually in it yet. Vegamite is right out. UnicodeError: ASCII unpalatable error: vegamite found, ham expected -- Tim Peters, comp.lang.python From guido at python.org Mon Dec 22 14:59:43 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 14:59:48 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Mon, 22 Dec 2003 14:45:43 EST." References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> Message-ID: <200312221959.hBMJxhY18697@c-24-5-183-134.client.comcast.net> > It doesn't get as far as the banner: > > Script started on Mon Dec 22 14:42:19 2003 > hydra /home/kbk/proj/sandbox/python_clean$ gdb ./python > GNU gdb 4.16.1 > Copyright 1996 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-unknown-openbsd3.3"... > (gdb) run > Starting program: /home/kbk/proj/sandbox/python_clean/./python > > Program received signal SIGSEGV, Segmentation fault. > 0x401900a0 in strchr () > (gdb) quit > The program is running. Quit anyway (and kill it)? (y or n) y > hydra /home/kbk/proj/sandbox/python_clean$ > Script done on Mon Dec 22 14:42:38 2003 > > Investigating. That suggests it's still in Py_Initialize(). What does the gdb command 'bt' say??? I'd also try another experiment: instead of "run" try "run -S". This passes the -S option to Python when it is started, so that it doesn't try to load site.py (which executes rather a lot of Python code). I'd be interested in seeing how much you can do interactively in that case, of if it still crashes in Py_Initialize(). --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Mon Dec 22 14:59:00 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 15:00:56 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? In-Reply-To: References: <20031222000819.10EC.JCARLSON@uci.edu> Message-ID: <20031222114715.0A76.JCARLSON@uci.edu> > "Josiah Carlson" wrote in > message > news:20031222000819.10EC.JCARLSON@uci.edu... > > Is None so weird? All I am proposing is a > symmetric value to None. > > To me that would be All of AllType. Some is in > between. Can you think of any uses for All other > than being max element? > > TJR Not really. However, I was thinking of Some in the context of graph and dynamic programming algorithms. When programming some algorithms in Python, you need to either pick some large constant (that you hope will be big enough), or have a test to see if a value is None. That is ugly and very not Pythonic. I do like All and AllType though, which gives a better name, as suggested by Guido. - Josiah From guido at python.org Mon Dec 22 15:03:40 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 15:03:47 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 20:55:38 +0100." References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> Message-ID: <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> > [Thomas Heller] > >> I find the current situation when 'embedding' Python rather unsatisfying. > > > > [Guido van Rossum] > > I'm interested in improve it. Can you explain what exactly your > > 'embedding' requirements are? > > see below. The only clue I find below is that you're trying to inmprove the error reporting of pythonw.exe, which is certainly laudable, but hardly qualifies as "embedding" IMO. > >> 1. Careful preparation (hacking?) of environment variables is needed to > >> avoid that Py_Initialize() takes the default actions needed for the > >> standard Python interpreter (for example, to avoid the default sys.path). > > > > I guess it depends on the embedding needs. Some embedded uses would > > be quite happy with the default sys.path. > > Sure, but others not. But the only use *you* are interested in is pythonw.exe? > >> Possible improvements I currently can think of are: > >> > >> - Provide a function like Py_InitializeEx(), which accepts parameters > >> specifying an initial sys.path, optimize flag, verbose flag, and so on. > > > > Except for sys.path, you can set most flags directly before calling > > Py_Initialize(), so I'm not sure if there's a need to pass these into > > Py_Initialize(). > > I want to specify the Python path before calling Py_Initialize(), and I > don't want it to use *any* environment variables which (again) set the > flags. So I thought of Py_InitializeEx() which would take parameters > specifying these things, and Py_Initialize() could be changed to call > Py_InitializeEx(NULL), maybe. If you don't want the environment to be used, set the global Py_IgnoreEnvironmentFlag before calling Py_Initialize(). > >> 2. A suggestion from /F on c.l.p was to change pythonw.exe so that > >> instead of hiding all output to stderr a console window would be opened > >> to show the tracebacks. This requires to construct an object with a > >> 'write' method doing all this magic, and it must be done after the call > >> to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the > >> code of Py_Main(), it seems impossible without rewriting most of the > >> code. > > > > I'm not sure what this has to do with embedding -- can you clarify? > > The Python interpreter (dll) embedded into pythonw.exe ;-) > > > > >> - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), > >> or let Py_Main() accept a callback function which will be called jsut > >> after Py_Initialize() has been run. > > > > OK, now I'm confused. If you're embedding Python, why would you be > > using Py_Main() at all? > > I'm probably not good at explaining these things. > > I was thinking of pythonw.exe (WinMain.c). I want to create a Python > object (an stderr writer), and assign this to sys.stderr, without > changing too much code. > > Does it make more sense now? Not quite -- I don't understand why *you* don't want the environment variables to be used, if all you want is a better pythonw.exe. If that's your only experience with embedding Python, you can't really speak for other embedders. --Guido van Rossum (home page: http://www.python.org/~guido/) From pf_moore at yahoo.co.uk Mon Dec 22 15:04:24 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Mon Dec 22 15:04:19 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <20031222000819.10EC.JCARLSON@uci.edu> Message-ID: Josiah Carlson writes: > I ask the rest of you, does Some sound so crazy? To be honest, yes. To me, None is not the low end of a scale, rather it is a unique distinguished value, more related to the concept of "uninitialised" or "not used" than to "smallest". The fact that it compares less than every other value is to me incidental, and arbitrary. I see None as inhabiting the same sort of niche as the SQL NULL value. >From the above, you'll gather that from my viewpoint, there really is no "other extreme" from None, any more than there is from zero. Paul. -- This signature intentionally left blank From tree at basistech.com Mon Dec 22 15:06:42 2003 From: tree at basistech.com (Tom Emerson) Date: Mon Dec 22 15:08:10 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> Message-ID: <16359.20178.766911.979542@magrathea.basistech.com> Guido van Rossum writes: > > Indeed. See patch 849278 for some of the work we did to deal with this. > > This mostly removes the use of the PYTHONXXX environment variables. > Can you explain why you don't want these to work? If the user installs our application and has a separate Python installation on their machine for which they use those environment variables, bad things can happen. I do not want to be in a position where I'm told our app doesn't work because of environment variables we have no control over, and it is the Wrong Answer to tell the user to change their environment. > Also note that you can already disable *all* use of the environment > (not just in Py_Initialize()) by setting the global > Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of > the patch for that reason. I did not know about or notice this flag when I made the changes --- in all the places I looked at the time the environment variables were read without such a check, at least as I remember. It certainly makes more sense to use the built in support. > Py_SetSysPaths() might be a valuable addition, but the game you play > with the prefiX name is abject. :) Sure, whatever. These were made for our own use and later submitted, They can be cleaned up as needed. > What do you suggest the interpreter should do instead of calling > abort()? This is only called in response to *fatal* errors -- > situations where raising an exception is *not* an acceptable > alternative for various reasons (maybe because it's in a piece of code > that is part of the implementation of exceptions, or maybe because a > corruption is detected in the very interpreter environment that's > needed to raise an exception). During initialization if there is a fatal error I can just disable the functionality that uses the interpreter --- having my app abort because Python cannot initialize for whatever reason is a Bad Thing. Similarly with writing messages directly to stderr: I do not necessarily want our users to see these since they don't know what to do with them. > Anout the only alternative I can think of would be a longjmp() call > to a handler you have set up, but that can't be safe when the fatal > error is discovered in a different thread. In our use of Python we build without threads, so the longjmp() would work. In the case of threads all bets are off. > What's your embedding environment? Why do you want to use Py_Main() > at all? I don't use Py_Main() in my embedding (multiple Unix flavors, 32- and 64-bit, and Win32 and Win64)... I wasn't reading closely enough. -tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever" From logistix at cathoderaymission.net Mon Dec 22 15:18:24 2003 From: logistix at cathoderaymission.net (logistix) Date: Mon Dec 22 15:09:52 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Message-ID: <000001c3c8c8$c0c116c0$20bba8c0@XP> > -----Original Message----- > From: > python-dev-bounces+logistix=cathoderaymission.net@python.org > [mailto:python-dev-bounces+logistix=cathoderaymission.net@pyth > on.org] On Behalf Of Kurt B. Kaiser > Sent: Monday, December 22, 2003 1:57 PM > To: python-dev@python.org > Subject: Re: [Python-Dev] OpenBSD anyone? > > > Guido van Rossum writes: > > > If anybody *does* see the bogus behavior and feels like > debugging it, > > I'd appreciate it! > > I've been running cvs Python but not previously with debug > switched on. > > I get a core when using OPT=-g. > > (Note that I'm running Patch 850977 "Modify setup.py to > detect Tcl/Tk on BSD", though I wouldn't think it would cause > this, and I doubt he's using it.) > > Investigating. Hints appreciated! > > -- > KBK > > I saw this a while ago and filed bug 712056. In my case it was problems with unicode and the newly introduced iconv module. At the time, this and a few other bugs caused the developers to yank the iconv module from the release and things compiled properly. Could be completely unrelated but I thought I'd mention it. From pf_moore at yahoo.co.uk Mon Dec 22 15:08:54 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Mon Dec 22 15:10:30 2003 Subject: [Python-Dev] Re: Relative import References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: >> I don't either, although my biggest problem with it is that it isn't >> clear just by looking what the punctuation actually means. I just >> fear this will be a permanent head scratcher. > > This argument by itself has very little value. The question isn't > whether it's self-describing (very few features beyond decimal > integers are); the question is whether it's easy to remember. There > are two subclasses of that: easy to remember what it means when you > next see it, and easy to remember how to spell it when you next need > it. IMO the dot syntax does fine here. The biggest "clarity" issue I have with the dot notation is that there's no obvious keyword to look up when you want to find out what's going on. The best you could do is look up the "import" section of the manual. Heaven help you if you want to use Google. (Has anyone tried to get any meaningful information from Google on Microsoft's two technologies, COM and .NET? You'd almost think they'd done it deliberately... Seriously, the "what do I search for?" factor bugs me every time I hit it. Paul. -- This signature intentionally left blank From guido at python.org Mon Dec 22 15:13:55 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 15:14:01 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 15:06:42 EST." <16359.20178.766911.979542@magrathea.basistech.com> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> <16359.20178.766911.979542@magrathea.basistech.com> Message-ID: <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> > > This mostly removes the use of the PYTHONXXX environment variables. > > Can you explain why you don't want these to work? > > If the user installs our application and has a separate Python > installation on their machine for which they use those environment > variables, bad things can happen. I do not want to be in a position > where I'm told our app doesn't work because of environment variables > we have no control over, and it is the Wrong Answer to tell the user > to change their environment. Fair enough. (Though you could have built your app to work even if Python wrote some random stuff to stderr or optimized its bytecode.) > > Also note that you can already disable *all* use of the environment > > (not just in Py_Initialize()) by setting the global > > Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of > > the patch for that reason. > > I did not know about or notice this flag when I made the changes --- > in all the places I looked at the time the environment variables were > read without such a check, at least as I remember. It certainly makes > more sense to use the built in support. Perhaps it's easy to overlook that it's using Py_GETENV() rather than getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first. > > Py_SetSysPaths() might be a valuable addition, but the game you play > > with the prefiX name is abject. :) > > Sure, whatever. These were made for our own use and later submitted, > They can be cleaned up as needed. Please do. Add some docs too -- undocumented new code isn't very welcome, since we've already got enough of that. :-) > > What do you suggest the interpreter should do instead of calling > > abort()? This is only called in response to *fatal* errors -- > > situations where raising an exception is *not* an acceptable > > alternative for various reasons (maybe because it's in a piece of code > > that is part of the implementation of exceptions, or maybe because a > > corruption is detected in the very interpreter environment that's > > needed to raise an exception). > > During initialization if there is a fatal error I can just disable the > functionality that uses the interpreter --- having my app abort > because Python cannot initialize for whatever reason is a Bad > Thing. Similarly with writing messages directly to stderr: I do not > necessarily want our users to see these since they don't know what to > do with them. > > > Anout the only alternative I can think of would be a longjmp() call > > to a handler you have set up, but that can't be safe when the fatal > > error is discovered in a different thread. > > In our use of Python we build without threads, so the longjmp() would > work. In the case of threads all bets are off. So that's an indirect answer to my question: you'd be happy if instead of calling abort() it would longjmp() to a location you specified? I guess a better (more general) solution would be to have a callback function that's called instead of abort(), and to initialize this with something that calls abort() by default. That's a useful feature; if you upload it to SF with reference to this thread it ought to be accepted. --Guido van Rossum (home page: http://www.python.org/~guido/) From theller at python.net Mon Dec 22 15:15:08 2003 From: theller at python.net (Thomas Heller) Date: Mon Dec 22 15:15:27 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 12:03:40 -0800") References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> Message-ID: >> I want to specify the Python path before calling Py_Initialize(), and I >> don't want it to use *any* environment variables which (again) set the >> flags. So I thought of Py_InitializeEx() which would take parameters >> specifying these things, and Py_Initialize() could be changed to call >> Py_InitializeEx(NULL), maybe. > > If you don't want the environment to be used, set the global > Py_IgnoreEnvironmentFlag before calling Py_Initialize(). I didn't know that, thanks for the hint. [...] >> Does it make more sense now? > > Not quite -- I don't understand why *you* don't want the environment > variables to be used, if all you want is a better pythonw.exe. I did probably mix two different things here. > If that's your only experience with embedding Python, you can't really > speak for other embedders. That may be - my other experiences are py2exe and the embedding done in bdist_wininst. Now knowing Py_IgnoreEnvironmentFlag I would be satisfied with the addition of a PySys_SetPath function ;-). Thomas From guido at python.org Mon Dec 22 15:15:44 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 15:15:50 2003 Subject: [Python-Dev] Re: Relative import In-Reply-To: Your message of "Mon, 22 Dec 2003 20:08:54 GMT." References: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <20031222075219.GA3426@performancedrivers.com> <200312220003.08810.troy@gci.net> <1072104915.18828.47.camel@anthem> <200312221536.hBMFa9S18026@c-24-5-183-134.client.comcast.net> Message-ID: <200312222015.hBMKFi918839@c-24-5-183-134.client.comcast.net> > The biggest "clarity" issue I have with the dot notation is that > there's no obvious keyword to look up when you want to find out what's > going on. The best you could do is look up the "import" section of the > manual. Heaven help you if you want to use Google. (Has anyone tried > to get any meaningful information from Google on Microsoft's two > technologies, COM and .NET? You'd almost think they'd done it > deliberately... > > Seriously, the "what do I search for?" factor bugs me every time I hit > it. There's no deep reason why the language reference couldn't contain non-alphanumeric characters, although this won't help for Google. But in this case, "from" would be the thing to look up, so I'm not sure I see it as such a serious problem. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh at python.net Mon Dec 22 15:24:12 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 15:24:17 2003 Subject: [Python-Dev] Got None. Maybe Some? In-Reply-To: (Tim Peters's message of "Mon, 22 Dec 2003 11:46:01 -0500") References: Message-ID: <2mwu8od2j7.fsf@starship.python.net> "Tim Peters" writes: > None is handy and I like it, but it's arbitrary enough in some cases > that it takes time to get used to its quirks At least we don't pretend it's the empty list as well... Cheers, mwh -- On the other hand, the following areas are subject to boycott in reaction to the rampant impurity of design or execution, as determined after a period of study, in no particular order: ... http://www.naggum.no/profile.html From tim.one at comcast.net Mon Dec 22 15:34:38 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 22 15:34:42 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? In-Reply-To: Message-ID: [Paul Moore] > ... > The fact that [None] compares less than every other value is to me > incidental, and arbitrary. Arbitrary?! I clearly recall when the decision was made that None would compare less than anything else. I was in the same room with Guido, and this is a verbatim transcript of the debate: "As long as we're changing everything here anyway, maybe comparing with None should yield a consistent result." "Like what, maybe smaller than anything else?" "Sure. Or larger." "Ya, one of those. It's not worth it if it just ends up in the middle somewhere." "What does Python do now?" ... [head-scratching] ... "So what did it do before?" ... [more head-scratching] ... "Well, that's no help, then." "The Reference Manual doesn't promise anything about how None will compare, just the 'consistent total ordering' bit". "So that doesn't constrain us." "No, it's neutral." "So maybe it should just be smaller than everything else." "Sure. Or larger." "If it were smaller, then when you sorted a mixed-type list, all the Nones would float to the start." "Good point. If it were larger, they'd float to the end." "I wouldn't like that much." "Me neither." "OK, it's settled." Two lines of code later, the Only Objectively Correct result was achieved. history-is-more-impressive-in-textbooks-ly y'rs - tim From guido at python.org Mon Dec 22 16:04:11 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:04:17 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) Message-ID: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> New-style classes inherit a default __hash__ from object, but this is incorrect if the inheriting object defines __eq__ or __cmp__. This has been reported several times; the roll-up SF entry is 660098, which has the proposed patch attached as newpatch.txt. I've long pondered this, and it seems the best solution is to simply not define __hash__ on the base object class. After all, you don't inherit __eq__ or __cmp__ either; the default comparison behavior is supplied by intrinsic behavior of comparisons, and the default hash() behavior is also supplied by intrinsic behavior of hash(). Does anybody have any reason why I shouldn't check in this change in 2.4? There's no unit test that fails if I do this. It *is* a slight incompatibility though: if someone implemented their __hash__ in terms of object.__hash__, they would get a somewhat puzzling error message after the change, because object.__hash__ will refer to the type.__hash__ function bound to object, and refuse to take an argument: >>> object.__hash__() 135328576 >>> object.__hash__(42) Traceback (most recent call last): File " ", line 1, in ? TypeError: expected 0 arguments, got 1 >>> Since the default hash simply takes id() of the object, it's easy to fix such code once the failure is understood though. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Mon Dec 22 16:23:26 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon Dec 22 16:20:00 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.ne t> Message-ID: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> At 13:04 22.12.2003 -0800, Guido van Rossum wrote: >Since the default hash simply takes id() of the object, it's easy to >fix such code once the failure is understood though. Jython 2.2a0 on java1.4.0_02 (JIT: null) Type "copyright", "credits" or "license" for more information. >>> class C: pass ... >>> id(C) 1 >>> hash(C) 9174506 >>> I see, but then there should be probably be a different way to spell the default hash, because id is not a sensible option for Jython etc From pje at telecommunity.com Mon Dec 22 16:19:10 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 22 16:26:29 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.ne t> Message-ID: <5.1.1.6.0.20031222161608.02e98d50@telecommunity.com> At 01:04 PM 12/22/03 -0800, Guido van Rossum wrote: >New-style classes inherit a default __hash__ from object, but this >is incorrect if the inheriting object defines __eq__ or __cmp__. > >This has been reported several times; the roll-up SF entry is 660098, >which has the proposed patch attached as newpatch.txt. > >I've long pondered this, and it seems the best solution is to simply >not define __hash__ on the base object class. After all, you don't >inherit __eq__ or __cmp__ either; the default comparison behavior is >supplied by intrinsic behavior of comparisons, and the default hash() >behavior is also supplied by intrinsic behavior of hash(). > >Does anybody have any reason why I shouldn't check in this change in >2.4? There's no unit test that fails if I do this. Would this mean that instances of the following class: class Dummy(object): pass would no longer be usable as dictionary keys? I guess the parts I'm not clear on are 1) whether dictionaries call the equivalent of 'hash(key)' or 'key.__hash__()', and 2) whether 'hash(Dummy())' will do something meaningful. From guido at python.org Mon Dec 22 16:29:23 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:29:29 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: Your message of "Mon, 22 Dec 2003 16:19:10 EST." <5.1.1.6.0.20031222161608.02e98d50@telecommunity.com> References: <5.1.1.6.0.20031222161608.02e98d50@telecommunity.com> Message-ID: <200312222129.hBMLTNT19225@c-24-5-183-134.client.comcast.net> > Would this mean that instances of the following class: > > class Dummy(object): > pass > > would no longer be usable as dictionary keys? I guess the parts I'm > not clear on are 1) whether dictionaries call the equivalent of > 'hash(key)' or 'key.__hash__()', and 2) whether 'hash(Dummy())' will > do something meaningful. That class would still be usable. hash(Dummy()) sees that there's no __eq__ or __cmp__ override and then uses the default hash. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 16:32:15 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:32:21 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: Your message of "Mon, 22 Dec 2003 22:23:26 +0100." <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> References: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> Message-ID: <200312222132.hBMLWFe19253@c-24-5-183-134.client.comcast.net> > >Since the default hash simply takes id() of the object, it's easy to > >fix such code once the failure is understood though. > > Jython 2.2a0 on java1.4.0_02 (JIT: null) > Type "copyright", "credits" or "license" for more information. > >>> class C: pass > ... > >>> id(C) > 1 > >>> hash(C) > 9174506 > >>> > > I see, but then there should be probably be a different way to spell > the default hash, because id is not a sensible option for Jython etc Ow, you're right. I bet this is why object.__hash__ was introduced in the first place. We're either back to square one, or we can add a __default_hash__ to object which has the default hash implementation -- this isn't very pretty but at least it works. (I should add that this topic rose to my attention when I had to debug the issue in some code of my own -- I had a class whose __eq__ was overridden but whose __hash__ wasn't, and it was being used as a dict key... It is indeed a pain to debug this. :-( ) --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Mon Dec 22 16:43:40 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon Dec 22 16:40:25 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222132.hBMLWFe19253@c-24-5-183-134.client.comcast.ne t> References: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20031222224015.02f0a640@pop.bluewin.ch> At 13:32 22.12.2003 -0800, Guido van Rossum wrote: > > >Since the default hash simply takes id() of the object, it's easy to > > >fix such code once the failure is understood though. > > > > Jython 2.2a0 on java1.4.0_02 (JIT: null) > > Type "copyright", "credits" or "license" for more information. > > >>> class C: pass > > ... > > >>> id(C) > > 1 > > >>> hash(C) > > 9174506 > > >>> > > > > I see, but then there should be probably be a different way to spell > > the default hash, because id is not a sensible option for Jython etc > >Ow, you're right. I bet this is why object.__hash__ was introduced in >the first place. > >We're either back to square one, or we can add a __default_hash__ to >object which has the default hash implementation -- this isn't very >pretty but at least it works. yes, or hash should grow a keyword argument to get access to the default impl, which would be id for CPython but not for all Python impls in general. The problem is that in general it cannot be expected from Python implementations to implement the default hash in terms of id because the latter can be much more costly beacuse of its uniqueness constraints. regards. From kbk at shore.net Mon Dec 22 16:41:13 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon Dec 22 16:41:18 2003 Subject: [Python-Dev] OpenBSD anyone? References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> <200312221959.hBMJxhY18697@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > That suggests it's still in Py_Initialize(). What does the gdb > command 'bt' say??? > > I'd also try another experiment: instead of "run" try "run -S". This > passes the -S option to Python when it is started, so that it doesn't > try to load site.py (which executes rather a lot of Python code). I'd > be interested in seeing how much you can do interactively in that > case, of if it still crashes in Py_Initialize(). Script started on Mon Dec 22 15:20:25 2003 hydra /home/kbk/proj/sandbox/python_clean$ gdb ./python GNU gdb 4.16.1 Copyright 1996 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-openbsd3.3"... (gdb) run -S Starting program: /home/kbk/proj/sandbox/python_clean/./python -S Program received signal SIGSEGV, Segmentation fault. 0x401900a0 in strchr () (gdb) bt #0 0x401900a0 in strchr () #1 0x1d11d in load_next (mod=0x10905c, altmod=0x10905c, p_name=0xcfbfd640, buf=0xcfbfd230 "__builtin__", p_buflen=0xcfbfd22c) at Python/import.c:2004 #2 0x1cc40 in import_module_ex (name=0x19786 "__builtin__", globals=0x0, locals=0x0, fromlist=0x0) at Python/import.c:1888 #3 0x1ce29 in PyImport_ImportModuleEx (name=0x19786 "__builtin__", globals=0x0, locals=0x0, fromlist=0x0) at Python/import.c:1922 #4 0x1dfe9 in PyImport_Import (module_name=0x115598) at Python/import.c:2333 #5 0x1caec in PyImport_ImportModule (name=0xb895e "__builtin__") at Python/import.c:1853 #6 0xb8b3d in _PyExc_Init () at Python/exceptions.c:1755 #7 0x25ad4 in Py_Initialize () at Python/pythonrun.c:205 #8 0x282f in Py_Main (argc=2, argv=0xcfbfd82c) at Modules/main.c:376 #9 0x17e3 in main (argc=2, argv=0xcfbfd82c) at Modules/python.c:23 (gdb) q The program is running. Quit anyway (and kill it)? (y or n) y hydra /home/kbk/proj/sandbox/python_clean$ Script done on Mon Dec 22 15:21:06 2003 Same result w/o -S ============================================================= Slightly later: (gdb) b import.c:2004 Breakpoint 1 at 0x1d10f: file Python/import.c, line 2004. (gdb) r Starting program: /home/kbk/proj/sandbox/python_clean/./python Breakpoint 1, load_next (mod=0xe98ec, altmod=0xe98ec, p_name=0xcfbfd88c, buf=0xcfbfd47c "", p_buflen=0xcfbfd478) at Python/import.c:2004 2004 char *dot = strchr(name, '.'); (gdb) p *p_name $1 = 0x19786 "__builtin__" (gdb) p name $2 = 0x19786 "__builtin__" (gdb) p strchr(name, '.') Program received signal SIGSEGV, Segmentation fault. OTOH, if I break at 2004 and then step once, I get by the strchr call OK. Also if I stepi through it. If I continue, it segfaults at the next execution of line 2004. Weird. Investigating. -- KBK From guido at python.org Mon Dec 22 16:43:09 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:43:16 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: Your message of "Mon, 22 Dec 2003 22:43:40 +0100." <5.2.1.1.0.20031222224015.02f0a640@pop.bluewin.ch> References: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> <5.2.1.1.0.20031222224015.02f0a640@pop.bluewin.ch> Message-ID: <200312222143.hBMLh9C19299@c-24-5-183-134.client.comcast.net> > > > I see, but then there should be probably be a different way to spell > > > the default hash, because id is not a sensible option for Jython etc > > > >Ow, you're right. I bet this is why object.__hash__ was introduced in > >the first place. > > > >We're either back to square one, or we can add a __default_hash__ to > >object which has the default hash implementation -- this isn't very > >pretty but at least it works. > > yes, or hash should grow a keyword argument to get access to the default > impl, which would be id for CPython but not for all Python impls in > general. The problem is that in general it cannot be expected from Python > implementations to implement the default hash in terms of id because the > latter can be much more costly beacuse of its uniqueness constraints. Hm, the keyword arg would be more work to implement in C, and I don't really like the ad-hoc extension of the __hash__ signature -- it may make subclasses think they have to support the keyword too. (Overriding the __default_hash__ would be possible, but silly, and I recommend not to support this; if this were Java I'd make it a final method. Can you make it final in Jython?) --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Mon Dec 22 16:55:48 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon Dec 22 16:52:22 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222143.hBMLh9C19299@c-24-5-183-134.client.comcast.ne t> References: <5.2.1.1.0.20031222221506.02f2ccf8@pop.bluewin.ch> <5.2.1.1.0.20031222224015.02f0a640@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20031222224820.02f31840@pop.bluewin.ch> At 13:43 22.12.2003 -0800, Guido van Rossum wrote: > > > > I see, but then there should be probably be a different way to spell > > > > the default hash, because id is not a sensible option for Jython etc > > > > > >Ow, you're right. I bet this is why object.__hash__ was introduced in > > >the first place. > > > > > >We're either back to square one, or we can add a __default_hash__ to > > >object which has the default hash implementation -- this isn't very > > >pretty but at least it works. > > > > yes, or hash should grow a keyword argument to get access to the default > > impl, which would be id for CPython but not for all Python impls in > > general. The problem is that in general it cannot be expected from Python > > implementations to implement the default hash in terms of id because the > > latter can be much more costly beacuse of its uniqueness constraints. > >Hm, the keyword arg would be more work to implement in C, and I don't >really like the ad-hoc extension of the __hash__ signature -- it may >make subclasses think they have to support the keyword too. > >(Overriding the __default_hash__ would be possible, but silly, and I >recommend not to support this; if this were Java I'd make it a final >method. Can you make it final in Jython?) internally yes. But I would probably hardcode is usage in hash, so that is just a way to access the default hash impl and nothing more. From guido at python.org Mon Dec 22 16:56:40 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 16:56:52 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Mon, 22 Dec 2003 16:41:13 EST." References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> <200312221959.hBMJxhY18697@c-24-5-183-134.client.comcast.net> Message-ID: <200312222156.hBMLueA19333@c-24-5-183-134.client.comcast.net> (JD: yes, it's real!) > Script started on Mon Dec 22 15:20:25 2003 > hydra /home/kbk/proj/sandbox/python_clean$ gdb ./python > GNU gdb 4.16.1 > Copyright 1996 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-unknown-openbsd3.3"... > (gdb) run -S > Starting program: /home/kbk/proj/sandbox/python_clean/./python -S > > Program received signal SIGSEGV, Segmentation fault. > 0x401900a0 in strchr () > (gdb) bt > #0 0x401900a0 in strchr () > #1 0x1d11d in load_next (mod=0x10905c, altmod=0x10905c, p_name=0xcfbfd640, > buf=0xcfbfd230 "__builtin__", p_buflen=0xcfbfd22c) at Python/import.c:2004 > #2 0x1cc40 in import_module_ex (name=0x19786 "__builtin__", globals=0x0, locals=0x0, > fromlist=0x0) at Python/import.c:1888 > #3 0x1ce29 in PyImport_ImportModuleEx (name=0x19786 "__builtin__", globals=0x0, locals=0x0, > fromlist=0x0) at Python/import.c:1922 > #4 0x1dfe9 in PyImport_Import (module_name=0x115598) at Python/import.c:2333 > #5 0x1caec in PyImport_ImportModule (name=0xb895e "__builtin__") at Python/import.c:1853 > #6 0xb8b3d in _PyExc_Init () at Python/exceptions.c:1755 > #7 0x25ad4 in Py_Initialize () at Python/pythonrun.c:205 > #8 0x282f in Py_Main (argc=2, argv=0xcfbfd82c) at Modules/main.c:376 > #9 0x17e3 in main (argc=2, argv=0xcfbfd82c) at Modules/python.c:23 > (gdb) q > The program is running. Quit anyway (and kill it)? (y or n) y > hydra /home/kbk/proj/sandbox/python_clean$ > Script done on Mon Dec 22 15:21:06 2003 > > > Same result w/o -S > > ============================================================= > Slightly later: > > (gdb) b import.c:2004 > Breakpoint 1 at 0x1d10f: file Python/import.c, line 2004. > (gdb) r > Starting program: /home/kbk/proj/sandbox/python_clean/./python > > Breakpoint 1, load_next (mod=0xe98ec, altmod=0xe98ec, p_name=0xcfbfd88c, buf=0xcfbfd47c "", > p_buflen=0xcfbfd478) at Python/import.c:2004 > 2004 char *dot = strchr(name, '.'); > (gdb) p *p_name > $1 = 0x19786 "__builtin__" > (gdb) p name > $2 = 0x19786 "__builtin__" > (gdb) p strchr(name, '.') > > Program received signal SIGSEGV, Segmentation fault. > > OTOH, if I break at 2004 and then step once, I get by the strchr call > OK. Also if I stepi through it. If I continue, it segfaults at the > next execution of line 2004. Weird. > > Investigating. The most likely cause then is some kind of bug in the platform's strchr(). This could explain why -O3 fixes the issue: I think I've heard of GCC replacing calls to strchr(), strcpy() etc. with inline code, thereby avoiding the buggy library version (and explaining why the buggy code could persist undetected in the library -- most system code is of course compiled fully optimized). As to why stepi doesn't trigger the segfault: possibly it's a timing bug that doesn't occur when run one instruction at a time. This would even make it CPU dependent, which would explain that some folks didn't see this. I don't have the OpenBSD strchr.c source code online here so I'll stop speculating here... --Guido van Rossum (home page: http://www.python.org/~guido/) From blunck at gst.com Mon Dec 22 16:58:55 2003 From: blunck at gst.com (Christopher Blunck) Date: Mon Dec 22 16:58:59 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312221929.hBMJTef18533@c-24-5-183-134.client.comcast.net> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> <200312221929.hBMJTef18533@c-24-5-183-134.client.comcast.net> Message-ID: <20031222215855.GA32420@homer.gst.com> On Mon, Dec 22, 2003 at 11:29:40AM -0800, Guido van Rossum wrote: > > Interesting... When I do OPT=-g, I get a Memory fault (core dumped) > > when linking python. Is that when you receive the core dump? > > >From looking at his output, he gets it just after the linking, when > using the freshly linked python executable to run the setup.py script. > > Are you saying you didn't try the debug build? I received the dump when trying to run setup.py build. -c -- 16:55:00 up 21 days, 6:39, 12 users, load average: 1.30, 0.60, 0.24 From whisper at oz.net Mon Dec 22 15:43:09 2003 From: whisper at oz.net (David LeBlanc) Date: Mon Dec 22 16:59:31 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? In-Reply-To: Message-ID: > [Paul Moore] > > ... > > The fact that [None] compares less than every other value is to me > > incidental, and arbitrary. > > Arbitrary?! I clearly recall when the decision was made that None would > compare less than anything else. I was in the same room with Guido, and > this is a verbatim transcript of the debate: > Two lines of code later, the Only Objectively Correct result was achieved. > > history-is-more-impressive-in-textbooks-ly y'rs - tim > How about "everything" or "all" as the obverse of "none", ordered higher than all other values? "Some" is far too indeterminate. Dave LeBlanc Seattle, WA USA From blunck at gst.com Mon Dec 22 16:59:54 2003 From: blunck at gst.com (Christopher Blunck) Date: Mon Dec 22 17:00:29 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <2m1xqwei9s.fsf@starship.python.net> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <20031222192653.GA31766@homer.gst.com> <2m1xqwei9s.fsf@starship.python.net> Message-ID: <20031222215954.GB32420@homer.gst.com> On Mon, Dec 22, 2003 at 07:58:55PM +0000, Michael Hudson wrote: > What version of gcc? Anthony and I hacked around on HP's compile farm > in the 2.3.2 timeframe and saw gcc problems which we hacked around, > but nothing like this. 2.95.3. But I think I was mistaken in my last email. I get the dump when I run setup.py build, not when linking python. -c -- 16:55:00 up 21 days, 6:39, 12 users, load average: 1.30, 0.60, 0.24 From barry at python.org Mon Dec 22 17:10:07 2003 From: barry at python.org (Barry Warsaw) Date: Mon Dec 22 17:10:20 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> References: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> Message-ID: <1072131005.13676.9.camel@anthem> On Mon, 2003-12-22 at 16:04, Guido van Rossum wrote: > >>> object.__hash__() > 135328576 > >>> object.__hash__(42) > Traceback (most recent call last): > File " ", line 1, in ? > TypeError: expected 0 arguments, got 1 > >>> Would it be better if object.__hash__() raised a NotImplementedError? -Barry From jepler at unpythonic.net Mon Dec 22 17:10:58 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 22 17:11:03 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031222140255.GH12586@unpythonic.net> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> Message-ID: <20031222221058.GD24252@unpythonic.net> On Mon, Dec 22, 2003 at 08:03:17AM -0600, Jeff Epler wrote: > Well, unlike the last two bugs I filed in redhat bugzilla, this one has > already seen some activity. I hope they don't take the line that this > is not a bug :( Unfortunately, this is redhat's position. ------- Additional Comments From roland@redhat.com 2003-12-22 16:37 ------- I think it is clear that the specification refers to the elements of the child process state that survive exec, so that the executed command can perceive them as part of its "environment". You could submit an interpretation request, but I think the committee would concur with my reading. The specification of pthread_atfork refers to calls to fork, not to other parts of the POSIX.1 implementation. If your application calls system, and not fork, those clauses do not apply to it. roland@redhat.com changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |NOTABUG Status|ASSIGNED |CLOSED From guido at python.org Mon Dec 22 17:13:02 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 17:13:15 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: Your message of "Mon, 22 Dec 2003 17:10:07 EST." <1072131005.13676.9.camel@anthem> References: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> <1072131005.13676.9.camel@anthem> Message-ID: <200312222213.hBMMD2m19449@c-24-5-183-134.client.comcast.net> > > >>> object.__hash__() > > 135328576 > > >>> object.__hash__(42) > > Traceback (most recent call last): > > File " ", line 1, in ? > > TypeError: expected 0 arguments, got 1 > > >>> > > Would it be better if object.__hash__() raised a NotImplementedError? It can't -- it's type.__hash__(object). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Dec 22 17:23:11 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 17:23:17 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: Your message of "Mon, 22 Dec 2003 16:10:58 CST." <20031222221058.GD24252@unpythonic.net> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> Message-ID: <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> > Unfortunately, this is redhat's position. > > ------- Additional Comments From roland@redhat.com 2003-12-22 16:37 ------- > I think it is clear that the specification refers to the elements of > the child process state that survive exec, so that the executed > command can perceive them as part of its "environment". You could > submit an interpretation request, but I think the committee would > concur with my reading. The specification of pthread_atfork refers to > calls to fork, not to other parts of the POSIX.1 implementation. If > your application calls system, and not fork, those clauses do not > apply to it. How hard would it be to reimplement our own system() and popen() using only POSIX calls, for POSIX systems? I've always thought of these to be pretty simple combinations of fork() and exec(), with an assumption of a working /bin/sh. Without error checking: int system(char *cmd) { pid = fork(); if (!pid) { /* child */ execl("/bin/sh", "-c", cmd, NULL); } else { /* parent */ int sts; waitpid(pid, *sts, 0); return sts; } --Guido van Rossum (home page: http://www.python.org/~guido/) From jepler at unpythonic.net Mon Dec 22 17:38:20 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 22 17:38:26 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> References: <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> Message-ID: <20031222223811.GA1685@unpythonic.net> On Mon, Dec 22, 2003 at 02:23:11PM -0800, Guido van Rossum wrote: > How hard would it be to reimplement our own system() and popen() using > only POSIX calls, for POSIX systems? I've always thought of these to > be pretty simple combinations of fork() and exec(), with an assumption > of a working /bin/sh. Without error checking: Yeah -- this is something we could do if necessary. the opengroup standard lists a few more things to "get right" in terms of signal handling for system(), but we can do all those in C or in Python.. Jeff From mwh at python.net Mon Dec 22 17:39:53 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 17:39:55 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 12:13:55 -0800") References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> <16359.20178.766911.979542@magrathea.basistech.com> <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> Message-ID: <2msmjccw92.fsf@starship.python.net> Guido van Rossum writes: >> > (not just in Py_Initialize()) by setting the global >> > Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of >> > the patch for that reason. >> >> I did not know about or notice this flag when I made the changes --- >> in all the places I looked at the time the environment variables were >> read without such a check, at least as I remember. It certainly makes >> more sense to use the built in support. > > Perhaps it's easy to overlook that it's using Py_GETENV() rather than > getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first. Py_IgnoreEnvironmentFlag and the -E command line option are both /fairly/ new... 2.2 era? Cheers, mwh -- Worryingly, DEFUN appears to be a function that removes all the fun from something: after using it all your code is converted to C++. -- Tim Bradshaw, comp.lang.lisp From mwh at python.net Mon Dec 22 17:41:37 2003 From: mwh at python.net (Michael Hudson) Date: Mon Dec 22 17:41:40 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Mon, 22 Dec 2003 14:23:11 -0800") References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> Message-ID: <2moeu0cw66.fsf@starship.python.net> Guido van Rossum writes: >> Unfortunately, this is redhat's position. >> >> ------- Additional Comments From roland@redhat.com 2003-12-22 16:37 ------- >> I think it is clear that the specification refers to the elements of >> the child process state that survive exec, so that the executed >> command can perceive them as part of its "environment". You could >> submit an interpretation request, but I think the committee would >> concur with my reading. The specification of pthread_atfork refers to >> calls to fork, not to other parts of the POSIX.1 implementation. If >> your application calls system, and not fork, those clauses do not >> apply to it. > > How hard would it be to reimplement our own system() and popen() using > only POSIX calls, for POSIX systems? I've always thought of these to > be pretty simple combinations of fork() and exec(), with an assumption > of a working /bin/sh. Without error checking: I think it's a bit harder than what you post, but there's code in APUE for it... Cheers, mwh -- I'd certainly be shocked to discover a consensus. ;-) -- Aahz, comp.lang.python From jcarlson at uci.edu Mon Dec 22 17:41:29 2003 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Dec 22 17:44:59 2003 Subject: [Python-Dev] Re: Got None. Maybe Some? In-Reply-To: References: Message-ID: <20031222144122.0A7F.JCARLSON@uci.edu> > > [Paul Moore] > > > ... > > > The fact that [None] compares less than every other value is to me > > > incidental, and arbitrary. > > > > Arbitrary?! I clearly recall when the decision was made that None would > > compare less than anything else. I was in the same room with Guido, and > > this is a verbatim transcript of the debate: > > > Two lines of code later, the Only Objectively Correct result was achieved. > > > > history-is-more-impressive-in-textbooks-ly y'rs - tim > > > How about "everything" or "all" as the obverse of "none", ordered higher > than all other values? "Some" is far too indeterminate. > > Dave LeBlanc > Seattle, WA USA All seems like a reasonable spelling for an object with the given behavior. I started with the name Some because the name made me smile. - Josiah From guido at python.org Mon Dec 22 17:45:04 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 22 17:45:13 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: Your message of "Mon, 22 Dec 2003 22:39:53 GMT." <2msmjccw92.fsf@starship.python.net> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> <16359.20178.766911.979542@magrathea.basistech.com> <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> <2msmjccw92.fsf@starship.python.net> Message-ID: <200312222245.hBMMj4W19537@c-24-5-183-134.client.comcast.net> > > Perhaps it's easy to overlook that it's using Py_GETENV() rather than > > getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first. > > Py_IgnoreEnvironmentFlag and the -E command line option are both > /fairly/ new... 2.2 era? Yeah, but the patch he suggested was using Py_GETENV(), so it's not like it didn't exist when they did the patch; they must have missed its significance. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From tree at basistech.com Mon Dec 22 18:49:37 2003 From: tree at basistech.com (Tom Emerson) Date: Mon Dec 22 18:51:32 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312222245.hBMMj4W19537@c-24-5-183-134.client.comcast.net> References: <1xqwodnr.fsf@python.net> <16359.18588.450934.263664@magrathea.basistech.com> <200312221954.hBMJs1M18648@c-24-5-183-134.client.comcast.net> <16359.20178.766911.979542@magrathea.basistech.com> <200312222013.hBMKDt118815@c-24-5-183-134.client.comcast.net> <2msmjccw92.fsf@starship.python.net> <200312222245.hBMMj4W19537@c-24-5-183-134.client.comcast.net> Message-ID: <16359.33553.818193.723995@magrathea.basistech.com> Guido van Rossum writes: > > > Perhaps it's easy to overlook that it's using Py_GETENV() rather than > > > getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first. > > > > Py_IgnoreEnvironmentFlag and the -E command line option are both > > /fairly/ new... 2.2 era? > > Yeah, but the patch he suggested was using Py_GETENV(), so it's not > like it didn't exist when they did the patch; they must have missed > its significance. :-) Yes, *they* did miss the significance of the use of the macro. -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever" From martin at v.loewis.de Mon Dec 22 15:01:28 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 22 20:46:49 2003 Subject: [Python-Dev] Re: python/dist/src/Objectsunicodeobject.c, 2.204, 2.205 In-Reply-To: <2mptegej2x.fsf@starship.python.net> References: <2mptegej2x.fsf@starship.python.net> Message-ID: <3FE74D98.1020307@v.loewis.de> Michael Hudson wrote: > As Martin regularly points out, we have rather too much of this kind > of thing. Given that we demand ANSI C, we could surely lose > HAVE_LIMITS_H, and probably much other cruft. Indeed, and I would encourage contributions in that direction. Make sure you put, in the commit message, an elaboration why it is safe to delete the code in question. Regards, Martin From martin at v.loewis.de Mon Dec 22 15:39:20 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 22 21:19:40 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> Message-ID: <3FE75678.6090508@v.loewis.de> Guido van Rossum wrote: > Not quite -- I don't understand why *you* don't want the environment > variables to be used, if all you want is a better pythonw.exe. I haven't embedded Python for a while, but I can sympathize with the desire to ignore environment variables. One approach to embedding is to provide a complete (stripped) copy of Python, inside the target directory of the application. You might even directly link with a static python23.lib, and freeze all modules, instead of linking with python23.dll, and providing a copy of all .py files. In such an environment, you don't want a regular Python installation to interfere with your embedded environment. For example, you may have incorporated the library Foo, but the target system has another (older) copy of Foo in PYTHONPATH. If your embedded application now picks up the older copy of Foo, it fails, and all your testing goes to hell (*) since arbitrary pieces of the library might get arbitrarily replaced with something else. So to allow reproducable results, you need a guarantee that nothing on the target system is incorporated in your application. Regards, Martin (*) DLL hell, to be precise. From pinard at iro.umontreal.ca Mon Dec 22 22:20:05 2003 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Mon Dec 22 22:22:34 2003 Subject: [Python-Dev] Improve Python for embedding In-Reply-To: <3FE75678.6090508@v.loewis.de> References: <1xqwodnr.fsf@python.net> <200312221933.hBMJXTI18559@c-24-5-183-134.client.comcast.net> <200312222003.hBMK3eH18732@c-24-5-183-134.client.comcast.net> <3FE75678.6090508@v.loewis.de> Message-ID: <20031223032005.GA16709@titan.progiciels-bpi.ca> [Martin von L?wis] > So to allow reproducable results, you need a guarantee that nothing > on the target system is incorporated in your application. I did not closely follow this thread, and apologise if I repeat arguments which have already been said. While I quite understand the argument and needs which Martin expresses, there are situations where one is happy to tune the behaviour of the embedded Python through the usual environment variables. It might depend if the embedding application intends to either expose or hide the existence of Python. If the application offers Python as an extension language for the system implemented by the application, Python is then exposed, and the user should be able to use environment variable to his/her advantage. If the application uses Python internally, all to itself, then the application likely wants to hide even the fact that it uses Python, exerts full control, or at least decide exactly how much or little the user is allowed to interfere, as Martin explained. If it is given that only one of these avenues is favoured, I would for one prefer that the "exposed" direction be the easier to implement, which may be the statu quo over the current situation. The "hiding" direction might be evaluated so it requires a bit more work for the implementor, about deleting or forcing the values of a few sensitive environment variables. Especially given that a Python hiding application is likely to control other environment variables as well (like maybe those related to loader path search, or localisation). -- Fran?ois Pinard http://www.iro.umontreal.ca/~pinard From lists at hlabs.spb.ru Tue Dec 23 06:05:37 2003 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Tue Dec 23 03:05:34 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <3FE6CC59.10100@iinet.net.au> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <3FE6D94A.2060904@hlabs.spb.ru> <3FE6CC59.10100@iinet.net.au> Message-ID: <3FE82181.50309@hlabs.spb.ru> Nick Coghlan wrote: > Dmitry Vasiliev wrote: > >> The main idea is to treating package as a program and run package >> initialization code from command line. The advantage is zipping all >> program modules in one zip archive and running the program from >> command line without unzipping it, like Java's jar's. But this idea >> need more thoughts however... > > python runzip.py mypkg.zip > > where runzip.py is a pure Python script that works as you suggest. > > I'm fairly sure zipimport would allow you to write the program you > describe in pure Python. I've obviously never written such a program, > though. Code is fairly simple. import sys sys.path.insert(0, "module.zip") import module > It's certainly an interesting packaging idea for small Python programs > with several files, but for which a full-blown installer would be overkill. -- Dmitry Vasiliev (dima at hlabs.spb.ru) From lists at hlabs.spb.ru Tue Dec 23 06:09:27 2003 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Tue Dec 23 03:09:19 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <200312221807.14352.aleaxit@yahoo.com> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <3FE6D94A.2060904@hlabs.spb.ru> <200312221807.14352.aleaxit@yahoo.com> Message-ID: <3FE82267.3030607@hlabs.spb.ru> Alex Martelli wrote: >>The main idea is to treating package as a program and run package >>initialization code from command line. The advantage is zipping all >>program modules in one zip archive and running the program from command >>line without unzipping it, like Java's jar's. But this idea need more >>thoughts however... > > Couldn't you use: > python -c "import package_name" > for this purpose even today? python -c "import sys; sys.path.insert(0, 'module.zip'); import module" Seems ugly... :) -- Dmitry Vasiliev (dima at hlabs.spb.ru) From aleaxit at yahoo.com Tue Dec 23 04:05:41 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Tue Dec 23 04:05:51 2003 Subject: [Python-Dev] Importing packages from command line In-Reply-To: <3FE82267.3030607@hlabs.spb.ru> References: <1071839460.3fe2f8e4dbbad@mcherm.com> <200312221807.14352.aleaxit@yahoo.com> <3FE82267.3030607@hlabs.spb.ru> Message-ID: <200312231005.41130.aleaxit@yahoo.com> On Tuesday 23 December 2003 12:09 pm, Dmitry Vasiliev wrote: > Alex Martelli wrote: > >>The main idea is to treating package as a program and run package > >>initialization code from command line. The advantage is zipping all > >>program modules in one zip archive and running the program from command > >>line without unzipping it, like Java's jar's. But this idea need more > >>thoughts however... > > > > Couldn't you use: > > python -c "import package_name" > > for this purpose even today? > > python -c "import sys; sys.path.insert(0, 'module.zip'); import module" > > Seems ugly... :) Well, if the zipfile isn't on your sys.path already, then you'd have to insert it explicitly anyway -- surely, even if a switch "-p" to mean import existed, you wouldn't want it to force python to snoop into EVERY zipfile around?! PYTHONPATH=module.zip python -c "import module" is one way you might express "insert into path and import" using a decent shell (cygwin's bash on Windows, for example). The proposed: PYTHONPATH=module.zip python -p module doesn't appear to offer a _major_ enhancement, just a very minor one, in my personal opinion. One potential advantage of -p might be to let it be present _together_ with one -c (even better might be to allow multiple -c, actually). If you want to run a "while foo.bep(): foo.zlup()" you can do it today only with a shell that allows easy input of one multiline argument. But again it seems a rather marginal issue to me, personally. Alex From ncoghlan at iinet.net.au Tue Dec 23 05:03:55 2003 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Tue Dec 23 05:04:00 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <200312222213.hBMMD2m19449@c-24-5-183-134.client.comcast.net> References: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> <1072131005.13676.9.camel@anthem> <200312222213.hBMMD2m19449@c-24-5-183-134.client.comcast.net> Message-ID: <3FE8130B.1080601@iinet.net.au> [Barry] >>Would it be better if object.__hash__() raised a NotImplementedError? [Guido] > It can't -- it's type.__hash__(object). If I understand Barry's suggestion correctly, he means to keep object.__hash__, but have it raise a specific, meaningful error instead of making a potentially incorrect assumption as it does now. E.g. (not real code, since this would behave differently in Py 2.3.3) >>> class Works(object): pass >>> class Breaks(object): ... def __cmp__(): return 0 # All instances are created equal! >>> work = Works() >>> break = Breaks() >>> hash(work) 45632543 >>> hash(break) Traceback (most recent call last): NotImplementedError: Must explicitly define __hash__ for non-default comparisons hash(work) is fine, since there is no __cmp__ or __eq__ override in Works, and hence object.__hash__ never gets called. hash(break) raises the exception because of the existence of the (rather useless) __cmp__ function in the Breaks class. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From martin at v.loewis.de Tue Dec 23 04:33:53 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 23 05:13:45 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312222156.hBMLueA19333@c-24-5-183-134.client.comcast.net> References: <200312221629.hBMGTV918147@c-24-5-183-134.client.comcast.net> <20031222170257.GA31352@homer.gst.com> <200312221737.hBMHbTh18276@c-24-5-183-134.client.comcast.net> <200312221907.hBMJ7hg18485@c-24-5-183-134.client.comcast.net> <200312221959.hBMJxhY18697@c-24-5-183-134.client.comcast.net> <200312222156.hBMLueA19333@c-24-5-183-134.client.comcast.net> Message-ID: <3FE80C01.9080203@v.loewis.de> Guido van Rossum wrote: > I don't have the OpenBSD strchr.c source code online here > so I'll stop speculating here... It's this: http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/index.c I can't see anything wrong in it, and it hasn't significantly changed in ages, either. I'd be curious what p_name points to when the process crashes. (gdb) p *p_name Regards, Martin From martin at v.loewis.de Tue Dec 23 03:51:01 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 23 05:48:17 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <20031222221058.GD24252@unpythonic.net> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> Message-ID: <3FE801F5.9010304@v.loewis.de> Jeff Epler wrote: > You could > submit an interpretation request, but I think the committee would > concur with my reading. Hmm. This is what I now did: I submitted an interpretation request to PASC: Edition of Specification (Year): 2003 Defect code : 3. Clarification required It is unclear whether calling system(3) invokes atfork handlers in a conforming implementation. system(3) specifies "The environment of the executed command shall be as if a child process were created using fork(), and the child process invoked the sh utility using execl() as follows:" In particular, usage of the word "environment" is confusing here. It may refer just to environment variables, however, the "Application usage" sections indicates that also signal handlers should be arranged as if the process was created through fork() and execl(). This still makes not clear whether handlers installed through pthread_atfork() are invoked. Action: The description of system() should change to "system() behaves as if a new process was created using fork(), and the child process invoked the sh utility using execl() ..." In addition, the application usage section should make it clear that atfork handlers are invoked. Regards, Martin From martin at v.loewis.de Tue Dec 23 04:01:58 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 23 08:43:12 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> Message-ID: <3FE80486.9060807@v.loewis.de> Guido van Rossum wrote: > How hard would it be to reimplement our own system() and popen() using > only POSIX calls, for POSIX systems? I've always thought of these to > be pretty simple combinations of fork() and exec(), with an assumption > of a working /bin/sh. I would be concerned that we bypass magic that the system vendor put into system(), which is essential for proper operation. For example, on Linux, system() blocks SIGINT in the parent process while the child is running. Also, the shell executable that system() uses may not be /bin/sh. OTOH, using the same underlying implementation on all systems makes Python behave more predictable. In the specific case, we would not even need pthread_atfork anymore, as we now could invoke PyOS_AfterFork in the child ourselves. Regards, Martin From jepler at unpythonic.net Tue Dec 23 09:05:48 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Tue Dec 23 09:06:02 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: <3FE80486.9060807@v.loewis.de> References: <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> <3FE80486.9060807@v.loewis.de> Message-ID: <20031223140547.GA7907@unpythonic.net> On Tue, Dec 23, 2003 at 10:01:58AM +0100, Martin v. Loewis wrote: > I would be concerned that we bypass magic that the system vendor put > into system(), which is essential for proper operation. For example, > on Linux, system() blocks SIGINT in the parent process while the > child is running. Also, the shell executable that system() uses > may not be /bin/sh. This behavior (blocking SIGINT and SIGQUIT in the parent) is part of the specification of system(): The system() function shall ignore the SIGINT and SIGQUIT signals, and shall block the SIGCHLD signal, while waiting for the command to terminate. If this might cause the application to miss a signal that would have killed it, then the application should examine the return value from system() and take whatever action is appropriate to the application if the command terminated due to receipt of a signal. Jeff From tim.one at comcast.net Tue Dec 23 09:20:49 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Dec 23 09:20:53 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <3FE80C01.9080203@v.loewis.de> Message-ID: [Guido] >> I don't have the OpenBSD strchr.c source code online here >> so I'll stop speculating here... [Martin v. Loewis] > It's this: > > http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/index.c > > I can't see anything wrong in it, and it hasn't significantly > changed in ages, either. Looks fine to me too -- and it's a very simple function. Google didn't turn up any suggestion of strchr problems under OpenBSD either. Hate to say it, but the pointer passed *to* strchr must be insane, and that makes it more likely a Python, or platform compiler, bug. > I'd be curious what p_name points to when the process crashes. > > (gdb) p *p_name From kbk at shore.net Tue Dec 23 10:10:54 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue Dec 23 10:11:00 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: (Tim Peters's message of "Tue, 23 Dec 2003 09:20:49 -0500") References: Message-ID: "Tim Peters" writes: > Hate to say it, but the pointer passed *to* strchr must be insane, > and that makes it more likely a Python, or platform compiler, bug. There are two calls to load_next() in import_module_ex(). The segfault is occuring during the second call. The code is somewhat pathological in that the callee, load_next(), is modifying the caller's /parameters/ by changing the contents of name. For some reason, the compiler emits code which makes a copy of import_module_ex()'s parameters in the stack frame. When load_next() is called, the reference &name is the location in the parameter area of the frame, but when name is tested in the while loop, the copy in the local area of the frame is used. Since this has not been modified by load_next(), the fact that name has been set to 0x00 is missed. load_next() gets called erroneously and passes a null pointer to strchr. I tried a volatile declaration, but no joy. Adding a proper local, mod_name, resolved the problem. -- KBK Index: Python/import.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/import.c,v retrieving revision 2.225 diff -c -r2.225 import.c *** Python/import.c 20 Nov 2003 01:44:58 -0000 2.225 --- Python/import.c 23 Dec 2003 14:56:40 -0000 *************** *** 1871,1876 **** --- 1871,1877 ---- PyObject *fromlist) { char buf[MAXPATHLEN+1]; + char *mod_name; int buflen = 0; PyObject *parent, *head, *next, *tail; *************** *** 1878,1891 **** if (parent == NULL) return NULL; ! head = load_next(parent, Py_None, &name, buf, &buflen); if (head == NULL) return NULL; tail = head; Py_INCREF(tail); ! while (name) { ! next = load_next(tail, tail, &name, buf, &buflen); Py_DECREF(tail); if (next == NULL) { Py_DECREF(head); --- 1879,1893 ---- if (parent == NULL) return NULL; ! mod_name = name; ! head = load_next(parent, Py_None, &mod_name, buf, &buflen); if (head == NULL) return NULL; tail = head; Py_INCREF(tail); ! while (mod_name) { ! next = load_next(tail, tail, &mod_name, buf, &buflen); Py_DECREF(tail); if (next == NULL) { Py_DECREF(head); From charleshixsn at earthlink.net Tue Dec 23 10:54:46 2003 From: charleshixsn at earthlink.net (Charles Hixson) Date: Tue Dec 23 11:00:33 2003 Subject: [Python-Dev] Relative import In-Reply-To: <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> Message-ID: <3FE86546.50003@earthlink.net> Guido van Rossum wrote: >> I like syntax that reads most important left-to-right, so what about >> from MODULE import NAMES as RENAME searching HOW >> or >> import NAMES as RENAME from MODULE searching HOW >> >> searching could be 'absolute' 'relative' or any of the other >> suggested words. If you wanted to get fancy it could be a list, if >> it isn't a list people who truly care could cascade try/except on >> ImportError. >> >> -jackdied >> >> ps, I like the second version better but it is less like the current >> version, >> which is a drawback. >> > > > This got several encouraging responses, so I have to consider it. > > Unfortunately it looks like 'searching' would have to become a > reserved word; if we use the same trick we used to avoid making 'as' a > reserved word, that would be too confusing to decode in the code > generator. The HOW could be just an identifier then. > > I'm not sure how well it'll read in actual examples. Let's try. > Adapting an example from Opal that was used here before... > > from COGS import generate searching relative > > To me, that looks a lot like someone left the commas out of > > from COGS import generate, searching, relative > > In general I'm not a big fan of "wordy" clauses like this -- they work > better in languages like SQL or COBOL because those are case-sensitive > so you can use a difference in capitalization there; there the former > would be > > FROM COGS IMPORT generate SEARCHING RELATIVE > > (using imaginary syntax) while the latter would be > > FROM COGS IMPORT generate, searching, relative > > while in Python the long sequence of lowercase words becomes a blur. > > So if the two alternatives are simply 'searching absolute' and > 'searching relative', with the default being 'searching absolute' in > Python 3.0 (but the current ambiguous search in Python 2.x) I'd still > prefer making the distinction with "from ...MODULE" vs. "from MODULE" > rather than with a searching clause. And I still prefer the > single-dot syntax over three dots, because it can be used to spell > importing from the parent or grandparent package explicitly. > > A separate suggestion is to switch from "from X import Y" to "import Y > from X". This seems a gratuitous change even if it reads or edits > marginally better. Now's not the time to tinker with marginal stuff > -- that should've been done 10 years ago. > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/charleshixsn%40earthlink.net > > > > Not necessarily. import could be a special function that doesn't need parentheses around it's arguments or comma separators. That could be syntax sugar for: import (names, namedAs, fromModule, searchmethod) or import (names =[theNames], namedAd=[localNames], fromModule=fromModule, searchMethod=upFromCurrent) OTOH, to me it looks more like Smalltalk than like Python. An explicit import function without special sugar looks more Pythonic. (But that would break backward compatibility.) Perhaps the current method could be deprecated, and an import function be the replacement? From guido at python.org Tue Dec 23 11:11:40 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 23 11:11:49 2003 Subject: [Python-Dev] Python threads end up blocking signals in subprocesses In-Reply-To: Your message of "Tue, 23 Dec 2003 10:01:58 +0100." <3FE80486.9060807@v.loewis.de> References: <20031219012347.GB4692@unpythonic.net> <2mwu8rebcp.fsf@starship.python.net> <20031221230655.GA4417@unpythonic.net> <3FE64412.3010500@v.loewis.de> <20031222020728.GA15895@unpythonic.net> <3FE65AF6.6010403@v.loewis.de> <20031222140255.GH12586@unpythonic.net> <20031222221058.GD24252@unpythonic.net> <200312222223.hBMMNBn19477@c-24-5-183-134.client.comcast.net> <3FE80486.9060807@v.loewis.de> Message-ID: <200312231611.hBNGBek20630@c-24-5-183-134.client.comcast.net> > > How hard would it be to reimplement our own system() and popen() using > > only POSIX calls, for POSIX systems? I've always thought of these to > > be pretty simple combinations of fork() and exec(), with an assumption > > of a working /bin/sh. > > I would be concerned that we bypass magic that the system vendor put > into system(), which is essential for proper operation. For example, > on Linux, system() blocks SIGINT in the parent process while the > child is running. Also, the shell executable that system() uses > may not be /bin/sh. In practice, I think we can do as well as vendors -- there really isn't that much to it. Systems where /bin/sh doesn't exist will have other problems... And we get to do it right when called from a thread. > OTOH, using the same underlying implementation on all systems makes > Python behave more predictable. > > In the specific case, we would not even need pthread_atfork anymore, > as we now could invoke PyOS_AfterFork in the child ourselves. Right! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Dec 23 11:20:08 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 23 11:20:14 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Tue, 23 Dec 2003 10:10:54 EST." References: Message-ID: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> > There are two calls to load_next() in import_module_ex(). The > segfault is occuring during the second call. > > The code is somewhat pathological in that the callee, load_next(), is > modifying the caller's /parameters/ by changing the contents of name. > > For some reason, the compiler emits code which makes a copy of > import_module_ex()'s parameters in the stack frame. When load_next() > is called, the reference &name is the location in the > parameter area of the frame, but when name is tested in the while > loop, the copy in the local area of the frame is used. Since this has > not been modified by load_next(), the fact that name has been set to > 0x00 is missed. load_next() gets called erroneously and passes a null > pointer to strchr. > > I tried a volatile declaration, but no joy. Adding a proper local, > mod_name, resolved the problem. Wow. Thanks for the analysis. But this is clearly a compiler bug. Where do we report that? And why would it be unique to OpenBSD? In the mean time, Kurt, please check in your fix -- it can't hurt, and we might as well avoid the pain for the next person who wants to build a debugging Python. The fix could use a comment referring to a compiler bug, to keep the next maintainer from unfixing it. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Tue Dec 23 11:47:02 2003 From: barry at python.org (Barry Warsaw) Date: Tue Dec 23 11:47:09 2003 Subject: [Python-Dev] getting rid of default object.__hash__ (SF 660098) In-Reply-To: <3FE8130B.1080601@iinet.net.au> References: <200312222104.hBML4Bc18950@c-24-5-183-134.client.comcast.net> <1072131005.13676.9.camel@anthem> <200312222213.hBMMD2m19449@c-24-5-183-134.client.comcast.net> <3FE8130B.1080601@iinet.net.au> Message-ID: <1072198021.31242.8.camel@geddy> On Tue, 2003-12-23 at 05:03, Nick Coghlan wrote: > If I understand Barry's suggestion correctly, he means to keep > object.__hash__, but have it raise a specific, meaningful error instead > of making a potentially incorrect assumption as it does now. Right! Thanks Nick. -Barry From klm at zope.com Tue Dec 23 12:03:00 2003 From: klm at zope.com (Ken Manheimer) Date: Tue Dec 23 12:11:50 2003 Subject: [Python-Dev] Relative import In-Reply-To: Message-ID: On Mon, 22 Dec 2003, Devin wrote: > The first part stems from Guido's idea about using a '.' at the beginning > of a relative import. It seemed a bit weird at first, but I eventually > became very comfortable with the idea after thinking about it in respect > to absolute imports. I'll explain what I mean. > > If you currently want to import a module 'a.b.c.d', you write: > > import a.b.c.d > > Now, if you're in the module 'a.b', you could still write: > > import a.b.c.d > > ... but the 'a.b' part is redundant because you're already in the module > 'a.b'. If you take away 'a.b', then you end up with: > > import .c.d I also look at it this way, and find it both intuitive and mnemonic. My model is slighly different than yours, though, in a way that simplifies the expression for going up the relative chain. I'll explain... > This flowed well in my head, and looks natural. However, the suggestion > about '.' or '..' referring to the parent looks _ugly and unnatural_. > > Let's say '.' was used to spell "parent module", and let's assume that > the module 'a.b.c.d' wants to import 'a.b.e'. The absolute import would > be spelled: > > import a.b.e > > ... and the relative import would be spelled: > > import .....e # .[parent].[parent].e > > Yuck! (no offense :) Truly, yuck. But in my model, the leading '.' dot, itself, stands for the containing package, and '..' stands for the containing package's package, and so forth: import ..e # [parent-of-parent].e I don't think it's yucky at all, this way. (I also don't think it's un-pythonic - python uses punctuation all over to denote type literals ('{'/'}'), cues which delineate statement suites (':'), and so forth.) In fact, i think it's a lot more mnemonic and encompassing than the alternatives that use words (though i haven't looked at all the alternatives very carefully yet). Even using the leading '.' syntax, it would be good to have a specific identifier for the containing package, realized in each module and respected as a special identifier by the package machinery. I like '__pkg__': import __pkg__.sibling # == 'import .sibling' and import __pkg__.__pkg__.uncle # == 'import ..uncle' import __pkg__.__pkg__.uncle.cousin # == 'import ..uncle.cousin' I think the leading '.' dot forms are a lot nicer, but the presence of a module named '__pkg__' would be useful for programmatically mucking navigating the package structure. Ken Manheimer klm@zope.com From mwh at python.net Tue Dec 23 12:24:27 2003 From: mwh at python.net (Michael Hudson) Date: Tue Dec 23 12:24:31 2003 Subject: [Python-Dev] Relative import In-Reply-To: <3FE86546.50003@earthlink.net> (Charles Hixson's message of "Tue, 23 Dec 2003 07:54:46 -0800") References: <200312151954.hBFJskd18127@c-24-5-183-134.client.comcast.net> <20031216141746.GA3145@burma.localdomain> <200312161826.hBGIQxg20322@c-24-5-183-134.client.comcast.net> <1071714822.27808.88.camel@anthem> <20031218030409.GG473@frobozz> <1071716900.27808.114.camel@anthem> <20031218031645.GH473@frobozz> <20031219213031.GD1246@wopr> <200312220448.hBM4m1K17301@c-24-5-183-134.client.comcast.net> <3FE86546.50003@earthlink.net> Message-ID: <2mad5jcur8.fsf@starship.python.net> Charles Hixson writes: > Not necessarily. import could be a special function that doesn't need > parentheses around it's arguments or comma separators. That could be > syntax sugar for: > import (names, namedAs, fromModule, searchmethod) or > import (names =[theNames], namedAd=[localNames], > fromModule=fromModule, searchMethod=upFromCurrent) > > OTOH, to me it looks more like Smalltalk than like Python. An > explicit import function without special sugar looks more Pythonic. > (But that would break backward compatibility.) > Perhaps the current method could be deprecated, and an import function > be the replacement? I really don't think you could make that work (unless you want to write math = import("math") everywhere, which you can almost do today...) Cheers, mwh -- Ya, ya, ya, except ... if I were built out of KSR chips, I'd be running at 25 or 50 MHz, and would be wrong about ALMOST EVERYTHING almost ALL THE TIME just due to being a computer! -- Tim Peters, 30 Apr 97 From devin at whitebread.org Tue Dec 23 13:45:18 2003 From: devin at whitebread.org (Devin) Date: Tue Dec 23 13:50:12 2003 Subject: [Python-Dev] Relative import In-Reply-To: Message-ID: On Tue, 23 Dec 2003, Ken Manheimer wrote: > > This flowed well in my head, and looks natural. However, the suggestion > > about '.' or '..' referring to the parent looks _ugly and unnatural_. > > > > Let's say '.' was used to spell "parent module", and let's assume that > > the module 'a.b.c.d' wants to import 'a.b.e'. The absolute import would > > be spelled: > > > > import a.b.e > > > > ... and the relative import would be spelled: > > > > import .....e # .[parent].[parent].e > > > > Yuck! (no offense :) > > Truly, yuck. But in my model, the leading '.' dot, itself, stands for > the containing package, and '..' stands for the containing package's > package, and so forth: > > import ..e # [parent-of-parent].e I can't quite get used to the idea of '.' having two different meanings depending on how many '.'s are found next to each other in an import statement. I believe that the notation above _is_ concise, but _is not_ very intuitive ... > Even using the leading '.' syntax, it would be good to have a specific > identifier for the containing package, realized in each module and > respected as a special identifier by the package machinery. I like > '__pkg__': > > import __pkg__.sibling # == 'import .sibling' > > and > > import __pkg__.__pkg__.uncle # == 'import ..uncle' > import __pkg__.__pkg__.uncle.cousin # == 'import ..uncle.cousin' ... while the notation here _is_ intuitive (to a python programmer), but _is not_ concise. I still favor this syntax to the former syntax. I like '__pkg__' better than my initial suggestion ('__parent__'). It's more aesthetically pleasing. :) -- Devin devin@whitebread.org http://www.whitebread.org/ From chrisandannreedy at comcast.net Tue Dec 23 15:03:37 2003 From: chrisandannreedy at comcast.net (Chris and Ann Reedy) Date: Tue Dec 23 15:03:43 2003 Subject: [Python-Dev] Another Strategy for Relative Import Message-ID: <3FE89F99.8BC0D067@comcast.net> Ken Manheimer wrote: > > Even using the leading '.' syntax, it would be good to have a specific > identifier for the containing package, realized in each module and > respected as a special identifier by the package machinery. I like > '__pkg__': > > import __pkg__.sibling # == 'import .sibling' > > and > > import __pkg__.__pkg__.uncle # == 'import ..uncle' > import __pkg__.__pkg__.uncle.cousin # == 'import ..uncle.cousin' > > I think the leading '.' dot forms are a lot nicer, but the presence of > a module named '__pkg__' would be useful for programmatically mucking > navigating the package structure. Delurking: Knowing that the statement: import foo.bar has the effect of setting the variable foo to reference package foo, even if the variable foo already references something else, I was wondering if it would make sense to changes the semantics of import so that if the variable foo is already assigned and is a package (or maybe a module in some cases), then importing foo.bar would have the effect of importing the module or package bar within foo, using foo.__path__ to search for the module bar and foo.__name__ + '.bar' as the name of the module. It seems to me that such an approach might be used in a number of cases: 1. If there is a standard name for the parent of a package, e.g. __pkg__, then import__pkg__.bar just works. 2. This approach allows the following kind of thing to work: if : import xyz as util else: import xyz_old as util import util.bar 3. This could be used to clean up the problem with os.path. The idea here is that since initialization of os creates the attribute os.path which references a module, then: import os.path would have the effect of importing os and then noting that os.path exists and is a module, so no further work is required. I haven't (yet?) attempted to work out whether this could be used to handle all the use cases for relative import. Chris P.S. I've been lurking on this list for a about six months. I just moved between coasts (US) and things are still a mess. I would like to get more involved with Python. I'll send a message in the near future introducing myself. From barry at barrys-emacs.org Wed Dec 24 12:37:06 2003 From: barry at barrys-emacs.org (Barry Scott) Date: Wed Dec 24 12:37:15 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.ne t> References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> Message-ID: <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> At 23-12-2003 16:20, Guido van Rossum wrote: >Wow. Thanks for the analysis. But this is clearly a compiler bug. >Where do we report that? And why would it be unique to OpenBSD? Because the OpenBSD folks have their own code generator to improve the security of OpenBSD. They code gen to make it impossible/improbable to use a stack overflow attack. I found they fix the compiler quickly once you tell then on the OpenBSD dev list whats wrong with a recipe to reproduce. You may find that they will not fix for 3.3 as 3.4 is out and the tool chain moved from a.out to ELF in 3.4. Barry From guido at python.org Wed Dec 24 12:47:22 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 24 12:47:27 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Wed, 24 Dec 2003 17:37:06 GMT." <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> Message-ID: <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> > >Wow. Thanks for the analysis. But this is clearly a compiler bug. > >Where do we report that? And why would it be unique to OpenBSD? > > Because the OpenBSD folks have their own code generator to improve the > security of OpenBSD. They code gen to make it impossible/improbable > to use a stack overflow attack. I found they fix the compiler quickly > once you tell then on the OpenBSD dev list whats wrong with a recipe to > reproduce. Sigh. So much for security. :-( > You may find that they will not fix for 3.3 as 3.4 is out and the > tool chain moved from a.out to ELF in 3.4. Double sigh. Strange approach to security. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 24 13:12:39 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 24 13:12:45 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: Your message of "Wed, 24 Dec 2003 09:47:22 PST." <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> Message-ID: <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> I've tried to look for a way to report OpenBSD bugs, but it appears the only way to report one is by using their sendbug command, which assumes you've got OpenBSD installed. Would someone here be so kind as to report the bug we've found? A pointer to Kurt's last message in this thread should be a good piece of evidence. --Guido van Rossum (home page: http://www.python.org/~guido/) From kbk at shore.net Wed Dec 24 15:22:07 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Dec 24 15:22:13 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Wed, 24 Dec 2003 10:12:39 -0800") References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > I've tried to look for a way to report OpenBSD bugs, but it appears > the only way to report one is by using their sendbug command, which > assumes you've got OpenBSD installed. Would someone here be so kind > as to report the bug we've found? A pointer to Kurt's last message in > this thread should be a good piece of evidence. I'm not ignoring you, shore.net is delaying inbound messages over 24h and that's slowing me down. The problem is related to the ProPolice "stack smash" protector. It does exactly what I saw: It copies any pointer arguments to an area in the frame below the locally declared buffers. The error appears to be in referencing the original location in the arguments when making the call to load_next(). One of the ProPolice ideas is that a pointer (especially a pointer to a function) can be hacked by a buffer overrun to point to rogue code. www.trl.ibm.com/projects/security/ssp/ The implementation appears to be copying all argument pointers for good measure. I'm looking into this to see if there is a better way to fix the import code. Making a local copy of the pointer fixes the segfault but it's not clear to me yet whether that's a solid fix for all uses of this code in Python. As far as reporting the bug, I can do that once I localize the problem in gcc and see whether it's been fixed. A quick search of the OpenBSD bug database came up dry, but maybe I don't know how to phrase the question yet. -- KBK From kbk at shore.net Wed Dec 24 15:45:16 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Dec 24 15:45:29 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Wed, 24 Dec 2003 09:47:22 -0800") References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> Message-ID: Guido van Rossum writes: > Double sigh. Strange approach to security. Their overall approach is outstanding. They understand that the right thing to do is fix insecure code and they audit their base distro extensively. I think they have around a dozen people, including some security professionals, on the audit team. However, it's not possible to audit all the code in the ports, so they have implemented several defensive measures which make it more difficult to mount an attack: 1. ProPolice - anti-stacksmash methods. Sets a canary after the return address, moves local and argument buffers just below that, moves other locals below the buffers, and copies argument pointers into the latter area. If a buffer gets overrun, it nukes the canary. 2. W^X - memory which can be written by an app can't be executed, and vice-versa, assuming MMU support. 3. Non-executable stack @ 3.2 (note that this can cause problems with some functional languages) 4. Greatly reduced use of SUID/GUID binaries. Use of chroot and priviledge separation. It's not surprising there are some residual bugs in the implementation, at least at 3.3. I haven't moved my box to 3.4 yet. -- KBK From andymac at bullseye.apana.org.au Thu Dec 25 08:40:35 2003 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Thu Dec 25 08:48:31 2003 Subject: [Python-Dev] status of 2.3 branch for maintenance checkins Message-ID: <20031226003609.O1176@bullseye.apana.org.au> I don't recall any advice as to whether the 2.3 branch is again open for checkins post 2.3 - Is Jack still sorting out the Mac release? Regards, Andrew. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From anthony at interlink.com.au Thu Dec 25 12:47:43 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Dec 25 12:48:04 2003 Subject: [Python-Dev] status of 2.3 branch for maintenance checkins In-Reply-To: <20031226003609.O1176@bullseye.apana.org.au> Message-ID: <200312251747.hBPHlham032180@localhost.localdomain> >>> Andrew MacIntyre wrote > I don't recall any advice as to whether the 2.3 branch is again open for > checkins post 2.3 - Is Jack still sorting out the Mac release? It's still closed, until Jack says otherwise. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From jeremy at alum.mit.edu Fri Dec 26 14:06:31 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri Dec 26 14:10:04 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects listobject.c,2.172,2.173 In-Reply-To: References: Message-ID: <1072465591.31901.71.camel@localhost.localdomain> On Fri, 2003-12-26 at 14:05, jhylton@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Objects > In directory sc8-pr-cvs1:/tmp/cvs-serv32521 > > Modified Files: > listobject.c > Log Message: > Revert previous two checkins to repair test failure. > > The special-case code that was removed could return a value indicating > success but leave an exception set. test_fileinput failed in a debug > build as a result. If there's some value to the optimization I just removed, let's discuss it on python-dev. I could have fixed the new code, but it seems a lot of hairy new code with magic constants that only covered a few corner cases. The checkin message pointed to a python-list discussion that didn't have much real evidence in it. Jeremy From jeremy at alum.mit.edu Fri Dec 26 15:22:52 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri Dec 26 15:26:33 2003 Subject: [Python-Dev] test_unicode_file fails on Linux Message-ID: <1072470171.31901.74.camel@localhost.localdomain> The utime() call is failing for one of the Unicode file names. build> ./python ../Lib/test/test_unicode_file.py test_directories (__main__.TestUnicodeFiles) ... ok test_equivalent_files (__main__.TestUnicodeFiles) ... ok test_single_files (__main__.TestUnicodeFiles) ... '@test-\xc3\xa0\xc3\xb2' '@test-\xc3\xa0\xc3\xb2' u'@test-\xe0\xf2' ERROR ====================================================================== ERROR: test_single_files (__main__.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "../Lib/test/test_unicode_file.py", line 143, in test_single_files self._test_single(TESTFN_UNICODE) File "../Lib/test/test_unicode_file.py", line 117, in _test_single self._do_single(filename) File "../Lib/test/test_unicode_file.py", line 33, in _do_single os.utime(filename, None) UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-7: ordinal not in range(128) I'm running on a RH9 system. I see that the code hasn't changed since the beginning of the month, so I'm surprised to only discover it now. I can't recall the last time I did a full test run, so I'm not sure how long it has been failing. Jeremy From skip at pobox.com Fri Dec 26 16:17:22 2003 From: skip at pobox.com (Skip Montanaro) Date: Fri Dec 26 16:17:31 2003 Subject: [Python-Dev] test_unicode_file fails on Linux In-Reply-To: <1072470171.31901.74.camel@localhost.localdomain> References: <1072470171.31901.74.camel@localhost.localdomain> Message-ID: <16364.42338.127559.952496@montanaro.dyndns.org> Jeremy> The utime() call is failing for one of the Unicode file names. Been failing for me for awhile (Mac OS X), but in a bit different fashion: % ./python.exe ../Lib/test/regrtest.py -v test_unicode_file test_unicode_file test_directories (test.test_unicode_file.TestUnicodeFiles) ... FAIL test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ... ok test_single_files (test.test_unicode_file.TestUnicodeFiles) ... FAIL ====================================================================== FAIL: test_directories (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 155, in test_directories self._do_directory(TESTFN_ENCODED+ext, TESTFN_ENCODED+ext, os.getcwd) File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 103, in _do_directory make_name) AssertionError: '@test-a\xcc\x80o\xcc\x80.dir' != '@test-\xc3\xa0\xc3\xb2.dir' ====================================================================== FAIL: test_single_files (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 141, in test_single_files self._test_single(TESTFN_ENCODED) File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 116, in _test_single self._do_single(filename) File "/Users/skip/src/python/head/dist/src/Lib/test/test_unicode_file.py", line 41, in _do_single self.failUnless(base in os.listdir(path)) AssertionError ---------------------------------------------------------------------- Ran 3 tests in 0.031s FAILED (failures=2) test test_unicode_file failed -- errors occurred; run in verbose mode for details 1 test failed: test_unicode_file I thought this was a known issue (Unicode normalization differences?), so didn't bother to report it. Skip From Jack.Jansen at cwi.nl Fri Dec 26 16:58:33 2003 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Fri Dec 26 16:58:37 2003 Subject: [Python-Dev] status of 2.3 branch for maintenance checkins In-Reply-To: <20031226003609.O1176@bullseye.apana.org.au> References: <20031226003609.O1176@bullseye.apana.org.au> Message-ID: On 25-dec-03, at 14:40, Andrew MacIntyre wrote: > I don't recall any advice as to whether the 2.3 branch is again open > for > checkins post 2.3 - Is Jack still sorting out the Mac release? Still busy. I expect to be done this wweekend, but I'll send a note here when I'm done. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From tim.one at comcast.net Fri Dec 26 17:01:32 2003 From: tim.one at comcast.net (Tim Peters) Date: Fri Dec 26 17:01:42 2003 Subject: [Python-Dev] test_unicode_file fails on Linux In-Reply-To: <1072470171.31901.74.camel@localhost.localdomain> Message-ID: [Jeremy Hylton] > The utime() call is failing for one of the Unicode file names. > > build> ./python ../Lib/test/test_unicode_file.py > test_directories (__main__.TestUnicodeFiles) ... ok > test_equivalent_files (__main__.TestUnicodeFiles) ... ok > test_single_files (__main__.TestUnicodeFiles) ... > '@test-\xc3\xa0\xc3\xb2' > '@test-\xc3\xa0\xc3\xb2' > u'@test-\xe0\xf2' > ERROR > > ====================================================================== > ERROR: test_single_files (__main__.TestUnicodeFiles) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "../Lib/test/test_unicode_file.py", line 143, in > test_single_files > self._test_single(TESTFN_UNICODE) > File "../Lib/test/test_unicode_file.py", line 117, in _test_single > self._do_single(filename) > File "../Lib/test/test_unicode_file.py", line 33, in _do_single > os.utime(filename, None) > UnicodeEncodeError: 'ascii' codec can't encode characters in position > 6-7: ordinal not in range(128) > > I'm running on a RH9 system. > > I see that the code hasn't changed since the beginning of the month, > so I'm surprised to only discover it now. I can't recall the last > time I did a full test run, so I'm not sure how long it has been > failing. It's been failing "quite a while" on Win98SE too, in the same way. I believe MarkH knows (or should know) about it already, but he's on vacation now: C:\Code\python\PCbuild>python ../lib/test/test_unicode_file.py test_directories (__main__.TestUnicodeFiles) ... ok test_equivalent_files (__main__.TestUnicodeFiles) ... ok test_single_files (__main__.TestUnicodeFiles) ... ERROR ====================================================================== ERROR: test_single_files (__main__.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 142, in test_single_files self._test_single(TESTFN_UNICODE) File "../lib/test/test_unicode_file.py", line 116, in _test_single self._do_single(filename) File "../lib/test/test_unicode_file.py", line 32, in _do_single os.utime(filename, None) UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-7: ordinal not in range(128) ---------------------------------------------------------------------- Ran 3 tests in 0.160s FAILED (errors=1) Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 171, in ? test_main() File "../lib/test/test_unicode_file.py", line 168, in test_main run_suite(suite) File "C:\CODE\PYTHON\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "../lib/test/test_unicode_file.py", line 142, in test_single_files self._test_single(TESTFN_UNICODE) File "../lib/test/test_unicode_file.py", line 116, in _test_single self._do_single(filename) File "../lib/test/test_unicode_file.py", line 32, in _do_single os.utime(filename, None) UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-7: ordinal not in range(128) test_urllib2 is also failing on Win98SE, and that's more recent: C:\Code\python\PCbuild>python ../lib/test/test_urllib2.py test_trivial (__main__.TrivialTests) ... ok test_handled (__main__.OpenerDirectorTests) ... ok test_handler_order (__main__.OpenerDirectorTests) ... ok test_http_error (__main__.OpenerDirectorTests) ... ok test_processors (__main__.OpenerDirectorTests) ... ok test_raise (__main__.OpenerDirectorTests) ... ok test_errors (__main__.HandlerTests) ... ok test_file (__main__.HandlerTests) ... ERROR test_ftp (__main__.HandlerTests) ... ok test_http (__main__.HandlerTests) ... ok test_redirect (__main__.HandlerTests) ... ok test_build_opener (__main__.MiscTests) ... ok ====================================================================== ERROR: test_file (__main__.HandlerTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_urllib2.py", line 345, in test_file r = h.file_open(Request(url)) File "C:\CODE\PYTHON\lib\urllib2.py", line 1058, in file_open return self.open_local_file(req) File "C:\CODE\PYTHON\lib\urllib2.py", line 1073, in open_local_file stats = os.stat(localfile) OSError: [Errno 2] No such file or directory: '\\test.txt' ---------------------------------------------------------------------- Ran 12 tests in 0.170s FAILED (errors=1) Traceback (most recent call last): File "../lib/test/test_urllib2.py", line 628, in ? test_main(verbose=True) File "../lib/test/test_urllib2.py", line 624, in test_main MiscTests, File "C:\CODE\PYTHON\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\CODE\PYTHON\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "../lib/test/test_urllib2.py", line 345, in test_file r = h.file_open(Request(url)) File "C:\CODE\PYTHON\lib\urllib2.py", line 1058, in file_open return self.open_local_file(req) File "C:\CODE\PYTHON\lib\urllib2.py", line 1073, in open_local_file stats = os.stat(localfile) OSError: [Errno 2] No such file or directory: '\\test.txt' test_bsddb3 is in such bad shape on Win98SE now I don't know where to begin. The compiler warnings about mixing signed and unsigned in comparisons in _sre.c also persist. Other than that, everything's great. From sjoerd at acm.org Fri Dec 26 15:39:00 2003 From: sjoerd at acm.org (Sjoerd Mullender) Date: Fri Dec 26 17:39:55 2003 Subject: [Python-Dev] Fixes for imageop module Message-ID: <3FEC9C64.5080805@acm.org> I have some fixes for the imageop module to make it work on systems with a different byte-order than IRIX (e.g. Linux). Shall I check them in or is it not worth it for such an old module? Anything else that needs to be changed when I check this in? The fixes of course will result in different results on Linux than before, so that's the main reason for asking. -- Sjoerd Mullender From guido at python.org Fri Dec 26 17:49:43 2003 From: guido at python.org (Guido van Rossum) Date: Fri Dec 26 17:49:49 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: Your message of "Fri, 26 Dec 2003 21:39:00 +0100." <3FEC9C64.5080805@acm.org> References: <3FEC9C64.5080805@acm.org> Message-ID: <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> > I have some fixes for the imageop module to make it work on systems with > a different byte-order than IRIX (e.g. Linux). Shall I check them in or > is it not worth it for such an old module? Anything else that needs to > be changed when I check this in? > The fixes of course will result in different results on Linux than > before, so that's the main reason for asking. I don't see imageop in the list of deprecated modules in PEP 4, and apparently *you* are still using it, so as long as you don't mind being potentially the sole maintainer and user, I don't mind these being in the distribution. What do you mean by different results on Linux? Was this module previously doing something bogus there? --Guido van Rossum (home page: http://www.python.org/~guido/) From sjoerd at acm.org Fri Dec 26 18:09:39 2003 From: sjoerd at acm.org (Sjoerd Mullender) Date: Fri Dec 26 18:09:44 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> References: <3FEC9C64.5080805@acm.org> <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> Message-ID: <3FECBFB3.7090906@acm.org> Guido van Rossum wrote: >>I have some fixes for the imageop module to make it work on systems with >>a different byte-order than IRIX (e.g. Linux). Shall I check them in or >>is it not worth it for such an old module? Anything else that needs to >>be changed when I check this in? >>The fixes of course will result in different results on Linux than >>before, so that's the main reason for asking. > > > I don't see imageop in the list of deprecated modules in PEP 4, and > apparently *you* are still using it, so as long as you don't mind > being potentially the sole maintainer and user, I don't mind these > being in the distribution. > > What do you mean by different results on Linux? Was this module > previously doing something bogus there? > > --Guido van Rossum (home page: http://www.python.org/~guido/) The main difference is: *************** *** 570,577 **** r = (r<<5) | (r<<3) | (r>>1); g = (g<<5) | (g<<3) | (g>>1); b = (b<<6) | (b<<4) | (b<<2) | b; ! nvalue = r | (g<<8) | (b<<16); ! *ncp++ = nvalue; } return rv; } --- 560,569 ---- r = (r<<5) | (r<<3) | (r>>1); g = (g<<5) | (g<<3) | (g>>1); b = (b<<6) | (b<<4) | (b<<2) | b; ! *ncp++ = 0; ! *ncp++ = b; ! *ncp++ = g; ! *ncp++ = r; } return rv; } where the old ncp is an "Py_UInt32 *" and the new ncp is an "unsigned char *". This results in different byte ordering on a little endian machine such as Intel. -- Sjoerd Mullender From jackjansen at mac.com Sat Dec 27 10:37:08 2003 From: jackjansen at mac.com (Jack Jansen) Date: Sat Dec 27 10:37:15 2003 Subject: [Python-Dev] 2.3.3: test_import wreaks havoc when it runs out of memory Message-ID: <87A87885-3882-11D8-87A5-000A27B19B96@mac.com> During testing 2.3.3 for MacOS9 I think I came across a bug somewhere in import, but I don't have the time to track it down (and there's a workaround), and it's probably only ever going to happen on OS9. Should I put it in the bugs database anyway? If test_import runs out of memory (probably in the test_module_with_large_stack test) it fails with a MemoryError (as it should). However, from that point on there is something fishy in the state of the interpreter: any later test that should simply fail with an ImportError gets a SystemError: NULL return without exception in stead. After a couple of these the interpreter either hangs completely or crashes (I think around the threading tests, but I'm not sure). Note that it's not a question of simply all memory being tied up: it seems test_import does release most of the memory used, and many tests also pass after test_import. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From guido at python.org Sat Dec 27 12:49:46 2003 From: guido at python.org (Guido van Rossum) Date: Sat Dec 27 12:49:52 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: Your message of "Sat, 27 Dec 2003 00:09:39 +0100." <3FECBFB3.7090906@acm.org> References: <3FEC9C64.5080805@acm.org> <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> <3FECBFB3.7090906@acm.org> Message-ID: <200312271749.hBRHnkN25210@c-24-5-183-134.client.comcast.net> [Sjoerd] > >>I have some fixes for the imageop module to make it work on systems with > >>a different byte-order than IRIX (e.g. Linux). Shall I check them in or > >>is it not worth it for such an old module? Anything else that needs to > >>be changed when I check this in? > >>The fixes of course will result in different results on Linux than > >>before, so that's the main reason for asking. > Guido van Rossum wrote: > > I don't see imageop in the list of deprecated modules in PEP 4, and > > apparently *you* are still using it, so as long as you don't mind > > being potentially the sole maintainer and user, I don't mind these > > being in the distribution. > > > > What do you mean by different results on Linux? Was this module > > previously doing something bogus there? [Sjoerd] > The main difference is: > *************** > *** 570,577 **** > r = (r<<5) | (r<<3) | (r>>1); > g = (g<<5) | (g<<3) | (g>>1); > b = (b<<6) | (b<<4) | (b<<2) | b; > ! nvalue = r | (g<<8) | (b<<16); > ! *ncp++ = nvalue; > } > return rv; > } > --- 560,569 ---- > r = (r<<5) | (r<<3) | (r>>1); > g = (g<<5) | (g<<3) | (g>>1); > b = (b<<6) | (b<<4) | (b<<2) | b; > ! *ncp++ = 0; > ! *ncp++ = b; > ! *ncp++ = g; > ! *ncp++ = r; > } > return rv; > } > > where the old ncp is an "Py_UInt32 *" and the new ncp is an "unsigned > char *". This results in different byte ordering on a little endian > machine such as Intel. Hm. Anybody who uses the imageop module currently on Linux will find their programs broken. The strings manipulated by imageop all end up either being written to a file or handed to some drawing code, and changing the byte order would definitely break things! So I don't think this is an acceptable change. I take it that for IRIX, the byte order implied by the old code is simply wrong? Maybe the module can be given a global (shrudder?) byte order setting that you can change but that defaults to the old setting? --Guido van Rossum (home page: http://www.python.org/~guido/) From sjoerd at acm.org Sat Dec 27 13:12:50 2003 From: sjoerd at acm.org (Sjoerd Mullender) Date: Sat Dec 27 13:13:02 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: <200312271749.hBRHnkN25210@c-24-5-183-134.client.comcast.net> References: <3FEC9C64.5080805@acm.org> <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> <3FECBFB3.7090906@acm.org> <200312271749.hBRHnkN25210@c-24-5-183-134.client.comcast.net> Message-ID: <3FEDCBA2.3030607@acm.org> Guido van Rossum wrote: > [Sjoerd] > >>>>I have some fixes for the imageop module to make it work on systems with >>>>a different byte-order than IRIX (e.g. Linux). Shall I check them in or >>>>is it not worth it for such an old module? Anything else that needs to >>>>be changed when I check this in? >>>>The fixes of course will result in different results on Linux than >>>>before, so that's the main reason for asking. > > >>Guido van Rossum wrote: >> >>>I don't see imageop in the list of deprecated modules in PEP 4, and >>>apparently *you* are still using it, so as long as you don't mind >>>being potentially the sole maintainer and user, I don't mind these >>>being in the distribution. >>> >>>What do you mean by different results on Linux? Was this module >>>previously doing something bogus there? > > > [Sjoerd] > >>The main difference is: >>*************** >>*** 570,577 **** >> r = (r<<5) | (r<<3) | (r>>1); >> g = (g<<5) | (g<<3) | (g>>1); >> b = (b<<6) | (b<<4) | (b<<2) | b; >>! nvalue = r | (g<<8) | (b<<16); >>! *ncp++ = nvalue; >> } >> return rv; >> } >>--- 560,569 ---- >> r = (r<<5) | (r<<3) | (r>>1); >> g = (g<<5) | (g<<3) | (g>>1); >> b = (b<<6) | (b<<4) | (b<<2) | b; >>! *ncp++ = 0; >>! *ncp++ = b; >>! *ncp++ = g; >>! *ncp++ = r; >> } >> return rv; >> } >> >>where the old ncp is an "Py_UInt32 *" and the new ncp is an "unsigned >>char *". This results in different byte ordering on a little endian >>machine such as Intel. > > > Hm. Anybody who uses the imageop module currently on Linux will find > their programs broken. The strings manipulated by imageop all end up > either being written to a file or handed to some drawing code, and > changing the byte order would definitely break things! That's why I asked. > So I don't think this is an acceptable change. I take it that for > IRIX, the byte order implied by the old code is simply wrong? Maybe > the module can be given a global (shrudder?) byte order setting that > you can change but that defaults to the old setting? The problem is, the documentation says: "This is the same format as used by gl.lrectwrite() and the imgfile module." This implies the byte order that you get on the SGI which is opposite of what you get on Intel. The code produced the correct results on the SGI, but not on Intel. (By the way, I'm away from computers for a week starting tomorrow extremely early.) -- Sjoerd Mullender From troels at 2-10.org Sat Dec 27 15:10:05 2003 From: troels at 2-10.org (Troels Therkelsen) Date: Sat Dec 27 15:10:11 2003 Subject: [Python-Dev] 2.3.3 Compilation warnings... Message-ID: Hey all, I don't know what the policy on compilation warnings are with the development of Python, but I'd imagine that it's "no warnings". I get the following warnings: gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes - I. -I./Include -DPy_BUILD_CORE -o Python/errors.o Python/errors.c Python/errors.c:602: warning: function declaration isn't a prototype gcc -pthread -Xlinker -export-dynamic -o python \ Modules/python.o \ libpython2.3.a -lpthread -ldl -lutil -lm libpython2.3.a(posixmodule.o): In function `posix_tmpnam': /usr/local/src/Python-2.3.3/./Modules/posixmodule.c:5833: the use of `tmpnam_r' is dangerous, better use `mkstemp' libpython2.3.a(posixmodule.o): In function `posix_tempnam': /usr/local/src/Python-2.3.3/./Modules/posixmodule.c:5788: the use of `tempnam' is dangerous, better use `mkstemp' building 'readline' extension gcc -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasin g -I. -I/usr/local/src/Python-2.3.3/./Include -I/usr/local/include -I/usr/local/ src/Python-2.3.3/Include -I/usr/local/src/Python-2.3.3 -c /usr/local/src/Python-2.3.3/Modules/readline.c -o build/temp.linux-i686-2.3/readline.o /usr/local/src/Python-2.3.3/Modules/readline.c: In function `flex_complete': /usr/local/src/Python-2.3.3/Modules/readline.c:583: warning: implicit declaration of function `completion_matches' /usr/local/src/Python-2.3.3/Modules/readline.c:583: warning: return makes pointer from integer without a cast I guess the warnings in posixmodule has always been there, I just haven't been paying much attention to it and I thought I might just as well report all warnings. System specs: $ uname -a Linux gateway 2.2.19 #22 Wed Jun 20 18:12:16 PDT 2001 i686 unknown $ gcc -v Reading specs from /usr/lib/gcc-lib/i386-slackware-linux/2.95.3/specs gcc version 2.95.3 20010315 (release) Regards, Troels Therkelsen From guido at python.org Sat Dec 27 17:39:47 2003 From: guido at python.org (Guido van Rossum) Date: Sat Dec 27 17:39:53 2003 Subject: [Python-Dev] 2.3.3 Compilation warnings... In-Reply-To: Your message of "Sat, 27 Dec 2003 21:10:05 +0100." References: Message-ID: <200312272239.hBRMdlo01094@c-24-5-183-134.client.comcast.net> > I don't know what the policy on compilation warnings are with the > development of Python, but I'd imagine that it's "no warnings". Right, if possible. > I get the following warnings: > > gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes - > I. -I./Include -DPy_BUILD_CORE -o Python/errors.o Python/errors.c > Python/errors.c:602: warning: function declaration isn't a prototype > > gcc -pthread -Xlinker -export-dynamic -o python \ > Modules/python.o \ > libpython2.3.a -lpthread -ldl -lutil -lm > libpython2.3.a(posixmodule.o): In function `posix_tmpnam': > /usr/local/src/Python-2.3.3/./Modules/posixmodule.c:5833: the use of `tmpnam_r' > is dangerous, better use `mkstemp' > libpython2.3.a(posixmodule.o): In function `posix_tempnam': > /usr/local/src/Python-2.3.3/./Modules/posixmodule.c:5788: the use of `tempnam' > is dangerous, better use `mkstemp' Note that this is the linker speaking, not the compiler. The only way to get rid of this warning is not to use those two entry points, which means removing the functions os.tempnam() and os.tmpnam(). That will eventually happy. These functions have been deprecated already, but for backwards compatibility, we need to provide them for a few more releases. In the mean time, I don't know how to tell the linker to suppress the warnings. > building 'readline' extension > gcc -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasin > g -I. -I/usr/local/src/Python-2.3.3/./Include -I/usr/local/include -I/usr/local/ > src/Python-2.3.3/Include -I/usr/local/src/Python-2.3.3 -c > /usr/local/src/Python-2.3.3/Modules/readline.c -o > build/temp.linux-i686-2.3/readline.o > /usr/local/src/Python-2.3.3/Modules/readline.c: In function `flex_complete': > /usr/local/src/Python-2.3.3/Modules/readline.c:583: warning: implicit > declaration of function `completion_matches' > /usr/local/src/Python-2.3.3/Modules/readline.c:583: warning: return makes > pointer from integer without a cast This seems a case of a missing function declaration in readline.h. Unfortunately being compatible with all versions of GNU readline that are around is very tricky. If you have a patch, I'd consider it, but it would have to be tested on a number of different platforms before being accepted. I don't have the time to do all the verification myself. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Dec 27 17:41:16 2003 From: guido at python.org (Guido van Rossum) Date: Sat Dec 27 17:41:23 2003 Subject: [Python-Dev] Fixes for imageop module In-Reply-To: Your message of "Sat, 27 Dec 2003 19:12:50 +0100." <3FEDCBA2.3030607@acm.org> References: <3FEC9C64.5080805@acm.org> <200312262249.hBQMnhf24306@c-24-5-183-134.client.comcast.net> <3FECBFB3.7090906@acm.org> <200312271749.hBRHnkN25210@c-24-5-183-134.client.comcast.net> <3FEDCBA2.3030607@acm.org> Message-ID: <200312272241.hBRMfHb01333@c-24-5-183-134.client.comcast.net> > > Hm. Anybody who uses the imageop module currently on Linux will find > > their programs broken. The strings manipulated by imageop all end up > > either being written to a file or handed to some drawing code, and > > changing the byte order would definitely break things! > > That's why I asked. > > > So I don't think this is an acceptable change. I take it that for > > IRIX, the byte order implied by the old code is simply wrong? Maybe > > the module can be given a global (shrudder?) byte order setting that > > you can change but that defaults to the old setting? > > The problem is, the documentation says: "This is the same format as used > by gl.lrectwrite() and the imgfile module." This implies the byte order > that you get on the SGI which is opposite of what you get on Intel. The > code produced the correct results on the SGI, but not on Intel. Your fix would be okay if until now it was *only* used on IRIX with gl.lrectwrite() and/or the imgfile module. But how can we prove that? > (By the way, I'm away from computers for a week starting tomorrow > extremely early.) It's okay to way a week before making a decision. --Guido van Rossum (home page: http://www.python.org/~guido/) From edcjones at erols.com Sat Dec 27 18:13:49 2003 From: edcjones at erols.com (Edward C. Jones) Date: Sat Dec 27 18:17:43 2003 Subject: [Python-Dev] Valgrind on Python? Message-ID: <3FEE122D.6040702@erols.com> Has anyone run valgrind on Python? I get a lot of messages, probably mostly incorrect. Most are from "PyObject_Free (Objects/obmalloc.c:711)". I am using Python 2.3.2 and gcc 3.2.2 with Linux. From skanigsvv at terland.ternopil.ua Sat Dec 27 20:19:54 2003 From: skanigsvv at terland.ternopil.ua (Skanigsvv) Date: Sat Dec 27 20:20:18 2003 Subject: [Python-Dev] Python-dev In-Reply-To: References: Message-ID: <9GLL219D5I8FF0CF@terland.ternopil.ua> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031228/e5f01921/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: v1.gif Type: image/gif Size: 7859 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20031228/e5f01921/v1-0001.gif From andymac at bullseye.apana.org.au Sat Dec 27 18:26:59 2003 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Sat Dec 27 20:49:33 2003 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Objects listobject.c, 2.172, 2.173 In-Reply-To: <1072465591.31901.71.camel@localhost.localdomain> References: <1072465591.31901.71.camel@localhost.localdomain> Message-ID: <20031228092926.D12899@bullseye.apana.org.au> On Fri, 26 Dec 2003, Jeremy Hylton wrote: > > Modified Files: > > listobject.c > > Log Message: > > Revert previous two checkins to repair test failure. > > > > The special-case code that was removed could return a value indicating > > success but leave an exception set. test_fileinput failed in a debug > > build as a result. > > If there's some value to the optimization I just removed, let's discuss > it on python-dev. I could have fixed the new code, but it seems a lot > of hairy new code with magic constants that only covered a few corner > cases. The checkin message pointed to a python-list discussion that > didn't have much real evidence in it. While Duncan Booth's tests didn't properly account for setup overhead and his coverage of the range of source element counts insufficient, the performance difference between the two approaches is real and can be quite significant - list_ass_slice() is a big win for large source lists, and a non-trivial loss for small lists (>35% for 10 or fewer elements, >14% up to 64 elements). The change was an attempt to maximise the performance for all cases. If performance of list() is not seen as particularly crucial, then I would suggest that the special case calling of list_ass_slice() should probably be removed completely. The special casing of empty source sequences is not really important IMO, but the timings looked good... Some of the interest in this area comes from that class of Python user who wants to make a shallow copy of a list, and looks for a copy() method on list objects, and failing that finds copy.copy(). Some of them eventually twig that list() also does this, and doesn't require importing the copy module. These users don't feel comfortable with the slice approach, despite the fact that it is considerably faster. In fact I just noticed a posting to python-list/c.l.p from Alex Martelli in which he suggests list() is not unreasonable to use for this purpose. I don't have any problem with the slice, myself. While I tested the change extensively, I'll confess to not trying a debug build :-(. BTW, my ISP's dialin is playing silly buggers at the moment, so I'm not seeing email in a timely fashion. :-( That's not an excuse for the PyDict_CheckExact goof, because I saw that checkin msg and still managed not to see the problem, and hadn't gotten around to building from a fresh CVS checkout :-( Thanks for fixing that. Regards, Andrew. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From tree at basistech.com Sat Dec 27 20:55:52 2003 From: tree at basistech.com (Tom Emerson) Date: Sat Dec 27 21:01:34 2003 Subject: [Python-Dev] Valgrind on Python? In-Reply-To: <3FEE122D.6040702@erols.com> References: <3FEE122D.6040702@erols.com> Message-ID: <16366.14376.408291.541535@magrathea.basistech.com> Yes: valgrind and pymalloc do not get along very well. If you either setup Valgrind exceptions, or better, build without pymaloc, you (and valgrind) will be much happier. -tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever" From skip at manatee.mojam.com Sun Dec 28 08:01:17 2003 From: skip at manatee.mojam.com (Skip Montanaro) Date: Sun Dec 28 08:01:26 2003 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200312281301.hBSD1HoM014593@manatee.mojam.com> Bug/Patch Summary ----------------- 618 open / 4455 total bugs (+70) 220 open / 2504 total patches (+22) New Bugs -------- socketmodule.c does not pickup all symbols with MS visC++ (2003-12-14) http://python.org/sf/860134 PythonIDE does not save for PythonLauncher (2003-12-15) http://python.org/sf/860242 fileinput does not use universal input (2003-12-15) http://python.org/sf/860515 Core dump problems (2003-12-15) http://python.org/sf/860547 UnboundLocalError in cgitb.py (2003-12-16) http://python.org/sf/861340 PythonIDE on osx can't run script with commandline python (2003-12-19) http://python.org/sf/862941 ConfigParser.getint failure on 2.3.3 (2003-12-21) http://python.org/sf/864063 2.3.3 tests on cygwin (2003-12-22) http://python.org/sf/864374 doctest chokes on recursive members (2003-12-23) http://python.org/sf/864944 Python 2.3.3 won't build on MacOSX 10.2 (2003-12-23) http://python.org/sf/864985 Hangs when opening 2nd shell with subprocess under windows (2003-12-23) http://python.org/sf/865014 bsddb test_all.py - incorrect (2003-12-23) http://python.org/sf/865120 SimpleHTTPServer docs out of date (2003-12-26) http://python.org/sf/866222 New Patches ----------- tests and processors patch for urllib2 (2003-12-02) http://python.org/sf/852995 PyErrr_Display and traceback.format_exception_only behaviour (2003-12-15) http://python.org/sf/860326 installing modules are explained in old filenames (2003-12-18) http://python.org/sf/862531 heapq: A way to change the compare function (2003-12-28) http://python.org/sf/866594 Closed Bugs ----------- Over restricted type checking on eval() function (2000-09-22) http://python.org/sf/215126 smtpd.py needs documentation (2001-08-14) http://python.org/sf/450803 Confusions in formatfloat (2002-03-20) http://python.org/sf/532631 email package does not work with mailbox (2002-07-26) http://python.org/sf/586899 Another dealloc stack killer (2002-08-25) http://python.org/sf/600007 doctest fails with TypeError (2003-07-02) http://python.org/sf/764504 doctest.DocTestSuite does not support __test__ (2003-07-15) http://python.org/sf/772091 bsddb3 hash craps out with threads (2003-07-21) http://python.org/sf/775414 Windows _bsddb link warnings (2003-08-01) http://python.org/sf/781614 Minor FP bug in object.c (2003-08-15) http://python.org/sf/789290 Parser wart (2003-08-27) http://python.org/sf/796116 shutil.copyfile fails when dst exists read-only (2003-09-22) http://python.org/sf/810879 Proto 2 pickle vs dict subclass (2003-10-20) http://python.org/sf/826897 os.makedirs() cannot handle "." (2003-10-24) http://python.org/sf/829532 reading shelves is really slow (2003-11-26) http://python.org/sf/849662 Typo in Popen3 description (2003-11-28) http://python.org/sf/850818 Doc/README has broken link (2003-11-28) http://python.org/sf/850823 Missing BuildRequires in src rpm specfile (2003-11-28) http://python.org/sf/851020 socket.recv() raises MemoryError exception (WindowsXP ONLY) (2003-12-03) http://python.org/sf/853507 UserDict.DictMixin's comments should be a docstring (2003-12-07) http://python.org/sf/856072 Remove notion of deprecated string.{atol,atof} functions (2003-12-10) http://python.org/sf/857821 Pathological case segmentation fault in issubclass (2003-12-10) http://python.org/sf/858016 typo in doc (2003-12-14) http://python.org/sf/859810 typo in docs (2003-12-14) http://python.org/sf/859811 Closed Patches -------------- 658254: accept None for time.ctime() and friends (2003-01-06) http://python.org/sf/663482 doctest handles comments incorrectly (2003-01-15) http://python.org/sf/668500 iconv_codec 3rd generation (2003-04-13) http://python.org/sf/720585 test_timeout updates (2003-04-28) http://python.org/sf/728815 Port tests to unittest (Part 2) (2003-05-13) http://python.org/sf/736962 traceback module caches sources invalid (2003-05-13) http://python.org/sf/737473 ast-branch: multiple function fixes (2003-06-11) http://python.org/sf/752911 itertools roundrobin() (2003-06-17) http://python.org/sf/756253 add time elapsed to gc debug output (2003-06-25) http://python.org/sf/760990 A ForwardingHandler for logging (2003-09-10) http://python.org/sf/804180 call com_set_lineno more often (2003-11-28) http://python.org/sf/850789 zipimport.c is broken on 64-bit SusE AMD, here's a fix (2003-12-11) http://python.org/sf/858317 documentation bool change fix (2003-12-12) http://python.org/sf/859286 Get rid of compiler warnings on unicodeobject.c (2003-12-13) http://python.org/sf/859573 From barry at barrys-emacs.org Sun Dec 28 14:48:23 2003 From: barry at barrys-emacs.org (Barry Scott) Date: Sun Dec 28 14:48:31 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.ne t> References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> Message-ID: <6.0.1.1.2.20031228194701.023490c8@torment.chelsea.private> At 24-12-2003 18:12, Guido van Rossum wrote: >I've tried to look for a way to report OpenBSD bugs, but it appears >the only way to report one is by using their sendbug command, which >assumes you've got OpenBSD installed. Would someone here be so kind >as to report the bug we've found? A pointer to Kurt's last message in >this thread should be a good piece of evidence. > >--Guido van Rossum (home page: http://www.python.org/~guido/) From the "How to report a problem" page on www.openbsd.org they say mail to bugs@openbsd.org. Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20031228/13980c19/attachment.html From barry at barrys-emacs.org Sun Dec 28 14:53:33 2003 From: barry at barrys-emacs.org (Barry Scott) Date: Sun Dec 28 14:53:39 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> Message-ID: <6.0.1.1.2.20031228195231.022a77c8@torment.chelsea.private> You can use -fno-stack-protector to turn off the propolice code. Why not do that until there is a working combination? Barry From perky at i18n.org Mon Dec 29 03:17:24 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Mon Dec 29 03:17:31 2003 Subject: [Python-Dev] Optimization versus code bloat Message-ID: <20031229081724.GA42076@i18n.org> Hello! I posted a patch[1] optimizing str.split with specialized splitter as unicode.split do. The patch accellates str.split about 10~20% and adds about 50 lines. I'm curious whether it's acceptable ratio for python-dev moods. [1] http://www.python.org/sf/866875 Hye-Shik From python-dev at zesty.ca Mon Dec 29 04:43:16 2003 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Mon Dec 29 04:43:31 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: <20031220151629.GA24651@panix.com> Message-ID: I've been distracted by holiday events, but this requires a reply. On Sat, 20 Dec 2003, Aahz wrote: > Exactly. From my observations of these discussions, there are > essentially only two reasons for restricted execution: > > * To simplify things by reducing the potential solution space > > * To protect a system against a hostile attacker There is a huge blind spot in your claim. You forgot: * To limit the damage caused by a bug in your program * To make your programs have more predictable behaviour Capabilities are about making clear what parts of your program can and can't do. > Supposedly there's a middle ground of untrusted but non-hostile code, > but what's the point of providing support for that? Have you ever used a library written by someone else? Have you ever found a bug in something you wrote yourself? "Untrusted but non-hostile code" is what all of us write every day. -- ?!ng From guido at python.org Mon Dec 29 09:52:46 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 29 09:53:02 2003 Subject: [Python-Dev] Optimization versus code bloat In-Reply-To: Your message of "Mon, 29 Dec 2003 17:17:24 +0900." <20031229081724.GA42076@i18n.org> References: <20031229081724.GA42076@i18n.org> Message-ID: <200312291452.hBTEql011394@c-24-5-183-134.client.comcast.net> > I posted a patch[1] optimizing str.split with specialized splitter > as unicode.split do. The patch accellates str.split about 10~20% > and adds about 50 lines. I'm curious whether it's acceptable ratio > for python-dev moods. > > [1] http://www.python.org/sf/866875 Works for me -- split is a pretty common operation! I might even use this speedup in the Parrot benchmark. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Mon Dec 29 09:56:06 2003 From: aahz at pythoncraft.com (Aahz) Date: Mon Dec 29 09:56:10 2003 Subject: [Python-Dev] Re: Capabilities - published interfaces In-Reply-To: References: <20031220151629.GA24651@panix.com> Message-ID: <20031229145606.GA24858@panix.com> On Mon, Dec 29, 2003, Ka-Ping Yee wrote: > On Sat, 20 Dec 2003, Aahz wrote: >> >> Exactly. From my observations of these discussions, there are >> essentially only two reasons for restricted execution: >> >> * To simplify things by reducing the potential solution space >> >> * To protect a system against a hostile attacker > > There is a huge blind spot in your claim. You forgot: > > * To limit the damage caused by a bug in your program > > * To make your programs have more predictable behaviour > > Capabilities are about making clear what parts of your program > can and can't do. My understanding is that capabilities are not the same thing as restricted execution; it's certainly the case that we could provide some kind of capability model that doesn't meet the demands of restricted execution. While your points are valid, they are not points that I've seen brought up before in the context of restricted execution. There are many other programming models that address those issues; why do you think that restricted execution makes a particularly good choice? (Say, as compared to design-by-contract.) >> Supposedly there's a middle ground of untrusted but non-hostile code, >> but what's the point of providing support for that? > > Have you ever used a library written by someone else? Have you ever > found a bug in something you wrote yourself? > > "Untrusted but non-hostile code" is what all of us write every day. Well, sure, but if that's the only issue in play, I don't think that the stringent demands of restricted execution are needed. The question is whether the benefits of restricted execution are worth the effort required (and possibly the performance penalty). Not-quite tangent: I was talking to someone a few days ago, an experienced programmer who hasn't yet tried Python (and who I'm trying to persuade -- but he's an old Lisp and Smalltalk hacker who hates B&D syntax). He's also running a web server at home; I commented that if I did that, I'd use a server written in Python because it's more secure. He said that he'd seen more security alerts for Zope than for Apache, which forced me to explain that Zope isn't a web server, it's an application framework that allows people to run arbitrary Python code. >From my POV, that's the context in which we either need to provide truly robust restricted execution or just decide that we can't do it. And as I've pointed out before, restricted execution in the context of something like a web server is almost certainly going to require something external to Python in order to prevent excessive CPU and memory usage -- which is an issue that has grown in difficulty given the new prevalence of event-driven programming (instead of forking and threading). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From martin at v.loewis.de Mon Dec 29 10:25:40 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 10:25:55 2003 Subject: [Python-Dev] First version of Python MSI available Message-ID: <3FF04774.9020404@v.loewis.de> I have committed python/nondist/sandbox/msi to the CVS. This is a MSI generator for Python on Windows, which I'd like to propose as a replacement for Wise. A sample .msi file is available at http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi This installer includes binaries compiled with MSVC 7.1 (aka VC.NET 2003). The MSI file tries to copy most features and user interface from the Wise installer, but adjusts to the feature set of MSI. I'm aware of a few issues, but I'd like to request feedback in any case. Known issues: - the C runtime DLL is not yet included. It turns out that the VC.NET merge module for msvcr71.dll is useless, so I have to extract the DLL from it and incorporate that; I'd also like to include some auto-detection of DLLs used. - MSI "administrative installs" (network installs) are not tested yet; neither is advertisement (this should not work well, in any case, as the MSI file won't advertise the shortcuts) - the package doesn't ask for confirmation when installing into an existing directory. Not sure how to do this with MSI. Please let me know what you think, Martin From martin at v.loewis.de Mon Dec 29 10:27:28 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 10:27:46 2003 Subject: [Python-Dev] Switching to VC.NET 2003 Message-ID: <3FF047E0.8070905@v.loewis.de> In my own sandbox, I have prepared project files for VC.NET. Unless there are any strong objections, I'd like to switch over to VC.NET shortly, disabling VC6 builds. In the process, I will also update the build dependencies to more recent versions of several packages. Regards, Martin From guido at python.org Mon Dec 29 10:32:30 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 29 10:32:35 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: Your message of "Mon, 29 Dec 2003 16:27:28 +0100." <3FF047E0.8070905@v.loewis.de> References: <3FF047E0.8070905@v.loewis.de> Message-ID: <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> > In my own sandbox, I have prepared project files for VC.NET. > > Unless there are any strong objections, I'd like to switch > over to VC.NET shortly, disabling VC6 builds. In the process, > I will also update the build dependencies to more recent > versions of several packages. Will the resulting installer still install a working version of Python on old versions of Windows, like win95 and win98? I presume this will require that the MSI support is present on the target system; does this exist for Win95 etc? How about Windows NT 4.0? If not, what's the oldest Windows version still supported? --Guido van Rossum (home page: http://www.python.org/~guido/) From logistix at cathoderaymission.net Mon Dec 29 11:09:10 2003 From: logistix at cathoderaymission.net (logistix) Date: Mon Dec 29 10:54:29 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> Message-ID: > In my own sandbox, I have prepared project files for VC.NET. > > Unless there are any strong objections, I'd like to switch > over to VC.NET shortly, disabling VC6 builds. In the process, > I will also update the build dependencies to more recent > versions of several packages. > Are you using a Visual Studio "project" or a makefile based build? I believe you need a makefile based build if you want to let people develop with the free SDK from microsoft. I have some notes here that I posted a while ago: http://tinyurl.com/34ljh From martin at v.loewis.de Mon Dec 29 11:00:47 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:01:01 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> References: <3FF047E0.8070905@v.loewis.de> <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> Message-ID: <3FF04FAF.40209@v.loewis.de> Guido van Rossum wrote: > Will the resulting installer still install a working version of Python > on old versions of Windows, like win95 and win98? I don't know; in principle, it should (or atleast, we should be able to make it work if desired). > I presume this will > require that the MSI support is present on the target system; does > this exist for Win95 etc? How about Windows NT 4.0? Yes; there is a four-file bootstrap mechanism that we could distribute. It consists of 1. a tiny setup.exe 2. an installer for installer for W9x 3. an installer for installer for WNT 4. the actual MSI file. I believe the setup.exe driver needs to be parametrized as part of the build process to hard-code the name of the MSI file; I have no procedure yet to do this. I would propose to make parts 1..3 optional downloads; we can also (alternatively) point users to the redistributable download at http://www.microsoft.com/msdownload/platformsdk/instmsi.htm With that procedure, they would explicitly need to install installer, then reboot (I think), then install Python. On some installations, installer might be present already, but in a too-old version (I use the schema of Installer 2.0). In principle, the already-present installer should find out that it is too old, and suggest some sort of upgrade. > If not, what's the oldest Windows version still supported? A different question is the version of shared libraries that the Python binaries depend on. On the one hand, this is msvcr71.dll, which is not present by default on any Windows release, so we need to ship that. Another issue is Winsock 2 (which, I believe, is in wsock32.dll). A quick research indicates that this was present in Win95 already. I don't know whether all of the entry points that we use was present already; confirmation appreciated. In principle, usage of IPv6 could be a problem. In another principle, the platform SDK arranges to transparently emulate the IPv6 API on systems where it is not natively available, by means of GetProcAddress. This all needs testing. Regards, Martin From martin at v.loewis.de Mon Dec 29 11:06:39 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:06:52 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: References: Message-ID: <3FF0510F.3070504@v.loewis.de> logistix wrote: > Are you using a Visual Studio "project" or a makefile based build? I > believe you need a makefile based build if you want to let people > develop with the free SDK from microsoft. I'm using project files, so users of the free compiler would have to write their makefiles themselves. It might be possible to write a generator that produces a makefile automatically from the VC.NET project files; contributions are welcome. Maintaining both project files and makefiles in parallel is not feasible; dropping the project files is also unacceptable. Regards, Martin From martin at v.loewis.de Mon Dec 29 11:12:44 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:13:04 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF04FAF.40209@v.loewis.de> References: <3FF047E0.8070905@v.loewis.de> <200312291532.hBTFWU011539@c-24-5-183-134.client.comcast.net> <3FF04FAF.40209@v.loewis.de> Message-ID: <3FF0527C.6010909@v.loewis.de> Martin v. Loewis wrote: > On some installations, installer might be present already, > but in a too-old version (I use the schema of Installer 2.0). > In principle, the already-present installer should find out > that it is too old, and suggest some sort of upgrade. Following up to myself: If this turns out to be a frequent problem, we could probably go back to an older installer scheme - in particular, if we don't want to fight Windows File Protection (which is only transparently supported in most recent installer versions). In the sample installer, I put python24.dll into TARGETDIR, not into System32Folder. I'd like to keep it that way, as Microsoft discourages deployment into system32. Of course, if there is a good reason to install python24.dll into system32, changing the installer generator to do so would be only a small effort. Regards, Martin From pje at telecommunity.com Mon Dec 29 11:36:06 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 29 11:37:31 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF047E0.8070905@v.loewis.de> Message-ID: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> At 04:27 PM 12/29/03 +0100, Martin v. Loewis wrote: >In my own sandbox, I have prepared project files for VC.NET. > >Unless there are any strong objections, I'd like to switch >over to VC.NET shortly, disabling VC6 builds. In the process, >I will also update the build dependencies to more recent >versions of several packages. Was the question of mingw32-built extensions ever resolved? That is, will we be able to build extensions for the standard Windows Python using the distutils' "mingw32" compiler, as is possible with Python 2.2? If this hasn't been resolved, and somebody can send me a binary for a .NET-built Python, I'll be happy to test. I have several Pyrex and C extensions of varying vintage that I can use for such a test. From tim.one at comcast.net Mon Dec 29 11:38:33 2003 From: tim.one at comcast.net (Tim Peters) Date: Mon Dec 29 11:38:40 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF0527C.6010909@v.loewis.de> Message-ID: [Martin v. Loewis] > ... > > In the sample installer, I put python24.dll into TARGETDIR, > not into System32Folder. I'd like to keep it that way, > as Microsoft discourages deployment into system32. Of course, > if there is a good reason to install python24.dll into > system32, changing the installer generator to do so would > be only a small effort. We need to hear from Mark Hammond about this, but he's on vacation now. The current installer's "non-admin install" option leaves system folders alone, and works fine for everything I routinely do on Windows. The point to stuffing the Python DLL into a system directory had (IIRC) something to do with a Windows Service embedding a Python component via COM, in which case "the virtual command line" that started the service has no idea where Python might live on the box. Or something like that -- or something not like that at all. That's why we need Mark . From martin at v.loewis.de Mon Dec 29 11:43:13 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:43:32 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> Message-ID: <3FF059A1.8090202@v.loewis.de> Phillip J. Eby wrote: > Was the question of mingw32-built extensions ever resolved? That is, > will we be able to build extensions for the standard Windows Python > using the distutils' "mingw32" compiler, as is possible with Python 2.2? I don't know; I don't use mingw32. OTOH, I cannot see what the problem might be - except that you will have to link with msvcr71.dll at the end, not with msvcrt40.dll. > If this hasn't been resolved, and somebody can send me a binary for a > .NET-built Python, I'll be happy to test. Sure: Please try my installer at http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi Notice that this doesn't include msvcr71.dll, which I could provide in private. It also does not include msvcr71.lib - I have no clue how you would get a copy of that. But then, somehow, msvcrt40.lib (or is it msvcrt40.a) must have gotten into mingw32 also. Regards, Martin From martin at v.loewis.de Mon Dec 29 11:45:52 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:46:17 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: References: Message-ID: <3FF05A40.4070709@v.loewis.de> Tim Peters wrote: > The point to > stuffing the Python DLL into a system directory had (IIRC) something to do > with a Windows Service embedding a Python component via COM, in which case > "the virtual command line" that started the service has no idea where Python > might live on the box. Ah, ok. I'll figure out in the mean time how to incorporate msvcr71.dll into the package in the first place; putting it then into a "shared component" (along with pythonxy.dll) should be trivial. Regards, Martin From martin at v.loewis.de Mon Dec 29 11:50:18 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 11:50:34 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF059A1.8090202@v.loewis.de> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: <3FF05B4A.2070706@v.loewis.de> Martin v. Loewis wrote: > It also does not include msvcr71.lib - I have no clue how you would > get a copy of that. But then, somehow, msvcrt40.lib (or is it > msvcrt40.a) must have gotten into mingw32 also. Some googling shows there is hope: http://sources.redhat.com/ml/cygwin-announce/2003-09/msg00089.html 2003-06-18 Earnie Boyd * Makefile.in (LIBS): Add new MSVCRT libraries libmsvcr70 and libmsvcr71, including debug versions. (msvcr70.def, msvcr70d.def, msvcr71.def, msvcr71.def): New targets. So atleast the part of linking with msvcr71.dll should be possible. As I said, I cannot imagine what else might be a problem. Regards, Martin From aleaxit at yahoo.com Mon Dec 29 11:53:09 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Mon Dec 29 11:53:23 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF0510F.3070504@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> Message-ID: <200312291753.09793.aleaxit@yahoo.com> On Monday 29 December 2003 05:06 pm, Martin v. Loewis wrote: ... > It might be possible to write a generator that produces a makefile > automatically from the VC.NET project files; contributions At the time of Visual Studio 6, such a "generator" had been written by Microsoft, and was part of the VS6 package -- i.e., you could produce and save a makefile for the project you had loaded. Is this no more the case for the current release of Visual Studio...? Alex From logistix at cathoderaymission.net Mon Dec 29 12:14:53 2003 From: logistix at cathoderaymission.net (logistix) Date: Mon Dec 29 12:00:38 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312291753.09793.aleaxit@yahoo.com> Message-ID: On Mon, 29 Dec 2003, Alex Martelli wrote: > On Monday 29 December 2003 05:06 pm, Martin v. Loewis wrote: > ... > > It might be possible to write a generator that produces a makefile > > automatically from the VC.NET project files; contributions > > At the time of Visual Studio 6, such a "generator" had been written > by Microsoft, and was part of the VS6 package -- i.e., you could produce > and save a makefile for the project you had loaded. Is this no more the case > for the current release of Visual Studio...? > > > Alex > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/logistix%40cathoderaymission.net > Yes, they removed that functionality (or at least I wasn't able to find it). I used Visual Studio 6.0 to generate a good makefile in my tests. In VC 7.1, you can feed the project file into the compiler via a command line flag. I think you needed to use the GUI to build in 6.0, so MS provided the makefile export feature for people who wanted to do automated builds and other stuff where you couldn't rely on the gui. So the feature became obsolete from MS's perspective. Unfortunately, I don't think the command-line compilers in the SDK will accept a Visual Studio project file. From martin at v.loewis.de Mon Dec 29 12:03:20 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 12:04:23 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312291753.09793.aleaxit@yahoo.com> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> Message-ID: <3FF05E58.1070108@v.loewis.de> Alex Martelli wrote: > At the time of Visual Studio 6, such a "generator" had been written > by Microsoft, and was part of the VS6 package -- i.e., you could produce > and save a makefile for the project you had loaded. Is this no more the case > for the current release of Visual Studio...? No. Microsoft has dropped that feature (perhaps they will restore it in VS.NET 2005 or something :-( However, it is now simpler to process project files; they are XML files with a fairly obvious vocabulary (*). A friend of mine once wrote a generator that produces GNU makefiles out of vcproj files; such generators tend to be quite application-specific, as they need to take custom build steps into account, and translate them properly. Most likely, the generator we would use would not be useful outside Python. I would like to see a single makefile generated, instead of one per project, as the VC6 generator would do; having so many makefiles is painful. That makefile should have an "all" target in addition to the targets for the individual projects. Regards, Martin (*) OTOH, the "solution" files (formerly workspaces) don't use XML syntax, and have changed between VS.NET 2002 and VS.NET 2003. They would need to be considered as well, as they contain the inter-project dependencies (including the list of all projects in the solution). From martin at v.loewis.de Mon Dec 29 12:24:57 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 12:25:42 2003 Subject: [Python-Dev] str.ascii_lower Message-ID: <3FF06369.9020508@v.loewis.de> Looking at python.org/sf/866982, I find it troubling that there are languages where "I".lower() != "i" (for those of you not familiar with Turkish: the lower-case letter of "I" is U+0131, LATIN SMALL LETTER DOTLESS I, which is \xfd in iso-8859-9). As a solution, I'd like to propose a new method ascii_lower, which is locale-unaware and only works for bytes 65..90 (returning the byte itself for all other characters). Similarly, ascii_upper might be needed "for symmetry"; I don't know whether the symmetry should extend beyond those two. This, in turn, should be used inside the codecs library where encoding names are normalized to lower case. What do you think? Regards, Martin From guido at python.org Mon Dec 29 12:37:37 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 29 12:37:42 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: Your message of "Mon, 29 Dec 2003 18:24:57 +0100." <3FF06369.9020508@v.loewis.de> References: <3FF06369.9020508@v.loewis.de> Message-ID: <200312291737.hBTHbbM11741@c-24-5-183-134.client.comcast.net> > Looking at python.org/sf/866982, I find it troubling that > there are languages where "I".lower() != "i" > (for those of you not familiar with Turkish: the lower-case > letter of "I" is U+0131, LATIN SMALL LETTER DOTLESS I, > which is \xfd in iso-8859-9). > > As a solution, I'd like to propose a new method ascii_lower, > which is locale-unaware and only works for bytes 65..90 > (returning the byte itself for all other characters). > > Similarly, ascii_upper might be needed "for symmetry"; > I don't know whether the symmetry should extend beyond > those two. > > This, in turn, should be used inside the codecs library > where encoding names are normalized to lower case. > > What do you think? I never though there were locales possible that affected the mappings inside ASCII either. But shouldnt' this work just as well if it's only for encoding names (which I'd hope would be ASCII themselves): def ascii_lower(s): return str(unicode(s).lower()) The unicode() call converts ASCII to Unicode, which should always work for encoding names, and the Unicode lower() is locale-independent. This seems more elegant than adding yet more methods to the str type. --Guido van Rossum (home page: http://www.python.org/~guido/) From jepler at unpythonic.net Mon Dec 29 12:40:08 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 29 12:40:11 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <3FF06369.9020508@v.loewis.de> References: <3FF06369.9020508@v.loewis.de> Message-ID: <20031229174008.GH6171@unpythonic.net> On Mon, Dec 29, 2003 at 06:24:57PM +0100, Martin v. Loewis wrote: > Looking at python.org/sf/866982, I find it troubling that > there are languages where "I".lower() != "i" > (for those of you not familiar with Turkish: the lower-case > letter of "I" is U+0131, LATIN SMALL LETTER DOTLESS I, > which is \xfd in iso-8859-9). This post caused me to notice the following behavior. Is it "right"? >>> import locale >>> locale.setlocale(locale.LC_CTYPE, "tr_TR") 'tr_TR' >>> locale.getlocale()[1] # Expected charset 'ISO8859-9' >>> "I".lower() # Expected behavior '\xfd' >>> u"I".lower() # Python bug? (should be u'\u0131') u'i' >>> locale.setlocale(locale.LC_CTYPE, "tr_TR.UTF-8") 'tr_TR.UTF-8' >>> "I".lower() # C library bug? (should be "\xc4\xb1")* 'I' >>> locale.setlocale(locale.LC_CTYPE, "en_US.UTF-8") 'en_US.UTF-8' >>> "I".lower() # (UTF-8 locale works properly in english) 'i' Jeff * RedHat 9, glibc-2.3.2-11.9 From skip at pobox.com Mon Dec 29 09:01:28 2003 From: skip at pobox.com (Skip Montanaro) Date: Mon Dec 29 12:42:06 2003 Subject: [Python-Dev] Optimization versus code bloat In-Reply-To: <20031229081724.GA42076@i18n.org> References: <20031229081724.GA42076@i18n.org> Message-ID: <16368.13240.502549.132898@montanaro.dyndns.org> Hye-Shik> I posted a patch[1] optimizing str.split with specialized Hye-Shik> splitter as unicode.split do. The patch accellates str.split Hye-Shik> about 10~20% and adds about 50 lines. I'm curious whether it's Hye-Shik> acceptable ratio for python-dev moods. Hye-Shik> [1] http://www.python.org/sf/866875 Seems like a reasonable addition to me, especially if the same optimization is performed for unicode objects. I'm a bit confused about all the changes to test_string.py though. Since your change is only for performance it seems to me that no changes to the test suite would have been necessary unless it wasn't covering some critical tests. Was that just a rearrangement of that file or did you add some new test cases? Whether or not your optimization is accepted, if your test cases beef up the test suite, I'd vote to add them. Skip From pje at telecommunity.com Mon Dec 29 12:44:46 2003 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Dec 29 12:46:08 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF059A1.8090202@v.loewis.de> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> Message-ID: <5.1.1.6.0.20031229123908.023122e0@telecommunity.com> At 05:43 PM 12/29/03 +0100, Martin v. Loewis wrote: OTOH, I cannot see what the problem might be - except that you will >have to link with msvcr71.dll at the end, not with msvcrt40.dll. > >>If this hasn't been resolved, and somebody can send me a binary for a >>.NET-built Python, I'll be happy to test. > >Sure: Please try my installer at > >http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi > >Notice that this doesn't include msvcr71.dll, which I could provide >in private. Please do. >It also does not include msvcr71.lib - I have no clue how you would >get a copy of that. But then, somehow, msvcrt40.lib (or is it >msvcrt40.a) must have gotten into mingw32 also. Actually, one generates that by using dlltool and various other programs to extract a .def file from python2x.dll. I haven't generated one in ages, so I'm going to have to dig back up how to do that. The rest is handled by the distutils, pretty much. I don't think there's actually any direct linking to the MS VC runtime, anyway, if one is only using Python API calls. But I guess I'll find out. Hopefully I'll have enough clock cycles to spare on Thursday to get everything set up and tested. From guido at python.org Mon Dec 29 12:47:39 2003 From: guido at python.org (Guido van Rossum) Date: Mon Dec 29 12:47:44 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: Your message of "Mon, 29 Dec 2003 11:40:08 CST." <20031229174008.GH6171@unpythonic.net> References: <3FF06369.9020508@v.loewis.de> <20031229174008.GH6171@unpythonic.net> Message-ID: <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> > This post caused me to notice the following behavior. Is it "right"? > > >>> import locale > >>> locale.setlocale(locale.LC_CTYPE, "tr_TR") > 'tr_TR' > >>> locale.getlocale()[1] # Expected charset > 'ISO8859-9' > >>> "I".lower() # Expected behavior > '\xfd' > >>> u"I".lower() # Python bug? (should be u'\u0131') > u'i' Why? Unicode strings are not affected by the locale. > >>> locale.setlocale(locale.LC_CTYPE, "tr_TR.UTF-8") > 'tr_TR.UTF-8' > >>> "I".lower() # C library bug? (should be "\xc4\xb1")* > 'I' > >>> locale.setlocale(locale.LC_CTYPE, "en_US.UTF-8") > 'en_US.UTF-8' > >>> "I".lower() # (UTF-8 locale works properly in english) > 'i' I have no idea what adding UTF8 to the local means. Is this something that Python's locale-awareness does or is it simply recognized by the C library? --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Mon Dec 29 12:59:18 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 13:00:16 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <200312291737.hBTHbbM11741@c-24-5-183-134.client.comcast.net> References: <3FF06369.9020508@v.loewis.de> <200312291737.hBTHbbM11741@c-24-5-183-134.client.comcast.net> Message-ID: <3FF06B76.3010004@v.loewis.de> Guido van Rossum wrote: > But shouldnt' this work just as well if it's only for encoding names > (which I'd hope would be ASCII themselves): > > def ascii_lower(s): > return str(unicode(s).lower()) Certainly. Another implementation would be lower_map = string.maketrans(string.ascii_upper, string.ascii_lower) def ascii_lower(str): return str.translate(lower_map) > This seems more elegant than adding yet more methods to the str type. Ok. I'll fix it in this direction, then. There would be two copies of this function that I can see: One for the codecs (which uses tolower() in C code), and one in the email module. And yes, codec names must be pure ASCII. Regards, Martin From martin at v.loewis.de Mon Dec 29 13:04:56 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 13:05:06 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <20031229174008.GH6171@unpythonic.net> References: <3FF06369.9020508@v.loewis.de> <20031229174008.GH6171@unpythonic.net> Message-ID: <3FF06CC8.5070705@v.loewis.de> Jeff Epler wrote: >>>>u"I".lower() # Python bug? (should be u'\u0131') > > u'i' As Guido says: unicode.tolower is locale-inaware; it uses the Unicode Consortium character properties instead to determine the lower-case character. >>>>"I".lower() # C library bug? (should be "\xc4\xb1")* > > 'I' This is really a limitation of the C language, not of the C library. The interface is char tolower(char input); so it can only accept and return a single char. Multi-byte characters are not supported in that interface. Traditionally, for characters that cannot be converted, tolower returns its argument. >>>>"I".lower() # (UTF-8 locale works properly in english) > > 'i' This is because "i" is a single byte in UTF-8. Regards, Martin From jepler at unpythonic.net Mon Dec 29 13:09:27 2003 From: jepler at unpythonic.net (Jeff Epler) Date: Mon Dec 29 13:09:30 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> References: <3FF06369.9020508@v.loewis.de> <20031229174008.GH6171@unpythonic.net> <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> Message-ID: <20031229180927.GI6171@unpythonic.net> I stand corrected about the behavior of Unicode in the presence of locales. On Mon, Dec 29, 2003 at 09:47:39AM -0800, Guido van Rossum wrote: > > >>> locale.setlocale(locale.LC_CTYPE, "tr_TR.UTF-8") > > 'tr_TR.UTF-8' > > >>> "I".lower() # C library bug? (should be "\xc4\xb1")* > > 'I' > > >>> locale.setlocale(locale.LC_CTYPE, "en_US.UTF-8") > > 'en_US.UTF-8' > > >>> "I".lower() # (UTF-8 locale works properly in english) > > 'i' > > I have no idea what adding UTF8 to the local means. Is this something > that Python's locale-awareness does or is it simply recognized by the > C library? "A locale name is typically of the form language[_territory] [.code-set][@modifier]" -- man setlocale() on my system RedHat 9 made a halfhearted attempt to use UTF-8 as the encoding for all locales. So it sets LANG=en_US.UTF-8 by default. In theory, tr_TR.UTF_8 should be the Turkish locale with UTF-8 characters, but it behaves incorrectly by having "I".lower() == "I". Well, since my earlier post combined a misunderstanding of how Python works with a possible C library bug, I guess I raised two non-issues. Sorry for wasting everyone's time. Jeff From martin at v.loewis.de Mon Dec 29 13:09:40 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 13:09:59 2003 Subject: [Python-Dev] str.ascii_lower In-Reply-To: <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> References: <3FF06369.9020508@v.loewis.de> <20031229174008.GH6171@unpythonic.net> <200312291747.hBTHldO11794@c-24-5-183-134.client.comcast.net> Message-ID: <3FF06DE4.1010406@v.loewis.de> Guido van Rossum wrote: > I have no idea what adding UTF8 to the local means. Is this something > that Python's locale-awareness does or is it simply recognized by the > C library? It's recognized by the C library, both on Windows and Unix. The full format is _ @ . By default, each implies a default , which can be overridden. On Linux, is a IANA name (e.g. de_DE.ISO-8859-15); on Windows, it is a code page (e.g. german_Germany.1252). Regards, Martin From paoloinvernizzi at dmsware.com Mon Dec 29 15:30:07 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Mon Dec 29 15:30:28 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF05E58.1070108@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> Message-ID: <3FF08ECF.4010302@dmsware.com> Martin v. Loewis wrote: > No. Microsoft has dropped that feature (perhaps they will restore it > in VS.NET 2005 or something :-( > > However, it is now simpler to process project files; they are XML > files with a fairly obvious vocabulary (*). A friend of mine once > wrote a generator that produces GNU makefiles out of vcproj files; > such generators tend to be quite application-specific, as they > need to take custom build steps into account, and translate them > properly. Most likely, the generator we would use would not be > useful outside Python. I can suggest the fact that Scons can handle multiple builders (cygwin, mingw, VC6, VC7, VC7.1) easily and for VC it can generate the proper Project files? Can be a solution to support a single scons (makefile like) file and from that generate the proper project files? Users that wants to keep working with the IDE can do that! A free solution for a user that want to build python in a win32 environment without Visual Studio (whatever edition) could be: - Download python (from 1.5.2 above) and scons (as a make replacement) - Build python with the free Microsoft SDK compiler via scons. Scons has also an integrated autoconf-like environment, so it's possible to have a path to unify the classic configure/make *nix way with the win32 one. BTW, scons makefiles are just python files! Think only about the *update the python build number* problem that arose some times ago! It's trivial if handled by a builder script with-python-steroids! --- Paolo Invernizzi From martin at v.loewis.de Mon Dec 29 16:55:38 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 16:56:00 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF08ECF.4010302@dmsware.com> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> Message-ID: <3FF0A2DA.4010205@v.loewis.de> Paolo Invernizzi wrote: > Can be a solution to support a single scons (makefile like) file and > from that generate the proper project files? That might be an option, but not the one I'm working on; it isn't even one that I'm particularly interested in. If I were maintaining the Windows port (which I'm not), I would dislike the solution because it would mean I have to learn scons... > Users that wants to keep working with the IDE can do that! ... and it would also mean that I *cannot* work with the IDE: I cannot make modifications to the projects in the IDE, because I would have to make the modifications in the scons files, not in the VC projects. > A free solution for a user that want to build python in a win32 > environment without Visual Studio (whatever edition) could be: > > - Download python (from 1.5.2 above) and scons (as a make replacement) > - Build python with the free Microsoft SDK compiler via scons. Such a free solution is already available, through cygwin/mingw. However, this is free software, so if you are interested, feel free to contribute. If you want your contribution be accepted, be prepared to answer "yes" to the question "Are you going to maintain the build process for the next three years"? If somebody volunteers to maintain such a build process, I would not object to having this process in parallel to the VC build process (to which I'm willing to contribute, for the next three years). Regards, Martin From paoloinvernizzi at dmsware.com Mon Dec 29 17:23:01 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Mon Dec 29 17:23:01 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF0A2DA.4010205@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> Message-ID: <3FF0A945.2020107@dmsware.com> Martin v. Loewis wrote: > If I were maintaining the > Windows port (which I'm not), I would dislike the solution because it > would mean I have to learn scons... It is only an alternative to learning a new tool for converting VC project files to nmake specific makefile. > I cannot > make modifications to the projects in the IDE, because I would have > to make the modifications in the scons files, not in the VC projects. That's true. You have to make the modifications in the scons file, but that modifications (usually adding or removing some c module, or I'm missing something?) can be used by VC/cygwin/mgwin/*nix user... potentially. > Such a free solution is already available, through cygwin/mingw. But that leave you with the original problem... makefiles for the free microsoft compiler... > However, this is free software, so if you are interested, feel free > to contribute. If you want your contribution be accepted, be prepared > to answer "yes" to the question "Are you going to maintain the build > process for the next three years"? > If somebody volunteers to maintain such a build process, I would not > object to having this process in parallel to the VC build process > (to which I'm willing to contribute, for the next three years). I'm not actually in the position to reply "yes" to that question, I have too few spare time... but you are tempting me... :-) It was just an alternative suggestion that come in my mind after having read previous threads. I've done some month ago a scratch scons script for VC6 and VC7 compilation of the core python dll, pythonw.exe and python.exe... I must take a look to my messy sandbox at works! --- Paolo Invernizzi From perky at i18n.org Mon Dec 29 19:34:08 2003 From: perky at i18n.org (Hye-Shik Chang) Date: Mon Dec 29 19:34:14 2003 Subject: [Python-Dev] Optimization versus code bloat In-Reply-To: <16368.13240.502549.132898@montanaro.dyndns.org> References: <20031229081724.GA42076@i18n.org> <16368.13240.502549.132898@montanaro.dyndns.org> Message-ID: <20031230003408.GA17100@i18n.org> On Mon, Dec 29, 2003 at 08:01:28AM -0600, Skip Montanaro wrote: > > Hye-Shik> I posted a patch[1] optimizing str.split with specialized > Hye-Shik> splitter as unicode.split do. The patch accellates str.split > Hye-Shik> about 10~20% and adds about 50 lines. I'm curious whether it's > Hye-Shik> acceptable ratio for python-dev moods. > > Hye-Shik> [1] http://www.python.org/sf/866875 > > Seems like a reasonable addition to me, especially if the same optimization > is performed for unicode objects. I'm a bit confused about all the changes > to test_string.py though. Since your change is only for performance it > seems to me that no changes to the test suite would have been necessary > unless it wasn't covering some critical tests. Was that just a > rearrangement of that file or did you add some new test cases? Nah. Because I added one more specialized splitter, I should add unittests for that subsequently. So we need to test each test cases for every specialized splitters (whitespace, string and a character). Thanks for your review! :-) Hye-Shik From martin at v.loewis.de Mon Dec 29 20:00:36 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Mon Dec 29 20:00:53 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF0A945.2020107@dmsware.com> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> <3FF0A945.2020107@dmsware.com> Message-ID: <3FF0CE34.4060409@v.loewis.de> Paolo Invernizzi wrote: >> If I were maintaining the >> Windows port (which I'm not), I would dislike the solution because it >> would mean I have to learn scons... > > > It is only an alternative to learning a new tool for converting VC > project files to nmake specific makefile. It's different. That tool would have to be written from scratch, which would be a large effort for a single person. Learning scons would be a relatively small effort for a relatively large group of people (not only immediate maintainers of the Windows port, but anybody contributing to it, also). > That's true. You have to make the modifications in the scons file, but > that modifications (usually adding or removing some c module, or I'm > missing something?) can be used by VC/cygwin/mgwin/*nix user... > potentially. The more typical modification is the addition of an extension module, which means adding a new target. That becomes more and more tricky these days, as these new targets often have prerequisites that are difficult to build themselves (e.g. finding an available copy of OpenSSL, invoking Perl to build OpenSSL, invoking nmake to build zlib, etc). I somewhat doubt that the scons-to-vcproj generator gets such complex build dependencies correct - at which point the maintainers would need to look into the inner workings of scons to find out if it really can represent the build step in all supported "output build procedures", and how to make scons do it correctly. > But that leave you with the original problem... makefiles for the free > microsoft compiler... That, of course, is not a problem for the majority of the Python-Windows contributors, who already have a copy of VC 7.1 (thanks to a generous offer earlier this year). So for whoever this is a problem, they would need to find a solution. They need to find a way today already (as Python does not provide nmake files out of the box at the moment, either). The solution does not need to materialize itself next week, as the release of Python 2.4 is still months away. > I've done some month ago a scratch scons script for VC6 and VC7 > compilation of the core python dll, pythonw.exe and python.exe... > > I must take a look to my messy sandbox at works! You could also maintain this "outside" of the standard Python distribution, as some kind of "drop-in" zip file. That would put no obligation onto you - your users could either use it, or leave it (and we could advertise it as an alternative somewhere). Regards, Martin From pf_moore at yahoo.co.uk Tue Dec 30 06:45:00 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 30 06:44:54 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 References: <3FF047E0.8070905@v.loewis.de> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> Message-ID: "Phillip J. Eby" writes: > At 04:27 PM 12/29/03 +0100, Martin v. Loewis wrote: >>In my own sandbox, I have prepared project files for VC.NET. >> >>Unless there are any strong objections, I'd like to switch >>over to VC.NET shortly, disabling VC6 builds. In the process, >>I will also update the build dependencies to more recent >>versions of several packages. > > Was the question of mingw32-built extensions ever resolved? That is, > will we be able to build extensions for the standard Windows Python > using the distutils' "mingw32" compiler, as is possible with Python > 2.2? It is (pretty close) to being solved - recent versions of mingw32 allow building with msvcr71 via a -lmsvcr71 flag. However, last time I tried this, the generated linker commands didn't quite work, and needed a bit of fiddling. I have the latest mingw, though, so I can easily try this. Paul -- This signature intentionally left blank From pf_moore at yahoo.co.uk Tue Dec 30 06:49:36 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 30 06:50:22 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: "Martin v. Loewis" writes: > OTOH, I cannot see what the problem might be - except that you will > have to link with msvcr71.dll at the end, not with msvcrt40.dll. There's an issue with mingw using malloc/free from msvcrt in its startup code - by the time msvcr71 gets linked in, the startup code has already resolved these two to msvcrt. I believe this is (nearly) resolved. Also, I've never seen a real problem caused by this, just dire hearsay about problems using incompatible runtimes... >> If this hasn't been resolved, and somebody can send me a binary for >> a .NET-built Python, I'll be happy to test. > > Sure: Please try my installer at > > http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi Does this install cleanly alongside a "production" Python 2.3? Ie, will it leave the meaning of the "python" command, and command line associations for .py files, etc, unchanged? As a test install, I'd like it to have no effect on my main Python (I have no test machine to install on separately). > Notice that this doesn't include msvcr71.dll, which I could provide > in private. Not a problem for me - I have this. > It also does not include msvcr71.lib - I have no clue how you would > get a copy of that. But then, somehow, msvcrt40.lib (or is it > msvcrt40.a) must have gotten into mingw32 also. The appropriate library is indeed supplied with mingw. Paul. -- This signature intentionally left blank From pf_moore at yahoo.co.uk Tue Dec 30 07:02:00 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 30 07:01:54 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> <5.1.1.6.0.20031229123908.023122e0@telecommunity.com> Message-ID: "Phillip J. Eby" writes: > The rest is handled by the distutils, pretty much. I don't think > there's actually any direct linking to the MS VC runtime, anyway, if > one is only using Python API calls. But I guess I'll find out. One thing that I imagine does need looking at is modifying distutils to check whether Python was built with VC7.1, and if so add a -lmsvcr71 flag to the gcc command line when compiling with mingw. This can be hacked by hand, but only at the expense of making setup.py version-specific (or doing your own test in setup.py). I don't know if Martin has already done this, but it needs doing. I'm not a distutils expert, but I am willing to look at it in the longer term. BTW, on another note, which 3rd party package developers have access to MSVC7.1, or can build with Mingw? Has anyone surveyed this? I, for one, have installed the following packages which have extension modules involved: cx_Oracle mxBase PIL pygame (not used much) win32all twisted wxPython pyXML Metakit If there wasn't a Windows binary version for 2.4 produced, this would cause me problems. At the very least, I'd suggest a warning post on c.l.p and c.l.p.announce, something to the effect of "Python 2.4 will be built with MSVC 7.1. Extension developers supplying binary distributions for Windows will need some way of building MSVC 7.1 compatible modules (MSVC 7.1 itself, or a recent version of the free Mingw compiler package) to continue providing binries." Paul. -- This signature intentionally left blank From skip at pobox.com Tue Dec 30 10:03:55 2003 From: skip at pobox.com (Skip Montanaro) Date: Tue Dec 30 10:04:05 2003 Subject: [Python-Dev] urllib2 doesn't grok URLs w/ user/passwd Message-ID: <16369.37851.229594.332869@montanaro.dyndns.org> SF seems to be down for some unscheduled reason. Posting here just so I don't completely forget about it should I exit my web browser before SF is back up... urllib2.urlopen("http://foo@www.python.org/") fails (at least in part) because it fails to separate the username and password from the hostname. Trying to open http://foo:bar@www.python.org/ reveals other shortcomings in its url parsing. It seems to me the syntactic bits shouldn't be difficult to resolve using urllib.spluituser(). I'm much less clear what to do with the username and password once they've been separated from the hostname. Skip From theller at python.net Tue Dec 30 11:10:09 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 30 10:10:55 2003 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/msi msi.py, 1.1.1.1, 1.2 msilib.py, 1.1.1.1, 1.2 In-Reply-To: (loewis@users.sourceforge.net's message of "Tue, 30 Dec 2003 07:02:45 -0800") References: Message-ID: loewis@users.sourceforge.net writes: > Update of /cvsroot/python/python/nondist/sandbox/msi > In directory sc8-pr-cvs1:/tmp/cvs-serv15532 > > Modified Files: > msi.py msilib.py > root.start_component("TARGETDIR", default_feature) > root.add_file("PCBuild/w9xpopen.exe") # XXX: separate component to only install on W9x Currently in the Wise script, w9xpopen.exe is also installed on NT/2k/XP, and I would like to leave it this way. py2exe needs it to create programs using popen that also run on w9x. Thomas From theller at python.net Tue Dec 30 11:25:17 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 30 10:26:01 2003 Subject: [Python-Dev] First version of Python MSI available In-Reply-To: <3FF04774.9020404@v.loewis.de> (Martin v. Loewis's message of "Mon, 29 Dec 2003 16:25:40 +0100") References: <3FF04774.9020404@v.loewis.de> Message-ID: "Martin v. Loewis" writes: > I have committed python/nondist/sandbox/msi to the CVS. > This is a MSI generator for Python on Windows, which I'd > like to propose as a replacement for Wise. A sample .msi > file is available at > > http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi > > This installer includes binaries compiled with MSVC 7.1 > (aka VC.NET 2003). > > The MSI file tries to copy most features and user interface > from the Wise installer, but adjusts to the feature set of MSI. > I'm aware of a few issues, but I'd like to request feedback > in any case. I still have MSVC 7.0 (not 7.1) installed, and trying to build an extension with distutils raises this error: running build_ext Traceback (most recent call last): File "setup.py", line 332, in ? options = options, File "c:\python24\lib\distutils\core.py", line 149, in setup dist.run_commands() File "c:\python24\lib\distutils\dist.py", line 907, in run_commands self.run_command(cmd) File "c:\python24\lib\distutils\dist.py", line 927, in run_command cmd_obj.run() File "c:\python24\lib\distutils\command\build.py", line 107, in run self.run_command(cmd_name) File "c:\python24\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "c:\python24\lib\distutils\dist.py", line 927, in run_command cmd_obj.run() File "c:\python24\lib\distutils\command\build_ext.py", line 243, in run force=self.force) File "c:\python24\lib\distutils\ccompiler.py", line 1177, in new_compiler return klass (None, dry_run, force) File "c:\python24\lib\distutils\msvccompiler.py", line 206, in __init__ self.__macros = MacroExpander(self.__version) File "c:\python24\lib\distutils\msvccompiler.py", line 112, in __init__ self.load_macros(version) File "c:\python24\lib\distutils\msvccompiler.py", line 128, in load_macros self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1") File "c:\python24\lib\distutils\msvccompiler.py", line 118, in set_macro self.macros["$(%s)" % macro] = d[key] KeyError: 'sdkinstallrootv1.1' > c:\python24\lib\distutils\msvccompiler.py(118)set_macro() -> self.macros["$(%s)" % macro] = d[key] Here is a dump of some variable: (Pdb) print d {'dbgmanageddebugger': 'C:\\Programme\\Gemeinsame Dateien\\Microsoft Shared\\VS7Debug\\vs7jit.exe PID %d APPDOM %d EXTEXT "%s" EVTHDL %d', 'dbgjitdebuglaunchsetting': 2, 'installroot': 'C:\\WINDOWS\\Microsoft.NET\\Framework\\', 'sdkinstallroot': 'C:\\Programme\\Microsoft Visual Studio .NET\\FrameworkSDK\\'} (Pdb) print key sdkinstallrootv1.1 (Pdb) print macro FrameworkSDKDir (Pdb) Thomas From theller at python.net Tue Dec 30 11:27:40 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 30 10:28:22 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: (Paul Moore's message of "Tue, 30 Dec 2003 11:49:36 +0000") References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: Paul Moore writes: > "Martin v. Loewis" writes: > >> Sure: Please try my installer at >> >> http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi > > Does this install cleanly alongside a "production" Python 2.3? Ie, > will it leave the meaning of the "python" command, and command line > associations for .py files, etc, unchanged? No, it changes the associations for .py files. It would be useful to have the option to create no associations. As a workaround, installing the MSI file, and afterwards simply reinstalling python-2.3.3.exe seems to work fine. Thomas From barry at python.org Tue Dec 30 10:41:25 2003 From: barry at python.org (Barry Warsaw) Date: Tue Dec 30 10:41:33 2003 Subject: [Python-Dev] Head broken when --enable-unicode=ucs4 is used Message-ID: <1072798884.9216.77.camel@anthem> I've been building Python with --enable-unicode=ucs4 ever since I upgraded to RedHat 9. IIUC, this option is required to get Python and RH9's default Tk to play nicely together. The Python cvs trunk (i.e. 2.4) is broken when using this option. The pain starts immediately: gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o Modules/python.o Modules/python.c In file included from Include/Python.h:82, from Modules/python.c:3: Include/unicodeobject.h:73:2: #error Must define Py_UNICODE_SIZE In file included from Include/Python.h:82, from Modules/python.c:3: Include/unicodeobject.h:131: parse error before "Py_UNICODE" Include/unicodeobject.h:131: warning: type defaults to `int' in declaration of `Py_UNICODE' Include/unicodeobject.h:131: warning: data definition has no type or storage class Include/unicodeobject.h:370: parse error before "Py_UNICODE" Include/unicodeobject.h:370: warning: no semicolon at end of struct or union Include/unicodeobject.h:375: parse error before '}' token Include/unicodeobject.h:375: warning: type defaults to `int' in declaration of `PyUnicodeObject' Include/unicodeobject.h:375: warning: data definition has no type or storage class [...continuing on for seemingly ever...] Without this switch, the compilation appears to work. I don't have time right now to investigate further, but if no one else can reproduce (or fix) this, I'll try to help out later. -Barry From aleaxit at yahoo.com Tue Dec 30 11:11:44 2003 From: aleaxit at yahoo.com (Alex Martelli) Date: Tue Dec 30 11:11:51 2003 Subject: [Python-Dev] urllib2 doesn't grok URLs w/ user/passwd In-Reply-To: <16369.37851.229594.332869@montanaro.dyndns.org> References: <16369.37851.229594.332869@montanaro.dyndns.org> Message-ID: <200312301711.44687.aleaxit@yahoo.com> On Tuesday 30 December 2003 04:03 pm, Skip Montanaro wrote: > SF seems to be down for some unscheduled reason. Posting here just so I > don't completely forget about it should I exit my web browser before SF is > back up... > > urllib2.urlopen("http://foo@www.python.org/") fails (at least in part) > because it fails to separate the username and password from the hostname. > Trying to open http://foo:bar@www.python.org/ reveals other shortcomings in > its url parsing. It seems to me the syntactic bits shouldn't be difficult > to resolve using urllib.spluituser(). I'm much less clear what to do with > the username and password once they've been separated from the hostname. Presumably they need to be kept somewhere and sent in the Authorization header in case the server returns a 401 error and challenge (or a proxy returns a 407 error and challenge) -- or maybe the Authorization header (with the base 64 encoding of user:pass) can be sent even as part of the first request to speed things up (assuming an authorization scheme of Basic). RFC 2617, I believe. urllib2's architecture delegates authorization to separate components, of course, so I guess the userid and password should just be handed over to such components if they're present, but I haven't looked into that in detail. Alex From kdart at kdart.com Tue Dec 30 13:01:42 2003 From: kdart at kdart.com (Keith Dart) Date: Tue Dec 30 13:01:46 2003 Subject: [Python-Dev] feedback on Enum class Message-ID: <1072807302.11336.37.camel@leviathan.kdart.com> Greetings everyone. I hope you all had a great Christmas holiday. 8-) I am posting this Enum (named number) class to this list for possible feedback. It can be used as a regular integer but when printed (stringified) it yeilds its name. The hash also makes named numbers with the same integer value into unique dictionary keys. That is the part I am not sure about... Comments appreciated. class Enum(int): __slots__ = ("_name") def __new__(cls, val, name): v = int.__new__(cls, val) v._name = str(name) return v def __str__(self): return self._name def __repr__(self): return "%s(%d, %r)" % (self.__class__.__name__, self,self._name) def __hash__(self): return int.__hash__(self) + hash(self._name) -- -- ------------------------------------------------------------------------- Keith Dart ---------------------------------------------------------------------------- Public key ID: B08B9D2C Public key: ============================================================================ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20031230/df76d679/attachment.bin From tismer at stackless.com Tue Dec 30 13:38:29 2003 From: tismer at stackless.com (Christian Tismer) Date: Tue Dec 30 13:38:35 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF047E0.8070905@v.loewis.de> References: <3FF047E0.8070905@v.loewis.de> Message-ID: <3FF1C625.7070103@stackless.com> Martin v. Loewis wrote: > In my own sandbox, I have prepared project files for VC.NET. > > Unless there are any strong objections, I'd like to switch > over to VC.NET shortly, disabling VC6 builds. In the process, > I will also update the build dependencies to more recent > versions of several packages. Ach du Sch***e. I don't want to buy another M$ compiler. Is there a freeware alternative? ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From paoloinvernizzi at dmsware.com Tue Dec 30 13:41:34 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Tue Dec 30 13:41:36 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF0CE34.4060409@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> <3FF0A945.2020107@dmsware.com> <3FF0CE34.4060409@v.loewis.de> Message-ID: Martin v. Loewis wrote: > The more typical modification is the addition of an extension module, > which means adding a new target. That becomes more and more tricky > these days, as these new targets often have prerequisites that are > difficult to build themselves (e.g. finding an available copy of > OpenSSL, invoking Perl to build OpenSSL, invoking nmake to build > zlib, etc). Ok. For what I have understood reading the PCbuild/readme.txt (I admit I never tried to build on win32 an *hard* module, like tk or bdb, or ssl), the building process of that modules is still mostly a manual job... "follow the Windows instructions for building the Sleepycat software" > I somewhat doubt that the scons-to-vcproj generator gets such > complex build dependencies correct - at which point the maintainers > would need to look into the inner workings of scons to find out if > it really can represent the build step in all supported "output > build procedures", and how to make scons do it correctly. The VC project generated by scons simply binds the IDE build command to scons itself, so when the IDE need to build something, it actually launch scons, which can handle a lot of mutual dependend target easily. Well, summarizing all, I can try some *non trivial* build process, like the python exes plus core dll, plus out-of-the-box modules, plus OpenSSL or bdb, and see what can be done with it... I'll post something as soon as I've done some tries. --- Paolo Invernizzi From pf_moore at yahoo.co.uk Tue Dec 30 13:56:24 2003 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Dec 30 13:56:13 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: <3cb2xhgn.fsf@yahoo.co.uk> Thomas Heller writes: > No, it changes the associations for .py files. It would be useful to > have the option to create no associations. Agreed, that would be useful. (More generally, if there's anything else needed to install a second Python version which is never used by default, but only used when explicitly requested, that would be useful too.) Hmm, for example, does the installer set the registry key HKLM\Software\Microsoft\Windows\CurrentVersion\App Paths\python.exe That's what makes the command "python" work at the console prompt. Maybe just a single option in the installer, "Make this the default version of Python", which is true by default, would be best. This could control file associations, App Paths, and anything else related. > As a workaround, installing the MSI file, and afterwards simply > reinstalling python-2.3.3.exe seems to work fine. Yup, that's a good workaround. Paul. -- This signature intentionally left blank From guido at python.org Tue Dec 30 14:28:52 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 30 14:26:16 2003 Subject: [Python-Dev] urllib2 doesn't grok URLs w/ user/passwd In-Reply-To: Your message of "Tue, 30 Dec 2003 17:11:44 +0100." <200312301711.44687.aleaxit@yahoo.com> References: <16369.37851.229594.332869@montanaro.dyndns.org> <200312301711.44687.aleaxit@yahoo.com> Message-ID: <200312301928.hBUJSqu01449@c-24-5-183-134.client.comcast.net> > On Tuesday 30 December 2003 04:03 pm, Skip Montanaro wrote: > > SF seems to be down for some unscheduled reason. Posting here just so I > > don't completely forget about it should I exit my web browser before SF is > > back up... > > > > urllib2.urlopen("http://foo@www.python.org/") fails (at least in part) > > because it fails to separate the username and password from the hostname. > > Trying to open http://foo:bar@www.python.org/ reveals other shortcomings in > > its url parsing. It seems to me the syntactic bits shouldn't be difficult > > to resolve using urllib.spluituser(). I'm much less clear what to do with > > the username and password once they've been separated from the hostname. > > Presumably they need to be kept somewhere and sent in the Authorization > header in case the server returns a 401 error and challenge (or a proxy > returns a 407 error and challenge) -- or maybe the Authorization header > (with the base 64 encoding of user:pass) can be sent even as part of the > first request to speed things up (assuming an authorization scheme of > Basic). This is what the ever-popular old urllib does. > RFC 2617, I believe. urllib2's architecture delegates authorization > to separate components, of course, so I guess the userid and password > should just be handed over to such components if they're present, but I > haven't looked into that in detail. Me neither. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Dec 30 14:34:25 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 30 14:31:50 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: Your message of "Tue, 30 Dec 2003 19:38:29 +0100." <3FF1C625.7070103@stackless.com> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> Message-ID: <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> > I don't want to buy another M$ compiler. > Is there a freeware alternative? Ten core Python developers received a free VC7.1 copy from Microsoft. Maybe someone who hasn't opened theirs yet is willing to step down? MS also alluded to a free downloadable compiler, but I haven't tracked it down yet. And of course theer's always mingw (or whatever it's called). --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Tue Dec 30 14:59:55 2003 From: aahz at pythoncraft.com (Aahz) Date: Tue Dec 30 14:59:59 2003 Subject: [Python-Dev] feedback on Enum class In-Reply-To: <1072807302.11336.37.camel@leviathan.kdart.com> References: <1072807302.11336.37.camel@leviathan.kdart.com> Message-ID: <20031230195954.GA26455@panix.com> On Tue, Dec 30, 2003, Keith Dart wrote: > > I am posting this Enum (named number) class to this list for possible > feedback. It can be used as a regular integer but when printed > (stringified) it yeilds its name. The hash also makes named numbers with > the same integer value into unique dictionary keys. That is the part I > am not sure about... Comments appreciated. This should probably go to comp.lang.python. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. From theller at python.net Tue Dec 30 16:17:19 2003 From: theller at python.net (Thomas Heller) Date: Tue Dec 30 15:17:17 2003 Subject: [Python-Dev] First version of Python MSI available In-Reply-To: <3FF04774.9020404@v.loewis.de> (Martin v. Loewis's message of "Mon, 29 Dec 2003 16:25:40 +0100") References: <3FF04774.9020404@v.loewis.de> Message-ID: "Martin v. Loewis" writes: > I have committed python/nondist/sandbox/msi to the CVS. > This is a MSI generator for Python on Windows, which I'd > like to propose as a replacement for Wise. A sample .msi > file is available at > > http://www.dcl.hpi.uni-potsdam.de/home/loewis/python2.4.0.msi > > This installer includes binaries compiled with MSVC 7.1 > (aka VC.NET 2003). > > The MSI file tries to copy most features and user interface > from the Wise installer, but adjusts to the feature set of MSI. > I'm aware of a few issues, but I'd like to request feedback > in any case. The installer does not install pyconfig.h into Python24\include. After copying this file manually (and installing VC7.1 - puh) I'm now able to build extensions with distutils. Also distutils\command\wininst.exe is not installed. Thomas From kbk at shore.net Tue Dec 30 15:17:29 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue Dec 30 15:17:36 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Tue, 30 Dec 2003 11:34:25 -0800") References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> Message-ID: <87znda12na.fsf@hydra.localdomain> Guido van Rossum writes: > Ten core Python developers received a free VC7.1 copy from Microsoft. > Maybe someone who hasn't opened theirs yet is willing to step down? Maybe there are a few more available? -- KBK From guido at python.org Tue Dec 30 15:28:23 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 30 15:25:49 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: Your message of "Tue, 30 Dec 2003 15:17:29 EST." <87znda12na.fsf@hydra.localdomain> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> <87znda12na.fsf@hydra.localdomain> Message-ID: <200312302028.hBUKSNN01582@c-24-5-183-134.client.comcast.net> > > Ten core Python developers received a free VC7.1 copy from Microsoft. > > Maybe someone who hasn't opened theirs yet is willing to step down? > > Maybe there are a few more available? I'm not going to ask. --Guido van Rossum (home page: http://www.python.org/~guido/) From jjl at pobox.com Tue Dec 30 16:55:45 2003 From: jjl at pobox.com (John J Lee) Date: Tue Dec 30 16:56:05 2003 Subject: [Python-Dev] urllib2 doesn't grok URLs w/ user/passwd In-Reply-To: <200312301711.44687.aleaxit@yahoo.com> References: <16369.37851.229594.332869@montanaro.dyndns.org> <200312301711.44687.aleaxit@yahoo.com> Message-ID: On Tue, 30 Dec 2003, Alex Martelli wrote: > On Tuesday 30 December 2003 04:03 pm, Skip Montanaro wrote: > > SF seems to be down for some unscheduled reason. Posting here just so I > > don't completely forget about it should I exit my web browser before SF is > > back up... > > > > urllib2.urlopen("http://foo@www.python.org/") fails (at least in part) > > because it fails to separate the username and password from the hostname. > > Trying to open http://foo:bar@www.python.org/ reveals other shortcomings in [...] > Presumably they need to be kept somewhere and sent in the Authorization > header in case the server returns a 401 error and challenge (or a proxy > returns a 407 error and challenge) -- or maybe the Authorization header [...] urllib2 already knows about this syntax for proxy auth (ProxyHandler.proxy_open). If somebody fixes this (assuming it needs fixing -- I haven't read what the standards say), note that the Basic authentication logic is duplicated, and perhaps broken in one place, which should probably be fixed at the same time: http://www.google.com/groups?threadm=87d6dq2jx0.fsf%40pobox.com If nobody does it first, I'll fix it *eventually* (I'm lazy about configuring a local proxy and web server...). BTW, is somebody planning a 2.3.4 (or 2.2.4)? When? That would motivate me to fix urllib2 bugs sooner. John From martin at v.loewis.de Tue Dec 30 17:48:08 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 17:48:52 2003 Subject: [Python-Dev] First version of Python MSI available In-Reply-To: References: <3FF04774.9020404@v.loewis.de> Message-ID: <3FF200A8.5090604@v.loewis.de> Thomas Heller wrote: > The installer does not install pyconfig.h into Python24\include. [...] > Also distutils\command\wininst.exe is not installed. Thanks, will fix. Regards, Martin From martin at v.loewis.de Tue Dec 30 17:52:07 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 17:52:26 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF1C625.7070103@stackless.com> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> Message-ID: <3FF20197.1090802@v.loewis.de> Christian Tismer wrote: > Ach du Sch***e. > I don't want to buy another M$ compiler. > Is there a freeware alternative? Depends on what you want to do. If all you want to do is build extensions, this should be possible with mingw. Actually, with some work, it might be possible to build all of Python with mingw. As for freeware alternatives: Linux is a good alternative to Windows, too :-) Regards, Martin From martin at v.loewis.de Tue Dec 30 18:05:57 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 18:06:11 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> Message-ID: <3FF204D5.4010501@v.loewis.de> Paul Moore wrote: > Does this install cleanly alongside a "production" Python 2.3? Ie, > will it leave the meaning of the "python" command, and command line > associations for .py files, etc, unchanged? No. It will overwrite the associations, and, upon uninstallation, it will entirely remove them. OTOH, it does install into a separate folder, and it creates shortcuts in a separate start menu folder. > As a test install, I'd like it to have no effect on my main Python (I > have no test machine to install on separately). You could generate such an installer yourself, setting a variable in msi.py. However, I can make installation of the shortcuts optional, in future builds of the installer (to be enabled through a public MSI property). Regards, Martin From martin at v.loewis.de Tue Dec 30 18:08:57 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 18:10:00 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3cb2xhgn.fsf@yahoo.co.uk> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> <3cb2xhgn.fsf@yahoo.co.uk> Message-ID: <3FF20589.2030702@v.loewis.de> Paul Moore wrote: > Hmm, for example, does the installer set the registry key > HKLM\Software\Microsoft\Windows\CurrentVersion\App Paths\python.exe Yes, it does. > Maybe just a single option in the installer, "Make this the default > version of Python", which is true by default, would be best. This > could control file associations, App Paths, and anything else related. Authoring new user interface interface is a tedious task. It is much easier to just have a public property, so you would have to install it as msiexec.exe python2.4.0.msi DEFAULTPYTHON=1 Regards, Martin From martin at v.loewis.de Tue Dec 30 18:13:36 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 18:13:54 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> <5.1.1.6.0.20031229123908.023122e0@telecommunity.com> Message-ID: <3FF206A0.3070002@v.loewis.de> Paul Moore wrote: > I don't know if Martin has already done this, but it needs doing. I'm > not a distutils expert, but I am willing to look at it in the longer > term. I haven't changed any Python file at all in the process of compiling with VC 7.1. However, Python (since 2.3) indicates the compiler used to build it in sys.version; I believe this could be used as an indication to find out whether it was build with VC6 or VC7.1 (dunno whether it could also tell apart 7.0 and 7.1). > If there wasn't a Windows binary version for 2.4 produced, this would > cause me problems. Python 2.4 is still months ahead, I expect more copies of VC 7.1 being available by that time (unless Microsoft releases the next compiler version before that - but they are unlikely to do so until the very end of 2004). > At the very least, I'd suggest a warning post on c.l.p and > c.l.p.announce, something to the effect of "Python 2.4 will be built > with MSVC 7.1. Extension developers supplying binary distributions for > Windows will need some way of building MSVC 7.1 compatible modules > (MSVC 7.1 itself, or a recent version of the free Mingw compiler > package) to continue providing binries." I expect the word will spread quickly; again, there is plenty of time to prepare for that. Regards, Martin From martin at v.loewis.de Tue Dec 30 18:17:01 2003 From: martin at v.loewis.de (Martin v. Loewis) Date: Tue Dec 30 18:17:12 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> <3FF0A945.2020107@dmsware.com> <3FF0CE34.4060409@v.loewis.de> Message-ID: <3FF2076D.7060806@v.loewis.de> Paolo Invernizzi wrote: > Ok. For what I have understood reading the PCbuild/readme.txt (I admit I > never tried to build on win32 an *hard* module, like tk or bdb, or ssl), > the building process of that modules is still mostly a manual job... > "follow the Windows instructions for building the Sleepycat software" It depends: yes for tk and sleepycat, no for ssl and bzip2. > I'll post something as soon as I've done some tries. Please keep us posted. It would be good if it involved no more manual intervention than the current process; it would be a plus if the manual intervention would be reduced. It would also be a plus if you could figure out some of the pending issues while you are at it, like how to stop the linker warnings when linking _bsddb.pyd :-) Regards, Martin From paoloinvernizzi at dmsware.com Tue Dec 30 18:47:52 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Tue Dec 30 18:47:53 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF206A0.3070002@v.loewis.de> References: <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <5.1.1.6.0.20031229113056.02e151a0@telecommunity.com> <3FF059A1.8090202@v.loewis.de> <5.1.1.6.0.20031229123908.023122e0@telecommunity.com> <3FF206A0.3070002@v.loewis.de> Message-ID: Martin v. Loewis wrote: > I believe this could be used as an > indication to find out whether it was build with VC6 or VC7.1 (dunno > whether it could also tell apart 7.0 and 7.1). For a version built with VC7.1 the sys.version is: 2.4a0 (#0, Dec 15 2003, 17:22:56) [MSC v.1310 32 bit (Intel)] If I remember well, for VC7.0 it's [MSC v.1300 bit (Intel)] --- Paolo Invernizzi From paoloinvernizzi at dmsware.com Tue Dec 30 18:54:59 2003 From: paoloinvernizzi at dmsware.com (Paolo Invernizzi) Date: Tue Dec 30 18:55:00 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 In-Reply-To: <3FF2076D.7060806@v.loewis.de> References: <3FF0510F.3070504@v.loewis.de> <200312291753.09793.aleaxit@yahoo.com> <3FF05E58.1070108@v.loewis.de> <3FF08ECF.4010302@dmsware.com> <3FF0A2DA.4010205@v.loewis.de> <3FF0A945.2020107@dmsware.com> <3FF0CE34.4060409@v.loewis.de> <3FF2076D.7060806@v.loewis.de> Message-ID: Martin v. Loewis wrote: > Please keep us posted. It would be good if it involved no more manual > intervention than the current process; it would be a plus if the manual > intervention would be reduced. Ok. > It would also be a plus if you could figure out some of the pending > issues while you are at it, like how to stop the linker warnings when > linking _bsddb.pyd :-) Ok ;-) -- Paolo Invernizzi From trentm at ActiveState.com Tue Dec 30 19:27:32 2003 From: trentm at ActiveState.com (Trent Mick) Date: Tue Dec 30 19:33:07 2003 Subject: [Python-Dev] "What's New" in docs In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04BF@au3010avexu1.global.avaya.com>; from tdelaney@avaya.com on Fri, Dec 12, 2003 at 10:30:39AM +1100 References: <338366A6D2E2CA4C9DAEAE652E12A1DEFE04BF@au3010avexu1.global.avaya.com> Message-ID: <20031230162732.B12400@ActiveState.com> [Delaney, Timothy C (Timothy) wrote] > What's New in... > Python 2.3 > 1 PEP 218: A Standard Set Datatype > Python 2.2 > 1 PEP 234: Iterators (Behind the times here I know but:) ActivePython's chm does this. The TOC has: What's New Release Notes What's New in Python 2.3? What's New in Python 2.2? What's New in Python 2.1? The NEWS file Trent -- Trent Mick TrentM@ActiveState.com From kbk at shore.net Tue Dec 30 20:31:22 2003 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue Dec 30 20:31:27 2003 Subject: [Python-Dev] OpenBSD anyone? In-Reply-To: <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> (Guido van Rossum's message of "Wed, 24 Dec 2003 10:12:39 -0800") References: <200312231620.hBNGK8Q20710@c-24-5-183-134.client.comcast.net> <6.0.1.1.2.20031224173051.022dfec8@torment.chelsea.private> <200312241747.hBOHlMZ22008@c-24-5-183-134.client.comcast.net> <200312241812.hBOICdK22045@c-24-5-183-134.client.comcast.net> Message-ID: <87brppdb85.fsf@hydra.localdomain> Guido van Rossum writes: > Would someone here be so kind as to report the bug we've found? Bug submitted with cut-down: http://cvs.openbsd.org/cgi-bin/query-pr-wrapper?full=yes&numbers=3622 It has been verified that this is fixed at OpenBSD 3.4. Therefore it's only a problem with 3.3 because there was no stack protector prior to that. Workaround when using -g option is to add -fno-stack-protector or -O1. I don't think any change to Python is warranted. -- KBK From raymond.hettinger at verizon.net Tue Dec 30 20:40:12 2003 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue Dec 30 20:40:41 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation Message-ID: <003901c3cf3f$08c19720$e841fea9@oemcomputer> Dmitry Vasiliev has submitted a C implementation for the bisect module. My thoughts are to accept it with the following changes: * leave the original code intact for teaching purposes * import the C version from _bisect * make sure the C code is not list specific and will work with any container supporting __len__(), __getitem__(), and insert(). Do you guys have any objections? Raymond Hettinger From guido at python.org Tue Dec 30 20:47:31 2003 From: guido at python.org (Guido van Rossum) Date: Tue Dec 30 20:44:57 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: Your message of "Tue, 30 Dec 2003 20:40:12 EST." <003901c3cf3f$08c19720$e841fea9@oemcomputer> References: <003901c3cf3f$08c19720$e841fea9@oemcomputer> Message-ID: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> > Dmitry Vasiliev has submitted a C implementation for the bisect module. > > My thoughts are to accept it with the following changes: > > * leave the original code intact for teaching purposes > * import the C version from _bisect > * make sure the C code is not list specific and will work with any > container supporting __len__(), __getitem__(), and insert(). > > Do you guys have any objections? Only that I wished you had done the same for heapq.py. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Tue Dec 30 22:23:34 2003 From: tim.one at comcast.net (Tim Peters) Date: Tue Dec 30 22:23:40 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> Message-ID: [Raymond] >> Dmitry Vasiliev has submitted a C implementation for the bisect >> module. >> >> My thoughts are to accept it with the following changes: >> >> * leave the original code intact for teaching purposes +1 >> * import the C version from _bisect +1 >> * make sure the C code is not list specific and will work with any >> container supporting __len__(), __getitem__(), and insert(). -0.6. "Gratuitous hyper-generality", IMO -- I'm willing to bet nobody has ever written a production class, with that collection of methods, that would also be suitable for using with bisect. "list subtype" is good enough, and will leave the code clearer, quicker, and more maintainable. >> Do you guys have any objections? Six tenths of one, as above . [Guido] > Only that I wished you had done the same for heapq.py. It's not too late. IIRC, neither you nor I could make time to review the heapq plans before the C code got checked in. I was able to make a little time after, and mourned the loss of the educational value of the Python version (to Raymond, in pvt). That's a big enough point that it deserves public airing, though: everyone (except Guido, I guess) please remember what it was like when you first learned Python! Mounds and mounds of library code written in Python, doing interesting things with interesting techniques, and written at a high enough level that you weren't wholly buried under details. Python's C code is generally clean and educational too, but has a *much* smaller potential audience than our pure-Python modules. Nobody can sanely accuse me of not caring about speed (see a decade of speed-obsessed Python checkins for counterexamples ), but I cry a little each time a piece of the system gets recoded in C. Keeping the original Python code around is a nice compromise (pioneered, IIRC, by string.py, a loooong time ago). From python at rcn.com Tue Dec 30 23:27:03 2003 From: python at rcn.com (Raymond Hettinger) Date: Tue Dec 30 23:28:13 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: Message-ID: <005801c3cf56$703a1e60$e841fea9@oemcomputer> [Raymond] > Dmitry Vasiliev has submitted a C implementation for the bisect > module. > > My thoughts are to accept it with the following changes: > > * leave the original code intact for teaching purposes > * import the C version from _bisect > * make sure the C code is not list specific and will work with any > container supporting __len__(), __getitem__(), and insert(). > > Do you guys have any objections? > [Guido] > > Only that I wished you had done the same for heapq.py. [Tim] > It's not too late. IIRC, neither you nor I could make time to review the > heapq plans before the C code got checked in. I was able to make a little > time after, and mourned the loss of the educational value of the Python > version (to Raymond, in pvt). I would be happy to bring back the python version and make it coexist with the C version. IMO, it was a programming pearl unto itself and efforts were already made to preserve the extensive module docstring and make the code match the original line for line with the variable names and logic intact. The bisect module warrants even more consideration because it is more basic and more venerable. [Tim] > Nobody can sanely accuse me of not > caring about speed (see a decade of speed-obsessed Python checkins for > counterexamples ), but I cry a little each time a piece of the > system > gets recoded in C. Keeping the original Python code around is a nice > compromise (pioneered, IIRC, by string.py, a loooong time ago). Also, with the PyPy project, there is a chance that cleanly coded pure python modules will survive the test of time longer than their C counterparts. In the itertools module, I documented pure python equivalents to make the documentation more specific, to enchance understanding, and to make it easier to backport. For the most part, that effort was successful. The only downside is the double maintenance problem of keeping the two in sync. Given the stability of bisect, that is not much of an issue. Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From skip at pobox.com Wed Dec 31 00:08:04 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 31 00:08:13 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: References: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> Message-ID: <16370.22964.429991.454763@montanaro.dyndns.org> Tim> Nobody can sanely accuse me of not caring about speed (see a decade Tim> of speed-obsessed Python checkins for counterexamples ), but Tim> I cry a little each time a piece of the system gets recoded in C. Tim> Keeping the original Python code around is a nice compromise Tim> (pioneered, IIRC, by string.py, a loooong time ago). It would be nice to keep the old Python implementations around for two reasons. First, there's the obvious educational value. Second, the PyPy folks will know where to find it when they need it. I think a Demo/Lib subdirectory (with appropriate README) would be a reasonable place to put such code out to pasture. It could be made available in source distributions but never installed. Skip From tjreedy at udel.edu Wed Dec 31 00:49:59 2003 From: tjreedy at udel.edu (Terry Reedy) Date: Wed Dec 31 00:50:00 2003 Subject: [Python-Dev] Re: urllib2 doesn't grok URLs w/ user/passwd References: <16369.37851.229594.332869@montanaro.dyndns.org><200312301711.44687.aleaxit@yahoo.com> Message-ID: "John J Lee" wrote in message news:Pine.LNX.4.58.0312302129080.472@alice... > http://www.google.com/groups?threadm=87d6dq2jx0.fsf%40pobox.com > > If nobody does it first, I'll fix it *eventually* (I'm lazy about > configuring a local proxy and web server...). > > BTW, is somebody planning a 2.3.4 #yes >(or 2.2.4)? #no > When? #April-May > That would motivate me to fix urllib2 bugs sooner. I suspect more backported bug fixes will motivate a release. tjr From anthony at interlink.com.au Wed Dec 31 02:49:35 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Dec 31 02:50:08 2003 Subject: [Python-Dev] Re: urllib2 doesn't grok URLs w/ user/passwd In-Reply-To: Message-ID: <200312310749.hBV7naqk011830@localhost.localdomain> >>> "Terry Reedy" wrote > > BTW, is somebody planning a 2.3.4 #yes > >(or 2.2.4)? #no > > When? #April-May > > That would motivate me to fix urllib2 bugs sooner. > > I suspect more backported bug fixes will motivate a release. I plan a 2.3.4 in the April-May-June timeframe. This will _probably_ be the final release of 2.3 before 2.4, although we could have a 2.3.5 if there's a need for it at the end of the year (shortly after 2.4 is done would be my pick). I don't know that we'd have a need for many more 2.3 releases after that - particularly since, at the moment at least, 2.4 seems to be continuing down 2.3's conservative path as far as potentially dangerous changes. Note that, aside from major bug fixes, I'm not particularly driven by volume of bug fixes for selecting a release schedule - mostly it's driven by my own personal estimation of when I'm likely to have time available to do the release. Each release I do seems to take less time (yay for automation! boo that the major part of the automation seems to have gone missing somehow in the laptop theft/recovery of data from tape.... ah well). Anthony -- Anthony Baxter It's never too late to have a happy childhood. From 4q89v20mku at msn.com Wed Dec 31 16:33:44 2003 From: 4q89v20mku at msn.com (Cara Curtis) Date: Wed Dec 31 08:39:02 2003 Subject: [Python-Dev] Refill your medication online! yyjw l Message-ID: Rated Top Rated Online Store. HOT & NEW - Levitra/Lipitor/Nexium Weekly speciasls on all our drugs. -Zocor -Soma -Ambien -Phentermine -V1agra -Discount Generic's on ALL -MORE Next day discrete shipping on all products! http://www.rxstoreusa.biz/shopping Please, I wish to receive no more discounts on valuable items. http://www.rxstoreusa.biz/a.html jet djjdnjqj nldxi From barry at python.org Wed Dec 31 09:07:24 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 31 09:07:30 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: References: Message-ID: <1072879643.28895.241.camel@anthem> On Tue, 2003-12-30 at 22:23, Tim Peters wrote: > That's a big enough point that it deserves public airing, though: everyone > (except Guido, I guess) please remember what it was like when you first > learned Python! Mounds and mounds of library code written in Python, doing > interesting things with interesting techniques, and written at a high enough > level that you weren't wholly buried under details. Python's C code is > generally clean and educational too, but has a *much* smaller potential > audience than our pure-Python modules. Nobody can sanely accuse me of not > caring about speed (see a decade of speed-obsessed Python checkins for > counterexamples ), but I cry a little each time a piece of the system > gets recoded in C. Keeping the original Python code around is a nice > compromise (pioneered, IIRC, by string.py, a loooong time ago). Yes, yes, yes. I'd be bummed if modules like pickle.py got lost or out of sync with its C cousin. -Barry From barry at python.org Wed Dec 31 09:10:23 2003 From: barry at python.org (Barry Warsaw) Date: Wed Dec 31 09:10:32 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: <16370.22964.429991.454763@montanaro.dyndns.org> References: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> <16370.22964.429991.454763@montanaro.dyndns.org> Message-ID: <1072879822.28895.245.camel@anthem> On Wed, 2003-12-31 at 00:08, Skip Montanaro wrote: > It would be nice to keep the old Python implementations around for two > reasons. First, there's the obvious educational value. Second, the PyPy > folks will know where to find it when they need it. I think a Demo/Lib > subdirectory (with appropriate README) would be a reasonable place to put > such code out to pasture. It could be made available in source > distributions but never installed. The problem is that relegating stuff to Demo would very likely lead to bitrot, which is not a good thing. Another advantage of having current, working (and unit tested) pure-Python alternatives is that they are much easier to modify if you're experimenting with new features, or bug fixes, etc. OTOH, some of the really ancient Python modules will occasionally need attention to bring them up to modern coding standards, idioms, and practices. -Barry From tismer at stackless.com Wed Dec 31 09:44:10 2003 From: tismer at stackless.com (Christian Tismer) Date: Wed Dec 31 09:44:16 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <3FF20197.1090802@v.loewis.de> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> <3FF20197.1090802@v.loewis.de> Message-ID: <3FF2E0BA.9090105@stackless.com> Martin v. Loewis wrote: [CT doesn't want to spend money on a new compiler] > Depends on what you want to do. If all you want > to do is build extensions, this should be possible > with mingw. Actually, with some work, it might be > possible to build all of Python with mingw. > > As for freeware alternatives: Linux is a good > alternative to Windows, too :-) The problem is: On Windows, and with VC++6.0, I really can fly, while on Linux I'm so-so. It would be a shame if I had to pass Stackless Windows support over to someone else, just because we changed compilers ;-) best wishes for the new year -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Wed Dec 31 09:45:28 2003 From: tismer at stackless.com (Christian Tismer) Date: Wed Dec 31 09:45:33 2003 Subject: [Python-Dev] Switching to VC.NET 2003 In-Reply-To: <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> References: <3FF047E0.8070905@v.loewis.de> <3FF1C625.7070103@stackless.com> <200312301934.hBUJYPL01492@c-24-5-183-134.client.comcast.net> Message-ID: <3FF2E108.9060101@stackless.com> Guido van Rossum wrote: >>I don't want to buy another M$ compiler. >>Is there a freeware alternative? > > > Ten core Python developers received a free VC7.1 copy from Microsoft. > Maybe someone who hasn't opened theirs yet is willing to step down? > > MS also alluded to a free downloadable compiler, but I haven't tracked > it down yet. And of course theer's always mingw (or whatever it's > called). If there is a way I can provide Windows binaries, and I can do some interactive debugging, I'm willing to switch tools, sure. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From skip at pobox.com Wed Dec 31 09:56:03 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 31 09:56:14 2003 Subject: [Python-Dev] SF patch 864863: Bisect C implementation In-Reply-To: <1072879822.28895.245.camel@anthem> References: <200312310147.hBV1lV602218@c-24-5-183-134.client.comcast.net> <16370.22964.429991.454763@montanaro.dyndns.org> <1072879822.28895.245.camel@anthem> Message-ID: <16370.58243.130247.574887@montanaro.dyndns.org> Barry> The problem is that relegating stuff to Demo would very likely Barry> lead to bitrot, which is not a good thing. True. Note, however, that since these files would be in a source distribution there's no reason they couldn't be factored into "make test" somehow. Barry> Another advantage of having current, working (and unit tested) Barry> pure-Python alternatives is that they are much easier to modify Barry> if you're experimenting with new features, or bug fixes, etc. Agreed. Skip From mcherm at mcherm.com Wed Dec 31 10:12:34 2003 From: mcherm at mcherm.com (Michael Chermside) Date: Wed Dec 31 10:12:38 2003 Subject: [Python-Dev] Re: Switching to VC.NET 2003 Message-ID: <1072883554.3ff2e762a975e@mcherm.com> [Paul Moore:] > Maybe just a single option in the installer, "Make this the default > version of Python", which is true by default, would be best. This > could control file associations, App Paths, and anything else related. [Martin v Loewis:] > Authoring new user interface interface is a tedious task. > It is much easier to just have a public property, so you would > have to install it as > > msiexec.exe python2.4.0.msi DEFAULTPYTHON=1 Yes, but it's a tedious task undertaken by a single individual who knows something about how msi files work. Whereas a checkbox in the installer is easily understood by any user, including the vast majority who wouldn't know what "msi" meant, much less how to run execute one by hand with public properties specified. Not that I'm going to complain about how you're handling this... after all, you're the one volunteering your time. But if you're interested in hearing random opinions from others, I am one who would find it awefully nice if the MS installer had an option to install a non-default version of Python. Then someday it would be much easier for me to keep around copies of Python 2.4, 2.4.1, 2.4.2, 2.5, 2.5.1, and 2.6 all on the same machine. If-the-PSF-were-paying-your-salary-THEN-I'd-complain -- Michael Chermside From barry at python.org Wed Dec 31 11:36:20 2003 From: barry at python.org (Barry A. Warsaw) Date: Wed Dec 31 11:36:25 2003 Subject: [Python-Dev] base64.py support for RFC 3548 Message-ID: <16370.64260.941889.163153@gargle.gargle.HOWL> Hi all, I found I needed more complete RFC 3548 support for a project I'm working on. This RFC contains the most up-to-date specification of Base64, Base32, and Base16, including a description of alternative alphabets, etc. I didn't want to create any new modules, so I modified the base64.py module to include a bunch of new functions, leaving the old API unchanged (i.e. encode(), decode(), encodestring(), decodestring()). I have new functions with docstrings, and an updated unit test. If you folks like the API I've chosen and are generally in favor of my changes, I'll update the documentation as well. The patch is available here: http://sourceforge.net/tracker/index.php?func=detail&aid=868496&group_id=5470&atid=305470 Currently assigned to Tim for review. -Barry From barry at python.org Wed Dec 31 11:41:11 2003 From: barry at python.org (Barry A. Warsaw) Date: Wed Dec 31 11:41:15 2003 Subject: [Python-Dev] regrtest.py mods for code coverage Message-ID: <16370.64551.432152.383782@gargle.gargle.HOWL> Hi all, I wanted to improve the code coverage for base64.py (see previous message, re: RFC 3548 support) in the test_base64.py unit test. So I did a fairly simpleminded adaptation of Zope3's test.py -T code coverage flag to regrtest.py. Very handy for upping coverage of a module's code! The patch is available here: http://sourceforge.net/tracker/index.php?func=detail&aid=868499&group_id=5470&atid=305470 Currently assigned to Jeremy for review. Please let me know what you think. -Barry From guido at python.org Wed Dec 31 12:54:15 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 12:54:25 2003 Subject: [Python-Dev] Pie-thon benchmark code ready Message-ID: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> While it's still 2003 in most of the US and Europe (and happy new year to the folks in Asia, Australia and New Zealand! :-), I present the official Pie-thon Parrot Benchmark: ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz I'll quote from the README.txt file: """ This is a benchmark to be run in front of a live audience at OSCON 2004 between Python and Parrot. The bytecode must be Python 2.3 bytecode frozen in December 2003 (which is almost over as I write this :-). For some more background, see the python-dev thread around http://mail.python.org/pipermail/python-dev/2003-December/040977.html The benchmark here is intended to make Dan Sugalski's life difficult: there are some standard benchmark thingies (simple random algorithms using basic data types) but also a lot of play with odd corners of the language definition, and Python's extremely dynamic object model: funky descriptors, mutable classes, that sort of thing. The benchmark is supposed to run with either Python 2.3 or Python 2.4. [...] On timing: there's a requirement that the benchmark runs for at least 30 seconds. It currently runs for nearly a minute on my home box, which is a four-year-old 650 MHz Pentium box. If the contest hardware is so fast that it runs under a minute, there's a number in b.py that can be cranked up to increase the number of runs. (It takes 27 seconds on my recent work desktop, and on my screaming IBM T40 laptop it runs in 15 seconds, so I suspect that we should at least double the number of runs from 2 to 4.) """ I'd be happy to answer any questions. Let the competition begin! --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 13:12:13 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 13:12:48 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> Message-ID: At 9:54 AM -0800 12/31/03, Guido van Rossum wrote: >While it's still 2003 in most of the US and Europe (and happy new year >to the folks in Asia, Australia and New Zealand! :-), I present the >official Pie-thon Parrot Benchmark: > > ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz Woohoo! The announcement's forwarded on to p6i for the denizens there to grovel over. I fully expect many hours of pain and misery ahead to make this work. :) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 13:27:51 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 13:27:57 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 13:12:13 EST." References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> Message-ID: <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> > > ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz > > Woohoo! The announcement's forwarded on to p6i for the denizens there > to grovel over. I fully expect many hours of pain and misery ahead to > make this work. :) Enjoy. :-) Is there a online archive where I can watch the fun without signing up? BTW I forgot to mention that parts of the test are self-checking, but other parts require comparing of the output. The file 'out' is all you should care about. And if you want the bytecode, try this in a Unix shell: $ python2.3 -O [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import b >>> ^D $ You should now have optimized bytecode in the 8 files b*.pyo. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 13:40:18 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 13:40:49 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> Message-ID: At 10:27 AM -0800 12/31/03, Guido van Rossum wrote: > > > ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz >> >> Woohoo! The announcement's forwarded on to p6i for the denizens there >> to grovel over. I fully expect many hours of pain and misery ahead to >> make this work. :) > >Enjoy. :-) Is there a online archive where I can watch the fun >without signing up? Yep. There's a news<->mail gateway for all the perl.org lists with web archive, so you can read via a newsreader at news://nntp.perl.org/perl.perl6.internals or webbed at http://www.nntp.perl.org/group/perl.perl6.internals. It's a bit of a rough web archive, unfortunately, but the gateway does maintain threads. (Assuming the participants aren't using a broken mail program. Alas, I sometimes do) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From dan at sidhe.org Wed Dec 31 14:06:10 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 14:06:31 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> Message-ID: At 10:27 AM -0800 12/31/03, Guido van Rossum wrote: > > > ftp://ftp.python.org/pub/python/parrotbench/parrotbench.tgz >> >> Woohoo! The announcement's forwarded on to p6i for the denizens there >> to grovel over. I fully expect many hours of pain and misery ahead to >> make this work. :) > >Enjoy. :-) Is there a online archive where I can watch the fun >without signing up? > >BTW I forgot to mention that parts of the test are self-checking, but >other parts require comparing of the output. The file 'out' is all >you should care about. And if you want the bytecode, try this in a >Unix shell: > > $ python2.3 -O > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import b > >>> ^D > $ Hrm. On my OS X laptop: lir:~/Desktop/parrotbench dan$ python -O Python 2.3 (#1, Sep 13 2003, 00:49:11) [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import b --> iteration 0 --> b0 Traceback (most recent call last): File " ", line 1, in ? File "b.py", line 12, in ? b0.main() File "b0.py", line 893, in main checkoutput(4201300315) File "b0.py", line 763, in checkoutput check(strhash(outputtext), n) File "b0.py", line 3, in check raise AssertionError("%.30r != %.30r" % (a, b)) AssertionError: 503203581L != 4201300315L >>> I'll try generating the bytecode on a linux box, but something seems amiss somewhere. (Perhaps just in my understanding of what should be happening) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 14:31:02 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 14:31:08 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 14:06:10 EST." References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> Message-ID: <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> (Thanks for the p6i archives link BTW; Googling for "p6i" wasn't too helpful since its true name is perl6.internals. :-) > > $ python2.3 -O > > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import b > > >>> ^D > > $ > > Hrm. On my OS X laptop: > > lir:~/Desktop/parrotbench dan$ python -O > Python 2.3 (#1, Sep 13 2003, 00:49:11) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import b > --> iteration 0 > --> b0 > Traceback (most recent call last): > File " ", line 1, in ? > File "b.py", line 12, in ? > b0.main() > File "b0.py", line 893, in main > checkoutput(4201300315) > File "b0.py", line 763, in checkoutput > check(strhash(outputtext), n) > File "b0.py", line 3, in check > raise AssertionError("%.30r != %.30r" % (a, b)) > AssertionError: 503203581L != 4201300315L > >>> > > I'll try generating the bytecode on a linux box, but something seems > amiss somewhere. (Perhaps just in my understanding of what should be > happening) No, this means that the benchmark fails on that platform. But you got quite far (the first 5 sections of b0.py worked without a hitch). I notice you're using the original Python 2.3 release thjat came with Panther. It's a bit old. If upgrading to Python 2.3.3 doesn't help, can you send me the output of python b0.py ? Send it in private mail, it's rather large. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 16:16:34 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 16:18:25 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> Message-ID: At 11:31 AM -0800 12/31/03, Guido van Rossum wrote: >(Thanks for the p6i archives link BTW; Googling for "p6i" wasn't too >helpful since its true name is perl6.internals. :-) > >> > $ python2.3 -O >> > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 >> > Type "help", "copyright", "credits" or "license" for more information. >> > >>> import b >> > >>> ^D >> > $ >> > > Hrm. On my OS X laptop: [Failure snipped] > > >> I'll try generating the bytecode on a linux box, but something seems >> amiss somewhere. (Perhaps just in my understanding of what should be >> happening) > >No, this means that the benchmark fails on that platform. > >But you got quite far (the first 5 sections of b0.py worked without a >hitch). > >I notice you're using the original Python 2.3 release thjat came with >Panther. It's a bit old. If upgrading to Python 2.3.3 doesn't help, Turns out not to. I built and installed the 2.3.3 kit and it still fails, albeit differently. >can you send me the output of > > python b0.py > >? Send it in private mail, it's rather large. :-) Sent under a separate cover. Traceback (for those folks following along at home) is: lir:~/Desktop/parrotbench dan$ /usr/local/bin/python2.3 b0.py >b0_output_darwin.lst Traceback (most recent call last): File "b0.py", line 924, in ? main() File "b0.py", line 893, in main checkoutput(4201300315) File "b0.py", line 763, in checkoutput check(strhash(outputtext), n) File "b0.py", line 3, in check raise AssertionError("%.30r != %.30r" % (a, b)) AssertionError: 2772929724L != 4201300315L lir:~/Desktop/parrotbench dan$ I did get bytecode on a linux system, so I've something to work with, though it'll be tough to test times as we go on. (Though I figure if I can beat the 1.4GHz Celeron system with my 600MHz G3 laptop I'll be doing good :) I'll give this a shot on the 10.2 OS X server box I have at home, just for chuckles. (I can set it up as an automated build server if you want, which is all I use it for right now) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From skip at pobox.com Wed Dec 31 18:33:27 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 31 18:33:37 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> Message-ID: <16371.23751.389244.816299@montanaro.dyndns.org> >> > Hrm. On my OS X laptop: Dan> [Failure snipped] >> I notice you're using the original Python 2.3 release thjat came with >> Panther. It's a bit old. If upgrading to Python 2.3.3 doesn't help, Dan> Turns out not to. I built and installed the 2.3.3 kit and it still Dan> fails, albeit differently. Works for me (Mac OS X 10.2.8, Python 2.3.2, GCC 3.3, parrotbench directory after cvs up around 5:30PM central time): montanaro:parrotbench% python2.3 -O Python 2.3.2 (#1, Dec 17 2003, 17:22:01) [GCC 3.3 20030304 (Apple Computer, Inc. build 1493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import b --> iteration 0 --> b0 3141592653 3141592653 --> b1 False False 0 --> b2 ... --> b4 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] --> b5 OK. --> b6 --> All done. Did you make sure you didn't have any .py[co] files before starting? Skip From guido at python.org Wed Dec 31 18:34:07 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 18:34:14 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 11:31:02 PST." Message-ID: <200312312334.hBVNY7506326@c-24-5-183-134.client.comcast.net> Dan reported a problem with the benchmark on Mac OS X: > > Hrm. On my OS X laptop: > > > > lir:~/Desktop/parrotbench dan$ python -O > > Python 2.3 (#1, Sep 13 2003, 00:49:11) > > [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import b > > --> iteration 0 > > --> b0 > > Traceback (most recent call last): > > File " ", line 1, in ? > > File "b.py", line 12, in ? > > b0.main() > > File "b0.py", line 893, in main > > checkoutput(4201300315) > > File "b0.py", line 763, in checkoutput > > check(strhash(outputtext), n) > > File "b0.py", line 3, in check > > raise AssertionError("%.30r != %.30r" % (a, b)) > > AssertionError: 503203581L != 4201300315L > > >>> After some off-line email exchanges I think I have a fix for this behavior, which must have to do with a different length of the addresses shown in the default repr(), e.g. " ". Version 1.0.1 of the benchmark is at the same place as before: ftp://python.org/pub/python/parrotbench/parrotbench.tgz (You can tell whether you have the fixed version by looking at first line of README.txt; if it says "Parrot benchmark 1.0.1" you do.) I haven't heard back from Dan, but I assume that the fix works. Happy New Year everyone! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 31 18:45:00 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 18:45:07 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 17:33:27 CST." <16371.23751.389244.816299@montanaro.dyndns.org> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> <16371.23751.389244.816299@montanaro.dyndns.org> Message-ID: <200312312345.hBVNj0v06358@c-24-5-183-134.client.comcast.net> > Dan> Turns out not to. I built and installed the 2.3.3 kit and it still > Dan> fails, albeit differently. > > Works for me (Mac OS X 10.2.8, Python 2.3.2, GCC 3.3, parrotbench directory > after cvs up around 5:30PM central time): Check the README.txt. Does it say 1.0.1? Then you have my fixed version! I hope so, then I can go home. :-) > montanaro:parrotbench% python2.3 -O > Python 2.3.2 (#1, Dec 17 2003, 17:22:01) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1493)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import b > --> iteration 0 > --> b0 > 3141592653 > 3141592653 > --> b1 > False > False ^^^^^ This actually means that you failed the stack overflow test. It should work when you run it like this: python -O b.py Or you could do sys.setrecursionlimit(1001) before importing b in the interactive session. But that's unrelated to Dan's problem (which happens before b0 completes). And even if that test fails, the .pyo files will be correct. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 18:49:36 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 18:50:03 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312312334.hBVNY7506326@c-24-5-183-134.client.comcast.net> References: <200312312334.hBVNY7506326@c-24-5-183-134.client.comcast.net> Message-ID: At 3:34 PM -0800 12/31/03, Guido van Rossum wrote: >After some off-line email exchanges I think I have a fix for this >behavior, which must have to do with a different length of the >addresses shown in the default repr(), e.g. " ". > >Version 1.0.1 of the benchmark is at the same place as before: > > ftp://python.org/pub/python/parrotbench/parrotbench.tgz > >(You can tell whether you have the fixed version by looking at first >line of README.txt; if it says "Parrot benchmark 1.0.1" you do.) > >I haven't heard back from Dan, but I assume that the fix works. Yup, works just fine. (You caught me during the east coast commute home :) >Happy New Year everyone! Ho, ho, ho! -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From allison at sumeru.stanford.EDU Wed Dec 31 18:56:00 2003 From: allison at sumeru.stanford.EDU (Dennis Allison) Date: Wed Dec 31 18:56:49 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: <200312312334.hBVNY7506326@c-24-5-183-134.client.comcast.net> Message-ID: I suspect there are other folks who have run the pie-con benchmarks on their machines. Perhaps we should construct a chart. Times below are in seconds. ~60 Guido's ancient 650Mhz Pentium ~27 Guido's desktop at work ~15 IBM T40 laptop 88.550 dual 450 MHz Pentium 2 22.340 Athlon XP2800 (1243.363 MHz clock) The latter two times are for Python 2.3.3 built out of the box using the Makefile (make time) shipped with the parrotbenchmarks ftp file on an unloaded machine. The times reported are the user time from the time triplet. From skip at pobox.com Wed Dec 31 18:56:25 2003 From: skip at pobox.com (Skip Montanaro) Date: Wed Dec 31 18:57:06 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312312345.hBVNj0v06358@c-24-5-183-134.client.comcast.net> References: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.net> <200312311827.hBVIRpb05769@c-24-5-183-134.client.comcast.net> <200312311931.hBVJV2o05897@c-24-5-183-134.client.comcast.net> <16371.23751.389244.816299@montanaro.dyndns.org> <200312312345.hBVNj0v06358@c-24-5-183-134.client.comcast.net> Message-ID: <16371.25129.466173.864597@montanaro.dyndns.org> >> Works for me (Mac OS X 10.2.8, Python 2.3.2, GCC 3.3, parrotbench >> directory after cvs up around 5:30PM central time): Guido> Check the README.txt. Does it say 1.0.1? Then you have my fixed Guido> version! Yes. Guido> I hope so, then I can go home. :-) I saw your checkin after my cvs up, so yes, you probably did fix the problem. Time to go find some champagne. Skip From guido at python.org Wed Dec 31 19:14:16 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 19:14:29 2003 Subject: [Python-Dev] Re: Are we collecting benchmark results across machines In-Reply-To: Your message of "Wed, 31 Dec 2003 15:56:00 PST." References: Message-ID: <200401010014.i010EGt06465@c-24-5-183-134.client.comcast.net> > I suspect there are other folks who have run the pie-con benchmarks on > their machines. Perhaps we should construct a chart. Times below are in > seconds. > > ~60 Guido's ancient 650Mhz Pentium > ~27 Guido's desktop at work > ~15 IBM T40 laptop > > 88.550 dual 450 MHz Pentium 2 > 22.340 Athlon XP2800 (1243.363 MHz clock) > > The latter two times are for Python 2.3.3 built out of the box using the > Makefile (make time) shipped with the parrotbenchmarks ftp file on an > unloaded machine. The times reported are the user time from the time > triplet. Perhaps we can turn this into a benchmark comparison chart. In particular, in my experience pystone is a pretty good indicator of system performance even if it's a lousy benchmark. I'll report the pystone numbers for those same three systems: home desktop: 10438.4 pystones/second work desktop: 17421.6 pystones/second laptop: 30198.1 pystones/second Multiplying the pystone numbers with the parrotbench times should give a constant if the benchmarks are equivalent. But this doesn't look good: I get home desktop: 626304 work desktop: 470383 laptop: 452972 The home desktop is a clear outlier; it's more than twice as slow on the parrot benchmark, but only 2/3rds slower on pystone... (This is begging for a spreadsheet. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 19:16:31 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 19:16:54 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: References: Message-ID: At 3:56 PM -0800 12/31/03, Dennis Allison wrote: >I suspect there are other folks who have run the pie-con benchmarks on >their machines. Perhaps we should construct a chart. Times below are in >seconds. > >~60 Guido's ancient 650Mhz Pentium >~27 Guido's desktop at work >~15 IBM T40 laptop > >88.550 dual 450 MHz Pentium 2 >22.340 Athlon XP2800 (1243.363 MHz clock) Well, add in: real 1m20.412s user 1m1.580s sys 0m2.100s for this somewhat creaky 600MHz G3 iBook... -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 19:26:03 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 19:26:09 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: Your message of "Wed, 31 Dec 2003 19:16:31 EST." References: Message-ID: <200401010026.i010Q3B06511@c-24-5-183-134.client.comcast.net> > Well, add in: > > real 1m20.412s > user 1m1.580s > sys 0m2.100s > > for this somewhat creaky 600MHz G3 iBook... I've been looking at user times only, but on that box the discrepancy between user and real time is enormous! It also suggests that a 600 MHz G3 and a 650 P3 are pretty close, and contrary to what Apple seems to claim, the G3's MHz rating isn't worth much more than a P3's MHz rating. Could you run pystone too? python -c 'from test.pystone import main; main(); main(); main()' and then report the smallest pystones/second value. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 19:34:43 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 19:34:56 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: <200401010026.i010Q3B06511@c-24-5-183-134.client.comcast.net> References: <200401010026.i010Q3B06511@c-24-5-183-134.client.comcast.net> Message-ID: At 4:26 PM -0800 12/31/03, Guido van Rossum wrote: > > Well, add in: >> >> real 1m20.412s >> user 1m1.580s >> sys 0m2.100s >> >> for this somewhat creaky 600MHz G3 iBook... > >I've been looking at user times only, but on that box the discrepancy >between user and real time is enormous! Yeah, that's not uncommon. I'm not sure if there's a problem with the time command, or if it's something else. This is a laptop (currently on battery power, though with reduced performance turned off) and it's got a 100MHz main memory bus, which is definitely a limiting factor once code slips out of L1 cache. It's faster than my Gameboy, but some days I wonder how much... :) >It also suggests that a 600 MHz G3 and a 650 P3 are pretty close, and >contrary to what Apple seems to claim, the G3's MHz rating isn't worth >much more than a P3's MHz rating. Possibly. I'm not sure it's necessarily the best machine for that comparison, given the power/performance tradeoffs with laptops. I may give it a whirl on one of the G3 desktop machines around here to see how much bus speed matters. (I won't be surprised, given the likely working set for an interpreter, to find that bus speed makes more of a difference than CPU speed) >Could you run pystone too? > > python -c 'from test.pystone import main; main(); main(); main()' > >and then report the smallest pystones/second value. Smallest is: Pystone(1.1) time for 50000 passes = 6.61 This machine benchmarks at 7564.3 pystones/second -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 19:39:05 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 19:39:11 2003 Subject: [Python-Dev] Are we collecting benchmark results across machines In-Reply-To: Your message of "Wed, 31 Dec 2003 19:34:43 EST." References: <200401010026.i010Q3B06511@c-24-5-183-134.client.comcast.net> Message-ID: <200401010039.i010d5U06553@c-24-5-183-134.client.comcast.net> > >Could you run pystone too? > > > > python -c 'from test.pystone import main; main(); main(); main()' > > > >and then report the smallest pystones/second value. > > Smallest is: > > Pystone(1.1) time for 50000 passes = 6.61 > This machine benchmarks at 7564.3 pystones/second Thanks -- though I made a mistake -- I should've asked for the largest pystone value -- larger is better for pystone, unlike for running times. Anyway, this value is reasonable given your parrotbench time. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Wed Dec 31 19:44:49 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Wed Dec 31 19:41:14 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200312311754.hBVHsFx05650@c-24-5-183-134.client.comcast.ne t> Message-ID: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> At 09:54 31.12.2003 -0800, Guido van Rossum wrote: >I'd be happy to answer any questions. Let the competition begin! wondering whether the correctness checks are too much dependent on details of the representation of objects, especially the order of keys in a dict repr etc... From guido at python.org Wed Dec 31 19:50:33 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 19:50:41 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Thu, 01 Jan 2004 01:44:49 +0100." <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> Message-ID: <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.net> > wondering whether the correctness checks are too much dependent on details > of the representation of objects, especially the order of keys in a dict > repr etc... Probably. :-( I could probably fix that by subclassing dict with something whose repr() sorts the keys -- it doesn't print the few large dicts that it uses, so this should be fine. The same approach could be used for fixing all repr()-related issues. I'll leave it to Dan to decide if he wants this to be fixed ASAP or if he's okay with putting it off until later -- I imagine it will take a while before he's got it running on Parrot well enough to hit the first assert... (There's also an issue with the distinction between repr() of int and long, which will disappear in Python 3.0, and repr() of str vs. unicode, which already doesn't exist in Jython. So I predict that the benchmark currently doesn't have a chance of passing on Jython. (But it would be interesting to time it anyway -- simply disable the raises in b0.check() and in b5.check().) --Guido van Rossum (home page: http://www.python.org/~guido/) From dan at sidhe.org Wed Dec 31 20:16:00 2003 From: dan at sidhe.org (Dan Sugalski) Date: Wed Dec 31 20:16:23 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.net> References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.net> Message-ID: At 4:50 PM -0800 12/31/03, Guido van Rossum wrote: > > wondering whether the correctness checks are too much dependent on details >> of the representation of objects, especially the order of keys in a dict >> repr etc... > >Probably. :-( > >I could probably fix that by subclassing dict with something whose >repr() sorts the keys -- it doesn't print the few large dicts that it >uses, so this should be fine. The same approach could be used for >fixing all repr()-related issues. I'll leave it to Dan to decide if >he wants this to be fixed ASAP or if he's okay with putting it off >until later -- I imagine it will take a while before he's got it >running on Parrot well enough to hit the first assert... Heh. I'm OK with either leaving it as-is if the current ordering is considered correct, or altering the benchmark to guarantee ordering. (Or having the output compared by an external program to guarantee correctness, which is fine as well) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido at python.org Wed Dec 31 20:21:27 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 20:21:34 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Wed, 31 Dec 2003 20:16:00 EST." References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.net> Message-ID: <200401010121.i011LRP06664@c-24-5-183-134.client.comcast.net> > >> wondering whether the correctness checks are too much dependent > >> on details of the representation of objects, especially the order > >> of keys in a dict repr etc... > > > >Probably. :-( > > > >I could probably fix that by subclassing dict with something whose > >repr() sorts the keys -- it doesn't print the few large dicts that it > >uses, so this should be fine. The same approach could be used for > >fixing all repr()-related issues. I'll leave it to Dan to decide if > >he wants this to be fixed ASAP or if he's okay with putting it off > >until later -- I imagine it will take a while before he's got it > >running on Parrot well enough to hit the first assert... > > Heh. I'm OK with either leaving it as-is if the current ordering is > considered correct, or altering the benchmark to guarantee ordering. > (Or having the output compared by an external program to guarantee > correctness, which is fine as well) The ordering is supposed to be an unimportant detail. The external program solution won't work due to the nature of the benchmark. I'll fix the benchmark some time next week. The fix should be fairly simple and localized. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Wed Dec 31 20:39:00 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Wed Dec 31 20:35:31 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: <200401010050.i010oXA06606@c-24-5-183-134.client.comcast.ne t> References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20040101023157.02851b70@pop.bluewin.ch> At 16:50 31.12.2003 -0800, Guido van Rossum wrote: >(There's also an issue with the distinction between repr() of int and >long, which will disappear in Python 3.0, and repr() of str >vs. unicode, which already doesn't exist in Jython. So I predict >that the benchmark currently doesn't have a chance of passing on >Jython. (But it would be interesting to time it anyway -- simply >disable the raises in b0.check() and in b5.check().) Jython is still not up to run it. But yes it would be an interesting thing. Indeed in any case some other things wrt representation would be a problem still: Jython 2.2a0 on java1.4.0_02 (JIT: null) Type "copyright", "credits" or "license" for more information. >>> u'a' 'a' >>> def f(): pass ... >>> f >>> class X: ... pass ... >>> X >>> ... In general <...> reprs vary. From guido at python.org Wed Dec 31 23:44:01 2003 From: guido at python.org (Guido van Rossum) Date: Wed Dec 31 23:44:07 2003 Subject: [Python-Dev] Pie-thon benchmark code ready In-Reply-To: Your message of "Thu, 01 Jan 2004 02:39:00 +0100." <5.2.1.1.0.20040101023157.02851b70@pop.bluewin.ch> References: <5.2.1.1.0.20040101014159.03a31f50@pop.bluewin.ch> <5.2.1.1.0.20040101023157.02851b70@pop.bluewin.ch> Message-ID: <200401010444.i014i1o06815@c-24-5-183-134.client.comcast.net> > Jython is still not up to run it. But yes it would be an interesting thing. Maybe you could find a sponsor for Jython development...? > Indeed in any case some other things wrt representation would be a problem > still: > > Jython 2.2a0 on java1.4.0_02 (JIT: null) > Type "copyright", "credits" or "license" for more information. > >>> u'a' > 'a' > >>> def f(): pass > ... > >>> f > > >>> class X: > ... pass > ... > >>> X > > >>> > ... > > In general <...> reprs vary. But I've seen enough people write code that parses <...> reprs in some way to make me think that maybe they should be standardized somewhat, at least to the point where different Python implementations should not differ gratuitously. E.g. Jython could be much closer to CPython by inserting 'at 0x'. It's not like standardizing this would close off an important implementation freedom for other Python implementation. (I won't go as far as requiring that the number should be the same as hex(id(x)). :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4