Showing content from http://mail.python.org/pipermail/python-dev/attachments/20140916/2d4ac346/attachment.html below:
<div dir="ltr">Great points here - I especially like the concluding statement "<span style="font-family:arial,sans-serif;font-size:12.6666669845581px">you can't treat it as a pure Unicode string - it's a Unicode </span><span style="font-family:arial,sans-serif;font-size:12.6666669845581px">string with smuggled bytes"</span><div><span style="font-family:arial,sans-serif;font-size:12.6666669845581px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:12.6666669845581px">Given that Jython uses UTF-16 as its representation, it is possible to frequently smuggle isolated surrogates in it. A surrogate pair must be a low surrogate in range (D800, DC00), then a high surrogate in range(</span><span style="font-family:arial,sans-serif;font-size:12.6666669845581px">DC00, E000). So one can likely assign an interpretation that this is in fact the isolated surrogate, and not an actual codepoint.</span><br></div><div><span style="font-family:arial,sans-serif;font-size:12.6666669845581px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:12.6666669845581px">Of course, if you do actually have a smuggled isolated low surrogate FOLLOWED by a smuggled isolated high surrogate - guess what, the only interpretation is a codepoint. Or perhaps more likely garbage. Of course it doesn't happen so often, so maybe we are fine with the occasional bug ;)</span></div><div><span style="font-family:arial,sans-serif;font-size:12.6666669845581px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:12.6666669845581px">I personally suspect that we will resolve this by also supporting UCS-4 as a representation in Jython 3.x for such Unicode strings, albeit with the limitation that we have simply moved the problem to when we try to call Java methods taking java.lang.String objects.</span></div><div><span style="font-family:arial,sans-serif;font-size:12.6666669845581px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:12.6666669845581px">- Jim</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 16, 2014 at 9:27 AM, Chris Angelico <span dir="ltr"><<a href="mailto:rosuav@gmail.com" target="_blank">rosuav@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Wed, Sep 17, 2014 at 1:00 AM, R. David Murray <<a href="mailto:rdmurray@bitdance.com">rdmurray@bitdance.com</a>> wrote:<br>
> That isn't the case in the email package. The smuggled bytes are not<br>
> errors[*], they are literally smuggled bytes.<br>
<br>
</span>But they're not characters, which is what Stephen and I were saying -<br>
and contrary to what Jim said about treating them as characters. At<br>
best, they represent characters but in some encoding other than the<br>
one you're using, and you have no idea how many bytes form a character<br>
or anything. So you can't, for instance, word-wrap the text, because<br>
you can't know how wide these unknown bytes are, whether they<br>
represent spaces (wrap points), or newlines, or anything like that.<br>
You can't treat them as characters, so while you have them in your<br>
string, you can't treat it as a pure Unicode string - it''s a Unicode<br>
string with smuggled bytes.<br>
<br>
ChrisA<br>
<div class="HOEnZb"><div class="h5">_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com" target="_blank">https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">- Jim<div><br></div><div>jim.baker@{<a href="http://colorado.edu" target="_blank">colorado.edu</a>|<a href="http://python.org" target="_blank">python.org</a>|<a href="http://rackspace.com" target="_blank">rackspace.com</a>|<a href="http://zyasoft.com" target="_blank">zyasoft.com</a>}</div><div><a href="http://twitter.com/jimbaker" target="_blank">twitter.com/jimbaker</a></div><div><a href="http://github.com/jimbaker" target="_blank">github.com/jimbaker</a></div><div><a href="http://bitbucket.com/jimbaker" target="_blank">bitbucket.com/jimbaker</a></div></div>
</div>
RetroSearch is an open source project built by @garambo
| Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4