RetroSearch Browse

Tue Feb 22 13:20:57 CET 2011 · https://mail.python.org/pipermail/python-dev/2011-February/108148.html

On 22/02/2011 12:14, Jesus Cea wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I have 10MB pickled structure generated in Python 2.7. I only use basic
> types (no clases) like sets, dictionaries, lists, strings, etc.
>
> The pickle stores a lot of strings. Some of them should be "bytes",
> while other should be "unicode". My idea is to import ALL the strings as
> bytes in Python 3.2 and navigate the data structure to convert the
> appropiate values to unicode, in a one-time operation (I version the
> structure, so I can know if this conversion is already done, simply
> storing a new version value).
>
> But I get this error:
>
> """
> Python 3.2 (r32:88445, Feb 21 2011, 13:34:07)
> [GCC 4.4.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> f=open("file.pickle", mode="rb").read()
>>>> len(f)
> 9847316
>>>> b=pickle.loads(f,encoding="latin1")
> Traceback (most recent call last):
>    File "<stdin>", line 1, in<module>
> ValueError: operation forbidden on released memoryview object
> """
>

That seems like an odd error, but the decision was made that Python 2 
byte-strings would be unpickled on Python 3 as Unicode strings.

See the discussion here:

     http://bugs.python.org/issue6784

This is basically because many people "do the wrong thing" and use 
Python 2 byte strings for restoring text. What it means though is that 
people who "do the right thing" and store binary data in Python 2 byte 
strings can't use Python 2 pickles from Python 3. It also means that 
only ascii data can be unpickled.

A custom pickler / unpickler is suggested as the solution in this issue.

All the best,

Michael Foord

> I use the encoding "latin1" for transparent byte/unicode conversion (do
> not touch the values!).
>
> This seems to be a bug in Python 3.2. Any suggestion?.
>
> PS: The bytestream is protocol 2.
>
> PPS: If there is consensus that this is a real bug, I would create an
> issue in the tracker and try to get a minimal testcase.
>
> - -- 
> Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
> jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
> jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
> .                              _/_/  _/_/    _/_/          _/_/  _/_/
> "Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
> "My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQCVAwUBTWOomplgi5GaxT1NAQLTYAP/U+eqhQ5nIJyBAqSgYwPmkH4DOlMj4JnH
> Jgt6okvOV0hRIXlZ7kbWI2l9OuQyUM4gAeTNDSjFaKs9Hswy26Ro6xhtjidivXDS
> TKw6ocRx92/eHvgsOdEZjrE0D8l0dOqodZddbXELp2DjpYs9aozzAsjTHqNZDE1L
> fujeTOhtUKw=
> =/bcO
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk

-- 
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://mail.python.org/pipermail/python-dev/2011-February/108148.html below:

[Python-Dev] Strange error importing a Pickle from 2.7 to 3.2