Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/pandas-dev/pandas/issues/54190 below:

`testing.assert_frame_equal` unhelpful error message for `string[pyarrow]` · Issue #54190 · pandas-dev/pandas · GitHub

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

import pandas as pd

df = (
    pd.DataFrame({"a": [1, 2, 3], "b": [1.0, 2.0, None], "c": [None, "x", "y"]})
    .astype({"a": "int64[pyarrow]", "b": "float64[pyarrow]", "c": "string[pyarrow]"})
    .set_index("a")
)
df.to_parquet("demo.parquet")
df2 = pd.read_parquet("demo.parquet", dtype_backend="pyarrow")
pd.testing.assert_frame_equal(df, df2)

results in the error message

AssertionError: Attributes of DataFrame.iloc[:, 1] (column name="c") are different

Attribute "dtype" are different
[left]:  string[pyarrow]
[right]: string[pyarrow]

The problem is that

print(type(df.dtypes[-1]))   # string_.StringDtype'
print(type(df2.dtypes[-1]))  # arrow.dtype.ArrowDtype

have the same string representation

import pyarrow as pa
import pandas as pd

print(repr(pd.ArrowDtype(pa.string())))  # string[pyarrow]
print(repr(pd.StringDtype("pyarrow")))  # string[pyarrow]

1. StringDtype("pyarrow") and ArrowDtype("string") need different repr
2. testing.assert_frame_equal should consider printing not just repr(dtype), but also repr(type(dtype)).
3. pd.read_parquet should round trip both index and column dtypes #

As per https://pandas.pydata.org/docs/user_guide/pyarrow.html

The string alias "string[pyarrow]" maps to pd.StringDtype("pyarrow") which is not equivalent to specifying dtype=pd.ArrowDtype(pa.string()). Generally, operations on the data will behave similarly except pd.StringDtype("pyarrow") can return NumPy-backed nullable types while pd.ArrowDtype(pa.string()) will return ArrowDtype.

INSTALLED VERSIONS
------------------
commit           : 0f437949513225922d851e9581723d82120684a6
python           : 3.11.3.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.19.0-46-generic
Version          : #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 2.0.3
numpy            : 1.24.3
pytz             : 2023.3
dateutil         : 2.8.2
setuptools       : 68.0.0
pip              : 23.2
Cython           : 0.29.36
pytest           : 7.4.0
hypothesis       : None
sphinx           : 7.0.1
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.9.3
html5lib         : None
pymysql          : 1.4.6
psycopg2         : None
jinja2           : 3.1.2
IPython          : 8.14.0
pandas_datareader: None
bs4              : 4.12.2
bottleneck       : None
brotli           : None
fastparquet      : 2023.7.0
fsspec           : 2023.6.0
gcsfs            : None
matplotlib       : 3.7.2
numba            : 0.57.1
numexpr          : 2.8.4
odfpy            : None
openpyxl         : 3.1.2
pandas_gbq       : None
pyarrow          : 12.0.1
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : 1.11.1
snappy           : None
sqlalchemy       : 1.4.49
tables           : 3.8.0
tabulate         : 0.9.0
xarray           : 2023.7.0
xlrd             : None
zstandard        : None
tzdata           : 2023.3
qtpy             : None
pyqt5            : None

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4