Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/pandas-dev/pandas/issues/40652 below:

Pandas StructDType · Issue #40652 · pandas-dev/pandas · GitHub

I already searched for a while for discussions of nested structures in Pandas, but I couldn't find anything corresponding.

Currently, there is no way to work with arbitrary nested data types in Pandas.
In Spark and PyArrow, one can have StructTypes. In NumPy, we have Compound types.
However, when we try something like this:

df = pd.DataFrame({'pos':[1,2,3], "val": ["a","b", "c"]})
struct = df.to_records(index=False).view(type=np.ndarray, dtype=list(df.dtypes.items()))
pd.Series(struct)

Then we only get an error:

ValueError: Cannot construct a Series from an ndarray with compound dtype.  Use DataFrame instead.

This would be useful for a number of use cases:

• Direct mapping between PySpark and Pandas nested columns
• Easy creation of custom data types; type checking of nested columns
• Efficient serialization with Arrow

My wish would be to have a generic Pandas StructDType that:

• can be composed of any other Pandas DType
• allows conversion to PyArrow StructType back and forth
• can be written to Parquet

A perfect example is the IntervalDType/IntervalArray that is already implemented in Pandas:

In my opinion, its implementation is a special case of a Struct dtype.

It also supports conversion to and from PyArrow (see

).

Therefore, by generalizing the IntervalDType to use any number of subtypes, we would have the StructDType implementation ready.

to_csv(), etc. could have difficulties with storing nested data.
That's maybe a followup problem to solve.

One can try to construct the Series as a list of tuples.
However, this has two drawbacks:

• No type checking
• to_parquet() fails

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4