Bases: Array
Concrete class for Arrow arrays of a struct data type.
Methods
Attributes
Return a list of Buffer objects pointing to this arrayâs physical storage.
To correctly interpret these buffers, you need to also apply the offset multiplied with the size of the stored data type.
Cast array values to another data type
See pyarrow.compute.cast() for usage.
DataType, default None
Type to cast array to.
True
Whether to check for conversion errors such as overflow.
CastOptions, default None
Additional checks pass by CastOptions
MemoryPool, optional
memory pool to use for allocations during function execution.
Array
Construct a copy of the array with all buffers on destination device.
This method recursively copies the arrayâs buffers and those of its children onto the destination MemoryManager device and returns the new Array.
pyarrow.MemoryManager or pyarrow.Device
The destination device to copy the array to.
Array
The device type where the array resides.
DeviceAllocationType
Compute dictionary-encoded representation of array.
See pyarrow.compute.dictionary_encode() for full usage.
str, default âmaskâ
How to handle null entries.
DictionaryArray
A dictionary-encoded version of this array.
Compare contents of this array against another one.
Return a string containing the result of diffing this array (on the left side) against the other array (on the right side).
Array
The other array to compare this array with.
str
A human-readable printout of the differences.
Examples
>>> import pyarrow as pa >>> left = pa.array(["one", "two", "three"]) >>> right = pa.array(["two", None, "two-and-a-half", "three"]) >>> print(left.diff(right))
@@ -0, +0 @@ -âoneâ @@ -2, +1 @@ +null +âtwo-and-a-halfâ
Remove missing values from an array.
pyarrow.Array
Retrieves the child array belonging to field.
Union[int, str]
Index / position or name of the field.
Array
See pyarrow.compute.fill_null() for usage.
any
The replacement value for null entries.
Array
A new array with nulls replaced by the given value.
Select values from an array.
See pyarrow.compute.filter() for full usage.
Array or array-like
The boolean mask to filter the array with.
str, default âdropâ
How nulls in the mask should be handled.
Array
An array of the same type, with only the elements selected by the boolean mask.
Return one individual array for each field in the struct.
MemoryPool, default None
For memory allocations, if required, otherwise use default pool.
List[Array]
DEPRECATED, use pyarrow.Array.to_string
dict
str
Construct StructArray from collection of arrays representing each field in the struct.
Either field names, field instances or a struct type must be passed.
Array
List[str] (optional)
Field names for each struct child.
List[Field] (optional)
Field instances for each struct child.
pyarrow.Array[bool] (optional)
Indicate which values are null (True) or not null (False).
MemoryPool (optional)
For memory allocations, if required, otherwise uses default pool.
pyarrow.StructType (optional)
Struct type for name and type of each child.
StructArray
Construct an Array from a sequence of buffers.
The concrete type returned depends on the datatype.
DataType
The value type of the array.
int
The number of values in the array.
List[Buffer]
The buffers backing this array.
int, default -1
The number of null entries in the array. Negative value means that the null count is not known.
int, default 0
The arrayâs logical offset (in values, not in bytes) from the start of each buffer.
List[Array], default None
Nested type children with length matching type.num_fields.
Array
Convert pandas.Series to an Arrow Array.
This method uses Pandas semantics about what values indicate nulls. See pyarrow.array for more general conversion from arrays or sequences to Arrow arrays.
ndarray, pandas.Series, array-like
array (bool), optional
Indicate which values are null (True) or not null (False).
pyarrow.DataType
Explicit type to attempt to coerce to, otherwise will be inferred from the data.
True
Check for overflows or other unsafe conversions.
pyarrow.MemoryPool, optional
If not passed, will allocate memory from the currently-set default memory pool.
pyarrow.Array or pyarrow.ChunkedArray
ChunkedArray is returned if object data overflows binary buffer.
Notes
Localized timestamps will currently be returned as UTC (pandasâs native representation). Timezone-naive data will be implicitly interpreted as UTC.
The sum of bytes in each buffer referenced by the array.
An array may only reference a portion of a buffer. This method will overestimate in this case and return the byte size of the entire buffer.
If a buffer is referenced multiple times then it will only be counted once.
Find the first index of a value.
See pyarrow.compute.index() for full usage.
Scalar or object
The value to look for in the array.
int, optional
The start index where to look for value.
int, optional
The end index where to look for value.
MemoryPool, optional
A memory pool for potential memory allocations.
Int64Scalar
The index of the value in the array (-1 if not found).
Whether the array is CPU-accessible.
Return BooleanArray indicating the NaN values.
Array
Return BooleanArray indicating the null values.
False)
Whether floating-point NaN values should also be considered null.
Array
Return BooleanArray indicating the non-null values.
Total number of bytes consumed by the elements of the array.
In other words, the sum of bytes from all buffer ranges referenced.
Unlike get_total_buffer_size this method will account for array offsets.
If buffers are shared between arrays then the shared portion will be counted multiple times.
The dictionary of dictionary arrays will always be counted in their entirety even if the array only references a portion of the dictionary.
A relative position into another arrayâs data.
The purpose is to enable zero-copy slicing. This value defaults to zero but must be applied on all operations with the physical storage buffers.
Compute zero-copy slice of this array.
int, default 0
Offset from start of array to slice.
int, default None
Length of slice (default is until end of Array starting from offset).
Array
An array with the same datatype, containing the sliced values.
Sort the StructArray
str, default âascendingâ
Which order to sort values in. Accepted values are âascendingâ, âdescendingâ.
str or None, default None
If to sort the array by one of its fields or by the whole array.
dict, optional
Additional sorting options. As allowed by SortOptions
StructArray
Statistics of the array.
Sum the values in a numerical array.
See pyarrow.compute.sum() for full usage.
dict, optional
Options to pass to pyarrow.compute.sum().
Scalar
A scalar containing the sum value.
Select values from an array.
See pyarrow.compute.take() for full usage.
Array or array-like
The indices in the array whose values will be returned.
Array
An array with the same datatype, containing the taken values.
Return a NumPy view or copy of this array.
By default, tries to return a view of this array. This is only supported for primitive arrays with the same memory layout as NumPy (i.e. integers, floating point, ..) and without any nulls.
For the extension arrays, this method simply delegates to the underlying storage array.
True
If True, an exception will be raised if the conversion to a numpy array would require copying the underlying data (e.g. in presence of nulls, or for non-primitive types).
False
For numpy arrays created with zero copy (view on the Arrow data), the resulting array is not writable (Arrow data is immutable). By setting this to True, a copy of the array is made to ensure it is writable.
numpy.ndarray
Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
MemoryPool, default None
Arrow MemoryPool to use for allocations. Uses the default memory pool if not passed.
list, default empty
List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.
False
Encode string (UTF8) and binary types to pandas.Categorical.
False
Raise an ArrowException if this function call would require copying the underlying data.
False
Cast integers with nulls to objects
True
Cast dates to objects. If False, convert to datetime64 dtype with the equivalent time unit (if supported). Note: in pandas version < 2.0, only datetime64[ns] conversion is supported.
False
Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful in pandas version 1.x if you have timestamps that donât fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). Non-nanosecond timestamps are supported in pandas version 2.0. If False, all timestamps are converted to datetime64 dtype.
True
Whether to parallelize the conversion using multiple threads.
True
Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.
False
If True, do not use the âpandasâ metadata to reconstruct the DataFrame index, if present
True
For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.
False
If True, generate one internal âblockâ for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger âconsolidationâ which may balloon memory use.
False
EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.
Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory canât be freed until all columns are converted.
str, optional, default None
Valid values are None, âlossyâ, or âstrictâ. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), â¦].
If âlossyâ or âstrictâ, convert Arrow Map arrays to native Python dicts. This can change the ordering of (key, value) pairs, and will deduplicate multiple keys, resulting in a possible loss of data.
If âlossyâ, this key deduplication results in a warning printed when detected. If âstrictâ, this instead results in an exception being raised when detected.
None
A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.
False
Only applicable to pandas version >= 2.0. A legacy option to coerce date32, date64, duration, and timestamp time units to nanoseconds when converting to pandas. This is the default behavior in pandas version 1.x. Set this option to True if youâd like to use this coercion when using pandas version >= 2.0 for backwards compatibility (not recommended otherwise).
pandas.Series or pandas.DataFrame depending on type of object
Examples
>>> import pyarrow as pa >>> import pandas as pd
Convert a Table to pandas DataFrame:
>>> table = pa.table([ ... pa.array([2, 4, 5, 100]), ... pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) ... ], names=['n_legs', 'animals']) >>> table.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede >>> isinstance(table.to_pandas(), pd.DataFrame) True
Convert a RecordBatch to pandas DataFrame:
>>> import pyarrow as pa >>> n_legs = pa.array([2, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.record_batch([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch pyarrow.RecordBatch n_legs: int64 animals: string ---- n_legs: [2,4,5,100] animals: ["Flamingo","Horse","Brittle stars","Centipede"] >>> batch.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede >>> isinstance(batch.to_pandas(), pd.DataFrame) True
Convert a Chunked Array to pandas Series:
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs.to_pandas() 0 2 1 2 2 4 3 4 4 5 5 100 dtype: int64 >>> isinstance(n_legs.to_pandas(), pd.Series) True
Convert to a list of native Python objects.
str, optional, default None
Valid values are None, âlossyâ, or âstrictâ. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), â¦].
If âlossyâ or âstrictâ, convert Arrow Map arrays to native Python dicts.
If âlossyâ, whenever duplicate keys are detected, a warning will be printed. The last seen value of a duplicate key will be in the Python dictionary. If âstrictâ, this instead results in an exception being raised when detected.
list
Render a âpretty-printedâ string representation of the Array.
Note: for data on a non-CPU device, the full array is copied to CPU memory.
int, default 2
How much to indent the internal items in the string to the right, by default 2.
int, default 0
How much to indent right the entire content of the array, by default 0.
int
How many primitive items to preview at the begin and end of the array when the array is bigger than the window. The other items will be ellipsed.
int
How many container items (such as a list in a list array) to preview at the begin and end of the array when the array is bigger than the window.
If the array should be rendered as a single line of text or if each element should be on its own line.
Alias of to_pylist for compatibility with NumPy.
Compute distinct elements in array.
Array
An array of the same data type, with deduplicated elements.
Perform validation checks. An exception is raised if validation fails.
By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).
False
If True, run expensive checks, otherwise cheap checks only.
ArrowInvalid
Compute counts of unique elements in array.
StructArray
An array of <input type âValuesâ, int64 âCountsâ> structs
Return zero-copy âviewâ of array as another data type.
The data types must have compatible columnar buffer layouts
DataType
Type to construct view as.
Array
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4