Specify a partitioning scheme.
The supported schemes include:
âDirectoryPartitioningâ: this scheme expects one segment in the file path for each field in the specified schema (all fields are required to be present). For example given schema<year:int16, month:int8> the path â/2009/11â would be parsed to (âyearâ_ == 2009 and âmonthâ_ == 11).
âHivePartitioningâ: a scheme for â/$key=$value/â nested directories as found in Apache Hive. This is a multi-level, directory based partitioning scheme. Data is partitioned by static values of a particular column in the schema. Partition keys are represented in the form $key=$value in directory names. Field order is ignored, as are missing or unrecognized field names. For example, given schema<year:int16, month:int8, day:int8>, a possible path would be â/year=2009/month=11/day=15â (but the field order does not need to match).
âFilenamePartitioningâ: this scheme expects the partitions will have filenames containing the field values separated by â_â. For example, given schema<year:int16, month:int8, day:int8>, a possible partition filename â2009_11_part-0.parquetâ would be parsed to (âyearâ_ == 2009 and âmonthâ_ == 11).
pyarrow.Schema, default None
The schema that describes the partitions present in the file path. If not specified, and field_names and/or flavor are specified, the schema will be inferred from the file path (and a PartitioningFactory is returned).
list of str, default None
A list of strings (field names). If specified, the schemaâs types are inferred from the file paths (only valid for DirectoryPartitioning).
str, default None
The default is DirectoryPartitioning. Specify flavor="hive" for a HivePartitioning, and flavor="filename" for a FilenamePartitioning.
dict[str, Array]
If the type of any field of schema is a dictionary type, the corresponding entry of dictionaries must be an array containing every value which may be taken by the corresponding column or an error will be raised in parsing. Alternatively, pass infer to have Arrow discover the dictionary values, in which case a PartitioningFactory is returned.
Partitioning or PartitioningFactory
The partitioning scheme
Examples
Specify the Schema for paths like â/2009/Juneâ:
>>> import pyarrow as pa
>>> import pyarrow.dataset as ds
>>> part = ds.partitioning(pa.schema([("year", pa.int16()),
... ("month", pa.string())]))
or let the types be inferred by only specifying the field names:
>>> part = ds.partitioning(field_names=["year", "month"])
For paths like â/2009/Juneâ, the year will be inferred as int32 while month will be inferred as string.
Specify a Schema with dictionary encoding, providing dictionary values:
>>> part = ds.partitioning(
... pa.schema([
... ("year", pa.int16()),
... ("month", pa.dictionary(pa.int8(), pa.string()))
... ]),
... dictionaries={
... "month": pa.array(["January", "February", "March"]),
... })
Alternatively, specify a Schema with dictionary encoding, but have Arrow infer the dictionary values:
>>> part = ds.partitioning(
... pa.schema([
... ("year", pa.int16()),
... ("month", pa.dictionary(pa.int8(), pa.string()))
... ]),
... dictionaries="infer")
Create a Hive scheme for a path like â/year=2009/month=11â:
>>> part = ds.partitioning(
... pa.schema([("year", pa.int16()), ("month", pa.int8())]),
... flavor="hive")
A Hive scheme can also be discovered from the directory structure (and types will be inferred):
>>> part = ds.partitioning(flavor="hive")
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4