Public Functions
Test that all set options are valid.
Public Members
Whether to check UTF8 validity of string columns.
Optional per-column types (disabling type inference on those columns)
Recognized spellings for null values.
Recognized spellings for boolean true values.
Recognized spellings for boolean false values.
Whether string / binary columns can have null values.
If true, then strings in ânull_valuesâ are considered null for string columns. If false, then all strings are valid string values.
Whether quoted values can be null.
If true, then strings in ânull_valuesâ are also considered null when they appear quoted in the CSV file. Otherwise, quoted values are never considered null.
Whether to try to automatically dict-encode string / binary data.
If true, then when type inference detects a string or binary column, it is dict-encoded up to auto_dict_max_cardinality distinct values (per chunk), after which it switches to regular encoding.
This setting is ignored for non-inferred columns (those in column_types).
Decimal point character for floating-point and decimal data.
If non-empty, indicates the names of columns from the CSV file that should be actually read and converted (in the vectorâs order).
Columns not in this vector will be ignored.
If false, columns in include_columns but not in the CSV file will error out.
If true, columns in include_columns but not in the CSV file will produce a column of nulls (whose type is selected using column_types, or null by default) This option is ignored if include_columns is empty.
User-defined timestamp parsers, using the virtual parser interface in arrow/util/value_parsing.h.
More than one parser can be specified, and the CSV conversion logic will try parsing values starting from the beginning of this vector. If no parsers are specified, we use the default built-in ISO-8601 parser.
Public Static Functions
Create conversion options with default values, including conventional values for null_values, true_values and false_values
Public Functions
Test that all set options are valid.
Public Members
Field delimiter.
Whether quoting is used.
Quoting character (if quoting is true)
Whether a quote inside a value is double-quoted.
Whether escaping is used.
Escaping character (if escaping is true)
Whether values are allowed to contain CR (0x0d) and LF (0x0a) characters.
Whether empty lines are ignored.
If false, an empty line represents a single empty value (assuming a one-column CSV file).
A handler function for rows which do not have the correct number of columns.
Public Static Functions
Create parsing options with default values.
Public Functions
Test that all set options are valid.
Public Members
Whether to use the global CPU thread pool.
Block size we request from the IO layer.
This will determine multi-threading granularity as well as the size of individual record batches. Minimum valid value for block size is 1
Number of header rows to skip (not including the row of column names, if any)
Number of rows to skip after the column names are read, if any.
Column names for the target table.
If empty, fall back on autogenerate_column_names.
Whether to autogenerate column names if column_names is empty.
If true, column names will be of the form âf0â, âf1â⦠If false, column names will be read from the first CSV row after skip_rows.
Public Static Functions
Create read options with default values.
A class that reads an entire CSV file into a Arrow Table.
Public Functions
Read the entire CSV file and convert it to a Arrow Table.
Read the entire CSV file and convert it to a Arrow Table.
A class that reads a CSV file incrementally.
Caveats:
For now, this is always single-threaded (regardless of ReadOptions::use_threads.
Type inference is done on the first block and types are frozen afterwards; to make sure the right data types are inferred, either set ReadOptions::block_size to a large enough value, or use ConvertOptions::column_types to set the desired data types explicitly.
Public Functions
Return the number of bytes which have been read and processed.
The returned number includes CSV bytes which the StreamingReader has finished processing, but not bytes for which some processing (e.g. CSV parsing or conversion to Arrow layout) is still ongoing.
Furthermore, the following rules apply:
bytes skipped by ReadOptions.skip_rows are counted as being read before any records are returned.
bytes read while parsing the header are counted as being read before any records are returned.
bytes skipped by ReadOptions.skip_rows_after_names are counted after the first batch is returned.
Public Static Functions
Create a StreamingReader instance.
This involves some I/O as the first batch must be loaded during the creation process so it is returned as a future
Currently, the StreamingReader is not async-reentrant and does not do any fan-out parsing (see ARROW-11889)
Public Functions
Test that all set options are valid.
Public Members
Whether to write an initial header line with column names.
Maximum number of rows processed at a time.
The CSV writer converts and writes data in batches of N rows. This number can impact performance.
Field delimiter.
The string to write for null values. Quotes are not allowed in this string.
IO context for writing.
The end of line character to use for ending rows.
Quoting style.
Public Static Functions
Create write options with default values.
Convert table to CSV and write the result to output.
Experimental
Convert batch to CSV and write the result to output.
Experimental
Convert batches read through a RecordBatchReader to CSV and write the results to output.
Experimental
Create a new CSV writer.
User is responsible for closing the actual OutputStream.
sink â [in] output stream to write to
schema â [in] the schema of the record batches to be written
options â [in] options for serialization
Result<std::shared_ptr<RecordBatchWriter>>
Create a new CSV writer.
sink â [in] output stream to write to (does not take ownership)
schema â [in] the schema of the record batches to be written
options â [in] options for serialization
Result<std::shared_ptr<RecordBatchWriter>>
Values:
Unexpected JSON fields are ignored.
Unexpected JSON fields error out.
Unexpected JSON fields are type-inferred and included in the output.
Public Members
Whether to use the global CPU thread pool.
Block size we request from the IO layer; also determines the size of chunks when use_threads is true.
Public Static Functions
Create read options with default values.
Public Members
Optional explicit schema (disables type inference on those fields)
Whether objects may be printed across multiple lines (for example pretty-printed)
If true, parsing may be slower.
How JSON fields outside of explicit_schema (if given) are treated.
Public Static Functions
Create parsing options with default values.
A class that reads an entire JSON file into a Arrow Table.
The file is expected to consist of individual line-separated JSON objects
Public Functions
Read the entire JSON file and convert it to a Arrow Table.
A class that reads a JSON file incrementally.
JSON data is read from a stream in fixed-size blocks (configurable with ReadOptions::block_size). Each block is converted to a RecordBatch. Yielded batches have a consistent schema but may differ in row count.
The supplied ParseOptions are used to determine a schema, based either on a provided explicit schema or inferred from the first non-empty block. Afterwards, the target schema is frozen. If UnexpectedFieldBehavior::InferType is specified, unexpected fields will only be inferred for the first block. Afterwards theyâll be treated as errors.
If ReadOptions::use_threads is true, each blockâs parsing/decoding task will be parallelized on the given cpu_executor (with readahead corresponding to the executorâs capacity). If an executor isnât provided, the global thread pool will be used.
If ReadOptions::use_threads is false, computations will be run on the calling thread and cpu_executor will be ignored.
Public Functions
Read the next RecordBatch asynchronously This function is async-reentrant (but not synchronously reentrant).
However, if threading is disabled, this will block until completion.
Get the number of bytes which have been successfully converted to record batches and consumed.
Public Static Functions
Create a StreamingReader from an InputStream Blocks until the initial batch is loaded.
stream â [in] JSON source stream
read_options â [in] Options for reading
parse_options â [in] Options for chunking, parsing, and conversion
io_context â [in] Context for IO operations (optional)
cpu_executor â [in] Executor for computation tasks (optional)
The initialized reader
Create a StreamingReader from an InputStream asynchronously Returned future completes after loading the first batch.
stream â [in] JSON source stream
read_options â [in] Options for reading
parse_options â [in] Options for chunking, parsing, and conversion
io_context â [in] Context for IO operations (optional)
cpu_executor â [in] Executor for computation tasks (optional)
Future for the initialized reader
Public Functions
Buffered stream reading allows the user to control the memory usage of parquet readers.
This ensure that all RandomAccessFile::ReadAt calls are wrapped in a buffered reader that uses a fix sized buffer (of size buffer_size()) instead of the full size of the ReadAt.
The primary reason for this control knobs is for resource control and not performance.
Enable buffered stream reading.
Disable buffered stream reading.
Return the size of the buffered stream buffer.
Set the size of the buffered stream buffer in bytes.
Return the size limit on thrift strings.
This limit helps prevent space and time bombs in files, but may need to be increased in order to read files with especially large headers.
Set the size limit on thrift strings.
Return the size limit on thrift containers.
This limit helps prevent space and time bombs in files, but may need to be increased in order to read files with especially large headers.
Set the size limit on thrift containers.
Set the decryption properties.
Return the decryption properties.
EXPERIMENTAL: Properties for configuring FileReader behavior.
Public Functions
Set whether to use the IO thread pool to parse columns in parallel.
Default is false.
Return whether will use multiple threads.
Set whether to read a particular column as dictionary encoded.
If the file metadata contains a serialized Arrow schema, then â¦
This is only supported for columns with a Parquet physical type of BYTE_ARRAY, such as string or binary types.
Return whether the column at the index will be read as dictionary.
Set the maximum number of rows to read into a record batch.
Will only be fewer rows when there are no more rows in the file. Note that some APIs such as ReadTable may ignore this setting.
Return the batch size in rows.
Note that some APIs such as ReadTable may ignore this setting.
Enable read coalescing (default false).
When enabled, the Arrow reader will pre-buffer necessary regions of the file in-memory. This is intended to improve performance on high-latency filesystems (e.g. Amazon S3).
Return whether read coalescing is enabled.
Set options for read coalescing.
This can be used to tune the implementation for characteristics of different filesystems.
Return the options for read coalescing.
Set execution context for read coalescing.
Return the execution context used for read coalescing.
Set timestamp unit to use for deprecated INT96-encoded timestamps (default is NANO).
Enable Parquet-supported Arrow extension types.
When enabled, Parquet logical types will be mapped to their corresponding Arrow extension types at read time, if such exist. Currently only arrow::extension::json() extension type is supported. Columns whose LogicalType is JSON will be interpreted as arrow::extension::json(), with storage type inferred from the serialized Arrow schema if present, or utf8 by default.
Set whether to load statistics as much as possible.
Default is false.
Return whether loading statistics as much as possible.
Public Functions
Returns the PageIndexReader.
Only one instance is ever created.
If the file does not have the page index, nullptr may be returned. Because it pays to check existence of page index in the file, it is possible to return a non null value even if page index does not exist. It is the callerâs responsibility to check the return value and follow-up calls to PageIndexReader.
WARNING: The returned PageIndexReader must not outlive the ParquetFileReader. Initialize GetPageIndexReader() is not thread-safety.
Returns the BloomFilterReader.
Only one instance is ever created.
WARNING: The returned BloomFilterReader must not outlive the ParquetFileReader. Initialize GetBloomFilterReader() is not thread-safety.
Pre-buffer the specified column indices in all row groups.
Readers can optionally call this to cache the necessary slices of the file in-memory before deserialization. Arrow readers can automatically do this via an option. This is intended to increase performance when reading from high-latency filesystems (e.g. Amazon S3).
After calling this, creating readers for row groups/column indices that were not buffered may fail. Creating multiple readers for the a subset of the buffered regions is acceptable. This may be called again to buffer a different set of row groups/columns.
If memory usage is a concern, note that data will remain buffered in memory until either PreBuffer() is called again, or the reader itself is destructed. Reading - and buffering - only one row group at a time may be useful.
This method may throw.
Retrieve the list of byte ranges that would need to be read to retrieve the data for the specified row groups and column indices.
A reader can optionally call this if they wish to handle their own caching and management of file reads (or offload them to other readers). Unlike PreBuffer, this method will not perform any actual caching or reads, instead just using the file metadata to determine the byte ranges that would need to be read if you were to consume the entirety of the column chunks for the provided columns in the specified row groups.
If row_groups or column_indices are empty, then the result of this will be empty.
hole_size_limit represents the maximum distance, in bytes, between two consecutive ranges; beyond this value, ranges will not be combined. The default value is 1MB.
range_size_limit is the maximum size in bytes of a combined range; if combining two consecutive ranges would produce a range larger than this, they are not combined. The default values is 64MB. This must be larger than hole_size_limit.
This will not take into account page indexes or any other predicate push down benefits that may be available.
Wait for the specified row groups and column indices to be pre-buffered.
After the returned Future completes, reading the specified row groups/columns will not block.
PreBuffer must be called first. This method does not throw.
Arrow read adapter class for deserializing Parquet files as Arrow row batches.
This interfaces caters for different use cases and thus provides different interfaces. In its most simplistic form, we cater for a user that wants to read the whole Parquet at once with the FileReader::ReadTable method.
More advanced users that also want to implement parallelism on top of each single Parquet files should do this on the RowGroup level. For this, they can call FileReader::RowGroup(i)->ReadTable to receive only the specified RowGroup as a table.
In the most advanced situation, where a consumer wants to independently read RowGroups in parallel and consume each column individually, they can call FileReader::RowGroup(i)->Column(j)->Read and receive an arrow::Column instance.
Finally, one can also get a stream of record batches using FileReader::GetRecordBatchReader(). This can internally decode columns in parallel if use_threads was enabled in the ArrowReaderProperties.
The parquet format supports an optional integer field_id which can be assigned to a field. Arrow will convert these field IDs to a metadata key named PARQUET:field_id on the appropriate field.
Public Functions
Return arrow schema for all the columns.
Read column as a whole into a chunked array.
The index i refers the index of the top level schema field, which may be nested or flat - e.g.
0 foo.bar foo.bar.baz foo.qux 1 foo2 2 foo3
i=0 will read the entire foo struct, i=1 the foo2 primitive column etc
Return a RecordBatchReader of all row groups and columns.
Deprecated in 19.0.0. Use arrow::Result version instead.
Return a RecordBatchReader of all row groups and columns.
Return a RecordBatchReader of row groups selected from row_group_indices.
Note that the ordering in row_group_indices matters. FileReaders must outlive their RecordBatchReaders.
Deprecated in 19.0.0. Use arrow::Result version instead.
error Status if row_group_indices contains an invalid index
Return a RecordBatchReader of row groups selected from row_group_indices.
Note that the ordering in row_group_indices matters. FileReaders must outlive their RecordBatchReaders.
error Result if row_group_indices contains an invalid index
Return a RecordBatchReader of row groups selected from row_group_indices, whose columns are selected by column_indices.
Note that the ordering in row_group_indices and column_indices matter. FileReaders must outlive their RecordBatchReaders.
Deprecated in 19.0.0. Use arrow::Result version instead.
error Status if either row_group_indices or column_indices contains an invalid index
Return a RecordBatchReader of row groups selected from row_group_indices, whose columns are selected by column_indices.
Note that the ordering in row_group_indices and column_indices matter. FileReaders must outlive their RecordBatchReaders.
error Result if either row_group_indices or column_indices contains an invalid index
Return a RecordBatchReader of row groups selected from row_group_indices, whose columns are selected by column_indices.
Note that the ordering in row_group_indices and column_indices matter. FileReaders must outlive their RecordBatchReaders.
row_group_indices â which row groups to read (order determines read order).
column_indices â which columns to read (order determines output schema).
out â [out] record batch stream from parquet data.
error Status if either row_group_indices or column_indices contains an invalid index
Return a generator of record batches.
The FileReader must outlive the generator, so this requires that you pass in a shared_ptr.
error Result if either row_group_indices or column_indices contains an invalid index
Read all columns into a Table.
Read the given columns into a Table.
The indicated column indices are relative to the internal representation of the parquet table. For instance : 0 foo.bar foo.bar.baz 0 foo.bar.baz2 1 foo.qux 2 1 foo2 3 2 foo3 4
i=0 will read foo.bar.baz, i=1 will read only foo.bar.baz2 and so on. Only leaf fields have indices; foo itself doesnât have an index. To get the index for a particular leaf field, one can use manifest().schema_fields to get the top level fields, and then walk the tree to identify the relevant leaf fields and access its column_index. To get the total number of leaf fields, use FileMetadata.num_columns().
Scan file contents with one thread, return number of rows.
Return a reader for the RowGroup, this object must not outlive the FileReader.
The number of row groups in the file.
Set whether to use multiple threads during reads of multiple columns.
By default only one thread is used.
Set number of records to read per batch for the RecordBatchReader.
Experimental helper class for bindings (like Python) that struggle either with std::move or C++ exceptions.
Build FileReader from Arrow file and MemoryPool.
Advanced settings are supported through the FileReaderBuilder class.
Deprecated in 19.0.0. Use arrow::Result version instead.
Build FileReader from Arrow file and MemoryPool.
Advanced settings are supported through the FileReaderBuilder class.
A class for reading Parquet files using an output stream type API.
The values given must be of the correct type i.e. the type must match the file schema exactly otherwise a ParquetException will be thrown.
The user must explicitly advance to the next row using the EndRow() function or EndRow input manipulator.
Required and optional fields are supported:
Required fields are read using operator>>(T)
Optional fields are read with operator>>(std::optional<T>)
Note that operator>>(std::optional<T>) can be used to read required fields.
Similarly operator>>(T) can be used to read optional fields. However, if the value is not present then a ParquetException will be raised.
Currently there is no support for repeated fields.
Public Functions
Terminate current row and advance to next one.
ParquetException â if all columns in the row were not read or skipped.
Skip the data in the next columns.
If the number of columns exceeds the columns remaining on the current row then skipping is terminated - it does not continue skipping columns on the next row. Skipping of columns still requires the use âEndRowâ even if all remaining columns were skipped.
Number of columns actually skipped.
Skip the data in the next rows.
Skipping of rows is not allowed if reading of data for the current row is not finished. Skipping of rows will be terminated if the end of file is reached.
Number of rows actually skipped.
Public Functions
Specify the memory pool for the writer. Default default_memory_pool.
Enable dictionary encoding in general for all columns.
Default enabled.
Disable dictionary encoding in general for all columns.
Default enabled.
Enable dictionary encoding for column specified by path.
Default enabled.
Enable dictionary encoding for column specified by path.
Default enabled.
Disable dictionary encoding for column specified by path.
Default enabled.
Disable dictionary encoding for column specified by path.
Default enabled.
Specify the dictionary page size limit per row group. Default 1MB.
Specify the write batch size while writing batches of Arrow values into Parquet.
Default 1024.
Specify the max number of rows to put in a single row group.
Default 1Mi rows.
Specify the data page size.
Default 1MB.
Specify the data page version.
Default V1.
Specify the Parquet file version.
Default PARQUET_2_6.
Define the encoding that is used when we donât utilise dictionary encoding.
This either apply if dictionary encoding is disabled or if we fallback as the dictionary grew too large.
Define the encoding that is used when we donât utilise dictionary encoding.
This either apply if dictionary encoding is disabled or if we fallback as the dictionary grew too large.
Define the encoding that is used when we donât utilise dictionary encoding.
This either apply if dictionary encoding is disabled or if we fallback as the dictionary grew too large.
Specify compression codec in general for all columns.
Default UNCOMPRESSED.
Specify max statistics size to store min max value.
Default 4KB.
Specify compression codec for the column specified by path.
Default UNCOMPRESSED.
Specify compression codec for the column specified by path.
Default UNCOMPRESSED.
Specify the default compression level for the compressor in every column.
In case a column does not have an explicitly specified compression level, the default one would be used.
The provided compression level is compressor specific. The user would have to familiarize oneself with the available levels for the selected compressor. If the compressor does not allow for selecting different compression levels, calling this function would not have any effect. Parquet and Arrow do not validate the passed compression level. If no level is selected by the user or if the special std::numeric_limits<int>::min() value is passed, then Arrow selects the compression level.
If other compressor-specific options need to be set in addition to the compression level, use the codec_options method.
Specify a compression level for the compressor for the column described by path.
The provided compression level is compressor specific. The user would have to familiarize oneself with the available levels for the selected compressor. If the compressor does not allow for selecting different compression levels, calling this function would not have any effect. Parquet and Arrow do not validate the passed compression level. If no level is selected by the user or if the special std::numeric_limits<int>::min() value is passed, then Arrow selects the compression level.
Specify a compression level for the compressor for the column described by path.
The provided compression level is compressor specific. The user would have to familiarize oneself with the available levels for the selected compressor. If the compressor does not allow for selecting different compression levels, calling this function would not have any effect. Parquet and Arrow do not validate the passed compression level. If no level is selected by the user or if the special std::numeric_limits<int>::min() value is passed, then Arrow selects the compression level.
Specify the default codec options for the compressor in every column.
The codec options allow configuring the compression level as well as other codec-specific options.
Specify the codec options for the compressor for the column described by path.
Specify the codec options for the compressor for the column described by path.
Define the file encryption properties.
Default NULL.
Enable statistics in general.
Default enabled.
Disable statistics in general.
Default enabled.
Enable statistics for the column specified by path.
Default enabled.
Enable statistics for the column specified by path.
Default enabled.
Define the sorting columns.
Default empty.
If sorting columns are set, user should ensure that records are sorted by sorting columns. Otherwise, the storing data will be inconsistent with sorting_columns metadata.
Disable statistics for the column specified by path.
Default enabled.
Disable statistics for the column specified by path.
Default enabled.
Allow decimals with 1 <= precision <= 18 to be stored as integers.
In Parquet, DECIMAL can be stored in any of the following physical types:
int32: for 1 <= precision <= 9.
int64: for 10 <= precision <= 18.
fixed_len_byte_array: precision is limited by the array size. Length n can store <= floor(log_10(2^(8*n - 1) - 1)) base-10 digits.
binary: precision is unlimited. The minimum number of bytes to store the unscaled value is used.
By default, this is DISABLED and all decimal types annotate fixed_len_byte_array.
When enabled, the C++ writer will use following physical types to store decimals:
int32: for 1 <= precision <= 9.
int64: for 10 <= precision <= 18.
fixed_len_byte_array: for precision > 18.
As a consequence, decimal columns stored in integer types are more compact.
Disable decimal logical type with 1 <= precision <= 18 to be stored as integer physical type.
Default disabled.
Enable writing page index in general for all columns.
Default disabled.
Writing statistics to the page index disables the old method of writing statistics to each data page header. The page index makes filtering more efficient than the page header, as it gathers all the statistics for a Parquet file in a single place, avoiding scattered I/O.
Please check the link below for more details: apache/parquet-format
Disable writing page index in general for all columns. Default disabled.
Enable writing page index for column specified by path. Default disabled.
Enable writing page index for column specified by path. Default disabled.
Disable writing page index for column specified by path. Default disabled.
Disable writing page index for column specified by path. Default disabled.
Set the level to write size statistics for all columns.
Default is None.
level â The level to write size statistics. Note that if page index is not enabled, page level size statistics will not be written even if the level is set to PageAndColumnChunk.
Build the WriterProperties with the builder parameters.
The WriterProperties defined by the builder.
Public Functions
Enable nested type naming according to the parquet specification.
Older versions of arrow wrote out field names for nested lists based on the name of the field. According to the parquet specification they should always be âelementâ.
The underlying engine version to use when writing Arrow data.
V2 is currently the latest V1 is considered deprecated but left in place in case there are bugs detected in V2.
Returns whether the writer will use multiple threads to write columns in parallel in the buffered row group mode.
Returns the executor used to write columns in parallel.
Public Functions
Disable writing legacy int96 timestamps (default disabled).
Enable writing legacy int96 timestamps (default disabled).
May be turned on to write timestamps compatible with older Parquet writers. This takes precedent over coerce_timestamps.
Coerce all timestamps to the specified time unit.
unit â time unit to truncate to. For Parquet versions 1.0 and 2.4, nanoseconds are casted to microseconds.
Allow loss of data when truncating timestamps.
This is disallowed by default and an error will be returned.
Disallow loss of data when truncating timestamps (default).
EXPERIMENTAL: Write binary serialized Arrow schema to the file, to enable certain read options (like âread_dictionaryâ) to be set automatically.
When enabled, will not preserve Arrow field names for list types.
Instead of using the field names Arrow uses for the values array of list types (default âitemâ), will use âelementâ, as is specified in the Parquet spec.
This is enabled by default.
Preserve Arrow list field name.
Set the version of the Parquet writer engine.
Set whether to use multiple threads to write columns in parallel in the buffered row group mode.
WARNING: If writing multiple files in parallel in the same executor, deadlock may occur if use_threads is true. Please disable it in this case.
Default is false.
Set the executor to write columns in parallel in the buffered row group mode.
Default is nullptr and the default cpu executor will be used.
Create the final properties.
Iterative FileWriter class.
For basic usage, can write a Table at a time, creating one or more row groups per write call.
For advanced usage, can write column-by-column: Start a new RowGroup or Chunk with NewRowGroup, then write column-by-column the whole column chunk.
If PARQUET:field_id is present as a metadata key on a field, and the corresponding value is a nonnegative integer, then it will be used as the field_id in the parquet file.
Public Functions
Try to create an Arrow to Parquet file writer.
11.0.0
schema â schema of data that will be passed.
pool â memory pool to use.
sink â output stream to write Parquet data.
properties â general Parquet writer properties.
arrow_properties â Arrow-specific writer properties.
Return the Arrow schema to be written to.
Write a Table to Parquet.
table â Arrow table to write.
chunk_size â maximum number of rows to write per row group.
Start a new row group.
Returns an error if not all columns have been written.
Start a new row group.
Deprecated in 19.0.0.
Write ColumnChunk in row group using an array.
Write ColumnChunk in row group using slice of a ChunkedArray.
Write ColumnChunk in a row group using a ChunkedArray.
Start a new buffered row group.
Returns an error if not all columns have been written.
Write a RecordBatch into the buffered row group.
Multiple RecordBatches can be written into the same row group through this method.
WriterProperties.max_row_group_length() is respected and a new row group will be created if the current row group exceeds the limit.
Batches get flushed to the output stream once NewBufferedRowGroup() or Close() is called.
WARNING: If you are writing multiple files in parallel in the same executor, deadlock may occur if ArrowWriterProperties::use_threads is set to true to write columns in parallel. Please disable use_threads option in this case.
Write the footer and close the file.
Add key-value metadata to the file.
WARNING: If store_schema is enabled, ARROW:schema would be stored in the key-value metadata. Overwriting this key would result in store_schema being unusable during read.
Note
This will overwrite any existing metadata with the same key.
key_value_metadata â [in] the metadata to add.
Error if Close() has been called.
Return the file metadata, only available after calling Close().
Write a Table to Parquet.
This writes one table in a single shot. To write a Parquet file with multiple tables iteratively, see parquet::arrow::FileWriter.
table â Table to write.
pool â memory pool to use.
sink â output stream to write Parquet data.
chunk_size â maximum number of rows to write per row group.
properties â general Parquet writer properties.
arrow_properties â Arrow-specific writer properties.
A class for writing Parquet files using an output stream type API.
The values given must be of the correct type i.e. the type must match the file schema exactly otherwise a ParquetException will be thrown.
The user must explicitly indicate the end of the row using the EndRow() function or EndRow output manipulator.
A maximum row group size can be configured, the default size is 512MB. Alternatively the row group size can be set to zero and the user can create new row groups by calling the EndRowGroup() function or using the EndRowGroup output manipulator.
Required and optional fields are supported:
Required fields are written using operator<<(T)
Optional fields are written using operator<<(std::optional<T>).
Note that operator<<(T) can be used to write optional fields.
Similarly, operator<<(std::optional<T>) can be used to write required fields. However if the optional parameter does not have a value (i.e. it is nullopt) then a ParquetException will be raised.
Currently there is no support for repeated fields.
Public Functions
Output operators for required fields.
These can also be used for optional fields when a value must be set.
Output operators for fixed length strings.
Output operators for variable length strings.
Output operator for optional fields.
Skip the next N columns of optional data.
If there are less than N columns remaining then the excess columns are ignored.
ParquetException â if there is an attempt to skip any required column.
Number of columns actually skipped.
Terminate the current row and advance to next one.
ParquetException â if all columns in the row were not written or skipped.
Terminate the current row group and create new one.
Helper class to write fixed length strings.
This is useful as the standard string view (such as std::string_view) is for variable length data.
Read an Arrow Table or RecordBatch from an ORC file.
Public Functions
Return the schema read from the ORC file.
the returned Schema object
Read the file as a Table.
The table will be composed of one record batch per stripe.
the returned Table
Read the file as a Table.
The table will be composed of one record batch per stripe.
Read the file as a Table.
The table will be composed of one record batch per stripe.
include_indices â [in] the selected field indices to read
the returned Table
Read the file as a Table.
The table will be composed of one record batch per stripe.
include_names â [in] the selected field names to read
the returned Table
Read the file as a Table.
The table will be composed of one record batch per stripe.
Read a single stripe as a RecordBatch.
stripe â [in] the stripe index
the returned RecordBatch
Read a single stripe as a RecordBatch.
stripe â [in] the stripe index
include_indices â [in] the selected field indices to read
the returned RecordBatch
Read a single stripe as a RecordBatch.
stripe â [in] the stripe index
include_names â [in] the selected field names to read
the returned RecordBatch
Seek to designated row.
Invoke NextStripeReader() after seek will return stripe reader starting from designated row.
row_number â [in] the rows number to seek
Get a stripe level record batch iterator.
Each record batch will have up to batch_size rows. NextStripeReader serves as a fine-grained alternative to ReadStripe which may cause OOM issues by loading the whole stripe into memory.
Note this will only read rows for the current stripe, not the entire file.
batch_size â [in] the maximum number of rows in each record batch
the returned stripe reader
Get a stripe level record batch iterator.
Each record batch will have up to batch_size rows. NextStripeReader serves as a fine-grained alternative to ReadStripe which may cause OOM issues by loading the whole stripe into memory.
Note this will only read rows for the current stripe, not the entire file.
batch_size â [in] the maximum number of rows in each record batch
include_indices â [in] the selected field indices to read
the stripe reader
Get a record batch iterator for the entire file.
Each record batch will have up to batch_size rows.
batch_size â [in] the maximum number of rows in each record batch
include_names â [in] the selected field names to read, if not empty (otherwise all fields are read)
the record batch iterator
The number of stripes in the file.
The number of rows in the file.
StripeInformation for each stripe.
Get the format version of the file.
Currently known values are 0.11 and 0.12.
The FileVersion of the ORC file.
Get the software instance and version that wrote this file.
a user-facing string that specifies the software version
Get the compression kind of the file.
The kind of compression in the ORC file.
Get the buffer size for the compression.
Number of bytes to buffer for the compression codec.
Get the number of rows per an entry in the row index.
the number of rows per an entry in the row index or 0 if there is no row index.
Get ID of writer that generated the file.
UNKNOWN_WRITER if the writer ID is undefined
Get the writer id value when getWriterId() returns an unknown writer.
the integer value of the writer ID.
Get the version of the writer.
the version of the writer.
Get the number of stripe statistics in the file.
the number of stripe statistics
Get the length of the data stripes in the file.
return the number of bytes in stripes
Get the length of the file stripe statistics.
the number of compressed bytes in the file stripe statistics
Get the length of the file footer.
the number of compressed bytes in the file footer
Get the length of the file postscript.
the number of bytes in the file postscript
Get the total length of the file.
the number of bytes in the file
Get the serialized file tail.
Useful if another reader of the same file wants to avoid re-reading the file tail. See ReadOptions.SetSerializedFileTail().
a string of bytes with the file tail
Return the metadata read from the ORC file.
A KeyValueMetadata object containing the ORC metadata
Public Static Functions
Creates a new ORC reader.
file â [in] the data source
pool â [in] a MemoryPool to use for buffer allocations
the returned reader object
Options for the ORC Writer.
Public Members
Number of rows the ORC writer writes at a time, default 1024.
Which ORC file version to use, default FileVersion(0, 12)
Size of each ORC stripe in bytes, default 64 MiB.
The compression codec of the ORC file, there is no compression by default.
The size of each compression block in bytes, default 64 KiB.
The compression strategy i.e.
speed vs size reduction, default CompressionStrategy::kSpeed
The number of rows per an entry in the row index, default 10000.
The padding tolerance, default 0.0.
The dictionary key size threshold.
0 to disable dictionary encoding. 1 to always enable dictionary encoding, default 0.0
The array of columns that use the bloom filter, default empty.
The upper limit of the false-positive rate of the bloom filter, default 0.05.
Write an Arrow Table or RecordBatch to an ORC file.
Public Functions
Write a table.
This can be called multiple times.
Tables passed in subsequent calls must match the schema of the table that was written first.
table â [in] the Arrow table from which data is extracted.
Write a RecordBatch.
This can be called multiple times.
RecordBatches passed in subsequent calls must match the schema of the RecordBatch that was written first.
record_batch â [in] the Arrow RecordBatch from which data is extracted.
Close an ORC writer (orc::Writer)
Public Static Functions
Creates a new ORC writer.
output_stream â [in] a pointer to the io::OutputStream to write into
write_options â [in] the ORC writer options for Arrow
the returned writer object
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4