Options for reading Arrow IPC messages.
Public Members
The maximum permitted schema nesting depth.
The memory pool to use for allocations made during IPC reading.
While Arrow IPC is predominantly zero-copy, it may have to allocate memory in some cases (for example if compression is enabled).
Top-level schema fields to include when deserializing RecordBatch.
If empty (the default), return all deserialized fields. If non-empty, the values are the indices of fields in the top-level schema.
Use global CPU thread pool to parallelize any computational tasks like decompression.
Whether to convert incoming data to platform-native endianness.
If the endianness of the received schema is not equal to platform-native endianness, then all buffers with endian-sensitive data will be byte-swapped. This includes the value buffers of numeric types, temporal types, decimal types, as well as the offset buffers of variable-sized binary and list-like types.
Endianness conversion is achieved by the RecordBatchFileReader, RecordBatchStreamReader and StreamDecoder classes.
How to align data if mis-aligned.
Data is copied to aligned memory locations allocated via the MemoryPool configured as arrow::ipc::IpcReadOptions::memory_pool. Some use cases might require data to have a specific alignment, for example, for the data buffer of an Int32 array to be aligned on a 4-byte boundary.
Default (kAnyAlignment) keeps the alignment as is, so no copy of data occurs.
Options to control caching behavior when pre-buffering is requested.
The lazy property will always be reset to true to deliver the expected behavior
Options for writing Arrow IPC messages.
Public Members
If true, allow field lengths that donât fit in a signed 32-bit int.
Some implementations may not be able to parse streams created with this option.
The maximum permitted schema nesting depth.
Write padding after memory buffers up to this multiple of bytes.
Write the pre-0.15.0 IPC message format.
This legacy format consists of a 4-byte prefix instead of 8-byte.
The memory pool to use for allocations made during IPC writing.
While Arrow IPC is predominantly zero-copy, it may have to allocate memory in some cases (for example if compression is enabled).
Compression codec to use for record batch body buffers.
May only be UNCOMPRESSED, LZ4_FRAME and ZSTD.
Minimum space savings percentage required for compression to be applied.
Space savings is calculated as (1.0 - compressed_size / uncompressed_size).
For example, if min_space_savings = 0.1, a 100-byte body buffer wonât undergo compression if its expected compressed size exceeds 90 bytes. If this option is unset, compression will be used indiscriminately. If no codec was supplied, this option is ignored.
Values outside of the range [0,1] are handled as errors.
Note that enabling this option may result in unreadable data for Arrow C++ versions prior to 12.0.0.
Use global CPU thread pool to parallelize any computational tasks like compression.
Whether to emit dictionary deltas.
If false, a changed dictionary for a given field will emit a full dictionary replacement. If true, a changed dictionary will be compared against the previous version. If possible, a dictionary delta will be emitted, otherwise a full dictionary replacement.
Default is false to maximize stream compatibility.
Also, note that if a changed dictionary is a nested dictionary, then a delta is never emitted, for compatibility with the read path.
Whether to unify dictionaries for the IPC file format.
The IPC file format doesnât support dictionary replacements. Therefore, chunks of a column with a dictionary type must have the same dictionary in each record batch (or an extended dictionary + delta).
If this option is true, RecordBatchWriter::WriteTable will attempt to unify dictionaries across each table column. If this option is false, incompatible dictionaries across a table column will simply raise an error.
Note that enabling this option has a runtime cost. Also, not all types currently support dictionary unification.
This option is ignored for IPC streams, which support dictionary replacement and deltas.
Format version to use for IPC messages and their metadata.
Presently using V5 version (readable by 1.0.0 and later). V4 is also available (readable by 0.8.0 and later).
Use either of these two classes, depending on which IPC format you want to read. The file format requires a random-access file, while the stream format only requires a sequential input stream.
Synchronous batch stream reader that reads from io::InputStream.
This class reads the schema (plus any dictionaries) as the first messages in the stream, followed by record batches. For more granular zero-copy reads see the ReadRecordBatch functions
Public Functions
Return current read statistics.
Public Static Functions
Create batch reader from generic MessageReader.
This will take ownership of the given MessageReader.
message_reader â [in] a MessageReader implementation
options â [in] any IPC reading options (optional)
the created batch reader
Record batch stream reader from InputStream.
stream â [in] an input stream instance. Must stay alive throughout lifetime of stream reader
options â [in] any IPC reading options (optional)
the created batch reader
Open stream and retain ownership of stream object.
stream â [in] the input stream
options â [in] any IPC reading options (optional)
the created batch reader
Reads the record batch file format.
Public Functions
The schema read from the file.
Returns the number of record batches in the file.
Return the metadata version from the file metadata.
Return the contents of the custom_metadata field from the fileâs Footer.
Read a particular record batch from the file.
Does not copy memory if the input source supports zero-copy.
i â [in] the index of the record batch to return
the read batch
Read a particular record batch along with its custom metadata from the file.
Does not copy memory if the input source supports zero-copy.
i â [in] the index of the record batch to return
a struct containing the read batch and its custom metadata
Return current read statistics.
Computes the total number of rows in the file.
Begin loading metadata for the desired batches into memory.
This method will also begin loading all dictionaries messages into memory.
For a regular file this will immediately begin disk I/O in the background on a thread on the IOContextâs thread pool. If the file is memory mapped this will ensure the memory needed for the metadata is paged from disk into memory
indices â Indices of the batches to prefetch If empty then all batches will be prefetched.
Get a reentrant generator of record batches.
coalesce â [in] If true, enable I/O coalescing.
io_context â [in] The IOContext to use (controls which thread pool is used for I/O).
cache_options â [in] Options for coalescing (if enabled).
executor â [in] Optionally, an executor to use for decoding record batches. This is generally only a benefit for very wide and/or compressed batches.
Collect all batches as a vector of record batches.
Collect all batches and concatenate as arrow::Table.
Public Static Functions
Open a RecordBatchFileReader.
Open a file-like object that is assumed to be self-contained; i.e., the end of the file interface is the end of the Arrow file. Note that there can be any amount of data preceding the Arrow-formatted data, because we need only locate the end of the Arrow file stream to discover the metadata and then proceed to read the data into memory.
Open a RecordBatchFileReader If the file is embedded within some larger file or memory region, you can pass the absolute memory offset to the end of the file (which contains the metadata footer).
The metadata must have been written with memory offsets relative to the start of the containing file
file â [in] the data source
footer_offset â [in] the position of the end of the Arrow file
options â [in] options for IPC reading
the returned reader
Version of Open that retains ownership of file.
file â [in] the data source
options â [in] options for IPC reading
the returned reader
Version of Open that retains ownership of file.
file â [in] the data source
footer_offset â [in] the position of the end of the Arrow file
options â [in] options for IPC reading
the returned reader
Open a file asynchronously (owns the file).
Open a file asynchronously (borrows the file).
Open a file asynchronously (owns the file).
Open a file asynchronously (borrows the file).
To read an IPC stream in event-driven fashion, you must implement a Listener subclass that you will pass to StreamDecoder.
A general listener class to receive events.
You must implement callback methods for interested events.
This API is EXPERIMENTAL.
0.17.0
Subclassed by arrow::ipc::CollectListener
Public Functions
Called when end-of-stream is received.
The default implementation just returns arrow::Status::OK().
Called when a record batch is decoded and OnRecordBatchWithMetadataDecoded() isnât overridden.
The default implementation just returns arrow::Status::NotImplemented().
record_batch â [in] a record batch decoded
Called when a record batch with custom metadata is decoded.
The default implementation just calls OnRecordBatchDecoded() without custom metadata.
13.0.0
record_batch_with_metadata â [in] a record batch with custom metadata decoded
Called when a schema is decoded.
The default implementation just returns arrow::Status::OK().
schema â [in] a schema decoded
Called when a schema is decoded.
The default implementation just calls OnSchemaDecoded(schema) (without filtered_schema) to keep backward compatibility.
13.0.0
schema â [in] a schema decoded
filtered_schema â [in] a filtered schema that only has read fields
Push style stream decoder that receives data from user.
This class decodes the Apache Arrow IPC streaming format data.
This API is EXPERIMENTAL.
0.17.0
Public Functions
Construct a stream decoder.
listener â [in] a Listener that must implement Listener::OnRecordBatchDecoded() to receive decoded record batches
options â [in] any IPC reading options (optional)
Feed data to the decoder as a raw data.
If the decoder can read one or more record batches by the data, the decoder calls listener->OnRecordBatchDecoded() with a decoded record batch multiple times.
data â [in] a raw data to be processed. This data isnât copied. The passed memory must be kept alive through record batch processing.
size â [in] raw data size.
Feed data to the decoder as a Buffer.
If the decoder can read one or more record batches by the Buffer, the decoder calls listener->RecordBatchReceived() with a decoded record batch multiple times.
Reset the internal status.
You can reuse this decoder for new stream after calling this.
the shared schema of the record batches in the stream
Return the number of bytes needed to advance the state of the decoder.
This method is provided for users who want to optimize performance. Normal users donât need to use this method.
Here is an example usage for normal users:
decoder.Consume(buffer1); decoder.Consume(buffer2); decoder.Consume(buffer3);
Decoder has internal buffer. If consumed data isnât enough to advance the state of the decoder, consumed data is buffered to the internal buffer. It causes performance overhead.
If you pass next_required_size() size data to each Consume() call, the decoder doesnât use its internal buffer. It improves performance.
Here is an example usage to avoid using internal buffer:
buffer1 = get_data(decoder.next_required_size()); decoder.Consume(buffer1); buffer2 = get_data(decoder.next_required_size()); decoder.Consume(buffer2);
Users can use this method to avoid creating small chunks. Record batch data must be contiguous data. If users pass small chunks to the decoder, the decoder needs concatenate small chunks internally. It causes performance overhead.
Here is an example usage to reduce small chunks:
buffer = AllocateResizableBuffer();
while ((small_chunk = get_data(&small_chunk_size))) {
auto current_buffer_size = buffer->size();
buffer->Resize(current_buffer_size + small_chunk_size);
memcpy(buffer->mutable_data() + current_buffer_size,
small_chunk,
small_chunk_size);
if (buffer->size() < decoder.next_required_size()) {
continue;
}
std::shared_ptr<arrow::Buffer> chunk(buffer.release());
decoder.Consume(chunk);
buffer = AllocateResizableBuffer();
}
if (buffer->size() > 0) {
std::shared_ptr<arrow::Buffer> chunk(buffer.release());
decoder.Consume(chunk);
}
the number of bytes needed to advance the state of the decoder
Return current read statistics.
Public Members
Number of IPC messages read.
Number of record batches read.
Number of dictionary batches read.
Note: num_dictionary_batches >= num_dictionary_deltas + num_replaced_dictionaries
Number of dictionary deltas read.
Number of replaced dictionaries (i.e.
where a dictionary batch replaces an existing dictionary with an unrelated new dictionary).
The IPC stream format is only optionally terminated, whereas the IPC file format must include a terminating footer. Thus a writer of the IPC file format must be explicitly finalized with Close() or the resulting file will be corrupt.
Create a new IPC stream writer from stream sink and schema.
User is responsible for closing the actual OutputStream.
sink â [in] output stream to write to
schema â [in] the schema of the record batches to be written
options â [in] options for serialization
Result<std::shared_ptr<RecordBatchWriter>>
Create a new IPC stream writer from stream sink and schema.
User is responsible for closing the actual OutputStream.
sink â [in] output stream to write to
schema â [in] the schema of the record batches to be written
options â [in] options for serialization
Result<std::shared_ptr<RecordBatchWriter>>
Create a new IPC file writer from stream sink and schema.
sink â [in] output stream to write to
schema â [in] the schema of the record batches to be written
options â [in] options for serialization, optional
metadata â [in] custom metadata for File Footer, optional
Result<std::shared_ptr<RecordBatchWriter>>
Create a new IPC file writer from stream sink and schema.
sink â [in] output stream to write to
schema â [in] the schema of the record batches to be written
options â [in] options for serialization, optional
metadata â [in] custom metadata for File Footer, optional
Result<std::shared_ptr<RecordBatchWriter>>
Abstract interface for writing a stream of record batches.
Subclassed by arrow::flight::MetadataRecordBatchWriter
Public Functions
Write a record batch to the stream.
batch â [in] the record batch to write to the stream
Write a record batch with custom metadata to the stream.
batch â [in] the record batch to write to the stream
custom_metadata â [in] the record batchâs custom metadata to write to the stream
Write possibly-chunked table by creating sequence of record batches.
table â [in] table to write
Write Table with a particular chunksize.
table â [in] table to write
max_chunksize â [in] maximum number of rows for table chunks. To indicate that no maximum should be enforced, pass -1.
Perform any logic necessary to finish the stream.
Return current write statistics.
Public Members
Number of IPC messages written.
Number of record batches written.
Number of dictionary batches written.
Note: num_dictionary_batches >= num_dictionary_deltas + num_replaced_dictionaries
Number of dictionary deltas written.
Number of replaced dictionaries (i.e.
where a dictionary batch replaces an existing dictionary with an unrelated new dictionary).
Total size in bytes of record batches emitted.
The ârawâ size counts the original buffer sizes, while the âserializedâ size includes padding and (optionally) compression.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4