Package xlangx contains various low-level utilities needed for adding cross-language transforms to the pipeline.
const ( Separator = ":" ClasspathSeparator = ";" )
This section is empty.
func DecodeStructPayload(plBytes []byte) (interface{}, error)
DecodeStructPayload takes a marshaled ExternalConfigurationPayload proto and returns a native Go struct, with its type converted from the Schema representation and its value decoded from the Row.
func EncodeStructPayload(pl interface{}) ([]byte, error)
EncodeStructPayload takes a native Go struct and returns a marshaled ExternalConfigurationPayload proto, containing a Schema representation of the original type and the original value encoded as a Row. This is intended to be used as the expansion payload for an External transform.
func Expand ¶Expand expands an unexpanded graph.ExternalTransform as a graph.ExpandedTransform and assigns it to the ExternalTransform's Expanded field. This requires querying an expansion service based on the configuration details within the ExternalTransform.
For framework use only. Users should call beam.CrossLanguage to access foreign transforms rather than calling this function directly.
QueryAutomatedExpansionService submits an external transform to be expanded by the expansion service and then eagerly materializes the artifacts for staging. The given transform should be the external transform, and the components are any additional components necessary for the pipeline snippet.
The address to be queried is determined by the Config field of the HandlerParams after the prefix tag indicating the automated service is in use.
QueryExpansionService submits an external transform to be expanded by the expansion service. The given transform should be the external transform, and the components are any additional components necessary for the pipeline snippet.
The address to be queried is determined by the Config field of HandlerParams.
This HandlerFunc is exported to simplify building custom handler functions that do end up calling a Beam ExpansionService, either as a fallback or as part of normal flow.
func RegisterHandler ¶ added in v2.35.0RegisterHandler associates a namespace with a HandlerFunc which can be used to replace calls to a Beam ExpansionService.
Then, expansion addresses of the forms
"<namespace>" or "<namespace>:<configuration>"
can be used with beam.CrossLanguage. Any configuration after the separator is provided to the HandlerFunc on call for the handler func to use at it's leisure.
func RegisterOverrideForUrn(urn, expansionAddr string)
RegisterOverrideForUrn overrides which expansion address is used to expand a specific transform URN. The expansion address must be a URL or be a namespaced handler registered with RegisterHandler.
When the expansion address is for a handler, it may take the forms
"<namespace>" or "<namespace>:<configuration>"
Require takes an expansionAddr and requires cross language expansion to use it and it's associated handler. If the transform's urn has a specific override, it will be ignored.
Intended for use by cross language wrappers to permit per-call overrides of the expansion address within a single pipeline, such as for testing purposes.
ResolveArtifacts acquires all dependencies for a cross-language transform
ResolveArtifactsWithConfig acquires all dependencies for cross-language transforms, but with some additional configuration to behavior. By default, this function performs the following steps for each cross-language transform in the list of edges:
The changes that can be configured are documented in ResolveConfig.
This returns a map of "local path" to "sdk path". By default these are identical, unless ResolveConfig.SdkPath has been set.
UpdateArtifactTypeFromFileToURL changes the type of the artifact from FILE to URL when the file path contains the suffix element ("://") of the URI scheme.
UseAutomatedJavaExpansionService takes a gradle target and creates a tagged string to indicate that it should be used to start up an automated expansion service for a cross-language expansion.
Intended for use by cross language wrappers to permit spinning up an expansion service for a user if no expansion service address is provided.
type ExpansionServiceOption func(*string)
ExpansionServiceOption provides an option for xlangx.UseAutomatedJavaExpansionService()
AddClasspaths is an expansion service option for xlangx.UseAutomatedExpansionService that accepts a classpaths slice and creates a tagged expansion address string suffixed with classpath separator and classpaths provided.
type HandlerFunc ¶ added in v2.35.0HandlerFunc abstracts making an ExpansionService request.
type HandlerParams ¶ added in v2.35.0HandlerParams is the parameter to an expansion service handler.
func (*HandlerParams) CoderMarshaller ¶ added in v2.35.0CoderMarshaller returns a coder marshaller initialized with the request's namespace.
func (*HandlerParams) Inputs ¶ added in v2.35.0Inputs returns the provided input PCollections, if any, for the PTransform to expand in this expansion service request.
func (*HandlerParams) Outputs ¶ added in v2.35.0Outputs returns the provided output PCollections, if any, for expected outputs for this expansion service request.
If no collections are returned, none are currently expected, but may be provided by the expansion.
PCol represents input or output pcollections to the cross language transform being expanded.
ID produces a standard format globally namespaced id for a PCollection from the local identifier.
WSID produces a standard format globally namespaced id for a WindowingStrategy from the local identifier.
WindowingStrategy returns the id to this PCollection's windowing strategy, and the associated proto.
TODO: intern windowing strategies.
ResolveConfig contains fields for configuring the behavior for resolving artifacts.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4