Package contains interfaces used specifically for splittable DoFns.
This section is empty.
This section is empty.
This section is empty.
type BoundableRTracker interface {
RTracker
IsBounded() bool
}
BoundableRTracker is an interface used to interact with restrictions that may be bounded or unbounded while processing elements in splittable DoFns (specifically, in the ProcessElement method and TruncateRestriction method). Each BoundableRTracker tracks the progress of a single restriction.
All BoundableRTracker methods should be thread-safe for dynamic splits to function correctly.
LockRTracker is a restriction tracker that wraps another restriction tracker and adds thread safety to it by locking a mutex in each method, before delegating to the underlying tracker.
NewLockRTracker creates a LockRTracker initialized with the specified restriction tracker as its underlying restriction tracker.
GetError locks a mutex for thread safety, and then delegates to the underlying tracker's GetError.
GetProgress locks a mutex for thread safety, and then delegates to the underlying tracker's GetProgress.
GetRestriction locks a mutex for thread safety, and then delegates to the underlying tracker's GetRestriction.
IsBounded locks a mutex for thread safety, and then delegates to the underlying tracker's IsBounded(). If BoundableRTracker is not implemented then the RTracker is considered to be bounded by default.
IsDone locks a mutex for thread safety, and then delegates to the underlying tracker's IsDone.
TryClaim locks a mutex for thread safety, and then delegates to the underlying tracker's TryClaim.
TrySplit locks a mutex for thread safety, and then delegates to the underlying tracker's TrySplit.
type ManualWatermarkEstimator struct {
State time.Time
}
ManualWatermarkEstimator is a watermark estimator that advances the current DoFn's output watermark when a user calls UpdateWatermark from within ProcessElement.
CurrentWatermark returns the most recent timestamp set from ProcessElement. It is used by the Sdk harness to set the current DoFn's output watermark on splits and checkpoints.
UpdateWatermark is a convenience function that can be used to update the current watermark from inside ProcessElement.
type ProcessContinuation interface {
ShouldResume() bool
ResumeDelay() time.Duration
}
ProcessContinuation is an interface used to signal that a splittable DoFn should be split and resumed at a later time. The ProcessContinuation can be returned from a DoFn when it returns, either complete or needing to be resumed.
ResumeProcessingIn returns a ProcessContinuation that will resume the process later with a suggested delay passed as a time.Duration.
StopProcessing returns a ProcessContinuation that will not resume the process later.
RTracker is an interface used to interact with restrictions while processing elements in splittable DoFns (specifically, in the ProcessElement method). Each RTracker tracks the progress of a single restriction.
All RTracker methods should be thread-safe for dynamic splits to function correctly.
TimestampObservingEstimator is an interface used to represent a user defined watermark estimator that has the ability to observe timestamps of elements outputted from a ParDo's emit function.
type TimestampObservingWatermarkEstimator struct {
State time.Time
}
TimestampObservingWatermarkEstimator is a watermark estimator that advances the current DoFn's output watermark to the timestamp of the most recently emitted element.
CurrentWatermark returns the current watermark. It is used by the Sdk harness to set the current DoFn's output watermark on splits and checkpoints.
ObserveTimestamp returns updates the watermark to the timestamp of the most recently emitted element. It is invoked by the Sdk after each emit. The updated watermark will not be reflected until a split or checkpoint occurs.
type WallTimeWatermarkEstimator struct{}
WallTimeWatermarkEstimator is a watermark estimator that advances the current DoFn's output watermark to the current wallclock time on splits or checkpoints.
CurrentWatermark returns the current time. It is used by the Sdk harness to set the current DoFn's output watermark on splits and checkpoints.
type WatermarkEstimator interface {
CurrentWatermark() time.Time
}
WatermarkEstimator is an interface used to represent a user defined watermark estimator. Watermark estimators allow users to advance the output watermark of the current sdf.
WrappedTracker wraps an implementation of an RTracker and adds an IsBounded() function that returns true in order to allow RTrackers to be handled as bounded BoundableRTrackers if necessary (like in self-checkpointing evaluation.)
NewWrappedTracker is a constructor for an RTracker that wraps another RTracker into a BoundedRTracker.
IsBounded returns true, indicating that the underlying RTracker represents a bounded amount of work.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4