Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/expressions/Aggregator.html below:

Aggregator (Spark 4.0.0 JavaDoc)

org.apache.spark.sql.expressions.Aggregator<IN,BUF,OUT>

All Implemented Interfaces:

Serializable, org.apache.spark.sql.internal.UserDefinedFunctionLike

A base class for user-defined aggregations, which can be used in

operations to take all of the elements of a group and reduce them to a single value.

For example, the following aggregator extracts an int from a specific class and adds them up:

   case class Data(i: Int)

   val customSummer =  new Aggregator[Data, Int, Int] {
     def zero: Int = 0
     def reduce(b: Int, a: Data): Int = b + a.i
     def merge(b1: Int, b2: Int): Int = b1 + b2
     def finish(r: Int): Int = r
     def bufferEncoder: Encoder[Int] = Encoders.scalaInt
     def outputEncoder: Encoder[Int] = Encoders.scalaInt
   }.toColumn()

   val ds: Dataset[Data] = ...
   val aggregated = ds.select(customSummer)
 

Based loosely on Aggregator from Algebird: https://github.com/twitter/algebird

Since:

1.6.0

See Also:

• Serialized Form

• Constructor Summary

  Constructors

• Method Summary

  Specifies the Encoder for the intermediate value type.

  Transform the output of the reduction.

  Merge two intermediate values.

  Specifies the Encoder for the final output value type.

  Combine two values to produce a new value.

  Returns this Aggregator as a TypedColumn that can be used in Dataset operations.

  A zero value for this aggregation.

  Methods inherited from interface org.apache.spark.sql.internal.UserDefinedFunctionLikename

• Constructor Details
  • Aggregator

    public Aggregator()

• Method Details
  • bufferEncoder

    Specifies the Encoder for the intermediate value type.

    Returns:

    (undocumented)

    Since:

    2.0.0

  • finish public abstract OUT finish(BUF reduction)

    Transform the output of the reduction.

    Parameters:

    reduction - (undocumented)

    Returns:

    (undocumented)

    Since:

    1.6.0

  • merge

    Merge two intermediate values.

    Parameters:

    b1 - (undocumented)

    b2 - (undocumented)

    Returns:

    (undocumented)

    Since:

    1.6.0

  • outputEncoder

    Specifies the Encoder for the final output value type.

    Returns:

    (undocumented)

    Since:

    2.0.0

  • reduce

    Combine two values to produce a new value. For performance, the function may modify b and return it instead of constructing new object for b.

    Parameters:

    b - (undocumented)

    a - (undocumented)

    Returns:

    (undocumented)

    Since:

    1.6.0

  • toColumn

    Returns this Aggregator as a TypedColumn that can be used in Dataset operations.

    Returns:

    (undocumented)

    Since:

    1.6.0

  • zero public abstract BUF zero

    ()

    A zero value for this aggregation. Should satisfy the property that any b + zero = b.

    Returns:

    (undocumented)

    Since:

    1.6.0

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4