org.apache.spark.sql.functions
public class functions extends ObjectCommonly used functions available for DataFrame operations. Using functions defined here provides a little bit more compile-time safety to make sure the function exists.
You can call the functions defined here by two ways: _FUNC_(...) and functions.expr("_FUNC_(...)").
As an example, regr_count is a function that is defined here. You can use regr_count(col("yCol", col("xCol"))) to invoke the regr_count function. This way the programming language's compiler ensures regr_count exists and is of the proper form. You can also use expr("regr_count(yCol, xCol)") function to invoke the same function. In this case, Spark itself will ensure regr_count exists when it analyzes the query.
You can find the entire list of functions at SQL API documentation of your Spark version, see also the latest list
This function APIs usually have methods with Column signature only because it can support not only Column but also other types such as a native string. The other variants currently exist for historical reasons.
Nested Classes
Constructors
Computes the absolute value of a numeric value.
Returns the date that is numMonths after startDate.
Returns the date that is numMonths after startDate.
Returns a decrypted value of input.
Returns a decrypted value of input.
Returns a decrypted value of input.
Returns a decrypted value of input using AES in mode with padding.
Returns an encrypted value of input.
Returns an encrypted value of input.
Returns an encrypted value of input.
Returns an encrypted value of input.
Returns an encrypted value of input using AES in given mode with the specified padding.
Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.
Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.
Aggregate function: returns true if at least one value of e is true.
Aggregate function: returns some value of e for a group of rows.
Aggregate function: returns some value of e for a group of rows.
Aggregate function: returns the approximate number of distinct items in a group.
Aggregate function: returns the approximate number of distinct items in a group.
Aggregate function: returns the approximate number of distinct items in a group.
Aggregate function: returns the approximate number of distinct items in a group.
Aggregate function: returns the approximate percentile of the numeric column col which is the smallest value in the ordered col values (sorted from least to greatest) such that no more than percentage of col values is less than the value or equal to that value.
Creates a new array column.
Creates a new array column.
Creates a new array column.
Creates a new array column.
Aggregate function: returns a list of objects with duplicates.
Returns an ARRAY containing all elements from the source ARRAY as well as the new element.
Remove all null elements from the given array.
Returns null if the array is null, true if the array contains value, and false otherwise.
Removes duplicate values from the array.
Returns an array of the elements in the first array but not in the second array, without duplicates.
Adds an item into a given array at a specified position
Returns an array of the elements in the intersection of the given two arrays, without duplicates.
Concatenates the elements of column using the delimiter.
Concatenates the elements of column using the delimiter.
Returns the maximum value in the array.
Returns the minimum value in the array.
Locates the position of the first occurrence of the value in the given array as long.
Returns an array containing value as well as all elements from array.
Remove all elements that equal to element from the given array.
Creates an array containing the left argument repeated the number of times given by the right argument.
Creates an array containing the left argument repeated the number of times given by the right argument.
Returns the total number of elements in the array.
Sorts the input array in ascending order.
Sorts the input array based on the given comparator function.
Returns an array of the elements in the union of the given two arrays, without duplicates.
Returns true if a1 and a2 have at least one non-null element in common.
Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.
Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.
Returns a sort expression based on ascending order of the column.
Returns a sort expression based on ascending order of the column, and null values return before non-null values.
Returns a sort expression based on ascending order of the column, and null values appear after non-null values.
Computes the numeric value of the first character of the string column, and returns the result as an int column.
Returns null if the condition is true, and throws an exception otherwise.
Returns null if the condition is true; throws an exception with the error message otherwise.
Aggregate function: returns the average of the values in a group.
Aggregate function: returns the average of the values in a group.
Computes the BASE64 encoding of a binary column and returns it as a string column.
An expression that returns the string representation of the binary value of the given long column.
An expression that returns the string representation of the binary value of the given long column.
Aggregate function: returns the bitwise AND of all non-null input values, or null if none.
Returns the number of bits that are set in the argument expr as an unsigned 64-bit integer, or NULL if the argument is NULL.
Returns the value of the bit (0 or 1) at the specified position.
Calculates the bit length for the specified string column.
Aggregate function: returns the bitwise OR of all non-null input values, or null if none.
Aggregate function: returns the bitwise XOR of all non-null input values, or null if none.
Returns the bucket number for the given input column.
Returns the bit position for the given input column.
Returns a bitmap with the positions of the bits set from all the values from the input column.
Returns the number of set bits in the input bitmap.
Returns a bitmap that is the bitwise OR of all of the bitmaps from the input column.
Computes bitwise NOT (~) of a number.
Aggregate function: returns true if all values of e are true.
Aggregate function: returns true if at least one value of e is true.
Marks a DataFrame as small enough for use in broadcast joins.
Returns the value of the column e rounded to 0 decimal places with HALF_EVEN round mode.
Round the value of e to scale decimal places with HALF_EVEN round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.
Round the value of e to scale decimal places with HALF_EVEN round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.
Removes the leading and trailing space characters from str.
Remove the leading and trailing trim characters from str.
(Java-specific) A transform for any type that partitions by a hash of the input column.
(Java-specific) A transform for any type that partitions by a hash of the input column.
Call an user-defined function.
Call an user-defined function.
Call an user-defined function.
Returns length of array or map.
Computes the cube-root of the given column.
Computes the cube-root of the given value.
Computes the ceiling of the given value of e to 0 decimal places.
Computes the ceiling of the given value of e to 0 decimal places.
Computes the ceiling of the given value of e to scale decimal places.
Computes the ceiling of the given value of e to 0 decimal places.
Computes the ceiling of the given value of e to scale decimal places.
Returns the character length of string data or number of bytes of binary data.
Returns the character length of string data or number of bytes of binary data.
Returns the ASCII character having the binary equivalent to n.
Returns the first column that is not null, or null if all inputs are null.
Returns the first column that is not null, or null if all inputs are null.
Returns a
Column
based on the given column name.
Marks a given column with specified collation.
Returns the collation name of a given column.
Aggregate function: returns a list of objects with duplicates.
Aggregate function: returns a list of objects with duplicates.
Aggregate function: returns a set of objects with duplicate elements eliminated.
Aggregate function: returns a set of objects with duplicate elements eliminated.
Returns a
Column
based on the given column name.
Concatenates multiple input columns together into a single column.
Concatenates multiple input columns together into a single column.
Concatenates multiple input string columns together into a single string column, using the given separator.
Concatenates multiple input string columns together into a single string column, using the given separator.
Convert a number in a string column from one base to another.
Converts the timestamp without time zone sourceTs from the current time zone to targetTz.
Converts the timestamp without time zone sourceTs from the sourceTz time zone to targetTz.
Aggregate function: returns the Pearson Correlation Coefficient for two columns.
Aggregate function: returns the Pearson Correlation Coefficient for two columns.
Aggregate function: returns the number of items in a group.
Aggregate function: returns the number of items in a group.
Aggregate function: returns the number of distinct items in a group.
Aggregate function: returns the number of distinct items in a group.
Aggregate function: returns the number of TRUE values for the expression.
Returns a count-min sketch of a column with the given esp, confidence and seed.
Returns a count-min sketch of a column with the given esp, confidence and seed.
Aggregate function: returns the number of distinct items in a group.
Aggregate function: returns the number of distinct items in a group.
Aggregate function: returns the number of distinct items in a group.
Aggregate function: returns the number of distinct items in a group.
Aggregate function: returns the population covariance for two columns.
Aggregate function: returns the population covariance for two columns.
Aggregate function: returns the sample covariance for two columns.
Aggregate function: returns the sample covariance for two columns.
Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint.
Window function: returns the cumulative distribution of values within a window partition, i.e.
Returns the current date at the start of query evaluation as a date column.
Returns the current catalog.
Returns the current database.
Returns the current date at the start of query evaluation as a date column.
Returns the current schema.
Returns the current timestamp at the start of query evaluation as a timestamp column.
Returns the current session local timezone.
Returns the user name of current execution context.
Returns the date that is days days after start
Returns the date that is days days after start
Returns the number of days from start to end.
Converts a date/timestamp/string to a value of string in the format specified by the date format given by the second argument.
Create date from the number of days since 1970-01-01.
Extracts a part of the date/timestamp or interval source.
Returns the date that is days days before start
Returns the date that is days days before start
Returns timestamp truncated to the unit specified by the format.
Returns the date that is days days after start
Returns the number of days from start to end.
Extracts a part of the date/timestamp or interval source.
Extracts the day of the month as an integer from a given date/timestamp/string.
Extracts the three-letter abbreviated day name from a given date/timestamp/string.
Extracts the day of the month as an integer from a given date/timestamp/string.
Extracts the day of the week as an integer from a given date/timestamp/string.
Extracts the day of the year as an integer from a given date/timestamp/string.
(Java-specific) A transform for timestamps and dates to partition data into days.
Computes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32').
Converts an angle measured in radians to an approximately equivalent angle measured in degrees.
Converts an angle measured in radians to an approximately equivalent angle measured in degrees.
Window function: returns the rank of rows within a window partition, without any gaps.
Returns a sort expression based on the descending order of the column.
Returns a sort expression based on the descending order of the column, and null values appear before non-null values.
Returns a sort expression based on the descending order of the column, and null values appear after non-null values.
Returns element of array at given index in value if column is array.
Returns the n-th input, e.g., returns input2 when n is 2.
elt(scala.collection.immutable.Seq<Column> inputs)
Returns the n-th input, e.g., returns input2 when n is 2.
Computes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32').
Returns same result as the EQUAL(=) operator for non-null operands, but returns true if both are null, false if one of the them is null.
Aggregate function: returns true if all values of e are true.
Returns whether a predicate holds for one or more elements in the array.
Computes the exponential of the given column.
Computes the exponential of the given value.
Creates a new row for each element in the given array or map column.
Creates a new row for each element in the given array or map column.
Computes the exponential of the given column minus one.
Computes the exponential of the given value minus one.
Extracts a part of the date/timestamp or interval source.
Computes the factorial of the given value.
Returns an array of elements for which a predicate holds in a given array.
Returns an array of elements for which a predicate holds in a given array.
Returns the index (1-based) of the given string (str) in the comma-delimited list (strArray).
Aggregate function: returns the first value of a column in a group.
Aggregate function: returns the first value of a column in a group.
Aggregate function: returns the first value in a group.
Aggregate function: returns the first value in a group.
Aggregate function: returns the first value in a group.
Aggregate function: returns the first value in a group.
Creates a single array from an array of arrays.
Computes the floor of the given column value to 0 decimal places.
Computes the floor of the given value of e to 0 decimal places.
Computes the floor of the given value of e to scale decimal places.
Returns whether a predicate holds for every element in the array.
Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with HALF_EVEN round mode, and returns the result as a string column.
Formats the arguments in printf-style and returns the result as a string column.
Formats the arguments in printf-style and returns the result as a string column.
(Java-specific) Parses a column containing a CSV string into a StructType with the specified schema.
Parses a column containing a CSV string into a StructType with the specified schema.
(Java-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.
(Scala-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.
(Scala-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType of StructTypes with the specified schema.
(Java-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType of StructTypes with the specified schema.
Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.
(Java-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.
(Scala-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.
Parses a column containing a JSON string into a StructType with the specified schema.
(Java-specific) Parses a column containing a JSON string into a StructType with the specified schema.
(Scala-specific) Parses a column containing a JSON string into a StructType with the specified schema.
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the yyyy-MM-dd HH:mm:ss format.
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone.
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone.
(Java-specific) Parses a column containing a XML string into a StructType with the specified schema.
(Java-specific) Parses a column containing a XML string into a StructType with the specified schema.
(Java-specific) Parses a column containing a XML string into a StructType with the specified schema.
Parses a column containing a XML string into the data type corresponding to the specified schema.
Parses a column containing a XML string into the data type corresponding to the specified schema.
Returns element of array at given (0-based) index.
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object.
Returns the value of the bit (0 or 1) at the specified position.
Returns the greatest value of the list of column names, skipping null values.
Returns the greatest value of the list of column names, skipping null values.
Returns the greatest value of the list of values, skipping null values.
Returns the greatest value of the list of values, skipping null values.
Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set.
Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set.
Aggregate function: returns the level of grouping, equals to
Aggregate function: returns the level of grouping, equals to
Aggregate function: returns the level of grouping, equals to
Aggregate function: returns the level of grouping, equals to
Calculates the hash code of given columns, and returns the result as an int column.
Calculates the hash code of given columns, and returns the result as an int column.
Computes hex value of the given column.
Aggregate function: computes a histogram on numeric 'expr' using nb bins.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.
Returns the estimated number of unique values given the binary representation of a Datasketches HllSketch.
Returns the estimated number of unique values given the binary representation of a Datasketches HllSketch.
Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.
Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.
Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.
Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.
Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.
Extracts the hours as an integer from a given date/timestamp/string.
(Java-specific) A transform for timestamps to partition data into hours.
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
Returns col2 if col1 is null, or col1 otherwise.
Returns true if str matches pattern with escapeChar('\') case-insensitively, null if any arguments are null, false otherwise.
Returns true if str matches pattern with escapeChar case-insensitively, null if any arguments are null, false otherwise.
Returns a new string column by converting the first letter of each word to uppercase.
Creates a new row for each element in the given array of structs.
Creates a new row for each element in the given array of structs.
Returns the length of the block being read, or -1 if not available.
Returns the start offset of the block being read, or -1 if not available.
Creates a string column for the file name of the current Spark task.
Locate the position of the first occurrence of substr column in the given string.
Locate the position of the first occurrence of substr column in the given string.
Returns true if the input is a valid UTF-8 string, otherwise returns false.
Check if a variant value is a variant null.
Return true iff the column is NaN.
Returns true if col is not null, or false otherwise.
Return true iff the column is null.
Calls a method with reflection.
Calls a method with reflection.
Returns the number of elements in the outermost JSON array.
Returns all the keys of the outermost JSON object as an array.
Creates a new row for a json column according to the given field names.
Creates a new row for a json column according to the given field names.
Aggregate function: returns the kurtosis of the values in a group.
Aggregate function: returns the kurtosis of the values in a group.
Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row.
Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.
Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row.
Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.
Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.
Aggregate function: returns the last value of the column in a group.
Aggregate function: returns the last value of the column in a group.
Aggregate function: returns the last value in a group.
Aggregate function: returns the last value in a group.
Returns the last day of the month which the given date belongs to.
Aggregate function: returns the last value in a group.
Aggregate function: returns the last value in a group.
Returns str with all characters changed to lowercase.
Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row.
Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.
Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row.
Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.
Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.
Returns the least value of the list of column names, skipping null values.
Returns the least value of the list of column names, skipping null values.
Returns the least value of the list of values, skipping null values.
Returns the least value of the list of values, skipping null values.
Returns the leftmost len(len can be string type) characters from the string str, if len is less or equal than 0 the result is an empty string.
Computes the character length of a given string or number of bytes of a binary string.
Computes the character length of a given string or number of bytes of a binary string.
Computes the Levenshtein distance of the two given string columns.
Computes the Levenshtein distance of the two given string columns if it's less than or equal to a given threshold.
Returns true if str matches pattern with escapeChar('\'), null if any arguments are null, false otherwise.
Returns true if str matches pattern with escapeChar, null if any arguments are null, false otherwise.
Aggregate function: returns the concatenation of non-null input values.
Aggregate function: returns the concatenation of non-null input values, separated by the delimiter.
Aggregate function: returns the concatenation of distinct non-null input values.
Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter.
Creates a
Column
of literal value.
Computes the natural logarithm of the given value.
Returns the current timestamp without time zone at the start of query evaluation as a timestamp without time zone column.
Locate the position of the first occurrence of substr.
Locate the position of the first occurrence of substr in a string column, after position pos.
Returns the first argument-base logarithm of the second argument.
Returns the first argument-base logarithm of the second argument.
Computes the natural logarithm of the given column.
Computes the natural logarithm of the given value.
Computes the logarithm of the given value in base 10.
Computes the logarithm of the given value in base 10.
Computes the natural logarithm of the given column plus one.
Computes the natural logarithm of the given value plus one.
Computes the logarithm of the given value in base 2.
Computes the logarithm of the given column in base 2.
Converts a string column to lower case.
Left-pad the binary column with pad to a byte length of len.
Left-pad the string column with pad to a length of len.
Left-pad the string column with pad to a length of len.
Trim the spaces from left end for the specified string value.
Trim the specified character string from left end for the specified string column.
Trim the specified character string from left end for the specified string column.
Make DayTimeIntervalType duration.
Make DayTimeIntervalType duration from days.
Make DayTimeIntervalType duration from days and hours.
Make DayTimeIntervalType duration from days, hours and mins.
Make DayTimeIntervalType duration from days, hours, mins and secs.
Make interval from years.
Make interval from years and months.
Make interval from years, months and weeks.
Make interval from years, months, weeks and days.
Make interval from years, months, weeks, days and hours.
Make interval from years, months, weeks, days, hours and mins.
Make interval from years, months, weeks, days, hours, mins and secs.
Create timestamp from years, months, days, hours, mins and secs fields.
Create timestamp from years, months, days, hours, mins, secs and timezone fields.
Create the current timestamp with local time zone from years, months, days, hours, mins and secs fields.
Create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields.
Create local date-time from years, months, days, hours, mins, secs fields.
Returns a new string in which all invalid UTF-8 byte sequences, if any, are replaced by the Unicode replacement character (U+FFFD).
Make year-month interval.
Make year-month interval from years.
Make year-month interval from years, months.
Creates a new map column.
map(scala.collection.immutable.Seq<Column> cols)
Creates a new map column.
Returns the union of all the given maps.
Returns the union of all the given maps.
Returns true if the map contains the key.
Returns an unordered array of all entries in the given map.
Returns a map whose key-value pairs satisfy a predicate.
Creates a new map column.
Returns a map created from the given array of entries.
Returns an unordered array containing the keys of the map.
Returns an unordered array containing the values of the map.
Merge two given maps, key-wise into a single map using a function.
Masks the given string value.
Masks the given string value.
Masks the given string value.
Masks the given string value.
Masks the given string value.
Aggregate function: returns the maximum value of the column in a group.
Aggregate function: returns the maximum value of the expression in a group.
Aggregate function: returns the value associated with the maximum value of ord.
Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string.
Aggregate function: returns the average of the values in a group.
Aggregate function: returns the average of the values in a group.
Aggregate function: returns the median of the values in a group.
Aggregate function: returns the minimum value of the column in a group.
Aggregate function: returns the minimum value of the expression in a group.
Aggregate function: returns the value associated with the minimum value of ord.
Extracts the minutes as an integer from a given date/timestamp/string.
Aggregate function: returns the most frequent value in a group.
Aggregate function: returns the most frequent value in a group.
A column expression that generates monotonically increasing 64-bit integers.
Extracts the month as an integer from a given date/timestamp/string.
Extracts the three-letter abbreviated month name from a given date/timestamp/string.
(Java-specific) A transform for timestamps and dates to partition data into months.
Returns number of months between dates start and end.
Returns number of months between dates end and start.
Creates a struct with the given field names and values.
Creates a struct with the given field names and values.
Returns col1 if it is not NaN, or col2 if col1 is NaN.
Returns the negated value.
Returns the first date which is later than the value of the date column that is on the specified day of the week.
Returns the first date which is later than the value of the date column that is on the specified day of the week.
Inversion of boolean expression, i.e.
Returns the current timestamp at the start of query evaluation.
Window function: returns the value that is the offsetth row of the window frame (counting from 1), and null if the size of window frame is less than offset rows.
Window function: returns the value that is the offsetth row of the window frame (counting from 1), and null if the size of window frame is less than offset rows.
Window function: returns the ntile group id (from 1 to n inclusive) in an ordered window partition.
Returns null if col1 equals to col2, or col1 otherwise.
Returns null if col is equal to zero, or col otherwise.
Returns col2 if col1 is null, or col1 otherwise.
Returns col2 if col1 is not null, or col3 otherwise.
Calculates the byte length for the specified string column.
Overlay the specified portion of src with replace, starting from byte position pos of src.
Overlay the specified portion of src with replace, starting from byte position pos of src and proceeding for len bytes.
Parses a JSON string and constructs a Variant value.
Extracts a part from a URL.
Extracts a part from a URL.
Window function: returns the relative rank (i.e.
Aggregate function: returns the exact percentile(s) of numeric column expr at the given percentage(s) with value range in [0.0, 1.0].
Aggregate function: returns the exact percentile(s) of numeric column expr at the given percentage(s) with value range in [0.0, 1.0].
Aggregate function: returns the approximate percentile of the numeric column col which is the smallest value in the ordered col values (sorted from least to greatest) such that no more than percentage of col values is less than the value or equal to that value.
Returns the positive value of dividend mod divisor.
Creates a new row for each element with position in the given array or map column.
Creates a new row for each element with position in the given array or map column.
Returns the position of the first occurrence of substr in str after position 1.
Returns the position of the first occurrence of substr in str after position start.
Returns the value of the first argument raised to the power of the second argument.
Returns the value of the first argument raised to the power of the second argument.
Returns the value of the first argument raised to the power of the second argument.
Returns the value of the first argument raised to the power of the second argument.
Returns the value of the first argument raised to the power of the second argument.
Returns the value of the first argument raised to the power of the second argument.
Returns the value of the first argument raised to the power of the second argument.
Returns the value of the first argument raised to the power of the second argument.
Returns the value of the first argument raised to the power of the second argument.
Formats the arguments in printf-style and returns the result as a string column.
Formats the arguments in printf-style and returns the result as a string column.
Aggregate function: returns the product of all numerical elements in a group.
Extracts the quarter as an integer from a given date/timestamp/string.
Converts an angle measured in degrees to an approximately equivalent angle measured in radians.
Converts an angle measured in degrees to an approximately equivalent angle measured in radians.
Throws an exception with the provided error message.
Generate a random column with independent and identically distributed (i.i.d.) samples uniformly distributed in [0.0, 1.0).
Generate a random column with independent and identically distributed (i.i.d.) samples uniformly distributed in [0.0, 1.0).
Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.
Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.
Returns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).
Returns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).
Returns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z.
Returns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z, with the chosen random seed.
Window function: returns the rank of rows within a window partition.
Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.
Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.
Calls a method with reflection.
Calls a method with reflection.
Returns true if str matches regexp, or false otherwise.
Returns a count of the number of times that the regular expression pattern regexp is matched in the string str.
Extract a specific group matched by a Java regex, from the specified string column.
Extract all strings in the str that match the regexp expression and corresponding to the first regex group index.
Extract all strings in the str that match the regexp expression and corresponding to the regex group index.
Searches a string for a regular expression and returns an integer that indicates the beginning position of the matched substring.
Searches a string for a regular expression and returns an integer that indicates the beginning position of the matched substring.
Returns true if str matches regexp, or false otherwise.
Replace all substrings of the specified string value that match regexp with rep.
Replace all substrings of the specified string value that match regexp with rep.
Returns the substring that matches the regular expression regexp within the string str.
Aggregate function: returns the average of the independent variable for non-null pairs in a group, where y is the dependent variable and x is the independent variable.
Aggregate function: returns the average of the independent variable for non-null pairs in a group, where y is the dependent variable and x is the independent variable.
Aggregate function: returns the number of non-null number pairs in a group, where y is the dependent variable and x is the independent variable.
Aggregate function: returns the intercept of the univariate linear regression line for non-null pairs in a group, where y is the dependent variable and x is the independent variable.
Aggregate function: returns the coefficient of determination for non-null pairs in a group, where y is the dependent variable and x is the independent variable.
Aggregate function: returns the slope of the linear regression line for non-null pairs in a group, where y is the dependent variable and x is the independent variable.
Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(x) for non-null pairs in a group, where y is the dependent variable and x is the independent variable.
Aggregate function: returns REGR_COUNT(y, x) * COVAR_POP(y, x) for non-null pairs in a group, where y is the dependent variable and x is the independent variable.
Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(y) for non-null pairs in a group, where y is the dependent variable and x is the independent variable.
Repeats a string column n times, and returns it as a new string column.
Repeats a string column n times, and returns it as a new string column.
Replaces all occurrences of search with replace.
Replaces all occurrences of search with replace.
Returns a reversed string or an array with reverse order of elements.
Returns the rightmost len(len can be string type) characters from the string str, if len is less or equal than 0 the result is an empty string.
Returns the double value that is closest in value to the argument and is equal to a mathematical integer.
Returns the double value that is closest in value to the argument and is equal to a mathematical integer.
Returns true if str matches regexp, or false otherwise.
Returns the value of the column e rounded to 0 decimal places with HALF_UP round mode.
Round the value of e to scale decimal places with HALF_UP round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.
Round the value of e to scale decimal places with HALF_UP round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.
Window function: returns a sequential number starting at 1 within a window partition.
Right-pad the binary column with pad to a byte length of len.
Right-pad the string column with pad to a length of len.
Right-pad the string column with pad to a length of len.
Trim the spaces from right end for the specified string value.
Trim the specified character string from right end for the specified string column.
Trim the specified character string from right end for the specified string column.
Parses a CSV string and infers its schema in DDL format.
Parses a CSV string and infers its schema in DDL format.
Parses a CSV string and infers its schema in DDL format using options.
Parses a JSON string and infers its schema in DDL format.
Parses a JSON string and infers its schema in DDL format.
Parses a JSON string and infers its schema in DDL format using options.
Returns schema in the SQL format of a variant.
Returns the merged schema in the SQL format of a variant column.
Parses a XML string and infers its schema in DDL format.
Parses a XML string and infers its schema in DDL format.
Parses a XML string and infers its schema in DDL format using options.
Extracts the seconds as an integer from a given date/timestamp/string.
Splits a string into arrays of sentences, where each sentence is an array of words.
Splits a string into arrays of sentences, where each sentence is an array of words.
Splits a string into arrays of sentences, where each sentence is an array of words.
Generate a sequence of integers from start to stop, incrementing by 1 if start is less than or equal to stop, otherwise -1.
Generate a sequence of integers from start to stop, incrementing by step.
Returns the user name of current execution context.
Generates session window given a timestamp specifying column.
Generates session window given a timestamp specifying column.
Returns a sha1 hash value as a hex string of the col.
Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.
Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.
Shift the given value numBits left.
(Signed) shift the given value numBits right.
Unsigned shift the given value numBits right.
Returns a random permutation of the given array.
Returns a random permutation of the given array.
Computes the signum of the given value.
Computes the signum of the given column.
Computes the signum of the given value.
Returns length of array or map.
Aggregate function: returns the skewness of the values in a group.
Aggregate function: returns the skewness of the values in a group.
Returns an array containing all the elements in x from index start (or starting from the end if start is negative) with the specified length.
Returns an array containing all the elements in x from index start (or starting from the end if start is negative) with the specified length.
Aggregate function: returns true if at least one value of e is true.
Sorts the input array for the given column in ascending order, according to the natural ordering of the array elements.
Sorts the input array for the given column in ascending or descending order, according to the natural ordering of the array elements.
Returns the soundex code for the specified expression.
Splits str around matches of the given pattern.
Splits str around matches of the given pattern.
Splits str around matches of the given pattern.
Splits str around matches of the given pattern.
Splits str by delimiter and return requested part of the split (1-based).
Computes the square root of the specified float value.
Computes the square root of the specified float value.
Separates col1, ..., colk into n rows.
Separates col1, ..., colk into n rows.
Aggregate function: alias for stddev_samp.
Aggregate function: alias for stddev_samp.
Aggregate function: alias for stddev_samp.
Aggregate function: returns the population standard deviation of the expression in a group.
Aggregate function: returns the population standard deviation of the expression in a group.
Aggregate function: returns the sample standard deviation of the expression in a group.
Aggregate function: returns the sample standard deviation of the expression in a group.
Creates a map after splitting the text into key/value pairs using delimiters.
Creates a map after splitting the text into key/value pairs using delimiters.
Creates a map after splitting the text into key/value pairs using delimiters.
Aggregate function: returns the concatenation of non-null input values.
Aggregate function: returns the concatenation of non-null input values, separated by the delimiter.
Aggregate function: returns the concatenation of distinct non-null input values.
Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter.
Creates a new struct column that composes multiple input columns.
Creates a new struct column that composes multiple input columns.
Creates a new struct column.
Creates a new struct column.
Returns the substring of str that starts at pos, or the slice of byte array that starts at pos.
Returns the substring of str that starts at pos and is of length len, or the slice of byte array that starts at pos and is of length len.
Substring starts at pos and is of length len when str is String type or returns the slice of byte array that starts at pos in byte and is of length len when str is Binary type
Substring starts at pos and is of length len when str is String type or returns the slice of byte array that starts at pos in byte and is of length len when str is Binary type
Returns the substring from string str before count occurrences of the delimiter delim.
Aggregate function: returns the sum of all values in the given column.
Aggregate function: returns the sum of all values in the expression.
Aggregate function: returns the sum of distinct values in the expression.
Adds the specified number of units to the given timestamp.
Gets the difference between the timestamps in the specified units by truncating the fraction part.
Creates timestamp from the number of microseconds since UTC epoch.
Creates timestamp from the number of milliseconds since UTC epoch.
Converts the number of seconds from the Unix epoch (1970-01-01T00:00:00Z) to a timestamp.
Converts the input e to a binary value based on the default format "hex".
Converts the input e to a binary value based on the supplied format.
Convert e to a string based on the format.
Converts a column containing a StructType into a CSV string with the specified schema.
(Java-specific) Converts a column containing a StructType into a CSV string with the specified schema.
Converts the column into DateType by casting rules to DateType.
Converts the column into a DateType with a specified format
Converts a column containing a StructType, ArrayType or a MapType into a JSON string with the specified schema.
(Java-specific) Converts a column containing a StructType, ArrayType or a MapType into a JSON string with the specified schema.
(Scala-specific) Converts a column containing a StructType, ArrayType or a MapType into a JSON string with the specified schema.
Convert string 'e' to a number based on the string format 'format'.
Converts to a timestamp by casting rules to TimestampType.
Converts time string with the given pattern to timestamp.
Parses the timestamp expression with the default format to a timestamp without time zone.
Parses the timestamp expression with the format expression to a timestamp without time zone.
Parses the timestamp expression with the default format to a timestamp without time zone.
Parses the timestamp_str expression with the format expression to a timestamp without time zone.
Returns the UNIX timestamp of the given time.
Returns the UNIX timestamp of the given time.
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC.
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC.
Convert e to a string based on the format.
Converts a column containing nested inputs (array/map/struct) into a variants where maps and structs are converted to variant objects which are unordered unlike SQL structs.
Converts a column containing a StructType into a XML string with the specified schema.
(Java-specific) Converts a column containing a StructType into a XML string with the specified schema.
Returns an array of elements after applying a transformation to each element in the input array.
Returns an array of elements after applying a transformation to each element in the input array.
Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new keys for the pairs.
Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new values for the pairs.
Translate any character in the src by a character in replaceString.
Trim the spaces from both ends for the specified string column.
Trim the specified character from both ends for the specified string column.
Trim the specified character from both ends for the specified string column.
Returns date truncated to the unit specified by the format.
Returns the sum of left and right and the result is null on overflow.
Returns a decrypted value of input.
Returns a decrypted value of input.
Returns a decrypted value of input.
This is a special version of aes_decrypt that performs the same operation, but returns a NULL value instead of raising an error if the decryption cannot be performed.
Returns the mean calculated from values of a group and the result is null on overflow.
Returns dividend/divisor.
(array, index) - Returns element of array at given (1-based) index.
This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
Try to create a timestamp from years, months, days, hours, mins, and secs fields.
Try to create a timestamp from years, months, days, hours, mins, secs and timezone fields.
Try to create the current timestamp with local time zone from years, months, days, hours, mins and secs fields.
Try to create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields.
Try to create a local date-time from years, months, days, hours, mins, secs fields.
Returns the remainder of dividend/divisor.
Returns left*right and the result is null on overflow.
Parses a JSON string and constructs a Variant value.
Extracts a part from a URL.
Extracts a part from a URL.
This is a special version of reflect that performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.
This is a special version of reflect that performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.
Returns left-right and the result is null on overflow.
Returns the sum calculated from values of a group and the result is null on overflow.
This is a special version of to_binary that performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.
This is a special version of to_binary that performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.
Convert string e to a number based on the string format format.
Parses the s to a timestamp.
Parses the s with the format to a timestamp.
This is a special version of url_decode that performs the same operation, but returns a NULL value instead of raising an error if the decoding cannot be performed.
Returns the input value if it corresponds to a valid UTF-8 string, or NULL otherwise.
Extracts a sub-variant from v according to path string, and then cast the sub-variant to targetType.
Extracts a sub-variant from v according to path column, and then cast the sub-variant to targetType.
typedlit(T literal, scala.reflect.api.TypeTags.TypeTag<T> evidence$2)
Creates a
Column
of literal value.
typedLit(T literal, scala.reflect.api.TypeTags.TypeTag<T> evidence$1)
Creates a
Column
of literal value.
Return DDL-formatted type string for the data type of the input.
Returns str with all characters changed to uppercase.
Obtains a UserDefinedFunction that wraps the given Aggregator so that it may be used with untyped Data Frames.
udaf(Aggregator<IN,BUF,OUT> agg, scala.reflect.api.TypeTags.TypeTag<IN> evidence$3)
Obtains a UserDefinedFunction that wraps the given Aggregator so that it may be used with untyped Data Frames.
Defines a Java UDF0 instance as user-defined function (UDF).
Defines a Java UDF1 instance as user-defined function (UDF).
udf(UDF10<?,?,?,?,?,?,?,?,?,?,?> f, DataType returnType)
Defines a Java UDF10 instance as user-defined function (UDF).
Defines a Java UDF2 instance as user-defined function (UDF).
Defines a Java UDF3 instance as user-defined function (UDF).
Defines a Java UDF4 instance as user-defined function (UDF).
Defines a Java UDF5 instance as user-defined function (UDF).
Defines a Java UDF6 instance as user-defined function (UDF).
Defines a Java UDF7 instance as user-defined function (UDF).
Defines a Java UDF8 instance as user-defined function (UDF).
udf(UDF9<?,?,?,?,?,?,?,?,?,?> f, DataType returnType)
Defines a Java UDF9 instance as user-defined function (UDF).
udf(scala.Function0<RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$4)
Defines a Scala closure of 0 arguments as user-defined function (UDF).
udf(scala.Function1<A1,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$5, scala.reflect.api.TypeTags.TypeTag<A1> evidence$6)
Defines a Scala closure of 1 arguments as user-defined function (UDF).
static <RT, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10>
UserDefinedFunction udf(scala.Function10<A1,A2,A3,A4,A5,A6,A7,A8,A9,A10,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$59, scala.reflect.api.TypeTags.TypeTag<A1> evidence$60, scala.reflect.api.TypeTags.TypeTag<A2> evidence$61, scala.reflect.api.TypeTags.TypeTag<A3> evidence$62, scala.reflect.api.TypeTags.TypeTag<A4> evidence$63, scala.reflect.api.TypeTags.TypeTag<A5> evidence$64, scala.reflect.api.TypeTags.TypeTag<A6> evidence$65, scala.reflect.api.TypeTags.TypeTag<A7> evidence$66, scala.reflect.api.TypeTags.TypeTag<A8> evidence$67, scala.reflect.api.TypeTags.TypeTag<A9> evidence$68, scala.reflect.api.TypeTags.TypeTag<A10> evidence$69)
Defines a Scala closure of 10 arguments as user-defined function (UDF).
udf(scala.Function2<A1,A2,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$7, scala.reflect.api.TypeTags.TypeTag<A1> evidence$8, scala.reflect.api.TypeTags.TypeTag<A2> evidence$9)
Defines a Scala closure of 2 arguments as user-defined function (UDF).
udf(scala.Function3<A1,A2,A3,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$10, scala.reflect.api.TypeTags.TypeTag<A1> evidence$11, scala.reflect.api.TypeTags.TypeTag<A2> evidence$12, scala.reflect.api.TypeTags.TypeTag<A3> evidence$13)
Defines a Scala closure of 3 arguments as user-defined function (UDF).
udf(scala.Function4<A1,A2,A3,A4,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$14, scala.reflect.api.TypeTags.TypeTag<A1> evidence$15, scala.reflect.api.TypeTags.TypeTag<A2> evidence$16, scala.reflect.api.TypeTags.TypeTag<A3> evidence$17, scala.reflect.api.TypeTags.TypeTag<A4> evidence$18)
Defines a Scala closure of 4 arguments as user-defined function (UDF).
udf(scala.Function5<A1,A2,A3,A4,A5,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$19, scala.reflect.api.TypeTags.TypeTag<A1> evidence$20, scala.reflect.api.TypeTags.TypeTag<A2> evidence$21, scala.reflect.api.TypeTags.TypeTag<A3> evidence$22, scala.reflect.api.TypeTags.TypeTag<A4> evidence$23, scala.reflect.api.TypeTags.TypeTag<A5> evidence$24)
Defines a Scala closure of 5 arguments as user-defined function (UDF).
udf(scala.Function6<A1,A2,A3,A4,A5,A6,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$25, scala.reflect.api.TypeTags.TypeTag<A1> evidence$26, scala.reflect.api.TypeTags.TypeTag<A2> evidence$27, scala.reflect.api.TypeTags.TypeTag<A3> evidence$28, scala.reflect.api.TypeTags.TypeTag<A4> evidence$29, scala.reflect.api.TypeTags.TypeTag<A5> evidence$30, scala.reflect.api.TypeTags.TypeTag<A6> evidence$31)
Defines a Scala closure of 6 arguments as user-defined function (UDF).
udf(scala.Function7<A1,A2,A3,A4,A5,A6,A7,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$32, scala.reflect.api.TypeTags.TypeTag<A1> evidence$33, scala.reflect.api.TypeTags.TypeTag<A2> evidence$34, scala.reflect.api.TypeTags.TypeTag<A3> evidence$35, scala.reflect.api.TypeTags.TypeTag<A4> evidence$36, scala.reflect.api.TypeTags.TypeTag<A5> evidence$37, scala.reflect.api.TypeTags.TypeTag<A6> evidence$38, scala.reflect.api.TypeTags.TypeTag<A7> evidence$39)
Defines a Scala closure of 7 arguments as user-defined function (UDF).
udf(scala.Function8<A1,A2,A3,A4,A5,A6,A7,A8,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$40, scala.reflect.api.TypeTags.TypeTag<A1> evidence$41, scala.reflect.api.TypeTags.TypeTag<A2> evidence$42, scala.reflect.api.TypeTags.TypeTag<A3> evidence$43, scala.reflect.api.TypeTags.TypeTag<A4> evidence$44, scala.reflect.api.TypeTags.TypeTag<A5> evidence$45, scala.reflect.api.TypeTags.TypeTag<A6> evidence$46, scala.reflect.api.TypeTags.TypeTag<A7> evidence$47, scala.reflect.api.TypeTags.TypeTag<A8> evidence$48)
Defines a Scala closure of 8 arguments as user-defined function (UDF).
udf(scala.Function9<A1,A2,A3,A4,A5,A6,A7,A8,A9,RT> f, scala.reflect.api.TypeTags.TypeTag<RT> evidence$49, scala.reflect.api.TypeTags.TypeTag<A1> evidence$50, scala.reflect.api.TypeTags.TypeTag<A2> evidence$51, scala.reflect.api.TypeTags.TypeTag<A3> evidence$52, scala.reflect.api.TypeTags.TypeTag<A4> evidence$53, scala.reflect.api.TypeTags.TypeTag<A5> evidence$54, scala.reflect.api.TypeTags.TypeTag<A6> evidence$55, scala.reflect.api.TypeTags.TypeTag<A7> evidence$56, scala.reflect.api.TypeTags.TypeTag<A8> evidence$57, scala.reflect.api.TypeTags.TypeTag<A9> evidence$58)
Defines a Scala closure of 9 arguments as user-defined function (UDF).
Decodes a BASE64 encoded string column and returns it as a binary column.
Returns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers.
Returns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers, with the chosen random seed.
Returns the number of days since 1970-01-01.
Returns the number of microseconds since 1970-01-01 00:00:00 UTC.
Returns the number of milliseconds since 1970-01-01 00:00:00 UTC.
Returns the number of seconds since 1970-01-01 00:00:00 UTC.
Returns the current Unix timestamp (in seconds) as a long.
Converts time string in format yyyy-MM-dd HH:mm:ss to Unix timestamp (in seconds), using the default timezone and the default locale.
Converts time string with given pattern to Unix timestamp (in seconds).
Unwrap UDT data type column into its underlying type.
Converts a string column to upper case.
Decodes a str in 'application/x-www-form-urlencoded' format using a specific encoding scheme.
Translates a string into 'application/x-www-form-urlencoded' format using a specific encoding scheme.
Returns the user name of current execution context.
Returns an universally unique identifier (UUID) string.
Returns the input value if it corresponds to a valid UTF-8 string, or emits a SparkIllegalArgumentException exception otherwise.
Aggregate function: returns the population variance of the values in a group.
Aggregate function: returns the population variance of the values in a group.
Aggregate function: returns the unbiased variance of the values in a group.
Aggregate function: returns the unbiased variance of the values in a group.
Aggregate function: alias for var_samp.
Aggregate function: alias for var_samp.
Extracts a sub-variant from v according to path string, and then cast the sub-variant to targetType.
Extracts a sub-variant from v according to path column, and then cast the sub-variant to targetType.
Returns the Spark version.
Returns the day of the week for date/timestamp (0 = Monday, 1 = Tuesday, ..., 6 = Sunday).
Extracts the week number as an integer from a given date/timestamp/string.
Evaluates a list of conditions and returns one of multiple possible result expressions.
Returns the bucket number into which the value of this expression would fall after being evaluated.
Generates tumbling time windows given a timestamp specifying column.
Bucketize rows into one or more time windows given a timestamp specifying column.
Bucketize rows into one or more time windows given a timestamp specifying column.
Extracts the event time from the window column.
Returns a string array of values within the nodes of xml that match the XPath expression.
Returns true if the XPath expression evaluates to true, or if a matching node is found.
Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.
Returns a float value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.
Returns an integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.
Returns a long integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.
Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.
Returns a short integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.
Returns the text contents of the first xml node that matches the XPath expression.
Calculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column.
Calculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column.
Extracts the year as an integer from a given date/timestamp/string.
(Java-specific) A transform for timestamps and dates to partition data into years.
Returns zero if col is null, or col otherwise.
Merge two given arrays, element-wise, into a single array using a function.
public functions()
Aggregate function: returns the number of distinct items in a group.
An alias of count_distinct, and it is encouraged to use count_distinct directly.
expr - (undocumented)
exprs - (undocumented)
Aggregate function: returns the number of distinct items in a group.
An alias of count_distinct, and it is encouraged to use count_distinct directly.
columnName - (undocumented)
columnNames - (undocumented)
expr - (undocumented)
exprs - (undocumented)
Aggregate function: returns the level of grouping, equals to
(grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)
cols - (undocumented)
Aggregate function: returns the level of grouping, equals to
(grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)
colName - (undocumented)
colNames - (undocumented)
cols - (undocumented)
colName - (undocumented)
colNames - (undocumented)
cols - (undocumented)
cols - (undocumented)
Returns the first column that is not null, or null if all inputs are null.
For example, coalesce(a, b, c) will return a if a is not null, or b if a is null and b is not null, or c if both a and b are null but c is not null.
e - (undocumented)
Creates a new struct column. If the input column is a column in a
DataFrame
, or a derived column expression that is named (i.e. aliased), its name would be retained as the StructField's name, otherwise, the newly generated StructField's name would be auto generated as
col
with a suffix
index + 1
, i.e. col1, col2, col3, ...
cols - (undocumented)
colName - (undocumented)
colNames - (undocumented)
exprs - (undocumented)
columnName - (undocumented)
columnNames - (undocumented)
exprs - (undocumented)
columnName - (undocumented)
columnNames - (undocumented)
cols - (undocumented)
cols - (undocumented)
cols - (undocumented)
cols - (undocumented)
This is a special version of
reflect
that performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.
cols - (undocumented)
Separates
col1
, ...,
colk
into
n
rows. Uses column names col0, col1, etc. by default unless specified otherwise.
cols - (undocumented)
sep - (undocumented)
exprs - (undocumented)
format - (undocumented)
arguments - (undocumented)
format - (undocumented)
arguments - (undocumented)
Returns the
n
-th input, e.g., returns
input2
when
n
is 2. The function returns NULL if the index exceeds the length of the array and
spark.sql.ansi.enabled
is set to false. If
spark.sql.ansi.enabled
is set to true, it throws ArrayIndexOutOfBoundsException for invalid indices.
inputs - (undocumented)
exprs - (undocumented)
json - (undocumented)
fields - (undocumented)
Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.
e - (undocumented)
Returns the union of all the given maps.
cols - (undocumented)
udfName - (undocumented)
cols - (undocumented)
Call an user-defined function. Example:
import org.apache.spark.sql._
val df = Seq(("id1", 1), ("id2", 4), ("id3", 5)).toDF("id", "value")
val spark = df.sparkSession
spark.udf.register("simpleUDF", (v: Int) => v * v)
df.select($"id", call_udf("simpleUDF", $"value"))
udfName - (undocumented)
cols - (undocumented)
funcName - function name that follows the SQL identifier syntax (can be quoted, can be qualified)
cols - the expression parameters of function
Returns a
Column
based on the given column name.
colName - (undocumented)
colName - (undocumented)
Creates a
Column
of literal value.
The passed in object is returned directly if it is already a Column. If the object is a Scala Symbol, it is converted into a Column also. Otherwise, a new Column is created to represent the literal value.
literal - (undocumented)
Creates a
Column
of literal value.
An alias of typedlit, and it is encouraged to use typedlit directly.
literal - (undocumented)
evidence$1 - (undocumented)
Creates a
Column
of literal value.
The passed in object is returned directly if it is already a Column. If the object is a Scala Symbol, it is converted into a Column also. Otherwise, a new Column is created to represent the literal value. The difference between this function and lit(java.lang.Object) is that this function can handle parameterized scala types e.g.: List, Seq and Map.
literal - (undocumented)
evidence$2 - (undocumented)
typedlit will call expensive Scala reflection APIs. lit is preferred if parameterized Scala types are not used.
Returns a sort expression based on ascending order of the column.
df.sort(asc("dept"), desc("age"))
columnName - (undocumented)
Returns a sort expression based on ascending order of the column, and null values return before non-null values.
df.sort(asc_nulls_first("dept"), desc("age"))
columnName - (undocumented)
Returns a sort expression based on ascending order of the column, and null values appear after non-null values.
df.sort(asc_nulls_last("dept"), desc("age"))
columnName - (undocumented)
Returns a sort expression based on the descending order of the column.
df.sort(asc("dept"), desc("age"))
columnName - (undocumented)
Returns a sort expression based on the descending order of the column, and null values appear before non-null values.
df.sort(asc("dept"), desc_nulls_first("age"))
columnName - (undocumented)
Returns a sort expression based on the descending order of the column, and null values appear after non-null values.
df.sort(asc("dept"), desc_nulls_last("age"))
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
rsd - (undocumented)
columnName - (undocumented)
rsd - (undocumented)
e - (undocumented)
columnName - (undocumented)
rsd - maximum relative standard deviation allowed (default = 0.05)
e - (undocumented)
rsd - maximum relative standard deviation allowed (default = 0.05)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
Returns a count-min sketch of a column with the given esp, confidence and seed. The result is an array of bytes, which can be deserialized to a
CountMinSketch
before usage. Count-min sketch is a probabilistic data structure used for cardinality estimation using sub-linear space.
e - (undocumented)
eps - (undocumented)
confidence - (undocumented)
seed - (undocumented)
Returns a count-min sketch of a column with the given esp, confidence and seed. The result is an array of bytes, which can be deserialized to a
CountMinSketch
before usage. Count-min sketch is a probabilistic data structure used for cardinality estimation using sub-linear space.
e - (undocumented)
eps - (undocumented)
confidence - (undocumented)
column1 - (undocumented)
column2 - (undocumented)
columnName1 - (undocumented)
columnName2 - (undocumented)
e - (undocumented)
columnName - (undocumented)
Aggregate function: returns the number of distinct items in a group.
An alias of count_distinct, and it is encouraged to use count_distinct directly.
expr - (undocumented)
exprs - (undocumented)
Aggregate function: returns the number of distinct items in a group.
An alias of count_distinct, and it is encouraged to use count_distinct directly.
columnName - (undocumented)
columnNames - (undocumented)
expr - (undocumented)
exprs - (undocumented)
column1 - (undocumented)
column2 - (undocumented)
columnName1 - (undocumented)
columnName2 - (undocumented)
column1 - (undocumented)
column2 - (undocumented)
columnName1 - (undocumented)
columnName2 - (undocumented)
Aggregate function: returns the first value in a group.
The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
e - (undocumented)
ignoreNulls - (undocumented)
Aggregate function: returns the first value of a column in a group.
The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
columnName - (undocumented)
ignoreNulls - (undocumented)
Aggregate function: returns the first value in a group.
The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
e - (undocumented)
Aggregate function: returns the first value of a column in a group.
The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
columnName - (undocumented)
e - (undocumented)
Aggregate function: returns the first value in a group.
The function by default returns the first values it sees. It will return the first non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
e - (undocumented)
ignoreNulls - (undocumented)
e - (undocumented)
columnName - (undocumented)
Aggregate function: returns the level of grouping, equals to
(grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)
cols - (undocumented)
Aggregate function: returns the level of grouping, equals to
(grouping(c1) <<; (n-1)) + (grouping(c2) <<; (n-2)) + ... + grouping(cn)
colName - (undocumented)
colNames - (undocumented)
e - (undocumented)
lgConfigK - (undocumented)
e - (undocumented)
lgConfigK - (undocumented)
columnName - (undocumented)
lgConfigK - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
allowDifferentLgConfigK - (undocumented)
e - (undocumented)
allowDifferentLgConfigK - (undocumented)
columnName - (undocumented)
allowDifferentLgConfigK - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
Aggregate function: returns the last value in a group.
The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
e - (undocumented)
ignoreNulls - (undocumented)
Aggregate function: returns the last value of the column in a group.
The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
columnName - (undocumented)
ignoreNulls - (undocumented)
Aggregate function: returns the last value in a group.
The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
e - (undocumented)
Aggregate function: returns the last value of the column in a group.
The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
columnName - (undocumented)
e - (undocumented)
Aggregate function: returns the last value in a group.
The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
e - (undocumented)
ignoreNulls - (undocumented)
e - (undocumented)
Aggregate function: returns the most frequent value in a group.
When multiple values have the same greatest frequency then either any of values is returned if deterministic is false or is not defined, or the lowest value is returned if deterministic is true.
e - (undocumented)
deterministic - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
ord - (undocumented)
e.
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
ord - (undocumented)
e.
Aggregate function: returns the exact percentile(s) of numeric column
expr
at the given percentage(s) with value range in [0.0, 1.0].
e - (undocumented)
percentage - (undocumented)
Aggregate function: returns the exact percentile(s) of numeric column
expr
at the given percentage(s) with value range in [0.0, 1.0].
e - (undocumented)
percentage - (undocumented)
frequency - (undocumented)
Aggregate function: returns the approximate
percentile
of the numeric column
col
which is the smallest value in the ordered
col
values (sorted from least to greatest) such that no more than
percentage
of
col
values is less than the value or equal to that value.
If percentage is an array, each value must be between 0.0 and 1.0. If it is a single floating point value, it must be between 0.0 and 1.0.
The accuracy parameter is a positive numeric literal which controls approximation accuracy at the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the relative error of the approximation.
e - (undocumented)
percentage - (undocumented)
accuracy - (undocumented)
Aggregate function: returns the approximate
percentile
of the numeric column
col
which is the smallest value in the ordered
col
values (sorted from least to greatest) such that no more than
percentage
of
col
values is less than the value or equal to that value.
If percentage is an array, each value must be between 0.0 and 1.0. If it is a single floating point value, it must be between 0.0 and 1.0.
The accuracy parameter is a positive numeric literal which controls approximation accuracy at the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the relative error of the approximation.
e - (undocumented)
percentage - (undocumented)
accuracy - (undocumented)
e - (undocumented)
e - (undocumented)
columnName - (undocumented)
Aggregate function: alias for
stddev_samp
.
e - (undocumented)
Aggregate function: alias for
stddev_samp
.
e - (undocumented)
Aggregate function: alias for
stddev_samp
.
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
delimiter - (undocumented)
e - (undocumented)
e - (undocumented)
delimiter - (undocumented)
Aggregate function: returns the concatenation of non-null input values. Alias for
listagg
.
e - (undocumented)
Aggregate function: returns the concatenation of non-null input values, separated by the delimiter. Alias for
listagg
.
e - (undocumented)
delimiter - (undocumented)
Aggregate function: returns the concatenation of distinct non-null input values. Alias for
listagg
.
e - (undocumented)
Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter. Alias for
listagg
.
e - (undocumented)
delimiter - (undocumented)
Aggregate function: alias for
var_samp
.
e - (undocumented)
Aggregate function: alias for
var_samp
.
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
Aggregate function: returns the average of the independent variable for non-null pairs in a group, where
y
is the dependent variable and
x
is the independent variable.
y - (undocumented)
x - (undocumented)
Aggregate function: returns the average of the independent variable for non-null pairs in a group, where
y
is the dependent variable and
x
is the independent variable.
y - (undocumented)
x - (undocumented)
Aggregate function: returns the number of non-null number pairs in a group, where
y
is the dependent variable and
x
is the independent variable.
y - (undocumented)
x - (undocumented)
Aggregate function: returns the intercept of the univariate linear regression line for non-null pairs in a group, where
y
is the dependent variable and
x
is the independent variable.
y - (undocumented)
x - (undocumented)
Aggregate function: returns the coefficient of determination for non-null pairs in a group, where
y
is the dependent variable and
x
is the independent variable.
y - (undocumented)
x - (undocumented)
Aggregate function: returns the slope of the linear regression line for non-null pairs in a group, where
y
is the dependent variable and
x
is the independent variable.
y - (undocumented)
x - (undocumented)
Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(x) for non-null pairs in a group, where
y
is the dependent variable and
x
is the independent variable.
y - (undocumented)
x - (undocumented)
Aggregate function: returns REGR_COUNT(y, x) * COVAR_POP(y, x) for non-null pairs in a group, where
y
is the dependent variable and
x
is the independent variable.
y - (undocumented)
x - (undocumented)
Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(y) for non-null pairs in a group, where
y
is the dependent variable and
x
is the independent variable.
y - (undocumented)
x - (undocumented)
Aggregate function: returns some value of
e
for a group of rows.
e - (undocumented)
Aggregate function: returns some value of
e
for a group of rows. If
isIgnoreNull
is true, returns only non-null values.
e - (undocumented)
ignoreNulls - (undocumented)
Aggregate function: returns the number of
TRUE
values for the expression.
e - (undocumented)
e - (undocumented)
nBins - (undocumented)
Aggregate function: returns true if all values of
e
are true.
e - (undocumented)
Aggregate function: returns true if all values of
e
are true.
e - (undocumented)
Aggregate function: returns true if at least one value of
e
is true.
e - (undocumented)
Aggregate function: returns true if at least one value of
e
is true.
e - (undocumented)
Aggregate function: returns true if at least one value of
e
is true.
e - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
()
Window function: returns the cumulative distribution of values within a window partition, i.e. the fraction of rows that are below the current row.
N = total number of rows in the partition
cumeDist(x) = number of values before (and including) x / N
()
Window function: returns the rank of rows within a window partition, without any gaps.
The difference between rank and dense_rank is that denseRank leaves no gaps in ranking sequence when there are ties. That is, if you were ranking a competition using dense_rank and had three people tie for second place, you would say that all three were in second place and that the next person came in third. Rank would give me sequential numbers, making the person that came in third place (after the ties) would register as coming in fifth.
This is equivalent to the DENSE_RANK function in SQL.
Window function: returns the value that is
offset
rows before the current row, and
null
if there is less than
offset
rows before the current row. For example, an
offset
of one will return the previous row at any given point in the window partition.
This is equivalent to the LAG function in SQL.
e - (undocumented)
offset - (undocumented)
Window function: returns the value that is
offset
rows before the current row, and
null
if there is less than
offset
rows before the current row. For example, an
offset
of one will return the previous row at any given point in the window partition.
This is equivalent to the LAG function in SQL.
columnName - (undocumented)
offset - (undocumented)
Window function: returns the value that is
offset
rows before the current row, and
defaultValue
if there is less than
offset
rows before the current row. For example, an
offset
of one will return the previous row at any given point in the window partition.
This is equivalent to the LAG function in SQL.
columnName - (undocumented)
offset - (undocumented)
defaultValue - (undocumented)
Window function: returns the value that is
offset
rows before the current row, and
defaultValue
if there is less than
offset
rows before the current row. For example, an
offset
of one will return the previous row at any given point in the window partition.
This is equivalent to the LAG function in SQL.
e - (undocumented)
offset - (undocumented)
defaultValue - (undocumented)
Window function: returns the value that is
offset
rows before the current row, and
defaultValue
if there is less than
offset
rows before the current row.
ignoreNulls
determines whether null values of row are included in or eliminated from the calculation. For example, an
offset
of one will return the previous row at any given point in the window partition.
This is equivalent to the LAG function in SQL.
e - (undocumented)
offset - (undocumented)
defaultValue - (undocumented)
ignoreNulls - (undocumented)
Window function: returns the value that is
offset
rows after the current row, and
null
if there is less than
offset
rows after the current row. For example, an
offset
of one will return the next row at any given point in the window partition.
This is equivalent to the LEAD function in SQL.
columnName - (undocumented)
offset - (undocumented)
Window function: returns the value that is
offset
rows after the current row, and
null
if there is less than
offset
rows after the current row. For example, an
offset
of one will return the next row at any given point in the window partition.
This is equivalent to the LEAD function in SQL.
e - (undocumented)
offset - (undocumented)
Window function: returns the value that is
offset
rows after the current row, and
defaultValue
if there is less than
offset
rows after the current row. For example, an
offset
of one will return the next row at any given point in the window partition.
This is equivalent to the LEAD function in SQL.
columnName - (undocumented)
offset - (undocumented)
defaultValue - (undocumented)
Window function: returns the value that is
offset
rows after the current row, and
defaultValue
if there is less than
offset
rows after the current row. For example, an
offset
of one will return the next row at any given point in the window partition.
This is equivalent to the LEAD function in SQL.
e - (undocumented)
offset - (undocumented)
defaultValue - (undocumented)
Window function: returns the value that is
offset
rows after the current row, and
defaultValue
if there is less than
offset
rows after the current row.
ignoreNulls
determines whether null values of row are included in or eliminated from the calculation. The default value of
ignoreNulls
is false. For example, an
offset
of one will return the next row at any given point in the window partition.
This is equivalent to the LEAD function in SQL.
e - (undocumented)
offset - (undocumented)
defaultValue - (undocumented)
ignoreNulls - (undocumented)
Window function: returns the value that is the
offset
th row of the window frame (counting from 1), and
null
if the size of window frame is less than
offset
rows.
It will return the offsetth non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.
This is equivalent to the nth_value function in SQL.
e - (undocumented)
offset - (undocumented)
ignoreNulls - (undocumented)
Window function: returns the value that is the
offset
th row of the window frame (counting from 1), and
null
if the size of window frame is less than
offset
rows.
This is equivalent to the nth_value function in SQL.
e - (undocumented)
offset - (undocumented)
Window function: returns the ntile group id (from 1 to
n
inclusive) in an ordered window partition. For example, if
n
is 4, the first quarter of the rows will get value 1, the second quarter will get 2, the third quarter will get 3, and the last quarter will get 4.
This is equivalent to the NTILE function in SQL.
n - (undocumented)
()
Window function: returns the relative rank (i.e. percentile) of rows within a window partition.
This is computed by:
(rank of row in its partition - 1) / (number of rows in the partition - 1)
This is equivalent to the PERCENT_RANK function in SQL.
Window function: returns the rank of rows within a window partition.
The difference between rank and dense_rank is that dense_rank leaves no gaps in ranking sequence when there are ties. That is, if you were ranking a competition using dense_rank and had three people tie for second place, you would say that all three were in second place and that the next person came in third. Rank would give me sequential numbers, making the person that came in third place (after the ties) would register as coming in fifth.
This is equivalent to the RANK function in SQL.
()
cols - (undocumented)
colName - (undocumented)
colNames - (undocumented)
cols - (undocumented)
cols - (undocumented)
keys - (undocumented)
values - (undocumented)
Creates a map after splitting the text into key/value pairs using delimiters. Both
pairDelim
and
keyValueDelim
are treated as regular expressions.
text - (undocumented)
pairDelim - (undocumented)
keyValueDelim - (undocumented)
Creates a map after splitting the text into key/value pairs using delimiters. The
pairDelim
is treated as regular expressions.
text - (undocumented)
pairDelim - (undocumented)
text - (undocumented)
Marks a DataFrame as small enough for use in broadcast joins.
The following example marks the right DataFrame for broadcast hash join using joinKey.
// left and right are DataFrames
left.join(broadcast(right), "joinKey")
df - (undocumented)
Returns the first column that is not null, or null if all inputs are null.
For example, coalesce(a, b, c) will return a if a is not null, or b if a is null and b is not null, or c if both a and b are null but c is not null.
e - (undocumented)
()
e - (undocumented)
e - (undocumented)
()
A column expression that generates monotonically increasing 64-bit integers.
The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record number within each partition in the lower 33 bits. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.
As an example, consider a DataFrame with two partitions, each with 3 records. This expression would return the following IDs:
0, 1, 2, 8589934592 (1L << 33), 8589934593, 8589934594.
()
A column expression that generates monotonically increasing 64-bit integers.
The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record number within each partition in the lower 33 bits. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.
As an example, consider a DataFrame with two partitions, each with 3 records. This expression would return the following IDs:
0, 1, 2, 8589934592 (1L << 33), 8589934593, 8589934594.
Returns col1 if it is not NaN, or col2 if col1 is NaN.
Both inputs should be floating point columns (DoubleType or FloatType).
col1 - (undocumented)
col2 - (undocumented)
Unary minus, i.e. negate the expression.
// Select the amount column and negates all values.
// Scala:
df.select( -df("amount") )
// Java:
df.select( negate(df.col("amount")) );
e - (undocumented)
Inversion of boolean expression, i.e. NOT.
// Scala: select rows that are not active (isActive === false)
df.filter( !df("isActive") )
// Java:
df.filter( not(df.col("isActive")) );
e - (undocumented)
seed - (undocumented)
seed - (undocumented)
length - (undocumented)
length - (undocumented)
seed - (undocumented)
()
e - (undocumented)
colName - (undocumented)
Returns the sum of
left
and
right
and the result is null on overflow. The acceptable input types are the same with the
+
operator.
left - (undocumented)
right - (undocumented)
e - (undocumented)
Returns
dividend/divisor
. It always performs floating point division. Its result is always null if
divisor
is 0.
left - (undocumented)
right - (undocumented)
Returns the remainder of
dividend/divisor
. Its result is always null if
divisor
is 0.
left - (undocumented)
right - (undocumented)
Returns
left*right
and the result is null on overflow. The acceptable input types are the same with the
*
operator.
left - (undocumented)
right - (undocumented)
Returns
left-right
and the result is null on overflow. The acceptable input types are the same with the
-
operator.
left - (undocumented)
right - (undocumented)
e - (undocumented)
Creates a new struct column. If the input column is a column in a
DataFrame
, or a derived column expression that is named (i.e. aliased), its name would be retained as the StructField's name, otherwise, the newly generated StructField's name would be auto generated as
col
with a suffix
index + 1
, i.e. col1, col2, col3, ...
cols - (undocumented)
colName - (undocumented)
colNames - (undocumented)
Evaluates a list of conditions and returns one of multiple possible result expressions. If otherwise is not defined at the end, null is returned for unmatched conditions.
// Example: encoding gender string column into integer.
// Scala:
people.select(when(people("gender") === "male", 0)
.when(people("gender") === "female", 1)
.otherwise(2))
// Java:
people.select(when(col("gender").equalTo("male"), 0)
.when(col("gender").equalTo("female"), 1)
.otherwise(2))
condition - (undocumented)
value - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
pos - (undocumented)
e - (undocumented)
pos - (undocumented)
Parses the expression string into the column that it represents, similar to
Dataset.selectExpr(java.lang.String...)
.
// get the number of words of each length
df.groupBy(expr("length(word)")).count()
expr - (undocumented)
e - (undocumented)
e - (undocumented)
e in radians, as if computed by java.lang.Math.acos
columnName - (undocumented)
columnName, as if computed by java.lang.Math.acos
e - (undocumented)
e
columnName - (undocumented)
columnName
e - (undocumented)
e in radians, as if computed by java.lang.Math.asin
columnName - (undocumented)
columnName, as if computed by java.lang.Math.asin
e - (undocumented)
e
columnName - (undocumented)
columnName
e - (undocumented)
e as if computed by java.lang.Math.atan
columnName - (undocumented)
columnName, as if computed by java.lang.Math.atan
y - coordinate on y-axis
x - coordinate on x-axis
java.lang.Math.atan2
y - coordinate on y-axis
xName - coordinate on x-axis
java.lang.Math.atan2
yName - coordinate on y-axis
x - coordinate on x-axis
java.lang.Math.atan2
yName - coordinate on y-axis
xName - coordinate on x-axis
java.lang.Math.atan2
y - coordinate on y-axis
xValue - coordinate on x-axis
java.lang.Math.atan2
yName - coordinate on y-axis
xValue - coordinate on x-axis
java.lang.Math.atan2
yValue - coordinate on y-axis
x - coordinate on x-axis
java.lang.Math.atan2
yValue - coordinate on y-axis
xName - coordinate on x-axis
java.lang.Math.atan2
e - (undocumented)
e
columnName - (undocumented)
columnName
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
Computes the ceiling of the given value of
e
to
scale
decimal places.
e - (undocumented)
scale - (undocumented)
Computes the ceiling of the given value of
e
to 0 decimal places.
e - (undocumented)
Computes the ceiling of the given value of
e
to 0 decimal places.
columnName - (undocumented)
Computes the ceiling of the given value of
e
to
scale
decimal places.
e - (undocumented)
scale - (undocumented)
Computes the ceiling of the given value of
e
to 0 decimal places.
e - (undocumented)
num - (undocumented)
fromBase - (undocumented)
toBase - (undocumented)
e - angle in radians
java.lang.Math.cos
columnName - angle in radians
java.lang.Math.cos
e - hyperbolic angle
java.lang.Math.cosh
columnName - hyperbolic angle
java.lang.Math.cosh
e - angle in radians
e - angle in radians
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
Computes the floor of the given value of
e
to
scale
decimal places.
e - (undocumented)
scale - (undocumented)
Computes the floor of the given value of
e
to 0 decimal places.
e - (undocumented)
columnName - (undocumented)
exprs - (undocumented)
columnName - (undocumented)
columnNames - (undocumented)
column - (undocumented)
column - (undocumented)
Computes
sqrt(a^2^ + b^2^)
without intermediate overflow or underflow.
l - (undocumented)
r - (undocumented)
Computes
sqrt(a^2^ + b^2^)
without intermediate overflow or underflow.
l - (undocumented)
rightName - (undocumented)
Computes
sqrt(a^2^ + b^2^)
without intermediate overflow or underflow.
leftName - (undocumented)
r - (undocumented)
Computes
sqrt(a^2^ + b^2^)
without intermediate overflow or underflow.
leftName - (undocumented)
rightName - (undocumented)
Computes
sqrt(a^2^ + b^2^)
without intermediate overflow or underflow.
l - (undocumented)
r - (undocumented)
Computes
sqrt(a^2^ + b^2^)
without intermediate overflow or underflow.
leftName - (undocumented)
r - (undocumented)
Computes
sqrt(a^2^ + b^2^)
without intermediate overflow or underflow.
l - (undocumented)
r - (undocumented)
Computes
sqrt(a^2^ + b^2^)
without intermediate overflow or underflow.
l - (undocumented)
rightName - (undocumented)
exprs - (undocumented)
columnName - (undocumented)
columnNames - (undocumented)
e - (undocumented)
e - (undocumented)
columnName - (undocumented)
base - (undocumented)
a - (undocumented)
base - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - (undocumented)
columnName - (undocumented)
expr - (undocumented)
columnName - (undocumented)
e - (undocumented)
e - (undocumented)
l - (undocumented)
r - (undocumented)
l - (undocumented)
rightName - (undocumented)
leftName - (undocumented)
r - (undocumented)
leftName - (undocumented)
rightName - (undocumented)
l - (undocumented)
r - (undocumented)
leftName - (undocumented)
r - (undocumented)
l - (undocumented)
r - (undocumented)
l - (undocumented)
rightName - (undocumented)
l - (undocumented)
r - (undocumented)
dividend - (undocumented)
divisor - (undocumented)
e - (undocumented)
columnName - (undocumented)
Returns the value of the column
e
rounded to 0 decimal places with HALF_UP round mode.
e - (undocumented)
Round the value of
e
to
scale
decimal places with HALF_UP round mode if
scale
is greater than or equal to 0 or at integral part when
scale
is less than 0.
e - (undocumented)
scale - (undocumented)
Round the value of
e
to
scale
decimal places with HALF_UP round mode if
scale
is greater than or equal to 0 or at integral part when
scale
is less than 0.
e - (undocumented)
scale - (undocumented)
Returns the value of the column
e
rounded to 0 decimal places with HALF_EVEN round mode.
e - (undocumented)
Round the value of
e
to
scale
decimal places with HALF_EVEN round mode if
scale
is greater than or equal to 0 or at integral part when
scale
is less than 0.
e - (undocumented)
scale - (undocumented)
Round the value of
e
to
scale
decimal places with HALF_EVEN round mode if
scale
is greater than or equal to 0 or at integral part when
scale
is less than 0.
e - (undocumented)
scale - (undocumented)
e - angle in radians
e - (undocumented)
numBits - (undocumented)
e - (undocumented)
numBits - (undocumented)
e - (undocumented)
numBits - (undocumented)
e - (undocumented)
numBits - (undocumented)
e - (undocumented)
numBits - (undocumented)
e - (undocumented)
numBits - (undocumented)
e - (undocumented)
e - (undocumented)
columnName - (undocumented)
e - angle in radians
java.lang.Math.sin
columnName - angle in radians
java.lang.Math.sin
e - hyperbolic angle
java.lang.Math.sinh
columnName - hyperbolic angle
java.lang.Math.sinh
e - angle in radians
java.lang.Math.tan
columnName - angle in radians
java.lang.Math.tan
e - hyperbolic angle
java.lang.Math.tanh
columnName - hyperbolic angle
java.lang.Math.tanh
e - (undocumented)
columnName - (undocumented)
e - angle in radians
java.lang.Math.toDegrees
columnName - angle in radians
java.lang.Math.toDegrees
e - (undocumented)
columnName - (undocumented)
e - angle in degrees
java.lang.Math.toRadians
columnName - angle in degrees
java.lang.Math.toRadians
v - value to compute a bucket number in the histogram
min - minimum value of the histogram
max - maximum value of the histogram
numBucket - the number of buckets
()
()
()
()
e - (undocumented)
e - (undocumented)
e - column to compute SHA-2 on.
numBits - one of 224, 256, 384, or 512.
e - (undocumented)
cols - (undocumented)
cols - (undocumented)
c - (undocumented)
c - (undocumented)
e - (undocumented)
c - (undocumented)
c - (undocumented)
columnName - (undocumented)
c1 - (undocumented)
c2 - (undocumented)
columnName1 - (undocumented)
columnName2 - (undocumented)
c1 - (undocumented)
c2 - (undocumented)
allowDifferentLgConfigK - (undocumented)
columnName1 - (undocumented)
columnName2 - (undocumented)
allowDifferentLgConfigK - (undocumented)
()
Returns an encrypted value of
input
using AES in given
mode
with the specified
padding
. Key lengths of 16, 24 and 32 bits are supported. Supported combinations of (
mode
,
padding
) are ('ECB', 'PKCS'), ('GCM', 'NONE') and ('CBC', 'PKCS'). Optional initialization vectors (IVs) are only supported for CBC and GCM modes. These must be 16 bytes for CBC and 12 bytes for GCM. If not provided, a random vector will be generated and prepended to the output. Optional additional authenticated data (AAD) is only supported for GCM. If provided for encryption, the identical AAD value must be provided for decryption. The default mode is GCM.
input - The binary value to encrypt.
key - The passphrase to use to encrypt the data.
mode - Specifies which block cipher mode should be used to encrypt messages. Valid modes: ECB, GCM, CBC.
padding - Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC.
iv - Optional initialization vector. Only supported for CBC and GCM modes. Valid values: None or "". 16-byte array for CBC mode. 12-byte array for GCM mode.
aad - Optional additional authenticated data. Only supported for GCM mode. This can be any free-form input and must be provided for both encryption and decryption.
Returns an encrypted value of
input
.
input - (undocumented)
key - (undocumented)
mode - (undocumented)
padding - (undocumented)
iv - (undocumented)
org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column, Column)Returns an encrypted value of
input
.
input - (undocumented)
key - (undocumented)
mode - (undocumented)
padding - (undocumented)
org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column, Column)Returns an encrypted value of
input
.
input - (undocumented)
key - (undocumented)
mode - (undocumented)
org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column, Column)Returns an encrypted value of
input
.
input - (undocumented)
key - (undocumented)
org.apache.spark.sql.functions.aes_encrypt(Column, Column, Column, Column, Column, Column)Returns a decrypted value of
input
using AES in
mode
with
padding
. Key lengths of 16, 24 and 32 bits are supported. Supported combinations of (
mode
,
padding
) are ('ECB', 'PKCS'), ('GCM', 'NONE') and ('CBC', 'PKCS'). Optional additional authenticated data (AAD) is only supported for GCM. If provided for encryption, the identical AAD value must be provided for decryption. The default mode is GCM.
input - The binary value to decrypt.
key - The passphrase to use to decrypt the data.
mode - Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB, GCM, CBC.
padding - Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC.
aad - Optional additional authenticated data. Only supported for GCM mode. This can be any free-form input and must be provided for both encryption and decryption.
Returns a decrypted value of
input
.
input - (undocumented)
key - (undocumented)
mode - (undocumented)
padding - (undocumented)
org.apache.spark.sql.functions.aes_decrypt(Column, Column, Column, Column, Column)Returns a decrypted value of
input
.
input - (undocumented)
key - (undocumented)
mode - (undocumented)
org.apache.spark.sql.functions.aes_decrypt(Column, Column, Column, Column, Column)Returns a decrypted value of
input
.
input - (undocumented)
key - (undocumented)
org.apache.spark.sql.functions.aes_decrypt(Column, Column, Column, Column, Column)This is a special version of
aes_decrypt
that performs the same operation, but returns a NULL value instead of raising an error if the decryption cannot be performed.
input - The binary value to decrypt.
key - The passphrase to use to decrypt the data.
mode - Specifies which block cipher mode should be used to decrypt messages. Valid modes: ECB, GCM, CBC.
padding - Specifies how to pad messages whose length is not a multiple of the block size. Valid values: PKCS, NONE, DEFAULT. The DEFAULT padding means PKCS for ECB, NONE for GCM and PKCS for CBC.
aad - Optional additional authenticated data. Only supported for GCM mode. This can be any free-form input and must be provided for both encryption and decryption.
Returns a decrypted value of
input
.
input - (undocumented)
key - (undocumented)
mode - (undocumented)
padding - (undocumented)
org.apache.spark.sql.functions.try_aes_decrypt(Column, Column, Column, Column, Column)Returns a decrypted value of
input
.
input - (undocumented)
key - (undocumented)
mode - (undocumented)
org.apache.spark.sql.functions.try_aes_decrypt(Column, Column, Column, Column, Column)Returns a decrypted value of
input
.
input - (undocumented)
key - (undocumented)
org.apache.spark.sql.functions.try_aes_decrypt(Column, Column, Column, Column, Column)Returns a sha1 hash value as a hex string of the
col
.
col - (undocumented)
()
()
cols - (undocumented)
cols - (undocumented)
This is a special version of
reflect
that performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.
cols - (undocumented)
()
col - (undocumented)
Separates
col1
, ...,
colk
into
n
rows. Uses column names col0, col1, etc. by default unless specified otherwise.
cols - (undocumented)
min - (undocumented)
max - (undocumented)
min - (undocumented)
max - (undocumented)
seed - (undocumented)
seed - (undocumented)
()
col - (undocumented)
col - (undocumented)
col - (undocumented)
col - (undocumented)
col - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
sep - (undocumented)
exprs - (undocumented)
value - (undocumented)
charset - (undocumented)
value - (undocumented)
charset - (undocumented)
str - (undocumented)
str - (undocumented)
str - (undocumented)
str - (undocumented)
Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with HALF_EVEN round mode, and returns the result as a string column.
If d is 0, the result has no decimal point or fractional part. If d is less than 0, the result will be null.
x - (undocumented)
d - (undocumented)
format - (undocumented)
arguments - (undocumented)
Returns a new string column by converting the first letter of each word to uppercase. Words are delimited by whitespace.
For example, "hello world" will become "Hello World".
e - (undocumented)
str - (undocumented)
substring - (undocumented)
str - (undocumented)
substring - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
Computes the Levenshtein distance of the two given string columns if it's less than or equal to a given threshold.
l - (undocumented)
r - (undocumented)
threshold - (undocumented)
Computes the Levenshtein distance of the two given string columns.
l - (undocumented)
r - (undocumented)
substr - (undocumented)
str - (undocumented)
substr - (undocumented)
str - (undocumented)
pos - (undocumented)
str - (undocumented)
len - (undocumented)
pad - (undocumented)
str - (undocumented)
len - (undocumented)
pad - (undocumented)
str - (undocumented)
len - (undocumented)
pad - (undocumented)
e - (undocumented)
Trim the specified character string from left end for the specified string column.
e - (undocumented)
trimString - (undocumented)
Trim the specified character string from left end for the specified string column.
e - (undocumented)
trim - (undocumented)
e - (undocumented)
e - (undocumented)
collation - (undocumented)
e - (undocumented)
Returns true if
str
matches
regexp
, or false otherwise.
str - (undocumented)
regexp - (undocumented)
Returns true if
str
matches
regexp
, or false otherwise.
str - (undocumented)
regexp - (undocumented)
Returns true if
str
matches
regexp
, or false otherwise.
str - (undocumented)
regexp - (undocumented)
Returns a count of the number of times that the regular expression pattern
regexp
is matched in the string
str
.
str - (undocumented)
regexp - (undocumented)
e - (undocumented)
exp - (undocumented)
groupIdx - (undocumented)
Extract all strings in the
str
that match the
regexp
expression and corresponding to the first regex group index.
str - (undocumented)
regexp - (undocumented)
Extract all strings in the
str
that match the
regexp
expression and corresponding to the regex group index.
str - (undocumented)
regexp - (undocumented)
idx - (undocumented)
e - (undocumented)
pattern - (undocumented)
replacement - (undocumented)
e - (undocumented)
pattern - (undocumented)
replacement - (undocumented)
Returns the substring that matches the regular expression
regexp
within the string
str
. If the regular expression is not found, the result is null.
str - (undocumented)
regexp - (undocumented)
str - (undocumented)
regexp - (undocumented)
str - (undocumented)
regexp - (undocumented)
idx - (undocumented)
e - (undocumented)
str - (undocumented)
len - (undocumented)
pad - (undocumented)
str - (undocumented)
len - (undocumented)
pad - (undocumented)
str - (undocumented)
len - (undocumented)
pad - (undocumented)
str - (undocumented)
n - (undocumented)
str - (undocumented)
n - (undocumented)
e - (undocumented)
Trim the specified character string from right end for the specified string column.
e - (undocumented)
trimString - (undocumented)
Trim the specified character string from right end for the specified string column.
e - (undocumented)
trim - (undocumented)
e - (undocumented)
str - a string expression to split
pattern - a string representing a regular expression. The regex string should be a Java regular expression.
str - a string expression to split
pattern - a column of string representing a regular expression. The regex string should be a Java regular expression.
str - a string expression to split
pattern - a string representing a regular expression. The regex string should be a Java regular expression.
limit - an integer expression which controls the number of times the regex is applied.
regex will be applied as many times as possible, and the resulting array can be of any size.str - a string expression to split
pattern - a column of string representing a regular expression. The regex string should be a Java regular expression.
limit - a column of integer expression which controls the number of times the regex is applied.
regex will be applied as many times as possible, and the resulting array can be of any size.Substring starts at
pos
and is of length
len
when str is String type or returns the slice of byte array that starts at
pos
in byte and is of length
len
when str is Binary type
str - (undocumented)
pos - (undocumented)
len - (undocumented)
Substring starts at
pos
and is of length
len
when str is String type or returns the slice of byte array that starts at
pos
in byte and is of length
len
when str is Binary type
str - (undocumented)
pos - (undocumented)
len - (undocumented)
str - (undocumented)
delim - (undocumented)
count - (undocumented)
Overlay the specified portion of
src
with
replace
, starting from byte position
pos
of
src
and proceeding for
len
bytes.
src - (undocumented)
replace - (undocumented)
pos - (undocumented)
len - (undocumented)
Overlay the specified portion of
src
with
replace
, starting from byte position
pos
of
src
.
src - (undocumented)
replace - (undocumented)
pos - (undocumented)
Splits a string into arrays of sentences, where each sentence is an array of words.
string - (undocumented)
language - (undocumented)
country - (undocumented)
Splits a string into arrays of sentences, where each sentence is an array of words. The default country('') is used.
string - (undocumented)
language - (undocumented)
Splits a string into arrays of sentences, where each sentence is an array of words. The default locale is used.
string - (undocumented)
Translate any character in the src by a character in replaceString. The characters in replaceString correspond to the characters in matchingString. The translate will happen when any character in the string matches the character in the
matchingString
.
src - (undocumented)
matchingString - (undocumented)
replaceString - (undocumented)
e - (undocumented)
Trim the specified character from both ends for the specified string column.
e - (undocumented)
trimString - (undocumented)
Trim the specified character from both ends for the specified string column.
e - (undocumented)
trim - (undocumented)
e - (undocumented)
Converts the input
e
to a binary value based on the supplied
format
. The
format
can be a case-insensitive string literal of "hex", "utf-8", "utf8", or "base64". By default, the binary format for conversion is "hex" if
format
is omitted. The function returns NULL if at least one of the input parameters is NULL.
e - (undocumented)
f - (undocumented)
Converts the input
e
to a binary value based on the default format "hex". The function returns NULL if at least one of the input parameters is NULL.
e - (undocumented)
Convert
e
to a string based on the
format
. Throws an exception if the conversion fails. The format can consist of the following characters, case insensitive: '0' or '9': Specifies an expected digit between 0 and 9. A sequence of 0 or 9 in the format string matches a sequence of digits in the input value, generating a result string of the same length as the corresponding sequence in the format string. The result string is left-padded with zeros if the 0/9 sequence comprises more digits than the matching part of the decimal value, starts with 0, and is before the decimal point. Otherwise, it is padded with spaces. '.' or 'D': Specifies the position of the decimal point (optional, only allowed once). ',' or 'G': Specifies the position of the grouping (thousands) separator (,). There must be a 0 or 9 to the left and right of each grouping separator. '$': Specifies the location of the $ currency sign. This character may only be specified once. 'S' or 'MI': Specifies the position of a '-' or '+' sign (optional, only allowed once at the beginning or end of the format string). Note that 'S' prints '+' for positive values but 'MI' prints a space. 'PR': Only allowed at the end of the format string; specifies that the result string will be wrapped by angle brackets if the input value is negative.
If e is a datetime, format shall be a valid datetime pattern, see Datetime Patterns. If e is a binary, it is converted to a string in one of the formats: 'base64': a base 64 string. 'hex': a string in the hexadecimal format. 'utf-8': the input binary is decoded to UTF-8 string.
e - (undocumented)
format - (undocumented)
Convert
e
to a string based on the
format
. Throws an exception if the conversion fails. The format can consist of the following characters, case insensitive: '0' or '9': Specifies an expected digit between 0 and 9. A sequence of 0 or 9 in the format string matches a sequence of digits in the input value, generating a result string of the same length as the corresponding sequence in the format string. The result string is left-padded with zeros if the 0/9 sequence comprises more digits than the matching part of the decimal value, starts with 0, and is before the decimal point. Otherwise, it is padded with spaces. '.' or 'D': Specifies the position of the decimal point (optional, only allowed once). ',' or 'G': Specifies the position of the grouping (thousands) separator (,). There must be a 0 or 9 to the left and right of each grouping separator. '$': Specifies the location of the $ currency sign. This character may only be specified once. 'S' or 'MI': Specifies the position of a '-' or '+' sign (optional, only allowed once at the beginning or end of the format string). Note that 'S' prints '+' for positive values but 'MI' prints a space. 'PR': Only allowed at the end of the format string; specifies that the result string will be wrapped by angle brackets if the input value is negative.
If e is a datetime, format shall be a valid datetime pattern, see Datetime Patterns. If e is a binary, it is converted to a string in one of the formats: 'base64': a base 64 string. 'hex': a string in the hexadecimal format. 'utf-8': the input binary is decoded to UTF-8 string.
e - (undocumented)
format - (undocumented)
e - (undocumented)
format - (undocumented)
Replaces all occurrences of
search
with
replace
.
src - A column of string to be replaced
search - A column of string, If search is not found in str, str is returned unchanged.
replace - A column of string, If replace is not specified or is an empty string, nothing replaces the string that is removed from str.
Replaces all occurrences of
search
with
replace
.
src - A column of string to be replaced
search - A column of string, If search is not found in src, src is returned unchanged.
Splits
str
by delimiter and return requested part of the split (1-based). If any input is null, returns null. if
partNum
is out of range of split parts, returns empty string. If
partNum
is 0, throws an error. If
partNum
is negative, the parts are counted backward from the end of the string. If the
delimiter
is an empty string, the
str
is not split.
str - (undocumented)
delimiter - (undocumented)
partNum - (undocumented)
Returns the substring of
str
that starts at
pos
and is of length
len
, or the slice of byte array that starts at
pos
and is of length
len
.
str - (undocumented)
pos - (undocumented)
len - (undocumented)
Returns the substring of
str
that starts at
pos
, or the slice of byte array that starts at
pos
.
str - (undocumented)
pos - (undocumented)
url - (undocumented)
partToExtract - (undocumented)
key - (undocumented)
url - (undocumented)
partToExtract - (undocumented)
url - (undocumented)
partToExtract - (undocumented)
key - (undocumented)
url - (undocumented)
partToExtract - (undocumented)
format - (undocumented)
arguments - (undocumented)
Decodes a
str
in 'application/x-www-form-urlencoded' format using a specific encoding scheme.
str - (undocumented)
This is a special version of
url_decode
that performs the same operation, but returns a NULL value instead of raising an error if the decoding cannot be performed.
str - (undocumented)
str - (undocumented)
Returns the position of the first occurrence of
substr
in
str
after position
start
. The given
start
and return value are 1-based.
substr - (undocumented)
str - (undocumented)
start - (undocumented)
Returns the position of the first occurrence of
substr
in
str
after position
1
. The return value are 1-based.
substr - (undocumented)
str - (undocumented)
str - (undocumented)
suffix - (undocumented)
str - (undocumented)
prefix - (undocumented)
Removes the leading and trailing space characters from
str
.
str - (undocumented)
Remove the leading and trailing
trim
characters from
str
.
str - (undocumented)
trim - (undocumented)
This is a special version of
to_binary
that performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.
e - (undocumented)
f - (undocumented)
This is a special version of
to_binary
that performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.
e - (undocumented)
Convert string
e
to a number based on the string format
format
. Returns NULL if the string
e
does not match the expected format. The format follows the same semantics as the to_number function.
e - (undocumented)
format - (undocumented)
str - (undocumented)
str - (undocumented)
Returns the ASCII character having the binary equivalent to
n
. If n is larger than 256 the result is equivalent to chr(n % 256)
n - (undocumented)
left - (undocumented)
right - (undocumented)
Returns the
n
-th input, e.g., returns
input2
when
n
is 2. The function returns NULL if the index exceeds the length of the array and
spark.sql.ansi.enabled
is set to false. If
spark.sql.ansi.enabled
is set to true, it throws ArrayIndexOutOfBoundsException for invalid indices.
inputs - (undocumented)
Returns the index (1-based) of the given string (
str
) in the comma-delimited list (
strArray
). Returns 0, if the string was not found or if the given string (
str
) contains a comma.
str - (undocumented)
strArray - (undocumented)
Returns true if str matches
pattern
with
escapeChar
, null if any arguments are null, false otherwise.
str - (undocumented)
pattern - (undocumented)
escapeChar - (undocumented)
Returns true if str matches
pattern
with
escapeChar
('\'), null if any arguments are null, false otherwise.
str - (undocumented)
pattern - (undocumented)
Returns true if str matches
pattern
with
escapeChar
case-insensitively, null if any arguments are null, false otherwise.
str - (undocumented)
pattern - (undocumented)
escapeChar - (undocumented)
Returns true if str matches
pattern
with
escapeChar
('\') case-insensitively, null if any arguments are null, false otherwise.
str - (undocumented)
pattern - (undocumented)
Returns
str
with all characters changed to lowercase.
str - (undocumented)
Returns
str
with all characters changed to uppercase.
str - (undocumented)
Returns the leftmost
len
(
len
can be string type) characters from the string
str
, if
len
is less or equal than 0 the result is an empty string.
str - (undocumented)
len - (undocumented)
Returns the rightmost
len
(
len
can be string type) characters from the string
str
, if
len
is less or equal than 0 the result is an empty string.
str - (undocumented)
len - (undocumented)
Returns the date that is
numMonths
after
startDate
.
startDate - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
numMonths - The number of months to add to startDate, can be negative to subtract months
startDate was a string that could not be cast to a date
Returns the date that is
numMonths
after
startDate
.
startDate - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
numMonths - A column of the number of months to add to startDate, can be negative to subtract months
startDate was a string that could not be cast to a date
()
()
()
()
()
Converts a date/timestamp/string to a value of string in the format specified by the date format given by the second argument.
See Datetime Patterns for valid date and time format patterns
dateExpr - A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
format - A pattern dd.MM.yyyy would return a string like 18.03.1993
dateExpr was a string that could not be cast to a timestamp
IllegalArgumentException - if the format pattern is invalid
year(org.apache.spark.sql.Column) whenever possible as they benefit from a specialized implementation.
Returns the date that is
days
days after
start
start - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
days - The number of days to add to start, can be negative to subtract days
start was a string that could not be cast to a date
Returns the date that is
days
days after
start
start - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
days - A column of the number of days to add to start, can be negative to subtract days
start was a string that could not be cast to a date
Returns the date that is
days
days after
start
start - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
days - A column of the number of days to add to start, can be negative to subtract days
start was a string that could not be cast to a date
Returns the date that is
days
days before
start
start - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
days - The number of days to subtract from start, can be negative to add days
start was a string that could not be cast to a date
Returns the date that is
days
days before
start
start - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
days - A column of the number of days to subtract from start, can be negative to add days
start was a string that could not be cast to a date
Returns the number of days from
start
to
end
.
Only considers the date part of the input. For example:
dateddiff("2018-01-10 00:00:00", "2018-01-09 23:59:59")
// returns 1
end - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
start - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
end or start were strings that could not be cast to a date. Negative if end is before start
Returns the number of days from
start
to
end
.
Only considers the date part of the input. For example:
dateddiff("2018-01-10 00:00:00", "2018-01-09 23:59:59")
// returns 1
end - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
start - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
end or start were strings that could not be cast to a date. Negative if end is before start
Create date from the number of
days
since 1970-01-01.
days - (undocumented)
Extracts the year as an integer from a given date/timestamp/string.
e - (undocumented)
Extracts the quarter as an integer from a given date/timestamp/string.
e - (undocumented)
Extracts the month as an integer from a given date/timestamp/string.
e - (undocumented)
Extracts the day of the week as an integer from a given date/timestamp/string. Ranges from 1 for a Sunday through to 7 for a Saturday
e - (undocumented)
Extracts the day of the month as an integer from a given date/timestamp/string.
e - (undocumented)
Extracts the day of the month as an integer from a given date/timestamp/string.
e - (undocumented)
Extracts the day of the year as an integer from a given date/timestamp/string.
e - (undocumented)
Extracts the hours as an integer from a given date/timestamp/string.
e - (undocumented)
field - selects which part of the source should be extracted.
source - a date/timestamp or interval column from where field should be extracted.
field - selects which part of the source should be extracted, and supported string values are as same as the fields of the equivalent function extract.
source - a date/timestamp or interval column from where field should be extracted.
field - selects which part of the source should be extracted, and supported string values are as same as the fields of the equivalent function EXTRACT.
source - a date/timestamp or interval column from where field should be extracted.
e - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
Extracts the minutes as an integer from a given date/timestamp/string.
e - (undocumented)
e - (undocumented)
year - (undocumented)
month - (undocumented)
day - (undocumented)
Returns number of months between dates
start
and
end
.
A whole number is returned if both inputs have the same day of month or both are the last day of their respective months. Otherwise, the difference is calculated assuming 31 days per month.
For example:
months_between("2017-11-14", "2017-07-14") // returns 4.0
months_between("2017-01-01", "2017-01-10") // returns 0.29032258
months_between("2017-06-01", "2017-06-16 12:00:00") // returns -0.5
end - A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
start - A date, timestamp or string. If a string, the data must be in a format that can cast to a timestamp, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
end or start were strings that could not be cast to a timestamp. Negative if end is before start
Returns number of months between dates end and start. If roundOff is set to true, the result is rounded off to 8 digits; it is not rounded otherwise.
end - (undocumented)
start - (undocumented)
roundOff - (undocumented)
Returns the first date which is later than the value of the
date
column that is on the specified day of the week.
For example, next_day('2015-07-27', "Sunday") returns 2015-08-02 because that is the first Sunday after 2015-07-27.
date - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
dayOfWeek - Case insensitive, and accepts: "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"
date was a string that could not be cast to a date or if dayOfWeek was an invalid value
Returns the first date which is later than the value of the
date
column that is on the specified day of the week.
For example, next_day('2015-07-27', "Sunday") returns 2015-08-02 because that is the first Sunday after 2015-07-27.
date - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
dayOfWeek - A column of the day of week. Case insensitive, and accepts: "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"
date was a string that could not be cast to a date or if dayOfWeek was an invalid value
Extracts the seconds as an integer from a given date/timestamp/string.
e - (undocumented)
Extracts the week number as an integer from a given date/timestamp/string.
A week is considered to start on a Monday and week 1 is the first week with more than 3 days, as defined by ISO 8601
e - (undocumented)
ut - A number of a type that is castable to a long, such as string or integer. Can be negative for timestamps before the unix epoch
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.
See Datetime Patterns for valid date and time format patterns
ut - A number of a type that is castable to a long, such as string or integer. Can be negative for timestamps before the unix epoch
f - A date time pattern that the input will be formatted to
ut was a string that could not be cast to a long or f was an invalid date time pattern
()
unix_timestamp within the same query return the same value (i.e. the current timestamp is calculated at the start of query evaluation).
s - A date, timestamp or string. If a string, the data must be in the yyyy-MM-dd HH:mm:ss format
Converts time string with given pattern to Unix timestamp (in seconds).
See Datetime Patterns for valid date and time format patterns
s - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
p - A date time pattern detailing the format of s when s is a string
s was a string that could not be cast to a date or p was an invalid format
Converts to a timestamp by casting rules to
TimestampType
.
s - A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
Converts time string with the given pattern to timestamp.
See Datetime Patterns for valid date and time format patterns
s - A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
fmt - A date time pattern detailing the format of s when s is a string
s was a string that could not be cast to a timestamp or fmt was an invalid format
Parses the
s
with the
format
to a timestamp. The function always returns null on an invalid input with
/
without ANSI SQL mode enabled. The result data type is consistent with the value of configuration
spark.sql.timestampType
.
s - (undocumented)
format - (undocumented)
Parses the
s
to a timestamp. The function always returns null on an invalid input with
/
without ANSI SQL mode enabled. It follows casting rules to a timestamp. The result data type is consistent with the value of configuration
spark.sql.timestampType
.
s - (undocumented)
Converts the column into
DateType
by casting rules to
DateType
.
e - (undocumented)
Converts the column into a
DateType
with a specified format
See Datetime Patterns for valid date and time format patterns
e - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
fmt - A date time pattern detailing the format of e when eis a string
e was a string that could not be cast to a date or fmt was an invalid format
e - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
Returns date truncated to the unit specified by the format.
For example, trunc("2018-11-19 12:01:19", "year") returns 2018-01-01
date - A date, timestamp or string. If a string, the data must be in a format that can be cast to a date, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
format - : 'year', 'yyyy', 'yy' to truncate by year, or 'month', 'mon', 'mm' to truncate by month Other options are: 'week', 'quarter'
date was a string that could not be cast to a date or format was an invalid value
Returns timestamp truncated to the unit specified by the format.
For example, date_trunc("year", "2018-11-19 12:01:19") returns 2018-01-01 00:00:00
format - : 'year', 'yyyy', 'yy' to truncate by year, 'month', 'mon', 'mm' to truncate by month, 'day', 'dd' to truncate by day, Other options are: 'microsecond', 'millisecond', 'second', 'minute', 'hour', 'week', 'quarter'
timestamp - A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
timestamp was a string that could not be cast to a timestamp or format was an invalid value
ts - A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
tz - A string detailing the time zone ID that the input should be adjusted to. It should be in the format of either region-based zone IDs or zone offsets. Region IDs must have the form 'area/city', such as 'America/Los_Angeles'. Zone offsets must be in the format '(+|-)HH:mm', for example '-08:00' or '+01:00'. Also 'UTC' and 'Z' are supported as aliases of '+00:00'. Other short names are not recommended to use because they can be ambiguous.
ts was a string that could not be cast to a timestamp or tz was an invalid value
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone. For example, 'GMT+1' would yield '2017-07-14 03:40:00.0'.
ts - (undocumented)
tz - (undocumented)
ts - A date, timestamp or string. If a string, the data must be in a format that can be cast to a timestamp, such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS
tz - A string detailing the time zone ID that the input should be adjusted to. It should be in the format of either region-based zone IDs or zone offsets. Region IDs must have the form 'area/city', such as 'America/Los_Angeles'. Zone offsets must be in the format '(+|-)HH:mm', for example '-08:00' or '+01:00'. Also 'UTC' and 'Z' are supported as aliases of '+00:00'. Other short names are not recommended to use because they can be ambiguous.
ts was a string that could not be cast to a timestamp or tz was an invalid value
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'.
ts - (undocumented)
tz - (undocumented)
Bucketize rows into one or more time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The following example takes the average stock price for a one minute window every 10 seconds starting 5 seconds after the hour:
val df = ... // schema => timestamp: TimestampType, stockId: StringType, price: DoubleType
df.groupBy(window($"timestamp", "1 minute", "10 seconds", "5 seconds"), $"stockId")
.agg(mean("price"))
The windows will look like:
09:00:05-09:01:05
09:00:15-09:01:15
09:00:25-09:01:25 ...
For a streaming query, you may use the function current_timestamp to generate windows on processing time.
timeColumn - The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType.
windowDuration - A string specifying the width of the window, e.g. 10 minutes, 1 second. Check org.apache.spark.unsafe.types.CalendarInterval for valid duration identifiers. Note that the duration is a fixed length of time, and does not vary over time according to a calendar. For example, 1 day always means 86,400,000 milliseconds, not a calendar day.
slideDuration - A string specifying the sliding interval of the window, e.g. 1 minute. A new window will be generated every slideDuration. Must be less than or equal to the windowDuration. Check org.apache.spark.unsafe.types.CalendarInterval for valid duration identifiers. This duration is likewise absolute, and does not vary according to a calendar.
startTime - The offset with respect to 1970-01-01 00:00:00 UTC with which to start window intervals. For example, in order to have hourly tumbling windows that start 15 minutes past the hour, e.g. 12:15-13:15, 13:15-14:15... provide startTime as 15 minutes.
Bucketize rows into one or more time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The windows start beginning at 1970-01-01 00:00:00 UTC. The following example takes the average stock price for a one minute window every 10 seconds:
val df = ... // schema => timestamp: TimestampType, stockId: StringType, price: DoubleType
df.groupBy(window($"timestamp", "1 minute", "10 seconds"), $"stockId")
.agg(mean("price"))
The windows will look like:
09:00:00-09:01:00
09:00:10-09:01:10
09:00:20-09:01:20 ...
For a streaming query, you may use the function current_timestamp to generate windows on processing time.
timeColumn - The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType.
windowDuration - A string specifying the width of the window, e.g. 10 minutes, 1 second. Check org.apache.spark.unsafe.types.CalendarInterval for valid duration identifiers. Note that the duration is a fixed length of time, and does not vary over time according to a calendar. For example, 1 day always means 86,400,000 milliseconds, not a calendar day.
slideDuration - A string specifying the sliding interval of the window, e.g. 1 minute. A new window will be generated every slideDuration. Must be less than or equal to the windowDuration. Check org.apache.spark.unsafe.types.CalendarInterval for valid duration identifiers. This duration is likewise absolute, and does not vary according to a calendar.
Generates tumbling time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The windows start beginning at 1970-01-01 00:00:00 UTC. The following example takes the average stock price for a one minute tumbling window:
val df = ... // schema => timestamp: TimestampType, stockId: StringType, price: DoubleType
df.groupBy(window($"timestamp", "1 minute"), $"stockId")
.agg(mean("price"))
The windows will look like:
09:00:00-09:01:00
09:01:00-09:02:00
09:02:00-09:03:00 ...
For a streaming query, you may use the function current_timestamp to generate windows on processing time.
timeColumn - The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType.
windowDuration - A string specifying the width of the window, e.g. 10 minutes, 1 second. Check org.apache.spark.unsafe.types.CalendarInterval for valid duration identifiers.
Extracts the event time from the window column.
The window column is of StructType { start: Timestamp, end: Timestamp } where start is inclusive and end is exclusive. Since event time can support microsecond precision, window_time(window) = window.end - 1 microsecond.
windowColumn - The window column (typically produced by window aggregation) of type StructType { start: Timestamp, end: Timestamp }
Generates session window given a timestamp specifying column.
Session window is one of dynamic windows, which means the length of window is varying according to the given inputs. The length of session window is defined as "the timestamp of latest input of the session + gap duration", so when the new inputs are bound to the current session window, the end time of session window can be expanded according to the new inputs.
Windows can support microsecond precision. gapDuration in the order of months are not supported.
For a streaming query, you may use the function current_timestamp to generate windows on processing time.
timeColumn - The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType.
gapDuration - A string specifying the timeout of the session, e.g. 10 minutes, 1 second. Check org.apache.spark.unsafe.types.CalendarInterval for valid duration identifiers.
Generates session window given a timestamp specifying column.
Session window is one of dynamic windows, which means the length of window is varying according to the given inputs. For static gap duration, the length of session window is defined as "the timestamp of latest input of the session + gap duration", so when the new inputs are bound to the current session window, the end time of session window can be expanded according to the new inputs.
Besides a static gap duration value, users can also provide an expression to specify gap duration dynamically based on the input row. With dynamic gap duration, the closing of a session window does not depend on the latest input anymore. A session window's range is the union of all events' ranges which are determined by event start time and evaluated gap duration during the query execution. Note that the rows with negative or zero gap duration will be filtered out from the aggregation.
Windows can support microsecond precision. gapDuration in the order of months are not supported.
For a streaming query, you may use the function current_timestamp to generate windows on processing time.
timeColumn - The column or the expression to use as the timestamp for windowing by time. The time column must be of TimestampType or TimestampNTZType.
gapDuration - A column specifying the timeout of the session. It could be static value, e.g. 10 minutes, 1 second, or an expression/UDF that specifies gap duration dynamically based on the input row.
Converts the number of seconds from the Unix epoch (1970-01-01T00:00:00Z) to a timestamp.
e - (undocumented)
e - (undocumented)
e - (undocumented)
unit - (undocumented)
start - (undocumented)
end - (undocumented)
unit - (undocumented)
quantity - (undocumented)
ts - (undocumented)
Parses the
timestamp
expression with the
format
expression to a timestamp without time zone. Returns null with invalid input.
timestamp - (undocumented)
format - (undocumented)
Parses the
timestamp
expression with the default format to a timestamp without time zone. The default format follows casting rules to a timestamp. Returns null with invalid input.
timestamp - (undocumented)
Parses the
timestamp_str
expression with the
format
expression to a timestamp without time zone. Returns null with invalid input.
timestamp - (undocumented)
format - (undocumented)
Parses the
timestamp
expression with the default format to a timestamp without time zone. The default format follows casting rules to a timestamp. Returns null with invalid input.
timestamp - (undocumented)
timeExp - (undocumented)
format - (undocumented)
timeExp - (undocumented)
timeExp - (undocumented)
timeExp - (undocumented)
Returns null if the array is null, true if the array contains value, and false otherwise.
column - (undocumented)
value - (undocumented)
column - (undocumented)
element - (undocumented)
Returns true if a1 and a2 have at least one non-null element in common. If not and both the arrays are non-empty and any of them contains a null, it returns null. It returns false otherwise.
a1 - (undocumented)
a2 - (undocumented)
Returns an array containing all the elements in
x
from index
start
(or starting from the end if
start
is negative) with the specified
length
.
x - the array column to be sliced
start - the starting index
length - the length of the slice
Returns an array containing all the elements in
x
from index
start
(or starting from the end if
start
is negative) with the specified
length
.
x - the array column to be sliced
start - the starting index
length - the length of the slice
Concatenates the elements of column using the delimiter. Null values are replaced with nullReplacement.
column - (undocumented)
delimiter - (undocumented)
nullReplacement - (undocumented)
Concatenates the elements of column using the delimiter.
column - (undocumented)
delimiter - (undocumented)
exprs - (undocumented)
column - (undocumented)
value - (undocumented)
column - (undocumented)
value - (undocumented)
(array, index) - Returns element of array at given (1-based) index. If Index is 0, Spark will throw an error. If index < 0, accesses elements from the last to the first. The function always returns NULL if the index exceeds the length of the array.
(map, key) - Returns value for given key. The function always returns NULL if the key is not contained in the map.
column - (undocumented)
value - (undocumented)
column - (undocumented)
index - (undocumented)
e - (undocumented)
e - (undocumented)
comparator - (undocumented)
column - (undocumented)
element - (undocumented)
column - (undocumented)
column - (undocumented)
element - (undocumented)
Removes duplicate values from the array.
e - (undocumented)
col1 - (undocumented)
col2 - (undocumented)
arr - (undocumented)
pos - (undocumented)
value - (undocumented)
col1 - (undocumented)
col2 - (undocumented)
col1 - (undocumented)
col2 - (undocumented)
Returns an array of elements after applying a transformation to each element in the input array.
df.select(transform(col("i"), x => x + 1))
column - the input array column
f - col => transformed_col, the lambda function to transform the input column
Returns an array of elements after applying a transformation to each element in the input array.
df.select(transform(col("i"), (x, i) => x + i))
column - the input array column
f - (col, index) => transformed_col, the lambda function to transform the input column given the index. Indices start at 0.
Returns whether a predicate holds for one or more elements in the array.
df.select(exists(col("i"), _ % 2 === 0))
column - the input array column
f - col => predicate, the Boolean predicate to check the input column
Returns whether a predicate holds for every element in the array.
df.select(forall(col("i"), x => x % 2 === 0))
column - the input array column
f - col => predicate, the Boolean predicate to check the input column
Returns an array of elements for which a predicate holds in a given array.
df.select(filter(col("s"), x => x % 2 === 0))
column - the input array column
f - col => predicate, the Boolean predicate to filter the input column
Returns an array of elements for which a predicate holds in a given array.
df.select(filter(col("s"), (x, i) => i % 2 === 0))
column - the input array column
f - (col, index) => predicate, the Boolean predicate to filter the input column given the index. Indices start at 0.
Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state. The final state is converted into the final result by applying a finish function.
df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
expr - the input array column
initialValue - the initial value
merge - (combined_value, input_value) => combined_value, the merge function to merge an input value to the combined_value
finish - combined_value => final_value, the lambda function to convert the combined value of all inputs to final result
Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.
df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x))
expr - the input array column
initialValue - the initial value
merge - (combined_value, input_value) => combined_value, the merge function to merge an input value to the combined_value
Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state. The final state is converted into the final result by applying a finish function.
df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
expr - the input array column
initialValue - the initial value
merge - (combined_value, input_value) => combined_value, the merge function to merge an input value to the combined_value
finish - combined_value => final_value, the lambda function to convert the combined value of all inputs to final result
Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.
df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x))
expr - the input array column
initialValue - the initial value
merge - (combined_value, input_value) => combined_value, the merge function to merge an input value to the combined_value
Merge two given arrays, element-wise, into a single array using a function. If one array is shorter, nulls are appended at the end to match the length of the longer array, before applying the function.
df.select(zip_with(df1("val1"), df1("val2"), (x, y) => x + y))
left - the left input array column
right - the right input array column
f - (lCol, rCol) => col, the lambda function to merge two input columns into one column
Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new keys for the pairs.
df.select(transform_keys(col("i"), (k, v) => k + v))
expr - the input map column
f - (key, value) => new_key, the lambda function to transform the key of input map column
Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new values for the pairs.
df.select(transform_values(col("i"), (k, v) => k + v))
expr - the input map column
f - (key, value) => new_value, the lambda function to transform the value of input map column
Returns a map whose key-value pairs satisfy a predicate.
df.select(map_filter(col("m"), (k, v) => k * 10 === v))
expr - the input map column
f - (key, value) => predicate, the Boolean predicate to filter the input map column
Merge two given maps, key-wise into a single map using a function.
df.select(map_zip_with(df("m1"), df("m2"), (k, v1, v2) => k === v1 + v2))
left - the left input map column
right - the right input map column
f - (key, value1, value2) => new_value, the lambda function to merge the map values
Creates a new row for each element in the given array or map column. Uses the default column name
col
for elements in the array and
key
and
value
for elements in the map unless specified otherwise.
e - (undocumented)
Creates a new row for each element in the given array or map column. Uses the default column name
col
for elements in the array and
key
and
value
for elements in the map unless specified otherwise. Unlike explode, if the array/map is null or empty then null is produced.
e - (undocumented)
Creates a new row for each element with position in the given array or map column. Uses the default column name
pos
for position, and
col
for elements in the array and
key
and
value
for elements in the map unless specified otherwise.
e - (undocumented)
Creates a new row for each element with position in the given array or map column. Uses the default column name
pos
for position, and
col
for elements in the array and
key
and
value
for elements in the map unless specified otherwise. Unlike posexplode, if the array/map is null or empty then the row (null, null) is produced.
e - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
path - (undocumented)
json - (undocumented)
fields - (undocumented)
(Scala-specific) Parses a column containing a JSON string into a
StructType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema to use when parsing the json string
options - options to control how the json is parsed. Accepts the same options as the json data source. See Data Source Option in the version you use.
(Scala-specific) Parses a column containing a JSON string into a
MapType
with
StringType
as keys type,
StructType
or
ArrayType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema to use when parsing the json string
options - options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use.
(Java-specific) Parses a column containing a JSON string into a
StructType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema to use when parsing the json string
options - options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use.
(Java-specific) Parses a column containing a JSON string into a
MapType
with
StringType
as keys type,
StructType
or
ArrayType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema to use when parsing the json string
options - options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use.
Parses a column containing a JSON string into a
StructType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema to use when parsing the json string
Parses a column containing a JSON string into a
MapType
with
StringType
as keys type,
StructType
or
ArrayType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema to use when parsing the json string
(Java-specific) Parses a column containing a JSON string into a
MapType
with
StringType
as keys type,
StructType
or
ArrayType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema as a DDL-formatted string.
options - options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use.
(Scala-specific) Parses a column containing a JSON string into a
MapType
with
StringType
as keys type,
StructType
or
ArrayType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema as a DDL-formatted string.
options - options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use.
(Scala-specific) Parses a column containing a JSON string into a
MapType
with
StringType
as keys type,
StructType
or
ArrayType
of
StructType
s with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema to use when parsing the json string
(Java-specific) Parses a column containing a JSON string into a
MapType
with
StringType
as keys type,
StructType
or
ArrayType
of
StructType
s with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing JSON data.
schema - the schema to use when parsing the json string
options - options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use.
json - a string column that contains JSON data.
json - a string column that contains JSON data.
col - a column with a nested schema or column name.
v - a variant column.
Extracts a sub-variant from
v
according to
path
string, and then cast the sub-variant to
targetType
. Returns null if the path does not exist. Throws an exception if the cast fails.
v - a variant column.
path - the extraction path. A valid path should start with $ and is followed by zero or more segments like [123], .name, ['name'], or ["name"].
targetType - the target data type to cast into, in a DDL-formatted string.
Extracts a sub-variant from
v
according to
path
column, and then cast the sub-variant to
targetType
. Returns null if the path does not exist. Throws an exception if the cast fails.
v - a variant column.
path - the column containing the extraction path strings. A valid path string should start with $ and is followed by zero or more segments like [123], .name, ['name'], or ["name"].
targetType - the target data type to cast into, in a DDL-formatted string.
Extracts a sub-variant from
v
according to
path
string, and then cast the sub-variant to
targetType
. Returns null if the path does not exist or the cast fails..
v - a variant column.
path - the extraction path. A valid path should start with $ and is followed by zero or more segments like [123], .name, ['name'], or ["name"].
targetType - the target data type to cast into, in a DDL-formatted string.
Extracts a sub-variant from
v
according to
path
column, and then cast the sub-variant to
targetType
. Returns null if the path does not exist or the cast fails..
v - a variant column.
path - the column containing the extraction path strings. A valid path string should start with $ and is followed by zero or more segments like [123], .name, ['name'], or ["name"].
targetType - the target data type to cast into, in a DDL-formatted string.
v - a variant column.
v - a variant column.
json - a JSON string.
json - a foldable string column containing a JSON string.
json - a foldable string column containing JSON data.
options - options to control how the json is parsed. accepts the same options and the json data source. See Data Source Option in the version you use.
Returns the number of elements in the outermost JSON array.
NULL
is returned in case of any other valid JSON string,
NULL
or an invalid JSON.
e - (undocumented)
e - (undocumented)
(Scala-specific) Converts a column containing a
StructType
,
ArrayType
or a
MapType
into a JSON string with the specified schema. Throws an exception, in the case of an unsupported type.
e - a column containing a struct, an array or a map.
options - options to control how the struct column is converted into a json string. accepts the same options and the json data source. See Data Source Option in the version you use. Additionally the function supports the pretty option which enables pretty JSON generation.
(Java-specific) Converts a column containing a
StructType
,
ArrayType
or a
MapType
into a JSON string with the specified schema. Throws an exception, in the case of an unsupported type.
e - a column containing a struct, an array or a map.
options - options to control how the struct column is converted into a json string. accepts the same options and the json data source. See Data Source Option in the version you use. Additionally the function supports the pretty option which enables pretty JSON generation.
Converts a column containing a
StructType
,
ArrayType
or a
MapType
into a JSON string with the specified schema. Throws an exception, in the case of an unsupported type.
e - a column containing a struct, an array or a map.
input - string value to mask. Supported types: STRING, VARCHAR, CHAR
input - string value to mask. Supported types: STRING, VARCHAR, CHAR
upperChar - character to replace upper-case characters with. Specify NULL to retain original character.
input - string value to mask. Supported types: STRING, VARCHAR, CHAR
upperChar - character to replace upper-case characters with. Specify NULL to retain original character.
lowerChar - character to replace lower-case characters with. Specify NULL to retain original character.
input - string value to mask. Supported types: STRING, VARCHAR, CHAR
upperChar - character to replace upper-case characters with. Specify NULL to retain original character.
lowerChar - character to replace lower-case characters with. Specify NULL to retain original character.
digitChar - character to replace digit characters with. Specify NULL to retain original character.
input - string value to mask. Supported types: STRING, VARCHAR, CHAR
upperChar - character to replace upper-case characters with. Specify NULL to retain original character.
lowerChar - character to replace lower-case characters with. Specify NULL to retain original character.
digitChar - character to replace digit characters with. Specify NULL to retain original character.
otherChar - character to replace all other characters with. Specify NULL to retain original character.
Returns length of array or map.
This function returns -1 for null input only if spark.sql.ansi.enabled is false and spark.sql.legacy.sizeOfNull is true. Otherwise, it returns null for null input. With the default settings, the function returns null for null input.
e - (undocumented)
Returns length of array or map. This is an alias of
size
function.
This function returns -1 for null input only if spark.sql.ansi.enabled is false and spark.sql.legacy.sizeOfNull is true. Otherwise, it returns null for null input. With the default settings, the function returns null for null input.
e - (undocumented)
e - (undocumented)
e - (undocumented)
asc - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
e - (undocumented)
seed - (undocumented)
Returns a reversed string or an array with reverse order of elements.
e - (undocumented)
Creates a single array from an array of arrays. If a structure of nested arrays is deeper than two levels, only one level of nesting is removed.
e - (undocumented)
start - (undocumented)
stop - (undocumented)
step - (undocumented)
start - (undocumented)
stop - (undocumented)
left - (undocumented)
right - (undocumented)
e - (undocumented)
count - (undocumented)
Returns true if the map contains the key.
column - (undocumented)
key - (undocumented)
Returns an unordered array containing the keys of the map.
e - (undocumented)
Returns an unordered array containing the values of the map.
e - (undocumented)
Returns an unordered array of all entries in the given map.
e - (undocumented)
Returns a map created from the given array of entries.
e - (undocumented)
Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.
e - (undocumented)
Returns the union of all the given maps.
cols - (undocumented)
Parses a column containing a CSV string into a
StructType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing CSV data.
schema - the schema to use when parsing the CSV string
options - options to control how the CSV is parsed. accepts the same options and the CSV data source. See Data Source Option in the version you use.
(Java-specific) Parses a column containing a CSV string into a
StructType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing CSV data.
schema - the schema to use when parsing the CSV string
options - options to control how the CSV is parsed. accepts the same options and the CSV data source. See Data Source Option in the version you use.
csv - a CSV string.
csv - a foldable string column containing a CSV string.
csv - a foldable string column containing a CSV string.
options - options to control how the CSV is parsed. accepts the same options and the CSV data source. See Data Source Option in the version you use.
(Java-specific) Converts a column containing a
StructType
into a CSV string with the specified schema. Throws an exception, in the case of an unsupported type.
e - a column containing a struct.
options - options to control how the struct column is converted into a CSV string. It accepts the same options and the CSV data source. See Data Source Option in the version you use.
Converts a column containing a
StructType
into a CSV string with the specified schema. Throws an exception, in the case of an unsupported type.
e - a column containing a struct.
Parses a column containing a XML string into the data type corresponding to the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing XML data.
schema - the schema to use when parsing the XML string
options - options to control how the XML is parsed. accepts the same options and the XML data source. See Data Source Option in the version you use.
(Java-specific) Parses a column containing a XML string into a
StructType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing XML data.
schema - the schema as a DDL-formatted string.
options - options to control how the XML is parsed. accepts the same options and the xml data source. See Data Source Option in the version you use.
(Java-specific) Parses a column containing a XML string into a
StructType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing XML data.
schema - the schema to use when parsing the XML string
(Java-specific) Parses a column containing a XML string into a
StructType
with the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing XML data.
schema - the schema to use when parsing the XML string
options - options to control how the XML is parsed. accepts the same options and the XML data source. See Data Source Option in the version you use.
Parses a column containing a XML string into the data type corresponding to the specified schema. Returns
null
, in the case of an unparseable string.
e - a string column containing XML data.
schema - the schema to use when parsing the XML string
xml - a XML string.
xml - a foldable string column containing a XML string.
xml - a foldable string column containing XML data.
options - options to control how the xml is parsed. accepts the same options and the XML data source. See Data Source Option in the version you use.
(Java-specific) Converts a column containing a
StructType
into a XML string with the specified schema. Throws an exception, in the case of an unsupported type.
e - a column containing a struct.
options - options to control how the struct column is converted into a XML string. It accepts the same options as the XML data source. See Data Source Option in the version you use.
Converts a column containing a
StructType
into a XML string with the specified schema. Throws an exception, in the case of an unsupported type.
e - a column containing a struct.
e - (undocumented)
e - (undocumented)
e - (undocumented)
xml - (undocumented)
path - (undocumented)
xml - (undocumented)
path - (undocumented)
xml - (undocumented)
path - (undocumented)
xml - (undocumented)
path - (undocumented)
xml - (undocumented)
path - (undocumented)
xml - (undocumented)
path - (undocumented)
xml - (undocumented)
path - (undocumented)
xml - (undocumented)
path - (undocumented)
xml - (undocumented)
path - (undocumented)
e - (undocumented)
Converts the timestamp without time zone
sourceTs
from the
sourceTz
time zone to
targetTz
.
sourceTz - the time zone for the input timestamp. If it is missed, the current session time zone is used as the source time zone.
targetTz - the time zone to which the input timestamp should be converted.
sourceTs - a timestamp without time zone.
Converts the timestamp without time zone
sourceTs
from the current time zone to
targetTz
.
targetTz - the time zone to which the input timestamp should be converted.
sourceTs - a timestamp without time zone.
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
days - (undocumented)
hours - (undocumented)
days - (undocumented)
()
This is a special version of
make_interval
that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
This is a special version of
make_interval
that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
This is a special version of
make_interval
that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
days - (undocumented)
hours - (undocumented)
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
days - (undocumented)
hours - (undocumented)
This is a special version of
make_interval
that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
days - (undocumented)
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
days - (undocumented)
This is a special version of
make_interval
that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
years - (undocumented)
months - (undocumented)
weeks - (undocumented)
This is a special version of
make_interval
that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
years - (undocumented)
months - (undocumented)
years - (undocumented)
months - (undocumented)
This is a special version of
make_interval
that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.
years - (undocumented)
years - (undocumented)
()
Create timestamp from years, months, days, hours, mins, secs and timezone fields. The result data type is consistent with the value of configuration
spark.sql.timestampType
. If the configuration
spark.sql.ansi.enabled
is false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
timezone - (undocumented)
Create timestamp from years, months, days, hours, mins and secs fields. The result data type is consistent with the value of configuration
spark.sql.timestampType
. If the configuration
spark.sql.ansi.enabled
is false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
Try to create a timestamp from years, months, days, hours, mins, secs and timezone fields. The result data type is consistent with the value of configuration
spark.sql.timestampType
. The function returns NULL on invalid inputs.
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
timezone - (undocumented)
Try to create a timestamp from years, months, days, hours, mins, and secs fields. The result data type is consistent with the value of configuration
spark.sql.timestampType
. The function returns NULL on invalid inputs.
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
Create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields. If the configuration
spark.sql.ansi.enabled
is false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
timezone - (undocumented)
Create the current timestamp with local time zone from years, months, days, hours, mins and secs fields. If the configuration
spark.sql.ansi.enabled
is false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
timezone - (undocumented)
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
Create local date-time from years, months, days, hours, mins, secs fields. If the configuration
spark.sql.ansi.enabled
is false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
years - (undocumented)
months - (undocumented)
days - (undocumented)
hours - (undocumented)
mins - (undocumented)
secs - (undocumented)
years - (undocumented)
months - (undocumented)
years - (undocumented)
()
numBuckets - (undocumented)
e - (undocumented)
numBuckets - (undocumented)
e - (undocumented)
Returns
col2
if
col1
is null, or
col1
otherwise.
col1 - (undocumented)
col2 - (undocumented)
Returns true if
col
is not null, or false otherwise.
col - (undocumented)
col1 - (undocumented)
col2 - (undocumented)
Returns null if
col1
equals to
col2
, or
col1
otherwise.
col1 - (undocumented)
col2 - (undocumented)
Returns null if
col
is equal to zero, or
col
otherwise.
col - (undocumented)
Returns
col2
if
col1
is null, or
col1
otherwise.
col1 - (undocumented)
col2 - (undocumented)
Returns
col2
if
col1
is not null, or
col3
otherwise.
col1 - (undocumented)
col2 - (undocumented)
col3 - (undocumented)
Returns zero if
col
is null, or
col
otherwise.
col - (undocumented)
Obtains a
UserDefinedFunction
that wraps the given
Aggregator
so that it may be used with untyped Data Frames.
val agg = // Aggregator[IN, BUF, OUT]
// declare a UDF based on agg
val aggUDF = udaf(agg)
val aggData = df.agg(aggUDF($"colname"))
// register agg as a named function
spark.udf.register("myAggName", udaf(agg))
agg - the typed Aggregator
evidence$3 - (undocumented)
Obtains a
UserDefinedFunction
that wraps the given
Aggregator
so that it may be used with untyped Data Frames.
Aggregator<IN, BUF, OUT> agg = // custom Aggregator
Encoder<IN> enc = // input encoder
// declare a UDF based on agg
UserDefinedFunction aggUDF = udaf(agg, enc)
DataFrame aggData = df.agg(aggUDF($"colname"))
// register agg as a named function
spark.udf.register("myAggName", udaf(agg, enc))
agg - the typed Aggregator
inputEncoder - a specific input encoder to use
Defines a Scala closure of 0 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$4 - (undocumented)
Defines a Scala closure of 1 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$5 - (undocumented)
evidence$6 - (undocumented)
Defines a Scala closure of 2 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$7 - (undocumented)
evidence$8 - (undocumented)
evidence$9 - (undocumented)
Defines a Scala closure of 3 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$10 - (undocumented)
evidence$11 - (undocumented)
evidence$12 - (undocumented)
evidence$13 - (undocumented)
Defines a Scala closure of 4 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$14 - (undocumented)
evidence$15 - (undocumented)
evidence$16 - (undocumented)
evidence$17 - (undocumented)
evidence$18 - (undocumented)
Defines a Scala closure of 5 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$19 - (undocumented)
evidence$20 - (undocumented)
evidence$21 - (undocumented)
evidence$22 - (undocumented)
evidence$23 - (undocumented)
evidence$24 - (undocumented)
Defines a Scala closure of 6 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$25 - (undocumented)
evidence$26 - (undocumented)
evidence$27 - (undocumented)
evidence$28 - (undocumented)
evidence$29 - (undocumented)
evidence$30 - (undocumented)
evidence$31 - (undocumented)
Defines a Scala closure of 7 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$32 - (undocumented)
evidence$33 - (undocumented)
evidence$34 - (undocumented)
evidence$35 - (undocumented)
evidence$36 - (undocumented)
evidence$37 - (undocumented)
evidence$38 - (undocumented)
evidence$39 - (undocumented)
Defines a Scala closure of 8 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$40 - (undocumented)
evidence$41 - (undocumented)
evidence$42 - (undocumented)
evidence$43 - (undocumented)
evidence$44 - (undocumented)
evidence$45 - (undocumented)
evidence$46 - (undocumented)
evidence$47 - (undocumented)
evidence$48 - (undocumented)
Defines a Scala closure of 9 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$49 - (undocumented)
evidence$50 - (undocumented)
evidence$51 - (undocumented)
evidence$52 - (undocumented)
evidence$53 - (undocumented)
evidence$54 - (undocumented)
evidence$55 - (undocumented)
evidence$56 - (undocumented)
evidence$57 - (undocumented)
evidence$58 - (undocumented)
Defines a Scala closure of 10 arguments as user-defined function (UDF). The data types are automatically inferred based on the Scala closure's signature. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
evidence$59 - (undocumented)
evidence$60 - (undocumented)
evidence$61 - (undocumented)
evidence$62 - (undocumented)
evidence$63 - (undocumented)
evidence$64 - (undocumented)
evidence$65 - (undocumented)
evidence$66 - (undocumented)
evidence$67 - (undocumented)
evidence$68 - (undocumented)
evidence$69 - (undocumented)
Defines a Java UDF0 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF1 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF2 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF3 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF4 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF5 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF6 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF7 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF8 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF9 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a Java UDF10 instance as user-defined function (UDF). The caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
f - (undocumented)
returnType - (undocumented)
Defines a deterministic user-defined function (UDF) using a Scala closure. For this variant, the caller must specify the output data type, and there is no automatic input type coercion. By default the returned UDF is deterministic. To change it to nondeterministic, call the API
UserDefinedFunction.asNondeterministic()
.
Note that, although the Scala closure can have primitive-type function argument, it doesn't work well with null values. Because the Scala closure is passed in as Any type, there is no type information for the function arguments. Without the type information, Spark may blindly pass null to the Scala closure with primitive-type argument, and the closure will see the default value of the Java type for the null argument, e.g. udf((x: Int) => x, IntegerType), the result is 0 for null input.
f - A closure in Scala
dataType - The output data type of the UDF
udfName - (undocumented)
cols - (undocumented)
Call an user-defined function. Example:
import org.apache.spark.sql._
val df = Seq(("id1", 1), ("id2", 4), ("id3", 5)).toDF("id", "value")
val spark = df.sparkSession
spark.udf.register("simpleUDF", (v: Int) => v * v)
df.select($"id", call_udf("simpleUDF", $"value"))
udfName - (undocumented)
cols - (undocumented)
funcName - function name that follows the SQL identifier syntax (can be quoted, can be qualified)
cols - the expression parameters of function
Unwrap UDT data type column into its underlying type.
column - (undocumented)
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4