ADF DataFlow Functions CheatSheet by Deepak Goyal Azurelib-H0X4sMxnVP-DsMku3fYRq
ADF DataFlow Functions CheatSheet by Deepak Goyal Azurelib-H0X4sMxnVP-DsMku3fYRq
ADF DataFlow Functions CheatSheet by Deepak Goyal Azurelib-H0X4sMxnVP-DsMku3fYRq
https://adeus.azurelib.com
Email at: [email protected]
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
Returns the
bitwise AND
bitwiseAnd bitwiseAnd(3, 6) returns 2.
of the input
expressions.
Returns the
bitwise OR
bitwiseOr bitwiseOr(3, 6) returns 7.
of the input
expressions.
Returns the
bitwise XOR
bitwiseXor bitwiseXor(3, 6) returns 5.
of the input
expressions.
Returns the
cbrt cube root of cbrt(27) returns 3.
the input.
Returns the
smallest
integer that
ceil is greater ceil(3.14) returns 4.
than or
equal to the
input.
Returns the
coalesce first non-null coalesce(null, "hello", null, "world") returns "hello".
input.
Returns a list
of column
columnNames(table) returns ["column1", "column2",
columnNames names from
"column3"].
the input
table.
Compares
two inputs
and returns
-1 if the first
input is less
than the
second, 0 if compare(1, 2) returns -1.
they are
equal, or 1 if
compare the first
input is
greater than
the second.
Concatenate
s two or
concat concat("hello", "world") returns "helloworld".
more input
strings.
Concatenate
s two or
concatWS(",", "apple", "banana", "cherry") returns
concatWS more input
"apple,banana,cherry".
strings with
a separator.
Returns the
cosine of the
cos cos(0) returns 1.
input (in
radians).
Returns the
hyperbolic
cosh cosh(1) returns 1.5431.
cosine of the
input.
Returns the
CRC-32
crc32 crc32("Hello, world!") returns 222957957.
checksum of
the input.
Converts the
input from
degrees degrees(3.14159) returns 180.
radians to
degrees.
Divides two
divide divide(10, 2) returns 5.
input values.
Removes a
specified
number of
dropLeft characters dropLeft("hello", 2) returns "llo".
from the
beginning of
the input.
Removes a
specified
number of
dropRight characters dropRight("hello", 2) returns "hel".
from the
end of the
input.
Returns true
endsWith if the input endsWith("hello", "lo") returns true.
ends with
Course Link: https://adeus.azurelib.com
Email at: [email protected]
Azurelib Academy By Deepak Goyal
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: [email protected]
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
the specified
substring.
Returns true
if the two
equals equals(1, 2) returns false.
input values
are equal.
Returns true
if the two
input strings
equalsIgnoreCase equalsIgnoreCase("hello", "HELLO") returns true.
are equal
(ignoring
case).
Escapes
special
escape("It's a beautiful day!") returns "It\'s a beautiful
escape characters in
day!".
the input
string.
Evaluates a
expr string expr("1 + 2") returns 3.
expression.
Returns the
factorial factorial of factorial(5) returns 120.
the input.
Returns the
FALSE boolean FALSE returns false.
value false.
Returns the
largest
integer that
floor floor(3.14) returns 3.
is less than
or equal to
the input.
Decodes a
base64-
fromBase64 fromBase64("SGVsbG8sIHdvcmxkIQ==") returns "Hello, world!".
encoded
input string.
Returns true
if the first
greater input is greater(2, 1) returns true.
greater than
the second.
Returns true
if the first
input is
greaterOrEqual greaterOrEqual(2, 2) returns true.
greater than
or equal to
the second.
Returns the
greatest
greatest value among greatest(1, 2, 3, 4) returns 4.
the input
expressions.
Returns true
if the input
table
hasColumn hasColumn(table, "column1") returns true.
contains the
specified
column.
Returns true
if the input
hasError contains an hasError(input) returns true if input contains an error message.
error
message.
Returns the
second
input if the
first input is iif(1 > 2, "One is greater than two", "One is not greater
iif
true, than two") returns `"One is not greater than
otherwise
returns the
third input.
Returns the
second
input if the
first input is iifNull(null, "Input is null", "Input is not null") returns
not null, "Input is null".
iifNull otherwise
returns the
third input.
Capitalizes
the first
letter of
initCap initCap("hello world") returns "Hello World".
each word in
the input
string.
Returns the
position of
the first
occurrence
instr instr("hello world", "world") returns 7.
of a
substring in
the input
string.
Returns true
if the input
isDelete isDelete(input) returns true if input is a delete operation.
is a delete
operation.
Returns true
if the input
isError contains an isError(input) returns true if input contains an error message.
error
message.
Returns true
if the input
isIgnore isIgnore(input) returns true if input is an ignore operation.
is an ignore
operation.
Returns true
if the input
isInsert isInsert(input) returns true if input is an insert operation.
is an insert
operation.
Returns true
if the input
matches the
isMatch isMatch("hello", "hel*o") returns true.
specified
regular
expression.
Returns true
isNull if the input isNull(null) returns true.
is null.
Returns true
if the input
isUpdate isUpdate(input) returns true if input is an update operation.
is an update
operation.
Returns true
if the input
isUpsert isUpsert(input) returns true if input is an upsert operation.
is an upsert
operation.
Returns the
Jaro-Winkler
distance
jaroWinkler jaroWinkler("hello", "world") returns 0.
between two
input
strings.
Returns the
least value
least among the least(1, 2, 3, 4) returns 1.
input
expressions.
Returns the
specified
number of
left characters left("hello", 2) returns "he".
from the
beginning of
the input.
Returns the
length length of the length("hello") returns 5.
input string.
Returns true
if the first
lesser input is less lesser(1, 2) returns true.
than the
second.
Returns true
if the first
input is less
lesserOrEqual lesserOrEqual(2, 2) returns true.
than or
equal to the
second.
Returns the
Levenshtein
distance
levenshtein levenshtein("hello", "world") returns 5.
between two
input
strings.
Returns true
if the input
like matches the like("hello world", "%world") returns true.
specified
pattern.
Returns the
position of
the first
occurrence
locate locate("world", "hello world") returns 6.
of a
substring in
the input
string.
Returns the
natural
log log(2.718) returns 1.
logarithm of
the input.
Returns the
base-10
log10 log10(100) returns 2.
logarithm of
the input.
Converts the
input string
lower lower("HELLO") returns "hello".
to
lowercase.
Pads the
input string
with the
specified
lpad lpad("hello", 7, "*") returns "**hello".
character
until it is the
specified
length.
Removes
leading
ltrim whitespace `ltrim
from the
input string.
Removes
leading
whitespace ltrim(" hello") returns "hello".
ltrim from the
input string.
Returns the
md5("Hello, world!") returns
md5 MD5 hash of
"ed076287532e86365e841e92bfc50d8c".
the input.
Subtracts
minus minus(10, 2) returns 8.
the second
input from
the first.
Returns the
remainder
when the
mod mod(10, 3) returns 1.
first input is
divided by
the second.
Multiplies
multiply two or more multiply(2, 3, 4) returns 24.
input values.
Negates the
negate negate(5) returns -5.
input value.
Returns the
nextSequence("mySequence") returns the next value in the
nextSequence next value in
"mySequence" sequence.
a sequence.
Normalizes
the input
string to
normalize Unicode normalize("NFD", "Å") returns "A\u030A".
NFC
normalizatio
n form.
Returns the
negation of
not not(true) returns false.
the input
boolean.
Returns true
if the two
notEquals input values notEquals(1, 2) returns true.
are not
equal.
null Returns null. null returns null.
Returns true
if any of the
input
or or(false, true, false) returns true.
expressions
evaluate to
true.
Returns the
positive
pMod pMod(-10, 3) returns 2.
remainder
when the
first input is
divided by
the second.
Returns the
partition ID
partitionId partitionId(input) returns the partition ID for input.
for the
input.
Returns the
first input
power raised to the power(2, 3) returns 8.
power of the
second.
Converts the
input from
radians radians(180) returns 3.14159.
degrees to
radians.
Returns a
random
random value random() returns a random value.
between 0
and 1.
Returns the
first
substring
regexExtract that matches regexExtract("hello world", "w.*") returns "world".
the specified
regular
expression.
Returns true
if the input
matches the
regexMatch regexMatch("hello", "hel*o") returns true.
specified
regular
expression.
Replaces all
occurrences
of a
substring regexReplace("hello world", "\\w+", "123") returns "123
regexReplace
that matches 123".
the specified
regular
expression
with another
string.
Splits the
input string
using the
specified
regexSplit regexSplit("hello,world", ",") returns ["hello", "world"].
regular
expression
as the
delimiter.
Replaces all
occurrences
of a
replace replace("hello world", "o", "0") returns "hell0 w0rld".
substring
with another
string.
Reverses the
characters in
reverse reverse("hello") returns "olleh".
the input
string.
Returns the
specified
number of
right characters right("hello", 2) returns "lo".
from the
end of the
input.
Returns true
if the input
matches the
specified
rlike rlike("hello", "hel*o") returns true.
regular
expression
(case-
sensitive).
Rounds the
input to the
round round(3.14) returns 3.
nearest
integer.
Pads the
input string
with the
rpad
specified
character
until it is the
Pads the
input string
with the
specified
rpad("hello", 7, "*") returns "hello**".
character
rpad until it is the
specified
length.
Removes
trailing
rtrim whitespace rtrim("hello ") returns "hello".
from the
input string.
Returns the
sha1("Hello, world!") returns
sha1 SHA-1 hash
"0a4d55a8d778e5022fab701977c5d840bbc486d0".
of the input.
Returns the
SHA-2 hash
of the input sha2("Hello, world!", 256) returns
sha2 with the "a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9a
specified bit d9f146".
length (256
or 512).
Returns the
sine of the
sin sin(0) returns 0.
input (in
radians).
Returns the
hyperbolic
sinh sinh(1) returns 1.1752.
sine of the
input.
Returns the
Soundex
soundex soundex("hello") returns "H400".
code for the
input string.
Splits the
split input string split("hello,world", ",") returns ["hello", "world"].
using the
specified
delimiter.
Returns the
sqrt square root sqrt(16) returns 4.
of the input.
Returns true
if the input
startsWith starts with startsWith("hello", "he") returns true.
the specified
substring.
Returns a
substring of
the input
starting at
substring the specified substring("hello", 1, 3) returns "ell".
position and
with the
specified
length.
Returns the
position of
the first
occurrence
of the
specified
delimiter in
substringIndex substringIndex("hello,world", ",", 1) returns 6.
the input
string,
starting
from the
beginning or
end of the
string.
Returns the
tangent of
tan tan(0) returns 0.
the input (in
radians).
Returns the
hyperbolic
tanh tanh(1) returns 0.7616.
tangent of
the input.
Replaces
each
character in
the input
string that
matches a
character in translate("hello world", "aeiou", "12345") returns "h2ll4
translate
the specified w4rld".
from string
with the
correspondi
ng character
in the to
string.
Removes
leading and
trailing
trim trim(" hello ") returns "hello".
whitespace
from the
input string.
Returns the
TRUE boolean TRUE returns true.
value true.
Returns true
if the input
typeMatch(input, "type") returns true if input matches the
typeMatch matches the
specified type.
specified
type.
Unescapes
special
unescape("It\\'s a beautiful day\\!") returns "It's a
unescape characters in
beautiful day!".
the input
string.
Converts the
input string
upper upper("hello") returns "HELLO".
to
uppercase.
Returns a
randomly
generated
uuid UUID uuid() returns a UUID.
(Universally
Unique
Identifier).
Returns true
if exactly
one of the
xor input xor(true, false) returns true.
expressions
evaluates to
true.
Short
Function
Description Example
Returns the
approximate
approxDistinctCoun number of approxDistinctCount(column) returns the approximate number
t distinct of distinct values in the column.
values in the
input.
Returns the
average of
avg avg(column) returns the average of the values in the column.
the input
values.
Returns the
average of
the input
avgIf(column, condition) returns the average of the values in
avgIf values that
the column that match the condition.
match a
specified
condition.
Returns an
collect array of the collect(column) returns an array of the values in the column.
input values.
Returns an
array of the collectUnique(column) returns an array of the unique values in
collectUnique
unique input the column.
values.
Returns the
number of
count count(*) returns the number of rows in the input.
rows in the
input.
Returns the
number of
countAll(column) returns the number of non-null values in the
countAll non-null
column.
values in the
input.
Course Link: https://adeus.azurelib.com
Email at: [email protected]
Azurelib Academy By Deepak Goyal
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: [email protected]
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
Returns the
number of
countDistinct(column) returns the number of distinct values in
countDistinct distinct
the column.
values in the
input.
Returns the
number of
non-null countAllDistinct(column) returns the number of non-null
countAllDistinct
distinct distinct values in the column.
values in the
input.
Returns the
number of
rows in the
countIf(condition) returns the number of rows in the input that
countIf input that
match the condition.
match a
specified
condition.
Returns the
population
covariancePopulatio covariancePopulation(column1, column2) returns the population
covariance
n covariance of the values in column1 and column2.
of two input
columns.
Returns the
population
covariance
covariancePopulationIf(column1, column2, condition) returns
covariancePopulatio of two input
the population covariance of the values in column1 and column2
nIf columns
that match the condition.
that match a
specified
condition.
Returns the
sample
covarianceSample(column1, column2) returns the sample
covarianceSample covariance
covariance of the values in column1 and column2.
of two input
columns.
Returns the
sample
covariance
covarianceSampleIf(column1, column2, condition) returns the
of two input
covarianceSampleIf sample covariance of the values in column1 and column2 that
columns
match the condition.
that match a
specified
condition.
Returns the
first first value in first(column) returns the first value in the column.
the input.
Returns true
if the input isDistinct(column) returns true if the values in the column are
isDistinct
values are distinct.
distinct.
Returns the
kurtosis of
kurtosis kurtosis(column) returns the kurtosis of the values in the column.
the input
values.
Returns the
kurtosis of
the input
kurtosisIf(column, condition) returns the kurtosis of the values
kurtosisIf values that
in the column that match the condition.
match a
specified
condition.
Returns the
last last value in last(column) returns the last value in the column.
the input.
Returns the
maximum
max max(column) returns the maximum value in the column.
value in the
input.
Returns the
maximum
value in the
maxIf(column, condition) returns the maximum value in the
maxIf input that
column that matches the condition.
matches a
specified
condition.
Returns the
mean mean of the `mean
input values.
Returns the
mean of the mean(column) returns the mean of the values in the column.
mean
input values.
Returns the
mean of the
input values meanIf(column, condition) returns the mean of the values in the
meanIf
that match a column that match the condition.
specified
condition.
Course Link: https://adeus.azurelib.com
Email at: [email protected]
Azurelib Academy By Deepak Goyal
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: [email protected]
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
Returns the
minimum
min min(column) returns the minimum value in the column.
value in the
input.
Returns the
minimum
value in the
minIf(column, condition) returns the minimum value in the
minIf input that
column that matches the condition.
matches a
specified
condition.
Returns the
skewness of skewness(column) returns the skewness of the values in the
skewness
the input column.
values.
Returns the
skewness of
the input
skewnessIf(column, condition) returns the skewness of the
skewnessIf values that
values in the column that match the condition.
match a
specified
condition.
Returns the
standard
stddev(column) returns the standard deviation of the values in the
stddev deviation of
column.
the input
values.
Returns the
standard
deviation of
the input stddevIf(column, condition) returns the standard deviation of
stddevIf
values that the values in the column that match the condition.
match a
specified
condition.
Returns the
population
standard stddevPopulation(column) returns the population standard
stddevPopulation
deviation of deviation of the values in the column.
the input
values.
Returns the
population
standard
deviation of stddevPopulationIf(column, condition) returns the population
stddevPopulationIf the input standard deviation of the values in the column that match the
values that condition.
match a
specified
condition.
Returns the
sample
standard stddevSample(column) returns the sample standard deviation of
stddevSample
deviation of the values in the column.
the input
values.
Returns the
sample
standard
deviation of
stddevSampleIf(column, condition) returns the sample standard
stddevSampleIf the input
deviation of the values in the column that match the condition.
values that
match a
specified
condition.
Returns the
sum sum of the sum(column) returns the sum of the values in the column.
input values.
Returns the
sum of the sumDistinct(column) returns the sum of the distinct values in the
sumDistinct
distinct column.
input values.
Returns the
sum of the
distinct
sumDistinctIf(column, condition) returns the sum of the
sumDistinctIf input values
distinct values in the column that match the condition.
that match a
specified
condition.
Returns the
sum of the
input values sumIf(column, condition) returns the sum of the values in the
sumIf
that match a column that match the condition.
specified
condition.
Returns the
topN top N values topN(column, N) returns the top N values in the column.
in the input.
Returns the
variance of
variance variance(column) returns the variance of the values in the column.
the input
values.
Returns the
variance of
the input
varianceIf(column, condition) returns the variance of the
varianceIf values that
values in the column that match the condition.
match a
specified
condition.
Returns the
population
variancePopulation(column) returns the population variance of
variancePopulation variance of
the values in the column.
the input
values.
Returns the
population
variancePopulation(column) returns the population variance of
variance of
variancePopulation the values in the column.
the input
values.
Returns the
population
variance of
variancePopulationIf(column, condition) returns the
variancePopulationI the input
population variance of the values in the column that match the
f values that
condition.
match a
specified
condition.
Returns the
sample
varianceSample(column) returns the sample variance of the
varianceSample variance of
values in the column.
the input
values.
Returns the
sample
variance of varianceSampleIf(column, condition) returns the sample
varianceSampleIf
the input variance of the values in the column that match the condition
values that
match a
specified
condition.
array that
match a
specified
condition.
Returns the
index of the
first
occurrence
of the
find(array, value) returns the index of the first occurrence of
find specified
the value in the array, or -1 if the value is not found.
value in the
input array,
or -1 if the
value is not
found.
Returns an
array
containing
all the
elements of flatten(array1, array2) returns an array containing all the
flatten
the input elements of array1 and array2.
arrays
flattened
into a single
array.
Returns true
if the input
in value is in(value, array) returns true if the value is found in the array.
found in the
input array.
Returns an
array
containing
the elements intersect(array1, array2) returns an array containing the
intersect
that are elements that are present in both array1 and array2.
present in all
of the input
arrays.
Returns an
array
containing map(array, function) returns an array containing the result of
map
the result of applying the function to each element of the array.
applying a
function to
Course Link: https://adeus.azurelib.com
Email at: [email protected]
Azurelib Academy By Deepak Goyal
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: [email protected]
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
each
element of
the input
array.
Returns an
array
containing
the result of
applying a
function to mapIf(array, condition, function) returns an array containing
mapIf each the result of applying the function to each element of the array
element of that matches the condition.
the input
array that
matches a
specified
condition.
Returns an
array
containing
the result of
applying a mapIndex(array, function) returns an array containing the result
mapIndex
function to of applying the function to each element and index of the array.
each
element and
index of the
input array.
Returns an
array
containing
the result of
applying a
function to mapLoop(array, function) returns an array containing the result
mapLoop each of applying the function to each element of the array and the
element of previous and next elements.
the input
array and
the previous
and next
elements.
Returns the
result of
applying a
function to
each
reduce(array, function) returns the result of applying the
element of
reduce function to each element of the array to produce a single output
the input
value.
array to
produce a
single
output
value.
Returns the
number of
size elements in size(array) returns the number of elements in the array.
the input
array.
Returns a
subarray of
the input
array
starting at slice(array, start, length) returns a subarray of the array
the specified starting at the start index and with the specified length.
index and
slice with the
specified
length.
Returns an
array
containing
sort(array) returns an array containing the array sorted in
sort the input
ascending order.
array sorted
in ascending
order.
Returns an
array
containing
the input
unfold(array) returns an array containing the array with each
unfold array with
element expanded into a separate row.
each
element
expanded
into a
separate
row.
Returns an
array
containing
the elements union(array1, array2) returns an array containing the elements
union
that are that are present in array1 or array2.
present in
any of the
input arrays.
Date Time Function in DataFlow
Short
Function
Description Example
Adds two
add add(2, 3) returns 5.
numbers.
Adds a
specified
addDays(date, days) returns a new date that is days days after
addDays number of
the date.
days to the
input date.
Adds a
specified
number of addMonths(date, months) returns a new date that is months
addMonths
months to months after the date.
the input
date.
Returns true
if the input
value is between(value, lower, upper) returns true if the value is
between
between two between lower and upper.
specified
values.
Returns the
currentDate currentDate() returns the current date.
current date.
Returns the
currentTimestamp current currentTimestamp() returns the current timestamp.
timestamp.
Returns the
currentUTC current UTC currentUTC() returns the current UTC datetime.
datetime.
Returns the
day of the
dayOfMonth month of dayOfMonth(date) returns the day of the month of the date.
the input
date.
Returns the
day of the
dayOfWeek dayOfWeek(date) returns the day of the week of the date.
week of the
input date.
Returns the
day of the
dayOfYear dayOfYear(date) returns the day of the year of the date.
year of the
input date.
Returns the
number of
days(date1, date2) returns the number of days between date1
days days
and date2.
between two
dates.
Converts the
input UTC
datetime to
fromUTC(utcDatetime, timezone) returns the local datetime for
fromUTC a local
the utcDatetime in the specified timezone.
datetime in
the specified
timezone.
Returns the
hour
hour component hour(datetime) returns the hour component of the datetime.
of the input
datetime.
Returns the
number of
hours(datetime1, datetime2) returns the number of hours
hours hours
between datetime1 and datetime2.
between two
datetimes.
Returns true
if the input
isDate isDate(value) returns true if the value is a valid date.
value is a
valid date.
Returns true
if the input
isTimestamp value is a isTimestamp(value) returns true if the value is a valid timestamp.
valid
timestamp.
Returns the
last day of
lastDayOfMonth(date) returns the last day of the month of the
lastDayOfMonth the month
date.
of the input
date.
Returns the
millisecond
millisecond(datetime) returns the millisecond component of the
millisecond component
datetime.
of the input
datetime.
Returns the
number of
milliseconds(datetime1, datetime2) returns the number of
milliseconds milliseconds
milliseconds between datetime1 and datetime2.
between two
datetimes.
Subtracts
minus two minus(5, 3) returns 2.
numbers.
Returns the
minute
minute component minute(datetime) returns the minute component of the datetime.
of the input
datetime.
Returns the
number of
minutes(datetime1, datetime2) returns the number of minutes
minutes minutes
between datetime1 and datetime2.
between two
datetimes.
Returns the
month
month component month(date) returns the month component of the date.
of the input
date.
Returns the
number of
monthsBetween(date1, date2) returns the number of months
monthsBetween months
between date1 and date2.
between two
dates.
Returns the
second
component second(datetime) returns the second component of the datetime.
second of the input
datetime.
Returns the
number of
seconds(datetime1, datetime2) returns the number of seconds
seconds seconds
between datetime1 and datetime2.
between two
datetimes.
Subtracts a
specified
number of subDays(date, days) returns a new date that is days days before
subDays
days from the date.
the input
date.
Subtracts a
specified
number of subMonths(date, months) returns a new date that is months
subMonths
months months before the date.
from the
input date.
Converts a
toDate(string, format) returns the date represented by the
toDate string to a
string in the specified format.
date.
Converts a
toTimestamp(string, format) returns the timestamp represented
toTimestamp string to a
by the string in the specified format.
timestamp.
Converts the
input
toUTC datetime to toUTC(datetime) returns the UTC datetime for the datetime.
a UTC
datetime.
Returns the
ISO week of
weekOfYear the year of weekOfYear(date) returns the ISO week of the year of the date.
the input
date.
Returns the
number of
weeks(date1, date2) returns the number of weeks between
weeks weeks
date1 and date2.
between two
dates.
Returns the
year year year(date) returns the year component of the date.
component
of the input
date.