Components of Spark

Q1. Can you explain the internal working of Apache Spark?

Apache Spark's internal working can be broken down into several components:

1. Driver: The driver is the program that creates a SparkContext and coordinates the execution of
tasks on the cluster.

2. SparkContext: The SparkContext is the entry point to Spark's functionality. It sets up the
internal services and configuration for the application.

3. DAGScheduler: The DAGScheduler is responsible for breaking down the user's program into
stages and tasks.

4. TaskScheduler: The TaskScheduler is responsible for scheduling tasks on the cluster.

5. Executor: The Executor is responsible for executing tasks on the cluster.

6. ShuffleManager: The ShuffleManager is responsible for managing data shuffle between

stages.

7. BlockManager: The BlockManager is responsible for managing data storage and retrieval.

Q2. Can you explain the workflow of a Spark job?

Here's the workflow:

1. Job submission: The user submits a job to the SparkContext.

2. DAG creation: The DAGScheduler creates a Directed Acyclic Graph (DAG) of stages and tasks.

3. Stage creation: The DAGScheduler breaks down the DAG into stages.

4. Task creation: The DAGScheduler breaks down each stage into tasks.
5. Task scheduling: The TaskScheduler schedules tasks on the Executor.

6. Task execution: The Executor executes tasks and stores output in memory or on disk.

7. Shuffle: The ShuffleManager manages data shuffle between stages.

8. Result retrieval: The BlockManager retrieves results from memory or disk.

Q3. Can you explain the concept of RDDs and DataFrames?

RDDs (Resilient Distributed Datasets) and DataFrames are Spark's core data structures.

1. RDDs: RDDs are immutable collections of data split into partitions and processed in parallel.

2. DataFrames: DataFrames are a higher-level abstraction built on top of RDDs, providing a

structured data model.

Q4. How does Spark handle failures?

Spark uses a concept called lineage to handle failures. Lineage is the sequence of
transformations applied to the data. If a task fails, Spark can recompute the data using the
lineage.

Q5. What is Apache Spark, and how does it differ from Hadoop MapReduce?
Apache Spark is an open-source, distributed computing system that offers an interface for
programming entire clusters with implicit data parallelism and fault tolerance. Unlike Hadoop
MapReduce, Spark processes data in-memory, which significantly boosts performance,
especially for iterative algorithms.

Q6. Can you explain how Spark handles fault tolerance?

Spark achieves fault tolerance using lineage information. If a partition of data is lost, Spark can
rebuild it by recomputing the lost partition using the original transformations applied to the data.
This is efficient and reduces the need for redundant data storage.

Q7. How does Spark's DAG (Directed Acyclic Graph) work?

In Spark, the DAG represents a sequence of computations to be performed on data. When an
action is called, Spark creates a DAG of stages that it executes sequentially. Each stage consists
of tasks based on the data's partitioning, optimizing the overall execution process.

Q8: What are some of the optimizations Spark performs during execution?
Spark uses several optimizations, such as pipelining transformations, broadcasting variables to
avoid data shuffling, and optimizing the DAG execution plan with techniques like Tungsten and
Catalyst optimizers. These enhance both speed and resource efficiency.

Q9. What is an RDD in Apache Spark?

RDD, or Resilient Distributed Dataset, is Spark's core abstraction representing an immutable,
distributed collection of objects. RDDs are fault-tolerant and allow for parallel processing across
a cluster, making them the backbone of Spark's data processing.
Q10. How does a DataFrame differ from an RDD?
A DataFrame is a higher-level abstraction built on top of RDDs. It represents data in a tabular
format, similar to a table in a relational database, with named columns and rows. DataFrames
optimize operations through Spark's Catalyst optimizer, making them more efficient than RDDs
for most use cases.

Q11. When would you use a Dataset over a DataFrame?

Datasets combine the best of both RDDs and DataFrames. They provide the benefits of a typed
API with the efficiency of DataFrames. Datasets are particularly useful when you need compile-
time type safety along with the optimization benefits of Catalyst.

Q12: Why might you still use RDDs despite these higher-level abstractions?
RDDs offer more control over low-level operations and are necessary when working with
unstructured or semi-structured data that doesn't fit well into a DataFrame or Dataset. They're
also useful for custom transformations or when you're dealing with legacy codebases.

RDD Transformations are Spark operations when executed on RDD, it results in a single or
multiple new RDD’s.

RDD are immutable in nature, transformations always create new RDD without updating an
existing one hence, this creates an RDD lineage.

RDD Transformation is of 2 types : Narrow Transformation and

Wide Transformation

Different ways to read data into PySpark:

1.Reading from CSV Files:

df = spark.read.csv("path/to/file.csv", header=True, inferSchema=True)

2. Reading from JSON Files:

df = spark.read.json("path/to/file.json")

3. Reading from Parquet Files:

df = spark.read.parquet("path/to/file.parquet")

4. Reading from Text Files:

df = spark.read.text("path/to/file.txt")

5. Reading from Database:

You can read from databases using JDBC:
```python
df = spark.read.format("jdbc").options(
url="jdbc:postgresql://host:port/dbname",
driver="org.postgresql.Driver",
dbtable="table_name",
user="username",
password="password"
).load()
6.Reading from Hive Tables:
If you have a Hive context set up:
df = spark.sql("SELECT * FROM hive_table_name")

7. Reading from ORC Files:

df = spark.read.orc("path/to/file.orc")

8. Reading from Avro Files** (requires the Avro package):

df = spark.read.format("avro").load("path/to/file.avro")

9. Reading from Kafka:

To read streaming data from Kafka:
df = spark.readStream.format("kafka").option("kafka.bootstrap.servers",
"server:port").option("subscribe", "topic").load()

10. Reading from Delta Lake:

If using Delta Lake:
df = spark.read.format("delta").load("path/to/delta_table")

Here are some basic transformations you'll use on a daily basis:

1. Adding a New Column: df = df.withColumn("NewColumn", col("ExistingColumn") + 1)

2. Dropping a Column: df = df.drop("ColumnName")
3. Renaming a Column: df = df.withColumnRenamed("OldColumnName", "NewColumnName")
4. Filter Null Values: df = df.filter(df["ColumnName"].isNotNull())
5. Where Clause: df = df.where(col("ColumnName") > 10)
6. Case Statement: df = df.withColumn("NewColumn", when(col("Age") >= 18,
"Adult").otherwise("Minor"))
7. Joining DataFrames: df = df1.join(df2, df1["id"] == df2["id"], "inner")
8. Default Value for a Column: df = df.withColumn("NewColumn", lit("default_value"))
9. Replacing Nulls: df = df.fillna({"ColumnName": "default_value"})
10. UDF (User Defined Function): df = df.withColumn("UpperColumn",
to_upper_udf(col("ColumnName")))
11. Handling Arrays: df = df.withColumn("ArrayColumn", array("Column1", "Column2"))
12. Explode Arrays: df = df.withColumn("ExplodedColumn", explode(col("ArrayColumn")))
13. Window Functions: df = df.withColumn("Rank", rank().over(windowSpec))
14. Selecting Specific Columns: df = df.select("Column1", "Column2")
15. Distinct Rows: df = df.select("ColumnName").distinct()
16. GroupBy and Aggregate: df = df.groupBy("Column1").agg({"Column2": "sum"})
17. Sorting Data: df = df.orderBy(col("ColumnName").desc())
18. Drop Duplicates: df = df.dropDuplicates(["Column1", "Column2"])
19. Casting Data Types: df = df.withColumn("NewColumn", col("OldColumn").cast("Integer"))
20. Pivot/Unpivot: df = df.groupBy("Category").pivot("Type").sum("Amount")

Statistical based questions on Pyspark :

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
# Initialize Spark session
spark = SparkSession.builder.appName("Statistics").getOrCreate()

# Sample DataFrame
data = [(1,), (2,), (3,), (4,), (5,), (6,), (7,), (8,), (9,)]
df = spark.createDataFrame(data, ["values"])

# Function to calculate Gini coefficient

def gini_coefficient(values):
sorted_values = sorted(values)
n = len(sorted_values)
cumulative_values = [sum(sorted_values[:i+1]) for i in range(n)]
sum_cumulative = sum(cumulative_values)
sum_values = sum(sorted_values)
if sum_values == 0:
return 0
gini = 1 - 2 * (sum_cumulative / (n * sum_values))
return gini

# Collect data and apply Gini function

values = df.select("values").rdd.flatMap(lambda x: x).collect()
gini = gini_coefficient(values)
print(f"Gini Coefficient: {gini}")

# Calculate range
range_val = df.select(F.max("values") - F.min("values")).collect()[0][0]
print(f"Range: {range_val}")

# Function to calculate Harmonic Mean

def harmonic_mean(values):
n = len(values)
return n / sum([1.0 / v for v in values])

# Collect data and apply Harmonic Mean function

values = df.select("values").rdd.flatMap(lambda x: x).collect()
harmonic_mean_val = harmonic_mean(values)
print(f"Harmonic Mean: {harmonic_mean_val}")

# Kurtosis
kurtosis_val = df.select(F.kurtosis("values")).collect()[0][0]
print(f"Kurtosis: {kurtosis_val}")

# Median (50th percentile)

median_val = df.approxQuantile("values", [0.5], 0.0)[0]
print(f"Median: {median_val}")

# Skewness
skewness_val = df.select(F.skewness("values")).collect()[0][0]
print(f"Skewness: {skewness_val}")
# Calculate mean and standard deviation
mean_val = df.select(F.mean("values")).collect()[0][0]
stddev_val = df.select(F.stddev("values")).collect()[0][0]

# Add a new column with Z-scores

df_zscore = df.withColumn("z_score", (F.col("values") - mean_val) / stddev_val)
df_zscore.show()

# Coefficient of variation
cv = stddev_val / mean_val
print(f"Coefficient of Variation (CV): {cv}")

# Calculate IQR
q1 = df.approxQuantile("values", [0.25], 0.0)[0]
q3 = df.approxQuantile("values", [0.75], 0.0)[0]
iqr = q3 - q1
print(f"Interquartile Range (IQR): {iqr}")

# Variance
variance_val = df.select(F.variance("values")).collect()[0][0]
print(f"Variance: {variance_val}")