Associate-Developer-Apache-Spark-3.5 무료 덤프문제 온라인 액세스

A data engineer is building a Structured Streaming pipeline and wants the pipeline to recover from failures or intentional shutdowns by continuing where the pipeline left off.
How can this be achieved?

• A.By configuring the option checkpointLocation during writeStream
• B.By configuring the option recoveryLocation during writeStream
• C.By configuring the option recoveryLocation during the SparkSession initialization
• D.By configuring the option checkpointLocation during readStream

17 of 55.
A data engineer has noticed that upgrading the Spark version in their applications from Spark 3.0 to Spark 3.5 has improved the runtime of some scheduled Spark applications.
Looking further, the data engineer realizes that Adaptive Query Execution (AQE) is now enabled.
Which operation should AQE be implementing to automatically improve the Spark application performance?

• A.Improving the performance of single-stage Spark jobs
• B.Dynamically switching join strategies
• C.Optimizing the layout of Delta files on disk
• D.Collecting persistent table statistics and storing them in the metastore for future use

A data analyst wants to add a column date derived from a timestamp column.
Options:

• A.dates_df.withColumn("date", f.unix_timestamp("timestamp")).show()
• B.dates_df.withColumn("date", f.to_date("timestamp")).show()
• C.dates_df.withColumn("date", f.from_unixtime("timestamp")).show()
• D.dates_df.withColumn("date", f.date_format("timestamp", "yyyy-MM-dd")).show()

What is the risk associated with this operation when converting a large Pandas API on Spark DataFrame back to a Pandas DataFrame?

• A.The operation will fail if the Pandas DataFrame exceeds 1000 rows
• B.Data will be lost during conversion
• C.The operation will load all data into the driver's memory, potentially causing memory overflow
• D.The conversion will automatically distribute the data across worker nodes

18 of 55.
An engineer has two DataFrames - df1 (small) and df2 (large). To optimize the join, the engineer uses a broadcast join:
from pyspark.sql.functions import broadcast
df_result = df2.join(broadcast(df1), on="id", how="inner")
What is the purpose of using broadcast() in this scenario?

• A.It ensures that the join happens only when the id values are identical.
• B.It filters the id values before performing the join.
• C.It increases the partition size for df1 and df2.
• D.It reduces the number of shuffle operations by replicating the smaller DataFrame to all nodes.