Associate-Developer-Apache-Spark 무료 덤프문제 온라인 액세스

Which of the following code blocks returns a 2-column DataFrame that shows the distinct values in column productId and the number of rows with that productId in DataFrame transactionsDf?

• A.transactionsDf.groupBy("productId").agg(col("value").count())
• B.transactionsDf.count("productId")
• C.transactionsDf.count("productId").distinct()
• D.transactionsDf.groupBy("productId").count()
• E.transactionsDf.groupBy("productId").select(count("value"))

The code block displayed below contains an error. The code block should configure Spark to split data in 20 parts when exchanging data between executors for joins or aggregations. Find the error.
Code block:
spark.conf.set(spark.sql.shuffle.partitions, 20)

• A.The code block uses the wrong command for setting an option.
• B.The code block expresses the option incorrectly.
• C.The code block sets the incorrect number of parts.
• D.The code block is missing a parameter.
• E.The code block sets the wrong option.

The code block displayed below contains an error. The code block should return a DataFrame in which column predErrorAdded contains the results of Python function add_2_if_geq_3 as applied to numeric and nullable column predError in DataFrame transactionsDf. Find the error.
Code block:
1.def add_2_if_geq_3(x):
2. if x is None:
3. return x
4. elif x >= 3:
5. return x+2
6. return x
7.
8.add_2_if_geq_3_udf = udf(add_2_if_geq_3)
9.
10.transactionsDf.withColumnRenamed("predErrorAdded", add_2_if_geq_3_udf(col("predError")))

• A.The operator used to adding the column does not add column predErrorAdded to the DataFrame.
• B.The Python function is unable to handle null values, resulting in the code block crashing on execution.
• C.UDFs are only available through the SQL API, but not in the Python API as shown in the code block.
• D.Instead of col("predError"), the actual DataFrame with the column needs to be passed, like so transactionsDf.predError.
• E.The udf() method does not declare a return type.

The code block displayed below contains at least one error. The code block should return a DataFrame with only one column, result. That column should include all values in column value from DataFrame transactionsDf raised to the power of 5, and a null value for rows in which there is no value in column value. Find the error(s).
Code block:
1.from pyspark.sql.functions import udf
2.from pyspark.sql import types as T
3.
4.transactionsDf.createOrReplaceTempView('transactions')
5.
6.def pow_5(x):
7. return x**5
8.
9.spark.udf.register(pow_5, 'power_5_udf', T.LongType())
10.spark.sql('SELECT power_5_udf(value) FROM transactions')

• A.The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and the SparkSession cannot access the transactionsDf DataFrame.
• B.The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and Spark driver does not call the UDF function appropriately.
• C.The pow_5 method is unable to handle empty values in column value, the UDF function is not registered properly with the Spark driver, and the name of the column in the returned DataFrame is not result.
• D.The returned DataFrame includes multiple columns instead of just one column.
• E.The pow_5 method is unable to handle empty values in column value and the name of the column in the returned DataFrame is not result.

Which of the following describes characteristics of the Spark driver?

• A.If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.
• B.The Spark driver requests the transformation of operations into DAG computations from the worker nodes.
• C.The Spark driver's responsibility includes scheduling queries for execution on worker nodes.
• D.In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.
• E.The Spark driver processes partitions in an optimized, distributed fashion.