CDP-3002 무료 덤프문제 온라인 액세스

You're building an Airflow DAG to automate data quality checks on the output of your ETL pipeline. The checks involve performing various data validation tasks like checking for missing values, ensuring data type consistency, and verifying data integrity based on specific business rules. How can you implement these checks within Airflow?

• A.Use the PythonOperator to write custom Python scripts for each individual check and chain them together in the DAG.
• B.Utilize Python libraries like Pandas or Spark for data manipulation and validation within the PythonOperator.
• C.Leverage dedicated Airflow operators like BigQueryCheckOperator or S3KeySensor (these operators are specific to certain data sources and not generally applicable for all data quality checks).
• D.All of the above

You're deploying your Airflow ETL pipelines to a production environment. What are some best practices to ensure reliability and scalability?

• A.Implement robust error handling and retry mechanisms within your DAGs.
• B.Configure Airflow to run with high resource limits to handle unexpected spikes in workload.
• C.Utilize version control for your DAG code and configuration files to track changes and facilitate rollbacks.
• D.All of the above

For automating the deployment of Spark applications within a Cloudera Data Engineering (CDE. environment using the CDE CLI, what is the primary consideration to ensure seamless integration with existing CI/CD pipelines?

• A.Converting all Spark code to be Kubernetes-native before deployment
• B.Embedding CDE CLI commands within pipeline scripts and managing credentials securely
• C.Utilizing the cde job create command with appropriate flags for version control integration
• D.Ensuring all Spark applications are containerized before deployment

In a PySpark application running on Kubernetes, you want to enable dynamic allocation of Executors. Which configuration setting is essential to turn on this feature?

• A.'spark.dynamicAllocation.enabled'
• B.'spark.executor.instances'
• C.'spark.kubernetes.executor.dynamicAllocation'
• D.'spark.kubernetes.dynamicAllocation.enabled'

When using Apache Airflow to schedule quality checks, which strategy helps ensure that checks are only run on the most recent data partition?
A Use the LatestOnlyOperator to skip tasks that are not the latest in a series of executions.

• A.Apply the depends_on_past=True parameter to the quality check task.
• B.Utilize the execution_date in your SQL query to target the latest partition.
• C.Implement a custom PythonOperator that queries the database for the latest partition.