Databricks-Certified-Professional-Data-Engineer-KR 문제 6
아래 쿼리를 사용하여 Delta Lake 테이블이 생성되었습니다. 원래 쿼리에 오타가 있음을 확인하고 아래 코드를 실행했습니다. ALTER TABLE prod.sales_by_store RENAME TO prod.sales_by_store 두 번째 명령어를 실행하면 어떤 결과가 나타날까요?
정답: A
The query uses the CREATE TABLE USING DELTA syntax to create a Delta Lake table from an existing Parquet file stored in DBFS. The query also uses the LOCATION keyword to specify the path to the Parquet file as /mnt/finance_eda_bucket/tx_sales.parquet. By using the LOCATION keyword, the query creates an external table, which is a table that is stored outside of the default warehouse directory and whose metadata is not managed by Databricks. An external table can be created from an existing directory in a cloud storage system, such as DBFS or S3, that contains data files in a supported format, such as Parquet or CSV. The result that will occur after running the second command is that the table reference in the metastore is updated and no data is changed. The metastore is a service that stores metadata about tables, such as their schema, location, properties, and partitions. The metastore allows users to access tables using SQL commands or Spark APIs without knowing their physical location or format. When renaming an external table using the ALTER TABLE RENAME TO command, only the table reference in the metastore is updated with the new name; no data files or directories are moved or changed in the storage system. The table will still point to the same location and use the same format as before. However, if renaming a managed table, which is a table whose metadata and data are both managed by Databricks, both the table reference in the metastore and the data files in the default warehouse directory are moved and renamed accordingly. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "ALTER TABLE RENAME TO" section; Databricks Documentation, under "Metastore" section; Databricks Documentation, under "Managed and external tables" section.
Databricks-Certified-Professional-Data-Engineer-KR 문제 7
데이터 엔지니어링 팀의 주니어 멤버가 Databricks 노트북의 언어 상호 운용성을 탐색하고 있습니다. 아래 코드의 목표는 geo_lookup 테이블에 있는 아프리카 대륙 국가에서 발생한 모든 판매 내역을 표시하는 것입니다. 코드를 실행하기 전에 현재 데이터베이스에서 SHOW TABLES 명령을 실행하면 데이터베이스에 geo_lookup과 sales라는 두 개의 테이블만 있는 것으로 나타납니다. 대화형 노트북에서 이러한 명령 셀을 순서대로 실행했을 때의 결과를 올바르게 설명하는 문장은 무엇입니까?
정답: E
This is the correct answer because Cmd 1 is written in Python and uses a list comprehension to extract the country names from the geo_lookup table and store them in a Python variable named countries af. This variable will contain a list of strings, not a PySpark DataFrame or a SQL view. Cmd 2 is written in SQL and tries to create a view named sales af by selecting from the sales table where city is in countries af. However, this command will fail because countries af is not a valid SQL entity and cannot be used in a SQL query. To fix this, a better approach would be to use spark.sql() to execute a SQL query in Python and pass the countries af variable as a parameter. Verified References: [Databricks Certified Data Engineer Professional], under "Language Interoperability" section; Databricks Documentation, under "Mix languages" section.
Databricks-Certified-Professional-Data-Engineer-KR 문제 8
데이터 엔지니어링 팀은 매일 밤 일괄 업데이트를 통해 집계 통계 테이블을 관리합니다. 이 테이블에는 전날 총 매출액과 함께 지난 7일간, 연간 누계, 분기 누계 등 다양한 기간의 총계 및 평균 매출액이 포함됩니다. 이 테이블의 이름은 store_saies_summary이며 스키마는 다음과 같습니다. daily_store_sales 테이블에는 store_sales_summary 테이블을 업데이트하는 데 필요한 모든 정보가 포함되어 있습니다. 이 테이블의 스키마는 다음과 같습니다. store_id INT, sales_date DATE, total_sales FLOAT daily_store_sales 테이블이 Type 1 테이블로 구현되어 있고, total_sales 열이 수동 데이터 검토 후 조정될 수 있는 경우, store_sales_summary 테이블에 정확한 보고서를 생성하는 가장 안전한 방법은 무엇일까요?
정답: E
The daily_store_sales table contains all the information needed to update store_sales_summary. The schema of the table is: store_id INT, sales_date DATE, total_sales FLOAT The daily_store_sales table is implemented as a Type 1 table, which means that old values are overwritten by new values and no history is maintained. The total_sales column might be adjusted after manual data auditing, which means that the data in the table may change over time. The safest approach to generate accurate reports in the store_sales_summary table is to use Structured Streaming to subscribe to the change data feed for daily_store_sales and apply changes to the aggregates in the store_sales_summary table with each update. Structured Streaming is a scalable and fault-tolerant stream processing engine built on Spark SQL. Structured Streaming allows processing data streams as if they were tables or DataFrames, using familiar operations such as select, filter, groupBy, or join. Structured Streaming also supports output modes that specify how to write the results of a streaming query to a sink, such as append, update, or complete. Structured Streaming can handle both streaming and batch data sources in a unified manner. The change data feed is a feature of Delta Lake that provides structured streaming sources that can subscribe to changes made to a Delta Lake table. The change data feed captures both data changes and schema changes as ordered events that can be processed by downstream applications or services. The change data feed can be configured with different options, such as starting from a specific version or timestamp, filtering by operation type or partition values, or excluding no-op changes. By using Structured Streaming to subscribe to the change data feed for daily_store_sales, one can capture and process any changes made to the total_sales column due to manual data auditing. By applying these changes to the aggregates in the store_sales_summary table with each update, one can ensure that the reports are always consistent and accurate with the latest data. Verified References: [Databricks Certified Data Engineer Professional], under "Spark Core" section; Databricks Documentation, under "Structured Streaming" section; Databricks Documentation, under "Delta Change Data Feed" section.
Databricks-Certified-Professional-Data-Engineer-KR 문제 9
미국에 본사를 둔 한 소규모 회사가 최근 인도의 컨설팅 회사와 계약을 맺고 인공지능 애플리케이션에 필요한 여러 새로운 데이터 엔지니어링 파이프라인을 구축하기로 했습니다. 이 회사의 모든 데이터는 미국 내 지역 클라우드 스토리지에 저장됩니다. 회사 워크스페이스 관리자는 계약직 직원들이 사용하는 Databricks 워크스페이스를 어디에 배포해야 할지 확신하지 못하고 있습니다. 모든 데이터 거버넌스 고려 사항을 감안할 때, 다음 중 어떤 설명이 이 결정에 정확한 근거를 제시합니까?
정답: C
This is the correct answer because it accurately informs this decision. The decision is about where the Databricks workspace used by the contractors should be deployed. The contractors are based in India, while all the company's data is stored in regional cloud storage in the United States. When choosing a region for deploying a Databricks workspace, one of the important factors to consider is the proximity to the data sources and sinks. Cross-region reads and writes can incur significant costs and latency due to network bandwidth and data transfer fees. Therefore, whenever possible, compute should be deployed in the same region the data is stored to optimize performance and reduce costs. Verified References: [Databricks Certified Data Engineer Professional], under "Databricks Workspace" section; Databricks Documentation, under "Choose a region" section.
Databricks-Certified-Professional-Data-Engineer-KR 문제 10
다음 로직을 사용하여 activity_details라는 Delta 테이블에 CHECK 제약 조건이 성공적으로 추가되었습니다. 배치 작업이 테이블에 새 레코드를 삽입하려고 시도하고 있으며, 여기에는 위도 = 45.50, 경도 = 212.67인 레코드가 포함됩니다. 다음 중 이 일괄 삽입 작업의 결과를 가장 잘 설명하는 문장은 무엇입니까?
정답: B
The CHECK constraint is used to ensure that the data inserted into the table meets the specified conditions. In this case, the CHECK constraint is used to ensure that the latitude and longitude values are within the specified range. If the data does not meet the specified conditions, the write operation will fail completely and no records will be inserted into the target table. This is because Delta Lake supports ACID transactions, which means that either all the data is written or none of it is written. Therefore, the batch insert will fail when it encounters a record that violates the constraint, and the target table will not be updated. References: * Constraints : https://docs.delta.io/latest/delta-constraints.html * ACID Transactions : https://docs.delta.io/latest/delta-intro.html#acid-transactions
