The monitoring and logging area of the certified Databricks data engineer professional exam carries 10% of the total available marks. Expect around 6 questions on this topic.
I broke my rule and included a video in this list! See intro to this series of posts here.
Section 5: Monitoring & Logging
- Describe the elements in the Spark UI to aid in performance analysis, application debugging, and tuning of Spark applications.
Diagnose cost and performance issues using the Spark UI – Azure Databricks | Microsoft Learn
- Inspect event timelines and metrics for stages and jobs performed on a cluster
- Draw conclusions from information presented in the Spark UI, Ganglia UI, and the Cluster UI to assess performance problems and debug failing applications.
Web UI – Spark 3.5.1 Documentation (apache.org)
Advancing Spark – Getting Started with Ganglia in Databricks
- Design systems that control for cost and latency SLAs for production streaming jobs.
- Deploy and monitor streaming and batch jobs
One thought on “Databricks Data Engineer Professional Monitoring & Logging”