Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill.
The cluster was never the problem. The rule that keeps your joins and group-bys correct is the same rule that lets one task carry the whole job, and no amount of extra workers moves that work off it. Sometimes the stuck task turns out to be worse than slow.
In this episode:
- Why more workers and more shuffle partitions both leave the stuck task exactly where it was
- Where heavy keys come from, and the one source that is usually a bug in a performance costume
- Why plenty of skewed aggregations never slow down, and where adaptive query execution on Databricks does and doesn't split the load
- When salting pays, when a different approach beats it, and when the right call is to leave skew alone
This episode is for Databricks data engineers whose reflex for a slow Spark stage is a bigger cluster. Whether you're mid-level and tired of paying for compute that changes nothing, or senior and about to be asked "how do you handle data skew?" in an interview, you'll walk away able to tell a skew problem from a capacity problem before you touch the cluster settings.
---
Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.
Follow The Databricks Data Engineer for new episodes every week.
LinkedIn: linkedin.com/in/jrlasak
Newsletter: dataengineer.wiki
#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
