Two proposals sit on a head of data's desk, both about the nightly pipeline that feeds the finance dashboard. One swaps the full reload for incremental loads with Auto Loader: it finishes in a fraction of the time and it's cheaper every month after the first. The other keeps the full reload, puts it on a bigger job cluster and starts it an hour earlier. It costs more. Forever. She asks one question, "What happens if this breaks on quarter close?", and approves the second one.
It's tempting to call that ignorance or politics. It's neither. The better proposal was arguing merit to someone who gets asked a completely different question upstairs, in a meeting the engineer who wrote it will never be in.
In this episode:
- The fear that filters every proposal on your director's desk, and why it isn't fear of change
- The question a tech lead, an engineering manager and a director each get asked, and why pitching a director like a tech lead loses
- Three kinds of data leader, and how to tell which one you work for from the first question they ask about any proposal
- Three things to add to any proposal with a real spread, so it gets funded without getting duller
- A two-sentence test to run before anything goes upstairs
This episode is for Databricks data engineers who have watched a better proposal lose to a duller one and couldn't explain why. Whether you're the mid-level engineer writing the pitch or the senior trying to get a platform change funded, you'll walk away able to read your leader's decisions and write proposals they can say yes to.
---
Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.
Follow The Databricks Data Engineer for new episodes every week.
LinkedIn: linkedin.com/in/jrlasak
Newsletter: dataengineer.wiki
#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
