Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.


Key Takeaways:

  • 00:00 Introduction.
  • 01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.
  • 02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.
  • 02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.
  • 04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.
  • 05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.
  • 08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.
  • 09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.
  • 11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.
  • 12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.
  • 14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.
  • 15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

Rss Apple Podcaster
Podden och tillhörande omslagsbild på den här sidan tillhör Astronomer. Innehållet i podden är skapat av Astronomer och inte av, eller tillsammans med, Poddtoppen.