Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off.


Key Takeaways:

  • 00:00 Introduction.
  • 01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry.
  • 03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run.
  • 05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action.
  • 06:58 Why XComs don't solve this problem: they get reinitialized on every retry.
  • 09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403.
  • 11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase.
  • 15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs.
  • 21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator.
  • 23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run.
  • 25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

Rss Apple Podcaster
Podden och tillhörande omslagsbild på den här sidan tillhör Astronomer. Innehållet i podden är skapat av Astronomer och inte av, eller tillsammans med, Poddtoppen.