When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to discuss how his team uses Airflow across client projects, including a financial services scoring use case and a POC applying production-grade orchestration to RAG and GenAI data pipelines.


Key Takeaways:

  • 00:00 Introduction.
  • 01:00 What Accion Labs does as a technology consulting and services firm working across BFSI, healthcare, and retail.
  • 02:00 Chandan's role at the intersection of data engineering and GenAI, building pipelines one week and RAG-based agents the next.
  • 04:20 Why Airflow tends to win client evaluations: infrastructure agnostic, no cloud lock-in, fine-grained control over pipeline logic.
  • 06:00 Containerizing Airflow on Kubernetes or VMs so migrations between clouds don't require a rewrite.
  • 08:14 The loan eligibility scoring use case for a financial services client, and replacing fragile cron jobs with a single Airflow DAG end to end, cutting effort by about 25%.
  • 10:42 Triggering strategy: S3 file sensors as the primary trigger handling 90% of runs, plus a scheduled fallback as a safety net.
  • 12:53 End-to-end flow inside Airflow: ingestion into the data lake, validation and transformation with credit bureau joins, containerized model inference, and writeback to the loan management system.
  • 15:56 The AI orchestration POC and why the data feeding GenAI models needs the same rigor as any production pipeline.
  • 18:09 Using Airflow to detect document changes and re-chunk and re-embed only what changed, with quality thresholds and rollback before promoting to live.
  • 20:23 The roadmap: model evaluation pipelines, multi-agent orchestration, and provider packages for LangChain, OpenAI, and Hugging Face.
  • 22:32 Wishlist for Airflow: native event-driven triggers beyond polling sensors, first-class observability for AI workloads, and better dynamic DAG generation at scale.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

Rss Apple Podcaster
Podden och tillhörande omslagsbild på den här sidan tillhör Astronomer. Innehållet i podden är skapat av Astronomer och inte av, eller tillsammans med, Poddtoppen.