Treceți offline cu aplicația Player FM !
How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari
Manage episode 450104898 series 2948506
When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs.
Key Takeaways:
(02:03) Airflow as a service streamlines Uber’s data workflows.
(06:16) Serialization boosts security and reduces errors.
(10:05) Java-based scheduler improves system reliability.
(13:40) Custom recovery model supports emergency pipeline switching.
(15:58) No-code UI allows easy pipeline creation for non-coders.
(18:12) Backfill feature enables historical data processing.
(22:06) Regular updates keep Uber aligned with Airflow advancements.
(26:07) Plans to leverage Airflow’s latest features.
Resources Mentioned:
https://www.linkedin.com/in/shahshobhit/
https://www.linkedin.com/in/maheshwarisumit/
Uber -
https://www.linkedin.com/company/uber-com/
https://airflow.apache.org/
https://airflowsummit.org/
Uber -
https://www.uber.com/tw/en/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
42 episoade
How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari
The Data Flowcast: Mastering Airflow for Data Engineering & AI
Manage episode 450104898 series 2948506
When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs.
Key Takeaways:
(02:03) Airflow as a service streamlines Uber’s data workflows.
(06:16) Serialization boosts security and reduces errors.
(10:05) Java-based scheduler improves system reliability.
(13:40) Custom recovery model supports emergency pipeline switching.
(15:58) No-code UI allows easy pipeline creation for non-coders.
(18:12) Backfill feature enables historical data processing.
(22:06) Regular updates keep Uber aligned with Airflow advancements.
(26:07) Plans to leverage Airflow’s latest features.
Resources Mentioned:
https://www.linkedin.com/in/shahshobhit/
https://www.linkedin.com/in/maheshwarisumit/
Uber -
https://www.linkedin.com/company/uber-com/
https://airflow.apache.org/
https://airflowsummit.org/
Uber -
https://www.uber.com/tw/en/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
42 episoade
Toate episoadele
×![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer 43:39
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar 26:00
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant 21:19
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall 38:54
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado 24:35
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Exploring the Power of Airflow 3 at Astronomer with Amogh Desai 30:24
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta 24:11
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Maximizing Business Impact Through Data at GlossGenius with Katie Bauer 25:49
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade 27:47
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari 28:44
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino 28:29
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Inside Airflow 3: Redefining Data Engineering with Vikram Koka 30:08
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Building a Data-Driven HR Platform at 15Five with Guy Dassa 20:25
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 The Intersection of AI and Data Management at Dosu with Devin Stein 20:18
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 AI-Powered Vehicle Automation at Ford Motor Company with Serjesh Sharma 26:11
Bun venit la Player FM!
Player FM scanează web-ul pentru podcast-uri de înaltă calitate pentru a vă putea bucura acum. Este cea mai bună aplicație pentru podcast și funcționează pe Android, iPhone și pe web. Înscrieți-vă pentru a sincroniza abonamentele pe toate dispozitivele.