Treceți offline cu aplicația Player FM !
Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade
Manage episode 453266231 series 2948506
Scaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade, Staff Software Engineer at LinkedIn, shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers.
Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility.
Key Takeaways:
(01:36) LinkedIn minimizes human involvement in production to reduce errors.
(02:00) Airflow powers LinkedIn’s Continuous Deployment platform.
(05:43) Continuous deployment adoption grew from 8% to a targeted 80%.
(11:25) Kubernetes ensures scalability and flexibility for deployments.
(12:04) A custom UI offers real-time deployment transparency.
(16:23) No-code YAML workflows simplify deployment tasks.
(17:18) Canaries and metrics ensure safe deployments across fabrics.
(20:45) A gateway service ensures redundancy across Airflow clusters.
(24:22) Abstractions let engineers focus on development, not logistics.
(25:20) Multi-language support in Airflow 3.0 simplifies adoption.
Resources Mentioned:
https://www.linkedin.com/in/rahul-gade-68666818/
LinkedIn -
https://www.linkedin.com/company/linkedin/
https://airflow.apache.org/
https://kubernetes.io/
https://www.openpolicyagent.org/
https://backstage.io/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
42 episoade
Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade
The Data Flowcast: Mastering Airflow for Data Engineering & AI
Manage episode 453266231 series 2948506
Scaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade, Staff Software Engineer at LinkedIn, shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers.
Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility.
Key Takeaways:
(01:36) LinkedIn minimizes human involvement in production to reduce errors.
(02:00) Airflow powers LinkedIn’s Continuous Deployment platform.
(05:43) Continuous deployment adoption grew from 8% to a targeted 80%.
(11:25) Kubernetes ensures scalability and flexibility for deployments.
(12:04) A custom UI offers real-time deployment transparency.
(16:23) No-code YAML workflows simplify deployment tasks.
(17:18) Canaries and metrics ensure safe deployments across fabrics.
(20:45) A gateway service ensures redundancy across Airflow clusters.
(24:22) Abstractions let engineers focus on development, not logistics.
(25:20) Multi-language support in Airflow 3.0 simplifies adoption.
Resources Mentioned:
https://www.linkedin.com/in/rahul-gade-68666818/
LinkedIn -
https://www.linkedin.com/company/linkedin/
https://airflow.apache.org/
https://kubernetes.io/
https://www.openpolicyagent.org/
https://backstage.io/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
42 episoade
Toate episoadele
×![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Overcoming Airflow Scaling Challenges at Apollo GraphQL with Jonathan Rainer 43:39
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar 26:00
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant 21:19
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall 38:54
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado 24:35
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Exploring the Power of Airflow 3 at Astronomer with Amogh Desai 30:24
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta 24:11
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Maximizing Business Impact Through Data at GlossGenius with Katie Bauer 25:49
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade 27:47
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari 28:44
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino 28:29
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Inside Airflow 3: Redefining Data Engineering with Vikram Koka 30:08
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 Building a Data-Driven HR Platform at 15Five with Guy Dassa 20:25
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 The Intersection of AI and Data Management at Dosu with Devin Stein 20:18
![The Data Flowcast: Mastering Airflow for Data Engineering & AI podcast artwork](/static/images/64pixel.png)
1 AI-Powered Vehicle Automation at Ford Motor Company with Serjesh Sharma 26:11
Bun venit la Player FM!
Player FM scanează web-ul pentru podcast-uri de înaltă calitate pentru a vă putea bucura acum. Este cea mai bună aplicație pentru podcast și funcționează pe Android, iPhone și pe web. Înscrieți-vă pentru a sincroniza abonamentele pe toate dispozitivele.