Practical workflow orchestration

Practical AI: Machine Learning, Data Science, LLM

Workflow orchestration has always been a pain for data scientists, but this is exacerbated in these AI hype days by agentic workflows executing arbitrary (not pre-defined) workflows with a variety of failure modes. Adam from Prefect joins us to talk through their open source Python library for orchestration and visibility into python-based pipelines. Along the way, he introduces us to things like Marvin, their AI engineering framework, and ControlFlow, their agent workflow system.

Join the discussion

Changelog++ members save 9 minutes on this episode because they made the ads disappear. Join today!

Sponsors:

• WorkOS – A platform that gives developers a set of building blocks for quickly adding enterprise-ready features to their application. Add Single Sign-On (Okta, Azure, Google, Microsoft OAuth), sync users from any SCIM directory, HRIS integration, audit trails (SIEM), free magic link sign-in. WorkOS is designed for developers and offers a single, elegant interface that abstracts dozens of enterprise integrations. Learn more and get started at WorkOS.com
• Shopify – Sign up for a $1/month trial period at shopify.com/practicalai
• Notion – Notion is a place where any team can write, plan, organize, and rediscover the joy of play. It’s a workspace designed not just for making progress, but getting inspired. Notion is for everyone — whether you’re a Fortune 500 company or freelance designer, starting a new startup or a student juggling classes and clubs.

Featuring:

• Adam Azzam – Twitter, LinkedIn
• Chris Benson – Twitter, GitHub, LinkedIn, Website
• Daniel Whitenack – Twitter, GitHub, Website

Show Notes:

• Prefect
• Marvin
• ControlFlow

Something missing or broken? PRs welcome!

Capitole

1. Welcome to Practical AI (00:00:00)

2. Sponsor: WorkOS (00:00:35)

https://workos.com

3. Workflow orchestration (00:04:09)

4. Common pain points (00:09:32)

5. What makes ML orchestration different (00:14:37)

6. Sponsor: Shopify (00:20:57)

https://www.shopify.com/practicalai

7. Intro to Prefect (00:22:46)

8. Conversion experience (00:29:52)

9. Sponsor: Notion (00:37:25)

http://notion.com/changelog

10. Adam's friend Marvin (00:39:43)

11. Agentic approach (00:44:32)

12. What's next? (00:51:50)

13. Thanks for joining us (00:57:01)

14. Outro (00:57:38)