Orchestrating Hybrid Workflows with Apache Airflow

What is the focus of your work these days?

I am a principal developer advocate for open source, which means that I work with builders to make AWS the best place to run open source workloads. Sometimes that is about helping to spread awareness of open source projects. Sometimes it's about helping open source projects figure out how to best work on AWS. This means I do demos, I write blogs and I speak at events like QCon.

And what is the motivation for your talk?

Over the last 12 months, I've been spending a lot of time on a very popular open source project called Apache Airflow, which is used by data engineering teams to help automate their data pipelines. And I also work with customers who are moving to the cloud, and they're increasingly looking at solutions that work both within the cloud but also outside of the cloud. So how do they work in hybrid environments? The motivation of this talk is how can you take an open source project such as Apache Airflow, which is a workflow orchestrator, and extend that so that they can work and orchestrate workflows anywhere.

You touched on this a little bit with the last question, but how would you describe the persona and level of the target audience for your session?

The personas that I typically go for are data engineer, and sysadmins. These are the folks that typically have to understand what solutions and tools are available to help them solve problems, understand the capabilities and how to use them,  architect the solution, and ultimately do the wiring and deployment of all this stuff. It's those two broad buckets of personas who actually I talk to within my demo.

What would you like this persona to walk away with at the end of your presentation?

The key thing here is to understand how, first of all, Apache Airflow can help you orchestrate data pipelines in hybrid environments, that you've got a number of different ways you can do that, and each of those have got different tradeoffs. I'm hoping that they'll come away with a better understanding of some of the options they've got, as well as some of the new capabilities, the new innovations that we've released that make that really easier for these teams.