Managed Workflow for Apache Airflow (MWAA): What's New? - Airflow Summit 2025

0:06 This is going to cover MWA or manage workflows for Apache Airflow from the from AWS. So I'll kick it away to you Crystal. Go ahead. >> Awesome. Hi everyone. My name is Crystal and Daddy. I'm a worldwide go to market specialist here at AWS covering Amazon managed workflows for Apache Airflow and I'm joined today by Rees. if you want to give a brief introdu >> I am the software developer manager for the open-source team that is sort of conjoined with the MWA team and also work very closely with the MWA service team as well too.

0:48 >> Nice. So we have a lot of great content here today. We hope that as you walk away from the session, you have a good understanding of how AWS contributes to the Apache Airflow project. Also, the core features of Amazon managed workflows for Apache Airflow, which we call MWAA because it is such a mouthful. We'll also talk about recent and upcoming releases of MWAA. And then throughout, I'll weave through how customers are using MWA today. And then we'll wrap up with resources on how you can get started. I'd love to know by a show of hands how many of you are interested in MWA or have not started using the service at all.

1:27 >> Cool. >> Cool. >> Awesome. Well, thank you so much for joining us. So to get started, Rees will go through AWS involvement in the Apache Airflow project. >> So AWS has this I would say unique at least to AWS in terms of our approach to Apache Airflow and the service team. as I mentioned we do have an open- source team that is solely dedicated to Apache Airflow in the open source community and we do have a service team and the service team is the team that really works on MWA as a product and there is a clear-cut boundary in terms of how these two teams kind of coexist.

2:09 if you think about it from a customer point of view wanting a feature in airflow that is consuming MWA and that feedback loop sort of goes to the open source team and the open source team will then work on said feature in an open-source perspective and given that MWA takes vanilla airflow as is as part of the product that is how that feature eventually percolates back into not just the community but to MWA customers. So if we look at this year's contributions, it's you know not a massive list as as because the community was very focused on Airflow 3. but the team small team of five with three committers and then from yesterday you guys heard that Vincent Beck who's part of this team just announced to be a PMC member. we did you know a lot of contributions in terms of bug fixing and testing on Airflow 3.0 O and you know subsequent minor versions. Vincent was the author and key deliverer for the event driven scheduling. That was also a talk that happened yesterday. and also he is community known as the O expert. So he worked on the simple off manager that was launched with the 3.0 platform and then also in yesterday's talk with Rummit and Dennis deadline alerts. So phase one was launched in 3.1 and they're also working on some additional stuff in new feature updates that it will be in 3.2 timeline and that's part of the you know what's coming next. There are Nico and Vincent are also very active on the whole multi-team functionality within Airflow and there is a project board and there's a lot of ways that a lot of folks can get involved from an open source perspective and that is one of the biggest things my team our team within AWS open source is targeting for the next big release in terms of Airflow 32.

4:10 >> Great. Now we're going to do an overview of MWAA. All right, so let's talk through why customers choose to run Apache Airflow on MWAA. I know there's a lot of text here, so we'll just go one column at a time. I think at a high level though, customers choose MWAA because you're able to run Apachi Airflow at scale with a heightened level of enterprise readiness. So one dimension is really performance and scalability. Our customers really appreciate how MWA has a wide variety of AWS native integrations and on the next slide I'll show you some of them. But this allows this enables you to seamlessly integrate with other AWS services and make that process much more easier for you. We also have autoscaling and this enables you to really limit performance bottlenecks. You can also run complex high concurrency DAGs with MWAA and then also run your workflow environment in an isolated environment. The next dimension is high availability and enterprise security and this is composed of a couple different features. One is we actually deploy components of MWA in various multiple in various availability zones. So you get that heightened level of availability. There's also automated failover. So you also get high availability as well. We take on the burden of managing the security updates.

5:33 So one less thing you have to worry about. We also have built-in security and compliance with leading compliance standards like SOCK123 and HIPPA. We also see customers have a lower overall cost of ownership because it is a managed service. That's less operational overhead that you have. Your team can focus on innovation rather than, you know, doing bug fixes and all of that manual stuff. In addition to autoscaling helping with performance, it also helps you to prevent overprovisioning. So you're not paying for resources that you're not using. And then we also have pay as you go pricing. And then last but not least is MWAA is a fully managed open-source compatible service. Like Reess said before, we take everything that's in open source Airflow and harden it, make it available for our enterprise customers when we feel like it's reached our level of security and performance standards. So everything that's in Apache Airflow, you can use in MWA. And this includes running MWA on other cloud service providers, which is I feel like a fact that not a lot of people know.

6:39 You don't have to use MWA only on AWS. You can use it on any cloud service provider. You can also use it on prem. And then we also do our image development in public in the open on GitHub. So if you want to see how we're developing our code, you're able to do local testing even before we deploy the code because it's available for you to see. We'll kind of breeze through this because we talked about some of the features already like how it's 100% oss Apache Airflow, but a couple other things is that you get granular monitoring and observability since it's integrated into Cloudatch. And then we also have integrations with firstparty and thirdparty services. On the right hand side, you'll see some of our native AWS integrations. We have more and you can see the entire list on the Amazon provider package. But in addition to offering the native integrations, we also have a health dashboard. And what this means is that you're able to monitor the health of your native AWS integrations within Airflow.

7:41 I really love this quote because I feel like it encapsulates the value of MWAA. And one of our customers is the Pokemon Company. Eric Smith is a data platform engineer and he says with Amazon MWA we can focus on building reliable data pipelines that achieve business goals rather than patching and securing instances. So again, we want you to free up your time to focus on innovating, doing value additive tasks, not just, you know, the manual parts of self-managing a service.

8:16 And we see customers use MWA for a wide variety of use cases. I would say the three most common are ETL, DevOps, LLM ops, and a IML genai. ETL is probably our most common use case and one customer that does this is Amazon Global Transportation and Technology Services and they essentially provide data pipelines that ensure that your package ultimately gets to your front door when you order something from Amazon. And so this includes the shipping teams, last mile delivery, freight, all of that good stuff. and they originally had an in-house data orchestration platform, but as the demand of the service grew, they were having performance bottlenecks. So, they transformed their data pipelines with MWA.

9:01 And for context, they're running about 20,000 ETL jobs per day. And once they migrated to MWA, they're able to reduce their end-to-end workflow run times by nearly 70%. Another use case is DevOps LLM ops and this is really using MWAA to orchestrate workflows that are repetitive and also manual. And then last but not least, we're seeing a lot of popularity with using MWA for A IML and Genai, which is really exciting. One customer that uses NWA to orchestrate machine learning pipelines is Yahoo Advertising. And they use MWAA to orchestrate the machine learning pipeline that ultimately serves ad predictions. So making the ad more relevant, more aware to the end user.

9:53 Next, we're going to talk about what's new with the service. Regress and I will both tag team this section, but to start off, I'll take this slide. We're going to talk through some of the key features that MWA released in 2025. just three that we wanted to highlight. The first is in place version downgrades. So this enables you to have more flexibility over your Airflow environment. Previously you could you could either upgrade doing an in place version upgrade or you can downgrade by creating a net new environment. Both in place version downgrades you're able to downgrade to a supported minor version of Airflow. We also have graceful updates and this enables you to update your workflow environment without disrupting any ongoing tasks. And then last but not least, we have our integration with SageMaker Unified Studio. We had a really great talk on this yesterday, but essentially to level set, Amazon SageMaker Unified Studio is a cohesive platform where you can combine your data and AI with a single unified interface. And behind the scenes, you actually have MWA working as a workflow builder. So in this case, you can actually log into Amazon SageMaker Unified Studio and visually build workflow pipelines. So for teams that don't want to write their pipelines, their workflow in code or Python, you can actually visually drag and drop your workflows. And then if you want to use code, you can do that as well. So it really bridges the gap I would say between users that aren't super familiar with writing Python or Airflow but still want to orchestrate pipelines and those power users that really like writing their own code and customizing it and then Mash will take the slide.

11:41 >> Yeah. So, I had a number of people come up and ask me over the last day or so and even up to this morning about whether or not MWA time the release of Airflow 3.0 to happen just prior to this summit. and obviously all the features listed there are things that come within the the platform itself. my answer to that was not really direct but the approach I think we all sort of look at it when I say we MWA as a team and organizationally as a team and the set of responsibilities that we have to our customers is when we launched from a community perspective launched airflow 3.0 know back in April M I feel that MWA as a team took a responsibility to make sure that the solution that we bring to our customers is the best possible solution for our customers and we provide some level of stability and confidence that if we have customers that want to you know upgrade and migrate to three that that's what makes sense and as you know from an open source I'm going to be flipping hats back and forth from the open source community we went through a 3.01 a 3.02 2 3.03 and so forth up until we felt as as a team that there was some stability in 3.06. The timing just happened to line up with the Airflow Summit but 3.06 know six support in MWA was released last announced last week and it is available there for customers to start working with and obviously working with you know our support engineering and our support ops teams to to to get to that get to that migration path. and then everything as we mentioned before what's there in open source is what we're providing there for you in MWA.

13:34 Next slide. And then obviously everybody wants to know what's next. as with you know traditional AWS approach regional parity and regional expansion is a big thing because we want to get great products and great functionality and great service out to everyone wherever you are and exist in the world. the next item there is a maintenance window. So a lot of things with you know cloud providers is keeping your security patching up to date and so forth. Now it does become a little bit troublesome if you are having longunning DAGs or you have an environment that's in the midst of doing all your important ETL function work and at the same time we're saying hey we need to do some updates we need to do some security patching and so forth so maintenance window is MWA's approach to having a graceful transitional period so we'll look and make sure that we're not killing your tasks or killing any of your workflows just to go and apply a security patch.

14:36 And this is something that allows us to do this transition without interrupting the our customers workflows. And the last one obviously we all heard 3.1 was released two weeks ago I believe from the open source community and that is something that the MW team again we will look at 3.1 and see if that is a good enough fit for our to bring to our customer base or maybe it might be 31.1 or 31.2 but that evaluation has to be done with diligence to make sure again that we're bringing something stable from a platform to our customers. Oh yeah, >> resources if you want to learn more too online.

15:16 >> Yeah. >> Yes.