Unlocking Apache Airflow upgrades - how to simplify updating your legacy DAGs
Let me show you how you can unlock your Apache Airflow upgrade path by using Amazon Q Developer Agent for software development to simplify how to update your Airflow DAG's.
Ricardo Sueiras
Amazon Employee
Published Sep 30, 2024
In the past year or so, I have had the chance to speak at meet-ups and conferences about some of the new innovations in Apache Airflow. I wanted to address an issue that kept coming up during those talks, when I asked what was stopping you from using the latest versions of Apache Airflow.
With the recent announcement on the launch of Apache Airflow 2.10, I thought I would share how you can use Amazon Q Developer Agent for software development to simplify this task, potentially unlocking your path to upgrades to the latest versions of Apache Airflow.
Getting started
I had a number of DAGs that I had created as part of the initial launch of Managed Workflows for Apache Airflow (MWAA), when it was at version 1.12. I wanted to run these on my local Apache Airflow 2.9.2 instance, but they failed with various errors. From speaking with many of you, this sounds like a typical use case.
I created a new workspace in my local machine, and I copied my legacy DAG. I then went and grabbed the source code from the Amazon Provider package, using version 8.24.0 (using the tag feature to search on the GitHub repo for the right branch of the repo). This is what my local workspace looked like.
Tip! I have found that when using /dev or @workspace to use local files to provide additional context, it is worth trying to shape the outcomes by reviewing and pruning the local files. The more specific you can make it, the better outcomes I have seen.
I then launched VSCode, opening up the legacy-dag.py file. Keeping this file open is important as it provides valuable context when invoking the Amazon Q Developer Agent.
Now that I had everything setup, I opened up the Amazon Q Developer chat interface, and invoked the Amazon Q Developer Agent for software development by using /dev
I then entered the following prompt:
In my local workspace I have a legacy Airflow DAG called legacy-dag.py that works only with Apache Airflow 1.x. Can you update the DAG so that it will work with Airflow 2.9.2 using the latest Airflow provider package that is included in the apache_airflow_providers_amazon_8_24 directory
After a few seconds we can see Amazon Q Developer Agent start to work. It starts by exploring the current code base and then using the prompt, start to break down the problem you have given it. As it does this, it will review existing files and start creating new ones - you will see this in realtime and be given a running commentary in the Amazon Q Developer chat interface.
After a minute or so, it will finish and you will be presented with code which you can review. In my example, it was able to determine changes in the imports that were needed, as well as modifications to the Airflow Operators. After reviewing the changes, I accept the code changes.
I now am able to take this updated file and try and deploy this on my local Airflow 2.9.2 environment. As you will see in the video below, it worked first time.
This is just a simple example, but I hope it shows the power of next generation coding assistants like Amazon Q Developer, and how they can really help developer to unlock better outcomes. No one likes updating legacy code, but if tools like Amazon Q Developer can help, this is going to make a lot of folk happy.
Check out this short video where I show you this in action.
If you want to learn more, check out the Amazon Q Developer Dev Centre which contains lots of resources to help get you started. You can also get more Amazon Q Developer related content by checking out the dedicated Amazon Q space, and if you are looking for more hands on content, take a look at my interactive tutorial that helps you get started with Amazon Q Developer.
I would love to hear how you are using Amazon Q Developer to improve your work as a developer, get in touch by using the comments below.
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.