How Amazon Q Developer helped me to save days of work

Recently, I had an eye-opening experience with Amazon Q Developer that I'm excited to share. In just 15 minutes, I created a Python script that crawls the entire official Bundesliga DataHub API – a task that would have typically taken one to two full days of work. Our team needed comprehensive sports data to enhance our AI services for content localization. The challenge was to efficiently gather data on seasons, competitions, clubs, players, and official team staff members from the Bundesliga, spanning from 1908 to 2024. This was no small feat, considering the vast amount of data and the complexity of the API structure.

In this blog post, I'll take you through my journey with Amazon Q Developer, from the initial prompt to the final, comprehensive Python script. I'll share the (few) challenges I faced and the key learnings from this experience. Get comfortable, for this 15-minute love story between a solution architect and an AI coding assistant.

One of our teams was tasked with a project requiring comprehensive data from the Bundesliga DataHub API. They needed everything from historical seasons dating back to 1908 up to the most recent 2024 data, including details about competitions, clubs, players, and official team staff members. In total, this amounted to 13,498 single API requests – a daunting task. I wanted to help the team by taking the lead on this.

My traditional approach would have involved using tools like Postman, which would require:

Conservatively, I estimated this would take 1-2 full days of focused work – a significant chunk of time, especially when juggling multiple projects and deadlines. Instead of falling into old habits, I turned to Amazon Q Developer. Could it help me automate this process? Would it understand the nuances of the Bundesliga sports data API? Could it handle and understand relationships in data? Most importantly, would it truly save me time, or would I end up spending hours correcting its output?

You might already guess how it ended, but please continue reading as I take you through this experience, step by step, and show you how Amazon Q Developer not only met but exceeded my expectations, proving that sometimes, the right tool can indeed feel like magic.

With a clear understanding of the task at hand, the whole love story began with a simple prompt (sensitive information replaced with placeholders in all examples):

The response from Amazon Q was a solid baseline script. Without further instructions, Amazon Q Developer made an educated guess on the potential data structure of the response. I helped Q out by providing an example of a response from our seasons endpoint. This is where the real magic began.

Amazon Q Developer responded with an updated and precise version capable of handling the expected data structure correctly. I continued to add more instructions to expand the script's capabilities:

The following prompt template was used to download and traverse various data types:

Without some further instructions from my side, Amazon Q Developer made a suggestion on how to organize the downloaded files. As the script evolved, I provided additional instructions for better organization of the output.

When running the script, I recognized that combinations of seasons and competitions resulted in an error payload. Instead of having all these files with error payloads in my .out folder, I stopped the script for a moment and gave Amazon Q Developer feedback of this runtime behaviour:

As more and more data were crawled, I observed situations, that resulted in SSL related errors. Amazon Q Developer to the rescue, it was able to help me out and implement a retry logic with exponential backoff using this simple prompt:

After implementing a retry logic with exponential backoff to handle SSL-related errors, the script was complete. Had to remove the feed processing parts of the script due to security and compliance reasons, but this is what Amazon Q Developer created:

I had a timer running in parallel: In just 15 minutes, Amazon Q Developer and me had created a robust Python script capable of requesting, processing and downloading all relevant data feeds.

The script efficiently handled 13,498 single API requests, saving the responses in a structured format, and implementing error handling for various scenarios.

What struck me most was the accuracy and completeness of the code generated by Amazon Q Developer. I didn't need to make any manual modifications to the script. It was purely created through our back-and-forth conversation, with Amazon Q understanding the context and requirements perfectly. The quality of the output was directly related to the clarity and specificity of my prompts.

Clear, specific prompts play a crucial role in getting the desired output from Amazon Q Developer. When I provided detailed context and examples of API responses, the quality and accuracy of the generated code improved dramatically. Effective communication skills along domain expertise makes the difference when working with such AI tools.

For instance, when I shared an example of the API response structure, Amazon Q Developer was able to adapt the script precisely to handle the correct data structure. This iterative process of providing examples and getting refined code became the cornerstone of our collaboration.

Amazon Q Developer just needed one or two examples about the data structure the API returns. After that, Amazon Q was able to make a right guess on most of the following data structures of players, clubs and referees added as features afterwards.

The success of this project was largely due to the iterative approach I took with Amazon Q Developer. Instead of expecting a perfect script on the first try, I built the functionality piece by piece, adding features incrementally. This approach allowed for better control and understanding of the process, and it helped in identifying and addressing potential issues early on. Each iteration brought us closer to the final product, with prompts like:

Breaking down prompts into smaller, more manageable parts resulted in a more robust script but also allowed me to learn and adapt my interaction style with the AI assistant.

While Amazon Q Developer proved to be an incredibly powerful tool, this experience reinforced the importance of maintaining human oversight in the coding process. The AI was excellent at generating code based on my prompts, but it was my responsibility to ensure that the generated code aligned with our overall objectives and met our specific requirements. For example, when the AI made assumptions about error handling, I was able to provide more specific instructions.

My role shifted from writing code to problem-solving and providing context, allowing for a more strategic approach to development. Although Amazon Q Developer did the hard work of typing, my instructions and how I framed the problem made the difference. It's not about replacing human expertise but augmenting it, allowing us to focus on higher-level problem-solving and creativity. The code generated by Amazon Q Developer was a testament to the potential of AI's understanding of the context and requirements if we as engineers are able to shift mindsets.

As we continue to integrate AI tools into our development processes, it's crucial to cultivate this new mindset that balances technical expertise with the ability to effectively leverage AI capabilities.

In just 15 minutes, Amazon Q Developer proved its value as a powerful assistant, helping me automate a task that would have otherwise taken days. Through clear communication, an iterative approach, and a balance of AI assistance with human oversight, I was able to create a robust Python script to efficiently crawl the Bundesliga DataHub API.

There is a critical shift in how we approach software development. It's no longer just about writing code; it's about how effectively we can leverage AI to augment our capabilities. Amazon Q Developer didn't replace my expertise—it complemented it, enabling me to focus on problem-solving and strategy while the AI handled the heavy lifting.

As AI tools become more integral to our workflows, embracing this shift will not only boost productivity but also free up time for creativity, innovation (or drink more coffee with my engineer fellows). The key is to see AI as a partner, working alongside us to push the boundaries of what we can achieve. And when used effectively, it truly can feel like magic.