Building a Serverless Web Scraper with a ~little~ lot of help from Amazon Q [Part 1]

Update: 2024-06-14: This turned out to be more than I initially though, so will be turned into a series of articles, this being the first. I'll be posting the followups in this series, and link them as a list.

All the code for this app can be found on GitHub, with the code as it was at the end of this article on the article-1 tag.

Understanding the data

Architecture Choices

Setting up Terraform

Important: You cannot use variables in the terraform block for the backend, so make sure to copy the exact names for the resources from _bootstrap.tf.

Building the Lambda Function

Thanks, can you show me how I would write a webpage scraper with Lambda to scrape all the pages linked from https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin.html with the format https://travel.state.gov/content/travel/en/legal/visa-law0/visa-bulletin/2024/visa-bulletin-for-july-2024.html - they should be linked from the first page listed. Each page has multiple sections and tables, I would like to only store the rows in the tables listed under the heading “A. FINAL ACTION DATES FOR EMPLOYMENT-BASED PREFERENCE CASES” and “B. DATES FOR FILING OF EMPLOYMENT-BASED VISA APPLICATIONS”. Can you show me how to create a Lambda function to do this and store the data in DynamoDB?

1. Install the dependencies in requirements.txt
2. Zip these up
3. Deploy it as a Lambda layer

Running the code locally

Transforming the data

Storing the data

Note: This code assumes that the employment_based_data collection contains unique combinations of filing_type, country, category, and bulletin_date. If there are duplicate entries, you may need to modify the code to handle them appropriately (e.g., by keeping only the latest entry or aggregating the data in some way).

Finishing up for now

1. Fix the data structure to store all the data
2. Test the Lambda function to see if it works
3. Add parameters to the function to be able to only select the specific set of dates you want
4. Decide how to expose the Lambda function - either using AWS API Gateway as I have before, or setting up an invocation URL
5. Set up a CI/CD pipeline to deploy it
6. Add in tests

What I Learnt

Using Amazon Q Developer

Approach and Tech Choices

Technical

Final Code

Source code for this article.

Conclusion

1. xkcd cartoon depicting if automating a task is worth the time.
2. xkcd cartoon depicting the fallacy of how you think automating a task to save you time will go vs the reality.

Prompts used in this article

Prompt 1

1. Amazon Bedrock: This is the core service you’ll use to build your generative AI application. With Bedrock, you can choose from various foundation models (FMs) like Anthropic’s Claude, AI21 Labs’ Jurassic-2, and Amazon’s Titan models. You can also create Knowledge Bases and Agents to integrate external data sources and execute complex tasks. [1]
2. AWS Lambda: You can use AWS Lambda functions to fetch data from public websites and integrate with Amazon Bedrock. Lambda functions can be triggered by various events, such as API Gateway requests or scheduled events.
3. Amazon API Gateway: If you want to expose your application as an API, you can use API Gateway to create, publish, and manage APIs that can invoke your Lambda functions.
4. Amazon S3: You might need to store the data fetched from public websites in Amazon S3 buckets, which can then be used by your Bedrock Knowledge Bases or Agents.
5. AWS IAM: You’ll need to manage permissions and roles using AWS Identity and Access Management (IAM) to allow your Lambda functions and Bedrock resources to access other AWS services securely.
6. Terraform: You can use Terraform to define and provision the necessary AWS resources, such as Lambda functions, API Gateway, S3 buckets, IAM roles, and Bedrock resources, in a declarative and reproducible manner.

1. Generative AI App Development - Amazon Bedrock Studio - AWS

This is mostly ok, except I prefer not to create the infrastructure to store the statefile by hand, so I usually have _bootstrap.tf that contains the details of my base infrastructure for Terraform, create the S3 bucket, DynamoDB locking table, and KSM key with that, and then I add in the backend config to use those resources.

Prompt 2

1. Create an S3 bucket: First, you’ll need to create an S3 bucket to store the state file. You can do this using the AWS Management Console, AWS CLI, or Terraform itself. Make sure to choose a unique bucket name and configure the appropriate access controls and versioning settings.
2. Configure the Terraform backend: In your Terraform configuration, you’ll need to specify the S3 backend for storing the state file. Create a backend.tf file with the following content:

1. Initialize the backend: Run terraform init to initialize the Terraform working directory and configure the S3 backend.
2. Configure AWS credentials: Terraform needs access to your AWS credentials to interact with the S3 bucket and other AWS resources. You can configure your AWS credentials using environment variables, shared credentials file, or other methods supported by the AWS provider.
3. Apply the configuration: Run terraform apply to create the S3 bucket (if it doesn’t exist) and store the initial state file in the bucket.

1. Best practices for managing Terraform State files in AWS CI/CD Pipeline | AWS DevOps Blog
2. Protecting sensitive data in the Terraform state file - AWS Prescriptive Guidance

Prompt 3

There is a change needed here, it should use the full heading I supplied as there are other sections starting with FINAL ACTION DATES and DATES FOR FILING. I do like how Q extracted '/visa-bulletin-for-' for the individual bulletin links.

Prompt 4

• resource "aws_dynamodb_table" "visa_bulletin_data" { ... } defines a new DynamoDB table resource.
• name = "VisaBulletinData" sets the name of the DynamoDB table.
• billing_mode = "PAY_PER_REQUEST" specifies that the table should use the pay-per-request billing mode, where you pay for the read and write throughput consumed by your application.
• hash_key = "table_name" defines the partition key (hash key) for the table as table_name.
• attribute { ... } block defines the attribute for the partition key, specifying its name (table_name) and type (S for String).

1. Step 1: Create a table - Amazon DynamoDB

Since we already defined the AWS provider for Terraform, we can discard that part.

Prompt 5

1. Create a ZIP archive (lambda_function.zip) containing your Python code (lambda_function.py).
2. Update the filename attribute in the aws_lambda_function resource with the correct path to the ZIP archive.
3. Replace the runtime attribute with the appropriate Python version if you’re using a different version.
4. Configure the AWS provider with your AWS credentials and region.

1. AWS Lambda adds support for Python 3.10

This is close, but not quite there, I don’t want to add a manual step to zip up the Python app. I’m also assuming I need to run a command to install the dependencies, or store them somewhere. I also notice that the IAM permission is only for writing to the table, not reading, but that is fine for now, we haven’t gotten to the part where we need to access the data. Let’s follow-up with another question on how to avoid the manual zipping of the file.

Prompt 6

1. The data "archive_file" "lambda_zip" block creates a ZIP archive of the Lambda function code located in the lambda_code/ directory.
2. The resource "null_resource" "lambda_package" block is used to install dependencies and create the final Lambda package. The triggers block ensures that this resource is re-created whenever the source code changes.
   • The local-exec provisioner runs a series of commands:
     • Create a Python virtual environment (venv).
     • Activate the virtual environment and install the dependencies listed in the requirements.txt file into the package/python directory.
     • Deactivate the virtual environment.
     • Change to the package/python directory and create a new ZIP archive containing the installed dependencies and the Lambda function code.
     • Remove the temporary package directory.
3. The aws_lambda_function resource is updated to use the filename and source_code_hash values from the archive_file data source. The depends_on attribute ensures that the null_resource is executed before creating or updating the Lambda function.

1. Deploying a simple Python script and it’s dependencies as AWS Lambda Function with only Terraform | by Maxim Pisarenko | Medium

The mention of ${path.module} makes me think this code is from a module, so I will need to update it. Looking at the steps, I don’t think this will work. The resource "null_resource" "lambda_package" block uses the data.archive_file.lambda_zip.output_base64sha256 as a trigger to run the commands in provisioner "local-exec", which makes sense as it is a way to figure out if it should trigger a new build based on if anything has changed. Further down it doesn’t use the zip created with this local-exec though, just the original with just the source code, so will need to update this.

Prompt 7

1. Install pipreqs using pip:pip install pipreqs
2. Navigate to the directory containing your Python file (lambda_function.py).
3. Run the pipreqs command with the path to your Python file:pipreqs /path/to/lambda_function.py
   This will generate a requirements.txt file in the same directory, containing the list of dependencies and their versions based on the imports in your Python file. [2]Alternatively, you can use the following command to generate the requirements.txt file in a specific directory:pipreqs /path/to/lambda_function.py --force --savepath /path/to/requirements/directory
   This command will create (or overwrite) the requirements.txt file in the specified /path/to/requirements/directory.
4. Review the generatedrequirements.txt file and make any necessary adjustments or additions. [3]
5. In your Terraform configuration, update the local-exec provisioner in the null_resource block to use the correct path to the requirements.txt file:

1. Replace /path/to/requirements/directory/requirements.txt with the actual path to your requirements.txt file.

1. Is there a way to automatically add dependencies to requirements.txt as they are installed?
2. How to Specify Non-PyPI Requirements in Your ‘requirements.txt’ File | by Dinesh Kumar K B | Python in Plain English
3. Requiring only one of two dependencies in a requirements file

Prompt 8

1. Create a new Python file, e.g., local_test.py, in the same directory as your lambda_function.py file.
2. In local_test.py, import the lambda_handler function from your lambda_function.py file:from lambda_function import lambda_handler
3. Define a mock event object and a mock context object. The event object should mimic the structure of the event that your Lambda function expects to receive when invoked. The context object can be a simple dictionary or a custom class that implements the required methods and attributes.Here’s an example of a mock event object:

1. Define Lambda function handler in Python - AWS Lambda

Prompt 9

For this prompt, I had unfortunately closed my VSCode instance, and had to restart the conversation.

1. Parse the HTML file using BeautifulSoup

1. Terraform code to create the DynamoDB table [3]

1. Working with DynamoDB tables in Java - Amazon DynamoDB

Prompt 10

1. Imported the re module for regular expression operations and the datetime module from the Python standard library.
2. Added a regular expression pattern date_pattern to match the date portion of the URL (e.g., “july-2024” in “visa-bulletin-for-july-2024.html”).
3. Used re.search to find the match in the URL, and extracted the month name and year from the match groups.
4. Created a datetime object date_obj using the extracted month name and year, setting the day to the first of the month.
5. Added the date_obj to the dictionary in the employment_based_data list as the value for the “Date” key.

Prompt 11

The code blocks are added here only for readability, the prompt contained it as pasted in-line code. Given that I have the following collection built up using:

1. We start by importing the defaultdict from the collections module, which will help us group the data efficiently. [1]
2. We create an empty list transformed_data to store the transformed data.
3. We use a nested defaultdict to group the data by filing_type and category. The outer defaultdict groups by filing_type, and the inner defaultdict groups by category. For each item inemployment_based_data, we append a dictionary containing the country, bulletin_date, and date to the appropriate group.
4. Next, we iterate over the grouped data (grouped_data). For each filing_type and category combination, we create another defaultdict to group the data by country.
5. For each country group, we create a list of dictionaries containing the bulletin_date and date for that country.
6. Finally, we append a dictionary to transformed_data with the filing_type, category, and a list of dictionaries containing the country and its history (the list of bulletin_date and date dictionaries).

1. defaultdict

Prompt 12

Prompt 13

Again, codeblock is just for readability. Thanks, that helped me find the issue. I now want to relook how the data will be stored in DynamoDB. I currently have the table defined using Terraform as:

• The hash_key is set to "pk" (partition key), and the range_key is set to "sk" (sort key). Both keys are of type "S" (String).
• A global_secondary_index named "CountryIndex" is defined, which has the same hash_key and range_key as the table's primary key. The projection_type is set to "ALL", which means that all attributes will be projected into the index. [1]

• We construct the pk (partition key) by combining the filing_type and category with a delimiter (#).
• We construct the prefix for the sk (sort key) by combining the country with a delimiter (#).
• We use the query method of the DynamoDB table to retrieve items where the pk matches the constructed value, and the sk begins with the constructed prefix.
• The retrieved items are stored in the items variable.

1. Creating an index - Amazon DynamoDB

Prompt 14

Prompt 15

1. We're using a nested defaultdict with three levels: filing_type, category, and country. This allows us to group the data by all three keys.
2. In the grouping step, we're appending the bulletin_date and date dictionaries to the innermost defaultdict for each unique combination of filing_type, category, and country.
3. In the transformation step, we iterate over the countries dictionary for each filing_type and category combination. For each country, we create a dictionary with the country key and the history data (the list of bulletin_date and date dictionaries).
4. We append these dictionaries to a countries_list, which is then added to the transformed_data list.

Prompt 16

1. A duplicates_found flag is initialized to False.
2. Before appending the bulletin_date and date dictionaries to the innermost defaultdict, we create a tuple key with the combination of filing_type, country, category, and bulletin_date.
3. We check if this key already exists in the innermost defaultdict for the corresponding filing_type, category, and country. If it does, it means we have a duplicate entry, and we print a message indicating the duplicate entry.
4. If the key doesn't exist, we append the bulletin_date and date dictionaries to the innermost defaultdict, and we also append the key tuple to the same list. This will help us identify and remove the key tuple later.
5. After the grouping step, we check the duplicates_found flag. If it's True, we print a message indicating that duplicates were found, and the user should review the data.
6. If no duplicates were found, we proceed with the transformation step. However, before appending the history data to the countries_list, we remove the key tuple from the history_data list using the pop() method.

Prompt 17

1. Create a new DynamoDB table using Terraform: [1]Create a new file called processed_urls_table.tf and add the following Terraform code to create a DynamoDB table for storing the processed URLs:

1. This Terraform code creates a DynamoDB table named ProcessedURLs with a partition key (hash key) named url of type string (S). You can adjust the read_capacity and write_capacity values based on your application's requirements.
2. Check if the URL has been processed before:Before processing a URL, you can check if it has already been processed by querying the ProcessedURLs table. Here's an example Python code snippet:

1. sdf
   In this code:
   • We create a DynamoDB resource and get a reference to the ProcessedURLs table.
   • For each URL in the visa_bulletin_links list, we use the get_item method to check if the URL already exists in the ProcessedURLs table.
   • If the URL exists (i.e., 'Item' is in the response), we print a message indicating that the URL has already been processed and skip to the next URL using continue.
   • If the URL hasn't been processed, we print a message indicating that we're processing the URL, and then execute your code to process the URL.
   • After successfully processing the URL, we store the URL in the ProcessedURLs table using the put_item method.

1. Creating a DynamoDB Table with CloudFormation and Adding Items at Creation Time | AWS re:Post

Prompt 18

• The first new statement allows the GetItem and PutItem actions on the ProcessedURLs DynamoDB table. This will allow your application to check if a URL has been processed and store processed URLs in the table.
• The second new statement allows the GetItem and Query actions on the VisaBulletinData DynamoDB table. This will allow your application to retrieve data from the VisaBulletinData table.

Prompt 19

1. We define the filing_type, category, and country values for which we want to extract the data.
2. We use a list comprehension to filter the employment_based_data collection based on the filing_type, category, and country values. The resulting filtered_data list contains only the entries that match the specified criteria.
3. We use the sorted function to sort the filtered_data list in descending order based on the bulletin_date value. The key parameter of the sorted function is set to itemgetter('bulletin_date'), which tells the sorted function to use the bulletin_date value as the sorting key. The reverse=True argument ensures that the sorting is done in descending order.
4. The sorted data is stored in the sorted_data list.
5. Finally, we iterate over the sorted_data list and print each entry's bulletin_date and date values.

Prompt 20

• I have learned a lot from this experience.
• She had learned to play the piano when she was a child.
• The students learned their lesson after failing the test.

• I have learnt a lot from this experience.
• She had learnt to play the piano when she was a child.
• The students learnt their lesson after failing the test.

1. Learning to Learn (again)