Adding parameters to my Serverless Web Scraper API [part 3]

In the previous articles, I built a serverless API that scrapes the data from the USCIS monthly visa bulletins, and stores them in an Amazon DynamoDB table. After jumping in and defining the whole infrastructure with Terraform, I realised that I should probably first ensure the code is working. Now it is time to add some parameters to our function so that it will return the data from categories other than the hard-coded EB-3 one, with some help from Amazon Q Developer :)

Updating the Scrape and Storage Logic

Before we get to that though, we need to update the logic around the lookups and storing the data. Currently, it will loop over all the URLs on the main page, extract the ones that contain /visa-bulletin-for-, and then do a lookup in the DynamoDB table ProcessedURLS to see if that URL was processed. Only the data from pages that are not in that database table will be scraped, added to a single collection, then that whole collection will be stored. We need to update the code so that it stores the data as it scrapes a URL, and before writing the processed entry - if there was an error, it currently will not reprocess that page.

Fixing Up Processing and Storing Our Data

To get started, let’s see what we need to change. Ok, only a small change is needed to lambda_handler for the loop where we process and store the data, updated version is now:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
    # Scrape data from each visa bulletin page
    for link in visa_bulletin_links:
        if '2022' in link or '2023' in link or '2024' in link:
            # Check if the URL has been processed
            response = processed_urls_table.get_item(Key={'url': link})
            if 'Item' in response:
                print(f"Skipping URL: {link} (already processed)")
                continue

            # Process the URL
            print(f"Processing URL: {link}")
            url = f"https://travel.state.gov{link}"
            url_data = scrape_visa_bulletin(url)
            data.extend(scrape_visa_bulletin(url_data))

            # Store the data
            store_data(url_data)

            # Store the processed URL in DynamoDB
            processed_urls_table.put_item(Item={'url': link})

Surprisingly little effort, so not much of a distraction.

Defining Some Enums

Looking at the existing parameters (with current defaults), I suspect an enum would work, but I don’t know how this is done in Python:

1
def read_data_locally(data, filing_type = 'Final Date', category = '3rd', country = 'All Chargeability Areas Except Those Listed'):

The response looks to be what I need (minus the incorrect country, plus the missing ones), so let’s add some enums!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from enum import Enum

class FilingType(Enum):
    FINAL_DATE = 'Final Date'
    DATES_FOR_FILING = 'Dates for Filing'

class Category(Enum):
    FIRST = '1st'
    SECOND = '2nd' 
    THIRD = '3rd'
    FOURTH = '4th'
    OTHER_WORKERS = 'Other Workers'

class Country(Enum):
    ALL_AREAS = 'All Chargeability Areas Except Those Listed'
    CHINA = 'CHINA-mainland born'
    INDIA = 'INDIA'
    MEXICO = "MEXICO"
    PHILIPPINES = 'PHILIPPINES'

Side-note: Interesting how Amazon Q autocompleted PHILIPPINES for me, I’m assuming it picked it up from the context of the linked URLs - I did a reboot of my laptop this morning and started a new chat, so don’t think it would be previous chat context, but could be mistaken:

Image not found

Amazon Q Developer auto-completing my enum contextually with data from the page I'm scraping.

Running the app now doesn’t return any data, so I suspect that I need to reference the enums differently, and yup, I do. Just append .value.

Adding Parameters

Now we need to change the code so we can pass these values in. Currently, I’m running the code locally via python3 local_test.py which calls the handler.py code via:

1
2
3
4
5
6
7
8
9
mock_context = MockContext()

mock_event = {
    "key1": "value1",
    "key2": "value2",
    # Add any other relevant data for your event
}

result = lambda_handler(mock_event, mock_context)

Hrm, this could be interesting, how would I pass the enum as a request? I could use integers, and do some kind of mapping between the number and the enum value, but honestly, I’m not sure how this works in Python. Time to find out though! The first part of how I add them to the event seems reasonable:

1
2
3
4
5
6
7
mock_event = {
    'queryStringParameters': {
        'filing_type': 'FINAL_DATE',
        'category': 'THIRD',
        'country': 'ALL_AREAS'
    }
}

I’m not quite sure that the way of extracting them looks right, adding in the 2nd parameter looks like hardcoding it to me:

1
2
3
4
    query_params = event.get('queryStringParameters', {})
    filing_type_str = query_params.get('filing_type', 'FINAL_DATE')
    country_str = query_params.get('country', 'ALL_AREAS')
    category_str = query_params.get('category', 'THIRD')

After clarifying how the get() method works, this does appear to be the correct way - that 2nd parameter is a default if it doesn’t find the value in the input. Having it as a string bugs me - if I ever change the name of the enum, this will not work. After a bit of back and forth (1, 2, 3) I had the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
    query_params = event.get('queryStringParameters', {})
    filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.name)
    country_str = query_params.get('country', Country.ALL_AREAS)
    category_str = query_params.get('category', Category.THIRD.name)

    # Convert string values to enum members
    filing_type = FilingType[filing_type_str.upper()]
    country = Country[country_str.upper()]
    category = Category[category_str.upper()]

    ...
    # Updated call to read_data() with the parameters

    eb3_data = read_data(filing_type, category, country)

Additionally, I removed the hardcoded defaults on read_data() and read_data_locally(). Running the code locally returns the data as expected, I also checked with 'category': 'SECOND', just to make sure.

Splitting the Scrape and Retrieve Methods

We are now at a point where the code does what we want, except that every time we retrieve data, it would also loop through all the bulletins and try to process them. Ideally this should be split into 2 separate functions, with a way to call the code to scrape any new bulletins one a schedule. Before I try to find the answer, I’m thinking I could create a 2nd functions similar to lambda_handler(event, context), and then set up a 2nd Lambda function using the same Lambda layer for the requirements. It does mean I will include some libraries not needed by each function as it will contain all the ones across both functions, but I’m fine with that approach. Splitting this into 2 different projects, or even just splitting the requirements.txt file feels like even more over the top than we are already are.

Image not found

Hard choices!

The suggested approach is to split the project into 2 files, scraper.py and retriever.py, and then to have a lambda_handler function in each. I’m tempted to just create a 2nd function inside the 1 file, but let’s go with the suggestion and split it into those 2 files. After I rename handler.py to scraper.py and create a copy named retriever.py, I realise we will need to split it further. At the top of the code, we set the tables via table = dynamodb.Table('VisaBulletinData') and processed_urls_table = dynamodb.Table('ProcessedURLs'), and we also have the enums defined as classes, and both of the functions need them. I look how I would do that, and merrily follow the suggestion.

After I remove the code not needed in each of the new functions, each in their own file, extract the enums into enums.py, and add the import statement, I stare at the import line in scraper.py for a few seconds:

Image not found

Why are they not being used?

Why are the 3 classes I import the darker colour of an unused import? Aaaaah! While I use the enums to retrieve the data, I never use them to store the data. I suddenly realise that I should probably change the code to use them as a mapping from the text in the bulletins. I noticed in some of the older ones that the 5th category was split less granularly, so we would need to address that as well.

Taking a Step Back

I’ve been building this app over the last 2 - 3 weeks spending an hour here and there between other work to build it out. Initially I intended to sit down and build it out in a day or 2, and expected it to not take any longer. I started out with the intention that I wanted to use a Lambda function, store the data in DynamoDB, and then be able to query it. Putting on my Captain Obvious cape, I realise that this is a lot of additional complexity for a problem I could have solved with a spreadsheet where I paste in a new row or two once a month. As the saying goes:

If all you have is a hammer, then all your problems look like a nail.

Ok, new plan. I’m going to continue down this path for only a short while longer till I have the Lambda functions deployed, and I can call them.

Updating the Infrastructure

Since we now have 2 separate functions, we also need to deploy them. This requires changing the existing Lambda since it referenced handler.py. Pasting all the Terraform resources again would take up quite a bit of space, but you can look at the current version at this point in time. We can keep the following resources as-is:

resource "null_resource" "pip_install" - we aren’t splitting requirements.txt per source file, so a single one stays the same.
data "archive_file" "layer" - used to trigger updating the Lambda layer we create for our dependencies.
resource "aws_lambda_layer_version" "layer" - creates the Lambda layer.
data "archive_file" "app" combined with the line source_code_hash = data.archive_file.app.output_base64sha256 to ensure we update the functions for any code changes. This will trigger for both functions even if we only update one of them, but I don’t feel it is worth the effort for this project.

I do want to ensure we provide the least amount of IAM permissions per function, so will duplicate the existing IAM role, policy, and attachments, and then reduce the scope of each IAM policy for each function with only the access needed. The last step will be to define a 2nd Lambda function after updating the existing one to use the renamed source file. Sorry, I lied, the last-last step will also be to update the DynamoDB table names to use the environment variables defined in the Lambda function, with a fallback if they are not set.

Adding the Environment Variables for DynamoDB Table Names

This change requires updating the Terraform resource for our Lambda function, below is the updated scraper.py function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Define the Lambda function
resource "aws_lambda_function" "visa_bulletin_scraper" {
  function_name    = "visa-bulletin-scraper"
  handler          = "scraper.lambda_handler" # <--- I've updated the handler's filename as well
  runtime          = "python3.12"
  filename         = data.archive_file.app.output_path
  source_code_hash = data.archive_file.app.output_base64sha256
  role             = aws_iam_role.lambda_role.arn
  layers           = [aws_lambda_layer_version.layer.arn]

  environment {
    variables = {
      BULLETIN_DATA = aws_dynamodb_table.visa_bulletin_data.name,
      PROCESSED_BULLETIN_URLS = aws_dynamodb_table.processed_urls.name
    }
  }
}

We also need to change the currently hard-coded table names at the top of scraper.py to the following to use the environment variables, with a fallback value:

1
2
3
4
5
6
7
8
9
10
11
12
import os # <--- new import added

...

# Initialize DynamoDB client
dynamodb = boto3.resource('dynamodb')

table_name = os.environ.get('BULLETIN_DATA', 'VisaBulletinData')
table = dynamodb.Table(table_name)

processed_urls_table_name = os.environ.get('PROCESSED_BULLETIN_URLS', 'ProcessedURLs')
processed_urls_table = dynamodb.Table(processed_urls_table_name)

For retriever.py, we only need to add the VisaBulletinData via BULLETIN_DATA.

Create a 2nd Lambda Function

As mentioned above, we will reuse the same Lambda layer, and only create a 2nd function for retriever.py, along with its own IAM policy and role. While doing this, I notice that past-Cobus was lazy with the resource names:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
resource "aws_iam_role" "lambda_role" {
  name = "visa-bulletin-scraper-role"

  ...

}

# Attach the necessary IAM policies to the role
resource "aws_iam_policy_attachment" "lambda_basic_execution" {
    name = "lambda_basic_execution"
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
  roles      = [aws_iam_role.lambda_role.name]
}

resource "aws_iam_policy_attachment" "dynamodb_access" {
    name = "dynamodb_access"
  policy_arn = aws_iam_policy.dynamodb_access_policy.arn
  roles      = [aws_iam_role.lambda_role.name]
}

# Define the IAM policy for DynamoDB access
resource "aws_iam_policy" "dynamodb_access_policy" {
...
}

We now need to duplicate the Lambda function resource, along with reduced permissions for the IAM role needed by retriever.py. It is also a good time to split out our Terraform resources into smaller files to make it easier to find each one of them. I decided to split them from the single app.tf into the following:

dynamodb.tf - defines the 2 tables we use.
lambda_layer.tf - builds and creates the Lambda layer used by both functions.
lambda_zip.tf - zips up all the application code into a single .zip used by both functions.
lambda_scraper.tf - defines the Lambda function, IAM policy, and IAM role for scraper.py.
lambda_retriever.tf - defines the Lambda function, IAM policy, and IAM role for retriever.py.

After making these changes, and running terraform apply, the following error is returned:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
│ Error: creating IAM Role (visa-bulletin-scraper-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: 94248aee-5615-4fbf-936e-81ceb9c24f0f, EntityAlreadyExists: Role with name visa-bulletin-scraper-role already exists.
│ 
│   with aws_iam_role.scraper_role,
│   on lambda_scraper.tf line 20, in resource "aws_iam_role" "scraper_role":
│   20: resource "aws_iam_role" "scraper_role" {
│ 
╵
╷
│ Error: creating IAM Policy (visa-bulletin-scraper-dynamodb-access): operation error IAM: CreatePolicy, https response error StatusCode: 409, RequestID: da27e143-5a7e-4008-81a8-49bf23fb99d8, EntityAlreadyExists: A policy called visa-bulletin-scraper-dynamodb-access already exists. Duplicate names are not allowed.
│ 
│   with aws_iam_policy.scraper_dynamodb_access_policy,
│   on lambda_scraper.tf line 53, in resource "aws_iam_policy" "scraper_dynamodb_access_policy":
│   53: resource "aws_iam_policy" "scraper_dynamodb_access_policy" {
│ 
╵

Since I renamed the aws_iam_role resource from lambda_role to scraper_role, we ran into a race condition where the delete for the old one didn’t complete before the new one’s create started. IAM role names need to be unique, and this is why we encountered this issue. Running terraform apply a second time will fix this - worth keeping in mind if you ever run into this after doing a cleanup.

1
Apply complete! Resources: 4 added, 1 changed, 0 destroyed.

Pro-tip: terraform fmt will format your .tf files in the current directory and fix the indentation, be kind and ~rewind~ clean up your source files before you commit them (unlike past-Cobus who also forgot that step).

But Does It Work

If you deploy a Lambda function, but you never call it, does it even exist?

In theory, we now have our 2 Lambda functions that would scrape or return the data. We haven’t called them yet though, so let’s see how we can call them from our terminal:

1
2
3
4
5
aws lambda invoke \
  --function-name visa-bulletin-retriever \
  --cli-binary-format raw-in-base64-out \
  --payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}' \
  response.json

Which returns the lovely error:

1
2
3
4
5
{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}

No It Does Not

Using the AWS Console to test the retriever function with the same payload as above, I see the issue:

Image not found

Looking at the stacktrace for the error.

When splitting out the enums into their own source file, I forgot to also add the import statement. After fixing that, I run terraform apply again, and I’m quite curious to see what the change looks like since we’re using Lambda layers. The following change to the IAM role makes me suspicious:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# aws_iam_policy_attachment.retriever_lambda_basic_execution will be updated in-place
  ~ resource "aws_iam_policy_attachment" "retriever_lambda_basic_execution" {
        id         = "retriever_lambda_basic_execution"
        name       = "retriever_lambda_basic_execution"
      ~ roles      = [
          - "visa-bulletin-scraper-role",
            # (1 unchanged element hidden)
        ]
        # (3 unchanged attributes hidden)
    }

  # aws_iam_policy_attachment.scraper_lambda_basic_execution will be updated in-place
  ~ resource "aws_iam_policy_attachment" "scraper_lambda_basic_execution" {
        id         = "scraper_lambda_basic_execution"
        name       = "scraper_lambda_basic_execution"
      ~ roles      = [
          - "visa-bulletin-retriever-role",
            # (1 unchanged element hidden)
        ]
        # (3 unchanged attributes hidden)
    }

This rings a bell from years ago, but I can’t quite remember, so off I go and do a search for aws_iam_policy_attachment so I can look at the documentation. Right at the top of the page, it has a big, red warning:

Image not found

Looks like the issue.

Aha! 🎵 It’s all coming back to me nooooow! 🎵 This was similar to setting the security group rules inside the aws_security_group - in the case of aws_iam_policy_attachment, it will play whack-a-mole since we need to attach that policy to 2 different IAM roles. On the first run, one of them will end up succeeding, but for future runs, it will try to attach it again since whichever one finished last would remove the one that finished first. To fix this, we need to use aws_iam_role_policy_attachment instead:

1
2
3
4
resource "aws_iam_role_policy_attachment" "retriever_lambda_basic_execution" {
  role       = aws_iam_role.retriever_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

Updates done, and ran terraform apply again, only for it to timeout after 2 minutes with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
╷
│ Error: reading IAM Role Policy Attachment (visa-bulletin-retriever-role:arn:aws:iam::631077426493:policy/visa-bulletin-retriever-dynamodb-access): empty result
│ 
│   with aws_iam_role_policy_attachment.retriever_dynamodb_access,
│   on lambda_retriever.tf line 45, in resource "aws_iam_role_policy_attachment" "retriever_dynamodb_access":
│   45: resource "aws_iam_role_policy_attachment" "retriever_dynamodb_access" {
│ 
╵
╷
│ Error: reading IAM Role Policy Attachment (visa-bulletin-scraper-role:arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole): empty result
│ 
│   with aws_iam_role_policy_attachment.scraper_lambda_basic_execution,
│   on lambda_scraper.tf line 40, in resource "aws_iam_role_policy_attachment" "scraper_lambda_basic_execution":
│   40: resource "aws_iam_role_policy_attachment" "scraper_lambda_basic_execution" {
│ 
╵
╷
│ Error: reading IAM Role Policy Attachment (visa-bulletin-scraper-role:arn:aws:iam::631077426493:policy/visa-bulletin-scraper-dynamodb-access): empty result
│ 
│   with aws_iam_role_policy_attachment.scraper_dynamodb_access,
│   on lambda_scraper.tf line 45, in resource "aws_iam_role_policy_attachment" "scraper_dynamodb_access":
│   45: resource "aws_iam_role_policy_attachment" "scraper_dynamodb_access" {
│ 

The hint here is the empty result - when you try to attach a policy the 2nd time, it will not return an error if it is already attached. So we have another race condition (I think): the request to remove the attachment (since we replaced the aws_iam_policy_attachment with aws_iam_role_policy_attachment) was was made in parallel to the new one in an order where it didn’t return a successful response for the new one. At least this is what I would speculate without digging into it too much. Regardless, you can get around this issue by just running terraform apply a 2nd time.

Let’s Try Again

Second time is the charm, right? Running our aws invoke-lambda command again returns:

1
2
3
4
5
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
(END)

And then it will wait for you to exit the output view - this is the standard behaviour, and you can exit it via ctrl + c or cmd + c. Alternatively, you can add the --no-cli-pager parameter:

1
2
3
4
5
6
aws lambda invoke \
  --function-name visa-bulletin-retriever \
  --cli-binary-format raw-in-base64-out \
  --payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}' \
  --no-cli-pager \
  response.json

Looking at the contents in response.json, we can see the data!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
    "statusCode": 200,
    "body": [
        {
            "date": "2021-12-01",
            "filing_type": "Final Date",
            "bulletin_date": "2024-07-01",
            "category": "3rd",
            "sk": "BULLETIN_DATE#2024-07-01",
            "pk": "FILING_TYPE#Final Date#CATEGORY#3rd#COUNTRY#All Chargeability Areas Except Those Listed",
            "country": "All Chargeability Areas Except Those Listed"
        },
        
        ...

While the data is all there, it isn’t in a very readable format - we are only really interested in the bulletin_date along with the date as a sorted list since all the rest of the data is what we sent in to filter on.

Making the Response Better

A better response would be one where it shows the filing_type, category, and country once, and then have a list of key-value pairs for bulletin_date and date. It looks like we can do this by changing the return statement to the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
...

    response = {
        'filing_type': filing_type,
        'category': category,
        'country': country,
        'data': data
    }

    return {
        'statusCode': 200,
        'body': response
    }

This does assume that the response from read_data is in the format that we need, which is that key-value pair list. We need to also update the return of read_data to accomplish this to the following:

1
2
3
4
5
6
7
8
9
10
...

    date_list = []    

    for item in sorted_items:
        date = item['date']
        bulletin_date = item['bulletin_date']
        date_list.append({"bulletin_date":bulletin_date, "date": date})

    return date_list

And another terraform apply later, it is deployed. You can also see from the Terraform output below that it only replaced the Lambda function code, not the Lambda layer as we didn’t add any additional dependencies - json is built into Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Terraform will perform the following actions:

# aws_lambda_function.visa_bulletin_retriever will be updated in-place
  ~ resource "aws_lambda_function" "visa_bulletin_retriever" {
        id                             = "visa-bulletin-retriever"
      ~ last_modified                  = "2024-07-03T22:28:49.000+0000" -> (known after apply)
      ~ source_code_hash               = "BmwRVIejGX4B1ual5JdhfulogEDuyTxyW9/3/g95WBE=" -> "eTHf0nHRxTy/Pi4u+tOvpKATsPWqagYcD3rEl0wwwcQ="
        tags                           = {}
        # (27 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # aws_lambda_function.visa_bulletin_scraper will be updated in-place
  ~ resource "aws_lambda_function" "visa_bulletin_scraper" {
        id                             = "visa-bulletin-scraper"
      ~ last_modified                  = "2024-07-03T22:29:02.000+0000" -> (known after apply)
      ~ source_code_hash               = "BmwRVIejGX4B1ual5JdhfulogEDuyTxyW9/3/g95WBE=" -> "eTHf0nHRxTy/Pi4u+tOvpKATsPWqagYcD3rEl0wwwcQ="
        tags                           = {}
        # (27 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

The body of the response will be JSON-encoded, so if we used this function as an API, we would need to deserialize the body into an object to access the data. The contents of response.json looks as follows:

Image not found

Reduced response.

Wrapping Up

As with most things, this took quite a bit longer than expected. The intention was just to add some parameters to the Lambda function for us to call, but we had to take a few detours along the way. As mentioned further up, I think I’m over-complicating this just a little bit, but I’m still enjoying myself, so I’ll do one more article tomorrow to set up a simple CI/CD pipeline that will deploy any future changes to this little app. Oh, and I need to set up a daily schedule to call the scraper function to auto-update if a new file is found. Actually, what would be really nice to also include a notification if it finds a new bulletin - today is 3 July 2024, and I’m already refreshing the page to see when the draft of the August bulletin will be published…

(PS: By “tomorrow”, I mean Friday 5 July as tomorrow is a public holiday here in the US.)

Final Code

As for the previous piece, I’ve tagged this with article-3, and pushed the tag.

Source code for this article.

Prompts used in this article

Prompt 1

Can you please show me how to update the highlighted code so that it will store the data after processing the URL, but before it saves the URL via processed_urls_table.put_item?

Response:

Sure, here’s how you can modify the highlighted code to store the data after processing the URL, but before saving the URL via processed_urls_table.put_item:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
for link in visa_bulletin_links:
    if '2022' in link or '2023' in link or '2024' in link:
        # Check if the URL has been processed
        response = processed_urls_table.get_item(Key={'url': link})
        if 'Item' in response:
            print(f"Skipping URL: {link} (already processed)")
            continue

        # Process the URL
        print(f"Processing URL: {link}")
        url = f"https://travel.state.gov{link}"
        url_data = scrape_visa_bulletin(url)
        data.extend(url_data)

        # Store the data
        store_data(url_data)

        # Store the processed URL in DynamoDB
        processed_urls_table.put_item(Item={'url': link})

store_data(data)

Here’s what’s changed:

After scraping the visa bulletin data from the URL using scrape_visa_bulletin(url), the scraped data is stored in a new variable url_data.
The url_data is then appended to the data list using data.extend(url_data). 3.A new line store_data(url_data) is added to store the data scraped from the current URL.

After storing the data, the URL is marked as processed in DynamoDB using processed_urls_table.put_item(Item={'url': link}).

This way, the data is stored immediately after being scraped from the URL, and before the URL is marked as processed in DynamoDB.

Note: This assumes that the store_data function is defined elsewhere in your code and correctly handles storing the data in the desired location (e.g., a database, a file, etc.).

Prompt 2

Looking at the highlighted line, does Python have an equivalent of C#’s enum? What would the approach be in Python to use something similar? I would like to define an easier way to provide the filing_type, category, and country?

Response:

Yes, Python has an equivalent to C#’s enum called the enum module, which was introduced in Python 3.4. It allows you to define a set of symbolic names (members) bound to unique, constant values. This can make your code more readable and less prone to errors caused by typos or invalid values. [1]

Here’s an example of how you could use the enum module to define the possible values for filing_type, category, and country in your read_data_locally function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from enum import Enum

class FilingType(Enum):
    FINAL_DATE = 'Final Date'
    DATES_FOR_FILING = 'Dates for Filing'

class Category(Enum):
    FIRST = '1st'
    SECOND = '2nd' 
    THIRD = '3rd'
    OTHER_WORKERS = 'Other Workers'

class Country(Enum):
    ALL_AREAS = 'All Chargeability Areas Except Those Listed'
    CHINA_MAINLAND = 'China-mainland born'
    EL_SALVADOR = 'El Salvador, Guatemala, Honduras'
    # Add more countries as needed

def read_data_locally(data, filing_type=FilingType.FINAL_DATE, category=Category.THIRD, country=Country.ALL_AREAS):
    # Your function logic here
    pass

In this example, we define three enum classes: FilingType, Category, and Country. Each class has members representing the valid values for that category.

When calling read_data_locally, you can now use the enum members instead of strings, like filing_type=FilingType.FINAL_DATE. This has several advantages:

Type safety : The function will only accept values from the defined enums, preventing typos or invalid values.
Readability : The code becomes more self-documenting and easier to understand.
Consistency : You ensure that the same values are used throughout your codebase.

Note that you can also assign custom values to enum members if needed, and you can add methods and properties to the enum classes.

Sources:

Is there any way to define the enum in python?

Prompt 3

Using the enum values in this way doesn’t return any data, could you please explain how I would access the string value for each enum in filtered_data code?

Response:

To access the string value of an enum member in the filtered_data code, you can use the value attribute of the enum member or call the str() function on the member.

Here’s an example of how you can modify the read_data_locally function to filter the data based on the string values of the enum members:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def read_data_locally(data, filing_type=FilingType.FINAL_DATE, category=Category.THIRD, country=Country.ALL_AREAS):
    # Filter the data based on filing_type, category, and country
    filtered_data = [entry for entry in data
                     if entry['filing_type'] == filing_type.value
                     and entry['category'] == str(category)
                     and entry['country'] == country.value]

    # Sort the filtered data in descending order by bulletin_date
    sorted_data = sorted(filtered_data, key=itemgetter('bulletin_date'), reverse=True)

    # Print the sorted data
    for entry in sorted_data:
        print(f"Bulletin Date: {entry['bulletin_date']}, Date: {entry['date']}")

    return sorted_data

In this modified version:

For filing_type, we use filing_type.value to get the string value of the enum member.
For category, we use str(category) to get the string representation of the enum member.
For country, we use country.value to get the string value of the enum member.

This way, the filtered_data list will contain only the entries where the filing_type, category, and country values match the string values of the corresponding enum members.

Note that when you call the read_data_locally function, you can still use the enum members directly, like read_data_locally(data, filing_type=FilingType.FINAL_DATE, category=Category.THIRD, country=Country.ALL_AREAS). By using the value attribute or str() function, you can access the string representation of the enum members, which is necessary for filtering the data based on the string values stored in the dictionary entries.

Prompt 4

Thanks, how would I add the names of the enums to the event when calling lambda_handler - I want to be able to inject it into the the mock_event object for local testing, but also be able to call the Lambda using query string parameters? It should then insert the values from the event when calling read_data().

Response:

To add the names of the enums to the event when calling lambda_handler, you can modify the function to accept the enum values as query string parameters or as part of the event payload. Then, you can use these values when calling read_data().

Here’s an example of how you can modify the lambda_handler function to accept the enum values as query string parameters:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import urllib.parse

def lambda_handler(event, context):
    # ... (existing code)

    # Parse query string parameters
    query_params = event.get('queryStringParameters', {})
    filing_type_str = query_params.get('filing_type', 'FINAL_DATE')
    country_str = query_params.get('country', 'ALL_AREAS')

    # Convert string values to enum members
    filing_type = FilingType[filing_type_str.upper()]
    category = Category[category_str.upper()]
    country = Country[country_str.upper().replace(' ', '_')]

    # Call read_data with the enum values
    eb3_data = read_data(filing_type=filing_type, category=category, country=country)

    return {
        'statusCode': 200,
        'body': eb3_data
    }

In this modified version:

The query_params dictionary is created from the queryStringParameters in the event object.

The filing_type_str, category_str, and country_str values are retrieved from the query_params dictionary, with default values provided if the parameters are not present.

The string values are converted to their corresponding enum members using the FilingType, Category, and Country enums. The upper() method is used to handle case-insensitive values, and the replace(' ', '_') is used to handle spaces in the country string.

The read_data function is called with the enum values.

To test this locally, you can create a mock_event object with the desired query string parameters:

1
2
3
4
5
6
7
8
9
10
mock_event = {
    'queryStringParameters': {
        'filing_type': 'FINAL_DATE',
        'category': 'THIRD',
        'country': 'ALL_AREAS'
    }
}

result = lambda_handler(mock_event, None)
print(result)

When invoking the Lambda function, you can pass the query string parameters in the URL, like https://your-lambda-function.aws/path?filing_type=FINAL_DATE&category=THIRD&country=ALL_AREAS.

If you prefer to pass the enum values as part of the event payload instead of query string parameters, you can modify the lambda_handler function accordingly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def lambda_handler(event, context):
    # ... (existing code)

    # Parse query string parameters
    query_params = event.get('queryStringParameters', {})
    filing_type_str = query_params.get('filing_type', 'FINAL_DATE')
    country_str = query_params.get('country', 'ALL_AREAS')

    # Convert string values to enum members
    filing_type = FilingType[filing_type_str.upper()]
    category = Category[category_str.upper()]
    country = Country[country_str.upper().replace(' ', '_')]

    # Call read_data with the enum values
    eb3_data = read_data(filing_type=filing_type, category=category, country=country)

    return {
        'statusCode': 200,
        'body': eb3_data
    }

In this case, you would pass the enum values as part of the event payload when invoking the Lambda function.

Prompt 5

Can you please explain how the line category_str = query_params.get('category', 'THIRD') works? Is the 2nd parameter “THIRD” a default value if it can’t extract it?

Response:

Yes, you’re correct. The line category_str = query_params.get('category', 'THIRD') is using the get() method of a dictionary (or dictionary-like object) to retrieve the value associated with the key 'category'.

The get() method takes two arguments:

The key to look up in the dictionary
An optional default value to return if the key is not found

In this case, query_params is likely a dictionary containing query parameters extracted from a URL or form data. The get('category', 'THIRD') part is attempting to retrieve the value associated with the 'category' key from query_params.

If the 'category' key exists in query_params, its associated value will be returned and assigned to category_str. However, if the 'category' key is not present in query_params, the default value 'THIRD' will be used instead and assigned to category_str.

So in summary, yes - the second argument 'THIRD' is acting as a default value that will be used for category_str if the 'category' key cannot be found in query_params.

Prompt 6

If I wanted to replace the string FINAL_DATE with the name of the enum, how would I do that? Would something like FilingType.FINAL_DATE.str() work?

Response:

In Python, you can access the value of an enum member directly using its name. There is no need for a separate method like str(). You can simply use FilingType.FINAL_DATE.value to get the string value associated with the FINAL_DATE member of the FilingType enum.

Here’s an example of how you can replace the string 'FINAL_DATE' with the value of the FilingType.FINAL_DATE enum member:

1
filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.value)

In this line, if the 'filing_type' key is not present in query_params, the value of FilingType.FINAL_DATE.value (which is 'Final Date') will be used as the default value for filing_type_str.

Similarly, you can use the same approach for the country_str and category_str assignments:

1
2
3
filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.value)
country_str = query_params.get('country', Country.ALL_AREAS.value)
category_str = query_params.get('category', Category.THIRD.value)

By using the value attribute of the enum members, you can directly access the string values associated with each enum member, without needing any additional methods like str().

Prompt 7

Thanks, but wouldn’t that set the value of the enum instead of the name since further down we look up the value of the enum via filing_type = FilingType[filing_type_str.upper()]?

Response:

You’re absolutely right, my previous suggestion would not work correctly in this case. Since you are using the string value to look up the corresponding enum member later on with FilingType[filing_type_str.upper()], you need to use the string name of the enum member as the default value, not the value attribute. [1]

To replace the string 'FINAL_DATE' with the name of the FilingType.FINAL_DATE enum member, you can use the following:

1
filing_type_str = query_params.get('filing_type', 'FINAL_DATE')

This way, if the 'filing_type' key is not present in query_params, the string 'FINAL_DATE' will be assigned to filing_type_str. Then, when you do filing_type = FilingType[filing_type_str.upper()], it will correctly look up the FilingType.FINAL_DATE member.

Similarly, for the other enum types, you can use the string names of the desired default enum members:

1
2
filing_type_str = query_params.get('filing_type', 'FINAL_DATE')
country_str = query_params.get('country', 'ALL_AREAS')

This way, the default values will be the string names of the enum members, which can then be used to look up the corresponding enum members correctly.

Sources:

Entity Framework(EF) Enum issue/bug/feature - DEV Community

Prompt 8

This isn’t quite what I’m looking for. I want a way to avoid using a hard-coded string for the enum key FINAL_DATE, and instead somehow reference FilingType.FINAL_DATE so that if I decide to rename that specific enum value to e.g. FINAL_DATE_NEW, I would get an error when trying to the code as FilingDate.FINAL_DATE would no longer exist.

Response:

I understand now. You want to avoid hard-coding the string name of the enum member and instead reference the enum member itself, so that if the enum member name changes, you’ll get an error at compile/run time.

To achieve this, you can use the name attribute of the enum member. The name attribute returns the string name of the enum member.

Here’s how you can modify the code:

1
2
3
4
5
6
7
8
9
10
from enum import Enum

class FilingType(Enum):
    FINAL_DATE = 'Final Date'
    # other enum members

default_filing_type = FilingType.FINAL_DATE

filing_type_str = query_params.get('filing_type', default_filing_type.name)
filing_type = FilingType[filing_type_str.upper()]

In this example:

We define a default_filing_type variable and assign it the FilingType.FINAL_DATE enum member.
When getting the filing_type_str from query_params, we use default_filing_type.name as the default value.

This way, if you rename the FilingType.FINAL_DATE enum member to something else (e.g., FilingType.FINAL_DATE_NEW), you’ll get an error when trying to access FilingType.FINAL_DATE, because it no longer exists.

The name attribute of the enum member is a string, so it can be used as the default value for filing_type_str. Later, when you look up the enum member using FilingType[filing_type_str.upper()], it will work as expected, because you’re using the string name of the enum member.

This approach ensures that if you rename or remove an enum member, you’ll get an error at compile/run time, rather than silently using an incorrect hard-coded string value.

Prompt 9

The current lambda_handler function has 2 purposes, the first is to scrape the dates from the USCIS website and store them in the database, and the 2nd is to return the data based on the parameters specified. What is the best approach given the current structure (Python for the code, and Terraform for the infrastructure) to split this into 2 different functions?

Response:

To split the current lambda_handler function into two separate functions, you can follow these steps: [1]

1. Create a new Lambda function for scraping and storing dataIn your Terraform configuration, create a new AWS Lambda Function resource. This function will be responsible for scraping the USCIS website and storing the data in the database. [2]

1
2
3
4
5
6
7
resource "aws_lambda_function" "scraper_function" {
  filename         = data.archive_file.scraper_lambda_zip.output_path
  function_name    = "uscis-scraper"
  role             = aws_iam_role.lambda_exec_role.arn
  handler          = "scraper.lambda_handler"
  runtime          = "python3.9"
}

2. Create a new Lambda function for retrieving dataCreate another AWS Lambda Function resource in your Terraform configuration. This function will be responsible for retrieving data from the database based on the provided parameters.

1
2
3
4
5
6
7
resource "aws_lambda_function" "retriever_function" {
  filename         = data.archive_file.retriever_lambda_zip.output_path
  function_name    = "uscis-retriever"
  role             = aws_iam_role.lambda_exec_role.arn
  handler          = "retriever.lambda_handler"
  runtime          = "python3.9"
}

3. Separate the Python codeIn your Python code, create two separate files: scraper.py and retriever.py. scraper.py will contain the code for scraping the USCIS website and storing the data in the database. The lambda_handler function in this file will be responsible for this task. retriever.py will contain the code for retrieving data from the database based on the provided parameters. The lambda_handler function in this file will be responsible for this task.

4. Update the Lambda function handlersIn your Terraform configuration, update the handler argument for each Lambda function to point to the correct file and function name. [3]

1
2
3
4
5
6
7
8
9
resource "aws_lambda_function" "scraper_function" {
# ...
handler          = "scraper.lambda_handler"
}

resource "aws_lambda_function" "retriever_function" {
# ...
handler          = "retriever.lambda_handler"
}

5. Configure triggers and permissions

Depending on your use case, you may want to configure triggers or permissions for each Lambda function. For example, you could set up a CloudWatch Event Rule to trigger the scraper function periodically, or configure API Gateway to invoke the retriever function based on HTTP requests.

Sources:

Prompt 10

Can you show me how I would extract the enum classes to their own file so I can reference them from both the scraper.py and retriever.py files?

Response:

Sure, here’s how you can extract the enum classes to their own file and reference them from both scraper.py and retriever.py:

Create a new file called enums.py in the same directory as scraper.py and retriever.py.

1. In enums.py, define your enum classes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# enums.py

from enum import Enum

class FilingType(Enum):
    FINAL_DATE = 'Final Date'
    INITIAL_DATE = 'Initial Date'
    # Add other enum members as needed

class Country(Enum):
    ALL_AREAS = 'All Areas'
    CHINA_MAINLAND_BORN = 'China Mainland-born'
    # Add other enum members as needed

class Category(Enum):
    FIRST = 'First'
    SECOND = 'Second'
    THIRD = 'Third'
    # Add other enum members as needed

2. In scraper.py and retriever.py, import the enum classes from enums.py:

1
2
3
4
5
6
7
8
9
# scraper.py
from enums import FilingType, Country, Category

# Your scraper code here

# retriever.py
from enums import FilingType, Country, Category

# Your retriever code here

Now, you can use the enum classes FilingType, Country, and Category in both scraper.py and retriever.py.

For example, in retriever.py, you can use the enum classes like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# retriever.py
from enums import FilingType, Country, Category

def lambda_handler(event, context):
    query_params = event.get('queryStringParameters', {})

    filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.name)
    filing_type = FilingType[filing_type_str.upper()]

    country_str = query_params.get('country', Country.ALL_AREAS.name)
    country = Country[country_str.upper()]

    category_str = query_params.get('category', Category.THIRD.name)
    category = Category[category_str.upper()]

    # Your retrieval logic here

By extracting the enum classes to a separate file (enums.py), you can easily import and use them in both scraper.py and retriever.py. This approach promotes code reusability and maintainability, as you only need to define the enum classes in one place, and any changes to the enum classes will be reflected in both files.

Prompt 11

Can you please show me how I would update the line table = dynamodb.Table('VisaBulletinData') to use an environment variable called “BULLETIN_DATA” instead of the hard-coded string, and only use that hard-coded string as a fallback if the environment variable is not set?

Response:

Sure, here’s how you can update the line table = dynamodb.Table('VisaBulletinData') to use an environment variable called BULLETIN_DATA and fallback to the hard-coded string if the environment variable is not set:

1
2
3
4
import os

table_name = os.environ.get('BULLETIN_DATA', 'VisaBulletinData')
table = dynamodb.Table(table_name)

Here’s what’s happening:

import os imports the os module, which provides a way to interact with the operating system, including accessing environment variables.
os.environ.get('BULLETIN_DATA', 'VisaBulletinData') retrieves the value of the environment variable BULLETIN_DATA. If the environment variable is not set, it will use the fallback value 'VisaBulletinData'.
The retrieved value (either from the environment variable or the fallback value) is assigned to the table_name variable.
dynamodb.Table(table_name) creates a DynamoDB Table object using the value stored in table_name. [1]

By using os.environ.get(), you can easily switch between using the environment variable or the hard-coded string, depending on whether the environment variable is set or not.

To set the BULLETIN_DATA environment variable when running your Lambda function, you can configure it in the AWS Lambda console or through your Terraform configuration. Here’s an example of how you can set the environment variable using Terraform:

1
2
3
4
5
6
7
8
9
resource "aws_lambda_function" "your_lambda_function" {
  # ... other configuration ...

  environment {
    variables = {
      BULLETIN_DATA = "YourTableName"
    }
  }
}

Replace "YourTableName" with the desired name of your DynamoDB table.

With this configuration, your Lambda function will use the value of the BULLETIN_DATA environment variable as the DynamoDB table name. If the environment variable is not set (or set to an empty string), it will fallback to using the hard-coded string 'VisaBulletinData'.

Sources:

aws-lambda-dynamodb - AWS Solutions Constructs

Prompt 12

Can you please show me how I would call the visa_bulletin_retriever Lambda function using the AWS CLI on my local terminal with an event payload of:

1
2
3
4
5
6
7
{
    'queryStringParameters': {
        'filing_type': 'FINAL_DATE',
        'category': 'THIRD',
        'country': 'ALL_AREAS'
    }
}

Response:

Sure, here’s how you can invoke the visa_bulletin_retriever Lambda function using the AWS CLI on your local terminal with the provided event payload:

1
2
3
4
5
aws lambda invoke \
  --function-name visa_bulletin_retriever \
  --cli-binary-format raw-in-base64-out \
  --payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}' \
  response.json

Let’s break down this command:

aws lambda invoke: This is the AWS CLI command to invoke a Lambda function.
--function-name visa_bulletin_retriever: Specifies the name of the Lambda function you want to invoke. Replace visa_bulletin_retriever with the actual name of your Lambda function.
--cli-binary-format raw-in-base64-out: This option tells the AWS CLI to send the payload as raw binary data and to decode the response from Base64.
--payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}': This is the JSON payload that you want to pass to your Lambda function. It matches the structure you provided in your example.
response.json: This is the name of the file where the Lambda function’s response will be saved. You can choose any file name you prefer.

After running this command, the Lambda function will be invoked with the provided payload, and the response will be saved in the response.json file.

Make sure you have the AWS CLI configured with the appropriate credentials and permissions to invoke the Lambda function.

If you want to see the response in the terminal instead of saving it to a file, you can remove the response.json part from the command:

1
2
3
4
aws lambda invoke \
  --function-name visa_bulletin_retriever \
  --cli-binary-format raw-in-base64-out \
  --payload '{"queryStringParameters": {"filing_type": "FINAL_DATE", "category": "THIRD", "country": "ALL_AREAS"}}'

This will print the Lambda function’s response directly in the terminal.

Note: If you need to pass more complex payloads or payloads from a file, you can use the --payload-json file://path/to/payload.json option instead of embedding the JSON payload directly in the command.

Prompt 13

Can you please show me how to update the return method for lambda_handler in retriever.py so that the data is in the following shape please?

1
2
3
4
5
6
7
8
9
{
    'filing_type': 'Final Date',
    'category': '3rd',
    'country': 'All Chargeability Areas Except Those Listed',
    'data': [
        {'bulletin_date': '2024-07-01', 'date': '2021-12-01' },
        {'bulletin_date': '2024-08-01', 'date': '2021-12-05' }
    ]
}

Response:

To update the return method for lambda_handler in retriever.py to return the data in the desired shape, you can modify the code as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import json

def lambda_handler(event, context):
    query_params = event.get('queryStringParameters', {})

    filing_type_str = query_params.get('filing_type', FilingType.FINAL_DATE.name)
    filing_type = FilingType[filing_type_str.upper()].value

    country_str = query_params.get('country', Country.ALL_AREAS.name)
    country = Country[country_str.upper()].value

    category_str = query_params.get('category', Category.THIRD.name)
    category = Category[category_str.upper()].value

    data = read_data(filing_type, category, country)

    response = {
        'filing_type': filing_type,
        'category': category,
        'country': country,
        'data': data
    }

    return {
        'statusCode': 200,
        'body': json.dumps(response)
    }

Here’s what’s happening:

We import the json module to convert the Python dictionary to a JSON string.
Inside the lambda_handler function, we retrieve the filing_type, country, and category values from the query_params dictionary, using the corresponding enum values as defaults.
Instead of using the string names of the enum members, we use the value attribute to get the actual string values associated with each enum member (e.g., FilingType.FINAL_DATE.value returns 'Final Date').
We call the read_data function to get the data based on the filing_type, category, and country values.
We create a new dictionary response with the desired shape, including the filing_type, category, country, and data keys.
We return a dictionary with a statusCode of 200 and a body containing the JSON-encoded response dictionary.

With this modification, the Lambda function will return a response in the desired shape:

1
2
3
4
{
    "statusCode": 200,
    "body": "{\"filing_type\": \"Final Date\", \"category\": \"3rd\", \"country\": \"All Chargeability Areas Except Those Listed\", \"data\": [{\"bulletin_date\": \"2024-07-01\", \"date\": \"2021-12-01\"}, {\"bulletin_date\": \"2024-08-01\", \"date\": \"2021-12-05\"}]}"
}

Note that the body is a JSON-encoded string, so you may need to parse it on the client-side (e.g., using JSON.parse() in JavaScript) to work with the data as a JavaScript object.

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Select your cookie preferences

Site Terms, Privacy, and more.

Adding parameters to my Serverless Web Scraper API [part 3]

Third installment of the series where I build a serverless API to scrape the historic dates from the USCIS monthly visa bulletin pages.

Updating the Scrape and Storage Logic

Fixing Up Processing and Storing Our Data

Defining Some Enums

Adding Parameters

Splitting the Scrape and Retrieve Methods

Taking a Step Back

Updating the Infrastructure

Adding the Environment Variables for DynamoDB Table Names

Create a 2nd Lambda Function

But Does It Work

No It Does Not

Let’s Try Again

Making the Response Better

Wrapping Up

Final Code

Prompts used in this article

Prompt 1

Prompt 2

Prompt 3

Prompt 4

Prompt 5

Prompt 6

Prompt 7

Prompt 8

Prompt 9

Prompt 10

Prompt 11

Prompt 12

Prompt 13

Comments