Resource Lister: Expanding Resource Lister Capabilities: to Support for Additional AWS Services and Functions
This article will guide you through a process to extend Resource Lister's capabilities to support any of the 350 services and 3,500+ list and describe functions offered by boto3. By creating a one-time configuration file and uploading it to Resource Lister, you can effortlessly enable access to a new service and its associated functions. Resource Lister will automatically detect the newly added configuration file and seamlessly integrate the supported service and functions.
The open-source Resource(https://github.com/awslabs/resource-lister) Lister utility enables near real-time generation of AWS resource lists across multiple accounts and regions without writing a code. It outputs .csv, JSON, or directly sends data to an S3 bucket. For example, after a one-time configuration of appropriate permissions, Resource Lister can quickly produce lists of resources like:
- All Lambda functions
- EC2 instances with operating systems
- Redshift clusters
It abstracts away the underlying boto3 functions complexity. Resource Lister recently expanded support to 38 common AWS services, with approximately 100 related listing and description functions.
Boto3 provides access to over 350 AWS services with around 3,500 functions just for listing and describing AWS resources. Resource Lister currently supports 38 of those services, which may be enough for majority of the use cases. There may arise situations where we require a near real time listing of AWS resources that are either not currently supported or have been recently introduced as new services.
I have recently updated the Open-source code base that provides Users the ability to extend the functionality of the Resource Lister to any of the 3,500+ functions supported by boto3 by just creating and uploading a one-time configuration file to the Resource Lister.
In this article, I will demonstrate how to easily add support for any of the 350 services and 3,500 list and describe functions by simply creating a one-time configuration file and uploading it to Resource Lister. Resource Lister will automatically detect the new configuration file and enable access to the newly supported service and functions. By leveraging this configuration file approach, Resource Lister can be easily extended to work with all AWS services with minimal effort.
Let's consider a scenario where we want to expand the capabilities of the Resource Lister application to include support for two additional AWS services: Kinesis and Firehose. Specifically, we would like to add functionality to list Kinesis streams and describe stream details, as well as list Firehose delivery streams. This will allow Resource Lister to provide visibility into Kinesis Streams and Firehose resources in your AWS accounts
You can find the boto3 API documentation for these functions at the links below:
The configuration files contain lists of functions and their associated metadata that the functions require.
Let's briefly examine the metadata attributes and their corresponding values.
Configuration file Metadata
Note that for json_response we will use Resource Lister to flatten the raw function response . We will see that in following example. You can also examine reference functions
Step 2: Refer to the boto3 API documentation to create a configuration file for each boto3 function
Since we understand the configuration file metadata. Now we will create three configuration files for each of the boto3 function we want to add to resource lister.
We will create a top level folder with any name. Inside the top level folder, we will create a separate folder for each service. Within each service folder, we will create a .json file for each boto3 function that the service uses.As shown in the diagram below,
Step 3: Use the Resource Lister to Create Service Configuration files from function configuration files.
We have created configuration files organized by function. The next step is to use the Resource Lister tool to consolidate these functional files into service files and format the json_response to align with the tool's formatting requirements.
Go to Resource Lister --> Help-->8[Generate Config Files]-->1[Generate Service Configuration Files]
Enter Input path : Complete folder path (Top Level Directory ) where configuration files are created in Step 2. Enter Output path : Complete folder path where service config files will be generated (It could be any folder)
Note :
For Linux/Mac Path seperator is /
Example
/home/environment/repository
/home/environment/services
For Windows Pather seperate is \\
C:\\DEMO\\repository
C:\\DEMO\\service
Resource lister consolidate functional files into service files and format the json_response attribute according to tool's requirement and finally generate firehose.json and kinesis.json files in output directory
Step 4: Upload the Service Configuration files to Resource Lister
We will upload the service configuration files to the Resource Lister one at a time. First, we will upload the kinesis.json file. After that is complete, we will repeat the same upload process for the firehose.json file.
Go to Resource Lister --> Help-->8[Generate Config Files]-->2[Upload service configuration file.] Enter Input path : Complete service configuration.json file path
Note :
For Linux/Mac Path seperator is /
Example /home/environment/services/kinesis.json
For Windows Pather seperate is \\
C:\\DEMO\\service\\kinesis.json
Repeat the same Upload process for firehose.json
Step 5: Test the Utility for New capabilities
Lets quickly check if these services are already supported in Resource Lister. You can do this by typing ENTER in the MAIN Menu.You will Notice now that Kinesis stream and Kinesis firehose are listed.
To view the newly configured Kinesis functions, go to the Main Menu and enter "kinesis". The Resource Lister will then display the new functions.
describe_stream 2. list_streams
You can now generate a list of Kinesis streams across accounts and regions without writing any code by selecting the list_streams option (option 2).
Similary can now generate a list of Firehose Delivery streams across accounts and regions without writing any code by selecting the firehose service and list_delivery_streams option
Conclusion :
This article demonstrated how Resource Lister allows users to easily add or update support for any of the 350 services and 3,500 list and describe functions. This is accomplished by simply creating a one-time configuration file and uploading it to Resource Lister.
