Automating Remediation of Containers with Vulnerabilities in AWS

Modern containerized environments are fast-paced, and new vulnerabilities can arise even after images are deployed in production. When new Common Vulnerabilities and Exposures (CVEs) are added to vulnerability databases, they may expose previously clean images to risk. The pattern discussed in this post outlines how AWS tooling can automatically rebuild and redeploy containers whenever new CVEs impact containers that are already running, without the need for immediate human action.

Using Amazon Elastic Container Registry (ECR), Amazon EventBridge, AWS CodeBuild, and AWS CodeDeploy, we’ll create a seamless pipeline that scans, detects, patches, and redeploys container images with no or minimal manual intervention. This solution is designed to secure containerized workloads continuously, making it ideal for security-conscious teams managing dynamic, production-grade applications at scale.

Prerequisites

Amazon ECR Repository: Ensure that your container images are stored in Amazon ECR, with enhanced image scanning enabled to allow Amazon Inspector to scan images for vulnerabilities.
To maximise on automation in this pattern, a pre-existing AWS CodePipeline to orchestrate the triggering of a deployment to, say, Amazon Elastic Container Service (ECS) once a successful image build job has completed.
IAM Permissions: Configure appropriate IAM roles and permissions for Amazon Inspector, EventBridge, CodeBuild, and ECS to enable seamless access and interaction between these services in your account.

Architecture

Developers commit code changes to their source control repository, which is integrated into the ‘Source‘ stage of an AWS CodePipeline pipeline.
AWS CodeBuild builds and pushes the container image to Amazon ECR.
Enhanced Scanning in Amazon ECR, which is powered by Amazon Inspector, allows for automatic vulnerability scanning on images pushed into ECR
The CodePipeline would include a custom action configured that allows CodeDeploy to kick of the deployment in the absence of concerning vulnerabilities, e.g. findings of high and critical severity.
However, if a new CVE gets published and is detected as a vulnerability in the pre-built image, Amazon Inspector generates a finding for the new CVE.
Amazon EventBridge captures the generated CVE findings and based on a defined rule and event target configuration, triggers a CodeBuild job.
CodeBuild rebuilds the container, with specific commands to pull the latest patches included, and pushes the updated image to ECR.
AWS CodeDeploy deploys the patched image to Amazon ECS using a blue/green deployment, ensuring minimal downtime.

Implementation

Enable Amazon ECR for Enhanced Continuous Scanning

Amazon ECR enhanced scanning is essential for automatically detecting new CVEs affecting stored images. This setup ensures that Amazon Inspector will scan each image pushed to ECR and will re-scan images when new CVEs are added to its database.

Create an Amazon EventBridge Rule to Trigger Rebuilds

When Amazon Inspector detects a new vulnerability, EventBridge captures this event, which then allows you to configure a remediation process. Set up an EventBridge rule to listen for Amazon Inspector image scan findings.

Example rule configuration for capturing findings of high and critical severity:

The below AWS CLI command ensures that when a vulnerability that matches the above rule is picked up in a container image, a CodeBuild job gets triggered to build a patched image.

You'll also have to create an IAM role with the necessary permissions for EventBridge to start your CodeBuild job.

Automate the Image Rebuild with CodeBuild

Using CodeBuild to rebuild and patch images allows for automated remediation. A CodeBuild project includes a buildspec.yml file which specifies the build commands that get executed when a build job runs. This file will handle package updates that will typically be required to address the new CVE.

Here’s an example buildspec.yml for this purpose:

Note on Using apt-get Commands

Ensure the following commands are always part of your Dockerfile or buildspec.yml to keep packages up-to-date:

apt-get update -y – Fetches updated lists of packages.
apt-get upgrade -y – Installs the newest versions of packages.

These commands allow for the clean image to be automatically rebuilt with the latest patches when a vulnerability is found in the previous image version.

Update ECS Service for Zero-Downtime Deployment

Finally, configure the automated deployment of the patched image on Amazon ECS using a CodeDeploy blue/green deployment strategy to ensure high availability. ECS seamlessly rolls out the updated container while maintaining availability of the previous version.

Benefits of This Pattern

Automated Vulnerability Detection: Amazon Inspector continuously scans images and alerts on any new CVEs.
Event-Driven Remediation: EventBridge enables the build and deployment pipeline to react to vulnerabilities as soon as they are detected, reducing manual intervention.
High Availability: Blue/green deployment ensures zero-downtime rollouts, preventing service interruptions while vulnerability remediation occurs.
Scalability: This architecture is scalable across multiple repositories and clusters, enabling consistent security across all container workloads.

Conclusion

By combining Amazon ECR, EventBridge, CodeBuild, and Amazon CodeDeploy, this pattern automates vulnerability detection and remediation for containerized workloads in AWS. This solution streamlines patching, minimizes manual intervention, and ensures your environment remains secure without impacting availability.

This solution is highly recommended for DevOps and security teams aiming for proactive, continuous security in their AWS container environments.

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.