Automating Cribl Security Agent Deployment Across Your AWS EC2 Fleet

Introduction

Security is paramount in cloud environments, and ensuring all your EC2 instances have the proper security agents installed is a critical but challenging task. This is especially true in dynamic environments where instances are constantly being created and terminated through Auto Scaling groups.

In this article, I'll share a battle-tested approach for deploying the Cribl security agent across your entire EC2 fleet - both existing instances and newly launched ones. This solution combines several AWS services to create a robust, automated deployment pipeline with compliance monitoring.

The Challenge

A common scenario many organizations face:

You need to install a security agent (in this case, Cribl) on all EC2 instances
Your environment includes hundreds or thousands of existing instances
New instances are constantly being launched through Auto Scaling groups
You need visibility into compliance and alerts when instances are running without the agent

Solution Overview

Our comprehensive solution addresses these challenges through a multi-layered approach:

Existing Instance Deployment: Using AWS Systems Manager (SSM) to deploy the agent to all running instances
New Instance Integration: Automatically installing the agent on newly launched instances via EC2 User Data
Golden AMI Strategy: Creating pre-baked AMIs with the agent already installed
Compliance Monitoring: Implementing checks and alerts for instances running without the agent

Let's dive into the implementation details.

Implementation

1. Preparation Steps

First, we need to set up the infrastructure to support our deployment:

Set Up S3 Infrastructure

Create a bucket to store your installation scripts:

Ensure SSM Agent Installation

For Systems Manager to work, all instances need the SSM Agent. Create an SSM document to install it on instances that don't have it:

2. Deployment Methods

Method A: Existing Instance Deployment via SSM

For your existing fleet, use Systems Manager to deploy the agent in batches:

1. Create an SSM Document:

2. Create a Maintenance Window:

3. Create a Python Script for Batched Deployments

For large fleets, you'll want to deploy in batches. Here's a Python script to help:

Run the script:

Method B: Bootstrap for New Instances

For new instances, we'll use EC2 User Data to install the agent during instance launch:
1. Configure User Data for Auto Scaling Groups:

2. Update Launch Templates/Configurations:

Method C: Golden AMI Creation

For a long-term solution, create Golden AMIs with the agent pre-installed:

3. Monitoring & Compliance

SSM State Manager Association for Compliance

Create a State Manager association to regularly check if the agent is running:

CloudWatch Alarms for Missing Agents

Create a Lambda function to set up per-instance alarms:

Lambda function content (`lambda_function.py`):

Schedule this Lambda to run daily:

4. Automating Launch Template Updates

To ensure all Auto Scaling groups launch instances with the Cribl agent, we need to update all launch templates. Here's a script that automates this process across all regions:

Required IAM Role Permissions

For this solution to work, you'll need to ensure your instances and services have the appropriate permissions:

Conclusion

This comprehensive approach ensures that:

1. All existing instances get the Cribl agent installed through SSM

2. All new instances automatically get the agent through User Data or Golden AMIs

3. You have visibility into compliance with CloudWatch metrics and alarms

4. The solution scales across regions and works with different launch template formats

By implementing this solution, you can achieve 100% coverage of your EC2 fleet with the Cribl security agent, ensuring consistent security monitoring across your environment.

The beauty of this approach is its flexibility - while we've focused on the Cribl agent, the same methodology can be applied to deploy any agent or software across your EC2 fleet.

What challenges have you faced when deploying agents across your EC2 fleet? Share your experiences in the comments below!

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.