AWS Logo
Menu

Use CloudWatch LogGroups for EC2 logging

When EC2 instances get replaced in an autoscaling group. You can loose your application logs. Learn how you can prevent this from happening using AWS CloudWatch LogGroups.

Published Apr 10, 2024
You can protect yourself from losing logs on Amazon EC2 by using CloudWatch Logs. Configure the CloudWatch Agent to stream your logs to a LogGroup. This protects you from losing logs. For example, when the instance is replaced by autoscaling. You are also protected against tampering of the logs. An attacker who has gained access to your system can remove the logs. But the logs in the LogGroup will contain the original log lines.

Preparations

We will need the following resources:
  • InstanceSecurityGroup, allows an outbound connection on 443. The CloudWatch Agent needs to stream the logs to the CloudWatch endpoint.
  • Role, holds the policies that allow the instance to send the logs to the LogGroup.
  • InstanceProfile, the profile used to attach the role to the EC2 instances.
  • LogGroup, the LogGroup used to store the logs in.

Setting up the EC2 Instance

In my previous blog I wrote about how you can create an EC2 instance using Infrastructure as Code. We will continue on that example. Instead of a single instance we will create an autoscaling group.
First we will need a LaunchTemplate. This LaunchTemplate will contain some metadata called AWS::CloudFormation::Init. The metadata has so called configSets and blocks of config. Here is an example that we will use:

CloudFormationInit

The CloudFormationInit holds a configuration. This configuration is used by the cfn-init agent. It comes pre-installed with the AmazonLinux2 and AmazonLinux2023 AMIs. Under the files section you can see that we are defining 2 files:
  • /etc/cfn/cfn-hup.conf
  • /etc/cfn/hooks.d/cfn-auto-reloader.conf
The syntax is quite explanatory. You set an owner, group, mode and the content of the file. The content of the tells the EC2 instance where it can find the metadata. During the initial boot this is given in the user-data. But you might change the metadata overtime. This makes sure that the running instances checks if there is an update available.
To recap this will make sure that the running EC2 instances stay in sync with the given metadata.

CloudWatchLogs

The Amazon Linux AMIs do not come with the CloudWatch Agent you need to install the agent first. You can do this by providing the amazon-cloudwatch-agent package, under packages.
Now it’s time to configure the agent to stream the log files to CloudWatch LogGroups. We have defined 3 files that will be watched by the agent:
  • /var/log/user-data.log
  • /var/log/cfn-hup.log
  • /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
You can extend this list to your own needs. In this example each EC2 instance will have 3 streams. 1 per file and the instance id is used as a prefix, example: i-0000000000/user-data.log.
You can also do measurements on the instance itself and expose this as a CloudWatch Metric. In this example we are capturing a percentage of the used space of the root volume.
We have now configured the agent. But we also need to make sure it’s running. When changes are made to the configuration file, we also need to restart it. We do this by pointing to the config file. Now cfn-init knows when it changes the configuration that it also needs to restart the agent.

User Data

The user data will make sure that the configuration is applied. Since we configure the cfn-init to check the configuration it will stay in sync from here on. The second thing it will do is it will send a signal to the AutoScalingGroup resource. If the cfn-init is executed successfully it will signal a success and if not it will signal a failure.
This means that if the configuration fails the autoscaling group will receive a failure. As a result it will rollback the stack.

AutoScalingGroup

The autoscaling group looks as followed:
We will have 2 policies defined, one for creation and one for updates. If the signal configured in the user-data fails the resource will fail and trigger a rollback. If the instance is configured successfully it will continue. In the auto scaling group you can define the amount of instances required. In this example the maximum is 2 and the minimum 1. This makes sure that at least one EC2 instance is running. When a replacement is needed a new instance will be added first and if it is deployed successfully the old one is removed. This is why the max is set to 2 so that we can replace the old one gracefully.

Conclusion

Configuring a LogGroup to stream your log files is not hard. Using CloudFormation Init you can install and configure applications. In our example we configured the CloudWatch agent that will stream the logs.
When you have to deal with autoscaling events and/or malicious attackers. You can prevent the loss of logs by centralising your application log files.
Photo by Pixabay
 

Comments