Automatic Golden Image Generation using CI/CD

Everyone, In every organization, security and compliance guardrails are measured in order to maintain the things are aligned with client expectations and agreement. There are many types of guardrails or compliance parameters out of which golden image creation is one of them. Before going into deep dive, let's under stand what is Golden Image.

Golden Image is basically an image that has all required or supporting packages to be installed like agent packages, software or utilities packages, vulnerability agent package etc. there can be other packages installed which are approved by client. So when you're going to build a golden image for the first time, you just have to make sure that all required tools are installed and running fine in that server(windows/linux) to support the environment. After all this needs to be aligned with approved SOE parameters document. Along with making sure all packages are installed, another thing which is OS needs to be updated with latest patches for current month. Once these all are done, then take a snapshot of that instance and consider as base image which is known as Golden Image. This image would be used for further server build activity in future.

In this project, I have planned to build golden image for the first time as I didn't have any image earlier, so it's kind of we are starting from scratch. So, let me tell you guys first, that below are the planned action items to be done for this project ->

Note: As this is done for the first time, so ansible is required because there is no OS hardening parameters implemented. After instance provisioning with latest patches and implementing all security standards, once image is created, then for next month activity, Ansible will not be required because OS hardening parameters would have baked up in last month.

I have taken a sample base image (not last month golden image) as a reference, fetched this image using terraform and created a new EC2 instance.

var.tf

data.tf:

instance.tf

output.tf

userdata.sh

Here, we have used a shell script to get prerequisites installed for Ansible like user creation and providing sudo access etc.

Note: Before triggering ansible job in GitLab, please make sure you login to the server you built from gitlab runner as gitlab-runner is going to login to new server for ansible provisioning and that time it will get an error if we don't perform below one ->

main.yml

There are 4 stages created for entire deployment activity. Initially it will start with validation to make sure if all required services are running fine as expected.

If yes, then it will proceed for resource(EC2) build using Terraform. Here, I have used Terraform Cloud to make things more reliable and store state file in managed memory location provided by Hashicorp. But terraform cli can be used without any issues.

After successful resource build, provisioning needs to be performed to implement basic security standards and complete OS hardening process using Ansible CLI.

At last, once provisioning with patching is completed, pipeline job will take an AMI using AWS CLI commands.

Below are the required stages for this pipeline ->

.gitlab-ci.yml

As per below images, we can see instances has been launched and provisioned successfully, post that AMI has been taken.

So, after all we are at the end of this blog, I hope we all get an idea or approach how pipeline can be set up to build image without any manual intervention. However, in the pipeline I have referred Continuous Delivery approach, hence few stages are set to be trigged manually. There is one thing to highlight mandatorily which is "Do not set Ansible stage(job3) in gitlab as automatic. Use when: manual key to set this stage manual. As I mentioned on the top, ansible stage requires gitlab runner to login to newly build server which I could have mentioned as a command in the pipeline, but I didn't, thought of lets verify things by entering into the server from gitlab runner.

Hopefully you have enjoyed this blog, please go through this one and do the hands-on for sure🙂🙂. Please let me know how did you feel, what went well and how and where I could have done little better. All responses are welcome💗💗.

For upcoming updates, please stay tuned and get in touch. In the meantime, let's dive into below GitHub repository -> 👇👇

https://github.com/dasanirban834/automated-soe-image-pipeline

Thanks Much!!

Anirban Das.