Using Flux to Implement GitOps on AWS
GitOps is an effective way to achieve continuous deployment based on Kubernetes clusters while meeting enterprise-level requirements such as security, separation of privileges, auditability, and agility. These are the best practices for GitOps based on AWS EKS.
Amazon EKS-based Best Practices for GitOps
1.Deploy Cloud Infrastructure with IaC
1.1 Create a Project with CDK CLI
1.2 Create an EKS Cluster with EKS Blueprints
2.Deploy Flux CD on Amazon EKS Cluster
2.2 Prepare AWS CodeCommit Credentials
2.3 Install Flux on the Cluster
3.Deploy GitOps Workflow with Flux CD
3.3 Multi-Cluster Configuration
3.4 Deploy Microservices with GitOps Workflow
3.4.1 Adding the Microservices Repository Address
3.4.2 Adding CodeCommit Credentials
4.Image-Based Automated Deployment with GitOps Workflow
4.1 Defining the CodePipeline CI
4.2.1 Adding an image policy to the front-end of Git repository
4.2.2 Registering the Front-end of a Microservice Under Flux-repo
4.2.3 Configuring the access credentials for Amazon ECR
4.2.4 Setting image update policy
4.3.1 Update the Front-end Code
4.3.3 ECR Image Version Changing Confirmation
4.3.4 Verify Flux Image Information
4.3.5 Microservice Source Code Automatically Updates
4.3.6 Verify Pod Image Version
About | |
---|---|
✅ AWS experience | 300 - Advanced |
⏱ Time to complete | 90 minutes to complete |
💰 Cost to complete | Free tier eligible |
🧩 Prerequisites | AWS Account |
📢 Feedback | Any feedback, issues, or just a 👍 / 👎 ? |
⏰ Last Updated | 2023-05-19 |
Traditional CD | GitOps |
---|---|
Triggered by push events, such as code commits, timed tasks, manual, etc. | System constantly polls for changes |
Deployment of changes only | Declare the entire system for any deployment |
System might drift between deployments | The system will correct any drift |
Access to the deployment environment is a requirement | Deployment pipeline is authorized to run within system |

flux-repo
, the configuration repository for Flux CD, which is used to define Flux-related resources. The other is microservices-repo
, which saves microservice application configurations and deployment files. The third one is the source repository app-repo
for business services. In this post, a front-end project will be used as an example. We used AWS CodePipeline for continuous integration in the CI/CD pipeline, built and stored the docker image in Amazon ECR, and deployed the CD engine Flux as a pod in the Amazon EKS environment.- Coding engineers write code and push the final code to app-repo.
- Code changes in the app-repo trigger AWS CodePipeline.
- AWS CodePipeline edits and packages code, generates container images, and pushes them to the container image repository/Amazon ECR.
- The CD engine Flux, running in the EKS environment, regularly scans the ECR container image repository and pulls container image metadata for applications.
- The new container image address is automatically synced to the application deployment file stored in microservices-repo via git commit/push when a new version of the container image detected.
- Flux regularly pulls application configurations and deployment files from the Flux-repo. Since the Flux-repo repository references the microservices-repo, Flux checks the consistency of the workload running state of the cluster with the expectations described in the microservices-repo files. If there is any difference, Flux will automatically enable the EKS cluster to synchronize the differences to ensure that workloads run in the expected state.
- Deploy the cloud infrastructure using Infrastructure as Code (IaC)
- Deploy Flux CD on AWS EKS cluster
- Deploy GitOps workflow using Flux CD
- Implement automatic deployment based on images using GitOps workflow
cdk init
, which will create the folder structure and install the modules that TypeScript CDK project needs.1
cdk init --language typescript
quickstart
directory, and then run the following codes to install project dependencies.1
2
3
mkdir quickstart
cd quickstart
npm install @aws-quickstart/eks-blueprints
lib/quickstart-stack.ts
and add the following EKS Blueprints code.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as blueprints from '@aws-quickstart/eks-blueprints';
import { KubernetesVersion } from 'aws-cdk-lib/aws-eks';
export class QuickstartStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const account = props?.env?.account!;
const region = props?.env?.region!;
const clusterProvider = new blueprints.GenericClusterProvider({
version: KubernetesVersion.V1_23,
managedNodeGroups: [
{
id: "default-ng",
desiredSize: 2,
minSize: 1,
maxSize: 2,
}
]
});
const cluster = blueprints.EksBlueprint.builder()
.clusterProvider(clusterProvider)
.account(account)
.region(region)
.addOns(
new blueprints.AwsLoadBalancerControllerAddOn,
)
.teams();
}
}
NoteWe suggest to customize the cluster parameters via clusterProvider and add plugins through the built-in addOns in EKS Blueprints.
NoteWe recommend defining infrastructure with CDK code and use pipelines to manage changes across multiple clusters that is also the best practice of GitOps.
cdk deploy
command to deploy the stack.1
2
3
4
5
6
7
8
kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
aws-load-balancer-controller-6bd49cfb7-2cvlk 1/1 Running 0 5m31s
aws-load-balancer-controller-6bd49cfb7-7lcwd 1/1 Running 0 5m31s
aws-node-9hsvz 1/1 Running 0 99m
coredns-66cb55d4f4-gdcqg 1/1 Running 0 106m
coredns-66cb55d4f4-gmzjt 1/1 Running 0 106m
kube-proxy-wr5jn 1/1 Running 0 99m
- Initialized a CDK project using cdk init.
- Defined an EKS cluster quickly with EKS Blueprint while adding the AWS Application Load Balancer plugin.
So let's install Flux.
1
curl -s https://fluxcd.io/install.sh | sudo bash
1
2
3
4
5
6
7
8
9
10
.
├── apps // Define Application
│ ├── base // Application Infrastructure Layer
│ └── overlays // Application Overlays Layer
├── clusters // Cluster configuration portal
│ └── dev-cluster
├── infrastructure // Infrastructure Shared Components
│ ├── base // Infrastructure Infrastructure layer
│ └── overlays // Infrastructure Overlays layer
└── README.md
NoteWe recommend to dividing Flux-related resources into the infrastructure layer, cluster management layer, and application layer. We support multi-cluster deployment with Kustomization (base, overlays).
username
and password
in the command below with the HTTPS Git credentials for AWS CodeCommit.1
2
3
4
5
6
7
flux bootstrap git \
--url=https://git-codecommit.us-west-2.amazonaws.com/v1/repos/gitops \
--username=__replace_with_your_Git_credential_username__ \
--password=__replace_with_your_Git_credential_password__ \
--token-auth=true \
--path="./clusters/dev-cluster" \
--components-extra=image-reflector-controller,image-automation-controller
WarningEnable the image automatic update feature, add the--components-extra=image-reflector-controller,image-automation-controller
parameter when bootstrapping Flux. Otherwise, Flux will not install image-reflector-controller and image-automation-controller by default, and configurations such as automatic image updates will not take effect.
gotk-components.yaml
: defined the six controllers of Flux: helm, Kustomize, source, notification, image-automation, and image-reflector.gotk-sync.yaml
: the Git source of Flux, the Source Controller in the cluster monitoring code changes in the GitOps repository and making the corresponding changes.kustomization.yaml
: multi-cluster configuration.
1
2
3
4
flux get kustomizations --watch
NAME REVISION SUSPENDED READY MESSAGE
flux-system master/83b7e66 False True Applied revision: master/83b7e66
infrastructure master/83b7e66 False True Applied revision: master/83b7e66
1
2
3
4
5
6
7
8
9
10
11
12
13
kubectl -n flux-system get pod,services
NAME READY STATUS RESTARTS AGE
pod/helm-controller-88f6889c6-sblvd 1/1 Running 0 11m
pod/image-automation-controller-6c5f7bbbd9-8xskk 1/1 Running 0 11m
pod/image-reflector-controller-78949bb4b4-bqn4m 1/1 Running 0 11m
pod/kustomize-controller-784bd54978-s82dq 1/1 Running 0 11m
pod/notification-controller-648bbb9db7-627zs 1/1 Running 0 11m
pod/source-controller-79f7866bc7-gdt95 1/1 Running 0 11m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notification-controller ClusterIP 172.20.60.72 <none> 80/TCP 11m
service/source-controller ClusterIP 172.20.133.44 <none> 80/TCP 11m
service/webhook-receiver ClusterIP 172.20.196.178 <none> 80/TCP 11m
gotk-components.yaml
, gotk-sync.yaml
, and kustomization.yaml
. The following is a summary of this section:- Flux client installation
- Creating an IAM user and CodeCommit credentials
- Installing Flux on an Amazon EKS cluster and enabling the image automatic update feature
NoteFlux regularly pulls the configurations and deployment files from the repository, and compares the current application load status of the cluster with the expected state described in the files. When differences are detected, Flux will automatically synchronize the differences to the EKS cluster, ensuring that workloads always run as expected.
- Best practices for microservices deployment (including examples of mistakes)
- Capabilities for Cross-platform deployment
- The advantages of continuous integration/deployment
- The complementary nature of DevOps and microservices
- A "real" testable application for various orchestration platforms
- Kustomize maintains application configuration across different environments through Base & Overlays.
- Kustomize uses Patch to reuse Base configuration and implementation, and resource reuse is achieved through the difference section between the Overlay description and the Base application configuration.
- Kustomize manages native Kubernetes YAML files, without requiring learning DSL syntax.
1
cp deploy/kubernetes/complete-demo.yaml deploy/kubernetes/base/complete-demo.yaml
kustomization.yaml
:1
2
3
4
5
6
# deploy/kubernetes/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./complete-demo.yaml
overlays/development/kustomization.yaml
file, without copying and modifying the existing complete-demo.yaml.NoteFlux will automatically merge base and overlays configurations according to the environment during service deployment. We recommend defining differential configurations across multiple environments, such as development, testing, and production. We do not favor the strategy of multi-repository and multi-branch strategy here, preferring instead to use different paths in a git repository to manage clusters in multiple environments. It helps simplify the process of code maintenance and merging, and which is also a Flux best practice.
1
2
3
4
5
6
7
8
9
10
11
12
13
.
├── base
│ ├── kustomization.yaml
│ └── sock-shop
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ ├── rbac.yaml
│ └── tenant.yaml
└── overlays
└── development
├── kustomization.yaml
└── sock-shop
└── kustomization.yaml
https://git-codecommit.xxx.amazonaws.com/v1/repos/microservices-repo
.1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: sock-shop-tenant
namespace: sock-shop
spec:
interval: 1m
url: __MICRO_SERVICES_REPO__
ref:
branch: main
secretRef:
name: microservices-basic-access-auth
......
base/sock-shop/basic-access-auth.yaml
, and replace BASE64_USERNAME and BASE64_PASSWORD with the generated base64 encoding:1
2
3
4
5
6
7
8
9
10
---
apiVersion: v1
kind: Secret
metadata:
name: microservices-basic-access-auth
namespace: sock-shop
type: Opaque
data:
username: __BASE64_USERNAME__
password: __BASE64_PASSWORD__
flux get kustomizations -watch
and wait for Flux to update. When the READY status of all kustomizations is True, the deployment is complete.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
kubectl get pod,service -n sock-shop
NAME READY STATUS RESTARTS AGE
pod/carts-b4d4ffb5c-z4jrj 1/1 Running 0 5m28s
pod/carts-db-6c6c68b747-jl5pd 1/1 Running 0 5m28s
pod/catalogue-759cc6b86-qdmvc 1/1 Running 0 5m28s
pod/catalogue-db-96f6f6b4c-zgp5z 1/1 Running 0 5m28s
pod/front-end-5c89db9f57-cvbdl 1/1 Running 0 5m28s
pod/orders-7664c64d75-lqwbm 1/1 Running 0 5m28s
pod/orders-db-659949975f-qv7pl 1/1 Running 0 5m28s
pod/payment-7bcdbf45c9-szrfq 1/1 Running 0 5m28s
pod/queue-master-5f6d6d4796-nkktx 1/1 Running 0 5m28s
pod/rabbitmq-5bcbb547d7-gzhn4 2/2 Running 0 5m28s
pod/session-db-7cf97f8d4f-9mz6v 1/1 Running 0 5m28s
pod/shipping-7f7999ffb7-95rlc 1/1 Running 0 5m27s
pod/user-68df64db9c-kh247 1/1 Running 0 5m27s
pod/user-db-6df7444fc-jlkp9 1/1 Running 0 5m27s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/carts ClusterIP 172.20.33.124 <none> 80/TCP 5m29s
service/carts-db ClusterIP 172.20.75.163 <none> 27017/TCP 5m29s
service/catalogue ClusterIP 172.20.92.254 <none> 80/TCP 5m29s
service/catalogue-db ClusterIP 172.20.242.255 <none> 3306/TCP 5m29s
service/front-end LoadBalancer 172.20.55.188 k8s-sockshop-frontend-12345678910-012345678910abc.elb.us-east-1.amazonaws.com 80:30001/TCP 5m29s
service/orders ClusterIP 172.20.8.252 <none> 80/TCP 5m29s
service/orders-db ClusterIP 172.20.40.212 <none> 27017/TCP 5m29s
service/payment ClusterIP 172.20.6.218 <none> 80/TCP 5m29s
service/queue-master ClusterIP 172.20.153.80 <none> 80/TCP 5m29s
service/rabbitmq ClusterIP 172.20.99.37 <none> 5672/TCP,9090/TCP 5m29s
service/session-db ClusterIP 172.20.37.111 <none> 6379/TCP 5m29s
service/shipping ClusterIP 172.20.43.252 <none> 80/TCP 5m29s
service/user ClusterIP 172.20.220.174 <none> 80/TCP 5m29s
service/user-db ClusterIP 172.20.70.57 <none> 27017/TCP 5m29s
- Sock Shop Introduction
- Learn a configuration management tool- Kustomize (base, overlays) and how to modify the microservice multi-cluster deployment
- Build a GitOps workflow and deploy microservices
buildspec.yml
file to the front-end project source code to define the CI process executed in the CodePipeline:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws --version
- AWS_ACCOUNT_ID=`echo $REPOSITORY_URI|cut -d"." -f1`
- AWS_DEFAULT_REGION=`echo $REPOSITORY_URI|cut -d"." -f4`
- echo $AWS_ACCOUNT_ID $AWS_DEFAULT_REGION
- aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $REPOSITORY_HOST
- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
- IMAGE_TAG=main-$COMMIT_HASH-$CODEBUILD_BUILD_NUMBER
- echo $IMAGE_TAG
build:
commands:
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $REPOSITORY_URI:latest .
- docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker images...
- docker push $REPOSITORY_URI:latest
- docker push $REPOSITORY_URI:$IMAGE_TAG
weaveworksdemos/front-end
if any front-end code changed. The format of the image tag is [branch]-[commit]-[build number].--components-extra=image-reflector-controller,image-automation-controller
when you repeat the Flux bootstrap to enable it.- Register the image repository of the front-end microservice to allow Flux to periodically scan the ECR image repository correspondent to the front-end project.
- Configure the credentials for accessing the image repository. Flux needs the credentials to access ECR image repository to read the image information.
- Set the image updating policy. In most cases, we do not want all the image versions changes to trigger CD every time. Instead, we only want the specified branch (main) code changes to trigger CD. A special update policy is needed to fulfill this need.
ACCOUNT_ID
with your own ACCOUNT_ID).1
2
3
4
5
6
7
8
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: weaveworksdemos/front-end
newName: __ACCOUNT_ID__.dkr.ecr.us-west-2.amazonaws.com/weaveworksdemos/front-end # {"$imagepolicy": "sock-shop:sock-shop-front-end:name"}
newTag: latest # {"$imagepolicy": "sock-shop:sock-shop-front-end:tag"}
Note
Warning: The annotation$imagepolicy
, which is for locating, is mandatory. If Flux discovers the image version is changed, it will locate and modify the file content according to this annotation.
apps/overlays/development/sock-shop/registry.yaml
, and replace ACCOUNT_ID
with your own ACCOUNT_ID
.1
2
3
4
5
6
7
8
9
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
name: sock-shop-front-end
namespace: sock-shop
spec:
image: __ACCOUNT_ID__.dkr.ecr.xxxx.amazonaws.com/weaveworksdemos/front-end
interval: 1m0s
- Automatic authentication mechanism (image-reflector-controller retrieves credentials by itself, only applicable to: Amazon ECR, GCP GCR, Azure ACR)
- Regularly refreshing credentials (stored in the cluster through Secret) with CronJob
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
- patch: |-
- op: add
path: /spec/template/spec/containers/0/args/-
value: --aws-autologin-for-ecr
target:
version: v1
group: apps
kind: Deployment
name: image-reflector-controller
namespace: flux-system
NoteWe used Amazon ECR to choose the automatic authentication mechanism, modifyclusters/dev-cluster/flux-system/kustomization.yaml
and add the--aws-autologin-for-ecr
parameter through patching. This approach is simpler and more efficient when compared to using CronJob to generate credentials regularly.
gitops/apps/overlays/development/sock-shop/policy.yaml
. The following rules match image versions such as master-d480788-1
, master-d480788-2
, and master-d480788-3
.1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
name: sock-shop-front-end
spec:
imageRepositoryRef:
name: sock-shop-front-end
filterTags:
pattern: '^main-[a-fA-F0-9]+-(?P<buidnumber>.*)'
extract: '$buidnumber'
policy:
numerical:
order: asc
gitops/apps/overlays/development/sock-shop/image-automation.yaml
. Flux's automatic image configuration will specify a Git repository for the application configuration, including branch, path, and other information.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: sock-shop-front-end
spec:
git:
checkout:
ref:
branch: main
commit:
author:
email: fluxcdbot@users.noreply.github.com
name: fluxcdbot
messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
push:
branch: main
interval: 5m0s
sourceRef:
kind: GitRepository
name: sock-shop-tenant
namespace: sock-shop
update:
path: ./deploy/kubernetes/overlays/development
strategy: Setters
weaveworksdemos/front-end
image version:1
2
3
4
5
6
7
8
9
flux get images all --all-namespaces
NAMESPACE NAME LAST SCAN SUSPENDED READY MESSAGE
sock-shop imagerepository/sock-shop-front-end 2022-09-18T14:46:45Z False True successful scan, found 20 tags
NAMESPACE NAME LATEST IMAGE READYMESSAGE
sock-shop imagepolicy/sock-shop-front-end 267314483271.dkr.ecr.us-west-2.amazonaws.com/weaveworksdemos/front-end:master-1f49071-24 True Latest image tag for '267314483271.dkr.ecr.us-west-2.amazonaws.com/weaveworksdemos/front-end' resolved to: master-1f49071-24
NAMESPACE NAME LAST RUN SUSPENDED READY MESSAGE
sock-shop imageupdateautomation/sock-shop-front-end 2022-09-18T14:43:51Z False True no updates made; last commit 1ff6d91 at 2022-09-18T14:34:40Z
1
2
kubectl describe pod/front-end-759476784b-9r2rt -n sock-shop | grep Image:
Image: 267314483271.dkr.ecr.us-west-2.amazonaws.com/weaveworksdemos/front-end:master-1f49071-24
- Implementing the CI process through CodePipeline to achieve continuous integration of front-end code.
- Locating and modifying business configuration file by annotating Flux.
- Configuring Flux's image update policy to enable Flux to monitor the specific versions of images and complete automatic deployment.
- How to Gray-Release with security and increments for critical online-production-systems?
- How to improve the GitOps Key Management on cloud when Sealed Secrets introduced additional private key management requirements.
- How to Coordinate manage for Coding of IaC and EKS GitOps
- How to develop Kubernetes manifests (YAML) more efficiently
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.