AWS Logo
Menu
Implement a fully managed shared file storage for Red Hat OpenShift Service on AWS (ROSA) with Amazon FSx for NetApp ONTAP

Implement a fully managed shared file storage for Red Hat OpenShift Service on AWS (ROSA) with Amazon FSx for NetApp ONTAP

ROSA offers an integration with Amazon FSx for NetApp ONTAP NAS – a scalable, fully managed shared storage service built on NetApp's ONTAP file system. With FSx for ONTAP, customers have access to popular ONTAP features like snapshots, FlexClones, cross-region replication with SnapMirror, and a highly available file server with seamless failover.

That Cloud Expert
Published Sep 17, 2024
Kubernetes is a popular choice among many developers for application deployments, and many of these deployments can benefit from a shared storage layer with greater persistency. Red Hat OpenShift Service on AWS (ROSA) is a managed OpenShift integration on AWS developed by Red Hat and jointly supported by AWS and Red Hat.
ROSA clusters typically store data on locally attached Amazon Elastic Block Store (EBS) volumes. Some customers need the underlying data to be persistent and shared across multiple containers, including containers deployed across multiple Availability Zones (AZs). These customers are looking for a storage solution that scales automatically and provides a more consistent interface to run workloads across on-prem and cloud environments.
ROSA offers an integration with Amazon FSx for NetApp ONTAP NAS – a scalable, fully managed shared storage service built on NetApp's ONTAP file system. With FSx for ONTAP, customers have access to popular ONTAP features like snapshots, FlexClones, cross-region replication with SnapMirror, and a highly available file server with seamless failover.
FSx for ONTAP NAS is integrated with the NetApp Trident driver, a dynamic Container Storage Interface (CSI) to handle Kubernetes Persistent Volume Claims (PVCs) on storage disks. The Trident CSI driver manages on-demand provisioning of storage volumes across different deployment environments and makes it easier to scale and protect data for your applications.
In this blog, I demonstrate the use of FSx for ONTAP as a persistent storage layer for ROSA applications. I'll walk through a step-by-step installation of the NetApp Trident CSI driver on a ROSA cluster, provision an FSx for ONTAP NAS file system, deploy a sample stateful application, and demonstrate pod scaling across multi-AZ nodes using dynamic persistent volumes. Finally, I'll cover backup and restore for your application. With this solution, you can set up a shared storage solution that scales across AZ and makes it easier to scale, protect and restore your data using the Trident CSI driver.

Prerequisites

You need the following resources:
The diagram below shows the ROSA cluster deployed in multiple availability zones (AZs). The ROSA cluster's master nodes, infrastructure nodes and worker nodes run in a private subnet of a customer's Virtual Private Cloud (VPC). You'll create an FSx for ONTAP NAS file system within the same VPC and install the Trident driver in the ROSA cluster, allowing all the subnets of this VPC to connect to the file system.
Figure 1 – ROSA integration with Amazon FSx for NetApp ONTAP NAS
Figure 1 – ROSA integration with Amazon FSx for NetApp ONTAP NAS

1. Create a ROSA cluster and clone the GitHub repository

First, install the ROSA cluster. Use Git to clone the GitHub repository. If you don't have Git, install it with the following command:
sudo dnf install git
Clone the Git repository:
git clone https://github.com/aws-samples/rosa-fsx-netapp-ontap.git

2. Provision FSx for ONTAP

Create a multi-AZ FSx for the ONTAP NAS file system in the same VPC as your ROSA cluster.
If you want to provision the file system with a different storage capacity and throughput, you can override the default values by setting StorageCapacity and ThroughputCapacity parameters in the CFN template.
The value for FSxAllowedCIDR is the allowed Classless Inter-Domain Routing (CIDR) range for the FSx for ONTAP security groups ingress rules to control access. You can use 0.0.0.0/0 or any appropriate CIDR to allow all traffic to access the specific ports of FSx for ONTAP.
Also, take note of the VPC ID, the two subnet IDs corresponding to the subnets you want your file system to be in, and all route table IDs associated with the ROSA VPC subnets. Enter those values in the command below.
Run this command in a terminal to create FSx for the ONTAP file system:
Verify that your file system and storage virtual machine (SVM) has been created using the Amazon FSx console, shown below:
Figure 2 – Amazon FSx Console
Figure 2 – Amazon FSx Console

3. Install and configure the Trident CSI driver for the ROSA cluster

Use the following Helm command to install the Trident CSI driver in the "trident" namespace on the OpenShift cluster.
First, add the Astra Trident Helm repository:
helm repo add netapp-trident https://netapp.github.io/trident-helm-chart
Next, use helm install and specify a name for your deployment as in the following example, where 23.01.1 is the version of Astra Trident you are installing:
helm install <name> netapp-trident/trident-operator --version 23.01.1 --create-namespace --namespace trident
Verify the Trident driver installation:
Finally, verify the installation:

4. Create a secret for the SVM username and password in the ROSA cluster

Create a new file with the SVM username and admin password and save it as svm_secret.yaml. A sample svm_secret.yaml file is included in the fsx folder.
svm_secret.yaml
The SVM username and its admin password were created in Step 2. You can retrieve it from the AWS Secrets Manager:
Figure 3 – AWS Secrets Manager Console
Add the secrets to the ROSA cluster with the following command:
oc apply -f svm_secret.yaml
To verify that the secrets have been added to the ROSA cluster, run the following command:
oc get secrets -n trident |grep backend-fsx-ontap-nas
You see the following output:
backend-fsx-ontap-nas-secret Opaque 2 16d

5. Configure the Trident CSI backend to FSx for ONTAP NAS

The Trident back-end configuration tells Trident how to communicate with the storage system (in this case, FSx for ONTAP). You'll use the ontap-nas driver to provision storage volumes.
To get started, move into the fsx directory of your cloned Git repository. Open the file backend-ontap-nas.yaml. Replace the managementLIF and dataLIF in that file with the Management DNS name and NFS DNS name of the Amazon FSx SVM and svm with the SVM name, as shown in the screenshot below.
Note: ManagementLIF and DataLIF can be found in the Amazon FSx Console under "Storage virtual machines" as shown below (highlighted as "Management DNS name" and "NFS DNS name").
Figure 4 - Amazon FSx console - SVM page
Figure 4 - Amazon FSx console - SVM page
Now execute the following commands in the terminal to configure the Trident backend using management DNS and SVM name configuration in the ROSA cluster.
cd fsx
oc apply -f backend-ontap-nas.yaml
Run the following command to verify backend configuration.
Now that Trident is configured, you can create a storage class to use the backend you've created. This is a resource object that describes and classifies the type of storage you can request from different storage types available to Kubernetes cluster. Review the file storage-class-csi-nas.yaml in the fsx folder.

6. Create storage class in ROSA cluster

Now, create a storage class:
oc apply -f storage-class-csi-nas.yaml
Verify the status of the trident-csi storage class creation by running the following command:
This completes the installation of Trident CSI driver and its connectivity to FSx for ONTAP file system. Now you can deploy a sample MySQL stateful application on ROSA using file volumes on FSx for ONTAP.
If you want to verify that applications can create PV using Trident operator, create a PVC using the pvc-trident.yaml file provided in the fsx folder.

Deploy a sample MySQL stateful application

In this section, you deploy a highly-available MySQL application onto the ROSA cluster using a Kubernetes StatefulSet and have the PersistentVolume provisioned by Trident.
Kubernetes StatefulSet verifies the original PersistentVolume (PV) is mounted on the same pod identity when it's rescheduled again to retain data integrity and consistency. For more information about the MySQL application replication configuration, refer to MySQL Official document.

1. Create MySQL Secret

Before you begin with MySQL application deployment, store the application's sensitive information like username and password in Secrets. Save the following manifest to a file name mysql-secrets.yaml and execute the command below to create the secret.
Create the mysql namespace:
oc create namespace mysql
Next, create a mysql secret file called mysql-secrets.yaml:
Apply the YAML to your cluster:
oc apply -f mysql-secrets.yaml
Verify that the secrets were created:
$ oc get secrets -n mysql | grep mysql
mysql-password opaque 1

2. Create application PVC

Next, you must create the PVC for your MySQL application. Save the following manifest in a file named mysql-pvc.yaml:
Apply the YAML to your cluster:
oc apply -f mysql-pvc.yaml
Confirm that the pvc exists:
Run the MySQL application
To deploy your MySQL application on your ROSA cluster, save the following manifest to a file named mysql-deployment.yaml:
Apply the YAML to your cluster:
oc apply -f mysql-deployment.yaml
Verify that the application has been deployed:
A Kuberetes service defines a logical set of pods and a policy to access pods. StatefulSet currently requires a headless service to control the domain of its pods, directly reaching each pod with stable DNS entries. By specifying None for the clusterIP, you can create a headless service.
$ oc apply -f mysql-service.yaml
Verify the service:

5. Create MySQL client for MySQL

You need a MySQL client so you can access the MySQL applications that you just deployed. Review the content of mysql-client.yaml, and then deploy it:
oc apply -f mysql-client.yaml
Verify the pod status:
Log in to the MySQL client pod:
$ oc exec --stdin --tty mysql-client -- sh
Install the MySQL client tool:
$ apk add mysql-client
Within the mysql-client pod, connect to the MySQL server using the following command:
$ mysql -u root -p -h mysql.mysql.svc.cluster.localal
Enter the password stored in the mysql-secrets.yaml. Once connected, create a database in the MySQL database:

Backup and Restore

Now create a Kubernetes VolumeSnapshotClass so you can snapshot the persistent volume claim (PVC) used for the MySQL deployment.

1. Create a snapshot

Save the manifest in a file called volume-snapshot-class.yaml:
Run the manifest:
$ oc create -f volume-snapshot-class.yaml
volumesnapshotclass.snapshot.storage.k8s.io/fsx-snapclass created
Next, create a snapshot of the existing PVC by creating VolumeSnapshot to take a point-in-time copy of your MySQL data. This creates an FSx snapshot that takes almost no space in the filesystem backend. Save this manifest in a file called volume-snapshot.yaml:
Apply it:
$ oc create -f volume-snapshot.yaml volumesnapshot.snapshot.storage.k8s.io/mysql-volume-snap-01 created
And confirm:

2. Delete the database erp

Now you can delete the database erp. Log in to the container console using a new terminal:
$ oc exec --stdin --tty mysql-client -n mysql -- sh
mysql -u root -p -h mysql.mysql.svc.cluster.local
Delete the database erp:
MySQL [(none)]> DROP DATABASE erp;
Query OK, 1 row affected

3. Restore the snapshot

To restore the volume to its previous state, you must create a new PVC based on the data in the snapshot you took. To do this, save the following manifest in a file named pvc-clone.yaml:
Apply it:
And confirm:

4. Redeploy the database

Now you can redeploy your MySQL application with the restored volume. Save the following manifest to a file named mysql-deployment-restore.yaml:
Apply the YAML to your cluster:
$ oc apply -f mysql-deployment-restore.yaml
Verify the application has been deployed with the following command:
To validate that the database has been restored as expected, go back to the container console and show the existing databases:

Summary

In this article, you've seen how quick and easy it can be to restore a stateful application using snapshots, nearly instantly. Combining all the capabilities of FSx for ONTAP NAS with the sub-millisecond latencies and multi-AZ availability, FSx for ONTAP NAS is a great storage option for your containerized applications running in ROSA on AWS.
For more information on this solution, refer to the NetApp Trident documentation. If you would like to improve upon the solution provided in this post, follow the instructions in the GitHub repository.
 

Comments