EKS Secret Management: Pod Identity & ASCP with Amazon Q CLI

In today's cloud-native world, managing secrets securely in Kubernetes environments is essential to maintaining robust application security. This blog post shares our journey of implementing a robust secret management solution for Amazon EKS using Pod Identity and the AWS Secrets and Configuration Provider (ASCP), with Amazon Q Developer as our guide. For comprehensive guidance on configuring ASCP with Pod Identity, we recommend referring to the official AWS Security Blog post: Announcing ASCP integration with Pod Identity: Enhanced security for secrets management in Amazon EKS.

The Challenge: Secure Secret Management in Kubernetes

There are several challenges with traditional Kubernetes secret management:

Native Kubernetes secrets are only base64 encoded, not encrypted at rest
Managing IAM roles and service account credentials was complex
Secret rotation and access control required custom solutions
Storing secrets in version control systems posed security risks

We needed a solution that would provide strong encryption, fine-grained access control, and seamless integration with our EKS environment.

Enter Amazon Q Developer

For this implementation, we used the new agentic coding experience within the Amazon Q Developer CLI, which allowed us to interactively develop, execute, and troubleshoot Terraform scripts as well as interact with our EKS cluster through kubectl. Learn more about this powerful tool at Introducing the enhanced command line interface in Amazon Q Developer. Amazon Q Developer suggested leveraging AWS Secrets Manager with the AWS Secrets and Configuration Provider (ASCP) and EKS Pod Identity - a modern approach that would address our security concerns while simplifying operations.

Disclaimer: The code in this blog was generated with Amazon Q Developer. While tested in our environment, AI-generated content may evolve over time. We recommend reviewing all scripts and testing thoroughly in non-production environments before deployment.

Interaction with Amazon Q Developer

Here are some examples of the prompts and Q's responses:

Prompt Examples:

Step 1: Setting Up the EKS Cluster

Q provided a comprehensive configuration that included:

A VPC with public and private subnets across 3 availability zones
An EKS cluster with version 1.32
A managed node group with t3.medium instances
All required IAM roles and policies- Essential EKS addons, including CoreDNS, kube-proxy, VPC CNI, and the critical Pod Identity Agent

Here is a short demo for the interaction:

Step 2: Creating a Secret in AWS Secrets Manager

Next, Q guided me in creating a secret in AWS Secrets Manager to store our database credentials. The Terraform code Q provided included:

A properly named secret with appropriate tags
A secret version with our database username and password
A recovery window configuration for enhanced security

Step 3: Setting Up IAM Permissions

For secure access to our secrets, Q helped to generate Terraform code for :

An IAM policy with least-privilege permissions to access only our specific secret
An IAM role with the appropriate trust policy for Pod Identity
The necessary role-policy attachment Q emphasized the importance of the trust policy, which allows pods in our EKS cluster to assume the role and access the secrets.

Step 4: Installing the ASCP Secret Store CSI Driver

To enable our pods to mount secrets, Q provided Helm configuration for the Secrets Store CSI Driver and the ASCP plugin. This setup included:

The Secrets Store CSI Driver Helm chart
Configuration to enable syncing to Kubernetes secrets
The ASCP plugin installation

Step 5: Creating the service account and Pod Identity Association

Amazon Q Developer provides the YAML configuration for creating service account and Pod Identity Association:

Step 6: Configuring the SecretProviderClass and Deployment

Finally, Q helped us create:

A SecretProviderClass that defines how our secrets should be mounted
A deployment that uses the mounted secrets

Step 7: Verifying Our Implementation

After applying our Terraform configuration, we checked the status of our pods and examined the logs:

Success! Our application could access the secrets without having any credentials stored in the application code or as environment variables.

How Amazon Q Developer Enhanced Our Implementation Process

Throughout this journey, Amazon Q Developer provided significant value:

Complete Infrastructure as Code: Q developed the entire Terraform configuration from scratch, including the EKS cluster, networking, IAM roles, and all necessary addons.
Contextual Understanding: Q understood our EKS environment and recommended appropriate configurations based on our cluster version and enabled addons.
Troubleshooting Assistance: Amazon Q Developer diagnosed the Terraform configuration errors, provided specific code fixes, and guided us through the deployment process.
Security Best Practices: Q Developer strengthened our security posture by implementing least privilege IAM policies, read-only volume mounts, namespace isolation, and secret rotation strategies. These automated security recommendations ensured our infrastructure adhered to AWS best practices from the start.

Conclusion

Amazon Q Developer accelerated our EKS secret management implementation using Pod Identity and ASCP. The solution automated infrastructure setup, reduced development time, and strengthened security controls.

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.