AWS open source newsletter, #185
A weekly round up of the latest open source news, projects, and events that every open source developer should know about.
Ricardo Sueiras
Amazon Employee
Published Jan 22, 2024
Welcome to issue #185 of the AWS open source newsletter, the newsletter where we try and provide you the best open source on AWS content.
As always, this week we start with a round up of some freshly baked new projects for you to practice your four freedoms. This week we have projects that allow you to export your Partyrock applications, a tool to help you reduce hallucinations in your large language models, a new client for Redis, a tool to help you access the AWS Partner Network, as well as sample projects that look at how you can use large langue models to build a new reader and building pipelines using Cloudformation.
We also feature posts that include some of your favourite open source technologies, which this issue include Redis, Backstage, AWS CDK, Kubernetes, Amazon Corretto, SnapStart, Dash, FluxCD, OpenSearch, Apache Airflow, Apache Kafka, Apache Hudi, Amazon EMR, MySQL, OpenShift, Linux, ROSA, and more.
When I was small I was always told to never skip my vegetables, so heed this excellent advice and do not skip the Events section at the end! I have added a new link to a repo that lists open source events that are coming up in 2024, which is well worth checking out.
Feedback
Please please please take 1 minute to complete this short survey.
The articles and projects shared in this newsletter are only possible thanks to the many contributors in open source. I would like to shout out and thank those folks who really do power open source and enable us all to learn and build on top of what they have created.
So thank you to the following open source heroes: Stephen Sennett, Zhuo-Wei Lee, Alontay Ellis, Rajarshi Das, Nearform, Wolfgang Unger, John McCloskey, Adam Keller, Vadym Kazulkin, Paulo Siecola, Sunny Bhambhani, Daniel Rivera, David Nalley, Parnab Basak, Chandan Rupakheti, Vinod Jayendra, Rupesh Tiwari, Arnav Mediratta, Dumlu Timuralp, Pratik R. Mankad, Praseeda Sathaye, Ovidiu Valeanu, and Olly Pomeroy.
The great thing about open source projects is that you can review the source code. If you like the look of these projects, make sure you that take a look at the code, and if it is useful to you, get in touch with the maintainer to provide feedback, suggestions or even submit a contribution. The projects mentioned here do not represent any formal recommendation or endorsement, I am just sharing for greater awareness as I think they look useful and interesting!
partysmith is an awesome project from AWS Community Builder Stephen Sennett, that provides an unofficial way to transform your AWS PartyRock apps into deployable full-stack SvelteKit applications. Users can enter the URL of a publicly published PartyRock app, select the desired settings, and PartySmith will forge an application into a ZIP archive which will be downloaded to your machine, and ready for use. How cool is that! (Very in case you were wondering). Find out more by reading the supporting blog post, PartySmith - Bring PartyRock apps to your place.
RefChecker For all their remarkable abilities, large language models (LLMs) have an Achilles heel, which is their tendency to hallucinate, or make assertions that sound plausible but are factually inaccurate. RefChecker provides automatic checking pipeline and benchmark dataset for detecting fine-grained hallucinations generated by Large Language Models. Check out the supporting post for this tool, New tool, dataset help detect hallucinations in large language models
glide-for-redis or General Language Independent Driver for the Enterprise (GLIDE) for Redis (mayeb GLIDER would have been cooler :-) is a new open source client for Redis that works with any Redis distribution that adheres to the Redis Serialization Protocol (RESP) specification. The client is optimised for security, performance, minimal downtime, and observability, and comes pre-configured with best practices learned from over a decade of operating Redis-compatible services used by hundreds of thousands of customers.
aws-signer-oci-artifacts this project is used to demonstrate how OCI artefacts can be signed and verified in a development pipeline. Zhuo-Wei Lee, Alontay Ellis, and Rajarshi Das have put together a blog post to help you get started, so if this project interests you, make sure you dive into Signing and Validating OCI Artifacts with AWS Signer.
codecatalyst-blueprints This repository contains common blueprint components, the base blueprint constructs and several public blueprints. Blueprints are code generators used to create and maintain projects in Amazon CodeCatalyst.
aws-apn-connector this project from the folks at Nearform provides a way of interacting with the AWS APN (AWS Partner Network) programatically, as this does not provide an API. If you are looking to automate your interactions with the AWS APN, you should check this project out.
get-the-news-rss-atom-feed-summary is a fantastic demo that demonstrates some of the capabilities that using large language models can help you introduce into your applications. The demo code provides a summary of the most recent news from an RSS or Atom feed using Amazon Bedrock.
cfn-pipeline is a repo from Wolfgang Unger that contains an AWS Codepipeline that will allow automated Cloudformation deployments from within AWS Codepipeline. To help you get started, Wolfgang has put together a detailed blog post that includes videos. Go check it out, Pipeline for automatic CloudFormation Deployments
What is new from around the community this week? John McCloskey from Bazaarvoice gets us going this week with his post, Can You Keep a Secret? which is a must read for anyone who is using or looking at using backstage, an open platform for building developer portals. If you are looking for a good resource for understanding and getting started with the AWS Cloud Development Kit (CDK), then Adam Keller has your back with the post, The evolution of Infrastructure as Code with the AWS Cloud Development Kit (CDK). AWS Community Builder Vadym Kazulkin continues his quest to help Java developers everywhere better understand modern Java applications in the thirteenth (yes, thats right!!) instalment of his series, AWS SnapStart - Part 13 Measuring warm starts with Java 21 using different Lambda memory settings. This is a must read series to dive deeper into how you can optimise your Java applications on AWS, so check it out.
Next up we have Paulo Siecola with How to host a Plotly Dash app on AWS ECS that shows you how to deploy Dash, a popular open source Python framework for building ML & data science web apps, using AWS CDK. If you are a Kubernetes developer who likes to use helm charts to deploy your applications, then Installing multiple helm charts in one go using helmfile from Sunny Bhambhani will be of interest. In this post he shows you how to install multiple helm charts. To finish up this week we have Daniel Rivera who has put together Smart-Cash Project - GitOps with FluxCD, the second post in a series that provides plenty of memes, but even more code, that shows you how you get started with GitOps using FluxCD.
The OpenSearch project has achieved another milestone, launching the inaugural OpenSearch Leadership Committee. David Nalley has written about this in his post, OpenSearch Expands Leadership Beyond AWS. The committee is another step toward open governance in which a broad community of users, contributors, and committers is included in a publicly visible and accessible process that determines an open source project’s direction. This weeks must read post, so check it out to find out more.
This is a post that is going to make a lot of Apache Airflow folks very happy. In Disaster recovery strategies for Amazon MWAA – Part 1, Parnab Basak, Chandan Rupakheti, Vinod Jayendra, and Rupesh Tiwari embark on an epic journey to explore the need for Amazon Managed Workflows for Apache Airflow (MWAA) disaster recovery and prescribe solutions that will sustain Amazon MWAA environments against unintended disruptions. Essential reading for Airflow fans.
Important news this week from the Amazon EKS team, so make sure you read this. In Amazon EKS extended support for Kubernetes versions pricing Arnav Mediratta talks about your extended support options, including pricing, for those of you wanting to understand more about running older versions of Kubernetes in your Amazon EKS clusters. This post covers what it is in more details, use cases, and cost.
Sticking with Kubernetes we have How to leverage Application Load Balancer’s advanced request routing to route application traffic across multiple Amazon EKS clusters where Dumlu Timuralp, Pratik R. Mankad, and Praseeda Sathaye show you how to use the advanced request routing capability of an existing Application Load Balancer to route traffic to micro services spread across multiple Amazon EKS clusters in a given AWS Region. [hands on]
Also touching on the Kubernetes universe was this post, Deploying and scaling Apache Kafka on Amazon EKS where Ovidiu Valeanu dives deep and walks you through how to build and deploy a Kafka cluster with Amazon Elastic Kubernetes Service (Amazon EKS) using Data on EKS (DoEKS). [hands on]
- The Journey to IPv6 on Amazon EKS: Foundation (Part 1) is a series of three posts that looks to provide everything you need to know to make informed decisions around moving to IpV6 for your Kubernetes clusters
- Real-time data distribution with Amazon MSK and AWS AppSync demonstrate how to leverage AWS services for ingesting and serving streaming data to client applications with minimal operational complexity [hands on]
- Real-time serverless data ingestion from your Kafka clusters into Amazon Timestream using Kafka Connect is a hands on guide on how to stream events from your Amazon MSK clusters to Amazon Timestream using the Timestream sink connector that helps you further analyse your time-series data [hands on]
- Enforce fine-grained access control on Open Table Formats via Amazon EMR integrated with AWS Lake Formation shows you how to implement fine-grained access controls on Apache Hudi tables using Amazon EMR integrated with Lake Formation [hands on]
- Scientific Data Management on AWS with Open Source Quilt Data Packages looks at what Quilt Data Packages are (they were new to me!), and explores the concept of data packages for managing scientific data, both for wet-lab and computational data
There were two end of support announcements last week that you should pay attention to. First up we had Announcing end-of-support for AWS SDK for Go (v1) effective July 31, 2025 that provides a time line and suggestions options for those of you who might be using this. Following this, we had Announcing end-of-support for AWS SDK for Java v1.x effective December 31, 2025 that provides similar help around showing you a time line and making recommendations for you.
On Jan 16, 2024 Amazon announced quarterly security and critical updates for Amazon Corretto Long-Term Supported (LTS) and Feature (FR) versions of OpenJDK. Corretto 21.0.2, 17.0.10, 11.0.22, 8u402 are now available for download. Amazon Corretto is a no-cost, multi-platform, production-ready distribution of OpenJDK. You can get your downloads over at the Amazon Corretto download page.
Amazon Relational Database Service (Amazon RDS) for MySQL now supports multi-source replication, which allows you to configure multiple RDS for MySQL database instances as sources for a single RDS for MySQL target database instance. Multi-source replication on RDS for MySQL enables you to merge multiple shards into a single target, or consolidate data in one RDS for MySQL instance for analytics, or long-term backups. For every target RDS for MySQL instance, you can configure up to 15 RDS for MySQL source database instances. For read scaling, you can configure read replicas on the multi-source replication target instance. Multi-source replication is available on RDS for MySQL versions 5.7.44, 8.0.35 and higher minor versions.
The Red Hat OpenShift Service on AWS (ROSA) with hosted control planes (HCP) deployment model is generally available. Under the classic ROSA deployment model, AWS infrastructure required to run the ROSA control plane is hosted on your AWS account. Now, you can create ROSA clusters with the control plane hosted and managed on a service account. Like ROSA classic, ROSA with HCP clusters incur on-demand service fees per worker node vCPU. Additionally, ROSA with HCP clusters incur an on-demand service fee per cluster hour. For service fee discounts, you can optionally purchase upfront ROSA with HCP contracts from the ROSA console. ROSA with HCP reduces the cost of running ROSA, speeds up cluster creation times, improves version upgrade flexibility, and uses AWS managed policies to provide ROSA IAM service roles with narrowly-scoped permissions. ROSA with HCP helps reduce the cost of running ROSA clusters, as ROSA control planes are hosted and managed by Red Hat on AWS. This approach mitigates the risk that actions taken on your account lead to a degraded ROSA control plane. The new deployment model reduces ROSA cluster creation times, and you can separately schedule OpenShift version upgrades for the control plane and the worker node machine pools.
AWS Systems Manager now supports instances running Ubuntu 23.04, Debian 12, and SUSE SP5. Systems Manager customers running these operating systems versions now have access to all AWS Systems Manager Node Management capabilities, including Fleet Manager, Compliance, Inventory, Hybrid Activations, Session Manager, Run Command, State Manager, Patch Manager, and Distributor.
Seekable OCI (SOCI) is open sourced technology that allows you to launch containers faster by lazily loading container images. In this video, Olly Pomeroy walks you through how this is used in Amazon ECS Fargate to help applications deploy and scale out faster by enabling containers to start without waiting to download the entire container image.
If you are planning any events in 2024, either virtual, in person, or hybrid, get in touch as I would love to share details of your event with readers.
I recently found this GitHub repo, open-source-events that is a curated set of open source events for 2024. Go check it out and see what 2024 is looking like.
FOSDEM Brussels, Belgium 3rd and 4th February, 2024
FOSDEM is a free event for software developers to meet, share ideas and collaborate. Every year, thousands of developers of free and open source software from all over the world gather at the event in Brussels. There are plenty of folks from AWS that will be there, so drop me a DM if you are going and want to meet up. You don't need to register. Just turn up and join in!
Find out more by heading to the site home page, FOSDEM 24
SOOCon 24 London, UK 7th and 8th February, 2024
The State of Open Conference is back in the new year, and what better opportunity to combine with your trip to Fosdem, than come along to the wonderful Brewery in the City of London and catch up with the hundreds of open source developers, enthusiasts, hackers, and more. Check out the event page, and if you are so inclined, there is still chance to submit a talk as the call for papers (cfp) is open until the 15th of December.
You can view more at the site page, SOOCon24
Cortex Every other Thursday, next one 16th February
The Cortex community call happens every two weeks on Thursday, alternating at 1200 UTC and 1700 UTC. You can check out the GitHub project for more details, go to the Community Meetings section. The community calls keep a rolling doc of previous meetings, so you can catch up on the previous discussions. Check the Cortex Community Meetings Notes for more info.
OpenSearch Every other Tuesday, 3pm GMT
This regular meet-up is for anyone interested in OpenSearch & Open Distro. All skill levels are welcome and they cover and welcome talks on topics including: search, logging, log analytics, and data visualisation.
Sign up to the next session, OpenSearch Community Meeting
Remember to check out the Open Source homepage for more open source goodness.
Also a reminder, we now have a GitHub repo that just lists all the great open source code that is shared within this newsletter. Where I can hear you all ask? Well as you ask so nicely, you can meander over to newsletter-oss-projects.
Made with ♥ from DevRel
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.