Updating your programs for S3 Express One Zone
I'll demonstrate some challenges you may encounter while updating your programs to support S3 Express One Zone
Published Jan 31, 2024
Last Modified Feb 1, 2024
At last year's annual re:Invent conference, AWS launched a new S3 storage class called S3 Express One Zone. With the launch of this storage class you are now able to, for the first time, specify which availability zone your objects are stored in, in order to maximize performance. But there's more to it than that, a lot more.
I have a couple of programs dedicated to performing specific S3 tasks, so let's see what it takes to upgrade them to support this new storage class, and document some of the challenges involved. But first, let's spend some time understanding what's new.
While reading the launch blog post, you are slowly introduced to another concept: the "directory bucket". This is a new bucket type that is required in order to use S3 Express One Zone. You can't just upload files into a regular S3 bucket and specify the new storage class, that won't work. The directory bucket brings with it new restrictions and limitations in order to maximize performance (read How is S3 Express One Zone different?). The more you read about it, the more you realize it is a way bigger change than you expected at first. It may be the biggest change to Amazon S3 since it was launched in 2006.
Normally, these kinds of revisions are hard to make to a service, but since directory buckets are basically a new version of S3 that is running on new endpoints that are serving new buckets created specifically for S3 Express One Zone, it is possible for AWS to make these changes without breaking existing users. You cannot use an older existing program and point it to a directory bucket and expect it to work, because it will not. The authorization scheme is completely different, again in order to maximize performance.
You need to upgrade the AWS SDK version that your program is using in order to support S3 Express One Zone. In many cases simply upgrading the SDK may be enough, depending on your program and how it uses S3. It's magic.
To explain the magic part, we have to look at the bucket name. Directory buckets have a special naming scheme, ending with
--azid--x-s3
. The SDK uses this information to automatically direct the request to the correct endpoint and perform the necessary authorization. There are no new parameters, all the necessary information is packed into the bucket name. It feels a bit unconventional coming from AWS to do it this way, but I think they correctly assumed that it would be the simplest way to roll this out. It makes me wonder how long it took for them to settle on this and find a naming scheme that didn't interfere with any existing bucket names.- Upgrade AWS SDK.
- Use S3 Express One Zone.
- ???
- Profit!
Since there are new restrictions there is a good chance that you actually do have to make changes to your program. As mentioned before, if you are lucky then you only need to upgrade the AWS SDK. Be sure to test your program extensively, though, as there are a lot of small changes that might bite you.
To take a closer look at the changes you may have to make, I went ahead and upgraded three of my programs and tested them with directory buckets and S3 Express One Zone. The programs are:
- shrimp
- s3sha256sum
- s3verify
I'll break down the changes required for each program below.
shrimp is a program that can upload large files to S3 buckets, and it would be great if it could also upload to S3 Express One Zone.
I have created a new directory bucket with the prefix
my-test-bucket
and put it in us-west-2, which makes the full bucket name my-test-bucket--usw2-az1--x-s3
. I will use this bucket in my testing.Let's see how shrimp behaves when attempting to use it with a directory bucket, before the SDK is upgraded:
It complains that the bucket does not exist, which makes sense as I explained previously, directory buckets are basically their own separate S3 service and the normal S3 service is not aware of them. It would be useful if the error message hinted that the user has to upgrade their program to make it compatible with S3 Express One Zone, as they should be able to make this determination on the server side. Compatible clients should never contact the normal S3 service using this bucket name.
Lets upgrade the AWS SDK. All three of my programs are written in Go so I simply have to run
go get -u
to upgrade my dependencies.After the upgrade I attempt to build the program using
go build
, but unexpectedly there are many errors:It turns out that the AWS SDK for Go, aws-sdk-go-v2, released some breaking changes that are unrelated to S3 Express One Zone. You can read more about these changes here.
These breaking changes are fairly easy to fix, luckily. You can take a look at the commit to fix them here: https://github.com/stefansundin/shrimp/commit/7273da630388462378417d0fdf502c5f003e202e
Let's try it again:
Hmm, this seems like a bug on AWS's side. It doesn't seem like you can currently use
GetBucketLocation
with directory buckets. As a workaround we can add --region us-west-2
and shrimp won't try to look up the bucket region. I will report this issue as a bug. In the worst case we may have to start parsing the bucket names ourselves.Let's add
--region us-west-2
and try again:Uh oh. I have programmed shrimp to automatically opt in to use the S3 dual-stack endpoints, and unfortunately S3 Express does not support this yet. AWS has been slow in launching IPv6 support for their service endpoints (see their progress here).
I'll just remove this automatic opt in for now and revisit it in the future. You can take a look at the commit to fix this here: https://github.com/stefansundin/shrimp/commit/3ce042d4f50068a6deb74f548e026e55710afddb
Let's try it again:
Ah, we've finally hit a more interesting problem. Both the blog post and the documentation notes that S3 Express One Zone requires that
/
is used as a delimiter for ListObjectsV2
. It seems like this limitation also applies to other listing operations such as ListMultipartUploads
.I can just remove the
Prefix
parameter in this request, as shrimp already paginates that response. The request would just take a little bit longer if the user many multi-part uploads in progress (which is unlikely). You can take a look at the commit to fix this here: https://github.com/stefansundin/shrimp/commit/576e4bde577d43eab01edbf1ffc5ca50fe65b804Let's try it again:
Yay, it finally worked. 🥳
There may be other minor problems but it seems like the main functionality is working. I will perform more extensive testing before releasing a new version. Let's move on to the next program.
s3sha256sum is a program that calculates SHA-256 checksums of S3 objects. I wrote this program before AWS launched their own feature to support checksums (which you should definitely be using as it makes your upload faster!).
Let's move directly to upgrading the AWS SDK:
There are far fewer errors for this program. You can take a look at the commit to fix them here: https://github.com/stefansundin/s3sha256sum/commit/2dc22699acc7c183ed01de95d04b588b0bd183e9
Let's try it again:
It worked, yay. 🥳 Looks like upgrading s3sha256sum was very simple indeed. Let's move on to the next program.
s3verify is essentially the sequel to s3sha256sum which uses the new checksum feature in S3.
I think you know the drill by now, let's start by upgrading the SDK:
You can take a look at the commit to fix the errors here: https://github.com/stefansundin/s3verify/commit/47401dba869c464b9fee6bd63e654a2935a5c500
Let's see if it works:
This looks interesting, the checksums appear to be the same but on a closer inspection you can see that the S3 checksum has
-1
added to the end. The AWS Management Console has long presented multi-part checksums this way, but the API has so far not appended the number of parts to the checksum value. This error only occurs with multi-part uploads, the program successfully verifies single-part uploads.The fix for this is fairly simple, we need to account for both cases since the regular S3 still computes the checksum the original way. You can take a look at the commit to fix this here: https://github.com/stefansundin/s3verify/commit/5c80b6f7cc76b601e2788ec50142765071915ace
It might be challenging to keep your program is compatible with both versions of S3, especially if they diverge further as time goes on. You will have to remember to routinely test with both going forward, especially with major changes.
Let's try it again:
Yay, it works. 🥳
This concludes the blog post. I hope it was informative for you. If I overlooked something or if you have any questions, please write a comment on dev.to.