logo
How to Access Amazon S3 Bucket like a Local File System

How to Access Amazon S3 Bucket like a Local File System

Access Amazon S3 Bucket like a Local File System using Mountpoint for Amazon S3.

Published Feb 9, 2024
Mountpoint for Amazon S3 is now generally available and ready for production workloads. This open-source simple file client is optimized for read-heavy workloads that require high throughput and focused on high-performance access to large data sets.
In this post, we'll see how to effortlessly access an S3 bucket as a local file system by leveraging Mountpoint for Amazon S3.

Based on your Linux distributions and hardware-platform architecture, download prebuilt packages.
1
2
3
4
5
6
7
8
abhijit@AwsJunkie:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal
abhijit@AwsJunkie:~$ uname -ri
5.15.90.1-microsoft-standard-WSL2 x86_64
I am using the Ubuntu distro of Windows Subsystem for Linux (WSL2) running on x86_64 architecture. So, I'll download DEB-based distributions for x86_64.
RPM-based distributions (Amazon Linux, Fedora, CentOS, RHEL):
DEB-based distributions (Debian, Ubuntu):
Other distributions:
1
2
3
4
5
6
7
8
9
10
11
12
13
abhijit@AwsJunkie:~$ wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
--2023-08-21 15:30:14-- https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
Resolving s3.amazonaws.com (s3.amazonaws.com)... 16.182.66.96, 16.182.66.32, 52.216.208.152, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|16.182.66.96|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9398148 (9.0M) [binary/octet-stream]
Saving to: ‘mount-s3.deb’

mount-s3.deb 100%[=======================>] 8.96M 3.57MB/s in 2.5s

2023-08-21 15:30:17 (3.57 MB/s) - ‘mount-s3.deb’ saved [9398148/9398148]
abhijit@AwsJunkie:~$ ls
mount-s3.deb
Install the downloaded Mountpoint for Amazon S3 package.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
abhijit@AwsJunkie:~$ sudo apt install ./mount-s3.deb
Building dependency tree
Reading state information... Done
Note, selecting 'mount-s3' instead of './mount-s3.deb'
The following package was automatically installed and is no longer required:
libxmlb1
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
mount-s3
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/9398 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 /home/abhijit/mount-s3.deb mount-s3 amd64 1.0.0 [9398 kB]
Selecting previously unselected package mount-s3.
(Reading database ... 42843 files and directories currently installed.)
Preparing to unpack /home/abhijit/mount-s3.deb ...
Unpacking mount-s3 (1.0.0) ...
Setting up mount-s3 (1.0.0) ...
To make sure the installation was successful, please check the version.
1
2
abhijit@AwsJunkie:~$ mount-s3 --version
mount-s3 1.0.0

Mountpoint for Amazon S3 uses the same credentials configuration options as the AWS CLI. So, either use aws configure AWS CLI to create the credential file or create it manually.
1
2
3
4
5
abhijit@AwsJunkie:~$ aws configure
AWS Access Key ID [None]: AKIAUI2UW222A6I6R6O
AWS Secret Access Key [None]: 81926124b-8fcd-4239-9ab0-630f772222
Default region name [None]: us-east-1
Default output format [None]:
or,
Create ~/.aws/credentials file and add aws_access_key_id and aws_secret_access_key.
1
2
abhijit@AwsJunkie:~$ mkdir ~/.aws
abhijit@AwsJunkie:~$ sudo nano ~/.aws/credentials
Sample ~/.aws/credentials content
1
2
3
[default]
aws_access_key_id = AKIAUI2UW222A6I6R6O
aws_secret_access_key = 81926124b-8fcd-4239-9ab0-630f772222
Now create a local directory (e.g. ~/local-dir) and mount it with an S3 bucket (e.g. aj-mounts3-bucket).
1
2
3
abhijit@AwsJunkie:~$ mkdir local-dir
abhijit@AwsJunkie:~$ mount-s3 aj-mounts3-bucket local-dir
bucket aj-mounts3-bucket is mounted at local-dir

Now, we are good to access the S3 bucket as a local file system using different file operations.
Let's create a few files and a directory inside the mounted local directory.
1
2
3
4
5
6
7
8
9
10
11
12
abhijit@AwsJunkie:~$ echo 'file1' > file1.txt
abhijit@AwsJunkie:~$ echo 'file2' > file2.txt
abhijit@AwsJunkie:~$ mkdir dir1
abhijit@AwsJunkie:~$ echo 'file3' > dir1/file3.txt
abhijit@AwsJunkie:~/local-dir$ tree
.
├── dir1
│ └── file3.txt
├── file1.txt
└── file2.txt

1 directory, 3 files
Refresh S3 bucket. And we'll see the exact same files and directories in the S3 bucket as objects.