Select your cookie preferences

We use essential cookies and similar tools that are necessary to provide our site and services. We use performance cookies to collect anonymous statistics, so we can understand how customers use our site and make improvements. Essential cookies cannot be deactivated, but you can choose “Customize” or “Decline” to decline performance cookies.

If you agree, AWS and approved third parties will also use cookies to provide useful site features, remember your preferences, and display relevant content, including relevant advertising. To accept or decline all non-essential cookies, choose “Accept” or “Decline.” To make more detailed choices, choose “Customize.”

AWS Logo
Menu

Back Up On-Premises Data Incrementally to Amazon S3 over SFTP using AWS Transfer Family

Business often want to securely backup on-premises data to the cloud using familiar SFTP protocol with a static IP for SFTP server. This blog demonstrates how to use AWS Transfer Family to create an SFTP endpoint which can be exposed via a static IP, enabling incremental backups from your local servers to Amazon S3.

backuptransferfamilys3
Sameer
Amazon Employee
Published Dec 11, 2024
Comments
This blog post covers the process of setting up an SFTP endpoint using AWS Transfer Family to facilitate daily incremental backups from an on-premises server to Amazon S3. This solution is helpful for organizations looking to securely transfer data to the cloud while maintaining their existing SFTP-based backup processes.

Prerequisite

  1. Allocate an Elastic IP, you can follow steps mentioned in section Allocate an Elastic IP address
  2. Create an Amazon S3 bucket to store your data. You can follow steps mentioned in Creating a bucket.
  3. Create an IAM role for Amazon S3 access. You can follow the steps mentioned in Create an IAM Role and Policy.
  4. For demonstration I will use Amazon EC2 instance as the source server from where I will upload data to Amazon S3. If you already have a server you can use that to send your data to Amazon S3 otherwise you can follow steps mentioned in the tutorial to launch an EC2 instance.
  5. Follow the steps mentioned in Generate SSH keys to get a private and public key pair which will be used in further steps.

Create an SFTP Server using AWS Transfer Family

  1. Open AWS Transfer Family console.
  2. In the navigation pane, click Servers then click Create server
    Image not found
    Create Server
  3. On the next page, choose SFTP (SSH File Transfer Protocol) - file transfer over Secure Shell, click next
  4. On the next page, for Identity Provider for SFTP, FTPS, or FTP choose Service managed
  5. For Endpoint configuration, configure the following inputs:
    • Choose VPC Hosted as Endpoint type.
    • Custom hostname: None, you can also use Amazon Route53 DNS alias or other DNS if required
    • Choose a VPC which also spans across the availability zone selected for Elastic IP.
  • Image not found
    Configure endpoint parameters
  • Choose the availability zone and elastic IP address created earlier. Then, click Next
Image not found
Configure Availability Zone and Elastic IP
  1. Choose Amazon S3 as Domain
  2. On the next page, keep parameters to default and click Next
  3. Review the details and click Create server
  4. Wait for the server status to be online
  5. Select the created server and click Add user
Image not found
Create User
  1. Enter Username: 'username'
  2. Select the IAM role created in step 2 of Prerequisited as the IAM role while adding user.
  3. For Home Directory, choose the S3 bucket that you have created to store your data.
  4. For SSH public keys, you need to paste the content of the public key. Paste the content of the Public Key created in the step 5 of prerequisites.
  5. Click Add

Configure script on the source server to backup data

In case you already have a source server from where you want to backup the data, you can perform further steps on the same source server. Or if your server only requires the IP address or endpoint you can use the endpoing and IP address of the SFTP server created in the previous section.
  1. Connect to your on-premises server or Amazon EC2 instance. You can follow steps for Connecting to Amazon EC2 instance
  2. I've created a folder app-logs which you will backup to Amazon S3 everyday. You can see the following screenshot having app-logs folder and keys stored under the same directory.
Image not found
  1. Create a python script which does the following:
    1. Established SSH connection with the SFTP server and opens a SFTP session.
    2. Identifies files which were changed in last 24 hours.
    3. Performs the file transfer operation over SFTP for the modified files.
    4. Ensures SFTP session and SSH connection is closed properly.
    5. Sample script for the above actions:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import os
import paramiko
from datetime import datetime, timedelta

SFTP_HOST = 'your-transfer-family-endpoint or IP address'
SFTP_USERNAME = 'your-sftp-username'
SFTP_PRIVATE_KEY = '/path/to/your/private-key.pem'
LOCAL_DIR = '/path/to/local/data'
REMOTE_DIR = 'backup/directory'
TIME_THRESHOLD = datetime.now() - timedelta(hours=24)

def incremental_backup():
# Set up the SSH client
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

# Load the EC private key
key = paramiko.ECDSAKey.from_private_key_file(SFTP_PRIVATE_KEY)

# Connect to the SFTP server
ssh.connect(SFTP_HOST, username=SFTP_USERNAME, pkey=key)

# Open an SFTP session
sftp = ssh.open_sftp()

try:
for root, dirs, files in os.walk(LOCAL_DIR):
for file in files:
local_path = os.path.join(root, file)
remote_path = os.path.join(REMOTE_DIR, os.path.relpath(local_path, LOCAL_DIR))

if os.stat(local_path).st_mtime > TIME_THRESHOLD.timestamp():
print(f"Uploading {local_path} to {remote_path}")
sftp.put(local_path, remote_path)
finally:
sftp.close()
ssh.close()

if __name__ == "__main__":
incremental_backup()
Now, you can create a cron job on the source server to execute the file everyday which will backup the data to Amazon S3.

Conclusion

In this blog, I have shown steps to create a SFTP server using AWS transfer family and highlighted how get a static IP for the SFTP server. I created an Amazon EC2 instance to show how to establish SSH connection and setup script to backup data incrementally everyday.
 

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Comments

Comments

Log in to comment