Back Up On-Premises Data Incrementally to Amazon S3 over SFTP using AWS Transfer Family

Prerequisite

1. Allocate an Elastic IP, you can follow steps mentioned in section Allocate an Elastic IP address
2. Create an Amazon S3 bucket to store your data. You can follow steps mentioned in Creating a bucket.
3. Create an IAM role for Amazon S3 access. You can follow the steps mentioned in Create an IAM Role and Policy.
4. For demonstration I will use Amazon EC2 instance as the source server from where I will upload data to Amazon S3. If you already have a server you can use that to send your data to Amazon S3 otherwise you can follow steps mentioned in the tutorial to launch an EC2 instance.
5. Follow the steps mentioned in Generate SSH keys to get a private and public key pair which will be used in further steps.

Create an SFTP Server using AWS Transfer Family

1. Open AWS Transfer Family console.
2. In the navigation pane, click Servers then click Create server
   

   

3. On the next page, choose SFTP (SSH File Transfer Protocol) - file transfer over Secure Shell, click next
4. On the next page, for Identity Provider for SFTP, FTPS, or FTP choose Service managed
5. For Endpoint configuration, configure the following inputs:
   • Choose VPC Hosted as Endpoint type.
   • Custom hostname: None, you can also use Amazon Route53 DNS alias or other DNS if required
   • Choose a VPC which also spans across the availability zone selected for Elastic IP.

• 

• Choose the availability zone and elastic IP address created earlier. Then, click Next

1. Choose Amazon S3 as Domain
2. On the next page, keep parameters to default and click Next
3. Review the details and click Create server
4. Wait for the server status to be online
5. Select the created server and click Add user

1. Enter Username: 'username'
2. Select the IAM role created in step 2 of Prerequisited as the IAM role while adding user.
3. For Home Directory, choose the S3 bucket that you have created to store your data.
4. For SSH public keys, you need to paste the content of the public key. Paste the content of the Public Key created in the step 5 of prerequisites.
5. Click Add

Configure script on the source server to backup data

1. Connect to your on-premises server or Amazon EC2 instance. You can follow steps for Connecting to Amazon EC2 instance
2. I've created a folder app-logs which you will backup to Amazon S3 everyday. You can see the following screenshot having app-logs folder and keys stored under the same directory.

1. Create a python script which does the following:
   1. Established SSH connection with the SFTP server and opens a SFTP session.
   2. Identifies files which were changed in last 24 hours.
   3. Performs the file transfer operation over SFTP for the modified files.
   4. Ensures SFTP session and SSH connection is closed properly.
   5. Sample script for the above actions:

Conclusion