One SFTP to Rule Them All: Consolidate and Conquer!
Why and how we build a Secure SFTP Platform to allow file and mass objects transfer
Published Oct 11, 2024
In this article, we will explore why and how we build a highly secure and scalable Secure File Transfer Protocol (SFTP) platform using AWS Transfer Family, Amazon S3 for storage, AWS Cognito for custom authentication, AWS Lambda for executing authentication logic, and AWS IAM for managing fine-grained permissions.
During the last years and acquisitions, we realized that we had a lot of file transfer services, running in different datacenters with different stack of technologies and protocols, like Microsoft Windows, Linux servers and in some cases running old and unsafe protocols, like pure FTP. Well, to organize it and trust in a unique and centralized platform we decided to build a platform that could be used to any product or service.
And doing this way, we got some key advantages, like:
Maintaining multiple SFTP platforms can be expensive in terms of infrastructure, operational overhead, and resource management. By consolidating into a single SFTP platform, we could:
- Reduce Infrastructure Costs: Operating one platform lowers the need for additional resources (e.g., servers, storage, networking).
- Optimize Licensing and Maintenance: Only one instance of software or services (e.g., AWS Transfer Family, S3, IAM) needs to be licensed, maintained, and monitored.
- Lower Storage Costs: Instead of having separate S3 buckets or other storage systems per product, storage can be consolidated, and lifecycle policies can be unified to optimize costs. We applied a lifecycle policy to keep the data only for 30 days, so, there is not a unlimited storage, because in our design the data is only temporary, not permanent.
Having one platform simplifies the management of the SFTP system:
- Centralized Monitoring: It becomes easier to track, monitor, and audit file transfer activities across all products from a single point.
- Unified Backup and Disaster Recovery Plans: Instead of handling separate recovery solutions per product, a single system allows for unified disaster recovery procedures, making backups and data protection more streamlined.
- Simplified Troubleshooting: When issues arise, IT teams have one place to investigate, diagnose, and resolve problems, reducing downtime and operational complexity.
A centralized SFTP platform allows for stronger security controls and easier compliance:
- Unified Security Policies: Security policies, such as encryption, access controls, and audit logging, can be applied consistently across all products. Managing multiple platforms introduces the risk of inconsistent security configurations.
- Centralized Access Control: Using AWS IAM, one set of roles and policies can be configured to provide fine-grained permissions for different product teams or users. This ensures that access is controlled and audited in a unified manner across all products.
- Reduced Attack Surface: By reducing the number of platforms, the overall attack surface is minimized, lowering the risk of vulnerabilities being exploited across multiple systems.
With a single SFTP platform:
- Shared Resources: Processing power, storage, and bandwidth are pooled and shared across all products. This maximizes the utilization of your AWS Transfer Family instance and other infrastructure components.
- Scalability: It’s easier to scale a single platform based on the combined needs of all products than to individually scale multiple systems for each product's requirements.
A centralized platform ensures consistency in the way file transfers are handled:
- Standardization: All products follow the same processes, protocols, and best practices for file transfers. This reduces the risk of inconsistencies between different product platforms.
- Common Authentication & Authorization Framework: With a single authentication mechanism (e.g., Cognito and Lambda) and common authorization (via IAM), the user experience is consistent across all product lines.
In some companies, different products may need to share data or collaborate with common partners. A single SFTP platform allows for easier data exchange:
- Cross-Product Data Sharing: When products require shared data access, it is simpler to manage and configure cross-product access using a single platform. IAM roles can be leveraged to control granular permissions across product boundaries.
- Partner Integration: External partners or customers who need access to multiple products’ data don’t need to manage separate credentials and endpoints for each product. This simplifies their interaction with your systems and improves user experience.
Launching new products becomes easier with a single SFTP platform:
- No Need to Recreate Infrastructure: New products can plug into the existing platform, avoiding the need to set up new file transfer systems from scratch.
- Faster Onboarding: Onboarding new users, partners, or teams for new products is streamlined when they can leverage an existing platform with standardized processes and controls.
When a company operates in industries with stringent compliance and regulatory requirements, a centralized SFTP platform simplifies compliance efforts:
- Easier Compliance Audits: Auditors can review a single system rather than multiple platforms, making it easier to demonstrate compliance with standards like GDPR, HIPAA, or PCI-DSS.
- Consistent Logging and Reporting: Logging and reporting for file transfers can be managed centrally, ensuring that all transfers across all products are tracked consistently, providing a clear audit trail.
Image not found
This platform can be extended and customized based on specific use cases, offering flexibility and scalability for organizations of all sizes. In my case, the application is integrating with its UI/API’s so users have one simple way to transfer files.
- AWS Network Inspection Layer: Thinking in secure the platform and following all infosec requirements related to exposing services to internet, we added the platform behind our network inspection layer, doing a solid and strong inspection over all traffic, and allowing only our external load balancer to get in touch with the SFTP service, customers use a NLB (network load balancer) to establish a connection.
- AWS Transfer Family: The primary service that enables SFTP, FTPS, and FTP protocols directly in AWS without the need for on-premise infrastructure. This acts as the gateway to securely transfer files into and out of your Amazon S3 buckets. Our platform only uses SFTP as a valid protocol, FTP and FTPS are disabled by default, and we do not intent to enable it.
- Amazon S3: Serves as the storage layer where all the incoming and outgoing files are stored. S3 provides scalability, durability, and native features like lifecycle policies for automatic data archiving or deletion.
- AWS Cognito: Used as a custom Identity Provider (IdP) for user authentication. This allows integration with federated authentication mechanisms or internal user directories. AWS Cognito's ability to integrate custom attributes and support multiple identity pools gives flexibility for managing user access.
- AWS Lambda: A serverless function responsible for handling the authentication workflow. In our setup, it interacts with Cognito to validate user credentials and performs any additional custom logic like logging or monitoring authentication events.
- AWS IAM: The authorization mechanism that ensures that users only have access to the resources (files/folders) they are authorized to manage. Policies are enforced based on IAM roles tied to user sessions.
By integrating AWS Transfer Family with Amazon S3, AWS Cognito, Lambda, and IAM, we build a robust and scalable SFTP platform that supports custom authentication and fine-grained access control. This architecture eliminates the need for traditional file transfer servers, reduces operational overhead, and provides native cloud features like security, auditability, and automation.
This platform can be extended and customized based on specific use cases, offering flexibility and scalability for organizations of all sizes.
Thanks all !!!
Comments
Log in to comment