Unifying Real-Time and Analytical Data with Tableflow and AWS
Confluent and AWS will explore how Tableflow seamlessly bridges real-time streaming data with AWS analytics and AI/ML services. Learn how Tableflow enables effortless movement of Apache Kafka data into Amazon Athena, Amazon Redshift, and Amazon SageMaker Lakehouse, unlocking new insights and accelerating decision-making. This session will dive into key use cases, architecture best practices, and real-world applications, offering a fresh perspective on unifying streaming and analytical data.
Tony Mullen
Amazon Employee
Published Apr 8, 2025
The episode featured a discussion about TableFlow, a new offering from Confluent that enables seamless integration between Apache Kafka streaming data and analytical workloads in AWS services like Athena, Redshift, and SageMaker. Mark demonstrated how TableFlow allows users to easily enable streaming data to be represented as tables in Apache Iceberg or Delta Lake format with just a few clicks, making it instantly accessible through various AWS analytics services.
The discussion covered key aspects of TableFlow including schema evolution, security considerations, and cost efficiency. The speakers emphasized how TableFlow simplifies the traditional ETL pipeline process by handling schema management, metadata publishing, and file compaction automatically in a serverless manner. They explained how the service integrates with AWS security features like IAM roles and private networking options, while maintaining schema consistency between streaming and analytical workloads.
The team shared insights about early customer adoption since TableFlow's general availability two weeks prior to the show, highlighting use cases across financial services, media, and gaming industries. They discussed future developments, including potential "reverse ETL" capabilities that would allow processed analytical data to flow back into operational systems. The conversation concluded with perspectives on how TableFlow fits into the broader evolution of data architecture, particularly in the context of AI/ML workloads and real-time analytics.
Key Highlights:
- TableFlow enables one-click integration between Kafka streams and AWS analytics services
- Serverless architecture with automatic scaling and maintenance
- Native support for Apache Iceberg and Delta Lake formats
- Simplified security model leveraging existing Kafka permissions and AWS IAM roles
- Schema evolution handled automatically through Confluent's Schema Registry
- Cost-efficient design with consumption-based pricing
- Strong early adoption in financial services and media/gaming sectors
- Future roadmap includes bi-directional data flow capabilities
Loading...
Tony Mullen - Senior Database Specialist @AWS
Weifan Liang - Data & Analytics Sr PSA**** @ AWS
Joseph Morais - Technical Champion @ Confluent
Marc Selwan - Staff Product Manager @ Confluent
- Unifying Apache Kafka® Topics with Apache Iceberg and Delta Lake Tables in a Few Clicks - https://www.confluent.io/blog/tableflow-is-now-generally-available/
- Tableflow GA blog post- https://www.confluent.io/blog/tableflow-is-now-generally-available/
- Tableflow overview video - https://www.youtube.com/watch?v=k6mn2Sb1Uh4
- Tableflow lightboard video - https://www.youtube.com/watch?v=O2l5SB-camQ
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.