Zero-ETL | S02 EP34 | Lets Talk About Data

Explaining what ETL (Extract, Transform, Load) means and how it allows companies to work with data from multiple sources. We discussed traditional ETL methods involving coding and cloud tools like AWS Glue.

After that, we discussed "Zero ETL", which is a concept that allows quickly setting up a proof-of-concept data warehouse without the overhead of building extensive ETL workflows upfront. Zero ETL enables connecting data sources like databases and data lakes to a data warehouse like Amazon Redshift with just a few clicks. This allows quickly evaluating the value of integrating the data, before investing in more complex ETL processes. The hosts explain the tradeoffs between using federated queries versus fully importing data into Redshift.

Finally, Kate demonstrated setting up a Zero ETL integration between an Aurora Postgres database and Amazon Redshift, as well as ingesting real-time data from Amazon Kinesis. They then build a machine learning model in Redshift to detect fraudulent credit card transactions based on the combined historical and streaming data. The model is trained on known fraud data, and then used to flag potentially fraudulent transactions in real-time as new data arrives.

- Zero ETL allows quickly setting up a data warehouse to evaluate the value of integrating data sources.
- It provides a "minimum viable product" approach to data integration.
- Federated queries allow querying external data sources, while fully importing data into Redshift provides better performance.

Ibrahim Emara, RDS Specialist Solutions Architect @ AWS

Kate Gawron, Leader in cloud databases