Select your cookie preferences

We use essential cookies and similar tools that are necessary to provide our site and services. We use performance cookies to collect anonymous statistics, so we can understand how customers use our site and make improvements. Essential cookies cannot be deactivated, but you can choose “Customize” or “Decline” to decline performance cookies.

If you agree, AWS and approved third parties will also use cookies to provide useful site features, remember your preferences, and display relevant content, including relevant advertising. To accept or decline all non-essential cookies, choose “Accept” or “Decline.” To make more detailed choices, choose “Customize.”

AWS Logo
Menu

Data Solutions Framework on AWS | S02 EP16

In this show we would be discussing about Data Solutions Framework on AWS (DSF), an opinionated open source framework that accelerates building data solutions on AWS. It can take days or weeks to build end-to-end solutions on AWS with infrastructure as code (IaC) and following best practices, but with DSF it takes hours and you can focus on your use case.

awsanalyticsdatawell-arch-framework
Prasad Matkar
Amazon Employee
Published May 3, 2024
Last Modified Nov 7, 2024
Comments
In this episode host Prasad is joined with guest Lotfi and they have an in-depth discussion about the data solutions framework on AWS.
The data solutions framework is an AWS CDK library built on top of the AWS CDK that helps deploy data workloads on AWS more quickly. It provides developers with pre-built constructs to accelerate the process of setting up common data architecture patterns. Lotfi explains that it was built based on feedback from customers about the complexity and time needed to build out data platforms. He gives examples of some key constructs provided such as:
  • S3-based data lake with encryption and lifecycle policies
  • Data catalog powered by AWS Glue
  • Utilities like bulk data copy between S3 buckets
  • Spark job orchestration using AWS Step Functions
  • Packaging PySpark code and dependencies for AWS EMR
  • Kafka API for Amazon MSK
The constructs are designed to be modular so they can be used independently or composed together. There is also flexibility to customize things like data tiering policies or encryption keys.
In the second half of the episode, Lotfi does a live demo showing how to use the framework to stand up an end-to-end data lake architecture on AWS. This includes S3 buckets.
To view this Twitch stream, please accept cookies.
Hosts of the show 🎤
Prasad Matkar - Database Specialist Solutions Architect @ AWS

Guests 🎤

Lotfi Mouhib - Principal Solutions Architect @ AWS

Links from today's episode

Check out Past Shows

You can check out our past shows from out community page -https://community.aws/livestreams/lets-talk-about-data
 

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Comments

Comments

Log in to comment