Data Lake, Firehose, Glue, Athena, S3 And Aws Sdk For .Net

Leverage AWS Kinesis Data Firehose, AWS Glue, S3, Athena and the AWS SDK to build a Data Lake.

What you’ll learn
How to build a Data Lake using Firehose API for .NET, S3, AWS Glue and Athena
Requirements
You must have an active AWS Account to utilized the included code.
Prior knowledge of AWS Web Services would be helpful, but not needed.
Description
The purpose of this class is to demonstrate a proof of concept using a series of lab exercise’s (in the AWS Console using AWS Kinesis Data Firehose, AWS Glue, S3, Athena and the AWS SDK, with C# code using the AWS SDK) of building a Data Lake in the AWS ecosystem.In this class, we will be sending data from a local SQL Server database to AWS RDS securely, and automatically.I am utilizing the .NET SDK for AWS, however, this could easily be migrated to your language of choice once you understand the concepts that I am teaching. I will be providing the full working source code of this proof of concept.I will also provide a working example of creating Parquet files and sending to S3 (without using Firehose).

Overview

Section 1: Introduction

Lecture 1 Introduction

Section 2: Overview of AWS Web Services in a Data Lake

Lecture 2 AWS Serverless pieces to our Data Lake jigsaw puzzle

Section 3: AWS RDS and S3

Lecture 3 S3 Bucket creation

Lecture 4 RDS creation

Section 4: AWS Kinesis Firehose, C# and the AWS SDK

Lecture 5 Creating Kinesis Data Firehose delivery streams

Lecture 6 Console .NET core C# entity-framework project

Lecture 7 Kinesis Firehose C# API class

Lecture 8 Kinesis Firehose C# API send data example

Section 5: AWS Glue, Athena, ETL jobs, Triggers, and using Hangfire

Lecture 9 Creating Glue Crawlers

Lecture 10 Query Data in Athena using SQL

Lecture 11 Creating AWS ETL jobs

Lecture 12 Hangfire – Automation

Section 6: AWS Lake Formation, Cognito and Parquet .NET

Lecture 13 AWS Lake Formation

Lecture 14 AWS Cognito

Lecture 15 Parquet .NET – Alternative to using Firehose

Section 7: Final thoughts and advice

Lecture 16 Conclusion

AWS developers interested in learning about building a data lake using AWS services and the AWS SDK.