Data Engineering

 Introduction to Big Data 

- Definition of Big Data
- Characteristics (Volume, Velocity, Variety, Veracity, Value)
- Importance and Use Cases 

Big Data Ecosystem

- Overview of Big Data Tools and Technologies
- Hadoop Ecosystem
- NoSQL Databases
- Data Warehousing Solutions

 Introduction to Hadoop

- What is Hadoop?
- History of Hadoop
- Use Cases and Advantages          

 Hadoop Architecture   

- HDFS (Hadoop Distributed File System)
- MapReduce
- YARN (Yet Another Resource Negotiator) 

Lab

- Setting up Hadoop cluster
- Basic HDFS commands
- Running a simple MapReduce job 

 Hadoop Installation   

- Installing Hadoop
- Configuring Hadoop   

Lab

- Hands-on installation of Hadoop on a local machine or a cloud instance                                              

 HDFS Architecture     

- HDFS Components (NameNode, DataNode, Secondary NameNode)
- HDFS Read/Write Process 

Lab

-  Exploring HDFS architecture and components                                       

 HDFS Commands         

- Basic HDFS Commands (mkdir, ls, put, get, rm, etc.)
- HDFS File Permissions

Lab

- Practicing HDFS commands
- Managing file permissions in HDFS                     

 MapReduce Basics      

- Introduction to MapReduce
- MapReduce Workflow 

 Lab

- Writing a basic MapReduce program                                                                            

 YARN Architecture     

- YARN Components (Resource Manager, Node Manager, Application Master)
- YARN Workflow

Lab

-  Exploring YARN architecture and components                                         

 Introduction to Hive 

- What is Hive?
- Hive vs RDBMS
- Use Cases of Hive               

 Hive Architecture    

- Hive Components (Driver, Compiler, Metastore, etc.)
- Hive Query Language (HQL)
- Data Storage in Hive 

Lab

- Setting up Hive
- Exploring Hive components                                           

 Hive Installation    

- Installing and Configuring Hive
- Hive Shell and Beeline

Lab

- Installation of Hive on Hadoop cluster
- Basic commands in Hive Shell and Beeline                 

 Hive Data Types      

- Primitive Data Types
- Collection Data Types      

Lab

-  Creating tables with various data types                                                                  

 Hive Tables          

- Managed Tables
- External Tables
- Partitioned Tables
- Bucketing 

Lab

- Creating and managing tables
- Working with partitioned and bucketed tables             

 Hive File Formats    

- TextFile
- SequenceFile
- ORC (Optimized Row Columnar)
- Parquet 

Lab

-  Creating and using tables with different file formats                                   

 HiveQL Basics        

- SQL Select statements
- Filtering and Sorting Data
- Joins in Hive 

Lab

- Writing basic HiveQL queries
- Performing joins in Hive                                 

HiveQL (Hive Query Language)

 Advanced HiveQL      

- Subqueries
- Views and Indexes
- User-Defined Functions (UDFs)
- Windowing and Analytics Functions 

Lab

- Writing advanced HiveQL queries
- Creating and using UDFs                               
 

 Hive DDL Operations  

- Create, Alter, Drop Table
- Create, Alter, Drop Database       

Lab

- Practicing DDL operations                                                                    

 Hive DML Operations  

- Insert, Update, Delete
- Load Data           

Lab

- Practicing DML operations                                                                                      

 Hive Optimization    

- Query Optimization Techniques
- Indexing and Bucketing
- Partition Pruning 

Lab

- Optimization techniques and performance tuning                                          

 Hive Indexes         

- Creating and Managing Indexes
- Using Indexes to Improve Performance 

Lab

- Creating and using indexes to optimize queries                                          

Hive Project

- Usecase of hive project

Lab

- create usecase using data

 Introduction to Spark 

- What is Apache Spark?
- Spark vs Hadoop MapReduce
- Intro to components of Spark (Spark Core, Spark SQL, Spark Streaming, MLlib, GraphX)

 Spark Architecture    

- Spark Architecture and Components
- RDD (Resilient Distributed Dataset)
- DAG (Directed Acyclic Graph)
- Spark Cluster Modes (Standalone, YARN, Mesos) 

Lab

- Setting up Spark cluster
- Understanding RDDs and DAGs                                

 Spark Installation    

- Installing and Configuring Spark
- Spark Shell and Notebooks       

Lab

- Installation of Spark on Hadoop cluster or local machine
- Using Spark Shell and Notebooks 

 Working with RDDs     

- Creating RDDs
- Transformations and Actions
- Lazy Evaluation 

Lab

- Creating and manipulating RDDs
- Performing transformations and actions                  

 Advanced RDD Operations 

- Key-Value Pair RDDs
- Aggregations
- RDD Persistence and Caching 

Lab

- Performing advanced RDD operations
- Persisting and caching RDDs                        

 Spark Configurations  

- Spark Configuration Properties
- Tuning Parallelism
- Memory Management 

Lab

- Configuring Spark for optimal performance
- Managing memory and parallelism             

 Introduction to Spark SQL 

- Spark SQL Overview
- DataFrames and Datasets
- SQL Queries    

Lab

- Working with DataFrames and Datasets
- Executing SQL queries in Spark                    

 DataFrame Operations  

- Creating DataFrames
- Transformations on DataFrames
- Aggregations and Joins 

Lab

- Creating and transforming DataFrames
- Performing aggregations and joins               

 Working with Structured Data 

- Schemas and Encoders
- Reading and Writing Data
- Integrating with Hive 

Lab

- Defining schemas and encoders
- Reading/writing data from various sources              

 Introduction to Spark Streaming 

- What is Spark Streaming?
- Discretized Stream (DStream)
- Spark Streaming Architecture 

Lab

- Setting up a streaming application
- Understanding DStream architecture                 

 Working with DStreams 

- Creating DStreams
- Transformations on DStreams
- Window Operations on DStreams 

Lab

- Creating and transforming DStreams
- Performing window operations on DStreams           

 Spark Performance Tuning 

- Optimization Techniques
- Memory Management
- Serialization
- Resources optimization
- Code optimzation
- Handelling skew 

Lab

- Performance tuning and optimization
- Managing memory and serialization

 Best Practices in Spark 

- Coding Best Practices
- Monitoring and Debugging Spark Applications 

Lab

- Writing efficient Spark code
- Monitoring and debugging Spark applications              

 Introduction to Kafka 

- What is Apache Kafka?
- Kafka vs Other Messaging Systems
- Use Cases of Kafka 

 Kafka Architecture    

- Kafka Components (Broker, Zookeeper, Producer, Consumer, Topic, Partition)
- Kafka Workflow
- Kafka Cluster Architecture 

Lab

- Setting up a Kafka cluster
- Understanding Kafka components and workflow              

 Kafka Installation    

- Installing and Configuring Kafka
- Kafka CLI Commands              

Lab

- Installation of Kafka on local machine or cloud instance
- Using Kafka CLI commands    

 Kafka Producers       

- Producing Messages to Kafka
- Producer API
- Advanced Producer Configurations 

Lab

- Writing a simple producer
- Sending messages to Kafka topics                          

 Kafka Consumers       

- Consuming Messages from Kafka
- Consumer API
- Advanced Consumer Configurations 

Lab

- Writing a simple consumer
- Reading messages from Kafka topics                        

 Producer and Consumer Best Practices 

- Producer Acknowledgments
- Consumer Offsets and Group Management
- Idempotent Producers and Exactly Once Semantics 

Lab

- Implementing best practices for producers and consumers
- Managing offsets and ensuring message delivery 

 Kafka Streams         

- Introduction to Kafka Streams
- Stream Processing Concepts
- KStream and KTable 

Lab

- Implementing stream processing applications using Kafka Streams                         

 Advanced Kafka Streams 

- State Stores and Windowing
- Joins and Aggregations
- Error Handling and Reprocessing 

Lab

- Building stateful stream processing applications
- Performing joins and aggregations on streams 

 Kafka Streams Best Practices 

- Scaling and Fault Tolerance
- Testing and Debugging
- Performance Tuning 

Lab

- Implementing best practices for Kafka Streams applications
- Optimizing and scaling stream processing 

 KSQL                  

- Introduction to KSQL
- Querying Kafka Topics with SQL
- Building Streaming Applications with KSQL

Lab

- Writing KSQL queries to process streams
- Building streaming applications using KSQL   

 Introduction to Kafka Connect 

- What is Kafka Connect?
- Connectors and Tasks
- Use Cases of Kafka Connect 

 Setting Up Kafka Connect 

- Installing and Configuring Kafka Connect
- Kafka Connect Properties and Workers 

Lab

- Setting up Kafka Connect on a local machine or cluster
- Configuring workers and properties 

 Source Connectors     

- Introduction to Source Connectors
- Common Source Connectors (JDBC, FileSource)
- Configuring and Running Source Connectors 

Lab

- Setting up and using source connectors to ingest data into Kafka topics                

 Sink Connectors       

- Introduction to Sink Connectors
- Common Sink Connectors (HDFS, Elasticsearch)
- Configuring and Running Sink Connectors 

Lab

- Setting up and using sink connectors to export data from Kafka topics                   

 Kafka Connect Transformations 

- Single Message Transforms (SMTs)
- Custom Transformations
- Error Handling in Connectors 

Lab

- Implementing single message transforms
- Writing custom transformations for connectors 

 Kafka Use Cases       

- Common Kafka Use Cases (Messaging, Log Aggregation, Stream Processing)
- Real-World Kafka Deployments 

Lab

- Discussing real-world Kafka use cases
- Designing and implementing a Kafka-based solution 

 Introduction to Sqoop 

- What is Apache Sqoop?
- Use Cases of Sqoop
- Sqoop Architecture 

 Sqoop Installation    

- Installing and Configuring Sqoop
- Sqoop Command-Line Interface    

Lab

- Installing Sqoop on Hadoop cluster or local machine
- Configuring Sqoop properties     

 Basic Sqoop Import    

- Importing Data from RDBMS to HDFS
- Importing Data to Hive
- Importing Data to HBase 

Lab

- Importing tables from MySQL to HDFS
- Importing data into Hive tables                  

 Advanced Sqoop Import 

- Incremental Imports
- Free-form Query Imports
- Importing Data to HBase 

Lab

- Performing incremental imports
- Using free-form queries for import
- Importing data into HBase 

 Basic Sqoop Export    

- Exporting Data from HDFS to RDBMS
- Exporting Data from Hive to RDBMS 

Lab

- Exporting data from HDFS to MySQL
- Exporting Hive table data to PostgreSQL           

 Advanced Sqoop Export 

- Exporting Data from HBase to RDBMS
- Performance Tuning for Export Operations 

Lab

- Exporting data from HBase to MySQL
- Tuning export performance                         

 Introduction to AWS    

- Overview of AWS
- Key AWS Services
- AWS Global Infrastructure (Regions and Availability Zones)
 

 Amazon S3 Basics       

- Introduction to S3
- Buckets and Objects
- S3 Storage Classes 

Lab

- Creating S3 buckets
- Uploading and managing objects in S3                             

 S3 Security and Management 

- Bucket Policies and IAM Policies
- Versioning and Lifecycle Policies
- Logging and Monitoring 

Lab

- Configuring bucket policies
- Implementing versioning and lifecycle rules
- Enabling S3 logging and monitoring 

 Advanced S3 Features   

- S3 Transfer Acceleration
- Cross-Region Replication
- S3 Select
- S3 Notification

Lab

- Using S3 Transfer Acceleration
- Setting up cross-region replication
- Querying data with S3 Select 

CloudWatch Basics      

- Introduction to CloudWatch
- CloudWatch Metrics
- CloudWatch Alarms 

Lab

- Creating CloudWatch alarms
- Monitoring AWS resources using CloudWatch metrics        

 Advanced CloudWatch    

- CloudWatch Logs
- CloudWatch Events
- CloudWatch Dashboards 

Lab

- Setting up CloudWatch Logs
- Creating CloudWatch Events
- Building CloudWatch Dashboards  

 Introduction to Athena 

- What is Amazon Athena?
- Use Cases of Athena
- Querying Data with Athena

Lab

- Basic operations on Athena 

 Athena Querying        

- Writing SQL Queries in Athena
- Using Athena with S3
- Performance Tuning 

Lab

- Querying S3 data using Athena
- Optimizing Athena queries for performance             

Redshift Basics        

- Introduction to Redshift
- Redshift Architecture
- Use Cases of Redshift 

Redshift Setup and Management 

- Creating Redshift Clusters
- Managing Cluster Performance
- Redshift Security 

Lab

- Creating and managing Redshift clusters
- Configuring security settings for Redshift  

Advanced Redshift    

- Redshift Spectrum
- Data Loading and Unloading
- Performance Tuning  

Lab

- Using Redshift Spectrum
- Loading and unloading data
- Optimizing Redshift performance  

RDS Basics   

- Introduction to RDS
- RDS Instances
- RDS Use Cases                  

RDS Setup and Management 

- Creating RDS Instances
- Backup and Restore
- RDS Security 

Lab

- Creating and managing RDS instances
- Configuring backup and restore options            

Advanced RDS Features 

- Reading data from RDS
- Dbeaver Connection 

Lab

- Querying data
 

AWS DynamoDB

- Introduction to NoSQL databases and DynamoDB
- Data modeling and schema design
- Provisioned vs. On-demand capacity  

Lab

- Setting up DynamoDB tables and indexes
- Querying and scanning data in DynamoDB 
- DynamoDB transactions and best practices 
- Participants will also learn about DynamoDB Streams for real-time data processing        

AWS Kinesis Family     

- Overview of Kinesis Streams, Firehose, and Analytics
- Real-time data processing with Kinesis Streams
- Analytics and insights using Kinesis Analytics

Lab

- Setting up Kinesis streams and data ingestion
- Data transformation with Kinesis Firehose
- Integrating Kinesis with Lambda for real-time processing. Participants will also explore Kinesis Data Analytics for real-time insights and anomaly detection

 AWS SQS                 

- Introduction to message queues and SQS
- FIFO vs. Standard queues
- Scaling and performance considerations 

Lab

- Lab: Creating SQS queues and managing message lifecycle
- Message visibility and handling; Integrating SQS with Lambda for event-driven processing
- Participants will also practice implementing message-driven architectures using SQS

AWS SNS                 

- Overview of pub/sub messaging with SNS
- Topic-based vs. Direct messaging
- Integrating SNS with other AWS services 

Lab 

- Creating SNS topics and subscriptions
- Message filtering and delivery policies
- Handling message delivery failures
- Participants will explore SNS message attributes for fine-grained message filtering and dead-letter queues for handling failed messages

AWS Lambda 

- Serverless computing concepts and introduction to Lambda
- Event sources and function invocation
- Managing Lambda functions and monitoring performance 

Lab

- Creating Lambda functions and triggers 
- Writing Lambda functions in different programming languages
- Integrating Lambda with other AWS services
- Participants will also learn about Lambda@Edge for extending Lambda to the edge of the AWS global network          

AWS EMR

- Overview of EMR (Elastic MapReduce) and big data processing
- Setting up EMR clusters and configuring applications
- Running and monitoring EMR jobs 

Lab 

- Setting up EMR clusters with specific configurations
- Running sample big data processing jobs 
- Monitoring job performance and scaling EMR clusters if needed 
- Participants will explore EMR integration with Apache Spark and Apache Hadoop for distributed data processing               

AWS Glue

- Introduction to AWS Glue and its role in ETL processes
- Options available in Glue
- Paramaters and libraries 

Lab

- Setting up AWS Glue jobs and understanding ETL workflows 
- Configuring Glue crawlers to discover and catalog data 
- Writing Glue scripts for data transformation               

AWS Glue Connection

- Overview of AWS Glue Connection and its role in data source access  

Lab

- Creating Glue connections to various data sources such as Amazon S3, RDS, Redshift, etc.
- Configuring connection properties and access permissions     

AWS Glue Crawler        

- Understanding AWS Glue Crawler for automated data cataloging         

Lab

- Configuring Glue crawlers to automatically discover and catalog data from various sources; 
- Defining crawler schedules and output formats

Git and Gitlab

- Git installation
- Git Commands
- Git Branches
- Gitlab integration

Lab

- Setup git and push changes to repository using git

DevOps

- CICD process
- Dbeaver
- Putty/SSH