DBATU · Third Year B.Tech Artificial Intelligence & Allied · NEP 2020
Data Engineering and Big Data Systems
26AFAIPC504 · Semester V · 3 credits · Continuous assessment 20 + mid-semester 20 + end-semester 60
Unit 1: Fundamentals of Data Engineering and Big Data
5 hoursThe evolution of data processing and the data engineering lifecycle; the five Vs of big data; data sources; batch and stream processing; big data architecture; data engineering roles and the modern ecosystem; distributed file systems.
Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then
- Data engineering lifecycle
- Characteristics of big data
- Batch and real-time processing
- Big data architecture
Unit 2: Distributed Storage and Processing Frameworks
6 hoursHDFS architecture, NameNode and DataNode, replication and fault tolerance; the Hadoop ecosystem; MapReduce with mappers, reducers, combiners and partitioning; Apache Spark, RDDs, DataFrames and Spark SQL; MapReduce versus Spark.
Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then
- HDFS architecture
- Replication and fault tolerance
- Hadoop ecosystem
- MapReduce programming model
- RDD transformations and actions
- DataFrames and Spark SQL
- MapReduce versus Spark
Unit 3: Data Warehousing and Data Integration
6 hoursOperational databases versus warehouses; warehouse architecture, fact and dimension tables, star and snowflake schemas and data marts; OLAP; ETL and ELT; data integration, quality and metadata; data lakes and lakehouses.
Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then
- Data warehouse architecture
- Star and snowflake schemas
- OLAP operations
- ETL and ELT
- Data lakes and lakehouses
Unit 4: Real-Time Data Engineering
5 hoursStreaming data and event-driven architecture; publish–subscribe and message brokers; Apache Kafka producers, consumers and topics; stream and window-based processing; streaming pipelines, log processing and monitoring pipeline performance.
Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then
- Event-driven architecture and publish-subscribe
- Apache Kafka architecture
- Stream processing and windowing
- Data pipeline architecture
Unit 5: Data Governance, Security and Applications
5 hoursData governance, ownership and stewardship; privacy, compliance and security; backup and recovery; lineage and catalogues; workflow orchestration with Apache Airflow; industrial uses in finance, healthcare, e-commerce, manufacturing and smart cities; serverless data engineering, data mesh and data fabric.
Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then
- Data governance
- Data lineage and catalogues
- Workflow orchestration with Apache Airflow
- Data mesh, data fabric and serverless data engineering
Unit contents are summarised from the official DBATU syllabus . AiCanCode.org is not affiliated with Dr. Babasaheb Ambedkar Technological University (DBATU).