DBATU TY AI syllabus

DBATU · Third Year B.Tech Artificial Intelligence & Allied · NEP 2020

Data Engineering and Big Data Systems

26AFAIPC504 · Semester V · 3 credits · Continuous assessment 20 + mid-semester 20 + end-semester 60

All notes for this subject
  1. Unit 1: Fundamentals of Data Engineering and Big Data

    5 hours

    The evolution of data processing and the data engineering lifecycle; the five Vs of big data; data sources; batch and stream processing; big data architecture; data engineering roles and the modern ecosystem; distributed file systems.

    Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then

    • Data engineering lifecycle
    • Characteristics of big data
    • Batch and real-time processing
    • Big data architecture
  2. Unit 2: Distributed Storage and Processing Frameworks

    6 hours

    HDFS architecture, NameNode and DataNode, replication and fault tolerance; the Hadoop ecosystem; MapReduce with mappers, reducers, combiners and partitioning; Apache Spark, RDDs, DataFrames and Spark SQL; MapReduce versus Spark.

    Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then

    • HDFS architecture
    • Replication and fault tolerance
    • Hadoop ecosystem
    • MapReduce programming model
    • RDD transformations and actions
    • DataFrames and Spark SQL
    • MapReduce versus Spark
  3. Unit 3: Data Warehousing and Data Integration

    6 hours

    Operational databases versus warehouses; warehouse architecture, fact and dimension tables, star and snowflake schemas and data marts; OLAP; ETL and ELT; data integration, quality and metadata; data lakes and lakehouses.

    Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then

    • Data warehouse architecture
    • Star and snowflake schemas
    • OLAP operations
    • ETL and ELT
    • Data lakes and lakehouses
  4. Unit 4: Real-Time Data Engineering

    5 hours

    Streaming data and event-driven architecture; publish–subscribe and message brokers; Apache Kafka producers, consumers and topics; stream and window-based processing; streaming pipelines, log processing and monitoring pipeline performance.

    Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then

    • Event-driven architecture and publish-subscribe
    • Apache Kafka architecture
    • Stream processing and windowing
    • Data pipeline architecture
  5. Unit 5: Data Governance, Security and Applications

    5 hours

    Data governance, ownership and stewardship; privacy, compliance and security; backup and recovery; lineage and catalogues; workflow orchestration with Apache Airflow; industrial uses in finance, healthcare, e-commerce, manufacturing and smart cities; serverless data engineering, data mesh and data fabric.

    Examined in the 60-mark end-semester paper, and in the mid-semester exam if it is taught by then

    • Data governance
    • Data lineage and catalogues
    • Workflow orchestration with Apache Airflow
    • Data mesh, data fabric and serverless data engineering

Unit contents are summarised from the official DBATU syllabus . AiCanCode.org is not affiliated with Dr. Babasaheb Ambedkar Technological University (DBATU).