Data Analytics Big Data and Modern Data Infrastructure Syllabus

Big Data and Modern Data Infrastructure syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Big Data and Modern Data Infrastructure in Data Analytics, not a summary of it.

1. 1

   Big Data Concepts

   4 topics

   • The Vs of Big Data
   • Distributed Computing
   • Hadoop Ecosystem
   • Apache Spark
2. 2

   Data Warehousing and ETL

   4 topics

   • Data Warehouse vs Data Lake
   • ETL vs ELT Pipelines
   • Dimensional Modeling
   • Cloud Data Platforms
3. 3

   Business Intelligence

   4 topics

   • BI Architecture and Tools
   • OLAP and Cubes
   • Self-Service Analytics
   • Reporting and Automation
4. 4

   Applied Analytics and Capstone

   5 topics

   • Framing Business Problems
   • A/B Testing and Experimentation
   • Domain Analytics
   • End-to-End Capstone Project
   • Presenting to Stakeholders

Big Data and Modern Data Infrastructure flashcards for Data Analytics

21 of 51 cards from the Big Data and Modern Data Infrastructure deck — real questions with worked answers.

1. What are the classic three Vs of Big Data, and what does each mean?

   Volume (the sheer amount of data), Velocity (the speed at which data is generated and processed), and Variety (the diversity of data types: structured, semi-structured, unstructured).

2. Beyond the original three Vs, name two additional Vs often added to characterize Big Data and define each.

   Veracity (the trustworthiness/quality and uncertainty of data) and Value (the usefulness or business worth extracted from the data). Variability (inconsistency of data flow/meaning) is also sometimes added.

3. In Big Data terminology, what distinguishes structured, semi-structured, and unstructured data?

   Structured data fits a fixed schema (relational tables); semi-structured data has tags/markers but no rigid schema (JSON, XML); unstructured data has no predefined model (text, images, video, audio).

4. What is distributed computing, and why is it central to Big Data?

   Distributed computing splits work across a cluster of many networked machines that operate in parallel. It is essential for Big Data because a single machine cannot store or process petabyte-scale data within acceptable time or cost.

5. Contrast horizontal scaling (scaling out) with vertical scaling (scaling up).

   Horizontal scaling adds more machines/nodes to a cluster to share load; vertical scaling adds more resources (CPU, RAM, disk) to a single machine. Big Data systems favor horizontal scaling for cost-effective near-linear growth.

6. State the CAP theorem and its core trade-off.

   The CAP theorem states a distributed data store can guarantee at most two of three properties simultaneously: Consistency, Availability, and Partition tolerance. Since network partitions are unavoidable, systems effectively trade off consistency versus availability during a partition.

7. What are the two core components of Apache Hadoop's original architecture?

   HDFS (Hadoop Distributed File System) for distributed storage, and MapReduce for distributed batch processing. YARN was later added as the resource/cluster manager.

8. In HDFS, what are the roles of the NameNode and DataNodes?

   The NameNode is the master that stores filesystem metadata (namespace, block locations); DataNodes are workers that store the actual data blocks and serve read/write requests. Blocks are replicated (default 3x) across DataNodes for fault tolerance.

9. Describe the two phases of the MapReduce programming model.

   The Map phase processes input splits in parallel, emitting intermediate key-value pairs; the Reduce phase groups those pairs by key (after a shuffle-and-sort) and aggregates them into final output. It is a batch, disk-based model.

10. What is YARN's function in the Hadoop ecosystem?

    YARN (Yet Another Resource Negotiator) is the cluster resource manager that schedules jobs and allocates CPU/memory across the cluster, decoupling resource management from processing engines so frameworks beyond MapReduce (e.g., Spark) can run on Hadoop.

11. Name three tools in the Hadoop ecosystem beyond HDFS/MapReduce and their purpose.

    Hive (SQL-like queries over HDFS via HiveQL), Pig (dataflow scripting), HBase (NoSQL wide-column store), Sqoop (RDBMS import/export), Flume/Kafka (data ingestion), Oozie (workflow scheduling), ZooKeeper (coordination).

12. What is Apache Spark, and what is its key performance advantage over MapReduce?

    Spark is a distributed, general-purpose in-memory data processing engine. Its key advantage is caching intermediate data in RAM (rather than writing to disk between stages), making iterative and interactive workloads dramatically faster than disk-based MapReduce.

13. What is a Spark RDD, and what are its two operation types?

    An RDD (Resilient Distributed Dataset) is Spark's fundamental immutable, partitioned, fault-tolerant collection. Its operations are transformations (lazy, e.g., map, filter, returning a new RDD) and actions (eager, e.g., count, collect, that trigger execution).

14. In Spark, what does 'lazy evaluation' mean and why is it beneficial?

    Transformations are not executed when defined; Spark builds a DAG (lineage) of operations and only computes when an action is called. This lets the Catalyst optimizer reorder/combine steps for efficiency and enables fault recovery via lineage.

15. Name the four main libraries in the Apache Spark ecosystem.

    Spark SQL (structured data/DataFrames), Spark Streaming / Structured Streaming (real-time), MLlib (machine learning), and GraphX (graph processing).

16. How do Spark DataFrames differ from RDDs?

    DataFrames are distributed collections organized into named columns (like a table) with a schema, enabling the Catalyst optimizer and Tungsten execution engine to optimize queries. RDDs are lower-level, untyped collections without automatic query optimization.

17. Define a data warehouse and its primary use.

    A data warehouse is a centralized repository of integrated, cleaned, structured data organized for analytical querying and reporting (OLAP). It uses schema-on-write and stores curated, historical data optimized for business intelligence.

18. Define a data lake and how it differs from a data warehouse.

    A data lake stores vast amounts of raw data in native format (structured, semi-structured, unstructured) using schema-on-read. Unlike a warehouse (schema-on-write, structured, curated), a lake is cheaper, more flexible, and used for exploration, ML, and diverse data types.

19. Contrast schema-on-write with schema-on-read.

    Schema-on-write (data warehouses) enforces structure when data is loaded, ensuring quality up front. Schema-on-read (data lakes) applies structure only when data is queried, offering flexibility but requiring processing/governance at read time.

20. What is a 'data lakehouse'?

    A data lakehouse is an architecture that combines the low-cost, flexible raw storage of a data lake with the management, ACID transactions, and performance features of a data warehouse (e.g., via Delta Lake, Apache Iceberg, or Hudi table formats).

21. What is a 'data mart'?

    A data mart is a subset of a data warehouse focused on a specific business line or department (e.g., sales, finance). It contains a curated slice of data tailored to that group's analytical needs.

See more Big Data and Modern Data Infrastructure flashcards →

Planning Big Data and Modern Data Infrastructure for Data Analytics

Big Data and Modern Data Infrastructure (Data Analytics) FAQ