Data Science Big Data Technologies Syllabus

Big Data Technologies syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Big Data Technologies in Data Science, not a summary of it.

1. 1

   Big Data Fundamentals

   2 topics

   • Introduction to Big Data
     • Definition and Characteristics
     • Importance and Applications
     • Big Data vs Traditional Data
   • Big Data Architecture
     • Components of Big Data Architecture
     • Data Storage and Processing
     • Data Ingestion and Extraction
2. 2

   Data Storage Technologies

   3 topics

   • Distributed File Systems
     • Hadoop Distributed File System (HDFS)
     • Google File System (GFS)
     • Amazon S3
   • NoSQL Databases
     • Introduction to NoSQL
     • Types of NoSQL Databases
     • Key-Value Stores
     • Document Stores
     • Column-Family Stores
     • Graph Databases
   • Relational Databases
     • SQL Basics
     • Scalability and Performance
     • NewSQL Databases
3. 3

   Data Processing Frameworks

   3 topics

   • Batch Processing
     • Hadoop MapReduce
     • Apache Pig
     • Apache Hive
   • Stream Processing
     • Apache Kafka
     • Apache Storm
     • Apache Flink
     • Apache Samza
   • In-Memory Processing
     • Apache Spark
     • Apache Ignite
4. 4

   Data Ingestion and ETL

   2 topics

   • Data Ingestion Tools
     • Apache Sqoop
     • Apache Flume
     • Kafka Connect
   • ETL Processes
     • Extract, Transform, Load (ETL)
     • Data Cleaning and Preprocessing
     • ETL Tools
5. 5

   Data Analytics and Visualization

   2 topics

   • Data Analytics Tools
     • Apache Drill
     • Apache Impala
     • Presto
   • Data Visualization Tools
     • Tableau
     • Power BI
     • D3.js
6. 6

   Machine Learning and AI in Big Data

   3 topics

   • Machine Learning Frameworks
     • Apache Mahout
     • H2O.ai
     • TensorFlow
   • Deep Learning Frameworks
     • Keras
     • PyTorch
     • MXNet
   • Big Data and AI Integration
     • Scalable Machine Learning
     • Real-Time AI Applications
     • AI in Data Lakes
7. 7

   Big Data Security and Governance

   2 topics

   • Data Security
     • Encryption and Decryption
     • Access Control
     • Data Masking and Tokenization
   • Data Governance
     • Data Quality Management
     • Metadata Management
     • Regulatory Compliance
8. 8

   Big Data Use Cases and Applications

   2 topics

   • Industry-Specific Applications
     • Healthcare
     • Finance
     • Retail
     • Telecommunications
   • Emerging Trends
     • IoT and Big Data
     • Edge Computing
     • Blockchain and Big Data

Big Data Technologies flashcards for Data Science

21 of 55 cards from the Big Data Technologies deck — real questions with worked answers.

1. What are the classic "3 Vs" that define Big Data, and what does each mean?

   Volume (the sheer amount of data), Velocity (the speed at which data is generated and processed), and Variety (the range of data types and sources: structured, semi-structured, unstructured).

2. Two additional "Vs" are often added to extend the Big Data model beyond the original three. What are they?

   Veracity (the trustworthiness/quality/uncertainty of the data) and Value (the usefulness or business worth that can be extracted from the data).

3. How does Big Data differ from traditional data in terms of storage and processing approach?

   Traditional data fits on a single machine with vertical scaling (scale-up); Big Data uses distributed storage and processing across clusters of commodity hardware with horizontal scaling (scale-out).

4. What is the difference between structured, semi-structured, and unstructured data?

   Structured: fixed schema in rows/columns (e.g., relational tables). Semi-structured: self-describing tags but no rigid schema (e.g., JSON, XML). Unstructured: no predefined model (e.g., text, images, video, audio).

5. What are the main layers of a typical Big Data architecture (Lambda-style)?

   Data sources/ingestion, storage (data lake/warehouse), processing (batch + stream), serving/analytics, and visualization/consumption layers, plus cross-cutting security and governance.

6. In Lambda Architecture, what are the three layers and their roles?

   Batch layer (computes accurate views over the full dataset), Speed layer (computes real-time incremental views with low latency), and Serving layer (merges batch and speed views to answer queries).

7. How does Kappa Architecture differ from Lambda Architecture?

   Kappa removes the separate batch layer and treats all data as a single immutable stream, reprocessing historical data by replaying the stream. This avoids maintaining two separate codebases (batch and speed).

8. What is a data lake, and how does it differ from a data warehouse?

   A data lake stores raw data in its native format with schema-on-read (flexible, cheap, all data types). A data warehouse stores cleaned, structured data with schema-on-write, optimized for fast SQL analytics on curated data.

9. What is schema-on-read versus schema-on-write?

   Schema-on-write applies and validates the schema when data is loaded (data warehouses). Schema-on-read stores raw data and applies a schema only when the data is queried (data lakes).

10. What is HDFS and what are its two primary node types?

    HDFS (Hadoop Distributed File System) is a distributed file system for storing large files across a cluster. NameNode manages metadata and the namespace; DataNodes store the actual data blocks.

11. What is the default HDFS block size and default replication factor?

    The default block size is 128 MB (configurable; 64 MB in older versions), and the default replication factor is 3 copies of each block.

12. With an HDFS replication factor of 3, how much raw storage does a 1 TB logical dataset consume?

    $1\ \text{TB} \times 3 = 3\ \text{TB}$ of raw storage, because each block is replicated three times across DataNodes.

13. What design principle motivates moving computation to the data in HDFS/Hadoop rather than moving data to computation?

    Data locality: moving the (small) computation code to the node where the (large) data block resides is far cheaper than transferring massive data across the network, reducing network congestion.

14. What guarantees do the CAP theorem state a distributed data store can simultaneously provide?

    Only two of three: Consistency, Availability, and Partition tolerance. Since network partitions are unavoidable, real systems trade off between consistency (CP) and availability (AP).

15. What are the four main categories of NoSQL databases?

    Key-value stores (e.g., Redis, DynamoDB), document stores (e.g., MongoDB), column-family/wide-column stores (e.g., Cassandra, HBase), and graph databases (e.g., Neo4j).

16. What does BASE stand for, and how does it contrast with ACID?

    BASE = Basically Available, Soft state, Eventually consistent (favored by NoSQL for scalability). ACID = Atomicity, Consistency, Isolation, Durability (favored by relational DBs for strong consistency).

17. What do the four ACID properties guarantee in a relational transaction?

    Atomicity (all-or-nothing), Consistency (valid state to valid state), Isolation (concurrent transactions don't interfere), and Durability (committed changes survive failures).

18. What type of NoSQL database is best suited for storing and querying highly connected data such as social networks?

    A graph database (e.g., Neo4j), which stores nodes and edges with properties and efficiently traverses relationships.

19. What is normalization in relational databases and what problem does it solve?

    Normalization organizes tables to reduce redundancy and dependency by splitting data into related tables (e.g., to 1NF, 2NF, 3NF), preventing insertion, update, and deletion anomalies.

20. What is denormalization and why is it sometimes used in Big Data systems?

    Denormalization deliberately adds redundancy by combining tables to avoid costly joins, improving read/query performance at the expense of storage and update complexity. Common in NoSQL/analytics workloads.

21. What is sharding (horizontal partitioning) in a distributed database?

    Splitting a dataset across multiple machines by rows, so each shard holds a subset of the data. It enables horizontal scaling of storage and throughput.

See more Big Data Technologies flashcards →

Planning Big Data Technologies for Data Science

Big Data Technologies (Data Science) FAQ