Data Science Big Data Technologies Syllabus

What is sharding (horizontal partitioning) in a distributed database?

Splitting a dataset across multiple machines by rows, so each shard holds a subset of the data. It enables horizontal scaling of storage and throughput.

Describe the two phases of the MapReduce programming model.

Map phase: processes input splits in parallel, emitting intermediate key-value pairs. Reduce phase: groups values by key (after a shuffle/sort) and aggregates them into final output.

What is the shuffle-and-sort step in MapReduce?

The intermediate phase between Map and Reduce that transfers map outputs to reducers, grouping and sorting all values by key so each reducer receives all values for its keys.

What is the core abstraction of Apache Spark, and why does Spark outperform MapReduce?

The RDD (Resilient Distributed Dataset), an immutable, partitioned, fault-tolerant collection. Spark is faster because it performs in-memory computation and lazy DAG-based execution rather than writing intermediate results to disk between stages.

What is the difference between a transformation and an action on a Spark RDD?

Transformations (e.g., map, filter) are lazy and build a lineage DAG without executing. Actions (e.g., count, collect, save) trigger execution of the DAG and return a result or write output.

What is the key difference between batch processing and stream processing?

Batch processing operates on large, bounded datasets at scheduled intervals (high latency, high throughput). Stream processing handles unbounded, continuous data in real time or near-real time (low latency, per-event/micro-batch).

What is the difference between event time and processing time in stream processing?

Event time is when the event actually occurred (timestamp in the data); processing time is when the system processes it. Differences arise from network delays and out-of-order/late-arriving events.

What is windowing in stream processing, and name three common window types.

Windowing groups unbounded streams into finite chunks for aggregation. Common types: tumbling (fixed, non-overlapping), sliding (fixed, overlapping), and session (gap-based, activity-driven) windows.

What is a watermark in stream processing?

A watermark is a heuristic timestamp marking progress in event time, signaling that events with timestamps earlier than the watermark are not expected anymore, so windows can be closed and emitted while handling late data.

What does "exactly-once" processing semantics mean, and how does it compare to at-least-once and at-most-once?

Exactly-once: each record affects state once, even with failures. At-least-once: records may be processed more than once (no loss, possible duplicates). At-most-once: records processed once or dropped (possible loss, no duplicates).

Why is in-memory processing faster than disk-based processing, and what is the trade-off?

RAM access is orders of magnitude faster than disk I/O, dramatically reducing latency for iterative and interactive workloads. The trade-off is higher cost per byte and limited capacity, plus volatility (data lost on failure unless persisted).

What is Apache Kafka and what role does it play in Big Data pipelines?

Kafka is a distributed, partitioned, replicated commit-log message broker used for high-throughput, fault-tolerant publish/subscribe streaming and data ingestion, decoupling producers from consumers.

In Kafka, what are topics, partitions, and consumer groups?

A topic is a named stream of records; it is split into partitions for parallelism and ordering within a partition. A consumer group lets multiple consumers share partitions so each partition is read by exactly one consumer in the group.

What is the difference between data ingestion tools like Apache Sqoop and Apache Flume?

Sqoop transfers bulk structured data between relational databases and Hadoop/HDFS (batch). Flume collects, aggregates, and moves large volumes of streaming log/event data into HDFS (streaming).

What does ETL stand for and what happens in each stage?

Extract (pull data from sources), Transform (clean, validate, aggregate, and reformat the data), and Load (write it into the target store such as a warehouse).

How does ELT differ from ETL, and when is ELT preferred?

In ELT the data is loaded raw into the target first and transformed inside it (using the target's compute). ELT is preferred with cloud data warehouses/lakes that have scalable compute and for handling large, varied raw data.

What is a fact table versus a dimension table in a star schema?

A fact table stores quantitative measures/metrics and foreign keys to dimensions (e.g., sales amount). Dimension tables store descriptive context attributes (e.g., product, date, customer) used to filter and group facts.

What is the difference between a star schema and a snowflake schema?

Star schema has denormalized dimension tables directly linked to a central fact table. Snowflake schema normalizes dimensions into multiple related sub-tables, reducing redundancy but adding join complexity.

What is Apache Hive and what query language does it use?

Hive is a data-warehouse layer on top of Hadoop that provides SQL-like querying over large datasets. It uses HiveQL (HQL), translating queries into MapReduce, Tez, or Spark jobs.

What is the role of Apache Pig in the Hadoop ecosystem?

Pig provides a high-level dataflow scripting language called Pig Latin for ETL and analysis of large datasets, compiling scripts into MapReduce jobs without writing low-level Java.

What is OLTP versus OLAP?

OLTP (Online Transaction Processing): many short read/write transactions, normalized, current operational data. OLAP (Online Analytical Processing): complex read-heavy analytical queries over historical, aggregated, denormalized data.

Name two leading data-visualization tools used with Big Data and one strength of each.

Tableau: powerful interactive dashboards and drag-and-drop exploration. Power BI: tight Microsoft ecosystem integration and affordability. (Others: Looker, Qlik, Apache Superset.)

What is the purpose of a dashboard in data visualization?

A dashboard consolidates and displays key metrics and KPIs in a single interactive view, enabling at-a-glance monitoring and data-driven decisions through charts, gauges, and tables.

Name three popular general-purpose machine learning frameworks/libraries.

scikit-learn (classical ML in Python), Apache Spark MLlib (distributed ML on Spark), and XGBoost (gradient-boosted trees). Others include H2O.ai and Mahout.

What distinguishes deep learning frameworks like TensorFlow and PyTorch from classical ML libraries?

They are built for constructing and training deep neural networks using automatic differentiation, GPU/TPU acceleration, and tensor computation, whereas classical ML libraries focus on traditional algorithms (trees, SVMs, regression).

What is the key architectural difference between TensorFlow (1.x) and PyTorch regarding computation graphs?

PyTorch uses dynamic (define-by-run) computation graphs built at runtime; original TensorFlow 1.x used static (define-then-run) graphs. (TensorFlow 2.x added eager execution to behave dynamically.)

What is a tensor in the context of deep learning frameworks?

A tensor is a multi-dimensional array (generalizing scalars, vectors, and matrices) that is the fundamental data structure for representing inputs, weights, and activations in neural networks.

In the context of Big Data and AI integration, what is a feature store?

A feature store is a centralized repository for storing, sharing, serving, and reusing curated ML features consistently across training and inference, ensuring train/serve consistency.

What are the three pillars of information security (the CIA triad) applied to data security?

Confidentiality (prevent unauthorized access), Integrity (prevent unauthorized modification), and Availability (ensure authorized access when needed).

What is the difference between data encryption at rest and in transit?

Encryption at rest protects stored data on disk/databases (e.g., AES-256). Encryption in transit protects data moving over networks (e.g., TLS/SSL), defending against interception.

What is the difference between data masking, anonymization, and pseudonymization?

Masking obscures data with realistic but fake values for non-prod use. Anonymization irreversibly removes identifying information. Pseudonymization replaces identifiers with reversible tokens that can be re-linked using a separate key.

What is data governance, and name three of its core components.

Data governance is the framework of policies, roles, and processes ensuring data is managed as an asset. Core components: data quality, data lineage, metadata management, stewardship, and compliance/security policies.

What is data lineage and why is it important in Big Data governance?

Data lineage tracks data's origin, movement, and transformations through the pipeline. It is vital for debugging, impact analysis, auditing, regulatory compliance, and establishing trust in data.

Give one example of a Big Data application in the healthcare industry and one in finance.

Healthcare: predictive analytics for patient outcomes/early disease detection and genomic analysis. Finance: real-time fraud detection and algorithmic risk modeling.

What is a recommendation engine, a common industry-specific Big Data application in retail and streaming?

A system that analyzes user behavior and item attributes (via collaborative filtering or content-based methods) to predict and suggest products or content a user is likely to want, increasing engagement and sales.