Data Science Big Data Technologies Flashcards

24 sample cards from the Big Data Technologies deck

Sampled from the end of the deck, so these are different cards from the ones shown on the syllabus page.

1. What is Apache Kafka and what role does it play in Big Data pipelines?

   Kafka is a distributed, partitioned, replicated commit-log message broker used for high-throughput, fault-tolerant publish/subscribe streaming and data ingestion, decoupling producers from consumers.

2. In Kafka, what are topics, partitions, and consumer groups?

   A topic is a named stream of records; it is split into partitions for parallelism and ordering within a partition. A consumer group lets multiple consumers share partitions so each partition is read by exactly one consumer in the group.

3. What is the difference between data ingestion tools like Apache Sqoop and Apache Flume?

   Sqoop transfers bulk structured data between relational databases and Hadoop/HDFS (batch). Flume collects, aggregates, and moves large volumes of streaming log/event data into HDFS (streaming).

4. What does ETL stand for and what happens in each stage?

   Extract (pull data from sources), Transform (clean, validate, aggregate, and reformat the data), and Load (write it into the target store such as a warehouse).

5. How does ELT differ from ETL, and when is ELT preferred?

   In ELT the data is loaded raw into the target first and transformed inside it (using the target's compute). ELT is preferred with cloud data warehouses/lakes that have scalable compute and for handling large, varied raw data.

6. What is a fact table versus a dimension table in a star schema?

   A fact table stores quantitative measures/metrics and foreign keys to dimensions (e.g., sales amount). Dimension tables store descriptive context attributes (e.g., product, date, customer) used to filter and group facts.

7. What is the difference between a star schema and a snowflake schema?

   Star schema has denormalized dimension tables directly linked to a central fact table. Snowflake schema normalizes dimensions into multiple related sub-tables, reducing redundancy but adding join complexity.

8. What is Apache Hive and what query language does it use?

   Hive is a data-warehouse layer on top of Hadoop that provides SQL-like querying over large datasets. It uses HiveQL (HQL), translating queries into MapReduce, Tez, or Spark jobs.

9. What is the role of Apache Pig in the Hadoop ecosystem?

   Pig provides a high-level dataflow scripting language called Pig Latin for ETL and analysis of large datasets, compiling scripts into MapReduce jobs without writing low-level Java.

10. What is OLTP versus OLAP?

    OLTP (Online Transaction Processing): many short read/write transactions, normalized, current operational data. OLAP (Online Analytical Processing): complex read-heavy analytical queries over historical, aggregated, denormalized data.

11. Name two leading data-visualization tools used with Big Data and one strength of each.

    Tableau: powerful interactive dashboards and drag-and-drop exploration. Power BI: tight Microsoft ecosystem integration and affordability. (Others: Looker, Qlik, Apache Superset.)

12. What is the purpose of a dashboard in data visualization?

    A dashboard consolidates and displays key metrics and KPIs in a single interactive view, enabling at-a-glance monitoring and data-driven decisions through charts, gauges, and tables.

13. Name three popular general-purpose machine learning frameworks/libraries.

    scikit-learn (classical ML in Python), Apache Spark MLlib (distributed ML on Spark), and XGBoost (gradient-boosted trees). Others include H2O.ai and Mahout.

14. What distinguishes deep learning frameworks like TensorFlow and PyTorch from classical ML libraries?

    They are built for constructing and training deep neural networks using automatic differentiation, GPU/TPU acceleration, and tensor computation, whereas classical ML libraries focus on traditional algorithms (trees, SVMs, regression).

15. What is the key architectural difference between TensorFlow (1.x) and PyTorch regarding computation graphs?

    PyTorch uses dynamic (define-by-run) computation graphs built at runtime; original TensorFlow 1.x used static (define-then-run) graphs. (TensorFlow 2.x added eager execution to behave dynamically.)

16. What is a tensor in the context of deep learning frameworks?

    A tensor is a multi-dimensional array (generalizing scalars, vectors, and matrices) that is the fundamental data structure for representing inputs, weights, and activations in neural networks.

17. In the context of Big Data and AI integration, what is a feature store?

    A feature store is a centralized repository for storing, sharing, serving, and reusing curated ML features consistently across training and inference, ensuring train/serve consistency.

18. What are the three pillars of information security (the CIA triad) applied to data security?

    Confidentiality (prevent unauthorized access), Integrity (prevent unauthorized modification), and Availability (ensure authorized access when needed).

19. What is the difference between data encryption at rest and in transit?

    Encryption at rest protects stored data on disk/databases (e.g., AES-256). Encryption in transit protects data moving over networks (e.g., TLS/SSL), defending against interception.

20. What is the difference between data masking, anonymization, and pseudonymization?

    Masking obscures data with realistic but fake values for non-prod use. Anonymization irreversibly removes identifying information. Pseudonymization replaces identifiers with reversible tokens that can be re-linked using a separate key.

21. What is data governance, and name three of its core components.

    Data governance is the framework of policies, roles, and processes ensuring data is managed as an asset. Core components: data quality, data lineage, metadata management, stewardship, and compliance/security policies.

22. What is data lineage and why is it important in Big Data governance?

    Data lineage tracks data's origin, movement, and transformations through the pipeline. It is vital for debugging, impact analysis, auditing, regulatory compliance, and establishing trust in data.

23. Give one example of a Big Data application in the healthcare industry and one in finance.

    Healthcare: predictive analytics for patient outcomes/early disease detection and genomic analysis. Finance: real-time fraud detection and algorithmic risk modeling.

24. What is a recommendation engine, a common industry-specific Big Data application in retail and streaming?

    A system that analyzes user behavior and item attributes (via collaborative filtering or content-based methods) to predict and suggest products or content a user is likely to want, increasing engagement and sales.

What this deck covers

Big Data Technologies flashcards FAQ