Data Science Data Wrangling and Exploration Syllabus

Data Wrangling and Exploration syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Data Wrangling and Exploration in Data Science, not a summary of it.

1. 1

   Data Collection

   4 topics

   • APIs and Web Scraping
   • Database Extraction
   • File Formats (CSV, JSON, XML)
   • Data Warehouses and Lakes
2. 2

   Data Cleaning

   5 topics

   • Handling Missing Data
   • Data Type Conversion
   • Removing Duplicates
   • Outlier Detection and Treatment
   • Data Normalization and Standardization
3. 3

   Data Transformation

   5 topics

   • Feature Engineering
   • Data Aggregation
   • Data Encoding (One-Hot, Label Encoding)
   • Date and Time Manipulation
   • Text Data Processing
4. 4

   Data Integration

   4 topics

   • Merging Datasets
   • Joining Data
   • Concatenation
   • Handling Schema Mismatches
5. 5

   Data Exploration

   5 topics

   • Descriptive Statistics
   • Data Visualization
   • Correlation Analysis
   • Hypothesis Testing
   • Exploratory Data Analysis (EDA)
6. 6

   Tools and Libraries

   5 topics

   • Pandas
   • NumPy
   • Matplotlib
   • Seaborn
   • Scikit-learn
7. 7

   Best Practices

   5 topics

   • Version Control for Data
   • Reproducibility
   • Documentation
   • Data Security and Privacy
   • Performance Optimization

Data Wrangling and Exploration flashcards for Data Science

23 of 50 cards from the Data Wrangling and Exploration deck — real questions with worked answers.

1. What is the primary purpose of a REST API in data collection?

   A REST (Representational State Transfer) API exposes data over HTTP using standard methods (GET, POST, PUT, DELETE) and returns structured responses (usually JSON), letting a program request specific data resources via URLs/endpoints without scraping HTML.

2. In web scraping, what is the difference between an HTML parser and a headless browser?

   An HTML parser (e.g., BeautifulSoup, lxml) reads static HTML returned by an HTTP request. A headless browser (e.g., Selenium, Playwright) renders the page including JavaScript-generated content, so it is needed for dynamic sites where data loads after the initial HTML.

3. What HTTP status code indicates a successful request, and which indicates rate limiting / too many requests?

   $200$ indicates success (OK). $429$ indicates 'Too Many Requests' (rate limiting). Other common ones: $404$ Not Found, $401$ Unauthorized, $403$ Forbidden, $500$ Server Error.

4. What file is used to declare scraping permissions for a website, and what is its role?

   robots.txt, located at the site root, specifies which paths crawlers are allowed or disallowed from accessing. It is a voluntary standard guiding ethical scraping but is not technically enforced.

5. What is pagination in the context of API extraction, and name two common schemes?

   Pagination splits large result sets into pages. Two common schemes: offset/limit (e.g., ?offset=100&limit=50) and cursor-based (a token pointing to the next page). Cursor-based is more stable for changing data.

6. What SQL clause set is the minimal pattern to extract a filtered subset of columns from a database table?

   SELECT (columns) FROM (table) WHERE (condition). Example: SELECT name, age FROM users WHERE age >= 18.

7. What is the difference between OLTP and OLAP database systems?

   OLTP (Online Transaction Processing) handles many short read/write transactions for operational apps. OLAP (Online Analytical Processing) is optimized for complex read-heavy analytical queries over large historical data, often with denormalized/star schemas.

8. What is a database index and what trade-off does it introduce?

   An index is a data structure (often a B-tree) that speeds up row lookup on indexed columns, reducing query time from $O(n)$ scans toward $O(\log n)$. Trade-off: it consumes extra storage and slows down inserts/updates because the index must be maintained.

9. Compare CSV, JSON, and XML formats in terms of structure.

   CSV: flat, tabular, comma-separated, no nesting. JSON: hierarchical key-value pairs and arrays, lightweight, native to web APIs. XML: hierarchical with tags and attributes, verbose, supports schemas (XSD) and namespaces.

10. Why can parsing CSV be error-prone, and how is it handled?

    Fields may themselves contain the delimiter (comma) or newlines. This is handled by quoting fields (e.g., "New York, NY") and escaping embedded quotes by doubling them. A proper CSV parser respects quoting rather than naive splitting on commas.

11. What is the key structural difference between a JSON object and a JSON array?

    A JSON object is an unordered set of key-value pairs enclosed in braces {}, e.g., {"a":1}. A JSON array is an ordered list of values enclosed in brackets [], e.g., [1,2,3]. Values can nest arbitrarily.

12. What distinguishes a Data Warehouse from a Data Lake?

    A Data Warehouse stores structured, processed data with a predefined schema (schema-on-write) for BI/reporting. A Data Lake stores raw data in any format (structured, semi-structured, unstructured) with schema-on-read, offering flexibility but requiring more processing at query time.

13. What is the ETL process and how does ELT differ?

    ETL = Extract, Transform, Load: data is transformed before loading into the warehouse. ELT = Extract, Load, Transform: raw data is loaded first (often into a data lake/cloud warehouse) and transformed afterward, leveraging the destination's compute power.

14. What is a 'data lakehouse'?

    A data lakehouse is a hybrid architecture combining the low-cost, flexible raw storage of a data lake with the structure, schema enforcement, and ACID transaction management of a data warehouse (e.g., Delta Lake, Apache Iceberg).

15. List the main mechanisms (missingness types) for missing data: MCAR, MAR, MNAR.

    MCAR (Missing Completely At Random): missingness unrelated to any data. MAR (Missing At Random): missingness depends on observed variables. MNAR (Missing Not At Random): missingness depends on the unobserved value itself. The type determines valid imputation strategies.

16. What is mean/median imputation, and when is median preferred?

    It replaces missing numeric values with the column's mean or median. Median is preferred when the variable is skewed or has outliers, because the median is robust to extreme values whereas the mean is pulled toward them.

17. What is listwise deletion and its main drawback?

    Listwise deletion (complete-case analysis) removes any row containing a missing value. Drawback: it can discard large amounts of data and introduce bias if the data is not MCAR, reducing statistical power.

18. What is KNN imputation?

    KNN imputation fills a missing value using the values of the $k$ nearest neighbor rows (by distance over the other features), typically averaging the neighbors' values for that feature. It captures relationships between features but is computationally expensive on large data.

19. Why is correct data type conversion (e.g., string to numeric or datetime) important before analysis?

    Operations behave differently by type: arithmetic and aggregation require numeric types, sorting differs for strings vs numbers, and time-based operations require datetime types. Wrong types cause errors or silently incorrect results (e.g., '10' < '9' as strings).

20. What is the difference between explicit (downcasting) and implicit type coercion?

    Explicit conversion is programmer-directed casting (e.g., int(x), astype(float)). Implicit coercion is automatic type promotion by the language/library during mixed operations (e.g., int + float -> float). Implicit coercion can cause surprises like upcasting to object/string.

21. How are duplicate records typically identified and removed in a dataset?

    Duplicates are identified by checking for rows with identical values across all columns or a chosen subset of key columns. They are removed by keeping the first or last occurrence (e.g., df.drop_duplicates(subset=..., keep='first')).

22. What is the difference between exact duplicates and fuzzy (near) duplicates?

    Exact duplicates have identical values in compared fields. Fuzzy duplicates are records referring to the same entity but differing slightly (typos, formatting), requiring approximate matching like edit distance or similarity thresholds to detect.

23. What is the IQR method for outlier detection, including the formula for the fences?

    Using quartiles $Q_1$ and $Q_3$ with $IQR = Q_3 - Q_1$, a point is an outlier if it lies below $Q_1 - 1.5 \times IQR$ or above $Q_3 + 1.5 \times IQR$.

See more Data Wrangling and Exploration flashcards →

Planning Data Wrangling and Exploration for Data Science

Data Wrangling and Exploration (Data Science) FAQ