Data Science Data Wrangling and Exploration Syllabus

How are duplicate records typically identified and removed in a dataset?

Duplicates are identified by checking for rows with identical values across all columns or a chosen subset of key columns. They are removed by keeping the first or last occurrence (e.g., df.drop_duplicates(subset=..., keep='first')).

What is the difference between exact duplicates and fuzzy (near) duplicates?

Exact duplicates have identical values in compared fields. Fuzzy duplicates are records referring to the same entity but differing slightly (typos, formatting), requiring approximate matching like edit distance or similarity thresholds to detect.

What is the IQR method for outlier detection, including the formula for the fences?

Using quartiles $Q_1$ and $Q_3$ with $IQR = Q_3 - Q_1$, a point is an outlier if it lies below $Q_1 - 1.5 \times IQR$ or above $Q_3 + 1.5 \times IQR$.

What is the z-score method for outlier detection and the common threshold?

The z-score is $z = \frac{x - \mu}{\sigma}$. A point is commonly flagged as an outlier if $|z| > 3$. This assumes an approximately normal distribution.

What is 'winsorization' as an outlier treatment?

Winsorization caps extreme values at a specified percentile (e.g., values above the 95th percentile are set to the 95th-percentile value, and below the 5th to the 5th-percentile value). It limits outlier influence without deleting rows.

Give the formula for min-max normalization and its output range.

$$x' = \frac{x - x_{min}}{x_{max} - x_{min}}$$ It rescales values to the range $[0, 1]$ (or any chosen $[a,b]$ via scaling).

Give the formula for z-score standardization and the resulting distribution properties.

$$x' = \frac{x - \mu}{\sigma}$$ The standardized feature has mean $0$ and standard deviation $1$. It does not bound values to a fixed range.

When should you prefer standardization over min-max normalization?

Prefer standardization when the data is roughly Gaussian or when algorithms assume zero-centered data (e.g., PCA, SVM, logistic/linear regression with regularization). Min-max is preferred when you need a bounded range and the distribution is non-Gaussian (e.g., image pixels, some neural nets).

What is robust scaling and when is it useful?

Robust scaling uses the median and IQR: $$x' = \frac{x - \text{median}}{IQR}$$ It is useful when the data contains outliers, since median and IQR are robust to extreme values unlike mean and standard deviation.

What is feature engineering?

Feature engineering is the process of creating, transforming, or selecting input variables (features) from raw data to improve model performance — e.g., deriving ratios, polynomial terms, interaction terms, binning, or domain-specific aggregates.

What is binning (discretization) and name two strategies?

Binning converts a continuous variable into discrete intervals/categories. Two strategies: equal-width binning (intervals of equal range) and equal-frequency (quantile) binning (each bin holds roughly the same number of observations).

What is a log transformation used for in feature engineering, and what is a caveat?

A log transform, $x' = \log(x)$, compresses right-skewed data and reduces the effect of large values, often making distributions more symmetric. Caveat: it is undefined for $x \leq 0$, so a shift like $\log(x+1)$ (log1p) is used for zeros.

What is data aggregation, and give an example with a SQL clause.

Aggregation summarizes multiple rows into single values using functions (SUM, AVG, COUNT, MIN, MAX) over groups. Example: SELECT region, SUM(sales) FROM orders GROUP BY region — collapses rows per region into total sales.

What does a GROUP BY operation conceptually do, and what is the role of HAVING vs WHERE?

GROUP BY partitions rows into groups by key columns so aggregates compute per group. WHERE filters rows before grouping; HAVING filters groups after aggregation (e.g., HAVING SUM(sales) > 1000).

What is a pivot (or crosstab) operation in data aggregation?

A pivot reshapes data from long to wide format, turning unique values of one column into new columns and aggregating a value column at each row-column intersection — e.g., months as columns, products as rows, with summed sales in cells.

What is one-hot encoding and what dimensionality does it produce for a categorical feature with k levels?

One-hot encoding represents each category as a binary indicator column (1 if present, else 0). A feature with $k$ categories produces $k$ binary columns (or $k-1$ if one is dropped to avoid the dummy-variable trap).

What is label encoding and a key risk when using it with non-tree models?

Label encoding maps each category to an integer (e.g., red=0, green=1, blue=2). Risk: it imposes an artificial ordinal/numeric relationship, so distance- or linear-based models may wrongly infer blue > green > red. It is safe for tree-based models.

What is the 'dummy variable trap' and how is it avoided?

The dummy variable trap is perfect multicollinearity that occurs when one-hot columns are linearly dependent (they sum to 1). It is avoided by dropping one category column (using $k-1$ dummies), so no column is a linear combination of the others.

Contrast one-hot encoding with target (mean) encoding.

One-hot creates a binary column per category, increasing dimensionality. Target encoding replaces each category with a statistic of the target (e.g., mean target value for that category), keeping a single column but risking target leakage and overfitting; it needs cross-validation/smoothing.

Why is parsing strings into a proper datetime type important, and name a common pitfall.

Datetime types enable correct sorting, arithmetic (differences, offsets), and extraction of components. A common pitfall is ambiguous formats like 03/04/2025 (DD/MM vs MM/DD), so an explicit format/locale must be specified to avoid silent misparsing.

What is the difference between a timezone-naive and timezone-aware timestamp?

A naive timestamp has no timezone information and is ambiguous about the absolute instant. An aware timestamp carries timezone/UTC-offset info, allowing correct conversion between zones and unambiguous ordering across regions.

What are common datetime feature extractions for modeling?

From a timestamp you can derive year, month, day, day-of-week, hour, quarter, is_weekend, is_holiday, and elapsed time since an event. Cyclical features (hour, month) are often encoded with sine/cosine: $\sin\!\left(\frac{2\pi t}{T}\right)$ and $\cos\!\left(\frac{2\pi t}{T}\right)$.

What is tokenization in text data processing?

Tokenization splits raw text into smaller units (tokens) — typically words, subwords, or characters. It is the first step that turns a string into a sequence of discrete elements for further processing or vectorization.

Differentiate stemming from lemmatization.

Stemming crudely chops word endings to a root form (e.g., 'studies' -> 'studi') using rules, possibly producing non-words. Lemmatization maps a word to its dictionary base form (lemma) using vocabulary and part-of-speech (e.g., 'studies' -> 'study'), which is more accurate but slower.

What is the TF-IDF weighting scheme and its formula?

TF-IDF weights a term by its frequency in a document against its rarity across the corpus: $$\text{tfidf}(t,d) = tf(t,d) \times \log\frac{N}{df(t)}$$ where $N$ is the number of documents and $df(t)$ the number containing term $t$. Common words get low weight; distinctive words get high weight.

What are stop words and why are they often removed?

Stop words are extremely common words (the, is, and, of) that carry little discriminative meaning. They are often removed to reduce dimensionality and noise, though they can be retained when word order/context matters (e.g., for some deep models).

What is the difference between concatenation and merging/joining of datasets?

Concatenation stacks datasets along an axis (rows or columns) by position without matching keys. Merging/joining combines datasets by matching values in key columns, aligning rows based on relationships rather than position.

Compare INNER, LEFT, RIGHT, and FULL OUTER joins.

INNER: only rows with matching keys in both tables. LEFT: all rows from the left table plus matches from the right (nulls where no match). RIGHT: all rows from the right plus matches from the left. FULL OUTER: all rows from both, matched where possible, nulls elsewhere.

What is a cross join (Cartesian product) and how many rows does it produce?

A cross join pairs every row of one table with every row of the other, with no join condition. For tables of $m$ and $n$ rows it produces $m \times n$ rows.

What is a schema mismatch when combining datasets, and name two common causes and a fix.

A schema mismatch occurs when datasets to be combined have incompatible structures. Common causes: differing column names for the same concept (e.g., 'DOB' vs 'birth_date') and differing data types/units. Fixes: rename/map columns to a common schema and cast types/units to align before merging.