Data Analytics Predictive Analytics and Modeling Syllabus

Define overfitting and its symptom in terms of errors.

Overfitting is when a model learns noise/idiosyncrasies of the training data, giving low training error but high validation/test error (poor generalization).

Define underfitting and its symptom in terms of errors.

Underfitting is when a model is too simple to capture the underlying pattern, giving high error on both the training and validation/test sets.

Name three techniques to reduce overfitting.

Regularization (L1/L2), gathering more training data, reducing model complexity, early stopping, cross-validation, and (in trees) pruning or ensembling.

Write the bias-variance decomposition of expected prediction error.

$$\mathbb{E}\!\left[(y-\hat{f}(x))^{2}\right] = \underbrace{\left(\text{Bias}[\hat{f}(x)]\right)^{2}}_{\text{bias}^2} + \underbrace{\text{Var}[\hat{f}(x)]}_{\text{variance}} + \sigma^{2}$$ where $\sigma^{2}$ is the irreducible error.

How do bias and variance relate to model complexity?

As complexity increases, bias decreases but variance increases. High bias corresponds to underfitting; high variance corresponds to overfitting. The goal is the sweet spot minimizing total error.

What is the irreducible error in the bias-variance tradeoff?

The $\sigma^{2}$ term: noise inherent in the data-generating process that no model can eliminate, regardless of how good it is.

How does a decision tree decide where to split a node (for classification)?

It chooses the feature and threshold that maximize purity gain, measured by reducing impurity such as Gini impurity $$G = 1 - \sum_{k}p_{k}^{2}$$ or entropy $$H = -\sum_{k}p_{k}\log_{2}p_{k}.$$

What is a random forest and how does it reduce variance?

An ensemble of many decision trees trained on bootstrap samples (bagging) with a random subset of features considered at each split. Averaging/voting across decorrelated trees reduces variance without greatly increasing bias.

What is bagging (bootstrap aggregating)?

Training multiple models on different bootstrap resamples of the data and aggregating their predictions (averaging for regression, majority vote for classification) to reduce variance.

How does the KNN algorithm classify a new point?

It finds the $k$ nearest training points by a distance metric (e.g., Euclidean) and assigns the majority class among those neighbors (or the average value for regression).

Write the Euclidean distance between two points $\vec{x}$ and $\vec{y}$ in $\mathbb{R}^{n}$.

$$d(\vec{x},\vec{y}) = \sqrt{\sum_{i=1}^{n}\left(x_{i}-y_{i}\right)^{2}}.$$

How does the choice of $k$ affect the bias-variance tradeoff in KNN?

Small $k$ gives low bias but high variance (sensitive to noise, overfitting); large $k$ gives smoother boundaries with higher bias but lower variance. $k$ is typically chosen by cross-validation.

Why is feature scaling important for KNN and K-Means?

Both rely on distance metrics, so features with larger numeric ranges dominate the distance. Standardizing/normalizing features ensures each contributes fairly.

What objective does K-Means clustering minimize?

The within-cluster sum of squares (inertia): $$J = \sum_{k=1}^{K}\sum_{x \in C_{k}}\lVert x - \mu_{k}\rVert^{2}$$ where $\mu_{k}$ is the centroid of cluster $C_{k}$.

Describe the two alternating steps of the K-Means algorithm.

Assignment step: assign each point to its nearest centroid. Update step: recompute each centroid as the mean of the points assigned to it. Repeat until assignments no longer change (convergence).

What is the elbow method for choosing $K$ in K-Means?

Plot within-cluster sum of squares (inertia) against $K$; the 'elbow' where the rate of decrease sharply flattens suggests a good number of clusters $K$.

What is the goal of Principal Component Analysis (PCA)?

To reduce dimensionality by projecting data onto a smaller set of orthogonal axes (principal components) that capture the maximum possible variance in the data.

What do the eigenvectors and eigenvalues of the covariance matrix represent in PCA?

Eigenvectors give the directions of the principal components; their eigenvalues give the amount of variance explained along each component. Components are ordered by decreasing eigenvalue.

How is the proportion of variance explained by the first $m$ principal components computed?

$$\text{PVE} = \frac{\sum_{i=1}^{m}\lambda_{i}}{\sum_{j=1}^{p}\lambda_{j}}$$ where $\lambda_{i}$ are the eigenvalues (sorted in decreasing order).

Why should data be standardized before applying PCA?

PCA is variance-based, so features measured on larger scales would dominate the components. Standardizing to zero mean and unit variance ensures each feature contributes comparably.

Write the formula for Mean Squared Error (MSE) and Root Mean Squared Error (RMSE).

$$\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}\left(y_{i}-\hat{y}_{i}\right)^{2}, \qquad \text{RMSE} = \sqrt{\text{MSE}}.$$

Write the formula for Mean Absolute Error (MAE) and contrast it with MSE.

$$\text{MAE} = \frac{1}{n}\sum_{i=1}^{n}\lvert y_{i}-\hat{y}_{i}\rvert.$$ MAE weights all errors linearly and is robust to outliers, whereas MSE squares errors and thus penalizes large errors more heavily.

What does the coefficient of determination $R^{2}$ measure, and what is its formula?

The proportion of variance in $y$ explained by the model: $$R^{2} = 1 - \frac{\sum_{i}(y_{i}-\hat{y}_{i})^{2}}{\sum_{i}(y_{i}-\bar{y})^{2}} = 1 - \frac{\text{RSS}}{\text{TSS}}.$$

Define accuracy as a classification metric and state its main weakness.

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}.$$ It is misleading on imbalanced datasets, where predicting the majority class can yield high accuracy despite poor minority-class performance.

Define precision and recall.

$$\text{Precision} = \frac{TP}{TP + FP}, \qquad \text{Recall} = \frac{TP}{TP + FN}.$$ Precision is correctness among positive predictions; recall is coverage of actual positives.

What is the F1 score and why is it used?

The harmonic mean of precision and recall: $$F_{1} = 2\cdot\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}.$$ It balances the two, which is useful for imbalanced classes where accuracy is misleading.

What is a confusion matrix and what are its four cells for binary classification?

A table comparing predicted vs actual labels with four entries: True Positives ($TP$), True Negatives ($TN$), False Positives ($FP$, Type I error), and False Negatives ($FN$, Type II error).

What do the ROC curve and AUC represent?

The ROC curve plots the True Positive Rate (recall) against the False Positive Rate $\left(\frac{FP}{FP+TN}\right)$ across thresholds. AUC is the area under it: $1.0$ is perfect, $0.5$ is random guessing.

What is k-fold cross-validation and how is the score computed?

The data is split into $k$ equal folds; the model is trained on $k-1$ folds and validated on the held-out fold, rotating so each fold is used once for validation. The overall estimate is the average of the $k$ validation scores.

What is leave-one-out cross-validation (LOOCV), and its main drawback?

A special case of $k$-fold with $k = n$: each single observation is the validation set once. It is nearly unbiased but computationally expensive and can have high variance.

Distinguish trend, seasonality, and cyclicity in a time series.

Trend: long-term upward or downward movement. Seasonality: regular fluctuations tied to a fixed, known period (e.g., monthly, yearly). Cyclicity: rises and falls over variable, non-fixed periods (e.g., business cycles) not tied to a calendar.