Data Science Machine Learning Syllabus

What cost function does linear regression minimize, and what is its formula?

Mean Squared Error (MSE): $J(\beta) = \dfrac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^{2}$. Ordinary Least Squares finds the coefficients that minimize this.

What is the closed-form normal equation solution for linear regression?

$\hat{\beta} = (X^{T} X)^{-1} X^{T} y$, which directly computes the coefficients minimizing the sum of squared errors without iteration.

What does the coefficient of determination $R^{2}$ measure?

The proportion of variance in the target explained by the model: $R^{2} = 1 - \dfrac{\sum (y_i - \hat{y}_i)^{2}}{\sum (y_i - \bar{y})^{2}}$. It ranges from $-\infty$ to $1$, where $1$ is a perfect fit.

What are the key assumptions of linear regression?

Linearity (relationship is linear), independence of errors, homoscedasticity (constant error variance), normality of residuals, and little or no multicollinearity among predictors.

What are common strategies for handling missing data?

Deletion (drop rows/columns), imputation (fill with mean/median/mode), model-based imputation (e.g., KNN or regression), or using a missingness indicator flag. The choice depends on the missingness mechanism (MCAR, MAR, MNAR).

What is the difference between an outlier and an anomaly, and name a method to detect outliers.

An outlier is a data point far from the rest of the distribution; an anomaly often implies a meaningful/rare event. Detection methods include the IQR rule (outside $[Q_1 - 1.5\,\text{IQR},\, Q_3 + 1.5\,\text{IQR}]$), z-scores ($|z| > 3$), and isolation forests.

What is the z-score standardization formula and what does it achieve?

$z = \dfrac{x - \mu}{\sigma}$. It rescales a feature to have mean $0$ and standard deviation $1$, making features comparable and helping gradient-based and distance-based algorithms.

What is min-max normalization?

$x' = \dfrac{x - x_{\min}}{x_{\max} - x_{\min}}$, which rescales features to a fixed range, typically $[0, 1]$.

What is feature engineering?

The process of creating, transforming, or selecting input variables (features) from raw data to improve model performance — e.g., creating interaction terms, binning, polynomial features, or domain-derived ratios.

What is one-hot encoding and when is it used?

A technique to convert a categorical variable with $k$ categories into $k$ binary (0/1) columns, used so that nominal categories (no inherent order) can be fed to numerical models without implying ordinal relationships.

What is the difference between feature selection and feature extraction?

Feature selection chooses a subset of the original features (e.g., via filter/wrapper/embedded methods). Feature extraction creates new features by transforming/combining originals (e.g., PCA), changing the feature space.

Why apply a log transformation to a feature?

To reduce right skewness, compress large ranges, stabilize variance, and make multiplicative relationships additive. Useful for heavy-tailed variables like income or counts.

What is the logistic regression model and its output?

It models the probability of a binary class using the sigmoid: $\hat{p} = \sigma(z) = \dfrac{1}{1 + e^{-z}}$ where $z = \beta_0 + \sum_j \beta_j x_j$. Output is a probability in $(0,1)$, thresholded (e.g., at $0.5$) for classification.

What loss function does logistic regression minimize?

Binary cross-entropy (log loss): $J = -\dfrac{1}{n} \sum_{i=1}^{n} \left[ y_i \log \hat{p}_i + (1 - y_i) \log(1 - \hat{p}_i) \right]$.

How does the k-Nearest Neighbors (k-NN) algorithm classify a point?

It finds the $k$ closest training points (by a distance metric like Euclidean) and assigns the majority class among them. It is a lazy, non-parametric, instance-based learner.

What is the core idea of a Support Vector Machine (SVM)?

It finds the hyperplane that maximizes the margin (distance) between classes. The closest points are support vectors. The kernel trick allows nonlinear separation by implicitly mapping to higher dimensions.

What classification assumption does the Naive Bayes classifier make?

It assumes all features are conditionally independent given the class label. Prediction: $\hat{y} = \arg\max_c P(c) \prod_{j} P(x_j \mid c)$.

How does a decision tree decide where to split, and name a common impurity measure.

It selects splits that maximize purity gain. Common impurity measures: Gini impurity $G = 1 - \sum_{i} p_i^{2}$ and entropy $H = -\sum_{i} p_i \log_2 p_i$; the split maximizing information gain (entropy reduction) or Gini decrease is chosen.

What is the difference between bagging and boosting?

Bagging trains models in parallel on bootstrap samples and averages them to reduce variance (e.g., Random Forest). Boosting trains models sequentially, each correcting the errors of the previous, to reduce bias (e.g., AdaBoost, Gradient Boosting).

What is a Random Forest?

An ensemble of decision trees trained on bootstrap samples (bagging) with random feature subsets at each split. Predictions aggregate via majority vote (classification) or averaging (regression), reducing overfitting/variance.

Describe the k-means clustering algorithm.

1) Choose $k$ and initialize $k$ centroids. 2) Assign each point to the nearest centroid. 3) Recompute each centroid as the mean of its assigned points. 4) Repeat steps 2-3 until convergence. It minimizes within-cluster sum of squares (inertia).

What is the elbow method used for in clustering?

To choose the number of clusters $k$ by plotting within-cluster sum of squares (inertia) against $k$ and selecting the 'elbow' point where adding more clusters yields diminishing reductions.

How does DBSCAN differ from k-means?

DBSCAN is density-based: it groups densely connected points (parameters $\varepsilon$ and minPts), can find arbitrarily shaped clusters, marks low-density points as noise, and does not require specifying the number of clusters. k-means requires $k$ and assumes spherical clusters.

What is the goal of dimensionality reduction?

To reduce the number of features while retaining as much relevant information as possible — combating the curse of dimensionality, reducing computation, removing redundancy/noise, and aiding visualization.

What does Principal Component Analysis (PCA) do?

PCA finds orthogonal axes (principal components) that capture maximum variance, given by the eigenvectors of the covariance matrix. It projects data onto the top components, reducing dimensionality while preserving variance.

What is the curse of dimensionality?

As the number of features grows, data becomes sparse, distances between points become less meaningful, and the data needed to generalize grows exponentially — degrading distance-based and density-based methods.

What does association rule learning find, and define support and confidence.

It finds frequent co-occurrence rules $X \Rightarrow Y$ (e.g., market basket analysis). Support: $\text{supp}(X) = \dfrac{\text{transactions containing } X}{\text{total transactions}}$. Confidence: $\text{conf}(X \Rightarrow Y) = \dfrac{\text{supp}(X \cup Y)}{\text{supp}(X)}$.

What is lift in association rule learning and how is it interpreted?

$\text{lift}(X \Rightarrow Y) = \dfrac{\text{conf}(X \Rightarrow Y)}{\text{supp}(Y)}$. Lift $> 1$ means $X$ and $Y$ are positively associated, $= 1$ means independent, $< 1$ means negatively associated.

In reinforcement learning, what are the key components of a Markov Decision Process (MDP)?

States $S$, actions $A$, transition probabilities $P(s' \mid s, a)$, reward function $R(s, a)$, and a discount factor $\gamma \in [0, 1]$. The agent seeks a policy $\pi$ maximizing expected cumulative discounted reward.

What is the Q-learning update rule?

$Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]$, where $\alpha$ is the learning rate, $\gamma$ the discount factor, and $r$ the observed reward.

What is the precision and recall of a classifier?

Precision $= \dfrac{TP}{TP + FP}$ (of predicted positives, how many are correct). Recall $= \dfrac{TP}{TP + FN}$ (of actual positives, how many were found). The $F_1$ score is their harmonic mean: $F_1 = 2 \cdot \dfrac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$.

What does ROC-AUC measure?

The ROC curve plots true positive rate vs. false positive rate across thresholds; AUC (area under it) measures overall ranking ability. AUC $= 1$ is perfect, $= 0.5$ is random guessing.

What is k-fold cross-validation?

The data is split into $k$ equal folds; the model is trained on $k-1$ folds and validated on the held-out fold, repeated $k$ times so each fold serves once as validation. Results are averaged, giving a robust performance estimate and reducing dependence on a single split.

What is the difference between L1 (Lasso) and L2 (Ridge) regularization?

L1 adds $\lambda \sum_j |\beta_j|$ to the loss, driving some coefficients to exactly zero (feature selection/sparsity). L2 adds $\lambda \sum_j \beta_j^{2}$, shrinking coefficients smoothly toward zero without eliminating them.

How do batch, stochastic, and mini-batch gradient descent differ?

Batch GD uses the entire dataset per update (stable, slow). Stochastic GD (SGD) uses one sample per update (fast, noisy). Mini-batch GD uses a small subset per update, balancing stability and speed — the most common in practice.

Why is the learning rate $\alpha$ important in gradient descent?

It controls the step size of updates. Too large $\alpha$ can overshoot or diverge; too small $\alpha$ makes convergence very slow. It is a key hyperparameter often tuned or scheduled (decayed) over training.