Data Science Machine Learning Flashcards

24 sample cards from the Machine Learning deck

Sampled from the end of the deck, so these are different cards from the ones shown on the syllabus page.

1. What is the logistic regression model and its output?

   It models the probability of a binary class using the sigmoid: $\hat{p} = \sigma(z) = \dfrac{1}{1 + e^{-z}}$ where $z = \beta_0 + \sum_j \beta_j x_j$. Output is a probability in $(0,1)$, thresholded (e.g., at $0.5$) for classification.

2. What loss function does logistic regression minimize?

   Binary cross-entropy (log loss): $J = -\dfrac{1}{n} \sum_{i=1}^{n} \left[ y_i \log \hat{p}_i + (1 - y_i) \log(1 - \hat{p}_i) \right]$.

3. How does the k-Nearest Neighbors (k-NN) algorithm classify a point?

   It finds the $k$ closest training points (by a distance metric like Euclidean) and assigns the majority class among them. It is a lazy, non-parametric, instance-based learner.

4. What is the core idea of a Support Vector Machine (SVM)?

   It finds the hyperplane that maximizes the margin (distance) between classes. The closest points are support vectors. The kernel trick allows nonlinear separation by implicitly mapping to higher dimensions.

5. What classification assumption does the Naive Bayes classifier make?

   It assumes all features are conditionally independent given the class label. Prediction: $\hat{y} = \arg\max_c P(c) \prod_{j} P(x_j \mid c)$.

6. How does a decision tree decide where to split, and name a common impurity measure.

   It selects splits that maximize purity gain. Common impurity measures: Gini impurity $G = 1 - \sum_{i} p_i^{2}$ and entropy $H = -\sum_{i} p_i \log_2 p_i$; the split maximizing information gain (entropy reduction) or Gini decrease is chosen.

7. What is the difference between bagging and boosting?

   Bagging trains models in parallel on bootstrap samples and averages them to reduce variance (e.g., Random Forest). Boosting trains models sequentially, each correcting the errors of the previous, to reduce bias (e.g., AdaBoost, Gradient Boosting).

8. What is a Random Forest?

   An ensemble of decision trees trained on bootstrap samples (bagging) with random feature subsets at each split. Predictions aggregate via majority vote (classification) or averaging (regression), reducing overfitting/variance.

9. Describe the k-means clustering algorithm.

   1) Choose $k$ and initialize $k$ centroids. 2) Assign each point to the nearest centroid. 3) Recompute each centroid as the mean of its assigned points. 4) Repeat steps 2-3 until convergence. It minimizes within-cluster sum of squares (inertia).

10. What is the elbow method used for in clustering?

    To choose the number of clusters $k$ by plotting within-cluster sum of squares (inertia) against $k$ and selecting the 'elbow' point where adding more clusters yields diminishing reductions.

11. How does DBSCAN differ from k-means?

    DBSCAN is density-based: it groups densely connected points (parameters $\varepsilon$ and minPts), can find arbitrarily shaped clusters, marks low-density points as noise, and does not require specifying the number of clusters. k-means requires $k$ and assumes spherical clusters.

12. What is the goal of dimensionality reduction?

    To reduce the number of features while retaining as much relevant information as possible — combating the curse of dimensionality, reducing computation, removing redundancy/noise, and aiding visualization.

13. What does Principal Component Analysis (PCA) do?

    PCA finds orthogonal axes (principal components) that capture maximum variance, given by the eigenvectors of the covariance matrix. It projects data onto the top components, reducing dimensionality while preserving variance.

14. What is the curse of dimensionality?

    As the number of features grows, data becomes sparse, distances between points become less meaningful, and the data needed to generalize grows exponentially — degrading distance-based and density-based methods.

15. What does association rule learning find, and define support and confidence.

    It finds frequent co-occurrence rules $X \Rightarrow Y$ (e.g., market basket analysis). Support: $\text{supp}(X) = \dfrac{\text{transactions containing } X}{\text{total transactions}}$. Confidence: $\text{conf}(X \Rightarrow Y) = \dfrac{\text{supp}(X \cup Y)}{\text{supp}(X)}$.

16. What is lift in association rule learning and how is it interpreted?

    $\text{lift}(X \Rightarrow Y) = \dfrac{\text{conf}(X \Rightarrow Y)}{\text{supp}(Y)}$. Lift $> 1$ means $X$ and $Y$ are positively associated, $= 1$ means independent, $< 1$ means negatively associated.

17. In reinforcement learning, what are the key components of a Markov Decision Process (MDP)?

    States $S$, actions $A$, transition probabilities $P(s' \mid s, a)$, reward function $R(s, a)$, and a discount factor $\gamma \in [0, 1]$. The agent seeks a policy $\pi$ maximizing expected cumulative discounted reward.

18. What is the Q-learning update rule?

    $Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]$, where $\alpha$ is the learning rate, $\gamma$ the discount factor, and $r$ the observed reward.

19. What is the precision and recall of a classifier?

    Precision $= \dfrac{TP}{TP + FP}$ (of predicted positives, how many are correct). Recall $= \dfrac{TP}{TP + FN}$ (of actual positives, how many were found). The $F_1$ score is their harmonic mean: $F_1 = 2 \cdot \dfrac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$.

20. What does ROC-AUC measure?

    The ROC curve plots true positive rate vs. false positive rate across thresholds; AUC (area under it) measures overall ranking ability. AUC $= 1$ is perfect, $= 0.5$ is random guessing.

21. What is k-fold cross-validation?

    The data is split into $k$ equal folds; the model is trained on $k-1$ folds and validated on the held-out fold, repeated $k$ times so each fold serves once as validation. Results are averaged, giving a robust performance estimate and reducing dependence on a single split.

22. What is the difference between L1 (Lasso) and L2 (Ridge) regularization?

    L1 adds $\lambda \sum_j |\beta_j|$ to the loss, driving some coefficients to exactly zero (feature selection/sparsity). L2 adds $\lambda \sum_j \beta_j^{2}$, shrinking coefficients smoothly toward zero without eliminating them.

23. How do batch, stochastic, and mini-batch gradient descent differ?

    Batch GD uses the entire dataset per update (stable, slow). Stochastic GD (SGD) uses one sample per update (fast, noisy). Mini-batch GD uses a small subset per update, balancing stability and speed — the most common in practice.

24. Why is the learning rate $\alpha$ important in gradient descent?

    It controls the step size of updates. Too large $\alpha$ can overshoot or diverge; too small $\alpha$ makes convergence very slow. It is a key hyperparameter often tuned or scheduled (decayed) over training.

What this deck covers

Machine Learning flashcards FAQ