Artificial Intelligence Machine Learning Syllabus

What does the regularization hyperparameter $\lambda$ control?

The strength of the penalty: larger $\lambda$ increases bias but reduces variance (more shrinkage/simpler model); $\lambda = 0$ recovers the unregularized model.

How does the k-Nearest Neighbors algorithm classify a new point?

It finds the $k$ closest training points (by a distance metric) and assigns the majority class among them (for regression, averages their values). It is a lazy, non-parametric method with no explicit training phase.

Give the Euclidean and Manhattan distance formulas used in kNN.

Euclidean: $d = \sqrt{\sum_j (x_j - x'_j)^2}$. Manhattan: $d = \sum_j |x_j - x'_j|$.

What is the effect of choosing a very small vs very large $k$ in kNN?

Small $k$ gives low bias but high variance (noisy, jagged boundaries); large $k$ gives high bias but low variance (smoother boundaries, may underfit).

What does a Support Vector Machine maximize, and what are support vectors?

It maximizes the margin, the distance between the decision hyperplane and the nearest points. The support vectors are those closest training points that lie on the margin and define the boundary.

Give the margin width of an SVM with weight vector $\vec{w}$.

The margin equals $\frac{2}{\lVert \vec{w} \rVert}$, so maximizing the margin is equivalent to minimizing $\frac{1}{2}\lVert \vec{w} \rVert^2$.

What is the kernel trick in SVMs?

Replacing inner products $\vec{x}_i^{\top}\vec{x}_j$ with a kernel $K(\vec{x}_i, \vec{x}_j)$ that computes similarity in a high-dimensional feature space without explicitly mapping the data, enabling nonlinear boundaries.

Give the RBF (Gaussian) kernel formula.

$$K(\vec{x}, \vec{x}') = \exp\!\left(-\gamma \lVert \vec{x} - \vec{x}' \rVert^2\right)$$

What does the soft-margin parameter $C$ control in an SVM?

The tradeoff between maximizing the margin and minimizing classification errors: large $C$ penalizes misclassification heavily (low bias, high variance); small $C$ allows more slack (wider margin, more tolerance).

State Bayes' theorem as used by the Naive Bayes classifier.

$$P(y \mid \vec{x}) = \frac{P(\vec{x} \mid y)\,P(y)}{P(\vec{x})}$$

What is the 'naive' assumption in the Naive Bayes classifier?

That features are conditionally independent given the class, so $P(\vec{x} \mid y) = \prod_j P(x_j \mid y)$.

What is Laplace (additive) smoothing and why is it needed in Naive Bayes?

Adding a constant $\alpha$ (often 1) to counts so no conditional probability is zero, preventing an unseen feature value from zeroing out the entire product $\prod_j P(x_j \mid y)$.

Compare bagging and boosting as ensemble methods.

Bagging trains models in parallel on bootstrap samples and averages them to reduce variance (e.g. Random Forest). Boosting trains models sequentially, each correcting the previous one's errors, primarily reducing bias (e.g. AdaBoost, Gradient Boosting).

What two sources of randomness make a Random Forest?

Bootstrap sampling of training rows (bagging) and random selection of a subset of features at each split, which decorrelates the trees.

How does AdaBoost adjust training as it proceeds?

It reweights training examples after each weak learner, increasing weights on misclassified points so subsequent learners focus on them; final prediction is a weighted vote of the weak learners.

Write the equation for a single perceptron's output.

$$y = \phi\!\left(\sum_{j} w_j x_j + b\right)$$ where $\phi$ is a step/activation function.

What key limitation of a single-layer perceptron did Minsky and Papert highlight?

It can only separate linearly separable data; it cannot represent the XOR function. Multilayer networks with nonlinear activations overcome this.

Why are nonlinear activation functions necessary in multilayer networks?

Without them, a composition of linear layers collapses to a single linear transformation, so the network could not model nonlinear relationships regardless of depth.

Name three common activation functions and one property of each.

Sigmoid $\sigma(z)=\frac{1}{1+e^{-z}}$ (saturates, vanishing gradients); $\tanh(z)$ (zero-centered, range $(-1,1)$); ReLU $\max(0, z)$ (cheap, non-saturating for $z>0$, but 'dying' units for $z<0$).

What is backpropagation?

An algorithm that computes gradients of the loss with respect to every weight by applying the chain rule backward through the network, enabling gradient-based optimization.

Give the gradient-descent weight update rule using learning rate $\eta$.

$$w \leftarrow w - \eta \frac{\partial J}{\partial w}$$

What is the vanishing gradient problem?

In deep networks, repeated multiplication of small derivatives (e.g. from saturating sigmoid/tanh) makes gradients shrink exponentially toward earlier layers, so those layers train extremely slowly. ReLU, residual connections, and careful initialization mitigate it.

What three key architectural ideas define a Convolutional Neural Network?

Local receptive fields (convolutional filters), parameter/weight sharing across spatial positions, and pooling for spatial downsampling and translation invariance.

Give the output size formula for a convolution with input size $W$, filter $F$, padding $P$, and stride $S$.

$$O = \frac{W - F + 2P}{S} + 1$$

What is the purpose of a pooling layer (e.g. max pooling) in a CNN?

To reduce spatial dimensions, lower computation and parameters, provide small-translation invariance, and enlarge the effective receptive field.

What problem do Recurrent Neural Networks address, and what limits vanilla RNNs?

They process sequential data by maintaining a hidden state $h_t = \phi(W_h h_{t-1} + W_x x_t + b)$. Vanilla RNNs suffer from vanishing/exploding gradients, limiting learning of long-range dependencies.

How do LSTM/GRU cells improve over vanilla RNNs?

They use gating mechanisms (input, forget, output gates in LSTM) and a cell state that lets gradients flow across many time steps, mitigating vanishing gradients and capturing long-term dependencies.

What is the core mechanism of the Transformer architecture, and give the scaled dot-product attention formula.

Self-attention, which relates all positions in a sequence in parallel. $$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$

Why do Transformers require positional encodings?

Self-attention is permutation-invariant and has no inherent notion of order, so positional encodings inject sequence-position information into the token representations.

Describe the k-means clustering algorithm and its objective.

Alternately assign each point to its nearest centroid and recompute centroids as cluster means, until convergence. It minimizes within-cluster variance $$J = \sum_{i}\sum_{x \in C_i} \lVert x - \mu_i \rVert^2$$

Contrast k-means with hierarchical (agglomerative) clustering, and name what PCA optimizes for dimensionality reduction.

k-means partitions data into a preset $k$ clusters via centroids; hierarchical clustering builds a dendrogram by successively merging closest clusters and needs no preset $k$. PCA reduces dimensions by projecting onto the orthogonal directions (eigenvectors of the covariance matrix) that capture maximum variance.