Machine Learning Project: Recommendation System Flashcards

24 sample cards from the Project: Recommendation System deck

Sampled from the end of the deck, so these are different cards from the ones shown on the syllabus page.

1. Define Precision@k and Recall@k for evaluating a top-N recommender.

   Precision@k = (relevant items in top $k$) / $k$. Recall@k = (relevant items in top $k$) / (total relevant items). Precision measures list quality; recall measures coverage of relevant items.

2. What is the RMSE formula used to evaluate rating-prediction recommenders?

   $$\text{RMSE} = \sqrt{\frac{1}{N}\sum_{(u,i)}(\hat{r}_{ui} - r_{ui})^{2}}$$ where $N$ is the number of test ratings. Lower is better; it penalizes large errors more heavily than MAE.

3. What is the MAE formula and how does it differ from RMSE?

   $$\text{MAE} = \frac{1}{N}\sum_{(u,i)} |\hat{r}_{ui} - r_{ui}|$$ MAE is the average absolute error and treats all errors linearly, while RMSE squares errors and thus penalizes large deviations more.

4. What does NDCG measure and why is it useful for recommendations?

   Normalized Discounted Cumulative Gain measures ranking quality by rewarding relevant items placed higher in the list, with a logarithmic position discount, then normalizing by the ideal ordering. It is useful because position matters in recommendation lists.

5. Write the DCG@k formula used in NDCG.

   $$\text{DCG@}k = \sum_{i=1}^{k} \frac{2^{rel_i}-1}{\log_2(i+1)}$$ where $rel_i$ is the relevance of the item at rank $i$. NDCG@k = DCG@k / IDCG@k.

6. Besides accuracy metrics, name three 'beyond-accuracy' evaluation metrics for recommenders.

   Coverage (fraction of catalog recommended), diversity (dissimilarity within a list), novelty (how non-obvious/unpopular recommended items are), and serendipity (useful surprise).

7. Why can't you split recommendation data with a naive random row split the way you might for standard classification?

   Random splitting can leak future interactions into the training set and may leave users/items in the test set with no training history. Recommenders often use per-user holdout or temporal (time-based) splits to respect chronology and avoid leakage.

8. What is a temporal (time-based) train/test split and when is it used?

   Interactions are split by time—earlier interactions for training, later ones for testing—so the model is evaluated on predicting future behavior. It is used when temporal dynamics matter and to realistically simulate deployment.

9. What is a leave-one-out split in recommender evaluation?

   For each user, the most recent (or one random) interaction is held out for testing while the rest are used for training. It is common for evaluating top-N ranking where you check if the held-out item appears in the recommended list.

10. What is the role of a validation set when training a recommender?

    A validation set (separate from train and test) is used to tune hyperparameters (latent factor count, regularization, learning rate) and to monitor for overfitting via early stopping, without contaminating the final test evaluation.

11. What regularized loss function does matrix factorization typically minimize?

    $$\min_{P,Q} \sum_{(u,i)\in\mathcal{K}} (r_{ui} - \vec{p}_u \cdot \vec{q}_i)^{2} + \lambda\left(\lVert \vec{p}_u \rVert^{2} + \lVert \vec{q}_i \rVert^{2}\right)$$ where $\mathcal{K}$ is the set of known ratings and $\lambda$ controls regularization.

12. What are the two common optimization algorithms for training matrix factorization models?

    Stochastic Gradient Descent (SGD), which updates factors per observed rating, and Alternating Least Squares (ALS), which fixes one factor matrix and solves the other in closed form, alternating until convergence. ALS parallelizes well for implicit data.

13. How does regularization parameter $\lambda$ affect the trained model?

    $\lambda$ penalizes large latent factor magnitudes. Too small $\to$ overfitting (memorizes training ratings); too large $\to$ underfitting (factors shrink toward zero, losing signal). It is tuned on the validation set.

14. What does it mean to evaluate the model on the test set, and why must it stay untouched during training?

    Evaluating on the held-out test set estimates generalization to unseen interactions. It must never be used for training or hyperparameter tuning, otherwise the metric is optimistically biased and no longer reflects real-world performance.

15. What are signs that a recommender is overfitting?

    Training error keeps decreasing while validation/test error rises; excellent accuracy on seen users/items but poor performance on new data; over-reliance on popular items. Remedies include more regularization, fewer latent factors, and early stopping.

16. Name three hyperparameters you would fine-tune in a matrix-factorization recommender.

    Number of latent factors (rank $k$), regularization strength $\lambda$, learning rate, number of epochs/iterations, and (for implicit ALS) the confidence weighting parameter.

17. What hyperparameter-search strategies are commonly used to fine-tune recommender models?

    Grid search (exhaustive over a defined grid), random search (samples configurations), and Bayesian optimization (models the objective to choose promising configurations efficiently), each evaluated on a validation set.

18. What does 'preparing the model for production' involve for a recommender?

    Serializing/exporting the trained model, precomputing item embeddings or recommendation lists, building efficient retrieval (e.g., approximate nearest neighbor / ANN indexes), defining the serving API, and ensuring latency and scalability meet requirements.

19. Why are approximate nearest neighbor (ANN) indexes used when serving recommendations at scale?

    Exact similarity search over millions of item embeddings is too slow for real-time serving. ANN indexes (e.g., HNSW, FAISS, LSH) retrieve the top similar items in sub-linear time with minimal accuracy loss, meeting latency requirements.

20. What are common challenges when integrating a recommender with existing production systems?

    Matching data schemas and feature pipelines, meeting latency/throughput SLAs, handling real-time vs batch scoring, fitting the API into existing front-end/services, fallback behavior for cold-start, and ensuring consistency between training and serving features (avoiding training-serving skew).

21. Why is monitoring a deployed recommender necessary, and what is model/data drift?

    User preferences and item catalogs change over time, so a static model degrades. Drift is the change in data distribution (data drift) or in the input-output relationship (concept drift) that causes accuracy and business metrics to decline, signaling the need for retraining.

22. What is the role of A/B testing and online metrics in maintaining a recommender?

    A/B tests compare a new model/variant against the current one on live traffic, measuring real business KPIs (CTR, conversion, engagement). They validate that offline improvements translate to real user value before full rollout.

23. How is user feedback collected and used to improve a recommender after deployment?

    Through explicit signals (ratings, likes, thumbs) and implicit signals (clicks, dwell time, purchases, skips). This feedback is logged, fed back into the training data, and used to retrain/update the model, closing the loop for continuous improvement.

24. What does 'continuous improvement' mean in the lifecycle of a production recommendation system?

    An ongoing loop of collecting new interaction data and feedback, periodically retraining or incrementally updating the model, re-evaluating offline and online, A/B testing improvements, and monitoring—so the system adapts to evolving users, items, and trends.

What this deck covers

Project: Recommendation System flashcards FAQ