Associate of the Society of Actuaries (ASA/FSA) Exam PA — Predictive Analytics Flashcards

24 sample cards from the Exam PA — Predictive Analytics deck

Sampled from the end of the deck, so these are different cards from the ones shown on the syllabus page.

1. Compare bagging (random forests) with boosting.

   Bagging/random forests build many deep trees in parallel on bootstrap samples (forests also subsample features per split) and average them to reduce variance. Boosting builds trees sequentially, each correcting the previous one's errors, reducing bias. Boosting often achieves higher accuracy but is more prone to overfitting and slower to tune.

2. What is a random forest's key mechanism for decorrelating its trees?

   At each split, the random forest considers only a random subset of $m$ predictors (commonly $m \approx \sqrt{p}$ for classification). This prevents strong predictors from dominating every tree, decorrelating the trees so that averaging reduces variance more effectively than plain bagging.

3. Name the key hyperparameters of a gradient boosting machine and their effect.

   Number of trees (more can overfit), learning rate/shrinkage $\eta$ (smaller is more robust but needs more trees), tree depth/interaction depth (controls complexity), and subsampling fraction. Learning rate and number of trees trade off and are tuned together.

4. Why is the train/test (and validation) split essential in model validation?

   Evaluating a model on the same data used to train it gives an optimistically biased estimate of performance. A held-out test set estimates generalization to unseen data; a separate validation set (or CV) is used for tuning so the test set remains untouched until final assessment.

5. Explain k-fold cross-validation and its advantage.

   Data are split into $k$ equal folds; the model is trained on $k-1$ folds and validated on the held-out fold, rotating so each fold serves once as validation. The $k$ errors are averaged. It uses data efficiently and gives a lower-variance estimate of out-of-sample performance than a single split.

6. Define the bias-variance trade-off and relate it to model complexity.

   Expected test error decomposes into $\text{Bias}^{2} + \text{Variance} + \text{Irreducible error}$. Simple models have high bias, low variance (underfit); complex models have low bias, high variance (overfit). The goal is the complexity that minimizes total test error.

7. List common metrics for evaluating a classification model.

   Accuracy, precision, recall (sensitivity), specificity, F1-score, confusion-matrix counts, the ROC curve with AUC, and log-loss. The choice depends on class balance and the relative cost of false positives vs false negatives.

8. Define precision and recall and give their formulas.

   Precision $= \frac{TP}{TP + FP}$ — of predicted positives, the fraction truly positive. Recall (sensitivity) $= \frac{TP}{TP + FN}$ — of actual positives, the fraction correctly identified. There is generally a trade-off between them, summarized by the F1-score.

9. What does the ROC curve plot and what does AUC of 0.5 versus 1.0 mean?

   The ROC curve plots the true positive rate (sensitivity) against the false positive rate ($1 - $ specificity) across classification thresholds. AUC (area under the curve) of $0.5$ indicates no better than random guessing; $1.0$ indicates perfect discrimination.

10. Name common metrics for evaluating a regression model.

    Mean squared error (MSE), root MSE (RMSE), mean absolute error (MAE), and the coefficient of determination $R^{2}$. For GLMs, deviance and AIC/BIC are also used to compare fit while penalizing complexity.

11. Give the formulas for AIC and BIC and state their purpose.

    $\text{AIC} = -2\ln(\hat{L}) + 2k$ and $\text{BIC} = -2\ln(\hat{L}) + k\ln(n)$, where $\hat{L}$ is the maximized likelihood, $k$ the number of parameters, and $n$ the sample size. Both penalize complexity to compare models; lower is better. BIC penalizes parameters more heavily for large $n$.

12. How is residual analysis used to validate a GLM?

    Residuals (e.g., deviance or Pearson residuals) are plotted against fitted values and predictors to check for systematic patterns, non-constant variance, or outliers. Randomly scattered residuals around zero support the chosen distribution and link; patterns signal missing terms or a poor model form.

13. What is a variable importance measure and how is it obtained for tree ensembles?

    Variable importance ranks predictors by their contribution to the model. For tree ensembles it is commonly the total reduction in impurity (or RSS) attributed to splits on that variable, summed over all trees, or a permutation-based drop in accuracy when the variable is shuffled.

14. What is a partial dependence plot (PDP) and what does it show?

    A PDP shows the marginal effect of one (or two) predictor(s) on the predicted response, averaging out the effects of all other predictors. It helps interpret 'black-box' models like boosted trees by revealing the shape and direction of a predictor's relationship with the target.

15. Distinguish statistical significance from practical (business) significance when interpreting model output.

    Statistical significance (e.g., a small p-value) indicates an effect is unlikely due to chance, but with large samples even tiny effects can be significant. Practical significance asks whether the effect's magnitude is large enough to matter for business decisions. Both should be considered.

16. How should an actuary tailor communication of model results to a non-technical stakeholder?

    Lead with the business answer and recommendation, avoid jargon, use intuitive visuals (charts, examples), translate coefficients into concrete impacts (e.g., effect on premium), state key assumptions and limitations, and quantify uncertainty in plain terms. Reserve technical detail for an appendix or technical audience.

17. What elements should an effective predictive analytics report or executive summary contain?

    The business problem and objective, data sources and key assumptions, the modeling approach and why it was chosen, main findings and their business implications, the recommendation, limitations/caveats, and next steps. Technical methodology supports but does not dominate the narrative.

18. Why is conveying model uncertainty and limitations important when communicating results?

    Stakeholders make decisions based on the model; failing to disclose uncertainty (confidence intervals, validation error), data limitations, and assumptions can lead to overconfidence, poor decisions, and erosion of trust. Transparency about limitations is part of professional responsibility.

19. What ethical risks arise from using protected or proxy variables in predictive models?

    Including protected attributes (race, gender, etc.) — or proxies highly correlated with them — can produce unfair discrimination and legal/regulatory violations, even unintentionally. Models must be checked for disparate impact and proxy discrimination, not just exclusion of the explicit attribute.

20. Define algorithmic bias and unfairness in a predictive model.

    Algorithmic bias occurs when a model systematically produces outcomes that disadvantage particular groups, often because the training data reflects historical bias or unrepresentative sampling. Fairness assessment checks whether error rates or predictions differ unjustifiably across protected groups.

21. What professional and regulatory considerations apply to an actuary building predictive models?

    Adherence to the Actuarial Standards of Practice (ASOPs) and the Code of Professional Conduct, data privacy and consent laws, documentation and reproducibility, transparency with stakeholders, avoiding unfair discrimination, and ensuring the model is appropriate, validated, and used within its intended scope.

22. Why must data privacy and consent be considered in an actuarial analytics project?

    Personal data is often subject to privacy regulations and policyholder expectations. Actuaries must ensure data is collected and used lawfully and with appropriate consent, is secured and anonymized where required, and is not used beyond its intended purpose, to protect individuals and the firm's reputation.

23. What is overfitting, what are its symptoms, and how is it mitigated?

    Overfitting is when a model learns noise specific to the training data, performing well in-sample but poorly out-of-sample. Symptoms: training error far below validation/test error. Mitigations: simpler models, regularization, pruning, cross-validation-guided tuning, more data, and early stopping.

24. Why is reproducibility and documentation emphasized in the Exam PA workflow?

    Documenting data sources, cleaning steps, modeling choices, assumptions, and code enables others to review, audit, reproduce, and maintain the work. It supports professional standards, regulatory scrutiny, knowledge transfer, and defensible decision-making.

What this deck covers

Exam PA — Predictive Analytics flashcards FAQ