Machine Learning · AI216

K-Fold and Stratified K-Fold Cross-Validation

Machine Learning Study guides

K-fold cross-validation splits the training set into K folds, trains on K-1 of them and evaluates on the remaining fold, repeating K times so every fold is used once for evaluation.

Measuring a model on the same data it trained on is optimistic. K-fold cross-validation gives a better estimate: split the training set into K folds, train on K-1 folds and evaluate on the held-out fold, K times, then average the K scores. In Scikit-Learn cross_val_score returns the evaluation scores for each fold, while cross_val_predict returns the predictions made on each fold when it was held out.

StratifiedKFold performs stratified sampling: each fold keeps the same class proportions as the full dataset. Regular KFold does not guarantee that, so with imbalanced classes a fold might contain very few of the rare class. Stratified is the right default for classification.

Note that the test set stays untouched until the very end. Cross-validation works only on the training set (and uses its own folds for validation).

Key points

  • K folds: K-1 train, 1 evaluate, repeat K times.
  • Average the K scores for a more reliable estimate.
  • Stratified keeps class ratios in every fold.
  • cross_val_score returns scores; cross_val_predict returns predictions.
  • Never touch the test set during cross-validation.

Worked example

With 5-fold cross-validation on 10,000 training images, each round trains on 8,000 and scores on 2,000. After five rounds you have five accuracy numbers and take their mean and spread.

Common questions

What is the difference between StratifiedKFold and KFold?

StratifiedKFold keeps the class proportions in every fold; plain KFold just splits without regard to class balance.

What is cross_val_predict for?

It returns the prediction made for every instance by a model that did not train on that instance, useful for building a confusion matrix.

Practice questions on this topic

Try these, then open the app to check your answers with explanations.

  1. ....................performs K-fold cross-validation and returns the predictions made on each test fold
    • A. cross_val_predict()
    • B. cross_val_score()
    • C. cross_test_predict()
    • D. none of them
  2. Which of the following is TRUE about the StratifiedKFold class
    • A. It produces folds that contain a representative ratio of each class in the dataset
    • B. It is a cross-validation method designed specifically for regression problems
    • C. It always guarantees higher accuracy on imbalanced datasets
    • D. It is the same as regular KFold
  3. What is the difference between StratifiedKFold and regular KFold cross-validation
    • A. StratifiedKFold performs stratified sampling, while regular KFold does not
    • B. StratifiedKFold is always slower than regular Kfold
    • C. StratifiedKFold is only useful for imbalanced datasets.
    • D. There is no difference between StratifiedKFold and regular Kfold
  4. Just like the ....................... function, ....................performs K-fold cross-validation, but instead of returning the evaluation scores, it returns the predictions made on each test fold.
    • A. cross_val_score(), cross_val_predict()
    • B. cross_val_predict() , cross_val_score()
    • C. cross_val_test() , cross_val_score()
    • D. none of them
  5. ..........................performs K-fold cross-validation and return the evaluation scores
    • A. cross_val_score()
    • B. cross_val_predict()
    • C. cross_test_predict()
    • D. none of them
  6. K-fold cross-validation means splitting the testing set into K folds, then training them on each fold using a model trained on the remaining folds.
    • A. false
    • B. true