4. Using blockCV with caret

Introduction

This tutorial shows how to use spatial cross-validation folds generated by blockCV in the caret modelling framework. The key step is to pass the training and testing row indices stored in folds_list to caret::trainControl() through its index and indexOut arguments.

Prepare the data

Load the simulated presence-absence data and environmental raster covariates included with blockCV.

library(blockCV)
library(sf)
library(terra)

# import presence-absence species data
points <- read.csv(system.file("extdata/", "species.csv", package = "blockCV"))

# make an sf object from the data.frame
pa_data <- sf::st_as_sf(points, coords = c("x", "y"), crs = 7845)

# load raster covariates
covars <- terra::rast(
  list.files(system.file("extdata/au/", package = "blockCV"), full.names = TRUE)
)

Extract the raster covariate values at the species records. This is the modelling table that will be supplied to caret.

training <- terra::extract(covars, pa_data, ID = FALSE)
training$occ <- as.factor(pa_data$occ)

head(training)
##      bio_12    bio_15    bio_4    bio_5 occ
## 1 1287.2784  93.80042 323.8961 31.22999   0
## 2 1114.8223 118.56091 240.8846 29.44557   0
## 3  958.7523  85.10988 346.9273 28.04835   1
## 4  610.1941  22.76476 559.0266 31.00196   1
## 5  553.3838  15.44879 584.8892 31.40681   0
## 6 1729.3408 113.51132 123.3365 32.81274   0

Create spatial folds

Create spatial blocks with cv_spatial(). Each item in folds_list contains two integer vectors: the first is the training row indices and the second is the held-out testing row indices.

set.seed(123)

sb1 <- cv_spatial(
  x = pa_data,
  column = "occ",
  r = covars,
  size = 450000,
  k = 5,
  selection = "random",
  iteration = 50,
  progress = FALSE,
  report = TRUE,
  plot = TRUE
)
## 
##   train_0 train_1 test_0 test_1
## 1     198     223     59     20
## 2     203     183     54     60
## 3     204     197     53     46
## 4     213     196     44     47
## 5     210     173     47     70

Fit a caret model

The only conversion needed for caret is to separate the training and testing indices from folds_list. The following example uses a random forest model through caret::train(). The modelling chunk is not evaluated when the vignette is built because caret and randomForest are optional modelling packages.

library(caret)

train_index <- lapply(sb1$folds_list, function(fold) fold[[1]])
test_index <- lapply(sb1$folds_list, function(fold) fold[[2]])

names(train_index) <- paste0("Fold", seq_along(train_index))
names(test_index) <- names(train_index)

control <- trainControl(
  method = "cv",
  number = length(train_index),
  index = train_index,
  indexOut = test_index,
  search = "random"
)

set.seed(123)

rf_model <- train(
  occ ~ .,
  data = training,
  method = "rf",
  trControl = control,
  tuneLength = 4,
  ntree = 500
)

rf_model
rf_model$resample
plot(rf_model)

The resampling results are now based on the spatial folds from blockCV, rather than on random non-spatial folds generated internally by caret.

The CAST package uses the same caret interface. Its CreateSpacetimeFolds() function creates train and test lists from pre-defined spatial, temporal, or spatio-temporal groups that can be passed directly to trainControl(index = ..., indexOut = ...).

Please cite blockCV by: Valavi R, Elith J, Lahoz-Monfort JJ, Guillera-Arroita G. blockCV: An R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models. Methods Ecol Evol. 2019; 10:225-232. doi: 10.1111/2041-210X.13107

References

  • Meyer H, Reudenbach C, Hengl T, Katurji M, Nauss T. 2018. Improving performance of spatio-temporal machine learning models using forward feature selection and target-oriented validation. Environmental Modelling & Software 101: 1-9.

  • Kuhn M. 2008. Building predictive models in R using the caret package. Journal of Statistical Software 28: 1-26.

  • Valavi R, Elith J, Lahoz-Monfort JJ, Guillera-Arroita G. blockCV: An R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models. Methods Ecol Evol. 2019; 10:225-232. doi: 10.1111/2041-210X.13107