rsatoolbox.inference.evaluate module

evaluate model performance

rsatoolbox.inference.evaluate.bootstrap_crossval(models, data, method='cosine', fitter=None, k_pattern=None, k_rdm=None, N=1000, n_cv=2, pattern_descriptor='index', rdm_descriptor='index', boot_type='both', use_correction=True)

evaluates a set of models by k-fold crossvalidation within a bootstrap

Crossvalidation creates variance in the results for a single bootstrap sample, because different assginments to the training and test group lead to different results. To correct for this, we apply a formula which estimates the variance we expect if we evaluated all possible crossvalidation assignments from n_cv different assignments per bootstrap sample. In our statistical evaluations we saw that many bootstrap samples and few different crossvalidation assignments are optimal to minimize the variance of the variance estimate. Thus, this function by default applies this correction formula and sets n_cv=2, i.e. performs only two different assignments per fold. This function nonetheless performs full crossvalidation schemes, i.e. in every bootstrap sample all crossvalidation folds are evaluated such that each RDM and each condition is in the test set n_cv times. For the even more optimized version which computes only two randomly chosen test sets see bootstrap_cv_random.

The k_[] parameters control the cross-validation per sample. They give the number of crossvalidation folds to be created along this dimension. If a k is set to 1 no crossvalidation is performed over the corresponding dimension. by default ks are set by rsatoolbox.util.inference_util.default_k_pattern and rsatoolbox.util.inference_util.default_k_rdm based on the number of rdms and patterns provided. the ks are then in the range 2-5.

Using the []_descriptor inputs you may make the crossvalidation and bootstrap aware of groups of rdms or conditions to be handled en block. Conditions with the same entry will be sampled in or out of the bootstrap together and will be assigned to cross-calidation folds together.

Using the boot_type argument you may choose the dimension to bootstrap. By default both conditions and RDMs are resampled. You may alternatively choose to resample only one of them by passing ‘rdm’ or ‘pattern’.

models should be a list of models. data the RDMs object to evaluate against method the method for comparing the predictions and the data. fitter may provide a non-default funcion or list of functions to fit the models.

Parameters:

• models (rsatoolbox.model.Model) – models to be evaluated

• data (rsatoolbox.rdm.RDMs) – RDM data to use

• method (string) – comparison method to use

• fitter (function) – fitting method for models

• k_pattern (int) – #folds over patterns

• k_rdm (int) – #folds over rdms

• N (int) – number of bootstrap samples (default: 1000)

• n_cv (int) – number of crossvalidation runs per sample (default: 1)

• pattern_descriptor (string) – descriptor to group patterns

• rdm_descriptor (string) – descriptor to group rdms

• boot_type (String) – which dimension to bootstrap over (default: ‘both’) alternatives: ‘rdm’, ‘pattern’

• use_correction (bool) – switch for the correction for the variance caused by crossvalidation (default: True)

Returns:

matrix of evaluations (N x k)

Return type:

numpy.ndarray

rsatoolbox.inference.evaluate.crossval(models, rdms, train_set, test_set, ceil_set=None, method='cosine', fitter=None, pattern_descriptor='index', calc_noise_ceil=True)

evaluates models on cross-validation sets

Parameters:

• models (rsatoolbox.model.Model) – models to be evaluated

• rdms (rsatoolbox.rdm.RDMs) – full dataset

• train_set (list) – a list of the training RDMs with 2-tuple entries: (RDMs, pattern_idx)

• test_set (list) – a list of the test RDMs with 2-tuple entries: (RDMs, pattern_idx)

• method (string) – comparison method to use

• pattern_descriptor (string) – descriptor to group patterns

Returns:

vector of evaluations

Return type:

numpy.ndarray

rsatoolbox.inference.evaluate.eval_bootstrap(models, data, theta=None, method='cosine', N=1000, pattern_descriptor='index', rdm_descriptor='index', boot_noise_ceil=True)

evaluates models on data performs bootstrapping to get a sampling distribution

Parameters:

• models (rsatoolbox.model.Model or list) – models to be evaluated

• data (rsatoolbox.rdm.RDMs) – data to evaluate on

• theta (numpy.ndarray) – parameter vector for the models

• method (string) – comparison method to use

• N (int) – number of samples

• pattern_descriptor (string) – descriptor to group patterns for bootstrap

• rdm_descriptor (string) – descriptor to group rdms for bootstrap

Returns:

vector of evaluations

Return type:

numpy.ndarray

rsatoolbox.inference.evaluate.eval_bootstrap_pattern(models, data, theta=None, method='cosine', N=1000, pattern_descriptor='index', rdm_descriptor='index', boot_noise_ceil=True)

evaluates a models on data performs bootstrapping over patterns to get a sampling distribution

Parameters:

• models (rsatoolbox.model.Model or list) – models to be evaluated

• data (rsatoolbox.rdm.RDMs) – data to evaluate on

• theta (numpy.ndarray) – parameter vector for the models

• method (string) – comparison method to use

• N (int) – number of samples

• pattern_descriptor (string) – descriptor to group patterns for bootstrap

• rdm_descriptor (string) – descriptor to group patterns for noise ceiling calculation

Returns:

vector of evaluations

Return type:

numpy.ndarray

rsatoolbox.inference.evaluate.eval_bootstrap_rdm(models, data, theta=None, method='cosine', N=1000, rdm_descriptor='index', boot_noise_ceil=True)

evaluates models on data performs bootstrapping to get a sampling distribution

Parameters:

• models (rsatoolbox.model.Model or list of these) – models to be evaluated

• data (rsatoolbox.rdm.RDMs) – data to evaluate on

• theta (numpy.ndarray) – parameter vector for the models

• method (string) – comparison method to use

• N (int) – number of samples

• rdm_descriptor (string) – rdm_descriptor to group rdms for bootstrap

Returns:

vector of evaluations

Return type:

numpy.ndarray

rsatoolbox.inference.evaluate.eval_dual_bootstrap(models, data, method='cosine', fitter=None, k_pattern=1, k_rdm=1, N=1000, n_cv=2, pattern_descriptor='index', rdm_descriptor='index', use_correction=True)

dual bootstrap evaluation of models i.e. models are evaluated in a bootstrap over rdms, one over patterns and a bootstrap over both using the same bootstrap samples for each. The variance estimates from these bootstraps are then combined into a better overall estimate for the variance.

This method allows the incorporation of crossvalidation inside the bootstrap to handle fitted models. To activate this set k_rdm and k_pattern as described below.

Crossvalidation creates variance in the results for a single bootstrap sample, because different assginments to the training and test group lead to different results. To correct for this, we apply a formula which estimates the variance we expect if we evaluated all possible crossvalidation assignments from n_cv different assignments per bootstrap sample. In our statistical evaluations we saw that many bootstrap samples and few different crossvalidation assignments are optimal to minimize the variance of the variance estimate. Thus, this function by default applies this correction formula and sets n_cv=2, i.e. performs only two different assignments per fold. This function nonetheless performs full crossvalidation schemes, i.e. in every bootstrap sample all crossvalidation folds are evaluated such that each RDM and each condition is in the test set n_cv times.

The k_[] parameters control the cross-validation per sample. They give the number of crossvalidation folds to be created along this dimension. If a k is set to 1 no crossvalidation is performed over the corresponding dimension. by default ks are set by rsatoolbox.util.inference_util.default_k_pattern and rsatoolbox.util.inference_util.default_k_rdm based on the number of rdms and patterns provided. the ks are then in the range 2-5.

Using the []_descriptor inputs you may make the crossvalidation and bootstrap aware of groups of rdms or conditions to be handled en block. Conditions with the same entry will be sampled in or out of the bootstrap together and will be assigned to cross-calidation folds together.

models should be a list of models. data the RDMs object to evaluate against method the method for comparing the predictions and the data. fitter may provide a non-default funcion or list of functions to fit the models.

Parameters:

• models (rsatoolbox.model.Model) – models to be evaluated

• data (rsatoolbox.rdm.RDMs) – RDM data to use

• method (string) – comparison method to use

• fitter (function) – fitting method for models

• k_pattern (int) – #folds over patterns

• k_rdm (int) – #folds over rdms

• N (int) – number of bootstrap samples (default: 1000)

• n_cv (int) – number of crossvalidation runs per sample (default: 1)

• pattern_descriptor (string) – descriptor to group patterns

• rdm_descriptor (string) – descriptor to group rdms

• random (bool) – randomize group assignments (default: True)

• boot_type (String) – which dimension to bootstrap over (default: ‘both’) alternatives: ‘rdm’, ‘pattern’

• use_correction (bool) – switch for the correction for the variance caused by crossvalidation (default: True)

Returns:

matrix of evaluations (N x k)

Return type:

numpy.ndarray

rsatoolbox.inference.evaluate.eval_dual_bootstrap_random(models, data, method='cosine', fitter=None, n_pattern=None, n_rdm=None, N=1000, n_cv=2, pattern_descriptor='index', rdm_descriptor='index', boot_type='both', use_correction=True)

evaluates a set of models by a evaluating a few random crossvalidation folds per bootstrap.

If a k is set to 1 no crossvalidation is performed over the corresponding dimension.

As especially crossvalidation over patterns/conditions creates variance in the cv result for a single variance the default setting of n_cv=1 inflates the estimated variance. Setting this value higher will decrease this effect at the cost of more computation time.

by default ks are set by rsatoolbox.util.inference_util.default_k_pattern and rsatoolbox.util.inference_util.default_k_rdm based on the number of rdms and patterns provided. the ks are then in the range 2-5.

Parameters:

• models (rsatoolbox.model.Model) – models to be evaluated

• data (rsatoolbox.rdm.RDMs) – RDM data to use

• method (string) – comparison method to use

• fitter (function) – fitting method for models

• k_pattern (int) – #folds over patterns

• k_rdm (int) – #folds over rdms

• N (int) – number of bootstrap samples (default: 1000)

• n_cv (int) – number of crossvalidation runs per sample (default: 1)

• pattern_descriptor (string) – descriptor to group patterns

• rdm_descriptor (string) – descriptor to group rdms

• boot_type (String) – which dimension to bootstrap over (default: ‘both’) alternatives: ‘rdm’, ‘pattern’

• use_correction (bool) – switch for the correction for the variance caused by crossvalidation (default: True)

Returns:

matrix of evaluations (N x k)

Return type:

numpy.ndarray

rsatoolbox.inference.evaluate.eval_fixed(models, data, theta=None, method='cosine')

evaluates models on data, without any bootstrapping or cross-validation

Parameters:

• models (list of rsatoolbox.model.Model or list) – models to be evaluated

• data (rsatoolbox.rdm.RDMs) – data to evaluate on

• theta (numpy.ndarray) – parameter vector for the models

• method (string) – comparison method to use

Returns:

evaluation

Return type:

float