rsatoolbox.util.inference_util module

Inference module utilities

rsatoolbox.util.inference_util.all_tests(evaluations, noise_ceil, test_type='t-test', model_var=None, diff_var=None, noise_ceil_var=None, dof=1)[source]

wrapper running all tests necessary for the model plot -> pairwise tests, tests against 0 and against noise ceiling

Parameters
  • evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)

  • noise_ceil (numpy.ndarray) – noise_ceiling estimate(s) to compare against

  • test_type (Strinng) – ‘t-test’ : t-test bases tests using variances ‘bootstrap’ : Direct bootstrap sample based tests ‘ranksum’ : Wilcoxon signed rank-sum tests

Returns

p_pairwise, p_zero, p_noise

Return type

numpy.ndarrays

rsatoolbox.util.inference_util.default_k_pattern(n_pattern)[source]

the default number of pattern divisions for crossvalidation minimum number of patterns is 3*k_pattern. Thus for n_pattern <=9 this returns 2. From there it grows gradually until 5 groups are made for 40 patterns. From this point onwards the number of groups is kept at 5.

bootstrapped crossvalidation also uses this function to set k, but scales n_rdm to the expected proportion of samples retained when bootstrapping (1-np.exp(-1))

rsatoolbox.util.inference_util.default_k_rdm(n_rdm)[source]

the default number of rdm groupsfor crossvalidation minimum number of subjects is k_rdm. We switch to more groups whenever the groups all contain more rdms, e.g. we make 3 groups of 2 instead of 2 groups of 3. We follow this scheme until we reach 5 groups of 4. From there on this function returns 5 groups forever.

bootstrapped crossvalidation also uses this function to set k, but scales n_rdm to the expected proportion of samples retained when bootstrapping (1-np.exp(-1))

rsatoolbox.util.inference_util.extract_variances(variance, nc_included=True)[source]

extracts the variances for the individual model evaluations, differences between model evaluations and for the comparison to the noise ceiling

for 1D arrays we assume a diagonal covariance is meant

for 2D arrays this is taken as the covariance of the model evals

for 3D arrays we assume this is the result of a dual bootstrap and perform the correction. Then there should be three covariances given from double, rdm & pattern bootstrap in that order.

nc_included=True jields the result if the last two columns correspond to the noise ceiling results

nc_included=False assumes that the noise ceiling is fixed instead.

rsatoolbox.util.inference_util.input_check_model(models, theta=None, fitter=None, N=1)[source]

Checks whether model related inputs to evaluations are valid and generates an evaluation-matrix of fitting size.

Parameters
  • model – [list of] rsatoolbox.rdm.RDMs the models to be evaluated

  • theta – numpy.ndarray or list , optional Parameter(s) for the model(s). The default is None.

  • fitter – [list of] function, optional fitting function to overwrite the model default. The default is None, i.e. keep default

  • N – int, optional number of samples/rows in evaluations matrix. The default is 1.

Returns

numpy.ndarray

empty evaluations-matrix

thetalist

the processed and checked model parameters

fitter[list of] functions

checked and processed fitter functions

Return type

evaluations

rsatoolbox.util.inference_util.pair_tests(evaluations)[source]

pairwise bootstrapping significance tests for a difference in model performance. Tests add 1/len(evaluations) to each p-value and are computed as two sided tests, i.e. as 2 * the smaller proportion

Parameters

evaluations (numpy.ndarray) – model evaluations to be compared

Returns

matrix of proportions of opposit conclusions, i.e.

p-values for the bootstrap test

Return type

numpy.ndarray

rsatoolbox.util.inference_util.pool_rdm(rdms, method='cosine')[source]

pools multiple RDMs into the one with maximal performance under a given evaluation metric rdm_descriptors of the generated rdms are empty

Parameters
  • rdms (rsatoolbox.rdm.RDMs) – RDMs to be pooled

  • method – String, optional Which comparison method to optimize for. The default is ‘cosine’.

Returns

the pooled RDM, i.e. a RDM with maximal performance

under the chosen method

Return type

rsatoolbox.rdm.RDMs

rsatoolbox.util.inference_util.ranksum_pair_test(evaluations)[source]

pairwise tests between models using the wilcoxon signed rank test

Parameters

evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)

Returns

matrix of proportions of opposit conclusions, i.e.

p-values for the test

Return type

numpy.ndarray

rsatoolbox.util.inference_util.ranksum_value_test(evaluations, comp_value=0)[source]

nonparametric wilcoxon signed rank test against a fixed value

Parameters
  • evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)

  • comp_value (float) – value to compare against

Returns

p-value

Return type

float

rsatoolbox.util.inference_util.t_test_0(evaluations, variances, dof=1)[source]

t-tests against 0 performance.

Parameters
  • evaluations (numpy.ndarray) – model evaluations to be tested, typically from a results object

  • variances (numpy.ndarray) – vector of model evaluation variances

  • dof (integer) – degrees of freedom used for the test (default=1)

Returns

p-values for the raw t-test of each model against 0.

Return type

numpy.ndarray

rsatoolbox.util.inference_util.t_test_nc(evaluations, variances, noise_ceil, dof=1)[source]

t-tests against noise_ceiling. Technically this can be used to test evaluations against any fixed number.

Parameters
  • evaluations (numpy.ndarray) – model evaluations to be tested, typically from a results object

  • variances (numpy.ndarray) – variance estimates for the comparisons to the noise ceiling

  • noise_ceil (float) – the average noise ceiling to test against.

  • noise_ceil_var (numpy.ndarray) – variance or covariance of the noise ceiling

  • dof (integer) – degrees of freedom used for the test (default=1)

Returns

p-values for the raw t-test of each model against the noise ceiling.

Return type

numpy.ndarray

rsatoolbox.util.inference_util.t_tests(evaluations, variances, dof=1)[source]

pairwise t_test based significant tests for a difference in model performance

Parameters
  • evaluations (numpy.ndarray) – model evaluations to be tested, typically from a results object

  • variances (numpy.ndarray) – vector of the variances of model evaluation differences

  • dof (integer) – degrees of freedom used for the test (default=1)

Returns

matrix of p-values for the test

Return type

numpy.ndarray