rsatoolbox.util.inference_util module

Inference module utilities

rsatoolbox.util.inference_util.all_tests(evaluations: NDArray, noise_ceil: NDArray, test_type: str = 't-test', model_var: NDArray | None = None, diff_var: NDArray | None = None, noise_ceil_var: NDArray | None = None, dof: int = 1)

wrapper running all tests necessary for the model plot -> pairwise tests, tests against 0 and against noise ceiling

Parameters:

• evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)

• noise_ceil (numpy.ndarray) – noise_ceiling estimate(s) to compare against

• test_type (String) – ‘t-test’ : t-test bases tests using variances ‘bootstrap’ : Direct bootstrap sample based tests ‘ranksum’ : Wilcoxon signed rank-sum tests

• model_var – variance estimates from the results object. These correspond to the variances of the individual model evals, pairs of model evals and differences from the noise ceiling.

• diff_var – variance estimates from the results object. These correspond to the variances of the individual model evals, pairs of model evals and differences from the noise ceiling.

• noise_ceil_var – variance estimates from the results object. These correspond to the variances of the individual model evals, pairs of model evals and differences from the noise ceiling.

Returns:

p_pairwise, p_zero, p_noise

Return type:

numpy.ndarrays

rsatoolbox.util.inference_util.bootstrap_pair_tests(evaluations)

pairwise bootstrapping significance tests for a difference in model performance. Tests add 1/len(evaluations) to each p-value and are computed as two sided tests, i.e. as 2 * the smaller proportion

Parameters:

evaluations (numpy.ndarray) – model evaluations to be compared

Returns:

matrix of proportions of opposit conclusions, i.e.

p-values for the bootstrap test

Return type:

numpy.ndarray

rsatoolbox.util.inference_util.default_k_pattern(n_pattern)

the default number of pattern divisions for crossvalidation minimum number of patterns is 3*k_pattern. Thus for n_pattern <=9 this returns 2. From there it grows gradually until 5 groups are made for 40 patterns. From this point onwards the number of groups is kept at 5.

bootstrapped crossvalidation also uses this function to set k, but scales n_rdm to the expected proportion of samples retained when bootstrapping (1-np.exp(-1))

rsatoolbox.util.inference_util.default_k_rdm(n_rdm)

the default number of rdm groupsfor crossvalidation minimum number of subjects is k_rdm. We switch to more groups whenever the groups all contain more rdms, e.g. we make 3 groups of 2 instead of 2 groups of 3. We follow this scheme until we reach 5 groups of 4. From there on this function returns 5 groups forever.

bootstrapped crossvalidation also uses this function to set k, but scales n_rdm to the expected proportion of samples retained when bootstrapping (1-np.exp(-1))

rsatoolbox.util.inference_util.extract_variances(variance, nc_included: bool = True, n_rdm: int | None = None, n_pattern: int | None = None)

extracts the variances for the individual model evaluations, differences between model evaluations and for the comparison to the noise ceiling

for 1D arrays we assume a diagonal covariance is meant

for 2D arrays this is taken as the covariance of the model evals

for 3D arrays we assume this is the result of a dual bootstrap and perform the correction. Then there should be three covariances given from double, rdm & pattern bootstrap in that order.

nc_included=True jields the result if the last two columns correspond to the noise ceiling results

nc_included=False assumes that the noise ceiling is fixed instead.

To get the more accurate estimates that take into account the number of subjects and/or the numbers of stimuli can be passed as n_rdm and n_pattern respectively. This function corrects for all ns that are passed. If you bootstrapped only one factor only pass the N for that factor!

rsatoolbox.util.inference_util.get_errorbars(model_var, evaluations, dof, error_bars='sem', test_type='t-test')

computes errorbars for the model-evaluations from a results object

Parameters:

• results – rsatoolbox.inference.Results the results object

• eb_type – str, optional which errorbars to compute

Returns:

limits (np.array)

rsatoolbox.util.inference_util.input_check_model(models, theta=None, fitter=None, N=1)

Checks whether model related inputs to evaluations are valid and generates an evaluation-matrix of fitting size.

Parameters:

• model – [list of] rsatoolbox.rdm.RDMs the models to be evaluated

• theta – numpy.ndarray or list , optional Parameter(s) for the model(s). The default is None.

• fitter – [list of] function, optional fitting function to overwrite the model default. The default is None, i.e. keep default

• N – int, optional number of samples/rows in evaluations matrix. The default is 1.

Returns:

numpy.ndarray

empty evaluations-matrix

thetalist

the processed and checked model parameters

fitter[list of] functions

checked and processed fitter functions

Return type:

evaluations

rsatoolbox.util.inference_util.nc_tests(evaluations, noise_ceil, test_type='t-test', noise_ceil_var=None, dof=1)

wrapper running tests against noise ceiling

Parameters:

• evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)

• noise_ceil (numpy.ndarray) – noise_ceiling estimate(s) to compare against

• test_type (Strinng) – ‘t-test’ : t-test bases tests using variances ‘bootstrap’ : Direct bootstrap sample based tests ‘ranksum’ : Wilcoxon signed rank-sum tests

• noise_ceil_var – variance estimate from the results object. These correspond to the variances of the differences from the noise ceiling.

Returns:

p_noise

Return type:

numpy.ndarrays

rsatoolbox.util.inference_util.pair_tests(evaluations: NDArray, test_type: str = 't-test', diff_var: NDArray | None = None, dof: int = 1)

wrapper running pair tests

Parameters:

• evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)

• test_type (Strinng) – ‘t-test’ : t-test bases tests using variances ‘bootstrap’ : Direct bootstrap sample based tests ‘ranksum’ : Wilcoxon signed rank-sum tests

Returns:

p_pairwise

Return type:

numpy.ndarray

rsatoolbox.util.inference_util.pool_rdm(rdms, method: str = 'cosine')

pools multiple RDMs into the one with maximal performance under a given evaluation metric rdm_descriptors of the generated rdms are empty

Parameters:

• rdms (rsatoolbox.rdm.RDMs) – RDMs to be pooled

• method – String, optional Which comparison method to optimize for. The default is ‘cosine’.

Returns:

the pooled RDM, i.e. a RDM with maximal performance

under the chosen method

Return type:

rsatoolbox.rdm.RDMs

rsatoolbox.util.inference_util.ranksum_pair_test(evaluations)

pairwise tests between models using the wilcoxon signed rank test

Parameters:

evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)

Returns:

matrix of proportions of opposit conclusions, i.e.

p-values for the test

Return type:

numpy.ndarray

rsatoolbox.util.inference_util.ranksum_value_test(evaluations, comp_value=0)

nonparametric wilcoxon signed rank test against a fixed value

Parameters:

• evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)

• comp_value (float) – value to compare against

Returns:

p-value

Return type:

float

rsatoolbox.util.inference_util.t_test_0(evaluations, variances, dof=1)

t-tests against 0 performance.

Parameters:

• evaluations (numpy.ndarray) – model evaluations to be tested, typically from a results object

• variances (numpy.ndarray) – vector of model evaluation variances

• dof (integer) – degrees of freedom used for the test (default=1)

Returns:

p-values for the raw t-test of each model against 0.

Return type:

numpy.ndarray

rsatoolbox.util.inference_util.t_test_nc(evaluations, variances, noise_ceil, dof=1)

t-tests against noise_ceiling. Technically this can be used to test evaluations against any fixed number.

Parameters:

• evaluations (numpy.ndarray) – model evaluations to be tested, typically from a results object

• variances (numpy.ndarray) – variance estimates for the comparisons to the noise ceiling

• noise_ceil (float) – the average noise ceiling to test against.

• noise_ceil_var (numpy.ndarray) – variance or covariance of the noise ceiling

• dof (integer) – degrees of freedom used for the test (default=1)

Returns:

p-values for the raw t-test of each model against the noise ceiling.

Return type:

numpy.ndarray

rsatoolbox.util.inference_util.t_tests(evaluations, variances, dof=1)

pairwise t_test based significant tests for a difference in model performance

Parameters:

• evaluations (numpy.ndarray) – model evaluations to be tested, typically from a results object

• variances (numpy.ndarray) – vector of the variances of model evaluation differences

• dof (integer) – degrees of freedom used for the test (default=1)

Returns:

matrix of p-values for the test

Return type:

numpy.ndarray

rsatoolbox.util.inference_util.zero_tests(evaluations, test_type='t-test', model_var=None, dof=1)

wrapper running tests against 0

Parameters:

• evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)

• test_type (Strinng) – ‘t-test’ : t-test bases tests using variances ‘bootstrap’ : Direct bootstrap sample based tests ‘ranksum’ : Wilcoxon signed rank-sum tests

Returns:

p_zero

Return type:

numpy.ndarray