rsatoolbox.util.inference_util module¶
Inference module utilities
- rsatoolbox.util.inference_util.all_tests(evaluations: NDArray, noise_ceil: NDArray, test_type: str = 't-test', model_var: NDArray | None = None, diff_var: NDArray | None = None, noise_ceil_var: NDArray | None = None, dof: int = 1)[source]¶
wrapper running all tests necessary for the model plot -> pairwise tests, tests against 0 and against noise ceiling
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)
noise_ceil (numpy.ndarray) – noise_ceiling estimate(s) to compare against
test_type (String) – ‘t-test’ : t-test bases tests using variances ‘bootstrap’ : Direct bootstrap sample based tests ‘ranksum’ : Wilcoxon signed rank-sum tests
model_var – variance estimates from the results object. These correspond to the variances of the individual model evals, pairs of model evals and differences from the noise ceiling.
diff_var – variance estimates from the results object. These correspond to the variances of the individual model evals, pairs of model evals and differences from the noise ceiling.
noise_ceil_var – variance estimates from the results object. These correspond to the variances of the individual model evals, pairs of model evals and differences from the noise ceiling.
- Returns:
p_pairwise, p_zero, p_noise
- Return type:
numpy.ndarrays
- rsatoolbox.util.inference_util.bootstrap_pair_tests(evaluations)[source]¶
pairwise bootstrapping significance tests for a difference in model performance. Tests add 1/len(evaluations) to each p-value and are computed as two sided tests, i.e. as 2 * the smaller proportion
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be compared
- Returns:
- matrix of proportions of opposit conclusions, i.e.
p-values for the bootstrap test
- Return type:
numpy.ndarray
- rsatoolbox.util.inference_util.default_k_pattern(n_pattern)[source]¶
the default number of pattern divisions for crossvalidation minimum number of patterns is 3*k_pattern. Thus for n_pattern <=9 this returns 2. From there it grows gradually until 5 groups are made for 40 patterns. From this point onwards the number of groups is kept at 5.
bootstrapped crossvalidation also uses this function to set k, but scales n_rdm to the expected proportion of samples retained when bootstrapping (1-np.exp(-1))
- rsatoolbox.util.inference_util.default_k_rdm(n_rdm)[source]¶
the default number of rdm groupsfor crossvalidation minimum number of subjects is k_rdm. We switch to more groups whenever the groups all contain more rdms, e.g. we make 3 groups of 2 instead of 2 groups of 3. We follow this scheme until we reach 5 groups of 4. From there on this function returns 5 groups forever.
bootstrapped crossvalidation also uses this function to set k, but scales n_rdm to the expected proportion of samples retained when bootstrapping (1-np.exp(-1))
- rsatoolbox.util.inference_util.extract_variances(variance, nc_included: bool = True, n_rdm: int | None = None, n_pattern: int | None = None)[source]¶
extracts the variances for the individual model evaluations, differences between model evaluations and for the comparison to the noise ceiling
for 1D arrays we assume a diagonal covariance is meant
for 2D arrays this is taken as the covariance of the model evals
for 3D arrays we assume this is the result of a dual bootstrap and perform the correction. Then there should be three covariances given from double, rdm & pattern bootstrap in that order.
nc_included=True jields the result if the last two columns correspond to the noise ceiling results
nc_included=False assumes that the noise ceiling is fixed instead.
To get the more accurate estimates that take into account the number of subjects and/or the numbers of stimuli can be passed as n_rdm and n_pattern respectively. This function corrects for all ns that are passed. If you bootstrapped only one factor only pass the N for that factor!
- rsatoolbox.util.inference_util.get_errorbars(model_var, evaluations, dof, error_bars='sem', test_type='t-test')[source]¶
computes errorbars for the model-evaluations from a results object
- Parameters:
results – rsatoolbox.inference.Results the results object
eb_type – str, optional which errorbars to compute
- Returns:
limits (np.array)
- rsatoolbox.util.inference_util.input_check_model(models, theta=None, fitter=None, N=1)[source]¶
Checks whether model related inputs to evaluations are valid and generates an evaluation-matrix of fitting size.
- Parameters:
model – [list of] rsatoolbox.rdm.RDMs the models to be evaluated
theta – numpy.ndarray or list , optional Parameter(s) for the model(s). The default is None.
fitter – [list of] function, optional fitting function to overwrite the model default. The default is None, i.e. keep default
N – int, optional number of samples/rows in evaluations matrix. The default is 1.
- Returns:
- numpy.ndarray
empty evaluations-matrix
- thetalist
the processed and checked model parameters
- fitter[list of] functions
checked and processed fitter functions
- Return type:
evaluations
- rsatoolbox.util.inference_util.nc_tests(evaluations, noise_ceil, test_type='t-test', noise_ceil_var=None, dof=1)[source]¶
wrapper running tests against noise ceiling
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)
noise_ceil (numpy.ndarray) – noise_ceiling estimate(s) to compare against
test_type (Strinng) – ‘t-test’ : t-test bases tests using variances ‘bootstrap’ : Direct bootstrap sample based tests ‘ranksum’ : Wilcoxon signed rank-sum tests
noise_ceil_var – variance estimate from the results object. These correspond to the variances of the differences from the noise ceiling.
- Returns:
p_noise
- Return type:
numpy.ndarrays
- rsatoolbox.util.inference_util.pair_tests(evaluations: NDArray, test_type: str = 't-test', diff_var: NDArray | None = None, dof: int = 1)[source]¶
wrapper running pair tests
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)
test_type (Strinng) – ‘t-test’ : t-test bases tests using variances ‘bootstrap’ : Direct bootstrap sample based tests ‘ranksum’ : Wilcoxon signed rank-sum tests
- Returns:
p_pairwise
- Return type:
numpy.ndarray
- rsatoolbox.util.inference_util.pool_rdm(rdms, method: str = 'cosine')[source]¶
pools multiple RDMs into the one with maximal performance under a given evaluation metric rdm_descriptors of the generated rdms are empty
- Parameters:
rdms (rsatoolbox.rdm.RDMs) – RDMs to be pooled
method – String, optional Which comparison method to optimize for. The default is ‘cosine’.
- Returns:
- the pooled RDM, i.e. a RDM with maximal performance
under the chosen method
- Return type:
rsatoolbox.rdm.RDMs
- rsatoolbox.util.inference_util.ranksum_pair_test(evaluations)[source]¶
pairwise tests between models using the wilcoxon signed rank test
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)
- Returns:
- matrix of proportions of opposit conclusions, i.e.
p-values for the test
- Return type:
numpy.ndarray
- rsatoolbox.util.inference_util.ranksum_value_test(evaluations, comp_value=0)[source]¶
nonparametric wilcoxon signed rank test against a fixed value
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)
comp_value (float) – value to compare against
- Returns:
p-value
- Return type:
float
- rsatoolbox.util.inference_util.t_test_0(evaluations, variances, dof=1)[source]¶
t-tests against 0 performance.
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be tested, typically from a results object
variances (numpy.ndarray) – vector of model evaluation variances
dof (integer) – degrees of freedom used for the test (default=1)
- Returns:
p-values for the raw t-test of each model against 0.
- Return type:
numpy.ndarray
- rsatoolbox.util.inference_util.t_test_nc(evaluations, variances, noise_ceil, dof=1)[source]¶
t-tests against noise_ceiling. Technically this can be used to test evaluations against any fixed number.
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be tested, typically from a results object
variances (numpy.ndarray) – variance estimates for the comparisons to the noise ceiling
noise_ceil (float) – the average noise ceiling to test against.
noise_ceil_var (numpy.ndarray) – variance or covariance of the noise ceiling
dof (integer) – degrees of freedom used for the test (default=1)
- Returns:
p-values for the raw t-test of each model against the noise ceiling.
- Return type:
numpy.ndarray
- rsatoolbox.util.inference_util.t_tests(evaluations, variances, dof=1)[source]¶
pairwise t_test based significant tests for a difference in model performance
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be tested, typically from a results object
variances (numpy.ndarray) – vector of the variances of model evaluation differences
dof (integer) – degrees of freedom used for the test (default=1)
- Returns:
matrix of p-values for the test
- Return type:
numpy.ndarray
- rsatoolbox.util.inference_util.zero_tests(evaluations, test_type='t-test', model_var=None, dof=1)[source]¶
wrapper running tests against 0
- Parameters:
evaluations (numpy.ndarray) – model evaluations to be compared (should be 3D: bootstrap x models x subjects or repeats)
test_type (Strinng) – ‘t-test’ : t-test bases tests using variances ‘bootstrap’ : Direct bootstrap sample based tests ‘ranksum’ : Wilcoxon signed rank-sum tests
- Returns:
p_zero
- Return type:
numpy.ndarray