Our postdoc, Ekaterina Pronizius, was part of the lead team that developed a step-by-step tutorial on conducting multiverse analyses through crowdsourcing, published in Psychological Methods.

When researchers process and analyze empirical data, they regularly face decisions, such as how to define or handle outliers, that may seem minor but can substantially influence results. Relying on a single analytic pathway can limit the usefulness and generalizability of findings because alternative, equally plausible approaches remain unexplored.

Multiverse analysis addresses this issue by systematically exploring multiple reasonable analytic choices to assess how these decisions affect conclusions. Yet even multiverse analyses can be biased if researchers selectively include or exclude certain options. To reduce this risk and enhance transparency, Dr. Pronizius and colleagues outline a novel, crowdsourcing-based approach that makes multiverse analyses more objective and robust.

Their step-by-step tutorial provides clear guidance for implementing this method, and a worked-out example, the Semantic Priming Across Many Languages project, demonstrates both its feasibility and its potential to improve rigor in psychological research.

Link to the article: 

Heyman, T., Pronizius, E., Lewis, S. C., Acar, O. A., Adamkovič, M., Ambrosini, E., Antfolk, J., Barzykowski, K., Baskin, E., Batres, C., Boucher, L., Boudesseul, J., Brandstätter, E., Collins, W. M., Filipović Ðurđević, D., Egan, C., Era, V., Ferreira, P., Fini, C., . . . Buchanan, E. M. (2025). Crowdsourcing multiverse analyses to explore the impact of different data-processing and analysis decisions: A tutorial. Psychological Methods. Advance online publication. doi.org/10.1037/met0000770

(c) image: pixabay.com/illustrations/ai-generated-telescope-observatory-9398107/