Metascience – reproducibilitynetwork

15 July 202515 July 2025

Preregistration For Student Assignments

How can we integrate open science practices into our curricula? In this webinar two lecturers told us how they are including preregistrations in their students’ curricula.

“When you preregister your research, you’re simply specifying your research plan in advance of your study and submitting it to a registry.” (Quote from the website of the Open Science Framework)

Ewout Meijer started the webinar and told us how he convinced thesis coordinators to include pre registration in their courses. As an Open Science Ambassador for his faculty, Ewout got backed by the dean to find out where in the various courses open science practices could be integrated. Pre registrations are just one example of those practices.

He shared several tips of how he convinced colleagues to integrate open science practices:

Make it easy for your colleagues – minimize the extra workload by sharing templates and offering introduction lecture materials
Make them want it – top down mandates to include open science practices don’t work well, but if your colleagues are convinced that this is the right thing to do, they will follow
Don’t be overambitious – first the science, then open science. Finding the balance between what students need to know in terms of scientific content and what they need to know about the scientific system is difficult and will differ between courses and student populations.
Searching for “project proposals” as a required component for students to pass a course is a good way to find courses where preregistration can be taught. You just need to replace the proposal with a registration.
Including new (open science) content means kicking out some other content – see what can be replaced and what can be tossed

At Maastricht University, students are asked to use the AsPredicted template and submit that as a pdf (i.e., they don’t upload it on aspredicted.com). Ewout mentioned that not all internship projects are suited for this format, so students might have to adjust it or come up with a project just to fill in the template and pass this grading component.

Students get exposed to the idea of preregistration and the same effect goes for workgroup tutors. Tutors come from a wide range of research groups and are learning themselves about pre registrations while helping students with their thesis work.

Elen Le Foll asked her seminar students to pre-register their term paper analyses. Adding this component required some extra time investment to make sure students understood what was expected of them and for extra feedback rounds. The preregistration adds at least one round of feedback to the term paper and requires students to plan ahead and submit their preregistration on time to have enough time left to incorporate the feedback into their final data analysis. On the positive side, students can learn from feedback and include it in their work. For normal term papers, students get feedback at the end but do not need to use that feedback or cannot improve their work anymore.

As Elen’s course is an advanced course for master students, some of her students want to turn their term paper into a research paper. For them, the preregistration is an excellent way to get a timestamp for their analysis.

In the discussion with Andrea and other attendees, we discussed how the AsPredicted format can be used by students and if a full registered report might be even more suitable. We briefly touched upon the difficulty of grading pre registrations and how much detail we should ask students. Another point of discussion was how we can sell pre registration to students who are not interested in becoming researchers. This led to a discussion on how to balance the need for academic training with content and application outside of academia.

Thanks to our presentors:

Elen Le Foll is a post-doctoral researcher and lecturer in linguistics at the Department of Romance Studies at the University of Cologne. She likes to integrate Open Science principles and practices in her seminars and recently asked her students to pre-register a study as part of a term paper assignment.

Ewout Meijer works at Maastricht University and coordinates the thesis module for the research master in psychology. He introduced preregistrations for thesis projects.

Useful Links:

Aspredicted: aspredicted.org

OSF Preregistration Templates: www.cos.io/initiatives/prereg

20 February 202516 March 2025

Platform for Young Meta-Scientists (PYMS): Discussing the Future of Meta-Science

Blogpost by Cas Goos

The goal of our organization, the Platform for Young Meta-Scientists (PYMS), is to bring together Early Career Researchers (ECRs) working on meta-science, while providing a place to network with peers and discuss research. We consider this an important initiative as ECRs have been at the forefront of many reform initiatives, and because despite the increasing number of meta-science ECRs, many still work disconnected from other researchers with shared interests. Bringing ECRs together is a crucial step in enabling large scale research and strengthening reform initiatives that are key to improving science.

The PYMS Meeting

To further these goals within the most recent PYMS meeting, which took place at the UMCG on December 5^th 2024 as a pre-symposium to the NLRN symposium on December 6^th, we invited an audience of ECRs from a variety of backgrounds. We are glad to have achieved a broad representation of ECRs, including from abroad. Presentations span a variety of meta-science topics such as spin, equivalence testing, and reproducibility. Not only work from the social sciences, but also work in computational sciences, sports and exercise science, and nanobioscience were presented and discussed.

Map of PYMS attendees and their areas of expertise/interest. Map backdrop was constructed by Tamarinde Haven.

Furthermore, during the meeting we created many opportunities for discussion and networking, including invitations for collaborations, throughout the day.

We were also fortunate enough to have Tracey Weissgerber give a keynote talk on having a career in meta-science followed by a series of provocative statements on the same topic. Both led to a practically informative discussion for the attendees on having a career in meta-science.

The Future of PYMS

Since the end of last year, the board for PYMS has been renewed. Four new ECRs have joined the board to carry the torch. The new members are Sajedeh Rasti from Eindhoven University, Raphael Merz from Ruhr University Bochum, and Anouk Bouma and myself from Tilburg University. Like the previous board, we plan to continue creating a community for meta-science ECRs with informal networking opportunities. However, we also plan to expand the reach of PYMS in new ways.

As a first step, PYMS will go international during our next meeting. In this way we expand our networking beyond the Netherlands to prompt broader collaboration in our field. This is especially important here, since projects requiring large investments of time and effort from multiple parties are of crucial importance to investigate and improve scientific practice effectively. This meeting will be held this summer, there will be plenty of opportunities to discuss future projects and financial compensation for international presenters to encourage a broad representation of young meta-scientists.

Interested ECRs can join our mailing list and get a link through there to our Discord server to connect and communicate with fellow ECRs on meta-science, as well as keep up to date about future events, including the next PYMS meeting this summer.

23 September 202416 March 2025

Striking a balance

In June, I attended the World Conference on Research Integrity in Athens. I am still inspired by the many fruitful and fun encounters with colleagues from different places in the world and by thought provoking presentations. One of these talks will form the basis of this blogpost. It is the keynote of Daniele Fanelli entitled “Cautionary tales from metascience” in a session on the effects of research integrity on innovation and policy making. Among his main messages were (1) that replication rates are not as bad as we make them to be,(2) that reproducibility is related to the complexity of the research at hand, (3) that changing policies as a reaction to the reproducibility crisis might do more harm than good and that (4) it is not a one size fits all. Below, I will discuss these issues and will try to conclude what this means for the work within NLRN.

Let’s start with his premises that replication rates from the literature are actually not that bad. He mentioned rates of 60-90%, taking the higher values from ranges reported in the literature. I think a more fair representation would be a median rate between 50-60%. Whether that means that we are in a crisis is a different question. Crises are usually associated with specific periods in time and it’s probably reasonable to assume that replication rates would not have been much different 20 or 30 years ago, if there had been replication studies at that time. Fanelli went on to mention the large variance in replication rates across studies and apparently also (sub) disciplines and there he has a good point.

Fanelli presented results from own work, performed with data from the Brazilian reproducibility initiative, showing that complexity might indeed be related to replication. So there is at least some empirical evidence for his statement. It also seems logical that simpler, more straightforward research or research in a mature field, where there is a high degree of consensus on methods and procedures, would be easier to reproduce and results to replicate.

Fanelli went on to argue that policies focusing on incentive structures are not effective in combating questionable research practices (QRP). He showed that bias and questionable research practices are overall only strongly related to country of first author. Additionally , within countries in which QPRs are prevalent, incentive structures and publication pressure seem to be important drivers, but in other countries, these things do not seem to be related to QPRs. This would, according to Fanelli, imply that policies focused on these things would not be effective in many countries. Here I think Fanelli jumps to the conclusion a bit too quickly. All of his evidence comes from meta-research, which is by nature observational and on an aggregated level. This means that there might be confounds underlying the relations he showed. Moreover, we would need intervention studies to explore whether intervening on these aspects change outcomes. Such studies are scarce. In the field of reproducibility, there is some evidence that rigorous-enhancing practices in both original studies and replication studies can lead to high replication rates and effect sizes that are virtually unchanged in the replications.¹ These practices included confirmatory tests, large sample sizes, preregistration and methodological transparency . However, this multi-lab study was done in social psychology and it is uncertain how results will be in other fields or (sub) disciplines.

All in all, there is not much evidence yet for policy interventions improving the reproducibility and replication of studies and it is probably not one size fits all. Fanelli concludes that policy should be light and adaptive and that makes sense. We will have to strike a balance between incorporating some generic principles and leaving enough room for discipline, field and country/region specific differences. How do we know what works for whom? By developing interventions / policies together with academic and non academic staff, piloting and evaluating these and when deemed viable, by implementing them on a broader scale and evaluating and adapting where necessary. These efforts need continuous monitoring. The reproducibility networks are ideally suited to support these efforts through their network of research performing institutions, communities of researchers and educators and other relevant stakeholders.

Within the Dutch Reproducibility Network we acknowledge the specificity of reproducibility and replication across disciplines and fields, which is why one of our focus areas for the coming years is non-quantitative research. We are eager to work on these and other pressing issues with our partners, striving for evidence-informed implementation of interventions and policies on reproducibility.

1 Protzko, J., Krosnick, J., Nelson, L. et al. High replicability of newly discovered social-behavioural findings is achievable. Nat Hum Behav 8, 311–319 (2024). https://doi.org/10.1038/s41562-023-01749-9

find the slides to Danielle Fanelli’s talk here: https://az659834.vo.msecnd.net/eventsairwesteuprod/production-pcoconvin-public/e2e0a11ab8514551a3376e9b49af030d

4 September 202416 March 2025

Platform for Young Meta-Scientists (PYMS): Empowering the Future of Meta-Science

Meta-science, the study of scientific practice itself, is a field crucial for fostering and monitoring research transparency, reproducibility, and integrity. Recognizing the need for a community among early-career meta scientists in The Netherlands (and its neighbouring countries), the Platform for Young Meta-Scientists (PYMS) formed in 2018. PYMS is dedicated to supporting and connecting young meta-scientists, providing them with a collaborative environment to share resources and discuss new research ideas.

NLRN and PYMS: strong bonds

Collaborating with meta-scientists is one of the focus areas of the NLRN. Evidence-based interventions and monitoring strategies are pivotal to a successful move towards more reproducible science. It is the meta-sciences that can generate such evidence. We are therefore happy to work together with PYMS and wholeheartedly support their mission.

Highlights from the PYMS Meeting in Tilburg: May 31, 2024

The recent PYMS meeting at the Meta-Research Center at Tilburg University was a testament to the vibrant and dynamic nature of the PYMS network. The program featured both formal presentations and informal networking opportunities. A crucial part of the meeting was the brainstorming session on the future of PYMS, where attendees provided valuable feedback and ideas for future events and organizational strategies.

Here are a few examples of presentations to illustrate the wide range of expertise that participants brought to the meeting:

Signe Glaesel (Leiden University): Discussed the challenges surrounding data sharing, including misinterpretations, intellectual property concerns, and the impact of data policies on participant willingness in sensitive studies.

Caspar van Lissa (Tilburg University): Presented on text mining systematic reviews, addressing issues of inductive bias and reproducibility.

Michele Nuijten (Tilburg University): Shared a four-step robustness check for research replicability, highlighting the prevalence of reproducibility problems and strategies to improve research robustness.

Ana Barbosa Mendez (Erasmus University and Promovendi Netwerk Nederland (PNN)): Spoke on best practices in Open Science, mapping the needs of PhD students, and the holistic approach required for effective science communication and community building.

Looking Ahead: The Future of PYMS

The enthusiasm and engagement at the Tilburg meeting underscored the need for regular PYMS gatherings. Participants expressed interest in a more holistic approach, broadening the scope to include researchers from other scientific fields. Formalizing PYMS through stronger links with organizations like PNN and NLRN was also a key takeaway.

A concrete outcome of the brainstorming session in the afternoon are plans for a satellite event on December 5^th, the day before the NLRN symposium in Groningen. All career-young researchers in the field of meta science are invited to join this event. More information will be shared via the NLRN newsletter and on social media.

For more information on upcoming PYMS events and how to get involved, visit metaresearch.nl and PYMS (metaphant.net).