Category: Metascience

Blogpost by Cas Goos

The goal of our organization, the Platform for Young Meta-Scientists (PYMS), is to bring together Early Career Researchers (ECRs) working on meta-science, while providing a place to network with peers and discuss research. We consider this an important initiative as ECRs have been at the forefront of many reform initiatives, and because despite the increasing number of meta-science ECRs, many still work disconnected from other researchers with shared interests. Bringing ECRs together is a crucial step in enabling large scale research and strengthening reform initiatives that are key to improving science. 

The PYMS Meeting 

To further these goals within the most recent PYMS meeting, which took place at the UMCG on December 5^th 2024 as a pre-symposium to the NLRN symposium on December 6^th, we invited an audience of ECRs from a variety of backgrounds. We are glad to have achieved a broad representation of ECRs, including from abroad. Presentations span a variety of meta-science topics such as spin, equivalence testing, and reproducibility. Not only work from the social sciences, but also work in computational sciences, sports and exercise science, and nanobioscience were presented and discussed.  

Map of PYMS attendees and their areas of expertise/interest. Map backdrop was constructed by Tamarinde Haven. 

Furthermore, during the meeting we created many opportunities for discussion and networking, including invitations for collaborations, throughout the day.   

We were also fortunate enough to have Tracey Weissgerber give a keynote talk on having a career in meta-science followed by a series of provocative statements on the same topic. Both led to a practically informative discussion for the attendees on having a career in meta-science.  

The Future of PYMS 

Since the end of last year, the board for PYMS has been renewed. Four new ECRs have joined the board to carry the torch. The new members are Sajedeh Rasti from Eindhoven University, Raphael Merz from Ruhr University Bochum, and Anouk Bouma and myself from Tilburg University. Like the previous board, we plan to continue creating a community for meta-science ECRs with informal networking opportunities. However, we also plan to expand the reach of PYMS in new ways. 

As a first step, PYMS will go international during our next meeting. In this way we expand our networking beyond the Netherlands to prompt broader collaboration in our field. This is especially important here, since projects requiring large investments of time and effort from multiple parties are of crucial importance to investigate and improve scientific practice effectively. This meeting will be held this summer, there will be plenty of opportunities to discuss future projects and financial compensation for international presenters to encourage a broad representation of young meta-scientists. 

Interested ECRs can join our mailing list and get a link through there to our Discord server to connect and communicate with fellow ECRs on meta-science, as well as keep up to date about future events, including the next PYMS meeting this summer. 

In June, I attended the World Conference on Research Integrity in Athens. I am still inspired by the many fruitful and fun encounters with colleagues from different places in the world and by thought provoking presentations. One of these talks will form the basis of this blogpost. It is the keynote of Daniele Fanelli entitled “Cautionary tales from metascience” in a session on the effects of research integrity on innovation and policy making. Among his main messages were (1) that replication rates are not as bad as we make them to be,(2) that reproducibility is related to the complexity of the research at hand, (3) that changing policies as a reaction to the reproducibility crisis might do more harm than good and that (4) it is not a one size fits all. Below, I will discuss these issues and will try to conclude what this means for the work within NLRN. 

Let’s start with his premises that replication rates from the literature are actually not that bad. He mentioned rates of 60-90%, taking the higher values from ranges reported in the literature. I think a more fair representation would be a median rate between 50-60%. Whether that means that we are in a crisis is a different question. Crises are usually associated with specific periods in time and it’s probably reasonable to assume that replication rates would not have been much different 20 or 30 years ago, if there had been replication studies at that time. Fanelli went on to mention the large variance in replication rates across studies and apparently also (sub) disciplines and there he has a good point.  

Fanelli presented results from own work,  performed with data from the Brazilian reproducibility initiative, showing that complexity might indeed be related to replication. So there is at least some empirical evidence for his statement. It also seems logical that simpler, more straightforward research or research in a mature field, where there is a high degree of consensus on methods and procedures, would be easier to reproduce and results to replicate.  

Fanelli went on to argue that policies focusing on incentive structures are not effective in combating questionable research practices (QRP). He showed that bias and questionable research practices are overall only strongly related to country of first author. Additionally , within countries in which QPRs are prevalent, incentive structures and publication pressure seem to be important drivers, but in other countries, these things do not seem to be related to QPRs. This would, according to Fanelli, imply that policies focused on these things would not be effective in many countries. Here I think Fanelli jumps to the conclusion a bit too quickly. All of his evidence comes from meta-research, which is by nature observational and on an aggregated level. This means that there might be confounds underlying the relations he showed. Moreover, we would need intervention studies to explore whether intervening on these aspects change outcomes. Such studies are scarce. In the field of reproducibility, there is some evidence that rigorous-enhancing practices in both original studies and replication studies can lead to high replication rates and effect sizes that are virtually unchanged in the replications.¹ These practices included confirmatory tests, large sample sizes, preregistration and methodological transparency . However, this multi-lab study was done in social psychology and it is uncertain how results will be in other fields or (sub) disciplines.  

All in all, there is not much evidence yet for policy interventions improving the reproducibility and replication of studies and it is probably not one size fits all. Fanelli concludes that policy should be light and adaptive and that makes sense. We will have to strike a balance between incorporating some generic principles and leaving enough room for discipline, field and country/region specific differences. How do we know what works for whom? By developing interventions / policies together with academic and non academic staff, piloting and evaluating these and when deemed viable, by implementing them on a broader scale and evaluating and adapting where necessary. These efforts need continuous monitoring. The reproducibility networks are ideally suited to support these efforts through their network of research performing institutions, communities of researchers and educators and other relevant stakeholders.  

Within the Dutch Reproducibility Network we acknowledge the specificity of reproducibility and replication across disciplines and fields, which is why one of our focus areas for the coming years is non-quantitative research. We are eager to work on these and other pressing issues with our partners, striving for evidence-informed implementation of interventions and policies on reproducibility.   

1 Protzko, J., Krosnick, J., Nelson, L. et al. High replicability of newly discovered social-behavioural findings is achievable. Nat Hum Behav 8, 311–319 (2024). https://doi.org/10.1038/s41562-023-01749-9

find the slides to Danielle Fanelli’s talk here: https://az659834.vo.msecnd.net/eventsairwesteuprod/production-pcoconvin-public/e2e0a11ab8514551a3376e9b49af030d

Meta-science, the study of scientific practice itself, is a field crucial for fostering and monitoring research transparency, reproducibility, and integrity. Recognizing the need for a community among early-career meta scientists in The Netherlands (and its neighbouring countries), the Platform for Young Meta-Scientists (PYMS) formed in 2018. PYMS is dedicated to supporting and connecting young meta-scientists, providing them with a collaborative environment to share resources and discuss new research ideas. 

NLRN and PYMS: strong bonds 

Collaborating with meta-scientists is one of the focus areas of the NLRN. Evidence-based interventions and monitoring strategies are pivotal to a successful move towards more reproducible science. It is the meta-sciences that can generate such evidence. We are therefore happy to work together with PYMS and wholeheartedly support their mission.   

Highlights from the PYMS Meeting in Tilburg: May 31, 2024 

The recent PYMS meeting at the Meta-Research Center at Tilburg University was a testament to the vibrant and dynamic nature of the PYMS network. The program featured both formal presentations and informal networking opportunities. A crucial part of the meeting was the brainstorming session on the future of PYMS, where attendees provided valuable feedback and ideas for future events and organizational strategies. 

Here are a few examples of presentations to illustrate the wide range of expertise that participants brought to the meeting: 

• Signe Glaesel (Leiden University): Discussed the challenges surrounding data sharing, including misinterpretations, intellectual property concerns, and the impact of data policies on participant willingness in sensitive studies. 

• Caspar van Lissa (Tilburg University): Presented on text mining systematic reviews, addressing issues of inductive bias and reproducibility. 

• Michele Nuijten (Tilburg University): Shared a four-step robustness check for research replicability, highlighting the prevalence of reproducibility problems and strategies to improve research robustness. 

• Ana Barbosa Mendez (Erasmus University and Promovendi Netwerk Nederland (PNN)): Spoke on best practices in Open Science, mapping the needs of PhD students, and the holistic approach required for effective science communication and community building. 

Looking Ahead: The Future of PYMS 

The enthusiasm and engagement at the Tilburg meeting underscored the need for regular PYMS gatherings. Participants expressed interest in a more holistic approach, broadening the scope to include researchers from other scientific fields. Formalizing PYMS through stronger links with organizations like PNN and NLRN was also a key takeaway. 

A concrete outcome of the brainstorming session in the afternoon are plans for a satellite event on December 5^th, the day before the NLRN symposium in Groningen. All career-young researchers in the field of meta science are invited to join this event. More information will be shared via the NLRN newsletter and on social media.  

For more information on upcoming PYMS events and how to get involved, visit metaresearch.nl and PYMS (metaphant.net).