Graphic by Eilidh McNaughton By rapha RedfernA disease is first detected by symptoms and then diagnosed. Next, the full range of biological and environmental factors contributing to its development must be analysed and understood. Only then can treatment be developed. We take this approach towards the replication crisis, an ongoing methodological dilemma that has shaken social sciences, unveiling many irreproducible studies. Before we consider possible solutions and their efficacy, it is essential to understand the underlying social, conceptual and statistical factors. The crisis is a learning experience that can catalyse progress and ideological change in scientific communities which can be, perhaps unknowingly, stuck in their ways. Symptoms and Diagnosis The most unpleasant symptom of the replication crisis is mistrust in psychological scientific findings. This symptom appeared in the 1960s, when Archibald Cochrane raised questions about methods in biomedical science, advocating for randomised control trials (Stavrou et al., 2013). More recently, Bem’s (2011) paper notoriously demonstrated deep systematic issues by producing evidence for psychic abilities (Engber, 2017). This illustrated concerns that most scientific findings would fail to replicate (Ioannidis, 2005), and that psychological findings disproportionately (91.5%) agree with their hypothesis (Fanelli, 2010). A global project, Many Labs 2, 28 psychological replications, and found that only 54% replicated, and 75% showed smaller effect sizes (Klein et al. 2018). Findings that failed to replicate did so under little variation, so failures did not result from sampling differences. Camerer et al. (2018) provide another key piece of information that captures the current extent of the crisis, showing that only 13 of 21 papers selected from Science and Nature replicated, and effect sizes were about half of those originally reported. This diagnosis of replication issues creates the need to be skeptical when viewing and applying psychological research. Contributing factors There are inevitable conceptual features of psychological research that leave it vulnerable to replication issues, like a genetic predisposition to disease. Many psychological variables are abstract and only indirectly measurable, for example, IQ tests measuring intelligence. Furthermore, psychological mechanisms are highly complicated. Therefore, the theoretical basis for psychology is not concrete. These characteristics make direct and conceptual replication pivotal. Effects may hold in direct replication, demonstrating methodological rigour (Diener & Biswas-Diener, 2021), but conceptual replications are also necessary to test theories under differently operationalised variables (Crandall & Sherman, 2016). Lifestyle choices can cause disease, and similarly, systematic pressures create a culture that is not conducive to truly reliable science. Many young researchers are vying for career-progression, but academic rewards systems arbitrarily use publication and securing funding as primary indicators of research value. This seems to ensure eye-catching reading more than it promotes authentic progression of science. Positive and novel results are published more often, despite the fact that these are the most likely to be retracted (Fang & Casdevall, 2011). There is also systematic pressure on researchers to produce papers which leads to the ‘publish or perish’ mindset (Rawat & Meena, 2014). The combination of these factors not only leads to undervaluing of replication work, but also means null results are likely to never see the light of day, known as the file drawer effect (Ferguson & Heene, 2012). Acts of misconduct can also occur as a means to jump through academic hoops, but they are uncommon, with 10% of researchers claiming to have witnessed misconduct, and 3% admitting to it (Gross, 2016). More often, researchers fall victim to the slippery slope of questionable research practices. P-hacking, which involves the manipulation of data to attain a desired level of statistical significance, increases risk of false positives and includes practices like optional stopping and hypothesising after results are known (aka HARKing) (Simmons et al., 2011). Treatment What is needed is a cultural change in research communities, where individual gains are aligned with the shared goal of scientific advancement (Moshontz et al., 2018). A promising implementation of this goal is open science, which encourages replications and makes them easy to achieve by providing complete research materials. Reporting of full data sets and methods also makes spotting genuine mistakes easier, and discourages misconduct by increasing the likelihood of detection. Incentives for engaging in open science are not fully developed (Moher et al, 2018), and there are practical difficulties, such as time-cost (Allen & Mehler, 2019). However, open science practices encourage a more cooperative environment that values engagement in science. For example, it is inclusive to researchers with limited funding, facilitating access to repositories of data, like the human connectome project (Van Essen, 2012). One key element of open-science is pre-registration, which has met criticism for restricting the research process. This is an unavoidable side-effect of clearly distinguishing between hypothesis testing and post hoc analysis, outweighed by positive effects pre-registration has on research practices. P-hacking is avoided by pre-determining experimental and analytical methodologies as well as hypothesis, in advance of collecting data (Nosek, 2018). Pre-registration also positively influences publishing practices. Pre-registered papers are reviewed on the basis of justification and methodological strength, not ability to produce eye-catching headlines. Furthermore, studies are published regardless of their result, offering resolution to the file drawer effect. Allen & Mehler (2019) found that 60.5% of 113 registered reports produced null findings, contrasting to 5-20% of null reports published under closed science. Systematic changes influence individual perspectives, as open science moves the onus away from producing positive, novel results to carrying out robust experiments. Adoption of open science creates positive career opportunities (Flier, 2017), such as recognition of good scientific protocol as well as collaboration and development of theory (Poldrak, 2019). Wider education on the importance of replication and statistical pitfalls will also allow individual researchers to take responsibility and guard against human biases. This manifests as high standards of research practice and strong critical analysis of literature. Conclusion We have reviewed factors that contribute to replication issues in psychology, and demonstrated that whilst conceptual hazards of the field won’t change, the way research is rewarded and consequently viewed can be revolutionised. Open science engenders systematic change, and paired with education, can positively influence research priorities. The replication crisis is not incurable but needs to be carefully treated by creating an environment where researchers that practice thorough science experience career progression, instead of having to choose between the two. References
Allen, C., & Mehler, D. (2019). Open science challenges, benefits and tips in early career and beyond. PLOS Biology, 17(5), e3000246. https://doi.org/10.1371/journal.pbio.3000246 Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality And Social Psychology, 100(3), 407-425. https://doi.org/10.1037/a0021524 Camerer, C.F., Dreber, A., Holzmeister, F. et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Natural Human Behaviour, 637–644. https://doi.org/10.1038/s41562-018-0399-z Diener, E. & Biswas-Diener, R. (2021). The replication crisis in psychology. In R. Biswas-Diener & E. Diener (Eds), Noba textbook series: Psychology. Champaign, IL: DEF publishers. http://noba.to/q4cvydeh Engber, D. (2021, June 7). Daryl Bem Proved ESP Is Real. Which Means Science Is Broken. Slate Magazine. https://slate.com/health-and-science/2017/06/daryl-bem-proved-esp-is-real-showed-science-is-broken.html. Fanelli, D. (2010). “Positive” Results Increase Down the Hierarchy of the Sciences. Plos ONE, 5(4), e10068. https://doi.org/10.1371/journal.pone.0010068 Fang, F., & Casadevall, A. (2011). Retracted Science and the Retraction Index. Infection And Immunity, 79(10), 3855-3859. https://doi.org/10.1128/iai.05661-11 Ferguson, C. J., & Heene, M. (2012). A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null. Perspectives on Psychological Science, 7(6), 555–561. https://doi.org/10.1177/1745691612459059 Flier, J. (2017). Faculty promotion must assess reproducibility. Nature, 549(7671), 133-133. https://doi.org/10.1038/549133a Gross, C. (2016). Scientific Misconduct. Annual Review Of Psychology, 67(1), 693-711. https://doi.org/10.1146/annurev-psych-122414-033437 Ioannidis, J. (2005). Why Most Published Research Findings Are False. Plos Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124 Klein, R., Vianello, M., Hasselman, F., Adams, B., Adams, R., & Alper, S. et al. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances In Methods And Practices In Psychological Science, 1(4), 443-490. https://doi.org/10.1177/2515245918810225 Moher D, Naudet F, Cristea IA, Miedema F, Ioannidis JPA, Goodman SN. (2018). Assessing scientists for hiring, promotion, and tenure. PLoS Biology. https://doi.org/10.1371/journal.pbio.2004089 Moshontz H, Campbell L, Ebersole CR, IJzerman H, Urry HL, Forscher PS, et al. (2018). The Psychological Science Accelerator: Advancing Psychology through a Distributed Collaborative Network. Advanced Methods of Practical Psychological Science. https://doi.org/10.31234/OSF.IO/785QU Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1708274114 Poldrack, R. (2019). The Costs of Reproducibility. Neuron, 101(1), 11-14. https://doi.org/10.1016/j.neuron.2018.11.030 Rawat, S., & Meena, S. (2014). Publish or perish: Where are we heading?. Journal of research in medical sciences: the official journal of Isfahan University of Medical Sciences, 19(2), 87–89. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3999612/ Simmons, J., Nelson, L., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632 Stavrou, A., Challoumas, D., & Dimitrakakis, G. (2013). Archibald Cochrane (1909-1988): the father of evidence-based medicine. Interactive Cardiovascular And Thoracic Surgery, 18(1), 121-124. https://doi.org/10.1093/icvts/ivt451 Van Essen, D. C., Ugurbil, K., Auerbach, E., Barch, D., Behrens, T. E., Bucholz, R., et al. (2012). The Human Connectome Project: a data acquisition perspective. NeuroImage, 62(4), 2222–2231. https://doi.org/10.1016/j.neuroimage.2012.02.018
29 Comments
9/5/2022 16:39:14
Really informative article, I had the opportunity to learn a lot, thank you. https://freecodezilla.net/download-smartpanel-smm-panel-script/
Reply
9/11/2022 23:43:53
Really informative article, I had the opportunity to learn a lot, thank you. https://kurma.website/
Reply
9/12/2022 08:33:31
Really informative article, I had the opportunity to learn a lot, thank you. https://odemebozdurma.com/
Reply
9/15/2022 01:30:02
Really informative article, I had the opportunity to learn a lot, thank you. https://bit.ly/site-kurma
Reply
10/4/2022 23:17:29
I think this post is useful for people. It has been very useful for me. Looking forward to the next one, thank you. https://escortnova.com/escort-ilanlari/gaziantep-escort/nizip-escort/
Reply
10/5/2022 11:35:13
It was a post that I found very successful. Good luck to you. https://escortnova.com/escort-ilanlari/artvin-escort/
Reply
10/6/2022 00:11:51
I follow your posts closely. I can find it thanks to your reliable share. Thank you. https://escortnova.com/escort-ilanlari/corum-escort/sungurlu-escort/
Reply
10/8/2022 06:19:31
Thank you for your sharing. I must say that I am successful in your content. https://escortnova.com/escort-ilanlari/ankara-escort/mamak-escort/
Reply
10/8/2022 19:10:55
Thoughtful and real content is shared. Thank you for these shares. https://escortnova.com/escort-ilanlari/gaziantep-escort/
Reply
11/24/2022 14:22:40
Hemen Göz At evde para kazanmaya basla: https://sites.google.com/view/evden-ek-is/
Reply
12/12/2022 05:47:25
Düşmeyen bot takipçi satın al: https://takipcialdim.com/
Reply
12/17/2022 22:54:55
uygun fiyatlardan takipçi Hemen Göz At: https://takipcim.com.tr/
Reply
1/9/2023 00:43:02
100 tl deneme bonusu veren siteleri öğrenmek istiyorsan tıkla.
Reply
4/28/2023 18:35:23
Halotestin Siparişi Ver: Anabolic Kapında Halotestin
Reply
6/30/2023 16:34:51
En iyi zonguldak ilan sitesi burada. https://zonguldak.escorthun.com/
Reply
Leave a Reply. |