Scientists are not immune to confirmation biases and motivated reasoning. Values influence each phase of the research process, including how people interpret research findings. Reviewers’ theoretical and ideological views can influence their evaluation of research reports, leading them to judge studies that oppose their beliefs more critically than studies supporting their views. Consequently, they are then less likely to recommend publication of studies with undesired findings or funding for studies based on undesirable theories or hypotheses.
There are powerful incentives to present a strong, compelling story when describing their research. Most of us are motivated to get the science right, but we are also motivated to get the studies published and our grants funded. We want our colleagues to find our research sufficiently interesting and important to support publishing it, and then to cite it, preferably a lot. We want jobs, promotions, and tenure. We want popular media to publicize our research and to disseminate our findings beyond the confines of our lab. We might even hope to tell a story so compelling we can produce a bestselling popular book and receive lucrative consulting and speaking engagements, or have our findings influence policy decisions.
In brief, powerful incentives exist that motivate us to achieve — or, at least, appear to achieve — a “Wow Effect”. A “Wow Effect” is some novel result that comes to be seen as having far- reaching theoretical, methodological, or practical implications. It is the type of work likely to be emulated, massively cited, and highly funded.
Compelling, persuasive narratives are amply rewarded by promotions, grants, named chairs, etc., but the relationship of “compellingness of narrative” to validity (effect size, replicability, generalizability, etc.) is currently unknown. This raises the possibility that for some unknown and possibly substantial portion of the time, we are rewarding research practices that produce Wow Effects that are false, distorted, or exaggeratedhttps://www.dropbox.com/s/elimpj5u0aaeu ... l.pdf?dl=0