Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and HypercompetitionMarc A. Edwards*, and Siddhartha Roy
Environ Eng Sci. 2017 Jan 1; 34(1): 51–61.
Published online 2017 Jan 1
AbstractOver the last 50 years, we argue that incentives for academic scientists have become increasingly perverse in terms of competition for research funding, development of quantitative metrics to measure performance, and a changing business model for higher education itself. Furthermore, decreased discretionary funding at the federal and state level is creating a hypercompetitive environment between government agencies (e.g., EPA, NIH, CDC), for scientists in these agencies, and for academics seeking funding from all sources—the combination of perverse incentives and decreased funding increases pressures that can lead to unethical behavior.
If a critical mass of scientists become untrustworthy, a tipping point is possible in which the scientific enterprise itself becomes inherently corrupt and public trust is lost, risking a new dark age with devastating consequences to humanity. Academia and federal agencies should better support science as a public good, and incentivize altruistic and ethical outcomes, while de-emphasizing output.
Perverse Incentives in Research Academia: The New Normal?When you rely on incentives, you undermine virtues. Then when you discover that you actually need people who want to do the right thing, those people don't exist…—Barry Schwartz, Swarthmore College (Zetter, 2009)
Academics are human and readily respond to incentives. The need to achieve tenure has influenced faculty decisions, priorities, and activities since the concept first became popular (Wolverton, 1998). Recently, however, an emphasis on quantitative performance metrics (Van Noorden, 2010), increased competition for static or reduced federal research funding (e.g., NIH, NSF, and EPA), and a steady shift toward operating public universities on a private business model (Plerou, et al., 1999; Brownlee, 2014; Kasperkevic, 2014) are creating an increasingly perverse academic culture. These changes may be creating problems in academia at both individual and institutional levels
(Table 1).
Quantitative performance metrics: effect on individual researchers and productivityThe goal of measuring scientific productivity has given rise to quantitative performance metrics, including publication count, citations, combined citation-publication counts (e.g., h-index), journal impact factors (JIF), total research dollars, and total patents. These quantitative metrics now dominate decision-making in faculty hiring, promotion and tenure, awards, and funding (Abbott et al., 2010; Carpenter et al., 2014). Because these measures are subject to manipulation, they are doomed to become misleading and even counterproductive, according to Goodhart's Law, which states that “when a measure becomes a target, it ceases to be a good measure” (Elton, 2004; Fischer et al., 2012; Werner, 2015).
Ultimately, the well-intentioned use of quantitative metrics may create inequities and outcomes worse than the systems they replaced. Specifically,
if rewards are disproportionally given to individuals manipulating their metrics, problems of the old subjective paradigms (e.g., old-boys' networks) may be tame by comparison. In a 2010 survey, 71% of respondents stated that they feared colleagues can “game” or “cheat” their way into better evaluations at their institutions (Abbott, 2010), demonstrating that scientists are acutely attuned to the possibility of abuses in the current system.
Thus, another danger of overemphasizing output versus outcomes and quantity versus quality is creating a system that is a
“perversion of natural selection,” which selectively weeds out ethical and altruistic actors, while selecting for academics who are more comfortable and responsive to perverse incentives from the point of entry. Likewise, if normally ethical
actors feel a need to engage in unethical behavior to maintain academic careers (Edwards, 2014), they may become complicit as per Granovetter's well-established Threshold Model of Collective Behavior (1978). At that point, unethical actions have become “embedded in the structures and processes” of a professional culture, and nearly everyone has been “induced to view corruption as permissible” (Ashforth and Anand, 2003).
Systemic Risks to Scientific IntegrityScience is a human endeavor, and despite its obvious historical contributions to advancement of civilization, there is growing evidence that today's research publications too frequently suffer from lack of replicability, rely on biased data-sets, apply low or substandard statistical methods, fail to guard against researcher biases, and their findings are overhyped (Fanelli, 2009; Aschwanden, 2015; Belluz and Hoffman, 2015; Nuzzo, 2015; Gobry, 2016; Wilson, 2016). A troubling level of unethical activity, outright faking of peer review and retractions, has been revealed, which likely represents just a small portion of the total, given the high cost of exposing, disclosing, or acknowledging scientific misconduct (Marcus and Oransky, 2015; Retraction Watch, 2015a; BBC, 2016; Borman, 2016). Warnings of systemic problems go back to at least 1991, when NSF Director Walter E. Massey noted that the size, complexity, and increased interdisciplinary nature of research in the face of growing competition was making science and engineering “more vulnerable to falsehoods” (The New York Times, 1991).
Misconduct is not limited to academic researchers. Federal agencies are also subject to perverse incentives and hypercompetition, giving rise to a new phenomenon of institutional scientific research misconduct (Lewis, 2014; Edwards, 2016). Recent exemplars uncovered by the first author in the Flint and Washington D.C. drinking water crises include “scientifically indefensible” reports by the U.S. Centers for Disease Control and Prevention (U.S. Centers for Disease Control and Prevention, 2004; U.S. House Committee on Science and Technology, 2010), reports based on nonexistent data published by the U.S. EPA and their consultants in industry journals (Reiber and Dufresne, 2006; Boyd et al., 2012; Edwards, 2012; Retraction Watch, 2015b; U.S. Congress House Committee on Oversight and Government Reform, 2016), and silencing of whistleblowers in EPA (Coleman-Adebayo, 2011; Lewis, 2014; U.S. Congress House Committee on Oversight and Government Reform, 2015). This problem is likely to increase as agencies increasingly compete with each other for reduced discretionary funding. It also raises legitimate and disturbing questions as to whether accepting research funding from federal agencies is inherently ethical or not—modern agencies clearly have conflicts similar to those that are accepted and well understood for industry research sponsors. Given the mistaken presumption of research neutrality by federal funding agencies (Oreskes and Conway, 2010), the dangers of institutional research misconduct to society may outweigh those of industry-sponsored research (Edwards, 2014).A “trampling of the scientific ethos” witnessed in areas as diverse as climate science and galvanic corrosion undermines the “credibility of everyone in science” (Bedeian et al., 2010; Oreskes and Conway, 2010; Edwards, 2012; Leiserowitz et al., 2012; The Economist, 2013; BBC, 2016). The Economist recently highlighted the prevalence of shoddy and nonreproducible modern scientific research and its high financial cost to society—posing an open question as to whether modern science was trustworthy, while calling upon science to reform itself (The Economist, 2013). And, while there are hopes that some problems could be reduced by practices that include open data, open access, postpublication peer review, metastudies, and efforts to reproduce landmark studies, these can only partly compensate for the high error rates in modern science arising from individual and institutional perverse incentives (Fig. 1).
The high costs of research misconduct
The National Science Foundation defines research misconduct as intentional “fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results” (Steneck, 2007; Fischer, 2011). Nationally, the percentage of guilty respondents in research misconduct cases investigated by the Department of Health and Human Services (includes NIH) and NSF ranges from 20% to 33% (U.S. Department of Health and Human Services, 2013; Kroll, 2015, pers. comm.). Direct costs of handling each research misconduct case are $525,000, and over $110 million are incurred annually for all such cases at the institutional level in the U.S (Michalek, et al., 2010). A total of 291 articles retracted due to misconduct during 1992–2012 accounted for $58 M in direct funding from the NIH, which is less than 1% of the agency's budget during this period, but each retracted article accounted for about $400,000 in direct costs (Stern et al., 2014).
Obviously, incidence of undetected misconduct is some multiple of the cases judged as such each year, and the true incidence is difficult to predict. A comprehensive meta-analysis of research misconduct surveys between 1987 and 2008 indicated that 1 in 50 scientists admitted to committing misconduct (fabrication, falsification, and/or modifying data) at least once and 14% knew of colleagues who had done so (Fanelli, 2009). These numbers are likely an underestimate considering the sensitivity of the questions asked, low response rates, and the Muhammad Ali effect (a self-serving bias where people perceive themselves as more honest than their peers) (Allison et al., 1989). Indeed, delving deeper, 34% of researchers self-reported that they have engaged in “questionable research practices,” including “dropping data points on a gut feeling” and “changing the design, methodology, and results of a study in response to pressures from a funding source,” whereas 72% of those surveyed knew of colleagues who had done so (Fanelli, 2009). One study included in Fanelli's meta-analysis looked at rates of exposure to misconduct for 2,000 doctoral students and 2,000 faculty from the 99 largest graduate departments of chemistry, civil engineering, microbiology, and sociology, and found between 6 and 8% of both students and faculty had direct knowledge of faculty falsifying data (Swazey et al., 1993).
Academia and science are expected to be self-policing and self-correcting. However, based on our experiences, we believe there are incentives throughout the system that induce all stakeholders to “pretend misconduct does not happen.” Science has never developed a clear system for reporting, investigating, or dealing with allegations of research misconduct, and those individuals who do attempt to police behavior are likely to be frustrated and suffer severe negative professional repercussions (Macilwain, 1997; Kevles, 2000; Denworth, 2008). Academics largely operate on an unenforceable and unwritten honor system, in relation to what is considered fair in reporting research, grant writing practices, and “selling” research ideas, and there is serious doubt as to whether science as a whole can actually be considered self-correcting (Stroebe et al., 2012). While there are exceptional cases where individuals have provided a reality check on overhyped research press releases in areas deemed potentially transformative (e.g., Eisen, 2010–2015; New Scientist, 2016), limitations of hot research sectors are more often downplayed or ignored. Because every modern scientific mania also creates a quantitative metric windfall for participants and there are few consequences for those responsible after a science bubble finally pops, the only true check on pathological science and a misallocation of resources is the unwritten honor system (Langmuir et al., 1953).
If nothing is done, we will create a corrupt academic cultureThe modern academic research enterprise, dubbed a “Ponzi Scheme” by The Economist, created the existing perverse incentive system, which would have been almost inconceivable to academics of 30–50 years ago (The Economist, 2010). We believe that this creation is a threat to the future of science, and unless immediate action is taken, we run the risk of “normalization of corruption” (Ashforth and Anand, 2003), creating a corrupt professional culture akin to that recently revealed in professional cycling or in the Atlanta school cheating scandal.
An uncontrolled perverse incentive system can create a climate in which participants feel they must cheat to compete, whether it is academia (individual or institutional level) or professional sports. While procycling was ultimately discredited and its rewards were not properly distributed to ethical participants, in science, the loss of altruistic actors and trust, and risk of direct harm to the public and the planet raise the dangers immeasurably.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5206685/