WikiFindings

Criteria for funding and promotion leads to bad science

Background

Scientists are trained to carefully assess theories by designing good experiments and building on existing knowledge. But there is growing concern that too many research findings may be wrong – a recent attempt by 270 scientists to reproduce the findings in psychology found that only about 40 per cent could be reproduced. Scientific researchers have to decide what proportion of time to invest in looking for exciting new results rather than confirming previous findings. The most prestigious journals publish only highly novel findings, and scientists often win grants and get promotions if they manage to publish just one paper in these journals. Scientists also must decide how much resource to invest in each experiment.

Findings

Here, we used a mathematical model to predict how a researcher should spend their research time and effort. The model shows that the best thing for career progression is carry out lots of smaller exploratory studies, rather than the larger confirmatory ones, because this is more likely to lead to surprising results. Even though each experiment is less likely to identify a real effect if it's there, they are likely to get some false positives, which unfortunately are often published too. The number of erroneous conclusions is highest if novel findings are very highly weighted and if reward very quickly levels off as the number of papers increases.

Implications

In the UK funding allocation from government (Research Excellence Framework) there is an emphasis on the 'quality' of very few papers. Our results suggests that there would be substantial benefits of simply reducing the magnitude of the weighting of 4* papers for determining funding, and taking into account 1* and 2* papers in the REF. In some other countries overall productivity is important, so an immediate change could be to take into account the gross number of publications. This is an important issue because so much money is wasted doing research from which the results can't be trusted; a significant finding might be just as likely to be a false positive as actually be measuring a real phenomenon. The best thing for scientific progress would be a mixture of medium-sized exploratory studies with large confirmatory studies. This work suggests that researchers would be more likely to do this if funding agencies and promotion committees rewarded asking important questions and good methodology, rather than surprising findings and exciting interpretations.