Publication Bias in Medical Research

Publication bias occurs when studies with positive results are more likely to be published than those with negative or inconclusive findings. If a researcher obtains a "negative" result, they may choose not to publish it or attempt to present it in a more favorable light. Such selective reporting can lead to an overestimation of results, such as treatment effects, and an unbalanced scientific evidence base.

Why should we avoid publication bias?

Meta-analyses, which combine results from multiple studies, are particularly vulnerable to publication bias. If negative studies are underrepresented, pooled effect estimates may be misleading, potentially leading e.g. to inappropriate clinical recommendations. Publication bias also affects original studies, as they rely on prior scientific literature. For instance, because of this, researchers with better sample and improved methodology may fail to find statistically significant results, unlike earlier studies. As a result, researchers may doubt the validity of their own results and leave them unpublished. Publishing these types of findings would be crucial to improving the validity of the scientific evidence base.

All this relates to the dominant role of statistical significance testing in medical research. It is important to remember that when interpreting study findings, the focus should be on magnitudes of effect sizes or risk estimates and their confidence intervals rather than statistical significances.

Publication bias contributes to the reproducibility crisis, i.e., the growing concern over the inability to replicate findings in medical research. A much larger pool of researchers is now asking a greater number of questions, often with lower prior odds of success, increasing the likelihood of chance findings. By favoring studies with significant results and neglecting those with null or negative outcomes, publication bias exacerbates this crisis.

Related biases

The credibility of findings in medical research is influenced by publication bias and other outcome and analysis reporting biases. Publication pressure fosters an environment where "negative" results are undervalued and left unpublished. For example, certain "unhealthy" behaviors – such as poor diet or substance use – may be demonized, leading to biases in study design, execution, and reporting. Results that barely meet statistical significance criteria may be overinterpreted due to prior biased research. Publication bias is closely linked to white hat bias and confirmation bias. White hat bias refers to the distortion of research-based information in the service of perceived "righteous ends," while confirmation bias occurs when exaggerated results align with preconceived views and hypotheses, leading to their acceptance even when they are weak or insignificant.

Publication bias is not limited to the non-publication of entire studies; it also involves outcome reporting bias. For example, comparisons between trial publications and protocols have shown that about half of studies have had at least one primary outcome changed, introduced, or omitted. Additionally, underreporting adverse events and negative outcomes can hinder accurate assessments of the long-term safety of medical treatments, potentially delaying the identification of harmful side effects and compromising patient safety.

How Can Effects of Publication Bias Be Reduced?

Various statistical methods, such as funnel plots and regression tests, have been developed to detect publication bias in meta-analyses. However, identifying publication bias remains challenging, and existing statistical methods may not always be effective. Even when bias is detected, correcting it in meta-analyses is difficult, as these correction methods rely on assumptions that may not always hold true – highlighting the need for preventive strategies.

Addressing publication bias requires transparency measures such as study pre-registration, openly available variable catalogues, open-access data, and comprehensive reporting. A shift in attitudes among researchers and journal editors is necessary to counteract publication bias. Increasing awareness of the importance of publishing negative results can help change publication practices. Encouraging the publication of all results –whether positive or negative – can promote more comprehensive reporting.

Some fields already have dedicated journals for non-significant findings, and certain journals instruct reviewers to evaluate studies based on methodological quality rather than statistical significance.

Terms such as "positive," "significant," or "negative" can be misleading, as all results are equally relevant to science – provided they are produced using sound logic and good methodology. Therefore, it is essential to spend less time repeating weak correlations and more time investigating associations using stronger methodologies.

Author:

Professor Jouko Miettunen

Further reading:

Dwan K, Gamble C, Williamson PR, Kirkham JJ; Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One 2013; 8(7):e66844. doi: 10.1371/journal.pone.0066844.

Ioannidis JP. Why most published research findings are false. PLoS Med 2005; 2(8):e124. doi: 10.1371/journal.pmed.0020124.

Schoenfeld JD, Ioannidis JP. Is everything we eat associated with cancer? A systematic cookbook review. Am J Clin Nutr 201; 97(1):127-34. doi: 10.3945/ajcn.112.047142.