Toward a More Nuanced Interpretation of Statistical Significance in Biomedical Research

Introduction

Statistical significance is the bedrock of evidence-based medicine, crucial for decision-making and framing biomedical science policies. The emergence of complex data in many disciplines, resulting from the analysis of large datasets, has amplified the popularity of p-values. Its simplicity allows investigators to conclude and disseminate their research findings in a manner understood by most. Thus, obtaining a p-value that indicates “statistical significance” against the null hypothesis is often required for publishing in medical journals. However, it creates challenges due to nonreproducibility, misuse and overinterpretation, which lead to serious methodological errors. This article aims to draw biomedical researchers' attention toward the appropriate use of p-values in clinical decision-making.

Historical Development

Fisher first introduced null hypothesis significance testing (NHST) in 1925.¹ Subsequently, p-values became the standard of reporting and judging scientific evidence's strength when testing the null hypotheses against the alternative proposition in most scientific disciplines, including biomedical research. The recent development of big data research has made p-values even more popular to test the significance of study findings and has become a sine qua non for publishing in medical journals. The basis of the p-value is that it computes the probability of observing results at least as extreme as the ones observed, given that the null hypothesis is correct and compared against a predetermined significance level (α). If the reported p-value is lesser than α, the test result is to be considered statistically significant (Fig. 1). Typically, α is set arbitrarily at 0.05 level to control false-positive rate, and other commonly used significance levels are 0.01 and 0.001.

Close

Fig. 1 Definition of p-value (https://en.wikipedia.org/wiki/Statistics).

Export to PPT

Debate on Statistical Significance

Many researchers have questioned the acceptability of p-values in medical decision-making, and considerable research exists into how p-values are misused.^2,3 For example, in his seminal paper, Cohen⁴ argued that: “NHST is highly flawed as it is relatively easy to achieve results that can be labeled significant when a ‘nil’ hypothesis is used rather than a true ‘null’ hypothesis.” In recent years, despite its success, there is an emerging debate about whether to use p-values to describe statistically significant scientific results due to its frequent failure to reproduce and replicate similar statistically significant findings. Halsey et al⁵ argued that: “the p-value is often used without the realization that in most cases the statistical power of a study is too low for p to assist the interpretation of the data.… Researchers would do better to discard the p-value and use alternative statistical measures for data interpretation.” In agreement with this thinking, the journal of Basic and Applied Social Psychology recently barred p-values and hypothesis testing from articles published in their journal.⁶p-Values are also susceptible to “hacking,” to demonstrate statistical significance when no association exists and encourages selective reporting of only positive findings. A recent methodological review of articles published in high impact journals suggests that significant results are about twice as likely to be reported as nonsignificant results.⁷

In our opinion, p-values alone cannot be responsible for the lack of reproducibility of research findings, as it is often a combination of methodological errors and interpretation. Like other statistical measures, the p-value is also a one-dimensional metric and based on data; thus, it could be misleading when calculated from relatively small samples. The overall selection of statistical methods, including the lack of randomness in the sample and missing data, can also influence the statistical significance that may result in misleading p-values. Misinterpretation of the p-value as a measure of the strength of association, rather than its true meaning (assessing the probability for a given result arising due to chance), is a glaring indictment of the current biomedical teaching standards. A recent methodological survey of statistical methods in Indian journals suggests that no significant progress has been achieved regarding the correct use of statistical analyses.⁸

Suggested Alternatives on Statistical Significance

Several alternatives to p-values have been suggested in the literature, such as confidence intervals and Bayesian statistics.⁹ A confidence interval provides the point estimate with uncertainty bounds that can be more informative than a p-value. However, confidence intervals are like p-values when testing the null hypothesis's acceptance or rejection and challenging to compare between studies due to unit-dependence. In Bayesian statistics, the credible interval, equivalent to the frequentist approach's confidence interval, is another possible alternative to the p-value. Both alternative methods are like the p-value when testing the null hypothesis for clinical decision-making and can misinterpret the results' clinical or biological importance. Recently, Benjamin et al proposed to lower the p-value (the conventional “statistical significance”) threshold from 0.05 to 0.005 for all novel claims with relatively low prior odds, to avoid high false positives and improve the reproducibility of scientific research.¹⁰ However, others have argued that research should be guided by rigorous scientific principles, not by heuristics and arbitrary thresholds. These principles include sound statistical analyses, replication, validation and generalization, avoidance of logical traps, intellectual honesty, research workflow transparency, and accounting for potential sources of error.^11,12

Guidance on Appropriate Interpretation of Statistical Significance

Therefore, educating researchers with appropriate training on concepts and relevant software could be one alternative to prevent misinterpretation of the p-value. It is worth reiterating Fisher’s initial view that p-values should be one part of the evidence used when deciding whether to reject the null hypothesis. Appropriate guidance should also be taken during the design, process and data analysis of a study from various available resources such as Consolidated Standards of Reporting Trials (CONSORT)¹³ for clinical trials, Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA)¹⁴ for systematic reviews and meta-analysis, Strengthening the Reporting of Observational Studies in Epidemiology (STROBE)15 for observational studies, and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD)¹⁶ for risk prediction from a multivariable model.

As mentioned in NHST approach, researchers commonly classify results as statistically “significant” or “not-significant,” based on whether the p-value is smaller than some prespecified cut point value (usually 0.05). However, this practice is becoming obsolete, and exact p-values are preferred by leading medical journals such as the British Medical Journal (BMJ), Journal of the American Medical Association (JAMA), and the Lancet. Guidance should be taken from Fisher’s¹ belief about p-values from his 1925 book and Efron’s¹⁷ interpretation on observed p-values (or achieved significance level) as presented in Table 1. The American Statistical Association (ASA) took this matter to their board in 2015 and discussed it with renowned statisticians in multiple rounds for several months. Further, a “statement on statistical significance and p-values” with six principles has been released by ASA at the beginning of 2016 to guide researchers and avoid any misuse and misinterpretation.¹⁸ In its statement, the ASA advised researchers to avoid drawing scientific conclusions or making policy decisions purely based on p-values. Additionally, they recommend describing the data analysis approach that produces statistically significant results, including all statistical tests and choices made in calculations. Otherwise, results may appear misleadingly robust.

Conclusion

In conclusion, we advocate that statistics should be used as a science rather than a recipe for the desired flavor. While researchers want certainty, they must understand that statistics is a science of uncertainty. Thus, solely relying on a single p-value to describe the scientific value of a study is a misuse of the p-value and when evaluating the strength of any evidence, researchers need to explain their results in the light of theoretical considerations such as scope, explanatory extent, and predictive power.