Comparing Clinical Significance & Statistical Significance - Similarities & Differences

In extremely broad terms, statistical significance means that it’s likely that something is happening, while clinical significance verifies to what extent that thing is happening. Put another way: statistical significance seeks to disprove a negative, and say an event probably didn’t happen by chance; clinical significance seeks to prove a positive, and say an event did happen in a particular, measured manner.

Unfortunately, it’s also a lot more complicated than that, and this goes beyond simple semantics: a crisis on the non-replicability of scientific studies in the last decade has caused vociferous arguing about what these terms mean, and how they’re decided, and how much importance they should convey.

There’s an informal fallacy that can help illustrate aspects of that crisis, and it’s called the Texan Sharpshooter Fallacy. In this thought exercise, a Texan starts shooting at the side of a barn. When he’s run out of bullets, he walks over to the barn and paints a giant target sign, making sure to draw the center of the target around the area where the most bullets landed. Hey look, he says, I’m a sharpshooter! From a statistically significant point of view, he’s right: it couldn’t possibly have happened by chance that all those shots landed so close to the bullseye.

Now, in the academic world, that type of misrepresentation is far less exaggerated, far less common, and practically never intentional. Still, certain forces have implicitly encouraged small errors in data collection and put a priority on the wrong kinds of significance.

To put it succinctly, some academic journals and studies placed far too much importance on statistical significance, which, it turns out, sometimes isn’t all that significant. If this sounds confusing, don’t worry; it’s confusing to some extremely smart people, too.

To learn the differences, similarities, and applications of clinical and statistical significance, check out our side-by-side comparison chart below.

Clinical Significance Statistical Significance

Definition

In medical terms, clinical significance (also known as practical significance) is assigned to a result where a course of treatment has had genuine and quantifiable effects. Broadly speaking, statistical significance is assigned to a result when an event is found to be unlikely to have occurred by chance.

Key Terms

  • The effect size is a measure of an observed phenomenon’s magnitude. This could include the correlation between two variables, the mean difference between two variables, or the risk of a particular event happening.
  • The number needed to treat (NNT) is a type of effect size that measures the average number of patients who need to be treated to prevent one additional bad outcome, or the number who need to be treated for one to benefit over the control.
  • The Jacobson-Truax approach calculates clinical significance through a Reliability Change Index (RCI), which is equal to the difference between a study participant’s pre-study and post-study scores, divided by a standard error of the difference. Participants are also scored into categories for deteriorated, unchanged, improved, and recovered.
  • The null hypothesis is the default assumption that there is no statistical significance: that nothing observed has changed, and/or there is no association or relationship between observed data sets.
  • The p-value is the probability of achieving a study’s results if the null hypothesis is assumed to be true. If P=0.038, for example, it means there’s a 3.8 percent chance that any observed difference was the result of random chance.
  • The significance level of a study is set in advance, before data is collected. Originally, it was defined as 0.05, or 5 percent, but it can be set much, much lower, depending on the field of study (particle physics or genomics, for example, may use up to nine decimal places). If, after all data is collected, The p-value is less than or equal to the significance level, the results are said to be statistically significant.

History

Historically, clinical significance has been applied primarily in pharmaceutical trials and medical research. It also has had major implications in psychology and psychotherapy, where the term clinical significance differs slightly in its criteria and formulation. There’s no simple equation for determining clinical significance, and numerous methods have developed over time for arriving at more accurate results.

In the 21st century, the emergence of big data and the field of data science have brought a renewed focus on the concept of clinical significance, which is often referred to as ‘practical significance’ when used outside of a medical research setting. Practical significance now allows researchers to dig deeply into data sets and draw conclusions with a high amount of confidence.

While the history of statistical significance dates back to the 18th century, it wasn’t until 1925 that British statistician and geneticist Ronald Fisher advanced the idea within statistical hypothetical testing. Fisher put forth the initial threshold of 0.05, or 5 percent, for determining statistical significance, but never intended it to be a standard cutoff point, instead recommending it be adjusted based on contextual characteristics.

By the early 2010s, academic journals began to note an alarming overuse of the 0.05 p-value, and an editorial bias towards publishing studies that met that threshold. Such bias led to practices such as ‘p-hacking’ that emphasized p-value over other forms of scientific and statistical reasoning.

In 2016, the American Statistical Association issued a statement saying that p-value should never be used as a substitute for scientific rigor, and that it hoped to steer research into a ‘post p<0.05 era’.

Applications

Clinical significance has key applications in vaccine testing, pharmaceutical testing, and other forms of medical research where the magnitude and specific implications of a particular intervention need to be measured and quantified. But it also has use in non-medical settings, too, where it can provide a more rigorous critique of a data set. Statistical significance has broad applications wherever one is looking to learn whether something happened by chance, including market research with A/B testing, and opinion research with surveys or polls. It can also be useful in the early stages of pharmaceutical testing to determine whether further research is warranted.

Examples

A pharmaceutical company is testing the efficacy of a new anti-pain drug with clinical trials. It cultivates a representative participant group, gauges their pre-test pain levels, and then closely monitors their progression, before recording their post-test pain levels. Participants are sorted into tiers of their level of effectiveness, and weighed against a control, with further contextual data taken into account (i.e., side-effects and pre-existing conditions). At the conclusion of the study, researchers will have a more informed sense of the new drug’s effect. A retail company is considering a new advertising campaign. It shows the new campaign to a sample group, and finds that they spend, on average, more than those who saw the old campaign. If the results are statistically significant, a correlation between the new ad and increased spend is likely, and the company may switch all their marketing material to the new campaign.

Importance

Clinically significant results are reproducible to a much higher degree than those which are merely statistically significant, and therefore can be used in scenarios with higher stakes, where even a modest margin of error would be too high. Clinical significance is a requirement when performing pharmaceutical testing, and the nuanced results that come with it can produce radical insights. Statistical significance helps scientists, companies, and other entities understand how strongly the results of an experiment, survey, or poll should influence their decisions. But it’s far from the only factor one should consider when assessing the importance of a particular result. Sample size, contextual characteristics, and more complete research must be taken into account as well.

Resources

The Bottom Line

Clinical significance seeks to understand the size and scope of an effect. It’s a critical tool for decision-makers who are dealing with high-stakes pharmaceutical, psychological, and medical research. However, the scientific rigor and reproducibility of this type of data analysis is also attractive to data scientists answering questions in other industries. Statistical significance seeks to verify that an effect is taking place. It can be a helpful tool for decision-makers when taking into account the results of a particular study. However, it should not be the primary determinant of truth, efficacy, or importance. A stricter focus on scientific rigor and smaller p-values will reduce the chances of false positives and irreproducibility in statistical research.
Matt Zbrog
Matt Zbrog
Writer

Matt Zbrog is a writer and researcher from Southern California. Since 2018, he’s written extensively about emerging issues in healthcare administration and public health, with a particular focus on progressive policies that empower communities and reduce health disparities. His work centers around detailed interviews with researchers, professors, and practitioners, as well as with subject matter experts from professional associations such as the American Health Care Association / National Center for Assisted Living (AHCA/NCAL) and the American College of Health Care Executives (ACHCA).

Related Posts

  • 17 April 2024

    A Day in the Life of a Clinical Research Associate

    The clinical research associate acts as a liaison between the study's sponsor (e.g., pharmaceutical company) and the clinics where the study takes place. Because results of a clinical trial must be kept entirely transparent and not influenced by the interests of the sponsor, this is a critical role.

  • 15 April 2021

    A Day in the Life of a Clinical Data Analyst

    Every day, health organizations like hospitals, clinics, and physician offices collect data about their patients. This information is used to make data-driven decisions in order to provide the highest level of care to their patients as well as reduce expenses and errors. From the outcome of a particular treatment to a large-scale clinical trial, health data is a critical part of the modern healthcare process. Clinical data analysts help to make sense of the extensive data that is at their fingertips, creating stories that turn numbers into actionable intelligence to improve healthcare outcomes.

  • 10 July 2020

    Healthcare Debates: Does Costly US Healthcare Fund R&D and Medical Research?

    This article explores research and development in the context of greater healthcare expenditures, including looking into the main drivers of healthcare spending, influences on pricing, and where R&D fits into the whole ecosystem.

  • 13 November 2019

    Pharmaceutical Quality Director – A Day in the Life

    Consumers and corporations need to know that the drugs dispensed by pharmaceutical companies are manufactured safely and efficiently. Pharmaceutical quality directors ensure that quality control protocols are followed within the industry’s manufacturing, testing, and inspection procedures.

  • 20 February 2019

    Closing the Gap: Women Leading Healthcare

    Women drive healthcare. They are the industry's biggest consumers and workers. They serve as caregivers in their homes and make most decisions for their family's health. Why is it, then, that of the 40 Fortune 500 healthcare companies, not a single one is helmed by a woman?