A P Value Does Not Tell You the Probability You Are Right
I remember sitting in a windowless seminar room during my second year of my PhD, watching a senior researcher present a dataset that was clearly noisy, yet he was treating a tiny significance threshold like it was a divine revelation. He spoke about the results as if they were absolute truth, completely ignoring the fact that his sample size was practically microscopic. It was my first real encounter with the pervasive issue of p values and their misuse, and it left a bitter taste in my mouth. We’ve reached a point in modern research where people treat a threshold of 0.05 like a magic gate between “reality” and “noise,” but that’s a dangerous simplification that ignores how data actually behaves in the wild.
I am not here to give you a textbook definition or a list of arbitrary rules to memorize for an exam. Instead, I want to pull back the curtain on what these numbers actually represent—and more importantly, what they don’t. My goal is to help you understand the underlying mechanics of statistical inference so you can spot when someone is using math to mask a lack of substance. We are going to look at the actual friction between mathematical theory and messy, real-world implementation.
Table of Contents
The Trap of Null Hypothesis Significance Testing
The core of the problem lies in the framework we use: Null Hypothesis Significance Testing (NHST). In theory, it’s a clean way to decide if an observation is meaningful or just noise. In practice, it often becomes a binary trap. We set an alpha level—usually 0.05—and treat it like a holy threshold. If our p-value is 0.049, we declare victory; if it is 0.051, we claim nothing happened. This creates a massive incentive for p-hacking in research, where investigators tweak their models or exclude outliers just enough to nudge that number across the line.
The real danger, however, is that NHST encourages us to ignore the magnitude of what we’ve actually found. You can run a study with ten thousand participants and find a “statistically significant” result that is utterly meaningless in the real world. This is the gap between statistical significance vs practical significance. A drug might lower blood pressure by a fraction of a millimeter, which is mathematically “real” but clinically useless. If we only report the p-value and ignore the effect size, we aren’t doing science; we’re just performing a mathematical ritual.
Alpha Levels and the Fragility of Type I Error
We usually set our alpha level—the threshold for “significance”—at 0.05. This is a convention, not a law of nature, but it carries a heavy weight. By choosing $alpha = 0.05$, you are essentially pre-deciding that you are willing to accept a 5% chance of being wrong; you are accepting a Type I error, where you claim an effect exists simply because of a lucky roll of the cosmic dice. The problem is that when we run dozens of tests, that 5% risk doesn’t stay isolated. It compounds.
This is where we see the mechanics of the reproducibility crisis start to grind into gear. If you test enough variables, you will eventually find a “significant” pattern purely by chance. This is the mathematical engine behind p-hacking in research: if you slice the data enough ways, one of those slices will eventually cross the arbitrary 0.05 line. We often mistake this mathematical inevitability for a discovery, forgetting that a low p-value doesn’t measure the strength of a phenomenon, only how much it deviates from a specific, often hollow, null model.
How to stop letting p-values lie to you
- Look for the effect size, not just the significance. A p-value can tell you that a result is unlikely to be a fluke, but it won’t tell you if the result actually matters in the real world. You can have a “statistically significant” result that is so tiny it has zero practical utility for your system or your users.
- Stop treating p < 0.05 like a magic threshold for truth. That number is a convention, not a law of nature. When you treat it as a binary switch between "discovery" and "failure," you end up ignoring interesting trends that just barely missed the cut and obsessing over noise that barely passed it.
- Beware of “p-hacking” through multiple comparisons. If you test fifty different variables against a single outcome, one of them is bound to look significant just by sheer coincidence. If you aren’t adjusting your significance threshold for the number of tests you’re running, you aren’t doing science; you’re just hunting for patterns in the static.
- Contextualize the null hypothesis. Remember that a low p-value only suggests your data is inconsistent with a specific, often arbitrary, null model. It doesn’t prove your alternative hypothesis is correct; it just means the “nothing is happening” model is a poor fit for what you observed.
- Demand replication over single-shot results. In my experience, a single p-value is a snapshot, not a video. If a result is truly robust, it should show up again when the experiment is run under slightly different conditions or with a different sample. If it only appears once, treat it as a curiosity, not a fact.
The Reality Check
A p-value is a measure of how surprising your data is if we assume nothing is happening; it is not a direct measurement of whether your theory is correct.
Setting an alpha level at 0.05 is a convention, not a law of nature, and treating it as a hard threshold for “truth” creates a binary logic that the math doesn’t actually support.
Statistical significance is not the same as practical importance; a massive dataset can produce a tiny, mathematically “significant” p-value for an effect that is too small to matter in the real world.
Beyond the Threshold
We have to stop treating the p-value as a binary switch that flips a result from “meaningless” to “discovered.” As we have seen, the rigid adherence to arbitrary alpha levels creates a culture of fragility where a single digit can determine the fate of a research project, regardless of the actual effect size or the physical reality of the system being studied. When we rely solely on the null hypothesis framework, we aren’t actually measuring truth; we are merely measuring how much our data disagrees with a specific, often arbitrary, mathematical assumption. If you find yourself chasing a p-value of 0.049 just to clear a hurdle, you aren’t doing science—you are performing statistical gymnastics.
My hope is that we move toward a more honest way of communicating uncertainty. Instead of hiding behind the shield of “statistical significance,” I want us to start talking about effect sizes, confidence intervals, and the actual mechanisms that drive our observations. The math is a tool, not a verdict. When we prioritize the underlying logic over the pursuit of a specific number, we stop being mere calculators and start being investigators again. Let’s aim for work that is robust enough to survive the scrutiny of a skeptic, rather than work that is just lucky enough to pass a threshold.
