What does a p-value of 0.05 actually mean?

It means that if there were truly no effect, you would expect to see a result this extreme or more extreme about five times out of a hundred by chance alone. It is a statement about how surprising the data are under that assumption, not a probability that the effect is real.

Can a result be statistically significant but not important?

Yes, this happens often, especially in large studies. A tiny, practically meaningless difference can clear the significance threshold if enough data points are collected, so significance alone never tells you whether an effect matters in real life.

Does statistical significance mean the study was well designed?

No. A poorly designed study with a biased sample or flawed measurement can still produce a statistically significant result. The test only checks whether a pattern looks like chance, it says nothing about the quality of the method that produced it.

Why do some scientists want to move away from p-values?

Because the label encourages a false sense of certainty, treating significant as a stand in for true or important when it is neither. Many researchers now argue for reporting effect sizes and confidence intervals alongside, or instead of, a single pass or fail number.

How many times does a result need to be repeated before I should trust it?

There is no fixed number, but one study is rarely enough on its own. Confidence grows when independent teams, using different samples and sometimes different methods, keep finding a similar pattern.

What Statistically Significant Really Means

25 September 2026

Statistically significant means the pattern a study found is unlikely to have appeared by chance alone. It does not mean the effect is large, important, or certain to be true. Those are separate questions, and the phrase never answers them.

What the phrase actually claims

Statistical significance is a verdict from a specific kind of test. Researchers start by assuming there is no real effect, no real difference between the group given a new medicine and the group given a placebo, for example. Then they ask how likely it would be to see a difference this size, or bigger, in their data if that assumption were true and nothing was really going on.

If the answer is quite unlikely, conventionally less than a five percent chance, the result is called statistically significant. The label just means the numbers cleared that one threshold. It says nothing else about the finding beyond that single calculation.

This threshold, known as the p-value cutoff, is a convention rather than a law of nature. Scientists agreed decades ago that five percent was a sensible line to draw, not because it carries any special truth, but because some line was needed and this one stuck.

What it never promises

Significance says nothing about size. A blood pressure drug can lower readings by half a point on average and still be statistically significant, if the trial included enough people. Half a point changes nothing for a patient's health, but the maths behind the test does not care about that, it only cares whether the pattern is likely to be chance.

It also says nothing about importance or relevance to real life. A study might find a statistically significant link between eating breakfast and mood, but the actual difference could be so small that no one would notice it in daily life. Significance and meaningfulness are separate questions, and only one of them gets tested by the number.

And it says nothing about how well the study was run. A badly designed trial, with a biased sample or a flawed measurement, can produce a statistically significant result just as easily as a good one. The test checks for chance, not for competence.

The one thing it never tells you

Here is the misunderstanding that trips up almost everyone, including some scientists. A p-value of 0.05 does not mean there is a 95 percent chance the finding is real, or a five percent chance it is due to chance. It cannot mean that, because the calculation starts by assuming there is no real effect, then asks how surprising the data would be under that assumption. It is a statement about the data given the assumption, not a statement about the assumption given the data.

Working out the probability that a finding is actually true would need more than one number from one study. It would need to account for how plausible the idea was before the study started, how many other researchers tested similar ideas and found nothing, and how the result was measured. None of that is in the p-value. The p-value only ever answers the narrow question it was built to answer.

This is why a single significant result should never be read as proof. It is one piece of evidence about how surprising the data are, not a verdict on whether the underlying claim is correct.

Why sample size changes everything

Statistical tests get more sensitive as the number of people or measurements grows. With a small study, only a fairly large, obvious effect will clear the significance threshold. With a very large study, involving hundreds of thousands of people, even a tiny, practically meaningless difference can clear it easily.

This means the label statistically significant tells you almost nothing about whether an effect matters, once you know the study was large. A huge trial can turn a difference too small to notice into a headline finding, simply because it had the statistical power to detect it. The number is doing exactly what it was designed to do, it is just not designed to answer the question most readers assume it answers.

Reading past the label

A more useful question than was it significant is how big was the effect, and how confident can we be about its size. Good reporting gives you both, the actual difference in real units such as points, percentages, or years, and a range called a confidence interval that shows how much that number might vary.

It also helps to ask whether the finding has been repeated. A result that only ever showed up once, in one lab, in one study, deserves more caution than one that keeps appearing across different teams and different samples. Replication tells you something significance testing cannot, it tells you whether the pattern is reliable rather than a one off.

None of this means statistical significance is worthless. It is a useful filter for ruling out pure chance as an explanation. It is just the first filter, not the last one, and the phrase was never built to carry the extra weight people often load onto it.

Common questions

What does a p-value of 0.05 actually mean?: It means that if there were truly no effect, you would expect to see a result this extreme or more extreme about five times out of a hundred by chance alone. It is a statement about how surprising the data are under that assumption, not a probability that the effect is real.
Can a result be statistically significant but not important?: Yes, this happens often, especially in large studies. A tiny, practically meaningless difference can clear the significance threshold if enough data points are collected, so significance alone never tells you whether an effect matters in real life.
Does statistical significance mean the study was well designed?: No. A poorly designed study with a biased sample or flawed measurement can still produce a statistically significant result. The test only checks whether a pattern looks like chance, it says nothing about the quality of the method that produced it.
Why do some scientists want to move away from p-values?: Because the label encourages a false sense of certainty, treating significant as a stand in for true or important when it is neither. Many researchers now argue for reporting effect sizes and confidence intervals alongside, or instead of, a single pass or fail number.
How many times does a result need to be repeated before I should trust it?: There is no fixed number, but one study is rarely enough on its own. Confidence grows when independent teams, using different samples and sometimes different methods, keep finding a similar pattern.

Lucent, once a week

We read the new research and write up what actually holds, in the same plain language as this page. No jargon, no hype, three papers a week.

Read the latest issue