What counts as a good p-value?

There is no universally good p-value, only one that is small relative to a threshold the researchers chose in advance, usually 0.05. A smaller number means the data would be rarer under the null hypothesis, but it says nothing about whether the effect is large or meaningful.

Can a study have a good p-value and still be wrong?

Yes, easily. A p-value only tests one narrow statistical question and cannot catch problems like a biased sample, a flawed measurement, or researchers trying several analyses until one crosses the threshold. A tidy p-value can sit on top of a badly designed study.

Why do some scientists want to get rid of p-values altogether?

Because the threshold gets treated as a pass or fail line rather than a rough guide, which encourages selective reporting and discourages nuance. Many statisticians now argue for reporting effect sizes and confidence intervals alongside or instead of a single p-value, so readers see the full picture rather than one number.

Does a p-value tell you anything about cause and effect?

No. A p-value is about how unlikely a pattern in the data would be under chance, not about why that pattern exists. Establishing cause and effect requires a well designed study, ideally a controlled experiment, on top of any statistical test.

What should I look for instead when a headline mentions a significant result?

Look for the actual size of the effect, how many people or samples were involved, and whether other researchers have found the same thing. Those three details tell you far more about whether a finding matters than the p-value on its own.

A P-Value Explained Simply, Plus Three Common Mistakes

28 September 2026

A p-value tells you how surprising your results would be if there were truly no effect at all. It says nothing about whether the effect is real, how big it is, or how important it is. Most news articles that mention a p-value get at least one of these wrong.

The One Sentence Definition

A p-value is the probability of seeing a result as extreme as the one you got, or more extreme, if the thing you are testing for actually does not exist. Imagine testing whether a coin is biased. You flip it 100 times and get 60 heads. The p-value answers a narrow question: if the coin were perfectly fair, how often would 100 flips produce 60 or more heads just by chance?

That is the whole idea. It is a statement about how unusual your data would look under a specific assumption, called the null hypothesis, which is usually the boring explanation that nothing interesting is going on. A small p-value means the data would be unusual under that assumption. A large one means the data would be quite ordinary.

It Does Not Prove Anything

The first mistake is treating a small p-value as proof that a finding is true. A p-value of 0.01 does not mean there is a 99 percent chance the effect is real. It means that if there were no effect, results this extreme would turn up about 1 time in 100 by chance alone.

Those are different claims. The first is about the probability of a hypothesis being correct. The second is about the probability of the data given a hypothesis. Confusing the two is one of the most common errors in science writing, and it makes weak findings sound far more certain than they are.

It Does Not Measure Size

The second mistake is assuming a smaller p-value means a bigger, more important effect. It does not. A p-value is heavily influenced by sample size. Test a tiny, trivial effect on a million people and you can get a p-value so small it looks dramatic, even though the actual difference is meaningless in practice.

This is why a headline like 'highly significant result' can describe an effect too small to matter in real life. A drug that lowers blood pressure by an amount you would never notice can still produce a striking p-value if the trial is large enough. Always ask how big the effect actually was, not just how small the p-value was.

It Is Not The Odds Of Chance

The third mistake is the phrase 'there is a 5 percent chance this result is due to chance', which appears constantly in reporting and is wrong. A p-value of 0.05 does not tell you the probability that chance produced your result. It tells you how often you would see data like this if chance, and only chance, were operating.

Working out the actual chance that a specific finding is a fluke requires more than the p-value. It depends on how plausible the hypothesis was before the study, how many other hypotheses were tested alongside it, and how the study was designed. Two studies can report the same p-value and have very different real chances of being false positives, depending on that background context.

Why 0.05 Became The Line

The common threshold of 0.05, meaning a 1 in 20 chance under the null hypothesis, was popularised by the statistician Ronald Fisher in the 1920s as a rough, practical cutoff. It was never meant to be a strict boundary between true and false, or important and unimportant. It was a convenience that stuck.

Because so much research and publishing now hinges on crossing that line, there is pressure to nudge results just under 0.05 by any means available, from dropping inconvenient data points to trying several analyses until one works. This has become one of the more damaging side effects of treating an arbitrary threshold as a verdict.

Reading A P-Value Properly

A p-value is more useful as one clue among several than as a final answer. Look for the effect size, which tells you how large the difference actually was. Look for the confidence interval, which gives a range of plausible values rather than a single number. Ask whether the result has been repeated by other researchers using different data.

None of this requires advanced maths. It just means treating a p-value as a statement about how surprising the data would be under a specific assumption, and nothing more. Once you hold that boundary, most of the confident-sounding claims built on small p-values start to look a lot more modest.

Common questions

What counts as a good p-value?: There is no universally good p-value, only one that is small relative to a threshold the researchers chose in advance, usually 0.05. A smaller number means the data would be rarer under the null hypothesis, but it says nothing about whether the effect is large or meaningful.
Can a study have a good p-value and still be wrong?: Yes, easily. A p-value only tests one narrow statistical question and cannot catch problems like a biased sample, a flawed measurement, or researchers trying several analyses until one crosses the threshold. A tidy p-value can sit on top of a badly designed study.
Why do some scientists want to get rid of p-values altogether?: Because the threshold gets treated as a pass or fail line rather than a rough guide, which encourages selective reporting and discourages nuance. Many statisticians now argue for reporting effect sizes and confidence intervals alongside or instead of a single p-value, so readers see the full picture rather than one number.
Does a p-value tell you anything about cause and effect?: No. A p-value is about how unlikely a pattern in the data would be under chance, not about why that pattern exists. Establishing cause and effect requires a well designed study, ideally a controlled experiment, on top of any statistical test.
What should I look for instead when a headline mentions a significant result?: Look for the actual size of the effect, how many people or samples were involved, and whether other researchers have found the same thing. Those three details tell you far more about whether a finding matters than the p-value on its own.

Lucent, once a week

We read the new research and write up what actually holds, in the same plain language as this page. No jargon, no hype, three papers a week.

Read the latest issue