How P-Hacking Happens Without Anyone Lying

19 October 2026

P-hacking is what happens when a researcher, without ever faking data or lying about results, makes a string of small, reasonable-looking decisions that push a study towards a significant finding. No single choice looks dishonest. The trouble is that dozens of small, defensible choices can add up to a result that would not survive if the analysis had been locked in advance.

A Maze of Small Decisions

Collecting data is only the start. Before a result reaches a headline, a researcher has to decide how to define their main measurement, which participants or data points to exclude, which other factors to control for, and which statistical test to run. Each of these is a genuine judgement call, and for most of them there is no single correct answer written down anywhere.

Because there is no correct answer, there are often several equally defensible ways to run the same analysis. One way to define 'improvement' in a health trial might include a symptom that another, equally sensible definition leaves out. One way to handle a participant who dropped out halfway through might differ from another. None of these choices is a lie. They are just choices.

The problem is that these choices are not neutral. Some versions of the analysis will produce a stronger result than others, purely by chance. If a researcher tries a few versions and reports the one that worked, they have not fabricated anything. They have simply let chance pick the version that looks best, and then mistaken it for the truth.

Why So Many Paths Exist

Statisticians sometimes call this the garden of forking paths. Picture a single study as a tree with dozens of branching points, each one a small decision about how to measure, exclude, group or test the data. Walk down any one path and you get a result. Walk down a different path, using data from the exact same experiment, and you can get a different one.

The number of plausible paths through a typical study is often much larger than people assume. A handful of decisions, each with two or three sensible options, multiplies out fast. With enough paths available, it becomes likely that at least one of them will cross the threshold researchers use to call a result significant, purely through the ordinary wobble of sampling, not because the effect is real.

This is the heart of p-hacking. It does not require a researcher to try every path on purpose. It only requires that the path actually reported is the one that happened to work, whether chosen deliberately or simply because it was the first one tried and it looked clean.

Testing Many Things at Once

A related version of the same problem comes from testing many outcomes instead of many analysis methods. A study set up to look at one treatment might measure ten different health outcomes. If none of them show anything interesting on their own, but one crosses the line into statistical significance, it is tempting to write the paper around that one outcome and treat the other nine as background detail.

Run enough separate tests and some will look significant by chance alone, the same way that rolling a die enough times will eventually turn up a six even if the die is fair. The mistake is not rolling the die many times. The mistake is reporting only the six and leaving the rest of the rolls out of the story.

This is sometimes called the multiple comparisons problem, and it sits right next to p-hacking because the two often travel together. A researcher measuring many things, then analysing each of them several different ways, is multiplying their chances of stumbling onto a false positive without ever touching the raw numbers.

Peeking at the Data Early

Another quiet route to a false result is checking the data as it comes in and stopping the study once a significant result appears. This feels efficient. Why keep running an experiment once the answer already looks clear? But a result that looks significant partway through a study is more likely to be a temporary blip than a stable finding, and stopping early locks that blip in.

If the researcher had kept going, the effect might well have faded back towards nothing as more data arrived. Deciding, after the fact, that the study had 'enough' data the moment it looked good is a subtle form of the same problem as trying several analyses and keeping the best one. The data were honestly collected. The decision about when to stop was not fixed in advance, and that is what breaks the statistics.

Why No Lie Is Needed

None of this requires dishonesty because each step, taken on its own, feels like ordinary scientific judgement. Researchers are trained to clean messy data, to choose sensible statistical models, and to follow up on interesting leads. The habit of stopping once a story looks good, and not mentioning the paths that did not work, is not usually a decision to deceive. It is closer to a blind spot, built from genuine belief in the hypothesis and a natural tendency to notice evidence that confirms it more than evidence that does not.

This is also why p-hacking is hard to catch from the outside. A paper describes the single path that was reported, not the garden of paths that were available. Unless a researcher has registered their analysis plan in advance, there is usually no record of how many versions were tried, which outcomes were quietly dropped, or when exactly the decision to stop collecting data was made. The honesty of the researcher is not really the issue. The structure of the decision-making is.

Common questions

Is p-hacking the same thing as fraud?

No. Fraud involves inventing or altering data, which is a deliberate act of deception. P-hacking usually involves real data and genuine statistical choices, made without any intention to mislead, that happen to inflate the chance of a false positive.

Can a careful, well-meaning researcher p-hack by accident?

Yes, and this is the normal way it happens. Trying a few sensible analysis options, checking results as data comes in, or quietly dropping an outcome that did not pan out can all feel like ordinary scientific care rather than manipulation.

How would I spot p-hacking in a paper I am reading?

Look for signs that the analysis was not fixed in advance, such as many subgroup results reported with only one highlighted, exclusion criteria that seem to shift between analyses, or a sample size that looks oddly specific rather than planned. A preregistered analysis plan that matches what was actually reported is a strong reassurance against this.

Does preregistering a study actually stop p-hacking?

It helps a great deal, because it forces the researcher to commit to one analysis path, one set of outcomes and one stopping point before seeing the results. It does not eliminate every judgement call, but it closes off most of the garden of forking paths described above.

Why does this matter if the researcher was not trying to cheat?

Because the result can still be false, and false results get cited, repeated and sometimes acted on. The point of understanding p-hacking is not to assign blame but to recognise that good intentions do not protect a study from flawed statistical structure.

Read next

• What a Meta-Analysis Actually Does to the Evidence

  A meta-analysis combines results from many studies into one estimate, but it only clarifies the truth if those studies were asking the same question well.

• Observational Study vs Experiment: Why Diet Advice Flip-Flops

  Most nutrition headlines come from observational studies that track habits, not experiments that test them, which is why the advice keeps changing.

• What a Randomised Controlled Trial Actually Buys You

  A randomised controlled trial sorts people into groups by chance, which cancels out hidden differences and lets researchers credit any gap to the treatment.