Reading a self-help book critically

Who is making the claim?

The National Center for Complementary and Integrative Health, part of the US National Institutes of Health, offers a set of questions for judging health information online: what the source is, how you know the information is accurate, whether experts have reviewed it, how current it is, and who runs and pays for the website. The same questions can reasonably be put to a book, its author and its publisher, including whether anyone stands to profit from a claim.^[1]

Ioannidis listed greater financial and other interest and prejudice among the conditions in which a research finding is less likely to be true. In the case of a book, this can include income from a method, course or product built around the claim. It does not make a claim wrong. It is a reason to look for independent evidence from people with nothing to gain, and to be wary where the only evidence comes from the person selling the approach.^[2]

Study design, sample size and anecdote

In a 2005 essay, Ioannidis argued that a research finding is less likely to be true when the studies in a field are smaller, when effect sizes are smaller, when there is greater flexibility in designs, definitions, outcomes and analysis, and when many teams compete to find significant results. His simulations suggested that for most study designs and settings it is more likely for a research claim to be false than true. Useful questions follow for any study a book cites: how many people took part, was there a comparison group, and were the outcomes chosen in advance?^[2]

The depression literature shows how design changes results. Gellatly and colleagues found larger effects of self-help where allocation concealment was unclear, where outcomes were rated by observers and where the comparison group was a waiting list. The same programme can therefore look better or worse depending on how it was tested, and in that review, studies with a waiting-list comparison were linked to larger estimates.^[3]

An anecdote, such as an author or reader describing how a method changed their life, cannot show what would have happened without the method. It also says nothing about the people for whom the method failed, because those stories are rarely told. This is a point of reasoning rather than a finding from a cited study, and it is why the trials discussed on this site compare groups of people rather than relying on individual accounts.

The replication problem in psychology

The Open Science Collaboration (2015) attempted to replicate 100 experimental and correlational studies from three psychology journals, using high-powered designs and the original materials where available. Replication effects were half the size of the original effects. Ninety-seven per cent of the original studies had statistically significant results, compared with 36 per cent of the replications, and 39 per cent of effects were subjectively rated as having replicated the original result.^[4]

The project also reported that replication success was better predicted by the strength of the original evidence than by the characteristics of the original or replicating teams. This suggests that a result resting on weak original evidence deserves more caution than one that has been repeated by independent groups. A book that builds a method on one striking experiment is therefore on less secure ground. That last point is an inference from the finding, not a result the project reported directly.^[4]

Three well-known cases

Power posing was the idea that holding an expansive posture changes hormones and behaviour. Ranehill and colleagues (2015) reported no effect on hormones and risk tolerance in a large sample of men and women. Dana Carney, first author of the original 2010 study, later published a statement saying she does not believe the effects are real. She described the original sample as tiny, with a final N of 42, and said participants were run in chunks while the effect was checked along the way. She wrote that the evidence against the existence of power poses is undeniable.^[5][6]

Ego depletion is the idea that using self-control in one task weakens performance on a later one. In a preregistered project across 36 laboratories (N = 3,531), Vohs and colleagues found a nonsignificant confirmatory result (d = 0.06), and the data were four times more likely under the null hypothesis than under their alternative. An exploratory analysis of the full sample gave a small significant effect (d = 0.08), but the data were about equally likely under the null and alternative. The study tested two specific procedures, so it does not settle every question about willpower.^[7]

The popular 10,000 hour rule is treated here as one version of an older view, which Macnamara, Hambrick and Oswald describe as a frequent topic of popular-science writing: that differences in performance largely reflect differences in deliberate practice. Their 2014 meta-analysis found that deliberate practice explained 26 per cent of the variance in performance for games, 21 per cent for music, 18 per cent for sports, 4 per cent for education and less than 1 per cent for professions. The authors conclude that practice is important but not as important as has been argued. The meta-analysis tests that underlying view, not the specific figure of 10,000 hours.^[8]