How to Not Suck at Evaluating Claims

From the Brockhaus and Efron Encyclopedic Dictionary by Double--M

You don’t need all this stuff to properly evaluate the veracity of a claim.

Most people really suck at evaluating claims.

It’s not their fault – to be honest at least in the U.S. very little in our society or education systems properly prepares us to evaluate claims and make proper reason-based decisions.

The good news is, even if you’ve been awful at it your entire life, you can easily learn how to evaluate claims properly by starting to use some basic guidelines.

But first, why is it even important to have this skill?

Sic Semper in Excretia Sumus, Solum Profundum Variat.

It is staggering at times just how much bullshit there is out there – even more staggering is how willing people are in general to accept it.

Whether it’s Deepak Chopra making rambling, nonsensical claims about quantum mechanics, Dr. Oz touting the latest scientifically unsupported miracle health fad or a pop up ad yelling to you about a guy who “language professors hate” because he came up with the secret formula for easy language acquisition – there are a lot of people asking you to believe a lot of claims out there.

Some of these claims are legitimate, others are not. When the consequences of failing to properly evaluate a claim can range from something mildly annoying like wasting $10 to something dire like serious injury or death it’s vital to separate out the two as best as we possibly can.

While this skill applies across the board to essentially all aspects of life due to the manner in which we’re assaulted with information and advertisements on a daily basis I think it’s extra important for those interested in fitness and looking to improve their health because the fields of health, fitness & nutrition are some of the most saturated with unsupported and potentially dangerous claims.

So what are some easy ways to begin separating out the claims you should probably accept from those you probably shouldn’t?

The Bullshit Detection Kit

I have to give credit to the incomparable Carl Sagan and his excellent book The Demon Haunted World: Science as a Candle in the Dark for the concept of a bullshit detection kit (sometimes also called the baloney detection kit). The idea is to have a set of tools that can be easily applied to new claims that you come across to help you determine if they are likely to be true or not. Our bullshit detection kit differs a little from his, but the concepts are the same.

A note on the word ‘true’ – ‘True’ here means supported by reality. If you’re the kind of person who vociferously claims there is no objective truth and that everyone determines their own equally viable personal truth I invite you to decide gravity is a subjective truth and step out of a second story window.

The bullshit detection kit is in no way infallible, but it can offer a little bit of guidance in the right direction. As we unpack our kit we’ll apply each piece to a fitness claim that was made recently enough to still be in memory but far enough back that it’s been solidly disproven – the efficacy of power bands.

So let’s get to work.

  • Who’s Making the Claim – The first thing to look at when evaluating a claim is who’s making it and with that do they benefit directly from acceptance of this claim. In other words, if the only people making the claim are the manufacturers and sellers of a product then that claim should be more suspect than one independently verified by a source with nothing to gain by verifying that claim.

    In the case of power bands while companies made claims that the efficacy of their bands was scientifically demonstrable, they never actually released any studies or proof that didn’t come from a place funded entirely by the company that sold them. When the only people making the claim are the people trying to sell the claim to you and they’re only backed up by their own non-independently verified research, you should be suspicious.

  • Has the Claim Been Tested Properly – Whenever possible you should always look for a properly controlled double-blind study on whatever claim you’re evaluating. A proper double-blind study with a control group does the best job of guaranteeing that any confounding factors (variables that affect the outcome of the test and obscure the validity of the claim) are removed and only the claim is being tested.

    Anecdotal evidence or evidence from personal experience is insufficient – people are very easily deceived and we’re always the best at deceiving ourselves. Someone may tell you that a claim is true, someone may tell you that something worked for them, they may even genuinely believe what they’re telling you but the claim still may be false.

    It’s equally important that the claim be tested properly because then other independent testers with no personal stake in the outcome of the testing can run the same test. If they come to the same conclusions and get the same results then that lends credibility to the truth of the claim. If not, either something was done improperly in the testing or the claim is more likely to be false.

    Many power band companies used a kinesthetic trick where they would put the band on you and tug on your arm. You would appear to be much stronger with the band on and, hopefully for the sales person, be so impressed by this ‘proof’ that you’d drop $50 or so on a 10 cent piece of rubber and tin foil. When these same products were put to a double-blind study wherein the participants didn’t know if they had the band on or not, there was absolutely zero effect.

  • Is There Definite Causation – The post hoc ergo propter hoc (after this, therefore because of this) fallacy is a common misstep in the evaluation of the truth of claims. It’s due to the tendency of people to conflate correlation and causation – or just to see causation where there isn’t any.

    A good example might be that you have a headache. You take two aspirin and spin around ten times. Your headache goes away, but which actually cured you? This can even happen sans the aspirin since sometimes headaches just go away. You might spin around ten times and your headache disappears, but did the spinning actually help or did the headache just go away on its own or for another reason.

    Another good example might be the fact that there’s a strong correlation between having black hair and eating a diet high in rice. Does that mean that having black hair makes you more likely to eat rice or that eating rice makes you more likely to have black hair? Clearly not – it just happens that black hair is genetically more common in Asia and agriculturally rice is a staple food there.

    In the case of power bands they claimed that big name athletes that used them were performing better in games and pointed to statistics of professional athletes who had bought into their marketing to back it up. Even though they pointed to improvements in the performance of these athletes it still didn’t prove that the bracelet itself was having any actual effect.

  • Does the Claimant Rely on Arguments from Authority or other Fallacies – A common tactic for puffing up a claim with little actual substance behind it is the argument from authority. The argument from authority is essentially the argument that because someone smart or important says it’s true, it must be true.

    While in certain cases the support of a knowledgeable expert can add credence to a claim – for example a respected astrophysicist agreeing with a claim about astrophysics – it should not be the sole reason for accepting it.

    Even if a ton of experts agree a claim is true, it may still not be. Even if everyone in the world agrees that a claim is true, it may still not be. A claim must stand on its own merits and not rely entirely on its acceptance by others.

    Power bands companies, either through genuinely convincing them or via large payments, secured the endorsements of a lot of pro athletes. This was a big selling point for most people who thought, “If NBA star X wears one it must work! I’ll buy one too!” Unfortunately for the consumer, just because someone else says something is true doesn’t actually mean it is.

There are a lot more ways people who make false claims will try to convince you of there veracity and many more ways to help sort out the true from the false – but this will at least get you started. If you have any other tips you think people should keep in mind in order to not suck so bad at evaluating claims chime in and share them!

Photo Credit: Double–M

Adam is a former English teacher turned personal trainer and writer. He’s addicted to learning, parkour and martial arts. In addition to being a voracious bibliophile Adam’s fascinated by anything related to health, fitness and language. When not studying or training he can usually be found curled up with a good piece of fiction. You can e-mail Adam at Adam@RoadtoEpic.com