Assertions Based on Single Studies – BS Vaccine For Fitness

Previous: What Should I Read to Learn About Fitness? | Table Of Contents

I wonder how many of my readers have recently read a news or magazine article, on the web or elsewhere, explaining the results of one study and making concrete conclusions based on that one study. All of you? I figured as much.

A study in Australia revealed that young women fight off colds better than young men. Case closed. Not. First of all, “a study in Australia” is not an appropriate reference. If there is no reference to the actual study, the article should be considered to have no credibility. Second, there is no way that ONE study could possibly “reveal” conclusively that young women have colds that go away quicker than young men.

I know that you probably don’t care about a study that says young women have shorter colds. But what if the study reveals something you’d like to believe? What then? We’ve already talked about how stubborn beliefs are and how tough they are to abandon. Well, it is just as tough not to cling tightly to any little tidbit that confirms our favored reality. Trust me, I know.

For instance, consider so-called toning shoes. Skechers, Shape-Ups, Reebok, EasyTone, MBT…you’ve seen the ads. There was one study from The American Council on Exercise which suggested that, low and behold, those shoes don’t do any of the things they claim to do, such as activate the calves and hamstrings better and burn more calories. The study by Porcori, et al.¹Porcori, John, PhD., et al. “Will Toning Shoes Really Give You A Better Body?” Ace Fitness. Web. 2012. was divided into two trials, with 12 females each, one group for muscle activation and one for caloric expenditure. Although I have very little doubt that these shoes are complete and utter crap one study of 12 isn’t enough for me to start proclaiming that we have conclusive proof. This study may be a good indication but it is far from conclusive proof. So, no matter how much we may want to believe that these shoes are crap, an article claiming that they are crap based on this one study and nothing more should not be considered a credible one.

---

Bullshit Example 3: Absolute Change vs Relative Change

Beware of percentages used in fitness information. Percentages often lie. They are, in fact, a favorite tool of the bullshitter and when you see them, they should always give you pause.

Bullshitters often compare percentages. It is almost always inappropriate to do so. Why? Because the base numbers from which the percentages are derived are important. Would you want 75% of my deadlift or 50% of Andy Bolton’s? If you’re smart, you’ll take the lower seeming number. Andy Bolton lifts over 1000 lbs. I’ve eked out a little more than half of that on my very best days, which may be long past. You shouldn’t accept 100% of my income either if offered an alternative of 0.5% of Bill Gates’ income. This is easy to understand because it is obvious.

The better way to bullshit with percentages is with relative percentages. If I told you that a lifter increased his reps, in ONE workout, by 67%, you’d be suitably impressed. You’d be angry with me if I then revealed that I meant they went from 3 to 5 reps.

Let’s say I told you I increased a lifters max lift in the barbell squat by 33% in three years. You might think, well, that’s not too bad, but not really that very impressive, either. But, what if the lifter started at 300 in year one, and then progressed like this:

Year 1: 300
Year 2: 375
Year 3: 400

Woah. Hold up a minute. Year one to year two, we had a 25% increase in the lift. Then, in year two to year three, we had a 6.7% increase. Those kinds of results may be reality. If you are a person who has been strength training for a few years, you know that. You get the reality of strength training. But, a bullshitter counts on the ignorance of his audience. It is better to tell them about a 33% increase, overall, then tell them about a 6.7% increase over an entire year. So, where you start matters, and you start wherever will get you a more impressive number.

I can use percentages to make almost nothing sound very impressive. Hey, I increased my client list by 100% last year! I went from one to two clients. Relative change, as expressed by percentages, makes that a 100% difference. But it is actually only an absolute increase of 1 client. If you go from 100 lbs on your bench press to 200 lbs it’s an increase of 100%. But if we compare your results to another lifter on another program, who went from 125 to 215, we get an increase of 72%. We then use those numbers to say you got around 39% better results, because of the difference between 72% and 100%. In reality, given the starting points and the ending points, the two results are actually very similar. And the second lifter, in absolute terms, is still stronger. We cannot say anything beyond that.

We see much more of these misleading percentages in messages about our health. We may hear that eating cured meats daily increases our risk of pancreatic cancer by 20%. In fact, every day, you will probably come across a statistic that tells you how your risk of some scary illness, especially cancer, can be increased. When we see numbers like 20%, we easily start thinking of those numbers in concrete terms, like “if I eat bacon every day I will have a 20% risk of pancreatic cancer. But, in reality, you do not even know your risk of pancreatic cancer. If you were to find out, as is likely, that the actual risk for your segment of the population is 1 to 1.4%, at most, you may feel a bit different. That means eating cured meats would increase your risk of pancreatic cancer to 1.7%. If it is a true statistic, which is probably is not.

Say you found out that your risks of colorectal cancer actually dramatically increases as you age, regardless of bacon. You’d suddenly be more worried about your next birthday than bacon or anything else deemed to increase your risks. And, aging does increase your risk. It’s still not huge, but aging continues to be the key. Say you’re a male at age 60. In the next ten years, you have a 1.32% chance. In the next 20 years, you have a 3.08% chance. In the next 30 years, you have a 4.39% chance. In other words, there’s not that much of a chance. Still worried about bacon?

Don’t hold me to those statistics. They may have changed. They are certainly not far from accurate.

---

To be fair, concerning the referenced study, the author of the related article ²Anders, Mark. “Will Toning Shoes Really Give You A Better Body?” Ace Fitness. Web. 10 Jan. 2012. <http://www.acefitness.org/certifiednewsarticle/720/>. on the Ace Website complains that the studies performed by the manufacturers of these shoes are not peer-reviewed, failing to see the irony in the fact that neither is the Ace study, having not been published in a peer-reviewed journal! Something is amiss, is it not? In fact, according to an article by Denise Mann ³Mann, Denise. “Toning Shoes: Can Shoes Tone Your Butt and Legs?” WebMD – Better Information. Better Health. Web. 10 Jan. 2012. <http://www.webmd.com/fitness-exercise/features/truth-about-toning-shoes>. the Skechers company president actually called out the study based on this fact. You have a problem when the company guilty of BS in-house “studies” proving their products effectiveness actually call you out on your study not being peer-reviewed. Could Ace be biased as an organization? Sure, it’s possible. Should we take Ace funded studies as seriously as other studies funded by independent sources? Maybe not. After all, Ace is about producing fitness professionals who make a living helping people get fit. Still, this does not mean that the study is completely bogus. It’s just not enough and questionable, all studies are, especially until they are replicated with similar results.

Admittedly, if you spend over 100 bucks on any of these shoes, I’ll still think you’re silly. I don’t need studies, most of the time, to tell me when a fitness product is not worth the material it’s made from. But I’ll need to wait for some confirming evidence before I start referencing research to support my opinion. Right now, it’s just an opinion backed up by one small study, which may have its flaws, including its lack of scholarly publication.

From my perspective, there are two problems. One is that news and magazine articles routinely fail to properly reference the research they mention. Studies absolutely must be properly listed, with a link, complete journal citation, or both. If you cannot easily go find the study and read it yourself, the article’s validity is suspect. Many times you will find, even once you search out the study in question, that the conclusions of the study’s authors are not anything like the conclusions of the magazine article. This happens constantly, in fact, all over the web.  Usually, people look for articles that seem at a glance to support what they already believe, and have no regard for what the study’s authors actually conclude, if, that is, they actually conclude anything at all!

The other problem, from my perspective, is that this practice nothing more than content baiting. The public is being led to believe that huge discoveries are being made in the realm of fitness and health almost on a daily basis. Single small studies are used for no other purpose than to support a provocative headline to draw you, the reader, in. These news and magazine outfits do not care if what they report is accurate and they surely do not care if it makes a difference in your fitness pursuit. If they did, they would research the articles much more widely. The typical formula for these articles goes like this:

1. A vague reference to some study, usually only with a place named, like Australia, or the last name of one of the researchers
2. Vaguely related quote by an expert in the same field, usually someone who has a vested interest in confirming the conclusions of the said study.

I would love to be able to have a subject for an article every time someone does a study related, in some way, to fitness. Then, all I’d have to do is check Pubmed every day looking for something juicy and write a hastily worded post to cover it. I’d have a website filled with content and provocative headlines. I wouldn’t even need to stick strictly to fitness. What about a “recent study” that shows that pregnant women who eat chocolate have healthier babies with better temperaments. What do you think the response to that would be? Yippee, I can eat chocolate! Like a pregnant woman needs an excuse, anyway. ⁴Raikkonen, K. “Sweet Babies: Chocolate Consumption during Pregnancy and Infant Temperament at Six Months.” Early Human Development 76.2 (2004): 139-45. Web. <http://gsdl.sld.cu/collect/chocolat/index/assoc/HASH42f5.dir/doc.pdf>.

To read and analyze single studies, though, there is a particular skill and expertise required, which the average journalist does NOT possess. There are, however, many good scientific blogs related to fitness that do a much better job. So, all you need to do is skip the big headline at the top of the Google pile and look for some blog entries by people like, say, Bryan Chung, who, it just so happens, weighed in on the barefoot shoe thing. If you aren’t familiar with these, they are products like Vibram FiveFinger shoes. It just so happens there is a study from Ace on the barefoot shoes as well. ⁵Anders, Mark “Like Barefoot Only Better/” Ace Fitness. Web. 2012 <https://www.acefitness.org/certifiednews/images/article/pdfs/ACEVibramStudy.pdf>

Researchers themselves are prone to a kind of halo effect. We envision the pure scientist, filled to the brim with integrity and passion, working diligently to uncover the secrets of the universe or, in this case, fitness and nutrition. Well, scientists can get up to no good as well. We’ll talk about that in the next section.

Next: Self Plagiarism, Salami Slicing, and Publish or Perish