Analyze the Evidence: Our Introductory Guide

Our handbooks are divided into many parts. One of these parts is the Evidence section.

The Evidence section is there to describe the quality, quantity, and consistency of the evidence backing a (possibly effective) treatment.

Each Evidence section may have some statements that are unfamiliar to some people.

Here, we go over what these statements mean.

Types of Studies

Clinical Trials

These are scientific studies that help reveal a cause and effect relationship. For example: green tea consumption (the cause) directly leads to a drop in blood pressure (the effect).

There are many different types of clinical trials. Some clinical trials (CTs) are called RCTs: randomized controlled (clinical) trials.

“Controlled” means that the clinical trial includes a comparison group that isn’t given a the experimental treatment. Randomization means that the study participants are randomly assigned to the treatment group or the control group.

In short, randomization and control groups help maximize the chances that a clinical trial will produce an accurate result. In other words: all else equal, a randomized controlled trial is more trustworthy than a simple clinical trial.

Observational Studies

These are scientific studies which are often used to reveal an association between two things. For example, gum disease is associated with high blood pressure.

This association doesn’t guarantee that gum disease causes high blood pressure directly (although it might). Other factors could explain this association. For example, people with gum disease may be more likely to smoke and it’s the smoking that leads to high blood pressure.

Generally speaking, there are three types of observational studies: cohort, case-control, and cross-sectional.

Of the three, cohort studies provide stronger evidence for an association between two factors.

Number of Studies

Generally speaking, the greater the number of studies on a topic—and the greater the total number of participants—the more we can trust their collective results.

The Comparison

Let’s say that olive oil causes a drop in blood pressure. Whenever you read that a treatment causes something, you should always ask yourself: “as compared to what?”

In high quality studies, people given the treatment are always compared to other people given something called a “control”.

In some studies, the control is a placebo. A placebo is sometimes referred to as a “sugar pill”. It’s a substance that’s supposed to have no effect on a person’s body.

If a treatment isn’t better than a placebo, this means you’re probably better off saving your time and money for something that actually works.

If a treatment is more effective than a placebo, this means the treatment might actually help you.

In other studies, the treatment is compared to an “active control”. This is a substance or factor that has a known effect on a person’s body. For example, it’s known that aerobic exercise can lower a person’s blood pressure.

So, a study might compare olive oil to aerobic exercise to see if olive oil is worse than, equal to, or more effective than aerobic exercise at lowering a person’s blood pressure.

Therefore, if you read that olive oil is good at lowering people’s blood pressure, ask yourself: “as compared to what?”

If there was no control group in the studies in question, then these results may be very misleading.

If the olive oil is better than a placebo, then the olive oil is probably better than doing nothing at all.

If the olive oil is better than an active control like exercise, then the olive oil is probably a very good treatment.

Advanced topic: in some studies, researchers want to find out if two treatments taken together are better than one.

For instance, a group of people taking blood pressure medication alone (active control) may be compared to a group of people taking blood pressure medication AND an additional treatment like olive oil.

In these cases though, it’s imperative that the active control group is given a placebo in order to match the olive oil in the other group.

This is because the olive oil might “lower” a person’s blood pressure due to the misleading placebo effect. This is when an ineffective substance appears effective because a person believes that it works.

So if the active control group isn’t given a placebo, they cannot experience this misleading drop in blood pressure. This will make the olive oil appear effective, when it isn’t.

Estimated Effects

The effects (results) in this handbook, like a drop in blood pressure by 5 mm Hg, are estimates only.

Therefore, their accuracy and predictability is rated based on the quality and consistency of the available data.

How Trustworthy Is the Estimate?

Strong Evidence

These estimates are most likely accurate. Further research is unlikely to change these numbers much, if at all.

Medium-Strength Evidence

These estimates are probably accurate. There’s still a chance that better research will change these numbers by quite a bit, though.

Weak Evidence

These estimates might be inaccurate. Most likely, future research will change these numbers substantially. What’s unclear is by how much and in which direction (up or down).

Mixed-Quality Evidence

It’s unclear if the data for these estimates is overall low, medium, or high-quality in nature.

How Predictable Are the Results?

Very Consistent Results

Very similar effects were found in almost all of the studies. This means that most people will probably experience these same (or very similar) results.

Reasonably Consistent Results

Relatively similar effects were found across most of these studies. However, some of the studies provided very different results. This means that most people will probably experience an effect somewhere around the estimated one. But some people may experience very different results.

Inconsistent Results

The results of the studies were very different from one another. While the average of everyone’s results will likely be somewhere around the estimate, each person’s individual results may be very different from the estimate.

For example: a treatment may be very effective for one person but only slightly effective for another. Or, it may be effective for one person but not effective at all for another.

These are general definitions only. Most importantly, evidence ratings are partially subjective and consistency ratings are by no means fail-proof. They are approximations only.

Why Are Some Results Inconsistent?

Here are just a couple of potential explanations:

1. The various studies on the point used related, but slightly different, treatments like refined olive oil vs. extra virgin olive oil. So the exact treatment may alter the effects a bit.

2. The studies may have used the same treatment (extra virgin olive oil) but examined slightly different people. One study may have focused on those with normal blood pressure while another focused on those with elevated blood pressure. A person’s medical conditions could influence the way a treatment affects them, leading to inconsistent results across studies.