A Layperson’s Guide to the Scientific Literature – Part 3
July 23rd, 2008
Before I start onto my discussion of metanalyses and the rest, I’d like to take a few moments to answer some of the common criticisms that skeptical evaluations of autism research often stir up.
The most common criticism, almost reflexive in its regularity, is the “Are you calling me (or ‘these parents’) a liar?” complaint. A frequently seen variant is the “We know what we see!” complaint.
Let me start out by saying that I don’t believe that any of the parents who report improvement, resolution or even “miraculous cure” of their children’s autism are liars. As far as I know, they are reporting what they see (or think they see) and (most importantly) their interpretation of what they see.
People - sad to say - are prone to misperceptions and misinterpretation.
As any lawyer could tell you, eye-witness accounts are often the least accurate part of any evidence presented in court. This is not because they are lying (although some certainly are) but because our brains “edit” what we see to fit our preconceived notions of “reality”.
Repeated studies have shown that, when presented with a rapid sequence of images of, for instance, a cup falling and breaking, people will fail to notice if one – or even several – of the images are out of order. When challenged, they will usually vigorously deny that there was anything unusual about the sequence. They aren’t “lying”; their brains have simply “edited” the information they saw to make it conform to what “common sense” dictates.
Additionally, people are “hard-wired” to see patterns, even when no patterns exist. This need to find a pattern in random events has led to any number of superstitious and non-rational beliefs, from astrology to witches. The need to find a “cause” for what are, in fact, random events continues to the present day (and will no doubt continue into the future). People want to think that something – God, witches, the stars or “the government” – is behind the bad (and often the good) things that happen in their lives.
This is no less true with parents of autistic children than it is with the rest of humanity. They want “answers”, they want a “cause” and they want to find a recognizable pattern in the random events of life. Autism plays into these needs by being - so the genetic studies suggest – due to random genetic mutations and by its start-and-stop developmental progress.
The latter is probably not unique to autism, since even typical child development shows periods of rapid skill acquisition followed by periods of relative stasis. In autism, this pattern may be accentuated or it may be that the development of autistic children is watched so closely that the pattern seems to be accentuated. Either way, the random pattern of improvement and stasis also feeds the “pattern-seeking center” of our brains. Anything that is done or changed prior to a period of improvement is perceived to have caused the improvement; likewise, anything done prior to a period of stasis is perceived to have caused the stasis.
This leads a number of parents to truly believe that some intervention – which in all likelihood did nothing – caused their child to either get worse or get better. In addition, there is a phenomenon I like to call “The Lucky Stockbroker Syndrome” that leads to reinforcement of these individual experiences.
The Lucky Stockbroker Syndrome is based on the following analogy:
A stockbroker decides to try a new money-making scheme. He gets a list of ten thousand (10,000) potential clients and sends half of them a letter (or e-mail, if you like) telling them that a certain stock will rise in value over the next week. To the other half of the list, he sends a letter saying that the same stock will FALL in value.
The next week, he sends letters to the half of his list (5000 people) that got the CORRECT prediction – again, half of them are told that a certain stock will go up in price and the other half are told that the stock price will fall. This goes on for a total of six weeks.
At the end of this time, he has a list of 156 people who have – by random chance – received six consecutive correct predictions about stock prices. He then offers these people a five-year subscription to his stock-picking service, which they gladly purchase, thinking that he has some amazing system for predicting the stock market.
The same thing is happening in the world of autism therapies. The parents that try a particular therapy – such as chelation or dolphin therapy – prior to their child coincidentally having a period of improvement will conclude (erroneously) that the therapy caused the improvement. Those that try the same therapy and don’t see an improvement generally “move on” and try something else.
Even worse, some practitioners exhort the parents to “stay the course”, “give it a chance to work” or “don’t leave before the miracle”, which leads them to keep trying the therapy until – again, by shear coincidence – their child goes into a period of improvement. Thus, the “pattern-seeking center” of their brains sees – erroneously – a “pattern” of the therapy causing improvement. In fact, they are simply fooling themselves into seeing a pattern that isn’t there.
Again, I have to emphasize that this isn’t stupidity, gullibility or a lack of intelligence – it is simply how human brains work. THIS is why science is so persnickety about placebo controls, blinding of observers and randomizing subjects. We do it to keep from fooling ourselves, which we are all too capable of doing, given the chance.
Now, on to the show!
Meta-analyses:
A meta-analysis is a way of combining the results of a number of smaller studies in order to get more meaningful results. In the usual scenario, there is a question (e.g. “Does guinea pig therapy help in autism?”) that a number of smallish studies have tried to answer but were individually unable to reach statistically significant conclusions. This is most often due to small numbers of subjects, which is where a meta-analysis can help.
The meta-analysis gathers together the results from a number of smaller studies and adds them together to get – in the ideal situation – a statistically significant answer. Again, this is what happens in the ideal situation.
The biggest problem with doing a meta-analysis is how to sort the wheat from the chaff. Many smaller studies are also not very well planned and executed. They may have problems with subject selection, or their outcome measures may be weak or they may not have randomized their subjects well enough. Including poor studies in a meta-analysis will lead to the all-too-familiar experience of “garbage in, garbage out” (GIGO).
On the other hand, weeding out too many studies will lead to a meta-analysis that fails to reach a meaningful conclusion. This is better than reaching the wrong conclusion, but still not very satisfying.
Another problem in meta-analyses is that the individual studies may be (and almost certainly are) looking at different things. They may have different criteria for what or who they include in the study; they may have different outcome measures; they may have different criteria for classifying their results. All of these can lead to a meta-analysis trying to compare “apples and oranges” (or worse, “apples and orangutans”).
The bottom line is that a meta-analysis is only as good as the worst study it includes.
Meta-analyses should NOT be used as a substitute for a coherent large-scale study. At best, they can be used to give a “second look” at a question, to decide if it is worth expending the time, effort and money to answer it. A good meta-analysis can suggest that there is (or is not) a detectable difference that is worth looking at in depth with a real study.
Pilot Studies:
Another way of deciding if a question is worth studying in depth is to do a pilot study. Pilot studies are – by definition – smaller and less rigorous than the double-blind, placebo-controlled studies that could definitively answer the question. Many interesting questions (e.g. “Does television watching cause autism?”) are not inherently plausible enough to warrant launching a full-scale study to investigate them, so a smaller “pilot study” is done to gather enough data to determine if there is sufficient plausibility to justify a larger study.
In order to reduce the cost, pilot studies almost always have fewer subjects than would be needed to give a definitive answer. Or, they omit controls, “piggyback” on clinical work, or omit observer and/or subject blinding. Maybe all of the above. At any rate, pilot studies do not meet the standards needed to give a definitive answer to a question. That’s because they’re not meant to give an answer to the question – they’re just meant to give an indication of whether or not it would be worthwhile to start a larger study to get the definitive answer.
So, pilot studies, in and of themselves, are fine, so long as you remember that they do not give answers to the question – they just indicate if there is any point in doing a larger study. A true pilot study can only give one of two answers:
[1] The question is worth further study.
[2] The question does not appear to be worth further study.
Answer [2] needs more explanation. Because a pilot study has – almost by definition – a small number of subjects, it cannot detect small differences that a larger study would be able to pick up. For that reason, a pilot study can never truly “close the door” on a question. However, it can say that the effect size is smaller than a certain number, which depends on the number of subjects and the study design. It’s then up to the researchers to decide if it is worth trying to find a smaller effect.
Case Reports / Case Studies:
At one time, almost all of the articles in medical journals were case reports. Now, they are in the minority. This is probably a good thing.
Case reports are essentially anecdotal evidence – a “narrative” of what happened and what caused it to happen written by the people who were involved in the case(s). There is usually some effort made to “dress up” the anecdote in scientific garb by citing references and showing laboratory data, but there are no controls, no blinding and the subjects are definitely not selected at random.
Don’t get me wrong – a lot of good information can come out of case reports. They are especially useful in conveying to clinicians what to do (or not do) in certain situations. They can even point out possible promising areas of diagnosis or treatment to research.
But case reports aren’t high quality data.
The biggest limitation of case reports is that there is no indication of how the subjects were selected. The authors themselves may have no idea how the subjects were selected – most of the time, case reports are written because the author(s) thought they “saw a pattern” in their clinical practice. Again, beware the “pattern-seeking center” of every human’s brain – it often sees patterns where none exist.
The concerns I have about case reports revolve around two issues:
[1] The effects observed may have simply been random – the treatment may not have had any effect or the exposure may have been coincidental rather than causative or….. you get the point.
[2] The subjects in the report may not be representative. This could be inadvertent or deliberate. The worst case is that the author(s) may have “cherry-picked” their patients to find the results they wanted – either consciously or unconsciously.
For example, a clinician in the ER notices that there are “runs” of “non-accidental trauma” (medico-legalese for deliberate acts of violence) that coincide with the full moon. These could then be reported as being due to the full moon. However, the clinician fails to notice that such “runs” occur at random – unrelated to the phase of the moon – due to random “clustering”. However, the “runs” of violence that don’t occur during the full moon are unconsciously discounted by the “pattern-seeking center” of their brain. [By the way, I have heard this very “hypothesis” from doctors and nurses on several occasions. It has been the subject of numerous studies in the medical literature.]
Well, that’s all for now. I hope that I have been of some service to people struggling with the intricacies of the scientific literature.
For next time (or whenever I get around to it): How some “alternative” practitioners duck the responsibility (and blame parents) for therapeutic failures.
Prometheus
Filed under: Autism Science, Critical Thinking, Myths About Science | 8 Comments »