Secretin Rises from the Grave!
September 30th, 2008
From time to time, I receive comments like:
"Why are you so opposed to studying ‘alternative’ therapies that have a low probability of working? If the study shows the therapy doesn’t work, people will stop using it."
Oh, if that were only true!
Let’s take – for example – the use of secretin as a therapy for autism. This has been repeatedly shown to be ineffective (no better than placebo), yet it keeps rising from the grave like some unquiet corpse.
Although it’s a bit early for Halloween, Stephen M. Edelson (PhD) of the Autism Research Institute has once again raised the "secretin-cure-autism" hypothesis from the dead. It is now roaming the "alternative" autism therapy community, munching on the brains of the "reality challenged". You can read it yourself in the ARI newsletter, Autism Research Review International (volume 22, number 2, pages 3 and 6).
In his editorial, Dr. Edelson starts with an abbreviated (and redacted) timeline of the secretin "controversy" (he says "controversy", I say "debacle"). He then goes on to explain why his unscientific parent survey of treatment effectiveness is a significantly better measure of the effectiveness of secretin than the over twenty studies that have been done to date. Among his "reasons" are:
[1] The largest studies (including the Repligen Corporation’s Phase 3 study) used human secretin instead of porcine secretin. Porcine secretin, you see, was what was used in the earlier, smaller, less well-designed studies that showed some effect. More about this later.
[2] The studies didn’t focus on the children with gastrointestinal problems – specifically chronic diarrhea and chronic constipation – which (again, in the few studies that showed an effect) were the children who showed the most effect.
Secretin Timeline
For those who are new to the scene, let me give you a capsule summary of the rise, fall, and (at least for a while) death of the "secretin-cures-autism" hypothesis. You’ll notice that many of the details I cover are not included in Dr. Edelson’s timeline. Case reports and studies showing that secretin is an effective treatment for autism are in red , studies that show that secretin is no better than placebo are in blue .
A few definitions that might come in handy:
Double-blind : Neither the subjects (or their parents) nor the researchers knew which treatment the subjects were receiving until after the final evaluations were recorded. This keeps the patients/parents and researchers from unconsciously "shading" their interpretation of the results based on their knowledge of which subjects are getting the drug.
Randomized : The subjects were assigned to the treatment (secretin) or placebo group at random. This helps to prevent a certain "type" of patient from being assigned to the drug or placebo groups. Without randomization, "sicker" children might be assigned to the drug group (or, conversely, they might be shunted over to the placebo group to keep from "screwing up" the results).
Crossover study : In a crossover study, the subjects are assigned to one of two (or more) groups. One group receives the study drug (secretin, in this case) and the other group receives the placebo. After a suitable "washout" period to allow the effects of the drug (if any) to dissipate, the group that initially received the drug gets a placebo and the group that initially received the placebo gets the drug. This allows the subjects to act as their own "controls" (sort of – there are some problems with this assumption).
1996
1996 – Parker Beck undergoes an upper GI endoscopy at the University of Maryland hospital. During this procedure he receives – as is typical – an injection of secretin to stimulate pancreatic secretion so that the gastroenterologist can confirm that the pancreatic duct is open (patent) without having to inject X-ray contrast, which carries the risk of causing pancreatitis.
A few days later, Parker – who was autistic – began to talk for the first time. [Note: at least one source – written by Bernie Rimland – states that Victoria Beck (Parker’s mother) had to INSIST on the secretin injection. This seems unlikely, since secretin is a routine part of upper GI endoscopy AND because she would have had no reason to insist on secretin, as it had never before been associated with improvement in autism.]
According to Ms. Beck – documented in numerous interviews – Parker remained verbal with improved social interaction even though he received only a single dose of secretin until a approximately year later.
1997
1997 (approx) – Parker Beck begins receiving repeated secretin injections.
December 1997 – Repligen Corporation stock trading at $0.78 on volumes of 65,600 shares.
1998
A case report of three (3) autistic children who received secretin as part of endoscopy and showed improvement.
May 19th, 1998 – Victoria Beck and Dr. Karoly Horvath file a patent (#6,020,310) on the use of secretin for the treatment of autism and for the use of secretin levels in the diagnosis of autism. This patent was later sold to the Repligen Corporation.
September 1998 – Repligen Corporation stock selling at $1.31 a share with volumes of about 41,000 shares.
October 1998 – Victoria Beck – Parker Beck’s mother – appears on both "Dateline" and "Good Morning America" to reveal to the world how secretin cured her child.
October 16, 1998 – Ferring Pharmaceuticals - the only manufacturer of secretin at the time – sells its last vial of secretin. The company had decided earlier to stop manufacturing secretin due to the low demand, a decision which should go down in economic texts as an example of epic bad timing.
November 1998 – Repligen stock trading at $1.69 on volumes of 37,000 shares.
1999
January 13, 1999 – Victoria Beck and Bernie Rimland file a patent (#6,197,746) on methods of administering secretin for treating autism that includes (besides injection):
"Other methods and compositions for administering the effective amount of secretin include other transdermal carrier substances, such as gels, lotions, or patches; oral carriers, such as tablets, capsules, or lozenges; inhalation through the nose or mouth (e.g., as an aerosol); suppository forms of secretin and secretin compositions; and using acoustic waves to cause the secretin to penetrate the skin."
This patent was later sold to the Repligen Corporation
March 1999 – Bernie Rimland publishes an "update" on secretin in the ARI newsletter . In it, he mentions how Victoria Beck patented the use of secretin for the treatment of autism and how she fought off attempts by the "university medical school doctors" to try to patent it.
He fails to mention that he is also a holder of a patent on the use of secretin for autism (see above). Dr. Rimland mentions in passing that the Becks sold their patent to Repligen Corporation and donated the proceeds and royalties to ARI, which at that time was owned and operated by Bernie Rimland.
March 1999 – Repligen stock trading at $3.06 on volumes of over 200,000 shares.
1999 – Victoria Beck’s book, "Confronting Autism: The Aurora on the Dark Side of Venus: A Practical Guide to Hope, Knowledge, and Empowerment" is published.
A study of 56 children with autism. Randomized, double-blind placebo controlled, single dose. No effect seen from secretin.
2000
February 2000 – Repligen stock trading at $13.13 per share on volumes of 1.3 million shares.
A study of 56 autistic children. Open label (both researchers and parents knew what the child was getting), single injection. A second double-blind placebo-controlled study was done with 17 children who had responded the most in the first study and 8 new subjects who got an additional injection of secretin or placebo with crossover at four weeks.
"Results of both inquiries indicate that although treatment with secretin was reported to cause transient changes in speech and behavior in some children, overall it produced few clinically meaningful changes when compared to children given placebo injections."
A study of 95 autistic children. Placebo-controlled, randomized, double blind – single injection of either secretin or placebo. "No significant differences in language or autistic behaviour measures were observed at the 3-week follow-up between the groups."
2001
March 5, 2001 – Victoria Beck and Bernie Rimland file a patent (#6,790,825) for "Method of using secretin and compositions made therefrom for the treatment of autism and other neurological, behavioral and immunological disorders". This patent was later sold to the Repligen Corporation.
April 2001 – The Repligen Corporation, which makes recombinant human secretin, completes a Phase 2 study of secretin in autistic children.
May 2001 – Coniglio et al. A randomized, double-blind, placebo-controlled trial of single-dose intravenous secretin as treatment for children with autism. J Pediatr . 2001 May;138(5):649-55.
A study of 60 children with autism. Placebo-controlled, randomized, double-blind – single dose of either secretin or placebo.
"A single dose of intravenous secretin does not appear to have significant effects on either parents’ perception of autistic behaviors or language skills at 6 weeks after injection. Transient, marginally significant improvements in autistic behaviors may occur in some children."
A study of 64 autistic children. Randomized, placebo-controlled, double-blind – each child received either two doses of porcine secretin or two doses of placebo. "No evidence is provided for the efficacy of repeated doses of porcine secretin in the treatment of children with autism."
“Secretin hormone given daily in transdermal cream was associated with marked and sustained developmental progress in an aphasic two-and-a-half year old child diagnosed with autism.”
A study of 12 autistic children. Randomized, placebo-controlled, double-blind – single dose crossover study. Outcome measures included gastrointestinal problems.
"Statistically significant differences were observed on measures of positive affect and activity level following secretin infusion. In general, the autistic children did not demonstrate the improvements described in the initial retrospective report."
October 2001 – Dr. Walter Herlihy of Repligen Corporation (the company trying to get FDA approval to label its recombinant human secretin for the treatment of autism) gave a presentation at the "Defeat Autism Now!" conference in San Diego, CA.
In this presentation, he reported that the Phase 2 trials [Note: Phase 2 trials are intended to show that the proposed treatment is safe in the intended patient population] did not show a statistically significant difference (in safety or efficacy) between secretin and placebo. He argued that the criteria for statistical significance were "arbitrary" and that secretin had shown more effect on younger children with gastrointestinal problems like chronic diarrhea and constipation.
A study of 56 autistic children. Randomized, double-blind, placebo-controlled – crossover study of one injection of either placebo or secretin followed in four weeks by the opposite treatment.
"There was no evidence for efficacy of secretin in this randomized, placebo-controlled, double-blind trial."
2002
April 2002 – Repligen Corporation starts a Phase 3 trial of secretin in autistic children. Based on preliminary (and statistically non-significant) results from their Phase 2 trial, they focus the trial on children ages 32 – 59 months (2 ½ - 5 years) and looked specifically at gastrointestinal problems – primarily chronic diarrhea and chronic constipation.
April 5, 2002 - Repligen obtained FDA approval to market its brand of synthetic porcine secretin (SecreFlo TM) for use in pancreatic assessment.
A study of 6 autistic children. Randomized, placebo-controlled, double-blind crossover study. Each child received six injections – three placebo and three secretin – in random order every four weeks. "In this placebo-controlled study, multiple doses of secretin did not produce any symptomatic improvement."
A study of 19 autistic children, 5 with "chronic diarrhea". Randomized, placebo-controlled, double-blind crossover study – single injection of either secretin or placebo followed by the opposite treatment at three weeks.
Analysis of the entire group showed no statistically significant effect. Limiting analysis to the five (5) children with chronic diarrhea (but not the two with chronic constipation) gave statistically significant results.
A study of 85 autistic children. Randomized, placebo-controlled, double-blind – single dose. The children were divided into trios matched by age and communication level – one child from each trio received porcine secretin; one received synthetic (human) secretin and one received placebo.
"No evidence that either biologic or synthetic secretin provided amelioration of symptoms beyond placebo was observed. This held true when children with and without gastrointestinal problems were examined separately ."
December 20, 2002 – Victoria Beck and Dr. Karoly Horvath file a patent (#7,091,182) for "Method for assisting in differential diagnosis and treatment of autistic syndromes" which is described as "comprising the administration of a therapeutically effective, preferably intravenous, dose of secretin to an individual with autistic syndrome" and "an analysis of an individual’s blood and/or intestinal tissue for the presence of secretin and comparison of the level of secretin to known norms". This patent was later sold to the Repligen Corporation.
A study of 42 autistic children. Randomized, placeb-controlled, double-blind – crossover study. Each child received either placebo or secretin and then the opposite treatment at six weeks. "There were no significant differences in the mean scores on any measure of language, behavior, or autism symptom severity after treatment with secretin compared to treatment with placebo."
2003
A study of 62 autistic children. Randomized, placebo-controlled, double-blind – crossover study. "Five children showed clinical improvement in standard scores: two after HSS [human synthetic secretin] and three after placebo. There were no significant changes in gastrointestinal symptoms after HSS or saline placebo."
From Part II:
"At the conclusion of the study, parents were asked to guess their child’s group assignment. Twenty seven families guessed their child’s group assignment correctly and 27 guessed incorrectly."
September 2003 – Repligen stock trading at $7.47 a share on volumes of about 400,000 shares. Its cash value is estimated at about $1 per share.
December 2003 – Repligen stock trading at $4.37 a share on volumes of 740,000 shares.
2004
January 5, 2004 – Repligen announces that there were no significant differences between the treatment and placebo groups in its Phase 3 study of secretin for autism. This study – although Dr. Edelson seems unaware of the fact – focused on younger children and also looking at chronic gastrointestinal problems.
January 2004 – Repligen stock trading at $3.33 on volumes of over 2 million shares.
Early 2004 (exact date difficult to determine) – Dr. Rimland sends a letter to Walter Herlihy of the Repligen Corporation essentially denouncing their decision to declare that secretin was no more effective than placebo.
July 2004 – Repligen stock bottoms out at $1.62 a share on volumes of 200,000 shares.
2005
A study of 15 autistic children. Randomized, placeb-controlled, double-blind – crossover study. Transdermal secretin or placebo was applied daily to the backs of children for four weeks. After a six-week "washout" period, the groups switched treatments.
"Overall, there were no statistically significant differences in speech, sociability, sensory, and health scores for treatment versus placebo periods. In addition, there were no differences in such scores for children with a history of diarrhea."
November 2005 – Repligen Corporation starts a Phase 2 trial of secretin for the treatment of schizophrenia.
Present
September 25, 2008 – Repligen stock trading at $4.89 on volumes of 45,000 shares. The company has a little over 31 million shares outstanding.
Human vs Porcine Secretin:
Between 1998 and today, over 20 studies have been done (and published) to determine the effect of secretin on autism. At least nine (9) used porcine secretin and at least four (4) used human secretin. For the others, the methods section didn’t specify or was unclear (initially, only porcine secretin was available, so there was no need to specify). The results of the Repligen Phase 3 study have never been published.
Only three secretin studies (with a total of 23 subjects – really only 9 if you consider that Kern et al found a positive effect only among the 5 children with chronic diarrhea) found any effect on autism. At least twelve other studies – with a total of 609 children enrolled – showed that secretin had no effect in the treatment of autism. Despite this, Dr. Edelson is convinced that secretin "works" for autism. He attributes the discrepancy to the "fact" that the largest studies (by Repligen) used synthetic (recombinant) human secretin instead of the porcine secretin used in the earlier studies.
Well, he’s wrong on a couple of counts.
To begin with, many of the larger studies used porcine secretin – one even compared it to both synthetic (recombinant) human secretin and placebo (all three were equally effective).
Secondly, the difference between human and porcine secretin is only two amino acids out of twenty-seven. Only dogs have a secretin molecule that’s closer to ours than the porcine secretin. In addition, the two substitutions – aspartate for glutamate and serine for glycine – would have minimal effect on the function or structure of the protein. Aspartate and glutamate both have carboxylic acid side chains and serine’s higher affinity for water – compared to glycine – would be masked by the fact that it’s situated right next to aspartate, which is very hydrophilic.
In fact, if Dr. Edelson had bothered to check, he would have found that porcine and human secretin were found to be functionally interchangeable. Of course, he could have guessed that from the fact that porcine secretin was used in humans for years to produce the same effect as human secretin. Think about the "good old days" when diabetic people used porcine or bovine insulin. The few amino acid differences didn’t change the function of the insulin, although they did lead to a few people developing antibodies to bovine insulin.
On more thing to ponder before you go off to read Dr. Edelson’s editorial. There are only two manufacturers making secretin these days (in the US, anyway):
Manufacturers of human secretin:
ChiRhoClin – ChiRhoStim (human secretin - $415 per 16 mcg/160 U vial)
Manufacturers of porcine secretin:
Repligen – SecreFlo (porcine secretin - $425 per 16 mcg/160 U vial)
Both products are recombinant – meaning that they are made by bacteria or yeast that contain the gene for human or porcine secretin. Since there is no conceivable reason why Repligen shouldn’t use its "house brand" of human secretin (RG1068) for pancreatic imaging, I wonder if they are positioning themselves for a "replay" of the "secretin-cures-autism madness of a few years ago. Unlike Ferring in 1998, Repligen would be in good position to profit from a resurgent interest in PORCINE secretin.
The Gastrointestinal Issue:
Dr. Edelson is also convinced that his survey of parental impressions is correct about the effectiveness of secretin because the Kern et al (July 2002) study showed that (porcine) secretin was significantly more effective in children with autism AND chronic diarrhea.
Let’s examine this claim, shall we?
The Kern et al study looked at 19 autistic children and was a double-blind, randomized crossover study of a single dose of porcine secretin. Of the 19 children in the study, five (5) had chronic diarrhea. These five (5) children apparently showed more improvement that the others (in whom secretin was no better than placebo).
That’s right – Dr. Edelson is basing his defense of secretin on the results seen in five (5) autistic children with chronic diarrhea (five out of nineteen in the Kern et al study).
Just a few months after the Kern et al study came out, Unis et al published their study of porcine (that’s right – porcine ) and recombinant human secretin in autism. They had eighty-five (85) autistic children in their study (one third received porcine secretin) and concluded:
"No evidence that either biologic or synthetic secretin provided amelioration of symptoms beyond placebo was observed."
"This held true when children with and without gastrointestinal problems were examined separately."
So, a larger study (to be fair, only about 28 received porcine secretin) found no difference, even when looking at children with gastrointestinal problems.
And what about Dr. Edelson’s parent survey?
The parent survey that Dr. Edelson keeps referring to is posted on the ARI website. Parents fill it out and mail, fax or e-mail the form back to ARI. There is no attempt to get a representative sample of parents of autistic children and the "data" are inclusive to the beginning of the survey, several years ago. Here’s what visitor’s to the ARI website see when they look at the survey’s assessment of secretin (dated February 2008):
Intravenous secretin (468) Got worse: 7%; Got better: 44%; No change 49%
Transdermal secretin (196) Got worse 10%; Got better: 37%; No change 54%
To "help" the parents in their assessment of the various treatment options listed, the survey gives a ratio of the number of children who "got better" to the number who "got worse" - this completely ignores the children who had "no change", which is often a large number. Now, Dr. Edelson’s better:worse ratio makes secretin look like a magic bullet, with a better:worse ratio of 6.3 for intravenous secretin and 3.6 for transdermal secretin.
However, if we look at "better" vs "not better" (i.e. either no change or got worse), the numbers don’t look so rosy: 0.8 for intravenous and 0.6 for transdermal secretin.
As luck would have it, I happen to have saved ARI’s survey results from last year:
IV secretin (422) Got worse: 7%; Got better 45%; No change 48%
TD secretin (176) Got worse 10%; Got better 37%; No change 53%
If we use these numbers to "dissect" that past year’s activity, the survey reports from Feb 2007 and Feb 2008 show that, in the course of a year, ARI received reports that after treatment with IV secretin:
3 children got worse
3 children got better
39 children showed no change
This is about what you’d expect from a treatment that doesn’t work.
What I find particularly disappointing is that Dr. Edelson has a PhD in psychology and should know that his survey results are meaningless. He should also know how to read the scientific literature and interpret the data. There are all of these things that he should know , and yet he acts as though he was unaware of them.
I’ll let you draw your own conclusions about why Dr. Edelson is trying to breathe life back into this dead hypothesis. As for the validity of his arguments, I think Dean Yeager in Ghostbusters (I) said it best:
"Your theories are the worst kind of popular tripe, your methods are sloppy and your conclusions are highly questionable."
Prometheus
ADDENDUM:
I forgot to mention that Dr. Edelson, at the conclusion of his editorial on secretin, puts out a call for researchers willing to do a study of porcine secretin in autistic children with chronic GI problems. He offers ARI funding, but I doubt that he realizes how expensive such a study would be.
I’d be willing to act as Principle Investigator for such a research project, even though I doubt that negative results would get people to stop injecting (or rubbing) autistic kids with secretin. This passed from a scientific issue to a religious one years ago. Still, there’s nothing I love more than a futile effort in a lost cause.
I happened to mention this idea to the head of the university IRB (the body that approves and monitors human research) and asked her to give me an unofficial opinion on the study. I gave her the available published research and a brief outline of a study design. Her response, which appeared in my e-mail this morning, was that the IRB would not likely approve such a study for the following reasons:
[1] The published research indicates that there is no reason to suspect that secretin would be helpful in the treatment of autism (minimal to zero chance of benefit).
[2] There is a small, but non-zero, risk of harm.
[3] The study population is an "at risk" group (minor and disabled); they cannot give informed consent AND their parents are likely to be willing to accept risks that parents of "typical" children would not.
[4] The proposed research is unlikely to significantly increase the scientific knowledge on the subject. As a result, the risk to the subjects, although it is very small, would exceed the benefit of the knowledge gained.
Filed under: Autism Policy, Autism Practitioners, Myths About Science, Uncategorized | 17 Comments »
A Layperson’s Guide to the Scientific Literature – Part 3
July 23rd, 2008
Before I start onto my discussion of metanalyses and the rest, I’d like to take a few moments to answer some of the common criticisms that skeptical evaluations of autism research often stir up.
The most common criticism, almost reflexive in its regularity, is the “Are you calling me (or ‘these parents’) a liar?” complaint. A frequently seen variant is the “We know what we see!” complaint.
Let me start out by saying that I don’t believe that any of the parents who report improvement, resolution or even “miraculous cure” of their children’s autism are liars. As far as I know, they are reporting what they see (or think they see) and (most importantly) their interpretation of what they see.
People - sad to say - are prone to misperceptions and misinterpretation.
As any lawyer could tell you, eye-witness accounts are often the least accurate part of any evidence presented in court. This is not because they are lying (although some certainly are) but because our brains “edit” what we see to fit our preconceived notions of “reality”.
Repeated studies have shown that, when presented with a rapid sequence of images of, for instance, a cup falling and breaking, people will fail to notice if one – or even several – of the images are out of order. When challenged, they will usually vigorously deny that there was anything unusual about the sequence. They aren’t “lying”; their brains have simply “edited” the information they saw to make it conform to what “common sense” dictates.
Additionally, people are “hard-wired” to see patterns, even when no patterns exist. This need to find a pattern in random events has led to any number of superstitious and non-rational beliefs, from astrology to witches. The need to find a “cause” for what are, in fact, random events continues to the present day (and will no doubt continue into the future). People want to think that something – God, witches, the stars or “the government” – is behind the bad (and often the good) things that happen in their lives.
This is no less true with parents of autistic children than it is with the rest of humanity. They want “answers”, they want a “cause” and they want to find a recognizable pattern in the random events of life. Autism plays into these needs by being - so the genetic studies suggest – due to random genetic mutations and by its start-and-stop developmental progress.
The latter is probably not unique to autism, since even typical child development shows periods of rapid skill acquisition followed by periods of relative stasis. In autism, this pattern may be accentuated or it may be that the development of autistic children is watched so closely that the pattern seems to be accentuated. Either way, the random pattern of improvement and stasis also feeds the “pattern-seeking center” of our brains. Anything that is done or changed prior to a period of improvement is perceived to have caused the improvement; likewise, anything done prior to a period of stasis is perceived to have caused the stasis.
This leads a number of parents to truly believe that some intervention – which in all likelihood did nothing – caused their child to either get worse or get better. In addition, there is a phenomenon I like to call “The Lucky Stockbroker Syndrome” that leads to reinforcement of these individual experiences.
The Lucky Stockbroker Syndrome is based on the following analogy:
A stockbroker decides to try a new money-making scheme. He gets a list of ten thousand (10,000) potential clients and sends half of them a letter (or e-mail, if you like) telling them that a certain stock will rise in value over the next week. To the other half of the list, he sends a letter saying that the same stock will FALL in value.
The next week, he sends letters to the half of his list (5000 people) that got the CORRECT prediction – again, half of them are told that a certain stock will go up in price and the other half are told that the stock price will fall. This goes on for a total of six weeks.
At the end of this time, he has a list of 156 people who have – by random chance – received six consecutive correct predictions about stock prices. He then offers these people a five-year subscription to his stock-picking service, which they gladly purchase, thinking that he has some amazing system for predicting the stock market.
The same thing is happening in the world of autism therapies. The parents that try a particular therapy – such as chelation or dolphin therapy – prior to their child coincidentally having a period of improvement will conclude (erroneously) that the therapy caused the improvement. Those that try the same therapy and don’t see an improvement generally “move on” and try something else.
Even worse, some practitioners exhort the parents to “stay the course”, “give it a chance to work” or “don’t leave before the miracle”, which leads them to keep trying the therapy until – again, by shear coincidence – their child goes into a period of improvement. Thus, the “pattern-seeking center” of their brains sees – erroneously – a “pattern” of the therapy causing improvement. In fact, they are simply fooling themselves into seeing a pattern that isn’t there.
Again, I have to emphasize that this isn’t stupidity, gullibility or a lack of intelligence – it is simply how human brains work. THIS is why science is so persnickety about placebo controls, blinding of observers and randomizing subjects. We do it to keep from fooling ourselves, which we are all too capable of doing, given the chance.
Now, on to the show!
Meta-analyses:
A meta-analysis is a way of combining the results of a number of smaller studies in order to get more meaningful results. In the usual scenario, there is a question (e.g. “Does guinea pig therapy help in autism?”) that a number of smallish studies have tried to answer but were individually unable to reach statistically significant conclusions. This is most often due to small numbers of subjects, which is where a meta-analysis can help.
The meta-analysis gathers together the results from a number of smaller studies and adds them together to get – in the ideal situation – a statistically significant answer. Again, this is what happens in the ideal situation.
The biggest problem with doing a meta-analysis is how to sort the wheat from the chaff. Many smaller studies are also not very well planned and executed. They may have problems with subject selection, or their outcome measures may be weak or they may not have randomized their subjects well enough. Including poor studies in a meta-analysis will lead to the all-too-familiar experience of “garbage in, garbage out” (GIGO).
On the other hand, weeding out too many studies will lead to a meta-analysis that fails to reach a meaningful conclusion. This is better than reaching the wrong conclusion, but still not very satisfying.
Another problem in meta-analyses is that the individual studies may be (and almost certainly are) looking at different things. They may have different criteria for what or who they include in the study; they may have different outcome measures; they may have different criteria for classifying their results. All of these can lead to a meta-analysis trying to compare “apples and oranges” (or worse, “apples and orangutans”).
The bottom line is that a meta-analysis is only as good as the worst study it includes.
Meta-analyses should NOT be used as a substitute for a coherent large-scale study. At best, they can be used to give a “second look” at a question, to decide if it is worth expending the time, effort and money to answer it. A good meta-analysis can suggest that there is (or is not) a detectable difference that is worth looking at in depth with a real study.
Pilot Studies:
Another way of deciding if a question is worth studying in depth is to do a pilot study. Pilot studies are – by definition – smaller and less rigorous than the double-blind, placebo-controlled studies that could definitively answer the question. Many interesting questions (e.g. “Does television watching cause autism?”) are not inherently plausible enough to warrant launching a full-scale study to investigate them, so a smaller “pilot study” is done to gather enough data to determine if there is sufficient plausibility to justify a larger study.
In order to reduce the cost, pilot studies almost always have fewer subjects than would be needed to give a definitive answer. Or, they omit controls, “piggyback” on clinical work, or omit observer and/or subject blinding. Maybe all of the above. At any rate, pilot studies do not meet the standards needed to give a definitive answer to a question. That’s because they’re not meant to give an answer to the question – they’re just meant to give an indication of whether or not it would be worthwhile to start a larger study to get the definitive answer.
So, pilot studies, in and of themselves, are fine, so long as you remember that they do not give answers to the question – they just indicate if there is any point in doing a larger study. A true pilot study can only give one of two answers:
[1] The question is worth further study.
[2] The question does not appear to be worth further study.
Answer [2] needs more explanation. Because a pilot study has – almost by definition – a small number of subjects, it cannot detect small differences that a larger study would be able to pick up. For that reason, a pilot study can never truly “close the door” on a question. However, it can say that the effect size is smaller than a certain number, which depends on the number of subjects and the study design. It’s then up to the researchers to decide if it is worth trying to find a smaller effect.
Case Reports / Case Studies:
At one time, almost all of the articles in medical journals were case reports. Now, they are in the minority. This is probably a good thing.
Case reports are essentially anecdotal evidence – a “narrative” of what happened and what caused it to happen written by the people who were involved in the case(s). There is usually some effort made to “dress up” the anecdote in scientific garb by citing references and showing laboratory data, but there are no controls, no blinding and the subjects are definitely not selected at random.
Don’t get me wrong – a lot of good information can come out of case reports. They are especially useful in conveying to clinicians what to do (or not do) in certain situations. They can even point out possible promising areas of diagnosis or treatment to research.
But case reports aren’t high quality data.
The biggest limitation of case reports is that there is no indication of how the subjects were selected. The authors themselves may have no idea how the subjects were selected – most of the time, case reports are written because the author(s) thought they “saw a pattern” in their clinical practice. Again, beware the “pattern-seeking center” of every human’s brain – it often sees patterns where none exist.
The concerns I have about case reports revolve around two issues:
[1] The effects observed may have simply been random – the treatment may not have had any effect or the exposure may have been coincidental rather than causative or….. you get the point.
[2] The subjects in the report may not be representative. This could be inadvertent or deliberate. The worst case is that the author(s) may have “cherry-picked” their patients to find the results they wanted – either consciously or unconsciously.
For example, a clinician in the ER notices that there are “runs” of “non-accidental trauma” (medico-legalese for deliberate acts of violence) that coincide with the full moon. These could then be reported as being due to the full moon. However, the clinician fails to notice that such “runs” occur at random – unrelated to the phase of the moon – due to random “clustering”. However, the “runs” of violence that don’t occur during the full moon are unconsciously discounted by the “pattern-seeking center” of their brain. [By the way, I have heard this very “hypothesis” from doctors and nurses on several occasions. It has been the subject of numerous studies in the medical literature.]
Well, that’s all for now. I hope that I have been of some service to people struggling with the intricacies of the scientific literature.
For next time (or whenever I get around to it): How some “alternative” practitioners duck the responsibility (and blame parents) for therapeutic failures.
Prometheus
Filed under: Autism Science, Critical Thinking, Myths About Science | 8 Comments »