Testimonials: Listening to People’s Stories
October 15th, 2007
Evidence-based medicine is based on data. “Alternative” medicine has “alternative” data : testimonials.
So, what’s the matter with testimonials? They’re just people’s stories, right?
Absolutely! So are reports of alien abduction, Bigfoot sightings and pixies in the garden.
Are all of these people lying? No.
They’re just telling the truth as they see it.
Let me tell you a little story. This is a business plan that anybody with access to the Internet can do - if you decide to try it, please don’t mention that you saw it here.
……….
The Tale of the Lucky Stockbroker
A poor stockbroker (after the dot.com bubble burst) was trying to make a living when he decided that instead of buying and selling stock, he would sell his advice. The only problem was that he first needed to establish his reputation.
That’s when he came up with a brilliant idea.
He spent a little money to get the e-mail addresses of two thousand people who had purchased stock on-line and sent them each a message. To half of them, he sent a message saying that ABC Inc. stock would go up in the next week - to the other half, he sent a message that the stock would go down. It went up.
At the end of the following week, he sent an e-mail message to the thousand people who had received the “correct” stock advice. Half of these people got a message saying that XYZ Co. stock would go up in the next week - to the other half, he sent a message saying that the stock would go down. It went down.
He kept this up for four weeks, at which time he sent the remaining 125 potential clients a message saying that they could continue getting these amazing stock tips for a mere $5000 a year - payable in advance. Almost all of them paid for a year’s subscription.
The End.
……….
If this story seems too far-fetched to be true, remember: these people had just seen this stockbroker make four correct predictions in a row.
In fact, the stockbroker would be well-advised to arrange a meeting so that all 125 of the people who had gotten four straight correct predictions could get together and compare their amazing stories.
Furthermore, he could also have gotten a few customers from the people who had gotten three out of four correct predictions.
But, let’s take this a few steps further.
What if the stockbroker had said that the stock would go up within a month - or two months? And he could have told them that the stock might go down before it went up; that would give him a wider “window” for his “prediction” to come true.
And what if he told his potential clients that not every stock “worked” for every person - that you had to keep trying stocks until you found one that “worked” for you?
Well, at that point, most folks would start slowly backing away and looking for the door.
And what about all those people who only got one wrong prediction? Or those who got a mix of right and wrong predictions? Well, they aren’t around to tell the other “potential clients” their stories.
And that’s what it’s all about, right? Telling people’s stories?
Right?
Let’s recap the “stories” in this tale:
One wrong prediction (batting avg. 0.000) - 1000
One right and one wrong prediction (0.500) - 500
Two right and one wrong prediction (0.667) - 250
Three right and one wrong prediction (0.750) - 125
Four right predictions (1.000) - 125
Overall “batting average” - 0.500 - the same as flipping a coin.
So, out of the original two thousand (2000), 1500 (75%) think this guy is no better than flipping a coin (and 1000 think he’s worse). But those stories don’t get told.
The same thing happens in “alternative” autism therapies.
Don’t get me wrong. I firmly believe that the majority of “alternative” autism practitioners are sincere and truly think that they are helping people. Honesty and sincerity do not prevent them from being wrong, however.
But, with the Tale of the Lucky Stockbroker in mind, think about these points:
……….
[1] Autism is a syndrome of developmental delay, not developmental stasis - as I’ve said nearly a million times before. Children will continue to develop whether they are treated or no.
[2] Like their typical peers, autistic children develop in spurts, with periods of rapid change followed by longer periods of little or no change. This provides opportunities for “therapies” to appear to work. This is true for “mainstream” and “alternative” therapies, alike. The difference is that “mainstream” (evidence-based) therapies are tested using lots of subjects, along with placebos and blinded observers, which helps “zero out” random events.
[3] Studies have shown that as much as 19% of autistic children will “move off the autistic spectrum” before their seventh birthday. Thus, children “recover” or even get “cured” spontaneously. Even if some people don’t like to hear that.
[4] What happens to parents whose kids don’t get better? For the most part, they are encouraged - by other parents and by the practitioners themselves - to “keep trying”. They are also encouraged to pay close attention and to “think positively” - but they are never encouraged to doubt.
In most cases, parents who give up on “alternative” therapies simply fade away. They have enough going on in their lives that they don’t feel the need to “tell their story”. Especially when it’s not wanted.
If you have the opportunity, check in to one of the many “biomedically oriented” Internet groups and see what happens when somebody questions the idea that “biomed” can “recover” autistic children. At the least, they will be admonished - “Don’t stand in the way of other parents getting their children the help they need!”. More likely, they will be told to “Shut up!” and banned from the group. In some cases, they will be harassed and even threatened.
So, it’s not too likely that you’ll hear many dissenting “stories”. It seems that everyone’s story is valuable…unless it contradicts the received wisdom of “biomed”.
……….
Keeping in mind the story of the Tale of the Lucky Stockbroker, you can now see the value of testimonials in “alternative” autism therapy (not much). It’s not because the parents are lying; it’s not because the parents are stupid - it’s because the parents can only see their own experience.
Since they can only see their own experience, the “success stories” in “biomed” have no way of knowing if their child was going to be one of the 19% that is spontaneously “cured” or the unknown percentage who will show at least one period of rapid improvement. The only way to know that is to look at hundreds (if not thousands) of autistic children, some of whom received the treatment and some of whom didn’t.
And it would be best if the people evaluating the children didn’t know who got the treatment and who didn’t.
In other words, the only way to really know what works and what doesn’t is to use science.
Too bad about those testimonials.
Prometheus
Filed under: Autism Science, Autism Treatments, Critical Thinking
October 15th, 2007 at 7:52 pm
Great explanation! Thank you.
October 15th, 2007 at 8:33 pm
You have a long list of really accurate blog posts.
How can I tell if you are just lucky?
October 15th, 2007 at 9:16 pm
Excellent! I love tall tales.
Cheers
October 15th, 2007 at 9:19 pm
Matt,
The answer to that is simple. Are my posts accurate more often than they would be by random chance?
If it is too difficult to determine the proportion of posts that would be accurate by chance, you could compare my posts to those of someone who knows nothing about the subject (I could suggest possible ‘blogs to use as a “control”).
If my posts are more accurate than those of somebody who knows nothing about the subject, then it is not just “luck”.
Thanks for reading!
Prometheus
October 15th, 2007 at 9:40 pm
Hi Promotheus -
Nice posting.
“[3] Studies have shown that as much as 19% of autistic children will “move off the autistic spectrum” before their seventh birthday. Thus, children “recover” or even get “cured” spontaneously. Even if some people don’t like to hear that.”
Other studies have shown exactly the opposite. By way of example, I found this paper:
Autism From 2 to 9 Years of Age
From the discussion:
“Diagnosis of autism in 2-year-olds was quite stable up through 9 years of age, with the majority of change associated with increasing certainty of classifications moving from ASD/PDD-NOS to autism. Only 1 of 84 children with best-estimate diagnoses of autism at age 2 years received a nonspectrum diagnosis at age 9 years, and more than half of children initially diagnosed with PDD-NOS later met autism criteria. Nevertheless, more than 10% of children with diagnoses of PDD-NOS at age 2 years received nonspectrum best-estimate diagnoses (ie, not autism or ASD) by age 9 years, and nearly 30% continued to receive diagnoses of PDD-NOS, indicating mild symptoms at age 9 years. A significant minority of children with milder difficulties within ASD at age 2 years showed only mild deficits in the clinical ASD range at age 9 years. Classifications changed substantially more often from ages 2 to 5 years than from ages 5 to 9 years. The bulk of change in diagnosis occurring in early years is consistent with another recent study.9 At age 2 years, diagnostic groups were more similar in functioning and IQ than the diagnostic groups identified at age 9 years, when the autistic group showed very poor adaptive functioning and the PDD-NOS group, much less abnormal verbal and nonverbal IQ. ” (my emphasis)
The abstract you posted earlier didn’t specify gains in autism vs. pdd. Do you have the values that could be used in comparison? I’d also be interested in knowing if any of the children that lost a diagnosis were getting biomedical treatments.
No doubt if I were to reference a paper defending biomedical practices for which other papers had found drastically different results, you would not hesitate to point out the discrepancy. If nothing else, we can say that the research on how many kids will spontaneously recover is in its infancy as much as anything else in autism research.
“[4] What happens to parents whose kids don’t get better? For the most part, they are encouraged - by other parents and by the practitioners themselves - to “keep trying”. They are also encouraged to pay close attention and to “think positively” - but they are never encouraged to doubt.”
LOL! If you are a reliable source of information for how the biomedical community acts ‘for the most part’, John Bests board cut and pastes from message boards are a reliable source of information for how the neurodiverse community acts for the most part. Likely you’ve done a post on how people lap up what conforms to what they already believe, which makes your belief that you can realistically speak towards how the biomedical community acts all the more hilarious.
As far as the stock tips go, unbounded claims of wild developmental improvements are easy targets; but in my experience they are rarely made as such. For many people, that isn’t the kind of thing that convinced them that biomedical approaches could help; but rather, more specific changes that might not mean a loss of label, but an improvement in the quality of life.
By way of example, my son used to visually stim all the time. He would go to a fence, countertop, or table, tilt his head forty five degrees and run past. He could do this twenty or thirty times a day. It was suggested that we try cod liver oil with vitamin A to alleviate this. Two weeks after doing so, my wife and I noticed that we hadn’t seen my son perform this kind of behavior.
Of course, he could have simply decided to stop visually stimming in this manner; or it could have been the nature of his particular disorder, stimming like mad for months on end only to stop at the same time we happened to give him vitamin A. People do win the lottery all the time, after all.
But then the strangest thing happened; my son had ear tubes put in to combat persisent ear infections, one of the requirements for pre-surgery was that he not be allowed to take cod liver oil for two weeks before surgery due to blood thinning effects. In the days before surgery, he started stimming on tables, fences, and tabletops once again. After a post surgery period, supplementation resumed and the tilting headed boy is gone again. Go figure.
I suppose there is the chance that this was simply a series of coincidences; and we certainly seem to be at the dangers of the single participant test outlined in other areas. But what should we really do? Stop spending seven dollars a month because we don’t have a 99% CI that the reason he stopped stimming was our efforts? What if his stimming returned? Perform another test which we cannot have any confidence in? Assume that we are exceedingly prone to coincidences?
And lets not forget, we didn’t come to the idea of giving our son cod liver oil in a vacuum, but rather, because other people had described similar results. Are we to assume that they all are experiencing the same coincidences we are?
Creating caricatures and moving to take them apart is simple to do, but as always the case with caricatures, they distort reality to whatever proportions the artist sees fit.
Take care!
- pD
(if this double posts, I apologize. Please delete one / strange behavior)
October 16th, 2007 at 5:35 am
Nice post, and argument, but with some formal and infromal fallacies.
“Studies have shown that as much as 19% of autistic children will “move off the autistic spectrum” before their seventh birthday. Thus, children “recover” or even get “cured” spontaneously. Even if some people don’t like to hear that.”
From the study ibid - Diagnostic Stability in Very Young Children with Autism Spectrum Disorders; “Movement off the spectrum (19%) may reflect true improvement based on maturation, intervention, or over-diagnosis at age 2.
hmmm… so they do not just “recover” or get “cured” SpontaneouslY huh; even if some people don’t like to hear that.
P
October 16th, 2007 at 6:59 am
You have a long list of really accurate blog posts.
How can I tell if you are just lucky?
Matt,
I think this is the point where we all get the e-mail asking us for $5000 to keep reading Prometheus’ posts.
———
But seriously, nice post.
Joe
October 16th, 2007 at 10:42 am
Specially if the “cured” child was labeled at age 2, the odds of loss of label are not too low. But from my read of the research, it appears that you might be able to tell the child will move off the spectrum as early as age 3. I’d suggest that if a child is obviously autistic at, say, age 5, it’s a near certainty they will always be autistic. They still have a chance of a good outcome (probably more than the literature says, depending on circumstances), but like Kanner said, they are “success stories”, not non-autistics. Basically, I think the phenomenon of loss of label in early childhood is divorced from good adult outcomes.
October 16th, 2007 at 11:36 am
Peelu,
The conclusion of the authors was that they could not tell if the movement off of the autistic spectrum was due to maturation, intervention or mis-diagnosis. Their data show that 19% of children moved off the spectrum.
I happen to agree with that conclusion - that you can’t tell from their study if the loss of the autism diagnosis was due to maturation, intervention or mis-diagnosis. Since we can’t tell, we also can’t say that any of the children moved off the spectrum due to interventions.
It would be interesting to go back and see how many of these children were receiving classical “biomedical” interventions and if they moved off the spectrum as frequently as those without said interventions. Unfortunately, we aren’t given that information.
pD,
The study you refer to examined children referred for autism evaluation at age 2 years and then again at 9 years of age. Two points:
[1] The fact that they were referred for evaluation suggests that there was some indication that they might have autism. This is very different from looking at the general population.
[2] Although the diagnosis of autism can be made at age two years, the peak age of diagnosis is later, which by itself explains the increase seen in the study.
Prometheus
October 16th, 2007 at 12:50 pm
Hi Prometheus -
“[1] The fact that they were referred for evaluation suggests that there was some indication that they might have autism. This is very different from looking at the general population.”
I’m not sure I understand your point. Don’t all of the children in both studies fall into the category of having been referred for evaluation? I guess I don’t see where the general population comes into either study; both looked at groups of children who were given a diagnosis for autism spectrum. Maybe this is a result of only having access to the abstract for the 19% paper. (?) I’m sorry if I’m confused, can you clarify?
“[2] Although the diagnosis of autism can be made at age two years, the peak age of diagnosis is later, which by itself explains the increase seen in the study.”
In which study? In the study I referenced, children were more likely to move up the spectrum, (i.e., towards autism) than the other way around:
“Diagnosis of autism in 2-year-olds was quite stable up through 9 years of age, with the majority of change associated with increasing certainty of classifications moving from ASD/PDD-NOS to autism.”
It seems unlikely you meant to make this point, as it tends to cast doubt on if improvements in other studies are really the result of spontaneous recovery, or just bad diagnosis.
Common sense would tell me that a diangosis at three would be more reliable than one at two, but the numbers in the study I posted seem to tell us observations made at age 2 are underestimating severity. Perhaps this is an artifact of the tests used. (?) Or maybe they had some very good evaluators?
If, on the other hand, you are refencing the 19% study, that population included children 16 month olds (not clear on how many b/c no copy). But, the authors of the 19% study take care to note that the improvement could have been the result of over diagnosis at two; but that means they didn’t really recover at all, they were simply not diagnosed appropriately. It would seem they failed your fourth test from your previous post.
Anyways, take care!
- pD
October 16th, 2007 at 5:00 pm
What that study suggests is that the PDD-NOS diagnosis is unstable, with 10% of children losing the label, and most moving into autism. The autistic disorder label was much more stable. There’s probably some difference in the methodology of this study compared to the other studies too. For one, the researchers were different, and in this kind of study, I’d say that matters.
October 17th, 2007 at 1:58 am
Well said. I really appreciate your explanation here.
October 17th, 2007 at 8:47 am
Hi Joseph -
“What that study suggests is that the PDD-NOS diagnosis is unstable, with 10% of children losing the label, and most moving into autism. The autistic disorder label was much more stable. There’s probably some difference in the methodology of this study compared to the other studies too. For one, the researchers were different, and in this kind of study, I’d say that matters.”
Exactly my thoughts. But what I don’t see addressed here, is how can we have confidence in any study telling us how many kids move off the spectrum ’spontaneously’ if two studies have such large differences, or if a change in researchers can make such a big difference? You realize that replication of findings is key to insuring their validity; how can we hope to do this if a change in researchers can cause us to have very different findings?
I’m not standing behind this study, or the other, but I am saying that using one study as ‘proof’ that X amount of children will spontaneously move off the spectrum, while others studies show very different results, seems a lot like cherry picking the studies that fit a particular message. Picking favorites has no place in critical analysis, which I sort of thought is what this thread, and indeed blog, was about.
Take care!
- pD
October 17th, 2007 at 12:53 pm
pD,
Here’s how I see it:
The study I cited (Kleinman et al, Oct 2007) looked at 77 children and found that 19% of them “moved off the autistic spectrum” over the course of a few years.
The study you cited (Lord et al, June 2006) looked at 172 9-year-olds who had been evaluated for autism as 2-year-olds (twenty having apparently been lost to follow-up) and concluded that there was good diagnostic stability for autism but not very good diagnostic stability for PDD-NOS (from their conclusion section).
When I looked at the numbers in the Lord et al study, I saw that, out of 172 children, the number diagnosed with autism went from 84 to 100, those diagnosed with PDD-NOS went from 46 to 35 and those assessed as being “non-spectrum” went from 42 to 37.
That’s a 3% increase in “spectrum disorders”.
Interestingly, the Lord et al study had more subjects with autism than with PDD-NOS, which is rather the opposite of what is generally reported in the community.
Still, my statement remains correct:
“Studies have shown that as much as 19% of autistic children will ‘move off the autistic spectrum’ before their seventh birthday.” (italics added)
There are other studies showing numbers in the 10 - 15% range, but the Kleinman et al study has the advantage of being the most recent.
It would be interesting to see if the percentage change in diagnosis with maturation has changed over the years. I’ll see if I can get enough studies to see a trend.
Prometheus
October 19th, 2007 at 1:55 am
Catherine Lord presented that data on moving around on the spectrum. She was comparing kids who had been diagnosed by the best of the best, as I remember, they still moved from Autism to PDD.nos and PDD,nos to Autism from age 2 to 9, using the ADOS instrument as their guide.
She showed a photo of 1 (ONE) boy who she had come across in years of studying autism who truly seemed to go from Autism to totally “off the spectrum”. She didn’t mention any therapies that took him there. The boy in the photo looked like he was nine or ten. I would like to see how normal he is as a teen and adult, that would be the real test. He might be quite normal, but I bet he would identify with many of the feelings of autistic people.
October 22nd, 2007 at 9:05 am
The “Jenny McCarthy effect”: More credulity towards autism quackery…
I was thinking of calling this post Jenny McCarthy and Julie Deardorff: Two crappy tastes that taste crappy together, but I’ve already used that joke with Jenny McCarthy and Oprah Winfrey. Besides, Julie Deardorff isn’t nearly as famous as Oprah,……
October 24th, 2007 at 2:25 pm
I was allegedly diagnosed with autism before age 2. After 18 years of my mother putting me on a strict diet that excluded all sugar (including honey and fruit juice) and all food additives– such that I couldn’t eay, among other things, any cereal or peanut butter at the regular supermarket, I left home and chucked the diet. I also attended countless visits to homeopathic doctors (about 2-3 per week my whole life) across the country. Meanwhile as these treatments “seemed to be working”, my mother widely publicized their success and my progress, including how I was “becoming almost normal”, with the thought that she could help other parents this way.
While my mother still watches me to this day with baited breath that my “autism will return” now that I am not dieting the way she thinks I should, I have come to the conclusion that I either never had autism or else it spontaneously remitted.
The problem with “pseudoscience” extends beyond the doubt is casts on true scientific findings, or the wasted capital that families spend pursuing pseudoscientific treatments in vain. There is an untold social and emotional cost to the children forced to undergo these treatments who, like me, lived in isolation, guilt, and peer rejection for 18 years as a direct correlate of being subjected to this pseudoscience.
Of note, I’m an asst. professor of psychology at a research university, which is probably not an accident given my background. I don’t study autism, which is also probably not an accident.
October 25th, 2007 at 5:31 pm
That’s a very interesting post, Emotional Toll. I have to wonder how many kids suffer because their parents are convinced they have autism — when they actually don’t. Not only because of having to endure quack therapies for no reason, but also because of this constant unstated expectation of failure.
My daughter was assessed by the local school district — it’s a routine thing done for all children to see if they qualify for free educational assistance. They said she would qualify, just, for autism-spectrum assistance. They emphasized heavily that this was not a medical diagnosis, but merely looking to see what kinds of educational assistance might benefit her. I had to wonder how many parents would actually listen past the “autism-spectrum” part and hear the “not a medical diagnosis” part. How many parents believe their child has been diagnosed with autism when in fact all that’s happened is that an educator expressed a concern, without even intending for it to be a diagnosis? I further wonder how many of the glowing testimonials for various “therapies” come from these parents and not from the parents of genuinely autistic children.
Normal children can be a real handful. Can a parent, intimately knowing nothing beyond their own family, necessarily tell the difference between normal and abnormal development, given the very wide range of normal variation? I have serious doubts. I know I’m not capable of making that distinction. I wonder what gives other parents so much confidence in their own armchair diagnoses. A lack of critical thinking skills, I suspect, as well as closed-mindedness. (Ironically, I find that the alternative medicine folks tend to be the most closed-minded, not the least, though they talk a lot about how closed-minded the scientific community is. I think they don’t really know what it means to be open minded. It means realizing that you might actually be wrong.)
November 23rd, 2007 at 2:44 pm
Post-holiday “The stupid, it burns,” part 1: Jenny McCarthy…
I know, I know. Picking on Jenny McCarthy over her now frequent idiotic statements about autism and her parroting of the myth that vaccines cause autism is like shooting fish in a barrel, boxing a one-armed opponent, playing tennis with……
February 4th, 2008 at 5:51 pm
I stumbled upon your page. Nice post.
My son was diagnosed with PPD-NOS one week ago. He’s exactly 2 years old. The doctor examined him for one hour, before he had even been examined for hearing problems. See, her calendar was less booked than the pediatric audiologist. Go figure. To recap, she diagnosed my son with having a life long mental disability after having examined him for one hour. Welcome to The System.
Anyway, she goes on to say “well he’s on the mild end of the spectrum, so with services, we have hope that he’ll move off the spectrum.”
Huh???? There’s a cure for autism? Social services can change the organic composition of my son’s brain?
To say I was, and still am, confused is a staggering understatement.
I’ve come to believe just what you suggest - there’s no cure. However, I can hope my son grows out of it. The diagnosis is unstable at age 2, as the study you cited indicates.
And if he doesn’t, fuck if I’ll submit to that DAN protocol.
April 28th, 2008 at 8:55 am
[...] as Steve Novella has pointed out, and, as Prometheus has pointed out, autism is a syndrome of developmental delay, not developmental stasis, which is why randomized studies with a placebo-control group are so critical. Particularly telling [...]
April 20th, 2009 at 4:00 am
[...] correlation with causation and believe that a given ineffective intervention “worked.” Prometheus points out this very phenomenon: If you have the opportunity, check in to one of the many [...]
September 28th, 2009 at 3:00 am
[...] because a not insignificant minority of autistic children develop, sometimes to the point of losing a diagnosis of being on the autistic spectrum, there are always a few stories that appear to show that [...]
October 8th, 2009 at 8:01 am
[...] because a not insignificant minority of autistic children develop, sometimes to the point of losing a diagnosis of being on the autistic spectrum, there are always a few stories that appear to show that [...]
November 23rd, 2009 at 2:58 pm
[...] why anecdotes about autistic children “recovering” after chelation therapy are inherently unreliable, not to mention that there’s no control group against which to compare. Indeed, in an [...]
January 4th, 2010 at 4:47 pm
[...] their pseudoscience than to discover what does and does not actually work. Indeed, Prometheus describes this phenomenon well as he’s seen it in “autism biomed” discussion [...]