“Let’s put on a Study!”
November 3rd, 2008
Once again, there is a hue and cry to “do a study” about vaccines and autism, this time pushed forward by commentors on David Gorski’s recent post on the “Science-Based Medicine” ‘blog. Whenever I hear people who have never tried to do any study - let alone a study of human children - say “We need to do a study!”, I can’t help thinking of Mickey Rooney and Judy Garland and their “Let’s put on a show!” musical films - ergo the title of this article.
As I expected, the commentors put forth many of the same arguments that haven’t worked before but this time they are doing their math, something I appreciate. So, in appreciation, I have prepared the following:
The Reason:
The interest in “doing a study” seems to stem from the idea that the only way to find if vaccines are connected with autism is to study unvaccinated children and compare their autism prevalence with vaccinated children. This is not true and is - in fact - not even the most effective way to study the postulated connection, as I will show later.
Much of the blame for this wrong-headed fixation on “Just study the unvaccinated!” can be laid at the feet of Dan Olmsted, who claimed that the Amish don’t vaccinate and don’t have autism. Both claims - it turns out - are untrue. The Amish do vaccinate and they do have autism. Apparently, Mr. Olmsted didn’t look in the right places for his information about the Amish. And if you don’t look, you never find.
The same argument - if you don’t look, you don’t find - has been used to argue for studying the “connection” between autism and vaccines. However, researchers have looked. They looked at thimerosal and autism - and found no correlation. They looked at the MMR vaccine and autism - and failed to find a correlation. Now, with their backs against the wall, the vaccines-cause-autism believers have shifted from distinct vaccines or vaccine components to vague “toxins” or the even more vague “too many, too soon”.
Despite an absolute lack of data supporting either claim (’toxins” or “too many, too soon”), the proponents have argued that “we need to do a study” of vaccines and autism. “We” (i.e. the taxpayers) may indeed end up “putting on a study” in response to the political pressure these groups may bring to bear. I have no illusions that any research will ever be able to convince these people that they are wrong. After all, they haven’t been swayed by any research done so far.
However, since we’re all going to end up paying for this study, we should ask that it be done in the most effective way possible, rather than simply following the uninformed dictates of the vaccines-cause-autism believers.
The Amish:
Let me dispose of one line of “reasoning” that often comes up when these “Let’s put on a study” promoters gather. It is often suggested that we “take advantage” of existing groups of people who do not vaccinate their children. Mr. Olmsted proposed the Amish, although it later turned out that the Amish do vaccinate their children, although not at the level seen in the population at large. Another proposed study group was a health care group in Chicago that practiced non-vaccination and also - we are told - had a low to zero autism rate.
The problem with selecting “special” populations to study is that they are not comparable to the general public. The Amish - probably the most dramatic example - have a much higher prevalence of several genetic disorders as a result of generations of inbreeding. They are not comparable to the US population for this reason and any results obtained from studying them would be irrelevant. I suspect the main reason for wanting to study the Amish was that Mr. Olmsted “found” that they don’t have any autism - which was, as I’ve said above, also incorrect.
Even if we were to study a group like that in Chicago, the question would arise, “What other differences (besides vaccination) are there?” Do they only eat organic foods? What else might account for any differences between them and the general population. And if no difference in autism prevalence were discovered, would that be enough to satisfy the folks crying to “put on a study” or would they simply shift their sights to another group?
No, the only way to correctly do a study of vaccines and autism is to compare children who are matched for age, sex, geographic location, rural/urban/suburban setting, socio-economic group and race. That way, the only differences (we hope) would be vaccination status and autism prevalence.
The Numbers:
If we’re to do a study (as opposed to “putting on” a study), we need to have an idea of the numbers of subjects we’ll need to test in order to come up with a statistically meaningful result.
In 2004, Smith et al published “Children who have received no vaccines: Who are they and where do they live?”. This was an extensive study of vaccination practices in the US, carried out from 1995 to 2001 on over 23,000 children selected at random to be representative of the larger population. Their results were interesting and bear directly on the desire to “put on a study” looking at the connection (or lack thereof) between vaccines and autism.
What they found was that 62.8% of their study population was “fully vaccinated” (i.e. had received all the vaccines on the then-current vaccination schedule - 15 total in 2001). An additional 36.9% were “undervaccinated” - although 58.5% of them (20.4% of the total sample) were missing only one vaccine. Only 0.3% of the sample - 111 children ages 19 - 35 months over five years - had not received any vaccination.
I’d like to emphasize a point. The Smith et al study only looked at children 19 - 35 months of age, which is well before school age, at which time undoubtedly a few more tenths of a percent of the unvaccinated were whittled away. However, this age includes the ages where children have been reported to have “regressed into autism” following vaccines.
So, any study of unvaccinated vs vaccinated children has to deal with the fact that the “pool” of unvaccinated children is about 0.3% (or less) of the total population. Even if we could extend the Smith et al findings to the entire 0 - 17 year age range (”children”, by most definitions), this works out to only about 223,000 completely unvaccinated children. Given that at least some of the children who were unvaccinated before 35 months will have gotten at least one vaccination by their teen years, this number is sure to be much smaller.
For the purposes of this demonstration, let’s limit the study group to younger children, both to eliminate the uncertainty about the number of unvaccinated children and to keep the diagnostic criteria somewhat comparable. After all, the criteria used to diagnose the 17 year-old autistic children were very different from those used today.
How about we settle on the 3 - 6 year age range? This puts them past the scheduled time for the majority of childhood vaccinations and yet cuts off before the “school vaccination mandate” years. According to the US census bureau, this age group (as of August 2008) contains 16,550,753 children. Of that number 49,652 (or less) might be completely unvaccinated (and over ten million “completely vaccinated”).
What sort of sample numbers do we need to “prove” the vaccine-austism question? Well, let’s make a few assumptions. The first assumption is that we will want to have a sample size large enough to be sure that we have less than a 5% chance of mistakenly saying there is a correlation if there isn’t (alpha error) - and we’ll want a less than 5% chance of saying there isn’t a correlation if there is one (beta error) in order to satisfy those who want to “put on a study” (normally, we’d accept a 20 - 50% chance of mistakenly saying that there is not a difference when there is one).
Secondly, we’ll have to stipulate what sort of difference we expect to find between the two groups. If the “fully vaccinated” group had 50% more autism than the “unvaccinated” group, would that be enough of a reason to change vaccination policies? Would a 10% difference be enough? More to the point, how small a difference will be small enough for the vaccine-cause-autism proponents to say “I was wrong, vaccines don’t cause autism.”?
That last point may be the one that actually torpedoes the study. Because if there is no number small enough to convince those who want the study that they are wrong, most of the reason for doing the study vanishes.
Let’s say - for the sake of argument - that we decide that a 10% difference in autism prevalence is enough to convince the skeptics that vaccines might cause autism and that a less than 10% difference will convince the believers that vaccines don’t cause autism. [I know, the latter assumption is pure fantasy.]
Well, plugging those numbers in - along with the current 1 in 150 autism prevalence - we find that we need over 360,000 children in each group to detect a 10% difference (you can try it yourself here). Unfortunately, that is more than the total number of unvaccinated children in the US, so that’s not going to happen.
What can we get with our “sample” of 49,652 unvaccinated children? If we manage to include each and every unvaccinated child in the US in the study, we could detect a 26% or more difference in autism prevalence.
Of course, it’s not even remotely practical to expect to get 100% of the unvaccinated children in the country into a study. How about a more practical number - say, 10% of them? That would allow us to detect a 70% or greater difference - about a three-fold difference in autism prevalence between the fully vaccinated and unvaccinated groups.
Now, if a difference of this magnitude or greater were found, that would pretty effectively silence those who say that the data haven’t shown a connection between vaccines and autism. I would certainly change my opinion (from “not shown” to “strong indication”).
But what if the results go the other way? What if the results are “negative”, meaning that the difference is less than 70%? Would that be enough to get the vaccines-cause-autism crowd to stop protesting and stop sending death threats? I doubt it.
Of course, there are those who believe that the results will be unambiguously in favor of their belief that vaccines-cause-autism. They want the study to go forward because they are confident that they will be vindicated. They are also the people who will rant about “bias”, “flaws” and “corruption” if the data don’t go “their way”.
Another other thing to consider is that a study with almost ten thousand children enrolled would be exorbitantly expensive, especially as they would all have to be tested carefully for autism. Even using a very conservative figure of $150 for an evaluation, that’s over $1.5 million just for the autism testing.
Don’t forget that research money, time and facilities are not infinite. Resources spent chasing the vaccines-cause-autism “connection” will have to come from other research areas, and the most fair way to do that would be to take them from other autism research. Thus, much time and money is spent to accomplish little…or nothing.
And is it likely that you could get the parents of 10% of the unvaccinated children in the country to enroll them in a study? Given that many of these children are unvaccinated because their parents harbor deep suspcions about doctors and “the government”, it seems unlikely.
Finally, let’s “run the numbers” on a more practical study - one where we are able to enroll 500 unvaccinated children and 5000 fully vaccinated controls matched for age, sex, socioeconomic group, geographic location, urban vs rural vs suburban setting and race.
This study - which would still be very difficult and expensive to do - would only be able to detect a more than 15-fold difference in autism prevalence between the two groups. It could detect as little as a 7-fold difference, but only if we were willing to accept a beta error (chance of erroneously saying there is no difference when there is a difference) of over 50%.
I doubt this would “satisfy” the vaccines-cause-autism believers if the results were negative.
So, before someone tells me - again - how easy it would be to study autism in unvaccinated children, first go out and try to find a few thousand completely unvaccinated children and then tell me how easy it is.
Ethical Considerations:
Another suggestion made was to have a study where children are placed into “no vaccination” and “vaccination” groups. This, of course, would be rejected out of hand by any Institutional Review Board because the risks of not vaccinating are well known and quite serious. On the other hand, the connection between vaccines and autism is tenuous at best. It would be unethical to expose adults to a known serious risk in order to test a weakly-supported (again, at best!) possible risk. In children, it would be unthinkable.
Another “modest proposal” was to vaccinate one group per the suggested schedule and vaccinate another group with four times the amount of vaccine. Well, at least this commentor understands the concept of “dose-response”. If - as some people argue - the current amount of vaccine is causing some amount of autism, then it would be reasonable to expect that more vaccine would lead to more autism.
Of course, the idea of doing something with the intent of causing harm to children is unlikely to pass the ethical hurdles of a real IRB. However, even if the hypothesis is incorrect - even if the researchers argue that harm is extremely unlikely, they would still be exposing children to four times the dose used during the safety trials of the vaccines, which would constitute an unreasonable risk, especially to test a hypothesis that is already weak.
Additionally, if the results are “negative”, the vaccines-cause-autism proponents could (and would) argue that their hypothesis is correct but that the threshold for causing autism is lower than the “standard” vaccine regimen (you know, the “too many, too soon” argument). Thus, the argument for doing the study (absent the obvious ethical considerations) has largely disappeared.
An Alternative:
Part of the difficulty in doing a study of unvaccinated children is tracking them down. There is no national or even statewide database that can “spit out” the names of unvaccinated children; you’d have to track them down one by one, a process that can take years to complete, given the numbers needed.
Additionally, the proposed study design wastes the effort used to collect the subjects based on a single variable - vaccination.
A much better study design would be to look at two age, sex, etc. matched sets of autistic and non-autistic children. Autistic children are more “visible” in the medical and educational systems than unvaccinated children (who are often “under the radar”). It would be easier (although not easy) to recruit autistic children and non-autistic control subjects than it would be to track down thousands of unvaccinated children.
Another advantage is that the researchers could collect data on a wide variety of exposures, genetic issues and other variables that could later be “mined” to find other promising avenues of research. This in contrast to the “single issue” study looking at unvaccinated children, which can only compare issues related to vaccination.
Even with those advantages, the study will be “swimming upstream” a bit.
A study of autistic children in the 3 - 6 year age range would need over 683,000 children in each group to detect a 10% difference in vaccination level. It would need nearly 22,000 in each group to detect a 50% difference. With a predicted number of 110,000 autistic children in that age range, that is a sizeable fraction of all autistic children.
A more manageable study - one with 10,000 children in each arm (which is still a HUGE study!) - would only be able to tell the difference between the national average of 0.3% unvaccinated in the non-autistic group and 0.1% unvaccinated in the autistic group (at the specified levels of confidence). If the difference is smaller than that, the results would be considered negative (i.e. that there is no effect of vaccination). For reference, a study with 1,000 children in each arm would show statistical significance (at our specified level of confidence) only when the autistic group was below 0.01% unvaccinated or above 1.7% unvaccinated.
Of course, we wouldn’t have to just look at unvaccinated vs fully vaccinated with this study, which is a large part of its superiority. We could look at a dose response of vaccination - to see if it really is “too many” - as well as the age at youngest vaccination - to see if it really is “too soon”. In fact, a few studies have already looked at those issues and found that there is no difference between the autistic and non-autistic groups. I suspect this is the reason the folks pushing to “put on a study” want to look at vaccinated vs unvaccinated - they hope that the numbers will be different (or, at least, not as definitive) the other way round.
Conclusions:
The current push to “put on a study” comparing the autism prevalence in vaccinated and unvaccinated children is likely to come to political fruition long before it generates any real scientific interest. Anyone who has actually done a human study will recognize that the proposed study format - find a bunch of unvaccinated kids and compare their autism prevalence to the general population - has serious problems.
If our political leaders eventually find it expedient to “put on a study” in a vain attempt to quiet the protests of the vaccines-cause-autism fringe, then we need to be prepared to insist that it be a proper study, not a rigged “study” pre-ordained to find the “correct” results.
Prometheus
Filed under: Autism Policy, Autism Science, Critical Thinking | 29 Comments »