In a famous (to readers of this blog) example of statistical error, a researcher reported that beautiful parents were 36% more likely to have girl babies. The saps at Freakonomics fell for this one hook, line, and sinker, but I was suspicious, being a bit familiar with the sex ratio literature, which has found that the proportion of girl births is very stable at around 48.5%-49% in various subpopulations (whites and blacks, younger and older mothers, etc.).
The published analysis compared the sex ratio of children of the “very attractive” parents to all others, and the difference was 8 percentage points—not the “36 percent” mistakenly reported in the journal article and credulously repeated by Freakonomics, but still about 100 times larger than any plausible effect.
But here are the data, which look kind of convincing:
This comes from a survey of 3000 people and, as you can see, the percentage really is highest in that “most attractive” category; indeed, the comparison has a p-value of less than 0.05, hence the result being published.
OK, the result has no scientific plausibility given the vast literature on the topic (large effects of this sort are only found with studies with small samples), also the statistical significance is entirely explainable by forking paths—the odd choice of comparing category 5 with 1 through 4, rather than comparing 4 and 5 with 1 through 3, or just running a regression:
That slope estimate is not statistically significant; in any case the estimate is too noisy to be useful, given the range of realistically possible effect size. It’s the kangaroo problem.
But let’s set all that aside. Let’s forget all our subject-matter understanding and statistical expertise.
Here’s my question. Is there a way that a researcher without that specialized knowledge could see the problem with this study?
The answer, I think, is Yes. And the method is fake-data simulation. Which we could also call simulated-data experimentation. Or bootstrapping. With the only difference that, conventionally, the bootstrap is used as a way to get a bias correction or uncertainty estimate for an interval, and here we’re using it to develop intuition about a statistical data-collection process.
We would like to understand the statistical properties of the beauty-and-sex-ratio study by simulating hypothetical replications. In this case, the data came from 2792 participants who had at least one biological child in Wave III of the National Longitudinal Study of Adolescent to Adult Health. The full sample had 4877 respondents, of whom 2% were characterized as “very unattractive,” 5% as “unattractive,” 45% as “about average,” 37% as “attractive,” and 11% as “very attractive.”
We simulate replications under a null model. Assuming the probability of a girl birth is 0.488, independent of parental attractiveness, we simulate the results of 2792 births with proportions in the five attractiveness categories as given above.
Here are 20 simulated datasets, with for convenience the fitted least squares line displayed for each:
We see patterns just as dramatic as that of the observed data shown earlier, indicating that those data should not be taken as evidence against the null model.
Now here’s the point. The wonderful thing about these simulation experiments is that they can reveal problems with a naive design, even if you didn’t anticipate any difficulties ahead of time. The original author of that paper could have saved all of us a lot of trouble by simulating 1000 replications of those survey data on the computer, either before or after he performed his data analysis. No math required, no subject-matter knowledge required.
It’s too late for him, but it’s not too late for you to do this for your next modeling and analysis problem and avoid the embarrassment.
This is, in a very real sense, frequentism. To me, “frequentism” is not about unbiased estimators or long-term coverage or whatever; it’s about thinking of the data you see as one draw from a distribution of possible realizations of the data. It’s about looking at this distribution, both to see how your data and inferences could fluctuate in the future, and to interpret the data you do see.
Why don’t people do this all the time?
One reason, I think, is that they have a naive understanding of statistical theory and think that if you act like statistical significance = truth then you’ll be ok 95% of the time. They don’t check because they think there’s nothing to check, or maybe it’s more accurate to say they think that the experts have already checked for them. They don’t check their statistical analysis any more than I check the cables every time I get on an elevator; I assume the fundamental questions of elevator safety have all been settled already.
The other reason is that simulations is that they take effort. A simulation experiment requires a fully generative model—a rule for defining the truth and simulating data from some specified random process—followed by analysis of the simulated data, all nested within a loop and ending with comparison of inferences to truth. This involves additional work compared to that required to conduct an experiment, first because it requires an automatic procedure for data analysis and second because it requires a generative model. We have found that this additional effort involved in constructing a generative model and automating the data-analysis process is itself helpful for thinking through the experimental process. Indeed, it has similarities to the steps of preregistration.
This example and discussion are in Section 10.5, “Simulated-data experimentation as virtual replication,” of our Bayesian Workflow book. But I’ve never published it as a standalone article or blog post before, and the message is so clear that I wanted to share it here with you.
When people (including me!) get things wrong, it’s salutary not just to figure out where they went wrong and how it happened, but also to step back and see if there’s some more general process they could’ve followed that would have flagged the problem. And here we can do so.