“Surprises” in BLS Jobs Revisions Became More Frequent After 2020

“Revisions are a feature, not a bug,” former BLS Commissioner Erica Groshen explained on a recent episode of Moody’s Talks — Inside Economics entitled “In Defense of the BLS.” Groshen argued that most BLS jobs numbers are really early reads that involve forecasting, and, when information comes in later that contradicts the forecast, surprises are useful. They often indicate “inflection points” where things are changing in the economy.

Because of these “surprises,” the article BLS Jobs Revisions Have Been “Out-of-Control” Since the 1950s established that no one distribution could represent the BLS 1-month jobs revisions. What about two distributions, a distribution of “surprise” revisions and normal revisions?

Using the same data, where all numbers are in thousands of jobs, the mclust package in R made this very easy (see the code). Above is a density plot of the two distributions found using data from after 1990. Quantile residuals can be computed with the following formula that also includes the likelihood-based parameter estimates.

The means are estimated to be slightly above zero at 10.86 and 39.44, which would correspond to a tendency for positive revisions, but they are dwarfed by the standard deviations. Rounded to the nearest 5, the standard deviations are 55 and 430 (55,000 and 430,000 on the original scale). That is, the “surprise” distribution’s standard deviation is almost 8 times larger than the distribution of ordinary revisions.

Out of curiosity, I plotted these quantile residuals on an individuals chart. The model is so flexible that it’s a bit hard to interpret, as Rule 1 violations can now be absorbed into the surprise distribution.

It’s notable that there are still a few rule violations, including two Rule 1 violations near the global financial crisis (GFC) and some other Western Electric rule violations in yellow that indicate trending, notably appearing around the dot-com crash, the GFC, and at the beginning of COVID. There are no rule violations in the past few years, but the higher proportion of surprises has the model sponging up the larger values.

The surprise probabilities

The last estimated parameter in the residual calculation is the proportion of the 1–month revisions that are surprises, which is 0.113, or 11.3%. Assuming independence, that’s 1.4 (0.113 x 12) surprise 1-month revisions per year, a bit frequent for anything called an “inflection point.” While the model itself assumes independence of surprises, we can look for trending in the surprise probability estimates.

High surprise probability revisions happen with apparent regularity from 1990 to around 2015. In the 90s and early 2000s, these surprises happened overwhelmingly in reporting month June (revising May’s report of April’s job numbers). After the GFC, they started happening in reporting month February. Post 2020, it became more difficult to distinguish whether the revision was even coming from the surprise distribution and probabilities became more evenly distributed over [0, 1]. The moving average does show a higher surprise probability on average post-2020, though the troughs are close to some of the sustained periods.

Conclusion

While on the podcast, former BLS Commissioner Erica Groshen described the BLS employees as dedicated, hard-working data professionals who care deeply about the quality of their forecasts. She did admit that response rates have been declining, but maintained that the survey quality itself remained high. She was open to the possibility that the data quality could decline if faith was lost in the organization due to the recent activities by the Trump administration.

I thought Groshen did a wonderful job describing the importance of the survey infrastructure. Our collective knowledge of national metrics like unemployment mean that better decisions are made and that prices, for instance, for bonds, are set more optimally. Once lost, this survey capability is a public good that is not easily replaceable by the private sector.

However, if the Employment Situation is actually a collection of early-read forecasts, survey response rates have been falling, and 1-month jobs revisions are coming from at least two distributions with wildly different variances, perhaps the organization could do a better job communicating what is actually going on with this crucial piece of public infrastructure.