Our weekly episode of Still Counting is ready. This was taped yesterday morning, so you can blame any bad takes on sleep deprivation:
And here’s the audio. We’re very grateful for any reviews, comments, likes and subscribes. Yes, I’m going to say that every week.
Despite results on Tuesday that, to a first approximation, made our coverage this year look smart, Clare, Galen and I were all in slightly different moods. Elections build up to a stressful crescendo, frankly. It’s in the final days when you both have the most on the line and the most to do.
And we’re still feeling that stress, to some extent. So rather than come up with the perfect transition paragraph, I’m going to semi-awkwardly pivot to a debate about models versus prediction markets.1
I don’t particularly like this debate. To a large extent, I think of pollsters, modelers, prediction markets and smart political wonks as all being within “the industry”. Their interests are mostly aligned with one another. And sometimes they’re aligned against the conventional wisdom or the media ecosystem more broadly.
If you’ve read the book, it’s kind of a Village vs. River thing. Or smart money vs. dumb money. The smart money was on Democrats having a good and possibly great election. The smart money doesn’t always get it right, though.
Still, I’m finding myself a little annoyed if prediction markets are trying to take a disproportionate share of the credit as opposed to “the industry” broadly. A few key points.
FLIPR was — correctly, given what happened — more confident in overall Democratic odds in the Senate throughout the stretch run of the campaign.2
For the Nov. 3 datapoint in this chart, I’m using the last “print” of prediction market data before any results were reported starting at 6 p.m. There was a weird little jitter back toward Republicans at prediction markets during the course of the day on Tuesday, which was directionally wrong and I don’t think had any real basis, as I wrote at the time. Prediction markets are smart about incorporating new information, and in principle that’s one of their advantages versus models — we have to wait for whatever happens to show up in the polls. At the same time, there’s sometimes a “buy the rumor, sell the news” thing going on here. Markets almost certainly overreacted to the Haley Stevens thing in Michigan and to random reports about turnout on Tuesday morning and afternoon.
Once results began to come in on Tuesday, prediction markets very, very quickly honed in on the right results. They were better than the networks, and faster than the NYT Needle. That was impressive, though I think we got to the right answer pretty quickly in the liveblog, too.
However, if we are going to turn this into a competition, I think the better models like FLIPR unambiguously had a better pre-election period. Our model arrived at the right answer faster, leading the markets by days to weeks.
Indeed, our forecasts moved the markets when they launched on Aug. 11. This is as clear an indication as you can get that traders believe we provide alpha that the markets alone are not capturing.
Furthermore, prediction market probabilities strongly converged toward ours in individual Senate and gubernatorial races. I was tracking this stuff closely, precisely because I envisioned that we might wind up in this sort of debate where prediction markets are low-key stealing valor from the models. I grabbed these prices late Tuesday morning. First for the Senate:
And then for gubernatorial races:
The differences are interesting. In the Nevada governor’s race, for instance, prediction markets were clearly incorporating some assumptions about early voting in correctly being more confident about Aaron Ford than we were. I think that was smart, although that’s partly because early-voting analysis tends to be unusually reliable in Nevada as compared to other states.
Sometimes, I have subjective takes that might override FLIPR, but we think projects like FLIPR are going to be more informative in the long run if we stick to the algorithm.
I am interested in the differences, however. I discussed the Senate races where prediction markets and FLIPR disagreed in a post on Oct. 25. The funny thing was, in several of the races like South Carolina, FLIPR was more accurate, but my “gut” was on the wrong side of FLIPR! Prediction markets do deserve some credit relative to FLIPR on Mississippi, though, where Scott Colom is probably going to win in a major upset.
Overall, though, prediction market prices converged toward FLIPR, and not the other way around. Here were the prices as of Oct. 25:
You can see that there were some bigger gaps then than we wound up with on Tuesday. Although, the correlation was >=.99 in both cases.
You can also debate how to compare them. If Peltola wins in Alaska but by an underwhelming margin, as seems likely, does FLIPR deserve credit for showing a closer race? Or not, because prediction markets also “called” the right winner, but did so more confidently? Any evaluation that anyone publishes is usually going to be low-key biased toward the most favorable “reasonable” method for their forecast.
In the end, prediction markets and the better models converged on basically the same prices. But as you can see by comparing the Oct. 25 graphic to the Nov. 3 graphic, prediction markets moved toward model prices and not the other way around. The prices they wound up with on Tuesday morning, we often had days to weeks sooner.
I don’t want to turn this into a competition between Silver Bulletin and the other models. Split Ticket and FiftyPlusOne in particular also have smart models. (There’s more or less a “right” way to build one of these, so we all use pretty similar approaches.) Different analyses put their performance in slightly different orders. Yes, there were one or two other models that I thought were slightly wacky, but in general, these differences have dissipated, and the modeling part of “the industry” has matured from, say, 2016.
But traders get to use our forecasts, and they trust them, and I’m sure they make up some proportion of our subscription business.
The best-performing models, including FLIPR, don’t use prediction market data, however. They provide truly independent information. If there aren’t reliable models to look at, prediction markets are still often smart. But they can also get unhinged, like in the Wisconsin and Michigan Democratic primaries this year.
Building these models is a lot of work, with some subtle dynamics that take a lot of time to explore fully. One of those, for instance, is that polling in primaries is much less accurate than polls of general elections. So the markets sometimes wind up trusting polls too much in primaries, and too little in general elections.
It’s flattering that traders tend to trust our prices. But in a world where these models didn’t exist, or if the models kept their forecasts private, I don’t think the prediction markets would be as accurate on either an absolute or relative basis.
However, there’s an irony here. Just like the prediction markets are downstream of the models and benefit from their existence, the models are downstream from the inputs they use, including the polls. If you ask me, who really deserves the most credit is pollsters like NYT/Siena and AtlasIntel that provided highly accurate results that defied the consensus to some degree. There would have been hell to pay if these polls had been off, especially given that the president has sued pollsters before that he didn’t like. It showed some courage for them to stick their necks out there and they deserve a victory lap.
So if I had to rank-order this, I’d go something like:
1) NYT, AtlasIntel and other high-quality pollsters who trusted their own data and published their results at some risk of embarrassment.
Close 2nd) The better models, including but not limited to FLIPR.
Close 3rd) Prediction markets. They did have a good year, certainly. And I think it shows them maturing that they increasingly trust the more reliable sources of information and shut out the noise.
Distant 4th) The conventional wisdom, more broadly, which incorrectly assumed that polls are persistently biased against Republicans. They aren’t. Polling bias is basically random from cycle to cycle, in part because pollsters adjust their methods in response to errors. This election wasn’t the first time that polls missed low on Democrats. The same thing happened in the New Jersey and Virginia gubernatorial races last year, and in the presidential election of 2012.
Clare, Galen and I will be at Crooked Con in Washington D.C. this weekend, with a live Still Counting episode at 9 a.m. on Saturday!
The post-election period can be tricky. As discussed yesterday, we’re going to look to strike a balance between reflections on the midterm, more forward-looking coverage, and posts that have nothing to do with the election at all. We have a big year planned, including one or two new sports model launches. We also need to get some rest.