When you try to calibrate a polling model, which is hard enough to begin with, the dynamics - how the model moves over time - are particularly mysterious.
Interesting piece, thanks. Also reminds me of some of the Taleb/Silver arguments from a few years back (https://nautil.us/nassim-talebs-case-against-nate-silver-is-bad-math-237369/). On the calibration point, also feels like having a prediction interval would shed some light on the usefulness of models, e.g. if my current 95% interval for probability of a win is X-Y, then this uncertainty should ideally narrow over time, with my election day probability still within this range (otherwise I’ve potentially been too overconfident/reactive to news along the way).
Yes, I was reminded of that debate too - although my recollection was mostly that it appeared obvious that Taleb was wrong: confusing a snapshot read: "what if the election was today?" with a projection "what if the election was on election day?" So I couldn't work out why he was picking the fight.
On the calibration - I think everyone has confidence intervals on their electoral college number projections (e.g., 95% that Harris will land in the range 170-307). And similarly for vote share. But as for "95% intervals for probabilities of a win" ... this sounds like appealing to distributions of probabilities, a concept I've always found troubling (though I've seen it as a motivation for the existence of Beta distributions). I rather feel as though probabilities of probabilities doesn't make sense, and should be replaceable by a point probability without loss of information, though I am not sure how to make this precise. But it also feels as this should be a solved question - so if you (or anyone else) knows of a good discussion, I'd really like to read it.
Yes, framing around distribution of probabilities isn’t very concrete (especially as a win is a binary outcome, whereas electoral college votes are a hard number that can be compared to predictions on the day). If I was approaching the problem fresh (on a Friday eve, having not thought about it nearly as deeply as you and others) it seems like calculating the amount of density of the electoral votes above 270 would provide a sensible (and testable) point probability estimate? And with a preference for a prediction interval on votes that narrows over time, we’d expect this point probability to be fairly non-jumpy over time?
Exactly this. Simply summing up the distribution to 270 is precisely how I extracted win probabilities for the Princeton Election Consortium model, https://paulmainwood.substack.com/p/us-election-2024-poll-aggregator. And then plotted them over time (you can see them in the plots above), and concluded from their massive jumpiness that they hadn't even come close to fixing their uncorrelated errors problem. And while their CIs do narrow over time, they do so to such a ridiculous, overconfident degree, that they're pretty much useless.
I would have thought a reasonable polling average methodology should also be able to predict the outcome of polls, at least over short time horizons (e.g. those currently in the field but not released yet).
I think this is the best way to think about the dynamic linear model-based ones (The Economist, FiveThirtyEight's new model, Data Diary). You are essentially selecting your state over time to be the one that best predicts the data coming in the next day.
And then the only thing special about 5 November is that it is an especially important poll.
Interestingly, this is exactly what his adjustments like the "convention bounce" violate. That is, it's an additional adjustment to take account of things like goodwill towards one candidate after their convention, which are assumed a) to exist, and b) to be temporary - so the model "aims off" for several weeks, in order (in his view) to get a better lead on the final 5 November result.
Of course this means it won't get the polls right in the interim, unless that adjustment is turned off again.
Model 1 predicts a candidate gets 310 EV during August to end of September (~9 weeks), then adjusts to new polling and switches to 260 EV for the 5 weeks to election day.
Model 2 predicts 260 EV during all of Aug-Sep, then adjusts to the same new polling to switch to a prediction of 310 EV which it retains for 5 weeks to election day.
Actual result is 315 EV.
The approach you suggest would score Model 1 as far superior to Model 2.
Interesting piece, thanks. Also reminds me of some of the Taleb/Silver arguments from a few years back (https://nautil.us/nassim-talebs-case-against-nate-silver-is-bad-math-237369/). On the calibration point, also feels like having a prediction interval would shed some light on the usefulness of models, e.g. if my current 95% interval for probability of a win is X-Y, then this uncertainty should ideally narrow over time, with my election day probability still within this range (otherwise I’ve potentially been too overconfident/reactive to news along the way).
Yes, I was reminded of that debate too - although my recollection was mostly that it appeared obvious that Taleb was wrong: confusing a snapshot read: "what if the election was today?" with a projection "what if the election was on election day?" So I couldn't work out why he was picking the fight.
On the calibration - I think everyone has confidence intervals on their electoral college number projections (e.g., 95% that Harris will land in the range 170-307). And similarly for vote share. But as for "95% intervals for probabilities of a win" ... this sounds like appealing to distributions of probabilities, a concept I've always found troubling (though I've seen it as a motivation for the existence of Beta distributions). I rather feel as though probabilities of probabilities doesn't make sense, and should be replaceable by a point probability without loss of information, though I am not sure how to make this precise. But it also feels as this should be a solved question - so if you (or anyone else) knows of a good discussion, I'd really like to read it.
Yes, framing around distribution of probabilities isn’t very concrete (especially as a win is a binary outcome, whereas electoral college votes are a hard number that can be compared to predictions on the day). If I was approaching the problem fresh (on a Friday eve, having not thought about it nearly as deeply as you and others) it seems like calculating the amount of density of the electoral votes above 270 would provide a sensible (and testable) point probability estimate? And with a preference for a prediction interval on votes that narrows over time, we’d expect this point probability to be fairly non-jumpy over time?
Exactly this. Simply summing up the distribution to 270 is precisely how I extracted win probabilities for the Princeton Election Consortium model, https://paulmainwood.substack.com/p/us-election-2024-poll-aggregator. And then plotted them over time (you can see them in the plots above), and concluded from their massive jumpiness that they hadn't even come close to fixing their uncorrelated errors problem. And while their CIs do narrow over time, they do so to such a ridiculous, overconfident degree, that they're pretty much useless.
I would have thought a reasonable polling average methodology should also be able to predict the outcome of polls, at least over short time horizons (e.g. those currently in the field but not released yet).
I think this is the best way to think about the dynamic linear model-based ones (The Economist, FiveThirtyEight's new model, Data Diary). You are essentially selecting your state over time to be the one that best predicts the data coming in the next day.
And then the only thing special about 5 November is that it is an especially important poll.
I think 538 previously (ie Silver Bulletin now) viewed this as something their model should do too
Interestingly, this is exactly what his adjustments like the "convention bounce" violate. That is, it's an additional adjustment to take account of things like goodwill towards one candidate after their convention, which are assumed a) to exist, and b) to be temporary - so the model "aims off" for several weeks, in order (in his view) to get a better lead on the final 5 November result.
Of course this means it won't get the polls right in the interim, unless that adjustment is turned off again.
Model 1 predicts a candidate gets 310 EV during August to end of September (~9 weeks), then adjusts to new polling and switches to 260 EV for the 5 weeks to election day.
Model 2 predicts 260 EV during all of Aug-Sep, then adjusts to the same new polling to switch to a prediction of 310 EV which it retains for 5 weeks to election day.
Actual result is 315 EV.
The approach you suggest would score Model 1 as far superior to Model 2.