Michael Traugott

The television networks and their major partners seem to be on the verge of recovering from two different kinds of exit polling disasters in the last two national election cycles, and that will be a good thing for the American public. But there is a long way to go before things are in good order for the news media so they can provide a detailed explanation of voting behavior in the 2004 presidential election.

This is important and necessary work, as the exit polls - in their explanatory function, not their predictive one - have become an independent voice of the public in explaining what happened and why. Without this voice, journalists and political commentators would regress to the old style of political reporting, relying upon party officials and strategists to provide the causal links and explanations for why one party or candidate succeeded over another. Imperfect as they may be, the exit polls are a significant empirical tool for understanding the public will and interest, and their availability on Election Night and in the next few news cycles is critical in a world where news production has speeded up and what enters the news stream first often becomes “common knowledge” that can only be refuted with difficulty and limited success. Deep understanding from the National Election Study and other post-election surveys is important, but the timeliness of the exit poll information is important in framing subsequent coverage.

What the Exit Polls Do

It is important to remember that the exit polls serve two distinct functions: they provide information on who won and explanations for that success. Since the late 1960s, the use of exit polling has provided electronic media with a significant advantage in election coverage, giving them the ability to produce news about outcomes long before the full set of votes have been counted. Prior to this period, samples of “key precincts” were used to estimate the outcomes, and additional sets of relatively homogeneous precincts enabled the networks to talk about how various groups were voting. In addition to being subject to the ecological fallacy, this technique could only be used for such characteristics as size of place of residence or in relation to residential patterns of segregation; they could not be used to analyze patterns of voting by gender, for example.

Many of these analytical and inferential problems were alleviated when exit polling was developed as a technique to intercept individual voters as they left the polls in samples of precincts. By collecting and analyzing individual-level information, there was useful data available for projecting races as well as for analyzing voter intent and behavior by attitudinal characteristics as well as a full set of demographic factors. But the technique added several layers of logistical complexity to the former “key precinct” analysis. First of all, there were many more pieces of information to collect and process, and the analysis required more sophisticated statistical approaches that were based upon extensive custom software systems.

How the Exit Polls Function

It is important to keep in mind that there are several distinct but interrelated parts of any exit polling operation. The first is problem of constructing stratified samples of precincts. Another is the logistical nightmare of hiring and training a large temporary staff of exit poll interviewers to work in those precincts. Furthermore, there are issues of assembling and processing the data. And then there is the matter of how to use the exit poll results on candidate preference in conjunction with other electoral information about the places in which they were collected.

While the networks spent more than a decade working on these issues on their own, all of these procedures were most recently taken care of in a consolidated fashion under the rubric of the Voters News Service (VNS), originally incarnated as Voter Research and Surveys (VRS) in 1990. While this operation resulted in significant cost savings for the individual networks when they pooled their operations, it had several other adverse, unintended consequences. One of them was that it resulted in the production of a single estimate of the outcome of each race, and no set of multiple independent indicators was available to calibrate against any one of them. And when there was a meltdown of data collection and analysis procedures as occurred in 2002, there were no other estimates of outcomes or set of analyses to explain what happened on Election Day.

There were two significant problems with the VNS operation, identified as part of the post-2000 reviews, both internally and in the Congressional testimony that followed, that were being worked on going into 2002.1 The first was whether the administrative software and hardware that had been supporting VNS (still a DOS-based system) were up to the task. The staff and board of VNS realized these problems and had contracted with Battelle to reprogram the software and bring it up to current technical standards, committing an estimated $10 million to the effort. Why the contract was given to Battelle has not been widely discussed, but they had no prior experience in the election field. For some time leading up to 2002, it was clear that there were difficulties with Battelle’s work and rate of progress.2 Battelle was behind on their programming; new technology including voice recognition software had not been adequately tested for data entry; and many data management modules had not received adequate pretesting. In the end, the load of data input on Election Day was more than the system could handle, and estimates could not be produced without data. So the operation began to close down at midday.

However, a second issue that has not received much discussion was whether the statistical modeling of the VNS system was up to the task. The projections were not based just upon the exit poll interviews. There was a detailed history for each precinct, including information on turnout and the partisan division of the vote over time. And the precincts were “aggregated” or combined to form state regions or subdivisions for additional calibration. The system included prior estimates of the outcome of each race, based upon intelligence that included pre-election polls and a variety of expert assessments. When the data on Election Day came in conforming to the priors, reliable estimates could be made quickly and with confidence. But if this was not the case, the models waited for raw vote totals to come in. In extremely close races, the models could conceivably wait for all the raw vote to come in. While these models were conceptually appropriate, they did not employ the very latest statistical theories and models. For example, the models used by the BBC, although employed in a parliamentary system, uses more sophisticated regression algorithms.3 There had not been any significant discussions about revising the estimation models at VNS for 2002.

Soon after Election Day,2002, the networks began another review, and they eventually decided to disband VNS. In its place, they agreed to support new exit poll operation, the National Elections Pool (NEP), to be headed by Warren Mitofsky and Joe Lenski, two very experienced hands at producing exit polls. There has not been any public discussion of revising the statistical models for this new operation either.

What Can Be Done

The task for Mitofsky and Lenski is not an enviable one. The old VNS software system, whatever the nature of its current ownership and availability, was not up to the task of handling large amounts of data with current technology. And VNS on its own was unable with almost two years’ notice to develop a functioning new system to produce a relatively limited number of estimates for an off-year election. The new team has less than a year before the first presidential caucuses and primaries to develop another new system to produce a larger number of improved estimates.

Nothing happens in an exit poll operation unless the data are available for processing. This suggests that the primary task for Mitofsky and Lenski is to develop a technologically advanced national data handling and processing system. They have been conducting exit polls for some time in other U.S. and international elections, and they were able to provide such a service for CNN in 2002 for ten states through their “RealVote” system. Because data collection and processing are likely to be their primary focus, it also means that less time and effort will be devoted to the estimation models. There are statistical experts in the United States and overseas at places like the BBC who could provide assistance in such an effort, but it remains to be seen whether such expertise could be translated into a working software system in less than 12 months.

Press reports of the new contract suggest that NEP will provide both exit poll data and projections of winners. VNS started by providing projections to all of its clients, but by 1994 ABC had decided to assemble its own group of political and statistical experts to try to produce faster estimates. Eventually each of the networks developed an independent analysis and projection unit to try to gain an edge on the others. This increased competition to be first is one part of what produced the problems on Election Night 2000. In the new contract, the sponsoring members have preserved their right to continue this practice, at the same time that they have given up some operational control over the data collection process to NEP.

The new organization is still likely to produce the only estimates of outcomes and explanations, so that will remain a problem and an issue. We need at least one other independent data collection activity, to produce another estimate of the outcomes but also to employ a different questionnaire, in theory reflecting additional news judgments about appropriate content. This will provide a fuller explanation and more reliable estimates of what happened and why.

After the 2000 election, Mitofsky ended a recap of the events and his personal experience by saying:

I look forward to the election in 2002. It will mark a new beginning. It will demonstrate that VNS and the various decision desks learned from their mistakes. It will also show that the last election was a fluke.4

With added responsibility, he and his partner will have a chance of their own to demonstrate the viability of election polls. It is an important task for which we should all wish them well.

Michael Traugott is Professor of Communication Studies and Political Science at the University of Michigan and a past president of the American Association for Public Opinion Research.

Return to the Roundtable

1. VNS commissioned a review by RTI, whose recommendations for change were incorporated into the record of the hearings of the House Committee on Energy and Commerce. The full set of testimony before the committee, including Paul Biemer's testimony on behalf of RTI can be found here. [return]
2. For a discussion of many of these problems, see Larry Barrett, "Voter News Service: What Went Wrong?" available here. [return]
3. The work of Clive Payn at Nuffield College, Oxford, is particularly illuminating on this point. [return]
4. Warren J. Mitofsky, "Fool Me Twice: An Election Nightmare," Public Perspective (May/June 2001), p. 38. [return]