Election data processing

We obtained our data from IDEA's website by extracting it from all pages with a Java program. This could be repeated in the future, if IDEA chooses again to update its database. However, it requires some manual setup to work correctly. The data was then read into the Access database as table rawDataIDEA. The Java program made the following modifications to the data:

• Parliamentary elections were designated as type "A",  presidential elections as type "B" in every row
•  If IDEA reported "n/a" in a field, it was replaced with "-9"
• All numbers were stripped off their formatting (e.g. thousand separators), so that Access could interpret them correctly as numbers. In Access, formatting was applied again
• All percentages were divided by 100 in order to scale from 0 to 1

As first step in access, certain rows were filtered out of the dataset, because they represent cases that would throw off our algorithm. The following table lists all these cases. In almost all these cases, two elections were held in the same year. We retained the election with the highest voteVAP (totalVote / VAP). The Malawi cases are different, as it seems that IDEA reported all elections for Malawi (parliamentary & presidential, 1994 & 1999) twice. Hence one of each has to be removed from the dataset. Unfortunately this mistake by IDEA required the unique identifier for each row (field ID) to be used for determining which rows to filter out of the dataset. Therefore if the original IDEA dataset were to be updated, the step of determining these cases and entering them in the special table would have to be repeated. This shouldn't be a problem though, because if IDEA updates its data, we would have to check anyhow again for years with two elections of the same type. The IDEA data without these records is stored in rawDataIDEA_Without_Deleted_Record.

As a next step, the polity country code was assigned to the IDEA records. Several "countries" covered by IDEA where assigned non-Polity codes in this step, because they are not independent countries. Those were: American Samoa (2004), Anguilla (1312), Aruba (1314), Cook Islands (3000), Guam (2276), Macau (1546) and the Occupied Palestinian Territories ("West Bank & Gaza", 2203). "Independence" years for these were set according to when they came into existence as their present-day entities according to the CIA World Facts Book.

Then updates and additions we made to account for the lack of IDEA data were added into the dataset. The result of that and the previous step is stored in IDEA_voteUpdates. It should be noted that cases in which IDEA and the update dataset contained votes for the same election, IDEA was always used. Cases in which the IDEA dataset contained any special code or 0 (hence <=0), but a vote count was available from the update dataset, the vote count from the update dataset was used.

• If the country was independent and in existence according to the CIA World Facts Book (Independence Year) in 1945, then every field in 1945 was assigned a "-1".
• If the country was not independent or in existence in 1945, then the first election was looked at:
  • If the first election was after the independence year as reported by the CIA World Facts Book, then a row was added for the independence year and every field assigned a "-1".
  • If the first election was before the independence year, the first election was retained and nothing was changed in the dataset.
• If a country had elections in one type earlier than in the other, then the "-1" rule was only applied to the earlier election type. For example, IDEA reports parliamentary elections for the US in 1946 and presidential elections in 1948. A "-1" record in 1945 was only added for the parliamentary elections.

The output of this step above is stored in IDEA_voteUpdates_With_IndependenceMarkers.

All records were then transposed such that we ended up with one row per year per country. For example the United States before this step was:

After the transpose step, the United States became:

The output of this step was stored in allVotes.

In the following step, the data was expanded to cover all years between independence or the first year in the dataset and 2006. We used the following rules for row expansion

• If the time between two reported elections is greater than 7 years, then a "-4" was entered into every field of a new row in the 8th year after the earlier election. For example, if there are two parliamentary elections, one in 1945 and the other one in 1975, then in 1953 a "-4" row will be inserted. This assignment will be done, as all row-based ones, for both election types. In the example this means that a "-4" row will be added for the parliamentary election even if there was a presidential election in 1950. Why all of this? The intention is to avoid labeling a dictatorship as democracy by carrying over an election result for a long time. We chose 7 years, because of some long terms of presidents, e.g. France which used to be 7 years until October 2000 (changed to five years since then).
• If there is a "-9" row (n/a as reported by IDEA) between two elections, the row was deleted if the time between the two elections was not greater than 7 years. IDEA designates an election as n/a if it was not a competitive election, meaning one candidate ran unopposed. In those cases, we believed it was better to use the results of the last competitive election for our maximum of 7 years than to drop the data completely. For example, Ireland had competitive presidential elections in 1973. The elected president died though and non-competitive elections were held in 1974, which were reported as n/a by IDEA. The president elected in 1974 resigned in 1976 and a successor was elected again without competition. That president was re-elected in 1983 in another non-competitive election. IDEA designated the elections in 1976 and 1983 as n/a as well. Using our method, we carried over the result of the last competitive presidential election in 1973 until, but not including, the year 1980 and then recorded a -4 for 1981-1989. Ireland held competitive presidential elections again in 1990.

The results of this step were stored in expandedVotes and exported as stataVotes. All conversions are performed by executing the macro createDataset. Election data with UN Population data was exported as stataVotes_UNPopulation.