US embassy cable - 04LJUBLJANA830

UNCLAS  LJUBLJANA 000830 
 
SIPDIS 
 
 
SENSITIVE 
 
DEPT FOR EUR/PPD, EUR/NCE AND INR 
 
E.O. 12958: N/A 
TAGS: KPAO, PGOV, PINR, SI 
SUBJECT:  BIAS IN SLOVENE OPINION POLLING:  MORE ABOUT 
AFFORDABILITY THAN POLITICS? 
 
 
Sensitive but unclassified -- protect accordingly 
 
1.  (SBU) SUMMARY.  As in most countries, the results 
of public opinion polls in Slovenia are widely 
discussed among media, politicians, pundits and the 
public.  However, few of these "consumers" are aware of 
the methodologies that are used or the ease with which 
polling data can be manipulated.  Although 
misrepresentation of polling data exists in virtually 
every country, the close relationships between polling 
centers, media outlets, and political parties in 
Slovenia make it incumbent on the consumer of opinion 
polls to study their methodologies closely and take all 
results with a grain of salt.  In an analysis of 
polling methodologies used by different survey research 
centers in Slovenia, we have determined that 
professional pollsters are highly knowledgeable and 
cognizant of potential biases inherent in their polling 
methods.  However, lack of sufficient funding in a 
small country often means that polls are done on the 
cheap and that many of the results contain considerable 
bias.  More difficult to determine is whether bias is 
ignored intentionally and whether political parties can 
"commission bias" when paying for a poll.  END SUMMARY. 
 
2.  (U) In late August, Post's Public Diplomacy Officer 
(who has a PhD in Political Science and a Master's- 
level certificate in Survey Research Analysis) 
conducted a series of interviews with researchers at 
three major polling centers in Slovenia:  the Research 
Center for Public Opinion and Mass Communication at the 
University of Ljubljana; the DELO STIK polling agency 
affiliated with the leading Slovenian daily "Delo"; and 
the NINA MEDIA polling agency affiliated with the daily 
"Dnevnik."  The following observations and comments are 
based on the interviews mentioned above and an analysis 
of the methodologies used by these polling centers. 
 
--------------------------------------------- -------- 
Sample Bias vs. Sampling Error: A crucial distinction 
--------------------------------------------- -------- 
 
3.  (U) One of the main problems with public opinion 
polls in Slovenia and one of the main causes of faulty 
analysis by the media lies with the fact that polling 
agencies often do not take adequate care to ensure that 
various biases are eliminated from their polling 
methodologies.  Every opinion poll contains some bias 
and reducing bias to a minimum often comes at a great 
cost.  Thus, every polling agency faces a tradeoff 
between bias and the cost of utilizing expensive 
procedures to minimize it.  Since most polling agencies 
are businesses looking to make a profit, cost-cutting 
is prevalent; nevertheless, lower costs often mean less 
accuracy. 
 
4.  (U) Another major problem is that the media, who 
are the primary consumers and interpreters of opinion 
polls, often do not adequately understand the 
distinction between sample bias and sampling error. 
Opinion polls that have considerable bias but that have 
minimal error are often presented to the public as if 
they were quite accurate.  Typically, for example, a 
media outlet will report that an opinion poll has a 
margin of error of plus or minus 2-5 percent.  What 
they do not say, however, is that in addition to this 
margin of error (known as "sampling error"), these 
polls also--in fact invariably--contain sample bias, 
which can often skew a given poll by as much as 10, 15, 
or even 25 percent.  A recent conversation with the 
news director of a major Slovenian television station 
revealed a complete failure to appreciate this crucial 
distinction. 
 
5.  (U) Sampling error is a measure of the likelihood 
that a perfectly random sample of the population will 
have the same characteristics as the population as a 
whole.  Sampling error can be measured using a 
statistical formula.  Sample bias, however, cannot be 
measured.  Sample bias occurs when a polling agency 
consistently fails to obtain perfectly random samples 
of the population (often for a multitude of different 
reasons).  For example, if a polling agency were to 
interview people only within a 50-mile radius of their 
headquarters, they would consistently fail to obtain a 
perfectly random sample of the population.  However, 
extensive polling within this 50-mile radius could very 
 
 
well reduce the margin of error to plus or minus 2 
percent, or even less. 
 
6.  (U) Sample bias is almost impossible to eliminate. 
Seemingly innocuous methodologies like phoning random 
numbers to conduct surveys can produce remarkably 
biased results.  For example, poorer households tend 
not to own telephones (or subscribe to fewer lines), so 
phone interviews consistently bias the sample in favor 
of wealthier respondents, who in turn often have 
different voting preferences.  On the other hand, if 
telephone polls are conducted during the day, they may 
consistently exclude working professionals (since they 
are less likely to answer their home phone) and thus 
overstate the views of the elderly, students, and the 
unemployed.  While telephone penetration rates in 
Slovenia are estimated at 94 percent, and while most of 
the centers poll from 3:00-9:00 PM, the prevalence of 
telephone bias still exists and impacts polling 
results.  This is true especially since "busier" 
members of household are less likely to answer the 
phone while those who have more free time are likelier 
to agree to participate in a survey. 
 
7.  (SBU) Polling agency directors were of course 
cognizant of such methodological flaws, but noted that 
telephone polls were cheaper than fieldwork and 
indicated that the exorbitant costs of fieldwork often 
prevented them from doing more methodologically 
rigorous polling.  Polling on the cheap, they argued, 
was better than no polling at all.  While it is true 
that some media in the U.S. regularly conduct 
methodologically substandard polls, these polls are 
usually discounted by polling professionals, who rely 
on the far more accurate analyses provided by academic 
institutions, major media outlets, and reputed polling 
organizations like Gallup, who have the financial means 
to conduct methodologically rigorous surveys.  In 
Slovenia, the small number of polling centers and the 
modest funding they receive means that few polls employ 
a rigorous methodology. 
 
-------------------------------------------- 
Weighted Samples and Representative Regions 
-------------------------------------------- 
 
8.  (U) One standard method for reducing bias involves 
the "weighting" of survey samples.  This involves 
factoring into the analysis the known differences 
between the sample and the population as a whole 
(usually determined from a census).  For example, if 
you randomly telephone ten people and the result is 
that 3 women and 7 men answer the phone, and yet you 
know the population is split evenly between men and 
women, then to weight the sample you would simply 
multiply the responses of the women by 7/10 and the 
responses of the men by 3/10.  The problem with this 
method, however, is that while it may be easy to weight 
responses by gender or area of residence, there are 
hundreds of other variables that simply cannot be 
weighted because their underlying values are unknown 
(e.g. religiosity, previous affiliation with the 
Communist Party, etc).  Most Slovenian research centers 
weight their samples by gender, education, and region 
of residence.  However, other variables like affluence, 
church attendance, and previous support for the 
Communist regime have all been shown to have a 
significant--and perhaps even greater--impact on 
political preferences than the variables currently in 
use. 
 
9.  (U) Another method for minimizing bias is to pick a 
representative region based on prior election results. 
This is sometimes called the "Peoria method."  If a 
particular region or municipality within a country had 
election results that perfectly mirrored the national 
averages, then polling agencies will sometimes conduct 
methodologically rigorous polls within those regions in 
order to predict future election results.  The problem 
with this method, however, is that populations shift 
and what may be a representative region during one 
election cycle may no longer be representative during a 
subsequent cycle.  This is especially true if a new set 
of issues is being raised from one election to the 
next. 
 
----------------------------------------- 
 
 
Slovenia's EP elections: What Went Wrong 
----------------------------------------- 
 
10.  (SBU) The failure of most Slovenian polling 
agencies to adequately predict the European 
Parliamentary elections is mainly due to various 
different types of sample bias.  While it is impossible 
to authoritatively determine why such bias occurred, 
one hypothesis that seems likely is that the winner of 
the EP elections--Nova Slovenija (NSi or New Slovenia)- 
-has a rather "unrepresentative" electorate.  To cite 
but one example, NSi attracts both less educated voters 
and highly educated voters.  So, if a polling agency 
samples a population and weights the sample by 
education, they may actually be missing the two 
extremes--the uneducated and the highly educated--and 
hence the result may under-represent the percentage of 
voters who would vote for NSi. 
 
11.  (SBU) Anecdotal evidence from field polls 
conducted by DELO STIK also indicates that New 
Slovenia's voters tend to be more fearful of the 
government (a relic of the Communist system), and hence 
less likely to answer public opinion polls out of fear 
that the government may be trying to keep tabs on them. 
If such suspicions really do exist, they could 
potentially result in a large understatement of the 
number of NSi voters within a given region.  This sort 
of bias is also more likely in a telephone poll than in 
a field survey. 
 
12.  (SBU) The inability of Slovenian polling agencies 
to adequately predict the turnout for the EP elections 
also reflects a different type of bias, one associated 
with the way the survey questions are phrased and 
perceived by respondents.  It is conventional wisdom 
that when asking respondents if they plan to vote in an 
election, 10-15 percent more respondents will claim 
they intend to vote than actually do.  Such "social 
desirability bias" (as it is commonly known) stems from 
their desire to appear as conscientious citizens.  A 
poll conducted by the Center for Public Opinion and 
Mass Communication indicated that even after the 
election was over, 12 percent more respondents 
indicated that they had voted than actually did.  While 
polling center researchers were well aware of this bias 
and generally predicted the low turnout in the EP 
elections rather accurately, the media seemed more 
surprised that so many fewer voters would show up than 
had claimed they would vote. 
 
13.  (SBU) A final reason for the poor polling results 
in the run-up to the EP elections also has to do with 
the problem of low voter turnout.  Many of the polling 
center directors we spoke with claimed that NSi voters 
were underrepresented in the pre-election voting 
because they are much more disciplined than the 
electorates of other parties (meaning that they are 
more likely to show up on election day--rain or shine). 
Since this is a quality that is difficult if not 
impossible to measure prior to election day itself, the 
pre-election polls generally overrepresented the other 
electorates and underrepresented NSi. 
 
-------------------------- 
The Tail Wagging the Dog? 
-------------------------- 
 
14.  (SBU) The bias inherent within different polling 
methodologies and its effect on the popularity of 
political parties is not unknown to polling agencies. 
Researchers at the Center for Public Opinion and Mass 
Communication told us that without properly weighting a 
sample for education, the results consistently favor 
right-of-center parties.  From these and other 
comments, we believe that the effects of different 
methodologies are well known to those conducting survey 
data analysis in Slovenia. 
 
15. (SBU) Gossip and conspiracy theories abound in 
Slovene political circles.  Polling results are not 
immune.  The chatter that we hear from our contacts, 
particularly among opposition members, is that polling 
is notoriously unreliable.  The result, they say, is 
that the ruling coalition gets a boost from those who 
want to go with the winner.  According to one source, 
the ruling LDS party commissions their own "real" polls 
 
 
for internal use that are cannily accurate, predicting 
the actual July European Parliament election results of 
one party to within half a percentage point. 
 
16.  (SBU) The director of Nina Media also admitted 
that he was personally friendly with many members of 
LDS and that they frequently joked with him about the 
results of his polls.  Although he categorically denied 
that pressure was ever put on him to fudge the results, 
he admitted that most political parties did maintain 
regular contact with his polling agency and that they 
sometimes commissioned their own polls.  He also 
confirmed that most parties are keen to benefit from 
higher polling results.  As he said, "no one wants to 
look unpopular." 
 
-------- 
Comment 
-------- 
 
17.  (SBU) The level of knowledge and expertise among 
professional pollsters in Slovenia is very high. 
However, public opinion polls are extremely easy to 
manipulate by expert pollsters.  Polling agencies that 
compete for contracts have an incentive to maintain a 
reputation for accurate polling, but that does not mean 
that other considerations do not come into play.  In 
Slovenia, polling agencies are generally eager to cut 
costs and conduct polls on the cheap.  Since the more 
accurate methodologies are usually the most expensive-- 
requiring field interviews and persistence in tracking 
down randomly chosen respondents--it is natural for 
agencies to sacrifice accuracy in order to cut costs. 
However, it is also true that if one wanted to bias a 
sample in favor of a particular political constellation- 
-either left-of-center or right-of-center--it would be 
very easy to do so.  In fact, intentional bias of this 
sort could easily be attributed simply to the high cost 
of a more rigorous methodology. 
 
18.  (SBU) While election results generally validate 
the accuracy of a given polling agency, it is not 
impossible for inflated or deflated pre-election polls 
to encourage or dissuade potential voters and thus 
influence the result itself.  Generally, therefore, one 
needs to take all polling results, and claims of 
manipulation, with a grain of salt and be aware of the 
fact that it is always easy to explain inaccuracies 
after an election is over.  Parliamentary elections in 
Slovenia will be held October 3. 
 
HAAS 
 
 
NNNN