There’s been much debate in the NZ blogosphere about whether the opinion polls conducted here suffer because of the growing number of households without landline telephones, especially in the poorer areas. We don’t have the data to determine how much this is a problem in NZ, but a recent US survey sheds some light on this question.
Pew Research’s latest national survey which included 503 respondents on cell phones out of a sample of 2,004 adults, finds that “the overall estimate of voter presidential preference is modestly affected by whether or not the cell phone respondents are included.”
Specifically, “Barack Obama holds a 48% to 40% lead in the sample that includes cell phones, and a 46% to 41% advantage in the landline sample. Estimates of congressional vote are the same in the landline and combined samples.” (The survey was conducted 18-29 June.) This is a similar effect to that found in other recent surveys, which they cite.
They also point out that, “Despite the fact that cell-only respondents are often very different from those reached by landline, the relatively small impact from including cell phone samples is a consequence of the statistical weighting applied to surveys as a standard practice among professional pollsters.” This applies here as well.
In the US, nearly one out of every six households (15.8%) did not have a landline telephone, but did have at least one wireless telephone in the last 6 months of 2007. I’m having trouble tracking down the NZ data. The Household Use of Information and Communication Technology (ICT) Survey doesn’t seem to have the figures.
Tags: polls
Jafapete's Weblog 
July 25, 2008 at 5:49 pm |
JP
Glad to see you picked up on this, my own post here focused more on the voter engagement issue, though touching on the cell phone issue.
It is interesting that I have not seen any other references to the Pew survey, especially given the commentary here recently when Mike Williams made some comments
http://adamsmith.wordpress.com/2008/07/14/3144/
Adam
July 25, 2008 at 5:53 pm |
[…] Jafapete, a left of centre blogger, has commented more specifically on the cell phone aspects in this post at his blog today 25 July […]
July 25, 2008 at 6:34 pm |
You might perhaps want to note though, that of the 503 cell phone responses, I think only 162 had no land line, so a modest potential impact
July 26, 2008 at 12:56 pm |
(from Adam’s blog, apologies if this is not de rigueur in blogworld!) Adam has wondered recently just how engaged the NZ electorate is as regards the forthcoming NZ election, as opposed to putative voting intentions.
Same here Adam: I’ve asked on the Standard whether anyone could confirm or deny the reputed 70% “refusal rate” from those asked to participate in recent landline polls – to no avail.
If the 70% is accurate, it indicates a huge disengaged cohort – even before considering the cellphone exclusion factor. The much-trumpetted (and celebrated by the Right) 15% or so “gap” between Labour and National is actually based on a sample from a mere 30% of voters (and thus represents an “actual” gap of only around 4%) – the intentions and opinions of the vast majority remaining unknown but indicating a huge “undecided” factor up for grabs in the time remaining.
I’d be very interested to know a); whether the 70% figure is accurate, and b); how it compares to previous elections.
July 26, 2008 at 1:20 pm |
Ak,
I can help shed a little light on this. Refusal rates have been rising fairly fast in recent years, for a number of reasons: work/life imbalance, the increasing amount of market research being done, numbers of immigrants without English skills… A 30% response rate would be fairly good in 2008, and I doubt any of the major polls get that much.
I ran (with a colleague) a telephone survey a few years ago and we did exceedingly well with a 50% response rate amoungst those who we could tell were eligible (i.e. not including those who simply hung up or couldn’t speak English). Even with a relatively very high response rate there was a fair amount of response bias towards the higher educated, palangi, semi-professional/technical and women. A composite would be your typical nurse-manager. Pacifica people were particularly poorly represented, to the point where we couldn’t include them as a separate category in higher order analysis.
But weighting by industry and occupation took most of that bias out. Most, if not all, of the major polls reported in NZ are weighted as a matter of course, and probably pre-stratified with regional or racial quotas as well. I got a call just two nights ago from one of them, but they only wanted to speak to PIs, perhaps for this reason.
The Pew people in the quote above seem to be implying that weighting takes care of a lot of the bias resulting from the absence of those in the population withour landlines.
July 26, 2008 at 1:49 pm |
Thanks Pete – I’d clean forgotten about weighting (despite having a piece of paper somewhere stating that I studied all this in minute detail aeons ago!)
Still can’t help thinking that the current polls are surreal though – and that voter volatility seems to have increased in recent years. Guess it might have a lot to do with the fact that National has accepted most of Labour’s policies.
And it’s curious that in such a seemingly sophisticated field, Colmar-Brunton polls appear to consistently over-state National support compared to other polls and actual election results – some slight difference in methodology perhaps?
July 26, 2008 at 11:17 pm |
It is my view that the polls may reflect voting intentions, possible ones, at the time of polling – but this is not necessarily a valid indicator of how things will pan out, especially as we will not get a full turnout of all eligible voters.
Re your comment on Colmar-Brunton it would be interesting to know whether the wording of their questions varies in any way from that of the others and/or the time of day they do their polling.
On a lighter note, and maybe not entirely out of bounds see:-
http://adamsmith.wordpress.com/2008/07/21/3434/
September 24, 2008 at 1:01 pm |
[…] on cell-phone skew I posted in July about the Pew Research Center’s findings that non-inclusion of cell phone respondents […]