Wednesday, December 01 2010

On Monday I posted an entry titled Lies, Damn Lies, and surveys, the focus being that surveys — especially online surveys — are often of dubious merit, and worse are often intentionally or ignorantly misinterpreted by the press.

Yet it works wonders for both the press and the marketing research companies: There is a whole industry of lazy tech writers who will run with whatever is sent to them. Any slant they provide merely globs extra vaseline on the lens of accuracy.

In that case I was spurred by some just released results of an online survey done by GfK.

Take a look at the Fortune summary of said survey. As is the case with most other retellings, the Fortune writer seemed to simply rewrite the original Reuters story which itself said:

The survey found that just 25 percent of smartphone owners planned to stay loyal to the operating system running their phone, with loyalty highest among Apple users at 59 percent, and lowest for Microsoft's phone software, at 21 percent. Of users of Research in Motion's BlackBerrys, 35 percent said they would stay loyal.

The figure was 28 percent for users of phones running Google's Android software, and 24 percent for users of Nokia Symbian phones.

The Fortune story includes a big graph with the title "Plan to stick with your smartphone OS?". The iPhone towers at 59%, with Android down at a miserable 28%.

Wow, looks pretty rough for Android!

So I emailed GfK and they kindly responded with the same press release that they had sent out to the press. Unfortunately it still is only a summation, including no notes on methodology or the actual questions asked (which can often be very leading), yet it is far better than most survey companies that simply hide their summary behind a paywall, enjoying the attention as the press mangles it into something more hit magnetic.

Here's the table that was the source for most of these stories. I have to guess (based upon the surrounding wording) that it was the responses to a question asking current smartphone owners what they're going to consider when they next upgrade their phone.

Smartphone Ecosystem
Overall Apple Nokia
(Symbian)
BlackBerry Windows
Mobile
Android
Will stay loyal to smartphone OS 25% 59% 24% 35% 21% 28%
Will stay loyal to smartphone OS
but switch handset make
7% - - - 8% 16%
I will look at all options 56% 36% 60% 58% 61% 49%
Will switch smartphone OS 6% 1% 8% * 5% 2%
Don't know 7% 4% 7% 6% 4% 4%

There's Android at 28% among these single choice options. That's the meat of a lot of easy "news" stories.

Note the second line, though: "Will stay loyal to smartphone OS but switch handset make". Sum those two lines and for Android suddenly you're at 44%. Sure, maybe someone with an HTC Evo 4G is looking longingly at a Samsung Galaxy S, or maybe they're imagining getting that Android-powered Playstation phone when it comes out, but we (meaning virtually every single press reference to this survey) are talking about the OS here.

So Reuters (and any that followed Reuter's lead) couldn't manage to achieve a grade-school reading ability. I'm serious: my daughter had table reading assignments in grade 1 that were just like this, a job that many in the tech press would fail.

Extraordinary.

But it gets even better. Among those who answered "I will look at all options" (which ideally should be 100% of respondants), here are what each current platform's users are considering as their considerations when they upgrade at some point in the future (multiple choices obviously being allowed).

Current Smartphone Owners
Overall Apple Blackberry Nokia
(Symbian)
Windows
Mobile
Android
Apple iOS 53% 85% 46% 47% 43% 38%
Blackberry OS 33% 22% 74% 34% 41% 22%
Symbian 23% 14% 18% 40% 18% 13%
Windows Phone 7 41% 28% 40% 47% 65% 31%
Android 51% 40% 41% 48% 42% 84%

It paints a rather different narrative than the common, egregiously wrong interpretation. You end up with hilarious ignorance like this garbage. With what you have learned here, check out the chart here and see if you can figure out what's wrong.

The iPhone still takes the lead (with loyalty oddly dramatically higher in Germany — the home country of the survey organization — than in any other nation), but look back at the original Reuters statement: "The survey found that just 25 percent of smartphone owners planned to stay loyal to the operating system running their phone...the figure was 28 percent for users of phones running Google's Android software".

I suppose that sounds better than the more accurate "44% of Android users have pledged undying allegience to the operating system, regardless of the endless and unpredictable changes among competing choices. Of the remainder, the vast majority still plan on hopping back on the Android train regardless.

   

Reader Comments

Kudos on getting your hands on the GfK survey results -- which were not, as you suggest, widely mailed out to the press. And kudos on raising a daughter who at age six can do better than the "lazy press." You must be very proud.
Philip Elmer-DeWitt @ 12/1/2010 10:52:16 AM
I've revised the piece based on the data you got from GfK. Here's the link:

http://tech.fortune.cnn.com/2010/11/29/survey-only-32-percent-of-smartphone-owners-is-loyal-to-his-or-her-brand/
Philip Elmer-DeWitt @ 12/1/2010 11:49:49 AM
Nice debunk!

I am a bit of a sucker for good, old fashioned, investigative journalism. The modern news stories often feel like someone re-hashing everyone else - no one thinks to look and analyze source material. I remember back in school when the only authoritative resource was the original source material for any papers I wrote. Now, you often see serious journalists (and bloggers tend to be even more like this though they can be forgiven since they are not generally paid to report news) writing stories and using *other news organizations* as the source! This seems absurd to me...

In any case, I also took this as a personal reminder to *always* check the source when reading something I find personally important, or when re-hashing something on a future blog post.

Thanks, and I hope to see more of the same!
Charlie B @ 12/2/2010 1:33:41 PM

Add Comment

Name *:

Email Address:

(your email address is not displayed)
Website:

Comment *:



About the Author
Dennis Forbes Dennis Forbes is a Toronto-based software architect. While focused primarily on the .NET and SQL Server worlds, Dennis frequently ventures outside of this comfort zone into game development and image processing. He has been published in several industry magazines, has been quoted in the Wall Street Journal and has been interviewed by NPR.

He is a vice president and lead software architect at an innovative New York City hedge fund back-office services firm.

Dennis has been working on solutions for the financial, telecommunications, and power generation markets for over 15 years.





 

Dennis Forbes