Why Statistics Suck

A report has surfaced from an investing firm that Activision-Blizzard’s stock was downgraded from “buy” to “neutral” because of a projected loss of 900k-1.7m subscribers when SW:TOR comes out.

Bear with me as I list the known facts from that report… which I wasn’t able to find on its own, just the articles talking about it which are all based on an original article somewhere.  Here’s a link to one of these articles.

The Facts:

  • 381 online gamers were polled
  • 57% of former WoW players said they reached end-game content and become “bored”
  • 43% of former WoW players had quit due to friends leaving for other games
  • 50% of those currently playing WoW plan to buy SW: The Old Republic, with an additional 38% saying they may buy it
  • 87% of the 95 players currently in the SW:TOR beta had already pre-ordered or planned to pre-order the game
  • 50% of former WoW players would consider coming back to WoW if Blizzard were to offer a special promotion
  • 33% of former WoW players said they plan to resubscribe when Mists of Pandaria launches


The Dumb People

Here are some comments on various posts about those facts:

  • They are basing their percentages on a survey of 381 online gamers. This is a far cry from the 10.3million that actually play WoW
  • So because they polled 381 gamers that automatically means everything in that article is 100% accurate? Seems legit..
  • I’m no WoW fanboy but 381 online users isn’t much of a survey group when comparing it to millions of people.
  • Interviewing 386 out of 10,6 million doesn’t seem to be a fair research”


Tirade Time

Ok, first things first.  Idiots on the internet, 381 people polled is –precisely- the right amount for a survey to have a 95% confidence level with a 5% margin of error in a population of 10 million.  This is the industry standard for polls, btw, the 95 with 5%.  What it means in simple terms (and keep in mind I don’t know anything about this except some common sense deductions) is that:

-the numbers the survey shows are accurate within 5% (so when they say that 50% of players… the real number could be 45% or it could be 55%… within 5%).
-95% of the time these numbers will be accurate.  So if you did this poll 100 times with different groups, you’d get the same numbers (within 5% as a margin of error) 95 times.

Got that?  The numbers are fine.  And surveys/polls are very easy to do.  No matter the size of the population, if you’re going for the industry standard of 95% with a margin of error of plus minus 5%, you never need more than 380ish people.  That’s just how the math works.  If you wanted a smaller margin of error, like 2%, you’d need about 2400 people. That’s all.


So where’s the problem?

Here is my list of problems with this poll and why it doesn’t actually translate into fact (though it may prove to be right for other reasons):

  • 381 online gamers were interviewed.  It doesn’t specify how many were/are wow players, how many base all their online gaming experience on Farmville, etc.  It doesn’t specify how the gamers were polled.  Did a general gaming site have a survey? Did a bunch of people on survey sites get chosen because they had “gaming” selected in their profile?  Did a SW:TOR site advertise the poll? Did a WoW site advertise it?
  • 57% of former wow players said they were bored with the game.  Now we’re getting into WoW numbers.  So out of the 381 gamers surveyed, only those that played wow before but no longer do so matter for this question.  We don’t know what the numbers are, and if they’re less than 381 then they no longer statistically represent the industry standard of surveys.
  • 87% of the 95 players in the SW:TOR beta had already pre-ordered or planned to pre-order the game.  That’s 95 out of how many in total? What’s the sample size vs population there?  And seriously, do you think we’re retarded? “Have already pre-ordered or PLANNED TO pre-order”.  So how many have already pre-ordered? Most likely not a very impressive number because that number would have been used otherwise.  “plan to pre-order” is trying to spin a “maybe” into a sheet with hard facts.
  • 50% of former wow players would come back if a special promotion happened. AND 33% of former wow players WILL come back for Mists of Pandaria.  Again, based on what total numbers?  What are the facts that we should get here? Are the 50 and the 33 pooled together?  Or does it mean that 33% plan to come back for MoP AND that 50% would come back with a special promotion so that WoW’s returning player rate could be seen as 83%?  I don’t believe that’s the case, but that’s the kind of spin they seem to have put on the 87% of SW:TOR players with plans to pre-order the game.

That’s just with the way the data is presented.

Here are overall problems that pissed me off with this survey:

  • The salient points are comparing bored former wow players with sw:tor players currently in the sw:tor beta.  The bored former wow players have a less than stellar opinion on wow, and the sw:tor beta players are in love with sw:tor.  This is fascinating.  Soon this survey will tell me the sky is usually blue and that snow is cold.
  • I’m still pissed about the numbers.  The survey company is trying to make it seem like a legitimate survey that applies to wow because the number of people polled is fine for a population of 10m, but they don’t take those people from that population of 10m, instead ONLY A PORTION of their sample actually played wow, nevermind that only a portion of THAT group still plays.  That’s like conducting a survey of 100 people living in Toronto that are of Polish heritage and using their answers to determine who will win the next elections in Poland.
  • Where in these answers is there any indication that wow players will leave to play other games? Nowhere.
  • Where in this survey is the question : “will you cancel your subscription to play another game?”
  • Where in this survey is the question : “will you play both games at the same time? Or maybe other games?”


That WoW is declining in population isn’t a secret.  There are many reasons, many posts around the blogosphere about it, many industry experts trying to understand it.

This survey, however, is not one of the things that should have a legitimate claim to anything.  It’s a load of crap that says absolutely nothing about anything.


8 Responses to “Why Statistics Suck”

  1. 1 Tania November 27, 2011 at 1:52 pm

    If the survey was of 381 online gamers, unless they were ALL WoW players or former WoW players, the actual sample population of WoW players would be less than 381, which means the confidence interval will be less than 95%. This doesn’t mean those numbers are actually wrong, of course, just that you can’t be as confident of them as you would be if the actual number of people polled was higher.

    I really wish that you were able to find the actual study itself, because it’s hard to know much about what the numbers were or anything like that just from the basic numbers, which mean very little without context.

    Like, it there were, out of the 381 players polled, only 126 WoW or former WoW players, the numbers are significantly less reliable.

  2. 2 Cynwise November 28, 2011 at 3:06 pm

    Thank you. It’s nice to see the methodology of a survey taken apart – surveys can be really powerful tools when done right, but drawing conclusions from numbers requires knowing a lot of details about how it was done, what the sample population was, how the questions were phrased…

    … why yes, I’ve had to do stats analysis in the past, why do you ask? 🙂

  3. 5 Bristal November 28, 2011 at 3:57 pm

    The number needed to reach statistical significance is not a constant proportion of the whole population, it is also dependent upon the predicted versus actual result (one of which is an estimate, the other is essentially unknowable).

    For instance, if I predict that the male:female ratio of 10 million players is 10:1, and the ratio is actually 1:1, it would take a smaller sample size to disprove the theory than if I was trying to prove that the ratio was 2:1.

    It’s also extremely dependent upon approaching true random sampling, which is extremely difficult in questionnaire studies. Typically questionnaires have less than a 30 percent return rate, unless of course they have sponsored incentives, another variable we domt know about in this study.

    • 6 Tania November 28, 2011 at 5:29 pm

      I’m afraid I don’t understand your comment at all. How can you base the number you’re polling on a result that you don’t HAVE yet? I’m taking a statistics course at the moment, and according to all my notes, you don’t base anything except conclusions on results.

      You certainly don’t base your choices on numbers to pull from the population for your sample based on actual results, because that makes no bloody sense at all.

      Or have you just phrased your entire comment really poorly?

  4. 7 Ratshag November 28, 2011 at 4:13 pm

    As it happens, one of them 381 were a friend of mine: Stop of The Stoppable Force. Bugger told Stop he found his blog through Google. Don’t know what exactlies his search terms was, but kinda the whole point of Google is ta *not* give a random selection, but rathers what it thinks you wants fer ta see.

    Now, Stop still plays a bit of WoW, but his passion (and his blog) now be with SW:ToR. Did everyone else this bugger talked to sail in the same canoe? No freakin’ clue. Does I have a lot of faith in this bugger’s random samplin’ and statistifyin’ methodolgies? Nopes.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


  • 237,799 victims
November 2011
« Oct   Dec »

%d bloggers like this: