Statistics 101 for MMO Designers

First off, let me just say that I love when Georg Zoeller answers posts on the TOR forums.  He is not above using sarcasm and insults to get his point across.  I’ve argued here before that I’m all for that.  Its pretty much a given when dealing with people of the..uh..caliber…that you will find posting on official forums.

So when Georg responded to the criticism that they were dumbing down the game by removing the option to kill off companions, I liked his style.  And by further side note, let me say that my opinion on that is that – its a waste of time.  Bioware gave ample warning that they had no interest in building anything sandbox-y…err, sorry, “Uncle Owen’s Moisture Farm”-y.    So the fact that they removed a dynamic choice element from their game should not come as a shock to anyone.

But what I do have interest in is his math:

As for blaming those ‘testers you selected’ – they are a statistically representative sample of the SWTOR community.

Now,  here is a little primer on sample size and statistics for you.  But let me give you the tl;dr version:

To be that confident about the sampling,  the current set of Alpha testers in TOR to be statistically representative sample of the SWTOR community, they would have to number in excess of 10,000.


I’ll let that sink in for a moment.

And I’ll let you chew on the fact that Alpha testing generally incorporates only around one to two thousand people, based on what I’ve heard, read, and experienced first hand.

So what’s the bottom line here?  The bottom line is that either there are a lot more people currently testing this game than anyone thought was possible, or Bioware is overconfident.  I guess a third posibility is that they are outright lying, but I’d like to give them the benefit of the doubt, if for no other reason than they have in past been pretty okay with telling people – “this is how we are going to do it, and we don’t give a flying flip what you think.”