[MUD-Dev] Quality Testing

[Dave Rickey]

-----Original Message-----
From: Michael Tresca <talien at toast.net>

> This leaves us with the "let the players test it" approach, which
> I'm all for with Beta tests.  But how effective are betas in
> cleaning the game up enough for release?

  http://camelot.allakhazam.com/Polls/Poll_Index.html

What is your initial impression of the game?

 Choice                        Votes   Approximate %
 It rocks!                      444     49%
 Very Enjoyable                 363     40%
 It's OK                         78      9%
 I'm not really that impressed   14      2%
 It sucks                         5      1%  

Non-scientific, but it fits fairly well with the proportion of our
beta testers that signed up for the special 3-month
pre-subscription.  I'd say the beta did a fairly good job of getting
the game ready for release.

Since the MPBT: Solaris beta in 96, I've been claiming from the
other side of the fence that online game developers were horribly
under-utilizing the potential of their beta-tests.  I'm not pointing
to this as a brag, but as a case-in-point to show that it's time to
rethink the QA methods used for these games.  Traditional QA was
batting 1 for 3 (AC having been the only OLRPG prior to Camelot with
a smooth launch, and some dev's have admitted that many basic
balance issues were never addressed there).  Betas are much more
than a chance to do hardware-compatibility testing on the cheap and
build up buzz.

> I'm wondering if this doesn't warrant a different sales approach,
> where you offer a game at half price or free for the first five
> months to "get it out there," thus satisfying the release schedule
> requirements, but at the same time, level-set player expectations
> so they know the game isn't perfect.

Half-price *is* free, actually free for that long would mean
bankruptcy.  EQ, AO, and WW2O did all wind up extending their "free
month", not starting the clock until the worst of the launch issues
had been solved.

> We've seen over and over the "rush to market" and the surprisingly
> high level of tolerance consumers show for a game's flaws.  Then
> Anarchy Online came along...

AO and WW2O hit the market like a 1-2 punch, *everyone* expected
Camelot's launch to be a mess.  Almost every time I saw Camelot
being discussed on the boards of other games before the last month
of beta, it was in the context of "Maybe a few weeks after they
launch, if the game lasts that long and doesn't suck."  The players
were, by and large, incredibly gunshy and a bad few days at launch
could have set us back months.

Of course, a patcher that corrupts your system registry is kind of
the Mother Of All Bugs, but those launches went so bad it was like
watching a train-wreck in slow motion.  Add in "There, but for the
grace of God, go I," and the recent history really had us worried
around here.  Last wednesday's crash was almost a relief, we could
stop anticipating trouble and actually *fix* something.  And the
other shoe never dropped, there was the one crash and then
everything just *worked*.

There is a "honeymoon period", when the game is shiny and new and
people haven't yet run up against the sucky bits.  Camelot's still
in that phase, the real test is going to be seeing our retention
rates.  The only votes that count start with dollar signs.

Players are cutting us a lot of slack for our shortcomings on
content, because what parts are there *work*.  They're cutting us a
lot of slack for our 6+ hour /appeal backlog, because not too many
things are happening to stop their gameplay in its tracks until a
CSR can straighten it out.  This is my second trip through this
process, and the difference is night and day.  We're only about a
week post-launch, and our average hold time for phone support is 4
*minutes*, when EQ launched our average ran more like 2 hours for
the first month, and 90%+ dropped out of the qeue.  This in spite of
the horrible backlog for in-game support, the things that are going
wrong in-game aren't bad enough to provoke someone to make a
long-distance call.

Again, I'm not bragging.  I'm pointing out that we did *not* get
"lucky" .  The only part of the successful launch I'd ascribe to
luck was that we were able to hire Spyke Alexander, who lived
through the nightmare that was EQ launch along with me, and knew how
to set up our network to avoid repeating it as far as our network
went.

Our beta testers told us what was broken, we fixed it.  They told us
what they didn't like, we changed it.  They whined, we ignored it,
they provided data and they got results.  Several tens of thousands
of iterations of that, all with minimal overhead, and the result was
a successful launch.  If we'd tried to document every single step of
that, we'd all be filling out paperwork for the next 3 years.

--Dave Rickey

_______________________________________________
MUD-Dev mailing list
MUD-Dev at kanga.nu
https://www.kanga.nu/lists/listinfo/mud-dev