workload simulation (was: Re: [MUD-Dev] MMORPG/MMOG Server design)

[ceo]

Damion Schubert wrote:
> From: ceo

>> I believe you're talking about three problems:

>>   1. simulating a "heavily loaded" system (to run continuously in
>>   parallel with all other simulations)
 
>> Number one is, well, either "very hard" or "pretty easy". If you
 
> From a designer's point of view: in the past, checking basic 'load
> balancing' usually meant logging on a gazillion faux clients who
> ...

Indeed; your examples help flesh out the "if you need to test
anything that depends on the realism of the connections", and they
provide interesting reading.

Our approach has largely been to use mathematical modelling to
predict performance, rather than stress-testing. Stress-testing
shows you how the system will behave given the exact traffic-pattern
and client-behaviour that you simulate; it is not generally a good
basis to extrapolate from. It is typically an observational form of
modelling, and doesn't take account of fundamental causes and
pressures. By contrast, in theory, a mathematical model is
inherently predictive.

If your stress-testing fails, it shows you some performance problems
you need to fix - but which might be irrelevant in a live system. If
it succeeds, it tells you absolutely nothing beyond a very basic
indication that your software is at least sometimes, in some
situations, able to handle that load :(. This, of course, is the
same situation as with all testing; but many projects tend to have
much greate similarities between normal testing (unit, system, and
regression tests etc) and actual usage than they do between
stress-testing and actual usage. As Damion says, "usually meant
logging on a gazillion faux clients who would move and talk";
assuming this behaviour was purely random, it bears little
relationship to actual usage and incited behaviour of the system
(again, as described eloquently by Damion).

Of course, GIGO (Garbage in, Garbage out). If the equations in your
model suck...But you can base the equations on the source-code, and
so guarantee that they do take account of the totality of at least
the programmed portion of the system. Where most of the uncertainty
comes in is that you don't (generally) have access to similar
details of the hardware, the OS, or the internet. So your models for
those three are often where the uncertainty gets introduced, and all
three can be very unpredictable.

Wouldn't it be wonderful [conveniently laying aside all the new
hardships it would introduce ;) ] if all hardware and software
vendors provided decent mathematical models for their systems as
part of the documentation? Since this is approximately the same
amount of information as the actual source code, I'd settle for just
three particular perspectives: Processing- , Storage- and Bandwidth-
complexity equations...

Well, actually, if I'm going to postulate a "perfect" scenario, I'd
probably demand quite a few more. It's useless to 95% of users, but
it could enable a whole new world of inter-operating software. Pity
it would double, treble (or much worse) the development time for all
projects...

Adam M


_______________________________________________
MUD-Dev mailing list
MUD-Dev at kanga.nu
https://www.kanga.nu/lists/listinfo/mud-dev