Server Test Suite

From Mugshot Developer Wiki

The 'hungry' module in Subversion has a framework for running black-box tests against a live server.

To understand these instructions you'll need to be familiar with Server Development Setup and Super.

Setting Up a Hungry Instance

While there are some things you can do with hungry against an existing instance of the server, most of the useful tests require modifying a database, so the best way to use hungry is to create an instance of Mugshot separate from the one that you use for normal development and debugging.

Or, to drive home the reason for this: hungry likes to begin by dropping all tables in the database.

First, create a file ~/.hungry.conf with the following contents. This is an alternate configuration for super; we'll repoint super to look in this location instead of the default ~/.super.conf.

<superconf>
   <!-- Location to deploy to -->
   <parameter name="basedir">$home/hungry</parameter>
   <!-- The hungry tests need to be able to create their own users -->
   <parameter name="disableAuthentication">true</parameter>
   <!-- Don't send out any email to our test addresses -->
   <parameter name="disableEmail">true</parameter>
   <!-- Relocate HTTP and the rest of the ports -->
   <parameter name="tomcatHttpPort">31080</parameter>
   <parameter name="baseport">31000</parameter>
   <!-- With the above configuration, it is then safe to turn on destructive tests -->
   <service name="hungry">
       <parameter name="enableDestructive">true</parameter>
   </service>
</superconf>

We now need a command alias to run super pointed to this configuration. Add the following line to your ~/.bashrc:

alias hsuper="~/workspace/trunk/super/super --config=$HOME/.hungry.conf"

Then start a new terminal, or reload the aliases for the current terminal:

$ . ~/.bashrc

And finally, create a ~/hungry directory for deployment (this replaces ~/dhdeploy):

$ mkdir ~/hungry

Running Hungry Tests

While it's safe to run both your normal instance of the services and the instance for hungry at the same time with the above configuration, unless you have several GB of ram, you probably don't want actually do so. So, if you are running a local instance of the server for debugging, stop it in the normal way:

$ ./super stop all

Now start the instance for hungry

$ hsuper start all

To access the tests, start the super 'console' for hungry:

$ hsuper console hungry

That will give you a list of available tests:

[1] Run read-only tests
[2] Run destructive tests
[q] Quit

The read-only tests only work when there is data already in the database, so you probably want to run the destructive tests at this point; type '2' and hit return.

This should take a minute or so to run, and give you a lot of debug spew. If the tests completes succesfully, you'll see something like:

   [junit] Testcase: testPage took 0.027 sec
   [junit] Testcase: testPage took 12.52 sec
   [junit] Testcase: testPage took 1.534 sec
   [junit] Testcase: testInviteMany took 2.863 sec
   [junit] Testcase: testAcceptAllInvitations took 9.728 sec
   [junit] Testcase: testPage took 7.866 sec
   [junit] Testcase: testShareLinkMany took 25.055 sec

BUILD SUCCESSFUL

If things go wrong, the server logs can be found in ~/hungry/jboss/log/. When you are done working with hungry, stop the hungry instance:

$ hsuper stop all

Performance

To run the performance tests, set up hungry as above, and first create a test set by selecting from the hungry console

[3] Create performance data

This creates a set of users with contacts, groups, and link shares. Then to run the performance tests, select

[4] Run performance tests

You probably want to do this a couple of times to make sure that you are getting somewhat consistent results. The output for one test run for one page looks something like:

 Threads   Iters    Wall Iters/s  Thread Iters/s Latency
 ------- ------- ------- ------- ------- ------- -------
       1       1   0.348   2.874   0.347   2.882   0.347
       1      10   3.054   3.274   3.053   3.275   0.305
       3      10   5.979   5.018   5.901   5.084   0.590
       5      10   9.933   5.034   9.677   5.167   0.968

The columns here are:

Threads

Number of threads run in parallel. Each thread simulates a different logged in user, so increasing the number of threads gives a simulation of a bunch of users using the site at once.

Iters

Number of iterations for each thread. Every iteration in a thread is from the same user, which is a little unrealistic, since a single user is unlikely to continually make requests.

Wall

Total wall clock time. This gives a slight overestimate of the time it takes to process all Threads * Iters page loads, since it includes overhead for starting up the threads, and because towards the beginning and end of the test the beneficial effects of parallelism are reduced.

Wall - iter/s

Number of pages a second we can handle based on the Wall time (Throughput = (N_ITERATIONS * N_THREADS) / WALL_TIME)

Thread

Average time to process all the requests for a single thread. Using this to estimate throughput gives a slight underestimate, since the operations of all the threads aren't totally in parallel.

Thread - iter/s

Number of pages a second we can handle based on the Thread time. (Throughput = (N_ITERATIONS * N_THREADS) / AVERAGE_THREAD_TIME)

Latency

Average turn-around time for a request, based on Thread time. (Latency = AVERAGE_THREAD_TIME / N_ITERATIONS)

So, we see as as more threads access the page in parallel, the number of accesses goes up but the time to process a request increases significantly. The difference between running 1 access to the page and 10 in a row is most likely simply that we have less overhead and more accuracy with a bigger sample size.