Server Clustering
From Mugshot Developer Wiki
Clustering is a project to increase the server's scalability by taking advantage of multiple server computers.
Work is presently focused on distributing web load, and on the code changes necessary to make the code correctly function on multiple computers. This work is taking place on the cluster branch in Subversion.
Contents |
Setting up a development instance
For testing of the clustered sofware, you don't really want to need a rack full of machines. So this section describes how to set up a virtual cluster of machines, with the software in each machine running on a different user.
Prerequisites
You'll need to pick a machine to deploy to; this can be a remote machine, or it can be your devel machine. Even if it's a local devel server, deployment will be done over ssh. Make sure:
- The home directory of your user on the deployment machine is world readable
$ chmod ~ 0755
- You can ssh to the deployment machine without typing a password; use ssh-keygen if you don't already have a public key, and copy the resulting ~/.ssh/id_rsa.pub into the ~/.ssh/authorized_keys file on the deployment machine. Test to make sure that you have the permissions on ~/.ssh correct.
Pick and configure network addresses
Each server instance needs a separate network address; you can do this by creating IP aliases, which allow a network adapter to bind to multiple addresses. Things are easiest if you can get addresses statically assigned on your network. Notes on the configuration for the 3 Lan Drive office network can be found at here (private link) (we just set up a block of addresses for each developer.)
Once you know which network addresses you'll be using, you need to create IP aliases to bind to each of the addresses you'll be using. You can do this manually with:
$ ifconfig eth0:1 192.168.1.11
And so forth, or you can use the script bind-addresses.sh in the tools/ directory of the mugshot sources. That will prompt you for a base address and number of addresses to bind, and bind them if they aren't already bound.
$ ./bind-addresses.sh How many addresses do you want to bind (3)? What is the network address base (192.168.1.30)? Bound 192.168.1.30 to eth0:0 Bound 192.168.1.31 to eth0:1 Bound 192.168.1.32 to eth0:2
You'll need to copy the bind-addresses.sh script to the machine you'll be deploying on if that's different from your devel server. The script also assumes that you have sudo set up.
Create and setup user accounts
The next step is to set up an account for each instance. You'll need to:
- Create a group that the accounts will share
- Create a user account for each instance
- Set up .ssh/authorized_keys so you can ssh to each instance
- Create ~/dhdeploy directories with the right permissions
- Create appropriate ~/.super.conf files
This is all scripted by the setup-fake-cluster.sh script, also in the tools directory. Run this on the deployment machine.
$ ./setup-fake-cluster.sh How many test instances do you want to create (3)? What is the network address base (192.168.1.31)?
Group: otaylor-test
Cluster config file: /home/otaylor/.super-cluster.conf
User: otaylor-test1
Slave: false
Home: /home/otaylor-test1
Network address: 192.168.1.31
User: otaylor-test2
Slave: true
Home: /home/otaylor-test2
Network address: 192.168.1.32
User: otaylor-test3
Slave: true
Home: /home/otaylor-test3
Network address: 192.168.1.33
Look good? (y)? y
The script is designed to be safe to run multiple times, if you want to change the network address, add additional cluster members, or whatever.
Configure clusterwide parameters
The last step will have created a file in your home directory with contents like:
<superconf>
<parameter name="serverHost">localinstance.mugshot.org</parameter>
</superconf>
Edit the serverHost line to the name by which you'll be accessing the first (master) server in the cluster, for example
<parameter name="serverHost">otaylor-test1.dumbhippo.com</parameter>
You may also want to add additional parameters settings here like adminConsole and disableAuthentication.
Deploying with ant
To deploy to your test instances, you can use the 'ant deploy' target in the toplevel mugshot ant file. You'll first need to create a file ~/.dumbhippo.properties with a line like:
deploy.hosts=otaylor-test1@myhost.mydomain,otaylor-test2@myhost.mydomain,otaylor-test3@myhost.mydomain
Change the username and the host name to match your username and the host you are deploying on. Then you can deploy to those servers with a simple
$ ant deploy
That:
- Builds a dar file
- Stops all the servers
- Deploys the dar file to all servers
- Starts all the servers
If you want to shut down everything (say, you are sharing your deployment test server with other people), you can do it with:
$ ant deploy-stop
Setting up Apache/mod_jk load balancing
First, make sure you have mod_jk installed. JPackage http://jpackage.org is a good way to install it for RHEL/Fedora systems. We have our own FC5 RPM http://devel.mugshot.org/download/extra/mod_jk-ap20-1.2.15-1jpp.i386.rpm] since JPackage doesn't have FC5 RPMs yet.
Sample /etc/httpd/conf.d/mugshot.conf:
<VirtualHost *:80> ServerName localinstance.mugshot.org JkMount /* jboss </VirtualHost>
Sample /etc/httpd/workers.properties:
worker.list=jboss,status worker.node1.port=8009 worker.node1.host=192.168.1.30 worker.node1.type=ajp13 worker.node2.port=8009 worker.node2.host=192.168.1.31 worker.node2.type=ajp13 worker.jboss.type=lb worker.jboss.balance_workers=node1,node2 worker.status.type=status
Be sure you change the IP addresses to match the ones you're using.
Setting up XMPP load balancing
We currently use a small custom server to load balance clients among available XMPP servers. The protocol for this is somewhat standardized, although not generally supported by most clients.
Enabling the balancer requires adding two things to your ~/.super.conf on your "master" server. First, you must specify the set of all cluster nodes:
<parameter name="clusterHosts">192.168.1.78 192.168.1.79</parameter>
Second, enable the balancer service:
<service name="balancer" enabled="yes"> <!-- <parameter name="balancerBind">192.168.1.80</parameter> --> </service>
You can uncomment the balancerBind parameter to specify a specific IP address to which the balancer should bind. This is useful if you are debugging all of this on one machine and want to run cluster nodes and the balancer together.
