(Yes it’s been over a month since my last blog entry, and that’s been mainly because of how insanely busy things have been at work. I dont think things are going to improve much in the rest of 2006.)

Recently my Xen hosting hobby business actually started to make slightly more per month than it costs me to operate. Of course it has been running at a loss for over a year now, so is still nowhere near breaking even, but it does make it reasonably cheap as a hobby and meant I could afford to look to expansion plans.

Even though the first server, curacao, had been filled to capacity, my original plans had been to wait until early 2007 to try and break even properly before spending something in the region of £1,000 to £1,500 on another 1U server. That was until Chris mentioned that he was getting rid of all his x86 1U servers and upgrading to 64-bit. This provided me a good opportunity to grab a second hand one at a nice discount, that would still be plenty good enough for my needs.

The server I ended up with is now named islay (pronounced “eye-lah”, by the way!) and when I got it it only had 1GiB of RAM and two 80GiB SATA disks. The RAM turned out to be rather more expensive than I expected as the motherboard required ECC registered DDR2 costing about £100/GiB, so I had to compromise and only buy two of those to bring it up to 3GiB total. I had hoped to put 4GiB in which would be as far as I intended to take it, meaning no reboots for later RAM upgrades, but it’s not to be; I will have to get some customers on it to recoup that investment and buy another 2x1GiB later on when it’s required. The two disks were replaced by two Seagate 320GB models in a RAID-1.

One thing that didn’t need upgrading was the CPU, being a 3.6GHz Xeon. curacao is only a 3GHz Pentium 4, but it also has hardware RAID and 4 disks so it is debatable which one is going to provide greater real-world performance. My colo provider, Jump, does charge extra for every 0.3A of power beyond the first 0.3A so I was a little concerned about just how much islay was going to cost me, with Xeons being known as rather power hungry.

Happily after plugging it into a power meter I was surprised to find that while running idle it only uses about 120W (about 0.55A), rising to about 174W (~0.75A) when 100% busy compiling a kernel. curacao was measured at about 0.8A so the cost to me will be similar; I can only assume the disks count for a lot.

The new server sat in our lounge for about a month being experimented on by me. After a while Phil and I got used to the enormous racket the fans make! I needed this time to do a nice clean install of Xen 3.x — curacao is running 2.x which is woefully outdated now and a little buggy, but upgrading it in place is far too scary a prospect. Finally last Thursday (28th September) we took it in to Telehouse to meet James and install it. Well, I went straight from work while Phil drove in with the server. Apart from hellish traffic that made Phil about 45 minutes late (sorry again James!) it all went pretty smoothly and islay‘s now in the same rack as curacao. When we got home the house seemed eerily quiet!

It was only afterwards that I realised I had completely forgotten to take any photos of the server.

This weekend, the first after installation, is the first time I’ve had to sit down and move some real Xen domains over to islay and so far things seem to be going well. I have 4 domains that are either my own or are internal administrative domains, and so far 3 of them (including the one that runs this blog and all my other web sites) have been moved to the new server seamlessly.

The domains keep their MAC address and IP address, so the only things that actually have to be moved are the filesystems. I’m not (yet!) doing well enough to afford a SAN or any other form of shared storage, so filestems for Xen domains are exported from LVM logical volumes in dom0. To move a domain to another server I need to create a matching LV on that server and then somehow transfer all the data across to it. My current method is a bit like this:

• Create a new LV in dom0 of islay, and mount it.
• Log in to running domain on curacao and do an rsync over ssh into where the destination LV is mounted in dom0 on islay.
• Create the necessary Xen config file and iptables rules on islay.
• Shut down the running domain on curacao.
• Mount the domain’s old LV in dom0 on curacao and do a final rsync to capture any files that changed or got deleted by the shutdown.
• Unmount the new LV in dom0 of islay.
• Start the new domain on islay.

As you can see, this is a little laborious and results in a couple of minutes of downtime between when the old domain is shut down and the new one is started. I will also run into problems when I come to do this for customer domains as well, because I don’t generally have even a user account in customer’s virtual servers, let alone root access. I’m not sure how I can improve on this process though.

Xen’s own migration feature isn’t going to help here because that only transfers a running domain’s memory image to the target server where it expects to find the same block devices to export to the domain — I’d still need shared storage, a SAN or something.

Xen also has a save and restore feature, kind of like suspend to disk, but I don’t know if the image file can be moved to another host. Also bear in mind that the CPU architectures are different (P4 vs Xeon). Also that only moves the memory image again, so the disk storage would still need to be there ready for when the domain is set running again.

Anyway, my plan is to have all the non-customer domains run on the new server for a week or so just to be sure that it’s stable and then I’ll be contacting each customer one by one to move them over to it. Once they’re all running on islay I’ll be spending a day in Telehouse reinstalling curacao completely with a matching Xen 3.x installation and then I’ll spread the domains back out between the servers depending on how well each server performs. curacao is currently more loaded than I would like so this does mean that for a short period of time islay is likely to be overloaded also, but that’s kind of unavoidable.

After I’m fully up and running with two servers I think I’ll be reverting to my original plan of steady growth until I have really broken even, but I’m much happier to be able to do so with multiple servers as it will mean I can take on additional customers and be a bit more flexible.