Month: October 2006

Hosting update (and thanks Gentoo)

Posted on by Andy

This post a bit delayed but as usual I’ve been rather busy.

curacao in the new rack

I spent last Saturday (21st) in Telehouse reinstalling curacao to make it similar to the new server, islay. I’d already moved everyone over to islay so I could do this and take as long as I wanted doing it. One slight snag however was that I knew there would be no crash carts, keyboards or monitors there for me to use. I’ve got no problem using serial, it’s all set up for that from the BIOS, but I’d not installed it with only access to serial before as it had originally been installed at home. Adding to the complexity is the fact that I wanted Debian Sarge on it, but the Sarge installer doesn’t support the 3ware RAID card.

It wasn’t really a big deal because I have installed Debian via debootstrap from a live cd more times than I can count, but it had reached 3am Saturday morning and I still hadn’t found a live cd environment that could totally work over serial! Eventually I decided to try the Gentoo 2006.1 live cd. This didn’t work straight off but I was able to mount it loopback and edit isolinux.cfg to make it usable over serial:

serial 0 9600
default gentoo-serial
timeout 150
prompt 1
display boot.msg
F1 kernels.msg
F2 F2.msg
F3 F3.msg
F4 F4.msg
F5 F5.msg
F6 F6.msg
F7 F7.msg

label gentoo
  kernel gentoo
  append root=/dev/ram0 init=/linuxrc dokeymap looptype=squashfs loop=/image.squashfs cdroot initrd=gentoo.igz vga=791 splash=silent,theme:livecd-2006.1 CONSOLE=/dev/tty1 quiet

label gentoo-nofb
  kernel gentoo
  append root=/dev/ram0 init=/linuxrc dokeymap looptype=squashfs loop=/image.squashfs cdroot initrd=gentoo.igz

label gentoo-serial
  kernel gentoo
  append root=/dev/ram0 init=/linuxrc dokeymap nox dolvm2 looptype=squashfs loop=/image.squashfs cdroot initrd=gentoo.igz console=ttyS0

On the day, it turned out I spent more time helping Andrew Whalley find his server! He’d forgotten which room in Telehouse it had been installed in, and we went through three of them checking every rack unit carefully before we located it.

The Gentoo live cd worked beautifully when told to not boot into X, to load LVM2 support and work over serial. I had installed debootstrap, created new logical volumes and copied over the setup from islay in no time. It’s things like that which make me glad that there are so many different competing Linux distributions all trying new things and stopping each other from becoming complacent. So thanks Gentoo for existing and making it easier for me to install Debian! In future now I am starting to collect machines though it may be time to look into netbooting them.

The rest of the Jump rack

Now a week on, curacao is fully in service again and has been hosting four of my own internal Xen domains as a form of load testing, but I’m now happy enough with it to start putting customers on it. islay is already basically full so all new ones will be going on curacao. There’s about 2.3GiB of available RAM for this; I can increase it by another 1GiB or so but I don’t think it will be necessary.

The Jump rack is looking almost full now; I guess my next UK server won’t be in that one…

I’m also still working out with Matt the details of a possible expansion into the US market.

Good luck Whiskey!

Posted on by Andy

I received a letter from The Retired Greyhound Trust today to let me know that Ringo, the greyhound that I have been sponsoring, has been happily homed in Wales. I make a small monthly contribution to the RGT to help them with the cost of caring for and rehoming abandoned greyhounds. It’s not as rewarding as owning a greyhound myself of course but it’s the best I can do at the present time, and it’s great to hear about “my” greyhound a few times per year. I know there must be several people sponsoring each dog, but I don’t care about that; they need all the help they can get.

WhiskeyNow that Ringo has been homed, I get to sponsor another dog. RGT have chosen Whiskey (racing name: Beat Them Java!) for me and I’m very happy with their choice. He reminds me a lot of my first greyhound pet, Tom. Tom didn’t like other dogs much either. Whiskey’s been at the RGT kennels for four years and realistically because of his lack of tolerance for other animals he will probably stay there for the rest of his life, unless he can be united with someone who has the time and patience to properly care for him.

TomIt’s a real shame because there is no inherent reason why a greyhound cannot get along with other dogs, or even other animals in general. Whiskey will have ended up this way because for the first four years of his life he will have been kept in a cage with no experience of the world other than humans taking him to races. As a result he most likely is bewildered by anything that’s alive and not human-shaped — it’s prey or competition. If someone had been willing to put in the minimum effort then Whiskey could have spent the last four years as a loyal companion and family pet, but instead he’s lucky he hasn’t been euthanised or worse.

Greyhound Data lists Beat Them Java’s pedigree and racing record which is interesting reading — his best time was 365 metres in 28.8 seconds; that’s over 36 miles per hour. It’s sad that the site only exists because of the money involved in the cruel sport of greyhound racing, but then, that’s the only reason why greyhounds exist today also. Still I wish I had known about it when I had Tom.

rsnapshot?

Posted on by Andy

aquarius, did you know about rsnapshot before doing this? It seems to do pretty much everything you want. I’ve been using it for several years, and currently back up two local and twelve remote machines ever four hours. I’m very happy with it.

Russell Brand apology

Posted on by Andy

Recently I’ve been slagging off Russell Brand, because the association with Big Brother and the constant appearance in the gutter press really turns me off.

It still does turn me off and I do think he’s in danger of everyone becoming sick of it, much like nobody cares to hear about Pete Doherty anymore, but I have to concede that I had formulated my overall opinion without actually giving a fair assessment of his other work.

To give him a chance I had a few listens to his show on 6 music of a Sunday late morning, and I must admit that I find it very funny. Sometimes laugh-out-loud funny.

They edit it up a bit and put it out as a weekly podcast which I’ve been listening to on my way to work, and I know I must make this apology because it’s given me that “oh god I am laughing on the train and people are looking at me” problem at least once with every episode so far.

So if like me you thought there’s no way you could find this annoying reject from Adam and the Ants amusing, give him a second chance and have a listen to his podcast.

Studio 60 on the Sunset Strip

Posted on by Andy

Please allow me to be probably the millionth person to rave about this.

Studio 60 on the Sunset Strip is the new Aaron Sorkin-produced TV drama now onto its third episode in the US, but from the first pilot show I could feel the classic magic of The West Wing coming back and I know I’ll become hopelessly addicted to this. The show was created by Sorkin and is produced by Sorkin and Thomas Schlamme — a previous producer and director of many West Wing episodes. Many West Wing acting alumni are also present in the cast list including Bradley Whitford who is probably my all-time favourite from that show. This is going to do no harm at all to Matthew Perry‘s career, either.

The concept is the show-within-a-show: The fictional Studio 60 on the Sunset Strip is a popular US network TV comedy sketch show, clearly meant to be very similar to Saturday Night Live. Like many Sorkin stories, the pilot episode appears to start with a dramatic ending. I hate spoilers so you’re just going to have to watch it.

If you’re in the US, watch it. If you’re not and your morals aren’t offended by the suggestion then download it. Otherwise, wait impatiently for it!

MBNA you really should know better

Posted on by Andy

I have just received an official email communication from MBNA which contains the following text:

For your additional peace of mind, every customer e-mail sent by MBNA Europe Bank Limited, including those on behalf of its Credit Card Partners, will include your postcode:

AB123CD

(Obviously I have removed my real postcode there)

In what universe is a postcode considered a secure piece of information? This does nothing for my peace of mind since my postcode is not hard to obtain by anyone who feels like doing so. You do not need to be MBNA to have my postcode. There will be some people out there though who don’t think it through and now when they receive an email saying it is from MBNA (or a Credit Card Partner! And let me tell you I am on the edge of my seat over the kinds of Partners I could soon be receiving email from!) quoting their postcode they will feel just that little bit more convinced that it’s real.

If a service provider wants to alert me to something or pass on some information that they consider it worth proving their identity over, then either they need to use proper digitally-signed email (yeah right) or at least just tell me to log in to my account in the usual way (no need for a link!) and view the message there. Which is what my bank, smile does, and what Egg Money does, and what Paypal does. But not MBNA; they must be new to the credit industry!

And while I am all worked up over this, what is with the email coming from mbnaapply.co.uk!? Were the domains mbna.co.uk and mbna.com not enough? Oh and a third one to do the actual online banking through? Get a clue. You need precisely one domain name; you can use http and https versions of this, and by this means build up a consistent brand and identity which really does go some way to promoting peace of mind. All these stupid extra domains represent the people with good sense at your company being forced to bend over and take it from the muppets known as “The Marketing Dept.”

This kind of stupidity is a great way to encourage me to use your plastic as a balance transfer tool and nothing more.

HantsLUG EGM

Posted on by Andy

I’m at the Hampshire Linux User Group Extraordinary General Meeting in Southampton today. It’s an EGM because the “proper” annual general meeting a month or two ago didn’t get enough people to be quorate.

I don’t usually physically attend Hants LUG meetings given they take place (naturally) in Hampshire and I live in Feltham, near Heathrow! This time though there was an item on the agenda to be voted on which I felt rather strongly about, so I thought I should make the effort to help reach quorum and get the amendment passed.

The amendment in question was regarding opening the mailing list archives. Traditionally, Hants LUG has had a “closed” list archive in that only members of the mailing list can view the archive. This meant that no archiving by major search engines could take place, and therefore none of the pearls of wisdom posted to the list were being published outside the confines of the LUG. When this topic was brought up on the list it was rather contentious with a clear divide of opinion — some people just didn’t like the idea of their posts to the LUG’s mailing list being publically visible and archived.

Graham Bleach had stepped forward to propose an amendment to the LUG’s constitution that would see the mailing list archives opened up, and it’s one I was eager to support. Happily the amendment was passed something like 23 to 3.

Tony Whitmore had also taken the time to write up several other amendments related to who can vote, when meetings will take place, etc., which were all sound ideas I was happy to support. I am rather concerned that this will pave the way to Internet voting for Hants LUG which is something I am vehemently opposed to; I can quite easily see the LUG dominated by people like myself — keyboard heroes who are quite happy to pontificate digitally but rarely get off their arse to actually do anything in the local Linux community. These sort of movers and shakers are exactly what the LUG depends on, they’re the people in shortest supply, and they are the people who will be disenfranchised by an indifferent voting population. Time will tell.

While here we’ve also had a very productive chat about what we’re going to do about the hardware for lug.org.uk and I hope we’ll see some movement on that front in the very near future.

An election took place for the officers of the LUG, the main news being that Adrian Bridgett is the new chairman. It’s certainly not a role I envy but I’m sure he’ll take it in his stride!

I think we’re going for a curry later. \o/

Xen hosting expands

Posted on by Andy

(Yes it’s been over a month since my last blog entry, and that’s been mainly because of how insanely busy things have been at work. I dont think things are going to improve much in the rest of 2006.)

Recently my Xen hosting hobby business actually started to make slightly more per month than it costs me to operate. Of course it has been running at a loss for over a year now, so is still nowhere near breaking even, but it does make it reasonably cheap as a hobby and meant I could afford to look to expansion plans.

Even though the first server, curacao, had been filled to capacity, my original plans had been to wait until early 2007 to try and break even properly before spending something in the region of £1,000 to £1,500 on another 1U server. That was until Chris mentioned that he was getting rid of all his x86 1U servers and upgrading to 64-bit. This provided me a good opportunity to grab a second hand one at a nice discount, that would still be plenty good enough for my needs.

The server I ended up with is now named islay (pronounced “eye-lah”, by the way!) and when I got it it only had 1GiB of RAM and two 80GiB SATA disks. The RAM turned out to be rather more expensive than I expected as the motherboard required ECC registered DDR2 costing about £100/GiB, so I had to compromise and only buy two of those to bring it up to 3GiB total. I had hoped to put 4GiB in which would be as far as I intended to take it, meaning no reboots for later RAM upgrades, but it’s not to be; I will have to get some customers on it to recoup that investment and buy another 2x1GiB later on when it’s required. The two disks were replaced by two Seagate 320GB models in a RAID-1.

One thing that didn’t need upgrading was the CPU, being a 3.6GHz Xeon. curacao is only a 3GHz Pentium 4, but it also has hardware RAID and 4 disks so it is debatable which one is going to provide greater real-world performance. My colo provider, Jump, does charge extra for every 0.3A of power beyond the first 0.3A so I was a little concerned about just how much islay was going to cost me, with Xeons being known as rather power hungry.

Happily after plugging it into a power meter I was surprised to find that while running idle it only uses about 120W (about 0.55A), rising to about 174W (~0.75A) when 100% busy compiling a kernel. curacao was measured at about 0.8A so the cost to me will be similar; I can only assume the disks count for a lot.

The new server sat in our lounge for about a month being experimented on by me. After a while Phil and I got used to the enormous racket the fans make! I needed this time to do a nice clean install of Xen 3.x — curacao is running 2.x which is woefully outdated now and a little buggy, but upgrading it in place is far too scary a prospect. Finally last Thursday (28th September) we took it in to Telehouse to meet James and install it. Well, I went straight from work while Phil drove in with the server. Apart from hellish traffic that made Phil about 45 minutes late (sorry again James!) it all went pretty smoothly and islay‘s now in the same rack as curacao. When we got home the house seemed eerily quiet!

It was only afterwards that I realised I had completely forgotten to take any photos of the server.

This weekend, the first after installation, is the first time I’ve had to sit down and move some real Xen domains over to islay and so far things seem to be going well. I have 4 domains that are either my own or are internal administrative domains, and so far 3 of them (including the one that runs this blog and all my other web sites) have been moved to the new server seamlessly.

The domains keep their MAC address and IP address, so the only things that actually have to be moved are the filesystems. I’m not (yet!) doing well enough to afford a SAN or any other form of shared storage, so filestems for Xen domains are exported from LVM logical volumes in dom0. To move a domain to another server I need to create a matching LV on that server and then somehow transfer all the data across to it. My current method is a bit like this:

  • Create a new LV in dom0 of islay, and mount it.
  • Log in to running domain on curacao and do an rsync over ssh into where the destination LV is mounted in dom0 on islay.
  • Create the necessary Xen config file and iptables rules on islay.
  • Shut down the running domain on curacao.
  • Mount the domain’s old LV in dom0 on curacao and do a final rsync to capture any files that changed or got deleted by the shutdown.
  • Unmount the new LV in dom0 of islay.
  • Start the new domain on islay.

As you can see, this is a little laborious and results in a couple of minutes of downtime between when the old domain is shut down and the new one is started. I will also run into problems when I come to do this for customer domains as well, because I don’t generally have even a user account in customer’s virtual servers, let alone root access. I’m not sure how I can improve on this process though.

Xen’s own migration feature isn’t going to help here because that only transfers a running domain’s memory image to the target server where it expects to find the same block devices to export to the domain — I’d still need shared storage, a SAN or something.

Xen also has a save and restore feature, kind of like suspend to disk, but I don’t know if the image file can be moved to another host. Also bear in mind that the CPU architectures are different (P4 vs Xeon). Also that only moves the memory image again, so the disk storage would still need to be there ready for when the domain is set running again.

Anyway, my plan is to have all the non-customer domains run on the new server for a week or so just to be sure that it’s stable and then I’ll be contacting each customer one by one to move them over to it. Once they’re all running on islay I’ll be spending a day in Telehouse reinstalling curacao completely with a matching Xen 3.x installation and then I’ll spread the domains back out between the servers depending on how well each server performs. curacao is currently more loaded than I would like so this does mean that for a short period of time islay is likely to be overloaded also, but that’s kind of unavoidable.

After I’m fully up and running with two servers I think I’ll be reverting to my original plan of steady growth until I have really broken even, but I’m much happier to be able to do so with multiple servers as it will mean I can take on additional customers and be a bit more flexible.