First I tried just calling modify from Net::LDAP. That worked but just set the new password as plain text. My passwords appear to be “md5crypt”, and normally look like this:

{CRYPT}$1$fywXcrPC$Uakrx8POGBf1WM9l6mkG6/

I could come up with some code to create the correct hash on the machine where I was running the Perl script, but I really wanted the LDAP server to do it for me, for consistency.

A little bit of searching revealed Net::LDAP::Extension::SetPassword, so I gave that a go. Well, that was progress, but it set MD5 passwords. They look like this:

{MD5}13dmpYRmooMYt50wdZBpSQ==

Why did it just decide to use MD5? The answer’s in the OpenLDAP FAQ. password-hash was indeed set to {md5} on the server.

Right, OK, so set password-hash to {md5crypt} then? No! It does not accept that value. It does accept {crypt}, but that ends up like:

{CRYPT}Q.nfbCdTMBuGU

It seems to have the right hash type ({CRYPT}) but it’s much shorter. It’s the POSIX crypt(3) based on DES. Not quite what I wanted.

The eventual answer was found in the archives of the openldap-software mailing list from almost 8 years ago! So once slapd.conf contained:

password-hash  {CRYPT}
password-crypt-salt-format "$1$%.8s"

the correct password hash was generated.

How did I know about the “.8” bit? In an md5crypt hash, the characters between the $1$ and the next $ are the salt, and there’s 8 of them, so that’s why .8s.

Yesterday morning Graham was visiting the datacenter and kindly agreed to replace the disks for me. As it happens I don’t have that much experience of software RAID since the production machines I’ve worked on tend to have hardware RAID and the home ones tend not to be hot swap. It didn’t go entirely smoothly, but I think it was my fault.

The server chassis doesn’t support drive identification (e.g. by turning a light on) so I had to generate some disk activity so that Graham could see which drive lights weren’t blinking. It was easy enough for him to spot that slots 0 and 1 were still blinking away with slots 2 and 3 dead. I checked /proc/mdstat to ensure that those disks weren’t still present in any of the arrays. If they had been then I would have done:

$ sudo mdadm --fail /dev/mdX /dev/sdbX

to remove each one.

They weren’t present, so I gave Graham the go-ahead to pull the hot swap drive trays out.

At first the server didn’t notice anything. I thought this was bad as I would like it to notice! This was confirmed to be bad when all disk IO blocked and the load went through the roof.

I think what I had forgotten to do was to remove the devices from the SCSI subsystem as described in this article. So for me, it would have been something like:

$ for disk in sd{a,b,c,d}; do echo -n "$disk: "; ls -d /sys/block/$disk/device/scsi_device*; done
sda: /sys/block/sda/device/scsi_device:0:0:0:0
sdb: /sys/block/sdb/device/scsi_device:0:0:1:0
sdc: /sys/block/sdc/device/scsi_device:1:0:0:0
sdd: /sys/block/sdd/device/scsi_device:1:0:1:0

From /proc/mdstat I knew it was sdb and sdd that were broken. I think I should have done:

$ sudo sh -c 'echo "scsi remove-single-device" 0 0 1 0 > /proc/scsi/scsi'
$ sudo sh -c 'echo "scsi remove-single-device" 1 0 1 0 > /proc/scsi/scsi'

Anyway, at the time what I had was a largely unresponsive server. I used Magic Sysrq to sync, mount filesystems read-only and then reboot. In Cernio’s console this would normally be “~b” to send a break, but Xen uses “ctrl-o“. So that was ctrl-o s to sync, ctrl-o u to remount read-only and then ctrl-o b to reboot the system.

Graham had by then taken the dead disks out of the caddies and replaced with new, re-inserted them and powered the server back on.

Happily it did come back up fine, I then had to set about adding the new disks to the arrays.

I’d already been forewarned that the new disks had 488397168 sectors whereas the existing ones had 490234752 — both described as 250GB of course! A difference of some 890MiB, despite them both being from the same manufacturer, from the same range even. I didn’t bother adding a swap partition on the two new disks which made them just about big enough for everything else.

$ sudo mdadm --add /dev/md1 /dev/sdb1
mdadm: Cannot open /dev/sdb1: Device or resource busy

Oh dear!

After lengthy googling, this article gave me a clue.

$ sudo mulitpath -l
SATA_WDC_WD2500SD-01WD-WCAL72844661dm-1 ATA,WDC WD2500SD-01K
[size=233G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:1:0 sdd 8:48  [active][undef]
SATA_WDC_WD2500SD-01WD-WCAL72802716dm-0 ATA,WDC WD2500SD-01K
[size=233G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 0:0:1:0 sdb 8:16  [active][undef]

There’s my disks!

Stopping multipath daemon didn’t help. Running multipath -F did help.

The usual

$ sudo mdadm --add /dev/md1 /dev/sdb1
$ sudo mdadm --add /dev/md1 /dev/sdd1
$ sudo mdadm --add /dev/md3 /dev/sdb3
$ sudo mdadm --add /dev/md3 /dev/sdd3
$ sudo mdadm --add /dev/md5 /dev/sdb5
$ sudo mdadm --add /dev/md5 /dev/sdd5

worked fine after that.

I hope that was useful to someone. I’ll be practising it some more on some spare hardware here to see if the fiddling with /proc/scsi/scsi really does work.

Update:

Dominic (author of the linked article about dm-multipath) says:

I think there’s also a “remove” or “delete” file you can echo to in the /sys device directory, bit more friendly than talking to /proc/scsi/scsi.

and provides this snippet for multipath.conf which should disable multipath:

# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
blacklist {
        devnode "*"
}

Out of nowhere just now:

Her: Erm. Erm. If you’ve separated all the wood stuff how are you going to separate the rest?

Me: What wood stuff?

Her: From my bit.

(voice manages to convey mild irritation at my lack of understanding)

Me: Your bit of what?

Her: Thing!

(Sleep-Jenny clearly losing patience)

Me: Okay then. We’ll work it out.

Her: Good.

(the world has been set to rights)

In the morning I shall endeavour to find out what her bit is and what apart from wood needs to be separated from it.

The actual lighting works fine, it’s just that if the lighting did fail then there’d be no emergency lights directing the shallow end of the gene pool to safety.

So the staff close the place up as soon as it starts to get a bit dusky out.

I’ve seen a few people comment that they didn’t think that a non-technical person would understand what they were all about, so I took the opportunity to ask my friend about it.

“See that ad over there? What do you think it’s for?”

“It looks like it’s for a search thing. A new kind of web search thing. That’s a big list of related things to what they searched for”

“What do you think Google Chrome is then?”

“Well it says on it, a new browser. That’s what you use to search isn’t it?”

“What other browsers are there then?”

“Well there’s the Yahoo! one, and then there’s the Google one.”

So to the extent that she even noticed the ad, she assumed it was just something to do with Google’s search engine, because of the search implications and prior meaning of the brand “Google”.

I don’t think this proves anything, but it was interesting hearing another point of view.

What other kinds of manual labour do people enjoy? It’s pretty much just smashing things up for me I think.

After having read through it, I suppose my first thought is to wonder why anyone who knows that this sort of thing goes on would ever go there. Why any rich person from a democratic country would move there to set up home. Why any Western sporting personalities or pop stars would put their name to Dubai resorts and hotels, to stadiums and their gigs. Coldplay, Bon Jovi, Justin Timberlake, Shakira, Christina Aguilera and Elton John aren’t really acts you associate with slavery, after all.

The urge to take the moral high ground and denounce the practice is strong, and rightly so. Something nags at me though about how much of our society must be built on injustice. The clothes and food we buy when we’re being economical, how much of it comes from oppressed workers? Do we just turn a blind eye all the time?

The people interviewed by Johann Hari in the article either avoid the subject or say they force themselves not to think about it, so a lot of them too are just turning a blind eye. I like to think that if I knew some product was the result of sweat shop labour (let alone slavery) then I would avoid it, but am I kidding myself?

When we could afford it we shopped at Waitrose/Ocado not just because it tasted better but because it’s got to be ethically superior, right? But as soon as we needed to live on a tighter budget this went out of the window and these days we mostly shop at Asda. Most of the bad publicity for supermarkets in the UK seems to be reserved for Tesco but Asda is even cheaper and I can’t really believe their practices are that much better. They have whole cooked chickens for £4, for goodness sake. I could stop shopping there but then that would be less money to spend on other things which would reduce the quality of my life, so this makes me a massive selfish hypocrite.

Of course none of this is comparable to the blatantly fucked Dubai society; if Asda were taking passports off of Filipino girls and forcing them to sell cheap goods for 19 hours a day for no money then certainly I’d consider paying more than 5p per 100ml of Pepsi Max at a competing supermarket.

It’s a bit of a coincidence that we’re hearing all this about Dubai now though, just as they enter the news for allowing their state bank to default massively. This stuff has been going on for years, right? The millions of slaves used to build that place. I confess I never really thought about it before. The worst I’d previously heard was about their insane laws on drugs which could get you locked up for minuscule traces on the outside of your shoes, or for stuff you bought over the counter in the airport.

We all turned a blind eye for years.

If you read this far and for some reason want to comment then it would be great if your comment was not solely based on what you do to consume more ethically than everyone else on the Internet.

At first I thought it was simply because when I had moved his VPS I had taken the opportunity to reconfigure it to the new way I was setting them up, which meant that his root file system would be mounted from /dev/xvda instead of /dev/xvda1 (or /dev/sda1). That would have accounted for it if his monitoring tool had been doing it by device name, but it turned out that it was more fundamental than that — neither mount nor df were showing his root file system at all!

This was highly confusing at first. /proc/mounts looked correct and anyway how does a machine boot if it doesn’t know where its root file system is?

The answer to that question was a bit of a clue really: the boot loader tells the kernel what device the root file system is on, and in this case it was doing it by UUID. The UUID in the boot loader configuration was not the same as the UUID listed in /etc/fstab. I had forgotten to update the customer’s /etc/fstab.

The machine was able to boot because the boot loader was correctly configured, but then after it had already mounted the root file system it was trying to mount everything in /etc/fstab and failing on a line for a UUID that wasn’t present. That line then never made it to /etc/mtab which is what mount and df are reading from.

After correcting the /etc/fstab, it is fixable without a reboot by just mounting / again over the top of the existing one. Or you could probably just edit /etc/mtab.

gnutls26 (2.4.2-6+lenny2) stable-security; urgency=high
  * Non-maintainer upload by the Security Team.
  * Fixed CVE-2009-2730: a vulnerability related to NUL bytes in
    X.509 certificate name fields. (Closes: #541439)
    GNUTLS-SA-2009-4

 -- Giuseppe Iuculano <iuculano@debian.org> Sun, 01 Nov 2009 21:29:06
+0100

and then found that your applications began failing to connect to your LDAP server, you may want to check that your SSL certificate is valid. Along with this update it seems that the default behaviour changed to being more strict. In my case I was using self-signed SSL certificates without the CA being available.

You can disable the verification if you don’t want it by adding:

TLS_REQCERT     never

in /etc/ldap/ldap.conf on each client machine.

• Humax PVR started displaying everything in a shade of pink when set to RGB output (like it always has been).
• One of my LCD panels had its display totally corrupted with a strange tartan effect for about a day, and then went back to working perfectly again without me doing anything.
• My hair clippers just stopped working, half way through using them.
• My electric toothbrush wouldn’t turn off for a few minutes, then it stopped and won’t turn back on again now.
• A brand new wall wart for a wifi access point died after 3 weeks.
• One of the ports on my desktop switch is permanently lit up while nothing is plugged in. When you plug something in the lights go out.