As a consequence though, I tend to find that many (sometimes the majority) of posts are uninteresting. Clearly if they are all uninteresting then I need to just unsubscribe, but I’m on plenty of lists that do come up with gems from time to time. I find Mutt, the text-based (console) email client to be really helpful at quickly getting through these mailing lists without missing too many interesting things, and I thought I’d share some ways I do that.

These are primarily simple tips for dealing with other people’s sub-optimal mailing list behaviour. In some cases it’s the poster who’s clearly in the wrong, but asking people to give a toss about those reading their words of wisdom is apparently considered offensive in many places, and doesn’t actually modify behaviour.

First off, Mutt is kind of a culture shock for most people. This post is only really for people who already use Mutt, or maybe who were already considering using it. I’m not even going to try convincing the typical gmail web user to switch. Or anyone really. Depending on how you configure it, Mutt looks a bit like this:

(Click for higher-resolution version)

If that freaks you out, it may be best to stop reading, move along. Run, don’t walk.

Context is useful

Like, I suspect, most Mutt users, I don’t really know most of its features. It’s pretty complicated to configure. When other Mutt users glance at my email they quite commonly ask, “how did you split the window like that!?” I didn’t, it’s not split, it’s just a different layout of message window. Here’s how:

set pager_index_lines=5

Just handy for being able to see a bit of the context of where the current post is in regards to the rest of the thread.

Threading matters

Once a thread has gone bad, it’s usually going to stay bad. The most useful tricks involve operating on whole threads at once, so you don’t have to tediously click something on every email. So while it might not seem like the most annoying thing at first, people not threading properly becomes one of the more annoying things later as it slows down whatever you are trying to do on the thread.

Sort your folders by thread:

set sort=threads

Unfortunately Mutt doesn’t seem to have a feature to break a thread when the subject header changes, so you might instead prefer to sort by subject. That had too many false positives for me though, even with sort_re.

Get rid of a whole thread

If you’re looking at the start of a thread, and it’s uninteresting, and you can see all of the thread below it, chances are that it’s all going to be uninteresting. You can mark it all read with ctrl-r. I prefer to see the whole thread on the screen before doing that, because there’s some chance that someone might change the subject line into something that becomes interesting.

I find myself reading some lists mostly with ctrl-r without even looking at the content of the posts. For example, a thread that starts with “Mandriva v Windows” isn’t very likely to contain anything except trolling and counter-trolling (If you are unaware of what Mandriva is, you are reading the wrong blog post and only need know that it is a brand name for a popular USB personal massager product).

The risk is that someone will go off at a tangent and post something that is actually interesting, without changing the subject. I’m willing to take the risk; they should have changed the subject IMHO. And anyway, you still have the mail, it’s only been marked as read.

Deal with subthreads

If you can see that the subject of a thread has changed or there’s some other reason why you might want to treat every message below the current one differently, then you can operate on subthreads.

The most simple thing is to break the subthread off into a new thread of its own. # will do that. You can then treat it differently, use the thread commands on it in isolation. That’s how I usually do it because if this happened then it means that the content of the subthread is very different to the rest of the thread, to me.

If you want to keep it as part of the same thread, you can mark the subthread read with esc-r (or alt-r).

Also useful for when someone decides that the right way to start a new email is to just press reply on some other random email.

Tagging

Occasionally a bunch of posts are the same sort of thing but they’re not in a thread. If you can find something about them that’s common then you can tag them based on that, with T. e.g. T followed by ~h @luser.example.com tags every email that has “@luser.example.com” in its headers.

If you can’t think of anything to match on then at the very least, just hitting t on each of the posts will tag it.

Once tagged, you’ll find that many existing Mutt commands that operate on a single email will work on a bunch of tagged emails as well, by prefixing the command with ;. So, if you imagine you’d tagged the above emails and wanted to mark them read, the next thing you’d do would be ;N.

Rejoin broken threading

Some people continue to use broken email clients that don’t do threading properly. All of their posts appear in a new thread. You can easily rejoin errant posts into an existing thread by tagging them, moving to the post they should be replies of, and using the & command. It may seem like a lot of hassle, but the benefit is that every reply to that one will then be in the right place too.

Useful for those pointless flamewars that just won’t die. “Oh look it’s that thread again, I’ll just mark it read again.”

More info

That’s about all I can think of in terms of the simple stuff I do every day when reading email. I hope it helped some newcomers to Mutt. There’s a lot of great tips in the documentation but it can seem a bit impenetrable at first:

• Reading Mail – the Index and Pager

Ingredients

I used the following recipes as a basis, reducing to serve 2:

• Mushrooms stuffed with sun-dried tomatoes
• Dauphinoise potatoes
• Aubergine rolls with pesto
• Tiramisu

I couldn’t find aubergines anywhere (tried 3 big supermarkets) so had to settle for baby aubergines which of course weren’t big enough to wrap anything in.

Method

Mushrooms stuffed with sun-dried tomatoes

• Preheat oven to 200C.
• Soak about 8g sun-dried tomatoes in a small bowl of hot water, covered for 5 minutes.
• Reserve a tablespoon of the liquid, drain the rest off and chop the tomatoes finely.
• Chop off the mushroom stems and chop them finely.
• Finely chop 1/8th cup of shallots.
• Finely chop a clove of garlic.
• Lightly beat a large egg yolk.
• Mince 1/8th cup of parsley leaves.
• Crumble 1/4 teaspoon of basil.
• Heat 2 tablespoons of olive oil in a frying pan over moderate heat until hot but not steaming. Add the mushroom stems and shallots, stirring until shallots are softened.
• In a bowl stir together the mushroom stems, shallots, 1/6th cup bread crumbs, sun-dried tomatoes, the reserved liquid, the egg yolk, parsely, basil and garlic. Add salt to taste. Mound into the mushroom caps.
• Brush the mushroom caps with sun-dried tomato oil.
• Arrange the cups in a lightly greased shallow baking dish.
• Sprinkle the caps with 2 tablespoons of grated parmesan.
• Bake in the middle of the oven for 15 minutes.

Aubergine rolls with pesto

• Preheat oven to 180C.
• Cut 1 small carrot into matchsticks.
• Deseed and slice half a red pepper.
• Trim 8 asparagus spears.
• Chop half a clove of garlic.
• Finely grate 40g parmesan.
• Top and tail 2 aubergines. Slice aubergines lengthways into about 8 4-5mm thick strips. Add salt and pepper, brush with extra-virgin olive oil and set aside for 5 minutes. Since I only had baby aubergines I could only slice them into small thin chips at this point.
• Brush a pan with olive oil, place over high heat, cook aubergines for 2 minutes each side. Set aside.
• Blanch vegetables in boiling water:
  • Carrots: 3 minutes
  • Pepper: 2 minutes
  • Asparagus: 1 minute

  Drain, pat dry and set aside.

• Coarsely blend the garlic, 50g of drained sun-dried tomatoes and 1 tablespoon of pine nuts.
• Add 10g fresh basil, the parmesan, and 65ml extra-virgin olive oil and blend again. Season and stir in 1 tablespoon of double cream.
• Lay the vegetables at the end of each aubergine slice, roll up to secure and place in a large baking dish. Drizzle with extra-virgin olive oil and bake for 6-7 minutes or until hot.
• Serve with pesto, garnish with lemon and salad leaves.

Dauphinoise potatoes

• Preheat oven to 190C.
• Thinly slice half an onion.
• Slice 500g of King Edward potatoes thinly with a mandolin.
• Grease up a shallow baking dish with butter (I used Pure soy spread).
• Mix ~140ml double cream and ~40ml milk.
• Layer the potatoes and onions evenly in the dish. Pour the cream and milk over, dot over with butter and cover the dish with foil.
• Bake for one hour.
• Discard the foil and bake for a further 15-20 minutes or until potatoes are golden.

Tiramisu

• Sift ~22g icing sugar into ~112g mascarpone. Add 1/2 teaspoon of vanilla extract and 2 tablespoons of Tia Maria.
• In a separate bowl, whisk ~85ml of double cream until soft peaks form.
• Mix the cream and mascarpone together and refrigerate.
• Break some sponge fingers to size and briefly dip them in cold, strong coffee.
• Place the sponge fingers in the bottom of serving glasses and spoon mascarpone cream over.
• Grate chocolate on top before serving.

Conclusion

We were pretty pleased with how it turned out, very yummy! Things I’d change:

Presentation could have been better. The vegetables on a bed of aubergines should have been aubergine wraps, but I couldn’t find big enough aubergines. Easy enough to fix. The Dauphinoise potatoes would have looked a lot better if they’d been nice equal sizes and shapes but I only had a grater to slice them with and it was the first time I’d tried. I might have done better with a proper mandolin, and/or more practice.

The Dauphinoise potatoes were rather too creamy for my liking, but Jenny really liked them. Maybe not so much a change needed as simply smaller portions for me; not a huge fan of potatoes.

There was probably a bit too much parmesan in everything, especially the stuffing for the mushrooms. That could have done with being toned down a bit.

No complaints about the Tiramisu!

First I tried just calling modify from Net::LDAP. That worked but just set the new password as plain text. My passwords appear to be “md5crypt”, and normally look like this:

{CRYPT}$1$fywXcrPC$Uakrx8POGBf1WM9l6mkG6/

I could come up with some code to create the correct hash on the machine where I was running the Perl script, but I really wanted the LDAP server to do it for me, for consistency.

A little bit of searching revealed Net::LDAP::Extension::SetPassword, so I gave that a go. Well, that was progress, but it set MD5 passwords. They look like this:

{MD5}13dmpYRmooMYt50wdZBpSQ==

Why did it just decide to use MD5? The answer’s in the OpenLDAP FAQ. password-hash was indeed set to {md5} on the server.

Right, OK, so set password-hash to {md5crypt} then? No! It does not accept that value. It does accept {crypt}, but that ends up like:

{CRYPT}Q.nfbCdTMBuGU

It seems to have the right hash type ({CRYPT}) but it’s much shorter. It’s the POSIX crypt(3) based on DES. Not quite what I wanted.

The eventual answer was found in the archives of the openldap-software mailing list from almost 8 years ago! So once slapd.conf contained:

password-hash  {CRYPT}
password-crypt-salt-format "$1$%.8s"

the correct password hash was generated.

How did I know about the “.8” bit? In an md5crypt hash, the characters between the $1$ and the next $ are the salt, and there’s 8 of them, so that’s why .8s.

Yesterday morning Graham was visiting the datacenter and kindly agreed to replace the disks for me. As it happens I don’t have that much experience of software RAID since the production machines I’ve worked on tend to have hardware RAID and the home ones tend not to be hot swap. It didn’t go entirely smoothly, but I think it was my fault.

The server chassis doesn’t support drive identification (e.g. by turning a light on) so I had to generate some disk activity so that Graham could see which drive lights weren’t blinking. It was easy enough for him to spot that slots 0 and 1 were still blinking away with slots 2 and 3 dead. I checked /proc/mdstat to ensure that those disks weren’t still present in any of the arrays. If they had been then I would have done:

$ sudo mdadm --fail /dev/mdX /dev/sdbX

to remove each one.

They weren’t present, so I gave Graham the go-ahead to pull the hot swap drive trays out.

At first the server didn’t notice anything. I thought this was bad as I would like it to notice! This was confirmed to be bad when all disk IO blocked and the load went through the roof.

I think what I had forgotten to do was to remove the devices from the SCSI subsystem as described in this article. So for me, it would have been something like:

$ for disk in sd{a,b,c,d}; do echo -n "$disk: "; ls -d /sys/block/$disk/device/scsi_device*; done
sda: /sys/block/sda/device/scsi_device:0:0:0:0
sdb: /sys/block/sdb/device/scsi_device:0:0:1:0
sdc: /sys/block/sdc/device/scsi_device:1:0:0:0
sdd: /sys/block/sdd/device/scsi_device:1:0:1:0

From /proc/mdstat I knew it was sdb and sdd that were broken. I think I should have done:

$ sudo sh -c 'echo "scsi remove-single-device" 0 0 1 0 > /proc/scsi/scsi'
$ sudo sh -c 'echo "scsi remove-single-device" 1 0 1 0 > /proc/scsi/scsi'

Anyway, at the time what I had was a largely unresponsive server. I used Magic Sysrq to sync, mount filesystems read-only and then reboot. In Cernio’s console this would normally be “~b” to send a break, but Xen uses “ctrl-o“. So that was ctrl-o s to sync, ctrl-o u to remount read-only and then ctrl-o b to reboot the system.

Graham had by then taken the dead disks out of the caddies and replaced with new, re-inserted them and powered the server back on.

Happily it did come back up fine, I then had to set about adding the new disks to the arrays.

I’d already been forewarned that the new disks had 488397168 sectors whereas the existing ones had 490234752 — both described as 250GB of course! A difference of some 890MiB, despite them both being from the same manufacturer, from the same range even. I didn’t bother adding a swap partition on the two new disks which made them just about big enough for everything else.

$ sudo mdadm --add /dev/md1 /dev/sdb1
mdadm: Cannot open /dev/sdb1: Device or resource busy

Oh dear!

After lengthy googling, this article gave me a clue.

$ sudo mulitpath -l
SATA_WDC_WD2500SD-01WD-WCAL72844661dm-1 ATA,WDC WD2500SD-01K
[size=233G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:1:0 sdd 8:48  [active][undef]
SATA_WDC_WD2500SD-01WD-WCAL72802716dm-0 ATA,WDC WD2500SD-01K
[size=233G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 0:0:1:0 sdb 8:16  [active][undef]

There’s my disks!

Stopping multipath daemon didn’t help. Running multipath -F did help.

The usual

$ sudo mdadm --add /dev/md1 /dev/sdb1
$ sudo mdadm --add /dev/md1 /dev/sdd1
$ sudo mdadm --add /dev/md3 /dev/sdb3
$ sudo mdadm --add /dev/md3 /dev/sdd3
$ sudo mdadm --add /dev/md5 /dev/sdb5
$ sudo mdadm --add /dev/md5 /dev/sdd5

worked fine after that.

I hope that was useful to someone. I’ll be practising it some more on some spare hardware here to see if the fiddling with /proc/scsi/scsi really does work.

Update:

Dominic (author of the linked article about dm-multipath) says:

I think there’s also a “remove” or “delete” file you can echo to in the /sys device directory, bit more friendly than talking to /proc/scsi/scsi.

and provides this snippet for multipath.conf which should disable multipath:

# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
blacklist {
        devnode "*"
}

Out of nowhere just now:

Her: Erm. Erm. If you’ve separated all the wood stuff how are you going to separate the rest?

Me: What wood stuff?

Her: From my bit.

(voice manages to convey mild irritation at my lack of understanding)

Me: Your bit of what?

Her: Thing!

(Sleep-Jenny clearly losing patience)

Me: Okay then. We’ll work it out.

Her: Good.

(the world has been set to rights)

In the morning I shall endeavour to find out what her bit is and what apart from wood needs to be separated from it.

The actual lighting works fine, it’s just that if the lighting did fail then there’d be no emergency lights directing the shallow end of the gene pool to safety.

So the staff close the place up as soon as it starts to get a bit dusky out.

I’ve seen a few people comment that they didn’t think that a non-technical person would understand what they were all about, so I took the opportunity to ask my friend about it.

“See that ad over there? What do you think it’s for?”

“It looks like it’s for a search thing. A new kind of web search thing. That’s a big list of related things to what they searched for”

“What do you think Google Chrome is then?”

“Well it says on it, a new browser. That’s what you use to search isn’t it?”

“What other browsers are there then?”

“Well there’s the Yahoo! one, and then there’s the Google one.”

So to the extent that she even noticed the ad, she assumed it was just something to do with Google’s search engine, because of the search implications and prior meaning of the brand “Google”.

I don’t think this proves anything, but it was interesting hearing another point of view.

What other kinds of manual labour do people enjoy? It’s pretty much just smashing things up for me I think.

After having read through it, I suppose my first thought is to wonder why anyone who knows that this sort of thing goes on would ever go there. Why any rich person from a democratic country would move there to set up home. Why any Western sporting personalities or pop stars would put their name to Dubai resorts and hotels, to stadiums and their gigs. Coldplay, Bon Jovi, Justin Timberlake, Shakira, Christina Aguilera and Elton John aren’t really acts you associate with slavery, after all.

The urge to take the moral high ground and denounce the practice is strong, and rightly so. Something nags at me though about how much of our society must be built on injustice. The clothes and food we buy when we’re being economical, how much of it comes from oppressed workers? Do we just turn a blind eye all the time?

The people interviewed by Johann Hari in the article either avoid the subject or say they force themselves not to think about it, so a lot of them too are just turning a blind eye. I like to think that if I knew some product was the result of sweat shop labour (let alone slavery) then I would avoid it, but am I kidding myself?

When we could afford it we shopped at Waitrose/Ocado not just because it tasted better but because it’s got to be ethically superior, right? But as soon as we needed to live on a tighter budget this went out of the window and these days we mostly shop at Asda. Most of the bad publicity for supermarkets in the UK seems to be reserved for Tesco but Asda is even cheaper and I can’t really believe their practices are that much better. They have whole cooked chickens for £4, for goodness sake. I could stop shopping there but then that would be less money to spend on other things which would reduce the quality of my life, so this makes me a massive selfish hypocrite.

Of course none of this is comparable to the blatantly fucked Dubai society; if Asda were taking passports off of Filipino girls and forcing them to sell cheap goods for 19 hours a day for no money then certainly I’d consider paying more than 5p per 100ml of Pepsi Max at a competing supermarket.

It’s a bit of a coincidence that we’re hearing all this about Dubai now though, just as they enter the news for allowing their state bank to default massively. This stuff has been going on for years, right? The millions of slaves used to build that place. I confess I never really thought about it before. The worst I’d previously heard was about their insane laws on drugs which could get you locked up for minuscule traces on the outside of your shoes, or for stuff you bought over the counter in the airport.

We all turned a blind eye for years.

If you read this far and for some reason want to comment then it would be great if your comment was not solely based on what you do to consume more ethically than everyone else on the Internet.

At first I thought it was simply because when I had moved his VPS I had taken the opportunity to reconfigure it to the new way I was setting them up, which meant that his root file system would be mounted from /dev/xvda instead of /dev/xvda1 (or /dev/sda1). That would have accounted for it if his monitoring tool had been doing it by device name, but it turned out that it was more fundamental than that — neither mount nor df were showing his root file system at all!

This was highly confusing at first. /proc/mounts looked correct and anyway how does a machine boot if it doesn’t know where its root file system is?

The answer to that question was a bit of a clue really: the boot loader tells the kernel what device the root file system is on, and in this case it was doing it by UUID. The UUID in the boot loader configuration was not the same as the UUID listed in /etc/fstab. I had forgotten to update the customer’s /etc/fstab.

The machine was able to boot because the boot loader was correctly configured, but then after it had already mounted the root file system it was trying to mount everything in /etc/fstab and failing on a line for a UUID that wasn’t present. That line then never made it to /etc/mtab which is what mount and df are reading from.

After correcting the /etc/fstab, it is fixable without a reboot by just mounting / again over the top of the existing one. Or you could probably just edit /etc/mtab.