The ongoing struggle

Ubuntu 12.04

Given the recent release of Ubuntu 12.04, I thought it was about time that I upgraded one of my machines to it so that I could make sure I could still work with it effectively.

You see, both my laptop and my desktop were on the previous long term support release, 10.04. These days I don’t have a lot of patience for upgrading things every 6 months so I’m glad that the LTS releases are supported for many years. But after 10.04 Ubuntu made a bold departure away from the GNOME desktop and onto Unity. Knowing that I would be forced to change the way I did many things I have been putting off trying Unity. No more putting it off.

Terminals, Terminals, Terminals, Terminals, Terminals, Terminals

Given what I do for a living it’s fair to say that the predominant applications running on any of my desktop machines are many instances of terminals running SSH to remote hosts. I try to automate and configuration manage the hell out of everything, but it’s hard to avoid having connections open to a bunch of different machines at any one time.

In 10.04 what I used to do was have a .desktop file for each host that I commonly log in to, something like:

#!/usr/bin/env xdg-open

[Desktop Entry]
Version=1.0
Type=Application
Terminal=false
Icon[en_GB]=/usr/share/icons/Humanity/apps/48/terminal.svg
Name[en_GB]=specialbrew
Exec=urxvtc -T specialbrew -e ssh specialbrew.localnet
Name=specialbrew
Icon=/usr/share/icons/Humanity/apps/48/terminal.svg

I’d then have a menu called “SSH” added to my top menu bar, with an entry for each of those files. This was quite nice as I could also have multiple levels of menu, thus segregating different classes of host, hosts I administer with different hats on, customers I do consulting work for, etc.

Unity’s Launchers

Sadly that all goes out of the window with Unity. For a start there is no top menu bar. You’ve got the launcher down the side where you can add the launcher for gnome-terminal, but if you click that launcher more than once all that happens is you get your first terminal window brought back to focus.

There’s an open bug report asking for ways to set different properties on launchers, but judging by the age it doesn’t seem to be much of a priority.

I haven’t got a clue about launchers in Unity but I had a quick read of some documentation and worked out how to add a launcher for urxvt (my preferred terminal as opposed to gnome-terminal), and how to put different options on it. For example:

$ cat ~/.local/share/applications/rxvt.desktop 
[Desktop Entry]
Name=rxvt
Comment=Use the command line
TryExec=/home/andy/bin/urxvtc
Exec=/home/andy/bin/urxvtc
Icon=utilities-terminal
Type=Application
Categories=Utility;TerminalEmulator;
StartupNotify=true
OnlyShowIn=Unity;
Keywords=Run;
Actions=New;specialbrew;backup1

[Desktop Action New]
Name=New Terminal (localhost)
Exec=/home/andy/bin/urxvtc -T stoli
OnlyShowIn=Unity

[Desktop Action specialbrew]
Name=New Terminal (specialbrew)
Exec=/home/andy/bin/urxvtc -T specialbrew -e ssh specialbrew.localnet
OnlyShowIn=Unity

[Desktop Action backup1]
Name=New Terminal (backup1)
Exec=/home/andy/bin/urxvtc -T backup1.bitfolk.com -e ssh backup1.bitfolk.com
OnlyShowIn=Unity

Once you do something like that and get the icon locked on the Launcher, you can right click on it and be offered “localhost”, “specialbrew”, “backup1″, etc.

Okay that is workable, but it kind of sucks. That list will get huge, and it’s a flat list.

Lenses

Lenses seem like a very powerful feature of Unity. When I was asking on IRC about how people handled this use case, someone suggested (sarcastically, I think!) that I needed to create a lens to view all my hosts.

I actually did have a look into it, and was initially rather put off by the task. Fortunately it seems that someone already had the idea of a lens that scrapes SSH hosts out of ~/.ssh/config and ~/.ssh/known_hosts.

The SSH Search Lens

After installing this, it worked pretty much as advertised. As noted in the README you do have to use “HashKnownHosts no” to take advantage of it being able to read ~/.ssh/known_hosts — some would consider that a security flaw. Rather than disabling known host hashing for all users, you can disable it just for yourself:

$ cat ~/.ssh/config
HashKnownHosts no
...

Note that it only re-parses the SSH configuration files when it starts, which means that if you SSH to somewhere new then it won’t be found in Dash Home until after you’ve logged out and in again (officially). I found that looking for the /usr/bin/python /opt/extras.ubuntu.com/unity-lens-sshsearch/unity-lens-sshsearch.py process and killing it would cause it to be restarted next time I went to Dash Home. That saves a logout/in (but might be Bad).

The current version is hardcoded to call gnome-terminal, and I wanted to change that. I edited /opt/extras.ubuntu.com/unity-lens-sshsearch/unity-lens-sshsearch.py and changed the following line:

TERMINAL_APP = 'gnome-terminal'

to

TERMINAL_APP = '/home/andy/bin/urxvtc'

(Yes, I compile rxvt-unicode from source and keep it in ~/bin. What of it? Wanna fight about it?)

After restarting the lens it failed to work. Nothing happened when clicking on the icons it found. It wasn’t sending anything to ~/.xsession-errors either.

In the end I had to strace it, only to find it was getting “permission denied” when trying to execute my TERMINAL_APP. What? I can execute it myself.

FFFFFUUUUUUUUUUUUUUUUU AppArmor

Yeah, unity-lens-sshsearch ships an AppArmor profile, /etc/apparmor.d/opt.extras.ubuntu.com.unity-lens-sshsearch.unity-lens-sshsearch.py to be exact. That specifies what it can execute, and it’s limited to gnome-terminal.

After adding the paths to my rxvt-unicode there (it’s pretty obvious how, if you look in the file) it was happy.

Deficiencies of the SSH Search Lens

So, obvious deficiences here:

• Have to log out or risk killing the process to get it to index newly-added entries.
• Hard-coded to gnome-terminal.
• Still limited in terms of configurability to <command>, <user>, <host> and <port>
• Still has a flat hierarchy — you’ve got a list of hosts that your search term will be matched against. Possibly greater knowledge of Lenses/Scopes could improve this.
• My rxvt-unicode doesn’t have a nice icon like gnome-terminal does! I’m guessing I will be able to fix this by reading up more about the Launcher.
• Would be nice if the stderr output of the lens went to ~/.xsession-errors like every other X application, instead of /dev/null that I note it is redirected to. I realise that ~/.xsession-errors tends to be known as “that multi-gigabyte file of garbage that no one ever looks at” but it’s marginally more useful than /dev/null!

But on the whole this is a fairly natural way for me to launch these SSH sessions — I can press the “super” key and start typing the host name and I’ll get a list of matching icons to click on.

Also even though I don’t know Python, the source of this lens seems quite readable so I may be able to improve it and/or make my own lenses in future.

Tonight I’m on my own as Jenny decided to go to bed early; she has to get up very very early tomorrow for work. I got up a bit late today and don’t feel tired at all so I’m just contemplating an evening of work.

When I work I like to have a soundtrack, so I’m picking out a playlist for the next 12 hours (yes I will probably stay up all night).

What struck me is how much great music I have and what a terrible loss it would be if my collection were to be taken from me.

I’m not saying I have great taste in music. I don’t go to gigs — in fact I’ve never actually been to a gig at any venue larger than a pub — and I tend to find my new music through radio and TV; Later…, coverage of Reading, Glastonbury, that sort of thing. My taste in music has been described as “mediocre” by others, so I’m not saying I’m any kind of opinion leader here.

I was having a conversation on IRC recently about the streaming music service Spotify and how I don’t really understand the use case for it — I do get the mobile streaming part, it’s the idea of using it at home as your main method of playing music that I fail to comprehend.

During that conversation someone said to me:

“I use Spotify because I don’t have a music collection [...] I don’t derive pleasure from having a music collection.”

This idea completely boggles my mind! Looking through my collection I find all kinds of things with personal attachment.

It’s not that I feel like I have every bit of music ever. I know people who just download every bit of music they can and have hundreds of thousands of tracks. I’m not like that; I have just over 3,200 tracks most of which were ripped from CDs or purchased as online singles. If I don’t find myself listening to something for years then I usually delete it. So, my collection is stuff I do still listen to.

When building a playlist, every time seeing the list of albums brings back so many memories. Music that came out at certain times in my life, or was listened to a lot at certain times in my life. It brings back memories of my teenage years, university, past relationships (girlfriends who stole my CDs!), people who have since died. I’m not into a lot of obscure music, but there’s things there you won’t even find on Amazon as a CD, let alone on Spotify for streaming.

Maybe I am just getting old and not embracing the cloud. But how does one build a big playlist with something like Spotify? What about when they remove things from the service? I should just try the free version and see what it’s like.

Perhaps there are people of an older generation who don’t like the idea of only keeping music on the computer, and regard me with pity for not being immediately able to lay my hands on the CD or vinyl for most of my collection? That really doesn’t bother me; to me it’s the music that matters and it’s there for playing.

What bothers me is the idea of marking some track in the cloud as “liked” by me, and then later finding it’s disappeared for some reason so I can no longer listen. Memories gone.

If I did use something like Spotify I’d probably have to do some report of things I listened to a lot and make sure I buy them. I will get around to trying out Spotify at some point but I can’t imagine it will replace the desire to buy and own music, rather I would hope it would help me find more music that I like.

Because having music is ace.

I use Firefox web browser, currently on Ubuntu 10.04 LTS. For many years I have set the config option middlemouse.contentLoadURL to true so that middle clicking anywhere in the page (that does not accept input) will load the URL that is in my clipboard.

After restarting my web browser somewhere near the end of January 2012 I found my Firefox 3.x had been upgraded to Firefox 9.x. Also the middle click behaviour no longer worked.

Perusing about:config showed that the option had been set to false again. I set it back to true but on restart of the browser it was set back to false. A bit of searching about found various suggestions about forcing it in my user.js file, but none of those worked either.

Finally, in desperation, I did a search of every file beneath /usr for the string “middlemouse”. Lo and behold:

/usr/lib/firefox-9.0.1/extensions/ubufox@ubuntu.com/defaults/preferences/ubuntu-mods.js

…
pref("middlemouse.contentLoadURL", false); //setting to false disables pasting urls on to the page
…

Commenting this line out once more allowed me to change the setting myself.

It seems this this override was discussed by Ubuntu as far back as 2004, but it only became something that I could not override upon the upgrade to Firefox 9.

I reported a bug about this, and one of the comments seems to suggest that the method Ubuntu uses to change these settings has changed because they were breaking Firefox Sync, and that this outcome (overriding middlemouse.contentLoadURL) is not as bad as breaking Firefox Sync.

Even so, I would suggest that this outcome is very confusing for people and that as middlemouse.contentLoadURL is a popular setting which is easy to change, it should not be overridden in some obscure file.

As of the recent upgrade to Firefox 11, the file with the override in it has now moved to /usr/share/xul-ext/ubufox/defaults/preferences/ubuntu-mods.js.

Right, System Integrators – those companies that buy components from Supermicro et al and build you a server out of them. You guys seem to have a bit of a fascination with screwing. Screwing things in as tight as you can. Please stop.

It’s 100% true that vibration of components like hard disks is bad. numerous studies have been done that prove that vibration causes performance problems as drives need to do more corrective work.

However, this does not mean that you have to screw in the drives to the caddies to the limit of what is physically possible. They just need to be tightened until a little force won’t tighten them any more.

When you supply me with a server that’s got four super-tightened screws for each drive in it, and I deploy that server, chances are that one of the first things that will break in that server is one of the disk drives.

During the years those screws have been there they haven’t got any looser. It’s likely that if you tightened them all to the limit of your strength and tools, by now the force required to unscrew them will be less than the force required to deform the screw head. Like this:

No, this is not an issue of using the wrong driver head. Yes, you will strip a screw if you use the wrong driver head. That’s why I carry this stuff every time I go to a datacentre:

There’s two exactly correct drivers in there, and several that should also work anyway despite being a little bit off. I have never had a problem unscrewing any screw that I originally put in. Probably because I don’t tighten them like I am some sort of lunatic. I can even unscrew them around a corner with the offline driver. Oh yeah baby. So far nothing I have screwed in with merely normal force has fallen apart.

And this is not an isolated occurrence! Nearly all of you seem to do this with every screw, everywhere. Stop it!

The drive in that caddy is a dead one, and luckily I had a spare caddy with me for the replacement drive to go in, otherwise I too would have been screwed beyond the limits of my endurance.

So, now I’ve got to drill those out just to get this caddy back to being useful again. Or more likely find someone else to drill it out for me as I don’t trust myself with power tools really.

Earlier this evening I received a marketing email from a company I had never heard of, for an event I wasn’t interested in, to an email address I had only ever given to a differently, seemingly-unrelated company that we shall call Company U.

When this sort of thing happens it may be an indication that Company U has leaked their customer address database or else decided to sell the contact details on, so I’d sometimes follow it up instead of just opting out and consigning the address to the bitbucket forever.

In this case as many times before, I decided to have a whinge on twitter about it first.

Anyway to cut a long story short they got in touch, and it turns out that this marketing email has been sent by Company S. Someone from Company S emailed me to apologise and to remove my address from their list.

I asked this person how they obtained my email address that had only been sent to Company U, and they admitted that they used to work for Company U and that their “rampant CRM system” had somehow “indexed all of my PERSONAL emails”.

Now, from my point of view, this would actually suggest that this person has likely taken a database of customers of Company U with him to his new employer.

So, Internauts, do I have a duty to shop this guy to his former employer Company U? Or am I just frothing in my nerdrage here at the terrible inconvenience of being sent a piece of email I don’t want? Is the man a menace, or should I just get over it?

I’ll let you know how it goes.

Update 2011-11-26:

The first attempt worked fairly well. It was not possible to remove the bottle from the ice without filling it with some warm water for a few seconds. I think this was probably because the bottle had a very slight bulge at the bottom.

It would be hard to find a perfectly cylindrical or even tapering plastic bottle, so I will next try the original plastic cup suggestion.

Today one of my rsync backups began failing with:

inflate (token) returned -5
rsync error: error in rsync protocol data stream (code 12) at token.c(604) [receiver=3.0.3]
rsync: writefd_unbuffered failed to write 373 bytes [generator]: Broken pipe (32)
rsync error: error in rsync protocol data stream (code 12) at io.c(1544) [generator=3.0.3]

It was repeatable when trying to transfer the same file (a large gzipped SQL dump file).

It turned out to be a bug in that version of rsync.

rsync 3.0.3 comes with Debian lenny. In order to get a newer version I have had to use lenny-backports for this. That gets me rsync v3.0.7, which does not exhibit this bug.

(Yes, I am aware that squeeze has been released and this host should be upgraded to that. There is security support for lenny until at least February 2012.)

On November 2nd I received this spam:

(some headers removed; xxxxxxxxxxx@strugglers.net is my censored email address)

Received: from mail15.soatube.com ([184.105.143.66])
        by mail.bitfolk.com with esmtp (Exim 4.72)
        (envelope-from <bounce@soatube.com%gt;)
        id 1RLikr-00070I-6U
        for xxxxxxxxxxx@strugglers.net; Wed, 02 Nov 2011 21:53:57 +0000
Received: from [64.62.145.53] (mail3.soatube.com [64.62.145.53])
        by mail15.soatube.com (Postfix) with ESMTP id 6B324181CFF
        for <xxxxxxxxxxx@strugglers.net>;
        Wed,  2 Nov 2011 14:46:01 -0700 (PDT)
To: xxxxxxxxxxx@strugglers.net
From: events@idevnews.com
Date: Wed, 02 Nov 2011 14:00:40 -0700
Subject: BPM Panel Discussion: IBM, Oracle and Progress Software

-------------
BPM-CON: BPM Panel Discussion - IBM, Oracle and Progress Software
-------------
Online Conference

Expert Speakers:
IBM, Oracle, Progress Software
etc..

The email address it arrived at was an email address I created in November 2004 in order to take a web-based test on Red Hat’s web site prior to going on an RHCE course. It has only ever been provided to Red Hat, and has not received any email since 2007 (and all of that was from Red Hat). Until November 2nd.

The spam email contains no reference to Red Hat and is not related to any Red Hat product.

From my point of view, I can only think that one of the following things has happened:

1. Spammers guessed this email address out of the blue, first time, without trying any of the other possible variations of it all of which would still reach me.
2. One of my computers has been cracked into and the only apparent repercussion is that someone spammed an email address that appears only in an email archive from 2004/2005.
3. Red Hat knowingly gave/sold my email address to some spammers.
4. Red Hat or one of its agents have accidentally lost a database containing email addresses.

Possibility #4 seems far and away the most likely.

I contacted Red Hat to ask them if they knew what had happened, but they ignored all of my questions and simply sent me the following statement:

“Hello.

Thank you for contacting Red Hat.

we apologies for the inconvenience caused however we would like to inform you that we have not provided your email address to anyone.

Thank You.

Red Hat Training coordinator.”

That wasn’t really what I was asking. Let’s try again.

“Hi Red Hat Training coordinator,

Thanks for your reply, but I’m afraid I am not very reassured by your response. Do you have any suggestions as to how an email address created in 2004 and used only by yourselves for my RHCE exam managed to be used for unrelated marketing by a third party in 2011, unless Red Hat either provided my email address or leaked my email address?

For clarity we are talking about the email address “xxxxxxxxxxx@strugglers.net” which has never ever received any email except from Red Hat, until yesterday, when it got some unwanted
marketing email from a third party.”

“Hi Andy,

Please be assured that Red Hat does not circulate student’s e-mail address to any third party.

Thanks,
Red Hat Training Coordinator”

I’m not getting anywhere am I? I was only after some reassurance that they would actually look into it. Maybe they are looking into it, and for some reason decided that the best way to assure me of this was to show complete disinterest.

Oh well, I can send that email address to the bitbucket, but I can’t help thinking it’s not just my email address that has been leaked.

Anyone else received similar email? If so, was it to an address you gave to Red Hat?

Update 2011-11-10: Someone suggested I politely ask the marketer where they obtained my email address. It’s worth a try.

“Hi Integration Developer News,

May I ask where you obtained my email address
“xxxxxxxxxxx@strugglers.net”? I’m concerned that it may have been
given to you without my authority.

Thanks,
Andy”

Also I have now been contacted by someone from Red Hat’s Information Security team, who is looking into it. Thanks!

September 4th 2011

I cut my left index finger somehow. Slightly painful, but it was only a little cut so I thought no more.

Early hours of September 5th 2011

I noticed that my finger was swollen, and its knuckle also. It was slightly more painful to bend my finger.

Early morning of September 5th 2011

I noticed that there was a red line between my swollen knuckle and my wrist, following the line of the tendon. It was hot and tender to the touch.

Late morning of September 5th 2011

I noticed that the red line had now gone past my wrist and was snaking its way up my arm. It was very warm to the touch. I started to get a bit worried about it. It varied over time in how angry it looked; the picture on the right was taken when it wasn’t very visible.

We had a grocery order arriving between 1pm and 3pm and I hadn’t had any sleep, so I decided to get some sleep and see what it looked like after the order had arrived.

The order arrived near 3pm. The red line on my arm had faded to almost invisibility. Unless you were looking for it you probably wouldn’t have spotted it. My finger wasn’t so sore any more either. I decided there was no need to worry about it.

Evening of September 5th 2011

We had dinner and started to watch a movie. My arm got more and more uncomfortable, the red line was showing more than ever, and I noticed that it was now just starting to go above my elbow.

At this point I decided that I did actually want to see a medical professional. Having recently seen what happened with Maria’s foot I had a bit of the fear.

We made our way to Ashford Hospital NHS walk-in centre, which is a short bus ride away. It’s open until 10pm seven days a week, with last consult at 9.30pm. We arrived at about 8.20pm and I described my situation, only to be told that they were not seeing anyone else that evening. I was given the choice of out of hours GP or accident and emergency.

I called the out of hours GP number. A callcentre operative took my details and a doctor phoned me back within about 15 minutes. He said that I should have someone look at my arm, and arranged an appointment with a doctor based at West Middlesex hospital. It was the last available appointment, 9.30pm, and I wasn’t sure we could get there in time. I knew there was also an A&E at West Middlesex though, so I figured that if we missed the appointment then I would just go to the A&E there anyway.

One £20 cab journey to Isleworth later, we arrived at West Middlesex hospital at about 9.27pm and I was seen by a doctor.

She immediately said that I had an infection which was travelling under my skin, with the red line being cellulitis. She said that it was good that I hadn’t delayed seeking medical attention any further, because if the infection reached the lymph nodes in my arm pit then I would likely need intravenous antibiotics as opposed to the oral antibiotics she prescribed me (500mg flucloxacillin four times a day, 500mg penicillin four times a day).

Early hours of September 6th 2011

My hand was by now even more swollen and finger near the wound very painful to the touch. Even resting my hand on a pillow was painful. My whole arm was stiff and slightly painful to move, with some pains around half way between elbow and arm pit. I couldn’t sleep.

The red line of cellulitis seemed to have faded away, however. It was still too soon to tell if this was due to the antibiotics (of which I had only taken 500mg of each thus far) doing their stuff, or just a coincidence.

September 11th 2011

There’s only 24 hours of medication left now. Pretty much since the 7th things got better very quickly with only my finger remaining sore.

Right now the skin all around the area that was infected has flaked off and shiny new skin is underneath. Thanks modern medicine! I like not dying of septicaemia.

By default, a Linux host on an IPv6 network will listen for and solicit router advertisements in order to choose an IPv6 address for itself and to set up its default route. This is referred to as stateless address autoconfiguration (SLAAC).

If you don’t want a host to automatically configure an address and route then you could disable this behaviour by writing “0″ to /proc/sys/net/ipv6/conf/*/accept_ra.

Additionally, if the Linux host considers itself to be a router then it will ignore all router advertisements.

In this context, what makes the difference between router or not are the settings of the /proc/sys/net/ipv6/conf/*/forwarding files (or the net.ipv6.conf.*.forwarding sysctl). If you turn your host into a router by setting one of those to “1″, you may find that your host removes any IPv6 address and default route it learnt via SLAAC.

There is a valid argument that a router should not be autoconfiguring itself, and should have its addresses and routes configured statically. Linux has IP forwarding features for a reason though, and sometimes you want to forward packets with a Linux box while still enjoying autoconfiguration. In my case I have some hosts running virtual machines, with IPv6 prefixes routed to the virtual machines. I’d still like the hosts to learn their default route via SLAAC.

It’s taken me a long time to work out how to do this. It isn’t well-documented.

Firstly, if you have a kernel version of 2.6.37 or higher then your answer is to set accept_ra to “2″. From ip-sysctl.txt:

accept_ra – BOOLEAN

Accept Router Advertisements; autoconfigure using them.

Possible values are:

• 0 Do not accept Router Advertisements.
• 1 Accept Router Advertisements if forwarding is disabled.
• 2 Overrule forwarding behaviour. Accept Router Advertisements even if forwarding is enabled.

Functional default:

• enabled if local forwarding is disabled.
• disabled if local forwarding is enabled.

This appears to be a type of boolean that I wasn’t previously familiar with – one that has three different values.

If you don’t have kernel version 2.6.37 though, like say, everyone running the current Debian stable (2.6.32), this will not work. Helpfully, it also doesn’t give you any sort of error when you set accept_ra to “2″. It just sets it and continues silently ignoring router advertisements.

Fortunately Bjørn Mork posted about a workaround for earlier kernels which I would likely have never discovered otherwise. You just have to disable forwarding for the interface that your router advertisements will come in on, e.g.:

# echo 0 > /proc/sys/net/ipv6/conf/eth0/forwarding

Apparently as long as /proc/sys/net/ipv6/conf/all/forwarding is still set to “1″ then forwarding will still be enabled. Obviously.

Additionally there are some extremely unintuitive interactions between “default” and “all” settings you may set in /etc/sysctl.conf and pre-existing interfaces. So there is a race condition on boot between IPv6 interfaces coming up and sysctl configuration being parsed. martin f krafft posted about this, and on Debian recommends setting desired sysctls in pre-up headers of the relevant iface stanza in /etc/network/interfaces, e.g.:

iface eth0 inet6 static
    address 2001:0db8:10c0:d0c5::1
    netmask 64
# Enable forwarding
    pre-up echo 1 > /proc/sys/net/ipv6/conf/default/forwarding
    pre-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
# But disable forwarding on THIS interface so we still get RAs
    pre-up echo 0 > /proc/sys/net/ipv6/conf/$IFACE/forwarding
    pre-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra
    pre-up echo 1 > /proc/sys/net/ipv6/conf/all/accept_ra
    pre-up echo 1 > /proc/sys/net/ipv6/conf/default/accept_ra

You will now have forwarding and SLAAC.