Supermicro IPMI remote console on Ubuntu 14.04 through SSH tunnel

I normally don’t like using the web interface of Supermicro IPMI because it’s extremely clunky, unintuitive and uses Java in some places.

The other day however I needed to look at the console of a machine which had been left running Memtest86+. You can make Memtest86+ output to serial which is generally preferred for looking at it remotely, but this wasn’t run in that mode so was outputting nothing to the IPMI serial-over-LAN. I would have to use the Java remote console viewer.

As an added wrinkle, the IPMI network interfaces are on a network that I can’t access except through an SSH jump host.

So, I just gave it a go without doing anything special other than launching an SSH tunnel:

$ ssh me@jumphost -L127.0.0.1:1443:192.168.1.21:443 -N

This tunnels my localhost port 1443 to port 443 of 192.168.1.21 as available from the jump host. Local port 1443 used because binding low ports requires root privileges.

This allowed me to log in to the web interface of the IPMI at https://localhost:1443/, though it kept putting up a dialog which said I needed a newer JDK. Going to “Remote Control / Console Redirection” attempted to download a JNLP file and then said it failed to download.

This was with openjdk-7-jre and icedtea-7-plugin installed.

I decided maybe it would work better if I installed the Oracle Java 8 stuff (ugh). That was made easy by following these simple instructions. That’s an Ubuntu PPA which does everything for you, after you agree that you are a bad person who should feel badaccept the license.

This time things got a little further, but still failed saying it couldn’t download a JAR file. I noticed that it was trying to download the JAR from https://127.0.0.1:443/ even though my tunnel was on port 1443.

I eventually did get the remote console viewer to work but I’m not 100% convinced it was because I switched to Oracle Java.

So, basic networking issue here. Maybe it really needs port 443?

Okay, ran SSH as root so it could bind port 443. Got a bit further but now says “connection failed” with no diagnostics as to exactly what connection had failed. Still, gut instinct was that this was the remote console app having started but not having access to some port it needed.

Okay, ran SSH as a SOCKS proxy instead, set the SOCKS proxy in my browser. Same problem.

Did a search to see what ports the Supermicro remote console needs. Tried a new SSH command:

$ sudo ssh me@jumphost \
-L127.0.0.1:443:192.168.1.21:443 \
-L127.0.0.1:5900:192.168.1.21:5900 \
-L127.0.0.1:5901:192.168.1.21:5901 \
-L127.0.0.1:5120:192.168.1.21:5120 \
-L127.0.0.1:5123:192.168.1.21:5123 -N

Apart from a few popup dialogs complaining about “MalformedURLException: unknown protocol: socket” (wtf?), this now appears to work.

Supermicro IPMI remote console

SSH launchers for Ubuntu Unity

Ubuntu 12.04 ^

Given the recent release of Ubuntu 12.04, I thought it was about time that I upgraded one of my machines to it so that I could make sure I could still work with it effectively.

You see, both my laptop and my desktop were on the previous long term support release, 10.04. These days I don’t have a lot of patience for upgrading things every 6 months so I’m glad that the LTS releases are supported for many years. But after 10.04 Ubuntu made a bold departure away from the GNOME desktop and onto Unity. Knowing that I would be forced to change the way I did many things I have been putting off trying Unity. No more putting it off.

Terminals, Terminals, Terminals, Terminals, Terminals, Terminals ^

Given what I do for a living it’s fair to say that the predominant applications running on any of my desktop machines are many instances of terminals running SSH to remote hosts. I try to automate and configuration manage the hell out of everything, but it’s hard to avoid having connections open to a bunch of different machines at any one time.

In 10.04 what I used to do was have a .desktop file for each host that I commonly log in to, something like:

#!/usr/bin/env xdg-open

[Desktop Entry]
Version=1.0
Type=Application
Terminal=false
Icon[en_GB]=/usr/share/icons/Humanity/apps/48/terminal.svg
Name[en_GB]=specialbrew
Exec=urxvtc -T specialbrew -e ssh specialbrew.localnet
Name=specialbrew
Icon=/usr/share/icons/Humanity/apps/48/terminal.svg

I’d then have a menu called “SSH” added to my top menu bar, with an entry for each of those files. This was quite nice as I could also have multiple levels of menu, thus segregating different classes of host, hosts I administer with different hats on, customers I do consulting work for, etc.

Unity’s Launchers ^

Sadly that all goes out of the window with Unity. For a start there is no top menu bar. You’ve got the launcher down the side where you can add the launcher for gnome-terminal, but if you click that launcher more than once all that happens is you get your first terminal window brought back to focus.

There’s an open bug report asking for ways to set different properties on launchers, but judging by the age it doesn’t seem to be much of a priority.

I haven’t got a clue about launchers in Unity but I had a quick read of some documentation and worked out how to add a launcher for urxvt (my preferred terminal as opposed to gnome-terminal), and how to put different options on it. For example:

$ cat ~/.local/share/applications/rxvt.desktop 
[Desktop Entry]
Name=rxvt
Comment=Use the command line
TryExec=/home/andy/bin/urxvtc
Exec=/home/andy/bin/urxvtc
Icon=utilities-terminal
Type=Application
Categories=Utility;TerminalEmulator;
StartupNotify=true
OnlyShowIn=Unity;
Keywords=Run;
Actions=New;specialbrew;backup1

[Desktop Action New]
Name=New Terminal (localhost)
Exec=/home/andy/bin/urxvtc -T stoli
OnlyShowIn=Unity

[Desktop Action specialbrew]
Name=New Terminal (specialbrew)
Exec=/home/andy/bin/urxvtc -T specialbrew -e ssh specialbrew.localnet
OnlyShowIn=Unity

[Desktop Action backup1]
Name=New Terminal (backup1)
Exec=/home/andy/bin/urxvtc -T backup1.bitfolk.com -e ssh backup1.bitfolk.com
OnlyShowIn=Unity

Once you do something like that and get the icon locked on the Launcher, you can right click on it and be offered “localhost”, “specialbrew”, “backup1”, etc.

Okay that is workable, but it kind of sucks. That list will get huge, and it’s a flat list.

Lenses ^

Lenses seem like a very powerful feature of Unity. When I was asking on IRC about how people handled this use case, someone suggested (sarcastically, I think!) that I needed to create a lens to view all my hosts.

I actually did have a look into it, and was initially rather put off by the task. Fortunately it seems that someone already had the idea of a lens that scrapes SSH hosts out of ~/.ssh/config and ~/.ssh/known_hosts.

The SSH Search Lens ^

After installing this, it worked pretty much as advertised. As noted in the README you do have to use “HashKnownHosts no” to take advantage of it being able to read ~/.ssh/known_hosts — some would consider that a security flaw. Rather than disabling known host hashing for all users, you can disable it just for yourself:

$ cat ~/.ssh/config
HashKnownHosts no
...

Note that it only re-parses the SSH configuration files when it starts, which means that if you SSH to somewhere new then it won’t be found in Dash Home until after you’ve logged out and in again (officially). I found that looking for the /usr/bin/python /opt/extras.ubuntu.com/unity-lens-sshsearch/unity-lens-sshsearch.py process and killing it would cause it to be restarted next time I went to Dash Home. That saves a logout/in (but might be Bad).

The current version is hardcoded to call gnome-terminal, and I wanted to change that. I edited /opt/extras.ubuntu.com/unity-lens-sshsearch/unity-lens-sshsearch.py and changed the following line:

TERMINAL_APP = 'gnome-terminal'

to

TERMINAL_APP = '/home/andy/bin/urxvtc'

(Yes, I compile rxvt-unicode from source and keep it in ~/bin. What of it? Wanna fight about it?)

After restarting the lens it failed to work. Nothing happened when clicking on the icons it found. It wasn’t sending anything to ~/.xsession-errors either.

In the end I had to strace it, only to find it was getting “permission denied” when trying to execute my TERMINAL_APP. What? I can execute it myself.

FFFFFUUUUUUUUUUUUUUUUU AppArmor ^

Yeah, unity-lens-sshsearch ships an AppArmor profile, /etc/apparmor.d/opt.extras.ubuntu.com.unity-lens-sshsearch.unity-lens-sshsearch.py to be exact. That specifies what it can execute, and it’s limited to gnome-terminal.

After adding the paths to my rxvt-unicode there (it’s pretty obvious how, if you look in the file) it was happy.

Deficiencies of the SSH Search Lens ^

So, obvious deficiences here:

  • Have to log out or risk killing the process to get it to index newly-added entries.
  • Hard-coded to gnome-terminal.
  • Still limited in terms of configurability to <command>, <user>, <host> and <port>
  • Still has a flat hierarchy — you’ve got a list of hosts that your search term will be matched against. Possibly greater knowledge of Lenses/Scopes could improve this.
  • My rxvt-unicode doesn’t have a nice icon like gnome-terminal does! I’m guessing I will be able to fix this by reading up more about the Launcher.
  • Would be nice if the stderr output of the lens went to ~/.xsession-errors like every other X application, instead of /dev/null that I note it is redirected to. I realise that ~/.xsession-errors tends to be known as “that multi-gigabyte file of garbage that no one ever looks at” but it’s marginally more useful than /dev/null!

But on the whole this is a fairly natural way for me to launch these SSH sessions — I can press the “super” key and start typing the host name and I’ll get a list of matching icons to click on.

Also even though I don’t know Python, the source of this lens seems quite readable so I may be able to improve it and/or make my own lenses in future.

Firefox, Ubuntu and middlemouse.contentLoadURL

I use Firefox web browser, currently on Ubuntu 10.04 LTS. For many years I have set the config option middlemouse.contentLoadURL to true so that middle clicking anywhere in the page (that does not accept input) will load the URL that is in my clipboard.

After restarting my web browser somewhere near the end of January 2012 I found my Firefox 3.x had been upgraded to Firefox 9.x. Also the middle click behaviour no longer worked.

Perusing about:config showed that the option had been set to false again. I set it back to true but on restart of the browser it was set back to false. A bit of searching about found various suggestions about forcing it in my user.js file, but none of those worked either.

Finally, in desperation, I did a search of every file beneath /usr for the string “middlemouse”. Lo and behold:

/usr/lib/firefox-9.0.1/extensions/ubufox@ubuntu.com/defaults/preferences/ubuntu-mods.js

…
pref("middlemouse.contentLoadURL", false); //setting to false disables pasting urls on to the page
…

Commenting this line out once more allowed me to change the setting myself.

It seems this this override was discussed by Ubuntu as far back as 2004, but it only became something that I could not override upon the upgrade to Firefox 9.

I reported a bug about this, and one of the comments seems to suggest that the method Ubuntu uses to change these settings has changed because they were breaking Firefox Sync, and that this outcome (overriding middlemouse.contentLoadURL) is not as bad as breaking Firefox Sync.

Even so, I would suggest that this outcome is very confusing for people and that as middlemouse.contentLoadURL is a popular setting which is easy to change, it should not be overridden in some obscure file.

As of the recent upgrade to Firefox 11, the file with the override in it has now moved to /usr/share/xul-ext/ubufox/defaults/preferences/ubuntu-mods.js.

Win a BitFolk VPS on the Ubuntu UK Podcast!

As well as sponsoring one of the mirrors, BitFolk has contributed a prize in the most recent episode of the Ubuntu UK Podcast, released on Saturday 30th August.

The prize is £200 of service credit with BitFolk, so for example the winner could put it towards a 300M VPS for one year or a 120M for over two years. Even existing customers should feel free to enter. You’ll have to listen to the podcast episode to find out the competition question and instructions for entering.

There is also an informative discussion of virtual server technology in general, so give it a listen.

BitFolk

Two mirrors for Ubuntu UK Podcast

BitFolk is providing additional mirrors in London and San Francisco for the Ubuntu UK Podcast.

I think my end is sorted out so hopefully the Ubuntu UK side of things will be ready in time for episode 2’s release early next week.

Obviously as it’s only on episode 2 it’s rather new so I’m not exactly expecting a deluge of requests, but it will be interesting to see actually what bandwidth is used.

(click images for more detail; if image is broken then the graph is currently updating)