Disabling the default IPMI credentials on a Supermicro server

In an earlier post I mentioned that you should disable the default ADMIN / ADMIN credentials on the IPMI controller. Here’s how.

Install ipmitool ^

ipmitool is the utility that you will use from the command line of another machine in order to interact with the IPMI controllers on your servers.

# apt-get install ipmitool

List the current users ^

$ ipmitool -I lanplus -H 192.168.1.22 -U ADMIN -a user list
Password:
ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
2   ADMIN            false   false      true       ADMINISTRATOR

Here you are specifying the IP address of the server’s IPMI controller. ADMIN is the IPMI user name you will use to log in, and it’s prompting you for the password which is also ADMIN by default.

Add a new user ^

You should add a new user with a name other than ADMIN.

I suppose it would be safe to just change the password of the existing ADMIN user, but there is no need to have it named that, so you may as well pick a new name.

$ ipmitool -I lanplus -H 192.168.1.22 -U ADMIN -a user set name 3 somename
Password:
$ ipmitool -I lanplus -H 192.168.1.22 -U ADMIN -a user set password 3
Password:
Password for user 3:
Password for user 3:
$ ipmitool -I lanplus -H 192.168.1.22 -U ADMIN -a channel setaccess 1 3 link=on ipmi=on callin=on privilege=4
Password:
$ ipmitool -I lanplus -H 192.168.1.22 -U ADMIN -a user enable 3
Password:

From this point on you can switch to using the new user instead.

$ ipmitool -I lanplus -H 192.168.1.22 -U somename -a user list
Password:
ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
2   ADMIN            false   false      true       ADMINISTRATOR
3   somename         true    true       true       ADMINISTRATOR

Disable ADMIN user ^

Before doing this bit you may wish to check that the new user you added works for everything you need it to. Those things might include:

  • ssh to somename@192.168.1.22
  • Log in on web interface at https://192.168.1.22/
  • Various ipmitool commands like querying power status:
    $ ipmitool -I lanplus -H 192.168.1.22 -U somename -a power status
    Password:
    Chassis power is on

If all of that is okay then you can disable ADMIN:

$ ipmitool -I lanplus -H 192.168.1.22 -U somename -a user disable 2
Password:

If you are paranoid (or this is just the first time you’ve done this) you could now check to see that none of the above things now work when you try to use ADMIN / ADMIN.

Specifying the password ^

I have not done so in these examples but if you get bored of typing the password every time then you could put it in the IPMI_PASSWORD environment variable and use -E instead of -a on the ipmitool command line.

When setting the IPMI_PASSWORD environment variable you probably don’t want it logged in your shell’s history file. Depending on which shell you use there may be different ways to achieve that.

With bash, if you have ignorespace in the HISTCONTROL environment variable then commands prefixed by one or more spaces won’t be logged. Alternatively you could temporarily disable history logging with:

$ set +o history
$ sensitive commend goes here
$ set -o history # re-enable history logging

So anyway…

$ echo $HISTCONTROL
ignoredups:ignorespace
$     export IPMI_PASSWORD=letmein
$ # ^ note the leading spaces here
$ # to prevent the shell logging it
$ ipmitool -I lanplus -H 192.168.1.22 -U somename -E power status
Chassis Power is on

Installing Debian by PXE using Supermicro IPMI Serial over LAN

Here’s how to install Debian jessie on a Supermicro server using PXE boot and the IPMI serial-over-LAN.

Using these instructions you will be able to complete an install of a remote machine, although you will initially need access to the BIOS to configure the IPMI part.

BIOS settings ^

This bit needs you to be in the same location as the machine, or else have someone who is make the required changes.

Press DEL to go into the BIOS configuration.

Under Advanced > PCIe/PCI/PnP Configuration make sure that the network interface through which you’ll reach your PXE server has the “PXE” option ROM set:

BIOS PSE option ROM

Under Advanced > Serial Port Console Redirection you’ll want to enable SOL Console Redirection.

BIOS serial console redirection

(Pictured here is also COM1 Console Redirection. This is for the physical serial port on the machine, not the one in the IPMI.)

Under SOL Console Redirection Settings you may as well set the Bits per second to 115200.

BIOS SOL redirection settings

Now it’s time to configure the IPMI so you can interact with it over the network. Under IPMI > BMC Network Configuration, put the IPMI on your management network:

IPMI network configuration

Connecting to the IPMI serial ^

With the above BIOS settings in place you should be able to save and reboot and then connect to the IPMI serial console. The default credentials are ADMIn / ADMIN which you should of course change with ipmitool, but that is for a different post.

There’s two ways to connect to the serial-over-LAN: You can ssh to the IPMI controller, or you can use ipmitool. Personally I prefer ssh, but the ipmitool way is like this:

$ ipmitool -I lanplus -H 192.168.1.22 -U ADMIN -a sol activate

The ssh way:

$ ssh ADMIN@192.168.1.22
The authenticity of host '192.168.1.22 (192.168.1.22)' can't be established.
RSA key fingerprint is b7:e1:12:94:37:81:fc:f7:db:6f:1c:00:e4:e0:e1:c4.
Are you sure you want to continue connecting (yes/no)?
Warning: Permanently added '192.168.1.22,192.168.1.22' (RSA) to the list of known hosts.
ADMIN@192.168.1.22's password:
 
ATEN SMASH-CLP System Management Shell, version 1.05
Copyright (c) 2008-2009 by ATEN International CO., Ltd.
All Rights Reserved 
 
 
-> cd /system1/sol1
/system1/sol1
 
-> start
/system1/sol1
press <Enter>, <Esc>, and then <T> to terminate session
(press the keys in sequence, one after the other)

They both end up displaying basically the same thing.

The serial console should just be displaying the boot process, which won’t go anywhere.

DHCP and TFTP server ^

You will need to configure a DHCP and TFTP server on an already-existing machine on the same LAN as your new server. They can both run on the same host.

The DHCP server responds to the initial requests for IP address configuration and passes along where to get the boot environment from. The TFTP server serves up that boot environment. The boot environment here consists of a kernel, initramfs and some configuration for passing arguments to the bootloader/kernel. The boot environment is provided by the Debian project.

DHCP ^

I’m using isc-dhcp-server. Its configuration file is at /etc/dhcp/dhcpd.conf.

You’ll need to know the MAC address of the server, which can be obtained either from the front page of the IPMI controller’s web interface (i.e. https://192.168.1.22/ in this case) or else it is displayed on the serial console when it attempts to do a PXE boot. So, add a section for that:

subnet 192.168.2.0 netmask 255.255.255.0 {
}
 
host foo {
    hardware ethernet 0C:C4:7A:7C:28:40;
    fixed-address 192.168.2.22;
    filename "pxelinux.0";
    next-server 192.168.2.251;
    option subnet-mask 255.255.255.0;
    option routers 192.168.2.1;
}

Here we set the network configuration of the new server with fixed-address, option subnet-mask and option routers. The IP address in next-server refers to the IP address of the TFTP server, and pxelinux.0 is what the new server will download from it.

Make sure that is running:

# service isc-dhcp-server start

DHCP uses UDP port 67, so make sure that is allowed through your firewall.

TFTP ^

A number of different TFTP servers are available. I use tftpd-hpa, which is mostly configured by variables in /etc/default/tftp-hpa:

TFTP_OPTIONS="--secure"
TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="0.0.0.0:69"

TFTP_DIRECTORY is where you’ll put the files for the PXE environment.

Make sure that the TFTP server is running:

# service tftpd-hpa start

TFTP uses UDP port 69, so make sure that is allowed through your firewall.

Download the netboot files from your local Debian mirror:

$ cd /srv/tftp
$ curl -s http://ftp.YOUR-MIRROR.debian.org/debian/dists/jessie/main/installer-amd64/current/images/netboot/netboot.tar.gz | sudo tar zxvf -
./                
./version.info    
./ldlinux.c32     
./pxelinux.0      
./pxelinux.cfg    
./debian-installer/
./debian-installer/amd64/
./debian-installer/amd64/bootnetx64.efi
./debian-installer/amd64/grub/
./debian-installer/amd64/grub/grub.cfg
./debian-installer/amd64/grub/font.pf2
…

(This assumes you are installing a device with architecture amd64.)

At this point your TFTP server root should contain a debian-installer subdirectory and a couple of links into it:

$ ls -l .
total 8
drwxrwxr-x 3 root root 4096 Jun  4  2015 debian-installer
lrwxrwxrwx 1 root root   47 Jun  4  2015 ldlinux.c32 -> debian-installer/amd64/boot-screens/ldlinux.c32
lrwxrwxrwx 1 root root   33 Jun  4  2015 pxelinux.0 -> debian-installer/amd64/pxelinux.0
lrwxrwxrwx 1 root root   35 Jun  4  2015 pxelinux.cfg -> debian-installer/amd64/pxelinux.cfg
-rw-rw-r-- 1 root root   61 Jun  4  2015 version.info

You could now boot your server and it would call out to PXE to do its netboot, but would be displaying the installer process on the VGA output. If you intend to carry it out using the Remote Console facility of the IPMI interface then that may be good enough. If you want to do it over the serial-over-LAN though, you’ll need to edit some of the files that came out of the netboot.tar.gz to configure that.

Here’s a list of the files you need to edit. All you are doing in each one is telling it to use serial console. The changes are quite mechanical so you can easily come up with a script to do it, but here I will show the changes verbosely. All the files live in the debian-installer/amd64/boot-screens/ directory.

ttyS1 is used here because this system has a real serial port on ttyS0. 115200 is the baud rate of ttyS1 as configured in the BIOS earlier.

adtxt.cfg

From:

label expert
        menu label ^Expert install
        kernel debian-installer/amd64/linux
        append priority=low vga=788 initrd=debian-installer/amd64/initrd.gz --- 
include debian-installer/amd64/boot-screens/rqtxt.cfg
label auto
        menu label ^Automated install
        kernel debian-installer/amd64/linux
        append auto=true priority=critical vga=788 initrd=debian-installer/amd64/initrd.gz --- quiet

To:

label expert
        menu label ^Expert install
        kernel debian-installer/amd64/linux
        append priority=low console=ttyS1,115200n8 initrd=debian-installer/amd64/initrd.gz --- 
include debian-installer/amd64/boot-screens/rqtxt.cfg
label auto
        menu label ^Automated install
        kernel debian-installer/amd64/linux
        append auto=true priority=critical console=ttyS1,115200n8 initrd=debian-installer/amd64/initrd.gz --- quiet
rqtxt.cfg

From:

label rescue
        menu label ^Rescue mode
        kernel debian-installer/amd64/linux
        append vga=788 initrd=debian-installer/amd64/initrd.gz rescue/enable=true --- quiet

To:

label rescue
        menu label ^Rescue mode
        kernel debian-installer/amd64/linux
        append console=ttyS1,115200n8 initrd=debian-installer/amd64/initrd.gz rescue/enable=true --- quiet
syslinux.cfg

From:

# D-I config version 2.0
# search path for the c32 support libraries (libcom32, libutil etc.)
path debian-installer/amd64/boot-screens/
include debian-installer/amd64/boot-screens/menu.cfg
default debian-installer/amd64/boot-screens/vesamenu.c32
prompt 0
timeout 0

To:

serial 1 115200
console 1
# D-I config version 2.0
# search path for the c32 support libraries (libcom32, libutil etc.)
path debian-installer/amd64/boot-screens/
include debian-installer/amd64/boot-screens/menu.cfg
default debian-installer/amd64/boot-screens/vesamenu.c32
prompt 0
timeout 0
txt.cfg

From:

default install
label install
        menu label ^Install
        menu default
        kernel debian-installer/amd64/linux
        append vga=788 initrd=debian-installer/amd64/initrd.gz --- quiet

To:

default install
label install
        menu label ^Install
        menu default
        kernel debian-installer/amd64/linux
        append console=ttyS1,115200n8 initrd=debian-installer/amd64/initrd.gz --- quiet

Perform the install ^

Connect to the serial-over-LAN and get started. If the server doesn’t have anything currently installed then it should go straight to trying PXE boot. If it does have something on the storage that it would boot then you will have to use F12 at the BIOS screen to convince it to jump straight to PXE boot.

$ ssh ADMIN@192.168.1.22
ADMIN@192.168.1.22's password:
 
ATEN SMASH-CLP System Management Shell, version 1.05
Copyright (c) 2008-2009 by ATEN International CO., Ltd.
All Rights Reserved 
 
 
-> cd /system1/sol1
/system1/sol1
 
-> start
/system1/sol1
press <Enter>, <Esc>, and then <T> to terminate session
(press the keys in sequence, one after the other)
 
Intel(R) Boot Agent GE v1.5.13                                                  
Copyright (C) 1997-2013, Intel Corporation                                      
 
CLIENT MAC ADDR: 0C C4 7A 7C 28 40  GUID: 00000000 0000 0000 0000 0CC47A7C2840  
CLIENT IP: 192.168.2.22  MASK: 255.255.255.0  DHCP IP: 192.168.2.252             
GATEWAY IP: 192.168.2.1
 
PXELINUX 6.03 PXE 20150107 Copyright (C) 1994-2014 H. Peter Anvin et al    
 
 
 
 
 
 
                 ┌───────────────────────────────────────┐
                 │ Debian GNU/Linux installer boot menu  │
                 ├───────────────────────────────────────┤
                 │ Install                               │
                 │ Advanced options                    > │
                 │ Help                                  │
                 │ Install with speech synthesis         │
                 │                                       │
                 │                                       │
                 │                                       │
                 │                                       │
                 │                                       │
                 │                                       │
                 └───────────────────────────────────────┘
 
 
 
              Press ENTER to boot or TAB to edit a menu entry
 
 
  ┌───────────────────────┤ [!!] Select a language ├────────────────────────┐
  │                                                                         │
  │ Choose the language to be used for the installation process. The        │
  │ selected language will also be the default language for the installed   │
  │ system.                                                                 │
  │                                                                         │
  │ Language:                                                               │
  │                                                                         │
  │                               C                                         │
  │                               English                                   │
  │                                                                         │
  │     <Go Back>                                                           │
  │                                                                         │
  └─────────────────────────────────────────────────────────────────────────┘
 
 
 
 
<Tab> moves; <Space> selects; <Enter> activates buttons

…and now the installation proceeds as normal.

At the end of this you should be left with a system that uses ttyS1 for its console. You may need to tweak that depending on whether you want the VGA console also.

Audience tickets for Stewart Lee’s Comedy Vehicle

Last night Jenny and I got the chance to be in the audience for a recording of what will become (some percentage of) four episodes of Stewart Lee’s Comedy Vehicle season 4. Once we actually got in it was a really enjoyable experience, although as usual SRO Audiences were somewhat chaotic with their ticketing procedures.

I’d heard about the chance to get priority audience tickets from the Stewart Lee mailing list, so I applied, but the tickets I got were just their standard ones. From past experience I knew this would mean having to get there really early and queue for ages and still not be sure of getting in, so for most shows on the SRO Audiences site I don’t normally bother. As I particularly like Stewart Lee I decided to persevere this time.

The instructions said they’d be greeting us from 6.20pm, so I decided getting there about an hour early would be a good idea. I know from past experience that they massively over-subscribe their tickets in order to never ever have empty seats. That makes it very difficult to guess how early to be, and I hadn’t been to a Comedy Vehicle recording before either.

The venue was The Mildmay Club in Stoke Newington which was also the venue for all previous recordings of Comedy Vehicle. A bit of a trek from Feltham – train to Richmond then most of the way along the Overground towards Stratford; a good 90 minutes door to door. Nearest station Canonbury but we decided to go early and get some food at Nando’s Dalston first.

We got to the Mildmay Club about 5.25pm and there were already about 15 people queuing outside. Pretty soon the doorman let us in, but only as far as a table just inside the doors where a guy gave us numbered wristbands and told us to come back at 7pm.

This was a bit confusing as we weren’t sure whether that meant we were definitely getting in or if we’d still have to queue (and thus should actually come back before 7). So I asked,

“does the wristband mean we’re definitely getting in?”

“We’ll do our best to get as many people in as we can. We won’t know until 7pm,”

was the non-answer. People piling up behind us and they wanted us out of the way, so off we went.

Having already eaten we didn’t really have anything else to do, so we had a bit of an aimless wander around Newington Green for half an hour or so before arriving back outside the club again, where the queue was now a crowd bustling around the entrance and trailing off in both directions along the street. We decided to get back in the queue going to the right of the club, which was slowly shrinking, with the idea of asking if we were in the right place once we got to the front. All of the people in this queue were yet to collect their wristbands.

Having got to the front of this queue it was confirmed that we should wait around outside until 7pm, though still no idea whether we would get in or by what process this would be decided. We shuffled into the other queue to the left of the club which consisted of people like us who already had wristbands.

While in this queue, we heard calls for various colours of wristband that weren’t ours (white), and eventually all people in front of us had been called into the club. By about quarter past 6 we’d watched quite a large number of people with colourful wristbands get into the building, and we were starting to seriously consider that we might not be getting into this thing, despite the fact that we were amongst the first 15 people to arrive.

At this point a different member of staff came out and told us off for queuing to the left of the club, because

“you’re not allowed to queue past the shops”

and told us to queue to the right with all the other people who still hadn’t got wristbands yet. Various grumblings on the subject of the queue being really long and how will we know what is going on were heard, to which the response was,

“it doesn’t matter where you are, your wristbands are numbered and we’ll call people in number order anyway. You can go away and come back at 7pm if you like. Nothing is happening before 7pm.”

Well, we didn’t have anything else to do for the next 45 minutes anyway, and there was lack of trust that everyone involved was giving us the same/correct information, so we decided to remain in this mostly-linear-collection-of-people-which-was-not-a-queue-because-it-would-be-called-in-number-order.

About 6.55pm a staff member popped their head out the door and shouted,

“we’re delayed by about ten minutes but we do love you and we’ll start getting you inside soon.”

And then just a minute or two later he’s back and shouting out,

“wristband numbers below 510, come this way!”

We were 506 and 507.

The exterior of the Mildmay Club isn’t in the best condition. It looks pretty shabby. Inside though it’s quite nice. We were ushered into the bar area which is pretty much the same as the bar of every working men’s club or British Legion club that you have ever seen.

Even though we were amongst the first few white wristband people in, the room was really full already. These must have been all the priority ticket people we saw going in ahead of us. Nowhere for us to sit except the edge of a low stage directly in front of a speaker pumping out blues and Hendrix. Again we started to worry that we would not be getting in to the recording.

It must have been about 7.20pm when they started calling the colourful wristband people out of the bar and in to the theatre. The room slowly drained until it seemed like there were only about ten of us left. And then,

“white wristbands numbered 508 and below please!”

We rushed into the theatre to be confronted with mostly full seating.

“You want to be sat together don’t you?”

“Yes!”

“Oh, just take those reserved seats, they’ve blown it now, they’re too late.”

Score! I prodded Jenny in the direction of a set of four previously reserved seats that were in a great position. We were amongst the last twenty or so people to get in. I think if we had shown up even ten minutes later to get the wristbands then we wouldn’t have made it.

In contrast to the outside of the building the theatre itself was really quite nice, very interesting decor, and surprisingly large compared to the impression you get from seeing it on television.

Stewart did two sets of 28 minute pieces, then a short interval and then another 2×28 minutes, so almost two hours. I believe there were recordings on three nights so that’s potentially 12 episodes worth of material, but given that

  1. All the previous series had 6 episodes.
  2. Stewart made a comment at one point about moving something on stage for continuity with the previous night’s recording.

then I assume there’s two recordings of each episode’s material from which they’ll edit together the best bits.

The material itself was great, so fans of Comedy Vehicle have definitely got something to look forward to. If you have previously attempted to consume Stewart Lee’s comedy and found the experience unpalatable then I don’t think anything is going to change for you – in fact it might upset you even more, to be honest. Other than that I’m not going to say anything about it as that would spoil it and I couldn’t do it justice anyway.

Oh, apart from that it’s really endearing to see Stew make himself laugh in the middle of one of his own rants and have to take a moment to compose himself.

As for SRO Audiences, I possibly shouldn’t moan as I have no actual experience of trying to cram hundreds of people into a free event and their first concern has got to be having the audience side of things run smoothly for the production, not for the audience. I get that. All I would say is that:

  • Being very clear with people at wristband issuing time that they will be called in by number, and giving a realistic time for when the numbers would be called, would be helpful. This wasn’t clear for us so on the one hand we hung around being in the way a bit, but on the other hand I’m glad that we didn’t leave it until 7pm to come back because our numbers were called before 7pm and we did only just get in.
  • Doing your best to turn people away early when they have no realistic chance of getting in would be good. There were loads of people with higher number wristbands than us that we did not see in the theatre later. Unsure if they got eventually sent home or if they ended up watching the recording on TV in the bar. At previous SRO Audiences recordings I’ve waited right up until show start time to be told to go home though.