The ongoing struggle

This weekend I attended London Hackspace’s Arduino For Beginners workshop.

Background

I’ve been a member of Hackspace for quite some time, though I have very little hands on experience with electronics or making things or anything like that, and have up until now tended to only use the Hackspace occasionally for somewhere to work from. When the Arduino workshop was announced I thought it sounded not only very good value for money at £80 for a two day instructor-led course, but also would be a great way to broaden my skills.

In the interest of full disclosure, as a minor fanboy of the Hackspace I am prone to want to portray it and all its endeavours in a good light. However I am trying to write this from the point of view of a paying customer of the workshop (which I was), and will try to be fair and objective.

What we got

Included in the £80 (£100 for non-members) workshop fee was an Earthshine Electronics Arduino compatible starter kit worth about £40, containing a DFRduino Duemilanove and over 150 components for use in the various tutorials.

Day one

I arrived early on the first day expecting to touch in with my Oyster card (Hackspace’s members can access the space 24/7 using any compatible RFID card), but the door was already open and the place was a hive of activity. Inside I found Andy “Bob” Brockhurst, Charles, Neil, Mike from Earthshine, Adrian and several others busy tidying the place up making it usable for a workshop. This included attaching a shelf to one wall for the projector to sit on!

Presentation

Eventually the projector was sorted out, a rather makeshift but legible screen put up for it and things began with a presentation from Bob. Bob’s presentation explained what an Arduino is, the sorts of things they might be used for, the concepts behind “open source hardware”, the available software and some resources for getting more information and hardware. Bob’s presentation is available on his github.

Arduino IDE

Once the presentation was concluded, Bob, Mike, Adrian et al went around ensuring we all had the Arduino IDE installed. This is a piece of Java software comprising a basic editor for the C-like language Arduino is programmed with, together with facilities to compile and upload the code to the board, and a few other features.

There was also a brief diversion into Fritzing, which is another piece of open source software used for producing layouts and prototypes of circuits for sharing with others, ease of manufacturing, etc. Fritzing is not necessary to use the Arduino, but was used to display the board layouts for the examples.

Amongst the delegates there was a mix of Windows, Mac and Linux laptops but we were all eventually able to download and install the IDE. The supplied USB cable once connected to an Arduino board appears as a serial port on the computer, and the IDE just has to be told which board model you have and which serial port to use.

I had a bit of a false start with installing the IDE on Linux, owing to me not reading the documentation. It is necessary on Ubuntu to already have installed gcc-avr and avr-libc otherwise the IDE will be unable to compile your sketch to object code. I think I was in the minority using Linux; it seems Windows is preferred amongst AVR coders. The usual Java blah is needed to run the IDE, but I already had that.

Refreshments

I’m not sure who prepared the refreshments; I know that Clare, Charles’s other half had some hand in it but I don’t know if she was solely responsible. In any case, tea, coffee, cordial, biscuits and flapjacks were provided throughout both days, with sandwiches (two different cheeses, tuna, egg, bacon), crisps and fruit for lunch.

Walk-through of examples

Once everyone had their IDE up and running and had confirmed that the IDE could talk to their Arduino, Bob went through some simple example circuits. While Bob put the circuits up on the projector and explained how they worked, both electronically and software-wise, Mike, Neil and other knowledgeable Arduino hackers moved around the room helping people out where necessary.

Bob’s examples built on each other progressively, and included:

• Making an LED blink
• Simulating a traffic light sequence with 3 LEDs
• Adding a button to simulate a pedestrian crossing feature
• Adding a buzzer which sounds when safe to cross

There was also a final example which used a motor as a spinner to simulate an accessible pedestrian crossing, but most people did not attempt this due to the risks of damaging the Arduino board.

Sample code and Fritzing files available on Bob’s github.

Experimentation

The last few hours of the day were taken at our own individual speeds, practising with the examples and seeing what we could get working. Some people surged ahead, completing all examples and then moving on to their own ideas, whereas others took longer or needed some assistance. As far as I could see everyone made their way through the first four examples.

One person’s Arduino was damaged towards the end of day one and was replaced by Mike.

Wrap up of day one

Some time between 4 and 5pm most people started to pack up and we were reminded to try to think of a personal project to work on the next day if we had not done so already.

Day two

Door debacle

I arrived just before 10am the next day owing to general Sunday public transport fail, Jubilee line closure etc., again expecting to swipe my Oyster and again being confounded. This time however it was by a group of people standing outside the Hackspace.

It transpired that we had perhaps tried to be too clever for our own good and the door computer had crashed some time in the night, preventing all of us from getting in. After about 30 minutes, some IRCing, unsuccessful nmapping and leaving of voicemails, Mark popped up on IRC and kindly rushed over to let us in with a real metal key.

Projects

For the rest of this day we all worked on our own individual ideas, with Bob, Neil, Adrian and others milling around dispensing masses of assistance.

I couldn’t really think of anything practically useful to work on so decided I just wanted to push the limits of what I’d already learned. I’d spied the 8×8 matrix display in the kit box and wanted to play with that, but this first would involve learning how to drive a shift register.

The 8×8 matrix has 64 LEDs (actually 128 since each one is either red or green), and connecting up 64 outputs would not only be extremely tedious, the Arduino doesn’t have that many outputs anyway. The component actually has only 16 pins but even so, 16 outputs is too many. A shift register helps solve this by allowing the control of 8 outputs using only 3 pins.

The basic idea is that you use one pin to tell the shift register when you’re ready to start or finish feeding it data, one pin to tell it there is a new piece of data to read, and a third pin for the data itself. After you’ve indicated that you’re finished, all the data you’ve fed in will be output. The 74HC595 shift registers in the kit box have 8 outputs each.

In order to get a working knowledge of the shift register I decided to first try using one to drive 8 LEDs, first to make them count in binary and then to display various patterns. After that I would go on to try the shift registers with the matrix display.

I put a circuit together by following the commentary in one of the examples in Earthshine’s Complete Beginners Guide to the Arduino. Unfortunately I couldn’t get it to produce sensible results. At times it seemed like I almost had it, with it counting up in binary correctly except for a couple of values. Other times it was just completely messed up with the LEDs cycling through all kinds of patterns.

I found debugging this to be very difficult. Dammit Jim, I’m a sysadmin, not a hardware hacker; when I write awful sysadmin Perl I at least have some idea how to debug it! After a couple of hours of frustration I had to ask Bob for help, and after a good while of him checking everything I’d done several times, he was stumped too.

Neil then wandered over and measured the frequencies the Arduino was producing, confirming that my code and the Arduino were working correctly, narrowing the breakage down to my circuit. He got his magnifying glass out, we got the datasheet for the shift register, and after a short while he noticed that I was trying to use the shift register’s latch pin as a clock and vice versa.

It turned out that while the pinout of the shift register was correct in the Earthshine guide, the commentary in the guide had switched around the purpose of the shift register clock input pin and the storage register clock input pin, which accounted for the bizarre behaviour I was seeing. It worked! A similar example on the Arduino site has things the correct way around.

I went ahead and wrote the code to do binary counting and then to put some simple patterns on the LEDs, but after all that I felt a bit too burnt out to go on to the matrix display and decided to quit while I was ahead. I definitely feel confident enough to tackle this on my own now though.

• Code
• Breadboard

I think two more people’s Arduinos were damaged on the second day.

Conclusion

Wins

This workshop was indeed incredible value for money. I think the components, presentation and walk through of examples alone were easily worth £80 and could have been done in one day. What was worth so much more was having knowledgeable people in the same room as you for two days, personally guiding you through the examples and discussing your own ideas. I’ve paid 5 times as much for one day courses that were nowhere near as good.

This could and probably should have justified a doubling or more of price, but I can understand that this was the first workshop put on in the Hackspace and there was a desire to sell it out and gain some experience from the host side.

I hope the workshop is repeated. If it is, and you’re interested in this sort of thing, you should book it quickly.

Suggestions for improvement

• Many Bothans diedTwo or three Arduinos died during the course of the workshop. As far as I could tell through overhearing conversation, every case was related to running them from the power supply instead of the USB power. I think someone put a multimeter on one and found it was supplying over voltage. I understand better PSUs will be sought in future.
• We struggled to fit everyone in. The last two late-comers had less than ideal seating during the presentation because Bob needed to stand in one of the seating places in order to do the presentation. This could probably be easily remedied by tidying up the Hackspace a bit before the day of the workshop. I don’t think there’s any need to reduce the number of participants.
• The screen for the projector was rather tatty and ramshackle. It didn’t matter much for Bob’s presentation because his slides were very simple and just about fitted on the screen, but anything more demanding than that would have real problems.
• The door episode was extremely unfortunate timing. A suggestion was made that at least two of the people running any workshop should have keys in case of similar emergencies. Sounds reasonable.

It looks like the Hackspace are conducting a debrief and some of these issues have already been raised, so it should be easy to improve for next time.

Benjamin wrote:

“I have learnt to minimize the amount of friends I have who are vegetarian, religious or have extreme views about something. If I didn’t, I’d probably be so depressed from being lectured and told off all the time.”

Benjamin,

If you have “friends” who are lecturing or preaching to you, I don’t think they’re really your friends. Even if they’re right and you’re very wrong. If someone continually brings a subject up even when they know you aren’t interested in talking about it, then I think they’re doing so more for their own purposes than yours.

I think it’s the case that almost everyone holds “extreme” views about some topic or other, but most don’t feel the need to bring them up. You singled out vegetarianism and religion, but anything can be a hot topic for someone.

Friends might need to do some sort of reality check or intervention on each other from time to time, and of course debate is good too. But there really is no need for frequent lecturing when amongst friends, I believe. It would have been much better if you had instead said, “I have learnt to disassociate myself from people who lecture me” rather than explicitly mentioning vegetarians, people who have faith, free software zealots, … by calling out these groups you unfortunately make yourself look like a troll who is lecturing.

You probably have more friends than you think who are religious or vegetarian, and you didn’t even know.

I’m looking for a free, open source web-based issue tracker that I can install and run. I would like it to be able to authenticate off LDAP and allow users to rate the importance of the fixing of each issue, as well as the usual commenting.

I already run Request Tracker and don’t really want to adapt that to do it, I’d prefer something simpler.

Any recommendations? Can trac do it?

Thanks!

Update:

davee pointed out that more info is probably needed.

It’s for feature requests for various bitfolk.com services. There’s lots of them outstanding, and they’re all good, but I have no idea which ones are in most demand from the customers. So I’d like:

• for users to be able to submit ideas by web and possibly email
• rate how much they want them implemented
• for further discussion to be possible inside the issue tracker (web/email)

I know RT can be made to do it, but I currently have it set up for mostly individual customers to raise issues about their individual service, not for everyone to comment on stuff.

Recap

Back in part 1 I discussed what entropy is as far as Linux is concerned, why I’ve started to look in to entropy as it relates to a Linux/Xen-based virtual hosting platform, how much entropy I have available, and how this might be improved.

If you didn’t read that part yet then you might want to do so, before carrying on with this part.

As before, click on any graph to see the full-size version.

Hosting server with an Entropy Key

Recently I colocated a new hosting server so it seemed like a good opportunity to try out the Entropy Key at the same time. Here’s what the available entropy looks like whilst ekeyd is running.

First impressions are, this is pretty impressive. It hovers very close to 4096 bytes at all times. There is very little jitter.

Trying to deplete the entropy pool, while using an Entropy Key

As per Hugo’s comment in part 1, I tried watch -n 0.25 cat /proc/sys/kernel/random/entropy_avail to see if I could deplete the entropy pool, but it had virtually no effect. I tried with watch -n 0.1 cat /proc/sys/kernel/random/entropy_avail (so every tenth of a second) and the available entropy fluctuated mostly around 4000 bytes with a brief dip to ~3600 bytes:

In the above graph, the first watch invocation was at ~1100 UTC. The second one was at ~1135 UTC.

Disabling the Entropy Key

Unfortunately I forgot to get graphs of urquell before the ekeyd was started, so I have no baseline for this machine.

I assumed it would be the same as all the other host machines, but decided to shut down ekeyd to verify that. Here’s what happened.

The huge chasm of very little entropy in the middle of this graph is urquell running without an ekeyd. At first I was at a loss to explain why it should only have ~400 bytes of entropy by itself, when the other hosting servers manage somewhere between 3250 and 4096 bytes.

I now believe that it’s because urquell is newly installed and has no real load. Looking into how modern Linux kernels obtain entropy, it’s basically:

• keyboard interrupts;
• mouse interrupts;
• other device driver interrupts with the flag IRQF_SAMPLE_RANDOM.

Bear in mind that headless servers usuallly don’t have a mouse or keyboard attached!

You can see which other drivers are candidates for filling up the entropy pool by looking where the IRQF_SAMPLE_RANDOM identifier occurs in the source of the kernel:

http://www.cs.fsu.edu/~baker/devices/lxr/http/ident?i=IRQF_SAMPLE_RANDOM

(as an aside, in 2.4.x kernels, most of the network interface card drivers had IRQF_SAMPLE_RANDOM and then they all got removed through the 2.6.x cycle since it was decided that IRQF_SAMPLE_RANDOM is really only for interrupts that can’t be observed or tampered with by an outside party. That’s why a lot of people reported problems with lack of entropy after upgrading their kernels.)

My hosting servers are typically Supermicro motherboards with Intel gigabit NICs and 3ware RAID controller. The most obvious device in the list that could be supplying entropy is probably block/xen-blkfront since there’s one of those for each block device exported to a Xen virtual machine on the system.

To test the hypothesis that the other servers are getting entropy from busy Xen block devices, I shut down ekeyd and then hammered on a VM filesystem:

The increase you see towards the end of the graph was while I was hammering the virtual machine’s filesystem. I was able to raise the available entropy to a stable ~2000 bytes doing this, so I’m satisfied that if urquell were as busy as the other servers then it would have similar available entropy to them, even without the Entropy Key.

Feeding entropy to other hosts

ekeyd by default feeds entropy from the key directly into the Linux kernel of the host it’s on, but it can be configured to listen on a Unix or TCP socket and mimic the egd protocol. I set it up this way and then put an instance of HAProxy into a VM with my ekeyd as a back end. So at this point I had a service IP which would talk egd protocol, and client machines could use to request entropy.

On the client side, ekeyd-egd-linux can be found in Debian lenny-backports and in Debian squeeze, as well as Ubuntu universe since Jaunty. This daemon can read from a Unix or TCP socket using the egd protocol and will feed the received entropy into the Linux kernel.

I took a look at which of my VMs had the lowest available entropy and installed ekeyd-egd-linux on them, pointing it at my entropy service IP:

Success!

Where next?

• Get some customers using it, explore the limits of how much entropy can be served.
• Buy another Entropy Key so that it doesn’t all grind to a halt if one of them should die.
• Investigate a way to get egd to read from another egd so I can serve the entropy directly from a VM and not have so many connections to my real hardware. Anyone interested in coding that?
• Monitor the served entropy both for availability and for quality.

A while back, a couple of BitFolk customers mentioned to me that they were having problems running out of entropy.

A brief explanation of entropy as it relates to computing

Where we say entropy, we could in layman’s terms say “randomness”. Computers need entropy for a lot of things, particularly cryptographic operations. You may not think that you do a lot of cryptography on your computer, and you personally probably don’t, but for example every time you visit a secure web site (https://…) your computer has to set up a cryptographic channel with the server. Cryptographic algorithms generally require a lot of random data and it has to be secure random data. For the purposes of this discussion, “secure” means that an attacker shouldn’t be able to guess or influence what the random data is.

Why would an attacker be able to guess or influence the random data if it is actually random? Because it’s not actually random. The computer has to get the data from somewhere. A lot of places it might be programmed to get it from may seem random but potentially aren’t. A silly implementation might just use the number of seconds the computer has been running as a basis for generating “random” numbers, but you can see that an attacker can guess this and may even be able to influence it, which could weaken any cryptographic algorithm that uses the “random” data.

Modern computers and operating systems generate entropy based on events like electrical noise, timings of data coming into the computer over the network, what’s going on with the disks, etc. fed into algorithms — what we call pseudo-random number generators (PRNGs). A lot of data goes in and a relatively small amount of entropy comes out, but it’s entropy you should be able to trust.

That works reasonably well for conventional computers and servers, but it doesn’t work so well for virtual servers. Virtual servers are running in an emulated environment, with very little access to “real” hardware. The random data that conventional computers get from their hardware doesn’t happen with emulated virtual hardware, so the prime source of entropy just isn’t present.

When you have an application that wants some entropy and the system has no more entropy to give, what usually happens is that the application blocks, doing nothing, until the system can supply some more entropy. Linux systems have two ways for applications to request entropy: there’s /dev/random and /dev/urandom. random is the high-quality one. When it runs out, it blocks until there is more available. urandom will supply high-quality entropy until it runs out, then it will generate more programmatically, so it doesn’t block, but it might not be as secure as random. I’m vastly simplifying how these interfaces work, but that’s the basic gist of it.

What to do when there’s no more entropy?

If you’re running applications that want a lot of high-quality entropy, and your system keeps running out, there’s a few things you could do about it.

Nothing

So stuff slows down, who cares? It’s only applications that want high-quality entropy and they’re pretty specialised, right?

Well, no, not really. If you’re running a busy site with a lot of HTTPS connections then you probably don’t want it to be waiting around for more entropy when it could be serving your users. Another one that tends to use all the entropy is secure email – mail servers talking to each other using Transport Layer Security so the email is encrypted on the wire.

Use real hosting hardware

Most of BitFolk’s customers are using it for personal hosting, this problem is common to virtual hosting platforms (it’s not a BitFolk-specific issue), and BitFolk doesn’t provide dedicated/colo servers, so arguably I don’t need to consider this my problem to fix. If the customer could justify greater expense then they could move to a dedicated server or colo provider to host their stuff.

Tell the software to use urandom instead

In a lot of cases it’s possible to tell the applications to use urandom instead. Since urandom doesn’t block, but instead generates more lower-quality entropy on demand, there shouldn’t be a performance problem. There are obvious downsides to this:

• If the application author wanted high-quality entropy, it might be unwise to not respect that.
• Altering this may not be as simple as changing its configuration. You might find yourself having to recompile the software, which is a lot of extra work.

You could force this system-wide by replacing your /dev/random with /dev/urandom.

Customers could get some more entropy from somewhere else

It’s possible to feed your own data into your system’s pseudo-random number generator, so if you have a good source of entropy you can help yourself. People have used some weird and wonderful things for entropy sources. Some examples:

• A sound card listening to electro-magnetic interference (“static”).
• A web camera watching a lava lamp.
• A web camera in a dark box, so it just sees noise on its CCD.

The problem for BitFolk customers of course is that all they have is a virtual server. They can’t attach web cams and sound cards to their servers! If they had real servers then they probably wouldn’t be having this issue at all.

BitFolk could get some entropy from somewhere else, and serve it to customers

BitFolk has the real servers, so I could do the above to get some extra entropy. I might not even need extra entropy; I could just serve the entropy that the real machines have. If it wasn’t for the existence of the Simtec Electronics Entropy Key then that’s probably what I’d be trying.

I haven’t got time to be playing about with sound cards listening to static, webcams in boxes and things like that, but buying a relatively cheap little gadget is well within the limit of things I’m prepared to risk wasting money on.

Customers would need to trust my entropy, of course. They already need to trust a lot of other things that I do though.

Entropy Key

Entropy Keys are very interesting little gadgets and I encourage you to read about how they work. It’s all a bit beyond me though, so for the purposes of this series of blog posts I’ll just take it as read that you plug in an Entropy Key into a USB port, run ekeyd and it feeds high quality entropy into your PRNG.

I’d been watching the development of the Entropy Key with interest. When they were offered for cheap at the Debian-UK BBQ in 2009 I was sorely tempted, but I knew I wasn’t going to be able to attend, so I left it.

Then earlier this year, James at Jump happened to mention that he was doing a bulk order (I assume to fix this same issue for his own VPS customers) if anyone wanted in. Between the Debian BBQ and then I’d had a few more complaints about people running out of entropy so at ~£30 each I was thinking it was definitely worth exploring with one of them; perhaps buy more if it works.

How much entropy do I have anyway?

Before stuffing more entropy in to my systems, I was curious how much I had available anyway. On Linux you can check this by looking at /proc/sys/kernel/random/entropy_avail. I think this value is in bytes, and tops out at 4096. Not hard to plug this in to your graphing system.

Click on the following images to see the full-size versions.

Typical host server, no Entropy Key

Here’s what some typical BitFolk VM hosting servers have in terms of available entropy.

That’s pretty good. The available entropy hovers close to 4096 bytes all the time. It’s what you’d expect from a typical piece of computer hardware. The weekly view shows the small jitter:

The lighter pink area is the highest 5-minute reading in each 30 minute sample. The dark line is the lowest 5-minute reading. You can see that there is a small amount of jitter where the available entropy fluctuates between about 3250 and 4096 bytes.

Here’s a couple of the other host servers just to see the pattern:

No surprises here; they’re all much the same. If these were the only machines I was using then I’d probably decide that I have enough entropy.

Typical general purpose Xen-based paravirtualised virtual machine

Here’s a typical general purpose BitFolk VPS. It’s doing some crypto stuff, but there’s a good mix of every type of workload here.

These graphs are very different. There’s much more jitter and a general lack of entropy to begin with. Still, it never appears to reach zero (although it’s important to realise that these graphs are at best 5-minute averages, so the minimum and maximum values will be lower and higher within that 5-minute span) so there doesn’t seem to be a huge problem here.

Virtual machines with more crypto

Here’s a couple of VMs which are doing more SSL work.

This one has a fair number of web visitors and they’re all HTTPS. You can see that it’s even more jittery, and spends most of its time with less than 1024 bytes of entropy available. It goes as low as ~140 bytes from time to time, and because of the 5-minute sampling it’s possible that it does run out.

Again, this one has some HTTPS traffic and is faring worse for entropy, with an average of only ~470 bytes available. I ran a check every second for several hours and available entropy at times was as low as 133 bytes.

Summary so far

BitFolk doesn’t have any particularly busy crypto-heavy VMs so the above was the best I could do. I think that I’ve shown that virtual machines do have less entropy generally available, and that a moderate amount of crypto work can come close to draining it.

Based on the above results I probably wouldn’t personally take any action since it seems none of my own VMs run out of entropy, although I am unsure if the 133 bytes I measured was merely as low as the pool is allowed to go before blocking happens. In any case, I am not really noticing poor performance.

Customers have reported running out of entropy though, so it might still be something I can fix, for them.

Where next?

Next:

• See what effect using an Entropy Key has on a machine’s available entropy.
• Assuming it has a positive effect, see if I can serve this entropy to other machines, particularly virtual ones.
• Can I serve it from a virtual machine, so I don’t have customers interacting with my real hosts?
• Does one Entropy Key give enough entropy for everyone that wants it?
• Can I add extra keys and serve their entropy in a highly-available fashion?

Those are the things I’ll be looking into and will blog some more about in later parts. This isn’t high priority though so it might take a while. In the meantime, if you’re a BitFolk customer who actually is experiencing entropy exhaustion in a repeatable fashion then it’d be great if you could get in touch with me so we can see if it can be fixed.

In part 2 of this series of posts I do get the key working and serve entropy to my virtual machines.

A nice email from Tuscany Networks in my inbox the other day:

Date: Mon, 26 Apr 2010 13:19:36 +0000
From: marketing
To: <elided>@bitfolk.com
Subject: DNSSEC- Become Compliant
Reply-To: marketing@tuscanynetworks.com

Your email client cannot read this email. To view it online, please go
here:
<URL elided>

To stop receiving these
emails:<URL elided>

So what happened here? They sent me a marketing email that they obviously considered too whizzy and shiny to allow me to read a plain text version of, so they inserted a plain text version that just says that my email client can’t read it. There actually is a HTML version and my mail client can probably read it fine, if I chose to ask it to, but since Tuscany Networks can’t work out how to send email properly I haven’t bothered looking.

Dear Tuscany Networks,

I would suggest that if you want to sell me on your DNSSEC knowledge then first you should try not making assumptions about what my email client supports.

Next you might like to try just putting a sensible text version of whatever your HTML was, since that would actually go beyond the bare minimum level of competency and start to approach actual usefulness.

Finally you might consider ditching the HTML entirely, since you got my address from a technical presentation on DNSSEC that was hosted by Nominet and I doubt flashy HTML emails go down all that well with the sort of people present. Save it for your fellow marketroids, who are more adept at finding ways to make each other’s utter shite show up in a different and annoying new way in Outlook Express than they are at deploying a secure DNS infrastructure.

No love,
Andy

PS Thanks for the unsubscribe link though; my mail client was still able to follow it despite not being good enough for the rest of your work, so that’s a problem that neither of us should have to face in the future.

I’ve been following a thread on NANOG about why the first versions of the Internet Protocol supported only a maximum of 256 different networks.

Back then, every organisation on the fledgling Internet got a range of IP addresses starting with a digit 0-255 and used the next three digits to number their hosts. eg. 192.168.3.4. That’s 2²⁴-2 (16,777,214) possible host addresses. When IP address classes were introduced that was known as “class A”, and today we’d call that a /8. A pretty big range of IP addresses by today’s standards.

With the impending exhaustion of IPv4 addresses, some people are looking at these /8 networks — many of which are no longer publicly in use or are only seen to have a few reachable addresses — and asking how come these organisations were ever allowed to have such a large allocation. I never really thought about it before, but for some of the older ones the answer is that there was no choice back then. An allocation was 8 bits of network and 24 bits of hosts.

I particularly enjoyed reading a contribution on the matter from Vint Cerf, an Internet legend:

Date: Sat, 3 Apr 2010 08:17:28 -0400
From: Vint Cerf

When the Internet design work began, there were only a few fairly large networks around. ARPANET was one. The Packet Radio and Packet Satellite networks were still largely nascent. Ethernet had been implemented in one place: Xerox PARC. We had no way to know whether the Internet idea was going to work. We knew that the NCP protocol was inadequate for lossy network operation (think: PRNET and Ethernet in particular). This was a RESEARCH project. We assumed that national scale networks were expensive so there would not be too many of them. And we certainly did not think there would be many built for a proof of concept. So 8 bits seemed reasonable. Later, with local networks becoming popular, we shifted to the class A-D address structure and when class B was near exhaustion, the NSFNET team (I think specifically Hans-Werner Braun but perhaps others also) came up with CIDR and the use of masks to indicate the size of the “network” part of the 32 bit address structure. By 1990 (7 years after the operational start of the Internet and 17 years since its basic design), it seemed clear that the 32 bit space would be exhausted and the long debate about IPng that became IPv6 began. CIDR slowed the rate of consumption through more efficient allocation of network addresses but now, in 2010, we face imminent exhaustion of the 32 bit structure and must move to IPv6.

Part of the reason for not changing to a larger address space sooner had to do with the fact that there were a fairly large number of operating systems in use and every one of them would have had to be modified to run a new TCP and IP protocol. So the “hacks” seemed the more convenient alternative. There had been debates during the 1976 year about address size and proposals ranged from 32 to 128 bit to variable length address structures. No convergence appeared and, as the program manager at DARPA, I felt it necessary to simply declare a choice. At the time (1977), it seemed to me wasteful to select 128 bits and variable length address structures led to a lot of processing overhead per packet to find the various fields of the IP packet format. So I chose 32 bits.

vint

There is a reason that Vint Cerf is often called “Father of the Internet”. It’s amazing to me to think that they honestly did not know back then that this Internet thing was going to be all that popular.

On this day we reflect that one or more hard disk sectors may have died for your sins; the Debian mdadm checkarray script was created so that your array may have constant life. Debian loves you. Amen.

I’ve got this server in my loft at home that’s mainly a file server for the data we use/view/listen to here. It looks like this:

A bit of a beast. When I bought it over 4 years ago I somehow thought I’d be adding a lot more drives. Anyway.

It’s been a good, reliable bit of kit and had no problems for a long time apart from overheating in the old house, but that was a problem with the room it was in. It’s never even lost a disk. A couple of months ago though the PSU went pop and ever since then it started occasionally giving me this sort of thing:

Mar 21 13:53:16 specialbrew kernel: [5875576.400044] ata3.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Mar 21 13:53:16 specialbrew kernel: [5875576.400095] ata3.01: cmd c8/00:50:9e:a2:1d/00:00:00:00:00/f2 tag 0 dma 40960 in
Mar 21 13:53:16 specialbrew kernel: [5875576.400098]          res 40/00:01:01:4f:c2/00:00:00:00:00/10 Emask 0x4 (timeout)
Mar 21 13:53:16 specialbrew kernel: [5875576.400167] ata3.01: status: { DRDY }
Mar 21 13:53:16 specialbrew kernel: [5875576.400196] ata3: soft resetting link
Mar 21 13:53:16 specialbrew kernel: [5875576.719196] ata3.00: configured for UDMA/33
Mar 21 13:53:16 specialbrew kernel: [5875576.759036] ata3.01: configured for UDMA/100
Mar 21 13:53:16 specialbrew kernel: [5875576.759075] ata3: EH complete
Mar 21 13:53:16 specialbrew kernel: [5875576.800851] sd 2:0:0:0: [sdc] 625134827 512-byte hardware sectors (320069 MB)
Mar 21 13:53:16 specialbrew kernel: [5875576.801386] sd 2:0:0:0: [sdc] Write Protect is off
Mar 21 13:53:16 specialbrew kernel: [5875576.801418] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
Mar 21 13:53:16 specialbrew kernel: [5875576.808855] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Mar 21 13:53:16 specialbrew kernel: [5875576.810058] sd 2:0:1:0: [sdd] 625134827 512-byte hardware sectors (320069 MB)
Mar 21 13:53:16 specialbrew kernel: [5875576.810452] sd 2:0:1:0: [sdd] Write Protect is off
Mar 21 13:53:16 specialbrew kernel: [5875576.810482] sd 2:0:1:0: [sdd] Mode Sense: 00 3a 00 00
Mar 21 13:53:16 specialbrew kernel: [5875576.867347] sd 2:0:1:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Mar 21 13:53:16 specialbrew kernel: [5875576.871943] sd 2:0:0:0: [sdc] 625134827 512-byte hardware sectors (320069 MB)
Mar 21 13:53:16 specialbrew kernel: [5875576.873744] sd 2:0:0:0: [sdc] Write Protect is off
Mar 21 13:53:16 specialbrew kernel: [5875576.873770] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
Mar 21 13:53:16 specialbrew kernel: [5875576.873966] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Mar 21 13:53:16 specialbrew kernel: [5875576.874062] sd 2:0:1:0: [sdd] 625134827 512-byte hardware sectors (320069 MB)
Mar 21 13:53:16 specialbrew kernel: [5875576.874125] sd 2:0:1:0: [sdd] Write Protect is off
Mar 21 13:53:16 specialbrew kernel: [5875576.874148] sd 2:0:1:0: [sdd] Mode Sense: 00 3a 00 00
Mar 21 13:53:16 specialbrew kernel: [5875576.874195] sd 2:0:1:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

There’s 6 drives in there and the above messages have been seen referring to all of them at one time or another, so I don’t believe it’s as simple as a broken disk.

The incidences of the above have become more and more frequent, so today I spent some time trying to work out where the problem lay.

The way it seemed to affect all ATA busses made me think maybe the (new) PSU was underperforming, but I tried two different ones and they seem fine.

The six disks are inserted into two 3-bay Icydocks. Here’s what they look like:

They’re pretty dumb devices which just let you fit three 3.5″ disks into two 5.25″ bays. On the back they have three SATA data connectors (one for each disk), two molex power, one SATA power and a fan. I bought them because I didn’t want to buy a really expensive disk chassis for home, but I also didn’t want to screw six drives inside the case where they’d be hard to get access to.

Inside I have four of the drives connected to the motherboard’s SATA controller, and two of them connected to an additional Si3112 SATA card. This setup has been in place for over four years.

When all the drives are removed from the Icydocks and directly connected to SATA and power, everything appears to be fine. When either of the Icydocks have three disks in, the problem reappears. I then put three disks in an Icydock, three disks directly connected, but popped one of the disks in the Icydock out. This appears to also work fine (the file systems are all RAID-10 so can stand to run with one disk missing).

I’m a bit confused by that. When I was testing the Icydocks individually, I was using the same set of three disks with each one (with the other three disks connected directly). I could believe that the disk I have now removed is bad in some way that causes the whole bus to reset, but I would have to ask why it affects the other busses, and why it doesn’t happen when it’s directly connected.

I know other people who bought Icydocks and had a real struggle getting them to behave reliably, but mine worked well from the start and have done for over four years. I could believe that one of them went bad when the power popped, even though they are very simple electro-mechanical devices, but it’s hard to believe that two of them did.

I can’t just remove the Icydocks from the picture and forget about it because that leaves six SATA drives running on the floor. They need to be inside some form of enclosure, and I don’t want to fork out for a new enclosure or two right now if I can help it.

I’ve left it there for this evening, but I’ll have to return to it tomorrow afternoon. I’ll probably start by putting the other three disks back in their Icydock to see if the removal of that one really does fix it.

Any ideas for ways to narrow the problem down?

I hate hardware.

Update 2010-03-31

I tentatively believe I’ve tracked down the issue.

Joel wins: despite the new PSU being a bit beefier in max output than the dead one I was replacing (500W vs 384W), the new one actually had a lower limit on the 12V rail: 2.5A vs the previous 3.3A.

I scavenged a PSU from elsewhere that also had 3.3A and everything seems fine now and has been for 2 days.

I think that things worked fine outside the Icydocks because the Icydocks have fans, which are probably not very good, and suck additional power. Or else they maybe don’t do any kind of staggered spinup that might happen without them.

We’re quite enjoying Burn Notice at the moment.

There’s something strangely familiar about Gabrielle Anwar. I looked at her IMDB and nothing really stood out (except possibly Things to Do in Denver When You’re Dead).

And then I got to Press Gang. Sam Black!

The other one I should really have known, but must admit I didn’t even consider until I noticed his name in the credits. How could I have failed to recognise Bruce Campbell!? I suppose there’s not enough zombies in Miami. Is that why his character is called Sam Axe?