Something broke ^
Today I had a look at a customer’s problem. They had a Perl application that connects to a third party API, and as of sometime today it had started failing to connect, although the remote site API still seemed to be responding in general.
The particular Perl module for this service (doesn’t really matter what it was) wasn’t being very verbose about what was going on. It simply said:
Failed to POST to https://api.example.com/api/v1/message.json
I started by writing a small test program using LWP::UserAgent to do a POST to the same URI, and this time I saw:
500 Can’t connect to api.example.com:443 (SSL connect attempt failed with unknown errorerror:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure)
So, it’s failing to do a TLS handshake. But this was working yesterday. Has anything changed? Yes, the remote service was under a denial of service attack today and they’ve just moved it behind a CDN. TLS connections are now being terminated by the CDN, not the service’s own backend.
And oh dear, the customer’s host is Debian squeeze (!) which comes with OpenSSL 0.9.8. This is badly out of date. Neither the OS nor the OpenSSL version is supported for security any more. It needs to be upgraded.
Unfortunately I am told that upgrading the OS is not an option at this time. So can we update Perl?
Well yes, we could build our own Perl reasonably easily. The underlying issue is OpenSSL, though. So it would be an upgrade of:
- LWP, as the app’s HTTP client is using that
It’s not actually that bad though. In fact you do not need to build a whole new Perl, you only need to build OpenSSL, Net::SSLeay and IO::Socket::SSL and then tell Perl (and the system’s LWP) to use the new versions of those.
Of course, everything else on the system still uses a dangerously old OpenSSL, so this is not really a long term way to avoid upgrading the operating system.
Building OpenSSL ^
After downloading and unpacking the latest stable release of OpenSSL, the sequence of commands for building, testing and installing it look like this:
$ ./config --prefix=/opt/openssl \ --openssldir=/opt/openssl \ -Wl,-rpath,'$(LIBRPATH)' $ make $ make test $ sudo make install
The rpath thing is so that the binaries will find the libraries in the alternate path. If you were instead going to add the library path to the system’s ld.so.conf then you wouldn’t have to have that bit, but I wanted this to be self-contained.
When I did this the first time, all the tests failed and at the install step it said:
ar: /opt/openssl/lib/libcrypto.so: File format not recognized
This turned out to be because the system’s Text::Template Perl module was too old. Version 1.46 or above is required, and squeeze has 1.45.
Installing a newer Text::Template ^
So, before I could even build OpenSSL I needed to install a newer Text::Template. Cpanminus to the rescue.
$ sudo mkdir /opt/perl $ cd /opt/perl $ sudo cpanm --local-lib=./cpanm Text::Template
That resulted in me having a newer Text::Template in /opt/perl/cpanm/lib/perl5/. So to make sure every future invocation of Perl used that:
$ export PERL5LIB=/opt/perl/cpanm/lib/perl5/ $ perl -e 'use Text::Template; print $Text::Template::VERSION,"\n";' 1.58
Repeating the OpenSSL build steps from above then resulted in an OpenSSL install in /opt/openssl that passed all its own tests.
Installing newer Net::SSLeay and IO::Socket::SSL ^
Cpanminus once again comes to the rescue, with a twist:
$ cd /opt/perl $ OPENSSL_PREFIX=/opt/openssl cpanm --local-lib=./cpanm Net::SSLeay IO::Socket::SSL
The OPENSSL_PREFIX is part of Net::SSLeay’s build instructions, and then IO::Socket::SSL uses that as well.
Using the result ^
Ultimately the customer’s Perl application needed to be told to use these new modules. This could be done with either the PERL5LIB environment variable or else by putting:
use lib '/opt/perl/cpanm/lib/perl5';
At the top of the main script.
The application was then once more able to talk TLS to the CDN and it all worked again.
Other recommendations ^
The customer could maybe consider putting the application into a container on a new install of the operating system.
That way, the kernel and whole of the OS would be modern and supported, but just this application would be running with a terribly outdated userland. Over time, more of the bits inside the container could be moved out to the modern host (or another container), avoiding having to do everything at once.