Month: April 2007

haproxy bafflement

Posted on by Andy

Update 2007-05-05: I sent some strace output to the author of haproxy, Willy Tarreau, and he replied within 24 hours with a full annotation of the strace and a one line patch to fix this issue. That’s what I call support! Here’s his comments and the patch:

I think this is caused by the fact that the end of connection from the client was received BEFORE the connection even established to the server, and when the connection status is checked, there is nothing anymore in the buffer because all data was just sent at once.

Could you please apply the following patch (against 1.2.17) and check that it fixes your problem (simply do “patch -p1” on the mail) ?

diff --git a/haproxy.c b/haproxy.c
index 8e57700..357a37a 100644
--- a/haproxy.c
+++ b/haproxy.c
@@ -5589,7 +5589,7 @@ int process_srv(struct session *t) {
     else if (s == SV_STCONN) { /* connection in progress */
        if (c == CL_STCLOSE || c == CL_STSHUTW ||
            (c == CL_STSHUTR &&
-            (t->req->l == 0 || t->proxy->options & PR_O_ABRT_CLOSE))) { /* give up */
+            ((t->req->l == 0 && t->res_sw == RES_SILENT) || t->proxy->options & PR_O_ABRT_CLOSE))) { /* give up */
            tv_eternity(&t->cnexpire);
            fd_delete(t->srv_fd);
            if (t->srv)

Original problem is described below…

I’m trying to use haproxy to load balance three spamassassin spamd servers. spamd uses a plain text TCP protocol so in theory it should be simple, but I’m getting intermittent connection problems. Here’s my config:

global
        log 127.0.0.1 local0 debug
        maxconn 100
        ulimit-n 512
        uid 999
        gid 999
        daemon
        pidfile /var/run/haproxy-spamd.pid

listen spamd
        bind 212.13.194.5:783
        mode tcp
        option tcplog
        log global
        balance roundrobin
        source 212.13.194.5:0
        clitimeout 150000
        srvtimeout 150000
        contimeout 30000
        server corona  212.13.194.122:783 weight 5
        server curacao 212.13.194.71:783  weight 5
        server islay   212.13.194.96:783  weight 6

The problem is that sometimes the client drops the connection immediately with the client (in my case my MTA, Exim) saying:

2007-04-28 23:10:32 1Hhw3k-0000xN-G2 spam acl condition: cannot parse spamd output
2007-04-28 23:10:32 1Hhw3k-0000xN-G2 SA: Action: scanned but message isn't spam: score=0.7 required=5.0 (scanned in 0/0 secs | Message-Id: SODIUM3tt4LQsJABSCu000006a6@sodium.lon.periodicnetwork.com). From  (host=mail.argon.lon.periodicnetwork.com [83.245.63.194]) for elided@snowblind.net
2007-04-28 23:10:32 1Hhw3k-0000xN-G2 <= noreply@periodicnetwork.com H=mail.argon.lon.periodicnetwork.com (ARGON.lon.periodicnetwork.com) [83.245.63.194] P=esmtp S=2322 id=B0009206781@ARGON.lon.periodicnetwork.com
2007-04-28 23:10:33 1Hhw3k-0000xN-G2 => elided@gmail.com  R=dnslookup T=remote_smtp H=gmail-smtp-in.l.google.com [66.249.93.114]
2007-04-28 23:10:33 1Hhw3k-0000xN-G2 Completed

at these times, haproxy’s log will report:

Apr 28 23:10:32 localhost haproxy[22910]: 212.13.194.70:32958 [28/Apr/2007:23:10:32] spamd islay 0/-1/7 0 CC 0/0/0 0/0
Apr 28 23:10:32 localhost haproxy[22910]: 212.13.194.70:32961 [28/Apr/2007:23:10:32] spamd corona 0/0/236 2285 -- 0/0/0 0/0

The “CC” means that the client dropped the connection before a connection to a backend server was made. That’s the first connection in Exim’s spam acl. The second connection from SA-Exim was successful.

(at the moment Exim on 212.13.194.70 is doing both spam acl connection to spamd and then an SA-Exim one as well, so two connections per email accepted. This is just a transitional thing while I move away from SA-Exim and isn’t a long-term plan.)

It’s not always the spamd islay that shows this error, and the problem doesn’t happen every time – both curacao and islay have successful and problem connections. Only corona is always successful. I don’t know why.

Also when this happens, although Exim drops its spamd connection immediately after sending data, haproxy does pass the connection through to a backend spamd which does process it as normal:

Apr 28 23:10:32 admin spamd[11394]: spamd: connection from 212.13.194.5 [212.13.194.5] at port 33761
Apr 28 23:10:32 admin spamd[11394]: spamd: checking message  aka  for Debian-exim:102
Apr 28 23:10:33 admin spamd[11394]: spamd: clean message (0.8/5.0) for Debian-exim:102 in 0.7 seconds, 1668 bytes.
Apr 28 23:10:33 admin spamd[11394]: spamd: result: . 0 - AWL,NO_REAL_NAME,PORN_URL_SEX,SPF_PASS scantime=0.7,size=1668,user=Debian-exim,uid=102,required_score=5.0,rhost=212.13.194.5,raddr=212.13.194.5,rport=33761,mid=,rmid=,autolearn=no

haproxy doesn’t have a support mailing list, it only has an IRC channel which I am reluctant to bring this up in. I mailed the author and he doesn’t know why it should be behaving like this either. Anyone else have any ideas?

Failing that, anyone know a decent, open source software load balancing solution for generic TCP? Bonus if I can direct to least busy backend, or if I can specify a limit of connections per server.

FirstDirect stupidity

Posted on by Andy

Email from First Direct:

Thank you for your recent request to join first direct.

Full details about a first direct e-ISA, along with an application form containing the information you have provided, are now on their way to you.

[…]

—————————————–
SAVE PAPER – THINK BEFORE YOU PRINT!

This is after filling in some 4 page online form. Really need to take their own advice.

By contrast IceSave manage to do the whole application process online, for a savings account. I am pretty sure if I complain to First Direct about this they will say something about security and money laundering laws. IceSave took driving licence and passport numbers for that.

Update: Despite me calling them three times, First Direct never managed to send me the correct forms to transfer my existing ISA, so around the end of May 2007 I stopped bothering to contact them and opened an ISA account with Yorkshire Building Society instead.

You need xargs replacement to do it that way..

Posted on by Andy

sheepeatingtaz, Omahn’s method seems fine for a one-off. The problem with resiak’s suggestion is that you need text after the data that xargs would put in. One way to do it would be:


$ cd ~/fonts
$ find . -type f -iname '*.ttf' -print0 | xargs -I{} -0 cp {} ~/.fonts/

The -I flag tells it to replace the string “{}” with the data from its standard input, so it allows you to put more of the command after the data.

However you also find that -I implies -L1 — only do one command at a time! So this is actually no better than the simpler find that Omahn suggested.

Looking at the manpage for xargs it is possible to force it to do more by setting -L yourself. So you could say -L100 for example. But I don’t know if find’s automatic abilities to not exceed the maximum command length are still in force then.

In all honesty I’d most likely use Omahn’s simple find command to do them one at a time, if this is just a one-off.

Great galah

Posted on by Andy

Work folks were all outside The Salisbury last night to give Jasper a good send-off when this strange Australian girl attached herself to our crowd. There’s been a lot of turnover of staff recently so at first I just thought she was someone new from the office, but her bizarre behaviour soon put that straight.

Eventually we worked out that she was waiting to see Avenue Q with some friends in the Noël Coward Theatre next door, but they hadn’t turned up yet and she had all the tickets. Also she had been given a lot of cocktails earlier by her boss which accounted for her drunken state.

I say “eventually” because establishing these simple things took around 10 minutes; she was very drunk and most of her conversation was punctuated with “oh my god!” and “Wait! Wait! Wait! Wait!” if she was trying to explain something and someone else was talking nearby. One of these overly-lengthy topics of conversation was when she tried to explain that the distance lengthwise across Australia is “the same as the moon.” I asked, “is that diameter or circumference?” which for some reason sent her off into fits of laughter for a while. I don’t know why; it was a perfectly serious question!

Surprisingly she wasn’t an annoying drunk, it was more comical than anything. In an enclosed space she would have been too much but out on the street in a big group you could at least move away and let someone else talk to her for a while. I have to say it also helped that she was quite cute!

Things started to take a turn for the worse though when she decided to start tipping her drink (cranberry and vodka I think, maybe a cape codder) into our drinks (beer), and when she started eating crisps out of my hand it all got a bit weird and people started to move away! Fortunately at this point she nipped off to the toilet then came racing back, grabbed her drink out of my hand and shouted, “my friends are here!” as she ran back towards the Noël Coward. I have to wonder how much of the plot of Avenue Q she will have been able to follow in that state.

The whole thing took me back to being that sort of age (she said she was 19) and getting completely wasted, being an annoying drunk. Fun, but too risky to get yourself into that sort of state where you can’t look after yourself. I learnt that lesson in a not too terrible way when I was 18 or so; I hope you get a chance to learn it the easy way too, Sarah.

Anyway, I left at about 22.30 because I was beginning to get wasted on the cheeky vimtos Jasper was buying and really didn’t fancy 3 hours on night buses to get home after missing the last train. But it was a good send-off, a good turn out and a good night. I don’t do post-work beers that often but I think I’ll miss The Salisbury when we move to Gray’s Inn Road.