Tag Archives: concurrency

Qmail per domain concurrency

Patches, ,

Problem

In my last post about qmail I said that once you solve the big concurrency problem you'll end up with another problem because your mail server would create too many outgoing connections to some domains and you risk having your ips banned by those servers.

Solution

The solution is to have a way of limiting the maximum concurrency rate by domains. To do that you'll need the   qmail channels patch or write your own patch like I did ( mostly because I was unaware of the existence of the qmail channels patch )

The home page of the qmail channels patch will explain how to setup and configure qmail to limit the concurrency by a domain or group of domains.

What I like about this patch is that it allows you to set a concurrency limit for a group of domains like set 100 for all yahoo.com, yahoo.co.uk, yahoo.ca, etc .

What I don't like is that it doesn't seem to be able to set a default concurrency level for any domain. If I'm wrong please correct me, but if I'm right then this seems like a major problem for an email server that sends to a large number of addresses distributed over a large number of domains because you would have to configure concurrency limits for a lot of domains.

The ideal solution would allow you to specify a default per domain concurrency and this would apply to any domain that doesn't have a specific concurrency. For example most email servers would be ok with 5 concurrency connections from the same ip but no way for AOL (unless you're white listed and maybe not even then ) .

Another feature I would like is to be able to specify concurrency by domain's MX records or ips/group of ips assigned to the MX servers instead of the actual domain. This would ease the configuration for ISPs that host a lot of domains like rr or yahoo.

Qmail Big concurrency

Linux, , , , ,

Wanna send messages faster with your qmail server? Everyone will tell you to increase the remote concurrency. Till you find out that it can only go as high as 255. If you want more then that you have to apply the big concurrency patch

The Problem

Applying the patch and setting concurrency ( conf-spawn ) bigger then 509 will break the compilation. I was hoping to get at least 1000 on that new quad core 🙁

Why 509? It seems the number depends on the maximum size of the FD_SET array used for "monitoring" ( using select ) the file descriptors ( connection sockets or opened files ) . This limit is set in FD_SETSIZE constant to 1024. In case you're wondering ... the formula that gets you from 1024 to 509 is (FD_SETSIZE-5)/2 ( from chkspawn.c )

Trying to define FD_SETSIZE to a higher value in conf-cc like this -DFD_SETSIZE=4096 doesn't work because FD_SETSIZE is redefined in sys/select.h like this

  1. #define FD_SETSIZE              __FD_SETSIZE

__FD_SETSIZE is defined somewhere in /usr/include/bits/types.h ( actually typesizes.h ) to 1024. Defining -D__FD_SETSIZE doesn't work either...I even tried both and still no luck.

The Solution

After hours of digging through mail archives and sites I found this mailing list post that really helped:
Re: fd_setsize

If you just want to get it working just download my patch [download id="14"] , apply it ( after you apply the big concurrency patch ) , set conf-spawn to something big ( but less then 65000 ) and then you should be able to compile qmail.

If you want to know how it's done, read bellow...

It seems like the solution is to include bits/types.h, undefine __FD_SETSIZE and then define it to a higher value. The author of that post says that this is not a good idea (from the portability point of view, but I don't care about that ) since programs should never directly include bits/types.h ( true ) but the alternative is to modify that system file, again not a good idea since it will be overwritten by a possible update.

My first idea was to just use that code from the mailing list post into the select.h2 , since this is the file used to generate select.h and select.h is included in spawn.c and ckhspawn.c but this didn't work because spawn.c was including "select.h" after "sys/types.h" so even if select.h would define __FD_SETSIZE it would be useless since FD_SETSIZE ( this is the one that really matters ) would have been already defined in sys/types.h .

The solution I found at the time was to just move "select.h" at the top of the file and remove "sys/types.h" since it was already included from select.h but now I realized I could have just as well undefined and defined FD_SETSIZE too inside select.h

And that's the story about how I got to run 1000 concurrent connection in qmail.

The real problem

Now that we can have so much concurrency we hit another wall. Qmail, as most other MTAs, doesn't have any way of controlling the remote concurrency per destination domain.

At 1000 simultaneous connections it's very likely that it would create a few tens or hundreds of connections simultaneously to the same domain.

When this happens that domain will just ban your ip.  So how do we fix this one?

PS: I have an answer but I want to see what you have for a solution 🙂 so hit the comments...