limiting queue runners
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=5710
Printed Date: 31 July 2025 at 3:51pm
Topic: limiting queue runners
Posted By: Guests
Subject: limiting queue runners
Date Posted: 10 July 2006 at 8:04pm
|
Hi All, I am currently using the trial version. I have noticed that the queue runner is exceptionally agressive... Opening up 5+ threads to the same mailhost within a second. Is there any way to stop this? (ideally cut down the number of queue runners, or limit the total number?) Thanks, Andrew |
Replies:
Posted By: lyndonje
Date Posted: 11 July 2006 at 3:46am
|
By queue runners do you mean TCP connections to your destination server? I don't think you can limit/set this in SF, but I'd have thought you should be able to do this on your receiving server - ie set the maximum number of listening threads. If they're all in use, SF will queue the emails it couldn't send through and retry after n minutes set under 'Process queue every n minutes' under Settings/Configuration. |
Posted By: Guests
Date Posted: 11 July 2006 at 8:17pm
|
Hi, Yes, this is what I meant... It is possible to set this in every other MTA that I have used :( - Oh well, I guess i'll just have to look at other software. Thanks for your time :) Cheers, A
|
lyndonje wrote:Posted By: LogSat
Date Posted: 11 July 2006 at 8:57pm
|
AndrewD, SpamFilter has a limit on the maximum number of concurrent incoming connections (on the Settings - Configuration tab). When it comes to forwarding queued emails to your destination SMTP server, SpamFilter will use one half of that value to calculate the number of maximum concurrent connections to your server. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: Guests
Date Posted: 12 July 2006 at 12:28am
|
Hi Roberto, Thank you for your reply. Is there any way to manually change the outgoing (forwarding) SMTP number of threads without touching the incoming limit? - I need to leave the incoming 40+, but I want to keep the outgoing to less than 5. I see no real need to have so many queue runners hitting a single remote server at a time (within one second)... Ideally the first queue runner should be spawned, and then another x (ie 5 mins later) - and so forth to deal with the load. This is a very effective method for mail delivery (don't forget that multiple emails can be sent down one established SMTP session). If this is not possible for the current version, would you consider allowing this configuration to be changed in future versions? Thanks again for your time. Cheers, Andrew.
|
Posted By: LogSat
Date Posted: 12 July 2006 at 12:48am
|
AndrewD, SpamFilter will forward any "clean" email it receives immediately in real-timeas it arrives. So assuming a listserver is sending massive legitimate emails to your users, and the incoming limit in SpamFilter is 40, in theory SpamFilter could have 40 concurrent incoming connections from that listserver. in this case, as emails are forwarded in real-time, SpamFilter will also have 40 outgoing concurrent connections to your destination mail server. The only time when emails will be queued up is if your destination SMTP server is offline. In that case SpamFilter will queue the clean emails, and will attempt delivery on regular intervals. When your server will come back online, then in this case SpamFilter will forward all the queued emails to your destination SMTP server, using a number of concurrent threads equal to one half of the "Max incoming connection". SpamFilter should forward as many emails as it can when flushing the queue, as when this happens it will be because your server was down. If it was down for several hours, there may potentially be hundred of thousands of clean emails queued for delivery, and they need to be processed as fast as possible to avoid further delays to end users. You should size the value of "Max incoming connections" to a value that your destination SMTP server can handle. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: lyndonje
Date Posted: 12 July 2006 at 3:37am
|
Hi Andrew, If SF was hadling 40 incomming connections, but only passing out 5, assuming your inbound feed was constant you'd end up with a bottleneck on SF. Why would you want that? Also, if SF is receiving 40 incomming connections, what would happen if you didn't have SF? Your destination server would only have to process the 40 concurrent connections anyway, plus processing the spam. If you inbound load is too high for your destination server to handle, you need to upgrade or introduce a second for load balancing. If you can limit the number of inbound connections on your destination server, which I assume by your earlier comment "It is possible to set this in every other MTA that I have used" that you can, why don't you do this? That way you'll have a bottleneck on your SF, which would be the same as if what you wanted to achieve was possible. |
Posted By: Guests
Date Posted: 13 July 2006 at 8:48pm
|
Hi Again, The reason for reducing threads is that the SMTP standard supports multiple smtp transactions in a single TCP connection (one after the after). This is what the RFC dictates, and every other good MTA I have used follows these rules. Typically you will have connections from all over the world (slow) being forwarded to a few internal severs. You also have the luxury of queuing internally (for a few seconds if needed). It makes sense to follow the standards and not just spew TCP connections through, but instead group them by destination server and pass through seqentially (with additional queue runners if needed). If you take this approach (and follow the standards) there is no bottle-neck, and things actually run much more efficiently on both ends). I have worked on some exceptionally high volume mail systems (in the millions/day) - and this is the only good way to approach the situation. I am not trying to be overly critical of this product, instead pointing out a weaknesses that could be improved upon. I just can't see your product being used in a high volume isp environment. Thanks, Andrew.
|
Posted By: LogSat
Date Posted: 13 July 2006 at 11:29pm
|
Andrew, SpamFilter is not strictly an MTA. It is a product that has to receive emails, completely reject some of them, archive others, and pass on to other servers clean ones. To do so efficiently, unlike what you state, the best way for us to do so is to pass on each email in the same way it was received. This means that each incoming connection will result in a outgoing connection if the email is clean. Performance-wise, please post a new query in this forum asking for other user's experience. This will allow you not to take our word when we say that a very small 3MB application can indeed handle millions of emails/day, on very simple hardware (as in a VMWare guest machine with low CPU and RAM available). ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |