LogSat wrote: Matt, As usual you have valid points. We are preparing (trying to) a new version (SpamFilter Enterprise) that will have a major new feature.

YES!!!

>>>>Vrspock...By not getting an NDR, they assume that their list is good and will continue using it.

I disagree. 99.9999% of spammers don't use conventional SMTP servers to send their spam. The programs that they use completely ignore conventional denials in the smtp conversation i.e. they ignore the "550 unknown recipient" reply. Also, do you honestly think that they send a real email address in the "Mail From:" header? Do you think that they actually pickup the NDR's that are sent?  err...no.

After careful investigation of our logs, it is clear to me that by SF accepting the mail, then our systems re-writing the "RCPT TO:" header (hence dropping the recipients that don't exist or don't want spam) we are not getting any more spam attempts than we used to get. Therefore, the spammers aren't sending us even more spam just because we accepted the first email and didn't send an NDR.

This subject is obviously up for debate, but our opinion is that we will block spam from getting to our customers inbox because that's what they pay us to do (apart from the occasional paranoid user who insists on an unfiltered inbox...).

Each of you must do what you feel is right for you. This is right for us.

SpamFilter should facilitate the the flow of good email while blocking the spam.  When the recipient list is not processed completely, the software falls short of this objective.  The difference is SpamFilter needs to potentially make more than one outbound connection to deliver an email message. I thought all email servers did this anyway for outbound email and SpamFilter should be no different as it now supports authenticated outbound and relayed email.

1. SpamFilter currently sends a single NDR on a failure of one or more spam checks per the ini settings.  That single NDR SpamFilter can send per config is the only NDR SpamFilter should continue to send; there is never more than ONE SENDER to report an NDR to.

2. SpamFilter does not need to send multiple NDR's in this case. One NDR is generated if a virus or spam test fails AND the configuration determines whether or not to send NDR for that failure.  The text "relay or delivery to one or more recipients failed" along with the failed recipient list SHOULD be included in the NDR message when one or more recipients in a multiple recipient email is processed and NDR is supposed to be sent.

3.  There is no email splitting.  It's a matter of issuing one connection for each authorized domain when sending out and issuing one rcpt to: command for each recipient in the connected domain.  Copies of an email are sent to forwardable recipient domains.  Forwardable recipient domains are defined as those that do not fail a spam test and whose domain is in the list of allowed domains.

4.  The recipient list SHOULD be sorted by domains and sent in a loop to recipients (rcpt to:) grouped by domains (connections per authorized domain list).

5. If there are recipients in the headers that are not in the allowed domains; they are irrelavant and are ignored.  Spamfilter would have ALREADY refused those rcpt to: commands when the message was received.  SpamFilter could always build it's own alternate recipient x-header based on the rcpt to: commands it accepted on the inbound email receive.

So, in summary, with all of the flexible and powerful features already in this program; adding a feature that provides greater degree of accuracy is an intelligent direction for the product to take. I'm sure I'm missing a bit and overlooking a few gotchas in my ideas above, but this recipient accuracy feature deserves more consideration.

-Matt

I agree, caution is definetly in order on here with regards to our methods of dealing with the constant flow of spam.

I have thought about using our mail server's own anti-spam features in combination with spamfilter to do something similar, but the whole point of the authorizedto list is to reduce the number of hits we get from people who are blind spamming using lists that are populated with a bunch of non-existant mailboxes.  By not getting an NDR, they assume that their list is good and will continue using it.

On a side note, I did find something kind of humerous on the web with regards to email scavenging.  It was a .asp script that allowed for the random and dynamic generation of an infinite number of cross-linked pages filled with fake email addresses...it'd keep dishing them out as long as the scavenging program kept scanning for them.

i advise you to be very  'paranoid' when giving out code and explaining how you do it *in public*  im convinced some of the registered users here are indeed spammers. Topics have been started on this subject before, just be aware.

I have a solution which works for us, although Roberto will go nuts when he hears it !!!

I configure SF to not quarantine anything, but instead to tag and deliver. It then delivers the mail to IIS's SMTP server on the same box, which is configured to allow all domains. I then run a custom script in the OnArrival SMTP event which splits the email, sending it only to the recipients that either are "unfiltered" of specifically want email from the sender/domain. I then just abort delivery to the rest of the recipients, as they either dont exist, or don't want the spam. Therefore, the people who want the email get it, and the people who dont....dont. Also, as the email delivery gets aborted, no NDR's are generated.

Yes, Roberto, I hear you....I know you don't like splitting emails, and it's not strictly to the letter of the RFC, but if you won't do it, then I'll have to ...  ....talking of the RFC...I understand being in strict compliance....but the RFC has major failings. If they had got the RFC right in the first place, we wouldn't be getting this spam now....

For the rest of you....this custom script includes a bunch of code that needs to remain confidential, but I'll try to pull out the relevant parts and post the code here just in case anyone wants to have a play. Give me a week or so as were quite busy at the moment.

We've still got some very persistant spam getting through....one appears to have been a connection from Korea that happen to hit a valid, excluded email address on the first try and had a field day with random from/to addresses from then on.  Managed to get quite a few spam to pass through and generate a bunch of NDR's when spam filter attempted to deliver the message to about 20 or so invalid mailboxes.  This was one of the pharmacuetical spams.

Since this causes a mass of NDR's to be generated anyhow when spam filter accepts an email that has 1 valid, excluded address and about 20 to 30 invalid addresses, what difference would it make if we just split the message and NDR the non-excluded addresses?  Either way, we are generating a bunch of bounce backs.  At least by splitting the message, only those on the exclude list are getting what they asked for while everyone else is still filtered.

Think I'm going to lower our acceptable number of CC's and see if that helps any. 

So far so good webguys...I will hear a report from our end users on Monday, but from our logs it appears to be doing the trick:  Authorized To in combination with disabling the bayesian filter learning new emails and using *|mailbox for exlcuding accounts on the autowhitelistforceddelivery list.

What appers to happen is when a multiple-recipeint email comes in with a hit-or-miss list of mailboxes, typically the very first mailbox is an invalid one, the authorized to list gives its error and the session is immediately disconnected without further processing.  Also, we've set our blacklist cache and limbo settings to some very agressive settings.  We had already done this before trying to disable the bayesian filter as mentioned in this thread.  I'm not sure how doable these settings would be for those with a much higher volume of email coming through, but we have ours set to 3 failures within a 2-hour period lands you on our blacklist cache for 6-hours.  With all of the authroized-to failures dropping connections, our black list cache limbo is starting to become quite extensive, but our actual black list cache will typically only have 4 or 5 ip's at any given time, which confirms that the spam that has been getting through is hitting us from zombie machines.

I've only seen one or two spams getting through to my mailbox this weekend which is a significant drop from the past few weeks.

"hurt performance" .....  even at the expense of letting in spam?? Doesn't that defeat the whole idea of SF ??