Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Updated Filter order
  FAQ FAQ  Forum Search   Register Register  Login Login

Updated Filter order

 Post Reply Post Reply
Author
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Topic: Updated Filter order
    Posted: 17 May 2005 at 11:03pm
We have moved the "official" filter order to a new standalone post at:

=======================

Below is the latest list containing the order in which filters are processed. In general, all whitelists take precedence over blacklists. There are two exceptions:
  1. Viruses - if an email is infected, there is no whitelist to save it... it will be blocked, period.
  2. Allowed Domains - to avoid becoming an open-relay, no email is ever delivered unless the recipient domain is listed in the "Allowed Domains" list. The only whitelist that can be used to allow delivery of emails to non-local addresses is the IP whitelist. The theory is that any spammer can eventually guess how to use/fake your whitelists to then abuse your SpamFilter as an open relay. The only thing that they can't really fake is the IP address (IP spoofing won't help here...).
In the list below, in red are the blacklists, in green the whitelists.


  1.         Whitelisted IP
  2. Allowed Domains
  3.         Whitelisted Email Address To
  4.         Whitelisted EMail Address From
  5.         Whitelisted Email From Domain
  6.         Whitelisted Auto White List Force Delivery
  7. Local Domain Blacklist
  8. Local Emails Blacklist
  9. Local Emails TO Blacklist
  10. Not in Authorized TO Emails
  11. Country Blacklist
  12. Reject No Reverse DNS
  13. Reject Empty Mail From
  14. Reject Same To From Email address
  15. Reject if Recipient email in Honeypot email list
  16. Reject if IP in Honeypot-generated autoban list
  17. Reject Same To From Domain
  18. Recipient Count > Max RCPTTO
  19. MX Record check
  20. SPF Filter
  21. MAPS check
  22.         Keyword Whitelist
  23. Attachment Filter
  24. Keywords
  25. Bayesian Filtering
  26. SURBL check
  27. Antivirus Plugin



Edited by LogSat - 31 July 2009 at 10:04am
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 06 December 2005 at 11:56pm

Where would the new IP Cache Black List you were talking about fit into the filter order? I am going nuts trying trying to think of a way to stop the 24x7 dictionary attacks that are hitting my mail server (yes, I use AuthorizedTo list extensivley). I can write a script to gather all the IP's of the rejected maurauders, but there is no SF file to stick them since in the order list the AuthorizedTo check comes after the Whitelist check and before the Block IP list. There would have to be a list that was checked upon connection for the IP address and if it was in the cache black list be stopped before the first whitelist check was ever performed.

Is that what the new blacklist you mentioned recently going to do?

My SF machine is getting pounded, but that means my regular mailserver gets to go about its business as usual and everyone is happy so SF is really doing a great job for me. Thanks Roberto!

http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 December 2005 at 3:50am
We're still not 100% sure, but so far it looks as if it's going to be the 1st in the list, so that if the remote IP is in the blacklist cache, it won't even be allowed to connect. This should greatly reduce the strain on the Spam Filter server, but will have the downside of the attempted email to not even being quarantined. But as this is a "repeated offender", the first few email attempts have already been quarantined, so we do not think this will be so bad.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 07 December 2005 at 9:46am
Great! Looking forward to this feature. These mail harvesters really p*ss me off. Tired of daily log files > 40 meg filled with AuthorizedTo rejects.
http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 21 December 2005 at 5:03pm
We have moved the "official" filter order to a new standalone post at:


As of SpamFilter ISP version 2.7.1.511, following is the updated order of the filters. In red are the blacklists, in green the whitelists:
  1. Cached IP blacklist
  2.         Whitelisted IP
  3. Allowed Domains
  4.         Whitelisted Email Address To
  5.         Whitelisted EMail Address From
  6.         Whitelisted Email From Domain
  7.         Whitelisted Auto White List Force Delivery
  8. Local IP Blacklist
  9. Local Domain Blacklist
  10. Local Emails Blacklist
  11. Local Emails TO Blacklist
  12. Not in Authorized TO Emails
  13. Country Blacklist
  14. Reject No Reverse DNS
  15. Reject Empty Mail From
  16. Reject Same To From Email address
  17. Reject if Recipient’s email in Honeypot email list
  18. Reject if IP in Honeypot-generated auto-ban list
  19. Reject Same To From Domain
  20. Recipient Count > Max RCPTTO
  21. MX Record check
  22. SPF Filter
  23. MAPS check
  24.         Keyword Whitelist
  25. Attachment Filter
  26. Keywords
  27. Bayesian Filtering
  28. SURBL check
  29. Antivirus Plugin


Edited by LogSat - 31 July 2009 at 10:04am
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Guests View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Guests Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 10:49am

Roberto,

Is there a way for users to customize the filter order?  For instance, I would prefer that the Keywords blacklist fires before the MAPS check, since I am not quarantining spam flagged due to keywords, but I am quarantining spam flagged due to MAPS.  I've found that a lot of the messages in quarantine have the keywords that I've blacklisted, and if the keywords filter would fire first, I would have less email to sift through in the quarantine list.

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 5:24pm
Customizing the filtering order is a rather complex update, and we've always been hesitant in implementing it do to the problems it may cause. SpamFilter is optimized for speed and the low number of resources used, and part of this is due to the way the various filters are employed. For example, the MAPS check is performed very soon after a new incoming connection is detected, before the email's contents are received. This allows SpamFilter to decide if an email is spam even before the actual content is received and analyzed, which can be very cumbersome (CPU-wise) for a server. If the order of these filters is reversed, every single email will have to be fully received before and analyzed any of the DNS-based tests can be applied. If your keyword file is large, this can cause the required server's resources to increase by 10x-100x...
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 16 April 2006 at 10:56pm
We have moved the "official" filter order to a new standalone post at:


As of SpamFilter ISP version 3.0.1.553, following is the updated order of the filters. In red are the blacklists, in green the whitelists:

  1. Cached IP blacklist
  2.         Whitelisted IP
  3.         Whitelisted Email Address To
  4.         Whitelisted EMail Address From
  5.         Whitelisted Email From Domain
  6.         Whitelisted Auto White List Force Delivery
  7. Allowed Domains
  8. Local IP Blacklist
  9. Local Domain Blacklist
  10. Local Emails Blacklist
  11. Local Emails TO Blacklist
  12. Not in Authorized TO Emails
  13. Country Blacklist
  14. Reject No Reverse DNS
  15. Reject Empty Mail From
  16. Reject Same To From Email address
  17. Reject if Recipient�s email in Honeypot email list
  18. Reject if IP in Honeypot-generated auto-ban list
  19. Reject Same To From Domain
  20. Recipient Count > Max RCPTTO
  21. MX Record check
  22. SFDB Filter
  23. SPF Filter
  24. MAPS check
  25.         Keyword Whitelist
  26. Attachment Filter
  27. Keywords
  28. Image Filtering
  29. Bayesian Filtering
  30. SURBL check
  31. Antivirus Plugin

 



Edited by LogSat - 31 July 2009 at 10:05am
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 23 February 2007 at 10:26am
Read the list, but am still unsure in my case:

I'm trying to take some load off my main smtp server and want to route domains through spamfilter although the customer does not want any spam filtering (yes, those still exist...)

Anyway, I'm using the "Authorized TO EMails" List, which is generated by script off the main mail server. Now I was thinking about adding those domains which do not want spamfiltering to the "Unfiltered Emails" (with :tag), but as I found out until now, then the "Authorized TO EMails" does not get checked and all emails get forwarded to the main smtp server, not only those with valid EMail addresses (which would be my goal - filter out viruses and invalid email addresses on the spamfilter server and forward everything else).

Any possibility to achieve this?

Regards,

Mike
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 23 February 2007 at 10:24pm
Sorry Mike. As the list states, the "Unfiltered emails" has a higher priority than the "not in Authorized TO". Thus as soon as the email is whitelisted, all other tests are skipped (except the antivirus one).
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 24 February 2007 at 6:09am
OK, so i would have to add each email address separately in the "Unfiltered emails" list with a :tag option, then it would work. I could write a script to do that. Do you think performance could be an issue with this workaround?
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 06 January 2008 at 11:25pm
We have moved the "official" filter order to a new standalone post at:


As of SpamFilter ISP version 4.0.0.766, following is the updated order of the filters. In red are the blacklists, in green the whitelists:

  1. Cached IP blacklist
  2. Greylist
  3.         Whitelisted IP
  4.         Whitelisted Email Address To
  5.         Whitelisted EMail Address From
  6.         Whitelisted Email From Domain
  7.         Whitelisted Auto White List Force Delivery
  8. Allowed Domains
  9. Local IP Blacklist
  10. Local Domain Blacklist
  11. Local Emails Blacklist
  12. Local Emails TO Blacklist
  13. Not in Authorized TO Emails
  14. Country Blacklist
  15. Reject No Reverse DNS
  16. Reject Empty Mail From
  17. Reject Same To From Email address
  18. Reject if Recipient’s email in Honeypot email list
  19. Reject if IP in Honeypot-generated auto-ban list
  20. Reject Same To From Domain
  21. Recipient Count > Max RCPTTO
  22. MX Record check
  23. SFDB Filter
  24. SPF Filter
  25. MAPS check
  26.         Exceeded MaxMsgSizeForSpamFiltering
  27. Keyword Whitelist
  28. SFCD Filter
  29. Blank emails with attachments only
  30. Spam Images in PDFs
  31. Attachment Filter
  32. Keywords
  33. Image Filtering
  34. Bayesian Filtering
  35. SURBL check
  36. Resolve URLs and check IPs in MAPS
  37. Antivirus Plugin


Edited by LogSat - 31 July 2009 at 10:11am
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
dcook View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 January 2005
Location: United States
Status: Offline
Points: 174
Post Options Post Options   Thanks (0) Thanks(0)   Quote dcook Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 3:40pm
Does that mean that we can't whitelist a sender to avoid the greylist?
 
I have a client that is not getting some mails and if the client is rejected by the greylist it appears whitelisting will not help?
 
Dwight
www.vividmix.com
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 4:00pm
That is correct. Please do note that the greylist will only delay the reception of the first email ever sent by a specific server. Once that email has been received (because the remote SMTP server has retried sending it), the IP will always be allowed to pass the greylist filter in the future.

If really necessary (but there has not been a need for anyone to do this yet...), you can manually add IPs to the greylist files in the \SpamFilter\domains\GreyListAllowed.txt file (which requires stopping/restarting SpamFilter for it to be reloaded).
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 4:13pm
Roberto,
 
I actually came up with one issue which also happened with sending to Yahoo and I am not sure there is any real answer.  Gammadyne, (a semi-mass mailer), totally chokes when it hits any grey-listing server including SpamFilter and Yahoo.  I have contacted their Tech Support on this problem and have not yet heard back.  It also happens when I stupidly forget to add the Gammadyne server (on our network) to the donothoneypot list and the IP get's put in the BlackList Cache.  It seems that the SMTP engine just doesn't want to disconnect and kill it's own thread it it gets a disconnect it was not expecting.  This is NOT a SpamFilter issue.  Just thought I would warn folks if they use direct SMTP mailers to send notices to their internal users.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
kitti View Drop Down
Newbie
Newbie
Avatar

Joined: 26 September 2006
Location: Thailand
Status: Offline
Points: 1
Post Options Post Options   Thanks (0) Thanks(0)   Quote kitti Quote  Post ReplyReply Direct Link To This Post Posted: 06 August 2008 at 12:33am
Roberto

Good day Roberto.
Is it possible to include updated filter order in the help file?  It's very hard to find in support forum, it's take alot of time to find it when I look for.


Many thanks
Kitti J
From Siam
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 06 August 2008 at 4:50pm
Very valid point. We've just updated the documentation, and it will be included in the next release of SpamFilter.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.096 seconds.