Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - SpamFilter v2.7 with IP cache is avail
  FAQ FAQ  Forum Search   Register Register  Login Login

SpamFilter v2.7 with IP cache is avail

 Post Reply Post Reply
Author
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Topic: SpamFilter v2.7 with IP cache is avail
    Posted: 11 December 2005 at 9:13pm
We have pre-released in the registered user area of our website a newbuild of SpamFilter ISP v2.7.1.508. The major additions to this version are (1) a new feature that stores in a memory cache a list of IP addresses that have been blocked, and (2) a greatly improved Connections tab showing in realtime what commands the remote IPs are sending

The new IP Cache:
After an IP address send a spam/virus email, it will be added to an IP "limbo" cache. If an IP in the limbo cache sends more that a certain number of spam emails (3) during a certain amount of time (10 minutes), that IP will be moved to a temporary "blacklist cache", and from that point on all connections from that IP will be immediately rejected. The IP will be automatically be removed from the cache after a period of time (60 minutes) and will be thus given more chances to send "clean" emails.

Important!
Any addresses in the DoNotAddIPToHoneypot list will also not be added to this blacklist cache, thus preventing the blocking of "friendly" servers.

This IP cache should greatly reduce the load on the SpamFilter server because since the connection is rejected, the IP will never have a chance to send an email, saving CPU and bandwidth resources.

All of the above parameters in parenthesis can be user-configured by changing their values in the SpamFilter.ini file as follows:

;If an IP sends more than this number of spams in a certain period of time then it is temporarily banned (blacklisted)
IPCacheLimboCountTrigger=3

;If an IP sends more than a certain number of spams during this number of minutes then it is temporarily banned (blacklisted)
IPCacheLimboTimeTrigger=10

;If an IP address was banned because it sent too many spams in a certain time interval, it will be un-banned after this number of minutes
IPCacheBlacklistDuration=60

The release notes for the latest builds are as follows:

// New to VersionNumber = '2.7.1.508';
{TODO -cNew : Implemented an IP cache to temporarily deny further connections to IPs that sent multiple spams recently. This can greatly reduce the load on the server}
{TODO -cNew : Improved "Connections" tab, showing in real-time what commands the remote IPs are sending}
{TODO -cFix : Sometimes the "Current Connections" counter could not decrease when a remote connection is dropped, thus displaying a number higher than reality}

// New to VersionNumber = '2.6.3.502';
{TODO -cFix : Duplicate entries were being created in the logfiles}
{TODO -cFix : Bug introduced in v2.6.3.491. When forwarding emails to the destination SMTP server, sometimes the leading "<" and trailing ">" where missing in the MAIL FROM}

// New to VersionNumber = '2.6.3.495';
{TODO -cNew : Added options to not quarantine or send to NULL viruse-infected emails}

// New to VersionNumber = '2.6.3.493';
{TODO -cNew : Added DNSTimeout option in SpamFilter.ini to customize the DNS timeout for all of SpamFilter's DNS queries}
{TODO -cNew : Added EnableDbgLogs SpamFilter.ini option to enable separate detailed logging for troubleshooting purposes}
{TODO -cNew : Added to SpamFilter.ini several of the optional entries with their default values for users to see}
{TODO -cFix : Clicking on "Check if IP in ORBS" button in GUI could result in Access Violations being logged}

// New to VersionNumber = '2.6.3.491';
{TODO -cNew : Added support for maximum message size in reply to EHLO and MAIL FROM, as per RFC1870}
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 12 December 2005 at 2:32pm
Does this include 'Not in AuthorizedTo List' failures?? <fingers crossed>
http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 12 December 2005 at 4:18pm
How could we ignore your pleas for help on the "Not in Authorized list" problems...!

Yes, 3 attempts from the same IP within 10 minutes will cause the sender's IP to be blacklisted in cache for 60 minutes. They will be immediately disconnected from then on without even giving them a chance to issue any commands. All these parameters are configurable in the spamfilter.ini file.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 12 December 2005 at 5:25pm

So the squeaky wheel does get the grease.  

Back to Top
WebGuyz View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 12 December 2005 at 9:01pm

I have run into a small issue, unless I'm mis-reading this. Below are my current settings. What I'm finding is the IP's are being released from cache blacklist after 10 minutes, not 60 like I thought my setting would give me. Also, I have to get a whole bunch of rejects (more then what I set at 8) before SF puts the IP in limbo. From the readme I infered that my setting mean: if 8 rejects occur in 10 minutes, the IP would be put in limbo for 60 minutes. But if thats the case, thats not what is happening.

Andy

 

----------spamfilter.ini------------
IPCacheLimboCountTrigger=8
IPCacheLimboTimeTrigger=10
IPCacheBlacklistDuration=60

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 12 December 2005 at 9:32pm
Andy,

Using your settings, let me describe what SpamFilter will do.

You receive a spam email from a sender. The IP address of the sender will immediately be added to the limbo cache.

From the time it's added to the limbo, should that IP address send 7 more spam emails within 10 minutes from the 1st one (a total of 8 spams within 10 minutes), then the IP address will be removed from the limbo and will be added to the blacklist cache, where it will remain for 60 minutes.

All connections from IP addresses in the blacklist cache will be immediately rejected. The limbo is just a temporary location that holds all IP addresses that recently sent spam, connections will *not* be rejected simply due to the presence of the IP in the limbo. Consider the limbo halfway between good and evil... The senders are not totally innocent, but they won't be punished immediately. If they insist on behaving badly, they'll be treated as evil.

Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 12 December 2005 at 9:43pm

aha!! But it seems like it takes a while for it to kick in. I'll have 50 AuthorizedTo rejects fly by in the log before I even see it put the IP into limbo. Will study the logs some more this evening and tomorrow.

I think I like it better than greylisting as a means to cut down on spammer connections.

THANKS!

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 12 December 2005 at 9:56pm
Please note that if the spammer is attempting multiple (50) RCPT TO commands to send the email to multiple recipients in a single mail session, then when it will be rejected that will only count as a single connection... If that's the case, you may want to limit the maximum number of recipients allowed in a single session, so that the spammer gets dropped sooner.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 12 December 2005 at 10:08pm

Starting to come to that same conclusion looking at the log.

I have to keep the recipients allowed in single session high becaue some of our larger customers are on mailing lists and some have a LOT of mailboxes. I was getting complaints from the senders that half their emails were being returned to them. <sigh> Can't make everyone happy.

Back to Top
WebGuyz View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 13 December 2005 at 10:25am

Do we have access to the IP Blacklist Cache, or is it builtin to SF? I would like to build a log parser to gather IP's from dictionary attacks but have never had a way to use it since the AuthorizedTo check came early in the filter list(before block IP's). If I was able to feed the IP Blacklist cache manually I could fix this. Is it possible? or not?

THANKS!

Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 13 December 2005 at 10:39am

The IP Cache is not a file based cache.  It can be viewed under the "Statistics" tab.  I was going to wait at least a full week before I started begging for features!   I rarely am "on" my server so everything I do is either SQL queries or log parsing.   I am trying to find time to make modifications to the SawMill log parse file to extract the information out and then an export of that data may help you.  However, the whole point is that the spammers are in "Hit & Run" mode so that the value may be limited.

Per Roberto, The log entries are as follows:
12/11/05 13:01:31:515 -- (2272) Connection from: 172.27.4.50  -  Originating country : N/A
12/11/05 13:01:31:515 -- (2272) IP is in local blacklist cache. Disconnecting: 172.27.4.50
12/11/05 13:01:32:468 -- (2272) No Data Received
12/11/05 13:01:32:468 -- (2272) Disconnect
 
So ... should be easy to parse out.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
WebGuyz View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 13 December 2005 at 10:51am

I'm not too proud to beg

Roberto, how about a txt file that gets checked periodically and if it finds any IP in there it adds it to the internal IP Blacklist Cache. I would even do my own cleanup of this text file.

I use MS Logparser with excellent results and could easily harvest the harvesters IP's and add them myself.

I'm shocked at how well these guys are organized and the hit and run tactics you mentioned are definitly there to get around most spam checkers. They will try a handful of addresses with one IP and switch to another IP in the blink of an eye and they have a LOT of IP's so if I  have my blacklist cache timer set to 10 minitues, they can get around this my using many, many different IP's. But I would have them in the log and their IP's would be mine to blacklist. I checked some of the IP's and they are cable connection or residential accounts.

I really hate these guys ....

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 13 December 2005 at 8:11pm
The cache is in memory only, and is emptied when SpamFilter is restarted. There's really no plans to allow updating it with external data. Can I ask why you do not use the already-available local IP blacklist to reject spammers? If that list is modified, SpamFilter will automatically reload it.

As a side note, we do realize that the blacklist IP check is performed after many other tests, thus needlessly wasting resources. What we could do it to  prioritize it, so that blacklisted IPs are tested and rejected before other tests occur (ex. before MAPS, SURBL, MX, Reverse DNS, SPF, AuthorizedTo). Would this help?
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 13 December 2005 at 9:05pm

Roberto,

    Putting the IP Blacklist test at the top of the filter list would be a very good thing.

Back to Top
WebGuyz View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 16 December 2005 at 8:44pm

Will the prioritized IP Blacklist be in 2.7.1.510 ?

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 16 December 2005 at 10:12pm
It didn't make it in 510, but it did in 511 which we just uploaded, along with a couple of tweaks that should increase performance if the ReverseDNS filter is not used.

The release notes for 511 are:

// New to VersionNumber = '2.7.1.511';
{TODO -cNew : Changed the priority of the IP blacklist filter, it is now placed before the local domains blacklist}
{TODO -cNew : Changed the logfile entry if the IP address is blacklisted to:  "IP is in local blacklist file..."}
{TODO -cNew : Performing reverse DNS queries only if the ReverseDNS filter is enabled, thus improving performance when it's off}

Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.078 seconds.