Hi Roberto,

I've been thinking about the spam problem a lot and think we need to take a new look at spammer's methods of sending spam.

The spammers are getting more and more ingenious in getting through our defenses, so i think we need to put some more 'brain' in our filtering techniques. The way i imagine this is by not looking at individual mails and filtering those, but looking at groups of incoming spams and trying to find patterns.

For example:

- If one single sender IP is triggering the keyword filter more than 5 times in one hour, that ip gets blacklisted, even if all 5 mails are different and are sent to different recipients.

- If an IP is found in MAPS, move it to blacklist immediately.

- If a message is triggering one of the filters refuse all mails that have exact same length for one hour from the same IP range.

- If more than 5 mails have the same subject and are triggering one of the keywords filters, automatically block all mails with the same subject for a day.

- If a mail is positively identified as spam, create a checksum of its body and compare that to all incoming mails, if checksum matches, refuse it catagorically, regardless of origin, sender or adressee. Forward all sending IP's to MAPS servers.

- If more than 20 identical mails (checksum controlled) are passed through the same server within an hour, chances are high it's either spam or a newsletter, an alert should be created and all subsequent mails with same content should be tagged as spam and/or blocked.

These are just a few thoughts that spring to mind, in essence the idea is to not treat mails individually, since they are never beeing sent individually (in case of the spammails), and as such they behave in a pattern, recognising the pattern and blocking all that adheres to that pattern would reduce the amount of spam that travels the globe and pollutes all of our networks tremendously. I'm not saying we should stop investigating the individual mails, i'm saying we should *also* look for patterns.

I do realise this would take a major effort to implement, nevertheless i wanted to bring it forward.

If such a system would get operational, and would be carried by a serious number of ISP's, the spams would get caught after only a few sent mails, and the rest of their broadcast would be refused by all major isp's.

The resources that would be spent on finding patterns would reduce the amount of resources waisted on treating the individual mails, since a bunch of them will be recognised immediately as beeing spam.

If anyone has more similar ideas on the subject please bring them forward. :)

Best regards,

Marco

Marco,

Many of these are very good ideas. We've saved them in our "wish list", which, as you may already have notcied, is often used to enhance SpamFilter's abilities.

FYI, one of the next features will be the presence of an "IP blacklist cache", which will cause all IP's that send an email identified to be spam to be placed in a temporary IP blacklist. from then on, all connections from that IP will be immediately rejected, even before they have a chance to send the email content. This will allow SpamFilter to save bandwidth, CPU and RAM resources, as multiple spams by the same IP will simply not be allowed to connect. The cache will expire on a configurable timeout (we're thinking somewhere between 10-60 minutes by default).

As allways, your openmindedness towards user suggestions is commendable Roberto.

Thanks for listening to us, the end users.

Ok Marco you opened the door so while we are brain storming let me add a little fuel to the fire. :)

One of the missing links in almost all spam software is the inablity to learn from others. Sites like Spamcop and other MAPS because the more I report my spam to them the stronger the entire system becomes.

I believe this is an area that were SF could be improved. The idea is for Logsat to create a repository that SF would automatically submit spam info to. For example if Marco's system confirms a series of spam messages then it submits that info to the "Collective" (like the borg). Then my SF could query and learn from the collective about other potential spam and therefore be proactive before I receive the same spam.

Now one of the biggest problems is ranking the spam and some how coming up with a common agreement on what is spam. For example maybe some of you want Cialis emails but the rest of us don't so how do we make sure we don't give up total control over our filters but yet still benefit from each others knowledge.

Maybe there is some thing we can learn from how the MAPS systems determine their blacklist. Or maybe its based on a threshold like Marco suggested. If I could set a threshold on my system of when to have a collective filter kick in then that would give me some level of control.

I obviously haven't flushed out this idea completely but I think the concept is on target. The system is currently way to manual and constantly having to update regex strings is silly and seems very 1970's. :)

Think about it, if someone else has a whole list of regex filters do we all need to all run those same filters ? What I really want is to block a spammer so if Marco has some great filters I don't need his filters what I need is the list of spammers. So the collective would download that info to my system and blacklist those spammers.

Roberto maybe we need to take a chapter out of the Peer to Peer handbook and develop a way for Spamfilter ISP to work together as a network instead of stand alone individuals all duplicating the same efforts.

Lee

Maby have a "shared"DB that collects the blacklisted IPs, and counts how many SFs report the same IP.

Then we could collect data from the DB based on a triggerlevel how many SFs have submitted the IP?

The peer to peer might be a little too forward thinking right now and not easy to implement but I think we should be moving in that direction.

As a stepping stone maybe we could think of this like a virus definition. The way it works now is I don't have to get infected to be protected. Once a new definition is defined all of systems are updated.

Roberto would be the one to jump in on the format but just thinking out loud maybe an IP and domain blacklist would be the logical starting point. This would be a file that SF checks daily and automatically imports and becomes part of a new filter type.

Lee

lee, this should be easy. either logsat or one of us can setup a website just like those MAPS servers and add our server on the first of "MAPS Servers"  This would require no modification on SF.

There would be a little programming necessary but I could handle that. You would need to be able to upload new blacklist and then on my end I would need to parse the files, check for dupes and time stamp them so you don't download existing ips.

But you are right it wouldn't be a big deal.

Lee

Not sure if anyone has ever suggested this before, but what about a common subject line threshold?  I.E. if Marcos mail server suddenly receives a few hundred emails from random From addresses all with the exact same subject line, then obviously the source IP's, the subject line, the URBL entry, etc are all tale tell signs of some spam that needs to be collectivelly squashed and should have all of the obvious "signatures" of that particular spam flagged and entered into the borg...umm...spam filter's collective blacklists.

heh yeah, resistance is futile, just not sure who is saying that line; us or the spammers.

One thing is sure, we are allways one step behind. Thats why we need more intelligent filters, an automated search for patterns in emails that are received in the last n minutes.

We are constantly being hammered by dictionary attacks and the IP Blacklist Cache sounds like a good idea. I would like to see it focus on the Not in AuthorizedToEmail list messages. I think after 5 rejects in a row the IP should be stuck in the IP Blacklist cache, or better yet make it more flexible by being settable by admin.

I have been watching the activity screen and these clever bas*&*&rds are cycling through my domains, but only using a single IP for about 5 attempts or so before moving to another IP and hammering some more. Very seldom do I see them stick around and hammer for 20 attempts so that my other settings kick in and drop the connection.

I think this should be incorporated in SF soon.