Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - SpamFilter ISP v3 beta available
  FAQ FAQ  Forum Search   Register Register  Login Login

SpamFilter ISP v3 beta available

 Post Reply Post Reply Page  12>
Author
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Topic: SpamFilter ISP v3 beta available
    Posted: 31 March 2006 at 4:43pm
We've made available as a pre-release the new SpamFilter ISP v3.0.0.547.

The major additions to this new release are:

The availability of a new filter that scans images embedded in emails for spam content.

A new SFDB (SpamFilter Distributed Blacklist) filter that will rely on the large network of SpamFilter users to create and maintain automatically a centralized blacklist of spammer's IP.

A new web management interface is also being worked on, but due to its complexity it still has not been enabled on this beta. We hope to do so in the coming weeks.

Please note that this version is to be considered as a beta, not as a pre-release, as further testing will be required.

The release notes are as follows:

// New to VersionNumber = '3.0.0.547';
{TODO -cNew : Added a new filter to detect spam in images embedded inside emails}
{TODO -cNew : Added new SpamFilter Distributed Blacklist filter}
{TODO -cNew : Cosmetic touchups to the Configuration tab GUI}
{TODO -cNew : Added safety code to remove duplicates added by external apps to AllowedDomains.txt local domains}
{TODO -cNew : Removed annoying "Exception occurred during OnConnect" exceptions in logs}
{TODO -cFix : Sometimes Socket Errors on MX test could cause rejects (catches even more cases than in build 535)}
{TODO -cFix : Regression error in build 541 caused SURBL and attachment filter to stop working}



Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 02 April 2006 at 11:31am

The config tab is MUCH better!

Kevin



Edited by kspare
Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 02 April 2006 at 5:13pm

The image filter caught 3 messages so far today. Two were obvious spam the 3rd was a sales rep blasting out emails to his customers. kida grey area but i'm personally fine with it sitting in there.

 

Good Job!

Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 03 April 2006 at 9:37am

Great work on the new 3.0 version.

Will run it a couple of days to find bugs.

found one allready :)

- The shrunk qdb lines are back im sorry to inform you about (you know, the qdb lines reducing height to 2 pixels after a refresh)

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 03 April 2006 at 11:54am
Roberto, can you explain SFDB a little bit more? What black lists is it based on?
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 03 April 2006 at 12:28pm
Kevin,

It's a new filter based on a suggestion by Lee ( www.logsat.com/spamfilter/forums/forum_posts.asp?TID=/5531&a mp;PN=1#7577).

A few months ago we created a new filter, the IP blacklist cache. This filter causes any IP that sends multiple spam emails in a short period of time to be added to a temporary memory cache in SpamFilter, creating sort of a local, superfast blacklist.

Starting from v3, when SpamFilter will add an IP address to this memory cache, it will also upload its information (what the IP is, why it was blocked) to our centralized database. As more users begin to deploy SpamFilter v3, this database will grow very rapidly, and will contain in essence a copy of all the blacklist caches of all the SpamFilters in the world. As the cache practically contains the IPs of repeated spammers, the centralized DB will contian a dynamic, ever changing, self-updating list of IPs that are currently being blocked by all the SpamFilters in the world.

Every copy of SpamFilter is then able to query this database in realtime, and see if the connecting IP is listed in the database. SpamFilter will then query the database to see how many different users have reported that IP address at that time, and will apply a "confidence check" (meaning more users have reported the spammer, the less likely it is a false positive, and will decide if the email should be blocked or not.

The database is updated in realtime when an IP is added and removed from the cache, and furthermore we perfrom hourly "cleanup" on it to remove stale entries that have not been updated in the last 24 hours.

This is all experimental with this new version, and in a few weeks we'll be able to see hwo this filter performs.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Dan B View Drop Down
Senior Member
Senior Member
Avatar

Joined: 09 February 2005
Location: United States
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote Dan B Quote  Post ReplyReply Direct Link To This Post Posted: 04 April 2006 at 3:14pm

R,

Within the AllowedDomainFilterMatrix.txt file I noticed that there are 2 additional parameters at the end with this version.  Can you let me know what they are?

Thanks,
Dan B

Back to Top
Shade View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Shade Quote  Post ReplyReply Direct Link To This Post Posted: 08 April 2006 at 6:19am

Dan B : The first is allow or disable image filtering, second is for SFDB

Back to Top
Lee View Drop Down
Groupie
Groupie


Joined: 04 February 2005
Location: United States
Status: Offline
Points: 50
Post Options Post Options   Thanks (0) Thanks(0)   Quote Lee Quote  Post ReplyReply Direct Link To This Post Posted: 08 April 2006 at 11:28pm

Roberto thank you very much for the SFDB. If you can make this work I believe this will be a unique feature that sets Spamfilter ISP from any other product on the market.

After reading your description of how this works I thought maybe I would make another suggestion (sorry) :) What about a ranking system. The idea is that if an IP reach X number of hits it stays in the cache longer or maybe for good.

My thought is if an IP is only seen a few times over some period of time it gets flushed but if an IP continues to show up OR reaches a high reporting level it never gets purged and is considered blacklisted. Who knows may be it can get added to the my local blacklist listing.

Lee

Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 10 April 2006 at 5:13am

allthough i agree with Lee on the suggestion in general, i must point out that lots of people own an infected computer and are completely unaware of this.

Putting an automatic permanent ipblock on would potentially disable a lot of people's mail traffic, as well as the spam/viruses.

Some simply don't have the money to buy decent virusscanners, some don't give a rat's beehind, some (elderly) are unaware of the problem. It just sounds a bit too radical to me, especially when other SF systems are copying the block.

 

 

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
Lee View Drop Down
Groupie
Groupie


Joined: 04 February 2005
Location: United States
Status: Offline
Points: 50
Post Options Post Options   Thanks (0) Thanks(0)   Quote Lee Quote  Post ReplyReply Direct Link To This Post Posted: 10 April 2006 at 8:54am

Marco my suggestion has more to do with the way the SF collective (my term for the body of Spamfilter nodes) holds on to IP addresses of those systems that are sending out spam.

If you think about it this is not really any different than any of the Blackhole servers. When an IP is found to be a relaying server or a source of spam it gets blocked and added to the world wide listing. Spamfilter ISP would do the same except it sounds like Roberto has this using a cache mode where it gets purged quite frequently.

My suggestion is that once an IP address reaches some threshold it does not get purged but stays on the list. Maybe the way to handle this is at some point in a future release of 3.0 this could be set by the SF owner.

Say for example I set my threshold at 50 or 100, once an IP is reported by that many nodes in the collective then my Spamfilter server adds that to the static list of blacklisted addresses on my server.

To be honest I am not sure if this is really necessary and we will all have to do testing with this new feature to determine that. My only concern was that cronic spammers don't keep showing up in the list. But in a way this might be self correcting because others will be reporting those addresses.

Either way I am excited to see how this feature evolves and I trust Roberto and the team to make this determination. There will no doubt be changes and tweaks but once this gets worked out I believe this will be another great tool we can all use to keep spam out of our systems.

Lee

Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 10 April 2006 at 11:21am

hehe, ok, but i do suggest that people should get an opportunity to knock on the SF-hive door (at the risk of beeing assimilated) and ask for a release of the block. but who should act as the SF-queen? :)

Issueing permanent blocks demands someone to control those, and since this is growing into a collective who is to play that role?

But as you said, we'll have to see what Roberto comes up with, something tells me he'll find a good way to handle it :)

 

 

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
pcmatt View Drop Down
Senior Member
Senior Member
Avatar

Joined: 15 February 2005
Location: United States
Status: Offline
Points: 116
Post Options Post Options   Thanks (0) Thanks(0)   Quote pcmatt Quote  Post ReplyReply Direct Link To This Post Posted: 10 April 2006 at 3:54pm
How do I turn off SFDB globally in the INI?  I want to use the other new features, just not SFDB.
-Matt R
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 10 April 2006 at 10:13pm
Matt,

As with most other filters, just enter a value of "0" in the Network Reliability field in the SFDB tab under the settings tab. This will disable the SFDB filter globally.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 11 April 2006 at 11:23am
OK ... Time for my 2 cents.  Where is the page http://www.logsat.com/SFDB/why.asp and would not this be a good place to request removal, if required?
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 11 April 2006 at 11:33am
Ha!
Perfect timing... We've actually been coding the why.asp page all morning, and is still in progress...
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
JohnD View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote JohnD Quote  Post ReplyReply Direct Link To This Post Posted: 11 April 2006 at 4:33pm
Wow,  A world wide SF-Blakclist.  I love it.  I also love Lee's idea.  Keep a count of how often an IP is reported per hour/day. Every hour/day reduce the count by some "Fairness" factor.  Once the count reaches 0 or below remove the ip address.  This way if granny fixes her virus issue, she can send mail to me again.  Otherwise, I don't want her emails, harmless or not.
Back to Top
pcmatt View Drop Down
Senior Member
Senior Member
Avatar

Joined: 15 February 2005
Location: United States
Status: Offline
Points: 116
Post Options Post Options   Thanks (0) Thanks(0)   Quote pcmatt Quote  Post ReplyReply Direct Link To This Post Posted: 11 April 2006 at 4:56pm

Sorry about my earlier brain dead post while the answer was right on the screen to enter 0 to disable.

Maybe this comment will invoke some productive thought:

I would only use this feature if the filter that caused the block was included in the central database and any SFDB blocks enforced on my server matched my filter selections.  

For example, I don't want to block using MX filter, but maybe Desperado does.  My program should not care about SFDB blocks that were caused by MX filter in this case.

Make sense?



Edited by pcmatt
-Matt R
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 11 April 2006 at 5:22pm
Matt,

I understand your logic, but it is to be noted that the SFDB filter will not block an email just becuse an IP address was blocked by a single SpamFilter user. The confidence level, or "Network Reliability" value tells SpamFilter how many *different* users need to have reported the IP address in question for your SpamFilter to block it. This will, statistically, allow for a greater accuracy in the block, as there need to be multiple users who report the same IP, with possibly different filtering settings, for ti to be blocked.

Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 4:39am

Quote .. or "Network Reliability" value tells SpamFilter how many *different* users need to have reported the IP address in question ..

 

Roberto, i suggest adding a percentage option. So one could say '3' for three users or '2%' for a percentage.

Example: if 2% of all SF users think it's spam, treat it as spam.

As the SFDB network grows, the setting of '3' will become more and more aggressive, since the chance that 3 or more people have ultra-radical filters installed  becomes greater. introducing a percentage of users setting counters this.

 

 

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
pcmatt View Drop Down
Senior Member
Senior Member
Avatar

Joined: 15 February 2005
Location: United States
Status: Offline
Points: 116
Post Options Post Options   Thanks (0) Thanks(0)   Quote pcmatt Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 8:28am

Roberto,

I was aware of the points in your comments and to me it would be a useless feature to block based on anonymous uncategorized data. 

Each installation's cached IP blocks are the result of unique filter configurations that match the goals and standards of the individual user.

To block messages based on blocks caused by any filter would be too aggressive for any environment where no false positives is the primary goal. Half of the filters in the program are too aggressive for low or no false positive results.  Some people are so sick of spam they are quite happy with blocking email as well as spam.  These differences in philosophy on how "I" will fight spam and the flexibility of the program is what makes it a superior solution. You should not disregard this value when creating a new feature.

If SFDB is not filter specific, it would be useless. Making it fitler specific would be quite simple.  Use a bit value and the additional data would be miniscule.  SpamFilter should query for common blocks that match the bit value of the running configuration. THEN, you really have something that upgrades the program.

Using a "confidence level" probably would not be needed if the data was filter specific.  A "confidence level" as it is used now does not achieve any type of data accuracy and is virtually useless. All it does is create an illogical damper on using non-specific data.  The path the feature is on now makes it an email blocking feature, not a spam blocking feature.

-MJR

-Matt R
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 8:50am

pcmatt,

Hmmm ... I kinda agree with you on the "Bit Value" thing.  I, too have been a little worried about getting overly agressive samples from other users.  HOWEVER, even with the "confidence level" of 2 which is lower than recomended,  I have see very good results.  My worry is how much additional overhead would be required to set a bit based on what filter caused the entry and for now anyway, the existing "bulk" values seem real good.  I am waiting until more usere upgrade before I go to jury.

The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
pcmatt View Drop Down
Senior Member
Senior Member
Avatar

Joined: 15 February 2005
Location: United States
Status: Offline
Points: 116
Post Options Post Options   Thanks (0) Thanks(0)   Quote pcmatt Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 9:43am

In less than a day I saw the filter with default confidence level block spam and legitimate email equally. 

The opportunity exists to follow this new feature on a path that is contiguous with the rest of the program and makes it a powerful "spam" blocking filter.  I have no use for a filter where I can not explain to the sender nor the recipient the reason an email was rejected or quarantined.

There are other design issues to be considered too. For example, this feature should likely use UDP only to ahchieve the best performance and lowest overhead.  How are "rogue" users posting garbage to the database going to be detected and blocked?  How will access to the SFDB database be restricted so it is secure?  What other controls and checks will be needed?

Don't misunderstand, I like this idea if it upgrades the program for us all or at least the majority.

-Matt R
Back to Top
pcmatt View Drop Down
Senior Member
Senior Member
Avatar

Joined: 15 February 2005
Location: United States
Status: Offline
Points: 116
Post Options Post Options   Thanks (0) Thanks(0)   Quote pcmatt Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 9:45am

I saw lots of false positives from the image filter before I disabled it.  Just like SFDB it was about equal, good blocks and false positives blocking email.

Is there any documentation on what criteria is being used?  How could I tell if an email would be blocked by this filter?

-Matt R
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 9:56am

just brainstorming here; how about this idea:

Suppose some users are catching an IP through the keyword filter, while others are catching the same ip through other means.

If the bitvalues Matt suggested AND the keywords are sent to SFDB, which in turn would 'validate' the triggering keywords,once enough users are confirming that the IP is indeed a spammer, and would return the keyword (RegEx), plus the found ip to all SF users. My bet is that loads of IP's would show up and in turn be blocked if confirmed by enough sf users.

This way we would be creating a network that uses all of our combined knowledge and distribute it to all of us.

I do see trouble with this system as well though.. a keyword that is too simple would catch nearly anything, so those will have to be discarded somehow.

what do you all think?

 

 



Edited by Marco
Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 10:21am
We're "digesting" everyone's comments, so please continue to brainstorm, we're listenign as always!

Security-wise, in case anyone didn't check, all the updates and queries are performed via encrypted parameters in the HTTP stream, so it will be rather difficult for anyone to purposely "inject" bad data without using SpamFilter. We're using http rather than UDP because SpamFilter is performing updates along with queries, and for practicality, as http is easier for us to manage on the backend.

For the "filter-specific", the SFDB is already collecting the specific reason for the block report, but we're not currently using it on the query. In the future, we make make the lokups more powerful by selecting which filters must have been used (and which not) to create a report for an IP.

Our short term plan is to release this version to free users as well to obtain a much larger IP database. After that, we'll wait and see...
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
JohnD View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote JohnD Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 10:31am

I think we have a great group of thinkers  here.  If the rest of the admins in the world were as vigilant, we wouldn't have a spam problem.  I like the Idea of a bit value, but I am more concerned about overhead, than false positives.  That said, it becomes a matter of choice really.  Marco's percent Idea would help to temper the more aggressive admins as well as offset the less aggressive admins. 

Matt,  as far as telling an offender why they were rejected, I agree that this is an important aspect of spam fighting.  For this reason I would side with the bitvalue approach.  I have given this some thought and aside from the fact that the SPF database would need new fields to id why something was rejected, there is little extra overhead for the clients.  I for one believe that I would not want someone else's list of keyword determining what is spam and what isn't. Also, there are MAPS servers out there that are way to aggressive for my liking.

 

Thats my 2 cents for now

 

John,

 

Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 12:01pm

pcmatt,

I am still finding it hard to understand why you are getting so many false positives as I have not yet seen any with one exception which I whitelisted.  Could the reason be that I have a semi large white list for many valid listservers?  Or, are you including adult content as False Positives.  Strickly speaking, Adult content is not automatically SPAM.  As an ISP, we have many users the WANT their adult content.

Any examples would help.

The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
Matt R View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Matt R Quote  Post ReplyReply Direct Link To This Post Posted: 12 April 2006 at 11:51pm

Obviously we run completely different operations. Luckily Spamfilter is flexible enough to serve many different types of operations. Our users are strictly corporate users and our strategy for fighting spam begain development years before SpamFilter was out.  Our configuration is obviously much different than yours. That does not make yours nor ours invalid nor problematic, just different.

Therefore my comments on hoping for new features that can serve us ALL and not just you OR me, for example.

 

Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 13 April 2006 at 9:42am
Well,  that was kinda harsh but ... we run totally separate instances for our corporate customers which comprise about 80% of our business and a "bulk" instance for our residential customers.  Our business customers have "Portals" to their own filters and blocking features and we hold short classes for their administrators explaining all the SpamFilter featurs and the good & bad of using each one.  As a result, they achieve a very good balance that they are happy with. So, most of my question above had nothing to do with pushing the feature but to see where it was failing for you so that, as a customer that tests the beta fearures, I could be on the look out for the type of problems ALL users are having.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
 Post Reply Post Reply Page  12>
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.094 seconds.