Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Submit Spam Feature
  FAQ FAQ  Forum Search   Register Register  Login Login

Submit Spam Feature

 Post Reply Post Reply Page  12>
Author
jerbo128 View Drop Down
Senior Member
Senior Member
Avatar

Joined: 06 March 2006
Status: Offline
Points: 178
Post Options Post Options   Thanks (0) Thanks(0)   Quote jerbo128 Quote  Post ReplyReply Direct Link To This Post Topic: Submit Spam Feature
    Posted: 24 April 2006 at 12:10pm

I know that this has been brought up before - I am not trying to beat a dead dog...

I have some users that despite how "tight" I make the filter, they continue to receive spam that the filter has missed.

I would really like to see a feature where I (as an administrator) can submit spam that was not caught by the server.  Right now, our users have been trained to copy email headers and forward messages to spam@domain.com.  We then go through those emails and manually blacklist ips, and add keywords to our list.  No only is this VERY time consuming, but it is not very effective in the big picture.

If anyone else has ideas - I would like to hear from you.

Jerbo128

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 24 April 2006 at 3:04pm

While wishing for features I'll bring up the dead horse I keep flogging

It would be great to have a spamassassin plugin (similar to anti-virus plugin) Currently I'm using Mdaemon (only for Spamassassin, all other filtering disabled)  behind my SFI and it catches that last little bit that manages to get by SF and keeps me from having to learn regex

I currently have it set up to test for the Autowhitelist flag recently incorporated into SFI and if it exists then it bypasses spamassassin check. Of course I can't teach SF about the spam but I can at least make sure my customers never see it.

Would be nice to have it all in one. I think thats the last thing SF is missing in an otherwise fantastic product.

http://www.webguyz.net
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 25 April 2006 at 9:41am

SUppose a user reports spam to spam@domain.com. , and your mailserver would relay mail to this specific adress back to the SPF server, which in turn would recognise the TO adress, and act on it.

In effect 'certain adresses' could be considered 'inbound' for the SPF server. This also opens the opportunity for internet users to report spam to your SPF system.

If the report is presented to the SPF system in a certain format, so that the SPF engine can use the data within to effectively issue a block on the reported spammer...

What do you think Roberto?

 

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 25 April 2006 at 10:56pm
This has indeed been brought up before. I'm afraid we're going to "stall" for a bit more and see how the new SFDB filter performs. Submitting spam has challenges, and what Marco mentions as a problem is indeed one... Another alternative is to store the sender's info in a database for some time to allow easy recovery/blacklisting via a web interface by the users. In any case there's several considerations to make as this in not going to be simple for us, for the admins and for the end-users...

Thus the "stalling" on our end !
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Dingo View Drop Down
Newbie
Newbie


Joined: 06 April 2006
Status: Offline
Points: 1
Post Options Post Options   Thanks (0) Thanks(0)   Quote Dingo Quote  Post ReplyReply Direct Link To This Post Posted: 17 May 2006 at 6:45am
What tests does the bayesian filter learn from ?

Could i simply forward a spam back through SF that i edit to intentionaly fail a test. If this could be a workaround, what test would be best to cause the bayesian filter to learn it and would not result in my IP etc being banned ?

Regards

Dave
Back to Top
sgeorge View Drop Down
Senior Member
Senior Member


Joined: 23 August 2005
Status: Offline
Points: 178
Post Options Post Options   Thanks (0) Thanks(0)   Quote sgeorge Quote  Post ReplyReply Direct Link To This Post Posted: 17 May 2006 at 10:35am
A good idea indeed Dave, but I know that Robert has considered that idea before... I remember his response being that even by just forwarding a message to SpamFilter for the Bayesian filter to learn, you add a whole new set of keywords (mostly headers) to the message.  The result is that the filter is not as likely to block that message as it really should.

Jerbo, I currently use the same method that you do (in fact, I have a spam@mydomain.com address too).  It is time consuming and honestly, I can only really act on about 20% of the spam that our users submit (If I added a keyword entry for every spam subject I came across, my keyword file would be in MB, not KB..).

In any case, for those who are interested in trapping that spam that still gets through, you may want to check out pcmatts post on this thread:
http://www.logsat.com/spamfilter/forums/forum_posts.asp?TID= 5314&KW=pcmatt

Stephen

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 17 May 2006 at 10:46am

For that last bit of spam we wrote some tools using some email COM components. Basically we strip out any header info that SF adds and remove the last received from address and feed it through but add a single ^ to the subject line and our keyword filter is looking for that ^ and will reject the mail. The only difference between the original message and the one we resend is ^ in the Subject and of course we have to use our IP address as we can not spoof that. Jury is still out on how this effective it is but if we had an option in SF to have a directory where only known spam was placed I could easily make the message look 100% like the original stripping out anything added by SF and removing out ip from the Recieved line.

Lots of work but I really hate spam.

http://www.webguyz.net
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 3:56pm

What I would like is to have a seperate DB where all "good" mail is archived to, exactly the same way as the quarantine DB works. Then, when a user gets an email that is spam, they forward it to spam@domain.com . When this mail arrives, we could easily figure which  origional email this is, and as we have the origional...we could retrain the bysian based on the origional email.

 

Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 3:59pm

forgot to add.....

ANY email filtering program that cannot be retrained is not that great.

IF a spam gets past the filters, then the filters arn't working properly and they need to be retrained. ?This can only be done when a user gets spam and sends it back to the server to be retrained.

Question...Logsat...what do you suggest users do with spam emails that your program failed to detect????

 

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 11:18pm
This topic has been discussed in the past. The method you describe to retrain the bayesian filter is certainly possible. It is to note however that the Bayesian filter stops a very small percentage of spam when compared to the other filters.
We opted to concentrate our resources this past year on developing new filters like the spam-image filter, and the extremely successful SFDB (SpamFilter Distributed Database). These filters block a huge amount spam when compared to the Bayesian filter.
We felt that rather "slightly" improve the Bayesian filter, SpamFilter would be more successful if we developed new filtering tecniques.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 11 August 2006 at 4:20am

Logsat,

I understand, however teo things spring to mind. Firstly, we are still receiving emails that get through all of the current filters and have no way to teach SF that these are indeed spam emails.

Secondly, can you please explain exactly how the SFDB works, as a HUGE customer (Tesco ... the largest supermarket in the UK) seems to have their mail server's IP listed in SFDB, but not in any other DB as in spamcop, spamhause, ordb etc.

Also, how can we get this IP delisted??

 

Cheers

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 11 August 2006 at 7:46am
If it's listed it's... probably because they are sending unsolicited emails. If you login the registered area on our website, you'll be able to enter an IP address and perform a lookup in the SFDB to see how many reports there are for that IP, and for what reason it was reported.

Our SFDB is updated in realtime, so the split second they would start an email campaign to their subscriber list, should they emails to unsolicited recipients, they will be blacklisted. If the campaign stops, they will be delisted automatically in either 6 or 24 hours, depending on the amount of spam they had.

If it's not listed in other services... well, that may just mean the SFDB is better than others
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 11 August 2006 at 8:01am
ok, no problem....exactly where would this registered area login page be??? I can't find it.
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 11 August 2006 at 4:04pm
The link is found from the download page. More specifically, it's:

http://www.logsat.com/sfi-login.asp

Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 11 August 2006 at 4:06pm
got it..cheers
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 9:42am

SFDB went straight to the top after it's implementation. It simply is the best performing filter (dare i say it) on the web....

 

How it works: if a (settable) number of SF users declare a certain mail as beeing spam, ALL SF members reject the mail in question. So it's a joint effort / joint intelect type of filter. ANY spam that is not caught by SFDB filter, is because of 'your own' doing, and SFDB is fed with that.

 

 



Edited by Marco
Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 9:45am

I'm sorry, but I don't share you'r feelings......

SFDB blacklisted Tesco, the largest supermarket in the UK.....well donw...NOT !!!!!

Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 9:48am

refresh and see what i added.. then you'll undersdtand

sorry, I didnt describe it correctly, it should be;

*my* SF will block the sender since 'N' number of users declared the mail as beeing spam...

Apparantly, a number of the SF users think the tesco mails are spam, and your system believes it, since 'n' number of users say so.



Edited by Marco
Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 10:06am

Marco,

I know how it works, and I think it has at least one fatal flaw in it's design.

If 10,000 users sign up for a mailing list, then some time later, 100 of then descide that they no longer want it....invariably they don't bother to unsubscribe, they just mark it as spam.

Using this scenario, lazy/stupid/ignorant users can quite easily report an email as being spam, whereas in fact it isn't. The problem here is that if the company in question uses a single mail server IP for both the mailing list and the normal company mail (as Tesco do), then all of the normal email (Tesco use email to place orders with their suppliers, which can be for hundreds of thousands of pounds) will also be blocked.

Therefore, the failing here is that SFDB has no confirmation that the email that is being submitted is in fact spam.

I would suggest that it is very likely that the number of users declaring mail as spam is far too small.

Just my opinion...

Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 10:17am

As far as i know, your spam/mail system is 'only' one user in the SFDB chain, so, no matter how many of your internal users decalre the mailer as spammer, your system will issue the sending IP ONCE to SFDB.

So, if 'n' number of antiispamsystems say it is spam (read: ISP's/businesses), your system will adhere to that, no matter HOW OFTEN those ISP/businesses report the sending IP as spammer, it will only keep the ip alive in the SFDB.

you know the fun thing? you can bypass the SFDB alltogether, or increase the number, so that the chance of a falce positive gets smaller.

the setting "network reliability' number is the one you need to set to zero, or raise...

 

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 10:53am

Excellent...didn't know what that setting was...cheers.

any suggestion on a suitable number...it defaults to 3.....maybe a bit low

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 11:09am

Had mine set to 3 but then a Hotmail server started getting rejected by SFDB so I bumped mine up to 6. Seems to be pretty good and haven't seen any more major ISP servers stopped.

http://www.webguyz.net
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 11:10am

yeah, 3 might be a bit too low for your organisation, try 10 and see what happens, if your false positives go away,  work your way down by one a day/week untill they return. The lower the number the more 'aggressive' the filter works, the higher the number, the lesser affective it becomes.

You need to find out what works best for you by trial and error.

 

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 11:33am

p.s. you can allways whitelist this tesco domain, if you think it ok all your internal users can receive their mails.

 

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 4:26pm
Marco,

Thanks for your explanation, you are right on the money. We actually slightly increased the lower limit on the SFDB lookups on our database server itself. This means that our SFDB lookups won't provide a positive match unless there are at least nnn other reports. I'd like to keep the number private so that lurker spammers don't see it. If you wish to know just PM/email us, we'll provide the value.

StevenJohns, if Tesco was listed in the SFDB, it's because a rather large number of recipients have complained. And while I have no proof, I'm guessing it's quite larger than 100... Please note that there have to be a certain number of *different* providers/companies running SpamFilter that have received emails from Tesco, not just one company. For example, if 100 users at LogSat.com complained that it was spam, the report would be only one, from logsat.com. However, the 100 users from LogSat.com would report the spam to MAPS RBL services, like sorbs or spamcop. In this case, if these service blacklist Tesco, and multiple SpamFilter administrators use these RBLs, then yes, each SpamFilter installation would send a report. In this case, the SFDB would then contain the same information as the RBL. If users believe this is wrong, or do not wish to use the SFDB for IPs that were blacklisted due to RBLs, the SFDB lookups can be configured to not use RBL (or any other filter you wish).
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 4:49pm

Roberto,

Now I'm confused...

An earlier post suggested that the "Network Reliability" number was the minimum number of "spam complaints" that an IP had to have before SF DB would block the IP. Then, (as the default is 3), Webguyz suggested 6, Marco suggested 10 as being a suitable number, but you are suggesting that the tesco IP might have more than 100 "complaints".

Please can you have a look at your database and give me some examples of how many "complaints" some IP's typically get. I understand that you may not wish prying spammers to know this, so feel free to PM it to me. This will give me a good idea as to what a realistic number should be.

 

Cheers

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 5:04pm
As I mentioned earlier, you can always login the registered user area of the website, and perform detailed lookups on the SFDB database to see more details on what is being blocked and for what reason.

Let me explain the "100 complaints".

Consider the following example:

SpamFilter is shipped with default RBL servers. One of them is spamcop's bl.spamcop.net. Tesco sends 10,000 emails, and 100 customers complain and report Tesco to spamcop. Spamcop will then blacklist Tesco in its blacklist. Hundreds of SpamFilter administrators are using spamcop, so when they receive an email form Tesco, they will report it as spam to the SFDB. If more than "nn" different SpamFilter's report that IP to the SFDB, then the SFDB will list the IP.

Please note that Tesco would have to send emails to users who's provider runs SpamFilter (and is using spamcop), in order for SpamFilter to report the IP.

Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
sgeorge View Drop Down
Senior Member
Senior Member


Joined: 23 August 2005
Status: Offline
Points: 178
Post Options Post Options   Thanks (0) Thanks(0)   Quote sgeorge Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 5:08pm
Steven, maybe I can try to explain...

The network reliability number is the number of registered SpamFilter installations that have reported one or more blocked emails from a particular email.

Let's try a hypothetical example...
  • Your network reliability level is set to 3
  • my domain, which uses the SFDB, blocks 20 messages from 123.123.123.123.  my domain submits 123.123.123.123 to the SFDB, and the SFDB accepts 1 and only 1 submission for 123.123.123.123 from me.
  • Joe's domain, also on the SFDB, blocks 5 messages from 123.123.123.123; that becomes the 2nd submission for 123.123.123.123 on the SFDB.
  • Bob's domain, also on the SFDB, blocks 5 message from 123.123.123.123 as well;that becomes the 2nd submission for 123.123.123.123 on the SFDB.
  • You are receiving a message from 123.123.123.123.  Your SpamFilter setup is trying to determine if it should accept the connection.  When you query the SFDB, it says 123.123.123.123 has been blocked 3 times.  Since this is greater than or equal to your network reliability limit, you block the message from 123.123.123.123.
If I'm not making sense, then that's just me being myself.

Stephen
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 5:46pm

Roberto, got the PM..thanks.

sgeorge, thanks for the explanation.

I think that I need to whitelist the tesco domain for certain local domains. I know that I can whitelist certain sender email addresses for specific recipients, but can I whitelist *@tesco.com for *@mydomain.com ?? if so, how??

 

Cheers

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 16 August 2006 at 11:30pm
... there's actually a bug in SpamFilter that helps in your request. Please see the thread at http://www.logsat.com/spamfilter/forums/forum_posts.asp?TID= 4970.

It will also explain why the bug won't be fixed
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
 Post Reply Post Reply Page  12>
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.093 seconds.