Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Not impressed
  FAQ FAQ  Forum Search   Register Register  Login Login

Not impressed

 Post Reply Post Reply
Author
Hyldsgaard View Drop Down
Newbie
Newbie


Joined: 08 August 2007
Status: Offline
Points: 7
Post Options Post Options   Thanks (0) Thanks(0)   Quote Hyldsgaard Quote  Post ReplyReply Direct Link To This Post Topic: Not impressed
    Posted: 27 November 2007 at 3:03am
Every day I receive lots of emails that is obviously spam but Spam Filter just lets it right through.

It catches a lot of spam to, so something is working.

Am I the only one that is not impressed by Spam Filter's efficiency? Do I need to tweak some settings?
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 27 November 2007 at 10:37am
Hyldsgaard,

The most common cause of problems is that SpamFilter is not seeing the real source IP address of the senders. If you have mail being routed in your network so that other servers/firewalls process incoming email and mask the IP address of the sender, then most of the IP-based filtering rules will fails. Another possible issue if you upgraded from SpamFilter "standard" to the "Enterprise" version can occur if you did not import all your file-based filtering settings during the upgrade.

If you can zip and email us a few hours worth of SpamFilter's activity logfile, we'll take a look to see if there's any major configuration problems in your setup.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 29 November 2007 at 6:36pm
Hyldsgaard,

We received your logfile, but it honestly looks as if SpamFilter is working very, very efficiently.

In the 24 hours of the log, SpamFilter received 10,016 connection attempts. Of these, it only allowed 475 emails to be forwarded to your server (of these, 75 of them were delivered as the recipients were in the WhitelistedTO whitelist).

Even if 1 out of 10 good emails you receive is spam, that will still mean that SpamFilter only missed 47 spam emails out of 10,000, and that's about 99.5% accuracy, which is a great ratio.

As a side-note, I noticed that some of your domains have multiple MX records. As you only forwarded us the logfile for one SpamFilter server, I assume you are not running any anti-spam software on the server handling your secondary MX. Please note that spammers are known to send emails to *all* servers listed in the MX records, not just the primary. This means that you are receiving unfiltered spam for the domains with multiple MX records. You can easily verify this by looking at the headers for some of the spam you are receiving. For most of them, you will probably not see the X-SF headers that SpamFilter adds to all the emails it processes. This will tell you which emails are not being processed by SpamFilter.

If we're wrong with our analysis, if you can please forward us 5-10 spam emails that you received on 11/27/07, we'll be able to locate those emails in the logfile you sent, and should be able to find out how those emails slipped thru the cracks.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 30 November 2007 at 3:13pm
Hyldsgaard,
 
I do not know if this will post correctly but, below is a spread of stats on my server for a 24 hour period.  The green are messages that were delivered and the rest were blocked.
 
So, out of 787K messages, only about 91K were delivered and most of those were delivered due to customer white-lists.  I find this to be rather impressive.
 
Roberto may be on to something with the multiple MX records.  My secondary MX has a whopping 98.8 block rate due to spammers attempting to go there because they think it may bypass spam filtering.  Their mistake!
 
Reason Messages Bytes
1 EmailTO is not in AuthorizedTOEmail list 348,986 44.3 % 254.16 M
2 IP is in local blacklist cache 98,118 12.5 % 61.83 M
3 Blacklisted by zen.spamhaus.org. 59,062 7.5 % 47.20 M
4 Whitelisted EMail Address To 37,975 4.8 % 186.14 M
5 SFDB filter match 37,676 4.8 % 29.54 M
6 was queued 35,503 4.5 % 479.35 M
7 Reverse DNS not found 31,541 4.0 % 25.56 M
8 Whitelisted EmailTO 30,365 3.9 % 20.00 M
9 Probe or Unknown 21,705 2.8 % 15.07 M
10 Whitelisted Peer IP 16,608 2.1 % 46.28 M
11 EmailTO is in local blacklist file 11,695 1.5 % 7.34 M
12 Invalid MX record 10,633 1.4 % 7.46 M
13 Blacklisted by SURBL: black.uribl.com. 10,139 1.3 % 7.69 M
14 Whitelisted EMail Address From 6,756 0.9 % 130.54 M
15 content filter 6,745 0.9 % 14.73 M
16 SPF test 5,432 0.7 % 4.39 M
17 Too many connections 3,845 0.5 % 6.50 M
18 no relay allowed 3,617 0.5 % 2.25 M
19 Blacklisted by SURBL: black.uribl.com 2,279 0.3 % 1.94 M
20 Blacklisted by psbl.surriel.com. 2,131 0.3 % 2.00 M
21 EmailFrom is in local blacklist file 2,100 0.3 % 1.78 M
22 Whitelisted - Email size exceeded MaxMsgSizeForSpamFiltering 2,017 0.3 % 2.86 G
23 Blacklisted by bl.spamcop.net. 849 0.1 % 510.00 k
24 infected with the virus 510 0.1 % 1.11 M
25 AutoWhiteList Force Delivery 177 0.0 % 1.38 M
26 Blacklisted by combined.njabl.org. 174 0.0 % 126.00 k
27 Exceeded maximum number of RCPT TO 160 0.0 % 123.00 k
28 IP is in local blacklist file 77 0.0 % 45.00 k
29 signature in embedded image 71 0.0 % 89.00 k
30 Domain is in local blacklist file 38 0.0 % 20.00 k
31 No Data Received 27 0.0 % 45.00 k
32 IP address is from a blacklisted country 16 0.0 % 4.00 k
33 Whitelisted Email From Domain 5 0.0 % 30.00 k
34 Blocked by Honeypot Autofilter 1 0.0 % 0 b
  Total 787,033 100 %   4.19 G
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 30 November 2007 at 3:21pm
Desperado,

How are you collecting these stats?

thanks for sharing.
best regards

Atif
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 30 November 2007 at 3:37pm
Atif,
 
I use "Sawmill" by Flowerpower.  It is a generic log parsing program which I have written a custom "filter Plugin" for.  When shipped, Sawmill does support the LogSat format but does not have my latest filter.  I will be sending that to them.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 30 November 2007 at 3:59pm
Atif,

As an FYI, there's a simple query you can execute that will give you the breakdown of the types of spam currently in the quarantined database. It is nowhere as detailed as the reports Sawmill provides, but could be useful for quick reports:

SELECT tblQuarantine.RejectID, tblRejectCodes.RejectDesc,
COUNT(tblQuarantine.RejectID) AS Total
FROM tblQuarantine INNER JOIN
tblRejectCodes ON tblQuarantine.RejectID =
tblRejectCodes.RejectID
GROUP BY tblQuarantine.RejectID, tblRejectCodes.RejectDesc
ORDER BY TOTAL DESC
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 30 November 2007 at 4:13pm
Atif,
 
Bare in mind that the SQL query is FAST.  The only issue is that if a customer has forced messages through or removed messages, those stats will not be included.  I do, however, rely on the SQL queries for most checking as it is WAY faster than parsing all the logs.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
Stupid View Drop Down
Senior Member
Senior Member


Joined: 28 November 2005
Status: Offline
Points: 127
Post Options Post Options   Thanks (0) Thanks(0)   Quote Stupid Quote  Post ReplyReply Direct Link To This Post Posted: 03 December 2007 at 9:25am
You need to tune it properly for about a month before declare it's ineffective.

After that, you will find you will face a totally different problem, not with Sapmfilter, but with spam - what are you going to do with them?
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 05 December 2007 at 2:58pm
Originally posted by LogSat LogSat wrote:

Atif,

As an FYI, there's a simple query you can execute that will give you the breakdown of the types of spam currently in the quarantined database. It is nowhere as detailed as the reports Sawmill provides, but could be useful for quick reports:

SELECT tblQuarantine.RejectID, tblRejectCodes.RejectDesc,
COUNT(tblQuarantine.RejectID) AS Total
FROM tblQuarantine INNER JOIN
tblRejectCodes ON tblQuarantine.RejectID =
tblRejectCodes.RejectID
GROUP BY tblQuarantine.RejectID, tblRejectCodes.RejectDesc
ORDER BY TOTAL DESC


Roberto, thanks, but please some consideration for us running a caseSensitive OS.

this should be

SELECT tblquarantine.rejectid, tblrejectcodes.rejectdesc,
COUNT(tblquarantine.rejectid) AS Total
FROM tblquarantine INNER JOIN
tblrejectcodes ON tblquarantine.rejectid =
tblrejectcodes.rejectid
GROUP BY tblquarantine.rejectid, tblrejectcodes.rejectdesc
ORDER BY TOTAL DESC
best regards

Atif
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 05 December 2007 at 3:01pm
Desperado,

Do you make reports from your queries every day or can you run your queries at any time.

I dont see how it can be really fast with 20-30 million rows in the tblquarantine.

The parsing the log way is much slower but at least it can be done offline on a different report-server.

Thanks for sharing

best regards

Atif
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 05 December 2007 at 3:17pm
Atif,
 
We try to keep our quarantine below a million records.  At that level, a "quick Query" like Robertos takes about 5 seconds MAX.  A full query which has to do tome text searches still only takes about 35 to 40 seconds.  I would not even think about doing a full query on 20 million records.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 05 December 2007 at 3:40pm
Desperado, I have to to keep 10 days record. So I keep 11 days.
It used to be 20-30 million entries but these days its getting like 35-40 million.

The good thing about this bad effect is that SF is working very well.! :-)


best regards

Atif
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 05 December 2007 at 3:42pm
I get an alert if there are more than 30K quarantines/hour.
These days I am filtering them out a a different folder!

best regards

Atif
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 05 December 2007 at 4:46pm
Originally posted by Hyldsgaard Hyldsgaard wrote:

Every day I receive lots of emails that is obviously spam but Spam Filter just lets it right through.



Not trying  to be a fan boy,  Here are the result of  ONE day analysis
sorry about the fomattig,

+----------+-------------------------------------------+---------+
| rejectid | rejectdesc                                | Total   |
+----------+-------------------------------------------+---------+
|       19 | URL in email found in SURBL search        | 1347328 |
|       13 | Keywords found in content                 | 1103101 |
|       21 | SFDB filter match                         |  607959 |
|       15 | SPF Sender Policy Framework match         |  150683 |
|        1 | Domain is in local blacklist file         |  116063 |
|        8 | IP address is from a blacklisted country  |   73295 |
|       18 | IP blocked by honeypot entry              |   36390 |
|        7 | Exceeded maximum number of RCPT TO        |    4452 |
|       23 | Found prohibited attachment               |    2835 |
|       20 | Detected spam signature in embedded image |    1312 |
|       24 | Blank email with PDF attachment only      |     902 |
|        9 | EmailTO is in local blacklist file        |     890 |
|       25 | Too many spaces in subject                |      92 |
+----------+-------------------------------------------+---------+
13 rows in set (1 min 45.03 sec)



Edited by atifghaffar - 05 December 2007 at 4:47pm
best regards

Atif
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.078 seconds.