Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - SawMill Log Filters
  FAQ FAQ  Forum Search   Register Register  Login Login

SawMill Log Filters
 Post Reply Post Reply
Author
  Topic Search Topic Search  Topic Options Topic Options
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Topic: SawMill Log Filters
    Posted: 07 March 2005 at 3:40pm
All,
 
For those of you that use "SawMill" to parse the SpamFilterISP logs, the existing "Log Format" files are useless due to the MANY changes in the SpamFilter log format.  I have re-written them from the ground up and get fairly good results with the exception that "Probes" are not logged as a "Reason" or "Action" and there is a limit to how accurate parsing can get anyway.  Without going crazy, I am happy with the results I get.  I will continue to refine them as the logs change until I then submit them to SawMill for update.  In the meantime, they are available at:
 
 
IMPORTANT:  I am still working on the logs so check the file dates.
 
Edited:
 
I have updated my Version 7 Filter.  I spent a lot of time parsing the logs with Perl to find the "errors" in my filters and found that the seemingly large discrepancy is actually a function of how SawMill handles a single message with a zillion RCPT To's. Also, if the logs have a lot of "Exceeded Max RCTP To" actions, the way SpamFilter logs them, SawMill only starts counting AFTER the limit is reached.  So, bottom line, I believe that my newest filters work as well as possible and even with the discrepancies, yield very usefull statistics.   As most of you know, log parsing is an "Art" not a science and as such, I will be thrilled to death if someone else can improve on what I hve spent many hours on.
 
Edited: ADDED:  IP "Attacks" (Too Many Connections)  das
Edited: ADDED:  "No Data"  (SpamFilter build 435 and above)  das
Edited: REMOVED  IP "Attacks" (Caused some error issues)  das
 
Thanks, and
Regards,
 


Edited by Desperado
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2005 at 4:08pm

Here is one table from Sawmill.  Not that the stats are MUCH lower than the SpamFilter Database.  This is because Spamfilters database is reporting ALL messages while SawMill is reporting just the connection and the one virus it had to delete even if it goes to a zillion addresses.  I hope this shows up correctly.

This is for The month of March.

  Virus Messages Bytes
1 Sober.K@mm 2,319 49.6 % 157.40 M
2 Netsky.P@mm 1,255 26.8 % 50.10 M
3 Netsky.B@mm 131 2.8 % 3.71 M
4 Bagle.AH@mm 90 1.9 % 2.86 M
5 Netsky.C@mm 74 1.6 % 2.44 M
6 Netsky.D@mm 70 1.5 % 1.57 M
7 Netsky.Q@mm 67 1.4 % 2.56 M
8 Mabutu.A@mm 58 1.2 % 3.34 M
9 Netsky.K@mm 55 1.2 % 1.56 M
10 Netsky.Z@mm 50 1.1 % 1.52 M
11 Lovgate.AB@mm 47 1.0 % 7.87 M
12 Bagle.J@mm 45 1.0 % 765.00 k
13 Bagle.N@mm 42 0.9 % 1.18 M
14 Bagle.BC@mm 38 0.8 % 1.02 M
15 Bifrose.D 38 0.8 % 2.36 M
16 MyDoom.I@mm 35 0.8 % 2.81 M
17 Bagle.AR@mm 32 0.7 % 983.00 k
18 Bagle.BB@mm 32 0.7 % 844.00 k
19 MyDoom.J@mm 32 0.7 % 2.12 M
20 MyDoom.L@mm 30 0.6 % 1.16 M
21 W32/Downloader 21 0.5 % 441.00 k
22 W32/FunLove.4099 14 0.3 % 693.00 k
23 W32/Bagle.Gen!Zip 13 0.3 % 266.00 k
24 Netsky.AB@mm 12 0.3 % 290.00 k
25 Netsky.AD@mm 11 0.2 % 462.00 k
26 Bagle.AF@mm 10 0.2 % 296.00 k
27 Netsky.W@mm 8 0.2 % 318.00 k
28 W32/Bagle.Gen!Rar 6 0.1 % 188.00 k
29 Netsky.T@mm 6 0.1 % 150.00 k
30 Zafi.D@mm 6 0.1 % 108.00 k
31 W95/Pinfi.A 4 0.1 % 1.10 M
32 Netsky.X@mm 4 0.1 % 140.00 k
  17 other items 24 0.5 % 1.14 M
  Total 4,679 100 % 253.63 M

  Reason Messages Bytes
1 Reverse DNS not found 68,879 28.0 % 334.19 M
2 Bypassed all rules 58,789 23.9 % 1.17 G
3 Blacklisted by sbl-xbl.spamhaus.org. 49,640 20.2 % 174.23 M
4 Blacklisted by dnsbl.sorbs.net. 10,649 4.3 % 65.63 M
5 content filter 10,192 4.1 % 482.93 M
6 Invalid MX record 8,473 3.4 % 55.19 M
7 EmailTO is in local blacklist file 7,095 2.9 % 236.00 k
8 SPF test 6,581 2.7 % 55.93 M
9 no relay allowed 6,452 2.6 % 2.00 k
10 Blacklisted by bl.spamcop.net. 5,105 2.1 % 102.29 M
11 infected with the virus 4,679 1.9 % 253.63 M
12 EmailFrom is in local blacklist file 3,159 1.3 % 10.48 M
13 Blacklisted by dnsbl.njabl.org. 2,513 1.0 % 9.65 M
14 Blacklisted 1,357 0.6 % 6.67 M
15 Exceeded maximum number of RCPT TO 1,323 0.5 % 6.22 M
16 IP address is from a blacklisted country 693 0.3 % 2.62 M
17 Found prohibited attachment 329 0.1 % 924.00 k
18 Domain is in local blacklist file 51 0.0 % 73.00 k
  Total 245,959 100 % 2.70 G

Regards,



Edited by Desperado
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.250 seconds.

Copyright (c)2002-2025 LogSat Software LLC - Sales: sales@LogSat.com
Contact Us