SawMill Log Filters
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=5092
Printed Date: 11 December 2025 at 11:27am
Topic: SawMill Log Filters
Posted By: Desperado
Subject: SawMill Log Filters
Date Posted: 07 March 2005 at 3:40pm
|
All,
For those of you that use "SawMill" to parse the SpamFilterISP logs, the existing "Log Format" files are useless due to the MANY changes in the SpamFilter log format. I have re-written them from the ground up and get fairly good results with the exception that "Probes" are not logged as a "Reason" or "Action" and there is a limit to how accurate parsing can get anyway. Without going crazy, I am happy with the results I get. I will continue to refine them as the logs change until I then submit them to SawMill for update. In the meantime, they are available at:
Sawmill Version 6: http://spamman.mags.net/sawmill/SpamFilterISP - http://spamman.mags.net/sawmill/SpamFilterISP
Sawmill Version 7: http://spamman.mags.net/sawmill/logsat_spam_filter_isp.cfg - http://spamman.mags.net/sawmill/logsat_spam_filter_isp.cfg
IMPORTANT: I am still working on the logs so check the file dates.
Edited:
I have updated my Version 7 Filter. I spent a lot of time parsing the logs with Perl to find the "errors" in my filters and found that the seemingly large discrepancy is actually a function of how SawMill handles a single message with a zillion RCPT To's. Also, if the logs have a lot of "Exceeded Max RCTP To" actions, the way SpamFilter logs them, SawMill only starts counting AFTER the limit is reached. So, bottom line, I believe that my newest filters work as well as possible and even with the discrepancies, yield very usefull statistics. As most of you know, log parsing is an "Art" not a science and as such, I will be thrilled to death if someone else can improve on what I hve spent many hours on.
Edited: ADDED: IP "Attacks" (Too Many Connections) das
Edited: ADDED: "No Data" (SpamFilter build 435 and above) das
Edited: REMOVED IP "Attacks" (Caused some error issues) das
Thanks, and
Regards,
-------------
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com
|
Replies:
Posted By: Desperado
Date Posted: 07 March 2005 at 4:08pm
|
Here is one table from Sawmill. Not that the stats are MUCH lower than the SpamFilter Database. This is because Spamfilters database is reporting ALL messages while SawMill is reporting just the connection and the one virus it had to delete even if it goes to a zillion addresses. I hope this shows up correctly.
This is for The month of March.
| |
javascript ;" target=_blank onclick="set_sort_column0, 'virus'; return false; - Virus |
javascript ;" target=_blank onclick="set_sort_column0, 'messages'; return false; - Messages |
javascript ;" target=_blank onclick="set_sort_column0, 'bytes'; return false; - Bytes |
| 1 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Sober.K__HexEsc__40mm'; return false; - Sober.K@mm |
2,319 |
49.6 % |
157.40 M |
| 2 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.P__HexEsc__40mm'; return false; - Netsky.P@mm |
1,255 |
26.8 % |
50.10 M |
| 3 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.B__HexEsc__40mm'; return false; - Netsky.B@mm |
131 |
2.8 % |
3.71 M |
| 4 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Bagle.AH__HexEsc__40mm'; return false; - Bagle.AH@mm |
90 |
1.9 % |
2.86 M |
| 5 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.C__HexEsc__40mm'; return false; - Netsky.C@mm |
74 |
1.6 % |
2.44 M |
| 6 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.D__HexEsc__40mm'; return false; - Netsky.D@mm |
70 |
1.5 % |
1.57 M |
| 7 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.Q__HexEsc__40mm'; return false; - Netsky.Q@mm |
67 |
1.4 % |
2.56 M |
| 8 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Mabutu.A__HexEsc__40mm'; return false; - Mabutu.A@mm |
58 |
1.2 % |
3.34 M |
| 9 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.K__HexEsc__40mm'; return false; - Netsky.K@mm |
55 |
1.2 % |
1.56 M |
| 10 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.Z__HexEsc__40mm'; return false; - Netsky.Z@mm |
50 |
1.1 % |
1.52 M |
| 11 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Lovgate.AB__HexEsc__40mm'; return false; - Lovgate.AB@mm |
47 |
1.0 % |
7.87 M |
| 12 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Bagle.J__HexEsc__40mm'; return false; - Bagle.J@mm |
45 |
1.0 % |
765.00 k |
| 13 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Bagle.N__HexEsc__40mm'; return false; - Bagle.N@mm |
42 |
0.9 % |
1.18 M |
| 14 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Bagle.BC__HexEsc__40mm'; return false; - Bagle.BC@mm |
38 |
0.8 % |
1.02 M |
| 15 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Bifrose.D'; return false; - Bifrose.D |
38 |
0.8 % |
2.36 M |
| 16 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'MyDoom.I__HexEsc__40mm'; return false; - MyDoom.I@mm |
35 |
0.8 % |
2.81 M |
| 17 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Bagle.AR__HexEsc__40mm'; return false; - Bagle.AR@mm |
32 |
0.7 % |
983.00 k |
| 18 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Bagle.BB__HexEsc__40mm'; return false; - Bagle.BB@mm |
32 |
0.7 % |
844.00 k |
| 19 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'MyDoom.J__HexEsc__40mm'; return false; - MyDoom.J@mm |
32 |
0.7 % |
2.12 M |
| 20 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'MyDoom.L__HexEsc__40mm'; return false; - MyDoom.L@mm |
30 |
0.6 % |
1.16 M |
| 21 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'W32/Downloader'; return false; - W32/Downloader |
21 |
0.5 % |
441.00 k |
| 22 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'W32/FunLove.4099'; return false; - W32/FunLove.4099 |
14 |
0.3 % |
693.00 k |
| 23 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'W32/Bagle.Gen__HexEsc__21Zip'; return false; - W32/Bagle.Gen!Zip |
13 |
0.3 % |
266.00 k |
| 24 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.AB__HexEsc__40mm'; return false; - Netsky.AB@mm |
12 |
0.3 % |
290.00 k |
| 25 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.AD__HexEsc__40mm'; return false; - Netsky.AD@mm |
11 |
0.2 % |
462.00 k |
| 26 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Bagle.AF__HexEsc__40mm'; return false; - Bagle.AF@mm |
10 |
0.2 % |
296.00 k |
| 27 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.W__HexEsc__40mm'; return false; - Netsky.W@mm |
8 |
0.2 % |
318.00 k |
| 28 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'W32/Bagle.Gen__HexEsc__21Rar'; return false; - W32/Bagle.Gen!Rar |
6 |
0.1 % |
188.00 k |
| 29 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.T__HexEsc__40mm'; return false; - Netsky.T@mm |
6 |
0.1 % |
150.00 k |
| 30 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Zafi.D__HexEsc__40mm'; return false; - Zafi.D@mm |
6 |
0.1 % |
108.00 k |
| 31 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'W95/Pinfi.A'; return false; - W95/Pinfi.A |
4 |
0.1 % |
1.10 M |
| 32 |
javascript ;" target=_blank onclick="zoom0, 'virus', 'Netsky.X__HexEsc__40mm'; return false; - Netsky.X@mm |
4 |
0.1 % |
140.00 k |
| |
17 other items |
24 |
0.5 % |
1.14 M |
| |
Total |
4,679 |
100 % |
253.63 M |
| |
Reason |
Messages |
Bytes |
| 1 |
Reverse DNS not found |
68,879 |
28.0 % |
334.19 M |
| 2 |
Bypassed all rules |
58,789 |
23.9 % |
1.17 G |
| 3 |
Blacklisted by sbl-xbl.spamhaus.org. |
49,640 |
20.2 % |
174.23 M |
| 4 |
Blacklisted by dnsbl.sorbs.net. |
10,649 |
4.3 % |
65.63 M |
| 5 |
content filter |
10,192 |
4.1 % |
482.93 M |
| 6 |
Invalid MX record |
8,473 |
3.4 % |
55.19 M |
| 7 |
EmailTO is in local blacklist file |
7,095 |
2.9 % |
236.00 k |
| 8 |
SPF test |
6,581 |
2.7 % |
55.93 M |
| 9 |
no relay allowed |
6,452 |
2.6 % |
2.00 k |
| 10 |
Blacklisted by bl.spamcop.net. |
5,105 |
2.1 % |
102.29 M |
| 11 |
infected with the virus |
4,679 |
1.9 % |
253.63 M |
| 12 |
EmailFrom is in local blacklist file |
3,159 |
1.3 % |
10.48 M |
| 13 |
Blacklisted by dnsbl.njabl.org. |
2,513 |
1.0 % |
9.65 M |
| 14 |
Blacklisted |
1,357 |
0.6 % |
6.67 M |
| 15 |
Exceeded maximum number of RCPT TO |
1,323 |
0.5 % |
6.22 M |
| 16 |
IP address is from a blacklisted country |
693 |
0.3 % |
2.62 M |
| 17 |
Found prohibited attachment |
329 |
0.1 % |
924.00 k |
| 18 |
Domain is in local blacklist file |
51 |
0.0 % |
73.00 k |
| |
Total |
245,959 |
100 % |
2.70 G |
Regards,
-------------
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com
|
|