Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - consider changing log file format
  FAQ FAQ  Forum Search   Register Register  Login Login

consider changing log file format

 Post Reply Post Reply
Author
MarvinFS View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote MarvinFS Quote  Post ReplyReply Direct Link To This Post Topic: consider changing log file format
    Posted: 07 April 2003 at 12:55pm

Roberto,

Please consider changing log files format to cvs or tabbed to be MORE suitable for analyzing with external tools.

i advise something like this: (copy&paste it to wide console\screen)

#Software: SpamFilter ISP
#Version: 1.1.0.82b REGISTERED
#Date: 2003-03-05T00:46:19
#Fields: x-event-source x-event-datetime x-event-class x-event-severity c-ip x-event-msg
SMTPSVC-1 2003-03-05T00:46:19 Block Info 195.64.195.129 Mail from nobody@linux11236.dn.net is blocked. Domain "linux11236.dn.net" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T01:13:37 Block Info 195.64.195.129 Mail from Bounces_WebSiteMgt@CWMAILIN.COMPUTERWORLD.COM is blocked. Domain "CWMAILIN.COMPUTERWORLD.COM" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T01:19:19 Block Info 195.64.195.129 Mail from bounce-ciscosys_2_4672-50143365@lyrisb.bellevue.com is blocked. Domain "lyrisb.bellevue.com" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T02:16:17 Block Info 195.64.195.129 Blocked. Sender address (janneke@bestcom.nl) is listed on the sender blacklist. SMTP response: 555 5.1.7 Sender rejected. This server does not accept mails from this SMTP address (due">janneke@bestcom.nl)<CRLF>duejanneke@bestcom.nl)<CRLF>duejanneke@bestcom.nl)<CRLF>due to security reasons! Please contact admin at mse-admin@microtest.ru
SMTPSVC-1 2003-03-05T02:37:05 Block Info 195.64.195.129 Mail from cio@UPDATE.CIO.COM is blocked. Domain "UPDATE.CIO.COM" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T03:00:43 Block Info 195.64.195.129 Mail from root@http://www.bizbook.ru is blocked. Domain "http://www.bizbook.ru" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T03:44:10 Block Info 195.64.195.129 Mail from nksoft@mail.nnz.ru is blocked. Domain "mail.nnz.ru" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T03:52:56 Block Info 195.64.195.129 Mail from admin@ericron.com is blocked. Domain "ericron.com" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T04:28:45 Block Info 195.64.195.129 Mail from bounce-byteback-html-3019994@list.cramsession.com is blocked. Domain "list.cramsession.com" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T04:35:34 Block Info 195.64.195.129 Mail from huuwx@cntre.ru is blocked. Domain "cntre.ru" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T04:47:39 Block Info 195.64.195.129 Mail from huuwx@cntre.ru is blocked. Domain "cntre.ru" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T06:07:56 Block Info 195.64.195.129 Mail from nobody@linux10966.dn.net is blocked. Domain "linux10966.dn.net" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T07:27:26 Block Info 195.64.195.129 Mail from huuwx@cntre.ru is blocked. Domain "cntre.ru" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 April 2003 at 7:37pm

We are going to have to find a balance between readability/format and troubleshooting usefulness.

Having a clean log like the one you mention is good for cleanliness and readability. However it does not provide any indication of the steps SpamFilter performed to reach the reject decision. We are currently indicating step-by-step what happens in the various stages of a connection. We now have to figure out how to provide the same step-by-step detail, but in a more readable format.

..any suggestions? :-)

Roberto Franceschetti
LogSat Software

Back to Top
MarvinFS View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote MarvinFS Quote  Post ReplyReply Direct Link To This Post Posted: 07 April 2003 at 11:00pm

I suggest to make some sort of events based logging system. I saw such a system is other software.

There may be several events types and may be several severety types:

1. whenever mail is blocked it's block event
mail is bypassed all filters - bypass or pass event,  etc. all with info urgency

2. dns error, uploading to mail server error, maps errors, logging errors it's warnings or errors severety events.

3. all internal errors are to be critical severety event group. where spamfilter should decide what to do may be it will be safer to shutdown.

make several check boxes in logging tab WHAT TO DISPLAY and LOG for each group of events (i.e i want to log only criticals and blocks, and to display only them too, i dont what to display passed messages)

and refering to my prev. post with log file, we had there ip addresses, e-mal addresses, severety of event, type of event (block) and THE LAST column description WHAT CAUSED the messages to be blocked. in our case it may be maps lookup, or the message was blocked because of keyword filter's entry or, in the futute, actual senders domain mx check.

I think something like that, i may give some more details.

--------
CU Round,
MarvinFS

Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 09 August 2006 at 3:58pm

Is there any chance that you could implement (maybe as an option that you could turn on or off)  a method of logging to the database.

I'm thinking about a master table and a details table.

The master table could hold emailID, date/time, from, to, subject, sender IP etc... while the details table could hold all of the transaction details for the emailID in question.

This would make reporting a piece of cake, and we could then run an sql script to either delete or export entried that were x days/months old.

SF would be the correct place to log this info, rather than trying to trawl through text based log files periodically.

 

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 09 August 2006 at 6:11pm

Db based logging sounds like a good option!! Would make life easier for running reports and getting stats for those who care about them, but keep text based for those who don't need that much info.

http://www.webguyz.net
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 09 August 2006 at 6:27pm
Oh my god!  Do you realize how many hours I have spent with SawMill getting the logs to parse REAL NICELY and now we want to change.  Will someone help me re-write the parsing plugin?
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 09 August 2006 at 6:41pm


    

http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 09 August 2006 at 10:09pm
Sorry... no plans to change logging format anytime soon. There are 3rd party utilities that rely on the current format, and we do not wish to create problems with them.

Logging to a database would prevent logging if there were any database issues, and for admins who have 100s of MB logs every day, the database would grow too large too soon.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 4:15am

As I suggested, this could be an option that you can turn on or off, also logging to text files could also be a turn on/off option so as to not upset people who have written apps based on the text logging.

If there is a problem with the database, then you could write these critical errors to text files.

With reference to the size of the logs, these people could, as I suggested earlier periodically either delete or export entries that are x days/months old. And, if you have customers with 100s MB of text logs every day, can you imagine the hassle they must have parsing these?? How do they efficiently trace a connection in real time to diagnose problems???? VERY hard I would suggest.

As mentioned in other forum posts, logging is an ESSENTIAL part of a professional ISP, not one that apperas to be an afterthought,Just search your forum and see how many people are having problems with parsing the current text logs....why???? just stick the logs into a database (it's not hard, we already have one !) ....I bet 99% of your customers would be happier.


Implementing this would cater for all current objections and you would have some great logging which we could use for both diagnostics and getting some easy statistics. For instance, we currently log everything to our main reporting DB so that each customer can get stats and graphs of exactly how many emails they have received within a certain time frame, and more importantly (for our billing) how much crap we have stopped going to their domain / mailbox. Our customers DEMAND this type of logging/reporting, othrewise how will they know what they are paying for ?????

 

 

Back to Top
Web123 View Drop Down
Newbie
Newbie
Avatar

Joined: 26 January 2005
Location: Finland
Status: Offline
Points: 31
Post Options Post Options   Thanks (0) Thanks(0)   Quote Web123 Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 4:26am

We really need to get all the stats directly from SF

 

Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 8:47am
StevenJohns,
 
Just a comment or 2 on your post.  As an ISP we are bound by "rules" about accountability.  My comments may be slanted by that and my general experience of using log files for everything we do.
 
Your Post: "With reference to the size of the logs, these people could, as I suggested earlier periodically either delete or export entries that are x days/months old. And, if you have customers with 100s MB of text logs every day, can you imagine the hassle they must have parsing these?? How do they efficiently trace a connection in real time to diagnose problems???? VERY hard I would suggest."
 
I just had a situation with the FBI where they needed information for 14 months ago.  Deleting logs is NEVER an option.  Also, I really do not have any issues (hassles) parsing my logs and tracing anything and I have 3 separate machines with their own logging that each message passes through.  I guess the "real time" ting is an issue but with 500,000 messages a day, real time is a relative term anyway.
 
Your Post: "we currently log everything to our main reporting DB so that each customer can get stats and graphs of exactly how many emails they have received within a certain time frame, and more importantly (for our billing) how much crap we have stopped going to their domain / mailbox. Our customers DEMAND this type of logging/reporting, otherwise how will they know what they are paying for"
 
We do all this with both SawMill, which looks at the actual log files and with custom SQL queries against the quarantine DB.
 
We also have scripts that run at midnight to archive all logs over 2 days old to a NAS server if we need to get back to them.  Our logging is over 250MB a day (uncompressed) and I shudder to think what kind of machine I would need to do this logging in a DB that would not impact performance.
 
My 3 cents.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 9:56am

Desperado,

  Since there is no 'One Size Fits All' then having a choice would be great. You can keep text logging or go with DB, the key is having the choice.

You must be a public company to have to keep records for that long. Mine are deleted after 3 months. Can't give anyone what I don't have.

http://www.webguyz.net
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 10:13am

Dan,

I can see where you are comming from. As Webguyz says, there is no one size fits all, give us the option of how WE want to do our logging, rather than how we are TOLD that we have to do it. That's all.

By the way, as  LogSat was clearly worried about the size of the database....exactly how big is your DB?? If you have 250MB of logs each day, then your quarantine DB must be huge. If you keep your logs for over 14 months, is it reasonable to assume that you keep the quarantined email for as log too??

What would be the point of having a log saying "you emailed fred at 5:30 on 5/5/06, but I have no idea what the email content was"?

I don't mean to sound picky, just wondered what an ISP of your size does.

 

Cheers

 

Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 11:02am

StevenJohns,

Our mail logs go back to 1999 but we do not care what the content of a message was/is except for our own internal mail and we use an exchange clone for that.  Our SpamFilter Quarantine expires between 2 and 14 days depending on the company we are supporting.  Our DB is about 13BG for that.

WebGuys,

We are not "public" but we are an ISP (privately owned).

The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2006 at 11:12am

Dan,

 

This 13GB DB, is it MySQL??

How well is the quarantine DB performing??

Cheers

Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.063 seconds.