Print Page | Close Window

consider changing log file format

Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: http://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=181
Printed Date: 23 October 2017 at 9:36am


Topic: consider changing log file format
Posted By: Guests
Subject: consider changing log file format
Date Posted: 07 April 2003 at 12:55pm

Roberto,

Please consider changing log files format to cvs or tabbed to be MORE suitable for analyzing with external tools.

i advise something like this: (copy&paste it to wide console\screen)

#Software: SpamFilter ISP
#Version: 1.1.0.82b REGISTERED
#Date: 2003-03-05T00:46:19
#Fields: x-event-source x-event-datetime x-event-class x-event-severity c-ip x-event-msg
SMTPSVC-1 2003-03-05T00:46:19 Block Info 195.64.195.129 Mail from mailto:nobody@linux11236.dn.net" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - nobody@linux11236.dn.net is blocked. Domain "linux11236.dn.net" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T01:13:37 Block Info 195.64.195.129 Mail from mailto:Bounces_WebSiteMgt@CWMAILIN.COMPUTERWORLD.COM" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - Bounces_WebSiteMgt@CWMAILIN.COMPUTERWORLD.COM is blocked. Domain "CWMAILIN.COMPUTERWORLD.COM" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T01:19:19 Block Info 195.64.195.129 Mail from mailto:bounce-ciscosys_2_4672-50143365@lyrisb.bellevue.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - bounce-ciscosys_2_4672-50143365@lyrisb.bellevue.com is blocked. Domain "lyrisb.bellevue.com" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T02:16:17 Block Info 195.64.195.129 Blocked. Sender address ( mailto:janneke@bestcom.nl" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - janneke@bestcom.nl ) is listed on the sender blacklist. SMTP response: 555 5.1.7 Sender rejected. This server does not accept mails from this SMTP address ( undefined" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - due">janneke@bestcom.nl)<CRLF>duejanneke@bestcom.nl)<CRLF>duejanneke@bestcom.nl)<CRLF>due to security reasons! Please contact admin at mailto:mse-admin@microtest.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - mse-admin@microtest.ru
SMTPSVC-1 2003-03-05T02:37:05 Block Info 195.64.195.129 Mail from mailto:cio@UPDATE.CIO.COM" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - cio@UPDATE.CIO.COM is blocked. Domain "UPDATE.CIO.COM" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T03:00:43 Block Info 195.64.195.129 Mail from mailto:root@www.bizbook.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - root@ http://www.bizbook.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://www.bizbook.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://www.bizbook.ru is blocked. Domain " http://www.bizbook.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://www.bizbook.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://www.bizbook.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://www.bizbook.ru " failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T03:44:10 Block Info 195.64.195.129 Mail from mailto:nksoft@mail.nnz.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - nksoft@mail.nnz.ru is blocked. Domain "mail.nnz.ru" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T03:52:56 Block Info 195.64.195.129 Mail from mailto:admin@ericron.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - admin@ericron.com is blocked. Domain "ericron.com" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T04:28:45 Block Info 195.64.195.129 Mail from mailto:bounce-byteback-html-3019994@list.cramsession.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - bounce-byteback-html-3019994@list.cramsession.com is blocked. Domain "list.cramsession.com" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T04:35:34 Block Info 195.64.195.129 Mail from mailto:huuwx@cntre.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - huuwx@cntre.ru is blocked. Domain "cntre.ru" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T04:47:39 Block Info 195.64.195.129 Mail from mailto:huuwx@cntre.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - huuwx@cntre.ru is blocked. Domain "cntre.ru" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T06:07:56 Block Info 195.64.195.129 Mail from mailto:nobody@linux10966.dn.net" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - nobody@linux10966.dn.net is blocked. Domain "linux10966.dn.net" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.
SMTPSVC-1 2003-03-05T07:27:26 Block Info 195.64.195.129 Mail from mailto:huuwx@cntre.ru" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - huuwx@cntre.ru is blocked. Domain "cntre.ru" failed on RDNS test (no DNS MX or A/CNAME record(s)). SMTP response: 501 5.1.8 Sender domain must have a DNS MX or A/CNAME record.




Replies:
Posted By: LogSat
Date Posted: 07 April 2003 at 7:37pm

We are going to have to find a balance between readability/format and troubleshooting usefulness.

Having a clean log like the one you mention is good for cleanliness and readability. However it does not provide any indication of the steps SpamFilter performed to reach the reject decision. We are currently indicating step-by-step what happens in the various stages of a connection. We now have to figure out how to provide the same step-by-step detail, but in a more readable format.

..any suggestions? :-)

Roberto Franceschetti
LogSat Software



Posted By: Guests
Date Posted: 07 April 2003 at 11:00pm

I suggest to make some sort of events based logging system. I saw such a system is other software.

There may be several events types and may be several severety types:

1. whenever mail is blocked it's block event
mail is bypassed all filters - bypass or pass event,  etc. all with info urgency

2. dns error, uploading to mail server error, maps errors, logging errors it's warnings or errors severety events.

3. all internal errors are to be critical severety event group. where spamfilter should decide what to do may be it will be safer to shutdown.

make several check boxes in logging tab WHAT TO DISPLAY and LOG for each group of events (i.e i want to log only criticals and blocks, and to display only them too, i dont what to display passed messages)

and refering to my prev. post with log file, we had there ip addresses, e-mal addresses, severety of event, type of event (block) and THE LAST column description WHAT CAUSED the messages to be blocked. in our case it may be maps lookup, or the message was blocked because of keyword filter's entry or, in the futute, actual senders domain mx check.

I think something like that, i may give some more details.

--------
CU Round,
MarvinFS



Posted By: StevenJohns
Date Posted: 09 August 2006 at 3:58pm

Is there any chance that you could implement (maybe as an option that you could turn on or off)  a method of logging to the database.

I'm thinking about a master table and a details table.

The master table could hold emailID, date/time, from, to, subject, sender IP etc... while the details table could hold all of the transaction details for the emailID in question.

This would make reporting a piece of cake, and we could then run an sql script to either delete or export entried that were x days/months old.

SF would be the correct place to log this info, rather than trying to trawl through text based log files periodically.

 



Posted By: WebGuyz
Date Posted: 09 August 2006 at 6:11pm

Db based logging sounds like a good option!! Would make life easier for running reports and getting stats for those who care about them, but keep text based for those who don't need that much info.



-------------
http://www.webguyz.net


Posted By: Desperado
Date Posted: 09 August 2006 at 6:27pm
Oh my god!  Do you realize how many hours I have spent with SawMill getting the logs to parse REAL NICELY and now we want to change.  Will someone help me re-write the parsing plugin?

-------------
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com



Posted By: WebGuyz
Date Posted: 09 August 2006 at 6:41pm


    



-------------
http://www.webguyz.net


Posted By: LogSat
Date Posted: 09 August 2006 at 10:09pm
Sorry... no plans to change logging format anytime soon. There are 3rd party utilities that rely on the current format, and we do not wish to create problems with them.

Logging to a database would prevent logging if there were any database issues, and for admins who have 100s of MB logs every day, the database would grow too large too soon.

-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP


Posted By: StevenJohns
Date Posted: 10 August 2006 at 4:15am

As I suggested, this could be an option that you can turn on or off, also logging to text files could also be a turn on/off option so as to not upset people who have written apps based on the text logging.

If there is a problem with the database, then you could write these critical errors to text files.

With reference to the size of the logs, these people could, as I suggested earlier periodically either delete or export entries that are x days/months old. And, if you have customers with 100s MB of text logs every day, can you imagine the hassle they must have parsing these?? How do they efficiently trace a connection in real time to diagnose problems???? VERY hard I would suggest.

As mentioned in other forum posts, logging is an ESSENTIAL part of a professional ISP, not one that apperas to be an afterthought,Just search your forum and see how many people are having problems with parsing the current text logs....why???? just stick the logs into a database (it's not hard, we already have one !) ....I bet 99% of your customers would be happier.


Implementing this would cater for all current objections and you would have some great logging which we could use for both diagnostics and getting some easy statistics. For instance, we currently log everything to our main reporting DB so that each customer can get stats and graphs of exactly how many emails they have received within a certain time frame, and more importantly (for our billing) how much crap we have stopped going to their domain / mailbox. Our customers DEMAND this type of logging/reporting, othrewise how will they know what they are paying for ?????

 

 



Posted By: Web123
Date Posted: 10 August 2006 at 4:26am

We really need to get all the stats directly from SF

 



Posted By: Desperado
Date Posted: 10 August 2006 at 8:47am
StevenJohns,
 
Just a comment or 2 on your post.  As an ISP we are bound by "rules" about accountability.  My comments may be slanted by that and my general experience of using log files for everything we do.
 
Your Post: "With reference to the size of the logs, these people could, as I suggested earlier periodically either delete or export entries that are x days/months old. And, if you have customers with 100s MB of text logs every day, can you imagine the hassle they must have parsing these?? How do they efficiently trace a connection in real time to diagnose problems???? VERY hard I would suggest."
 
I just had a situation with the FBI where they needed information for 14 months ago.  Deleting logs is NEVER an option.  Also, I really do not have any issues (hassles) parsing my logs and tracing anything and I have 3 separate machines with their own logging that each message passes through.  I guess the "real time" ting is an issue but with 500,000 messages a day, real time is a relative term anyway.
 
Your Post: "we currently log everything to our main reporting DB so that each customer can get stats and graphs of exactly how many emails they have received within a certain time frame, and more importantly (for our billing) how much crap we have stopped going to their domain / mailbox. Our customers DEMAND this type of logging/reporting, otherwise how will they know what they are paying for"
 
We do all this with both SawMill, which looks at the actual log files and with custom SQL queries against the quarantine DB.
 
We also have scripts that run at midnight to archive all logs over 2 days old to a NAS server if we need to get back to them.  Our logging is over 250MB a day (uncompressed) and I shudder to think what kind of machine I would need to do this logging in a DB that would not impact performance.
 
My 3 cents.


-------------
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com



Posted By: WebGuyz
Date Posted: 10 August 2006 at 9:56am

Desperado,

  Since there is no 'One Size Fits All' then having a choice would be great. You can keep text logging or go with DB, the key is having the choice.

You must be a public company to have to keep records for that long. Mine are deleted after 3 months. Can't give anyone what I don't have.



-------------
http://www.webguyz.net


Posted By: StevenJohns
Date Posted: 10 August 2006 at 10:13am

Dan,

I can see where you are comming from. As Webguyz says, there is no one size fits all, give us the option of how WE want to do our logging, rather than how we are TOLD that we have to do it. That's all.

By the way, as  LogSat was clearly worried about the size of the database....exactly how big is your DB?? If you have 250MB of logs each day, then your quarantine DB must be huge. If you keep your logs for over 14 months, is it reasonable to assume that you keep the quarantined email for as log too??

What would be the point of having a log saying "you emailed fred at 5:30 on 5/5/06, but I have no idea what the email content was"?

I don't mean to sound picky, just wondered what an ISP of your size does.

 

Cheers

 



Posted By: Desperado
Date Posted: 10 August 2006 at 11:02am

StevenJohns,

Our mail logs go back to 1999 but we do not care what the content of a message was/is except for our own internal mail and we use an exchange clone for that.  Our SpamFilter Quarantine expires between 2 and 14 days depending on the company we are supporting.  Our DB is about 13BG for that.

WebGuys,

We are not "public" but we are an ISP (privately owned).



-------------
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com



Posted By: StevenJohns
Date Posted: 10 August 2006 at 11:12am

Dan,

 

This 13GB DB, is it MySQL??

How well is the quarantine DB performing??

Cheers




Print Page | Close Window