Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Bayesian Filter questions amd problems
  FAQ FAQ  Forum Search   Register Register  Login Login

Bayesian Filter questions amd problems

 Post Reply Post Reply
Author
robj View Drop Down
Newbie
Newbie


Joined: 06 February 2006
Location: United States
Status: Offline
Points: 3
Post Options Post Options   Thanks (0) Thanks(0)   Quote robj Quote  Post ReplyReply Direct Link To This Post Topic: Bayesian Filter questions amd problems
    Posted: 06 February 2006 at 2:06pm

How long does this take to "learn"
our emails?  It kicked in on Friday and I've been releasing every email it seems from the quaratine.  Once I release it it does seem to learn, mostly.  But a first time email to a user and it's always held.  It's running me ragged.  Any help?

I understand I can disable it by putting the value to 0.  I don't want to do that but I want to make sure I'm doing everything correctly.

I have also a created a white-to.txt with a list of only valid emails in our system to try and cut out a lot of unwanted junk.  Is this advisable?

Thanks for the help,

Rob



Edited by robj
Back to Top
Jason View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Jason Quote  Post ReplyReply Direct Link To This Post Posted: 06 February 2006 at 8:51pm
I would like some more info on this as well.  I recently purchased the software and am testing it on 1 domain currenlty that received quite a bit of spam.   So far the system has taken 10,000 emails and only 1,800 of them have been passed. 

I have noticed that the Bayesian Filter stilll shows that everything is at 0% and passes the email that is clearly spam.

Any help oy mabye I just have something setup incorrectly.
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 06 February 2006 at 9:07pm
Jason, Robj,

The Bayesian filter kicks in after SpamFilter has received and processed 5,000 good emails and 5,000 spam emails. Before those limits are reached, SpamFilter will only build its internal statistical database.

Please note that the statistical analysis only occurs after all other filters have failed to catch spam, so that even the the Bayesian filter becomes active, the number of emails it will block will be very small when compared to others.

As a comparison, the following shows the current number of emails blocked by the various filter on our own SpamFilter installation. You'll see that the Bayesian filter has a very low count, but that is normal as that simply means that all the other filters combined allowed 107 emails to slip thru...

64937  IP found in MAPS search
16230  IP address is from a blacklisted country
15726  SPF Sender Policy Framework match
12604  Exceeded maximum number of RCPT TO
9693   Invalid sender domain MX record
6434   URL in email found in SURBL search
3399   Keywords found in content
588    Mail From and Mail To domains are equal
486    IP blocked by honeypot entry
395    Virus Found in email
107    Statistical filter match
9      Mail From and Mail To are equal
1    Domain is in local blacklist file
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
robj View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote robj Quote  Post ReplyReply Direct Link To This Post Posted: 06 February 2006 at 9:21pm

But my problem is everything is coming in at 100%, tonight I'm still pulling out good emails.  I can get some counts tomorrow (I'm at home now), but my quaratine DB is 175meg, (starts at 3800, ends about 18000),  when I sort on the reject over half is the statistical 100% spam.  so that's several thousand.  My counts are

connections 82000,  forwarded 9300, blocked 44700, attempts 40500.

This was an upgrade from a 2+ year old V 1.0.  What am I missing.  I guess I'd rather have it working less than more because of all the calls I'm getting.

Thanks, Rob

Back to Top
RobJ View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote RobJ Quote  Post ReplyReply Direct Link To This Post Posted: 06 February 2006 at 9:23pm
oh good emails are 9300
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 06 February 2006 at 10:33pm
Robj,

It may be better to start with a fresh/clean statistical corpus database in case the one you have became corrupted.

To do so, can you please stop SpamFilter, then delete or rename the SpamFilter\corpus directory, and then restart SpamFilter.

Please note that the Bayesian filter will again need to process the initial 5,000 good and 5,000 spam emails.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
robj View Drop Down
Newbie
Newbie


Joined: 06 February 2006
Location: United States
Status: Offline
Points: 3
Post Options Post Options   Thanks (0) Thanks(0)   Quote robj Quote  Post ReplyReply Direct Link To This Post Posted: 07 February 2006 at 9:22am

Ok Stopped and reset this morning.  Usual junk is flowing again.  My Bays number on the pie chart was 6600.  Not sure how to reset this but I did reset the main counters.  Stuff showing 0% spam match getting through.

Any interest in checking out the corpus DB?  It was full, lots of the tokens had the same number, many were different.

Rob

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 February 2006 at 4:17pm
Sure, go ahead and zip us the whole corpus subdirectory at support at logsat dot com.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Lee View Drop Down
Groupie
Groupie


Joined: 04 February 2005
Location: United States
Status: Offline
Points: 50
Post Options Post Options   Thanks (0) Thanks(0)   Quote Lee Quote  Post ReplyReply Direct Link To This Post Posted: 01 March 2006 at 11:39pm

Roberto I noticed that all of the sudden my Bayesian filter is all of the sudden stopping emails from friends that I could receive on a few days ago.

It is possible that there is a problem with my Corpus database but is there any way to repair it without losing all of the tokens that it has collected over such a long period of time ?

Lee

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 02 March 2006 at 8:35pm
Lee,

You can't directly modify the corpus database, but you can cheat... If you have the original, unmodified source of the emails that you received from them, you could forward them so that SpamFilter processes them again, ensuring that when you re-send them you are whitelisting them. This way the Bayesian filter will "learn" that they are good emails and will adapt. You should also make sure you force-delivery of the good quarantined emails as this will cause SF to "undo" the entries it added to the Baysian database, and will actually additionally update to "heavily mark" those tokens as good for the future.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Robj View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Robj Quote  Post ReplyReply Direct Link To This Post Posted: 03 March 2006 at 9:16am

Well since I corrected the cache blacklist thing I've been working fine.  But my Bays filter hasn't caught a single email. 

 

Rob

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 03 March 2006 at 4:12pm
Have you upgraded to the latest versions of SpamFilter? One of the most visible improvements in the new version is a greater effectiveness of the Bayesian filter. Its spam catch rate has, sometimes, increased 100-fold.
 
This is the release note that applies:
 
 // New to VersionNumber = '2.7.1.526';
{TODO -cNew : Added DoNotStartWithoutAV option in SpamFilter.ini file to prevent SpamFilter from running unless the antivirus is working}
{TODO -cFix : Greatly improved Bayesian filter accuracy}

Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Clator View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Clator Quote  Post ReplyReply Direct Link To This Post Posted: 06 March 2006 at 11:30am
Per Dan's suggestion, Ive created a new corpus directory.  Meanwhile some of the domains that are failing MX checks are elon.edu, aapa.org, and gci.net.  Sorry I don't have the full headers at the moment.  I went ahead and forced the messages through.
Back to Top
Clator View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Clator Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2006 at 8:13am

Some more potential false positives ...

Received: from 205.188.139.137 by clator.com (LogSat Software SMTP Server - Unlicensed Evaluation Copy) Tue, 7 Mar 2006 08:01:11 -0500
Received: from [...] by imo-d23.mx.aol.com (mail_out_v38_r7.3.) id 3.214.1439293c (3657)
  for <...>; Tue, 7 Mar 2006 08:01:07 -0500 (EST)
From: [...]Message-ID: <...@aol.com>
Date: Tue, 7 Mar 2006 08:01:07 EST
Subject: Club Mom info from [...]
To: [...]MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="part1_214.1439293c.313ede13_boundary"
X-Mailer: 9.0 SE for Windows sub 5021
X-Spam-Flag: NO
X-Server: LogSat Software SMTP Server - Unlicensed Evaluation Copy
X-SF-RX-Return-Path: <...>
X-SF-HELO-Domain: imo-d23.mx.aol.com

Back to Top
Clator View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Clator Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2006 at 8:18am

and another ...  personall details removed again to keep the bots from picking it up.

Received: from 204.127.192.82 by clator.com (LogSat Software SMTP Server - Unlicensed Evaluation Copy) Mon, 6 Mar 2006 19:34:08 -0500
Received: from mack ([node].hsd1.va.comcast.net[69.143.209.237])
          by comcast.net (rwcrmhc12) with SMTP
          id <20060307003406m12001ichve>; Tue, 7 Mar 2006 00:34:07 +0000
Message-ID: <000a01c6417e$9c3c9aa0$6401a8c0@mack>
Reply-To: "..." <...@comcast.net>
From: "..." <...@comcast.net>
To: <...>
Subject: Simpsons video
Date: Mon, 6 Mar 2006 19:32:26 -0500
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="----=_NextPart_000_0007_01C64154.B2EC5990"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2741.2600
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2742.200
X-Server: LogSat Software SMTP Server - Unlicensed Evaluation Copy
X-SF-RX-Return-Path: <...@comcast.net>
X-SF-HELO-Domain: rwcrmhc12.comcast.net

Back to Top
Clator View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Clator Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2006 at 2:06pm

more.  several legit mails from Amazon.com getting caught.  Here's an example.

Received: from 207.171.160.42 by clator.com (LogSat Software SMTP Server - Unlicensed Evaluation Copy) Tue, 7 Mar 2006 13:33:57 -0500
Received: from na-rte-app-5102.iad5.amazon.com ([10.216.250.37])
  by mm-notify-out-2103.amazon.com with ESMTP; 07 Mar 2006 10:32:55 -0800
Received: by na-rte-app-5102.iad5.amazon.com
 id AAA-notification-29959,8591; 7 Mar 2006 10:32:37 -0800
Date: 7 Mar 2006 10:32:37 -0800
Message-ID: <...@na-rte-app-5102.iad5.amazon.com>
X-AMAZON-TRACK: notification
To: ...@clator.com
From: "Amazon.com Payments" <gameowner@msn.com>
Subject: Your Amazon Marketplace Purchase
Cc: payments-mail@amazon.com
Bounces-to: ...@bounces.amazon.com
Content-Type: text/plain
MIME-Version: 1.0
X-AMAZON-MAIL-RELAY-TYPE: notification
X-Server: LogSat Software SMTP Server - Unlicensed Evaluation Copy
X-SF-RX-Return-Path: <...@bounces.amazon.com>
X-SF-HELO-Domain: mm-notify-out-2103.amazon.com

Back to Top
Clator View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Clator Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2006 at 2:08pm

a legit one from turner.com ...

Received: from 64.236.240.147 by clator.com (LogSat Software SMTP Server - Unlicensed Evaluation Copy) Tue, 7 Mar 2006 13:52:01 -0500
Received: from CNNCIMSS05.turner.com (cnncimss05.turner.com [10.188.171.204])
 by smtpgw2.turner.com (8.12.10/8.12.11) with ESMTP id k27Ipw4c020541
 for <...@clator.com>; Tue, 7 Mar 2006 13:51:59 -0500 (EST)
Received: from ATLBH01.turner.com ([10.188.157.231]) by CNNCIMSS05.turner.com with InterScan Messaging Security Suite; Tue, 07 Mar 2006 13:51:58 -0500
Received: from ATLPF02.turner.com ([10.188.156.206]) by ATLBH01.turner.com with Microsoft SMTPSVC(6.0.3790.211);
  Tue, 7 Mar 2006 13:51:58 -0500
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="----_=_NextPart_001_01C64218.363A93AC"
X-MimeOLE: Produced By Microsoft Exchange V6.5
Subject: Top Stories
Date: Tue, 7 Mar 2006 13:51:58 -0500
Message-ID: <BA010952EAB04749A22AE36AB1A1037C0265C3FF@ATLPF02.turner.com>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: General Comments
Thread-Index: AcZCGDY1/nirXhlcS1SlO6wZI18H7gAAAAAK
From: "News in General" <Topstories4@turner.com>
To: <...@clator.com>
X-OriginalArrivalTime: 07 Mar 2006 18:51:58.0531 (UTC) FILETIME=[365C6130:01C64218]
X-Server: LogSat Software SMTP Server - Unlicensed Evaluation Copy
X-SF-RX-Return-Path: <Topstories4@turner.com>
X-SF-HELO-Domain: smtpgw2.turner.com

Back to Top
Clator View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Clator Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2006 at 2:09pm

and a legit one from ebay

Received: from 66.135.209.211 by clator.com (LogSat Software SMTP Server - Unlicensed Evaluation Copy) Tue, 7 Mar 2006 09:32:18 -0500
Received: from sjcbat03.sjc.ebay.com (sjcbat03.sjc.ebay.com [10.6.37.42])
 by mx33.smf.ebay.com (8.13.5/8.13.5) with ESMTP id k27EWGwX002986
 for <...@clator.com>; Tue, 7 Mar 2006 06:32:17 -0800
DomainKey-Signature: a=rsa-sha1; s=dk; d=ebay.com; c=nofws; q=dns;
 h=x-ebay-mailtracker:to:from:mime-version:content-type :subject:date:message-id;
 b=kjHoXTu0OJkocdhD7jpQb8TeK6un9jhK4UxfXbeZjRj+AAUds5rm Q29+lSiLB0UDc
 KnHdvzITNMFJjcx/TXXMQIO3w3Xhzz4xaquXIxz4hfiGPb+hku9yyY oH5LH0L+nCW7L
 wXWkFETSXy0yraUwJz996/G35Bg3ywRHDUq1bCo=
X-eBay-MailTracker: 10102.425.0.0
To: ...@clator.com
From: favoritesellers@ebay.com
Mime-Version: 1.0
Content-Type: multipart/alternative; boundary=3635219.1141741916156.JavaMail.ebba.sjcbat03
Subject: Your eBay Favorite Seller's New Items!
Date: Tue, 7 Mar 2006 06:31:56 PST
Message-ID: <2904989.1141741916198.JavaMail.ebba@sjcbat03>
X-Server: LogSat Software SMTP Server - Unlicensed Evaluation Copy
X-SF-RX-Return-Path: <favoritesellers@ebay.com>
X-SF-HELO-Domain: mx33.smf.ebay.com

Back to Top
Clator View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Clator Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2006 at 2:13pm

and lastly, three of these were caught.   All of the above postings were just in the past six hours and were only to me (not being an actual ISP, it's jsut me and the missus using the clator.com domain).

Hopefully these will point to some issues.  In case it hasn't been said, thanks for any help you can provide.

Received: from 68.230.240.34 by clator.com (LogSat Software SMTP Server - Unlicensed Evaluation Copy) Tue, 7 Mar 2006 13:56:40 -0500
Received: from willowoffice ([70.187.202.109]) by eastrmmtao05.cox.net
          (InterMail vM.6.01.05.02 201-2131-123-102-20050715) with ESMTP
          id <...eastrmmtao05.cox.net@willowoffice>
          for <...@clator.com>; Tue, 7 Mar 2006 13:54:21 -0500
From: "..." <...@willowtreemedia.com>
To: <...@clator.com>
Subject: FW: BIOS and Photos for Fertility C.A.R.E
Date: Tue, 7 Mar 2006 13:50:19 -0500
MIME-Version: 1.0
Content-Type: multipart/mixed;
 boundary="----=_NextPart_000_0033_01C641EE.12D79CE0"
X-Mailer: Microsoft Office Outlook, Build 11.0.6353
Thread-Index: AcYQqJIy9GiFULfASdW33ufOnhb1nQxboDEg
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1506
Message-Id: <....eastrmmtao05.cox.net@willowoffice>
X-Server: LogSat Software SMTP Server - Unlicensed Evaluation Copy
X-SF-RX-Return-Path: <...@willowtreemedia.com>
X-SF-HELO-Domain: eastrmmtao05.cox.net

Back to Top
Clator View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Clator Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2006 at 2:15pm

P.S. Humble apologies.  I posted these into the wrong thread.   My intent was for them to go into the MX filter thread.  I'll try to move them.

Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.172 seconds.