Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Emails Quarantined for Nonexistent users
  FAQ FAQ  Forum Search   Register Register  Login Login

Emails Quarantined for Nonexistent users

 Post Reply Post Reply
Author
jerbo128 View Drop Down
Senior Member
Senior Member
Avatar

Joined: 06 March 2006
Status: Offline
Points: 178
Post Options Post Options   Thanks (0) Thanks(0)   Quote jerbo128 Quote  Post ReplyReply Direct Link To This Post Topic: Emails Quarantined for Nonexistent users
    Posted: 27 March 2008 at 12:40pm
I am getting a fair amount of mail in the quarantine for email addresses that do not appear in my tblwl_authorizedtoemails. 
 
I have a script that expires any message in tbl_quarantine where the email field does not have a match in tblwl_authorizedtoemails.
 
I cleaned such messages this am, and now 2 hours later, I have 2600 more messages that match this rule.  The first time that I ran this rule, it removed 200K messages from my database.
 
After looking at the logs, It appears that the messages are being quarantined because they failed the local blacklist check.  So, the authorized To Email list is never checked.
 
Roberto - is this the intended behavior?  What can we do to stop these from clogging my database up? 
 
Running 773 SFE.
 
Any idea's?
 
Jeremy


Edited by jerbo128 - 27 March 2008 at 2:39pm
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 27 March 2008 at 4:23pm
Jeremy,

The order in which the filters is applied is listed at http://www.logsat.com/SpamFilter/Forums/forum_posts.asp?TID=5171#5826. If a filter blocks an email, all subsequent filters will be ignored to avoid wasting resources. This will indeed cause a number of emails where the recipient is not in the "Autorized TO" list to be quarantined instead of being rejected right away.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
jerbo128 View Drop Down
Senior Member
Senior Member
Avatar

Joined: 06 March 2006
Status: Offline
Points: 178
Post Options Post Options   Thanks (0) Thanks(0)   Quote jerbo128 Quote  Post ReplyReply Direct Link To This Post Posted: 27 March 2008 at 4:44pm
Roberto,
 
I understand, but in my opinion, this could be a huge isse.  With my retention of 7 days - that is 200,000 messages.  To me, that is a waste of resources also.
 
Maybe we need to weigh it out -to see which has a higher "cost"
 
Anyone else have thoughts on this?  How about some of you bigger guys who get a lot of mail, have you ever looked at the number of orphaned messages?
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 27 March 2008 at 5:47pm
jeremy,

The only three filters that are before the "Authorized TO" list are:

  1. Local Domain Blacklist
  2. Local Emails Blacklist
  3. Local Emails TO Blacklist


emails blocked by these are usually very, very small in numbers, if any at all. Are you actually receiving 200,000 emails each week blocked by these 3? If you have a huge blacklist of blacklisted sender's domains and emails, that may explain this large number, but if not, you may be targeted by very specific spammers/attackers.

In "normal" installations, the 3 lists above are often much smaller than the "Authorized TO" list, which is why SpamFilter examines the smaller ones first to try saving some resources. An option may be to change the order of the filters by moving up the "Authorized TO". We'll be monitoring the thread for feedback on this.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
jerbo128 View Drop Down
Senior Member
Senior Member
Avatar

Joined: 06 March 2006
Status: Offline
Points: 178
Post Options Post Options   Thanks (0) Thanks(0)   Quote jerbo128 Quote  Post ReplyReply Direct Link To This Post Posted: 27 March 2008 at 7:16pm
Originally posted by LogSat LogSat wrote:

jeremy,

The only three filters that are before the "Authorized TO" list are:

  1. Local Domain Blacklist
  2. Local Emails Blacklist
  3. Local Emails TO Blacklist
Local domain blacklist =23,000 rows
Emails to/from = less than 100 each
Local IP blacklist = 250,000 rows (does not count for subnets)
 
Most of these emails that I am seeing to unAutorized addresses are quarantined with ID 12 - Local IP Blacklist.  I see a few with reason 7 and 9.
 
Yes, you are correct - I am getting over 200,000 per week to these unauthorized addresses.  Looking at the IP's and subjects of the connections - they are scattered everywhere.
 
You said above that  local IP blacklist is AFTER the Authorized User Check, then why am I seeing all of these?   For clarification -This post lists a different filter order.   http://www.logsat.com/SpamFilter/Forums/forum_posts.asp?TID=5171 
Which is right?
  1. Cached IP blacklist
  2. Greylist
  3.         Whitelisted IP
  4.         Whitelisted Email Address To
  5.         Whitelisted EMail Address From
  6.         Whitelisted Email From Domain
  7.         Whitelisted Auto White List Force Delivery
  8. Allowed Domains
  9. Local IP Blacklist
  10. Local Domain Blacklist
  11. Local Emails Blacklist
  12. Local Emails TO Blacklist
  13. Not in Authorized TO Emails

Jeremy



Edited by jerbo128 - 27 March 2008 at 7:20pm
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 27 March 2008 at 10:49pm
Geez, I can't even read my own warning. And think that I had even super bolded the notice "Please always check the latest post in this thread" on that thread.

You're right, I did not see the updated order. What you posted is correct, the 4 filters (not 3) that cause emails to be quarantined before the "Authorized TO" list is hit are:
  1. Local IP Blacklist
  2. Local Domain Blacklist
  3. Local Emails Blacklist
  4. Local Emails TO Blacklist

I did not mention the others that appear first (allowed domains, and the "forgotten" blacklist cache and greylist) because these ones reject emails immediately, without allowing them to be quarantined, so do not apply in ths discussion.
c
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
jerbo128 View Drop Down
Senior Member
Senior Member
Avatar

Joined: 06 March 2006
Status: Offline
Points: 178
Post Options Post Options   Thanks (0) Thanks(0)   Quote jerbo128 Quote  Post ReplyReply Direct Link To This Post Posted: 27 March 2008 at 11:35pm

Do you have any suggestions for dealing with such a huge amount of crap mail?

My SQL server stays much happier with 200K less emails in it's tummy.
 
 
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 28 March 2008 at 5:45pm
That depends on the reliability of your IP blacklist, which is causing the bulk of these blocks. If you are note receiving false positives from it, you could enable the option to "do not quarantine" emails from this filter.

Also, if you have the list of allowed users in a database, it's rather simple to create a query that will weed out the unwanted recipients. If the list is only in a text file, there's a few extra steps involved to bulk import it in a temp table in MySQL (don't know yet if MySQL can perform a query directly against a text file, as MS SQL can do). I'll take a stab at it tonite to see if I can help you here, but please email me to stop me if the data is already in the DB!
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
jerbo128 View Drop Down
Senior Member
Senior Member
Avatar

Joined: 06 March 2006
Status: Offline
Points: 178
Post Options Post Options   Thanks (0) Thanks(0)   Quote jerbo128 Quote  Post ReplyReply Direct Link To This Post Posted: 28 March 2008 at 5:51pm
Don't worry about the script.  The users are in the DB.  And I have a query to weed out the bad emails. 
 
My point is that when they come in at a rate of 30,000 per day, I just see it as a lot of unneeded SQL traffic. Both inbound and then deleting.
 
As for the reliability of my blacklist, it's pretty good.  But when you have that many entries, it only takes 1 bed entry to get a chewing from the customer.  So I don't know if I am ready to flip the flag for "do not quarantine"
 
Jeremy
 
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4065
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 28 March 2008 at 8:57pm
Oh well.. since I've already started, and as it can be used by others who do have a file-based list, here it is anyways :-)


CREATE TABLE test(
email varchar(320)
);

LOAD DATA INFILE "/tmp/WL_AuthorizedTOEmails.txt"
INTO TABLE test
LINES TERMINATED BY '\r\n';

UPDATE tblquarantine  LEFT JOIN test ON tblquarantine.emailto = test.email
SET expire=1 WHERE ISNULL(test.email)

DROP TABLE test;



The above will mark the emails for deletion so that SpamFilter can then delete them during regular cleanup. If you want to delete them right away, this one will do the job:

CREATE TABLE test(
email varchar(320)
);

LOAD DATA INFILE "/tmp/WL_AuthorizedTOEmails.txt"
INTO TABLE test
LINES TERMINATED BY '\r\n';

DELETE tblquarantine FROM tblquarantine LEFT JOIN test ON tblquarantine.emailto = test.email
WHERE ISNULL(test.email);

DELETE tblmsgs FROM tblmsgs LEFT JOIN tblquarantine ON tblmsgs.msgid = tblquarantine.msgid
WHERE ISNULL(tblquarantine.msgid);

DROP TABLE test;


Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.109 seconds.