Print Page | Close Window

Feature Request: Autoblacklist

Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: http://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6761
Printed Date: 17 December 2017 at 8:38am


Topic: Feature Request: Autoblacklist
Posted By: kspare
Subject: Feature Request: Autoblacklist
Date Posted: 15 October 2009 at 10:18am
We have the auto white list for users, and it works great.

I was thinking a great idea would be to have an auto black list.

If a match is found, it simply deletes the message rather than commit it to the quarantine.

This helps the system auto manage itself by even further reducing the amount of spam that gets into the system, and reducing the spam a user has to sift through.

Thoughts?



Replies:
Posted By: LogSat
Date Posted: 16 October 2009 at 5:37am
kspare,

If I understand your request, this is already possible. Most blacklists, including the two that blocks specific email addresses and email domains, have an option to "Do not quarantine rejected emails from this blacklist". Is this what you're asking for?


-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP


Posted By: kspare
Date Posted: 16 October 2009 at 7:24am
No not exactly.

I want it to work just like the auto white list, but instead an auto black list.

Right now if a spam comes in, I can send it through to my mailbox, this creates an auto whitelist so that it always comes through.

I'd like to see the opposite.

If a user deletes a spam, it creates a black list from that email to their email, if that spam comes in again, it gets send to null so they don't ever have to deal with it again.

The biggeest complaint I get is that, i've deleted that spam, why does it keep coming in.


Posted By: yapadu
Date Posted: 17 October 2009 at 6:26am
How often when someone deletes a spam message does it come from the same sender?  I get spam, maybe for home loans or something but it is quite rare that the same type of spam would come from the same sender email address.

They usually send them from different sender addresses.


Posted By: kspare
Date Posted: 17 October 2009 at 10:38am
We see it alot, its a common complaint.


Posted By: kspare
Date Posted: 22 October 2009 at 12:01pm
Roberto, any chance of implementing this?


Posted By: WebGuyz
Date Posted: 23 October 2009 at 3:23pm
This would be a good idea. Never paid attention to how often spammers switch email addresses but maybe they might eventually run out.
 
But instead of making it a FROM|TO pair like autowhitelist, just make it a single FROM::NULL, that way every time one of our domains customers blacklists an email address I could query a table and apply it to all my SFE domains customers blacklists. Eventually they will start running out of fake email address. Would really like to see less mailto:cr@p - cr@p in the quarantine day after day.
 
 


-------------
http://www.webguyz.net


Posted By: kspare
Date Posted: 23 October 2009 at 3:27pm
The only problem with not doing a FROM|TO pair, is that it would affect other people as well.

doing a from|to:null would prevent people from getting duplicates in the future to their specific email address and gradually cut down the spam overall.

The other problem with a from:null is that if someone deleted a false positive from spam that goes to others, it would create confusion. This was it only affect them and only them, also it makes it easier to correct any problems if an email needs to be removed from the list.


Posted By: LogSat
Date Posted: 23 October 2009 at 4:22pm
We've always been hesitant in "feeding back" spam emails to SpamFilter, as there is no simple way to allow end-users the ability to do so. The only practical way we see so far to implement what you are asking is to allow users to enter an email to block once they login into their quarantine area (and only block the sender for themselves without affecting other users). Is this what you had in mind?

-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP


Posted By: WebGuyz
Date Posted: 23 October 2009 at 4:28pm
Originally posted by LogSat LogSat wrote:

We've always been hesitant in "feeding back" spam emails to SpamFilter, as there is no simple way to allow end-users the ability to do so. The only practical way we see so far to implement what you are asking is to allow users to enter an email to block once they login into their quarantine area (and only block the sender for themselves without affecting other users). Is this what you had in mind?
 
And have that entry NULL'ed so its never seen again. Goal is to cut down the entries in the Quarantine that customers have to wade through everyday.


-------------
http://www.webguyz.net


Posted By: kspare
Date Posted: 27 October 2009 at 4:37pm
I'm thinking of something exactly like the autowhitelist, but the opposite.

Whitelist works by having a user send a message out of spam, it gets marked in the database and spamfilter picks it up and adds that email from and to into the autowhitelist file.

I'd like to see if a user deletes the message, the message gets marked in the db, and added into the autoblacklist file, if a match is made the spam is nulled and sent away.

It should be fairly easy to do, the concept already exists for whitelisting messages.


Posted By: kspare
Date Posted: 29 October 2009 at 6:37pm
Roberto, what needs to be done to make this happen? I've got several of my larger customers complaining to me now that they are tired of deleting the same messages over and over out of spam, and are considering other options. I don't want to loose my customers!


Posted By: yapadu
Date Posted: 29 October 2009 at 10:25pm
If you have your own GUI that users access, you could do it at that level as well.  When someone deletes a message, you place the user in your own blacklist. 

Then from time to time clean out any blacklisted email address in the quarantine.

I've never had this complaint before though from users.


Posted By: LogSat
Date Posted: 30 October 2009 at 12:04am
kspare,

The way I "read" this thread is that you would like to have this new blacklist act upon the emails that have already been blocked by SpamFilter. This means that if SpamFilter has blocked successfully an email, and the users delete this email from the quarantine, then you (they) also wish to also never see that email in the quarantine again.

If this a correct interpretation, the issue becomes not as much in having SpamFilter stopping more emails, but one of SpamFilter dropping the emails without archiving them. This has some potential issues. Besides the need for an additional database table and the modification of the tblQuarantine (to differentiate which emails are deleted due to a user forcing their deletion and which ones are being deleted due to their age), this can have some pretty serious side effects. For example, some users routinely wipe out all of their quarantined emails all at once as they're tired of seeing all the junk. Chance are that there are some legitimate emails in the list that were incorrectly stopped by SpamFilter. If this auto-blacklist process was in place, the user could also then be causing a valid email blocked by mistake to also be permanently blacklisted for the future. This could be a bigger issue than having to deal with sifting thru duplicate quarantined emails.


-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP


Posted By: kspare
Date Posted: 30 October 2009 at 12:36am
I disagree, the only problem it creates is a problem for that user and it's a simple fix to remove that from|to rule from the server.

But it solves a much bigger problem in that users aren't constantly deleting emails that either come in multiple times each day, after they have already deleted them once before.

Why wouldn't spamfilter be smart enough to know that that email has been deleted from spam, so it should not be received and there for not show up again.


Posted By: yapadu
Date Posted: 30 October 2009 at 12:43am
Yep, seems like a bad idea to have it being done across the board.  A user deletes a message from their mother by accident, and it then blacklists any future email from the mother.

How many messages would it take before the mother realizes the kid is not responding to any of HER messages.

Implementing something on your local GUI and not in the heart of the SF system seems much better.  Maintain a list of blacklists, and purge any matching senders from the quarantine.


Posted By: kspare
Date Posted: 30 October 2009 at 12:48am
It's a bad idea if you over exaggerate it.

I think bigger picture, satisfying customers needs and not having the same messages coming in over and over it a good option. It might be easy for you to go in each day and add individual rules to block the spam, but when you have hundreds of customers on the server it's not something thats possible. Other solutions out there ensure that if you delete an email it doesn't come back.

The other option is that this could be turned off. If you don't want to use it or implement it, Don't.

But just because you are worried about someone and their mothers emails doesn't mean there isn't a legitimate use for this to solve customer problems.


Posted By: kspare
Date Posted: 30 October 2009 at 9:28am
Roberto, which blacklist will permit from|to filters? I can't find anything in the documentation, but it says on the website this can be done.


Posted By: WebGuyz
Date Posted: 30 October 2009 at 10:30am
I have to admit we hear this complaint a lot, expecially those  spam magnets who have hundreds of emails in their quarantine. We tried doing things like making the current day spam a different color to help those who check daily, but still we get questions like 'I deleted this spam, why does it keep coming back' . kspares suggestion would be helpful. Yes, it would require an extra table, but I think it would be worth it.

-------------
http://www.webguyz.net


Posted By: yapadu
Date Posted: 30 October 2009 at 11:59am
Are you guys sure your customers are saying 'I deleted this spam, why does it keep coming back' in relation to the senders email address?

I suspect they are saying that because they keep getting a message with a certain subject or message content.  They delete it, and think that is the end of it.  Spammers never use the same sender, they are usually random or from a pool.

But us humans certainly see the relationship between messages that have the same subject or pattern.

If a sender sends a message from star8923@discountmedicine.com, and then another one shows up a couple days later from star8934@discountmidicine.com your automatic blacklist does not work.

But guess what, the user thinks it is the same message and I suspect that is what they would be complaining about.

If it was just a matter of blocking star8923@discountmedicine.com after receiving their discount medicine advert, spam would not be such a massive problem.  It just does not seem that simple to me.


Posted By: kspare
Date Posted: 30 October 2009 at 12:07pm
Yes yapadu, we're quite capable of diagnosing this problem as well. As are my quite capable customers who run their own it departments as well. It's a problem for us as a provider and them as a customer.

I've had many complaints from a large customer that they get the *exact* same spam message back in the quarantine after it had been deleted before.,


Posted By: yapadu
Date Posted: 30 October 2009 at 2:24pm
Hope you are able to find a solution for your customers kspare.


Posted By: LogSat
Date Posted: 31 October 2009 at 5:36pm
Originally posted by kspare kspare wrote:

Roberto, which blacklist will permit from|to filters? I can't find anything in the documentation, but it says on the website this can be done.

I checked all the filters, but I did not se any one of them, except the autowhitelist, that supports the from|to notation. Can you let me know where on the website you saw this so I can double-check?

Also, can you please let me know what database platform/version (SQL Server/MySQL) you're using? I may have a workaround for this request that doesn't involve modifying SpamFilter, and would like to test it using your same setup while we give it a shot.


-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP


Posted By: kspare
Date Posted: 01 November 2009 at 2:25am
In looking for it found that I read it wrong:
Reject if "Mail From" = "Mail To" - Reject all emails where the sender's email is the same as the recipient's email.

We're using MSSQL 2005 for our db at the moment.

Thansk!


Posted By: LogSat
Date Posted: 01 November 2009 at 11:13am

I think what you asked for can be done by working directly against the database. 

This is the theory (step 1 and 2 are performed by the SQL script below - you will need to do step 2a, then step #3 and then step #4c).


1. We create a new table called "My_AutoBL", which will contain all the From/To email addresses for the emails that the end users have marked for deletion

2. We then create a new stored procedure called "PurgeQuarantineAndAutoBL" that then should be.....

a.  ...scheduled to be executed via the SQL Server Agent every 10 minutes (simply schedule to execute the SQL command "EXEC PurgeQuarantineAndAutoBL")

3. The automatic database cleanup that SpamFilter performs to remove old entries in the database should be disabled by entering "0" in SpamFilter's database tab, specifically under "Enter the interval in minutes for when the expired emails are deleted from the quarantine database".

4. The stored procedure "PurgeQuarantineAndAutoBL" will perform the following tasks:

a. When a user marks a quarantined email for deletion, it will add the From/To email address to the My_AutoBL table

b. It will remove from the tblQuarantine table all records where the From/To addresses match entries in the My_AutoBL table, which will prevent all entries users have opted to permanently delete from appearing in their spam list

c. It will perform the "standard" database cleanup by removing all archived emails older than a preset number of days (7 by default, please see comments in the stored procedure to modify this).


Please let us know if this works out for you.


This script can be also downloaded from:

http://www.logsat.com/spamfilter/pub/CreateMyAutoBL.zip - www.logsat.com/spamfilter/pub/CreateMyAutoBL.zip



-- Create My_AutoBL Auto-Blacklist Accessory table

CREATE TABLE [dbo].[My_AutoBL](

[ID] [int] IDENTITY(1,1) NOT NULL,

[EmailFrom] [nvarchar](100) NOT NULL,

[EmailTo] [nvarchar](100) NOT NULL,

 CONSTRAINT [PK_My_AutoBL] PRIMARY KEY CLUSTERED 

(

[ID] ASC

)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]

) ON [PRIMARY]

GO


CREATE NONCLUSTERED INDEX [IX_My_AutoBL_EmailFrom] ON [dbo].[My_AutoBL] 

(

[EmailFrom] ASC

)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]

GO


CREATE NONCLUSTERED INDEX [IX_My_AutoBL_EmailTo] ON [dbo].[My_AutoBL] 

(

[EmailTo] ASC

)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]

GO


-- Create the stored procedure that handles the addition of entries to the 

-- My_AutoBL automatic blacklist and to perform routine cleanup of the quarantine database

-- replacing the one being performed by SpamFilter

CREATE PROCEDURE [dbo].[PurgeQuarantineAndAutoBL] 

AS

BEGIN

-- SET NOCOUNT ON added to prevent extra result sets from

-- interfering with SELECT statements.

SET NOCOUNT ON;


-- Locate and add to the My_AutoBL table all from/to email addresses that have been marked

-- by users to be deleted

INSERT INTO My_AutoBL (EmailFrom, EmailTo)

SELECT     tblQuarantine.EmailFrom, tblQuarantine.EmailTo

FROM         My_AutoBL RIGHT OUTER JOIN tblQuarantine ON 

My_AutoBL.EmailFrom = tblQuarantine.EmailFrom AND 

My_AutoBL.EmailTo = tblQuarantine.EmailTo

WHERE     (tblQuarantine.Expire = 1) AND (My_AutoBL.ID IS NULL)


-- Now delete all entries in the tblQuarantine that match the to/from email addresses that have been

-- added in the past to the My_AutoBL blacklist.

DELETE FROM tblQuarantine

FROM   My_AutoBL INNER JOIN tblQuarantine ON 

My_AutoBL.EmailFrom = tblQuarantine.EmailFrom AND

My_AutoBL.EmailTo = tblQuarantine.EmailTo



-- Perform routine cleanup for the tblQuarantine by removing all emails older than the specified number

-- of days below (7 by default in the:

-- DATEDIFF(day, MsgDate, GETDATE()) > 7 

-- we are deleting 500 entries at a time to avoid locking rown in the database for too long

SET ROWCOUNT 500

delete_more1:

UPDATE tblQuarantine SET Expire = 1 WHERE (Expire <>1) AND (DATEDIFF(day, MsgDate, GETDATE()) > 7)

IF @@ROWCOUNT > 0 GOTO delete_more1

SET ROWCOUNT 0


SET ROWCOUNT 500

delete_more2:

DELETE FROM tblQuarantine WHERE tblQuarantine.Expire <> 0

IF @@ROWCOUNT > 0 GOTO delete_more2

SET ROWCOUNT 0


SET ROWCOUNT 500

delete_more3:

  DELETE tblMsgs FROM tblMsgs LEFT JOIN tblQuarantine 

ON tblMsgs.MsgID = tblQuarantine.MsgID WHERE (tblQuarantine.MsgID IS NULL)

IF @@ROWCOUNT > 0 GOTO delete_more3

SET ROWCOUNT 0

END




-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP


Posted By: WebGuyz
Date Posted: 09 November 2009 at 12:03pm
Roberto,
 
  Testing this now, but came across 1 item that would argue for a Blacklist that works the same way as the AutoWhitelist.
 
In the autowhitelist you can use wildcards like:
 
for these type entries:
mailto:bo-bum0edza67pz50axh8pd4d6zfgbvjw@b.email.lowes.com|andy@mymail.com - bo-bum0edza67pz50axh8pd4d6zfgbvjw@b.email.lowes.com|andy@mymail.com
you could actually add:
mailto:*@b.email.lowes.com|andy@mymail.com - *@b.email.lowes.com|andy@mymail.com
 
So if spammers are cycling thru different numbers in the FROM address, then the chances of stopping them from appearing are nil.
 
In my autowhitelist retrieval I have a script that automatically replaces strings of number and '.' with an asterisk. Would be great if we had wildcard functionality in a blacklist.
 
Never happy ..... Wink


-------------
http://www.webguyz.net


Posted By: WebGuyz
Date Posted: 13 November 2009 at 12:29pm
Just had an instance of where a autoblacklist that works like autowhitelist would have come in handy.
 
We are getting flooded with emails from bestinternetreport.info domain coming from a class C with all different kinds of IP's. They are also 'rotating' the left hand part of the FROM email so deleteing them in quarantine does no good. If we could have the capabilty to do something like mailto:*@bestinternetreport.info::NULL - *@bestinternetreport.info::NULL then we could stop these from appearing. Right now they are being stopped by one of our bl_keywords [Credit Check], but for me to use that filter I would have to go into each of the 450 SFE entries and move that entry to the top of the list (since its not sorted like the others) which is not an easy option.
 
Just one more thing to think about...


-------------
http://www.webguyz.net


Posted By: lyndonje
Date Posted: 29 January 2010 at 6:31am
Hi Roberto,

Just to confirm I understand what your above scripts do....

If I were to manually add a FROM|TO combination into the MyAutoBL table, and then receive an email matching these two address which was not otherwise being caught as spam, and therefore did not end up in the quarantine - would the email still be delivered to my Inbox?

As you have made no changes to the spamfilter binary's (from what I can see) to check the MyAutoBL table upon receiving an email, I presume my assumption above would be correct? 


Posted By: LogSat
Date Posted: 29 January 2010 at 10:39pm
That would be a "no" actually. SpamFilter does not use the "MyAutoBL" table at anytime. This table is only used as a placeholder to keep track of what emails users have forcefully deleted from their quarantine by adding the To/From address in a record in this table. The scheduled job on the SQL Server then uses these records in the MyAutoBL table to delete the quarantined emails that have those same To/From entries.

The end result is simply the following:

A user (Joe) deletes a spam email (from "SpamGuy") from their quarantine. A record for Joe/SpamGuy is added to MyAutoBL. A few hours later SpamGuy sends more spam to Joe. The scheduled job in the SQL Server simply removes from the quarantine table (tblQuarantine) any emails that have the To/From present in MyAutoBL. So all future emails from SpamGuy to Joe (that SpamFilter will have blocked due to other filters) that will be quarantined, will be removed from the quarantine by this scripted job, so that Joe does not see any more emails form SpamGuy in his quarantine.

This will not "help" SpamFilter stopping more spam as this MyAutoBL table is not checked. It simply helps reducing the clutter from the quarantined emails by removing emails in the quarantine tables that were forcefully removed in the past.
The email 
end-users do not see emails th


-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP


Posted By: lyndonje
Date Posted: 01 February 2010 at 4:29am
Hi Roberto,

Going by what you have said, it would seem as though my assumption is correct... Maybe you misunderstood what I was saying?

Example, I receive emails from Joe, who does not unsubscribe me, or who's unsubscribe does not work, or maybe you have to unsubscribe by sending a reply email where my from address does not match the to address they send the mailshots to.

Joe's emails never end up in my quarantine, therefore I can not ForceDelete, so instead I add our address pair to the MyAutoBL table manually.

What I was saying is that if the SpamFilter binary's do not check this table, and as his emails are not being rejected, my manual addition to MyAutoBL would have absolutely no effect on blocking his email, and so these emails would still be delivered to my Inbox?

SF checks the auto whitelist, and whitelists and emails which match any email pair found. In the same way, it would be better if SF also checked the MyAutoBL table, and simply rejected without quarantining any emails which match an address pair found in the MyAutoBL table.


Posted By: LogSat
Date Posted: 01 February 2010 at 9:47pm
Yeap, I sure did misunderstood you :-)
You are perfectly correct. Adding the entry to the MyAutoBL will not alter what SpamFilter does, so the emails will still be delivered to your Inbox.

The use of this extra table was added per user request, and as it does indeed help cleaning out the user's quarantine a bit, we liked the idea as well and implemented. It is was in no way meant as a means to actually block emails, since that was done already by the existing emails/domains blacklists. This MyAutoBL would however be more specific and could pinpoint blocking specific emails to specific users however... so we may have SpamFilter use it in the future as well to block incoming spam as you're hinting!


-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP


Posted By: lyndonje
Date Posted: 02 February 2010 at 4:08am
Hi Roberto,

Thanks for the confirmation. I shall keep on the look out for that one :)


Posted By: WebGuyz
Date Posted: 03 February 2010 at 10:38am
Originally posted by LogSat LogSat wrote:

... This MyAutoBL would however be more specific and could pinpoint blocking specific emails to specific users however... so we may have SpamFilter use it in the future as well to block incoming spam as you're hinting!
 
Roberto,
 
  We've been hinting this for years. Maybe we were hinting incorrectly LOL


-------------
http://www.webguyz.net


Posted By: lyndonje
Date Posted: 23 June 2010 at 7:32am
Hi Roberto,

I've been asked about this feature again by another client. Is there any likelihood of a to|from blacklist feature being introduced at some point in the future?

Thanks.




Posted By: LogSat
Date Posted: 27 June 2010 at 1:16pm
lyndon,

We're "trying" to release a new, long overdue, official build within the next couple of weeks. We've always postponed this as new features are continuously added, making it hard to have a "stable", proven build ready. If you can ping me back in 304 weeks, we'll see what can be done for this latter request.


-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP


Posted By: lyndonje
Date Posted: 28 June 2010 at 3:00am
Thanks Roberto, I'll make a note in my diary ;)


Posted By: AndrewD
Date Posted: 28 September 2010 at 8:58pm
it seems to me that this is being thought of all wrong. i think the answer needs to come from the frontend GUI not the backend filter.
The problem with the filter doing the deletion:-
user receives a spam from a known (faked) email address, he then deletes it from quarantine. Now that KNOWN user can not send to the user. and as you are requesting it deletes the message rather than quarantineing it, there is no way that the user can get the message back.
 
The Answer is with the GUI. simply create another DB (i would recomend against adding tables to the spamfilter one as it is easier to manage if all your custom tables exist in there own DB) then at the front end webpage create a link on the senders email address that gives the user the option to "Hide this address", Hide this domain" then store store it into your custom table along with the users name, the web script then marks the message for deletion, and reloads the page.
 
you need to customize the SQL query for the page load so that it pulls all the emails as long as the address does not appear in the custom table as mentioned above.
 
Also on the quarantine page i have a Toggle button to show hidden messages, in case they are looking for something in particular.
 
This proves to be the best option for a number of reasons:-
1. You want the Client to see that there are a large number of emails that you are stopping from getting to there network, otherwise what are they paying you for. It is bad enough that greylisting cuts out about 90% of spam so the users never get to see that. (Please read "Bad Enough" in the context that i meant it. There is no way i would ever run without it, i will happily live with the issue of explaining to my clients the positives.)
 
2. What you are requesting is that emails that match the autoBlackList be instantly dropped and not sent to the quarantine archive. So then getting them back seems to be a major issue.
 
 
Lastly
this thread was started on 15/10/09 and still being actively chased 1 month later. Why wouldnt you just get the GUI changed. Shouldnt take anymore than 8 hours work for a developer to add the feature.
 
Thats my two cents worth.



Print Page | Close Window