Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Queuing Quarantined msgs w/ database down
  FAQ FAQ  Forum Search   Register Register  Login Login

Queuing Quarantined msgs w/ database down

 Post Reply Post Reply
Author
Derk View Drop Down
Groupie
Groupie


Joined: 26 January 2005
Location: United States
Status: Offline
Points: 39
Post Options Post Options   Thanks (0) Thanks(0)   Quote Derk Quote  Post ReplyReply Direct Link To This Post Topic: Queuing Quarantined msgs w/ database down
    Posted: 02 September 2007 at 4:58pm
We understand that while the database is off-line that messages the spam filter considers to be "bad" are just plain lost.  In a previous post that is considered to be a "feature".  We don't see how this could possibly be a positive feature.  While the spam filter does a great job, it is not perfect and there are often false positives.  So if the database is down those messages are just gone-too-bad-hope-you-didn't-want-them!

On the flip side if the destination mail server is down the messages are queued and delivered when the server returns.  If these messages can be queued why can't the quarantine messages to be queued or at least give us the option?

If not, what about a secondary database if the primary fails?

if not, what about the option to forward all messages to the destination if the database is down?  Personally, I would rather get garbage delivered and than not get legitimate messages.

we're curious how other users feel about the "feature".


Back to Top
Vermin View Drop Down
Newbie
Newbie


Joined: 25 January 2006
Location: South Africa
Status: Offline
Points: 2
Post Options Post Options   Thanks (0) Thanks(0)   Quote Vermin Quote  Post ReplyReply Direct Link To This Post Posted: 03 September 2007 at 9:20am

I would also prefer a option or someway of storing the messages - until the database is back up - we get blamed for loosing anything so always good to have proof of some sort.

Back to Top
Web123 View Drop Down
Newbie
Newbie
Avatar

Joined: 26 January 2005
Location: Finland
Status: Offline
Points: 31
Post Options Post Options   Thanks (0) Thanks(0)   Quote Web123 Quote  Post ReplyReply Direct Link To This Post Posted: 03 September 2007 at 10:36am
Me too!
Have tried to convince Roberto into this....maybe now?
 
/Kim
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 04 September 2007 at 11:30pm
We're watching the thread...

Weve always been hesitant in queing emails  if the DB  was offline, to then resume the archival when it came back online. This is  because there could be a HUGE amount of queued emails that has to e uploaded to the database, and we are afraid that this traffic cold overwhelm the DB server. Throttling the amount of upload to the DB is not simple, as what will fit one customer will not work for others.

This said, is a compromise reachable? What if SpamFilter simply spooled the emails to be archived to a temporary folder if the database is offline? That wold allow admis the ability to manually retrieve an email if needed. It wold also be up to the admins to maintain the temp directory by deleting old emails from it...

Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Web123 View Drop Down
Newbie
Newbie
Avatar

Joined: 26 January 2005
Location: Finland
Status: Offline
Points: 31
Post Options Post Options   Thanks (0) Thanks(0)   Quote Web123 Quote  Post ReplyReply Direct Link To This Post Posted: 05 September 2007 at 12:18am

...an option to queue them to a directory is much better than just deleting themClap

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 05 September 2007 at 1:35am
Originally posted by LogSat LogSat wrote:

We're watching the thread...

 Throttling the amount of upload to the DB is not simple, as what will fit one customer will not work for others.

 
How about a settable throttle where you could set it at a low number for uploading of catch up quaratine emails  during peak periods and set it higher during slower periods to try and catchup faster. Just losing quarantined mail if the DB is down could cause a lot of screaming by our customers.
http://www.webguyz.net
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 05 September 2007 at 10:56am
My 2 cents.  Spool them to disk and let the Administrator write a stored procedure to do what ever they want at what ever rate.  Offload the entire quarantine process for spooled messages (if SQL is down) away from the spamfilter application and put it in the hands of the SQL server and the Administrator.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
Derk View Drop Down
Groupie
Groupie


Joined: 26 January 2005
Location: United States
Status: Offline
Points: 39
Post Options Post Options   Thanks (0) Thanks(0)   Quote Derk Quote  Post ReplyReply Direct Link To This Post Posted: 05 September 2007 at 3:16pm
Any option besides just dumping the message is better than nothing!
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 05 September 2007 at 4:15pm
Originally posted by LogSat LogSat wrote:


This said, is a compromise reachable? What if SpamFilter simply spooled the emails to be archived to a temporary folder if the database is offline?


Roberto, yes. We can write script to read this directory and upload to the datbase when needed without overloading the db.

PS: I am asuming that SF-HEADERS will be present to they can be extracted and inserted in the db accordingly.

I can provide a perl re-inject script to do this if you provide the spooling functionality.



best regards

Atif
Back to Top
caratking View Drop Down
Groupie
Groupie


Joined: 13 March 2006
Location: United States
Status: Offline
Points: 79
Post Options Post Options   Thanks (0) Thanks(0)   Quote caratking Quote  Post ReplyReply Direct Link To This Post Posted: 07 September 2007 at 10:16am
I tried to get the concept of not just deleting messages into the product while SFE was being developed...  I think it made it onto some wish list, but never got into the final product.

The logic I find most interesting is how SFE was designed to work from the ground up with the thought that the DB would be down at times.  By keeping the config / settings locally and always getting updates from the DB.

Totally written to handle the failure of a DB.

Then to turn around and just delete messages when the DB is down...  does not seem right. 

I vote for placing them in a que, with a configuration setting to throttle how fast they are placed in the DB when the DB comes back online.  X number per minute etc.  If the DB server is just rebooted, or hangs for a short time there should not be so much mail to process and probably no one would even notice.

If someone ends up with 50,000 messages in the queue because the server was off over a weekend they can adjust the insert speed, review & delete confirmed spam etc. to help clear the que.
Back to Top
Derk View Drop Down
Groupie
Groupie


Joined: 26 January 2005
Location: United States
Status: Offline
Points: 39
Post Options Post Options   Thanks (0) Thanks(0)   Quote Derk Quote  Post ReplyReply Direct Link To This Post Posted: 07 October 2007 at 4:38pm
has there been any more thought about this?
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 October 2007 at 4:45pm
We are testing a beta version with this feature internally with pretty good results. Would you be interested in testing it as well? 
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Derk View Drop Down
Groupie
Groupie


Joined: 26 January 2005
Location: United States
Status: Offline
Points: 39
Post Options Post Options   Thanks (0) Thanks(0)   Quote Derk Quote  Post ReplyReply Direct Link To This Post Posted: 07 October 2007 at 5:12pm
absolutely.
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 October 2007 at 11:57pm
I'll be sending you a PM shortly with the download URL.

As far as configurations, there's none needed. SpamFilter will spool emails to be archived in the \SpamFilter\quarantine directory. On successful archival, the files will be deleted. If the database operation fails, the files will be  renamed and SpamFilter will re-attempt delivery every 5 minutes.

Each email will consist of two files in plain text so they can be examined if needed. One will be the email body itself, the other will contain a few database-specific entries.

If the DB is offline for several hours, there will be quite a bit of emails to be inserted into the database. SpamFilter will "throttle" the archival process so that the number of concurrent connections into the database is one half the maximum number of connections allowed nto SpamFilter.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 08 October 2007 at 2:13am
me too, me too! :-)

while our database is pretty stable, we do often have the problem that the tables are locked by the "delete old quarantine items" process, so queuing would help us a lot in that case!
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 08 October 2007 at 6:57am
Just sent you a PM as well with the download URL...
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 08 October 2007 at 8:24am
Thanks, Roberto!
Queueing works, but the .info files stay in the quarantine queue and I can't seem to find any of the queued items in the quarantine db...
will check the log and keep you updated.


Edited by mikek - 08 October 2007 at 8:43am
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 08 October 2007 at 8:38am
10.08.07 14:13:12:252 -- (3976) Starting to process quarantine directory...

was the only line in the log I could find...
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 08 October 2007 at 7:59pm
I apologize for the early beta.. Some last-minute changes in the nightly builds had some less-than-desired side effects. We saw the let-over .info files as well, along with the fact that some queued emails are still not being quarantined.

We've just uploaded an updated beta in the same location I sent in the PM. Just update the build number from 724 to 725 in the URL and you will be ale to download it.

You can delete any .info files that are left over from yesterday's build.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 09 October 2007 at 3:48am
Hi Roberto

Build 725 is running smoothly!

As far as I can tell, the queuing and delayed inserting is working as intended.

Thanks for a great feature! Now, about that greylisting feature... :-)
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 09 October 2007 at 6:01pm
Roberto,

approximately when will be this feature available?

My logs make me look bad...
see.
mysql> select * from stats_quar_count;
+----+------------+--------+
| id | date       | count  |
+----+------------+--------+
|  1 | 2007-09-13 | 516335 |
...
...
| 15 | 2007-09-27 | 273103 |
| 16 | 2007-09-28 | 180868 |

No quarantine for 29,30,01 :-(

| 17 | 2007-10-02 | 266506 |
| 18 | 2007-10-03 | 451231 |
| 19 | 2007-10-04 | 476447 |
| 20 | 2007-10-05 | 457875 |
| 21 | 2007-10-06 | 423162 |
| 22 | 2007-10-07 | 409856 |
| 23 | 2007-10-08 | 483850 |
| 24 | 2007-10-09 | 511022 |

+----+------------+--------+



best regards

Atif
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 09 October 2007 at 6:51pm
Atif,

We've just upgraded the status from beta to pre-release for this latest 3.5.4.725 build, and have made it available in the registered user area. There are currently no known issues with it (and hope it will remain like this Smile).
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 16 October 2007 at 5:06pm
Roberto,

Thanks. I will ask our sysadmin to test it soon.
Currently I am a bit busy getting around the limbo blacklist cache not available to mortals... :-)


best regards

Atif
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.482 seconds.