Spam Filter ISP Forums : Anyone using SFE with a few hundred domai https://www.logsat.com/spamfilter/forums/ Thu, 12 Mar 2026 04:24:16 +0000 Thu, 19 Jul 2007 07:30:20 +0000 http://blogs.law.harvard.edu/tech/rss Web Wiz Forums 11.04 360 https://www.logsat.com/spamfilter/forums/RSS_post_feed.asp?TID=6107 <![CDATA[Spam Filter ISP Forums]]> https://www.logsat.com/spamfilter/forums/forum_images/web_wiz_forums.png https://www.logsat.com/spamfilter/forums/ <![CDATA[Anyone using SFE with a few hundred domai : I just realized that we already...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10502&title=anyone-using-sfe-with-a-few-hundred-domai#10502 Author: LogSat
Subject: 6107
Posted: 19 July 2007 at 7:30am

I just realized that we already have a detailed description of how to perform the upgrade to SpamFilter Enterprise offline in SpamFilter's new help file!

The full install of the current SpamFilter official release always includes the full manual, and the procedure is described in the chapter:

Upgrading to SpamFilter Enteprise "offline"

...I guess I'm included in the very popular group of admins who doesn't read manuals...

I must admit that I didn't remember it being there either, but that page pooped up on a Google search I was performing, and that's how I then realized it was there

We'll post the article to the forum as well soon.
]]> Thu, 19 Jul 2007 07:30:20 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10502&title=anyone-using-sfe-with-a-few-hundred-domai#10502 <![CDATA[Anyone using SFE with a few hundred domai : Yes, you may indeed want to differentiate...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10500&title=anyone-using-sfe-with-a-few-hundred-domai#10500 Author: LogSat
Subject: 6107
Posted: 18 July 2007 at 3:06pm

Yes, you may indeed want to differentiate the backups for the two sets of tables.

As a side-note, WebGuyz is absolutely correct when stating that if the SFE tables contain *lots* of data, the startup time will increase. It is also to note however, that the examples him and I were referring to assumed that each of the hundreds of domains was customized to have unique settings.

If you use SFE, but have default settings for the most part of the domains, the startup time will decrease dramatically. For example, while for starting up SFE with 200 domains, each customized, takes about 60 seconds on the 2GHz laptop, starting up with 600 domains, all inheriting the default settings, takes only about 10 seconds on the same laptop.
]]> Wed, 18 Jul 2007 15:06:49 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10500&title=anyone-using-sfe-with-a-few-hundred-domai#10500 <![CDATA[Anyone using SFE with a few hundred domai : Roberto, I understand your reasons...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10499&title=anyone-using-sfe-with-a-few-hundred-domai#10499 Author: atifghaffar
Subject: 6107
Posted: 18 July 2007 at 2:36pm

Roberto, I understand your reasons and myisam/innodb is not the issue.

One example:
Backup database without quarantine (just domains, prefs, etc) : 5MB
Backup database quarantine 50GB+

Perhaps we can exclude some tables from the backup...



]]> Wed, 18 Jul 2007 14:36:56 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10499&title=anyone-using-sfe-with-a-few-hundred-domai#10499 <![CDATA[Anyone using SFE with a few hundred domai : Atif,Both SFI and SFE share the...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10498&title=anyone-using-sfe-with-a-few-hundred-domai#10498 Author: LogSat
Subject: 6107
Posted: 18 July 2007 at 11:33am

Atif,

Both SFI and SFE share the same database. The SFI and SFE tables are completely independent, so there is no interaction between the data they contain, but they must reside in the same schema. This was done because in the future SpamFilter may add interaction between the tables, and also to simplify the design of future web interfaces to manage SFE (we're beta testing our soon-to-be-released one).

Please note that in the same database, if you're using MySQL, you can have some tables use the InnoDB engine and others using the MyISAM engine, so you can mix and match them as you please.


]]> Wed, 18 Jul 2007 11:33:08 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10498&title=anyone-using-sfe-with-a-few-hundred-domai#10498 <![CDATA[Anyone using SFE with a few hundred domai : atifghaffar, Same o Same o. Thats...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10495&title=anyone-using-sfe-with-a-few-hundred-domai#10495 Author: WebGuyz
Subject: 6107
Posted: 17 July 2007 at 11:38pm

atifghaffar,

 Same o Same o.

Thats why I said you needed a solid db. Doing the upgrade as Roberto described did not mess with my production SFI because there was no actual traffic going to the copied spamfilter directory that I was using to do the upgrade. Was a little hesitant about that and it took Roberto a couple of trys to convince me that I wouldn't screw my production system even though I was upgrading the same database my production system was using. A little unnerving at first but he was right. You can switch back and for between SFE and SFI and each system would be independant of the other. Pretty good design in that regard.

]]> Tue, 17 Jul 2007 23:38:36 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10495&title=anyone-using-sfe-with-a-few-hundred-domai#10495 <![CDATA[Anyone using SFE with a few hundred domai : Roerto, Another glitch that I...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10494&title=anyone-using-sfe-with-a-few-hundred-domai#10494 Author: atifghaffar
Subject: 6107
Posted: 17 July 2007 at 11:16pm

Roerto,

Another glitch that I saw when moving to SFE was that the SFE expects the
SFE database to be on the same place as the quarantine database.

I hope, I am wrong.

The quarantine database needs a very poweful innodb based machine
while the SFE configuration can come from another moderate or a replica
machine.

Please correct me if I am wrong, I could not find where to separate these
2 databases.

]]> Tue, 17 Jul 2007 23:16:17 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10494&title=anyone-using-sfe-with-a-few-hundred-domai#10494 <![CDATA[Anyone using SFE with a few hundred domai : Roberto, I am moving the current...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10493&title=anyone-using-sfe-with-a-few-hundred-domai#10493 Author: atifghaffar
Subject: 6107
Posted: 17 July 2007 at 11:07pm

Roberto,

I am moving the current infrastructure from the shared-isp-infrastructure
to the mail-only-infrastructure and my first insinct was also to move to
SFE, but:

1. I want the move to be very smooth. just IP address changes at the MX
level.
2. Already separating smtpauth and MX (SF(I|E) is great for MX but as
smtpauth machine, I would prefer the basebones (syslogging) postfix
3. Fallback. In case, there is any problems, I would like to fallback on the
current machines (running untouched version of SFI, to be replaced as
soon as the migration is done)


When the 1 to 1 migration is done, I will have the previous set of
machines to play with SFE and to acquire more licences :-)

]]> Tue, 17 Jul 2007 23:07:58 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10493&title=anyone-using-sfe-with-a-few-hundred-domai#10493 <![CDATA[Anyone using SFE with a few hundred domai : atifghaffar, I was just wondering...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10492&title=anyone-using-sfe-with-a-few-hundred-domai#10492 Author: WebGuyz
Subject: 6107
Posted: 17 July 2007 at 11:06pm

atifghaffar,

   I was just wondering how big some of your files must have been getting with that much data.

  Like yourself we also use SpamAssassin after our primary SFE's feeding into a second instance on one of the SFE's that checks for specific keywords and quarantines the email if found. It would have been nice to have SA incorporated inside of SFE, but in retrospect it makes more sense to use it as a final filter after SFE has done all the heavy lifting since SA is so very CPU intensive. Having SA check on each email would really require a lot of horsepower.

  I think you'll like SFE, but make sure the db is solid.



Edited by WebGuyz]]> Tue, 17 Jul 2007 23:06:05 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10492&title=anyone-using-sfe-with-a-few-hundred-domai#10492 <![CDATA[Anyone using SFE with a few hundred domai : LogSat wrote:Atif,For comparison,...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10491&title=anyone-using-sfe-with-a-few-hundred-domai#10491 Author: WebGuyz
Subject: 6107
Posted: 17 July 2007 at 10:56pm

Originally posted by LogSat LogSat wrote:

Atif,

For comparison, we just tested SFE on a laptop (the machine I have available right now) with a single Intel Core 2 Duo 2GHz CPU, running both SFE and SQL. SFE started up in just about 60 seconds with slightly over 200 domains.

Yeah, 60 seconds, but does each of the 200 domains have over 8,000 entries in the tblbl_keywords and do you have over 350k entries in your autowhitelist table.  

I guess its all relative, but you will need some ponies with 20+K domains.

I love SFE and glad we 'talked' Roberto and his team into writing it.

]]> Tue, 17 Jul 2007 22:56:44 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10491&title=anyone-using-sfe-with-a-few-hundred-domai#10491 <![CDATA[Anyone using SFE with a few hundred domai : Hello Webguyz, I was about to...]]> https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10490&title=anyone-using-sfe-with-a-few-hundred-domai#10490 Author: atifghaffar
Subject: 6107
Posted: 17 July 2007 at 10:41pm

Hello Webguyz,

I was about to post a detailed reply, but these damn mac osx keyboard
shooot-cuts came across my way (not the same as kde/win), for example
Apple+Home took me to apple.com... welll whatever...

So here is some more information again.
We have a couple of machines running 2 instances of SPI (Okayed by
Roberto in terms of licensing).

Instance1 (incoming emails (mx and smtp-authed))
Instance 2 (outoging emails (this one can use a lot of help))

There is a lot in between, nothing very special though:
you can get an idea at (perhaps outdated when u read this) this diagram:
http://www.worldsoft-postmaster.info/


For the incoming spamfilter(the internet facing mx and smtpauth
machine): this is what I have from one of our machines
xyz # cat AllowedDomainFilterMatrix.txt | wc -l
20362

This is the amount of domains that SFI is handling. This file is updated
whenever we get/lose a domain.

This was the simple task, but we do not allow anything at anydomain
(obvioulsy), so we use the AuthorizedRecipients.

xyz: # cat AuthorizedRecipients.txt | wc -l
45093


The quarantine database is ofcoarse on a different machine running linux
+mysql (innodb)

There is a mailadmin machine that makes these different Authorized
maps and scp's(cygwin) them to the spamfilter-isp machines.

Hope this helps

and

looking forward to moving to SFE soon to avoid this hastle.

]]> Tue, 17 Jul 2007 22:41:00 +0000 https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107&PID=10490&title=anyone-using-sfe-with-a-few-hundred-domai#10490