Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Anyone using SFE with a few hundred domai
  FAQ FAQ  Forum Search   Register Register  Login Login

Anyone using SFE with a few hundred domai

 Post Reply Post Reply
Author
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Topic: Anyone using SFE with a few hundred domai
    Posted: 16 June 2007 at 11:39pm

Interested in hearing from anyone using SFE with a few hundred domains. Wanted to compare notes and see what your performance is like.

Thanks!

http://www.webguyz.net
Back to Top
jerbo128 View Drop Down
Senior Member
Senior Member
Avatar

Joined: 06 March 2006
Status: Offline
Points: 178
Post Options Post Options   Thanks (0) Thanks(0)   Quote jerbo128 Quote  Post ReplyReply Direct Link To This Post Posted: 17 June 2007 at 12:49pm

Not running a few hundred - but...

Running about 50 domains with SFE
~3000 users
~400000 emails per day
~25-40 concurrent connections
MSSQL 2000 - Win2003 Server
Running on a single Quad Core Xeon @2.4Ghz with 4GB RAM
mem and Virt mem are both ~850MB
CPU usage usually less than 10%

PM me if you 'd like more info.

jerbo128

 

Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 17 July 2007 at 9:45pm
WebGuys, I am running the SFI with 20K+ domains with the intention of
moving to SFE.

What should I be careful about?
best regards

Atif
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 17 July 2007 at 10:01pm

Note that it takes about 3 minutes to load 300 domains. Probably take 10-13 or more for that many domains. Have a powerful box for that many domains. Have your db on a separate server

Can't believe you had that many on SFI. How big was your autowhitelistforcedelivery.txt file?

Other than that its been working just great. We have 2 SFE's to split the load and for redunadancy. SFE has surpassed all my expections with the excellent support of Roberto. I ran into a few minor issues initally but he fixed them all and now everything is great.

We created scripts to update our tblAuthorizedTo list and also have scripts to update the Autowhitelistforcedelivery with entries from our outbound mail server. Totally automated! The ability to use SQL code to update everything instead of text files is truly wonderful.

One nice thing that you can do is copy your existing spamfilter directory into a new directory and bring up the exe. There you can go thru the upgrade procedure while your production SFI is still cranking away. You can make sure everything is 100% before you shutdown your production server and convert it over.

Roberto,
   Maybe you should sticky the instructions for a painless upgrade like the ones you sent me which worked extremely well.

http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 17 July 2007 at 10:25pm
Atif,

WebGuyz is correct, we can provide you with a rather simple set of instructions to install another instance of SpamFilter that can run in Enterprise mode, using your existing settings and database, but without interfering with your live install. This will allow you to perform any tests you wish to see if SFE will work for you. If you can contact us by email I'll provide them to you, and yes, we really should add them to the documentation and/or the forum :-)

For comparison, we just tested SFE on a laptop (the machine I have available right now) with a single Intel Core 2 Duo 2GHz CPU, running both SFE and SQL. SFE started up in just about 60 seconds with slightly over 200 domains.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 17 July 2007 at 10:41pm
Hello Webguyz,

I was about to post a detailed reply, but these damn mac osx keyboard
shooot-cuts came across my way (not the same as kde/win), for example
Apple+Home took me to apple.com... welll whatever...

So here is some more information again.
We have a couple of machines running 2 instances of SPI (Okayed by
Roberto in terms of licensing).

Instance1 (incoming emails (mx and smtp-authed))
Instance 2 (outoging emails (this one can use a lot of help))

There is a lot in between, nothing very special though:
you can get an idea at (perhaps outdated when u read this) this diagram:
http://www.worldsoft-postmaster.info/


For the incoming spamfilter(the internet facing mx and smtpauth
machine): this is what I have from one of our machines
xyz # cat AllowedDomainFilterMatrix.txt | wc -l
20362

This is the amount of domains that SFI is handling. This file is updated
whenever we get/lose a domain.

This was the simple task, but we do not allow anything at anydomain
(obvioulsy), so we use the AuthorizedRecipients.

xyz: # cat AuthorizedRecipients.txt | wc -l
45093


The quarantine database is ofcoarse on a different machine running linux
+mysql (innodb)

There is a mailadmin machine that makes these different Authorized
maps and scp's(cygwin) them to the spamfilter-isp machines.

Hope this helps

and

looking forward to moving to SFE soon to avoid this hastle.

best regards

Atif
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 17 July 2007 at 10:56pm

Originally posted by LogSat LogSat wrote:

Atif,

For comparison, we just tested SFE on a laptop (the machine I have available right now) with a single Intel Core 2 Duo 2GHz CPU, running both SFE and SQL. SFE started up in just about 60 seconds with slightly over 200 domains.

Yeah, 60 seconds, but does each of the 200 domains have over 8,000 entries in the tblbl_keywords and do you have over 350k entries in your autowhitelist table.  

I guess its all relative, but you will need some ponies with 20+K domains.

I love SFE and glad we 'talked' Roberto and his team into writing it.

http://www.webguyz.net
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 17 July 2007 at 11:06pm

atifghaffar,

   I was just wondering how big some of your files must have been getting with that much data.

  Like yourself we also use SpamAssassin after our primary SFE's feeding into a second instance on one of the SFE's that checks for specific keywords and quarantines the email if found. It would have been nice to have SA incorporated inside of SFE, but in retrospect it makes more sense to use it as a final filter after SFE has done all the heavy lifting since SA is so very CPU intensive. Having SA check on each email would really require a lot of horsepower.

  I think you'll like SFE, but make sure the db is solid.



Edited by WebGuyz
http://www.webguyz.net
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 17 July 2007 at 11:07pm
Roberto,

I am moving the current infrastructure from the shared-isp-infrastructure
to the mail-only-infrastructure and my first insinct was also to move to
SFE, but:

1. I want the move to be very smooth. just IP address changes at the MX
level.
2. Already separating smtpauth and MX (SF(I|E) is great for MX but as
smtpauth machine, I would prefer the basebones (syslogging) postfix
3. Fallback. In case, there is any problems, I would like to fallback on the
current machines (running untouched version of SFI, to be replaced as
soon as the migration is done)


When the 1 to 1 migration is done, I will have the previous set of
machines to play with SFE and to acquire more licences :-)

best regards

Atif
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 17 July 2007 at 11:16pm
Roerto,

Another glitch that I saw when moving to SFE was that the SFE expects the
SFE database to be on the same place as the quarantine database.

I hope, I am wrong.

The quarantine database needs a very poweful innodb based machine
while the SFE configuration can come from another moderate or a replica
machine.

Please correct me if I am wrong, I could not find where to separate these
2 databases.

best regards

Atif
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 17 July 2007 at 11:38pm

atifghaffar,

 Same o Same o.

Thats why I said you needed a solid db. Doing the upgrade as Roberto described did not mess with my production SFI because there was no actual traffic going to the copied spamfilter directory that I was using to do the upgrade. Was a little hesitant about that and it took Roberto a couple of trys to convince me that I wouldn't screw my production system even though I was upgrading the same database my production system was using. A little unnerving at first but he was right. You can switch back and for between SFE and SFI and each system would be independant of the other. Pretty good design in that regard.

http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 18 July 2007 at 11:33am
Atif,

Both SFI and SFE share the same database. The SFI and SFE tables are completely independent, so there is no interaction between the data they contain, but they must reside in the same schema. This was done because in the future SpamFilter may add interaction between the tables, and also to simplify the design of future web interfaces to manage SFE (we're beta testing our soon-to-be-released one).

Please note that in the same database, if you're using MySQL, you can have some tables use the InnoDB engine and others using the MyISAM engine, so you can mix and match them as you please.


Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 18 July 2007 at 2:36pm
Roberto, I understand your reasons and myisam/innodb is not the issue.

One example:
Backup database without quarantine (just domains, prefs, etc) : 5MB
Backup database quarantine 50GB+

Perhaps we can exclude some tables from the backup...



best regards

Atif
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 18 July 2007 at 3:06pm
Yes, you may indeed want to differentiate the backups for the two sets of tables.

As a side-note, WebGuyz is absolutely correct when stating that if the SFE tables contain *lots* of data, the startup time will increase. It is also to note however, that the examples him and I were referring to assumed that each of the hundreds of domains was customized to have unique settings.

If you use SFE, but have default settings for the most part of the domains, the startup time will decrease dramatically. For example, while for starting up SFE with 200 domains, each customized, takes about 60 seconds on the 2GHz laptop, starting up with 600 domains, all inheriting the default settings, takes only about 10 seconds on the same laptop.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 19 July 2007 at 7:30am
I just realized that we already have a detailed description of how to perform the upgrade to SpamFilter Enterprise offline in SpamFilter's new help file!

The full install of the current SpamFilter official release always includes the full manual, and the procedure is described in the chapter:

Upgrading to SpamFilter Enteprise "offline"


...I guess I'm included in the very popular group of admins who doesn't read manuals...

I must admit that I didn't remember it being there either, but that page pooped up on a Google search I was performing, and that's how I then realized it was there

We'll post the article to the forum as well soon.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.328 seconds.