I kinda have it setup....

I run a script against the database to make every message appear to come from server id 2.

This will cause only one server to update the autowhitelist.

I then have a script that will copy the autowhitelist to a repository for all the other servers to download.

The server that generates the whitelist doesn't write to the proper file name. I've had problems where the server can overwrite the data in the file due to a copy conflict, so the that servers sync script copies the autowhitelist to the repository as the proper name for the other servers to download.

The repository has the master copy of all the other files, domain names etc.

The only thing I don't replicate is the honeypot. Hasn't been a big deal to do anyway.

I just remembered why I thought the .txt files were a good idea. Lets say you have a enterprise version with a centralized db and getting all the info from there. Lets say the DB goes down and your SFS (SpamFilter Satellites ) can no longer get info. Your spam filtering comes to a grinding halt.

If however your SFS's get the info and copy it to a local text file then update every minute like they do now they can keep working autonamously until the SF main db comes back up.

Roberto, we are doing the hard part, coming up with the names for the compenents in SFE. All you have to do is code it....

What I think a scalable SF would look like:

1) Keep master copy of all current text files in DB. Local SF server would download each list from DB and create text files and read them into memory as it does now. Instead of checking the text file every 60 seconds to see if it has been updated it would check to see if a db master file has been updated and if so dowload that one file to a textfile and read it into memory.(if the DB were down keep working off the local text file copy). If any customization by us was done it could be done right to the db and SF would propagate it, no more worrying about copying files back and forth or syncronizing them multiple SF servers.

2)The autowhitelistdelivery.txt file is powerfull. But the achilles heel is that it can get too big over time. Find a way to let each domain have its own autowhitelist. For example, if mydomain.com's autowhitelist gets updated in the SQL db, then just reload mydomains.com's autowhitelist to disk and read it into memory. The structure on the disk could look like:

\autowhitelist
    \domaina.com.txt
    \domainb.com.txt
    \domainc.com.txt

3)Make the equivelant of autowhitelist but for blacklists.
a) *@somedomain.com|*@mydomain.com 
b) turkey@yahoo.com|joe@mydomain.com

a) do not allow any mail from anyone at somedomain.com to reach anyone at mydomain.com.
b) do not allow email from turkey@yahoo.com to reach a single user joe@mydomain.com.

Roberto, SFI is very good as it is now but I'm thinking of areas that could be bottlenecks as your cutomers (everbody on this forum ;-) go forth and prosper and need to scale upward and want to bring SF along.

The overhead for the db queries would be slower unless your saying read them directly into memory from the DB which would be OK. But keeping multiple db's synced could be a big pain in the rear.

How do you keep local db copies synced with a master db? do you look for differences in time stamps or what? How often does it run?

I forgot to add:

The ability to add a check of our own into the datastream so that we can use products like SA SpamC. Kind of like what you doing with the anti-virus product but make it generic so that we can run a command line scanner against it. I know this would slow down SF, but the ability to do SA would be worth it. Also the ability to use a anti-virus like clam-v or f-prot thorugh that entry point as well.