Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Beta questions
  FAQ FAQ  Forum Search   Register Register  Login Login

Beta questions

 Post Reply Post Reply Page  12>
Author
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Topic: Beta questions
    Posted: 06 January 2008 at 5:40pm
The Greylisting beta is VERY imipressive. The question is what kind of problems might we see with use over time. A few questions I'm sure will come up:
 
Where does Greylisting fit in Filter Order? Before whitelists?
 
In the even ANY ip has an issue and we need to make sure it gets through, is there any way to force an IP like adding it to an IP whitelist. Also some SFE users have custom filters for ALL domains so it would have to be a generic text file or table.
 
Also in the SFDC, what do the log entries look like if successful (or not)
 
Thanks for a truly impressive filter (that we've been bugging you for forever LOL)
 
 
http://www.webguyz.net
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 06 January 2008 at 7:02pm
Never mind the qestion about the SFDC, it is working.
 
Can you tell me what the SFDC threshold is? How many 'hits' before an entry is quarantined and where does this filter fit in the Filter Order.
 
Thanks!
http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 06 January 2008 at 11:43pm
We've updated the filter order at logsat.com/spamfilter/forums/forum_posts.asp?TID=5171&PID=11418#11418 to show the new filter order with all the latest updates.

In regards to the SFDC thresholds, in this first beta there is only one paramenter for it in the various "Filters.ini" files under the \SpamFilter\Domains directories, and it's:
SFDC_Threshold=1

We are overriding that minimm vale on our server by setting it to (currently) 6, however on our SFC server we also take into consideration the separate number of installations that report the same hash, and will only blacklist it if there is a minimum number of SpamFilter's reporting the same hash, and that hash is being send by another minimum of separate source IPs.. We won't go into further details as o not give away the innerworkings of this to spammers, sorry!
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 11:22am
Looking at the filter order it does not appear there is a way to manually add an IP that might be having a problem, short of shutting down SFE and manually updating the greylistallowed.txt file and then restarting SFE to read the list it. Haven't had that need occur but today is the first full day of testing and its the busiest
 
Also, would have thought that greylisting would be at the top of the list instead of Blacklist cache.
 
Looking very good ...
http://www.webguyz.net
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 11:38am
Originally posted by WebGuyz WebGuyz wrote:

Looking at the filter order it does not appear there is a way to manually add an IP that might be having a problem, short of shutting ....
 
WebGuyz,
 
What do you mean exactly?  An IP that may be having a problem?  The Greylist is not supposed to be manually edited as I understand it and it is not an "allow" per se'.


Edited by Desperado - 07 January 2008 at 11:43am
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 11:49am
Desperado,
 
 Postini uses about 20 (or more) different outgoing IP's to send outbound mail. They rotate those IP's when sending mail to avoid looking like they are spamming when they send hotmail.com or yahoo.com users a bunch of email.
 
Any mail from Postini (there are other ISP's who have banks of ougoing servers with different IP's) may take a LONG time to get to our users for  them to cycle thru all their IP's and in our case we have 2 SFE's (others have more) so it can cause enough of a delay that our customers might complain.
 
Playing 'what if' and trying to be proactive and think of ways that greylisting might be a liability and find a way around them.
 
So far so good. We hold all quarantined email for 3 days and I can see whats going to start happening in about 2-3 days. People will go into the quarantine and notice there are so few entries they will think something is wrong and start calling ...Wink
http://www.webguyz.net
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 12:04pm
WebGuys,
 
On the first part ... Hotmail and Yahoo and MANY others use many IP's also and it really did not take very long for our "GreyListAllowed.txt" to populate with most of the IP's.  Prior to the population, the delay was only 5 minutes plus a couple of seconds and most of the major services *seem* to be GreyList aware ... meaning that they retried within seconds of the GreyList time-out.  During the "GreyListAllowed.txt" build up, we had ZERO customer complaints ... which frankly did surprise me.
 
On the second part, Grey-Listed messages are not quarantined but rather are rejected with an SMPT reject of "421 This server implements greylisting, please try again in %Time% seconds" where %Time% is the differance between the connection and the "GreyListInterval" (300 seconds by default).  So, I am not sure what you are getting at in the second part of your message.
 
What I can say is I was totally against the GreyList theory since Yahoo started using it but see a HUGE reduction in garbage in my quarantine (2/3 reduction) and a huge reduction in overall server load as a direct result so I am becoming a convert!
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 12:09pm
Currently the greylist file is only imported once when SpamFilter starts up. We were going to change thing so it would be re-imported when it changed, however...:
1 - the filter is working so well, with such almost undetectable delays after a couple hours of implementing it, that we may not see the need for this
2 - this greylist file can easily contain millions of IPs, and allowing SpamFilter to read changes by an external program while SpamFilter itself writes to it may introduce too many problems..

We're leaving things "as-is" right now and we'll see how this filter evolves.

For the order, both the blacklist cache and the greylist will immediately terminate a connection if it doesn't pass the tests. The blacklist cache is smaller, and is thus slightly more efficient to check it first so we can block any spammer that will pound SpamFilter with multiple connection attempts before checking them against the greylist.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 12:20pm
Originally posted by LogSat LogSat wrote:

The blacklist cache is smaller, and is thus slightly more efficient to check it first so we can block any spammer that will pound SpamFilter with multiple connection attempts before checking them against the greylist.
I thought I would mention that on Jan 1, we were "pounded" by 2.6 MILLION connections from the same IP over a 4 hour period.  SpamFilter handled it so well that I only saw the resulting HUGE log file!  Other than that, the system never seemed to notice or care.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 12:27pm
:-)  the "problem" with the blacklist cache is that, since it blocks connections at the TCP level right away, all the spam that would have been received is never seen, and thus, unless looking at the logs, you never see how much spam was really blocked (a lot!!)
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 2:03pm
I have a question regarding the greylistinterval setting. Are most people leaving this at the 300 seconds? I'm wondering if 9 minutes wouldn't be more effective? I know it would make customer mail servers have to try twice but maybe it would help reduce spam just that much more??? Just a thought going through my head.
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 3:44pm
My 2 Cents:
GreyListInterval=420
GreyListLimboHold=8
GreyListAllowedHold=30
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 5:39pm
Whats your reasoning Dan? I'm curious if we're on the same page?
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 6:54pm
Nothing very scientific:
 
I felt 5 minuts was a little short but 10 is really too long to wait for a message.
 
The 8 Hour ... Really a server should not wait any longer than 4 hours to retry and most do not wait that long.  I did not want to go too short for fear that messages may NEVER get delivered if the sendmail default of a 4 hour queue flush was in place (most admins speed that up)
 
Last one ... Jury is still out.  I felt that my IP list would get way to big (already at half a million) and it also may be too long to allow possible "bad" ip's to not be grey-listed.  I did not want to go too short because I do not want IP's like hotmail's to have to re-establish a "trust" more often than not.  So ... I still do not know on this value but 90 is longer than I wanted.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 6:57pm
Thats kind of what I was thinking too. I'm pretty impressed with how much less spam is even coming through to the queue now...it's very impressive!
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 7:04pm

kspare,

I have fully 1/3 the load on my Database and most of the dictionary attacks and address probes have been nearly eliminated so I think we have a winner here.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
atifghaffar View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 May 2006
Location: Switzerland
Status: Offline
Points: 104
Post Options Post Options   Thanks (0) Thanks(0)   Quote atifghaffar Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 7:16pm
Roberto,

When something is runnign so fine, I usually suspect that something is wrong.
Still cant find it though.

kudos.

best regards

Atif
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 7:28pm
Atif,

I don't know what to say here... this was supposed to be an alpha version as we were just about to start testing it here at LogSat internally. I got tricked into leaking it here on the forums, and that same build then suddenly became a beta. As of now we still did not receive a single bug report on it, so it may as well be promoted to official release...
With this kind of luck, I may just disappear for a few days as I'll be spending them in Las Vegas!!
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 7:31pm

The botnet herds really do get stopped with the addition of the SFE version of greylisting.

But thinking about the original Greylisting spec (using triplet data) I think that over time it might be more desireable to go that route instead of just IP.
 
Looking at the logs I see junk coming thru from IP's that have been added to the greylist, probably from compromised mail servers since a mailserver will retry as its supposed to. Once that IP is added all spammers using that IP will have their junk come thru. If the triplet info was used, then it would probably stop more (or less would get thru) over time.
 
Roberto, you know us end user, never happy LOL
 
 
http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 7:36pm
As usual, our ears are always open to advice. We'll keep an eye on this, but please do note that with our "flavor" of greylisting, we are greatly reducing the risk of delaying delivery of emails due to the greylisting. yes, the side effect is that more IPs will slip thru, but (1) the other filters should get them, and (2) we can always tweak the greylisting parameters to reduce the number of days (90 by default, which is maybe excessive) permitted IPs remain in the "permitted" state.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 7:42pm
Hey Roberto,
 
Pick me up one of those new 150" flat-screen TV's while you are in LV!
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 07 January 2008 at 7:43pm
I hear ya Dan, I just reset greylisting to the same settings as you and wow, the difference on my database is just night and day, there is no way you could write a sql script to remove all this automatically and be 100% accurate.

I'd normally have 1000+ spams alone in the queue for me personally and now I have zero, so i'm pretty impressed!
Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 1:45am
Without a word of a lie. I'm seeing a 90-95% reduction in spam. The 10-5% that make it in are a combination of actual emails and spam that made it through. This is amazing.
Back to Top
ImInAfrica View Drop Down
Groupie
Groupie
Avatar

Joined: 27 June 2006
Location: FL, USA
Status: Offline
Points: 60
Post Options Post Options   Thanks (0) Thanks(0)   Quote ImInAfrica Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 10:32am
although we are very impressed with the greylisting, i see a lot (ten of thousands) of ip's which are clearly spammers ip's.

I'm thinking the following:
As an ISP we host email for over 2000 domains.
lets say spambots start sending emails at 15:00 and domain 1 is first on the list.
connection is rejected, and delayed for 300 seconds (or whatever the setting is)
even is this ip then tries to reconnect, it will only be allowed to reconnect at 15:05 right?
at 15:01 there are a couple of emails for domain 2 - 10
at 15:02 there are a couple of emails for domains 20-50
and so on.
at 15:05 we start receiving emails from this ip, bypassing the greylist.

The problem here, is that with this method of greylisting if you're places under spam attack for lets same 15 minutes, from the same ip range, then after 5 minutes they've broken through the first barrier. chances are they'll get caught straight away, but this situation is theoretical only.

We've observed that an ip range was sending emails (spam) for various domains, for over 1 hour. not spam flood, just a trickle. however since the greylist (on an installation with a lot of domains) "stops" working after 5 minutes,  it kinda defeats the point.

My suggestion is as follows:
on first connection start counting the time (by default 300 secs)
If further connection attempt are tried BEFORE the full 300 secs have expired, reset the count.
example:
first connect from 196.197.101.101 at 15:00
time to allow ip 15:05
second connect from 196.197.101.101 at 15:01
time to allow ip 15:06
third connect from 196.197.101.101 at 15:05
time to allow ip 15:10
and so on.

a correctly configured smtp server SHOULD not retry in a less then 5 minute period.
this change will greatly reduce the number of spam bots which are bypassing the greylist, as the timeout will continuously increment.  of course this 'may' lead to issues if the connection delay is set to too long.

any thoughts?

Amir
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 10:51am

Amir,

I understand your point and I, myself was worried about the Spammers getting through after the initial timeout was satisfied but I do not really see a meaningful degradation in the effectiveness and I have other filters that grab most of the persistent abusers.  I also feel that simple is better ... both from a functionality / reliability view but also I can't imagine my help-desk guys (and they are very sharp indeed) following the trail of an IP that has a moving target time-out.  Just my 2 cents.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 11:05am
The greylisting has helped a lot, but your right, the spammers find a way a usual to circumvent this.
 
I am working on a vbs script to run against previous days log file and extract all IP's of SFDB, SURBL, and AuthorizedTo failures and from this list any sending IP's that have at least 10 failures will get added to our blacklist or possibly to our firewall block list.
 
Also, some of these IP's are from compromised servers so those will always get through the greylist.
 
At least the Bot herders have been slowed down a bit, as that where I see the biggest difference. Fire and forget  spamming is where greylisting really shines.
http://www.webguyz.net
Back to Top
dcook View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 January 2005
Location: United States
Status: Offline
Points: 174
Post Options Post Options   Thanks (0) Thanks(0)   Quote dcook Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 3:42pm
 
Suggestion for Beta: I often search the logs for information and it would be helpful if the greylist entry in the logs included the senders email address as well - for ease in quickly tracing greylist false positives.  I usually have the senders email address but not necessarly the IP number.  Also the returned message from the greylist should be customizable in the final release.
Will the greylisting be per domain in enterprise spamfilter?
 
Dwight
www.vividmix.com
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 4:02pm
The greylisting occurs at the TCP level right after a connection attempt is detected. The server is disconnected before they even have a chance to output the commands that specify the sender and the recipient. For this reason, neither the "from" nor the "to" domains are known, and thus the filter can't be customized per domains in SFE, nor can we log that in the SpamFilter logs, sorry!
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
dcook View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 January 2005
Location: United States
Status: Offline
Points: 174
Post Options Post Options   Thanks (0) Thanks(0)   Quote dcook Quote  Post ReplyReply Direct Link To This Post Posted: 08 January 2008 at 5:55pm
I really knew that answer before I asked it. But I always want the impossible, don't you?
I am already getting real fast at log trace now and you do have a winner here! 
 
I think the 150" TV is much better than the new thin 11" to watch.  Enjoy CES Roberto!


Edited by dcook - 08 January 2008 at 5:56pm
Dwight
www.vividmix.com
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4068
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 09 January 2008 at 1:44pm
FYI - an updated beta is available in the registered user area.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
 Post Reply Post Reply Page  12>
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.094 seconds.