Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - gmail greylisting
  FAQ FAQ  Forum Search   Register Register  Login Login

gmail greylisting

 Post Reply Post Reply
Author
dave.m View Drop Down
Newbie
Newbie


Joined: 05 August 2009
Status: Offline
Points: 5
Post Options Post Options   Thanks (0) Thanks(0)   Quote dave.m Quote  Post ReplyReply Direct Link To This Post Topic: gmail greylisting
    Posted: 05 August 2009 at 1:44pm
We recently downloaded SpamFilterISP and are currently doing tests with it.  We are a website design company that hosts sites and e-mails for several companies and would like to migrate off of our existing spam solution that uses sendmail/spamassassin/etc on a Linux platform.

Installation went fine, and some tests have come through successfully. 

We're having an issue with gmail clients being able to send mail to us.  I looked through the forums, and saw a couple of other posts related to this, and it appears to have to do with greylisting.

If I understand correctly, when a mail comes in from gmail if the IP isn't on the greylisting "whitelist" it's told to check back in X minutes determined the greylisting setting.

Apparently, gmail does try back at whatever time interval the gmail servers are set to try back at - but is most likely coming from a different IP address.  Eventually this mail gets bounced back enough that it doesn't come through and the sender receives an undeliverable e-mail.

We would like to know if anyone has found a way to to whitelist all gmail sending IP's.  I know it can't be done by ranges from a previous post, but is there a way to import these into the database perhaps? 

Or has anyone else found a solution to this issue?

Thanks,

Dave

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 05 August 2009 at 3:59pm
Dave,

If you have a list of gmail's IPs, you can prevent specific IPs from being blocked by the GreyList filter as follows. SpamFilter stores the IPs that have been allowed to pass the greylist filter in the file:
\SpamFilter\Domains\GreyListAllowed.txt

The entries in that file are in the form:
aaa.bbb.ccc.ddd~nnnnn.ddddddddddd

where aaa.bbb.ccc.ddd is the IP address, and nnnnn.ddddddddddd is a decimal date value. For example:

80.69.70.3~39495.5492583796

The decimal date value indicates the number of days that have passed since 12/30/1899. The fractional part of the value is fraction of a 24 hour day that has elapsed.

SpamFilter will delete daily IPs whose date indicated above older than the number of days indicated in the following SpamFilter parameter (60 days by default):
GreyListAllowedHold=60

You can manually insert the IP you want in the GreyListAllowed.txt file, and assign it a date that is a few years in the future.  For example, adding 10 years would require adding 365x10=3650 to the value indicated:

80.69.70.3~39495.5492583796  plus 3650=
80.69.70.3~43145.5492583796

So, in short, simply add the above line to the GreyListAllowed.txt file. Please note that SpamFilter will only load this file during startup, it does not support changes made to it while it is running. You thus need to stop SpamFilter before making any manual changes to it.

Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
dave.m View Drop Down
Newbie
Newbie


Joined: 05 August 2009
Status: Offline
Points: 5
Post Options Post Options   Thanks (0) Thanks(0)   Quote dave.m Quote  Post ReplyReply Direct Link To This Post Posted: 06 August 2009 at 10:40am
Thanks for the quick response.

So I guess our options are to add the 65,000 IP addresses that gmail supposedly sends from:

216.239.32.0/19
64.233.160.0/19
66.249.80.0/20
72.14.192.0/18
209.85.128.0/17
66.102.0.0/20
74.125.0.0/16

And these have to be added individually, correct?  We can't add blocks of IP addresses from what I saw in other forum topics.

Or turn off greylisting - but this is still an effective way to block spam, right?

Has anyone else found a good resolution for this other than these two options? 

Thanks again,

Dave
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 August 2009 at 12:53am
Correct - the greylist filter does not support subnets/wildcards. The reason for this is that the greylist filter works on a TCP level by blocking IPs directly and immediately upon connections. It must be very fast to avoid DDOS attacks, and the most efficient way we could implement it was by having a plain text list in RAM to perform fast lookups. Mixing the list with subnets would cause the lookups to be much slower. Of course adding 65,000 IPs will also make it slower... but not as much as having admins enter 100s of subnets mixed with thousands of IPs.

There are however very, very good admins on this forum, so if anyone has any opinions on this we always listen!
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2009 at 12:22am
Roberto,
 
  Understand you want peak performance, but the reality is that we need to be able to compensate for these large farms of smtp gateways, as well as handling greylisting sharing amongst multiple SFE's. I had to setup a non-greylisting version of SFE  for a number of customer because of horrible delivery times because of  the issue of greylisting and multiple IP's from a single large ISP.  That really hurts when customers threaten to leave because of slow delivery when I bust my butt to deliver the best service possible. I would gladly give up performance to be able to not have this happen. What good is performance when your service is called poor because it can take an hour or more for an email to get from point A to B. Customer could care less that I have a high performance spam engine, all he knows is that his business contact  who was on the phone with him sends him an email, but my customer does not get it until over an hour later, he calls to tell me my service sucks. Dead
http://www.webguyz.net
Back to Top
Wayne View Drop Down
Groupie
Groupie
Avatar

Joined: 29 August 2006
Location: Switzerland
Status: Offline
Points: 60
Post Options Post Options   Thanks (0) Thanks(0)   Quote Wayne Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2009 at 7:18am
Roberto

I totally agree with WebGuyz!
We need this option and I'm also willing to sacrifice performance for that.
I cannot believe that this whitelist implementation would took about an hour for most customers. Otherwise i cannot explain how the other anti-spam manufacturers have solved this problem and you know, they have. Wink

Regards
Wayne


Edited by Wayne - 10 August 2009 at 7:21am
SF4.5.0.1-beta
Back to Top
yapadu View Drop Down
Senior Member
Senior Member


Joined: 12 May 2005
Status: Offline
Points: 297
Post Options Post Options   Thanks (0) Thanks(0)   Quote yapadu Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2009 at 10:42am
I often feel the delay of having to wait.  I know every time I sign up for an online service I am going to have to wait an extended period of time... knowing the email will come in 15 minutes or maybe an hour (or sometimes the next day).

Greylisting is doing what it is supposed to be doing though, just having it on the server keeps the CPU load levels down quite a bit by keeping the trash connections from even getting to the server.

I suspect there may be a suitable comprise, such as allowing individual IP addresses as well as a class C.  That reduces the number of entries people have to make and should still keep the complexity of the lookups down.

I don't know how SpamFilter works internally, but I have noticed that IP Addresses are all stored in 123.456.789.123 format in the databases.  When dealing with IP addresses I find it much quicker to store them as a numeric base 10 value - some binary format is probably much better than that.

Roberto is the master, and knows what spamfilter is doing behind the scenes but maybe storing the values in a different format would allow a speed improvement enough to allow some additions of range based lists.

Another option might be to maintain two lists in memory for the greylists, one of just IP's and another of class C network ranges etc.  You just need to search two lists, and you know the format of the addresses in both so mixing of IP's vs. Class C list searches should not cause big problems.

I also vote for some type of  shared list among servers.  If you have a cluster of spamfilter servers the issue of inbound greylisting gets even worse, causing even longer delays.  Imagine a cluster of 4 servers, some external server connects to server 1, gets rejected.  The next time it tries it connects to server 2, then a different server from their cluster tries and gets server 3.  Could take a very long time before the message gets through.

I'm happy with the grey listing at the moment though.


Back to Top
yapadu View Drop Down
Senior Member
Senior Member


Joined: 12 May 2005
Status: Offline
Points: 297
Post Options Post Options   Thanks (0) Thanks(0)   Quote yapadu Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2009 at 11:04am
Again, I'm just thinking out loud here...  If Roberto is storing the addresses in memory as strings, there could be quite a bit of improvement if the addresses were converted to base 10.  Then you could use some quick search methods, such as the discussion here:

http://leepoint.net/notes-java/algorithms/searching/binarysearch.html
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2009 at 5:45pm
Originally posted by WebGuyz WebGuyz wrote:

Understand you want peak performance, but the reality is that we need to be able to compensate for these large farms of smtp gateways, as well as handling greylisting sharing amongst multiple SFE's.

WebGuyz,

When you mention "performance", am I interpreting correctly in that you'd prefer to loose some performance in exchange for the flexibility of being able to manually add subnets in the greylist?

In regards to the sharing of the greylist amongst multiple servers, this is a *much* bigger challenge to address. Just to get the ball possibly rolling, could you let me know how many entries you currently have in the GreyListAllowed.txt file? 
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2009 at 10:33pm
Our greylist file has about 235K entries and service about 5k mailboxes. We have 2 SFE's.
<soapbox>
Greylisting is a excellent tool and really has decreased the amount of traffic hitting our servers, but if it causes long delivery delays and causes perception that your service is 'slow' because sometimes it takes over an hour for an email to get delivered (because it was bouncing around between different IP or in our case multiple SFE's), then it becomes a liablilty.
 
We have an organ donation firm that needed timely emails from all around the country, but even when they whitelisted everyone they would still get some emails over an hour old because of the greylisting. They threatened to leave so I put them on a copy of SFE with no greylisting. Since then I've had to put 6 other companies on that same SFE with no greylisting because of complaints of slow delivery of emails, usually from senders in companies like IBM and other large corps that have a LOT of different IP's used to deliver mail and which they alternate retries.
I hate it when people tell me my service sucks and I can't do anything about it. I love SFE, but greylisting is its achilles heel in a multi SFE environment and where its not easily possible to put complete subnets in for accepting emails from large ISP's.
 
</soapbox>
 
 
 
 
http://www.webguyz.net
Back to Top
yapadu View Drop Down
Senior Member
Senior Member


Joined: 12 May 2005
Status: Offline
Points: 297
Post Options Post Options   Thanks (0) Thanks(0)   Quote yapadu Quote  Post ReplyReply Direct Link To This Post Posted: 10 August 2009 at 11:34pm
My greylist file on the primary server has 200k, and the backup server has 66k.  These have all been automatically added, we have not put in any manual entries.
Back to Top
Wayne View Drop Down
Groupie
Groupie
Avatar

Joined: 29 August 2006
Location: Switzerland
Status: Offline
Points: 60
Post Options Post Options   Thanks (0) Thanks(0)   Quote Wayne Quote  Post ReplyReply Direct Link To This Post Posted: 11 August 2009 at 2:25am

So it seems I'm a little smaler, I only have 22k entries Smile Therefrom are 200 manually added.
SF4.5.0.1-beta
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 13 August 2009 at 12:00am
Within the next 24/36 hours we'll have an early beta for SpamFilter (both Standard and Enterprise) that allows the entry of IP wildcards (ex. 192.168.255.* or 192.168.*) in the GreyListAllowed.txt greylist file. You may contact us at support at logsat.com if you wish to test it before it's released on our website (please include your order number in the email so we can verify the current software maintenance).
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 13 August 2009 at 12:10am
You know what else would be really good, a way to do a soft reset to reload the greylist file without having to do a complete restart. The same way the spamfilter.ini is read in every minute, but it wouldn't have to be that frequently.
 
 I know, I know, we keep asking for the sky, but when you have over 400 domains and a huge autowhitelistdelivery table when using SFE it take almost 5 minutes to load SFE into memory.
 
Thanks! Wink


Edited by WebGuyz - 13 August 2009 at 12:12am
http://www.webguyz.net
Back to Top
Wayne View Drop Down
Groupie
Groupie
Avatar

Joined: 29 August 2006
Location: Switzerland
Status: Offline
Points: 60
Post Options Post Options   Thanks (0) Thanks(0)   Quote Wayne Quote  Post ReplyReply Direct Link To This Post Posted: 13 August 2009 at 2:11am
Originally posted by LogSat LogSat wrote:

Within the next 24/36 hours we'll have an early beta for SpamFilter (both Standard and Enterprise) that allows the entry of IP wildcards (ex. 192.168.255.* or 192.168.*) in the GreyListAllowed.txt greylist file. You may contact us at support at logsat.com if you wish to test it before it's released on our website (please include your order number in the email so we can verify the current software maintenance).


Great news Roberto! Tongue

Thanx!!
SF4.5.0.1-beta
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 17 August 2009 at 2:31pm
Thought .... are we using the "dnsbl white list"?  Can that be used to generate entries in the gray-list allow list that expire in a year or so?
Just letting ideas rattle around in my head!
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
dave.m View Drop Down
Newbie
Newbie


Joined: 05 August 2009
Status: Offline
Points: 5
Post Options Post Options   Thanks (0) Thanks(0)   Quote dave.m Quote  Post ReplyReply Direct Link To This Post Posted: 17 August 2009 at 4:41pm
Roberto,

We're currently testing the software (see original post :))  Would like to know if we can get a copy of this beta so I can try and get it approved for purchase.

Thanks!

Dave
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 17 August 2009 at 10:13pm
Dave,

Yes, it's not going to be a problem. If you can you please contact us via email at support at logsat.com we'll see how to proceed.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.416 seconds.