Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - SFDB HTTP Errors
  FAQ FAQ  Forum Search   Register Register  Login Login

SFDB HTTP Errors

 Post Reply Post Reply
Author
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Topic: SFDB HTTP Errors
    Posted: 31 October 2006 at 10:01am
Hi Roberto

I'm seeing quite a lot of HTTP Errors lately when SF queries and updates the SFDB.

I counted 6549 successful queries/updates today and 1992 HTTP Errors.

Would you like me to send you our log file or is it a known problem?

Cheers, Mike
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 31 October 2006 at 10:04am
"SFDB_Weberror has reached it's limit, SFDB checks are paused temporarily" was output 812 times today in the log... :-(
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4066
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 31 October 2006 at 10:06am
mikek,

What build of SpamFilter are you using? Are they timeout problems? If so, we may have added too short of a timeout on the new 605 build to fix a problem with the previous versions. We're actually seing this on our own server as well, but with a much lower percentage than you.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 31 October 2006 at 10:08am
maybe the timeouts are too short when the SFDB Server is a bit busy... or on trans-atlantic internet connections as ours (22 hops, ~150ms RTT)...
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 31 October 2006 at 10:08am
using build 605, and yes, the usual message is "read timeout"...

Edited by mikek
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 01 November 2006 at 4:54am

I think maybe a couple of SFDB mirrors across the net are called for. Perhaps some of the major SFI users (ISP's) would/should volunteer? Afterall, major ISP's have the most to gain by having a good SFDB lookup system.

SFDB can only go one direction: bigger, with more query's to handle with every new customer logsat gets. Roberto, maybe it's wise to test the water to see who would cooperate. It's certainly wise to plan ahead on this one. Afterall, resources are allways limited.

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4066
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 01 November 2006 at 8:47am
Actually CPU on the server we use for the SFDB is practically never above 10% peaks, and the average load is only 4%-5%. Mirorring won't help as the load is not the problem.

The issue is a bit different, and has to do with locking issues in SQL server when inserting new records.

We're looking into this now to see what can be done on the server's end to minimize these timeouts.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4066
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 01 November 2006 at 8:21pm
Update.

This is one of the times where a person could take a server and throw it out of the window.

We just found out that on our IIS server, the Microsoft's ADOVBS.INC file, which contains all the ADO-related constants used by IIS and Active Server Pages, is missing the following line:

Const adAsyncExecute = &H00000010

That happens to be the constant we use (or better, we *THOUGHT* we were using) to allow the SFDBUpload queries to execute asynchronously.

In simple terms, this means that once the web page on our SFDB web server that uploads new SFDB records is called by SpamFilter, the ASP code tells our database server to process the stored procedure, and to then return immediately. The key piece here is "then return immediately". This is done by using the adAsyncExecute constant in the ASP code. But since that constant was missing from the Microsoft's file, the ASP pages were waiting for the SQL statement to complete before terminating. This is why we have all been seing timeout errors recently, as while usually the query takes milliseconds to run, sometimes it does take a few seconds if the records are locked.

At 20:08:00 we added the missing entry, so the timeout errors should hopefully stop, or at least greatly reduce. We'll be monitoring the situation. In the meantime I'll have a camomille to cool down the nerves...

We apologize to everyone for the problems!



Edited by LogSat
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 02 November 2006 at 2:14am
Thanks for the update, I will observe the logs today and report later...

BTW: as an ASP programmer myself, I can give you the following tip: instead of including adovbs.inc just place the following line at the top of your script, or in global.asa:

<!--METADATA TYPE="typelib" NAME="Microsoft ActiveX Data Objects 2.8 Library" UUID="{2A75196C-D9EB-4129-B803-931327F72D5C}" VERSION="2.8"-->
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 02 November 2006 at 6:34am
A bit better, but not much.

Statistics for the last 12 hours:

Successful queries: 6666
HTTP Errors: 1005
SFDB checks are paused temporarily: 509
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4066
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 02 November 2006 at 7:32am
Yes, we see the same on our end. Looks like the timeout is not caused by the query, as we've verified that the calls are now indeed non-blocking, and they were not before. Back to the drawing board.


Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 02 November 2006 at 7:40am
ok, here's a wild guess: check the ASP -> Requests Queued with perfmon and see if there are a lot of queued requests...
Back to Top
dcook View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 January 2005
Location: United States
Status: Offline
Points: 174
Post Options Post Options   Thanks (0) Thanks(0)   Quote dcook Quote  Post ReplyReply Direct Link To This Post Posted: 03 November 2006 at 12:15pm
I too have had 980 of these errors in thae last 6 days --- HTTP Error in SFDBUploadIP check:Socket Error # 10054 -- Connection reset by peer.

I searched under socket error to find them.
Dwight
www.vividmix.com
Back to Top
Lee View Drop Down
Groupie
Groupie


Joined: 04 February 2005
Location: United States
Status: Offline
Points: 50
Post Options Post Options   Thanks (0) Thanks(0)   Quote Lee Quote  Post ReplyReply Direct Link To This Post Posted: 11 November 2006 at 9:53pm
I am also seeing quite a few of the same errors.

Lee


11/11/06 21:51:52:656 -- (2133) HTTP Error in SFDBUploadIP check:HTTP/1.1 500 Internal Server Error
11/11/06 21:51:52:826 -- (2133) SFDB - Added 71.251.35.252 - Response:
11/11/06 21:51:53:006 -- (2133) Disconnect
Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 29 November 2006 at 6:09pm

Roberto,

 

Looks like this isn't quite fixed yet... I'm getting quite a few of these...

11/29/06 01:13:55:531 -- (5536) HTTP Error in DoSFDBCheck:Read Timeout

Back to Top
StevenJohns View Drop Down
Senior Member
Senior Member


Joined: 03 August 2006
Status: Offline
Points: 119
Post Options Post Options   Thanks (0) Thanks(0)   Quote StevenJohns Quote  Post ReplyReply Direct Link To This Post Posted: 29 November 2006 at 6:11pm

Actually,

 

I'm also getting a few of these....

 

11/29/06 02:51:13:390 -- (7136) HTTP Error in SFDBUploadIP check:HTTP/1.1 500 Internal Server Error

11/29/06 06:23:10:125 -- (9204) HTTP Error in GetSFDBStats:Connect timed out.

 

 

Back to Top
Wayne View Drop Down
Groupie
Groupie
Avatar

Joined: 29 August 2006
Location: Switzerland
Status: Offline
Points: 60
Post Options Post Options   Thanks (0) Thanks(0)   Quote Wayne Quote  Post ReplyReply Direct Link To This Post Posted: 02 December 2008 at 10:31am
Push up

Roberto, is there something new about this problem?

Because we've got 1'410 "HTTP Error in GetSFDBStats:Connect timed out" in 24h.

Thx
Wayne
SF4.5.0.1-beta
Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 02 December 2008 at 12:09pm
Hmmm ... not sure if this means anything or helps but .... in 5 days:
 
1 of these: HTTP Error in GetSFDBStats:HTTP/1.1 503 Service Unavailable
3 of these: HTTP Error in SFDBUploadIP check:HTTP/1.1 503 Service Unavailable
 
All events happened around the same time ... within a 1 minute period.
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4066
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 02 December 2008 at 11:00pm
Wayne,

SpamFilter will poll stats from our SFDC and SFDB databases every two minutes to display the count of IPs and of spam hashes being blacklisted at any time. Twice every two minutes indicates 1440 queries every 24 hours, which is pretty close to the number of errors you see. If you see those errors exactly 2 minutes apart, it means that your SpamFilter server is not able to communicate with our website (specifically http://sfdb.logsat.com) to retrieve that information. Please also note that if your server is not able to make HTTP requests to that same URL, SpamFilter will not be able to use our SFDB and SFDC filters, which are very effective in stopping spam.

Can you please ensure that the server running SpamFilter is able to make those web connections? Proxy servers and/or web filtering appliances/applications in your network could be preventing that from occurring. You can verify that this process works by going to the "Settings" - "Configuration" tab in SpamFilter, and then clicking on the "SFDB Settings" button. In the window that appears, you should see in the bottom right a value for "Current IPs listed in the SFDB" in the order of 100,000-500,000.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Wayne View Drop Down
Groupie
Groupie
Avatar

Joined: 29 August 2006
Location: Switzerland
Status: Offline
Points: 60
Post Options Post Options   Thanks (0) Thanks(0)   Quote Wayne Quote  Post ReplyReply Direct Link To This Post Posted: 03 December 2008 at 4:52am
Roberto

Thank you for your input, the blame was the protection profile at the spamfilter rule of our firewall

Sorry for that!

Kind regards

Wayne
SF4.5.0.1-beta
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.047 seconds.