Scan protection with rate limiting

Abstract

Techniques described herein improve network security and traffic management. In an embodiment, a request associated with an identifier (ID) is received. It is determined whether the ID exists in a first membership database (MDB). If the ID exists in the first MDB, the request is serviced subject to a rate limit. If the ID does not exist in the first MDB, it is determined whether the ID exists in a second MDB. If the ID exists in the second MDB, the request is serviced. If the ID does not exist in the second MDB, the request is serviced subject to another rate limit. A response is received. The first and second MDBs can be updated based on the type of received response. In an embodiment, the response is classified as indicative of degraded or typical network performance, and the first and second MDBs are updated accordingly.

What is claimed is: 
     
         1 - 20 . (canceled) 
     
     
         21 . A method for processing content requests, the method comprising:
 receiving content requests from a plurality of sources;   based on identities of the sources, forwarding the requests to content servers at different rates for at least two different sources, said content servers responding to the content requests from the sources;   based on responses from the content servers, adjusting at least one rate for processing requests from at least one source.   
     
     
         22 . The method of  claim 21 , wherein
 each source is associated with one identifier,   said forwarding comprises forwarding the requests based on the identifiers associated with the requests.   
     
     
         23 . The method of  claim 22 , wherein said forwarding the requests based on the identifiers comprises:
 using each request's source identifier to associate the request with one type of request from a plurality of different types of requests, each type of request associated with a different rate for processing requests of that type;   using each request's associated type to identify a rate for forwarding the request to a content server that responds to the request.   
     
     
         24 . The method of  claim 23 , wherein said adjusting comprises changing the association of a particular identifier from one type to another in order to change the rate for processing requests from the particular source associated with the particular identifier. 
     
     
         25 . The method of  claim 23 , wherein using each request's identifier to associate the request with a request type comprises determining whether the identifier exists in one of a plurality of data storages associated with different request types. 
     
     
         26 . The method of  claim 25 , wherein the different data storages comprise at least a first data store storing identifiers known to be associated with sources sending requests that have detrimental effect on performance of the content servers, and a second data store storing identifiers known to be associated with sources sending requests that do not have detrimental effect on performance of the content servers. 
     
     
         27 . The method of  claim 26 , wherein
 the first data store is associated with a first request type, and the second data store is associated with a second request type,   forwarding the requests further comprises forwarding the requests that are associated with a third type that is different than the first and second types to the content servers at a rate faster than a first rate used to forward requests of the first type but slower than a second rate used to forward requests of the second type.   
     
     
         28 . The method of  claim 26 , wherein the third type is a request type defined for requests from unknown sources. 
     
     
         29 . A method for processing content requests to content servers, the method comprising:
 classifying content requests as first, second, third types of requests, said first type being requests known to have adverse impact on content servers, said second type being requests known not to have adverse impact on content servers, and said third type, and third type being unknown; and   forwarding requests of the first type at a first rate to the content servers, requests of the second type at a second rate to the content servers, and requests of the third type at a third rate to the content servers, the first rate being slower than the second and third rates, and the third rate is slower than the second rate.   
     
     
         30 . The method of  claim 29 , wherein the first, second and third types correspond to first, second and third types of request sources, and said classifying comprises classifying content requests as requests coming from first, second and third types of sources, with the first source type comprising sources known to send requests that have adverse impact on content servers, the second source type comprising sources known to send requests that do not have adverse impact on content servers, and the third source type comprises unknown sources. 
     
     
         31 . A non-transitory machine readable medium storing a program for processing content requests, the program executable by at least one processing unit, the program comprising sets of instructions for:
 receiving content requests from a plurality of sources;   based on identities of the sources, forwarding the requests to content servers at different rates for at least two different sources, said content servers responding to the content requests from the sources;   based on responses from the content servers, adjusting at least one rate for processing requests from at least one source.   
     
     
         32 . The non-transitory machine readable medium of  claim 31 , wherein
 each source is associated with one identifier,   the set of instructions for forwarding comprises a set of instructions for forwarding the requests based on the identifiers associated with the requests.   
     
     
         33 . The non-transitory machine readable medium of  claim 32 , wherein the set of instructions for forwarding the requests based on the identifiers further comprises sets of instructions for:
 using each request's source identifier to associate the request with one type of request from a plurality of different types of requests, each type of request associated with a different rate for processing requests of that type;   using each request's associated type to identify a rate for forwarding the request to a content server that responds to the request.   
     
     
         34 . The non-transitory machine readable medium of  claim 33 , wherein the set of instructions for adjusting comprises a set of instructions for changing the association of a particular identifier from one type to another in order to change the rate for processing requests from the particular source associated with the particular identifier. 
     
     
         35 . The non-transitory machine readable medium of  claim 33 , wherein the set of instructions for using each request's identifier to associate the request with a request type comprises a set of instructions for determining whether the identifier exists in one of a plurality of data storages associated with different request types. 
     
     
         36 . The non-transitory machine readable medium of  claim 35 , wherein the different data storages comprise at least a first data store storing identifiers known to be associated with sources sending requests that have detrimental effect on performance of the content servers, and a second data store storing identifiers known to be associated with sources sending requests that do not have detrimental effect on performance of the content servers. 
     
     
         37 . The non-transitory machine readable medium of  claim 36 , wherein
 the first data store is associated with a first request type, and the second data store is associated with a second request type,   the set of instructions for forwarding the requests further comprises a set of instructions for forwarding the requests that are associated with a third type that is different than the first and second types to the content servers at a rate faster than a first rate used to forward requests of the first type but slower than a second rate used to forward requests of the second type.   
     
     
         38 . The non-transitory machine readable medium of  claim 36 , wherein the third type is a request type defined for requests from unknown sources.