Dynamically populated manifests and manifest-based prefetching

Abstract

Described herein are, among other things, systems and methods for generating and using manifests in delivering web content, and for using such manifests for prefetching. Manual and automated generation of manifests are disclosed. Such manifests preferably have placeholders or variables that can be populated at the time of the client request, based on data known from the request and other contextual information. Preferably though without limitation an intermediary device such as a proxy server, which may be part of content delivery network (CDN), performs the function of populating the manifest given a client request for a page. An intermediary or other computer device with a populated manifest can utilize that completed manifest to make anticipatory forward requests to an origin to obtain web resources specified on the manifest, before receiving the client's requests for them. In this way, many kinds of content may be prefetched based on the manifest.

1 - 26 . (canceled) 
     
     
         27 . A method implemented by one or more computer apparatus, the computer apparatus comprising one or more servers that have one or more microprocessors and memory holding computer program instructions for execution by the one or more microprocessors to perform the method, the method comprising:
 upon execution of the computer program instructions, the one or more servers performing:
 generating a generic manifest, said generation comprising:
 (a) analyzing of any of: page rendering and log files; 
 (b) providing in the generic manifest a variable for a URL query parameter, the variable denoting a string to be populated at the time of a client request; 
 
 receiving a first request from a first client device for a first HTML document; 
 retrieving the first HTML document from any of: (i) a remote server and (ii) a local cache; 
 determining that the first HTML document is associated with the generic manifest, and retrieving the generic manifest; 
 wherein said retrieval of the generic manifest comprises retrieving the generic manifest from any of: (i) the remote server and (ii) the local cache, the generic manifest specifying a set of one or more resources by one or more URLs; 
 sending the first HTML document to the first client device, in response to the first client device's request for the HTML document; 
 rewriting one or more strings in at least one of the URLs in the generic manifest based on data in the first client device's request for the first HTML document, to create a first populated manifest; 
 receiving a second request from a second client device for a second HTML document; 
 retrieving the second HTML document from any of: (i) the remote server and (ii) the local cache; 
 determining that the second HTML document is associated with the generic manifest, and retrieving the generic manifest; 
 wherein said retrieval of the generic manifest comprises retrieving the generic manifest from any of: (i) the remote server and (ii) the local cache, the generic manifest specifying a set of one or more resources by URLs; 
 sending the second HTML document to the second client device, in response to the second client device's request for the second HTML document; 
 rewriting one or more strings in at least one of the URLs in the generic manifest based on data in the second client device's request for the second HTML document, to create a second populated manifest. 
   
     
     
         28 . The method of  claim 27 , wherein the data in the first client device's request that is used for rewriting the one or more strings comprises any of: a value of a request header field, a universal resource locator (URL) parameter, and a value in a cookie. 
     
     
         29 . The method of  claim 27 , further comprising requesting a particular resource on the first populated manifest from any of: (i) the remote server and (ii) the local cache, said request being prior to receiving a request from the first client device for the particular resource. 
     
     
         30 . The method of  claim 29 , wherein the particular resource is embedded in an HTML document other than the first HTML document that the first client device requested. 
     
     
         31 . The method of  claim 29 , wherein the first HTML document includes a script having instructions executable by the first client device to generate a request for the particular resource. 
     
     
         32 . The method of  claim 27 , wherein the generic manifest further comprises a default value for the variable. 
     
     
         33 . The method of  claim 27 , wherein said generation of the generic manifest further comprises:
 identifying first and second manifests as similar manifests, said identification comprising comparing the first and second manifests to determine that a first URL on the first manifest and a second URL on the second manifest differ by a query parameter.   
     
     
         34 . The method of  claim 27 , wherein the first populated manifest comprises a particular resource, the particular resource being the target of a request generated by a script included in the first HTML document, the script having instructions executable by the client device to generate the request for the particular resource. 
     
     
         35 . The method of  claim 34 , wherein the script has instructions executable by the client device to generate an asynchronous request for the particular resource. 
     
     
         36 . The method of  claim 35 , wherein the script is an AJAX script. 
     
     
         37 . The method of  claim 27 , wherein the first populated manifest comprises a particular resource that is embedded in an HTML document other than the first HTML document that the client device requested. 
     
     
         38 . A system, comprising:
 one or more computers communicating over one or more networks, the one or more computers having one or more hardware processors computer memory storing instructions executed by the one or more hardware processors, the instructions comprising:
 program code executable to generate a generic manifest, said generation comprising:
 (a) analyzing of any of: page rendering and log files; 
 (b) providing in the generic manifest a variable for a URL query parameter, the variable denoting a string to be populated at the time of a client request; 
 
 program code executable to receive a first request from a first client device for a first HTML document; 
 program code executable to retrieve the first HTML document from any of: (i) a remote server and (ii) a local cache; 
 program code executable to determine that the first HTML document is associated with the generic manifest, and retrieving the generic manifest; 
 wherein said retrieval of the generic manifest comprises retrieving the generic manifest from any of: (i) the remote server and (ii) the local cache, the generic manifest specifying a set of one or more resources by one or more URLs; 
 program code executable to send the first HTML document to the first client device, in response to the first client device's request for the HTML document; 
 program code executable to rewrite one or more strings in at least one of the URLs in the generic manifest based on data in the first client device's request for the first HTML document, to create a first populated manifest; 
 program code executable to receive a second request from a second client device for a second HTML document; 
 program code executable to retrieve the second HTML document from any of: (i) the remote server and (ii) the local cache; 
 program code executable to determine that the second HTML document is associated with the generic manifest, and retrieving the generic manifest; 
 wherein said retrieval of the generic manifest comprises retrieving the generic manifest from any of: (i) the remote server and (ii) the local cache, the generic manifest specifying a set of one or more resources by URLs; 
 program code executable to sending the second HTML document to the second client device, in response to the second client device's request for the second HTML document; 
 program code executable to rewrite one or more strings in at least one of the URLs in the generic manifest based on data in the second client device's request for the second HTML document, to create a second populated manifest. 
   
     
     
         39 . The system of  claim 38 , wherein the data in the first client device's request that is used for rewriting the one or more strings comprises any of: a value of a request header field, a universal resource locator (URL) parameter, and a value in a cookie. 
     
     
         40 . The system of  claim 38 , further comprising requesting a particular resource on the first populated manifest from any of: (i) the remote server and (ii) the local cache, said request being prior to receiving a request from the first client device for the particular resource. 
     
     
         41 . The system of  claim 40 , wherein the particular resource is embedded in an HTML document other than the first HTML document that the first client device requested. 
     
     
         42 . The system of  claim 40 , wherein the first HTML document includes a script having instructions executable by the first client device to generate a request for the particular resource. 
     
     
         43 . The system of  claim 38 , wherein the generic manifest further comprises a default value for the variable.. 
     
     
         44 . The system of  claim 38 , wherein said generation of the generic manifest further comprises:
 identifying first and second manifests as similar manifests, said identification comprising comparing the first and second manifests to determine that a first URL on the first manifest and a second URL on the second manifest differ by a query parameter.   
     
     
         45 . The system of  claim 38 , wherein the first populated manifest comprises a particular resource, the particular resource being the target of a request generated by a script included in the first HTML document, the script having instructions executable by the client device to generate the request for the particular resource. 
     
     
         46 . The system of  claim 45 , wherein the script has instructions executable by the client device to generate an asynchronous request for the particular resource. 
     
     
         47 . The system of  claim 45 , wherein the script is an AJAX script. 
     
     
         48 . The system of  claim 38 , wherein the first populated manifest comprises a particular resource that is embedded in an HTML document other than the first HTML document that the client device requested.