Bidirectional data transfer optimization and content control for networks

Abstract

A method of transferring data objects over a network comprises intercepting a network transfer message with a passing object, creating a unique identifier for the object using a predetermined function, the same function having been used to provide identifiers for objects stored at network nodes, removing the object and sending on the network transfer message with the unique identifier in place of the object. Then, at the recipient end it is possible to obtain the unique identifier and use it as a key to search for a corresponding object in the local nodes. The search starts with a node closest to the recipient and steadily spreads outwards. The object when found is reattached for the benefit of the recipient and network bandwidth has been saved by the avoidance of redundant transfer since the object is brought to the recipient from the node which is the closest to him.

1 . A method comprising:
 receiving, with a first intermediate device, data output by a source device for a plurality of destination devices, wherein the first intermediate device is positioned between the source device and a second intermediate device, and wherein the second intermediate device is positioned between the first intermediate device and a plurality of destination devices;   forwarding, with the first intermediate device, the data received from the source device to the second intermediate device;   comparing, at the first intermediate device, received unique identifiers of the data with previously received unique identifiers to detect repeated data from the source device;   at the first intermediate device, sending a message from the first intermediate device to the second intermediate device directing the second intermediate device to output the detected data corresponding to the repeated unique identifier; and   maintaining, with the first intermediate device, a directory that includes the unique identifier cached in the second intermediate device.   
     
     
         2 . The method of  claim 1 , wherein the data comprises movie data. 
     
     
         3 . An apparatus for transferring data objects over a network, comprising:
 at a sending locality on said network:   an interception unit configured for intercepting and caching passing objects on the way to respective recipients; and   an identification unit, associated with said interception unit, configured for generating a unique identifier for said passing object, said unique identifier being generated from said passing object, said interception unit being further configured to replace said passing object with said unique identifier to pass over said network and further to use said unique identifier to label said object in said cache; and   at a receiving locality on said network:   a search unit configured for searching for a corresponding object in at least one node dictionary using said unique identifier to identify said corresponding object, said identifying comprising matching, thereby to supply said corresponding object to a respective recipient from a relatively nearby node dictionary without carrying out redundant network transfer.   
     
     
         4 . The apparatus according to  claim 3 , wherein said search unit is configured to provide:
 a) local searching,   b) if said local searching is unsuccessful, then searching at a closest network node, and   c) if said local searching is unsuccessful then continuing to search at successively distant network nodes.   
     
     
         5 . The apparatus according to  claim 4 , wherein said search unit is further configured to modify said searching in accordance with load balancing considerations, and wherein said load balancing considerations are obtained from a central load balancing control node. 
     
     
         6 . The apparatus according to  claim 3 , wherein said object is segmented prior to storage such that each segment is searchable and transportable on its own, and wherein, upon modification of said segmented object, each segment is treated independently, the apparatus further configured to include a change in a segment as a delta with the original segment. 
     
     
         7 . The apparatus according to  claim 3 , wherein said identification function is substantially injective, and said identification unit is configured to use a hashing function as said substantially injective identification function. 
     
     
         8 . The apparatus according to  claim 3 , wherein said identification unit is configured to use a fully injective identification function. 
     
     
         9 . The apparatus according to  claim 3 , wherein copies of respective passing objects are stored at a plurality of nodes of said network, each in association with said respective unique identifier, such that said copy is retrievable from each node using said respective unique identifier, and wherein said intercepting unit is configured to segment passing objects prior to caching, the apparatus further comprising a load balancing unit associated with said search unit, such that said load balancing unit is able to retrieve different segments of said object from different nodes of said network, thereby to provide relatively balanced network utilization. 
     
     
         10 . The apparatus according to  claim 4 , wherein distance of nodes is determined by networking parameters, and wherein said networking parameters comprise at least one of a group including: physical distance, bandwidth, roundtrip time, latency, number of routing hops and economical cost. 
     
     
         11 . The apparatus according to  claim 3 , further comprising a content control unit configured to store unique identifiers relating to data objects whose distribution it is intended to control, together with rules for said control, such that said control unit can be searched using a given unique identifier to retrieve a corresponding rule to be applied to further distribution of said object, wherein said rule is any one of a group comprising transfer blockage, object altering, and object replacement. 
     
     
         12 . The apparatus according to  claim 3 , wherein said sending locality further comprises one member of the group consisting of:
 a receiving locality identification unit configured to identify a system component at said receiving locality by sending an ICMP message to said recipient, and allowing a response to said ICMP message to be recognized and altered by said system component, thereby to identify said system component as a component at said receiving locality; and   a receiving locality identification unit configured to identify a closest system component to said recipient by sending an ICMP message to said recipient, and allowing a response to said ICMP message to be recognized and altered by a first system component that said response passes, thereby to identify said system component as a closest system component to said recipient.   
     
     
         13 . A method of transferring data objects over a network, comprising:
 at a sending locality on said network:   intercepting and caching passing objects on the way to respective recipients;   generating a unique identifier for said passing object, said unique identifier being generated from said passing object,   replacing said passing object with said unique identifier to pass over said network, and   using said unique identifier to label said object in said cache; and   at a receiving locality on said network:   searching for a corresponding object in at least one network node using said unique identifier to identify said corresponding object, said identifying comprising matching, thereby to supply said corresponding object to a respective recipient from a relatively nearby network node without carrying out redundant network transfer.   
     
     
         14 . A node of a communication network configured to minimize redundant object transfer over said network, the node comprising:
 an interception unit configured for intercepting data objects within network communications carrying said data objects;   a node dictionary configured for caching said data objects;   a labeling unit associated with said node dictionary, configured for   a) generating a unique identifier of each data object stored in said node dictionary, said unique identifier being generated from said data object, thereby providing each said data object with an electronic signature;   b) storing a first copy of said identifier in association with said object; and   c) replacing said data object with a second copy of said identifier in said network communication to continue over said network; and   a retrieval unit associated with said node dictionary, configured for   d) receiving a query for an object identified by an electronic signature from said network,   e) comparing said electronic signature with stored signatures in said node dictionary, said comparing comprising matching,   f) if a match is found then retrieving an object corresponding to said matched signature, and   g) if a match is not found then sending said query on to a neighboring node.   
     
     
         15 . A caching dictionary, associated with a node of a communication network, said caching dictionary being configured to minimize redundant object transfer over said network, the caching dictionary being capable of caching said data objects and further being associated with:
 an interception unit configured for intercepting data objects within network communications carrying said data objects;   a labeling unit, configured for   a) generating a unique identifier of each data object cached in said caching dictionary, said unique identifier being generated from said data object, thereby providing each said data object with an electronic signature;   b) storing a first copy of said identifier in association with said object; and   c) replacing said data object with a second copy of said identifier in said network communication to continue over said network; and   a retrieval unit, for   d) receiving a query for an object identified by an electronic signature from said network,   e) comparing said electronic signature with stored signatures in said caching dictionary, said comparing comprising matching,   f) if a match is found then retrieving an object corresponding to said matched signature, and   g) if a match is not found then sending said query on to a neighboring node.   
     
     
         16 . A method of transferring data objects over a network comprising a plurality of nodes, comprising:
 intercepting and caching any passing object at a node located relatively close to a sender,   generating a unique identifier of said object, said unique identifier being generated from said passing object;   storing said unique identifier in association with said object in said relatively close node;   sending said unique identifier towards an intended recipient of said object;   at a node relatively close to said intended recipient, using said sent unique identifier to compare with identifiers of objects cached therein to find an exact match;   if a match is found then retrieving a corresponding cached object and sending to said intended recipient;   if no match is found then repeating said match at a further node, until a match is found.   
     
     
         17 . An apparatus for remotely identifying over a network a system component closest to a given location, the apparatus comprising:
 a message generator configured for generating a response request identifiable to respective system components, and   a sending unit configured for sending said response request to said given location such that said response request is received by said given location and a response is sent by return, said response being identifiable to said system components such that a first system component receiving said response identifies itself to said apparatus, each system component being provided with a unique identifier, each unique identifier being generated from said respective system component, and wherein said identifier is used in said identification to find an exact match.   
     
     
         18 . A method for remotely identifying over a network a system component closest to a given location, the method comprising:
 generating a response request identifiable to respective system components,   sending said response request to said given location such that said response request is received by said given location and a response is sent by return, said response being identifiable to said system components such that a first system component receiving said response identifies itself, each system component being provided with a unique identifier, each unique identifier being generated from said respective system component, and wherein said identifier is used in said identification to find exact matches.   
     
     
         19 . An apparatus for finding a closest neighboring destination node to a receiving locality, wherein a sending locality comprises a receiving locality identification unit configured to identify a system component at said receiving locality by sending an ICMP message to said recipient, and allowing a response to said ICMP message to be recognized and altered by said system component, thereby to identify said system component as a component at said receiving locality, each system component being provided with a unique identifier for exact matching, each unique identifier being generated from said respective system component, that identifier being used in said alteration. 
     
     
         20 . An apparatus for finding a closest receiving locality to a sending locality, wherein said sending locality further comprises a receiving locality identification unit configured to identify a closest system component to said recipient by sending an ICMP message to said recipient, and allowing a response to said ICMP message to be recognized and altered by a first system component that said response passes, thereby to identify said system component as a closest system component to said recipient, each system component being provided with a unique identifier for exact matching, each unique identifier being generated from said respective system component, that identifier being used in said alteration. 
     
     
         21 . A method for finding a closest neighboring destination node to a receiving locality, the method comprising:
 identifying a system component at said receiving locality by sending an ICMP message to said recipient,   recognizing a response to said ICMP message, and   altering a response to said ICMP message, thereby to identify said system component as a component at said receiving locality, each system component being provided with a unique identifier for exact matching, each unique identifier being generated from said respective system component, that identifier being used in said alteration.   
     
     
         22 . A method for finding a closest receiving locality to a sending locality, the method comprising:
 identifying a closest system component to said recipient by sending an ICMP message to said recipient, and   allowing a response to said ICMP message to be recognized and altered by a first system component that said response passes, thereby to identify said system component as a closest system component to said recipient, each system component being provided with a unique identifier for exact matching, each unique identifier being generated from said respective system component, that identifier being used in said alteration.