Scaling a rest cluster using feature based steering

Abstract

Illustrative embodiments process large amounts of unstructured data by producing, from the unstructured data, a plurality of work items. Each work item includes an item of unstructured data, and corresponding work item metadata. The work items are stored in a work item queue, where they may be claimed by, and processed by, a consumer node from a set of consumer nodes.

What is claimed is: 
     
         1 . A computer-implemented system for distributing data to processing nodes, the system comprising:
 a set of producer nodes, each such producer node configured to:
 receive a plurality of data items, the plurality of data items comprising a set of data items, and a plurality of operation tags, the plurality of operation tags comprising a set of operation tags, each operation tag of the set of operation tags specifying an operation selected from a set of operations, each operation tag uniquely associated with a corresponding data item from the set of data items; 
 create a plurality of work items, each work item comprising a data item from the set of data items bound to the operation tag uniquely associated with said data item; 
 populate a work item queue with the set of work items; and 
 expose the work item queue to a plurality of consumer nodes, each consumer node of the plurality of consumer nodes configured to selectively claim one or more work items from the work item queue. 
   
     
     
         2 . The computer-implemented system of  claim 1 , wherein each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations. 
     
     
         3 . The computer-implemented system of  claim 1 , wherein:
 each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations; and   each consumer node of the plurality of consumer nodes is configured to assess work items from the work item queue to determine, for each such work item, whether said consumer node is specialized to perform the operation specified by the operation tag of the work item.   
     
     
         4 . The computer-implemented system of  claim 1 , wherein: 
 each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations; and   each consumer node of the plurality of consumer nodes is configured to:
 (a) assess work items from the work item queue to ascertain, for each such work item, that said consumer node is specialized to perform the operation specified by the operation tag of the work item, each such work item being a compatible work item, and 
 (b) to claim the compatible work item for processing by the consumer node, to the exclusion of other consumer nodes from the plurality of consumer nodes, said compatible work item being a claimed work item. 
   
     
     
         5 . The computer-implemented system of  claim 4 , wherein:
 each work item comprises a status tag indicating whether such work item has been claimed by a consumer node from the plurality of consumer nodes, and   each consumer node is configured to update the status tag of each claimed work item claimed by said consumer node, so that the status tag of said work item indicates that said work item has been claimed by a consumer node from the plurality of consumer nodes.   
     
     
         6 . The computer-implemented system of  claim 1 , wherein the producer node is further configured to include, in each work item, a status tag, said status tag editable by a consumer node to indicate that said consumer node has claimed the work item. 
     
     
         7 . The computer-implemented system of  claim 1 , wherein the plurality of data items comprise plurality of unstructured data items. 
     
     
         8 . A method of distributing data to processing nodes, the method comprising:
 receiving, at set of produce nodes, a plurality of data items, the plurality of data items comprising a set of data items, and a plurality of operation tags, the plurality of operation tags comprising a set of operation tags, each operation tag of the set of operation tags specifying an operation selected from a set of operations, each operation tag uniquely associated with a corresponding data item from the set of data items;   creating a plurality of work items, each work item comprising a data item from the set of data items bound to the operation tag uniquely associated with said data item;   populating a work item queue with the set of work items; and   exposing the work item queue to a plurality of consumer nodes, each consumer node of the plurality of consumer nodes configured to selectively claim one or more work items from the work item queue.   
     
     
         9 . The method of  claim 8 , wherein each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations. 
     
     
         10 . The method of  claim 8 , wherein:
 each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations; and   each consumer node of the plurality of consumer nodes is configured to assess work items from the work item queue to determine, for each such work item, whether said consumer node is specialized to perform the operation specified by the operation tag of the work item.   
     
     
         11 . The method of  claim 8 , wherein:
 each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations; and   each consumer node of the plurality of consumer nodes is configured to:
 (a) assess work items from the work item queue to ascertain, for each such work item, that said consumer node is specialized to perform the operation specified by the operation tag of the work item, each such work item being a compatible work item, and 
 (b) to claim the compatible work item for processing by the consumer node, to the exclusion of other consumer nodes from the plurality of consumer nodes, said compatible work item being a claimed work item. 
   
     
     
         12 . The method of  claim 11 , wherein:
 each work item comprises a status tag indicating whether such work item has been claimed by a consumer node from the plurality of consumer nodes, and   each consumer node is configured to update the status tag of each claimed work item claimed by said consumer node, so that the status tag of said work item indicates that said work item has been claimed by a consumer node from the plurality of consumer nodes.   
     
     
         13 . The method of  claim 8 , wherein the producer node is further configured to include, in each work item, a status tag, said status tag editable by a consumer node to indicate that said consumer node has claimed the work item. 
     
     
         14 . The method of  claim 8 , wherein the plurality of data items comprise plurality of unstructured data items. 
     
     
         15 . A non-transitory computer-readable medium storing computer-executable code thereon, the code comprising:
 code for receiving, as a set of producer nodes, a plurality of data items, the plurality of data items comprising a set of data items, and a plurality of operation tags, the plurality of operation tags comprising a set of operation tags, each operation tag of the set of operation tags specifying an operation selected from a set of operations, each operation tag uniquely associated with a corresponding data item from the set of data items;   code for creating a plurality of work items, each work item comprising a data item from the set of data items bound to the operation tag uniquely associated with said data item;   code for populating a work item queue with the set of work items; and   code for exposing the work item queue to a plurality of consumer nodes, each consumer node of the plurality of consumer nodes configured to selectively claim one or more work items from the work item queue.   
     
     
         16 . The non-transitory computer-readable medium of  claim 15 , wherein each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations. 
     
     
         17 . The non-transitory computer-readable medium of  claim 15 , wherein:
 each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations; and   each consumer node of the plurality of consumer nodes is configured to assess work items from the work item queue to determine, for each such work item, whether said consumer node is specialized to perform the operation specified by the operation tag of the work item.   
     
     
         18 . The non-transitory computer-readable medium of  claim 15 , wherein:
 each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations; and   each consumer node of the plurality of consumer nodes is configured to:
 (a) assess work items from the work item queue to ascertain, for each such work item, that said consumer node is specialized to perform the operation specified by the operation tag of the work item, each such work item being a compatible work item, and 
 (b) to claim the compatible work item for processing by the consumer node, to the exclusion of other consumer nodes from the plurality of consumer nodes, said compatible work item being a claimed work item. 
   
     
     
         19 . The non-transitory computer-readable medium of  claim 18 , wherein:
 each work item comprises a status tag indicating whether such work item has been claimed by a consumer node from the plurality of consumer nodes, and   each consumer node is configured to update the status tag of each claimed work item claimed by said consumer node, so that the status tag of said work item indicates that said work item has been claimed by a consumer node from the plurality of consumer nodes.   
     
     
         20 . The non-transitory computer-readable medium of  claim 15 , wherein the producer node is further configured to include, in each work item, a status tag, said status tag editable by a consumer node to indicate that said consumer node has claimed the work item.