Checkpointing in Distributed Streaming Platform for Real-Time Applications

Abstract

Software receives a data stream for an application running on a distributed streaming platform over a networked cluster of servers. The software converts the data into a plurality of data tuples structured according to a schema. The software repeatedly emits a plurality of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window. The streaming window is a sequential sequence of tuples that is associated with a recovery policy. Then the software emits a checkpointing tuple after a plurality of streaming windows. The checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance. Each of the operations is executed by one or more processors in real time or near real time rather than offline.

What is claimed is: 
     
         1 . A method, comprising:
 receiving a data stream for an application;   converting the data stream into a plurality of data tuples structured according to a schema;   emitting the plurality of data tuples as a plurality of streaming windows, each of the plurality of streaming windows being separated from other streaming windows by a leading control tuple and a trailing control tuple, both the leading control tuple and the trailing control tuple being associated with a same ordinal identifier specific to each respective streaming window; and   emitting a checkpointing tuple, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.   
     
     
         2 . The method of  claim 1 , wherein checkpointing includes serializing the instance to persistent storage. 
     
     
         3 . The method of  claim 2 , wherein the persistent storage is Hadoop Distributed File System (HDFS) and wherein the serialization uses Kryo. 
     
     
         4 . The method of  claim 1 , wherein the data stream is accessed through a web Application Programming Interface (“API”). 
     
     
         5 . The method of  claim 1 , wherein the data tuples are displayed in a graphical user interface (GUI) following removal of leading control tuples and/or trailing control tuples. 
     
     
         6 . The method of  claim 5 , wherein the GUI is dashboard for decision support. 
     
     
         7 . The method of  claim 1 , wherein each of the plurality of streaming windows is associated with a recovery policy that is one of at least once, at most once, or exactly once. 
     
     
         8 . A non-transitory computer-readable storage medium storing a computer program, the computer-readable medium comprising:
 program instructions for receiving a data stream for an application;   program instructions for converting the data stream into a plurality of data tuples structured according to a schema;   program instructions for emitting the plurality of data tuples as a plurality of streaming windows, each of the plurality of streaming windows being separated from other streaming windows by a leading control tuple and a trailing control tuple, both the leading control tuple and the trailing control tuple being associated with a same ordinal identifier specific to each respective streaming window; and   program instructions for emitting a checkpointing tuple, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed in real time or near real time rather than offline.   
     
     
         9 . The non-transitory computer-readable storage medium of  claim 8 , wherein checkpointing includes serializing the instance to persistent storage. 
     
     
         10 . The non-transitory computer-readable storage medium of  claim 9 , wherein the persistent storage is Hadoop Distributed File System (HDFS) and wherein the serialization uses Kryo. 
     
     
         11 . The non-transitory computer-readable storage medium of  claim 8 , wherein the data stream is accessed through a web Application Programming Interface (“API”). 
     
     
         12 . The non-transitory computer-readable storage medium of  claim 8 , wherein the data tuples are displayed in a graphical user interface (GUI) following removal of leading control tuples and/or trailing control tuples. 
     
     
         13 . The non-transitory computer-readable storage medium of  claim 12 , wherein the GUI is a dashboard for decision support. 
     
     
         14 . The non-transitory computer-readable storage medium of  claim 8 , wherein each of the plurality of streaming windows is associated with a recovery policy that is one of at least once, at most once, or exactly once. 
     
     
         15 . A method, comprising:
 receiving a data stream for an application, the application using one or more stream modes from the group of stream modes consisting of in-node and in-rack;   converting the data stream into a plurality of data tuples structured according to a schema;   emitting the plurality of data tuples as a plurality of streaming windows, which is separated from other streaming windows by a leading control tuple and a trailing control tuple associated with a same ordinal identifier specific to each respective streaming window; and   emitting a checkpointing tuple, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.   
     
     
         16 . The method of  claim 15 , wherein checkpointing includes serializing the instance to persistent storage. 
     
     
         17 . The method of  claim 16 , wherein the persistent storage is Hadoop Distributed File System (HDFS) and wherein the serialization uses Kryo. 
     
     
         18 . The method of  claim 15 , wherein the data stream is accessed through a web Application Programming Interface (“API”). 
     
     
         19 . The method of  claim 15 , wherein the data tuples are displayed in a graphical user interface (GUI) following removal of leading control tuples and/or trailing control tuples. 
     
     
         20 . The method of  claim 19 , wherein the GUI is a dashboard for decision support.