Parallel, block-based data encoding and decoding using multiple computational units

Abstract

Massively parallel, block-based encoding and decoding technology that includes an encoded block format uses a plurality of processing cores to perform block-based encoding and decoding operations. The encoded block format includes a header and a payload. The encoded block format's headers represent unique single-Byte and multi-Byte event parameters that occur in the original data block from which each encoded block was generated. The encoded block format's payloads represent a sequence of single-Byte and multi-Byte events using tokens that associate each event with its corresponding parameter(s). Metadata can include an array of encoded block sizes that support random access.

What is claimed is: 
     
       1. An encoding method that encodes an input dataset, comprising:
 reading a plurality of data blocks of block size N, where N is a number of bytes, of the input data set; 
 using a plurality of cores of a multicore processor, encoding data blocks of the input data set into a plurality of encoded data blocks having variant block sizes, at least one of the encoded data blocks having a block size less than N, including concurrently encoding
 a first data block of the plurality of data blocks of the input dataset into a first losslessly encoded data block using a first core in the plurality of cores, and 
 a second data block of the plurality of data blocks of the input dataset into a second losslessly encoded data block using a second core in the plurality of cores; and 
 storing or transmitting the plurality of encoded data blocks; 
 
 wherein at least one of the losslessly encoded data blocks contains one or more headers and one or more payloads, and:
 each header represents a list of unique single-Byte parameters and a list of unique multi-byte parameters having an encoding format different from that of the list of single-byte parameters; and 
 each payload includes tokens that identify a sequence of single-Byte and multi-Byte parameters whose contents are identified in at least one header. 
 
 
     
     
       2. The method of  claim 1 , including generating metadata that includes the sizes of encoded data blocks of the plurality of encoded data blocks. 
     
     
       3. A computer system that encodes an input dataset, comprising:
 a multicore processor, memory and a communication interface, including instructions executable by the processor to 
 read a plurality of data blocks of block size N, where N is a number of bytes, of the input data set; 
 use a plurality of cores of the multicore processor, to encode data blocks of the input data set into a plurality of encoded data blocks having variant block sizes, at least one of the encoded data blocks having a block size less than N, including concurrently encoding
 a first data block of the plurality of data blocks of the input dataset into a first losslessly encoded data block using a first core in the plurality of cores, and 
 a second data block of the plurality of data blocks of the input dataset into a second losslessly encoded data block using a second core in the plurality of cores; and 
 store or transmit the plurality of encoded data blocks; 
 
 wherein at least one of the losslessly encoded data blocks contains one or more headers and one or more payloads, and:
 each header represents a list of unique single-Byte parameters and a list of unique multi-byte parameters having an encoding format different from that of the list of single-byte parameters; and 
 each payload includes tokens that identify a sequence of single-Byte and multi-Byte parameters whose contents are identified in at least one header. 
 
 
     
     
       4. The system of  claim 3 , further including instructions executable by the processor to generate metadata that includes the sizes of encoded data blocks of the plurality of encoded data blocks. 
     
     
       5. A computer program product comprising a non-transitory computer readable medium for a method that encodes an input dataset, comprising:
 instructions executable by a multicore processor stored in the computer readable medium to read a plurality of data blocks of block size N, where N is a number of bytes, of the input data set; 
 use a plurality of cores of the multicore processor, to encode data blocks of the input data set into a plurality of encoded data blocks having variant block sizes, at least one of the encoded data blocks having a block size less than N, including concurrently encoding
 a first data block of the plurality of data blocks of the input dataset into a first losslessly encoded data block using a first core in the plurality of cores, and 
 a second data block of the plurality of data blocks of the input dataset into a second losslessly encoded data block using a second core in the plurality of cores; and 
 store or transmit the plurality of encoded data blocks; 
 
 wherein at least one of the losslessly encoded data blocks contains one or more headers and one or more payloads, and:
 each header represents a list of unique single-Byte parameters and a list of unique multi-byte parameters having an encoding format different from that of the list of single-byte parameters; and 
 each payload includes tokens that identify a sequence of single-Byte and multi-Byte parameters whose contents are identified in at least one header. 
 
 
     
     
       6. The computer program product of  claim 5 , further including instructions executable by the processor to generate metadata that includes the sizes of encoded data blocks of the plurality of encoded data blocks.