Partitioning, processing, and protecting data

Abstract

A technique for managing data objects in a storage cluster includes splitting a data object into multiple portions at boundaries within the data object. The technique further includes transforming the portions of the data object into segments that provide individually processable units, and distributing the segments among multiple computing nodes of the storage cluster for storage therein.

What is claimed is: 
     
         1 . A method of managing data objects, comprising:
 splitting a data object into multiple portions at boundaries within the data object, the boundaries providing separators between processable units of the data object in accordance with a type of the data object, said splitting including scanning, by a gateway computing device, the data object to identify the boundaries;   transforming the portions into segments that provide individually processable units of a same type as the type of the data object; and   distributing the segments among multiple computing nodes of a storage cluster for protected storage therein.   
     
     
         2 . The method of  claim 1 , wherein transforming the portions into segments includes adding or modifying metadata of a set of the portions. 
     
     
         3 . The method of  claim 2 , wherein adding or modifying the metadata includes identifying header metadata of the data object and adding the header metadata or a modified version thereof to each of the set of portions. 
     
     
         4 . The method of  claim 2 , wherein the data object is a CSV (comma-separated values) file having a header that identifies column names, and wherein adding or modifying the metadata includes identifying the header of the CSV file and adding the header to each of the set of portions. 
     
     
         5 . The method of  claim 2 , wherein adding or modifying the metadata includes identifying footer metadata in one of the portions and adding the footer metadata or a modified version thereof to each of the set of portions. 
     
     
         6 . The method of  claim 2 , wherein the data object is a Parquet file that includes multiple row groups and a footer, wherein splitting the data object includes providing a respective set of row groups in each of the set of portions, and wherein adding or modifying the metadata includes adding the footer or a modified version thereof to each of the set of portions. 
     
     
         7 . The method of  claim 6 , wherein adding or modifying the metadata further includes adding a header to each of the set of segments, the header including a code that designates the Parquet file, and wherein adding the footer or modified version thereof to each of the set of portions includes (i) adding footer metadata that describes the set of row groups in the respective portion but not row groups in other portions, (ii) adding a length of the footer metadata, and (iii) adding the code that designates the Parquet file. 
     
     
         8 . The method of  claim 1 , wherein the data object is a video file or stream that includes multiple frames, and wherein splitting the data object includes (i) identifying an i-frame in the video file or stream (ii) defining an end of a current segment as a frame located one frame prior to the identified i-frame, and (iii) defining a start of a next segment as the identified i-frame. 
     
     
         9 . The method of  claim 1 , further comprising, prior to splitting the data object, reading a set of regions of the data object and identifying the type of the data object based on the set of regions, wherein splitting the data object includes searching the data object for a separator used to separate processable units of data objects of the identified type. 
     
     
         10 . The method of  claim 1 , further comprising performing a distributed processing task by the storage cluster, the distributed processing task independently executing by multiple respective computing nodes of the storage cluster on respective segments or sets of segments stored therein. 
     
     
         11 . The method of  claim 10 , further comprising storing object metadata that associates segments of the data object with locations in the storage cluster where the respective segments are stored. 
     
     
         12 . The method of  claim 11 , wherein storing the object metadata includes storing an association between byte ranges of the data object and segments of the data object. 
     
     
         13 . The method of  claim 12 , wherein performing the distributed processing task includes identifying, based on accessing the object metadata, a subset of the segments that contain data required by the processing task, and directing the processing task to computing nodes that store the subset of segments but not to computing nodes that do not store segments that are part of the subset of segments. 
     
     
         14 . The method of  claim 11 , further comprising receiving, by a gateway, output of the processing task from a plurality of the computing nodes, and combining the output to produce output data. 
     
     
         15 . The method of  claim 14 , wherein the output received from the plurality of computing nodes is received in an order, and wherein combining the output includes providing the output in the order received. 
     
     
         16 . The method of  claim 15 , wherein the output received by the gateway from the plurality of computing nodes arrives in multiple RDMA (remote direct memory access) transmissions. 
     
     
         17 . The method of  claim 11 , wherein the data object includes data records, and wherein the processing task defines an aggregate query that returns less than 1 kB of output data as results of the aggregate query. 
     
     
         18 . The method of  claim 1 , wherein transforming the portions into segments includes copying a region of data from a first portion adjacent to a boundary between a first portion and a second portion to the second portion. 
     
     
         19 . The method of  claim 1 , wherein transforming the portions into segments includes copying an initial region a first portion to respective initial regions of each of a set of other portions. 
     
     
         20 . The method of  claim 19 , further comprising:
 receiving a processing request that specifies processing of the data object without use of a header; and   in response to the processing request, processing the initial region of the first portion but ignoring the initial region of each of the set of other portions.   
     
     
         21 . The method of  claim 1  wherein, when splitting the data object into multiple portions, the method further comprises identifying a feature in a portion of the data object that enables faster processing and updating metadata associated with the portion to indicate that the portion includes the identified feature. 
     
     
         22 . A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of managing data objects, the method comprising:
 splitting a data object into multiple portions at boundaries within the data object, the boundaries providing separators between processable units of the data object in accordance with a type of the data object, said splitting including scanning, by computerized apparatus, the data object to identify the boundaries;   transforming the portions into segments that provide individually processable units of a same type as the type of the data object; and   distributing the segments among multiple computing nodes of a storage cluster for protected storage therein.   
     
     
         23 . A method of managing data objects, comprising:
 splitting a data object into multiple segments, said splitting including scanning, by a gateway computing device, the data object to identify the boundaries;   distributing the segments among multiple computing nodes of a storage cluster for protected storage therein; and   performing a distributed processing task by the storage cluster, the distributed processing task executing independently by multiple respective computing nodes of the storage cluster on respective segments or sets of segments stored therein.