System and Methods for Distributed Machine Learning with Multiple Data Sources, Multiple Programming Languages or Frameworks, and Multiple Devices or Infrastructures

Abstract

Methods and systems are presented for consuming different data sources, and deploying artificial intelligence and machine learning programs on different target devices or infrastructures. Many data types can be transformed into machine learning data shards (MLDS) while many machine learning programs written in various programming languages or frameworks are transformed to common operator representations. Operator representations are transformed into execution graphs (EG) for a chosen target device or infrastructure. The MLDS and EG are input to the targeted devices and infrastructures, which then execute the machine learning programs (now transformed to EGs) on the MLDS to produce trained models or predictions with trained models.

1 - 9 . (canceled) 
     
     
         10 . A method of representing data from a plurality of data sources in a consistent format, comprising:
 reading data from a data source of the plurality of data sources; determining a data signature based on the data source;   dividing the data into a plurality of data pieces;   distributing each data piece to a respective data engine machine of a plurality of data engine machines;   selecting, at each data engine machine, one or more filters based on the data signature; transforming, at each data engine machine, a respective data piece of the plurality of data pieces into a machine learning data shard using the one or more filters; and   combining the respective machine learning data shards from each data engine machine into a machine learning data shard database record.   
     
     
         11 . The method of  claim 10 , wherein each machine learning data shard from a respective data engine machine is represented as a vector having a number of dimensions, D. 
     
     
         12 . The method of  claim 11 , wherein the machine learning data shard database record is represented as a matrix having N rows and D columns. 
     
     
         13 . The method of  claim 10 , wherein the data source is a streaming data source and receiving the data includes continually receiving streaming data from the streaming data source. 
     
     
         14 . The method of  claim 10  further comprising:
 reading a second set of data from a second data source of the plurality of data sources; 
 determining a second data signature based on the second data source; 
 transforming a respective second data piece of a plurality of second data pieces into a second machine learning data shard using one or more second filters, and 
 combining the respective second machine learning data shards into a second machine learning data shard database record, wherein the second machine learning data shard database record has the same consistent format as the machine learning data shard database record. 
 
     
     
         15 . The method of  claim 10 , further comprising:
 partitioning the machine learning data shard database record into a plurality of machine learning data shard record pieces, and   distributing the machine learning data shard record pieces to one or more target devices.   
     
     
         16 - 20 . (canceled)