Managed tables for data lakes

Abstract

Aspects of the disclosure are directed to merging data lake openness with scalable metadata for managed tables in a cloud database platform, allowing for atomicity, consistency, isolation, and durability (ACID) transactions, performant data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and/or automatic storage optimization. Table data is stored in various open-source file formats in cloud storage while physical metadata of the table data is stored in a scalable metadata storage system.

The invention claimed is: 
     
         1 . A method for processing queries, comprising:
 receiving, by one or more processors, a first request from a query engine to read one or more data files;   determining, by the one or more processors, based on instructions in the first request, whether to read the one or more data files through a storage application programming interface (API) or directly from read-optimized cloud storage;   selecting, by the one or more processors, based on the instructions indicating to read the one or more data files through the storage API, to read the one or more data files through the storage API;   retrieving, by the one or more processors, the one or more data files from either the read-optimized cloud storage or a write-optimized buffer by determining whether the one or more data files are stored in the read-optimized cloud storage or the write-optimized buffer using column-level metadata stored in an appendable distributed file system separate from the read-optimized cloud storage;   reading, by the one or more processors, the one or more data files in response to the first request from the query engine;   receiving, by the one or more processors, a second request to read one or more additional data files;   selecting, by the one or more processors, to directly read the one or more additional data files from the read-optimized cloud storage based on instructions in the second request;   retrieving, by the one or more processors, the one or more additional data files from the read-optimized cloud storage using a metadata snapshot of the column-level metadata; and   reading, by the one or more processors, the one or more additional data files in response to the second request.   
     
     
         2 . The method of  claim 1 , further comprising exporting, by the one or more processors, the metadata stored in the appendable distributed file system to the read-optimized cloud storage in one or more formats compatible with the query engine. 
     
     
         3 . The method of  claim 2 , wherein the exporting is automatically triggered in response to one or more additions to a table transaction log stored in the appendable distributed file system. 
     
     
         4 . The method of  claim 3 , wherein the one or more additions to the table transaction log are in response to requests to write one or more data files to the read-optimized cloud storage. 
     
     
         5 . The method of  claim 1 , wherein the column-level metadata is stored in a table transaction log in the appendable distributed file system. 
     
     
         6 . The method of  claim 5 , wherein the table transaction log is periodically compacted into a read-optimized format compatible with the query engine. 
     
     
         7 . The method of  claim 1 , wherein the query engine comprises a data lake query engine or a data warehouse query engine. 
     
     
         8 . The method of  claim 1 , further comprising performing, by the one or more processors, one or more maintenance tasks based on the column-level metadata in the distributed file system. 
     
     
         9 . The method of  claim 8 , wherein the one or more maintenance tasks comprise at least one of garbage collection, data file merging, data file splitting, or data file reclustering. 
     
     
         10 . A system comprising:
 one or more processors; and   one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for processing queries, the operations comprising:
 receiving a first request from a query engine to read one or more data files; 
 determining, based on instructions in the first request, whether to read the one or more data files through a storage application programming interface (API) or directly from read-optimized cloud storage; 
 selecting, based on the instructions indicating to read the one or more data files through the storage API, to read the one or more data files through the storage API; 
 retrieving the one or more data files from either the read-optimized cloud storage or a write-optimized buffer by determining whether the one or more data files are stored in the read-optimized cloud storage or the write-optimized buffer using column-level metadata stored in an appendable distributed file system separate from the read-optimized cloud storage; 
 reading the one or more data files in response to the request from the query engine; 
 receiving a second request to read one or more additional data files; 
 selecting to directly read the one or more additional data files from the read-optimized cloud storage based on instructions in the second request; 
 retrieving the one or more additional data files from the read-optimized cloud storage using a metadata snapshot of the column-level metadata; and 
 reading the one or more additional data files in response to the second request. 
   
     
     
         11 . The system of  claim 10 , wherein the operations further comprise exporting the column-level metadata stored in the appendable distributed file system to the read-optimized cloud storage in one or more formats compatible with the query engine. 
     
     
         12 . The system of  claim 11 , wherein the exporting is automatically triggered in response to one or more additions to a table transaction log stored in the appendable distributed file system. 
     
     
         13 . The system of  claim 12 , wherein the one or more additions to the table transaction log are in response to requests to write one or more data files to the read-optimized cloud storage. 
     
     
         14 . The system of  claim 10 , wherein the column-level metadata is stored in a table transaction log in the appendable distributed file system. 
     
     
         15 . The system of  claim 14 , wherein the table transaction log is periodically compacted into a read-optimized format compatible with the query engine. 
     
     
         16 . The system of  claim 10 , wherein the operations further comprise performing one or more maintenance tasks based on the column-level metadata in the distributed file system. 
     
     
         17 . A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for processing queries, the operations comprising:
 receiving a first request from a query engine to read one or more data files;   determining, based on instructions in the first request, whether to read the one or more data files through a storage application programming interface (API) or directly from read-optimized cloud storage;   selecting, based on the instructions indicating to read the one or more data files through the storage API, to read the one or more data files through the API;   retrieving the one or more data files from either the read-optimized cloud storage or a write-optimized buffer by determining whether the one or more data files are stored in the read-optimized cloud storage or the write-optimized buffer using column-level metadata stored in an appendable distributed file system separate from the read-optimized cloud storage;   reading the one or more data files in response to the request from the query engine;   receiving a second request to read one or more additional data files;   selecting to directly read the one or more additional data files from the read-optimized cloud storage based on instructions in the second request;   retrieving the one or more additional data files from the read-optimized cloud storage using a metadata snapshot of the column-level metadata; and   reading the one or more additional data files in response to the second request.