Systems and methods for a time series forecasting transformer network

Abstract

Embodiments described herein provide a Transformer-based neural network architecture that comprises mixture-of-experts time series foundation models to predict different types of time series data. Specifically, given an input multi-variate time series data, a single projection layer may be used to generate patch embeddings for the different time series patterns. The patch embeddings are then passed to a Transformer self-attention layer to compute attention weights, based on which a gating function assigns the patch embeddings into different time series clusters to be further fed to different expert such as feed-forward layers. The feed-forward layers in turn predict a distribution. The output tokens of forecasted time series data are then decoded via the output projection layers from the predicted distribution.

What is claimed is: 
     
         1 . A method of forecasting time series data for a future time period by a neural network based model, the method comprising:
 receiving, via a data interface, time series data collected at a first frequency of time-varying activities corresponding to a first period of time;   splitting an input sequence of the time series data into one or more non-overlapping patches of a pre-defined patch size independent of the first frequency;   encoding, by a first neural network projection layer, the one or more non-overlapping patches into one or more patch embeddings;   generating, by a Transformer neural network layer comprising a set of specialized modules each specializes in a distinct patter of time series data, layer outputs corresponding to the one or more patch embeddings,
 wherein a subset of the set of specialized modules are selectively activated for each token in the one or more patch embeddings based on a respective pattern of the time series data; and 
   generating, by a second neural network projection layer, a predicted distribution of time series data over a second period of the time based on the layer outputs.   
     
     
         2 . The method of  claim 1 , wherein the input sequence comprises a first subsequence corresponding to values of a first variate over the first period of time and a second subsequence concatenated to the first subsequence, corresponding to values of a second variate over the first period of time. 
     
     
         3 . The method of  claim 1 , wherein the Transformer neural network layer comprises a self-attention module that generate attention outputs indicating correlations between tokens of the one or more patch embeddings. 
     
     
         4 . The method of  claim 3 , wherein the Transformer neural network layer comprises a gating module that at least one affinity score between at least one of the set of specialized modules and a token from the one or more patch embeddings from the attention outputs. 
     
     
         5 . The method of  claim 4 , wherein the at least one affinity score is computed as a Softmax operation over a top-K logits of a linear projection applied on the attention outputs. 
     
     
         6 . The method of  claim 4 , wherein the at least one affinity score is computed as a Softmax operation over a top-K Euclidean distances between the attention outputs and cluster centroids corresponding to the set of specialized modules,
 wherein the cluster centroids are computed by performing k-means clustering on the attention outputs using a batch of training time series data.   
     
     
         7 . The method of  claim 4 , wherein the subset of the set of specialized modules are selectively activated for the each token based on the at least one affinity score, and
 wherein at least one specialized module is a feed forward layer.   
     
     
         8 . The method of  claim 4 , wherein the layer outputs are generated by:
 multiplying the at least one affinity score with a module output from at least one selectively activated specialized module; and   aggregating multiplication results over the set of specialized modules.   
     
     
         9 . The method of  claim 1 , wherein the first neural network projection layer encodes times series patches from time series data of different frequencies, wherein the time series data of different frequencies are split into time series patches of a same pre-defined patch size. 
     
     
         10 . The method of  claim 1 , further comprising:
 receiving a training dataset of time series data samples having different frequencies;   dividing each time series data sample into a context window and a prediction window;   encoding, by the first neural network projection layer, the time series data samples having different frequencies;   generating, by the neural network based model comprising the Transformer neural network layer, a predicted training distribution of time series data within the prediction window;   training the neural network based model based on a first loss computed based on the predicted training distribution of time series data and a second loss computed based on token allocation to the set of specialized modules.   
     
     
         11 . A system of forecasting time series data for a future time period by a neural network based model, the system comprising:
 a data interface receiving time series data collected at a first frequency of time-varying activities corresponding to a first period of time;   a memory storing a plurality of processor-executable instructions, the processor-executable instructions being executed by one or more processors to perform operations comprising:
 splitting an input sequence of the time series data into one or more non-overlapping patches of a pre-defined patch size independent of the first frequency; 
 encoding, by a first neural network projection layer, the one or more non-overlapping patches into one or more patch embeddings; 
 generating, by a Transformer neural network layer comprising a set of specialized modules each specializes in a distinct patter of time series data, layer outputs corresponding to the one or more patch embeddings,
 wherein a subset of the set of specialized modules are selectively activated for each token in the one or more patch embeddings based on a respective pattern of the time series data; and 
 
 generating, by a second neural network projection layer, a predicted distribution of time series data over a second period of the time based on the layer outputs. 
   
     
     
         12 . The system of  claim 11 , wherein the input sequence comprises a first subsequence corresponding to values of a first variate over the first period of time and a second subsequence concatenated to the first subsequence, corresponding to values of a second variate over the first period of time. 
     
     
         13 . The system of  claim 11 , wherein the Transformer neural network layer comprises a self-attention module that generate attention outputs indicating correlations between tokens of the one or more patch embeddings. 
     
     
         14 . The system of  claim 13 , wherein the Transformer neural network layer comprises a gating module that at least one affinity score between at least one of the set of specialized modules and a token from the one or more patch embeddings from the attention outputs. 
     
     
         15 . The system of  claim 14 , wherein the at least one affinity score is computed as one of:
 a Softmax operation over a top-K logits of a linear projection applied on the attention outputs; or   a Softmax operation over a top-K Euclidean distances between the attention outputs and cluster centroids corresponding to the set of specialized modules,
 wherein the cluster centroids are computed by performing k-means clustering on the attention outputs using a batch of training time series data. 
   
     
     
         16 . The system of  claim 14 , wherein the subset of the set of specialized modules are selectively activated for the each token based on the at least one affinity score, and
 wherein at least one specialized module is a feed forward layer.   
     
     
         17 . The system of  claim 14 , wherein the layer outputs are generated by:
 multiplying the at least one affinity score with a module output from at least one selectively activated specialized module; and   aggregating multiplication results over the set of specialized modules.   
     
     
         18 . The system of  claim 11 , wherein the first neural network projection layer encodes times series patches from time series data of different frequencies, wherein the time series data of different frequencies are split into time series patches of a same pre-defined patch size. 
     
     
         19 . The system of  claim 11 , wherein the operations further comprise:
 receiving a training dataset of time series data samples having different frequencies;   dividing each time series data sample into a context window and a prediction window;   encoding, by the first neural network projection layer, the time series data samples having different frequencies;   generating, by the neural network based model comprising the Transformer neural network layer, a predicted training distribution of time series data within the prediction window;   training the neural network based model based on a first loss computed based on the predicted training distribution of time series data and a second loss computed based on token allocation to the set of specialized modules.   
     
     
         20 . A non-transitory processor-readable medium storing a plurality of processor-executable instructions for forecasting time series data for a future time period by a neural network based model, the instructions being executed by one or more processors to perform operations comprising:
 receiving, via a data interface, time series data collected at a first frequency of time-varying activities corresponding to a first period of time;   splitting an input sequence of the time series data into one or more non-overlapping patches of a pre-defined patch size independent of the first frequency;   encoding, by a first neural network projection layer, the one or more non-overlapping patches into one or more patch embeddings;   generating, by a Transformer neural network layer comprising a set of specialized modules each specializes in a distinct patter of time series data, layer outputs corresponding to the one or more patch embeddings,
 wherein a subset of the set of specialized modules are selectively activated for each token in the one or more patch embeddings based on a respective pattern of the time series data; and 
   generating, by a second neural network projection layer, a predicted distribution of time series data over a second period of the time based on the layer outputs.