Wide and deep machine learning models

Abstract

A system includes one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the computers to implement a combined machine learning model for processing an input including multiple features to generate a predicted output for the machine learning input. The combined model includes: a deep machine learning model configured to process the features to generate a deep model output; a wide machine learning model configured to process the features to generate a wide model output; and a combining layer configured to process the deep model output generated by the deep machine learning model and the wide model output generated by the wide machine learning model to generate the predicted output, in which the deep model and the wide model have been trained jointly on training data to generate the deep model output and the wide model output.

What is claimed is: 
     
         1 .- 20 . (canceled) 
     
     
         21 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement a combined machine learning model for processing a machine learning input comprising a plurality of features to generate a predicted output for the machine learning input, the combined machine learning model comprising:
 a deep machine learning model configured to process a subset of the plurality of features comprising continuous features to generate a deep model intermediate predicted output;   a wide machine learning model configured to process binary features of the plurality of features to generate a wide model intermediate predicted output; and   a combining layer configured to process the deep model intermediate predicted output generated by the deep machine learning model and the wide model intermediate predicted output generated by the wide machine learning model to generate the predicted output,   wherein the deep machine learning model and the wide machine learning model have been trained jointly on training data to generate the deep model intermediate predicted output and the wide model intermediate predicted output.   
     
     
         22 . The system of  claim 21 , wherein the binary features are one-hot encoded representations of sparse features. 
     
     
         23 . The system of  claim 21 , wherein the binary features are binarized representations of categorical features. 
     
     
         24 . The system of  claim 21 , wherein the plurality of features are features of a content presentation setting, and wherein the predicted output is a predicted output for the content presentation setting. 
     
     
         25 . The system of  claim 24 , wherein the features of the content presentation setting include features of a content item to be presented in the content presentation setting, and wherein the predicted output for the content presentation setting represents a likelihood that a particular objective will be satisfied if the content item is presented in the content presentation setting. 
     
     
         26 . The system of  claim 25 , wherein the combining layer is a logistic regression layer that is configured to process the deep model intermediate predicted output generated by the deep machine learning model and the wide model intermediate predicted output generated by the wide machine learning model to generate a score that represents the likelihood that the particular objective will be satisfied if the content item is presented in the content presentation setting. 
     
     
         27 . The system of  claim 24 , wherein the predicted output for the content presentation setting is a respective score for each content item in a predetermined set of content items, and wherein each respective score represents a respective likelihood that a particular objective will be satisfied if the corresponding content item is presented in the content presentation setting. 
     
     
         28 . The system of  claim 27 , wherein the combining layer is a softmax layer that is configured to process the deep model intermediate predicted output generated by the deep machine learning model and the wide model intermediate predicted output generated by the wide machine learning model to generate the respective score for each content item in a predetermined set of content items. 
     
     
         29 . The system of  claim 24 , wherein the plurality of features include user features characterizing a user to whom a content item is to be presented in the content presentation setting. 
     
     
         30 . The system of  claim 24 , wherein the plurality of features include contextual information characterizing a context of the content presentation setting. 
     
     
         31 . The system of  claim 21 , wherein the deep machine learning model includes a deep neural network. 
     
     
         32 . The system of  claim 31 , wherein the deep machine learning model includes an embedding layer that is configured to map each of the features to a respective numeric embedding of the feature. 
     
     
         33 . The system of  claim 21 , wherein the wide machine learning model is a generalized linear model. 
     
     
         34 . The system of  claim 21 , wherein the wide machine learning model is configured to process the binary features and transformed features generated from the binary features to generate the wide model intermediate output. 
     
     
         35 . The system of  claim 21 , wherein the wide machine learning model intermediate output and the deep model intermediate output are log odds outputs. 
     
     
         36 . A method of training a combined machine learning model, the combined machine learning model being configured to process a machine learning input to generate an output for the machine learning input, the combine machine learning model including a deep machine learning model, a wide machine learning model, and a combining layer that is configured to process outputs of the deep machine learning model and the wide machine learning model to generate the output for the machine learning input, and the method comprising:
 obtaining training data comprising, for each of a plurality of training inputs, (i) a plurality of features of the training input and (ii) a known output for the training input; and   for each of the training inputs:
 processing a subset of the plurality of features comprising continuous features using the deep machine learning model to generate a deep model intermediate predicted output for the training input in accordance with current values of parameters of the deep machine learning model; 
 processing binary features of the plurality of features of the training input using the wide machine learning model to generate a wide model intermediate predicted output for the training input in accordance with current values of parameters of the wide machine learning model; 
 processing the deep model intermediate predicted output and the wide model intermediate predicted output for the training input using the combining layer to generate a predicted output for the training input; 
 backpropagating a gradient determined from an error between the predicted output for the training input and the known output for the training input through the combining layer to the wide machine learning model and the deep machine learning model to jointly adjust the current values of the parameters of the deep machine learning model and the wide machine learning model. 
   
     
     
         37 . The method of  claim 35 , wherein the binary features are one-hot encoded representations of sparse features. 
     
     
         38 . The method of  claim 35 , wherein the binary features are binarized representations of categorical features. 
     
     
         39 . The method of  claim 35 , further comprising adjusting the current values of the parameters of the deep machine learning model and the wide machine learning model using mini-batch stochastic optimization. 
     
     
         40 . The method of  claim 35 , further comprising adjusting the current values of the parameters of the wide machine learning model using a Follow-the-regularized-leader (FTRL) algorithm with L1 regularization. 
     
     
         41 . The method of  claim 35 , further comprising adjusting the current values of the parameters of the deep machine learning model using stochastic gradient optimization with an adaptive learning rate. 
     
     
         42 . A non-transitory computer-readable medium having instructions stored thereon which, when executed by at least one computer, cause the at least one computer to perform operations to train a combined machine learning model, the combined machine learning model being configured to process a machine learning input to generate an output for the machine learning input, the combine machine learning model including a deep machine learning model, a wide machine learning model, and a combining layer that is configured to process outputs of the deep machine learning model and the wide machine learning model to generate the output for the machine learning input, and the operations comprising:
 obtaining training data comprising, for each of a plurality of training inputs, (i) a plurality of features of the training input and (ii) a known output for the training input; and   for each of the training inputs:
 processing a subset of the plurality of features comprising continuous features using the deep machine learning model to generate a deep model intermediate predicted output for the training input in accordance with current values of parameters of the deep machine learning model; 
 processing binary features of the plurality of features of the training input using the wide machine learning model to generate a wide model intermediate predicted output for the training input in accordance with current values of parameters of the wide machine learning model; 
 processing the deep machine learning model intermediate predicted output and the wide model intermediate predicted output for the training input using the combining layer to generate a predicted output for the training input; 
 backpropagating a gradient determined from an error between the predicted output for the training input and the known output for the training input through the combining layer to the wide machine learning model and the deep machine learning model to jointly adjust the current values of the parameters of the deep machine learning model and the wide machine learning model. 
   
     
     
         43 . The non-transitory computer readable medium of  claim 42 , wherein the binary features are one-hot encoded representations of sparse features. 
     
     
         44 . The non-transitory computer readable medium of  claim 42 , wherein the binary features are binarized representations of categorical features. 
     
     
         45 . The non-transitory computer readable medium of  claim 42 , wherein the operations further comprise adjusting the current values of the parameters of the wide machine learning model using a Follow-the-regularized-leader (FTRL) algorithm with L1 regularization. 
     
     
         46 . The non-transitory computer readable medium of  claim 42 , wherein the operations further comprise adjusting the current values of the parameters of the deep machine learning model using stochastic gradient optimization with an adaptive learning rate.