Detecting and responding to malicious acts directed towards machine learning model

Abstract

A system detects and responds to malicious acts directed towards machine learning models. Data fed into and output by a machine learning model is collected by a sensor. The data fed into the model includes vectorization data, which is generated from raw data provided from a requester, such as for example a stream of timeseries data. The output data may include a prediction or other output generated by the machine learning model in response to receiving the vectorization data. The vectorization data and machine learning model output data are processed to determine whether the machine learning model is being subject to a malicious act (e.g., attack). The output of the processing may indicate an attack score. A response for handling the request by a requester may be selected based on the output that includes the attack score, and the response may be applied to the requestor.

What is claimed is: 
     
         1 . A method for monitoring a machine learning-based system for malicious acts, comprising:
 receiving vectorization data by a sensor a server, the vectorization data derived from input data intended for a first machine learning model and provided by a requestor;   receiving, by the sensor, an output generated by the machine learning model, the machine learning model generating the output in response to receiving the vectorization data;   transmitting the vectorization data and the output to a processing engine by the sensor;   processing the vectorization data and the output by the processing engine to generate an attack score, the attack score indicating a likelihood of a malicious action towards the machine learning model via the vectorization data; and   applying a response to a request associated with the requestor, the response based at least in part on the attack score, the response applied in place of the output of the first machine learning model.   
     
     
         2 . The method of  claim 1 , wherein applying the response includes selecting, by a response engine, a response based on an output by a second machine learning model within the processing engine, the output of the second machine learning model including a prediction of an attack on the first machine learning model. 
     
     
         3 . The method of  claim 1 , further comprising collecting the vectorization data by a sensor component, the sensor component transmitting the collected vectorization data to the processing engine on the server. 
     
     
         4 . The method of  claim 3 , wherein the sensor component is created in a computing environment that proxies the first machine learning model. 
     
     
         5 . The method of  claim 3 , further including:
 collecting the output generated by the first machine learning model by the sensor component;   coupling the vectorization data and output by the sensor component; and   transmitting the coupled vectorization data and output to the processing engine by the sensor component.   
     
     
         6 . The method of  claim 3 , further including:
 intercepting the output of the first machine learning model by a sensor component; and   transmitting a response generated by the sensor to the requestor in place of the output, the response generated based at least in part on the attack score.   
     
     
         7 . The method of  claim 1 , further comprising generating an alert based on the attack score. 
     
     
         8 . The method of  claim 1 , further comprising reporting attack data to a user through a graphical interface, the attack data based at least in part on the attack score. 
     
     
         9 . A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for monitoring a machine learning-based system for malicious acts, the method comprising:
 receiving vectorization data by a sensor, the vectorization data derived from input data intended for a first machine learning model and provided by a requestor;   receiving, by the sensor, an output generated by the machine learning model, the machine learning model generating the output in response to receiving the vectorization data;   transmitting the vectorization data and the output to a processing engine by the sensor;   processing the vectorization data and the output by the processing engine to generate an attack score, the attack score indicating a likelihood of a malicious action towards the machine learning model via the vectorization data; and   applying a response to a request associated with the requestor, the response based at least in part on the attack score, the response applied in place of the output of the first machine learning model.   
     
     
         10 . The non-transitory computer readable storage medium of  claim 9 , wherein applying the response includes selecting, by a response engine, a response based on an output by a second machine learning model within the processing engine, the output of the second machine learning model including a prediction of an attack on the first machine learning model. 
     
     
         11 . The non-transitory computer readable storage medium of  claim 9 , the method further comprising collecting the vectorization data by a sensor component, the sensor component transmitting the collected vectorization data to the processing engine on the server. 
     
     
         12 . The non-transitory computer readable storage medium of  claim 11 , wherein the sensor component is created in a computing environment that implements the first machine learning model. 
     
     
         13 . The non-transitory computer readable storage medium of  claim 11 , the method further including:
 collecting the output generated by the first machine learning model by the sensor component;   coupling the vectorization data and output by the sensor component; and   transmitting the coupled vectorization data and output to the processing engine by the sensor component.   
     
     
         14 . The non-transitory computer readable storage medium of  claim 11 , the method further including:
 intercepting the output of the first machine learning model by a sensor component; and   transmitting a response generated by the sensor to the requestor in place of the output, the response generated based at least in part on the attack score.   
     
     
         15 . The non-transitory computer readable storage medium of  claim 9 , the method further comprising generating an alert based on the attack score. 
     
     
         16 . The non-transitory computer readable storage medium of  claim 9 , the method further comprising reporting attack data to a user through a graphical interface, the attack data based at least in part on the attack score. 
     
     
         17 . A system for monitoring a machine learning-based system for malicious acts, comprising:
 one or more servers including a memory and a processor; and   one or more modules stored in the memory and executed by the processor to receive vectorization data, by sensor, the vectorization data derived from input data intended for a first machine learning model and provided by a requestor, receive, by the sensor, an output generated by the machine learning model, the machine learning model generating the output in response to receiving the vectorization data, transmit the vectorization data and the output to a processing engine by the sensor, process the vectorization data and the output by the processing engine to generate an attack score, the attack score indicating a likelihood of a malicious action towards the machine learning model via the vectorization data, and apply a response to a request associated with the requestor, the response based at least in part on the attack score, the response applied in place of the output of the first machine learning model.   
     
     
         18 . The system of  claim 17 , wherein applying the response includes selecting, by a response engine, a response based on an output by a second machine learning model within the processing engine, the output of the second machine learning model including a prediction of an attack on the first machine learning model. 
     
     
         19 . The system of  claim 17 , the modules further executable to collect the vectorization data by a sensor component, the sensor component transmitting the collected vectorization data to the processing engine on the server. 
     
     
         20 . The system of  claim 19 , wherein the sensor component is created in a computing environment that implements the first machine learning model.