Automated control of microfluidic devices based on machine learning

Abstract

A system is provided to automatically monitor and control the operation of a microfluidic device using machine learning technology. The system receives images of a channel of a microfluidic device collected by a camera during operation of the microfluidic device. Upon receiving an image, the system applies a classifier to the image to classify the operation of the microfluidic device as normal, in which no adjustment to the operation is needed, or as abnormal, in which an adjustment to the operation is needed. When an image is classified as normal, the system may make no adjustment to the microfluidic device. If, however, an image is classified as abnormal, the system may output an indication that the operation is abnormal, output an indication of a needed adjustment, or control the microfluidic device to make the needed adjustment.

1 - 45 . (canceled) 
     
     
         46 . A method performed by a computing system, the method comprising:
 collecting images relating to operation of a training microfluidic device during operation of the training microfluidic device;   generating a label for each image relating to operation of the training microfluidic device; and   training a classifier using the images and labels as training data, the classifier for receiving an image relating to operation of a production microfluidic device and generating a label relating to operation of the production microfluidic device.   
     
     
         47 . The method of  claim 46  wherein the classifier includes a convolutional neural network and a sub-classifier. 
     
     
         48 . The method of  claim 47  wherein the sub-classifier is a support vector machine. 
     
     
         49 . The method of  claim 47  wherein the sub-classifier is a linear regression classifier. 
     
     
         50 . The method of  claim 47  wherein the convolutional neural network inputs and generates a feature vector for the image and wherein the sub-classifier inputs the feature vector and generates a label for the image. 
     
     
         51 . The method of  claim 47  wherein the convolutional neural network include multiple convolutional layers, wherein each convolutional layer is associated with a rectified linear unit and a max pooling technique, wherein the input to the first convolutional layer is an image, wherein the output of each convolutional layer is processed by the associated rectified linear unit and max pooling technique to generate output that is input to the next convolutional layer if any and input to a fully connected layer if there is no next convolutional layer. 
     
     
         52 . The method of  claim 47  wherein the sub-classifier generates the label for an image further based on an additional feature not generated by the convolution neural network. 
     
     
         53 . The method of  claim 46  wherein a label indicates an abnormal operation of the training microfluidic device. 
     
     
         54 . The method of  claim 46  wherein the microfluidic device includes a micromixer for mixing two or more fluids with a microchannel. 
     
     
         55 . The method of  claim 46  wherein the microfluidic device includes a microswitch and the labels indicate shapes of cells. 
     
     
         56 . The method of  claim 46  wherein the microfluidic device is a microencapsulation device and a label indicates whether an image is of a well-formed microcapsule or a malformed microcapsules. 
     
     
         57 - 73 . (canceled) 
     
     
         74 . One or more computing systems for training a classifier for generating a label for an image by applying a classifier, the one or more computing systems comprising:
 one or more computer-readable storage mediums for storing computer-executable instructions for controlling the one or more computing systems to:
 access training data for training the classifier, the training data including images having labels the images and labels relating to operation of a fluidic device; 
 training the classifier using the training data 
 wherein the classifier includes a convolutional neural network and a sub-classifier and wherein during classification, an image is input to the convolutional neural network, output of the convolutional neural network is input to the sub-classifier as a feature vector for the image, and output of the sub-classifier is a label for the image; and 
   one or more processors for executing the computer-executable instructions stored in the one or more computer-readable storage mediums.   
     
     
         75 . The one or more computing systems  claim 74  wherein the sub-classifier is a support vector machine. 
     
     
         76 . The one or more computing systems of  claim 74  wherein the sub-classifier is a linear regression classifier. 
     
     
         77 . The one or more computing systems of  claim 74  wherein the convolutional neural network includes multiple convolutional layers, wherein each convolutional layer is associated with a rectified linear unit and a max pooling technique, wherein the input to the first convolutional layer is the image, wherein the output of each convolutional layer is processed by the associated rectified linear unit and max pooling technique to generate output that is input to the next convolutional layer if any and input to a fully connected layer if there is no next convolutional layer. 
     
     
         78 . The one or more computing systems of  claim 74  wherein the sub-classifier inputs an additional feature not generated by the convolutional neural network. 
     
     
         79 . The method of  claim 46  wherein the microfluidic device includes a micromixer for mixing two or more fluids with a microchannel. 
     
     
         80 . The method of  claim 79  wherein the two or more fluids are reactants. 
     
     
         81 . The method of  claim 79  wherein the two or more fluids include an acid or a base to control pH balance in another fluid. 
     
     
         82 . The method of  claim 46  wherein a label relates to a change in a flow rate of a fluid. 
     
     
         83 . The computing system of  claim 46  wherein a label relates to discarding malformed droplets of a double emulsion. 
     
     
         84 . The method of  claim 46  wherein during initial operation of the microfluidic device, a label relates to controlling the microfluidic device to operate normally. 
     
     
         85 . The method of  claim 46  wherein the microfluidic device includes a switch and wherein a label relates to controlling the switch to sort cells of a target shape into a target bin and other cells into another bin. 
     
     
         86 . The method of  claim 46  wherein the microfluidic device is a microencapsulation device with a switch and wherein a label relates to controlling the switch to deliver well-formed microcapsules to one bin and malformed microcapsules to another bin. 
     
     
         87 . The one or more computing systems of  claim 74  wherein a label relates to controlling the fluidic device. 
     
     
         88 . The one or more computing systems of  claim 74  wherein a label relates to a condition of operation of the fluidic device 
     
     
         89 . The one or more computing systems of  claim 74  wherein the classifier includes a convolution neural network to generate features for the images and a sub-classifier to generate labels for the images based on the generated features of the images. 
     
     
         90 . The one or more computing systems of  claim 74  wherein the sub-classifier generates the labels for the images further based on an additional feature not generated by the convolution neural network. 
     
     
         91 . One or more computing systems comprising:
 one or more computer-readable storage mediums for storing
 images relating to operation of a fluidic device, each image having a label relating to operation of the fluidic device; and 
 computer-executable instructions for controlling the one or more computing systems to train a classifier using the images and labels as training data, the classifier for receiving an image relating to operation of a production fluidic device and for generating a label relating to operation of the production fluidic device; and 
   one or more processors for executing the computer-executable instructions stored in the one or more computer-readable storage mediums.   
     
     
         92 . The one or more computing systems of  claim 91  wherein a label indicates an abnormal operation of a fluidic device. 
     
     
         93 . The one or more computing systems of  claim 91  wherein when a label indicates that flow rate of a fluid that enters a fluidic device is to be adjusted. 
     
     
         94 . The one or more computing systems of  claim 91  wherein a fluidic device includes a mixer for mixing two or more fluids with a channel. 
     
     
         95 . The one or more computing systems of  claim 94  wherein the two or more fluids are reactants. 
     
     
         96 . The one or more computing systems of  claim 94  wherein the two or more fluids include an acid or a base to control pH balance in another fluid. 
     
     
         97 . The one or more computing systems of  claim 91  wherein a label relates to discarding malformed droplets of a double emulsion. 
     
     
         98 . The one or more computing systems of  claim 91  wherein a fluidic device includes a switch and wherein a label relates to directing the switch to sort cells of a target shape into a target bin and other cells into another bin. 
     
     
         99 . The one or more computing systems of  claim 91  wherein a fluidic device is an encapsulation device with a switch and wherein the label relates to directing the switch to deliver well-formed capsules to one bin and malformed capsules to another bin. 
     
     
         100 . The one or more computing systems of  claim 91  wherein the images were collected during operation of one or more training fluidic devices.