Method for training and refining an artificial intelligence

Abstract

One variation of a method for training and refining an artificial intelligence includes: training a neural network on a training set to identify objects in optical images; receiving a manual label attributed to an optical image—recorded by a road vehicle during its operation—by a human annotator; passing the optical image through the neural network to generate an automated label attributed to the optical image; in response to the manual label differing from the automated label, serving the optical image to a human annotator for manual confirmation of one of the manual label and the automated label; appending the training set with the optical image containing one of the manual label and the automated label based on confirmation received from the human annotator; and retraining the neural network, with the training set, to identify objects in optical images.

I claim: 
     
         1 . A method comprising:
 accessing a training set comprising optical images, each optical image in the training set comprising a label identifying an object represented in the optical image;   training a neural network, with the training set, to identify objects in optical images;   receiving a first optical image recorded by an optical sensor integrated into a road vehicle, the first optical image recorded during operation of the road vehicle;   serving the first optical image to an annotation portal executing on a local computer system for manual labeling;   receiving a first manual label attributed to a first manually-defined location on the first optical image by a human annotator at the local computer system;   passing the first optical image through the neural network to generate a first automated label attributed to a first automatically-defined location on the first optical image;   in response to the first manually-defined location approximating the first automatically-defined location and in response to the first manual label differing from the first automated label:
 serving the first optical image, the first manual label, and the first automated label to the annotation portal for manual confirmation of one of the first manual label and the first automated label proximal the first manually-defined location; 
 receiving confirmation of one of the first manual label and the first automated label proximal the first manually-defined location from the human annotator via the annotation portal; 
   appending the training set with the first optical image comprising one of the first manual label and the first automated label based on confirmation received from the human annotator; and   retraining the neural network, with the training set, to identify objects in optical images.   
     
     
         2 . The method of  claim 1 :
 wherein training the neural network comprises training the neural network at a remote computer system;   further comprising, at the local computer system:
 rendering the first optical image, the first manual label, and the first automated label, the first manual label and the first automated label addressed to the first manually-defined location; 
 prompting the human annotator to enter a selection of one of the first manual label and the first automated label within the annotation portal to confirm a type of an object represented proximal the first manually-defined location within the first optical image; and 
 returning the selection to the remote computer system via a computer network. 
   
     
     
         3 . The method of  claim 1 :
 wherein receiving the first optical image comprises downloading a LIDAR feed from the road vehicle, the LIDAR feed recorded by a LIDAR sensor integrated into the road vehicle, the first optical image defining a frame in the LIDAR feed and comprising a point cloud representing external surfaces in the field of view of the LIDAR sensor at a time the first optical image was recorded;   further comprising, at the local computer system:
 receiving selection of the first manual label; 
 rendering a virtual bounding box coupled to a cursor within the annotation portal, the virtual bounding box defining a geometry associated with a first type of the first manual label; 
 locating the bounding box within the first optical image based on a position of a cursor input over the first optical image; and 
 labeling a cluster of points contained within the bounding box as representing an object of the first type; and 
   wherein receiving the first manual label comprises:
 retrieving identifiers of points in the cluster of points and the first manual label linked to each point in the cluster of points; and 
 defining the first manually-defined location based on positions of points in the cluster of points within the first optical image. 
   
     
     
         4 . The method of  claim 1 :
 wherein receiving confirmation of one of the first manual label and the first automated label proximal the first manually-defined location comprises receiving confirmation of the first manual label of a first type;   further comprising, in response to receiving confirmation of the first manual label at the first manually-defined location in the first image, the first manual label conflicting with the first automated label:
 aggregating a subset of optical training images, from the training set, containing labels of a second type represented by the first automated label; 
 distributing the subset of optical training images to a set of annotation portals for manual confirmation of labels of one of the first type and the second type within the subset of optical training images by a set of human annotators; and 
 receiving confirmation of labels of one of the first type and the second type for the subset of optical training images from the set of human annotators via the set of annotation portals; and 
   wherein retraining the neural network comprises retraining the neural network according to:
 the first optical image and the first manual label; and 
 the subset of optical training images and labels of one of the first type and the second type confirmed for the subset of optical training images by the set of human annotators. 
   
     
     
         5 . The method of  claim 4 :
 wherein passing the first optical image through the neural network further comprises calculating a first confidence score for the first automated label;   wherein aggregating the subset of optical training images and distributing the subset of optical training images to the set of annotation portals for manual confirmation comprises aggregating the subset of optical training images and distributing the subset of optical training images to the set of annotation portals for manual confirmation in response to the first confidence score exceeding a threshold score; and   further comprising in response to the first confidence score remaining below the threshold score:
 appending the training set with the first optical image comprising the first manual label directly; and 
 retraining the neural network, with the training set, to identify objects in optical images. 
   
     
     
         6 . The method of  claim 1 :
 wherein receiving the first optical image comprises, at a remote computer system, downloading a LIDAR feed from the road vehicle, the LIDAR feed recorded by a LIDAR sensor integrated into the road vehicle, the first optical image defining a frame in the LIDAR feed and comprising a point cloud representing external surfaces in the field of view of the LIDAR sensor at a time the first optical image was recorded;   further comprising, at the remote computer system, downloading a first video feed from the road vehicle, the first video feed recorded by a first camera integrated into the road vehicle and comprising digital photographic images of a first field around the road vehicle;   wherein serving the first optical image to the annotation portal for manual labeling comprises:
 serving frames in the LIDAR feed to the annotation portal; and 
 serving digital photographic images in the first video feed to the annotation portal for rendering adjacent and synchronized with the LIDAR feed; and 
   wherein receiving the first manual label comprises receiving the first manual label applied to the first manually-defined location within the first optical image in the LIDAR feed.   
     
     
         7 . The method of  claim 6 :
 wherein downloading the LIDAR feed from the road vehicle comprises:
 downloading first optical image comprising a three-dimensional point cloud representing external surfaces proximal the road vehicle at the time the first optical image was recorded by the LIDAR sensor; 
 removing a subset of points representing a ground surface from the first optical image; and 
 compressing remaining points in the first optical image onto a horizontal plane to form a two-dimensional plan point cloud; 
   wherein downloading the first video feed comprises downloading a first digital photographic image temporally aligned to the first optical image and approximating a two-dimensional elevation view;   further comprising, at the local computer system:
 rendering the first optical image and the first digital photographic image substantially simultaneously within the annotation portal; and 
 in response to insertion of the first manual label onto the first manually-defined location within the first optical image, projecting the first manual label from the first manually-defined location in the first optical image onto a corresponding location in the first digital photographic image rendered within the annotation portal in real-time. 
   
     
     
         8 . The method of  claim 6 :
 further comprising, at the remote computer system:
 downloading a second video feed from the road vehicle, the second video feed recorded by a second camera integrated into the road vehicle and comprising digital photographic images of a second field, distinct from the first field, around the road vehicle; 
 linking the first video feed to a first sector within the LIDAR feed based on a known position of the first camera on the road vehicle; 
 linking the second video feed to a second sector within the LIDAR feed based on a known position of the second camera on the road vehicle, the second sector different from the first sector; 
   further comprising, at the local computer system:
 querying the remote computer system for digital photographic images in the first video feed in response to detecting the cursor over the first sector of the LIDAR feed; 
 querying the remote computer system for digital photographic images in the second video feed in response to detecting the cursor over the second sector of the LIDAR feed; and 
 selectively rendering digital photographic images from the first video feed and the second video feed adjacent and synchronized with the LIDAR feed within the annotation portal; and 
   wherein serving digital photographic images in the first video feed to the annotation portal comprises selectively serving digital photographic images in the first video feed and the second video feed to the annotation portal responsive to queries received from the local computer system.   
     
     
         9 . The method of  claim 1 :
 wherein receiving the first optical image comprises, at a remote computer system, downloading a LIDAR feed from the road vehicle, the LIDAR feed recorded by a LIDAR sensor integrated into the road vehicle, the first optical image defining a frame in the LIDAR feed and comprising a point cloud representing external surfaces in the field of view of the LIDAR sensor at a time the first optical image was recorded;   wherein serving the first optical image to the annotation portal comprises serving a first subset of frames intermittently distributed throughout the LIDAR feed to the local computer system, the first subset of frames comprising the first optical image and a second optical image succeeding the first optical image in the LIDAR feed;   further comprising, at local computer system:
 recording the first manual label defining a first dynamic object type at the first manually-defined location in the first optical image; 
 recording a second manual label defining the first dynamic object type at a second manually-defined location within the second optical image; and 
 returning the first manual label, a definition of the first manually-defined location, the second manual label, and a definition of the second manually-defined location to the remote computer system; and 
   further comprising, at the remote computer system, interpolating a third location of a third label of the first dynamic object type in a third frame based on the first manually-defined location and the second manually-defined location, the third frame between the first frame and the second frame in the LIDAR feed and excluded from the first subset of frames.   
     
     
         10 . The method of  claim 1 , wherein receiving the first manual label comprises receiving, from the local computer system, the first manual label representing a fixed infrastructure object. 
     
     
         11 . The method of  claim 1 :
 wherein passing the first optical image through the neural network comprises passing the first optical image through a version of the neural network executing on the road vehicle during operation of the road vehicle to generate the first automated label attributed to the first automatically-defined location on the first optical image; and   wherein receiving the first optical image comprises receiving the first optical image, the first automated label, and a pointer to the first automatically-defined location from the road vehicle.   
     
     
         12 . The method of  claim 1 :
 wherein passing the first optical image through the neural network further comprises calculating a first confidence score for the first automated label representing a first object type and calculating a second confidence score for a second automated label representing a second object type for the first automatically-defined location within the first optical image, the second confidence score less than the first confidence score;   further comprising, at the local computer system:
 in response to the first confidence score exceeding a preset threshold score and the second confidence score, rendering the first optical image with the first automated label linked to the first automatically-defined location within the first optical image in the annotation portal; and 
 in response to the first confidence score and the second confidence score remaining below the preset threshold score: 
   rendering the first optical image with the first automated label and the second automated label linked to the first automatically-defined location within the first optical image in the annotation portal; and   rendering a prompt to select one of the first automated label and the second automated label in the annotation portal; and   wherein receiving the first manual label comprises receiving selection of the second automated label for the first optical image by the human annotator at the annotation portal.   
     
     
         13 . A method comprising
 accessing a training set comprising optical images, each optical image in the training set comprising a label identifying an object represented in the optical image;   training a neural network, with the training set, to identify objects in optical images;   receiving a first optical image recorded by an optical sensor integrated into a road vehicle, the first optical image recorded during operation of the road vehicle;   passing the first optical image through the neural network to generate a first automated label attributed to the first optical image;   serving the first optical image to a first annotation portal executing on a local computer system for manual labeling;   receiving a first manual label attributed to the first optical image by a first human annotator at the local computer system;   in response to the first manual label differing from the first automated label:
 serving the first optical image, the first manual label, and the first automated label to a set of annotation portals for manual confirmation of one of the first manual label and the first automated label for the first optical image by a set of human annotators; 
 receiving confirmation of one of the first manual label and the first automated label for the first optical image from the set of human annotators; 
   appending the training set with the first optical image comprising one of the first manual label and the first automated label based on confirmations received from the set of human annotators through the set of annotation portals; and   retraining the neural network, with the training set, to identify objects in optical images.   
     
     
         14 . The method of  claim 13 :
 wherein serving the first optical image, the first manual label, and the first automated label to the set of annotation portals comprises serving the first optical image, the first manual label, and the first automated label to the set of human annotators for manual confirmation of one of the first manual label and the first automated label for the first optical image, the set of human annotators comprising the first human annotator and a second human annotator;   wherein receiving confirmations of one of the first manual label and the first automated label for the first optical image comprises calculating a binary combination of a first confirmation of one of the first manual label and the first automated label received from the first human annotator and a second confirmation of one of the first manual label and the first automated label received from the second human annotator; and   wherein appending the training set with the first optical image comprises appending the training set with the first optical image and the binary combination.   
     
     
         15 . The method of  claim 13 :
 wherein receiving confirmation of one of the first manual label and the first automated label comprises receiving confirmation of the first manual label of a first type within the first optical image;   further comprising, in response to receiving confirmation of the first manual label in the first image from the set of human annotators, the first manual label conflicting with the first automated label:
 aggregating a subset of optical training images, from the training set, containing labels of a second type represented by the first automated label; 
 distributing the subset of optical training images to the set of annotation portals for manual confirmation of labels of one of the first type and the second type by the set of human annotators; and 
 receiving confirmation of labels of one of the first type and the second type for the subset of optical training images from the set of human annotators via the set of annotation portals; and 
   wherein retraining the neural network comprises retraining the neural network according to:
 the first optical image and the first manual label; and 
 the subset of optical training images and labels of one of the first type and the second type confirmed for the subset of optical training images by the set of human annotators. 
   
     
     
         16 . A method comprising:
 accessing a training set comprising discrete sequences of optical images, each discrete sequence of optical images in the training set comprising a label identifying a navigational action represented in the discrete sequence of optical images;   training a neural network, with the training set, to identify navigational actions represented in sequences of optical images;   receiving a first sequence of optical images recorded by an optical sensor integrated into a road vehicle, the first sequence of optical images recorded during operation of the road vehicle;   passing the first sequence of optical images through the neural network to generate a first automated label attributed to the first sequence of optical images;   serving the first sequence of optical images to a first annotation portal executing on a first local computer system for manual labeling;   receiving a first manual label attributed to the first sequence of optical images by a first human annotator at the first local computer system;   in response to the first manual label differing from the first automated label in the first sequence of optical images:
 serving the first sequence of optical images, the first manual label, and the first automated label to a set of annotation portals for manual confirmation of one of the first manual label and the first automated label for the first sequence of optical images by the first set of human annotators; 
 receiving confirmations of one of the first manual label and the first automated label for the first sequence of optical images from the set of human annotators through the set of annotation portals; 
   appending the training set with the first sequence of optical images comprising one of the first manual label and the first automated label based on confirmations received from the set of human annotators; and   retraining the neural network, with the training set, to identify navigational actions in discrete sequences of optical images.   
     
     
         17 . The method of  claim 16 :
 wherein passing the first sequence of optical images through the neural network to generate the first automated label comprises:
 passing the first sequence of optical images through a version of the neural network executing on the road vehicle during autonomous execution of a route by the road vehicle to select a next action of the road vehicle; and 
 writing the first automated label identifying the next action to the first sequence of optical images; and 
   wherein receiving the first sequence of optical images comprises receiving the first sequence of optical images, the first automated label, and a first automatically-defined activation time and a first automatically-defined deactivation time of a first vehicle navigational action represented by the first automated label from the road vehicle.   
     
     
         18 . The method of  claim 17 :
 wherein receiving the first sequence of optical images comprises, at a remote computer system, downloading a first video feed from the road vehicle, the first video feed recorded by a first camera integrated into the road vehicle;   wherein serving the first sequence of optical images to the first annotation portal for manual labeling comprises serving the first video feed to the first annotation portal;   further comprising, at the first local computer system:
 replaying the first video feed; and 
 receiving a sequence of inputs to activate and deactivate vehicle navigational action labels throughout a duration of the first video feed, each vehicle navigational action label specifying a unique navigational action in a predefined set of navigational actions; and 
   wherein receiving the first manual label attributed to the first sequence of optical images comprises receiving, from the first local computer system, manually-defined activation times and manually-defined deactivation times of vehicle navigational action labels activated and deactivated by the first human annotator throughout the duration of the first video feed.   
     
     
         19 . The method of  claim 18 :
 further comprising, at the first local computer system:
 ranking the predefined set of navigational actions presented within the annotation tool based on the first automated label from the first automatically-defined activation time to the first automatically-defined deactivation time during playback of the first sequence of optical images at the first annotation portal; and 
 recording the first manual label activated by the first human annotator at a first manually-defined activation time and deactivated by the first human annotator at a first manually-defined deactivation time; and 
   wherein receiving the first manual label comprises receiving the first manual label, the first manually-defined activation time, and the first manually-defined deactivation time from the local computer system.   
     
     
         20 . The method of  claim 19 :
 further comprising:
 in response to the first manually-defined activation time of the first manual label differing from the first automatically-defined activation time of the first automated label by more than a threshold time, flagging the first sequence of optical image for a time conflict; and 
 in response to the first manual label conflicting with the first automated label and the first manually-defined activation time approximating the first automatically-defined activation time, flagging the first sequence of optical images for a navigational action conflict; and 
   wherein serving the first sequence of optical images to the set of annotation portals for manual confirmation by the set of human annotators comprises serving the first sequence of optical images to the set of annotation portals for manual confirmation by the set of human annotators in response to flagging the first sequence of optical images.