Network for detecting edge cases for use in training autonomous vehicle control systems

Abstract

Embodiments relate to the detection of edge cases through application of a neural network to predict future vehicle environment data and identifying an edge case when the prediction error exceeds a given threshold. This allows edge cases to be identified based on unexpected vehicle environmental conditions or conditions that otherwise cause the neural network to make inaccurate predictions. These edge cases can then be utilised to better train machine learning systems, for instance, to train autonomous vehicle control systems. Alternatively, the identification of an edge case can highlight the need for remedial action, and can therefore trigger an alert to a vehicle control system to take remedial action. Further methods and systems described herein improve environmental sensing by providing a computationally efficient and accurate means for fusing sensor data and using this fused data to control sensors to focus on areas that would most reduce the uncertainty in the sensing system.

1 . A computer-implemented method for detecting edge cases, the method comprising:
 obtaining environment data for a first time detailing one or more observed features of a vehicle environment at the first time;   processing the environment data for the first time using a neural network to determine predicted environment data for a second time that is later than the first time;   obtaining environment data for the second time detailing one or more observed features of the vehicle environment at the second time;   determining a prediction error based on the predicted environment data and the environment data for the second time; and   in response to the prediction error exceeding a threshold, outputting an alert indicating detection of an edge case.   
     
     
         2 . The method of  claim 1  wherein:
 the alert comprises information detailing the environment data for one or both of the first time and the second time as an edge case for use in training one or more autonomous vehicle control systems; or 
 the alert is to a vehicle control system within the vehicle to adapt to potential danger. 
 
     
     
         3 . The method of  claim 1  wherein:
 environment data for the first time further describes one or more observed actions performed by a driver of the vehicle or an autonomous control system of the vehicle at the first time; 
 the predicted environment data for the second time comprises one or more predicted actions to be taken by the driver or autonomous vehicle control system of the vehicle at the second time; and 
 the environment data for the second time further details one or more observed actions performed by the driver or autonomous vehicle control system of the vehicle at the second time. 
 
     
     
         4 . The method of  claim 3  wherein the one or more observed actions for the first and second times and the one or more predicted actions comprise one or more of breaking, accelerating, engaging a clutch, turning, indicating and changing gear. 
     
     
         5 . The method of  claim 4  wherein the alert is to a vehicle control system within the vehicle to adapt to potential danger and wherein the alert comprises the one or more predicted actions. 
     
     
         6 . The method of  claim 1  further comprising predicting one or more emergency actions for adapting to the vehicle environment and wherein the alert is to a vehicle control system within the vehicle comprising the one or more emergency actions. 
     
     
         7 . The method of  claim 6  wherein the one or more emergency actions comprise at least one action from a group comprising:
 adjusting a configuration of, engaging or disengaging a braking system for the vehicle; 
 adjusting a configuration of, engaging or disengaging a steering system for the vehicle; 
 adjusting a configuration of, engaging or disengaging an acceleration system for the vehicle; and 
 adjusting a configuration of, engaging or disengaging one or more external lights or indicators of the vehicle. 
 
     
     
         8 . The method of  claim 1  wherein the method is performed in a computing system carried by the vehicle and the alert comprises information detailing the environment data for one or both of the first time and the second time as an edge case for use in training one or more autonomous vehicle control systems, wherein the alert is output by sending the information to a server for further analysis. 
     
     
         9 . The method of  claim 8  wherein the output information is utilised to recreate the vehicle environment at the first and second times within a simulator in order to train an autonomous vehicle control system to learn from the edge case. 
     
     
         10 . The method of  claim 1  further comprising:
 receiving an updated neural network; 
 obtaining environment data for a third time detailing one or more observed features of a vehicle environment at the third time; 
 processing the environment data for the third time using the updated neural network to determine predicted environment data for a fourth time that is later than the third time; 
 obtaining environment data for the fourth time detailing one or more observed features of the vehicle environment at the fourth time; 
 determining a prediction error based on the predicted data for the fourth time and the environment data for the fourth time; and 
 in response to the prediction error for the fourth time exceeding a threshold, outputting information detailing the environment data for the fourth time as an edge case for further analysis. 
 
     
     
         11 . The method of  claim 1  wherein
 the one or more features of the environment comprise an autonomous vehicle prediction error from an autonomous vehicle control system for the vehicle. 
 
     
     
         12 . The method of  claim 1  further comprising training the neural network by updating parameters of the neural network to reduce prediction error. 
     
     
         13 . The method of  claim 1  wherein:
 the environment data for the first time comprises a plurality of observed features for the first time; 
 the predicted environment data for the second time comprises a plurality of sub-predictions for the second time; 
 the environment data for the second time comprises a plurality of observed features for the second time; and 
 the prediction error comprises a weighted combination of sub-prediction errors, each sub-prediction error relating to a corresponding pair of observation and sub-prediction for the second time. 
 
     
     
         14 . A computing system for detecting edge cases, the computing system comprising one or more processors configured to:
 obtain environment data for a first time detailing one or more observed features of a vehicle environment at the first time;   process the environment data for the first time using a neural network to determine predicted environment data for a second time that is later than the first time;   obtain environment data for the second time detailing one or more observed features of the vehicle environment at the second time;   determine a prediction error based on the predicted environment data and the environment data for the second time; and   in response to the prediction error exceeding a threshold, output an alert indicating detection of an edge case.   
     
     
         15 . A computer readable medium comprising computer executable instructions that, when executed by a processor, cause the processor to:
 obtain environment data for a first time detailing one or more observed features of a vehicle environment at the first time;   process the environment data for the first time using a neural network to determine predicted environment data for a second time that is later than the first time;   obtain environment data for the second time detailing one or more observed features of the vehicle environment at the second time;   determine a prediction error based on the predicted environment data and the environment data for the second time; and   in response to the prediction error exceeding a threshold, output an alert indicating detection of an edge case.   
     
     
         16 . A computer-implemented method for alerting a vehicle control system to potential danger, the method comprising:
 obtaining environment data for a first time detailing one or more observed features of a vehicle environment at the first time;   processing the environment data for the first time using a neural network to determine predicted environment data for a second time that is later than the first time;   determining whether the predicted environment data indicates a predicted environmental state associated with danger; and   in response to the determining that predicted environment data indicates an environmental state associated with danger, issuing an alert to a vehicle control system to recommend taking remedial action to adapt to the predicted environmental state.   
     
     
         17 . The method of  claim 16  wherein determining whether the predicted environment data indicates a predicted environmental state associated with danger comprises one or more of:
 determining that the predicted environment data comprises a predicted emergency action by a driver of the vehicle or the vehicle control system; 
 determining that the predicted environment data indicates a predicted collision with an object external to the vehicle; and 
 determining that the predicted environment data comprises data indicative of a high risk object.