Method and system for vision-based vehicle interior environment sensing guided by vehicle prior information

Abstract

A method for operating a vehicle including a vehicle sensing system includes generating a baseline image model of an cabin of the vehicle based on image data of the cabin of the vehicle generated by an imaging device of the vehicle sensing system, the baseline image model generated before a passenger event, and generating an event image model of the cabin of the vehicle based on image data of the cabin of the vehicle generated by the imaging device, the event image model generated after the passenger event. The method further includes identifying image deviations by comparing the event image model to the baseline image model with a controller of the vehicle sensing system, the image deviations corresponding to differences in the cabin of the vehicle from before the passenger event to after the passenger event, and operating the vehicle based on the identified image deviations.

What is claimed is: 
     
       1. A method for operating a vehicle including a vehicle sensing system, the method comprising:
 generating a plurality of baseline image models of a cabin of the vehicle based on image data of the cabin of the vehicle generated by an imaging device of the vehicle sensing system, the plurality of baseline image models generated before a passenger event; 
 generating an event image model of the cabin of the vehicle based on image data of the cabin of the vehicle generated by the imaging device, the event image model generated after the passenger event; 
 detecting a vehicle configuration based on vehicle setting data; 
 selecting a baseline image model from the plurality of baseline image models for comparison with the event image model based on the detected vehicle configuration; 
 identifying image deviations by comparing the event image model with the selected baseline image model with a controller of the vehicle sensing system, the image deviations corresponding to differences in the cabin of the vehicle from before the passenger event to after the passenger event; and 
 operating the vehicle based on the identified image deviations. 
 
     
     
       2. The method as claimed in  claim 1 , wherein identifying image deviations further comprises:
 identifying an item located in the cabin of the vehicle after the passenger event that was not located in the cabin of the vehicle before the passenger event. 
 
     
     
       3. The method as claimed in  claim 1 , wherein generating the event image model of the cabin further comprises:
 generating at least one high-dynamic-range (HDR) image of the cabin of the vehicle, 
 wherein the at least one HDR image is configured to reduce an impact of environmental lighting on the generation of the event image model of the cabin of the vehicle. 
 
     
     
       4. The method as claimed in  claim 3 , wherein:
 generating the event image model of the cabin further comprises performing an image decomposition de-lighting process to decompose the at least one HDR image into a decomposed image including a reflectance layer and a shading layer, and 
 identifying the image deviations further comprises comparing the decomposed image with a reference image included in the selected baseline image model of the cabin of the vehicle. 
 
     
     
       5. The method as claimed in  claim 4 , wherein identifying the image deviations further comprises:
 uses a per-pixel or a graph-based system to detect the deviations during the comparison of the decomposed image with the reference image. 
 
     
     
       6. The method as claimed in  claim 1 , wherein operating the vehicle based on the identified image deviations comprises:
 generating notification data corresponding to the identified image deviations, and 
 transmitting the notification data to an electronic device of a passenger associated with the passenger event. 
 
     
     
       7. The method as claimed in  claim 1 , wherein operating the vehicle based on the identified image deviations comprises:
 causing the vehicle to travel autonomously to a service center. 
 
     
     
       8. A vehicle sensing system for a corresponding vehicle, the vehicle sensing system comprising:
 an imaging device configured to generate image data of a cabin of the vehicle; 
 a memory configured to store a plurality of baseline image models of the cabin of the vehicle that is generated prior to a passenger event; and 
 a controller operably connected to the imaging device and the memory, the controller configured to (i) generate an event image model of the cabin of the vehicle based on the generated image data after the passenger event, (ii) detect a vehicle configuration based on vehicle setting data, (iii) select a baseline image model from the plurality of baseline image models for comparison with the event image model based on the detected vehicle configuration, (iv) identify image deviations by comparing the event image model with the selected baseline image model, and (v) operate the vehicle based on the identified image deviations, 
 wherein the image deviations correspond to differences in the cabin of the vehicle from before the passenger event to after the passenger event. 
 
     
     
       9. The vehicle sensing system as claimed in  claim 8 , wherein the identified image deviations correspond to damage to the cabin of the vehicle and/or to an item located in the cabin of the vehicle after the passenger event that was not located in the cabin of the vehicle before the passenger event. 
     
     
       10. The vehicle sensing system as claimed in  claim 8 , wherein:
 the generated image data includes at least one high-dynamic-range (HDR) image of the cabin of the vehicle, and 
 the at least one HDR image is configured to reduce an impact of environmental lighting on the generation of the event image model by the controller. 
 
     
     
       11. The vehicle sensing system as claimed in  claim 10 , wherein the controller is further configured (i) to generate the event image model of the cabin by performing an image decomposition de-lighting process to decompose the at least one HDR image into a decomposed image including a reflectance layer and a shading layer, and (ii) to identify the image deviations by comparing the decomposed image with a reference image included in the selected baseline image model of the cabin of the vehicle. 
     
     
       12. The vehicle sensing system as claimed in  claim 11 , wherein the controller is configured to use at least one of a per-pixel and a graph-based system to detect the deviations during the comparison of the decomposed image with the reference image. 
     
     
       13. The vehicle sensing system of  claim 8 , further comprising:
 a transceiver operably connected to the controller, 
 wherein the controller is further configured to generate notification data corresponding to the identified image deviations, and 
 wherein the transceiver is configured to transmit the notification data to an electronic device of a passenger associated with the passenger event. 
 
     
     
       14. The vehicle sensing system of  claim 8 , wherein the controller is further configured to cause the vehicle to travel autonomously to a service center based on the identified image deviations. 
     
     
       15. The vehicle sensing system of  claim 8 , wherein the controller is further configured to prevent further passenger events from occurring based on the identified image deviations.