Situational awareness systems and methods and micromobility platform

Abstract

A system for providing situational awareness to a cyclist or other user of a micromobility vehicle comprises a stereo camera assembly and processing logic configured to determine, based on images acquired by the stereo camera assembly, a distance between the cyclist and an object of interest (e.g., a vehicle). The system is configured to determine a threat level of the object based one or more factors such as, e.g., a speed of the object and/or a category of the object. In some examples, the system includes a display and/or an audio indicator to convey information to the cyclist about detected threats. In some examples, the system is configured to produce an audio indication in response to a threat exceeding a threshold threat level. A software platform may be configured to store and/or process micromobility data gathered from one or more users.

The invention claimed is: 
     
       1. A system for providing situational awareness, the system comprising:
 a stereo camera assembly comprising a first camera and a second camera disposed in a stereo configuration and configured to be mounted to a vehicle; and 
 processing logic coupled to the first and second cameras, the processing logic configured to:
 control the first and second cameras to acquire respective first and second stereo images, wherein each of the first and second stereo images includes an object of interest; 
 determine a bounding shape encompassing the object of interest within the first stereo image, the bounding shape having a first position within the first stereo image; 
 determine a second position within the second stereo image at which the bounding shape encompasses the object of interest in the second stereo image; 
 determine a displacement between the first position and the second position; 
 determine, based on the displacement, a distance between the stereo camera assembly and the object of interest; 
 based on at least one of the first and second stereo images, classify the object of interest as belonging to a first category of a plurality of categories, wherein the plurality of categories includes at least one of “motor vehicle” and “bicycle”; 
 determine, based on at least one of the first and second stereo images, an angular position of the object of interest relative to the stereo camera assembly; and 
 determine, based on a plurality of factors including the first category and the angular position, a threat level indicating a degree of threat posed to the vehicle by the object of interest. 
 
 
     
     
       2. The system of  claim 1 , wherein the first position and the second position are each defined by an index identifying an image pixel at which a predetermined portion of the bounding shape is disposed, such that determining the first and second positions comprises determining the respective indices. 
     
     
       3. The system of  claim 1 , wherein the bounding shape is a rectangular box. 
     
     
       4. The system of  claim 1 , wherein the processing logic is further configured to:
 control the first and second cameras to, after acquiring the first and second stereo images, acquire updated first and second stereo images; 
 determine, based on the updated first and second stereo images, an updated distance between the stereo camera assembly and the object of interest; and 
 determine, based on the distance, the updated distance, and a time interval between acquiring the first and second stereo images and acquiring the updated first and second stereo images, a speed of the object of interest. 
 
     
     
       5. The system of  claim 4 , wherein the plurality of factors further includes the speed. 
     
     
       6. The system of  claim 1 , further comprising a sound-emitting device, and wherein the processing logic is further configured to:
 in response to identifying a threat level meeting or exceeding a predetermined threshold threat level, cause the sound-emitting device to emit an audio indication. 
 
     
     
       7. The system of  claim 6 , wherein at least one property of the audio indication is selected based on the threat level. 
     
     
       8. A system for providing situational awareness for a user of a personal vehicle, the system comprising:
 a stereo camera assembly configured to capture pairs of stereo images; and 
 a processor configured to:
 cause the stereo camera assembly to capture successive pairs of stereo images, each pair of stereo images including a first image and a second image; 
 detect a region of interest corresponding to at least a portion of a real-world object appearing in the first and second images of each of the successive pairs of stereo images; 
 for each pair of the successive pairs of stereo images:
 determine a bounding outline bounding the region of interest in the first image, wherein determining the bounding outline includes determining a first position of the bounding outline in the first image, and wherein the bounding outline has first image content when disposed at the first position within the first image; 
 determine a second position of the bounding outline in the second image such that, when the bounding outline is disposed at the second position in the second image, the bounding outline has second image content matching the first image content according to a predetermined criterion; 
 determine, based on the first position and the second position, a stereo disparity of the detected region of interest between the first image and the second image; and 
 determine, based on the stereo disparity and a stereo baseline of the stereo camera assembly, a distance between the stereo camera assembly and the real-world object at the time the pair of stereo images was captured; 
 
 determine, based on the distances determined for each of the successive pairs of stereo images, a speed of the real-world object relative to the stereo camera assembly; 
 determine, based on at least one of the first or second images of at least one of the successive pairs of stereo images, an azimuthal position of the real-world object relative to the stereo camera assembly; and 
 determine, based on a plurality of factors including the speed and the azimuthal position, a threat level of the real-world object. 
 
 
     
     
       9. The system of  claim 8 , wherein the processor is further configured to classify the real-world object as belonging to a first category of a plurality of predetermined categories of object, and wherein the plurality of predetermined categories includes at least the following categories: motor vehicle, bicycle. 
     
     
       10. The system of  claim 8 , wherein a ratio of at least one of the distances to the stereo baseline of the stereo camera assembly is at least 125 to 1. 
     
     
       11. A method for providing situational awareness for a user of a micromobility vehicle, the method comprising:
 using a stereo image capture device traveling with the micromobility vehicle, capturing a first pair of stereo images at a first time and capturing a second pair of stereo images at a second time, the second time being later than the first time, and each of the first and second pairs of stereo images including a respective first image and a respective second image; 
 for each of the first and second pairs of stereo images:
 determining a boundary encompassing an object in the first image of the pair; 
 determining a disparity between a first position of the boundary, at which the boundary encompasses the object in the first image, and a second position of the boundary, at which the boundary encompasses the object in the second image; and 
 determining, based on the disparity, a distance from the object; and 
 
 determining, based on the distances determined for the first and second pairs of stereo images, a rate of approach of the object; 
 wherein the boundary has first image content when disposed at the first position in the first image, and wherein determining the disparity between the first and second positions includes rasterinq the boundary across the second image between a plurality of positions including at least the second position, and identifying the second position as the position of the plurality of positions at which contents of the boundary have a greatest correlation with the first image content. 
 
     
     
       12. The method of  claim 11 , further comprising:
 based on at least one of the first and second images of at least one of the first and second pairs of stereo images, determining an angular position of the object relative to the micromobility vehicle; and 
 determining a threat level posed by the object to the micromobility vehicle based on the rate of approach and the angular position of the object. 
 
     
     
       13. The method of  claim 12 , further comprising displaying, on a display traveling with the micromobility vehicle, first data including the threat level and a relative position of the object to the micromobility vehicle, the relative position being based on the distance from the object and the angular position of the object. 
     
     
       14. The method of  claim 12 , further comprising producing an auditory alert in response to determining a threat level meeting or exceeding a threshold threat level. 
     
     
       15. The system of  claim 1 , wherein the stereo camera assembly is configured to be mounted facing backward on the vehicle, such that the stereo camera assembly acquires images of an area behind the vehicle. 
     
     
       16. The system of  claim 2 , wherein the bounding shape has first image content when disposed at the first position in the first image and second image content when disposed at the second position in the second image, and wherein determining the second position includes:
 shifting the bounding shape across the second image among a plurality of positions including the second position, and 
 selecting the second position from the plurality of positions based on a correlation between the first image content and the second image content. 
 
     
     
       17. The system of  claim 9 , wherein the plurality of factors further includes the first category. 
     
     
       18. The system of  claim 8 , further comprising producing an auditory alert in response to the threat level meeting or exceeding a threshold threat level. 
     
     
       19. The system of  claim 18 , wherein at least one property of the produced auditory alert is based on the threat level. 
     
     
       20. The system of  claim 8 , wherein determining the second position of the bounding outline in the second image includes:
 shifting the bounding outline across the second image between a plurality of positions including at least the second position, 
 at each of the plurality of positions, evaluating a match between the first image content and current image content of the bounding outline; and 
 determining that the second image content of the bounding outline when disposed at the second position matches the first image content according to the predetermined criterion.