Systems and Methods for Image-Based Location Determination and Parking Monitoring

Abstract

Embodiments relate to systems, methods and computer readable media for parking monitoring in an urban area by image processing operations. Embodiments perform parking monitoring by capturing images of an urban area, comparing captured images with reference images to determine location and parking conditions. Embodiments processes captured images to detect licence plates, vehicles or parking signs to determine compliance of vehicles with parking conditions.

1 . A system for parking monitoring in an urban area, the system comprising:
 at least one camera, wherein the at least one camera is positioned to capture images of the urban area;   a computing device in communication with the at least one camera to receive the captured images;   the computing device comprising at least one processor and a memory accessible to the at least one processor;   wherein the memory comprises a library of reference background images and metadata for each reference background image, wherein the metadata comprises parking location information and parking condition information;   wherein the memory stores program code executable by the at least one processor to configure the at least one processor to:
 process a first captured image to determine a licence plate number corresponding to a target vehicle in the first captured image; 
 process a second captured image using a background matching module to identify a matching reference background image; 
 determine an identified parking location of the target vehicle and at least one parking condition based on the metadata of the matching reference background image; 
 determine compliance of the target vehicle with the determined at least one parking condition. 
   
     
     
         2 . The system of  claim 1 , wherein the first captured image is the same captured image as the second captured image. 
     
     
         3 . A system for parking monitoring in an urban area, the system comprising:
 at least one camera, wherein the at least one camera is positioned to capture images of the urban area;   a computing device in communication with the at least one camera to receive the captured images;   the computing device comprising at least one processor and a memory accessible to the at least one processor;   wherein the memory comprises a library of reference background images and metadata for each reference background image, wherein the metadata comprises parking location information;   the memory also comprises program code executable by the at least one processor to configure the at least one processor to:
 process the captured images using a parking indicator detection machine learning model to identify a parking indicator in at least one of the captured images; 
 on identifying the parking indicator, process the captured images using a background matching module to identify a matching reference background image that matches one of the captured images; 
 determine a parking location based on the metadata associated with the matching reference background image; 
 determine parking conditions based on the identified parking indicator; 
 process the captured images to determine a licence plate number corresponding to a target vehicle; and 
 determine compliance of the target vehicle to the determined parking conditions. 
   
     
     
         4 . The system of any one of  claims 1  to  3 , wherein the licence plate number corresponding to the target vehicle is determined using a licence plate detection machine learning model. 
     
     
         5 . The system of  claim 3  or  claim 4 , wherein the parking indicators comprise parking signs or licence plates and the parking indicator detection machine learning model detects parking signs or licence plates in the captured images. 
     
     
         6 . The system of any one of  claims 1  to  5 , wherein the memory further comprises parking perimeter metadata associated with each reference background image, and the at least one processor is further configured to:
 process the at least one captured image to identify an image portion corresponding to the target vehicle in one of the captured images; 
 determine compliance of the target vehicle to the determined parking conditions based on the parking perimeter metadata associated with the matching reference background image and the image portion corresponding to the target vehicle. 
 
     
     
         7 . The system of  claim 6 , wherein the image portion corresponding to the target vehicle is identified using a vehicle detection machine learning model. 
     
     
         8 . The system of any one of  claims 1  to  7 , wherein the background matching module comprises: a background feature extractor neural network, and the at least one processor is further configured to identify the matching reference background image by:
 extracting background descriptors from the at least one captured image using the background feature extractor neural network; 
 selecting one or more candidate matching images from the library of background images based on the extracted background descriptors; 
 performing geometric matching between the at least one captured image and the candidate matching images to select the matching reference background image. 
 
     
     
         9 . The system of  claim 8 , wherein the geometric matching is performed using a random sample consensus process. 
     
     
         10 . The system of any one of  claims 1  to  9 , wherein:
 the parking location information comprises a street name and a street number; or 
 the parking location information comprises a street name, a street number and a parking bay identifier; or 
 the parking location information comprises a longitude coordinate and a latitude coordinate associated with the parking location. 
 
     
     
         11 . The system of any one of  claims 1  to  10 , wherein the one or more cameras are mounted on a surveillance vehicle,
 the computing device is carried by the surveillance vehicle, and 
 the system further comprises a communication module to enable wireless communication between the computing device and a remote computer system. 
 
     
     
         12 . The system of  claim 11 , wherein the system is configured to perform parking monitoring in real time as the surveillance vehicle moves in the urban area. 
     
     
         13 . The system of  claim 11  or  claim 12 , wherein the system comprises at least two cameras, with at least one camera positioned on each side of the surveillance vehicle to perform parking monitoring on both sides of the surveillance vehicle. 
     
     
         14 . The system of  claim 11  or  claim 12 , wherein the system comprises at least two cameras, both cameras are positioned to capture images on one side of the surveillance vehicle; and
 the background matching module is configured to perform background matching using captured images from each of the at least two cameras to identify a matching reference background image. 
 
     
     
         15 . The system of any one of  claims 11  to  14 , wherein the at least one processor is further configured to track the target vehicle across the captured images as the surveillance vehicle moves in the urban area. 
     
     
         16 . The system of  claim 11 , wherein the at least one processor is further configured to transmit to the remote computer system via the communication module one or more of:
 the determined compliance of the target vehicle with the determined parking conditions;   the determined licence plate number corresponding to the target vehicle;   the determined parking location; or   captured images of the target vehicle.   
     
     
         17 . The system of  claim 4 , wherein the licence plate detection machine learning model is configured to identify a portion of the captured image corresponding to a license plate of the target vehicle, and
 the licence plate number is determined based on processing the portion of the captured image corresponding to the license plate by a character recognition module.   
     
     
         18 . The system of  claim 3 , wherein the parking conditions are determined based on characters recognised by processing a portion of the at least one captured image corresponding to the identified parking signs using a character recognition module. 
     
     
         19 . The system of any one of  claims 1  to  18 , wherein at least one reference background image relates to a parking zone start location and another at least one reference background image relates to a parking zone end location, and
 determination of compliance of the target vehicle to the identified at least one parking condition is based on:
 distance between the identified parking location and the parking zone start location; or 
 distance between the identified parking location and the parking zone end location. 
 
 
     
     
         20 . A computer implemented method for parking monitoring performed by a computing device comprising at least one processor in communication with a memory, the memory comprising a library of reference background images, the method comprising:
 receiving images of an urban area captured by a camera in communication with the computing device;   processing the captured images using a parking indicator detection machine learning model to identify one or more parking indicators in at least one captured image;
 on identifying at least one parking indicator in the at least one captured image, process the at least one captured image using a background matching module to identify a matching reference background image; 
 determining a parking location based on the matching reference background image; 
 determining parking conditions based on the determined parking location or the identified one or more parking indicators; 
 processing the at least one captured image to determine a licence plate number corresponding to a target vehicle in the at least one captured image; and 
 determining compliance of the target vehicle to the determined parking conditions based on the determined licence plate number and the determined parking conditions. 
   
     
     
         21 . A system for location determination, the system comprising:
 a computing device comprising at least one processor and a memory accessible to the at least one processor;   wherein the memory comprises a library of reference background images and metadata for each reference background image, wherein the metadata comprises location information; and   wherein the memory stores program code executable by the at least one processor to configure the at least one processor to:
 receive an input image data from a remote computing device, wherein the input image data includes image data of at least one image captured by the remote computing device at a location to be determined; 
 process the received input image data using a background matching module to identify matching reference background image; 
 determine location information corresponding to the input image data based on the metadata of the matching reference background image in the library; and 
 transmit the determined location information to the remote computing device. 
   
     
     
         22 . The system of  claim 21 , wherein the background matching module comprises: a background feature extractor neural network, and the at least one processor is further configured to identify the matching reference background image by:
 extracting background descriptors from the at least one captured image using the background feature extractor neural network;   selecting one or more candidate matching images from the library of background images based on the extracted background descriptors;   performing geometric matching between the at least one captured image and the candidate matching images to select the matching reference background image.   
     
     
         23 . The system of  claim 22 , wherein the geometric matching comprises identifying common visual features in the at least one captured image and each of the candidate matching images. 
     
     
         24 . The system of  claim 22  or  claim 23 , wherein the geometric matching is performed using a random sample consensus process. 
     
     
         25 . The system of any one of  claims 22  to  24 , wherein the background feature extractor neural network is trained to extract background descriptors corresponding to one or more stationary features in the at least one captured image. 
     
     
         26 . The system of any one of  claims 21  to  25 , wherein the memory stores program code executable by the at least one processor to further configure the at least one processor to:
 receive GPS data corresponding to the input image from the remote computing device, wherein the GPS data comprises a low data quality indicator; 
 generate a GPS correction signal based on the determined location information; 
 transmit the GPS correction signal to the remote computing device. 
 
     
     
         27 . A vehicle mounted system for location determination in an urban area, the system comprising:
 at least one camera, wherein the at least one camera is positioned to capture images of the urban area;   a computing device in communication with the at least one camera to receive the captured images, the computing device comprising at least one processor and a memory accessible to the at least one processor, the memory comprising a library of reference background images;   wherein the memory comprises program code executable by the at least one processor to configure the at least one processor to:   extract background descriptors from at least one captured image;   select one or more candidate matching reference images from the library of background images based on the extracted background descriptors;   perform geometric matching between the at least one captured image and the one or more candidate matching reference images to select a single matching reference background image; and   determine a location of the vehicle based on the single matching reference background image.   
     
     
         28 . The system of  claim 27 , wherein the memory further comprises location metadata corresponding to each reference background image; and
 the location of the vehicle is determined based on the location metadata corresponding to corresponding to the single matching reference background image.   
     
     
         29 . The system of  claim 27  or  claim 28 , wherein the background descriptors are extracted from at least one captured image using a background feature extractor neural network. 
     
     
         30 . The system of any one of  claims 27  to  29 , wherein determining the location is performed in real time. 
     
     
         31 . A computer implemented method for determining a location of a vehicle, the method performed by a vehicle mounted computing device comprising at least one processor in communication with a memory, the memory comprising a library of reference background images, the method comprising:
 receiving images of an urban area captured by a camera in communication with the computing device;   extracting background descriptors from at least one image captured by the camera;   selecting one or more candidate matching reference images from the library of background images based on the extracted background descriptors;   perform geometric matching between the at least one captured image and the one or more candidate matching reference images to select a single matching reference background image; and   determine a location of the vehicle based on the single matching reference background image.   
     
     
         32 . A computer implemented method for determining a location of a vehicle in an urban area, the method performed by a vehicle mounted computing device comprising at least one processor in communication with a memory and at least one camera, the memory comprising a library of reference background images, the method comprising:
 capturing an image of the urban area while the at least one camera is moving in the urban area;   processing the captured image using a background matching module to identify a matching reference background image;   determining a location of the vehicle based on a metadata of the matching reference background image.   
     
     
         33 . The method of  claim 32 , wherein the at least one camera is mounted on the vehicle. 
     
     
         34 . The method of  claim 32  or  claim 33 , wherein the determination of the location of the vehicle is performed in real time by the vehicle mounted computing device. 
     
     
         35 . A system for location determination in an urban area, the system comprising:
 at least one camera, wherein the at least one camera is positioned to capture images of the urban area while the at least one camera is moving in the urban area;   a computing device moving with the at least one camera and in communication with the at least one camera to receive the captured images;   the computing device comprising at least one processor and a memory accessible to the at least one processor;   wherein the memory comprises a library of reference background images and metadata for each reference background image, wherein the metadata comprises location information;   wherein the memory stores program code executable by the at least one processor to configure the at least one processor to:
 process a captured image using a background matching module to identify a matching reference background image; 
 determine a location of the at least one camera and the computing device based on the metadata of the matching reference background image. 
   
     
     
         36 . The system of  claim 35 , wherein processing the captured image using a background matching module comprises:
 extracting background descriptors from the captured image;   selecting one or more candidate matching images from the library of reference background images based on the extracted background descriptors;   performing geometric matching between the captured image and the candidate matching images to select the matching reference background image.   
     
     
         37 . The system of  claim 36 , wherein the background matching module comprises a background feature extractor neural network configured to extract background descriptors corresponding to one or more stationary features in the at least one captured image. 
     
     
         38 . The system of  claim 36  or  claim 37 , wherein the geometric matching is performed using a random sample consensus process; and
 wherein the geometric matching comprises identifying common visual features in the at least one captured image and each of the candidate matching images. 
 
     
     
         39 . The system of any one of  claims 35  to  38 , wherein the computing device is configured to determine the location in real-time. 
     
     
         40 . A vehicle mounted with the system of any one of  claims 35  to  39 , wherein the at least one camera is mounted on the vehicle to capture images of a vicinity of the vehicle. 
     
     
         41 . The vehicle of  claim 40 , wherein the vehicle is an autonomous driving vehicle. 
     
     
         42 . The vehicle of  claim 41 , wherein the vehicle comprises an on-board GPS receiver and the vehicle is configured to trigger location determination using the system for location determination in response to an image based location determination trigger event. 
     
     
         43 . The vehicle of  claim 42 , wherein the image based location determination trigger event may comprise at least one of:
 low precision GPS data being generated by the on-board GPS receiver; or   crossing of a predefined geo-fence by the vehicle.   
     
     
         44 . A computer implemented method for location determination, the method performed by a computing device comprising at least one processor in communication with a memory, the method comprising:
 receiving an input image by the computing device from a remote computing device, wherein the input image corresponds to a location to be determined;   processing the received input image using a background matching module provided in the memory of the computing device to identify a matching reference background image from among a library of reference background images stored in the memory;   determining location information corresponding to the input image based on the metadata of the matching reference background image; and   transmitting the determined location information to the remote computing device.   
     
     
         45 . The method of  claim 44 , wherein the background matching module comprises:
 a background feature extractor neural network, and the method further comprises identifying the matching reference background image by:
 extracting background descriptors from the at least one captured image using the background feature extractor neural network; 
 selecting one or more candidate matching images from the library of background images based on the extracted background descriptors; 
 performing geometric matching between the at least one captured image and the candidate matching images to select the matching reference background image. 
   
     
     
         46 . The method of  claim 45 , wherein the geometric matching comprises identifying common visual features in the at least one captured image and each of the candidate matching images. 
     
     
         47 . The method of  claim 45  or  claim 46  wherein the geometric matching is performed using a random sample consensus process. 
     
     
         48 . The method of any one of  claims 45  to  47 , wherein the background feature extractor neural network is trained to extract background descriptors corresponding to one or more permanent stationary features in the at least one captured image. 
     
     
         49 . The method of any one of  claims 45  to  48 , wherein the method further comprises:
 receiving GPS data corresponding to the input image from the remote computing device, wherein the GPS data comprises a low data quality indicator; 
 generating a GPS correction signal based on the determined location information; 
 transmitting the GPS correction signal to the remote computing device; 
 wherein the GPS correction signal comprises information accessible by the remote computing device to determine a more accurate GPS location data. 
 
     
     
         50 . A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform the method of any one of  claims 20 ,  31  to  34  and  44  to  49 . 
     
     
         51 . A system for parking monitoring in an urban area, the system comprising:
 at least one camera, wherein the at least one camera is positioned to capture images of the urban area;   a computing device in communication with the at least one camera to receive the captured images;   the computing device comprising at least one processor and a memory accessible to the at least one processor;   wherein the memory comprises a library of reference background images;   the memory also comprises program code executable by the at least one processor to configure the at least one processor to:
 process the captured images using a parking indicator detection machine learning model to identify one or more parking indicators in at least one captured image; 
 on identifying at least one parking indicator in the at least one captured image, process the at least one captured image using a background matching module to identify a matching reference background image; 
 determine a parking location based on the matching reference background image; 
 determine parking conditions based on the determined parking location or the identified one or more parking indicators; 
 process the at least one captured image to determine a licence plate number corresponding to a target vehicle in the at least one captured image; and 
 determine compliance of the target vehicle to the determined parking conditions based on the determined licence plate number and the determined parking conditions.