Augmented reality assisted parking with driver-in-loop selections and alignments of feature point vicinities

Abstract

A system for augmented-reality (AR)-assisted vehicle parking with operator-in-loop selections and alignments of feature point vicinities (FPVs) includes a vehicle, human-machine interfaces (HMIs), sensors detecting FPVs of a parking location and controllers. The controllers have a processor, memory, and input/output (I/O) ports in communication with the HMIs and sensors. The memory stores an AR-assisted parking application (ARAPA) executed by the processor. The AR-assisted parking application (ARAPA) has a training session (TS) and a live session (LS), and is activated upon completing parking at a parking spot at the parking location. The TS portion trains the ARAPA to recognize FPVs of the parking spot. The LS portion prompts an operator to select the parking spot, guides the operator into the parking spot by displaying trained FPVs onto live sensor data displayed on the HMI, and instructs the operator to align the trained FPVs with corresponding real-world FPVs by maneuvering the vehicle.

What is claimed is: 
     
         1 . A system for augmented-reality (AR)-assisted vehicle parking with operator-in-loop selections and alignments of feature point vicinities (FPVs), the system comprising:
 a vehicle;   one or more human-machine interfaces (HMIs) disposed within the vehicle;   one or more sensors disposed on the vehicle and detecting FPVs of a parking location;   one or more controllers, each of the one or more controllers having a processor, a memory, and one or more input/output (I/O) ports, the I/O ports in communication with the one or more sensors and the one or more HMIs; the memory storing programmatic control logic; the processor executing the programmatic control logic; the programmatic control logic including an AR-assisted parking application (ARAPA), a vehicle operator activating the ARAPA via the HMI upon completing parking at a parking spot at the parking location; the ARAPA comprising:   a training session (TS) portion; and   a live session (LS) portion,   wherein in the TS portion, the ARAPA is trained to recognize FPVs of the parking spot at the parking location; wherein the ARAPA suggests, via the HMI, FPVs and semantic tags corresponding to parking spots that the system has previously identified at the parking location; offers, via the HMI, automatically generated proposed parking spot names and opportunities for the vehicle operator to alter the proposed parking spot names; and stores the proposed parking spot names and semantic tags in memory for use in the LS portion of the ARAPA; and   wherein in the LS portion, the ARAPA prompts a vehicle operator to select the parking spot, and guides the vehicle operator into the parking spot by displaying previously trained FPVs overlaid onto live sensor data displayed on the HMI and instructing the vehicle operator to align the previously trained FPVs with corresponding real-world FPVs of the parking spot by maneuvering the vehicle.   
     
     
         2 . The system of  claim 1 , wherein the TS portion further comprises:
 a first TS control logic portion wherein the vehicle operator indicates that the vehicle has been parked in a preferred parking spot by selecting a first feature of the ARAPA via the HMI.   
     
     
         3 . The system of  claim 2 , wherein the TS portion further comprises:
 a second TS control logic portion wherein the ARAPA utilizes the HMI to present a plurality of proposed FPVs overlaid on a real-time display of a view from the one or more sensors, wherein the proposed FPVs at least partially define a current or an idealized position of the vehicle within the preferred parking spot; and   control logic that receives, via the HMI, inputs from the vehicle operator selecting some or all of the plurality of proposed FPVs, each of the proposed FPVs having appropriate separations from other proposed FPVs, wherein the appropriate separation is chosen such that all selected FPVs may be simultaneously displayed on the HMI and so that different FPVs do not overlap at any line of sight of the vehicle operator; and wherein the appropriate separation reduces a potential for vehicle mis-alignment within the parking spot from a first level to a second level less than the first level as the vehicle is parked.   
     
     
         4 . The system of  claim 1 , wherein the LS portion further comprises:
 a first LS control logic portion that determines that a current distance between the vehicle and one or more previously trained parking spots is less than or equal to a predefined FPV preview threshold distance, wherein the FPV preview threshold distance is between approximately six meters and approximately 10 meters; and   a second LS control logic portion that offers, via the HMI, one or more prompts to the vehicle operator, wherein the one or more prompts request the vehicle operator to select one of the one or more previously trained parking spots within the FPV preview threshold distance that the vehicle operator intends to park within.   
     
     
         5 . The system of  claim 4 , wherein the LS portion further comprises:
 a third LS control logic portion that provides augmented reality (AR) assistance to the vehicle operator as the vehicle operator provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle towards and into the previously trained parking spot, wherein the AR assistance overlays FPVs of the previously trained parking spot on one or more of live sensor data displayed on the HMI and on a heads-up display (HUD) of the vehicle so that the FPVs are displayed directly in a line-of-sight of the vehicle operator and in positions corresponding to actual physical locations of the FPVs from an optical perspective of the vehicle operator.   
     
     
         6 . The system of  claim 5 , wherein the LS portion further comprises:
 a fourth LS control logic that determines that the vehicle is within the FPV preview threshold distance of one or more previously saved FPVs of parking spots in a parking location; and   a fifth LS control logic that conditionally displays a preview of the identified previously saved FPVs on one or more of the HMI and a head-sup-display (HUD), and uses previously-identified FPVs as enhanced steering guidance for the vehicle operator to align the vehicle within a parking spot to a distance that is equal to or less than a predefined FPV alignment threshold distance.   
     
     
         7 . The system of  claim 6 , wherein the LS portion further comprises:
 a sixth LS control logic that provides AR assistance to the vehicle operator as the vehicle operator provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle towards and into the parking spot identified by saved previously-identified FPVs, wherein the AR assistance overlays the saved previously-identified FPVs on one or more of live sensor data displayed on the HMI and on the HUD of the vehicle so that the FPVs are displayed directly in a vehicle operator's line-of-sight and in positions corresponding to actual physical locations of the FPVs from the vehicle operator's optical perspective, and wherein a path to the saved previously-identified FPVs is displayed on one or more of the HUD and the HMI in a different style, color, or layout, in comparison to a path to a previously trained parking spot.   
     
     
         8 . The system of  claim 7 , wherein the LS portion further comprises:
 a seventh LS control logic that determines a current distance of the vehicle to a parking spot is less than or equal to the FPV alignment threshold distance and engages an FPV alignment mode of the ARAPA; and   an eighth LS control logic that utilizes AR to display previously-identified or saved FPVs on one or more of the HMI and the HUD, thereby providing the vehicle operator with references that the vehicle operator utilizes the align the vehicle within the parking spot in real-time;   a ninth LS control logic that causes the vehicle operator to check alignments of the saved FPVs with corresponding features of a real-world environment for enhanced parking assistance and steers, or otherwise maneuvers the vehicle; and   a tenth LS control logic that notifies the vehicle operator that the vehicle has been completely parked when the saved FPV are aligned to real-world FPVs or environmental objects as viewed on either or both of the HMI and the HUD.   
     
     
         9 . A method for augmented-reality (AR)-assisted vehicle parking with operator-in-loop selections and alignments of feature point vicinities (FPVs), the method comprising:
 detecting FPVs of a parking location with one or more sensors disposed on a vehicle;   executing programmatic control logic stored within a memory of one or more controllers in communication with the one or more sensors and in communication with one or more human-machine interfaces (HMIs) disposed within the vehicle, each of the one or more controllers having a processor, the memory, and one or more input/output (I/O) ports, the I/O ports in communication with the one or more sensors and the one or more HMIs; the processor executing control logic including an AR-assisted parking application (ARAPA);   activating, via a vehicle operator input through the HMI, the ARAPA upon completing parking at a parking spot at the parking location; the ARAPA comprising:   training the ARAPA, in a training session (TS) portion, to recognize FPVs of a parking spot at the parking location; and   prompting the vehicle operator, in a live session (LS) portion of the ARAPA, to select the parking spot; wherein the LS portion further comprises:
 determining that a current distance between the vehicle and one or more previously trained parking spots is less than or equal to a predefined FPV preview threshold distance, wherein the FPV preview threshold distance is between approximately six meters and approximately 10 meters; 
 offering, via the HMI, one or more prompts to the vehicle operator, wherein the one or more prompts request the vehicle operator to select one of the one or more previously trained parking spots within the FPV preview threshold distance that the vehicle operator intends to park within; and 
 providing augmented reality (AR) assistance to the vehicle operator as the vehicle operator provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle towards and into the previously trained parking spot, wherein the AR assistance overlays FPVs of the previously trained parking spot on one or more of live sensor data displayed on the HMI and on a heads-up display (HUD) of the vehicle so that the FPVs are displayed directly in a line-of-sight of the vehicle operator and in positions corresponding to actual physical locations of the FPVs from an optical perspective of the vehicle operator; 
   guiding the vehicle operator into the parking spot by displaying previously trained FPVs overlaid onto live sensor data displayed on the HMI; and   instructing the vehicle operator to align the previously trained FPVs with corresponding real-world FPVs of the parking spot by maneuvering the vehicle.   
     
     
         10 . The method of  claim 9 , wherein the TS portion further comprises:
 indicating, via a vehicle operator input to a first feature of the ARAPA in the HMI, that the vehicle has been parked in a preferred parking spot.   
     
     
         11 . The method of  claim 10 , wherein the TS portion further comprises:
 presenting, via the HMI, a plurality of proposed FPVs overlaid on a real-time display of a view from the one or more sensors, wherein the proposed FPVs at least partially define a current or an idealized position of the vehicle within the preferred parking spot; and   receiving, via the HMI, inputs from the vehicle operator selecting some or all of the plurality of proposed FPVs, each of the proposed FPVs having appropriate separations from other proposed FPVs, wherein the appropriate separation is chosen such that all selected FPVs may be simultaneously displayed on the HMI and so that different FPVs do not overlap at any line of sight of the vehicle operator; and wherein the appropriate separation reduces a potential for vehicle mis-alignment within the parking spot from a first level to a second level less than the first level as the vehicle is parked.   
     
     
         12 . The method of  claim 11 , wherein the TS portion further comprises:
 suggesting to the vehicle operator, via the HMI, FPVs and semantic tags corresponding to parking spots that the ARAPA has previously identified at the parking location;   offering, via the HMI, automatically generated proposed parking spot names and opportunities for the vehicle operator to alter the proposed parking spot names; and   storing the proposed parking spot names and semantic tags in memory for use in the LS portion of the ARAPA.   
     
     
         13 . The method of  claim 9 , wherein the LS portion further comprises:
 determining that the vehicle is within the FPV preview threshold distance of one or more previously saved FPVs of parking spots in a parking location; and   conditionally displaying a preview of the previously saved FPVs on one or more of the HMI and a head-sup-display (HUD), and using previously saved FPVs as enhanced steering guidance for the vehicle operator to align the vehicle within a parking spot to a distance that is equal to or less than a predefined FPV alignment threshold distance.   
     
     
         14 . The method of  claim 13 , wherein the LS portion further comprises:
 providing AR assistance to the vehicle operator as the vehicle operator provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle towards and into the parking spot identified by previously-identified FPVs, wherein the AR assistance overlays the previously-identified FPVs on one or more of live sensor data displayed on the HMI and on the HUD of the vehicle so that the FPVs are displayed directly in a line-of-sight of the vehicle operator and in positions corresponding to actual physical locations of the FPVs from an optical perspective of the vehicle operator, and wherein a path to the identified, previously saved FPVs is displayed on one or more of the HUD and the HMI in a different style, color, or layout, in comparison to a path to a previously trained parking spot.   
     
     
         15 . The method of  claim 14 , wherein the LS portion further comprises:
 determining a current distance of the vehicle to a parking spot is less than or equal to the FPV alignment threshold distance and engages an FPV alignment mode of the ARAPA; and   utilizing AR to display previously-identified or saved FPVs on one or more of the HMI and the HUD, thereby providing the vehicle operator with references that the vehicle operator utilizes the align the vehicle within the parking spot in real-time;   causing the vehicle operator to check alignments of the saved FPVs with corresponding features of a real-world environment for enhanced parking assistance and steers, or otherwise maneuvers the vehicle; and   notifying the vehicle operator that the vehicle has been completely parked when the saved FPV are aligned to real-world FPVs or environmental objects as viewed on either or both of the HMI and the HUD.   
     
     
         16 . A method for augmented-reality (AR)-assisted vehicle parking with operator-in-loop selections and alignments of feature point vicinities (FPVs), the method comprising:
 detecting FPVs of a parking location with one or more sensors disposed on a vehicle;   executing programmatic control logic stored within a memory of one or more controllers in communication with the one or more sensors and in communication with one or more human-machine interfaces (HMIs) disposed within the vehicle, each of the one or more controllers having a processor, the memory, and one or more input/output (I/O) ports, the I/O ports in communication with the one or more sensors and the one or more HMIs; the processor executing control logic including an AR-assisted parking application (ARAPA);   activating, via a vehicle operator input through the HMI, the ARAPA upon completing parking at a parking spot at the parking location; the ARAPA including control logic comprising:   training the ARAPA, in a training session (TS) portion, to recognize FPVs of a parking spot at the parking location, including:
 indicating, via a vehicle operator input to a first feature of the ARAPA in the HMI, that the vehicle has been parked in a preferred parking spot; 
 presenting, via the HMI, a plurality of proposed FPVs overlaid on a real-time display of a view from the one or more sensors, wherein the proposed FPVs at least partially define a current or an idealized position of the vehicle within the preferred parking spot; 
 receiving, via the HMI, inputs from the vehicle operator selecting some or all of the plurality of proposed FPVs, each of the proposed FPVs having appropriate separations from other proposed FPVs, wherein the appropriate separation is chosen such that all selected FPVs may be simultaneously displayed on the HMI and so that different FPVs do not overlap at any line of sight of the vehicle operator; and wherein the appropriate separation reduces a potential for vehicle mis-alignment within the parking spot from a first level to a second level less than the first level as the vehicle is parked; 
 suggesting to the vehicle operator, via the HMI, FPVs and semantic tags corresponding to parking spots that the ARAPA has previously identified at the parking location; 
 offering, via the HMI, automatically generated proposed parking spot names and opportunities for the vehicle operator to alter the proposed parking spot names; and 
 storing the proposed parking spot names and semantic tags in memory for use in a live-session (LS) portion of the ARAPA; 
   prompting the vehicle operator, in an LS portion of the ARAPA, to select a parking spot as the vehicle operator navigates through a parking location with one or more parking spots;
 determining that a current distance between the vehicle and one or more previously trained parking spots is less than or equal to a predefined FPV preview threshold distance, wherein the FPV preview threshold distance is between approximately six meters and approximately ten meters; 
 offering, via the HMI, one or more prompts to the vehicle operator, wherein the one or more prompts request the vehicle operator to select one of the one or more previously trained parking spots within the FPV preview threshold distance that the vehicle operator intends to park within; 
 providing augmented reality (AR) assistance to the vehicle operator as the vehicle operator provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle towards and into the previously trained parking spot, wherein the AR assistance overlays FPVs of the previously trained parking spot on one or more of live sensor data displayed on the HMI and on a heads-up display (HUD) of the vehicle so that the FPVs are displayed directly in a line-of-sight of the vehicle operator and in positions corresponding to actual physical locations of the FPVs from an optical perspective of the vehicle operator; 
 determining that the vehicle is within the FPV preview threshold distance of one or more previously saved FPVs of parking spots in a parking location; 
 conditionally displaying a preview of the previously saved FPVs on one or more of the HMI and a head-sup-display (HUD), and using previously-identified FPVs as enhanced steering guidance for the vehicle operator to align the vehicle within a parking spot to a distance that is equal to or less than a predefined FPV alignment threshold distance; 
 providing AR assistance to the vehicle operator as the vehicle operator provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle towards and into the parking spot identified by previously-identified FPVs, wherein the AR assistance overlays the previously-identified FPVs on one or more of live sensor data displayed on the HMI and on the HUD of the vehicle so that the FPVs are displayed directly in the vehicle operator's line-of-sight and in positions corresponding to actual physical locations of the FPVs from the vehicle operator's optical perspective, and wherein a path to the identified, previously saved FPVs is displayed on one or more of the HUD and the HMI in a different style, color, or layout, in comparison to a path to a previously trained parking spot; and 
   instructing the vehicle operator to align the previously trained FPVs with corresponding real-world FPVs of the parking spot by maneuvering the vehicle by:
 determining a current distance of the vehicle to a parking spot is less than or equal to the FPV alignment threshold distance and engages an FPV alignment mode of the ARAPA; 
 utilizing AR to display previously-identified or saved FPVs on one or more of the HMI and the HUD, thereby providing the vehicle operator with references that the vehicle operator utilizes the align the vehicle within the parking spot in real-time; 
 causing the vehicle operator to check alignments of the saved FPVs with corresponding features of a real-world environment for enhanced parking assistance and steers, or otherwise maneuvers the vehicle; and 
   notifying the vehicle operator via one or more of the HUD and the HMI that the vehicle has been completely parked when the saved FPV are aligned to real-world FPVs or environmental objects as viewed on either or both of the HMI and the HUD.