Systems and Methods of Detecting and Responding to a Visitor to a Smart Home Environment

Abstract

A method of presenting appropriate actions for responding to a visitor to a smart home environment via an electronic greeting system of the smart home environment, including detecting a visitor of the smart home environment; obtaining context information from the smart home environment regarding the visitor; based on the context information, identifying a plurality of appropriate actions available to a user of a client device for interacting with the visitor via the electronic greeting system; and causing the identified actions to be presented to the user of the client device.

What is claimed is: 
     
         1 . A method comprising:
 receiving, by a remote server system, an indication that a visitor activated a video doorbell;   causing, by the remote server system, the video doorbell to output an audio greeting;   receiving, by the remote server system, audio-visual data from the video doorbell after outputting the audio greeting;   determining, by the remote server system and based on the audio-visual data, a response for the video doorbell; and   causing, by the remote server system, the video doorbell to perform the response.   
     
     
         2 . The method of  claim 1 , wherein the audio-visual data is a first voice response, the method further comprising:
 after causing the video doorbell to perform the response, receiving, by the remote server system, a second voice response from the visitor; and   storing, by the remote server system, the second voice response as an audio message.   
     
     
         3 . The method of  claim 2 , further comprising:
 converting, by the remote server system, the audio message to text using speech-to-text; and   sending, by the remote server system and to a device associated with a user associated with the video doorbell, the text of the audio message.   
     
     
         4 . The method of  claim 1 , wherein:
 the audio-visual data includes an indication that the visitor is dropping off a package; and   the response for the video doorbell includes an indication of a location to leave the package.   
     
     
         5 . The method of  claim 1 , further comprising:
 receiving, by the remote server system, video data captured by the video doorbell; and   determining, by the remote server system and based on the video data, that the visitor is approaching the video doorbell.   
     
     
         6 . The method of  claim 5 , further comprising:
 responsive to determining that the visitor is approaching, causing the video doorbell to initiate an observation window; and   during the observation window:
 continuously receiving, by the remote server system and from the video doorbell, additional video data; and 
 determining, by the remote server system and using facial recognition, whether the visitor is a known visitor based on the additional video data. 
   
     
     
         7 . The method of  claim 6 , further comprising:
 responsive to determining that the visitor is a known visitor, determining whether a pre-assigned verbal announcement is associated with the known visitor; and   responsive to determining that the pre-assigned verbal announcement is associated with the known visitor, determining that the pre-assigned verbal announcement is the audio greeting.   
     
     
         8 . The method of  claim 6 , wherein determining the response for the video doorbell is further based on whether the visitor is the known visitor. 
     
     
         9 . The method of  claim 6 , further comprising, responsive to determining that the visitor is an unknown and potentially unsafe visitor, determining the response for the video doorbell to include an action-based security function. 
     
     
         10 . The method of  claim 9 , wherein the action-based security function includes one or more of adjusting an alarm sensitivity level, sounding an alarm, adjusting a brightness level of one or more lights, alerting law enforcement personnel, alerting a preselected contact of a user associated with the video doorbell, capturing image or video data of the visitor and sending the captured image or video data to the law enforcement personnel or to the preselected contact of the user. 
     
     
         11 . The method of  claim 5 , further comprising:
 responsive to determining that the visitor is approaching the video doorbell, determining, by the remote server system, context information associated with the visitor, wherein the context information includes at least one of a facial recognition analysis result of the visitor, one or more behavioral characteristics of the visitor, one or more physical characteristics of the visitor, one or more clothing or accessory characteristics of the visitor, a time of day during which the visitor approaches the video doorbell, a day of a week during which the visitor approaches the video doorbell, proximity of a time at which the visitor approaches the video doorbell to a time of a prescheduled event, proximity of a time at which the visitor approaches the video doorbell to a time of a prescheduled status, a known or an unknown status of a user associated with the video doorbell, or an expected or an unexpected status of the user,   wherein determining the response for the video doorbell includes determining the response from a plurality of suggested responses based on the context information, wherein the plurality of suggested responses includes at least one of a status of the user, a request for the visitor to perform an action, a request for the visitor to leave a message, a preprogrammed customized communication, or an audio version of a user-composed text message.   
     
     
         12 . The method of  claim 11 , wherein causing the video doorbell to perform the response includes causing the video doorbell to provide visual feedback via a user interface of the video doorbell or via one or more lights associated with the video doorbell. 
     
     
         13 . The method of  claim 12 , wherein causing the video doorbell to provide visual feedback via the user interface includes causing the video doorbell to display at least one of an image, a video, or a message via the user interface of the video doorbell. 
     
     
         14 . A computing system comprising:
 one or more processors; and   a memory that stores instructions that, when executed by the one or more processors, cause the one or more processors to:
 receive, from a remote server system, an indication that a visitor activated a video doorbell; 
 cause the video doorbell to output an audio greeting; 
 receive, by the remote server system, audio-visual data from the video doorbell after outputting the greeting; 
 determine, based on the audio-visual data, a response for the video doorbell; and 
 cause the video doorbell to perform the response. 
   
     
     
         15 . The computing system of  claim 14 , wherein the audio-visual data is a first voice response, and wherein the instructions further cause the one or more processors to:
 after the video doorbell performs the response, receive a second voice response from the visitor; and   store the second voice response as an audio message.   
     
     
         16 . The computing system of  claim 15 , wherein the instructions further cause the one or more processors to:
 convert the audio message to text using speech-to-text; and   send, to a device associated with a user associated with the video doorbell, the text of the audio message.   
     
     
         17 . The computing system of  claim 14 , wherein:
 the audio-visual data includes an indication that the visitor is dropping off a package; and   the response includes an indication of a location to leave the package.   
     
     
         18 . The computing system of  claim 14 , wherein the instructions further cause the one or more processors to:
 receive, from the video doorbell, video data captured by the video doorbell; and   determine, based on the video data, that the visitor is approaching the video doorbell.   
     
     
         19 . The computing system of  claim 18 , wherein the instructions further cause the one or more processors to:
 responsive to determining that the visitor is approaching, cause the video doorbell to initiate an observation window; and   during the observation window:
 continuously receive, from the video doorbell, additional video data; and 
 determine, using facial recognition, whether the visitor is a known visitor based on the additional video data. 
   
     
     
         20 . The computing system of  claim 19 , wherein the instructions further cause the one or more processors to:
 responsive to determining that the visitor is a known visitor, determine whether a pre-assigned verbal announcement is associated with the known visitor; and   responsive to determining that the pre-assigned verbal announcement is associated with the known visitor, determine that the pre-assigned verbal announcement is the audio greeting.   
     
     
         21 . The computing system of  claim 19 , wherein the instructions further cause the one or more processors to, responsive to determining that the visitor is an unknown and potentially unsafe visitor, determine the response for the video doorbell to include an action-based security function. 
     
     
         22 . The computing system of  claim 21 , wherein the action-based security function includes one or more of adjusting an alarm sensitivity level, sounding an alarm, adjusting a brightness level of one or more lights, alerting law enforcement personnel, alerting a preselected contact of a user associated with the video doorbell, capturing image or video data of the visitor and sending the captured image or video data to the law enforcement personnel or to the preselected contact of the user. 
     
     
         23 . The computing system of  claim 18 , wherein:
 the instructions further cause the one or more processors to, responsive to determining that the visitor is approaching the video doorbell, determine context information associated with the visitor;   the context information includes at least one of a facial recognition analysis result of the visitor, one or more behavioral characteristics of the visitor, one or more physical characteristics of the visitor, one or more clothing or accessory characteristics of the visitor, a time of day during which the visitor approaches the video doorbell, a day of a week during which the visitor approaches the video doorbell, proximity of a time at which the visitor approaches the video doorbell to a time of a prescheduled event, proximity of a time at which the visitor approaches the video doorbell to a time of a prescheduled status, a known or an unknown status of a user associated with the video doorbell, or an expected or an unexpected status of the user;   the instructions cause the one or more processors to determine the response for the video doorbell from a plurality of suggested responses further based on the context information; and   the plurality of suggested responses includes at least one of a status of the user, a request for the visitor to perform an action, a request for the visitor to leave a message, a preprogrammed customized communication, or an audio version of a user-composed text message.   
     
     
         24 . The computing system of  claim 23 , wherein the instructions cause the one or more processors to cause the video doorbell to perform the response by causing the video doorbell to provide visual feedback via a user interface of the video doorbell or via one or more lights associated with the video doorbell. 
     
     
         25 . The computing system of  claim 24 , wherein the instructions cause the one or more processors to cause the video doorbell to provide visual feedback via the user interface by causing the video doorbell to display at least one of an image, a video, or a message via the user interface of the video doorbell. 
     
     
         26 . A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors of a remote computing system, cause the one or more processors to:
 receive an indication that a visitor activated a video doorbell;   cause the video doorbell to output an audio greeting;   receive audio-visual data from the video doorbell after outputting the audio greeting;   determine, based on the audio-visual data, a response for the video doorbell; and   cause the video doorbell to perform the response.   
     
     
         27 . The non-transitory computer-readable storage medium of  claim 26 , wherein:
 the audio-visual data is a first voice response from the visitor; and   the instructions further cause the one or more processors to:
 after the video doorbell performs the response, receive a second voice response from the visitor; 
 convert the second voice response to text using speech-to-text; and 
 send, to a device associated with a user associated with the video doorbell, the text of the second voice response. 
   
     
     
         28 . The non-transitory computer-readable storage medium of  claim 26 , wherein:
 the audio-visual data includes an indication that the visitor is dropping off a package; and   the response includes an indication of a location to leave the package.   
     
     
         29 . The non-transitory computer-readable storage medium of  claim 26 , wherein the audio-visual data includes a first voice response, wherein the instructions further cause the one or more processors to:
 after the video doorbell performs the response, receive a second voice response from the visitor;   store the second voice response as an audio message;   convert the audio message to text using speech-to-text; and   send, to a device associated with a user associated with the video doorbell, the text of the audio message.   
     
     
         30 . The non-transitory computer-readable storage medium of  claim 26 , wherein the instructions further cause the one or more processors to:
 receive, from the video doorbell, video data captured by the video doorbell;   determine, based on the video data, that the visitor is approaching the video doorbell;   responsive to determining that the visitor is approaching, cause the video doorbell to initiate an observation window; and   during the observation window:
 continuously receive, from the video doorbell, additional video data; and 
 determine, using facial recognition, whether the visitor is a known visitor based on the additional video data.