Generating Smart Reminders by Assistant Systems

Abstract

In one embodiment, a method includes receiving initial sensory data of visual data captured by cameras of a head-mounted client device at the head-mounted client device, proactively generating a reminder associated with a first entity responsive to proactively identifying the first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, wherein the knowledge about the user comprises one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity, determining an activation condition associated with the reminder, which is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user, and presenting the reminder when the activation condition is satisfied at the head-mounted client device.

What is claimed is: 
     
         1 . A method comprising, by a head-mounted client device associated with a user:
 receiving, at the head-mounted client device, initial sensory data captured by one or more cameras of the head-mounted client device, wherein the initial sensory data is visual data;   proactively generating, by the head-mounted client device responsive to proactively identifying a first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, a reminder associated with the first entity for the user, wherein the knowledge about the user comprises one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity;   determining, by the head-mounted client device, an activation condition associated with the reminder, wherein the activation condition is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user; and   presenting, at the head-mounted client device, the reminder when the activation condition is satisfied.   
     
     
         2 . The method of  claim 1 , wherein the activation condition is based on a location, wherein the method further comprises:
 applying one or more scene recognition algorithms to current sensory data captured by the one or more cameras of the head-mounted client device;   recognizing that the user is at the location; and   determining the activation condition is satisfied.   
     
     
         3 . The method of  claim 2 , wherein the current sensory data comprises one or more of an image or a video clip captured by the one or more cameras. 
     
     
         4 . The method of  claim 1 , wherein the first entity is identified further based on a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein the plurality of nodes comprise a node corresponding to the user and a node corresponding to the first entity. 
     
     
         5 . The method of  claim 1 , wherein the first entity is identified further based on prior user requests by the user. 
     
     
         6 . The method of  claim 1 , wherein the initial sensory data comprises one or more of an image or a video clip, wherein the initial sensory data is based on a field of view of the one or more cameras, and wherein the visual analysis of the visual data is based on one or more machine-learning algorithms. 
     
     
         7 . The method of  claim 6 , wherein the one or more machine-learning algorithms are based on one or more of facial recognition, gait recognition, or object recognition. 
     
     
         8 . The method of  claim 1 , wherein determining the activation condition is further based on one or more of the routine of the user or the episodic memory of the user. 
     
     
         9 . The method of  claim 1 , further comprising:
 determining contextual information associated with the initial sensory data; and   accessing a plurality of episodic memories of the user;   wherein the reminder comprises one or more references to one or more other users, respectively, the referenced users being based on the contextual information and one or more of the accessed episodic memories of the user.   
     
     
         10 . The method of  claim 9 , wherein the reminder further comprises information associated with the referenced users from the one or more of the accessed episodic memories of the user. 
     
     
         11 . The method of  claim 9 , further comprising:
 retrieving one or more content objects associated with the one or more of the accessed episodic memories of the user, wherein each content object comprises one or more of a post, a comment, an image, or a video clip;   wherein the reminder further comprises one or more of the retrieved content objects.   
     
     
         12 . The method of  claim 1 , wherein the reminder is generated based on one or more reminder-templates. 
     
     
         13 . The method of  claim 1 , wherein the head-mounted client device is associated with an assistant system, and wherein the reminder is generated by a response-execution module of the assistant system. 
     
     
         14 . The method of  claim 1 , wherein the reminder comprises a social summary associated with the first entity. 
     
     
         15 . The method of  claim 1 , wherein the reminder comprises a social recommendation referencing one or more other users. 
     
     
         16 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
 receive, at a head-mounted client device, initial sensory data captured by one or more cameras of the head-mounted client device, wherein the initial sensory data is visual data;   proactively generate, by the head-mounted client device responsive to proactively identifying a first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, a reminder associated with the first entity for the user, wherein the knowledge about the user comprises one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity;   determine, by the head-mounted client device, an activation condition associated with the reminder, wherein the activation condition is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user; and   present, at the head-mounted client device, the reminder when the activation condition is satisfied.   
     
     
         17 . The media of  claim 16 , wherein the first entity is identified further based on a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein the plurality of nodes comprise a node corresponding to the user and a node corresponding to the first entity. 
     
     
         18 . The media of  claim 16 , wherein determining the activation condition is further based on one or more of the routine of the user or the episodic memory of the user. 
     
     
         19 . A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
 receive, at a head-mounted client device, initial sensory data captured by one or more cameras of the head-mounted client device, wherein the initial sensory data is visual data;   proactively generate, by the head-mounted client device responsive to proactively identifying a first entity based on a visual analysis of the visual data and correlating the first entity with knowledge about the user, a reminder associated with the first entity for the user, wherein the knowledge about the user comprises one or more of a routine of the user related to the first entity or an episodic memory of the user referencing the first entity;   determine, by the head-mounted client device, an activation condition associated with the reminder, wherein the activation condition is based on one or more of a time or a location, wherein the time and/or location are determined based on the analysis of the visual data and the knowledge about the user; and   present, at the head-mounted client device, the reminder when the activation condition is satisfied.   
     
     
         20 . The system of  claim 19 , wherein the first entity is identified further based on a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein the plurality of nodes comprise a node corresponding to the user and a node corresponding to the first entity