Intelligent interface based on augmented reality

Abstract

An interaction method implemented by a terminal apparatus includes: obtaining an interface underlayer of a space that has direction information and distance information; establishing a mapping relationship between an external device and an identification corresponding to the external device marked on the interface underlayer, the identification indicating a physical location of the external device; receiving, from the external device, property information of the external device; obtaining an image of a real scene in the space; determining that a target object is depicted in the image of the real scene; determining target presentation information of the target object according to the property information of the external device; and displaying the image of the real scene together with the target presentation information, the target presentation information being added to the image as an augmented reality (AR) mark at an image location corresponding to the target object.

What is claimed is: 
     
       1. An interaction method, implemented by a terminal apparatus, comprising:
 obtaining an interface underlayer of a space, the interface underlayer including a measurement drawing having direction information and distance information; 
 establishing a mapping relationship between an external device and an identification corresponding to the external device marked on the interface underlayer, the identification indicating a physical location of the external device; 
 receiving, from the external device, property information of the external device; 
 obtaining an image of a real scene in the space; 
 determining that a target object is depicted in the image of the real scene; 
 determining target presentation information of the target object according to the property information of the external device; and 
 displaying, on a display screen corresponding to the terminal apparatus, the image of the real scene together with the target presentation information, the target presentation information being added to the image as an augmented reality (AR) mark at an image location corresponding to the target object, 
 wherein the target object is the external device, and determining that the target object is depicted in the image of the real scene comprises:
 determining a target location of the external device relative to the terminal apparatus according to a location of the terminal apparatus in the interface underlayer and the identification of the external device in the interface underlayer; and 
 determining that the external device is in the image of the real scene if an orientation of the terminal apparatus indicates that the terminal apparatus is facing toward the external device. 
 
 
     
     
       2. The method according to  claim 1 , wherein the target object is the external device or an object of interest corresponding to the external device. 
     
     
       3. The method according to  claim 1 , wherein:
 the property information of the external device includes at least one of characteristics information of the external device, current status information of the external device, record information on the external device, or control options of the external device. 
 
     
     
       4. The method according to  claim 1 , after displaying the AR mark, further comprising:
 generating a control instruction upon detecting a user operation on the AR mark; and 
 sending the control instruction to the external device, the control instruction being configured to instruct the external device to perform a corresponding action. 
 
     
     
       5. The method according to  claim 4 , wherein the AR mark is displayed as one or more user interface (UI) items corresponding to one or more control options of the external device, and generating the control instruction comprises:
 upon detecting that the user operation is performed on a UI item having one or more sub-items, updating the AR mark to display the one or more sub-items; and 
 upon detecting that the user operation is performed on a UI item or a sub-item corresponding to a control option of the external device, generating the control instruction according to the user operation and the control option. 
 
     
     
       6. The method according to  claim 4 , after sending the control instruction, further comprising:
 receiving a feedback information corresponding to the control instruction; 
 updating the target presentation information according to the feedback message; and 
 displaying the updated target presentation information as an updated AR mark. 
 
     
     
       7. The method according to  claim 6 , wherein:
 when the control instruction corresponds to a write action of adjusting a parameter of the external device, the feedback message includes an execution result indicating whether the corresponding action is executed successfully; and 
 when the control instruction corresponds to a read action of querying a record of the external device, the feedback message includes a query result. 
 
     
     
       8. The method according to  claim 1 , wherein determining that the target object is depicted in the image of the real scene further comprises: before determining the target location of the external device relative to the terminal apparatus,
 performing a calibration process and determining the location and the orientation of the terminal apparatus in the interface underlayer. 
 
     
     
       9. The method according to  claim 8 , wherein performing the calibration process comprises:
 displaying a calibration icon on the image of the real scene at a preset image location, the calibration icon indicating an expected outline of a calibration object; and 
 upon determining that the calibration object included in the image of the real scene matches the expected outline, determining that terminal apparatus is located at a reference point with a reference orientation on the interface underlayer. 
 
     
     
       10. The method according to  claim 9 , wherein:
 a location of the reference point and the reference orientation, the calibration icon, and the preset image location are recorded during a process of generating the measurement drawing with a ranging device. 
 
     
     
       11. The method according to  claim 8 , further comprising:
 determining an expected image location of the external device in the image of the real scene according to the identification of the external device in the interface underlayer and the location and the orientation of the terminal apparatus in the interface underlayer. 
 
     
     
       12. The method according to  claim 11 , wherein:
 comparing current image data at the expected image location of the image of the real scene with prestored image data of the external device; and 
 upon determining that the current image data does not match the prestored image data, displaying a reminder message indicating that a current location of the external device in the real scene is not consistent with the physical location of the external device recorded in the interface underlayer. 
 
     
     
       13. A terminal apparatus, comprising: a memory, and a processor coupled to the memory and configured to perform:
 obtaining an interface underlayer of a space, the interface underlayer including a measurement drawing having direction information and distance information; 
 establishing a mapping relationship between an external device and an identification corresponding to the external device marked on the interface underlayer, the identification indicating a physical location of the external device; 
 receiving, from the external device, property information of the external device; 
 obtaining an image of a real scene in the space; 
 determining that a target object is depicted in the image of the real scene; 
 determining target presentation information of the target object according to the property information of the external device; and 
 displaying, on a display screen corresponding to the terminal apparatus, the image of the real scene together with the target presentation information, the target presentation information being added to the image as an augmented reality (AR) mark at an image location corresponding to the target object, 
 wherein the target object is the external device, and determining that the target object is depicted in the image of the real scene comprises:
 determining a target location of the external device relative to the terminal apparatus according to a location of the terminal apparatus in the interface underlayer and the identification of the external device in the interface underlayer; and 
 determining that the external device is in the image of the real scene if an orientation of the terminal apparatus indicates that the terminal apparatus is facing toward the external device. 
 
 
     
     
       14. The terminal apparatus according to  claim 13 , wherein the target object is the external device or an object of interest corresponding to the external device. 
     
     
       15. The terminal apparatus according to  claim 13 , wherein:
 the property information of the external device includes at least one of characteristics information of the external device, current status information of the external device, record information on the external device, or control options of the external device. 
 
     
     
       16. The terminal apparatus according to  claim 13 , wherein after displaying the AR mark, the processor is further configured to perform:
 generating a control instruction upon detecting a user operation on the AR mark; and 
 sending the control instruction to the external device, the control instruction being configured to instruct the external device to perform a corresponding action. 
 
     
     
       17. The terminal apparatus according to  claim 16 , wherein the AR mark is displayed as one or more user interface (UI) items corresponding to one or more control options of the external device, and generating the control instruction comprises:
 upon detecting that the user operation is performed on a UI item having one or more sub-items, updating the AR mark to display the one or more sub-items; and 
 upon detecting that the user operation is performed on a UI item or a sub-item corresponding to a control option of the external device, generating the control instruction according to the user operation and the control option. 
 
     
     
       18. The terminal apparatus according to  claim 16 , wherein after sending the control instruction, the processor is further configured to perform:
 receiving a feedback information corresponding to the control instruction; 
 updating the target presentation information according to the feedback message; and 
 displaying the updated target presentation information as an updated AR mark. 
 
     
     
       19. The terminal apparatus according to  claim 18 , wherein:
 when the control instruction corresponds to a write action of adjusting a parameter of the external device, the feedback message includes an execution result indicating whether the corresponding action is executed successfully; and 
 when the control instruction corresponds to a read action of querying a record of the external device, the feedback message includes a query result. 
 
     
     
       20. The terminal apparatus according to  claim 13 , wherein determining that the target object is depicted in the image of the real scene further comprises: before determining the target location of the external device relative to the terminal apparatus,
 performing a calibration process and determining the location and the orientation of the terminal apparatus in the interface underlayer.