Systems and methods for computer assisted operation

Abstract

An augmented reality method includes selecting an image of an object to be installed; converting the image into a 3D model; capturing a view of an environment and selecting a place to insert the 3D model; overlaying the 3D model with the view of the environment in a mashed-up and aligned manner wherein the 3D model view is displayed contemporaneously and contiguously to form an augmented reality view; and enabling user interaction with the displayed 3D model view to update the 3D model of the environment in the 3D modeling system.

What is claimed is: 
     
         1 . An augmented reality method, comprising:
 selecting an image of an object to be installed;   converting the image into a 3D model;   performing motion tracking and understanding an environment with points or planes using accelerometer sensor and estimating light or color in the environment using a video camera in a mobile phone;   determining a target volume or space;   capturing images from a plurality of angles of the environment;   acquiring sensor data from sensors and optimizing features extracted from each image and sensor data, where a feature conveys data unique to the image at a specific pixel location;   capturing a view of an environment and selecting a place to insert the 3D model;   overlaying the 3D model with the view of the environment, wherein the 3D model view is displayed contemporaneously and contiguously to form an augmented reality view; and   enabling user interaction with the displayed 3D model view to update the 3D model of the object in the 3D modeling system.   
     
     
         2 . The method of  claim 1  wherein the environment is a location, farm, building, plant, factory, refinery, city, a vehicle, a car, a plane, a ship, an entity or a physical asset, comprising applying a color for the 3D model. 
     
     
         3 . The method of  claim 1  comprising downloading the image from a shopping web site. 
     
     
         4 . The method of  claim 1  wherein display of the augmented reality view and user interaction therewith are implemented by an augmented reality application; and the 3D database is one of external to or embedded in the augmented reality application. 
     
     
         5 . The method of  claim 4  further comprising the step of federating the 3D modeling system with the augmented reality application. 
     
     
         6 . The method of  claim 1  wherein the step of enabling user interaction with the displayed 3D model view is implemented by the 3D modeling system that was originally used to generate the 3D model of the environment or used to store said 3D model in the 3D database. 
     
     
         7 . The method of  claim 1 , comprising in response to user interaction with the displayed 3D model view, the 3D modeling system updating the corresponding 3D model of the object and the 3D model of the environment in the 3D database. 
     
     
         8 . The method of  claim 1  wherein displaying of the 3D model view of the one environment object overlaying the augmented reality view is aligned in size, dimensions and/or orientation. 
     
     
         9 . The method of  claim 1 , comprising:
 determining body landmarks or key positions;   identifying a surgical path to operate and providing instruction on surgical technique;   overlaying images of the anatomy to guide a user;   accessing a target to replace or repair and show instruction to repair or replace.   
     
     
         10 . The method of  claim 1 , comprising:
 identifying buildings and building structures positioned thereon; and   retrieving a building repair sequence.   
     
     
         11 . The method of  claim 1 , comprising:
 determining type and location of a building structure, and   highlighting on the augmented reality view each building structure to be opened or removed and provides instructions on opening or removal techniques.   
     
     
         12 . The method of  claim 11 , comprising:
 iteratively providing instructions to allow each structure to be opened or removed and then show next structure;   accessing a target structure to replace or repair and showing instructions to repair or replace; and   showing instructions in sequence to put back structures previously opened or removed.   
     
     
         13 . The method of  claim 1 , comprising rendering the 3D model as a perspective view relative to the environment. 
     
     
         14 . The method of  claim 1 , comprising rendering in the augmented view one or more maintenance or repair techniques. 
     
     
         15 . The method of  claim 1 , comprising:
 motion-tracking with visual features of an environment, in combination with accelerometer and gyroscope data, to track movements in space;   storing environment data in a map that can be re-used with metadata including notes, instructions, or points of interest; and   detecting distances, sizes, and surfaces in the environment for depth perception.   
     
     
         16 . The method of  claim 1 , comprising:
 displaying a 3D model view of the object overlaying on the augmented reality view of the machine in a mashed-up and aligned manner such that the 3D model view is displayed contemporaneously and contiguously to the augmented reality view; and   identifying in the augmented reality view each object to be opened or removed and providing instructions on how to open or move the object in order to access a targeted object.   
     
     
         17 . The method of  claim 1 , comprising automatically identifying the machine and retrieving parts and components in the machine. 
     
     
         18 . The method of  claim 1 , comprising
 after each component is opened or removed, detecting the next object to be opened or removed to reach the targeted object to be replaced or repaired;   showing instructions to repair or replace the object; and   showing instructions in the augmented reality view to put back objects previously opened or removed.   
     
     
         19 . A computer aided design (3D) modeling method comprising:
 performing motion tracking and understanding an environment with points or planes using accelerometer sensor and estimating light or color in the environment using a video camera in a mobile phone;   determining a target volume or space;   capturing images from a plurality of angles of the environment;   acquiring sensor data from sensors and optimizing features extracted from each image and sensor data, where a feature conveys data unique to the image at a specific pixel location;   capturing a view of an environment and selecting a place to insert the 3D model;   generating an augmented reality view of a environment, the environment having various objects displayed in the augmented reality view;   in response to user interaction with one of the environment objects as displayed in the augmented reality view, searching a 3D database for a corresponding 3D model of the one object, the 3D database storing a 3D model of the environment from a 3D modeling system, said searching resulting in obtaining the 3D model of the environment that contains the corresponding 3D model of the one object;   displaying a 3D model view of the one environment object overlaying on the augmented reality view of the one environment object in a mashed-up and aligned manner such that the 3D model view is displayed contemporaneously and contiguously to the augmented reality view; and   enabling user interaction with the displayed 3D model view to update the 3D model of the environment in the 3D modeling system.   
     
     
         20 . The method of  claim 19 , comprising:
 motion-tracking with visual features of an environment, in combination with accelerometer and gyroscope data, to track movements in space;   storing environment data in a map that can be re-used with metadata including notes, instructions, or points of interest; and   detecting distances, sizes, and surfaces in the environment for depth perception.