System and method for fusing three-dimensional shape data on distorted images without correcting for distortion

Abstract

A system and method for intra-operatively providing a surgeon with visual evaluations of possible surgical outcomes ahead of time, and generating simulated data, includes a medical imaging camera, a registration device for registering data to a physical space, and to the medical imaging camera, and a fusion mechanism for fusing the data and the images to generate simulated data, without correcting for distortion. The simulated data (e.g., such as augmented X-ray images) is natural and easy for a surgeon to interpret. In an exemplary implementation, the system preferably includes a data processor which receives a three-dimensional surgical plan or three-dimensional plan of therapy delivery, one or a plurality of two-dimensional intra-operative images, a three-dimensional model of pre-operative data, registration data, and image calibration data. The data processor produces one or a plurality of simulated post-operative images, without correcting for distortion, by integrating a projection of a three-dimensional model of pre-operative data onto one or a plurality of two-dimensional intra-operative images.

What is claimed is:  
     
       1. A method of fusing three-dimensional image data on an image, comprising: 
       receiving a potentially distorted image;  
       computing an apparent contour of said three-dimensional image data on said potentially distorted image;  
       for each pixel of the image, determining one of a ray in three-dimensional space and computing a distance from the ray to the apparent contour; and  
       selectively adjusting a pixel value of said potentially distorted image based on said distance.  
     
     
       2. The method according to  claim 1 , further comprising: 
       calibrating said potentially distorted image,  
       wherein said computing is based on said potentially distorted image having been calibrated.  
     
     
       3. The method according to  claim 2 , wherein said calibrating comprises: 
       associating a center of perspective to the image and a ray destination for each pixel of the potentially distorted image.  
     
     
       4. The method according to  claim 3 , wherein said computing comprises: 
       decomposing said three-dimensional image data into predetermined sub-shapes; and  
       computing a set of three-dimensional apparent contours based on said center of perspective and the three-dimensional image data being decomposed into said predetermined sub-shapes.  
     
     
       5. The method according to  claim 4 , wherein said sub-shapes comprise sub-shapes having one of a triangular shape and a polygonal shape. 
     
     
       6. The method according to  claim 3 , wherein said computing further comprises: 
       based on said center of perspective, defining and extracting a three-dimensional apparent contour.  
     
     
       7. The method according to  claim 6 , wherein said defining and extracting comprises: 
       for each surface sub-shape, defining a viewing direction as a vector originating from the center of perspective to a centroid of the sub-shape.  
     
     
       8. The method according to  claim 7 , wherein if the sub-shape normal, as defined by a cross product of ordered oriented sub-shape edges, makes an obtuse angle with the viewing direction, the sub-shape is considered visible, 
       wherein a surface apparent contour is a subset of surface edges, such that a sub-shape on one side of the edge is visible and a sub-shape on another side of the edge is invisible,  
       said apparent contour having edges linked to form non-planar polygonal curves in three dimensions.  
     
     
       9. The method according to  claim 8 , wherein said forming of said apparent contour comprises: 
       identifying edges belonging to any apparent contour, and adding said edges to a list, wherein said edges are oriented such that a visible sub-shape is on a predetermined side of the edge, thus defining an edge origin and an edge destination.  
     
     
       10. The method according to  claim 9 , wherein said forming of said apparent contour further comprises: 
       based on a first edge in the list, creating a new apparent contour starting with said first edge; and  
       completing said apparent contour, containing said first edge.  
     
     
       11. The method according to  claim 10 , wherein said completing said apparent contours comprises: 
       starting from the destination of the current edge, completing a next edge, wherein sub-shapes incident to a destination vertex in a counter-clockwise fashion, are visited, and the first edge is determined that belongs to the list of apparent contour edges; and  
       reapplying said completing until a next edge is the same as the first edge that was processed previously.  
     
     
       12. The method according to  claim 11 , wherein said forming said apparent contour further comprises: 
       removing all the edges forming that contour from the list of apparent contour edges.  
     
     
       13. The method according to  claim 12 , wherein said forming said apparent contour further comprises: 
       reapplying said creating a new apparent contour, completing the apparent contour, and said removing until the list of apparent contour edges is empty.  
     
     
       14. The method according to  claim 3 , wherein said determining comprises: 
       for each pixel of the potentially distorted image, determining the corresponding ray from the center of perspective, and computing the distance to the apparent contour.  
     
     
       15. The method according to  claim 14 , wherein computing said distance from a given line in three-dimensions to an apparent contour, comprises: 
       computing the distance from a line in three-dimensions to a line segment in three-dimensions.  
     
     
       16. The method according to  claim 3 , wherein said adjusting said pixel value comprises: 
       updating the pixel value with a distance ray-shape that was determined.  
     
     
       17. The method according to  claim 3 , further comprising: 
       fusing by projecting a silhouette curve of said data by considering in turn each new pixel, determining a line in three-dimensions corresponding to that pixel by image calibration, computing a distance from a line to the silhouette curve, and assigning a pixel gray-scale value depending on the distance.  
     
     
       18. The method according to  claim 17 , wherein said fusing comprises assigning pixel gray-scale values corresponding to a distance, 
       wherein if the distance is less than a first predetermined value, then the gray-scale value is set to a first predetermined number.  
     
     
       19. The method according to  claim 18 , wherein if the distance is less than a second predetermined value, then the gray-scale value is set to a second predetermined number larger than said first predetermined number. 
     
     
       20. The method according to  claim 19 , wherein if the distance is less than a third predetermined value greater than said first and second predetermined values, then the gray-scale value is set to a third predetermined number larger than said first and second predetermined numbers. 
     
     
       21. The method according to  claim 20 , wherein if the distance is greater than or equal to said third predetermined value, then the gray-scale value is not modified for projecting the silhouette curves. 
     
     
       22. The method according to  claim 2 , further comprising: 
       fusing said three-dimensional image data with said potentially distorted image by integrating a two-dimensional projection of a silhouette of a three-dimensional implant model in an X-ray image.  
     
     
       23. The method according to  claim 22 , wherein said fusing uses a calibration of the X-ray image, to determine a center of perspective whose location represents an estimate of a location of an X-ray source, 
       said center of perspective being used to compute silhouette curves on the three-dimensional implant model.  
     
     
       24. The method according to  claim 23 , wherein said silhouette curves are such that rays emanating from the center of perspective and tangent to the three-dimensional model meet the three-dimensional implant model on a silhouette curve. 
     
     
       25. An apparatus for fusing three-dimensional image data on an image, comprising: 
       means for receiving a potentially distorted image;  
       a processor for computing an apparent contour of said three-dimensional image data on said potentially distorted image;  
       means for determining, for each pixel of the image, one of a ray in three-dimensional space and computing a distance from the ray to the apparent contour; and  
       means for selectively adjusting a pixel value of said potentially distorted image based on said distance.  
     
     
       26. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform a method for computer-implemented fusing of three-dimensional image data on a distorted image without correcting for distortion, comprising: 
       computing an apparent contour of three-dimensional image data of a potentially distorted image;  
       for each pixel of the image, determining a ray in three-dimensional space and computing a distance from the ray to the apparent contour; and  
       selectively adjusting a pixel value of said potentially distorted image based on said distance.