Method for planning a movement path for a robotic arm

Abstract

A method is provided for planning a movement path for a non-fixed end of a robotic arm. A mixed reality (MR) device is used to capture hand images of a user to obtain multiple target coordinate sets in an MR coordinate system. A computer device is used to convert the target coordinate sets into multiple path coordinate sets in a robot base coordinate system, and controls the non-fixed end of a robotic arm to move along a movement path as indicated by the path coordinate sets. A camera device is used to capture images of a base that the robotic arm is fixed to for conversion of the target coordinate sets into the path coordinate sets.

What is claimed is: 
     
         1 . A method for planning a movement path for a non-fixed end of a robotic arm, the robotic arm having a fixed end that is opposite to the non-fixed end and that is fixed to a base, the method comprising:
 providing a robotic arm control system that is communicatively connected to a camera device which is configured to capture images of the base, wherein the robotic arm control system includes:
 a mixed reality (MR) device that is to be worn on a user; and 
 a computer device that is communicatively connected to the robotic arm and the MR device, and that stores a first transformation matrix and a second transformation matrix, 
 the first transformation matrix being used to perform coordinate transformation from a robot base coordinate system related to the fixed end of the robotic arm to a navigation coordinate system related to the camera device, 
 the second transformation matrix being used to perform coordinate transformation from the navigation coordinate system to an MR coordinate system related to the MR device; 
   the method further comprising:
 A) by the MR device, capturing a target-setting hand image that is related to a hand of the user, and obtaining a target coordinate set in the MR coordinate system based on the target-setting hand image, where the target coordinate set is related to the hand of the user; 
 B) by the MR device, determining whether an instruction for terminating path planning is received; 
 C) by the MR device, repeating step A) upon determining that the instruction for terminating path planning has not been received yet; 
 D) by the MR device, after determining that the instruction for terminating path planning is received, generating and sending an execution request to the computer device, the execution request including the target coordinate sets that are obtained respectively in multiple executions of step A) and that are arranged in a user-defined order; 
 E) by the computer device, upon receipt of the execution request, converting, based on the first transformation matrix and the second transformation matrix, the target coordinate sets in the MR coordinate system respectively to path coordinate sets in the robot base coordinate system, the path coordinate sets being arranged in the user-defined order and cooperatively indicating a movement path; and 
 F) by the computer device, controlling the non-fixed end of the robotic arm to move along the movement path by generating and sending to the robotic arm an operating instruction that includes the path coordinate sets. 
   
     
     
         2 . The method as claimed in  claim 1 , wherein step A) includes:
 A-1) by the MR device, capturing a user-hand image, which is an image of the hand of the user, and acquiring a plurality of hand node coordinate sets in the MR coordinate system based on the user-hand image, the hand node coordinate sets being related to a plurality of hand nodes of the hand, respectively;   A-2) by the MR device, determining whether a first portion of the hand node coordinate sets indicates a first gesture; and   A-3) by the MR device, upon determining that the first portion of the hand node coordinate sets indicates the first gesture, making the user-hand image serve as the target-setting hand image, and acquiring the target coordinate set based on the first portion of the hand node coordinate sets.   
     
     
         3 . The method as claimed in  claim 2 , wherein:
 in sub-step A-1), the hand node coordinate sets include a first fingertip coordinate set that corresponds to a first fingertip of the hand, and a second fingertip coordinate set that corresponds to a second fingertip of the hand;   in sub-step A-2), the first portion of the hand node coordinate sets includes the first fingertip coordinate set and the second fingertip coordinate set, and the MR device determines whether the first portion of the hand node coordinate sets indicates the first gesture by determining whether a distance between the first fingertip coordinate set and the second fingertip coordinate set is smaller than a first predetermined distance; and   in sub-step A-3), the target coordinate set is one of the first fingertip coordinate set and the second fingertip coordinate set.   
     
     
         4 . The method as claimed in  claim 3 , wherein:
 sub-step A-1) further includes, by the MR device, acquiring a hand rotational angle data piece based on the target-setting hand image, the hand rotational angle data piece being related to the hand of the user and corresponding to the hand node coordinate sets;   sub-step A-3) further includes, by the MR device, upon determining that the first portion of the hand node coordinate sets indicates the first gesture, storing a rotational angle data piece that corresponds to the target coordinate set, the rotational angle data piece including a pitch angle, a yaw angle and a roll angle with respect to the MR coordinate system; and   said method further comprises:
 G) by the MR device, determining whether a second portion of the hand node coordinate sets indicates a second gesture that is different from the first gesture upon determining in sub-step A-2) that the first portion of the hand node coordinate sets does not indicate the first gesture, the second portion of the hand node coordinate sets including a reference node coordinate set; 
 H) by the MR device, upon determining in step G) that the second portion of the hand node coordinate sets indicates the second gesture, determining whether at least one target coordinate set has been obtained; 
 I) by the MR device, determining whether the at least one target coordinate set includes a to-be-adjusted coordinate set upon determining in step H) that at least one target coordinate set has been obtained, where a distance between the to-be-adjusted coordinate set and the reference node coordinate set is smaller than a second predetermined distance and is a minimum of all distance(s) each between the reference node coordinate set and a corresponding one of the at least one target coordinate set; 
 J) by the MR device, upon determining in step I) that the at least one target coordinate set includes the to-be-adjusted coordinate set, determining whether a first angle adjustment data piece is found in the MR device, the first angle adjustment data piece being used to update the rotational angle data piece that corresponds to the to-be-adjusted coordinate set; 
 K) by the MR device, upon determining in step J) that the first angle adjustment data piece is not found in the MR device, making the hand rotational angle data piece serve as the first angle adjustment data piece, and performing step B); 
 L) by the MR device, upon determining in step J) that the first angle adjustment data piece is found in the MR device, making the hand rotational angle data piece serve as the second angle adjustment data piece; 
 M) by the MR device, updating the to-be-adjusted coordinate set based on the reference node coordinate set, and updating the rotational angle data piece that corresponds to the to-be-adjusted coordinate set based on the first angle adjustment data piece and the second angle adjustment data piece, followed by deleting the second angle adjustment data piece, and performing step B); and 
 N) by the MR device, deleting the first angle adjustment data piece and the second angle adjustment data piece upon determining in step G) that the second portion of the hand node coordinate sets does not indicate the second gesture. 
   
     
     
         5 . The method as claimed in  claim 4 , wherein the hand node coordinate sets further include a hand joint coordinate set indicating a joint of the hand, and a wrist joint coordinate set indicating a wrist joint of the hand, the second portion of the hand node coordinate sets includes the hand joint coordinate set, the wrist joint coordinate set and the second fingertip coordinate set; and
 wherein step G) includes:
 G-1) acquiring a first vector based on the hand joint coordinate set and the wrist joint coordinate set; 
 G-2) acquiring a second vector based on the hand joint coordinate set and the second fingertip coordinate set; and 
 G-3) determining whether the second portion of the hand node coordinate sets indicates the second gesture by determining whether an angle between the first vector and the second vector is smaller than a predetermined angle. 
   
     
     
         6 . The method as claimed in  claim 4 , wherein the MR device stores a virtual three-dimensional (3D) object;
 wherein sub-step A-3) further includes presenting, based on the target coordinate set and the rotational angle data piece that corresponds to the target coordinate set, the virtual 3D object in a virtual space defined by the MR coordinate system; and   wherein step M) further includes presenting the virtual 3D object in the virtual space based on the to-be-adjusted coordinate set that has been updated and the rotational angle data piece that corresponds to the to-be-adjusted coordinate set and that has been updated.   
     
     
         7 . The method as claimed in  claim 1 , wherein the base includes a first positioning label, and the robotic arm control system further includes a second positioning label disposed to be captured by the camera device and the MR device; and
 wherein the computer device stores a third transformation matrix that is used to perform coordinate transformation from the robot base coordinate system to a first positioning coordinate system defined by the first positioning label;   said method further comprising, before step A):
 i) by the computer device, upon receipt of a first image that is captured by the camera device and that includes the first positioning label, obtaining the first transformation matrix based on the first image and the third transformation matrix, and sending the first transformation matrix to the MR device; and 
 ii) by the computer device, upon receipt of a second image and a third image, obtaining the second transformation matrix based on the second image and the third image, and sending the second transformation matrix to the MR device, where the second image is captured by the camera device and includes the second positioning label, and the third image is captured by the MR device and includes the second positioning label. 
   
     
     
         8 . The method as claimed in  claim 7 , wherein step i) includes:
 i-1) obtaining, based on a first-positioning-label region that is in the first image and that is related to the first positioning label, a label-based transformation matrix that is used to perform coordinate transformation from the first positioning coordinate system to the navigation coordinate system; and   i-2) obtaining the first transformation matrix based on the third transformation matrix and the label-based transformation matrix, and sending the first transformation matrix to the MR device.   
     
     
         9 . The method as claimed in  claim 7 , wherein step ii) includes:
 ii-1) obtaining, based on a second-positioning-label region that is in the second image and that is related to the second positioning label, a first label-based transformation matrix that is used to perform coordinate transformation from a second positioning coordinate system that is defined by the second positioning label to the navigation coordinate system;   ii-2) obtaining, based on another second-positioning-label region that is in the third image and that is related to the second positioning label, a second label-based transformation matrix that is used to perform coordinate transformation from the second positioning coordinate system to the MR coordinate system; and   ii-3) obtaining the second transformation matrix based on the first label-based transformation matrix and the second label-based transformation matrix, and sending the second transformation matrix to the MR device.   
     
     
         10 . The method as claimed in  claim 7 , wherein the robotic arm includes a plurality of links, one of the links having the fixed end of the robotic arm, and another one of the links having the non-fixed end of the robotic arm;
 wherein the MR device stores a world coordinate system, and a virtual robotic arm that has a same structure as the robotic arm and that is used in a virtual object coordinate system to simulate motion of the robotic arm based on a joint angle data piece; and   wherein the virtual robotic arm includes a plurality of virtual links, one of the virtual links including a virtual fixed end of the virtual robotic arm that corresponds to the fixed end of the robotic arm and that is located at a base point coordinate set in the virtual object coordinate system, another one of the virtual links including a virtual non-fixed end that corresponds to the non-fixed end of the robotic arm and that is opposite to the virtual fixed end;   said method further comprising:
 G) by the MR device, obtaining an environmental image data piece that is related to a surrounding environment, and obtaining, based on the environmental image data piece, a first dynamic transformation matrix that is used to perform coordinate transformation from the MR coordinate system to the world coordinate system; 
 H) by the MR device, obtaining an alignment point coordinate set in the MR coordinate system based on the second transformation matrix, the first transformation matrix, the first dynamic transformation matrix, and a coordinate set of an origin in the robot base coordinate system; 
 I) by the MR device, generating and sending to the computer device a request for the joint angle data piece; 
 J) by the computer device, upon receipt of the request for the joint angle data piece, sending an initial joint angle data piece to the MR device, where the initial joint angle data piece includes a plurality of configured angles that correspond to the links of the robotic arm; and 
 K) by the MR device, upon receipt of the initial joint angle data piece, using the initial joint angle data piece as the joint angle data piece, calibrating the base point coordinate set in the virtual object coordinate system based on the alignment point coordinate set in the MR coordinate system, and presenting the virtual robotic arm in a virtual space defined by the MR coordinate system in such a way that the virtual fixed end of the virtual robotic arm is located at the alignment point coordinate set, and that a plurality of virtual configured angles that correspond to the virtual links are respectively set to equal the configured angles included in the initial joint angle data piece. 
   
     
     
         11 . The method as claimed in  claim 10 , wherein step H) includes:
 H-1) obtaining, based on the second transformation matrix and the first dynamic transformation matrix, a second dynamic transformation matrix that is used to perform coordinate transformation from the navigation coordinate system to the world coordinate system;   H-2) obtaining, based on the first transformation matrix, the second dynamic transformation matrix and the first dynamic transformation matrix, a third dynamic transformation matrix that is used to perform coordinate transformation from the robot base coordinate system to the MR coordinate system; and   H-3) obtaining the alignment point coordinate set based on the coordinate set of the origin in the robot base coordinate system and the third dynamic transformation matrix.   
     
     
         12 . The method as claimed in  claim 10 , further comprising, before step B):
 L) by the MR device, determining whether an instruction for simulating a currently planned path is received;   M) by the MR device, upon determining in step L) that the instruction for simulating a currently planned path is not received, performing step B);   N) by the MR device, upon determining in step L) that the instruction for simulating a currently planned path is received, generating and sending a simulation request to the computer device, the simulation request including the target coordinate sets that are obtained respectively in the multiple executions of step A) and that are arranged in the user-defined order;   O) by the computer device, upon receipt of the simulation request, converting, based on the first transformation matrix and the second transformation matrix, the target coordinate sets in the MR coordinate system respectively to the path coordinate sets in the robot base coordinate system, the path coordinate sets being arranged in the user-defined order and cooperatively indicating a simulation path;   P) by the computer device, obtaining, based on the path coordinate sets obtained in step O), a plurality of simulated joint angle data pieces to be used when the virtual non-fixed end of the virtual robotic arm is to move along the simulation path, each of the simulated joint angle data pieces including a plurality of simulated angles that respectively correspond to the virtual links; and   Q) by the MR device, setting, for each of the simulated joint angle data pieces, the virtual configured angles based on the simulated joint angle data piece, and making the virtual non-fixed end of the virtual robotic arm move along the simulation path based on the virtual configured angles that are set for the simulated joint angle data pieces, followed by performing step B).