The current worldwide trend around the latest R&D achievements involves active introduction of robotic equipment in all sectors of the economy. A paper by The Boston Consulting Group (BCG) states that by 2030 the total global market of professional robotics will reach $260 billion [1].

Medical robotics is one of the leading segments of professional robotics by the level of technology employed and demand present. As Joseph Engelberger, father of commercial robotics, said, "... hospitals are the ideal place and the ideal environment for use of robots" [2].

Nevertheless, despite the broad introduction of robotic medical systems, the matters of safety of patients and medical staff that use such systems have not been investigated sufficiently. Some of the published studies point to the documents reporting results of operations that employed robots, and the number of subsequent adverse consequences exceeded one and a half thousand [3]. During the period from 2000 through 2013, surgeries done with the help of robots resulted in death of 144 people. Between 2000 and 2013, the equipment ignited or failed on more than 190 cases. Almost 800 other cases of adverse consequences of robotics-enabled operations resulted from systemic errors such as loss of the video feed [3].

According to the authors of the article, robotic rehabilitation implies slightly different risks for patients: robot arms may move incorrectly (along an unacceptable trajectory or at an unacceptable angle) and thus injure the patient or medical personnel. With this in mind, we undertook to make use of medical rehabilitation robots safer for people by developing a method installing an additional control loop for the robot manipulator's movements that relies on a computer vision system (CVS). "Robot arm movement control" in the context of this study means establishing the fact of that arm reaching a preset point within a local coordinate system. Thus, we employed CVS only to confirm the successfully performed movements without assessing the controlled manipulator's final configuration. For the stated purpose, we analyzed the 3D coordinates for each manipulator's structural components.

The subject we tested the developed method of positioning robotic manipulator's components on was an AR-600E anthropomorphic robot (NPO Androidnaya Tekhnika; Russia). The testing sought to confirm the possible way to improve safety of use and accuracy of positioning of anthropomorphic robotic prostheses of human upper limbs. We paid special attention to finding a solution to the problem of establishing the coordinates of individual components of the manipulator to design its configuration to mimic movements of a human arm in the best possible way. The solution allows designing missing or dysfunctional arm replacements with kinematics matching kinematics of a human arm as close as possible, which makes them safer in use.

It should be taken into account that, unlike industrial robot arms, an anthropomorphic robotic rehabilitation manipulator is not fixed on a rigid base. Coupled with mechanical complexity and a large number of interconnected components, this translates into considerably inaccurate positioning values, which, in some cases, can make the robotic prosthesis dangerous to its owner or people around him/her. Under such conditions, the task of accurate and safe positioning is not a trivial one; it largely depends on the method of designing the mechanism itself.

A suggested solution to this task involves a CVS module integrated into the robotic rehabilitation anthropomorphic manipulator's control loop. This module would monitor the position of the manipulator in its field of view and generate commands to interrupt or correct a potentially dangerous movement. The module is supposed to recognize and track both the manipulator's grip and special markers attached to its components [4, 5]. However, even with a CVS enhancing the traditional methods of estimation of the manipulator's position (relative to the elements of the CVS), the absolute mean calibration error between the system [6, 7] and the manipulator's grip is more than ± 5 cm [4].

The traditional approach to determining the current position of the manipulator's components involves requesting their 3D coordinates from the direct kinematics logic. The inputs are the values transmitted by the angular position sensors of the corresponding drives. The logic also contains the current coordinates by the CVS. At the initial stages, the software must search for the target object and calculate its spatial position relative to the elements of the system or the absolute zero of the kinematics logic. This can be done with the help of position-based (PBVS), image-based (IBVS) visual servoing or hybrid methods. In general, the above methods calculate the needed coordinates by analyzing images; they are applicable to both industrial manipulators and anthropomorphic robots and robotic prostheses of the human hand.

Image-based visual servoing allows comparing the calculated needed and current positions of the manipulator or object on a plane. The difference between the needed and the current positions (the error) is used as feedback. The connection between the received information and changes in the position of the components is made through a Jacobi matrix and the direct kinematic logic of the robot [8]. There are a large number of methods to determine this connection [9–11]. It should be taken into account that a single marker on the object (either the manipulated object or the manipulator itself) enables control over only two degrees of freedom. At least four markers are needed to control six degrees of freedom. Greater number of markers also increases the probability of an unambiguous decision supporting the control command [11, 12]. The IBVS method does not allow linear control of the robot components and does not rely on 3D information about position of the manipulated object. This leads to generation of non-optimal or unrealizable trajectories, a problem that can be solved through adjustments by selected visual parameters [13–15].

The PBVS method implies the coordinates of objects inside the manipulator operating space are determined relative to the coordinate system of the camera part of the CVS. The parameters of the geometric model of the tracked object and camera parameters are factored in together for this purpose. The parameters of the tracked manipulator in the operating space are known; the changes of these parameters can be tracked by responses from the robot's kinematics logic. Geometrical parameters of the manipulated object, on the contrary, strongly depend on the CVS parameters and the adopted methods of 3D localization [16–22].

Hybrid visual servo-enabled control involves both the IBVS and PBVS methods. This approach improves the accuracy of the generated commands through separation of control over manipulator's degrees of freedom [23–27]. Systems that rely on such an approach are less dependent on the robot cameras' calibration accuracy and give a more true idea of the objects' geometry. However, such systems harder to build and consume more computing resources. Moreover, they do not eliminate the risk of non-optimal or unrealizable trajectories, which can still impair the process due to positioning errors and/ or incorrect estimation of the 3D coordinates of the tracked objects by the CVS.

Calculation of the optimal positions of the manipulator's grip should factor in mechanical limitations of the joints peculiar to both the initial positioning and the subsequent manipulation stages. In addition, since of anthropomorphic prostheses are supposed to be a safer robotic rehabilitation device fir for use in "human" environment, positioning of the manipulator may include stopping before obstacles while moving to target position.

With the current methods of processing data inputs from the CVS, development of the software generating movement trajectories for the AR-600E manipulators does not exclude the possibility of generation of a trajectory (arm to target object) that is either unsafe or unrealizable. In addition, we undertook to enable generation of the trajectories that mimic natural movements of the human arm. In the context of this study, we assessed various options of solutions to the manipulator's inverse kinematics problem and formulated a method that ensures building the movement trajectory as expected.

This study aimed to develop a method making use of robotic medical rehabilitation devices safer by designing and testing an algorithm for calculation of the angular positions of robotic manipulators or robotic prostheses used for rehabilitation purposes and capable of reproducing the natural human.

METHODS

For this study, we developed a number of algorithmic solutions allowing to build a movement trajectory for an anthropomorphic manipulator that closely resembles that of a human arm. The solutions were implemented as software that controls the manipulator of an AR-600E anthropomorphic robot in a simulation environment enabled with quaternion algebra. The manipulator includes the following components (fig. 1):

1. the Shoulder groups of components moves the Elbow component and the lower components of the manipulator along the frontal (ShoulderS), sagittal (ShoulderF) and vertical axes (ElbowR);
2. the Elbow component moves its child components along the sagittal axis;
3. the WristR component moves its child component along the vertical axis;
4. the WristF and WristS components move the hand along the frontal and sagittal axes, respectively.

The fingers are driven with actuators located on the WristS component.

The Forward and Backward Reaching Inverse Kinematics (FABRIK) method [28] was used to calculate the spatial position of the manipulator's components. This method accounts for the constraints and represents the components of the components, which, combined, allow bringing the hand to the target position in the 3D space.

Two cameras on the head of the anthropomorphic mechanism captured color images within the operating space. Machine learning methods enabled control of the movements of manipulator's grip and tracking thereof, assessment point being whether it has reached the set coordinate in the operating space or not. The main task set before the routines that incorporate machine learning is to detect the grip and assess how it performs a given task within the operating space.

Two approaches to assess correctness of the grip positioning were applied: 1) by requesting responses from the manipulator drives and modeling the current configuration of the manipulator based thereon, and 2) with the CVS of anthropomorphic mechanism. Configuration of the manipulator in its entirety is not controlled, since the CVS' field of view does not cover all of its components. This task is solved after modeling the spatial position of the manipulator, through calculation of angular positions of the drives that bring it to the set 3D coordinates. Below, we consider the possible ways of their calculation on the example of the Shoulder group (fig. 1) the movements of which affect the spatial position of all other components of the manipulator. The following methods of angular positions calculation were compared:

1) using orthogonal projections of specific points on the manipulator's components. The angle between two points (for example, axis of the Elbow component and the Shoulder group) was calculated by the following formula:

form. 1

where a is the module of the y coordinate for the corresponding axis of the component, and c is the distance between components in the corresponding orthogonal plane;

2) as angle of rotation of the orthogonal plane between projection of the Elbow component during initialization of the kinematic logic and projection in the analyzed plane of its movement. We used the following formula in this case:

form. 2

where ā and ƀ are the 3D vectors between which the angle is calculated, and a_n, b_n are the corresponding coordinates of 3D vectors after modeling of the orthogonal projections;

3) through the Elbow component's 3D coordinates projection onto the corresponding Shoulder group movement planes:

form. 3

where dist is the distance between the current coordinate on the Elbow component axis and the common axis (null) of ShoulderF, ShoulderS components (Shoulder group). We used both (1) and (2) to calculate the angular positions;

4) with the help of the "Shortest arc" method used by game developers to calculate the shortest arc of movement of connected components of virtual objects from the initial point to a given point in 3D space. Practically, implementation by this method means generation of the rotation quaternion from the double rotation and zero rotation quaternions, with the resulting quaternion being the sum of the double rotation quaternion and the identity quaternion;

5) using the algorithm suggested by the authors of this article.

The angular positions calculation method suggested and tested by the authors makes use of a matrix containing interrelated positions of the manipulator components, i.e., a set of coordinates corresponding to certain angles. The increment between them is set during initialization of the algorithm, which prevents repeated generation of the matrix. The input data for the suggested algorithm is the 3D coordinate of a point (center of the hand; fig. 1) on the manipulator's grip. Next, application of the FABRIK approach yields a set of coordinates of the axis points of the main components’ nodes, which, combined, form configuration of the manipulator in 3D space. After that, inside the generated matrix of interrelated positions, using the principle of minimum distances (dist; formula (3)), the most suitable angular position of the drive is selected between the coordinates of the estimated position of the Elbow component and the coordinates stored in the matrix. If necessary, the angular position is adjusted through generation of a local matrix of interrelated positions with a more accurate increment within a small range of rotation angles. Then the angular position of the Elbow component drive is calculated by formula (1), with a being the y axis value by the orthogonal projection of the target position and c being the length of the component. The ElbowR drive's angular position is calculated relative to the axis of the WristR component (fig. 1). This step involves a number of iterations: sequentially formed quaternions enable rotation of WristR to a given increment. The procedure stops upon reaching the minimum distance between the coordinates calculated by the FABRIK method and the coordinates extracted from the matrix of interrelated positions.

RESULTS

The experimental part of the study involved application of the considered methods to calculate angular positions of drives of the AR-600E anthropomorphic robot's manipulator in a simulation environment. For this purpose, operator manually set target positions of the components and registered the resulting angular positions of their drives and 3D coordinates of the axes of Elbow and WristS drives. The next step was to compare the results and select the best method by proximity of the resulting coordinates with the coordinates of axes of the components registered by the operator. Figure 2 (fig. 2) shows one of the manipulator configurations recorded during the experimental part of the study.

Table 1 (tab. 1) shows angular positions of the components of the Shoulder group and Elbow component:

1. 0 — initial position for the anthropomorphic mechanism;
2. 1 — angular positions set by the operator;
3. 2 — angular positions for the configuration calculated by formula (1);
4. 3 – angular positions for the configuration calculated by formula (2);
5. 4 — angular positions for the configuration calculated through the projection of the coordinate onto the respective axis by formula (1);
6. 5 — angular positions for the configuration calculated through the projection of the coordinate onto the respective axis by formula (2);
7. 6 — angular positions calculated by the suggested method.

Elbow and WristS cells of Table 1 contain the 3D coordinates of axes of these components that were discovered through application of the formed angular positions of the drives to them. Cells Elbow and WristR present information about angular positions of the drives (in degrees) that move the respective components to target points in 3D space.

Table 2 (tab. 2) below illustrates the absolute difference between the coordinates of the Elbow and WristR axes, i.e., the difference between the coordinates resulting from operator's actions (manual movement of the components) and coordinates discovered through application of each of the considered method to calculate angular positions to which the drives moved. Table 2 does not include information on positions 0 and 1.

DISCUSSION

Analysis of the results given in tab. 1 and tab. 2 allows deducing that the suggested drive angular position calculation method ensures generation of the target configuration of the manipulator and movement of its components to the target points. There are also visible differences between the manipulator components' position coordinates calculated by the suggested algorithm and learned as a result of manual movements by the operator. Axis coordinate of the WristS component presents the greatest discrepancy between the obtained results and the reference values. To compensate for this error, we applied an algorithm that, like the one discussed above, uses a matrix of interrelated positions. The WristS component should be moved to its initial position for this purpose. Then, the matrix of interrelated positions is scanned for the drive's angular position value that fits the minimum distances condition. To speed up performance of the algorithm, we decided to calculate the initial value of the WristF drive's angular position:

form. 4

where angle is a precalculated rotation angle of the WristS drive along the sagittal axis; nullWristF is the static angle of rotation of WristS relative to WristF, calculated during the initialization of the anthropomorphic robot's software; a is the length of the forearm of the anthropomorphic robot; b is the distance from the WristF axis to the WristS axis; c is the distance from the axis of WristS to the axis of WristR.

We calculated the angular position of the WristR drive in a similar way to the respective calculation for ElbowR. Table 3 (tab. 3) presents adjustments of the manipulator configurations shown in tab. 1.

The "Difference between deviations" line in tab. 3 reflects the magnitude and direction of changes of distances (in millimeters) between the target 3D coordinates of the manipulator components and their values calculated before and after adjustment for the angular positions of the WristS drive. The analysis of the presented data allows deducing that an additional stage of adjustment of the coordinates makes manipulator movements more accurate. In such a case, the movement error does not exceed 0.5 mm. This mechanism can be used when setting a large search increment in generation of the interrelated positions matrices. A more accurate adjustment of the configuration is undertaken at an additional adjustment stage, which ultimately speeds up the suggested algorithm.

CONCLUSIONS

We developed an algorithm for calculation of angular positions of the manipulator components' drives that produces the most accurate and predictable result (tab. 2, tab. 3), which ultimately allows forming the configuration designed by the operator. The algorithm also minimizes the probability of an unpredictable result and robot arm movements along a trajectory unsafe for human beings. It should be noted separately that the accuracy of calculation of the coordinates with the help of the suggested algorithm depends directly on the search increment value (in degrees) set for the interrelated positions matrices generation stage. The algorithm was coded and optimized in C++. Its execution time on a personal computer (Intel Core i7–4770 3.40 GHz, RAM 16 Gb) ranged from 5 to 8 ms, which is sufficient for the purpose of controlling the manipulator of an anthropomorphic robot and a robotic prosthesis. The accuracy of the drives' angular position calculations can be improved by reducing the increment used at interrelated positions matrices generation stage. This, however, would require more RAM capacity for the control software and slow down execution of the algorithm. The way to mitigate this problem is to add a modified version of the suggested algorithm to the control software, i.e. a version that would calculate angular positions of the manipulator drives factoring in its previous configuration. In this case, the search in the matrices of interrelated positions will be significantly more narrow, which will speed up generation of the move command by the manipulator/robotic prosthesis control software while also improving the accuracy of its positioning. In the context of application of the suggested algorithm, CVS will be used only to control the correctness of execution of an action by the manipulator, which greatly simplifies the algorithmic part, reduces the computational load and time to generate a command. This approach will increase the safety of medical rehabilitation robotic devices for which the key performance criteria are speed of reaction and accuracy of execution of actions.