Volume 16 Number 6
December 2019
Article Contents
Hong-Xuan Ma, Wei Zou, Zheng Zhu, Chi Zhang and Zhao-Bing Kang. Selection of Observation Position and Orientation in Visual Servoing with Eye-in-vehicle Configuration for Manipulator. International Journal of Automation and Computing, vol. 16, no. 6, pp. 761-774, 2019. doi: 10.1007/s11633-019-1181-z
Cite as: Hong-Xuan Ma, Wei Zou, Zheng Zhu, Chi Zhang and Zhao-Bing Kang. Selection of Observation Position and Orientation in Visual Servoing with Eye-in-vehicle Configuration for Manipulator. International Journal of Automation and Computing, vol. 16, no. 6, pp. 761-774, 2019. doi: 10.1007/s11633-019-1181-z

Selection of Observation Position and Orientation in Visual Servoing with Eye-in-vehicle Configuration for Manipulator

Author Biography:
  • Hong-Xuan Ma received the B. Sc. degree in intelligent science and technology from Central South University, China in 2016. He is currently a Ph. D. degree candidate in control science and engineering at Institute of Automation, Chinese Academy of Sciences, China. He is also with University of Chinese Academy of Sciences, China. His research interests include visual servoing, robot localization, computer vision and navigation. E-mail: mahongxuan2016@ia.ac.cnORCID iD: 0000-0001-6422-1441

    Wei Zou received the B. Eng. degree in control science and engineering from Baotou University of Iron and Steel Technology, China in 1997, the M. Eng. degree in control science and engineering from Shandong University of Technology, China in 2000, and the Ph. D. degree in control science and engineering from Institute of Automation, Chinese Academy of Sciences (IACAS), China in 2003. Currently, he is a professor at the Research Center of Precision Sensing and Control, IACAS. His research interests include intelligent robotics, visual servoing, robot localization and navigation. E-mail: wei.zou@ia.ac.cn (Corresponding author)ORCID iD: 0000-0003-4215-5361

    Zheng Zhu received the B. Sc. degree in automation from Zhenzhou University, China in 2014. He is currently a Ph. D. degree candidate in control science and engineering at Institute of Automation, Chinese Academy of Sciences, China. He is also with University of Chinese Academy of Sciences, China. His research interests include computer vision, deep learning and robotics. E-mail: zhuzheng2014@ia.ac.cn

    Chi Zhang received the B. Sc. degree in automation from Shenyang University of Chemical Technology, China in 2015, and the M. Sc. degree in control engineering from Harbin University of Science and Technology, China and Institute of Automation, Chinese Academy of Sciences, China in 2018. He is currently a Ph. D. degree candidate in control science and engineering at Institute of Automation, Chinese Academy of Sciences, China. His research interests include intelligent control and reinforcement learning. E-mail: zhangchi2015@ia.ac.cn

    Zhao-Bing Kang received the B. Eng. degree in mechanical engineering and automation from Dezhou University, China in 2008, the M. Eng. degree in mechanical and electronic engineering from Harbin Institute of Technology, China in 2011. Currently, he is a Ph. D. degree candidate in control science and engineering at Institute of Automation, Chinese Academy of Science, China. His research interests include visual servoing and robot location and navigation. E-mail: kangzhaobing2017@ia.ac.cn

  • Received: 2018-11-21
  • Accepted: 2019-04-09
  • Published Online: 2019-06-13
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Figures (19)

Metrics

Abstract Views (1239) PDF downloads (41) Citations (0)

Selection of Observation Position and Orientation in Visual Servoing with Eye-in-vehicle Configuration for Manipulator

Abstract: In this paper, we propose a method to select the observation position in visual servoing with an eye-in-vehicle configuration for the manipulator. In traditional visual servoing, the images taken by the camera may have various problems, including being out of view, large perspective aberrance, improper projection area of object in images and so on. In this paper, we propose a method to determine the observation position to solve these problems. A mobile robot system with pan-tilt camera is designed, which calculates the observation position based on an observation and then moves there. Both simulation and experimental results are provided to validate the effectiveness of the proposed method.

Hong-Xuan Ma, Wei Zou, Zheng Zhu, Chi Zhang and Zhao-Bing Kang. Selection of Observation Position and Orientation in Visual Servoing with Eye-in-vehicle Configuration for Manipulator. International Journal of Automation and Computing, vol. 16, no. 6, pp. 761-774, 2019. doi: 10.1007/s11633-019-1181-z
Citation: Hong-Xuan Ma, Wei Zou, Zheng Zhu, Chi Zhang and Zhao-Bing Kang. Selection of Observation Position and Orientation in Visual Servoing with Eye-in-vehicle Configuration for Manipulator. International Journal of Automation and Computing, vol. 16, no. 6, pp. 761-774, 2019. doi: 10.1007/s11633-019-1181-z
    • Visual servoing is a technique that utilizes visual information from a camera to control the relative pose between a robot and a target, which has already been applied to various platforms for different tasks[18]. Relative to the position of the camera, traditional visual servoing configuration can be classified into eye-in-hand and eye-to-hand configurations[9, 10].

      In eye-in-hand configuration, a camera is installed on a manipulator, and there is a fixed transformation between the camera coordinate system and the manipulator coordinate system. A marker is attached to a target, and based on the visual feedback, the manipulator adjusts its posture. As the camera is moving to the target, the relative pose errors between the manipulator and the target are gradually reduced. However, some problems will occur in this configuration: sometimes the marker on the target may be on the edge of images, and it is easily out of the field of view (FOV) when the manipulator is moving; at some positions, because the camera is too close to the object, it is impossible to keep the whole marker in the field of view; the marker in image has a great deal of perspective aberrance, and multiple feature points may be squeezed onto a few pixels in images, which will affect the accuracy of feature point extraction; sometimes markers on the object may be blocked in images.

      In eye-to-hand configuration, a camera is fixed in a place to observe a target and a manipulator simultaneously. The relative pose between the manipulator and camera can be obtained by eye-hand calibration. According to visual feedback, the manipulator completes the visual servo task. However, when the manipulator operates the target, there are also some image problems: the camera′s field of view is fixed; sometimes markers on the object may be blocked in images.

      Current research on the image problems mentioned above is mostly concentrated on the FOV problem. Thuilot et al.[11] proposed a method to plan a trajectory in the image to prevent the target out of view. Chesi et al.[12] presented a method to rotate the camera to keep the target in the center of images when it tends to the edge of vision. Mezouar and Chaumette[13] proposed an approach to plan a path in image space, for the camera mounted on a manipulator, to keep the target in the field of view. Chesi and Hung[14] presented a method to achieve visual servoing for manipulators with visibility constraints while minimizing a cost function. These methods solve the FOV problem by planning a trajectory and controlling the robot manipulator to track the trajectory.

      However, the quality of imaging, including perspective aberrance and imaging area of target, is rarely researched such as perspective aberrance and imaging area of target. Perspective aberrance and small projection areas bring difficulty perspective aberrance and small projection area bring difficulty to the recognition and extraction accuracy of feature points. Having too large a projection area for the target means the target can easily be out of view. In the traditional visual servoing configuration, the motion space of the camera is not enough, which leads to limitations in finding observation positions. Thus, a novel visual servoing configuration named eye-in-vehicle is designed. The camera is mounted on a vehicle, and the vehicle moves to the optimal observation position to observe the target. Based on the images taken by the camera mounted on the vehicle, the manipulator is controlled to operate the target.

      In this paper, we mainly focus on the vehicle (mobile robot) control system, which is used to determine the observation position. The system consists of a mobile robot equipped with an active camera. There is some research about how to control a pan-tilt camera[15, 16]. Tinh et al.[17] proposed a new control law for an integrated mobile robot pan-tilt camera system, allowing the mobile robot to track the desired position. Le and Pham[18] proposed a dynamic and control method to track a moving object by a mobile robot with a pan-tilt binocular camera, and the integrated system was highly optimal response qualities for minimizing error of velocity and energy of the system. Fang et al.[19] designed an adaptive pan camera controller to keep the target in the center of view, and the convergence of its control law was proved. In this paper, an adaptive pan-tilt camera controller previously designed by us is applied, which adds compensation of velocity and angular velocity of mobile robot and does not need the depth information of the target.

      In the visual servoing process, it is important to select the observation position for the camera. Some researchers proposed various methods to determine camera position. Sharma and Hutchinson[20] proposed a measure of manipulability and deigned a cost function based on manipulability to determine the optimal observation position. Tarabanis et al.[21] presented a method to determine the observation position for a robotic vision system and the visibility was taken into consideration. Zheng et al.[22] proposed a new scheme that moved the camera to the desired position with proper distance and viewing angle from the objects to acquire proper images for visual feedback control. However, the research does not solve the image problems facing eye-in-hand and eye-to-hand configuration as we mentioned above. Hence, we propose a novel method for the camera to find an observation position, which can solve the image problems mentioned above.

      In this paper, our main contributions can be summarized as follows:

      1) A new visual servoing configuration is designed. A camera is mounted on a pan-tilt platform of the mobile robot, which is able to select more observation positions.

      2) In this paper, a method to determine the optimal observation position is presented. By this method, a position can be determined where the images taken by camera have less perspective aberrance.

      3) Some experiments are conducted to validate the effectiveness of the proposed method.

      The remainder of this paper is organized as follows. In Section 2, some basic concepts are defined and a series of coordinate systems are described. In Section 3, the eye-in-vehicle configuration is proposed. In Section 4, the vehicle control system is described. In Section 5, the optimal and suboptimal observation positions for the camera are defined and a strategy to determine these positions is proposed. In Section 6, both simulations and experiments are conducted and the validity of our method is verified. In Section 7, some conclusions and directions for future work are given.

    • The mobile robot is equipped with a pan-tilt pinhole camera. According to the feedback from images, the mobile robot can be driven to the calculated observation position while the camera keeps the target in the center of view. In the camera model, the relationship between camera space and image space is denoted as follows:

      $ c_z\left[ {\begin{array}{*{20}{c}} u\\ v\\ 1 \end{array}} \right] = \left[ {\begin{array}{*{20}{c}} {{\alpha _x}}&0&{{u_0}}\\ 0&{{\alpha _y}}&{{v_0}}\\ 0&0&1 \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {c_x }\\ {c_y}\\ {c_z } \end{array}} \right]= M\left[ {\begin{array}{*{20}{c}} {c_x }\\ { c_y}\\ {c_z } \end{array}} \right] $

      (1)

      where M is the camera intrinsic matrix, $ {\alpha _x} $ and $ {\alpha _y} $ are the focal lengths of the camera in pixel dimensions, $ {({u_0},{v_0})} $ are the coordinates of the principal point, $ (^cx,\;^cy,\;^c\!z) $ is a point in camera coordinate and $ {({u},{v})} $ is its corresponding image coordinate.

    • As illustrated in Fig. 1, there are five coordinate systems. Let $ F_r $, $ F_c $, $ F_t $ denote the coordinate systems of vehicle, camera and target, respectively. The other two coordinate systems are the auxiliary coordinate systems. The direction of $ Z_r $, $ Y_a $ and $ Y_b $ are the same, which are the shaft of the pan motor. The direction of $ X_b $ and $ X_c $ are the opposite, which are the shaft of the tilt motor. The direction of the $ Z_a $ and $ Z_b $ are the same. The angle from $ X_r $ to $ Z_a $ is denoted as $ \alpha $, and the angle from $ Z_b $ to $ Z_c $ is denoted as $ \beta $. The height between the origin of $ F_a $ and the origin of $ F_b $ is denoted as H. A rectangular marker as a kind of common marker, is attached to the object. In this paper, the coordinates of its four feature points are known.

      Figure 1.  Coordinate system. Ft is the target frame. Fr is the mobile robot frame. Fc is the camera frame. The origin of Fa is the same with Fr, and the origin of Fb is the same with Fc. When pan motor rotates, Fa will rotate around axis Ya. When tilt motor rotates, Fc will rotate around axis Xc. Color versions of the figures in this paper are available online.

    • The traditional visual servoing configuration can be classified into eye-in-hand and eye-to-hand configurations. Based on the actual situation, they are changed into many kinds of configuration as shown in Fig. 2.

      Figure 2.  Common visual servoing configuration

      However, the cameras are still restricted in motion range; they are not able to find more observation positions with less perspective aberrance and appropriate imaging area of target. Thus, a novel visual servoing configuration named eye-in-vehicle is presented in Fig. 3.

      Figure 3.  Eye-in-vehicle configuration

      In the eye-in-vehicle configuration, the vehicle is equipped with a pan-tilt camera, which is used to track the target. A laser positioning device is mounted on the vehicle, and the relative pose between the manipulator and the vehicle can be calculated. A marker is attached to the target, and by observing the marker, the mobile robot can calculate the relative pose between the vehicle and the target. Then, the relative pose between the manipulator and the target can also be obtained. Next, the manipulator will adjust its pose to operate the target. In the process of operation, the mobile robot will adjust its observation position in real time according to the observation effect, which can guarantee the observability of marker.

      Compared with the traditional configuration, the camera mounted on a pan-tilt platform of a mobile robot has more observation positions to select. By this configuration, the mobile robot is able to move to a position where the image taken by camera has less perspective aberrance. It is helpful to reduce visual measurement errors. This configuration is also beneficial to select a observation position with proper imaging area of target.

    • In this paper, we mainly focus on the mobile robot control system, which is used to adjust the observation position. It consists of an image processing module, pan-tilt camera controller, vehicle controller, and the observation position planner. The image processing module is used to extract feature points in images. The pan-tilt camera controller is used to control the pan motor and tilt motor to track the target. The vehicle controller is used to drive the vehicle to the calculated observation position. The whole process is shown in Fig. 4.

      Figure 4.  Coordinate system. Ft is the target frame. Fr is the mobile robot frame. Fc is the camera frame. The origin of Fa is the same with Fr, and the origin of Fb is the same with Fc. When pan motor rotates, Fa will rotate around axis Ya. When tilt motor rotates, Fc will rotate around axis Xc.

    • Inspired by [19], we have previously designed an adaptive pan-tilt camera controller. It can keep the target in the view during the motion of mobile robot. This controller can compensate the angular velocity of the pan motor and tilt motor according to the vehicle′s velocity and angular velocity.

      In order to keep the target in the center of images, the image errors are defined as

      $ \left\{ \begin{aligned}&{e_u} = u - {u_0}\\ &{e_v} = v - {v_0} \end{aligned} \right. $

      (2)

      where $ {({u},{v})} $ represents the center point of target in image plane. The adaptive pan-tilt camera controller is

      $ {\omega _\alpha } = - {\omega _r} + \frac{{{K_1}D{e_u} + \hat{\lambda}D{\varOmega _1} - {K_2}B{e_v} - \hat{\lambda}B{\varOmega _2}}}{{BC-AD}}{\rm{ }} \\ $

      (3)

      $ {\omega _\beta } = - \frac{{{K_1}C{e_u} + \hat{\lambda}C{\varOmega _1}-{K_2}A{e_v} -\hat{\lambda}A{\varOmega _2}}}{{BC-AD}} $

      (4)

      where $ {\varOmega _1} $ and $ {\varOmega _2} $ are determined as

      $ \begin{array}{l} {\varOmega _1} = \dfrac{{V_r}}{{{\alpha _y}}}({\alpha _x}\sin \alpha - {e_u}\cos \alpha \cos\beta)({e_v}\cos\beta + {\alpha _y}\sin \beta ) \end{array} $

      (5)

      ${\varOmega _2} \!=\! -\dfrac{{V_r}}{{{\alpha _y}}}({e_v}\cos \alpha \cos\beta \!+\! {\alpha _y}\cos\alpha \sin \beta )({e_v}\cos\beta \!+\! {\alpha _y}\sin \beta )$

      (6)

      and A, B, C, D are

      $ A = {\rm{ }}\frac{{{\alpha _x}^2{\alpha _y}\cos \beta + {\alpha _x}^2\sin \beta {e_v} + {e_u}^2{\alpha _y}\cos \beta }}{{{\alpha _x}{\alpha _y}}} $

      (7)

      $ B = -\frac{{{e_u}{e_v}}}{{{\alpha _y}}} $

      (8)

      $ C = \frac{{{\alpha _y}\sin \beta {e_u} + {e_u}{e_v}\cos \beta }}{{{\alpha _x}}} $

      (9)

      $ D = -\frac{{\alpha _y^2 + e_v^2}}{{{\alpha _y}}} $

      (10)

      where $ K_1 $ and $ K_2 $ are the positive gains of $ \omega _\alpha $ and $ \omega _\beta $, $ e_u $ and $ e_v $ are the image errors which can be obtained by image processing, $ V_r $ and $ \omega_r $ are the robot′s velocity and angular velocity respectively, which can be determined by encoders of two wheels, $ \alpha_x $ and $ \alpha_y $ are the camera internal parameter, which can be obtained by camera calibration, $ \lambda $ is an adaptive parameter, and it is on-line updated by

      $ \begin{array}{l} \dot {\hat{\lambda}} = r({\varOmega _1}{e_u} + {\varOmega _2}{e_v}){\rm{}} \end{array} $

      (11)

      where $ r $ is a positive update gain.

      By the Lyapunov method, the pan-tilt camera controller (3) and (4) with the adaptive law (11) can guarantee that the image coordinate of the tracked point converges to $ (u_0,v_0) $, that is

      $ {\lim\limits_{t \to +\infty}} e_u(t) = 0 $

      (12)

      $ {\lim\limits_{t \to +\infty}} e_v(t) = 0 .$

      (13)
    • After acquiring the coordinate of the observation position in $ F_r $, the mobile robot is driven by a proportional controller. According to Fig. 5, we define the state $ \theta_e $:

      Figure 5.  Kinematics model of the vehicle. The camera keeps the target in view and the mobile robot moves to the observation position.

      $ \theta_e = \arctan \frac{y_r}{x_r} $

      (14)

      where $ (x_r,y_r) $ represents the planned coordinates. Then a simple proportional controller is designed to drive the robot to the calculated observation position:

      $ \omega_r = K_{\omega_r}\theta _e $

      (15)

      $ {V_r} = {K_v}{\rm sgn}(x_r)(x_r^2 + y_r^2) $

      (16)

      where $ K_{\omega_r} $ and $ {K_v} $ represent the gains of velocity and angular velocity. In addition, to avoid fast velocity, a velocity limit is set.

    • In visual servoing systems, the images taken by camera have an importance to the control of the manipulator. If the observation position of the camera is not constrained, the camera may be placed at many inappropriate places, which will cause some image problems: the camera is too close to the target, and the projection area of target in image is not proper; the perspective aberrance of target in images is large, and multiple feature points may be squeezed onto a few pixels in images; the camera may be in the back of marker, which will cause the visual servo task to fail.

      To avoid these problems, we propose a method to determine the observation position based on the rectangular target. At the beginning, some conceptions will be described.

      Optimal observation positions (OOP): at the OOP, $ \overline{O_cP_1} = \overline{O_cP_2} = \overline{O_cP_3} = \overline{O_cP_4} $. According to the geometric relationship, it can be known that in the image the length ratio of the opposite side of the square target is 1, and the image has least perspective aberrance, the length ratio of the opposite side of the square target is 1. It can be known that the OOP locates on the $ Z_t $ axis in Fig. 6. The projection area of target is neither too large nor too small. It should be within an appropriate range, otherwise a suboptimal observation position needs to be determined.

      Figure 6.  Target frame. P5 is at the origin of the target frame, and P6 is at (0, 0, –1). Space I represents the space in front of the target where Zt < 0, and Space II represents the space in the back of the target where Zt ≥ 0.

      Suboptimal observation positions (SOP): because the height of the camera is limited, sometimes the camera can not reach the OOP, some compromises are made. At the SOP, $ \overline{O_cP_1} = \overline{O_cP_4} $ and $ \overline{O_cP_2} = \overline{O_cP_3} $. The images taken by the camera have less perspective aberrance. The camera is in front of the target. And the projection area of target in image is within the set threshold range.

      At first, the mobile robot determines whether the OOP exists. If the OOP exists, the mobile robot will move to the OOP; otherwise, it will move to the SOP.

      To illustrate, we define two homogeneous coordinates $ ^t\!{P_5} = [0\!\quad\! 0\!\quad\! 0\!\quad\! 1]^{\rm T} $ and $ ^t\!{P_6} = [0\!\quad\! 0\!\quad\! -1\!\quad\! 1]^{\rm T} $ in Fig. 6. $ ^b\!{P_5} $ and $ ^b\!{P_6} $ are their corresponding coordinates in $ F_b $.

      $ {}^b\!{P_5} =\; ^b\!T_c\;{}^cT_t\;{^t\!P_5} $

      (17)

      $ {}^b\!{P_6} =\, ^b\!T_c\;{}^cT_t\;{^t\!P_6} $

      (18)

      where $ ^cT_t $ can be obtained by method[23], and $ ^bT_c $ can be obtained by tilt angle $ \beta $, which are read from the encoder of tilt motor.

      Let $ L_t $ denote the ray from $ P_5 $ to $ P_6 $, and let $ \overrightarrow {L_t} $ denote the unit vector from $ P_5 $ to $ P_6 $. In $ F_b $, the horizontal plane where the camera′s optical center locates is denoted as $ ^b\!y = 0 $. If $ L_t $ intersects with Plane A, we have

      $ {}^b\!{P_{5y}} + t({}^b\!{P_{6y}} - {}^b\!{P_{5y}}) = 0 $

      (19)

      where the index y denotes the y-coordinate of a point, and $ b $ denotes the coordinate in $ F_b $. From (19), $ t $ can be obtained. It is worth mentioning that $ |t| $ denotes the distance from the target to the camera′s optical center, and the sign of t denotes whether the intersection locates in Space I or Space II. Small value of $ |t| $ can cause the target images to be too large or even incomplete, and great value of $ |t| $ can cause the target images to be too small. Hence, t needs to be constrained. Based on t, whether the OOP exists can be determined.

    • Based on experience, two distance thresholds $ T_1 $ and $ T_2 $ are set. If $ t > 0 $ and $ T_1<t<T_2 $, it indicates that the intersection locates in Space I (see Fig. 7) and the distance t is proper, and the OOP exists.

      Figure 7.  Diagram about the OOP. Plane A is the horizontal plane where the camera′s optical center locates.

      According to the results of $ t $, we can obtain the coordinates of OOP by

      $ {}^b\!{P_{optimal}} = {}^b\!{P_{5}} + t({}^b\!{P_{6}} - {}^b\!{P_{5}}) $

      (20)

      $ {}^r{P_{optimal}} = {}^r{T_b}{}^a{T_b}{}^b\!{P_{optimal}} $

      (21)

      where $ {}^{b}{P_{optimal}} $ and $ {}^{r}{P_{optimal}} $ represent the OOP in $ F_b $ and $ F_r $, respectively. Based on $ {}^{r}{P_{optimal}} $, the mobile robot will be driven to the OOP.

      During the movement of the mobile robot to the OOP, the pan-tilt camera is controlled to track the center of the target. With this configuration as shown in Fig. 8, $ {{{O}}_c}{{O}} \bot ABCD $ and $ {{{O}}_c}{{{O}}_1} \bot {A_1}{B_1}{C_1}{D_1} $ hold. Based on this, the conclusion that the images taken at the OOP by camera have least perspective aberrance can be obtained.

      Figure 8.  Diagram of camera projection at the OOP when the camera points to the center of the target. ABCD is the rectangular target, and ${ A_1B_1C_1D_1 }$ is the projection of target in the image. ${ O_cO }$ is the optical axis of camera, and ${ O }$ is the center of the target.

      Proof.

      $ \begin{array}{l} \because OA = OB,\;{{\rm{O}}_c}{\rm{O}} \bot ABCD\\ \therefore \Delta AO{O_c} \buildrel \Delta \over = \Delta BO{O_c},\;\angle A{O_c}{\rm{O = }}\angle {\rm{B}}{{\rm{O}}_c}O\\ {\rm similarly},\;\Delta DO{O_c} \buildrel \Delta \over = \Delta CO{O_c}\\ \because \angle {A_1}{{\rm{O}}_c}{{\rm{O}}_1} = \angle {B_{\rm{1}}}{O_{\rm{c}}}{O_1},\;{{\rm{O}}_c}{{\rm{O}}_1} \bot {A_1}{B_1}{C_1}{D_1}\\ \therefore \Delta {A_{\rm{1}}}{O_{\rm{1}}}{O_c} \buildrel \Delta \over = \Delta {B_{\rm{1}}}{O_{\rm{1}}}{O_c}\\ \therefore {A_{\rm{1}}}{O_c}={B_{\rm{1}}}{O_c},\;{\rm similarly},\; {{\rm{D}}_{\rm{1}}}{O_c}={{\rm{C}}_{\rm{1}}}{O_c}\\ \because AO_c ={B}{O_c}, {D}{O_c}={C}{O_c},{A}{D}={B}{C}\\ \therefore \Delta A{O_c}D \buildrel \Delta \over = \Delta B{O_c}C,\;\angle A{O_c}{\rm{D = }}\;\angle {\rm{B}}{{\rm{O}}_c}C\\ \because {A_{\rm{1}}}{O_c}={B_{\rm{1}}}{O_c},\;{{\rm{D}}_{\rm{1}}}{O_c}={{\rm{C}}_{\rm{1}}}{O_c},\;\angle A{O_c}{\rm{D = }}\;\angle {\rm{B}}{{\rm{O}}_c}C\\ \therefore \Delta {A_{\rm{1}}}{{\rm{D}}_{\rm{1}}}{O_c} \buildrel \Delta \over = \Delta {B_{\rm{1}}}{{\rm{C}}_{\rm{1}}}{O_c},\;{A_{\rm{1}}}{{\rm{D}}_{\rm{1}}}={B_{\rm{1}}}{{\rm{C}}_{\rm{1}}}\\ {\rm similarly},\;{A_{\rm{1}}}{{\rm{B}}_{\rm{1}}}={{\rm{D}}_{\rm{1}}}{{\rm{C}}_{\rm{1}}}.\end{array} $

      Because $ A_1D_1 = B_1C_1 $ and $ A_1B_1 = D_1C_1 $, the perspec- tive aberrance in images taken at the OOP is least.

    • If $ t\leq0 $, it means that the observation position is at the back of the target; if $ t\notin[T_1,T_2] $, it means that the distance between the target and the observation position is improper. In both cases, the OOP does not exist and the mobile robot will move to the SOP. Its coordinates are obtained by the following method.

      For ease of description, there is a plane passing through the target center, which is also perpendicular to the target plane and parallel to $ L_3 $ and $ L_4 $. The plane is denoted as Plane A in Fig. 9. The horizontal plane that the camera′s optical center locates on is denoted as Plane B, and the intersection of Plane A and Plane B is denoted as $ L_r $, which is the red line in Fig. 9. Based on the camera′s perspective projection relationship, it can be obtained that the SOP is on $ L_r $. According to the current and desired area of target in images, the coordinate of SOP on $ L_r $ can be obtained.

      Figure 9.  Schematic of the method to determine the SOP. ${ O_c }$ denotes the origin of ${ F_c }$. ${ P_5 }$ denotes the center point of target. ${ P_5 }$ and ${ P_6 }$ correspond to the points in Fig. 3, respectively. ${ L_t }$ denotes the vector from ${ P_5 }$ to ${ P_6 }$. ${ L_r }$ denotes the red line. ${ B_1 }$ and ${ B_2 }$ are two points on ${ L_r }$.

      To reach the SOP, the whole process is divided into two steps: in the first step, the mobile robot moves to the line $ L_r $; in the second step, the robot calculates the next point on $ L_r $ based on the areas of the target in images. All the geometrical elements, such as points and planes, are described in $ F_r $ in this section, so their subscript or superscript is omitted for simplicity.

      In the first step, the mobile robot approaches to $ L_r $. The coordinate of desired position at each step on $ L_r $ can be obtained by the following method.

      The normal vector $ \overrightarrow {A\;}$ of Plane A can be obtained by the coordinates of the four vertices of the target as

      $ \begin{array}{l} \overrightarrow {A\;} = {P_2} - {P_3} = {P_1} - {P_4} \end{array} $

      (22)

      where $ P_{1-4} $ are the points of target as shown in Fig. 6.

      According to the normal vector $ \overrightarrow {A\;} (A_x,A_y,A_z) $ and the center point $ P_5 (x_5,y_5,z_5) $ of the target, the equation of Plane A can be determined as

      ${A_x}(x - {x_5}) + {A_y}(y - {y_5}) + {A_z}(z - {z_5}) = 0 . $

      (23)

      Plane B is parallel to the ground, and the y coordinate of the camera′s optical center in $ F_r $ is known, which is denoted as H. So Plane A can be written as

      $ \quad z = H.$

      (24)

      Based on (23) and (24), the equation of $ L_r $ can be obtained by

      ${A_x}(x - {x_5}) + {A_y}(y - {y_5}) + {A_z}(H - {z_5}) = 0 . $

      (25)

      The coordinate of mobile robot is denoted as (0, 0, 0) in $ F_r $, the point closest to the robot on $ L_r $ is denoted as $ P_{nearest} $. When $ A_x \neq 0 $ and $ A_y\neq 0 $, $ P_{nearest} $ can be obtained as follows:

      $ \left\{ \begin{aligned}& x = \frac{{A_x^2{x_5} + {A_x}{A_z}({z_5} - H) + {A_x}{A_y}{y_5}}}{{A_x^2 + A_y^2}}\\ &y = \frac{{{A_y}}}{{{A_x}}} \cdot \frac{{A_x^2{x_5} + {A_x}{A_z}({z_5} - H) + {A_x}{A_y}{y_5}}}{{A_x^2 + A_y^2}} \end{aligned} \right. $

      (26)

      when $ A_x = 0 $ and $ A_y\neq 0 $, $ P_{nearest} $ is

      $ \left\{ \begin{aligned}&x = 0\\ &y = \frac{{{A_z}}}{{{A_y}}}({z_5} - H) + {y_5} \end{aligned} \right. $

      (27)

      when $ A_x \neq0 $ and $ A_y = 0 $, $ P_{nearest} $ is

      $ \left\{ \begin{aligned}& x = \frac{{{A_z}}}{{{A_x}}}({z_5} - H) + {x_5}\\ &y = 0. \end{aligned} \right. $

      (28)

      According to $ P_{nearest} $, the robot approaches to $ L_r $. When the distance between the mobile robot and $ L_r $ is less than a threshold $ T_{distance} $, the mobile robot will perform the first and second step in order, otherwise, the mobile robot will only perform the first step.

      In the second step, according to the current and desired target area in the images, the mobile robot moves along this line. Denote $ l_1 $ as the unit direction vector from $ P_5 $ to $ P_6 $. Two points $ B_1 $ and $ B_2 $ are taken randomly on $ L_r $, and $ l_2 $ represents the unit direction vector from $ B_1 $ to $ B_2 $. The mobile robot can judge whether the direction of $ l_1 $ and $ l_2 $ is consistent by

      $ \cos \theta = \frac{{{l_1} \cdot {l_2}}}{{|{l_1}||{l_2}|}}.$

      (29)

      Let $ L $ represent the vector with the same direction to $ l_1 $. If $ \cos \theta \ge 0 $, the two vectors are in the same direction, $ L = l_2 $. If $ \cos \theta < 0 $, they are in the opposite direction, $ L = -l_2 $. The next point can be acquired iteratively by

      $P_{next} = P_{nearest}+ L \cdot K_s \cdot (S_d-S)$

      (30)

      where $ K_s $ represents a positive gains about areas, $ P_{next} $ represents the next planned coordinate of mobile robot. $ S_d $ and $ S $ represent the desired area and current area of target in images, respectively. By (30), a series of sequential points $ {{}^{r}}P_{next} $ can be obtained to guide the robot to the SOP.

    • In this section, two series of simulations are provided to verify our method. The first and the second simulations are about the strategy to determine the OOP and SOP, respectively. In the simulations the pan-tilt camera is mounted on a mobile robot and a virtual camera is applied, whose intrinsic matrix is determined by calibration of a real camera. The intrinsic matrix is

      $M = \left[ {\begin{array}{*{20}{c}} {1\,356.54}&0&{738.96}\\ 0&{1\,358.15}&{553.24}\\ 0&0&1 \end{array}} \right]. $

      In both simulations, a square target is used to locate the mobile robot. The distance H between the origin of $ F_a $ and the origin of $ F_b $ is 0.2 m. In order to test the robustness of the system, Gaussian white noise with 0.2 variance is added in the image. The initial angle between the forward direction of the mobile robot and the positive direction of the X-axis of the world coordinate system is $-\dfrac{\pi}{7} $. The initial states of the pan-tilt camera are: $ \alpha = 0 $ and $ \beta = 0 $.

      In the simulation to determine the OOP, the relative pose between the camera and target is like Fig. 10 (a) and the OOP exists. The image taken by the camera at the initial position is shown in Fig. 10 (a) and there is some perspective aberrance in the image. The image captured by the camera at the OOP is shown in Fig. 10 (b). In Fig. 10 (b), it is seen that the target is in the center of view and each side of the target is equal in the image. Hence, the perspective aberrance due to different perspectives is the least.

      Figure 10.  Simulation to determine the OOP: (a) The image taken by the camera at initial position. The small circle represents the center of the image and the point in the target represents the center point of the target. (b) The image taken by the camera at the OOP. (c) The black fold line is the real-time trajectory of mobile robot. The blue line is the ray emitted by the center point of the target, which is perpendicular to the target plane. The red pentagram is the OOP.

      The trajectory of the mobile robot is shown in Fig. 10(c). The initial position of the mobile robot locates at (0, 0, 0), and the OOPs are calculated in real time. Finally the mobile robot arrives at (5.517, –4.993) and stabilize, the optimal position is at (5.508, –5.000).

      During the movement of the mobile robot, the image errors between the center of images and the target are shown in Fig. 11(a). It can be seen that the errors converge to zero, which means the adaptive camera controller can keep the target in view during the motion of mobile robot. The position errors between the mobile robot and the OOP are shown in Fig. 11(b), it can be seen that the position errors become 0, and it indicates that the mobile robot can reach the optimal position and stabilize.

      Figure 11.  Curves of the simulation to determine the OOP: (a) The tracking errors in image plane. (b) The position errors between the robot and the OOP. (c) The length ratios of the opposite sides in images. Ratio 1 denotes the ratio of ${ \overline{P_1P_4} }$ to ${ \overline{P_2P_3} }$. Ratio 2 denotes the ratio of ${ \overline{P_1P_2} }$ to ${ \overline{P_3P_4} }$. (d) The angles in image plane corresponding to the target′s four corners. (e) The angular velocity of the camera and the mobile robot.

      In Fig. 11(c), the length ratios of the opposite sides of the square target in images are given and it is seen that Ratio 1 and Ratio 2 converge to 1 which means that the lengths of the four sides of the target in images tends to be the same while the robot is driven to the OOP. In Fig. 11(d) four angles of the target in images are given and it is known that they converge to $ \dfrac{\pi}{2} $ as the robot moves to the OOP, which means that the perspective aberrance due to different perspectives becomes less and less. Based on Figs. 11(c) and 11(d), it can be concluded that the mobile robot moves to the OOP where the images taken by camera have least perspective aberrance.

      In the second simulation, because the height of camera is fixed, the camera can not reach the OOP. So the mobile robot will move to the SOP. The image captured at the initial position is shown in Fig. 12(a) and the image captured at the SOP is shown in Fig. 12(b). From Figs. 12(a) and 12(b), it can be known that the image taken at the SOP has less perspective aberrance than the image taken at the initial position. The trajectory of the mobile robot is shown as Fig. 12(c), the initial position locates at (–2, –3), the SOP locates at (2.96, –4.994). In Fig. 12(c), it can be seen that the mobile robot arrives and stabilizes at (2.96, –4.994).

      Figure 12.  The second simulation to determine the SOP: (a) The image taken by the camera at initial position. (b) The image taken by the camera at the SOP. (c) The black fold line is the real-time trajectory of mobile robot. The blue line corresponds to $ L_r $. The red pentagram is the SOP.

      In the simulation, $ T_{distance} $ is set to 2.5 m. Because the distance between the mobile robot and $ L_r $ is greater than $ T_{distance} $ at the beginning, the robot starts to approach $ L_r $. After the distance is less than $ T_{distance} $, the robot starts to adjust its distance from the target by calculating $ P_{next} $. According to the sequential points $ P_{next} $, the mobile robot moves to the SOP.

      From Fig. 13 (a), it can be known that the $ e_u $ and $ e_v $ converge to zero quickly, which means the pan-tilt camera has a good performance.

      Figure 13.  Curves of the simulation to determine the SOP: (a) The tracking errors in image plane. (b) The ratio of the current area to the desired area in images. (c) The length ratios of the opposite sides in images. Ratio 1 denotes the ratio of ${ \overline{P_1P_4} }$ to ${ \overline{P_2P_3} }$. Ratio 2 denotes the ratio of ${ \overline{P_1P_2} }$ to ${ \overline{P_3P_4} }$. (d) The angles in the image plane correspond to the target′s four corners.

      The ratio of current area to desired area in images is shown as Fig. 13(b) and it is seen that the ratio of current area to desired area converges to 1. It means that the projection area of the target is close to the desired area. The length ratios of opposite sides of a square target in images are given in Fig. 13(c), and it is seen that Ratio 1 converges to 1.4 and Ratio 2 converges to 1, which means the perspective aberrance in one direction is much less. Based on Figs. 13(b) and 13(c), it can be concluded that the mobile robot reaches the SOP.

    • After simulation, a series of experiments have been conducted by a mobile robot equipped with a pan-tilt camera as Fig. 14(a) shows. The mobile robot is made by Haifeng intelligent technology, and the radius of the two wheels is 10 cm. The distance between the two wheels is 42 cm. The OK_AC2167 camera is produced by Beijing JoinHope Image Tecnology Ltd. The pan-tilt platform is equipped with two encoders that can read the rotation angles of the two servo motors at any time, respectively. The image resolution used is $ 768\times576 $, and the calibrated camera intrinsic M is the following:

      Figure 14.  Experiment to determine the OOP: (a) the mobile robot; (b) the image taken by the camera at initial position; (c) the image taken by the camera at the OOP.

      $ {{M}} = \left[ {\begin{array}{*{20}{c}} {642.96}&0&{408.55}\\ 0&{643.67}&{286.43}\\ 0&0&1 \end{array}} \right] $

      In our experiment, we use a square target with length 12.9 cm in Fig. 14(c) and its four vertices are detected with the aid of the OpenCV library (find ChessboardCorners function). Some parameters are as follows:

      $ {K_{{\omega _r}}} = 0.15,\;{K_{{V_r}}} = 0.2.$

      In the experiment to find the OOP, we set $ T_1 = 0.2\;{\rm m} $ and $ T_2 = 4.5\;{\rm m}$. When the target tilts up as shown in Fig. 14(b), the OOP exists. The image taken at the initial position is shown in Fig. 14(b) and the image taken at the OOP is shown in Fig. 14(c). The image errors are given in Fig. 15(a), and it can be seen that $ e_v $ has a little change, but $ e_u $ changes a lot, which is mainly caused by the fact that the images of target change little in the vertical direction but a lot in the horizontal direction as the mobile robot is moving. When the robot arrives at the OOP the tracking errors ($ e_u $, $ e_v $) are (4.919 pixs, –1.728 pixs). Fig. 15(d) shows the change of the four angles in images, it is seen that the angles converge towards $\dfrac{\pi}{2}$ and the perspective aberrance gradually decreases. The final four angles are 1.571 rad, 1.570 rad, 1.562 rad, 1.578 rad, respectively. In Fig. 15(c) the length ratios of the opposite side of the target in the images are given and it is seen that the ratios converge to 1. Based on the results, it can be known the image has a very small perspective aberrance.

      Figure 15.  Curves of the experiment to determine the OOP: (a) The tracking errors in image plane. (b) The angular velocity of the camera and the mobile robot. (c) The length ratios of the opposite sides in images. Ratio 1 denotes the ratio of ${ \overline{P_1P_4} }$ to ${ \overline{P_2P_3} }$. Ratio 2 denotes the ratio of ${ \overline{P_1P_2} }$ to ${ \overline{P_3P_4} }$. (d) The angles in image plane corresponding to the target′s four corners.

      Another experiment is conducted when the OOP does not exist. The image that is captured at initial position is shown in Fig. 16(a) and the image taken at the SOP is shown in Fig. 16(b).

      Figure 16.  Experiment to determine the SOP: (a) the image taken by the camera at initial position; (b) the image taken by camera at the SOP.

      Fig. 17(a) gives the position errors of the mobile robot. At the first stage when the distance between the mobile robot and $ L_r $ is larger than $ T_{distance} $ (we set $ T_{distance} $ to 0.2 m), $ e_x $ and $ e_y $ denote the errors between the mobile robot and $ P_{nearest} $. And $ P_{nearest} $ is obtained by (14) and (29)–(30). At the second stage when the the distance between the mobile robot and $ L_r $ is less than $ T_{distance} $, $ e_x $ and $ e_y $ denote the errors between the mobile robot and $ P_{next} $ calculated by (16). From Fig. 17(a), one can find that $ e_x $ immediately becomes larger after the position errors are less than $ T_{distance} $. Because when the distance between mobile robot and $ L_r $ is less than the set threshold, the robot will plan its next new position and the position errors will become larger immediately until the coordinates of planned points converge to (0 m, 0 m). In our experiment, the final position errors are (0.030 5 m, 0.000 474 m).

      Figure 17.  Curves of the experiment to determine the SOP: (a) The errors between the mobile robot and the planned points. (b) The tracking errors in the image plane. (c) The angular velocity of the camera and the mobile robot. (d)The length ratios of the opposite sides in images. Ratio 1 denotes the ratio of ${ \overline{P_1P_2} }$ to ${ \overline{P_3P_4} }$. Ratio 2 denotes the ratio of ${ \overline{P_2P_3} }$ to ${ \overline{P_1P_4} }$. (e)The ratio of current area to desired area in images.

      In Fig. 17(a), it can be seen that all angular velocity becomes 0, which means the mobile robot system can be stable after reaching the optimal position. In Fig. 17(d) the length ratios of the opposite side of the target in images are given, it is seen that Ratio 1 is close to 1 and Ratio 2 is less than 1. Because in this case, the lengths of L3 and L4 are similar but L2 is shorter than L1. That means the image perspective aberrance is decreasing, but it still exists. In the experiment, we set the desired area to 20 000 pix2. And in Fig. 17(e), one can find that the area of the target in current images is approaching the desired area and the target area in images taken at the SOP is 19 694 pix2.

      Finally, in order to validate the effectiveness of this method, another two experiments are conducted. The first experiment is about the relation between the perspective aberrance and the accuracy of relative pose estimation. The second experiment is about the change of re-projection errors during the motion of mobile robot.

      In the experiment, the accuracy of the relative pose between the camera and object is evaluated by the re-projection errors. The smaller the re-projection error is, the more accurate the relative pose is. In Fig. 18, images with different perspective aberrance are collected. The perspective aberrance is measured by (31), where $ l_1 $ and $ l_2 $ are opposite sides, and $ l_3 $ and $ l_4 $ are opposite sides. From Fig. 18, it can be known that as the perspective aberrance becomes larger, the re-projection errors become more scattered and larger. In images with larger perspective aberrance, many pixels are projected into a single pixel. It reduces the accuracy of feature point extraction, which affects the accuracy of positioning. When the perspective aberrance is serious, it even leads to the failure of feature point extraction.

      Figure 18.  Experimental data that reflects the relation between the re-projection errors and the perspective aberrance

      $ \begin{array}{l} F = 0.25 \left[\left(\dfrac{{{l_1}}}{{{l_2}}}\right) + \left(\dfrac{{{l_2}}}{{{l_1}}}\right) + \left(\dfrac{{{l_3}}}{{{l_4}}}\right) + \left(\dfrac{{{l_4}}}{{{l_3}}}\right)\right]. \end{array} $

      (31)

      In Fig. 19, the re-projection errors become less as the mobile robot moves to the optimal observation position. At the initial position the perspective aberrance is large (about 29.2), then the mobile robot moves to the optimal observation position. Although the re-projection error increases temporarily as the mobile robot moves, it moves to a position maintaining a small error (about 12.88) finally. It means that the relative pose can be estimated more accurately.

      Figure 19.  Change of re-projection errors when the mobile robot moves to the OOP

    • In this paper, we propose a new visual servo configuration eye-in-vehicle to solve the image problems that may occur in eye-in-hand and eye-to-hand configuration. We have implemented the mobile robot system in the proposed eye-in-vehicle configuration. The mobile robot will calculate the observation position in real time according to the observation effect. By the simulation and experiment, we verify effectiveness of the proposed method. In future work, the mobile robot will coordinate the movement with a manipulator. The mobile robot will move to the OOP or SOP, and the manipulator will ensure that it will not block the marker on target.

    • This work was supported by Natural Science Foundation of China (No. 61773374) and Key Research and Development Program of China (No. 2017YFB1300104).

Reference (23)

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return