Real-Time Kinect Fingers Tracking and Recognition for Servo Gripper Controller

Rosdiyana Samad, Lim Siong Hee, Mahfuzah Mustafa, Dwi Pebrianti, Nor Rul Hasma Abdullah, Nurul Hazlina Noordin


The conventional method to control something like machines such as remote controllers or wearable sensors have its limitation and cannot cater the high demands in some scenarios. To overcome this situation, a real-time Kinect finger tracking and recognition is developed to control a servo gripper via a single board microcontroller, Arduino. This paper presents hand and finger tracking method that can determine the number of fingers and also the angle of the finger’s position. The hand gesture is captured by the Kinect and go through a series of image processing and finally, the information will send to the Arduino. The image processing method includes a detection of hand using depth sensor on the Kinect, then finds the finger and calculates the angle of the finger’s position. The convex hull algorithm is used to represent the region of the hand. The fingertips are recognized by calculating the angle of the fingertip and each angle is compared with threshold angle. The orientation and position of the fingers are obtained by finding the middle line of the finger and compares to the vertical line of the middle finger, then calculates the angle. The result shows that the number of fingers appeared in the display can be recognized and the orientation and position of each finger can be determined. A gripper also able to react simultaneously (open and close) based on the detected finger.


Finger Tracking; Gesture Recognition; Kinect; Real-Time; Servo Gripper Control;

Full Text:



C. A. Burande, R. M. Tugnayat, and N. K. Choudhary, “Advanced recognition techniques for human computer interaction,” in Proc. of The 2nd International Conference on Computer and Automation Engineering (ICCAE), 2010, pp. 480-483.

Bulling and K. Kunze, “Eyewear computers for human-computer interaction,” in Interactions, pp. 70-73, 2016. DOI=

M. M. F. M. Fareed, Q. I. Akram, S. B. A. Anees and A. H. Fakih, “Gesture based wireless single-armed robot in cartesian 3D space using Kinect,” in Proc. of The Fifth International Conference on Communication Systems and Network Technologies, 2015, pp.1210- 1215

C. Li, C.Yang, P. Liang, A. Cangelosi and J. Wan, “Development of Kinect based teleoperation of Nao robot,” in Proc. of International Conference on Advanced Robotics and Mechatronics (ICARM), 2016, pp.133-138.

R. Afthoni, A. Rizae and E. Susanto, “Proportional derivative control based robot arm system using Microsoft Kinect,” in Proc. of International Conference on Robotics, Biomimetics, Intelligent Computational Systems (ROBIONETICS), Yogyakarta, Indonesia, 2013, pp. 24-29.

M. M. Ali, H. Liu, N. Stoll, K. Thuro, “Intelligent arm manipulation system in Life Science Labs using H20 mobile robot and Kinect sensor,” in Proc. of IEEE 8th International Conference on Intelligent Systems,2016, pp.382-387.

M. V. Liarokapis, P. K. Artemiadis, and K. J. Kyriakopoulos, “Mapping human to robot motion with functional anthropomorphism for teleoperation and telemanipulation with robot arm hand systems,” in Proc. of IEEE International Conference on Intelligent Robots and Systems (IROS), 2013, pp. 2075-2075.

J. Shin and C. M. Kim, “Non-touch character input system based on hand tapping gestures using Kinect sensor,” in IEEE Access Open Access Journal, 2017. Doi: 10.1109/ACCESS.2017.2703783.

Z. Lai, Z. Yao, C. Wang, H. Liang, H. Chen and W. Xia, “Fingertips detection and hand gesture recognition based on Discrete Curve Evolution with a Kinect sensor,” in Proc. of The International Conference on Visual Communications and Image Processing (VCIP), 2016.

PrimeSense, “OpenNI NiTE2.2.0.11,” 2012. [Online]. Available:

W. Yan, H. Chuanyan, Y. Guanghui, and W. Changbo, “A robust method of detecting hand gestures using depth sensors,” in Proc. of IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE), 2012, pp. 72-77.

R. Hartanto, A. Susanto, and P. I. Santosa, “Real time hand gesture movements tracking and recognizing system,” in Electrical Power, Electronics, Communications, Controls and Informatics Seminar (EECCIS), 2014, pp. 137-141

Arduino, “Arduino Uno Rev3,”. 2015. [Online]. Available:


  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.

ISSN: 2180-1843

eISSN: 2289-8131