Control and programming of a robotic arm using gestures and skeleton tracking

The advent of the Microsoft Kinect sensor and the release of the NITE middleware by PrimeSense revolutionized the field of Computer Vision. The depth camera along with NITE offered various advanced features at a very low cost that were not once available to a civilian. Skeleton Tracking is one such feature that made tracking the joints of a human user, very easy and hence provided new possibilities in Human Computer Interaction. The Skeleton Tracking framework allows the computer to understand a person’s body position in 3D and to have a quite accurate idea of where the person’s joints stand in space at every point in time. Using this information it is possible to calculate the angle of these joints at every point in time. This paper explores the possibility of designing a robotic arm which will be controlled and can also be programmed using the skeleton tracking data obtained by using the NITE middleware along with the SimpleOpenNI library. The arm built will have 3 degrees of freedom with a gripper to pick objects. An Arduino board is used to control the actuators. Processing is used for programming and will act as an interface between the Arduino and the Kinect. Once the processing sketch is run, it will display the depth image obtained from the Kinect. Once the user is identified and his joints are tracked, various methods are used to calculate the angles of these joints tracked. These angles are then optimized and sent to the Arduino using serial communication. The elbow and the shoulder angles are directly calculated from the kinect data. As its not possible to calculate the shoulders swivel using a single kinect we use a separate gesture to control the swivel. These gestures are derived from the hand tracking feature available in SimpleOpenNI library. Moving a tracked hand in a circular motion will control the swivel action of the arm. Swiping the hand in a particular direction will close or open the gripper accordingly. All these data sent to the Arduino is stored in an Array List. In other words, from the start of the program all the movement the user imparts to the robotic arm is stored. On a mouse click, the skeleton tracking is stopped and theses stored data are continuously sent to the Arduino. This will ensure that the arm repetitively performs the motion that the user programmed it to do my moving his hand around in real 3D space. This method will provide a very easy and highly interactive way of programming a robotic arm. Instead of typing in a code or by moving the arm step by step and recording its position, the proposed method will be a lot easier and requires very minimal programming knowledge to use.