Intelligent Interactive Robot with Gesture Recognition and Voice Feedback

Intelligent Interactive Robot with Gesture Recognition and Voice Feedback

Prof. S. S. Turakne

Department of Electronics and Telecommunication Engineering, PREC, Loni

Savitribai Phule Pune University, Maharashtra, India

Abstract – Now-a-days with the help of wireless robot we can do different work. This paper is based on wireless robot using gesture recognition and voice feedback. Robot is designed with the help of flex sensor for the movement of robotic arm. Also voice command is given with the help of microphone for the movement of robot. In industry such a robot can work without any external assistance. Wireless robot system with voice and hand gesture recognition commands is useful in the hazardous industry or area where human being cannot work. Here we are using the voice recognition with pattern recognition in MATLAB software.

KeywordsWireless robot, Flex sensor, RF Module, PC

1. INTRODUCTION

   This robot is a prototype but its applications are vast [2]. There are many types of robot used in industry and other fields are designed to increase [5] efficiency and accuracy of work. Wireless robot can be designed to work for the maximum range.

   By using flex sensor robotic arm can grip the object or move arm in up/down directions. With help of Flex sensors it is very easy to measure the finger bending, also hand position and orientation for gesture recognition [6].

   Voice command is the easier way of controlling any type of robot. In any hazardous industry or area where the man is unable to do the work, in this condition by giving the voice command to robot, commands like Start/stop, forward/reverse and left/right. These commands are processes in MATLAB software and then send to microcontroller with the help of RS232 serial communication bus.

2. LITRATURE SURVEY

   In this paper of greedy extension of localized auction based protocols for wireless robot-robot coordination we presented improvements of previously proposed auction aggregation protocols to improve the sub optimality of localized decisions in robot-robot communication for the task assignment 2009[9].Then the paper Virtual WRSN Modeling and Simulation of Wireless Robot-Sensor Network Systems concerns with wireless robot . The user oriented software like Simulink/Matlab are used to design the human operational system. Main application of the software simulator and background theory used for the development of the system are indicated in the paper 2010[10]. The paper on Wireless Sensor Networks robot localization method describes robot localization. In the wireless sensor networks, robot works as the navigation prop and it is localized by using the wireless sensor networks range-based location method and range-free location method 2011[11]. The paper

   wireless gesture control robot represents many techniques have been determined and referred with their advantages and drawbacks under various operational and functional methods. so, it can be verified that features like user affable interface, light weight and portability of smart phones with OS android has over taken the greatness of technology like programmable glove, fixed cameras etc. making them obsolete 2012[3].

   Further the paper hand gesture control robot system describes a hand gesture based Control interface for navigating a robot. Two axis accelerometer is utilized to record a users hand movements. Then data from accelerometer is transmitted using wireless RF module. The received data from RF module are then classified to any one command from four commands used for navigating a robot and two commands for claw/robotic arm 2013[2]. The wireless robot is implemented by using Android Based Wireless Gesture Controlled system. There is great demand for such type of robotic systems 2014[8]. This research paper finger pattern recognition and gesture based Security System of device is an introduction of a hand gesture recognition based system to identify real time gestures in natural environment and match up to patterns with image database to trigger unlocking of mobile devices 2015[4].

3. HARWARE DISCRIPTION

   Flex sensors are connected to the analog channels of ARM processor LPC 2138. Voice signal is given with the help of microphone. PC consists of MATLAB software for generation of code.ARM microcontroller communicates with PC using MAX 232 bus. Generated signal transmitted from human interaction circuit to the robot circuit via RF module. The robot is operated by two DC motors. The voice signal given to microcontroller operates these DC motors and controls the robot accordingly. Griper motor operates as robotic arm. The movement of this griper motor is controlled by hand gesture which is used to grip the object for completion of task.

   Fig.1. Block diagram of human interaction circuit

   1. Microcontroller

      ARM microcontroller LPC2138 is heart of the system. In this system there are two microcontrollers, one at transmitter section and other at receiver section. The function of transmitter section microcontroller is to read gesture command from microphone and flex sensor, make the decision and send it to receiver section.

      Fig.2. Block diagram of robot circuit

      The microcontroller at receiver section receives command and acts accordingly. The LPC2138 microcontroller is 16/32- bit ARM7TDMI-S CPU with embedded traces support and emulation facility. It has 512 KB flash memory, two 10 bit ADC, two UARTs from which we are using UART0.

   2. RF Module

      RF stands for Radio Frequency. This module consists of Transmitter (TX) and Receiver (RX) pair. The transmitter and receiver has the own frequency ranges. The transmitter transmit the signal with known frequency range is calibrated first then the receiver side should ranged in same way as the transmitter side is calibrated. An Encoder (HT12E) circuit with transmitter and a Decoder (HT12D) circuit with receiver is used in order to transmit and receive signal.

      The whole commands such as start, stop, forward, reverse, left and right using voice command and gripper movement using flex sensor is transmitted and received with the help of RF module. The RF module used in this paper works on the frequency of 315MHz with an operating range of 400-500 meters.

   3. Flex sensor

   Fig.3. Flex sensor bending movement

   Flex sensor are sensors whose change in resistance depends on the bend on the sensor. They convert the change in bending to change in resistance more the bend, more the resistance value. Flex sensors are analog resistor. They work as variable analog voltage dividers. The flex sensor consists of carbon resistive elements within a thin flexible element. More carbon value means minimum resistance. When the particular sensor is bent the output of that sensor produces resistive output relative to the bend radius.

   1. DC motor driver

      The L293D motor driver is providing User with easy interfacing with ARM (LPC2138). L293D motor driver is mounted on a good quality, single sided non-PTH PCB. The motor O/P pins are connected to connecters through freewheeling diode to minimize the voltage spikes which can damage the components of the circuit. The L293D is a Dual Full Bridge driver which drives up to 1Amp per bridge with supply voltage range 9V to 24V. It can drive two DC motors, relays, etc. The code vice is TTL supported. Two H bridges of driver IC connected in shunt which can increase its current capacity from 1Amp to 2 Amp.

      The driver IC L293D is quad push-pull drivers is able to deliver output currents to 1A per channel respectively. Each channel is controlled by a TL-compatible logic I/P and two drivers are equipped with an enable input available at pin 1 and 9. The motor will be start running only when chip enable is at high logic i.e. chip inhibit is enabled.

   2. DC motor

      DC motor is one of the rotating device which is used to convert the electrical energy into mechanical rotary motion. This motor is used for the rotary motion of the robot and the robotic arm for appropriate actions and the routing of the robot. Right movement is performed when the voice command right is given with the help of RF module. All the commands are given in same manner for the navigation.

   3. Gripper

      Grippers allow the robot to interact with its surroundings. Gripper motors are used to grip the object. Hence in robotic system Gripper motor is used to drive the gripper on the optional arm assembly.

      The gripper is prepared by using two dc motors one for the up, down movements of gripper and one for the open and closing of the gripper jaw which is usable for pick the object.

   4. MAX 232

      MAX 232 is use for the serial interface between PC and microcontroller. For the logic 1 voltage range is from -3V to – 25V and for logic 0 it ranges between +3V to +25V. It gives appropriate voltage 0V or 5V to the microcontroller.

   5. Liquid crystalline display

   The LCD requires 3 control lines and either 4 or 8 I/O lines for the data bus.

   Fig.4. 2*16 LCD display

   In this paper the LCD is to operate with a 4-bit data bus. If a 4-bit data bus is used the LCD will require a total of 7 data lines (3 control lines and the 4 lines for the data bus). If an 8-bit data bus is used the LCD will require a total of 11data lines (3 control lines and the 8 lines for the data bus).

4. Speech Recognition

   Fig.5. speech features extraction

   1. Speech extraction using matlab

      The main aim of pattern recognition is to classify data based on either prior knowledge or on statistical data extracted from the patterns. The patterns to be classified are usually for the use of measurements/observations, defining points in an appropriate multidimensional space. This is contrast to pattern matching, where the pattern is specified. A whole pattern recognition system consists of a method that collects the information that is to be classified or described a feature extraction mechanism. The same mechanism is used to calculate numeric/symbolic information from the observations, and a classification/description scheme that does the actual job of classifying or observations, relying on the extracted features. The classification and description scheme generally based on the accessibility of a set of patterns that have already been classified. The model which is fixed for the particular person by measuring the frequency and accent is supervised to give the proper command. The commands should be uniquely supervised which cannot be unsupervised that means priority is not given for the specific pattern.

   2. Command procedure

   Open MATLAB 12 software. Add new sounds from headphone to the database. By clicking on my test check recorded sounds. Switch on the power supply. Check for the LEDs are glowing(if LEDs are glowing there is a power supply). Commands such as start, stop, left, right, forward, reverse are given from microphone randomly. Find match for the given command (sound) from previously added database. (if match found then robot will move accordingly).

   Fig.6. generated speech signal using microphone on matlab

   Fig.7. voice command added to database

   The voice commands which are saved in the data base are stored in both the ARMs i.e. at human interface as well as at robot circuit. The commands are stored in the form of strings like for forward- #F, reverse-#B, right-#R, left-#L, stop-#S, etc. The receiver circuit accept these strings which are send from the transmitter side. The voice command procedure is done in above manner to accomplish the robotic navigation movements using DC motor driver and DC motor.

5. HAND GESTURE COMMAND FLOW

   START

   START

6. RESULTS

   TABLE I

   SR. NO.	VOICE COMMAND	ROBOT MOVEMENTS
   1	Start	Robot initializes
   2	Forward	Robot moves forward
   3	Backward	Robot moves backward
   4	Left	Robot moves to left
   5	Right	Robot moves to right
   6	Stop	Robot stops

   SR. NO.	VOICE COMMAND	ROBOT MOVEMENTS
   1	Start	Robot initializes
   2	Forward	Robot moves forward
   3	Backward	Robot moves backward
   4	Left	Robot moves to left
   5	Right	Robot moves to right
   6	Stop	Robot stops

   RESULTS FOR VOICE FEEDBACK

   Initialize LCD

   Initialize LCD

   If serial NO

   interrupt

   detected

   YES

   Receive serial interrupt from pc as start, stop, left, right, forward, reverse

   Receive serial interrupt from pc as start, stop, left, right, forward, reverse

   Select analog channel 3 & read ADC

   Select analog channel 3 & read ADC

   FOR NO

   Robotic Arm Down

   Robotic Arm Down

   FLEX1 If R<800

   YES

   Robotic Arm Up

   TABLE 2

   RESULTS FOR FLEX SENSOR BENDING

   FLEX SENSOR (1/2)	FLEX SENSOR POSITION	GRIPPER MOVEMENTS OF ROBOT
   1	Bent	Gripper moves downward
   1	Nominal	Grippe moves upward
   2	Bent	Gripper jaw get closed
   2	Nominal	Gripper jaw get opened

   Store and display FLEX1 reading on LCD

   Store and display FLEX1 reading on LCD

   Select analog channel 4 & read ADC

   Select analog channel 4 & read ADC

   FOR NO

   FLEX2 If R<800

   Gripper Close

   Gripper Close

   YES

   Gripper Open

   Store and display FLEX2 reading on LCD

   Store and display FLEX2 reading on LCD

   END

   END

   Fig.8. Human interface model snap

   Fig.8. Robot prototype model snap

7. CONCLUSION

The possibility of working in hazardous area is accomplished with the help of wireless system. The hand gesture and voice control robot system gives an alternative way of controlling robot. We have provided two techniques for giving inputs to the system that are hand gesture and voice recognition. Therefore the conclusion is that features like user friendly interface and accuracy makes the robot absolute. Human efforts are minimized effectively and have been done without causing any harm.

REFERANCES

1. Love Aggarwal, Varnika Gaur, Puneet Verma Design and implementation of a wireless gesture controlled robotic arm with vision in International Journal of Computer Applications, Vol. 79- no. 13,

   October 2013

2. Riyaz Mansuri, Sandesh Vakale, Ashish Shine, Tanveer Patel Hand gesture control robot vehical in Inetrnational Journal of Electronics and Communication Technology, Vol. 4, Issue Spl – 2, Jan-March 2013

3. Monika Jain, Aditi, Ashwini Lohiya, Mohammad Fahad Khan, Abhishek Maurya Wireless gesture control robot: an analysis in International Journal of Advanced Research in Computer and Comm. Engineering, Vol. 1, Issue 10, December 2012

4. Shivam Khare Fingure gesture and pattern recognition based device security system

5. Haslina Jaafar, Azura Che Soh, Wan Zuha Wan Hasan, Mohammad Hamiruce Marhaban, Shaufiq Abd Rashid A wireless flipper robot using interface free contoller IEEE 8th Conference on Industrial Electronics and Apllications, 2013

6. Harish Kumar Kaura, Vipul Honrao, Sayali Patil, Pravish Shetty Gesture controlled robot using image processing in International Journal of Advanced Research in Artificial Intelligence, Vol. 2, no. 5 , 2013

7. G. Pacnik, K. Benkic, B. Brecko, Voice operated intelligent wheelchair

   – VOICE IEEE ISIE 2005

8. Altaf Ganihar, Shreyas Joshi, Rahul G, Rohini Hongal, Uma Mudenagudi, Android based wireless gesture controlled robot in International Conference on Advanced in Electronics, Computers and Communications, 2014

9. I. Mezei, V. Malbasa, I. Stojmenovic, Greedy extention of localized auction based protocols for wireless robot- robot coordination 2009 IEEE

10. Aleksandar Rodic, Gyula Mester Virtual WRSN- modeling and simulation of wireless robot-sensor networked systems IEEE 8th International Symposium on Intelligent Systems and Informatics 2010

11. Zhang Guo-wei, Wang Jia-yu, Lu Qiu-hong Localization of robots based on wireless sensor networks 2011 IEEE