Hand Gesture Gaming using Ultrasonic Sensors & Arduino

Hand Gesture Gaming using Ultrasonic Sensors & Arduino

Gopi Manoj Vuyyuru Tata Consultancy Services Bangalore, Karnataka

Dimple Talasila Department of ECE, GITAM Hyderabad, Telangana

Abstract- In the industrial 4.0 revolution, hand gestures are playing significant roles in the domains of IoT and robotics. Hand gestures are widely used in smart house, wearable devices, automobiles, virtual reality, etc in the domain of IoT. In robotics, these are used to control them and guide them. Now a days research scholars are working on the development of hand gesture computer that can be worked without the help of any hardware devices but with the help of hand gestures and sensors. There are few researchers who actually proved by showing that we can operate the video player, web browser and a text document with the help of hand gestures by using arduino and ultrasonic sensors. In this paper, we are going to show how we can control small games using hand gestures efficiently.

Keywords Hand gesture, IoT, Ultrasonic sensors Arduino-uno, Python, Gesture control

I INTRODUCTION

In today's world, vision and gesture are playing a crucial role in establishing communication between humans and computers just like physical devices (keyboard and mouse) do. There are various ways available for establishing communication between humans and computers. Out of those techniques, the hand gesture is very easy and fast to perform operations done by keyboard and mouse. Hand gestures can be implemented in various ways by using IR sensors, digital cameras, color bands and ultrasonic sensors. Each technique has its advantages as well as disadvantages. But out of all given techniques, using ultrasonic sensors is considered to be more comfortable as well as faster.

Initially, we take two ultrasonic sensors and fix them on the top of the laptop separated by a distance. Then we make circuit connections between the ultrasonic sensors and the arduino uno board which is attached on the top panel of the laptop. Now we use the ultrasonic sensor to measure the distance between the sensor and our hand placed in front of it. Based on the distance particular operation is performed. The primary objective of our paper is to play small games using hand gestures without the help of keyboard, mouse or any other physical device but using only arduino and ultrasonic sensors.

II LITERATURE REVIEW

In the year 2006, university of Moncton used infrared sensors to perform mouse operations using hand gestures. Here a hand glove is made by fitting IR sensors which are used to recognize hand gestures of disabled people and convert these into meaningful messages. These messages are used in real-time to perform mouse operations [3].

In the year 2011, the Institute of Space Technology of Islamabad published a computer-based on hand gestures with the help of web camera and projector. Here the projector is used to project the display on the wall or any other plain surface. Users can interact with this screen using his fingertips which are tracked using 'camshaft' tracker in the air. A robust method has implemented to detect and recognize these fingertips, which are used to perform actions [4].

In the year 2015, University of Malaya developed computer and presented in an IEEE conference using color bands. The first color and depth sensor called Kinect was developed by Microsoft for the Xbox console and released in November 2010. This sensor projects an infrared pattern of 307,200 dots in a 640 × 480 mesh and receives the reflected pattern through a CMOS monochrome sensor. This structured light application allows the device to measure the depth of every point using triangulation. Moreover, an RGB camera provides synchronized color information for each point. Microsoft Kinect was formerly developed for full-body tracking to interact with video games using body movements and gestures [5].

III HARDWARE COMPONENTS SETUP

1. Basic Hardware Setup

   ULTRASONIC SENSOR

   ULTRASONIC SENSOR

   To implement the proposed method we require an arduino uno board, a pair of ultrasonic sensors, a USB cable and finally jumper wires.

   ULTRASONIC SENSOR

   ARDUINO-UNO

   ARDUINO-UNO

   LAPTOP

   LAPTOP

   Figure 1: Basic connections of hardware components

   Initially, we attach the arduino uno board on the front face or panel of a laptop using double side tape. Then we use the USB cable and test the power supply from computer to board. After testing the power supply we then attach the

   sensors are attached on either side of the top of the screen. The sample layout is shown below.

2. Working of Ultrasonic Sensors

   The Ultrasonic sensor operates on a similar principle as that of the radar system. Here the electrical energy is used to generate waves and these waves are transmitted by a transmitter module into the air to detect an object. The echo waves that are reflected by hitting an object are received at receiver module and the distance is measured based on the following formula.

   Distance = (Speed of the waves × Time taken to travel)/2 Here we divide the total distance into half because the waves are travelled to and fro.

   Figure 2: Working of Ultrasonic sensor

   HC-SR04 is the most popular ultrasonic sensor and has 4 pins. The 4 pins are Vcc, Trigger, Echo and Ground pin. The layout of HS-SR04 is as shown below.

   Figure 3: Pin layout of Ultrasonic sensor

3. Overview of Arduino Uno Board

   The Arduino-Uno is an open source and programmable microcontroller board developed as a simplified version of arduino mega 328. The board has mainly 14 digital input/output pins,analog pins which can be programmable by using arduino IDE (Integrated Development Environment). It can be programmed by simple C/C++ programming by connecting with type B USB cable. It accepts voltage between 7 – 20 volts by connecting it with a power source. The clock speed of the board is 16MHz and SRAM capacity is 2KB. The clear details of configurations and technical specifications can be obtained from official arduino website as mentioned below.

   https://store.arduino.cc/usa/arduino-uno-rev3

   Figure 4: Arduino Uno Board

4. Circuit Connection

For left sensor we connect VCC to 5V power pin in Arduino-Uno board, then we connect ground pin to first GND pin. Then we connect trigger pin to pin 2 and finally echo pin to pin 3 in arduino board. For the right sensor, we connect VCC pin to 3.3V power pin, then we connect ground pin to second GND pin. Finally we connect trigger pin to pin 4and echo pin to pin 5 in arduino board.

Figure 5: Connection of Ultrasonic Sensors to Arduino Uno Board

IV INSTALLATION OF SOFTWARES & PACKAGES

1. Installation of Arduino IDE

   arduino IDE is open-source software developed in order to program circuit boards easily and efficiently. We can perform simple operations like turning on LED to complex operations like controlling robots. We can tell our board to do whatever operation we want by our arduino board using a set of programming instructions. So to perform these operations we use a software called arduino IDE. Using this software we can perform coding for various projects like IoT, wearable devices, 3D printing, circuit boards etc. So in order to install the software we can go to the below- mentioned link and select our operating system and download the software and install it. https://www.arduino.cc/en/Main/Software

2. Installation of Python IDE

   We have to install latest version of python i.e 3.83 (at the time paper was written). It can b installed from the below link based on your operating system: https://www.python.org/downloads/

   The main purpose of installing this software is to use python packages that will simplify the code and also helps in performing various operations effectively.

3. Installation of PIP Package Installer

   Pip is one of the best and widely used package manager used in Python for installing, deleting and managing packages. For some of the packages, pip comes preinstalled. So in order to check whether Pip is installed in your system or not we type "pip version" in python console. If we get any error then we have to install it from below documentation.

   https://pip.pypa.io/en/stable/

4. Installation of PyAutoGUI Library

   Pyautogui is a library in python which allows us to control keyboard as well as mouse to perform operations over a computer without the help of them. So to install it we use following command "pip install pyautogui".

5. Installation of PySerialI Library

PySerial library is used to provide backend support for python running on our computer. The major role of this library is it enables our program to communicate with the serial port. So to install it we use following command "pip install pyserial".

V OUTPUT & RESULTS

For many small games and even for big games the main keys are arrow keys or w, a, s, d keys. Here for proving this, we are taking a small car race game. Where the major operations are going forward, sideways and break. So in this, we have implemented those operations as listed below:

Table 1: Implementation of Arrow Keys

1. Implementation of UP Arrow

   To implement this operation initially we place our right hand in front of the right sensor at a distance less than 40 cm. Then the car starts moving forward with increasing speed. If we remove the hand then the speed automatically falls slowly to zero. The following picture depicts the increase in speed when we place our hand as specified.

   Figure 6: Increasing the speed of the car using hand gesture.

2. Implementation of Down Arrow

   To perform this operation initially we place our left hand in front of the left sensor at a distance less than 40 cm. Then the car starts slowing down to zero. The following picture depicts the decrease in speed when we place our hand as specified above.

   Figure 7: Decreasing the speed of the car using hand gesture.

3. Implementation Right Arrow

   To perform this task we place our right hand in front of the right sensor at a distance greater than 50 cm and less than 80 cm. Then the car starts turning right until we remove hand it keeps on turning towards right. The following picture depicts how car turn towards right when we place our hand as mentioned.

   Figure 8: Car turning right using hand gesture.

4. Implementation Left Arrow

   To perform this task we place our left hand in front of the left sensor at a distance greater than 50 cm and less than 80 cm. Then the car starts turning left until we remove hand it keeps on turning towards left. The following picture depicts how car turn left when we place our hand as mentioned.

   Figure 9: Car turning left using hand gesture.

   CONCLUSION

   In this paper, we have presented how we can play smaller games with hand gestures with the help of arduino and ultrasonic sensors by taking small call race as an example. Here we use sensors for measuring the distance between our hand and sensors. Based on that distance we can perform various operations by programming the arduino board with the help of arduino IDE. We have used python package pyautogui for controlling keyboard using hand gestures. In future, we implement the same approach for playing large games by using sensors of high quality and performance.

   REFERENCES

   1. G. Anil Kumar, B. Sathwik, Gesture Controlled Video Playback, IJCESR, Vol. 5, Issue. 4, 2018

   2. "Gesture recognition", En.wikipedia.org, 2020. [Online]. Available: https://en.wikipedia.org/wiki/Gesture_recognition. [Accessed: 15- May- 2020]

   3. Cost effective hand gesture computer control interface, GergelySziládi; Tibor Ujbányi; JózsefKatona 2016 7th IEEE International Conference on Cognitive Info communications.

   4. S. Thakur, R. Mehra and B. Prakash, "Vision based computer mouse control using hand gestures," 2015 International Conference on Soft Computing Techniques and Implementations (ICSCTI), Faridabad, 2015, pp. 85-89, doi: 10.1109/ICSCTI.2015.7489570.

   5. C. J. Cohen; G. Beach; G.Foulk A basic hand gesture control system for PC applications 30th Applied Imagery Pattern Recognition Workshop (AIPR 2001). Analysis and Understanding of Time Varying Imagery.

   6. Mollo, Luca & Bellotti, Francesco & Berta, Riccardo & De Gloria, Alessandro. (2016). Building Arduino-Based Tangible Serious Games for Elementary Mathematics and Physics. 60- 69. 10.1007/978-3-319-50182-6_6.

   7. Hoshino K. (2017) Hand Gesture Interface for Entertainment Games. In: Nakatsu R., Rauterberg M., Ciancarini P. (eds) Handbook of Digital Games and Entertainment Technologies.

      Springer, Singapore

   8. S. Belgamwar and S. Agrawal, "An Arduino Based Gesture Control System for Human-Computer Interface," 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA), Pune, India, 2018, pp. 1-3, doi: 10.1109/ICCUBEA.2018.8697673.

   9. Robust Hand Gesture Recognition for Robotic Hand Control(first published 15th June 2017 by Springer) by Ankit Chaudhary.