- Open Access
- Total Downloads : 0
- Authors : Prof. Keerthi Prasad G, Dixit R Jain, Krishna Chaitanya S Holla, Rashmi M G, Vanitha P
- Paper ID : IJERTCONV6IS15094
- Volume & Issue : ICRTT – 2018 (Volume 06 – Issue 15)
- Published (First Online): 05-01-2019
- ISSN (Online) : 2278-0181
- Publisher Name : IJERT
- License: This work is licensed under a Creative Commons Attribution 4.0 International License
Magic Mirror-An Intelligent Tool for Elderly People
Magic Mirror-An Intelligent Tool for Elderly People
Prof. Keerthi Prasad G Assistant Professor Dept. of ISE, GMIT
Dixit R Jain
Dept. of ISE, GMIT Davangere, Karnataka
Krishna Chaitanya S Holla
Dept. of ISE, GMIT Davangere, Karnataka
Rashmi M G
Dept. of ISE, GMIT Davangere, Karnataka
Dept. of ISE, GMIT Davangere, Karnataka
Abstract:- The world of Internet transformed our lives by connecting us more conveniently to information and other people in the virtual world. Mobile phones then became smart-phones and since then this concept has erupted and morphed into the Internet of Things, things which connect us to everyday objects. There is no end of objects that could be made Smarter, some being more suited to this than others. Mirrors, for example, provide a large surface which is ideal for displaying information and interacting with. Most people have mirrors at home so the concept of a smart mirror that we can interact with is attractive. The device was to go beyond an ordinary mirror. The device was to look like a regular mirror but would have a screen inside and you would be able to interact with it. The main features would be showing basic weather, location-based time information, reminders etc.
General Terms:- Magic Mirror, Raspberry Pi 3b, Amazon Voice Services, Artificial Intelligence, Cloud Computing, Face recognition, Amazon Lambda Servies, IOT Home Automation, OpenCV, Amazon SQS service
Keywords:- Smart Mirror, Magic Mirror, Home Automation, IoT, Alexa Voice Assistant, Alexa Skill, AWS IoT sdk, Amazon Lambda Services, Face recognition
This paper concentrates on design and development of a smart mirror that represents an elegant interface for glancing information for multiple people in a home environment powered with the interaction provided by Amazon Voice services. Face-recognition based authentication is used to detect the user. It provides a web-based interface to access data feeds and other services. It is interdisciplinary work comprising of domain knowledge of Artificial Intelligence, Image Processing, Internet of Things, Cloud Platform and home automation.
1.1 The Problem
In our rapidly developing world, information is always right at your fingertips – on your phone, on your
computer, maybe even on your watch. Staying connected with new information is both important for entertainment and daily life. With such a variety of options, there is difficulty in following all of your data streams. Often, during your day, you may end up in a position where it is inconvenient, or even impossible, to take out your phone or computer and check the newest update.
You cannot commit to a slower interaction. You need a display to glance at, with the information you need ready to go. However, aesthetics is just as important as displaying information. Keeping an extra computer in your bathroom or hall would be inconvenient and would not fit well with the look of a modern room. A sleek, simple display, easy for an average consumer, is a necessity in todays world. Also, people find it friendly and easy to use when interaction is involved.
Below we briefly comment on some related research in this direction. The Smart Mirror contains some devices equipped with a touch screen or TV enhanced externals devices. However, most of them support entertainment and some interactive tasks. The work has been reviewed as follows:
Mirror 2.0  combines the advantages of a smartphone and a mirror. It contains an LCD display positioned behind the glass. It provides news and weather information and it allows the playback of both videos and
Smart Washbasin  displays different information in a washbasin mirror such as mails, weather forecast, the water temperature and pressure, the calendar and the users
weight measured through a built-in-scale in the base portion. The device consists of an Android Tablet that displays the widgets on the basin mirror, made with a semi- reflecting glass put on top of an LCD display. It is possible to control it
without touching the screen surface, since it is equipped with proximity sensors able to track the hands position and motion.
– NEOD Framed Mirror TV  is a standard LCD
screen (up to 50 inches), covered by a mirror, and specifically designed for the screen. The screen provides some TV
functionalities, but it does not provide more interactive
– Smart Mirror for home environment  allows to control all the smart devices at home. It relays on face
recognition for authenticating the user and displays
personalized information (news, mail, messages etc.). The system exploits a touch screen monitor and two webcams, one
for the face recognition and one for the home surveillance.
Multi Display in Black Mirror  by Toshiba is a prototype that combines the functionalities of a tablet together
with the reflecting surface of a mirror. It provides two
configurations taking into account two different home environments: the bathroom and the kitchen. Considering the bathroom, the prototype provides useful information for the beginning of the day such as the weather forecast and fitness information coming from personal devices. In the kitchen, the setting includes a camera allowing the user to interact through gestures while preparing recipes and controlling the appliances.
The Reveal Project , created in the New York Times research and development, consists of an LCD Display covered by a mirror glass. The device exploits a Microsoft Kinect for tracking users movements in real-time. It
visualizes different information on its surface (calendar, mail, news, online shopping websites, instant messenger etc.). In addition, it responds to vocal commands. A peculiar feature is the medicine box scanner, which allows the user to buy
medicines recognizing their packages.
Cybertecture Mirror , is a complete PC contained into a 37 inches mirror, equipped with a 32 inches LCD screen. Through a smartphone application, the user accesses
different information overlaid on the reflected image. The interface allows to visualize instant messages, the calendar,
the mailbox, and the weather forecast. In addition, it provides
information on the users physical state. Indeed, the device
provides a set of external wireless sensors that allow to
measure the users weight, fat, muscle and bone mass.
– Interactive Mirror  by Panasonic seems to be an ordinary mirror: neither camera nor the other sensors suggest
the features of a smart object. Once the user sits down in front of it, the mirror displays an enlarged frame for her face, together with menus for accessing different functionalities. The system analyses the face hydration, wrinkles and other
details in order to recommend products and treatments to take care of her skin (e.g., to make it softer etc.), to slow ageing and so on. The mirror supports the user in buying such products. In addition, it provides make-up style previews,
simulating lighting and ambient conditions (e.g., at home, outdoor, shopping center, etc.).
Connected Store Demo  by eBay and Rebecca Minkoff provides interactive experiences in both the store showcase and in the fitting room. Once finished, the shopperprepares the fitting room with all the items. Inside the fitting room, the user exploits the mirror for looking for other items and/or providing feedback. In addition, she may select some of them for buying.
Brushing Teeth Mirror  displays the information collected by a smart brush about inflammations or infections of the teeth and gum.
Medical Mirror  combines computer vision and signal
processing technique for measuring the heart rate from the optical signal reflected of the face. The prototype consists of an LCD display with built-in camera and a two-way mirror fitted onto the frame. The smart mirror recognizes the presence of a user when she stands in front of it and, after about 15 seconds, it displays the heart rate below the users reflected image.
Our solution is an open platform for discrete display development. We offer an aesthetically pleasing mirror, with a hidden smart display underneath. With a generic display, the mirror can be built to any size so the information can be both in your face while showing you your face. Our product differs from the competition with an easy-to-use interface that is both simple for the average user and interaction provided by smart assistant Alexa from Amazon, which can be used to control different modules of the mirror as well as provide its basic services for user queries. A sleek display gives all levels of users a modern hub of technology for their personal daily interaction, one which both displays visually all the information you could need or want and operates with a simple interaction that you could fit into your daily routine.
By creating a platform open to modification, developers will also be able to add new functionality at their own pace. This will allow our display to be a tailorable and adaptable platform. A web application provides the interface that the user sees and interacts with. Also, an online configurator will relieve the frustration and difficulty of personalizing your information, as well as allow streamlined development of new modules. Powered by a small computer, the smart mirror will have great potential for expansion by developers. Along with these functionalities, this system can be used to control home appliances like lights, fans and Doors etc. by giving voice. Our product will be a step in the future of IoT, connecting your daily mirror to your tech-savvy world.
The system is designed to offer all the required services using various components. An Acrylic two mirror is the most important component used which provides a reflection layer on front surface and a transparent layer on the back surface. Beneath the mirror a standard LCD monitor is employed, which is the displaying component of our mirror. A System on Chip (SoC) microcontroller, Raspberry-Pi 3b is used that has computational capability and ability to control other devices with the help of a relay device. An USB microphone is connected to RPI board to receive input audio from the user. A speaker is utilized to to play audio feedback of Alexa assistant. Power module is used to supply AC power for the operation of smart mirror. On receiving voice command from the user, the system fetches the corresponding information from the cloud or executes any requested services like locking the door, turning on lights, showing CCTV footage of door and controlling fans etc. Systems interface is designed in
modular fashion and it comprises of many modules like date, time, weather and many more. User may also get multimedia playback facility like playing music and videos.
Figure 1 represents the block diagram of a Smart Mirror, which depicts the connection of all these components integrated into the system.
Figure 1. Block-Diagram of Raspberry Pi and Alexa enabled smart mirror.
3.1 Data Flow Diagram
Data flow diagram depicts the overall functioning of the smart mirror from the stage of being turned on till its shut down. The external elements are the devices connected to RPI 3B as a part of home automation. Initially when the smart mirror is powered on, it displays the default modules. Now the face recognition module gets activated and waits for the
Figure 2. Data flow in the Smart Mirror
user to stand in front it. Once the user comes in front of mirror, it detects the authorized user and loads his personalized modules. The voice recognition system is now ready for user queries or commands.
Upon receiving commands, the audio to text conversion takes place and necessary features from the converted text is extracted. Once the intent has been extracted, corresponding services will be executed by fetching data
from the cloud APIs. The response is produced in terms of text, which is then again transformed to audio and finally the results will be displayed and spoken out. After a cycle of execution, it again rolls back to face recognition stage.
3.1 Face Recognition Algorithm
OpenCV is a popular computer vision library started by Intel in 1999. The cross-platform library sets its focus on real-time image processing and includes patent-free implementations of the latest computer vision algorithms. In 2008 Willow Garage took over support and OpenCV
now comes with a programming interface to C, C++, Python and Android. OpenCV is released under a BSD license so it is used in academic projects and commercial products alike.
OpenCV 2.4 now comes with the very new Face- Recognizer class for face recognition, so you can start experimenting with face recognition right away. The currently available algorithms are:
Eigenfaces (see createEigenFaceRecognizer())
Fisherfaces (see createFisherFaceRecognizer())
LBHS (see createLBPHFaceRecognizer()) 
The Smart Mirror uses Local Binary Patterns Histograms algorithm. A more formal description of the LBP operator can be given as:
, with (xc, yc) as central pixel with intensity; and being the intensity of the neighboring pixel. S is the sign function defined as:
This description enables you to capture very fine- grained details in images. In fact, the authors were able to compete with state of the art results for texture classification shown in Figure
3. Soon after the operator was published it was noted, that a fixed neighborhood fails to encode details differing in scale. So, the operator was extended to use a variable neighborhood. The idea is to align an arbitrary number of neighbors on a circle with a variable radius, which enables to capture the following neighborhoods:
Figure 3. Texture Classification
For a given Point (xc, yc), the position of the neighbor (xp, yp), p P can be calculated by:
Where, R is the radius of the circle and P is the number of sample points.
The operator is an extension to the original LBP codes, so its sometimes called Extended LBP (also referred to as Circular LBP).
If a points coordinate on the circle
doesnt correspond to image coordinates, the point gets interpolated. Computer science has a bunch of clever interpolation schemes, the OpenCV implementation does a bilinear interpolation:
By definition the LBP operator is robust against monotonic gray scale transformations. We can easily verify this by looking at the LBP image of an artificially modified image.
Figure 4. Test Faces
Alexa Voice Command Processing
Figure 5. Alexa Commands processing flow diagram
Following is an instance which shows how Alexa command works:
Initially the user says, Alexa, Turn on the lamp.
Raspberry-Pi uses Alexa AVS package to capture the audio.
Alexa AVS sends the audio to Alexa Skills to extract
Alexa skills then sends the intent to AWS Lambda function.
Lambda function passes the extracted intent to AWS
IoT dk hosted in it.
AWS IoT sdk sends this intent to AWS IoT sdk integrated in Magic Mirror package.
Mirror on receiving intent, gives command to
Raspberry Pi to execute scripts or to change modules.
The scripts control different pins of the Relay device.
On receiving signal, the device connected to particular pin in relay turns on (lamp).
4. COMPONENTS OVERVIEW
The following are the hardware components employed to implement this work:
Two-way mirror: A special mirror known as a two- way mirror or observation mirror is used in this project.
mirror is special as compared to an ordinary household mirror. Unlike a household mirror, the two- way mirror is not painted with an opaque color on the back, instead its left untouched. This gives the property of the mirror being
reflective one side and transparent/translucent from the other. Hence the two-way mirror acts as mirror as long as there is no light send from the back of mirror.
Raspberry-Pi 3B: Raspberry Pi is a credit-card sized computer manufactured and designed in the United Kingdom by the Raspberry Pi foundation with the intention of teaching basic computer science to school students and every other person interested in computer hardware, programming and DIY-Do-it Yourself projects.
Webcam: A simple USB powered webcam is used to
recognize peoples faces.
LCD panel: An LCD panel placed behind the mirror is used to present to the user the desired interface.
USB Microphone: One mode of interaction with the smart mirror is through microphones. Two microphones are used to power the voice recognition capabilities of the device. USB microphones has to be used because the
Raspberry Pi does not have regular microphone input.
Speakers: Speakers are used to support audio feedback from mirror and for multimedia functionality.
Relay Device: Relay device is the hardware component which acts as switch to control devices.
The following are the software packages employed to implement our work:
OpenCV: OpenCV (Open Source Computer Vision
Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a
common infrastructure for computer vision applications
and to accelerate the use of machine perception in the commercial products. Being a BSD-licensed product,
OpenCV makes it easy for businesses to utilize and modify the code. The library has more than 2500 optimized algorithms, which includes a comprehensive set of both classic and state-of-the-art computer vision and machine
learning algorithms. These algorithms can be used to detect and recognize faces, identify objects, classify human actions in videos, track camera movements, track moving objects etc. OpenCV has more than 47 thousand people of user community and estimated number of downloads exceeding 7 million. The library is used extensively in companies, research groups and by governmental bodies. It has C++, C, and Python, Java and MATLAB interfaces, and supports Windows, Linux, Android and Mac OS. OpenCV leans mostly towards real-time vision applications and takes advantage of MMX and SSE instructions when available. OpenCV is written natively in C++ and has a templated interface that works seamlessly with STL containers . This project will utilize the facial recognition algorithm from the OpenCV library to recognize users.
Raspbian OS: Raspbian is a free operating system based on Debian optimized for the Raspberry Pi hardware . Raspbian comes with over 35,000 packages, pre- compiled software bundled in a nice format for easy installation on Raspberry Pi computer.
NodeJS: NodeJS is a java-script engine for server- side applications. It comes included with Electron and we used it to launch processes to control things that are not available in web APIs such as the sensors and microphones
for voice recognition. We also used it to access the filesystem and read the app files.
Python: Python is a high-level, general purpose,
interpreted programming language. It is very popular in the Raspberry Pi community and it has lots of support and libraries. We used it with the microphone to detect sounds and also for control of sensors.
Electron: Electron allows for the use of standard web technologies to implement a front end. This allowed
electron-boilerplate. This package provided a faster start to application development and allows for more integrated testing as well as the potential to build release executables
for multiple systems.
The Smart Mirror thus accomplishes all set objectives, by still being a mirror with all the technology inside it, making it very approachable to use and integrating seamlessly into our lives. The Smart Mirror has scope in the field of IoT and home automation. The Smart Mirror can be connected to the home appliances, mobile devices, etc. which can expand the functionality of the mirror. The facial recognition technology used can be future enhanced as a means of security. Adding security means that no one can try to access sensitive data that maybe displayed on your mirror via the use of APIs. The artificial intelligence embedded using Amazon Voice Services in the form of Alexa enables the user to interact with and control other modules of the mirror. We believe that the future of the home will be a brilliantly connected ecosystem of smart technology designed to make your life easier, more enjoyable, and efficient. Obviously, there are a ton of opportunities in the home for technology integration but a mirror is one of the best places to start.
23. NEOD: NEOD Framed Mirror TV. http://www.neod.org/  M. A. Hossain, P. K. Atrey, A. El. Saddik, Smart Mirror for ambient home environment, 2007.  Toshiba: Toshiba to Unveil Leading edge Technologies at CES 2014.
http://www.toshiba.co.jp/about/press/2014_01/pr0702.html. B. House, A. Lloyd, M. Zimbalist, Reveal Project, http://brianhouse.net/works/reveal/  J. Law, Cybertecture Mirror, http://www.jameslawcybertecture.com/index.php?section
=Company Panasonic: The Future Mirror. http://youtube/- 2kc9GQYIE  Ebay: Rebecca Minkoff Connected Store Demo.  T. Mullins, Briushing Teeth Mirror, http://wwwdesignboom.com/contest/view.php?contest_pk=36
&item_pk=44258&p=1. M. Z. Poh, D. McDuff, R. Picard, A medical mirror for non- contact health monitring, In ACM SIGGRAPH 2011 Emerging Technologies SIGGRAPH 11, New York, NY, USA, ACM (2011) 2:1-2:1.  Ahonen, Timo, Abdenour Hadid, and Matti Pietikainen. Face description with local binary patterns: Application to face recognition. IEEE transactions on pattern analysis and machine intelligence 28.12 (2006): 20372041  ABOUT OpenCV, Accessed May 04, 2016. http://opencv.org/about.html.  "Home" FrontPage. Accessed May 07, 2016. https://www.raspbian.org