Read My Mood – Expressive Emotion Detection AI

Read My Mood – Expressive Emotion Detection AI

Neha

Dept. of Computer Applications, St. Joseph Engineering College, Mangaluru, India

Murari B K,

Dept. of Computer Applications, St. Joseph Engineering College, Mangaluru, India

Abstract – Emotion-aware systems are becoming increasingly important in the development of human-centered AI applications. This paper reports on a working model called "Read My Mood", an expressive emotion detection chatbot with a facial recognition/sentiment analysis technology to identify and respond to the emotional states of users. The underlying technology uses deep learning to classify a users facial expression into one of seven basic emotions (happiness, sadness, anger, fear, surprise, disgust, and neutral). Coupled with an intelligent chatbot, the application can adapt its reply to meet the empathy and contextual awareness dimensions. The study describes the system architecture, methods and application of an emotionally intelligent system in three areas of application (i.e. mental health, customer service and education). The paper closes with a discussion on future directions such as multimodal emotion identification and multilingual responses.

Index Term – Emotion detection, facial expression recognition, chatbot, deep learning, affective computing

1. INTRODUCTION

   Artificial Intelligence (AI) has progressed considerably over the last many yearsfrom rule-based approaches, to models that utilize machine learning to imitate human thought processes and behaviors. However, one of the key human interaction componentsemotionremains relatively underrepresented in most AI systems. Emotions, whether positive or negative, are central to human communication, decision-making, and relationship building. An intelligent system must understand not just the content of a given message, but also the emotional context of the message. In this paper, we initiate "Read My Mood," an expressive and adaptive emotion detection system, to address this emotional void in human- computer interaction. By leveraging facial expression analysis along with traditional natural language processing (NLP) approaches, the system will detect a user's emotional state and adjust its conversational tone/characteristics accordingly. By integrating a chatbot into the application, the system-enabled interaction will be more socially empathetic and human-like or human-typed, negating the need for the messenger receiver components of human messages from a human-to-human

   interaction. The proposed system addresses a fundamental gap that exists in traditional chatbot technologies: the absence of an understanding and responding to emotional signals. Most existing chatbots follow a script or only match keywords, which results in unfeeling, robotic interactions. Conversely, our emotion-aware chatbot detects real-time facial signals of emotion, pulls relevant sentiment, and alters its response strategy to personalize the experience. The relevance of emotional intelligence in AI extends into a myriad of fields across a spectrum of human authority, from mental health management to education, customer support, or entertainment. A chatbot that can recognize and respond to distress, joy, or confusion can improve emotional reassurance, engaging learning strategies, or manage frustrated customers, and thus improve the quality and engagement of their experience. The paper is structured to first detail the design system, architecture and implementation of Read My Mood the potential applications of the work and future work. The research is focused on adding to the field of affective computing and emotionally intelligent AI systems.

2. LITERATURE REVIEW

   The research of emotion recognition in machines is part of the discipline of Affective Computing, which Rosalind Picard coined in the 1990s, with the goals of addressing the emotional divide between humans and computers. Over the years, emotion recognition has progressed by combining computer vision, natural language processing and machine learning algorithms. The literature indicates that these advancements represent significant progress in both unimodal and multimodal emotion detection systems.

   1. Facial Expression Recognition (FER) Facial expressions are one of the more informative and universally recognized signals of human emotion. Ekman and Friesen [1] proposed six basic emotions, happiness, sadness, fear, anger, surprise, and disgust, which often serve as the framework for most FER systems. There have been recent studies utilizing convolutional neural networks (CNNs) and deep learning architectures to do real-time, accurate classification of facial emotions. For

      example, Mollahosseini et al. [2] indicate using deep neural networks trained on datasets such as FER2013 and CK+ offer some level of success in FER. Open-source frameworks such as OpenCV and libraries such as MediaPipe, and Dlib have made it easier to detect facial landmarks and pre-process facial images for the models. There is considerable utilization of these tools in academic FER systems and commercial FER systems.

   2. Emotion Detection in Chatbots Chatbots have assumed a model based on keyword matching with intent classification. Emotionally aware chats aim to recognize users' emotional interaction via text analysis through some underlying sentiment analysis or multimodal capabilities (facial aspects or tonal aspects). Poria et al. [3] identified the lack of emotion recognition in dialogue systems, and emphasized consideration of emotion when creating something that is user-specific and situational. The use of Natural Language Processing libraries such as Transformers, BERT and Rasa allow for greater understanding of user sentiment through textual data, along with additional capability for recognizing emotional cues through integration of speech-to-text (STT) and text-to-speech (TTS) to enable emotion recognition through voice interaction. This research is currently being explored for its potential for accessibility and usability.

   3. Multimodal Emotion Recognition Multimodal approaches are strategies that leverage visual data, auditory data, and textual data to provide more extensive emotion detection. Zeng et al. [4] indicate that combining facial expressions and voice tone with language cues greatly improves emotion classification performance. This paper primarily deals with the visual aspect of emotion detection, but planned future work will enhance this study with voice-based emotion analysis to engage in comprehensive emotion-aware interactions.

   4. Applications and Trends Emotion detection systems are being implemented in more and more applications, such as mental health assessment, e-learning, customer relationship bank assessments, and interactive gaming. Companies such as Affectiva and Emotient (purchased by Apple) are demarcating FER technologies on the commercial side, demonstrating an increasingly healthy market and research interest.

3. PROBLEM STATEMENT

   While recent advances in conversational AI have improved the ability to understand user intent and produce humanlike responses, the next evolution will be in the technology's ability to understand and respond to a user's emotional state.

   Currently, traditional chatbots are often still chained to rigid decision trees or keyword scripting, with slight improvements to generating responses emotionally, leading to emotionally flat, depersonalized interactions. This limit on the capabilities for which the chatbot may produce a response leads to conversations that personal approach leads conversations that are impersonal, conversational imperonalize the user and simply disengage from them.

   The inability to respond empathically to an emotional state that could be sadness, anger, or frustration, has a negative impact on user satisfaction and retention for users because of the serious limitations that provide disengaged persona as it can be a particularly sensitive area to the eventual user. Particularly in domains like mental health support, customer service, and educational settings where trust and emotional connection is important for a user to have satisfaction.

   • Having no emotional intelligence in chatbot systems has the following drawbacks:

   • Lack of personalization makes for generic conversations that disengage users.

   • Chatbots that cannot respond empathetically to an emotional states should hold very limited chances of being successful.

   • Conversations with no emotion content to draw on to create even a small positive experience will likely lead a user to have lower satisfaction and retention rates, particularly in domains that need trust and emotional connection.

   With chatbot systems aiming to lessen these issues, our objective is to build a chatbot system that integrates live facial emotion recognition with sentiment-aware dialogue-based conversation logic, that changes its behaviour based on the user's emotional state.

   The aim of this endeavor is to enhance engagement, empathy, and effectiveness in human-computer interaction.

4. METHODOLOGY

   The proposed system is taking a modular approach that integrates computer vision with deep learning and chatbots. The approach can be divided into several components:

   1. Data Collection and Preprocessing

      • Datasets: Emotion classification models can be trained using publicly available facial expression datasets such as FER2013, CK+, or AffectNet.

      • Image Preprocessing: We will perform facial detection using either OpenCV or MediaPipe, this detected region of interest (ROI) will be normalized, resized, and converted into grayscale or RGB for the model input.

   2. Emotion Classification Model

      • A Convolutional Neural Network (CNN) or transfer learning model (i.e., MobileNetV2,erran Nv2-) will be used to classify facial expressions into seven emotions.

        Happiness , Sadness, Anger, Fear, Surprise, Disgust and Neutral

      • Libraries/Frameworks – TensorFlow/Keras or PyTorch, will be used for Model Development and Training.

   3. Chatbot Integration

      • The Emotion Detection Module is combined with a

        rule-based or NLP-based chatbot (i.e, using ASM's Rasa, Dialogflow …)

      • The chatbot will use a combination of intent- classification and emotion input to create response logic.

      • The chatbot response logic can dynamically: Down taper or ramp up emotion language Comfort or soothe language for negative emotions Energize and excite language when the subject displays positive emotions.

   4. Web User Interface

      • Frontend: Built as similar to HTML, CSS, JavaScript, having optional built-in functionalities with React.js for state management.

      • Backend: Flask or Node.js can be used to give functionalities to REST API calls and real time video streaming.

      • Features:

        Detect emotion in real time with the webcam video feed.

        Text and voice input for chat.

        Live updates for detected emotion and responses from the chatbot.

   5. Storage and Logging.

      • Storage can be done using a MongoDB or Firebase database which can be used to store:

      • User credentials and session- Detected emotions with timestamp. Chat log for personalization later and analysis.

5. SYSTEM ARCHITECTURE AND DESIGN

   Fig. 1. System architecture of read my mood

   A. Dataset Preparation

   The "Read My Mood" system is implemented as a modular client-server architecture that handles streaming video, emotion detection, chatbot interaction, and user management in real- time. The architecture is composed of three constituent layers– a Presentation Layer, a Business Logic Layer, and a Data Access Layer–each responsible for different parts of the application.

   1. A High Level Overview of the Components of the Systems

      1. Frontend (Client side)

         • Offers the interface.

         • Takes webcam input for facial emotion recognition.

         • Enables text or voice-based chat.

      2. Backend (Server side)

         • Processes facial data and classifies emotions using a trained deep learning model.

         • Interoperates with the chatbot logic to respond accordingly.

         • Manages multilingual interactions as well as text-to- speech.

      3. Database

         • Keeps the user logs, chat logs and emotion logs.

         • Handles the user sessions for tracking and analysis.

   2. Layer Architecture

      1. Presentation Layer (View Layer)

         • Technologies: HTML, CSS, JavaScript (or React.js)

         • Flask Features: Live webcam streaming ,Accepts voice (speech-to-text) inputs Emotion display overlay ,Chatbot (text + audio) interface

      2. Business Logic Layer

         • Components:

   1. Emotion Detection Module:

      • Deep Learning Models (Convolutional Neural Networks in TensorFlow/Keras)

      • Uses crops of faces, and outputs seven emotions classified

   2. Chatbot Engine

      • Built with Rasa, or Dialogflow

      • Perform random response actions from Intents and Emotion

   3. Sentiment and Tone Analyzer (not core component)

      • Natural Language Processing of the user messages that make a determination if the emotional state across – not included

   1. Access Layer (User Management)

      • all registrations, logins and authentications User: Admin/User

   access control on either the chatbot configuration, or logs

   Fig. 2. Data Flow Diagram: Level 1

   1. Data Flow Diagram (Described) Phase 1 – User Flow

      • User login – webcam detects face – emotion

        detected – chatbot changes – display reply phase

        Phase 2 – Internal Process

      • Emotion model – predict emotion

      • Sentiment analyzer – analyze text / voice tone

      • Chatbot – match intent + emotion – generate reply

      • Logger – persist emotion + chat log to DB.

   2. The user starts by registering in the system, typically through a sign-up or login process.

      • While registering or logging in, the system captures the user's facial image using the camera.

      • The captured face is analyzed using deep learning to detect the user's current emotion.

      • Once the emotion is recognized (e.g., happy, sad, angry), it is passed to the chatbot.

      • The chatbot then generates and sends a personalized, empathetic response based on that emotion.

        These interactions are illustrated in Fig. 3.

        Fig. 3. DFD: Level -2

   3. The system involves two primary actors:

      • Admin responsible for managing chatbot training, updating emotional response logic, and configuring system settings.

      • User interacts with the application by registering, logging in, and using the emotion-aware chatbot.The chatbot supports emotion detection through facial expressions, voice input, and multiple languages.

        This diagram highlights how each actor interacts with the system components during registration, emotion capture, and conversation flow.

        Fig. 4. Use Case Diagram

   4. Sequence Diagram – The user interaction sequence outlines the step-by-step flow of the chatbot system. The process begins when the user registers or logs into the application. Once authenticated, the system activates the webcam to detect the users facial emotion in real time. The user then selects a preferred language and engages with the chatbot using either text or voice input. Based on the detected emotion and chosen language, the chatbot generates a personalized response, making the interaction empathetic and human-like.

      Fig. 6. Admin Sequence Diagram

6. IMPLEMENTATION DETAILS

   Bringing "Read My Mood" to life includes folding together facial recognition, emotion identification, chatbot intelligence

   Fig. 5. User Sequence Diagram

   1. Admin Sequence- The admin plays a crucial role in managing the backend operations of the emotion-aware chatbot system. The process begins with the admin logging into a secure dashboard interface. Once authenticated, the admin can update or fine-tune the chatbot's dataset, which includes emotional responses, language options, and user intent recognition. This ensures that the chatbot continues to deliver relevant and empathetic replies based on the evolving needs of users.

   The admin is also responsible for configuring system parameters such as language models, emotion thresholds, and chatbot behavior rules. Additionally, the admin has the ability to monitor ongoing user sessions to ensure the system is functioning smoothly and without bias or errors. This includes reviewing emotion detection accuracy, user engagement metrics, and overall system performance.

   In case of issues, the admin can intervene, re-train models, or deploy updates. The sequence ensures that the system remains scalable, updated, and aligned with user expectations, making the admin a critical component in sustaining system efficiency and emotional intelligence.

   and frontend UI design, as all are folded into a single real-time application. This section explains the technology, algorithms, and procedures that went into developing the system.

   1. Emotion Detection Model

      1. Model Design

         • A Convolutional Neural Network (CNN) is used for emotion recognition.

         • Inputs: 48×48 grayscale image of detected face.

         • Output: One of the 7 emotion types (happiness, sadness, anger, surprise, fear, disgust, neutral).

         • Layers: Convolution + ReLU, MaxPooling,

           Dropout, Dense (fully-connected), Softmax output

      2. Training method

         • Dataset: FER2013 (or optionally CK+ or AffectNet).

         • Preprocessing: Face align and resize , Normalizing (pixels values scaled to [0,1])

         • Framework: TensorFlow/Keras or PyTorch

         • Loss: Categorical Crossentropy

         • Optimizer: Adam

         • Epochs: ~3050

         • Validation Accuracy: ~6575% depending on the dataset

   2. Integration of Chatbot

      • Framework: Rasa (Open Source) or Dialogflow

        (Google)

      • NLU Pipeline: Intent classification , Entity extraction

      • Custom Actions: Get the current detected emotion from the model , Change tone/interaction based on sentiment (e.g., comforting for sadness, calming for anger)

      • Training Data: Intents such as greeting, problem report, general questions , Emotion modifiers added to each intent (e.g., greet_happy, greet_sad)

   3. Speech and Language Processing

      • Speech-to-Text (STT): Google STT API for converting the user's voice to text input

      • Text-to-Speech (TTS): Google TTS or pyttsx3 for voice reply output

      • Language Support: Multilingual chatbot reply templates – to be added later

   4. Frontend

      • WebCam Capture: HTML5 + JavaScript with OpenCV.js or MediaPipe

      • Chat Interface : React.js (Flask also can be used for simplified version) , Embeds emotion detection with chat reply in real-time

      • Overlay : Shows emotion description above a live web web-cam feed (ex. Detected : Sadness)

   5. Database & Storage

      • MongoDB or Firebase was used for storing: User profiles and login , Emotion logs which includes time stamps , Chat transcripts

      • Thereby allowing tracking of users for personalization and allowing future data analytics.

   6. Step by Step Workflow Summary

   1. User logs into the web-ui

   2. WebCam captures live video

   3. Emotion model detects user facial expression

   4. User sends message (text/voice) for chatbot to respond to

   5. Chatbot obtains emotion and interprets the message intent

   6. Chatbot generates reply with language to support emotion

   7. Chat and emotion logs are stored to db

7. RESULTS AND DISCUSSION

   The "Read My Mood" system was evaluated for its emotional detection function, along with its ability to adjust the chatbots responses in real-time. In this section, we discuss some of the most relevant performance metrics, observable behaviours during testing, and what we took away from our findings.

   1. Emotion detection accuracy

      • Overall, our CNN model on the FER2013 dataset achieved an average detection accuracy of 72.4% on

        our validation set.

      • Emotion types such as, happiness and neutrality had high precision (>80%), while disgust and fear were much lower accuracy, which is a well-known problem with imbalanced datasets.

      • Real-time detection provided stable results for frontal faces with good light conditions, and stable results with side profiles, but none for occluded faces. We saw a minor decrease in performance when we detected side profiles or occluded faces (i.e., angle away, hats, sunglasses, etc.).

   2. Reactivity of Chatbots

      • Using scenario-based testing, the chatbot was assessed qualitatively.

      • It displayed emotionally suitable reactions like: Providing consolation when sadness was detected: "I'm here for you." Would you like to discuss your concerns?, Calm tone when angry: "Let's take a deep breath and work through this together." Positive comments for contented users: "That's fantastic! Tell me more about the highlights of your wonderful day.

      • With local hosting, integration delays were negligible (response time < 2 seconds).

   3. The Experience of the User 90% of the 10 volunteers in a small user study, who were between the ages of 18 and 30, reported higher engagement as a result of emotional responsiveness.

   • Eighty percent of respondents thought the chatbot was more human-like than other bots. According to one user, "It felt like the system could actually understand how I was feeling, which made it easier to open up." These results suggest that the system has the ability to enhance emotional engagement in online discussions.

   1. ATS Score (out of 100): Depending on keyword matching, section coverage, tools and certifications, metrics used, job role relevance, and grammar.

   2. FormattingScore (out of 100): Depending on formatting guidelines like section presence, font uniformity, absence of all-caps, etc.

   3. Skill Match (%): Percentage of required job role skills found in the resume using semantic cosine similarity ([13]).

   4. Visual Score Representation: Scorecards and gauges generated using Plotly provide quick feedback to users.

8. USES

   Numerous useful applications are made possible by the "Read My Mood" system's capacity to recognize and respond to emotions:

   1. Assistance for Mental Health It serves as a virtual tool for emotional check-in.

      • Encourages expression by providing nonjudgmental

        answers.

      • Capable of offering relaxation methods, crisis information, or professional referral (future enhancement).

   2. Client Support

      • Early detection of frustration or discontent.

      • Modifies tone to be soothing and comforting. If emotional escalation is detected, it can alert human agents.

   3. Learning Environments

      • Recognizes when students are bored, confused, or frustrated.

      • Modifies the material or recommends breaks.

      • Empathetic tutoring increases student motivation and retention.

   4. Games & Entertainment

      • Story branching or NPC actions motivated by emotions.

      • Enhanced user immersion according to mood.

      • Tailored interactive storytelling.

   5. Wellbeing at Work

      • Tracks employee mood during breaks or logins.

      • Suitable for use in HR wellness programs' daily emotion dashboards.

9. FUTURE EXTENT

   The current iteration of the "Read My Mood" system lays the groundwork for artificial intelligence that is emotionally intelligent. Nonetheless, there are numerous chances to improve it in order to increase its impact and capabilities:

   1. Recognition of Voice Emotions

      • Future Direction: To identify emotion from spoken input, incorporate voice analysis using acoustic characteristics like pitch, energy, and speech rate. Benefit: Increases the system's resilience and multimodality by adding a second emotional channel.

   2. Customized Emotional Characteristics

      • Method: Record recurrent moods and reactions to create an emotional history for every user.

      • Use Case: Customize chatbot interactions according to user preferences, emotional patterns, and past actions.

   3. Cross-Cultural and Multilingual Support

      • Implementation: Include support for various languages and cultural expressions of emotion.

      • Significance: Promotes inclusivity and opens the system to audiences around the world.

   4. Emotion-Adaptive Content

      • Idea: Adapt chatbot responses to emotional input by changing their tone, content depth, language complexity, and topic sensitivity.

      • Use: Particularly beneficial in fields like education or therapy.

   5. Deployment Across Platforms

      • Growth: Create desktop and mobile app versions with Electron, React Native, or Flutter.

      • Objective: Expand the system's reach by making it accessible on desktop, kiosk, and smartphone platforms.

   6. Wearable Technology Integration

      • Possibility: Integrate biometric signals (e.g., heart rate, stress levels) by connecting with fitness bands or smartwatches.

      • Benefit: Develops a hybrid emotional detection system by combining physiological, aural, and visual information.

10. CONCLUSION

    The next step up from functional communication to empathetic engagement in human-computer interaction is represented by emotionally intelligent systems. This study introduces "Read My Mood," an AI-driven chatbot that can recognize and react to human emotions through real-time facial emotion detection.

    Warm, sympathetic interaction and cold, logic-based AI are successfully bridged by the system. It improves the caliber of digital communication in fields like mental health, education, customer service, and entertainment by utilizing deep learning, natural language processing, and user-centric design. The project establishes the foundation for more sophisticated, multimodal emotion-aware systems, even though the current implementation concentrates on facial expression analysis.

    Future developments like voice emotion recognition and personalization could revolutionize how machines perceive and react to human emotions, transforming AI into more than just intelligent technology.

11. REFERNCES

1. W. V. Friesen and P. Ekman (1971). Face and emotion transcend cultural boundaries.

2. Hasani, B., Mahoor, M. H., & Mollahosseini, A. (2016). AffectNet: A Database for Wild People's Arousal, Valence, and Facial Expressions. IEEE Affective Computing Transactions.

3. Poria, S., Bajpai, R., Cambria, E., & Hussain, A. (2017). A Comprehensive Overview of Affective Computing: From Multimodal Fusion to Unimodal Analysis. Fusion of Information, 37, 98125.

4. Roisman, G. I., Huang, T. S., Zeng, Z., & Pantic, M. (2009). A Summary of Methods for Identifying Visual, Aural, and Unplanned Expressions. IEEE Transactions on Machine Intelligence and Pattern Analysis, 31(1),

   3958.

5. Bengio, Y., Courville, A., & Goodfellow, I. (2016). deep learning. The Press of MIT

6. E. W. Pamungkas, "Emotionally-Aware Chatbots: A Survey," arXiv preprint arXiv:1906.06083, 2019.

7. S. Li and W. Deng, "Deep Facial Expression Recognition: A Survey," arXiv preprint arXiv:1804.08348, 2018.

8. S. Poria, N. Majumder, R. Mihalcea, and E. Hovy, "Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances,"

   *IEEE Access*, vol. 7, pp. 100943100953, 2019.

9. "Emotion-Regulatory Chatbots for Enhancing Consumer Servicing,"

*Journal of Retailing and Consumer Services*, vol. 73, 2023, by R. Dev and A. Mukherjee.