Enhancing Opinion Mining using Deep Learning Models

Enhancing Opinion Mining using Deep Learning Models

Dr. K. Narasimhulu

Associate Professor, Dept. Of CSE Rajeev Gandhi Memorial College of Engineering and Technology Nandyal, India

R. Nitheesha

Dept. Of CSE Rajeev Gandhi Memorial College of Engineering and Technology Nandyal, India

S. Faiza

Dept. Of CSE Rajeev Gandhi Memorial College of Engineering and Technology Nandyal, India

P. Renu Sree

Dept. Of CSE Rajeev Gandhi Memorial College of Engineering and Technology Nandyal, India

S. Mohammed Shyaan

Dept. Of CSE Rajeev Gandhi Memorial College of Engineering and Technology Nandyal, India

Abstract: Sentiment analysis is used for understanding user-text as it is positive, negative or neutral the traditional lexicon-based models often fail to analyze linguistic nuances within complex sentences. This research addresses these limitations by developing a hybrid deep learning architecture that combines the Bidirectional Encoder Representations from Transformers (BERT) with the Enhanced Bidirectional Long Short-Term Memory (BiLSTM) network. This method utilizes BERT to understand the context of the data and then BiLSTM is used for the sequential dependencies it remembers the long reviews and mixed opinions . Additionally, a specialized emoji classification module is added to analyze non-verbal signal(emojis), combing of both visual intelligence with textual intelligence. This model is evaluated with social media reviews data set it reduced the neutrality and also this model has increased the accuracy of analyzing the text-data.

Key Words: BERT, Bidirectional Long Short-Term Memory (BiLSTM), Deep Learning, Emoji Analysis, Natural Language Processing (NLP), Opinion Mining, Sentiment Analysis, Transformer Models.

1. INTRODUCTION

   The rapid increase of digital communication and online academic platforms has generated an unprecedented volume of unstructured textual data. Sentiment analysis, or opinion mining, is being used

   for computing the feelings of the text and analyzing it as positive,negative and neutral from the given data set for improving the decision-making and user experience. While early methods like lexicon it depends on a fixed words list so it is very difficult to address and analyzing of sarcasm and non verbal signals like emojis.

   The evolution of Natural Language Processing (NLP) has seen a transition from fixed list dictionaries to models to analyzing architectures. Traditional machine learning classifiers, such as Support Vector Machines (SVM), often fail to capture the long-range dependencies and all-around view is more important for analyzing the context [19]. Even though Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM)is used to address sequential data loss, they are struggling with speed and wide-range vision [22]. The introduction of the Transformer architecture and Bidirectional Encoder Representations from Transformers (BERT) has defined the state-of-the-art by utilizing self-attention mechanisms to generate deep contextual embeddings [1].

   Even though it consists of many benefits, there is a clear gap in maintaining track of long-form reviews while simultaneously resolving the indecision of the lexicon based models [16]. This project addressed these challenges and proposed a hybrid deep learning model. By combing the conceptual intelligence BERT

   with the sequential memory of Bidirectional LSTMs (BiLSTM) and a lexicon based emoji module, this study achieves a robust classification accuracy of 94.2%.

   Problem:

   The main challenge in this sentiment analysis is the inability of architecture of the single models to analyze complex grammar, sequential dependencies, and non-verbal emotional signals. Traditional lexicon- based models are used to relay on the fixed dictionaries so they frequently fail to understand emotions, inverters, and modifiers of the text [17], [18]. Furthermore,the traditional models makes use of fixed list words and ignores sentences flow cannot understand real meaning and also cant remember long reviews [19], [20].whereas modern transformer architectures like BERT has resolved these issues through By looking at the whole sentence at once and understanding the neutral reviews correctly the main point of review from comments[1], [16]. therefore,this system combines brain and memory along with emotional signals this is combined into a single architecture to secure accurate analysis.

   Proposed Methodology:

   Fig. 1. Architectural workflow of the proposed Hybrid BERT-BiLSTM framework for sentiment classification and emoji-intensity fusion.

   Data set collection :

   To built this project, we gathered a social media feedback comments to analyze the machine and human conversations to understand the Sentiment field, providing categorical labels (0, 1, and 2). It addresses a positive negative and neutral of text data in modern training models making it ideal for training hybrid BERT-BiLSTM architectures and emoji combined modules. The data is more accurate for academic and organizational , serving as a standard model for evaluating opinion mining systems.

   Comment	segment
   guy face made movies voice made radio great personality wanna see garrett	2
   need august well best series	2
   awesome variety products also like five six items love budget finds	2
   wasnt prepared immediate start lol	0
   one actually watches snacks drinks since	2
   Canvas one wouldnt bad price didnt pay monthly membership	1
   floating cities sound pretty promising free land already scarce question rescue cities violent storms protect armed pirates floating city need defenses allowed within mile territorial limit	2
   gotta say one best episodes series actually think every single one things would something id go buy especially led sign finally picking back plan build pc flexibility sign adds lot well flexibility want endgame turn	2
   seems like fun person omg keep calm smoke joint let teach english tracy thats stupid question idk school youre going maybe change schools im dead	1

   Fig 2: Table contains information about dataset

   BERT (Bidirectional Encoder Representations from Transformers)

   Data Preprocessing:

   The cleaning of the data for the most optimized processing of data adds value to better classification accuracy by converting raw text into an NLP-ready format.

   Noise Reduction: We eliminated irrelevant data including HTML tags, URLs, and non-alphanumeric noise.

   Emoji Polarity Mapping: Before text cleaning, emojis were separated and converted into numerical sentiment scores. Text Standardization: To maintain consistency with the BERT-base-uncased model, all text was converted to lowercase. Tokenization and Sequence Alignment: Using the WordPiece tokenizer, text was broke into sub-words.

   Phase 2: Contextual Encoding (BERT)

   Phase 3: Sequential Dependency Analysis (BiLSTM)

   To overcome the limitations of static context, the BERT-derived vectors are fed into a Bidirectional Long Short-Term Memory (BiLSTM) layer.

   Temporal Tracking:gating method is used by bilstm to keep track of input sequence.

   positive

   Classification layer

   negative

   neutral

   FIG: Ber Architecture

   The separated text is mapped through a Transformer- based Encoder.

   Input Representation: The model provides a Integrated embedding (Etotal) from Token, contextualized, and component embeddings.

   High-Level Extraction: Through Multi-Head Self- Attention, the mechanism estimates global global long-range correlations throughout its ordered sequence.

   Feature Isolation: Rather than of using all hidden states, this model particularly extract the [CLS] token hidden state, which provides the context summary to input.

   • Forget Gate (ft): Decides which information from the previous state is discarded.

     ft =cr(Wf ·[ht-1,xt] + bf)

   • Input Gate (it): Determines which new information will be added to the cell state.

     It = cr(Wi ·[ht-1,xt] + bi)

   • Cell State Update (Ct): Combines the gated old memory with new candidate values (c~t)

     c~t = tanh(Wc ·[ht-1,xt] + bc) Ct = ft 0C t-1 + it0c~t

   • Output Gate (Ot): Selects the information to be passed to the next hidden state.

   Ot = cr(Wi ·[ht-1,xt] + bi) ht = Ot 0 tanh(Ct)

   Bidirectional Context:adding front pass h- t and backend pass h..- t the calculation is much easirer for

   capturing more dependencies,sarcasm and emotional shifts.

   t

   1. Forward Pass ( -h ): Processes the BERT hidden states from t=1 to T.

   2. Backward Pass( ..h- t): Processes the BERT hidden states from t=T to 1.

      The final output for each time step is the concatenation of these two directions, providing a unified representation of the past and future context:

      HBiLSTM = [h- t][ h..-t]

      Consequently, the TextBlob output for the 41,145 records shows an unreliable score, frequently misclassifying objective feedback as Positive (38.0%) simply because it lacks the depth understanding towards sarcasm and complex sentences with grammar.

      The final layer of this hybrid model is softmax which acts as the decision-making layer.After completion of the merging of textual data with emoji vectors it next processed to dense layer and non-linear activation and then scores mapped to probability distribution.

      	precisio n	recal l	F1_scor e	Suppor t
      Positive	0.96	0.95	0.96	8245
      Negative	0.94	0.94	0.94	7510
      Neutral	0.92	0.93	0.92	8580
      Accurac y			0.94	41145
      Macro avg	0.94	0.94	0.94	41145
      Weighte d avg	0.94	0.94	0.94	41145

      Result Display:

      The Hybrid BERT-BiLSTM model got 94.5% accuracy were as textblob cant get much accuracy as the hybrid model because textblob classifies mostly text as positive because it cant process sarcasm.but this hybrid model can process Neutrality bias to 35.6% as it can understand the content of the data and sequential dependencies along with the emoji detection.

      Textblob :

      The TextBlob results represents a traditional lexicon- based approaches which depends on the fixed lists of wors libraries for classifying of the sentences cant process the words in common because of contextual blindness,and can not process sarcasm and also it consider emojis as the noise and clear it which are important for accurate classification.

      The Hybrid BERT-BiLSTM model provides a more accurate and contextually accounts for classification by understanding sentences as a continuous sequence of meaning. This project architecture consists of four stages: a BERT Encoder to generate 768-dimensional contextual embedding, a BiLSTM Layer to capture bidirectional sequential dependencies an Emoji Extraction Module for intensity scoring, and a Feature Fusion layer to combines text and visual signals. By utilizing this deep learning pipeline, the hybrid model achieved a 94.5% accuracy rate, identifying a much more realistic Neutral sentiment (35.6%) and effectively resolving the ambiguities that caused the baseline to fail.

      Baseline: TextBlob Lexicon Results (41,000 Records)

      The TextBlob Baseline utilizes a lexicon-based approach where sentiment is analyzed by matching individual words against a pre-defined dictionary.

      Limitations: This method struggles from contextual blindness, meaning it only understand the individual words not according to sentence . It also lacks the ability to process emojis and the complex, informal language found in labor market data.

      Confusion Matrix Analysis: As seen in the diagram above, there is a high degree of error are Specifically in a large number of Neutral and Negative comments are incorrectly pushed into the Positive category because the model focuses on individual keywords rather than the overall sentence meaning.

2. Proposed: Hybrid BERT-BiLSTM Results (41,000 Records)

   The Hybrid BERT-BiLSTM model represents the advanced neural network developed in this model. It consists of:

   1. BERT Encoder: To understand the deep, context of every word.

   2. BiLSTM Layer: To understand the sequential dependencies of the text.

   3. Emoji-Intensity Module: To extract and weigh the emotional signal of emojis.

   4. Fusion Layer: To merge all features for a final classification.

Performance: On the same data set which contains 41,000 records, the hybrid model classification is more accurate than traditional lexicon based textblob model as hybrid model contains understanding of content with sequential dependencies.

Confusion Matrix Analysis:The above confusion matrix shows hybrid BERT-BiLSTM model is more accurate not confused for classifying the neutral values too along with the

emoji classification of the reviews it is accurate than textblob.

The performance of the textblob is not classified accurately when compared to this Hybrid BERT_BiLSTM as this model contains contextual understanding not only word reading and classifying it according to the fixed dictionaries like traditional models. it also contains sequential dependencies that makes more accurate at long reviews also in this data set also contains emoji detection with all this hybrid model achieved neutrality-bias with accuracy of 94%

. Conclusion and future Enhancement

This study demonstrates that hybrid model BERT- BiLSTM is successfully outperformed traditional textblob model in classifying sentiments of the social media reviews 41,000 records of data set it is done on the basis of understating the text and sequential dependencies along with the emoji module which makes the outcome of this model is more accurate in emoji detection also it gave a total accuracy of 94%in classification of the emotion of the text.so multi-model is more efficient than traditional lexicon based models like textblob.

Future Enhancements

Stream Processing: Integrating the architecture into Apache Spark/Kafka pipelines for real-time monitoring of job boards and social feeds.

References:

1. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in

   Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 41714186, Jun. 2019.

2. S. Hochreiter and J. chmidhuber, “Long Short- Term Memory,” Neural Computation, vol. 9, no. 8, pp. 17351780, Nov. 1997.

3. A. Vaswani et al., “Attention is All You Need,” in Advances in Neural Information Processing Systems (NIPS), pp. 59986008, 2017.

4. S. Loria, “TextBlob: Simplified Text Processing,” 2014. [Online]. Available: https://textblob.readthedocs.io/en/dev/

5. P. Felbo, A. Mislove, A. Søgaard, I. Rahwan, and

   S. Lehmann, “Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 16151625, 2017.

6. Z. Huang, W. Xu, and K. Yu, “Bidirectional LSTM- CRF Models for Sequence Tagging,” arXiv preprint arXiv:1508.01991, 2015.

7. Y. Kim, “Convolutional Neural Networks for Sentence Classification,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 17461751, Oct. 2014.

8. C. J. Hutto and E. Gilbert, “VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text,” in Proceedings of the International AAAI Conference on Web and Social Media, vol. 8, no. 1, pp. 216225, May 2014.

9. M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Transactions on Signal Processing, vol. 45, no. 11, pp. 26732681,

   Nov. 1997.

10. F. Chollet, “Keras,” 2015. [Online]. Available: https://github.com/fchollet/keras