Technology Trend Prediction from Social Media using Long Short Term Memory Network

The present study aims to develop a deep learning based data analysis model which would act as a framework for efficient implementation of Social Media Analysis (SMA). This framework combines four process for data analysis i) Data collection, ii) Pre-processing, iii) Technology Classification and iv) Technology Trend prediction. The study uses hybrid model of Deep Feed Forward Neural Network (DNN) and Long Short Term Memory (LSTM) network. The technology and tools learned by people becomes obsolete in short period of time. As I.T. industry required frequent upgrades in knowledge and new technologies are being released, it is essential to track and know new upcoming technology trend of the field. To achieve this aim deep learning model is developed to identify upcoming technologies from social media threads. This paper presents LSTM trend prediction model to predict technology trends from unstructured text content of social media sources. The proposed method ensembles classification and regression process in single architecture. First, it uses deep learning algorithm to build a classifier to correctly predict the technology topic of a discussion thread from its description. After technology identification, how frequently that technology is discussed with respect to time is calculated to generate temporal series of frequencies. The Long Short Term Memory (LSTM) network is combined with Deep Feed Forward Neural Network model for processing temporal topic sequence and frequencies recursively to predict technology trends from of social posts generated on social platforms. Keywords— Technology Trend Prediction, Social Media, Deep Feed Forward Neural Network, Long Short Term Memory Network


I. INTRODUCTION
The distillation of knowledge from unstructured social big data is, an extremely challenging tasks. Existing Social Media Analytics approaches includes machine learning and deep learning based model which are having limitations in identifying patterns and trends from unstructured content. Deep Learning algorithms extract high-level, complex abstractions as data representations through a hierarchical learning process. Deep Feed Forward Neural Network model learns underlying representations from data itself. Deep Neural Network model outperforms machine learning models in the task of classification and are considered as good classifiers, yet main component of Deep Neural Network is feed forward neural network which is not designed for time dependant data. Deep Neural Network is not suitable for time series based trend prediction problem where network requires context of previous observations.
Deep learning models do not have any understanding of their input, at least not in any human sense. There is fundamental differences between the straightforward geometric morphing from input to output that deep learning models do, and the way that humans think and learn. Humans can easily identify trending topics/things/places by what is being talked most and discussed most. In contrast, machine/model cannot perform this task in a single step. To achieve this goal, computing model should be able to identify technology from social threads, its relevance with time and requires mathematical quantification. In the presented work, process of trend analysis is decomposed into Classification and Tagging, Temporal Mapping and Trend Prediction.
The key characteristic of social platform is the continuous generation of content which leads to derivation of new knowledge out of it. Social platforms are dynamic in nature and they can be considered as new type of information resources for future trend predictions with application of data analytics techniques. The discussion threads and content available on the websites such as Twitter, Facebook and other web portals includes latest information and people's view on different subjects and topics. Using social media data to solve domainspecific problem is challenging due to complexity of the domain, lack of context, colloquial nature of language. Considering the aspect of problem-solving, the content diversity and data volume inherent in social media create significant practical challenges for extracting relevant information as it is similar to searching for a needle in a haystack. Identifying the "Emerging Technology Trends" from social media is an open challenge and appealing work which leads to the exploration of presented research problem. This type of research work can help stakeholders of education community including students, academic institutions, management, university etc. to support their decision making process in areas of employment, career path selection and curriculum designing process. The uniqueness of social media data calls for novel data mining techniques that can effectively handles user generated content. The motivation and idea behind  (Joachims, 2002). By feeding a large training corpus of annotated texts to a machine-learning algorithm, it is possible for the system to learn keywords, punctuation, and word cooccurrence frequencies. Statistical methods are semantically weak which identifies only obvious keywords and model will have less predictive value. As a result, statistical text classifiers only work with acceptable accuracy when given a sufficiently large text input. While these methods are able to classify text on the page-or-paragraph level, they do not work well on smaller text units such as sentences or clauses. The social posts have certain character limits and facts are presented in short text only. As social posts are having less words and limited content in it, to classify them better approach is requireed.

B. Machine Learning based Methods
In Data Analytics, ML based methodologies are used to devise and generate complex algorithms and models which lend themselves to a prediction. Traditional ML model performs feature extraction and model construction in a separated manner. The handcrafted features are firstly extracted by transforming raw data into a form of statistical quantity, frequency, and time-frequency to take the representative information before applying predictive models. The performance of the constructed model not only relies on the optimization of adopted algorithms but also is heavily affected by the handcrafted features. The feature extraction and selection are time-consuming which requires the complex data pre-processing methods depending upon the type data. Machine learning models requires manual feature engineering for individual data sources. In case of heterogeneous social data sources, pre-processing of data increases the model complexity.

C. Deep Learning based Methods
In deep learning based methods, features are learned by transforming data into abstract representations. Recently, deep learning approaches have gained attention from the research community and industry for their ability to automatically learn optimal feature representation for a given task, while claiming state-of-the-art performance in many tasks in computer vision, speech recognition and natural language processing. It has been observed that with large amount of training data, deep neural networks are able to efficiently map the raw input of text to a low-dimensional vector representation, which preserves important syntactic and semantic aspects of the input text. Deep learning models outperforms other models in task of classification, but such model can handle one problem at a time. Either classification or regression type of problems are handled at a time for data analysis. Deep Neural Network model generates only single output y for series of input. Such network does not support multiple outputs/series based on many given inputs. Main component of DNN is feed forward neural network which is not designed for time dependant data. Problem of trend analysis and prediction requires to process data which is having time parameter or time based sequence pattern. Each input shown to DNN is processed independently, with no state kept in between inputs. With such networks, in order to process sequence or temporal series of data points, entire sequence needs to be presented to network at once which is difficult in case of large amount of data. DNN is not suitable for time series or trend prediction problem where network requires context of previous observations as it suffers from vanishing gradient [5] issue with more hidden layers, where adding more layers to network does not work and model training will become still. DNN is sufficient for single task of classification, with less computational cost and less training time but cannot work with temporal data.
LSTM networks are designed for time series based prediction. While feed-forward networks map many input to only one output, recurrent networks can map one input to many output, many to many (translation), or many to one (classifying a voice). LSTM networks carries information across time steps, and they require more processing resources and time to train the model. Thus, use of LSTM is preferable for sequential and time based data only.

D. Deep Feed Forward and LSTM Model
In the presented work, the deep learning based data analysis model is designed to perform following task of social media analysis and technology trend prediction in single end-to-end architecture.
More formally, given a series of social posts S= {s1 ,s2 , . . . ,sn} which are created on time stamp TS ={ts1 ,ts2 , . . . ,tsT } where T ∈ R, we aim at predicting a future technology trends by generating time based sequences from unstructured textual content of social posts. To predict Tt+h where h is the desirable horizon ahead of the current time stamp, social posts are classified in to technology topics by deep feed forward network and new dataset S = {{y1 ,y2 , . . . ,yT },{ts1 ,ts2 , . . . ,tsT }} is generated where y1, y2 are predicted technologies from content and ts1, ts2 are timestamp. To predict the value of the next time stamp Tt+h+1, LSTM network is combined with deep feed forward neural network.
This research work focuses on the development of method where deep learning model is implemented for trend prediction from text content of social media sources. A major challenge in developing such deep learning model is to manage the data representation and transform data such that network models can be trained to generate expected output. DL models are capable to predict future outcome from data, but learning happens from the data itself. This model is developed to work with unstructured content without any quantitative values. The data representation from which neural network learn to predict pattern is managed at each step such that models are trained to predict short and long term patterns together in single architecture.   Phase 2: After technology topic identification, the topic sequence and time mapping needs to be performed for identifying the past trend patterns of posted topics. The datasets which contains details of date-time and topic, will serve as input to another network for future trend prediction. Time based sequence is prepared in this phase to provide input to LSTM neural network.

International
Phase 3: In this phase, LSTM Network is trained to perform prediction on time series data. LSTM network is capable to process long term patterns and able to determine how long to hold onto old information, when to remember and forget, and how to make connections between old memory with the new input. Here historical data is transformed to time series and existing data patterns are used to predict future trend pattern. Deep neural network is opted for technology classification from social text which is first component of model. After technology classification, quantification of data is performed to map technology term with time parameter. Day-wise, demand of each technology is calculated in second phase of model development to generate temporal sequence for trend prediction. In second phase of model development, the researcher has created component to map technologies with time parameter. After classification and tagging of data, day wise technology demand count is calculated to analyse trend of technology over the years and prediction of technology trend can be done by learning from historical sequence of data. Following figure 2 presents the structure of data after mapping technology demands with timestamp. Above generated sequence is passed as input to next model component LSTM for analyzing and prediction trend patterns from date-wise technology demands. For initial years, demand of python technology is less, which is showing increasing pattern gradually in data. After temporal sequence generation, trend prediction is performed in third phase. In last phase of model development process, trend predication component is developed to find up-down technology pattern from historical data and prediction of technology trend is done using deep LSTM network.
Researcher has developed model which can combine deep feed forward network and LSTM network to serve the purpose of technology classification and trend prediction both. This joint model implementation performs social media analytics and trend prediction task in single end-to-end architecture automatically.
For trend prediction from time sequence data, following model are experimented to evaluate performance of LSTM prediction.

A. Technology Trend Prediction using Auto Arima
Auto regressive statistical models are applied for prediction of time series data and regression analysis. Given a set of training examples, each marked as belonging to one or the other of two categories, an ARIMA algorithm builds a model that predict new data points based on analysis of past data trend pattern. Following configuration are selected for experiment and analysis. Auto ARIMA is applied with regression as predictor variable is separable linearly and having numeric value to predict. Model training details and model performance results are shown in Table 1 and Table 2.

B. Technology Trend Prediction using Linear Regression
Linear regression are supervised learning models with associated learning algorithms that analyze data used for and regression analysis. the regression model allows for a linear relationship between the forecast variable y and a single predictor variable x.Following configuration are selected for experiment and analysis. Linear Regression is applied with linear kernel as predictor variable is separable linearly and having numeric value to predict. Model training details and model accuracy results are shown in Table 4 and Table 5. The time complexity of linear regression model is 567s for training the dataset which is least compared to other models.

C. Technology Trend Prediction using LSTM
Support-vector machines are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis. Given a set of training examples, each marked as belonging to one or the other of two categories, an SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier. Following configuration are selected for experiment and analysis. SVM classifier is applied with linear kernel as predictor variable is separable linearly and having binary value to predict. Model training details and model accuracy results are shown in Table  2 and Table 3. V. CONCLUSION Researcher has selected deep feed-forward neural network for technology classification as discussed in above sections and extended study of deep neural network architectures for making predictive model scalable to process data volume of year 2008 to 2016. As a part of methodology, this paper covers construction, refinement and development of Deep Feed Forward and LSTM based Trend Prediction model. Model construction started with technology trend prediction using hybrid neural network approach. Refinement of model is carried out by experiment with three different regression models for evaluation of proposed architecture. Temporal Sequence Mapping is created to quantify text data and generate sequence of date wise technology demand. Technology trend prediction component is made using LSTM network. Subsequent to deciding the deep neural network based approach, studies were extend to develop end-to-end generalized model where hybrid neural network architecture of Feed Forward Deep Network and LSTM was introduced. The research study has successfully integrated multitier data sources, managed data representation for model generalization to facilitate accurate data analytics and prediction for the future. This model has been validated by the use case of predicting the emerging technology trends of information technology domain.