 Open Access
 Total Downloads : 169
 Authors : M. Divya, Dr. S . Chitrakala
 Paper ID : IJERTV3IS031075
 Volume & Issue : Volume 03, Issue 03 (March 2014)
 Published (First Online): 27032014
 ISSN (Online) : 22780181
 Publisher Name : IJERT
 License: This work is licensed under a Creative Commons Attribution 4.0 International License
Topic Modeling over Asynchronous Text Sequences
1M. Divya, 2Dr. S. Chitrakala
1Anna University, 2 AnnaUniversity,Chennai
Abstract – Topic models have been widely used to identify topics in text corpora. Topic models are a suite of algorithms that uncover the hidden thematic structure in document collections. These algorithms help us develop new ways to search, browse and summarize large archives of texts. They provide a simple way to analyze large volumes of unlabelled text. A "topic" consists of a cluster of words that frequently occur together. Using contextual clues, topic models can connect words with similar meanings and distinguish between uses of words with multiple meanings. One key challenge in topic modeling is to develop a fast solution for indexing high dimensional data which is crucial to build large scale application data. This survey highlights on the current probabilistic topic models. It also point out scope and challenges in topic modeling.

INTRODUCTION
The traditional method of learning about the particular topic is to read a book or a survey paper related to the topic. To buy a book is a time consuming and is inconvenient .This traditional method is not been applicable in most of the cases because of the topics and the technologies emerge in the fast growing world. To know the depth knowledge about a particular topic among the collection of documents that has become increasingly important and popular. No one will spend time by writing a book on some particular topic. To have knowledge of such emerging topic one can resort to research papers. But the research papers are not understandable by nonresearchers and few research papers cover all aspects of the topic, there came into existence of topic modeling.
Topic modeling is the way of extracting common topics that occurs among the collection of documents. A document may have depth about a particular topic or multiple topics with different proportions. By using mathematical framework topic model examines each and every documents and discovers based on statistical words in each documents.
Text sequences are in many forms namely news streams, web log articles, email, instant messages, and research papers. The first step is to extract topic from sequence with both semantic and temporal information that are described by two distribution namely word distribution describing the semantic of the topics and time distribution that describes the topics intensity over time.
Multiple sequences that shared the common topic having correlating with each other provides meaningful and comprehensive topic than the individual stream. The recent work is explored in the multiple sequence of temporal correlation in order to get the semantic correlation of topics.
In the case of asynchronism among multiple sequences (i.e) document from different sequences among the same topic that have different timestamps is common in practice. It is no guarantee in the news feed that the news covering the same topic will not be indexed with the same timestamp. The newspaper which takes hourly basis to publish the news , in the periodic it takes weekly basis and even some source will take monthly basis to publish the news that discuss about the common topic. The another example in the research archives is that papers published in the researches will undergo the proceeding of newsletters that is within the week or a month and followed by the conference which are published annually and finally in the journals that takes more than a year to publish. The paper published will share a common topic but at different time stamp. To visualize it the occurrence of two terms warehouse and mining in the title of all research papers published in two different sequences namely SIGMOD and TKDE respectively. In the case of asynchronism if the conventional topic mining is used it fails to find that mining and warehouse as a common topic. For mining common topics from multiple text sequences an effective method is been proposed. It defines the problem by introducing a principled probabilistic framework and put forward an algorithm to optimize the objective function by exploiting the mutual impact between the topic discovered and time synchronization. The key idea is to utilize the semantic and temporal correlation among the sequences and to build up a reinforcement process. To start, by extracting a set of common topics from given sequences using their original time stamps. And to update the time stamp of the document based on the extracted time stamp and word distribution. Finally update the time stamps of documents in all sequences by assigning them to most relevant topic.

RELATED WORK
Topic Detection and Tracking (TDT) works with the clustering based technique that aims to find and track in news sequences [12], [13] then came into existence of
probabilistic generative models such as PLSA [14], LDA
[15] and their derivatives.Hoffman et.al [3] has proposed PLSA is a statistical model for analyzing two modes and finding the cooccurrence of data. This model aims creating each document in corpus, where each word in a document is sampled from a mixture of multinomial distribution that can be interpreted as topics, and proportions corresponding to mixture weights are sampled from a separate multinomial distribution for each document. Problem with PLSI has mixture parameter of each document that has too many parameters.
Bhei et.Al [4] used each document in the LDA to be viewed as the mixture of topics. LDA is similar to PLSA expect that the topic distribution is Dirichlet Prior. The problem of PLSA can be overcome by LDA by increasing the number of estimates by using Dirichlet prior distribution. The complexity of LDA is more when compared to PLSI that helps to find exact inferences from generative model. SparseLDA is approximately 20 times faster than highly optimized traditional LDA and twice the speed of previously published fast sampling method. To efficiently cope with this problem, several approximate inference algorithms are derived such as Variational Inference, and various Markov Chain Monte Carlo algorithms, such as Gibbs Sampling. LDA is more expensive than PLSA and LDA does not have some of the features such as more complex model like finding relation between the topics.
Bhei et.Al [6] introduced hierarchical LDA which is the LDAs extension model is Hierarchical LDA and is introduced by Bhei in 2003. LDA model a flat topic structure instead HLDA models tree of topics. Hierarchical LDA uses nonparametric Bayesian approach to model hierarchies. Tree of topic is constructed hierarchically of nodes by an algorithm. In topic tree model each and every node is represented by random number and has got corresponding wordtopic distribution assigned to it. The tree can be traversed from the root till its leaves while sampling topics along the path
Text sequences is that the text collection which carries the temporal information. The text sequences will be seen in many applications. In order to capture the dynamic topic various methods has been proposed over time in text sequences. Even from the single sequences also these methods can be used. The individual sequences [5],

can be modeled with the random variable can either be discrete or continuous. It is assumed that in the given sequences, the timestamp of the document is independent from word. In [1] the author introduced hyper parameter that evolves over time in state transfer in the sequence. For each time slice, a hyper parameter is assigned with a state by a probability distribution, given the stat on the former time slice.(The timestamp of the document is independent from word)
.
In [7] for each slice the time dimension is cut into time slice and topics. As a result in multiple text sequences, topic in each sequence can only be estimated separately and potentially correlated between topics in different sequences both semantically, temporally cannot be fully explored. From the static text collection the semantic correlation [16], [17], [18] between different topics was considered. Similarly, Zhai et al.[19] explored common topic in multiple static text sequences.
Wang et al.[11] has proposed a very recent work in topic mining methods that aims to discover common (bursty) topic over multiple text sequences. Their approach is entirely different from this method because this method deals with asynchronism (i.e) common topic occurring over different sequences with different time stamp. But Wang proposed to find common topic from different sequences that are synchronized and coordinated. Based on this premise, documents with same time stamp are combined together over different sequences so that the word distributions of topics in individual sequences can be discovered, As a contrast, this method aim to find topics that are common in semantics, while having asynchronous time distributions in different sequences.
Asuncion et al. [20] studied a generalized asynchronous distributed learning scheme with applications in topic mining. However, in their work the term asynchronous means a set independent Gibbs sampler which communicates with each other in an asynchronous manner. Therefore, their problem setting is fundamentally different this method.
We also note that there is a whole literature on similarity measure between time series (sequences). Various similarity functions have been proposed, many of which addressed the asynchronous nature between time series [21], [22]. However, defining an asynchronism robust similarity measure alone does not necessarily solve our problem. In fact, most of the similarity measures deal with asynchronism implicitly, rather than fix the asynchronism explicitly. It introduces a generative topic model which incorporates both temporal and semantic information function, which is to maximize the likelihood subject to certain constraints.


PROPOSED SYSTEM
The proposed work is based on the topic modeling PLSI. PLSI is a statistical model for analyzing two modes and finding the cooccurrence of data. This model aims creating each document in corpus, where each word in a document is sampled from a mixture of multinomial distribution that can be interpreted as topics, and proportions corresponding to mixture weights are sampled from a separate multinomial distribution for each document.
Common
Topic
Time offset
Figure 1 System Design
The text sequences which include the detail of title name, author name and time (in the form of date) is given as input. The text sequence is given to the first module as an input called preprocessing. PreProcessing module will remove the stop word and will undergo stemming process that will lead to the meaningful sequences without the meaningful words like of, the, an etc. The meaningful sequence is given as input to the second module namely topic extraction which will extract the common topic from the meaningful text sequences and store it in the database. Based on the topic extracted in the topic extraction module along with the time stamp of the each document, synchronization is done and finally the topic is extracted based on the selected timestamp.
The Topic modeling over asynchronous text sequence will undergo the following modules

PreProcessing

Topic Extraction

Time Synchronization
Preprocessing is the first module in the topic modeling. The preprocessing has two process namely stop word removal and stemming. The dataset or the text corpus is given as input in which the stop word. The stop word will remove some common, short function words such as the, is, at, which, and on. So as a result the stop words output is given as input for stemming in which the root words will be removed like the words ending with ing, ive, ate, able.
The input of the topic extraction is the processed sequence of the preprocessing and the output retrieved by this module is the common topic. Topic Extraction evaluates the common topics with no meaningful word from multiple text sequences. It then performs extracting of same topic from different sequences with different timestamps. The extraction is done by using
an EM algorithm which will extract the top list words from the whole dataset.
The EStep,
and MStep
The EM step is used to find the maximum likelihood function. These steps are repeated alternately and the objective function guarantees to converge to a local optimum. This process will collect the unique topic by using pair wise divergence.
KLdivergence indicates that the two topics are more discriminative to each other and 0 divergence means that two topics are identical. This method extracts much more discriminative topics than those extracted by the previous method.
Figure 2 Topic Extracted by PLSI method
The input given to the time synchronization is the common topic extracted in the topic extraction module. The
Precision:
Number of retrieved relevant documents
matching of the document is done based on the time stamp
adjustment of the document that will extract the topic published during the period of time.
Recall:
Total number of retrieved documents
Number of retrieved relevant document
Total number of relevant documents
Accuracy :
Relevant documents retrieved in top T
T
Perplexity is a commonly used intrinsic evaluation metric in the topic model literature (Blei et al. 2003; Griffiths and Steyvers 2004). Perplexity originates from language modelling, and is calculated by estimating the probability of words in held out/test data based on training data.
Figure 3 Maximum count of words extracted in dataset
Figure 3 and figure 4 shows the extracted common words from the dataset by using both topic extraction and time synchronization algorithm alternately and it finds the maximum likelihood of the words that shows the most research topic that is been published in the archives of both conferences and also journals. The collected topics will help the researches to concentrate in their focused area.


PARAMETERS USED FOR EXPERIMENTAL EVALUATION:
The standard parameters which are used for experimental evaluation are precision, recall and accuracy.
Precision is defined as number of retrieved relevant documents divided by total number of retrieved documents and the recall is the number of retrieved relevant document divided by total number of relevant documents in the database. Accuracy can be calculated as relevant document retrieved in top T returns divided by T. The formulas for calculation of these evaluation parameters can be given as following:
Where M is the number of documents in test; wm are the words in document m; and
Nm is the number of words in document m.
60%
50%
40%
30%
20%
10%
0%
Precision
Recall
Accuracy
PLSA LDA HLDA ATM
Fig. 2. Graphs showing the average recall, the average precision as in [9]

CONCLUSION:
The proposed method discovers and fixes potential asynchronism among sequences and consequentially extracts better common topics. It performs topic extraction and time synchronization alternatively to optimize a unified objective model. This method significantly outperforms
both in quality and in quantity. So the performance of the method becomes more robust and stable. The framework is not only useful in practice but also valuable to machine learning because human beinglearn knowledge and even topics change over time and past knowledge is not discarded but used to solve new problems.
REFERENCES:

A. David, J. Li, L. Zhou, and F. Muhammad, Knowledge discovery through directed probabilistic topic models: a survey, Frontiers of Computer Science in China, vol. 4, no. 2, pp. 280301, Jun. 2010.

David M. Blei. Introduction to Probabilistic Topic Models. Communications of the ACM, 2011 pp.

Mark Steyvers, Tom Griffiths. Probabilistic Topic Models. In Landauer

Zhu, Jun, and Eric P Xing. Conditional Topic Random Fields. Forbes. Ed. Johannes FÃ¼rnkranz& Thorsten Joachims.

Steven Abney and Marc Light. 1999. Hiding a semantic hierarchy in a markov model. In Proceedings of the Workshop on Unsupervised Learning in Natural Language Processing, pages 18.

T. L. Griffiths, M. Steyvers, D. M. Blei, and J. B. Tenenbaum, Integrating topics and syntax, In Advances in Neural Information Processing Systems 17, vol. 17, 2005, pp. 537544.

A. Gruber, M. RosenZvis, and Y. Weiss, Hidden topic markov models, in Artificial Intelligence and Statistics, 2007.

T. L. Griffiths and M. Steyvers, Finding scientific topics, Proceedings of the National Academy of Sciences of the United States of America, vol. 101, no. Suppl 1, pp. 52285235, Apr. 2004.

Wang Wei, PayamBarnaghi, Probabilistic Topic Models for Learning Terminological Ontologies Member, IEEE, and AndrzejBargiela, Member, IEEE.

X. Wang, A. McCallum, and X. Wei, Topical ngrams: Phrase and topic discovery, with an application to information retrieval, in ICDM 2007: Proceeding of the seventh IEEE International Conference On Data Mining, Ed., 2007, pp. 697702.

X. Wang and A. McCallum, Topics over time: a nonMarkov continuoustime model of topical trends, in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery
and data mining, ser. KDD 06. New York, NY, USA: ACM, 2006, pp. 424433.

D. M. Blei and J. D. Lafferty, Dynamic topic models, in Proceedings of the 23rd international conference on Machine learning, ser. ICML 06. New York, NY, USA: ACM, 2006, pp. 113120.

D. Blei, T. Gri, M. Jordan, and J. Tenenbaum, Hierarchical topic models and the nested Chinese restaurant process, 2003.

D. M. Blei and J. D. Lafferty. (2006) Correlated topic models.

T. Hofmann. (1999) Probabilistic latent semantic indexing.

S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R.Harshman,Indexing by latent semantic analysis, Journal of the American Society for Information Science, vol. 41, pp. 391407, 1990.

D. M. Blei and J. D. Mcauliffe, Supervised topic models,, in Proceedings of th Neural Information Processing Systems NIPS, 2007.

D. Mimno and A. McCallum, Topic Models Conditioned on Arbitrary Features with Dirichletmultinomial Regression, in Proceedings of the 24th Conference on Uncertainty in Artificial Intelligence (UAI 08), 2008.

M. R. Zvi, C. Chemudugunta, T. Griffiths, P. Smyth, and M. Steyvers, Learning authortopic models from text corpora, ACM Trans. Inf. Syst., vol. 28, no. 1, pp. 138, Jan. 2010.

D. M. Blei and P. J. Moreno, Topic segmentation with an aspect hidden markov model, in Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, ser. SIGIR 01. New York, NY, USA: ACM, 2001, pp. 343348.

J.P. Yamron, I. Carp, L. Gillick, S. Lowe, and P. van Mulbregt, A Hidden Markov Model Approach to Text Segmentation and Event Tracking, Proceedings ICASSP98, Seattle, May 1998.

Y. Wang, H. Bai, M. Stanton, W. Y. Chen, and E. Y. Chang, PLDA: Parallel latent dirichlet allocation for LargeScale applications, in Proceedings of the 5th International Conference on Algorithmic Aspects in Information and Management, ser. AAIM
09, vol. 5564. Berlin, Heidelberg: SpringerVerlag, 2009, pp. 301
314.

W. Li and A. McCallum, Pachinko allocation: DAGstructured mixture models of topic correlations, in Proceedings of the 23rd international conference on Machine learning, ser. ICML 06. New York, NY, USA: ACM, 2006, pp. 577586