Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

project sentiment analysis


Published on

  • My brother found Custom Writing Service ⇒ ⇐ and ordered a couple of works. Their customer service is outstanding, never left a query unanswered.
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! I can recommend a site that has helped me. It's called ⇒ ⇐ They helped me for writing my quality research paper.
    Are you sure you want to  Yes  No
    Your message goes here
  • The difference between "like" and "love" (most women miss this) ✄✄✄
    Are you sure you want to  Yes  No
    Your message goes here
  • Überprüfen Sie die Quelle ⇒ ⇐ . Diese Seite hat mir geholfen, eine Diplomarbeit zu schreiben.
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi there! I just wanted to share a list of sites that helped me a lot during my studies: .................................................................................................................................... - Write an essay .................................................................................................................................... - Summary of books .................................................................................................................................... - Online coursework .................................................................................................................................... - proquest dissertations .................................................................................................................................... - Movies reviews .................................................................................................................................... - Best powerpoint presentations .................................................................................................................................... - Write a research paper .................................................................................................................................... - Homework help online .................................................................................................................................... - Professional resume writing service .................................................................................................................................. - Help with writing any papers ......................................................................................................................................... Save so as not to lose
    Are you sure you want to  Yes  No
    Your message goes here

project sentiment analysis

  1. 1. A Project Report on SENTIMENT ANALYSIS OF MOBILE REVIEWS USING SUPERVISED LEARNING METHODS A Dissertation submitted in partial fulfillment of the requirements for the award of the degree of BACHELOR OF TECHNOLOGY IN COMPUTER SCIENCE AND ENGINEERING BY Y NIKHIL (11026A0524) P SNEHA (11026A0542) S PRITHVI RAJ (11026A0529) I AJAY RAM (11026A0535) E RAJIV (11026A0555) Under the esteemed guidance of Dr. L. SUMALATHA Professor, CSE Department DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING UNIVERSITY COLLEGE OF ENGINEERING JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY KAKINADA, KAKINADA – 533003, A.P 2011 - 2013
  2. 2. UNIVERSITY COLLEGE OF ENGINEERING JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY KAKINADA, KAKINADA – 533003 ANDHRA PRADESH, INDIA CERTIFICATE This is to certify that the dissertation titled “Sentiment Analysis of Mobile Reviews Using Supervised Learning Techniques” is submitted by Y.NIKHIL (11026A0524), P. SNEHA (11026A0542), S.PRITHVI RAJ (11026A0529), I.AJAYRAM (11026A0535) , E.RAJIV (11026A0555), students of B.Tech.(CSE - IIMDP) , in partial fulfillment of the requirements for the award of the degree of Bachelor Of Technology in Computer Science and Engineering is a record of bonafide work carried out by them under my supervision. Dr. L. Sumalatha Internal Guide, Head of the Department, Professor, Department of CSE, University College of Engineering, JNT University, Kakinada.
  3. 3. DECLARATION This is to certify that the thesis titled “Sentiment Analysis of Mobile Reviews Using Supervised Learning Techniques” is a bonafide work done by us, in partial fulfillment of the requirements for the award of the degree B.Tech.(CSE-IIMDP) and submitted to the Department of Computer Science & Engineering, University College of Engineering, Jawaharlal Nehru Technological University, Kakinada. I also declare that this project is a result of my own effort and that has not been copied from anyone and I have taken only citations from the sources which are mentioned in the references. This work was not submitted earlier at any other University or Institute for the award of any degree. Place: UCEK, JNTUK Y NIKHIL (11026A0524) Date: P SNEHA (11026A0542) S PRITHVI RAJ (11026A0529) I AJAY RAM (11026A0535) E RAJIV (11026A0555)
  4. 4. ii ACKNOWLEDGEMENTS We express our deep gratitude and regards to Dr. L. Sumalatha, Internal Guide and Professor, Head of Department of Computer Science & Engineering for her encouragement and valuable guidance in bringing shape to this dissertation. We thankful to all the Professors and Faculty Members in the department for their teachings and academic support and thanks to Technical Staff and Non-teaching staff in the department for their support. Y NIKHIL (11026A0524) P SNEHA (11026A0542) S PRITHVI RAJ (11026A0529) I AJAY RAM (11026A0535) E RAJIV (11026A0555)
  5. 5. iii ABSTRACT Sentiment analysis or opinion mining is the computational study of people’s opinions, sentiments, attitudes, and emotions expressed in written language. It is one of the most active research areas in natural language processing and text mining in recent years. Its popularity is mainly due to two reasons. First, it has a wide range of applications because opinions are central to almost all human activities and are key influencers of our behaviors. Whenever we need to make a decision, we want to hear other’s opinions. Second, it presents many challenging research problems, which had never been attempted before the year 2000. Part of the reason for the lack of study before was that there was little opinionated text in digital forms. It is thus no surprise that the inception and the rapid growth of the field coincide with those of the social media on the Web. In fact, the research has also spread outside of computer science to management sciences and social sciences due to its importance to business and society as a whole. In this talk, I will start with the discussion of the mainstream sentiment analysis research and then move on to describe some recent work on modeling comments, discussions, and debates, which represents another kind of analysis of sentiments and opinions. Sentiment classification is a way to analyze the subjective information in the text and then mine the opinion. Sentiment analysis is the procedure by which information is extracted from the opinions, appraisals and emotions of people in regards to entities, events and their attributes. In decision making, the opinions of others have a significant effect on customers ease, making choices with regards to online shopping, choosing events, products, entities. The approaches of text sentiment analysis typically work at a particular level like phrase, sentence or document level. This paper aims at analyzing a solution for the sentiment classification at a fine-grained level, namely the sentence level in which polarity of the sentence can be given by three categories as positive, negative and neutral.
  6. 6. iv TABLE OF CONTENTS 1 INTRODUCTION 1.1 Objective 4 1.2 Proposed Approach and Methods to be Employed 4 2 LITERATURE SURVEY 2.1 Models 6 2.1.1 Naïve Bayes 6 2.1.2 Bag Of Words 10 2.1.3 Support Vector Machine 14 2.1.4 Principal Component Analysis 21 3 SYSTEM ANALYSIS AND DESIGN 3.1 Software and Hardware Requirements 30 3.2 Matlab Technology 30 3.3 Data Flow Diagrams 32 4 IMPLEMENTATION 4.1 Elimination of Special Characters and Conversion to Lower Case38 4.2 Word Count 38 4.3 Testing and Training 39 4.3.1 Naïve Bayes 39 4.3.2 Bag of Words 39 4.3.3 Support Vector Machine 40 4.4 Sample Code 40
  7. 7. v 5 TESTING 5.1 Testing Strategies 49 5.1.1 Unit Testing 49 5.1.2 Integration Testing 49 Top Down Integration Testing 49 Bottom Up Integration Testing 50 5.1.3 System Testing 50 5.1.4 Accepting Testing 50 Alpha Testing 50 Beta Testing 50 5.2 Testing Methods 50 5.2.1 White Box Testing 50 5.2.2 Black Box Testing 51 5.3 Validation 51 5.4 Limitations 51 5.5 Test Results 51 6 SCREEN SHOTS 6.1 Naïve Bayes 53 6.2 Bag of Words 56 6.3 Support Vector Machine 59 CONCLUSION 63 REFERENCES 64
  8. 8. vi LIST OF FIGURES Figure 1 : Example of Bag Of Words Figure 2 : Hyperplane separating two classes Figure 3:Geometrical Representation of the SVM Margin Figure 4 : A Graph plotted with 3 Support Vectors Figure 5 : Hyperplane separating different classes with an intercept Figure 6 : MATLAB Interfaces Figure 7: Level 0 and Level 1 Data Flow Diagram Figure 8: Level 2 Data Flow Diagram for the process 1 (Naïve Bayes) Figure 9 : Level 2 Data Flow Diagram for the process 2 (Bag of Words) Figure 10 : Level 2 Data Flow Diagram for the process 3 (Support Vector Machine) Figure 11: Phases of Software Development Figure 12: Naïve Bayes Main Screen for Input Figure 13: Naïve Bayes Screen with Test Data Figure 14: Naïve Bayes Screen with output Figure 15: Bag Of Words Main Screen for Input Figure 16 : Bag Of Words screen with Test Data Figure 17: Bag Of Words screen with output Figure 18 : Support Vector Machine Main Screen for Input Figure 19: Support Vector Machine screen with Test Data Figure 20: Support Vector Machine Screen with Output
  9. 9. vii LIST OF TABLES Table 1: Principle Component Analysis Data Set Table 2: Principle Component Analysis Mean and Variance calculation Table 3: Centered data matrix Table 4: Principle Component Projection Calculation
  11. 11. 2 1.INTRODUCTION Sentiment analysis refers to the use of natural language processing, text analysis and computational linguistics to identify and extract subjective information in source materials. Generally speaking, sentiment analysis aims to determine the attitude of a speaker or a writer with respect to some topic or the overall contextual polarity of a document .The attitude may be his or her judgment or evaluation affective state, or the intended emotional communication. Sentiment analysis is the process of detecting a piece of writing for positive, negative, or neutral feelings bound to it .Humans have the innate ability to determine sentiment; however, this process is time consuming, inconsistent, and costly in a business context It’s just not realistic to have people individually read tens of thousands of user customer reviews and score them for sentiment . For example if we consider Semantria’s cloud based sentiment analysis software .Semantria’s cloud-based sentiment analysis software extracts the sentiment of a document and its components through the following steps:  A document is broken in its basic parts of speech, called POS tags, which identify the structural elements of a document, paragraph, or sentence (ie Nouns, adjectives, verbs, and adverbs) .  Sentiment-bearing phrases, such as “terrible service”, are identified through the use of specifically designed algorithms .  Each sentiment-bearing phrase in a document is given a score based on a logarithmic scale that ranges between -10 and 10 .  Finally, the scores are combined to determine the overall sentiment of the document or sentence Document scores range between -2 and 2 . Semantria’s cloud-based sentiment analysis software is based on Natural Language Processing and delivers you more consistent results than two humans. Using automated sentiment analysis, Semantria analyzes each document and its components based on sophisticated algorithms developed to extract sentiment from your content in a similar manner as a human – only 60,000 times faster. Existing approaches to sentiment analysis can be grouped into three main categories:  Keyword spotting  Lexical affinity  Statistical methods
  12. 12. 3 Keyword spotting is the most naive approach and probably also the most popular because of its accessibility and economy .Text is classified into affect categories based on the presence of fairly unambiguous affect words like ‘happy’, ‘sad’, ‘afraid’, and ‘bored’ .The weaknesses of this approach lie in two areas: poor recognition of affect when negation is involved and reliance on surface features .About its first weakness, while the approach can correctly classify the sentence “today was a happy day” as being happy, it is likely to fail on a sentence like “today wasn’t a happy day at all” About its second weakness, the approach relies on the presence of obvious affect words that are only surface features of the prose . In practice, a lot of sentences convey affect through underlying meaning rather than affect adjectives For example, the text “My husband just filed for divorce and he wants to take custody of my children away from me” certainly evokes strong emotions, but uses no affect keywords, and therefore, cannot be classified using a keyword spotting approach . Lexical affinity is slightly more sophisticated than keyword spotting as, rather than simply detecting obvious affect words, it assigns arbitrary words a probabilistic ‘affinity’ for a particular emotion For example, ‘accident’ might be assigned a 75% probability of being indicating a negative affect, as in ‘car accident’ or ‘hurt by accident’ These probabilities are usually trained from linguistic corpora .Though often outperforming pure keyword spotting, there are two main problems with the approach First, lexical affinity, operating solely on the word-level, can easily be tricked by sentences like “I avoided an accident” (negation) and “I met my girlfriend by accident” (other word senses) Second, lexical affinity probabilities are often biased toward text of a particular genre, dictated by the source of the linguistic corpora This makes it difficult to develop a reusable, domain-independent model . Statistical methods, such as Bayesian inference and support vector machines, have been popular for affect classification of texts .By feeding a machine learning algorithm a large training corpus of affectively annotated texts, it is possible for the system to not only learn the affective valence of affect keywords (as in the keyword spotting approach), but also to take into account the valence of other arbitrary keywords (like lexical affinity), punctuation, and word co-occurrence frequencies. However, traditional statistical methods are generally semantically weak, meaning that, with the exception of obvious affect keywords, other lexical or co-occurrence elements in a statistical model have little predictive value individually .As a result, statistical text classifiers only work with acceptable accuracy when given a sufficiently large text input .So, while these methods may be able to affectively classify user’s text on the
  13. 13. 4 page- or paragraph- level, they do not work well on smaller text units such as sentences or clauses . 1.1 Objective Sentiment classification is a way to analyze the subjective information in the text and then mine the opinion .Sentiment analysis is the procedure by which information is extracted from the opinions, appraisals and emotions of people in regards to entities, events and their attributes. In decision making, the opinions of others have a significant effect on customers ease, making choices with regards to online shopping, choosing events, products, entities . 1.2 Proposed Approach and methods to be Employed Sentiment Analysis or Opinion Mining is a study that attempts to identify and analyze emotions and subjective information from text Since early 2001, the advancement of internet technology and machine learning techniques in information retrieval make Sentiment Analysis becomes popular among researchers. Besides, the emergent of social networking and blogs as a communication medium also contributes to the development of research in this area Sentiment analysis or mining refers to the application of Natural Language Processing, Computational Linguistics, and Text Analytics to identify and extract subjective information in source materials .Sentiment mining extracts attitude of a writer in a document includes writer’s judgement and evaluation towards the discussed issue. Sentiment analysis allows us to identify the emotional state of the writer during writing, and the intended emotional effect that the author wishes to give to the reader .In recent years, sentiment analysis becomes a hotspot in numerous research fields, including natural language processing (NLP), data mining (DM) and information retrieval (IR) This is due to the increasing of subjective texts appearing on the internet .Machine Learning is commonly used to classify sentiment from text .This technique involves with statistical model such ad Support Vector Machine (SVM) , Bag of Words and Näive Bayes (NB) .The most commonly used in sentiment mining were taken from blog, twitter and web review which focusing on sentences that expressed sentiment directly .The main aim of this problem is to develop a sentiment mining model that can process the text in the mobile reviews .
  15. 15. 6 2.LITERATURE SURVEY Sentiment analysis or opinion mining is the computational study of people’s opinions, sentiments, attitudes, and emotions expressed in written language .It is one of the most active research areas in natural language processing and text mining in recent years . 2.1 Models 2.1.1 Naïve bayes A naïve bayes classifier is a simple probability based algorithm. It uses the bayes theorem but assumes that the instances are independent of each other which is an unrealistic assumption in practical world naïve bayes classifier works well in complex real world situations . The naïve bayes classifier algorithm can be trained very efficiently in supervised learning for example an insurance company which intends to promote a new policy to reduce the promotion costs the company wants to target the most likely prospects the company can collect the historical data for its customers ,including income range ,number of current insurance policies ,number of vehicles owned ,money invested ,and information on whether a customer has recently switched insurance companies .Using naïve bayes classifier the company can predict how likely a customer is to respond positively to a policy offering. With this information,the company can reduce its promotion costs by restricting the promotion to the most likely customers . The naïve bayes algorithm offers fast model building and scoring both binary and multiclass situations for relatively low volumes of data this algorithm makes prediction using bayes theorem which incorporates evidence or prior knowledge in its prediction bayes theorem relates the conditional and marginal probabilities of stochastic events H and X which is mathematically stated as
  16. 16. 7 P stands for the probability of the variables within parenthesis . P(H) is the prior probability of marginal probability of H it’s prior in the sense that it has not yet accounted for the information available in X . P(H/X) is the conditional probability of H, given X it is also called the posterior probability because it has already incorporated the outcome of event X . P(X/H) is the conditional probability of X given H . P(X) is the prior or marginal probability of X, which is normally the evidence . It can also represented as Posterior=likelihood ∗ prior normalising constant⁄ The ratio of P(X/H)/P(X) is also called as standardised likelihood . The naive Bayesian classifier works as follows: Let T be a training set of samples, each with their class labels .There are k classes [C1,C2, , Ck] Each sample is represented by an n-dimensional vector, X = {x1, x2, , xn} depicting n measured values of the n attributes [A1, A2, , An] respectively . Given a sample X, the classifier will predict that X belongs to the class having the highest a posteriori probability, conditioned on X That is X is predicted to belong to the class Ci if and only if P(Ci / X) > P(Cj /X) for 1 ≤j ≤ m, j ≠ i Thus we find the class that maximizes P(Ci/X) The class Ci for which P(Ci /X) is maximized is called the maximum posteriori hypothesis . By Baye’s theorem P(Ci/X) = (P(X/Ci)*P(Ci)) / P(X) As P(X) is the same for all classes, only P(X/Ci)* P(Ci) need be maximized If the class a priori probabilities, P(Ci), are not known, then it is commonly assumed that the classes are equally likely [P(C1) = P(C2) = = P(Ck)] and we would therefore maximize P(X/Ci) Otherwise we maximize P(X/Ci) * P(Ci) . Given data sets with many attributes, it would be computationally expensive to compute P(X/Ci). In order to reduce computation in evaluating P(X/Ci) * P(Ci), the naive assumption of class conditional independence is made This presumes that the values of the attributes are conditionally independent of one another, given the class label of the sample .
  17. 17. 8 Mathematically this means that P(X/Ci) ≈ ∏ 𝑛 𝑘=1 P(xk /Ci) The probabilities [P(x1/Ci), P(x2 /Ci) … P(xn /Ci)] can easily be estimated from the training set .Recall that here xk refers to the value of attribute Ak for sample X .If Ak is categorical, then P(xk /Ci) is the Ak number of samples of class Ci in T having the value xk for attribute , divided by freq(Ci, T), the number of sample of class Ci in T . In order to predict the class label of X, P(X/Ci)* P(Ci) is evaluated for each class Ci .The classifier predicts that the class label of X is Ci if and only if it is the class that maximizes P(X/Ci) * P(Ci) . The naïve bayes example for text classification The training set consists of 10 Positive Reviews and 10 negative reviews and considered word counts are as follows Positive Reviews Database Negative Reviews Database I = 5 I=4 Love= 20 Love=6 This= 5 This=5 Film=4 Film=3 Given test set as “I love this film” Find the sentiment for the given test set Given training set consists of the following information positive reviews =10 Negative reviews=10 Total no of Reviews=positive reviews+ negative reviews=20 Prior probability: The prior probability for the positive reviews is P(positive)=10/20=0 5 The prior probability for the negative reviews is P(negative)=10/20=0 5 Conditional probability The conditional probability is the probability that a random variable will take on a particular value given that the outcome for another random variable is known The conditional probability for the word ‘I’ in positive review is
  18. 18. 9 P(I/positive)=5/10=0 5 The conditional probability for the word ‘LOVE’ in positive review is P(Love/positive)=20/10=2 The conditional probability for the word ‘THIS’ in positive review is P(This/positive)=5/10=0 5 The conditional probability for the word ‘FILM’ in positive review is P(Film/positive)=4/10=0 4 The conditional probability for the word ‘I’ in negative review is P(I/negative)=4/10=0 4 The conditional probability for the word ‘LOVE’ in negative review is P(Love/negative)=6/10=0 6 The conditional probability for the word ‘THIS’ in negative review is P(This/negative)=5/10=0 5 The conditional probability for the word ‘FILM’ in negative review is P(Film/negative)=3/10=0 3 Posterior probability The posterior probabilities is the product of prior probability and conditional probabilities Posterior probability= prior probability *conditional probability The posterior probability for the positive review is P(positive)=0 5*0 5*0 5*0 4*2=0 1 The posterior probability for the negative review is P(negative)=0 5*0 6*0 3*0 5*0 4=0 018 The posterior probability for the positive reviews is greater than the posterior probability of the negative review P(positive)>P(negative) The given test set “I Love This Film” is predicted by naïve bayes as a positive Sentiment
  19. 19. 10 2.1.2 Bag of words In Multinomial document model a document is represented by a feature vector with integer elements whose value is the frequency of that word in the document .Text classifiers often don’t use any kind of deep representation about language often a document is represented as a bag of words (A bag is like a set that allows repeating elements ) .This is an extremely simple representation it only knows which words are included in the document (and how many time search word occurs), and throws away the word order In the multinomial document model, the document feature vectors capture the frequency of words, not just their presence or absence . Let xi be the multinomial model feature vector for the ith document Di The tth element of xi, written xit, is the count of the number of times word wt, occurs in document Di Let ni=∑ xit𝑡 be the total number of words in document Di . Let P(wt|C) again be the probability of word wt occurring in class C, this time estimated using the word frequency information from the document feature vectors .We again make the naive Bayes assumption, that the probability of each word occurring in the document is independent of the occurrences of the other words .We can then write the document likelihood P(Di |C) as a multinomial distribution where the number of draws corresponds to the length of the document, and the proportion of drawing item t is the probability of word type t occurring in a document of class C, P(wt|C) P(Di |C) ~ P(xi|C) = ( ni! ∏ xit! |v| t=1⁄ ) ∏ P(wt|C)xit |v| t=1 ∝ ∏ P(wt|C)xit |v| t=1 We often won’t need the normalisation term ( ni! ∏ xit! |v| t=1⁄ ) because it does not depend on the class, C .The numerator of the right hand side of this expression can be interpreted as the product of word likelihoods for each word in the document, with repeated words taking part for each repetition . As for the Bernoulli model, the parameters of the likelihood are the probabilities of each word given the document class P(wt|C), and the model parameters also include the prior probabilities P(C). To estimate these parameters from a training set of documents labelled with class C = k, let zik be an indicator variable which equals 1 when Di has class C=k, and equals 0 otherwise If N is again the total number of documents, then we have: P(wt|C=k) = ∑ xitzik 𝑁 𝑖=1 (∑ ∑ xiszik)𝑁 𝑖=1 |𝑣| 𝑠=1⁄
  20. 20. 11 An estimate of the probability P(wt|C=k) as the relative frequency of wt in documents of class C=k with respect to the total number of words in documents of that class . The prior probability of class C=k is estimated as P(C=k) = Nt 𝑁⁄ Thus given a training set of documents (each labelled with a class) and a set of K classes, we can estimate a multinomial text classification model as follows:  Define the vocabulary V the number of words in the vocabulary defines the dimension of the feature vectors .  Count the following in the training set: N the total number of documents . Nk the number of documents labelled with class C=k, for each class k=1,...,K . xit the frequency of word wt in document Di, computed for every word wt in V .  Estimate the priors P(C=k) .  Estimate the likelihoods P(wt | C=k) . To classify an unlabelled document Dj, we estimate the posterior probability for each class in terms of words u which occur in our document as P(C|Di )∝ P(C) ∏ p(uh/𝐶) 𝑙en(Di) h=1 Where uh is the ith word in document Di The Zero Probability Problem A drawback of relative frequency estimates for the multinomial model is that zero counts result in estimates of zero probability .This is a bad thing because the Naive Bayes equation for the likelihood involves taking a product of probabilities if any one of the terms of the product is zero, then the whole product is zero .This means that the probability of the document belonging to that particular class is zero which is impossible . Just because a word does not occur in a document class in the training data does not mean that it cannot occur in any document of that class .The problem is that equation of likelihood underestimates the likelihoods of words that do not occur in the data .Even if word w is not observed for class C=k in the training set, we would still like P(w | C=k) > 0 . Since probabilities must sum to 1, if unobserved words have underestimated probabilities, then those words that
  21. 21. 12 are observed must have overestimated probabilities .Therefore, one way to alleviate the problem is to remove a small amount of probability allocated to observed events and distribute this across the unobserved events .A simple way to do this, sometimes called Laplace’s law of succession or add one smoothing, adds a count of one to each word type .If there are W word types in total, then instead of previous likelihood formula replaced with: P(wt| C=k) = 1 + ∑ xitzik 𝑛 𝑖=1 (|v| + ∑ ∑ xiszik)𝑁 𝑖=1 |v| 𝑠=1 ⁄ P(wt| C) = 𝑐𝑜𝑢𝑛𝑡(wt, 𝑐) + 1 ∑ count(𝑤, c) + |v|𝑤∈𝑉⁄ The denominator was increased to take account of the |V| extra “observations” arising from the “add 1” term, ensuring that the probabilities are still normalised . Bag of words example on text classification Figure 1 : Example for Bag Of Words Given dataset consists of Total no of positive classes=3 Total no of negative classes=1 Total no of classes =positive classes + negative classes = 4 Prior probability: It is defined as the ratio of no of objects in that class to total no of objects Doc Words Class Training 1 Chinese Beijing Chinese c 2 Chinese Chinese Shanghai c 3 Chinese Macao c 4 Tokyo Japan Chinese j Test 5 Chinese Chinese Chinese Tokyo Japan ?
  22. 22. 13 P = 𝑁𝑐 𝑁⁄ P= (no of objects in that class/total no of objects) The prior probability for the class c is P(c) = 3/4 The prior probability for the class j is P(j) = 1/4 Conditional Probability: ||)( 1),( )|(ˆ Vccount cwcount cwP    The conditional probability for the word “Chinese” in c class is P(Chinese | c) =(5+1)/(8+6)=6/14=3/7 The conditional probability for the word “Tokyo” in c class is P(Tokyo | c) =(0+1)/(8+6)=1/14 The conditional probability for the word “Japan” in c class is P(Japan | c) =(0+1)/(8+6)=1/14 The conditional probability for the word “Chinese” in c class is P(Chinese | c) =(1+1)/(3+6)=2/9 The conditional probability for the word “Tokyo” in c class is P(Tokyo | j) =(1+1)/(3+6)=2/9 The conditional probability for the word “Japan” in c class is P(Japan | j) =(1+1)/(3+6)=2/9 Posterior probability The posterior probability for the class c is P(c) = 3/4*3/7*3/7*3/7*1/14*1/14 = 0 0003 The posterior probability for the class j is P(j) = 1/4*2/9*2/9*2/9*2/9*2/9 = 0 0001
  23. 23. 14 The posterior probability of the class c is greater than the posterior probability of the class j P(c) > P(j) Hence the given test data belongs to Class c 2.1.3 Support vector machine Support vector machines (SVMs) are a set of related supervised learning methods used for classification and regression they belong to a family of generalized linear classifiers .In another terms, Support Vector Machine (SVM) is a classification and regression prediction tool that uses machine learning theory to maximize predictive accuracy while automatically avoiding over-fit to the data .Support Vector machines can be defined as systems which use hypothesis space of a linear functions in a high dimensional feature space, trained with a learning algorithm from optimization theory that implements a learning bias derived from statistical learning theory. Support vector machine was initially popular with the NIPS community and now is an active part of the machine learning research around the world. SVM becomes famous when, using pixel maps as input; it gives accuracy comparable to sophisticated neural networks with elaborated features in a handwriting recognition task It is also being used for many applications, such as hand writing analysis, face analysis and so forth, especially for pattern classification and regression based applications .The foundations of Support Vector Machines (SVM) has gained popularity due to many promising features such as better empirical performance .The formulation uses the Structural Risk Minimization (SRM) principle, which has been shown to be superior to traditional Empirical Risk Minimization (ERM) principle, used by conventional neural networks.SRM minimizes an upper bound on the expected risk,where as ERM minimizes the error on the training data .It is this difference which equips SVM with a greater ability to generalize, which is the goal in statistical learning .SVMs were developed to solve the classification problem, but recently they have been extended to solve regression problems. A support vector machine (SVM) is preferred when data has exactly two classes .An SVM classifies data by finding the best hyperplane that separates all data points of one class from those of the other class.The best hyperplane for an SVM means the one with the largest margin between the two classes .Margin means the maximal width of the slab parallel to the hyperplane that has no interior data points .The support vectors are the data points that are closest to the separating hyperplane; these points are on the boundary of the slab
  24. 24. 15 Figure 2: Hyperplane separating two classes Assume, there is a new company j which has to be classified as solvent or insolvent according to the SVM score. In the case of a linear SVM the score looks like a DA or Logit score, which is a linear combination of relevant financial ratios xj= (xj1, xj2, …xjd), where xjis a vector with d financial ratios and 𝑥𝑗𝑘is the value of the financial ratio number k for company j ( k=1,…,d) So 𝑧𝑗 the score of company j, can be expressed as: 𝑧𝑗=𝑥𝑗 𝑇 +b Where w is a vector which contains the weights of the d financial ratios and b is a constant The comparison of the score with a benchmark value (which is equal to zero for a balanced sample) delivers the “forecast” of the class – solvent or insolvent – for company j . To use this decision rule for the classification of company j, the SVM has to learn the values of the score parameters w and b on a training sample. Assume this consists of a set of n companies (i=1, 2, … , n) .From a geometric point of view, calculating the value of the parameters w and b means looking for a hyperplane that best separates solvent from insolvent companies according to some criterion . The criterion used by SVMs is based on margin maximization between the two data classes of solvent and insolvent companies .The margin is the distance between the hyper planes bounding each class, where in the hypothetical perfectly separable case no observation may lie. By maximizing the margin, we search for the classification function that can most safely separate the classes of solvent and insolvent companies .The graph below represents a binary space with two input variables .Here crosses represent the solvent companies of the training sample and circles the insolvent ones .The threshold separating solvent and insolvent companies is the line in the middle between the two margin boundaries, which are canonically
  25. 25. 16 represented as xT w+b=1 and xT w+b=-1 .Then the margin is 2 / ||w|| where ||w|| is the norm of the vector w . In a non-perfectly separable case the margin is “soft” .This means that in-sample classification errors occur and also have to be minimized .Let 𝜀𝑖 be a non-negative slack variable for in- sample misclassifications . In most cases 𝜀𝑖=0, that means companies are being correctly classified .In the case of a positive 𝜀𝑖 the company i of the training sample is being misclassified .A further criterion used by SVMs for calculating w and b is that all misclassifications of the training sample have to be minimized . Let 𝑦𝑖 be an indicator of the state of the company, where in the case of solvency 𝑦𝑖 =-1 and in the case of insolvency 𝑦𝑖 =1 . By imposing the constraint that no observation may lie within the margin except some classification errors, SVMs require that either xi T w + b ≥ 1- εi or xi T w + b ≤ -1+ εi Figure 3: Geometrical Representation of the SVM Margin The optimization problem for the calculation of w and b can thus be expressed by: minw 1 2 ||𝑤||2 +C∑ 𝑛 𝑖=1 εi xi T w + b ≥ -1+ εi We maximize the margin 2 ⁄ ||𝑤|| by minimizing ||w||2/ 2, where the square in the form of w comes from the second term, which originally is the sum of in-sample misclassification errors 2 ⁄ ||𝑤|| times the parameter C .Thus SVMs maximize the margin width while minimizing
  26. 26. 17 errors .This problem is quadratic i e convex C = “capacity” is a tuning parameter, which weights in-sample classification errors and thus controls the generalization ability of an SVM The higher is C, the higher is the weight given to in-sample misclassifications , the lower is the generalization of the machine . Low generalization means that the machine may work well on the training set but would perform miserably on a new sample .Bad generalization may be a result of overfitting on the training sample, for example, in the case that this sample shows some untypical and non-repeating data structure .By choosing a low C, the risk of overfitting an SVM on the training sample is reduced .It can be demonstrated that C is linked to the width of the margin .The smaller is C, the wider is the margin, the more and larger in-sample classification errors are permitted . Solving the above mentioned constrained optimization problem of calibrating an SVM means searching for the minimum Lagrange function and considering 𝛼𝑖≥ 0 are the Lagrange multipliers for the inequality constraint and 𝑣𝑖≥0 are the Lagrange multipliers for the condition 𝜀𝑖 ≥0 .This is a convex optimization problem with inequality constraints, which is solved my means of classical non-linear programming tools and the application of the Kuhn-Tucker Sufficiency Theorem .The solution of this optimisation problem is given by the saddle-point of the Lagrangian, minimized with respect to w, b, and 𝜀 and maximized with respect to α and ν. The entire task can be reduced to a convex quadratic programming problem in 𝛼𝑖 .Thus, by calculating 𝛼𝑖 we solve our classifier construction problem and are able to calculate the parameters of the linear SVM model using the formulas w = ∑ 𝑛 𝑖=1 𝑦𝑖 𝛼𝑖 𝑥𝑖 b = (𝑥+1 𝑇 , 𝑥−1 𝑇 )/2 𝛼𝑖 must be non-negative, weighs different companies of the training sample The companies, whose 𝛼𝑖 are not equal to zero, are called support vectors and are the relevant ones for the calculation of w .Support vectors lie on the margin boundaries or, for non-perfectly separable data, within the margin .By this way, the complexity of calculations does not depend on the dimension of the input space but on the number of support vectors .Here x+1 and x-1 are any two support vectors belonging to different classes, which lie on the margin boundaries. After simplifying the equations we obtain the score 𝑧𝑗 as a function of the scalar product of the financial ratios of the company to be classified and the financial ratios of the support vectors in the training sample, of 𝛼𝑖 and of 𝑦𝑖 .By comparing 𝑧𝑗 with a benchmark value, we are able to estimate if a company has to be classified as solvent or insolvent
  27. 27. 18 𝑧𝑗 = ∑ 𝑛 𝑖=1 𝑦𝑖 𝛼𝑖(𝑥𝑖,𝑥𝑗)+b the support vector machine returns one class as output which is our result . Example : Given 3 support vectors as 𝑆1= 2 1 𝑆2= 2 −1 𝑆3= 4 0 Where 𝑆1 and 𝑆2 belongs to Negative class , 𝑆3 belongs to Positive class . Find 𝑆4= 6 2 belongs to which class either positive class or negative class using support vector machine Given 3 support vectors as 𝑆1, 𝑆2, 𝑆3 where 𝑆1 , 𝑆2 belongs to Negative class and 𝑆3 belongs to Positive class. 𝑆1= 2 1 𝑆2= 2 −1 𝑆3= 4 0 A graph is plotted for these 3 support vectors Figure 4: A graph plotted with 3 support vectors we will use vectors augmented with a 1 as a bias input .we will differentiate these with an over- tilde.
  28. 28. 19 𝑆1= 2 1 𝑆̃1= 2 1 1 𝑆2= 2 −1 𝑆̃2= 2 −1 1 𝑆3= 4 0 𝑆̃3= 4 0 1 Find the 3 parameters α1, α2, α3 from the 3 linear equations Substitute the values of 𝑆̃1, 𝑆̃2 , 𝑆̃3 in the above 3 Linear equations 𝑆̃1= 2 1 1 𝑆̃2= 2 −1 1 𝑆̃3= 4 0 1 After substituting the values of 𝑆̃1, 𝑆̃2 , 𝑆̃3 in 3 linear equations and simplifying it reduces into By solving the above 3 equations we will get the α1, α2, α3 values
  29. 29. 20 α1= -3 25 α2= -3 25 α3= 3 50 The Hyperplane that seperates the positive class from negative class is given by formula 𝑤̃=∑ ∝𝑖 𝑆̃𝑖𝑖 Now substituting the values of Ŝ1, Ŝ2 ,Ŝ3 in the above formula After simplifying Our vectors are augmented with a bias Hence we can equate the entry in as the hyper plane with an offset b The separating hyper plane equation 𝑦 = 𝑤𝑥 + 𝑏 with w= 1 0 and offset 𝑏 = −3 Hence the positive class is seperated from the negative class at offset b=-3 . b=3 Figure 5 : Hyperplane separating two different classes with an intercept
  30. 30. 21 Given S4 = 6 2 for finding the class for this new support vector we use the formula = w x in this case if w x > offset positive class w x < offset negative class we know w = 1 0 and x= 6 2 and offset=3 w x = 1 0 * 6 2 =6 w x > offset = 6 > 3 hence support vector machine classifies this newly added point belongs to the positive class. 2.1.4 Principle component analysis PCA is a dimensionality reduction method in which a covariance analysis between factors takes place .The original data is remapped into a new coordinate system based on the variance within the data PCA applies a mathematical procedure for transforming a number of (possibly) correlated variables into a (smaller) number of uncorrelated variables called principal components .The first principal component accounts for as much of the variability in the data as possible, and each succeeding component accounts for as much of the remaining variability as possible . PCA is useful when there is data on a large number of variables, and (possibly) there is some redundancy in those variables .In this case, redundancy means that some of the variables are correlated with one another and because of this redundancy, PCA can be used to reduce the observed variables into a smaller number of principal components that will account for most of the variance in the observed variables. PCA is recommended as an exploratory tool to uncover unknown trends in the data .The technique has found application in fields such as face recognition and image compression, and is a common technique for finding patterns in data of high dimension. For a given data set with n no of observations .Each observation consists of x variants for calculating the principal components the PCA algorithm follows the following 5 main steps Subtract the mean from each of the data dimensions: The mean subtracted is the average across each dimension .This produces a data set whose mean is zero Mean 𝑥̅=∑ 𝑥𝑖 𝑛 𝑖=1 𝑛⁄
  31. 31. 22 Calculate the covariance matrix: Variance and Covariance are a measure of the spread of a set of points around their center of mass (mean).Variance is the measure of the deviation from the mean for points in one dimension .Variance is calculated with the formula 𝜎2 =∑(𝑥 − 𝑥̅)2 𝑛 − 1⁄ Covariance as a measure of how much each of the dimensions vary from the mean with respect to each other .Covariance is measured between 2 dimensions to see if there is a relationship between the 2 dimensions .The covariance between one dimension and itself is the variance . For example a 3-dimensional data set (x,y,z), then you could measure the covariance between the x and y dimensions, the y and z dimensions, and the x and z dimensions .Measuring the covariance between x and x , or y and y , or z and z would give you the variance of the x , y and z dimensions respectively C=[ 𝑣𝑎𝑟(𝑥, 𝑥) 𝑐𝑜𝑣(𝑥, 𝑦) 𝑐𝑜𝑣(𝑥, 𝑧) 𝑐𝑜𝑣(𝑦, 𝑥) 𝑣𝑎𝑟(𝑦, 𝑦) 𝑐𝑜𝑣(𝑦, 𝑧) 𝑐𝑜𝑣(𝑧, 𝑥) 𝑐𝑜𝑣(𝑧, 𝑦) 𝑣𝑎𝑟(𝑧, 𝑧) ] Diagonal is the variances of x, y and z cov(x,y) = cov(y,x) hence matrix is symmetrical about the diagonal For a N-dimensional data will result in NxN covariance matrix Exact value is not as important as it’s sign A positive value of covariance indicates both dimensions increase or decrease together example as the number of hours studied increases, the marks in that subject increase.A negative value indicates while one increases the other decreases, or vice-versa If covariance is zero: the two dimensions are independent of each other. Calculate the eigenvectors and eigenvalues of the covariance matrix: The covariance matrix is a square, the calculation of eigenvectors and eigenvalues are possible for this matrix. These are rather important, as they tell us useful information about our data Eigen values are calculated with the formula |𝐶 − 𝜆𝐼| = 0 Where I is the identity matrix Eigen vectors are calculated with the formula [𝐶 − 𝜆𝐼][𝑘] = 0
  32. 32. 23 It is important to notice that these eigenvectors are both unit eigenvectors ie Their lengths are both 1 which is very important for PCA . Choose components and form a feature vector: The eigenvectors and eigenvalues obtained from the covariance matrix will have quite different values .In fact, it turns out that the eigenvector with the highest eigenvalue is the principle component of the data set . In general, once eigenvectors are found from the covariance matrix, the next step is to order them by eigenvalue, highest to lowest This gives you the components in order of significance Now, if you like, you can decide to ignore the components of lesser significance .You do lose some information, but if the eigenvalues are small, you don’t lose much If you leave out some components, the final data set will have less n dimensions than the original .To be precise, if you originally have dimensions in your data, and so you calculate n eigenvectors and eigenvalues, and then you choose only the first p eigenvectors, then the final data set has only p dimensions . To form a feature vector, which is just a fancy name for a matrix of vectors . This is constructed by taking the eigenvectors that you want to keep from the list of eigenvectors, and forming a matrix with these eigenvectors in the columns FeatureVector = (eigenvector1, eigenvector2, ,eigenvector 𝑛) Deriving the new data set: FinalData = RowFeatureVector x RowDataAdjusted Where RowFeatureVector is the matrix with the eigenvectors in the columns transposed so the eigenvectors are now in the rows and the most significant are in the top. RowDataAdjusted is the mean-adjusted data transposed i.e the data items are in each column, with each row holding a separate dimension. It will give us the original data solely in terms of the vectors we chose . Principle component analysis example
  33. 33. 24 Table 1 : Principle Component Analysis Data Set Given a two dimensional data as x1,x2 Dimensionality reduction using Principle component analysis can be done in 5 steps:  To obtain covariance matrix .  To obtain eigen values .  To obtain eigen vectors .  To obtain coordinates of data point in the direction of eigen vectors . Covariance matrix: The covariance matrix is obtained by using the formula = Cov(x1, x1) Cov(x1, x2) Cov(x2, x1) Cov(x2, x2) X1 X2 1 4000 1 6000 -1 4000 -2 0000 -3 0000 2 4000 1 5000 2 3000 -3 2000 -4 1000 1 6500 1 9700 -1 7750 -2 5250 -3 9500 3 0750 2 0250 2 7500 -4 0500 -4 8500
  34. 34. 25 Table 2 : Principle Component Analysis Mean and Variance calculation Cov(x1,x1)=6 4228 Cov(x1,x2)=7 9876 Cov(x2,x1)=7 9876 Cov(x2,x2)=9 9528 Covariance matrix = 6 4228 7 9876 7 9876 9 9528 Eigen values The eigen values are calculated by using the formula |Covariance matrix-λI| = 0 Where Covariance matrix = 6 4228 7 9876 7 9876 9 9528 Identity matrix I = 1 0 0 1 After substituting the covariance matrix and identity matrix in the formula 6 4228 7 9876 7 9876 9 9528 - λ 1 0 0 1 = 0 X1 X2 C= X1-X1bar D= X2-X2bar C*D 1 4000 1 6500 1 8500 2 2175 4 1024 1 6000 1 9750 2 0500 2 5425 5 3121 -1 4000 -1 7750 -0 9500 -1 2075 1 1471 -2 0000 -2 5250 -1 5500 -1 9575 3 0341 -3 0000 -3 9500 -2 5500 -3 3825 8 6254 2 4000 3 0750 2 8500 3 6425 10 3811 1 5000 2 0250 1 9500 2 5925 5 0554 2 3000 2 7500 2 7500 3 3175 9 1231 -3 2000 -4 0500 -2 7500 -3 4825 9 5769 -4 1000 -4 8500 -3 6500 -4 2825 15 6311 A=-0 4500 B= -0 5675
  35. 35. 26 After simplifying we get Eigen values as λ1= 16 36809984 λ2= 0 007462657 Eigen vector The eigen vectors for the given two dimension data set are obtained by using the formula ( A − λ I) x = 0 If we consider λ =16 36809984 ( A − λ I) = 6 4228 7 9876 7 9876 9 9528 - ( 16 36809984) 1 0 0 1 = −9 9453 7 9876 7 9876 −6 4153 ( A − λ I) x = 0 = −9 9453 7 9876 7 9876 −6 4153 * 𝑎 𝑏 =0 By using row reduction and simplifying we get eigen vector as Eigen vector = 0 6262 0 7797 If we consider λ = 0 007462657 ( A − λ I) = 6 4228 7 9876 7 9876 9 9528 - (0 007462657) 1 0 0 1 = 6 4153 7 9876 7 9876 8 9925 ( A − λ I) x = 0 = 6 4153 7 9876 7 9876 8 9925 * 𝑎 𝑏 =0 By using row reduction and simplifying we get eigen vector as Eigen vector= 0 7797 −0 6262 coordinates of data point in the direction of eigen vectors This is obtained by multiplying centered data matrix to the eigen vector matrix Eigen vector matrix = 0 6262 0 7797 0 7797 −0 6262
  36. 36. 27 Table 3 : Centered data matrix 0 6262 0 7797 0 7797 −0 6262 = variance Table 4 : Principle Component Projection Calculation X1-X1bar X2-X2bar 1 8500 2 2175 2 0500 2 5425 -0 9500 -1 2075 -1 5500 -1 9575 -2 5500 -3 3825 2 8500 3 6425 1 9500 2 5925 2 7500 3 3175 -2 7500 -3 4825 -3 6500 -4 2825 Projection on the line of 1st principle component Projection on the line of 2nd principle component 2 88737 0 05380 3 26600 00622 -1 53633 0 01545 -2 49680 0 01729 -4 23402 0 12995 4 62459 -0 05886 3 2237 -0 10306 4 30858 0 06669 -4 43722 0 03664 -5 62453 -0 16411 16 3680977 0 007462657 X1-X1bar X2-X2bar 1 8500 2 2175 2 0500 2 5425 -0 9500 -1 2075 -1 5500 -1 9575 -2 5500 -3 3825 2 8500 3 6425 1 9500 2 5925 2 7500 3 3175 -2 7500 -3 4825 -3 6500 -4 2825
  37. 37. 28 The variance of projections on the line of first principle component is 16 36809775 . The variance of projections on the line of first principle component is 0 007462657 . The variances of projections in the line of principle component is equal to the eigen values of the principle components .First eigen vector is able to explain around 99% of total variance .
  39. 39. 30 3.SYSTEM ANALYSIS AND DESIGN 3.1 Software and Hardware Requirements Software Requirements  Operating System : windows 7, windows vista, windows xp,windows 8 and higher versions  Language : MATLAB 2013a Hardware requirements  Ram : 1 GB Ram and more  Processor : Any Intel Processor  HardDisk : 6 GB and more  Speed : 1GHZ and more 3.2 MATLAB Technology MATLAB is a high-performance language for technical computing .It integrates computation, visualization, and programming in an easy-to-use environment where problems and solutions are expressed in familiar mathematical notation .Typical uses include:  Math and computation  Algorithm development  Modeling, simulation, and prototyping  Data analysis, exploration, and visualization  Scientific and engineering graphics  Application development, including Graphical User Interface building MATLAB is an interactive system whose basic data element is an array that does not require dimensioning .This allows you to solve many technical computing problems, especially those with matrix and vector formulations, in a fraction of the time it would take to write a program in a scalar noninteractive language such as C or Fortran. The name MATLAB stands for matrix laboratory .MATLAB was originally written to provide easy access to matrix software developed by the LINPACK and EISPACK projects, which together represent the state-of-the-art in software for matrix computation. MATLAB has evolved over a period of years with input from many users .In university environments, it is the standard instructional tool for introductory and advanced courses in
  40. 40. 31 mathematics, engineering, and science .In industry, MATLAB is the tool of choice for high- productivity research, development, and analysis. MATLAB features a family of application-specific solutions called toolboxes .Very important to most users of MATLAB, toolboxes allow you to learn and apply specialized technology. Toolboxes are comprehensive collections of MATLAB functions (M-files) that extend the MATLAB environment to solve particular classes of problems Areas in which toolboxes are available include signal processing, control systems, neural networks, fuzzy logic, wavelets, simulation, and many others . The MATLAB system consists of five main parts: The MATLAB language This is a high-level matrix/array language with control flow statements, functions, data structures,input/output, and object-oriented programming features.It allows both "programming in the small" to rapidly create quick and dirty throw-away programs, and "programming in the large" to create complete large and complex application programs . The MATLAB working environment This is the set of tools and facilities that you work with as the MATLAB user or programmer. It includes facilities for managing the variables in your workspace and importing and exporting data .It also includes tools for developing, managing, debugging, and profiling M-files, MATLAB's applications. Handle Graphics This is the MATLAB graphics system .It includes high-level commands for two-dimensional and three-dimensional data visualization, image processing, animation, and presentation graphics .It also includes low-level commands that allow you to fully customize the appearance of graphics as well as to build complete Graphical User Interfaces on your MATLAB applications . The MATLAB mathematical function library This is a vast collection of computational algorithms ranging from elementary functions like sum, sine, cosine, and complex arithmetic, to more sophisticated functions like matrix inverse, matrix eigenvalues, Bessel functions, and fast Fourier transforms.
  41. 41. 32 The MATLAB Application Program Interface (API) This is a library that allows you to write C and Fortran programs that interact with MATLAB. It include facilities for calling routines from MATLAB (dynamic linking), calling MATLAB as a computational engine, and for reading and writing MAT-files. The main components in the MATLAB are  Command Window  Command History  Workspace  Current Directory Figure 6: MATLAB Interfaces 3.3 Dataflow Diagrams A data flow diagram (DFD) is a graphical representation of the "flow" of data through an information system, modeling its process aspects .A DFD is often used as a preliminary step to create an overview of the system, which can later be elaborated.  Process: A process takes data as input, execute some steps and produce data as output.  External Entity: Objects outside the system being modeled, and interact with processes in system.  Data Store: Files or storage of data that store data input and output from process.
  42. 42. 33  Data Flow: The flow of data from process to process. Data flow diagram : Figure 7 : Level 0 and Level 1 Data Flow Diagram
  43. 43. 34 Figure 8: Level 2 Data Flow Diagram for the process 1 (Naïve Bayes)
  44. 44. 35 Figure 9: Level 2 Data Flow Diagram for the process 2 (Bag of Words)
  45. 45. 36 Figure 11: Level 2 Data Flow Diagram for the process 3 (Support Vector Machine)
  47. 47. 38 4.IMPLEMENTATION We used MATLAB Technology for the implementation of “sentiment analysis of mobile reviews using supervised learning techniques”. There are several stages involved during implementation of our problem using different supervised learning techniques. Among them training and testing are the two main phases that are involved . 4.1 Elimination of Special Characters and Conversion to Lower Case The given Dataset of positive and negative reviews are passed through the elimination of special Characters and Conversion to lower case part .This stage eliminates all the special Characters from the reviews and converts all the upper case to lower case . The implementation is while ischar(line) temp_line1=lower(line); temp_line2=strrep(temp_line1,' ',''); temp_line3=strrep(temp_line2,',',''); temp_line4=strrep(temp_line3,';',''); temp_line5=strrep(temp_line4,':',''); temp_line6=strrep(temp_line5,'"',''); temp_line7=strrep(temp_line6,'',''); temp_line8=strrep(temp_line7,')',''); temp_line9=strrep(temp_line8,'(',''); temp_line10=strrep(temp_line9,'?',''); temp_line11=strrep(temp_line10,'!',''); end 4.2 Word Count In this stage the number of occurences of each distinct word in the data set of both positive and negative reviews is calculated .These word counts are considered as the values for the attributes in Naïve Bayes and Bag of Words . The implementation is for i=1:Length(DataSet) Value 1=DataSet(i,1);
  48. 48. 39 Count=0; for j=i+1:Length(DataSet) Value 2= DataSet(j,1); If Strcmp(Value 1, Value 2) Count=count+1; end end for k=1:Length(DataSetCount) Value 3=DataSetCount(k,1); if strcmp(Value 1, Value 3) else DataSetCount(k,1)=Value 1; DataSetCount(k,1)=count; K=k+1; end end end 4.3 Training And Testing 4.3.1 Naïve Bayes : This method mainly concentrates on the attributes Training Phase involves the Elimination of Special Characters and Conversion to Lower case and Word Count stages are used in this method .The DataSet obtained from the WordCount is passed to another stage where all the neutral words are eliminated by using Positive and Negative words .Finally the obtained DataSet is given as a input to the Naïve Bayes method and along with that Sentiment polarities are calculated carefully and given as input. In the testing phase the test data is passed through the Elimination of Special Characters and Conversion to Lower case stage .The prior,conditional and posterior probabilities are calculated using the input data . 4.3.2 Bag of Words : The Elimination of Special Characters and Conversion to Lower case and Word Count stages are used in this method .The DataSet obtained from the WordCount is passed to
  49. 49. 40 another stage where all the neutral words are eliminated by using Positive and Negative words. Finally the obtained DataSet is given as a input to the Bag of Words method . In Testing Phase, the Special characters and upper case letters are eliminated from the test data . In this method to eliminate the Zero probability problem each word is considered as repeated once .The prior,conditional and posterior probabilities are calculated using the input data . 4.3.3 Support Vector Machine : The Support Vector Machine is implemented along with the Principal Component Analysis .In this method , a dictionary with both positive and negative words are maintained .By applying window concept, the word count for every pair of words (in the dictionary) are calculated from positive and negative reviews .The obtained DataSet is passed to Principal Component Analysis where it reduces the dimensionality without any loss of data PCA Cofficient data is used to train the support Vector Machine . In testing phase, the word count is calculated for the test data and passed as an input to Principle Component Analysis where it reduces the dimensionality and gives the Cofficient matrix. This coefficient matrix is then compared with trained Support Vector Machine (struct) where it returns one class as an output . 4.4 Sample Code Bag of Words format short g ; distinctwords={}; positivereviewwords={}; negativereviewwords={}; [~,~,raw]=xlsread('positivereviewwords.xlsx'); positivereviewwords=raw; distinctwords=positivereviewwords; po_size=size(positivereviewwords); po_size1=po_size(1); display('total no of positive words with no repetition'); display(po_size1);
  50. 50. 41 cal_postivewordscount=0; for lop=1:po_size1 po_value=positivereviewwords(lop,2); po_value1=cell2mat(po_value); cal_postivewordscount=cal_postivewordscount+po_value1; end display(' total no of positive words with repetition '); display(cal_postivewordscount); [~,~,raw1]=xlsread('negativereviewwords.xlsx'); negativereviewwords=raw1; ne_size=size(negativereviewwords); ne_size1=ne_size(1); display('total no of negative words with no repetition'); display(ne_size1); cal_negativewordscount=0; for lon=1:ne_size1 ne_value=negativereviewwords(lon,2); ne_value1=cell2mat(ne_value); cal_negativewordscount=cal_negativewordscount+ne_value1; end display('total no of negative words with repetition '); display(cal_negativewordscount); lowt=po_size1+1; for tyew=1:ne_size1 qwe1= negativereviewwords(tyew,1); qwe2= negativereviewwords(tyew,2); distinctwords(lowt,1)=qwe1; distinctwords(lowt,2)=qwe2; lowt=lowt+1; end distinct_size=size(distinctwords);
  51. 51. 42 distinct_size1=distinct_size(1); distinctwordset={}; gfd=1; for utr=1:distinct_size1 distinct_word=distinctwords(utr,1); if utr==1 distinctwordset(gfd,1)=distinct_word; gfd=gfd+1; else distinctwrdset_size=size(distinctwordset); distinctwrdset_size1=distinctwrdset_size(1); jfd=0; for hur=1:distinctwrdset_size1 distinct_word1=distinctwordset(hur,1); if strcmp(distinct_word1,distinct_word) jfd=1; break; end end if jfd==0 distinctwordset(gfd,1)=distinct_word; gfd=gfd+1; end end end display('distinct words i e v is'); dwq=size(distinctwordset); dwq1=dwq(1); display(dwq1); [positivereviews,~,~]=xlsread('positivereviewcount.xlsx'); [negativereviews,~,~]=xlsread('negativereviewcount.xlsx');
  52. 52. 43 totalreviews=positivereviews+negativereviews; piy1=positivereviews; piy2=negativereviews; positivepriorprobability=piy1/totalreviews; negativepriorprobability=piy2/totalreviews; display(totalreviews); display(positivepriorprobability); display(negativepriorprobability); testdatawords={}; mi=''; ts=''; bt=1; ky=0; mt=0; op1=fopen('testfile.txt','r'); file_pointer=fgets(op1); while ischar(file_pointer) file_pointer1=strtrim(file_pointer); line_size=size(file_pointer1); line_size1=line_size(2); if line_size1==0 ts=''; mt=0; end for ta=1:line_size1 if ta==1 mi=''; ts=''; fed=''; ky=0; mt=0; end fed=file_pointer1(ta);
  53. 53. 44 if strcmp(fed,' ') if strcmp(edr,' ') ky=ky+1; else ts1=cellstr(ts); testdatawords(bt,1)=ts1; bt=bt+1; edr=fed; ky=ky+1; mi=''; end else ts=strcat(mi,fed); mi=ts; edr=fed; ky=ky+1; if ky==line_size1 mt=1; break; end end end if mt==1 ts1=cellstr(ts); testdatawords(bt,1)=ts1; bt=bt+1; end file_pointer=fgets(op1); end display('test data with repeated words'); display(testdatawords); positivewordscountvalues={}; poqa=size(testdatawords);
  54. 54. 45 poqa1=poqa(1); hjo=1; for poty=1:poqa1 testdata_wordvalue=testdatawords(poty,1); loi=0; for koty=1:po_size1 testdata_wordvalue1=positivereviewwords(koty,1); if strcmp(testdata_wordvalue,testdata_wordvalue1) testdata_word2value=positivereviewwords(koty,2); testdata_word2value1=cell2mat(testdata_word2value); povalue=testdata_word2value1+1; povalue1=num2cell(povalue); positivewordscountvalues(hjo,1)=testdata_wordvalue; positivewordscountvalues(hjo,2)=povalue1; hjo=hjo+1; loi=1; break; end end if loi==0 positivewordscountvalues(hjo,1)=testdata_wordvalue; povalue1=num2cell(1); positivewordscountvalues(hjo,2)=povalue1; hjo=hjo+1; end end display('After eliminating zero probability the dataset will be); display(positivewordscountvalues); negativewordscountvalues={}; neqa=size(testdatawords); neqa1=neqa(1); hjo=1; for nety=1:neqa1
  55. 55. 46 testdata_wordvalue=testdatawords(nety,1); loi=0; for koty=1:ne_size1 testdata_wordvalue1=negativereviewwords(koty,1); if strcmp(testdata_wordvalue,testdata_wordvalue1) testdata_word2value=negativereviewwords(koty,2); testdata_word2value1=cell2mat(testdata_word2value); povalue=testdata_word2value1+1; povalue1=num2cell(povalue); negativewordscountvalues(hjo,1)=testdata_wordvalue; negativewordscountvalues(hjo,2)=povalue1; hjo=hjo+1; loi=1; break; end end if loi==0 negativewordscountvalues(hjo,1)=testdata_wordvalue; povalue1=num2cell(1); negativewordscountvalues(hjo,2)=povalue1; hjo=hjo+1; end end display('after eliminating zero probability the word data set is’); display(negativewordscountvalues); negativeconditionalprobability=1; negativewrdset_size=size(negativewordscountvalues); divn=cal_negativewordscount+dwq1; display(divn); negativewrdset_size1=negativewrdset_size(1); for ted=1:negativewrdset_size1 nbgc=negativewordscountvalues(ted,2); nbgc1=cell2mat(nbgc);
  56. 56. 47 negativeconditionalprobability=negativeconditionalprobability*(nbgc1/divn); end negativeposteriorprobability=negativepriorprobability*negativeconditionalprobability; display(‘posterior probability for negative class is’); display(negativeposteriorprobability); positiveconditionalprobability=1; positivewrdset_size=size(positivewordscountvalues); divp=cal_postivewordscount+dwq1; display(divp); positivewrdset_size1=positivewrdset_size(1); for ted1=1:positivewrdset_size1 pbgc=positivewordscountvalues(ted1,2); pbgc1=cell2mat(pbgc); positiveconditionalprobability=positiveconditionalprobability*(pbgc1/divp); end positiveposteriorprobability=positivepriorprobability*positiveconditionalprobability; display(‘posterior probability for positive class is’); display(positiveposteriorprobability); if positiveposteriorprobability>negativeposteriorprobability display('It is a positive review'); elseif negativeposteriorprobability>positiveposteriorprobability display('It is a negative review'); else display(' neutral '); end
  57. 57. 48 CHAPTER 5 TESTING
  58. 58. 49 5.TESTING Testing is the process of evaluating a system or its component’s with the intent to find that whether it satisfies the specified requirements or not .This activity results in the actual, expected and difference between their results i.e testing is executing a system in order to identify any gaps, errors or missing requirements in contrary to the actual desire or requirements. 5.1 Testing Strategies In order to make sure that system does not have any errors, the different levels of testing strategies that are applied at different phases of software development are Figure 11 : Phases of Software Development 5.1.1 Unit Testing The goal of unit testing is to isolate each part of the program and show that individual parts are correct in terms of requirements and functionality. 5.1.2 Integration Testing The testing of combined parts of an application to determine if they function correctly together is Integration testing .This testing can be done by using two different methods Top Down Integration testing
  59. 59. 50 In Top-Down integration testing, the highest-level modules are tested first and then progressively lower-level modules are tested. 5 .1.2.2 Bottom-up Integration testing Testing can be performed starting from smallest and lowest level modules and proceeding one at a time .When bottom level modules are tested attention turns to those on the next level that use the lower level ones they are tested individually and then linked with the previously examined lower level modules.In a comprehensive software development environment, bottom-up testing is usually done first, followed by top-down testing. 5.1.3 System Testing This is the next level in the testing and tests the system as a whole .Once all the components are integrated, the application as a whole is tested rigorously to see that it meets Quality Standards. 5.1.4 Acceptance Testing The main purpose of this Testing is to find whether application meets the intended specifications and satisfies the client’s requirements .We will follow two different methods in this testing. Alpha Testing This test is the first stage of testing and will be performed amongst the teams .Unit testing, integration testing and system testing when combined are known as alpha testing. During this phase, the following will be tested in the application:  Spelling Mistakes.  Broken Links.  The Application will be tested on machines with the lowest specification to test loading times and any latency problems. Beta Testing In beta testing, a sample of the intended audience tests the application and send their feedback to the project team .Getting the feedback, the project team can fix the problems before releasing the software to the actual users.
  60. 60. 51 5.2 Testing Methods 5.2.1 White Box Testing White box testing is the detailed investigation of internal logic and structure of the Code. To perform white box testing on an application, the tester needs to possess knowledge of the internal working of the code .The tester needs to have a look inside the source code and find out which unit/chunk of the code is behaving inappropriately. 5.2.2 Black Box Testing The technique of testing without having any knowledge of the interior workings of the application is Black Box testing .The tester is oblivious to the system architecture and does not have access to the source code.Typically, when performing a black box test, a tester will interact with the system’s user interface by providing inputs and examining outputs without knowing how and where the inputs are worked upon. 5.3 Validation All the levels in the testing (unit,integration,system) and methods (black box,white box)are implemented on our application successfully and the results obtained as expected . 5.4 Limitations The execution time for support vector machine is more so that the user may not receive the result fast. 5 5 Test Results The testing is done among the team members and by the end users. It satisfies the specified requirements and finally we obtained the results as expected.
  62. 62. 53 6.SCREEN SHOTS 6.1 Naïve Bayes Figure 12 : Naïve Bayes Main Screen for Input
  63. 63. 54 Figure 13 : Naïve Bayes Screen with Test Data
  64. 64. 55 Figure 14 : Naïve Bayes Screen with output
  65. 65. 56 6.2 Bag of words Figure 15 : Bag Of Words Main Screen for Input
  66. 66. 57 Figure 16 : Bag Of Words screen with Test Data
  67. 67. 58 Figure 17 : Bag Of Words screen with output
  68. 68. 59 6.3 Support Vector Machine using Principle Component Analysis Figure 18 : Support Vector Machine Main Screen for Input
  69. 69. 60 Figure 19 : Support Vector Machine screen with Test Data
  70. 70. 61 Figure 20 : Support Vector Machine Screen with Output
  72. 72. 63 7.CONCLUSION We considered three statistical models for solving our problem Support Vector Machine,Bag Of Words and Naive Bayes. The first method that we approached for our problem is Naïve Bayes. It is mainly based on the independence assumption .Training is very easy and fast In this approach each attribute in each class is considered separately.Testing is straightforward, calculating the conditional probabilities from the data available . One of the major task is to find the sentiment polarities which is very important in this approach to obtain desired output . In this naïve bayes approach we only considered the words that are available in our dataset and calculated their conditional probabilities . we have obtained successful results after applying this approach to our problem. In supervised learning methods next we adopted bag of words .This approach assumes that every single word in the test data is repeated atleast once, which eliminates the zero probability problem . After applying this approach, the results are obtained correctly and their execution is also very fast .The third method that we applied for our problem is support vector machine along with principal component analysis .The main reason for using principal component analysis is because of its dimensionality reduction .It reduces the large dimensions into smaller without any loss of data . A window (comprised of five words on either side of the given word) is used to count the number of appearances of each word in our data set ,to find the combinations of words .Training is very long compared to naïve bayes, bag of words .The training of support vector machine is done only with a small dataset. The outcomes of this approach are obtained partially . However, we were successful at predicting sentiment on topics in mobile reviews on a small scale using three different approaches Naïve Bayes, Bag Of Words, Support Vector Machine and also gained a lot of information in machine learning.
  73. 73. 64 REFERENCES [1] Chih-Wei Hsu, Chih-Chung Chang, Chih-Jen Lin,” A Practical Guide to Support Vector Classification”, http://www csie ntu edu tw ,web, July 22 2014 . [2] N Cristianini, J Shawe-Taylor, “An Introduction to Support Vector Machines and Other Kernel-based Learning Methods”, Cambridge University Press, 2000 . [3] Hiroshi Shimodaira,“Text classifying using Naïve Bayes ”, Document models, http://www inf ed ac uk/teaching/courses/inf2b/learnnotes/inf2b-learn-note07-2up,11 Feb 2014,web, 15 August 2014 . [4]Lindsay I Smith,“Principle Component Analysis”, http://www cs otago ac nz/cosc453/student_tutorials/principal_components, 2002,web,August 5 2014 . [5] Laura Auria , Rouslan A Moro , “Support Vector Machines ” , 2008,web,20 August 2014. [6] H Kim, P Howland, and H Park “Dimension reduction in text classification with support vector machines”, Journal of Machine Learning Research,2005 . [7]Pang- Ning Tan ,Michael Steinbach,Vipin Kumar ,”Introduction to Data Mining”, pearson publications. [8] Dan Jurafsky “Text Classification and Naïve Bayes”, The Task Of Text Classification, https://web stanford edu/class/cs124/lec/naivebayes ,web, July 28 2014 .