RealCode4You

Supervised learning as the name specifies shows the presence of a superviser as an educator. Essentially supervised learning is a learning wherein we teach or train the machine utilizing information in which the target lables are present. Once the machine has been trained then we present it with data without the presence of a target variable. In the second case the machine itself determines the correct targets for the data presented to it. Some of the most famous Supervised

algorithms include:
1: Support Vector Machines
2: Linear Regression
3: Logistic Regression
4: Naive Bayes
5: Linear Discriminanat Analysis
6: Neural Networks

​

Unsupervised Learning

Unsupervised learning as the name specifies has no superviser present, this means that there are no targets or labels present in the training phase. The data consists of just the feature variables and is grouped based only on the feature set. Some of the most famous Unsupervised algorithms in to-days time include:

1: K-means clustering
2: K-NN
3: Principal Component Analysis
4: Singular Value Decomposition
5: Hierarchical Clustering

​

Semi-Supervised Learning

Semi-Supervised learning is derived by the cumulation of supervised and unsupervised learning. In semi-supervised learning the data set is partially labeled this means in some of the cases we are provided with the targets but in most of the cases the data is still not labeled. Semi-supervised algorithms solve problems by making different assumptions. Some of these assumptions are listed below:

1: Continuity assumption
2: Cluster assumption

3: Manifold assumption
4: Generative models
5: Low-density separation
6: Graph-based Methods

​

Deep Learning

The field of Artificial Intelligence at its core is based on the idea that machine should be able to mimic human intelligence. Deep learning is a sub-field of artificial intelligence, where machines can learn form experience and acquire skills without human involvement. It consists of Artificial Neural networks, algorithms which try to emulate working of a human brain and learn from large amount of data provided to them. Similar to how we learn from experience, the deep learning algorithm would perform a task repeatedly, each time tweaking it a little to improve the outcome.

​

Natural Language Processing

Natural language processing abbreviated as NLP is a sub-field of Artificial Intelligence deals with automatic manipulation of speach in different forms. Everbody know that machines don’t understand human language but what if they could? This question gave rise to more and more research in the field so much so that almost every next year a new research paper is published describing a new methodology or a new model aiming to make the process more and more better as well as accurate. As of now Google has come with the BERT which is the state of art model till 2020, still there is so much research going in the field that it won’t be a surprise if a new better model comes along in 2020.

​

Application of NLP :

Few applications of Natural Language Processing include:

1 : Sentiment Analyzer
2 : Text classification/Clustering
3 : Email Spam and Malware Filtering
4 : Chatbots
5 : Machine Translation
6 : Question Answering
7 : Named Entity Recognition (NER)
8 : Speech Recognition

​

1 : Sentiment Analyser

Sentiment Analysis is relevant mining of content which recognizes emotional data in source material. In todays world a sentiment analyser is usally used over different social media sites. This is because there is plethora of data being posted everday and analysing this huge data has proven to be very useful in recent years. Deriving emotions behind this large textual data is called sentiment analysis. A sentiment analyser could help businesses to understand how their customers feel about their products and also their company as a whole. With the recent advances in deep learning, the ability of algorithms to analyse text has improved considerably. Creative use of advanced artificial intelligence techniques can be an effective tool for doing in-depth research. Sentiment Analysis is the most common text classification tool that analyses an incoming message and tells whether the underlying sentiment is positive, negative our neutral. How it works

There are 5 main steps involved in Building a basic sentiment analyser. These steps are :

1: Data Collection
2: Data Preprocessing
3: Feature Selection
4: Model Selection
5: Model Evaluation

Data Collection

This is the fundamental step of any machine learning model since data is the basic necessity for applying any ML algorithim. In our case of sentiment analysis data collection depends on the problem statement but mostly sentiment analysis is done on social media data such as twitter. Data from twitter could be extracted using the twitter api namely tweepy. This extracted data is not only raw and totally unstructured but also incomplete as sentiment analysis is a classification problem and needs a target variable. There are different methods of providing these targets such we can annote the data manually or we can use some lexicon based algorithm to do annotation
for us(manual annotation is preffered as lexicon based may not annote with perfection).

Data preprocessing

This one of the most important as well as most time consuming step in the whole process of building a sentiment analyser. The pre-processing of data implies the processing of raw data into a more convenient format which could be fed to a classifier in order to better the accuracy of the classifier. Here, in our case the raw data extracted from twitter using an API is initially totally unstructured and bogus as the availability of various useless characters seems very common in it. For this matter we need to remove all the unnecessary characters and words from this data using a module in python known as Regular Expressions, RE for short. This module adopts symbolic techniques to represent different noise in the data and therefore makes it easy to drop them. Specifically in twitter terminology there are various common useless phrases and spelling mistakes present in the data, which need to be removed to boost the accuracy of our resultant.

These could be summoned up as follows: •

Hash tags:
These are very common in tweets. Hash tags represent a topic of interest about which the tweet is being written. Hashtags look something like #topic. •

@Usernames:
These represent the user mentions in a tweet. Some times a tweet is written and then is associated with some twitter user, for this purpose these are used. •

Retweets(RT):
As the name suggests retweets are used when a tweet is posted twice by same or different user. •

Emoticons:
These are very commonly found in the tweets. Using punctuation facial expressions are formed in order to represent the a smile or other expressions, these are known as emoticons. •

Stop words:
Stop words are those word which are useless when it comes to sentiment analysis. Words such as it, is, the etc are known as stop words.

Feature Selection
Features is the oil on which a Machine Learning model runs. Choosing correct features is directly linked with how well an algorithm is going to perform. Different researchers have used different features for the classification of the tweets. These features include :

1: Unigram
2: Bigram
3: N-gram
4: POS tagging
5: Subjective features
6: objective features and so on.

NLTK short for Natural Language Tool Kit is a module available in python which is open source
and could be used for extraction of these features.

Model Selection

Once the data is being pre-processed, this data is to be fed to a classification model for further processing. There are different classification algorithms on which these models are built on.

​

For example KNN or k-Nearest Neighbor algorithm represents a machine learning technique used for classifying a set of data into its given target values . KNN could also be used for regression problems but is widely used for classification problems. Now, any classification model needs a target set on which we train the model for its further use. Most of Data Scientists manually set these target values to positive, negative or null. There are other ways of doing this for example there is a library in python known as Textblob that automatically set the target for each tweet just in few lines of code. The data set then is divided into two half's training set and testing set. Now this training as well as test set needs to be transformed into vector formation so as to be fed to the model. The models don’t understand any values other than the vectors. This could be done using another module of the python known as sklearn which contains many classification model as well as different encoders in it. Most commonly used encoding methods include:

1: Count vectorizor
2: Tfidf encoder

Alternately, we can use deep learning for building a sentiment analyser. In this case libraries like keras, tensorflow and Theono could turn to be helpful

​

Model evaluation
One of the most common and appropriate technique used for evaluation of a classifier is through confusion matrix. A generalized form of confusion matrix is given below:

​

By applying this technique we can derive the generalized evaluation parameters. These parameters include:

​

Accuracy : accuracy of a classifier indicates how accurately the classifier has predicted the result.

It can be calculated using the formula:

TP + TN / TP +TN+FP+FN

​

Precision: precision shows how often the result that is being predicted by the classifier is actually true, when it indicates true. The formula for precision is:

​

TP / TP+FP

​

Recall: it indicates the true positive rate of the classifier. Formula for recall is:

​

recall(r) = TP / TP + FN

​

F1 score : it indicates the weighed average of recall and precision.Formula for recall is:

​

F1 score = 2 p.r / p – r

​

​

2 : Text Classification

​

Text Classification is a technique in which we assign targets or categories to textual data in accordance with the context of the data. This method is included in the fundamentals of NLP techniques. Sentiment analysis is actually an application of text classification. Other applications of text classification include spam detection , also a faster emergency response system can be made by classifying panic conversation on social media. Textual data is everywhere, may it be emails , web sites , social media , books or chats. Everywhere the sight goes there is some form of unstructured textual data present. All this data could be made usefull only if we know how to extract it and find usefull patterns in it. Structuring this large data needs scrutinizing effort but this effort could bring a lot of benifit to an individual or organization. Almost all of the text Classification techniques could be expressed in following steps :

1: Data Collection

2: Data Preprocessing
3: Feature Selection
4: Model Selection
5: Model Evaluation

Data Collection
This is the first and a necessary step for building any Machine Learning algorithm as all the machine learning algorithms require some data to train on. Data Collection totally depends on the problem at hand , for example in case of sentiment analysis which is an application text classification, needs any raw text with attached target annotations as positive , negative and null. Similarly depending on the problem we can collect data in any form , may it be reviews of some product of some organization or it could be genre-labeled songs etc.

Data Preprocessing
The textual data that we collect is too messy and unprocessed. The messier the data we train our model on the poorer will be the model’s accuracy of prediction. Therefore for this reason preprocessing is one of the most important step of this process. Cleaning textual data is a lot different from cleaning any other data since it contains vocabulary words from any language instead of numbers. There are various preprocessing steps one could follow for better results some of these are written below:

1: Tokenization
Tokenization is the process of splitting a textual corpora in to a set of words or sentences known as tokens . A corpora when split into words are called word tokenization and when split into sentences is known as sentence tokenization. There are many libraries in python as well as in R which deal with tokenization for example keras, sklearn , textblob , preprocessor etc

2: Remove punctuation
Normally when we extract text from an online source say twitter , the data consists of a lot of punctuations since people tend to build emojis using different punctuation symbols. And also people sometimes use unnecessary punctuations everywhere for example some people use many fullstop symbols together to build a line. All these need to be removed from the data set as these are useless and make no sence. We can remove these punctuations simply by using Regex library.

3: Stemming
A stemming algorithm is a process of linguistic normalisation, in which the variant forms of a word are reduced to a common form, for example,

connection
connections
connective >>> connect
connected
connecting

4: Lemmatization
Lemmatization is the process of grouping together the different inflected forms of a word so they can be analysed as a single item. Lemmatization is similar to stemming but it brings context to the words. So it links words with similar meaning to one word.Text preprocessing includes both stemming as well as Lemmatization. Many times people find these two terms confusing. Some treat these two as same. Actually, lemmatization is preferred over Stemming because lemmatization does morphological analysis of the words. Lemmatization could be done in python
using NLTK framework.

​

5: Removing html
Sometimes what happenes is when we extract data from a web page or using some API, the data which we recieve consists of many html tags such as <h></h> or <p></p>. These tags add no meaning to our data thus these need to be removed. Feature Selection Choosing correct features is directly linked with how well an algorithm is going to perform. Different researchers have used different features for different problems. Choosing the right features for a given problem is very important. Some features used for textual classification problem include :

1: Unigram
2: Bigram
3: N-gram
4: POS tagging
5: Subjective features
6: objective features and so on.

After selecting the features , these features need to be vectorized since the model only understands numbers. This could be done in a number of ways. Some of recommended methods include :

1 : Using a Countvectorizer (sklearn.feature_extraction.text.CountVectorizor)
Countvectorizer is present in python’s sklearn package.The CountVectorizer provides a simple way to both tokenize a collection of text documents and build a vocabulary of known words, but also to encode new documents using that vocabulary.

​

2 : Using a TFIDF vectorizer(sklearn.feature_extraction.text.TfidfVectorizer)
Tf-idf stands for term frequency-inverse document frequency, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus.

Model Selection
There are many parameters on which choice of model depends for example if the target variable shows some sort of continuity then it is prefered to choose a regression based model. Similarly if the target variable consists of categorical values , then in that case it would be better not to choose regression.There are different classification algorithms on which these models are built on. For example logistic regression algorithm represents a machine learning technique used for classifying a set of data into its given target values . Logistic regression could also be used for regression problems but is widely used for classification problems.

​

Model evaluation
One of the most common and appropriate technique used for evaluation of a classifier is through confusion matrix. A generalized form of confusion matrix is given below:

​

By applying this technique we can derive the generalized evaluation parameters. These parameters include:

Accuracy : accuracy of a classifier indicates how accurately the classifier has predicted the result.

It can be calculated using the formula:

TP + TN / TP +TN+FP+FN

​

Precision: precision shows how often the result that is being predicted by the classifier is actually true, when it indicates true. The formula for precision is:

​

TP / TP+FP

​

Recall: it indicates the true positive rate of the classifier. Formula for recall is:

​

recall(r) = TP / TP + FN

​

F1 score : it indicates the weighed average of recall and precision.Formula for recall is:

​

F1 score = 2 p.r / p – r