TF(Term Frequency)
- Assign each term in a document a weight for that term.
- The weight of a term t in a document d is a function of the number of times t appears in d.
The weight can be simply set to the number of occurrences of t in d :
tf (t, d) = count (t, d)
The term frequency may optionally be normalized
Inverse Document Frequency (Idf)
idf(t) = log [N/df(t)]
N: Number of documents in the corpus
df(t): Number of documents in the corpus that contain a term t
- Measures term uniqueness in corpus
"phone" vs. "brick"
- Indicates the importance of the term
Search (relevance)
Classification (discriminatory power)
TF-IDF and Modified Retrieval Algorithm
- term t in document d:
tfidf(t, d) = tf (t, d) * idf(t)
query: brick, phone
- Document with "brick" a few times more relevant than document with "phone" many times
- Measure of Relevance with tf-idf
- Call up all the documents that have any of the terms from the query, and sum up the tf-idf of each term:
TF-IDF and Modified Retrieval Algorithm, example
The process to find meaning of documents using TF-IDF is very similar to Bag of words,
Clean data / Preprocessing — Clean data (standardise data) , Normalize data( all lower case) , lemmatize data ( all words to root words ).
Tokenize words with frequency
Find TF for words
Find IDF for words
Vectorize vocab
Example
Let’s cover an example of 3 documents -
Document 1 It is going to rain today. 1/6
Document 2 Today I am not going outside.
Document 3 I am going to watch the season premiere. 1/8
To find TF-IDF we need to perform the steps we laid out above, let’s get to it.
- Step 1: Clean data and Tokenize
- Step 2: Find TF for all docs
TF = (Number of repetitions of word in a document) / (# of words in a document)
- Step 3: Find IDF
IDF =Log[(Number of documents) / (Number of documents containing the word)]In Excel use LN(3/3)
- Step 4: Build model i.e. stack all words next to each other
IDF Value and TF value of 3 documents
- Step 5: Compare results and use table to ask questions
Remember, the final equation = TF-IDF = TF * IDF
Analysis and outcomes
You can easily see using this table that words like ‘it’,’is’,’rain’ are important for document 1 but not for document 2 and document 3 which means Document 1 and 2&3 are different w.r.t talking about rain.
You can also say that Document 1 and 2 talk about something ‘today’, and document 2 and 3 discuss something about the writer because of the word ‘I’.
This table helps you find similarities and non similarities between documents, words and more much better than Bag Of Words.