{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","metadata":{"id":"u0t1UALWvzbq"},"source":["# 1st LAB: Document Representation"]},{"cell_type":"code","metadata":{"id":"9jQRNFjwTuFb","colab":{"base_uri":"https://localhost:8080/"},"outputId":"79954504-2cb4-41b7-fdb9-c4f5e7f645d0","executionInfo":{"status":"ok","timestamp":1668081959795,"user_tz":-60,"elapsed":18003,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["from google.colab import drive\n","drive.mount(\"/content/gdrive\")\n","\n","from collections import defaultdict\n","from pathlib import Path\n","import pandas as pd\n","pd.set_option('display.max_colwidth', -1)"],"execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["Mounted at /content/gdrive\n"]},{"output_type":"stream","name":"stderr","text":["/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:7: FutureWarning: Passing a negative integer is deprecated in version 1.0 and will not be supported in future version. Instead, use None to not limit the column width.\n"," import sys\n"]}]},{"cell_type":"markdown","metadata":{"id":"IW14-Hn8JvQj"},"source":["# Importing the data"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":431},"id":"ertxZLqZKQH8","outputId":"2966df1b-3105-4e48-d727-d623a1e467c0","executionInfo":{"status":"ok","timestamp":1668081959797,"user_tz":-60,"elapsed":18,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["print('Sample Text:\\n')\n","text0 = \"DEAR FRIEND,\\n\\nI AM ( DR.) JAMES NGOLA, THE PERSONAL ASSISTANCE TO THE LATE CONGOLESE (PRESIDENT LAURENT KABILA) WHO WAS ASSASSINATED BY HIS BODY GUARD ON 16TH JAN. 2001.\\n\\n\\nTHE INCIDENT OCCURRED IN OUR PRESENCE WHILE WE WERE HOLDING MEETING WITH HIS EXCELLENCY OVER THE FINANCIAL RETURNS FROM THE DIAMOND SALES IN THE AREAS CONTROLLED BY (D.R.C.) DEMOCRATIC REPUBLIC OF CONGO FORCES AND THEIR FOREIGN ALLIES ANGOLA AND ZIMBABWE, HAVING RECEIVED THE PREVIOUS DAY (USD$100M) ONE HUNDRED MILLION UNITED STATES DOLLARS, CASH IN THREE DIPLOMATIC BOXES ROUTED THROUGH ZIMBABWE.\\n\\nMY PURPOSE OF WRITING YOU THIS LETTER IS TO SOLICIT FOR YOUR ASSISTANCE AS TO BE A COVER TO THE FUND AND ALSO COLLABORATION IN MOVING THE SAID FUND INTO YOUR BANK ACCOUNT THE SUM OF (USD$25M) TWENTY FIVE MILLION UNITED STATES DOLLARS ONLY, WHICH I DEPOSITED WITH A SECURITY COMPANY IN GHANA, IN A DIPLOMATIC BOX AS GOLDS WORTH (USD$25M) TWENTY FIVE MILLION UNITED STATES DOLLARS ONLY FOR SAFE KEEPING IN A SECURITY VAULT FOR ANY FURTHER INVESTMENT PERHAPS IN YOUR COUNTRY. \\n\\nYOU WERE INTRODUCED TO ME BY A RELIABLE FRIEND OF MINE WHO IS A TRAVELLER,AND ALSO A MEMBER OF CHAMBER OF COMMERCE AS A RELIABLE AND TRUSTWORTHY PERSON WHOM I CAN RELY ON AS FOREIGN PARTNER, EVEN THOUGH THE NATURE OF THE TRANSACTION WAS NOT REVEALED TO HIM FOR SECURITY REASONS.\\n\\n\\nTHE (USD$25M) WAS PART OF A PROCEEDS FROM DIAMOND TRADE MEANT FOR THE LATE PRESIDENT LAURENT KABILA WHICH WAS DELIVERED THROUGH ZIMBABWE IN DIPLOMATIC BOXES. THE BOXES WERE KEPT UNDER MY CUSTODY BEFORE THE SAD EVENT THAT TOOK THE LIFE OF (MR. PRESIDENT).THE CONFUSION THAT ENSUED AFTER THE ASSASSINATION AND THE SPORADIC SHOOTING AMONG THE FACTIONS, I HAVE TO RUN AWAY FROM THE COUNTRY FOR MY DEAR LIFE AS I AM NOT A SOLDIER BUT A CIVIL SERVANT I CROSSED RIVER CONGO TO OTHER SIDE OF CONGO LIBREVILLE FROM THERE I MOVED TO THE THIRD COUNTRY GHANA WHERE I AM PRESENTLY TAKING REFUGE. \\n\\nAS A MATTER OF FACT, WHAT I URGENTLY NEEDED FROM YOU IS YOUR ASSISTANCE IN MOVING THIS MONEY INTO YOUR ACCOUNT IN YOUR COUNTRY FOR INVESTMENT WITHOUT RAISING EYEBROW. FOR YOUR ASSISTANCE I WILL GIVE YOU 20% OF THE TOTAL SUM AS YOUR OWN SHARE WHEN THE MONEY GETS TO YOUR ACCOUNT, WHILE 75% WILL BE FOR ME, OF WHICH WITH YOUR KIND ADVICE I HOPE TO INVEST IN PROFITABLE VENTURE IN YOUR COUNTRY IN OTHER TO SETTLE DOWN FOR MEANINGFUL LIFE, AS I AM TIRED OF LIVING IN A WAR ENVIRONMENT. \\n\\nTHE REMAINING 5% WILL BE USED TO OFFSET ANY COST INCURRED IN THE CAUSE OF MOVING THE MONEY TO YOUR ACCOUNT. IF THE PROPOSAL IS ACCEPTABLE TO YOU PLEASE CONTACT ME IMMEDIATELY THROUGH THE ABOVE TELEPHONE AND E-MAIL, TO ENABLE ME ARRANGE FACE TO FACE MEETING WITH YOU IN GHANA FOR THE CLEARANCE OF THE FUNDS BEFORE TRANSFRING IT TO YOUR BANK ACCOUNT AS SEEING IS BELIEVING. \\n\\nFINALLY, IT IS IMPORTANT ALSO THAT I LET YOU UNDERSTAND THAT THERE IS NO RISK INVOLVED WHATSOEVER AS THE MONEY HAD NO RECORD IN KINSHASA FOR IT WAS MEANT FOR THE PERSONAL USE OF (MR. PRESIDEND ) BEFORE THE NEFARIOUS INCIDENT OCCURRED, AND ALSO I HAVE ALL THE NECESSARY DOCUMENTS AS REGARDS TO THE FUNDS INCLUDING THE (CERTIFICATE OF DEPOSIT), AS I AM THE DEPOSITOR OF THE CONSIGNMENT.\\n\\n\\nLOOKING FORWARD TO YOUR URGENT RESPONSE.\\n\\nYOUR SINCERELY,\\n\\n\\t\\t\\tMR. JAMES NGOLA. \" \n","text = \"DEAR FRIEND,\\n\\nI AM ( DR.) JAMES NGOLA, THE PERSONAL ASSISTANCE TO THE LATE CONGOLESE (PRESIDENT LAURENT KABILA) WHO WAS ASSASSINATED BY HIS BODY GUARD ON 16TH JAN. 2001.\\n\\n\\nTHE INCIDENT OCCURRED IN OUR PRESENCE WHILE WE WERE HOLDING MEETING WITH HIS EXCELLENCY OVER THE FINANCIAL RETURNS FROM THE DIAMOND SALES IN THE AREAS CONTROLLED BY (D.R.C.) DEMOCRATIC REPUBLIC OF CONGO FORCES AND THEIR FOREIGN ALLIES ANGOLA AND ZIMBABWE, HAVING RECEIVED THE PREVIOUS DAY (USD$100M) ONE HUNDRED MILLION UNITED STATES DOLLARS, CASH IN THREE DIPLOMATIC BOXES ROUTED THROUGH ZIMBABWE.\\n\\nMY PURPOSE OF WRITING YOU THIS LETTER IS TO SOLICIT FOR YOUR ASSISTANCE AS TO BE A COVER TO THE FUND AND ALSO COLLABORATION IN MOVING THE SAID FUND INTO YOUR BANK ACCOUNT THE SUM OF (USD$25M) TWENTY FIVE MILLION UNITED STATES DOLLARS ONLY, WHICH I DEPOSITED WITH A SECURITY COMPANY IN GHANA, IN A DIPLOMATIC BOX AS GOLDS WORTH (USD$25M) TWENTY FIVE MILLION UNITED STATES DOLLARS ONLY FOR SAFE KEEPING IN A SECURITY VAULT FOR ANY FURTHER INVESTMENT PERHAPS IN YOUR COUNTRY. \\n\\nYOU WERE INTRODUCED TO ME BY A RELIABLE FRIEND OF MINE WHO IS A TRAVELLER,AND ALSO A MEMBER OF CHAMBER OF COMMERCE AS A RELIABLE AND TRUSTWORTHY PERSON WHOM I CAN RELY ON AS FOREIGN PARTNER, EVEN THOUGH THE NATURE OF THE TRANSACTION WAS NOT REVEALED TO HIM FOR SECURITY REASONS.\\n\\n\\nTHE (USD$25M) WAS PART OF A PROCEEDS FROM DIAMOND TRADE MEANT FOR THE LATE PRESIDENT LAURENT KABILA WHICH WAS DELIVERED THROUGH ZIMBABWE IN DIPLOMATIC BOXES. THE BOXES WERE KEPT UNDER MY CUSTODY BEFORE THE SAD EVENT THAT TOOK THE LIFE OF (MR. PRESIDENT).THE CONFUSION THAT ENSUED AFTER THE ASSASSINATION AND THE SPORADIC SHOOTING AMONG THE FACTIONS, I HAVE TO RUN AWAY FROM THE COUNTRY FOR MY DEAR LIFE AS I AM NOT A SOLDIER BUT A CIVIL SERVANT I CROSSED RIVER CONGO TO OTHER SIDE OF CONGO LIBREVILLE FROM THERE I MOVED TO THE THIRD COUNTRY GHANA WHERE I AM PRESENTLY TAKING REFUGE. \\n\\nAS A MATTER OF FACT, WHAT I URGENTLY NEEDED FROM YOU IS YOUR ASSISTANCE IN MOVING THIS MONEY INTO YOUR ACCOUNT IN YOUR COUNTRY FOR INVESTMENT WITHOUT RAISING EYEBROW. FOR YOUR ASSISTANCE I WILL GIVE YOU 20% OF THE TOTAL SUM AS YOUR OWN SHARE WHEN THE MONEY GETS TO YOUR ACCOUNT, WHILE 75% WILL BE FOR ME, OF WHICH WITH YOUR KIND ADVICE I HOPE TO INVEST IN PROFITABLE VENTURE IN YOUR COUNTRY IN OTHER TO SETTLE DOWN FOR MEANINGFUL LIFE, AS I AM TIRED OF LIVING IN A WAR ENVIRONMENT. \\n\\nTHE REMAINING 5% WILL BE USED TO OFFSET ANY COST INCURRED IN THE CAUSE OF MOVING THE MONEY TO YOUR ACCOUNT. IF THE PROPOSAL IS ACCEPTABLE TO YOU PLEASE CONTACT ME IMMEDIATELY THROUGH THE ABOVE TELEPHONE AND E-MAIL, TO ENABLE ME ARRANGE FACE TO FACE MEETING WITH YOU IN GHANA FOR THE CLEARANCE OF THE FUNDS BEFORE TRANSFRING IT TO YOUR BANK ACCOUNT AS SEEING IS BELIEVING. \\n\\nFINALLY, IT IS IMPORTANT ALSO THAT I LET YOU UNDERSTAND THAT THERE IS NO RISK INVOLVED WHATSOEVER AS THE MONEY HAD NO RECORD IN KINSHASA FOR IT WAS MEANT FOR THE PERSONAL USE OF (MR. PRESIDEND ) BEFORE THE NEFARIOUS INCIDENT OCCURRED, AND ALSO I HAVE ALL THE NECESSARY DOCUMENTS AS REGARDS TO THE FUNDS INCLUDING THE (CERTIFICATE OF DEPOSIT), AS I AM THE DEPOSITOR OF THE CONSIGNMENT.\\n\\n\\nLOOKING FORWARD TO YOUR URGENT RESPONSE.\\n\\nYOUR SINCERELY,\\n\\n\\t\\t\\tMR. JAMES NGOLA. \" \n","import pandas as pd\n","##reading the data\n","data_spam = pd.read_csv(\"/content/gdrive/MyDrive/LAB-1_IR_2022-2023/spam.csv\",encoding=\"ISO-8859-1\")\n","display(data_spam.head())\n","data_spam = data_spam[['v1','v2']]\n","display(data_spam.head())"],"execution_count":3,"outputs":[{"output_type":"stream","name":"stdout","text":["Sample Text:\n","\n"]},{"output_type":"display_data","data":{"text/plain":[" v1 \\\n","0 ham \n","1 ham \n","2 spam \n","3 ham \n","4 ham \n","\n"," v2 \\\n","0 Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat... \n","1 Ok lar... Joking wif u oni... \n","2 Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's \n","3 U dun say so early hor... U c already then say... \n","4 Nah I don't think he goes to usf, he lives around here though \n","\n"," Unnamed: 2 Unnamed: 3 Unnamed: 4 \n","0 NaN NaN NaN \n","1 NaN NaN NaN \n","2 NaN NaN NaN \n","3 NaN NaN NaN \n","4 NaN NaN NaN "],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
v1v2Unnamed: 2Unnamed: 3Unnamed: 4
0hamGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...NaNNaNNaN
1hamOk lar... Joking wif u oni...NaNNaNNaN
2spamFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18'sNaNNaNNaN
3hamU dun say so early hor... U c already then say...NaNNaNNaN
4hamNah I don't think he goes to usf, he lives around here thoughNaNNaNNaN
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":[" v1 \\\n","0 ham \n","1 ham \n","2 spam \n","3 ham \n","4 ham \n","\n"," v2 \n","0 Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat... \n","1 Ok lar... Joking wif u oni... \n","2 Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's \n","3 U dun say so early hor... U c already then say... \n","4 Nah I don't think he goes to usf, he lives around here though "],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
v1v2
0hamGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...
1hamOk lar... Joking wif u oni...
2spamFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's
3hamU dun say so early hor... U c already then say...
4hamNah I don't think he goes to usf, he lives around here though
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{}}]},{"cell_type":"code","source":["import nltk\n","from nltk.corpus import movie_reviews\n","nltk.download('movie_reviews') ###< run once\n","nltk.download('punkt') ###< run once\n","raw = movie_reviews.raw()\n","print()\n","print(\"RAW representation of movies reviews dataset \\n\")\n","print(raw[:500])"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"IPDwcDPy5UPn","executionInfo":{"status":"ok","timestamp":1668081961737,"user_tz":-60,"elapsed":1950,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}},"outputId":"cb2102d5-7042-4493-d2f0-56e371bf6c9f"},"execution_count":4,"outputs":[{"output_type":"stream","name":"stderr","text":["[nltk_data] Downloading package movie_reviews to /root/nltk_data...\n","[nltk_data] Unzipping corpora/movie_reviews.zip.\n","[nltk_data] Downloading package punkt to /root/nltk_data...\n","[nltk_data] Unzipping tokenizers/punkt.zip.\n"]},{"output_type":"stream","name":"stdout","text":["\n","RAW representation of movies reviews dataset \n","\n","plot : two teen couples go to a church party , drink and then drive . \n","they get into an accident . \n","one of the guys dies , but his girlfriend continues to see him in her life , and has nightmares . \n","what's the deal ? \n","watch the movie and \" sorta \" find out . . . \n","critique : a mind-fuck movie for the teen generation that touches on a very cool idea , but presents it in a very bad package . \n","which is what makes this review an even harder one to write , since i generally applaud films which attempt\n"]}]},{"cell_type":"markdown","metadata":{"id":"RuRxTTDO_HQc"},"source":["# Text Pre-processing"]},{"cell_type":"code","metadata":{"id":"WkXSY_GM_Hjm","colab":{"base_uri":"https://localhost:8080/","height":395},"outputId":"d31aeaf1-b1c8-4ed1-ce4f-3409004d5492","executionInfo":{"status":"ok","timestamp":1668081961738,"user_tz":-60,"elapsed":18,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["import regex as re\n","import string\n","#defining the function to remove punctuation\n","def remove_punctuation(text):\n"," punctuationfree=\"\".join([i for i in text if i not in string.punctuation])\n"," return punctuationfree\n","\n","def character_repeatation(text):\n"," # Pattern matching for all case alphabets\n"," # \\1 It refers to the first capturing group.\n"," # {1,} It means we are matching for repetition that occurs more than one time.\n"," # r’\\1\\1' → It limits all the repetition to two characters.\n"," Pattern_alpha = re.compile(r\"([A-Za-z])\\1{1,}\", re.DOTALL)\n"," # Limiting all the repeatation to two characters.\n"," Formatted_text = Pattern_alpha.sub(r\"\\1\\1\", text) \n"," # Pattern matching for all the punctuations that can occur\n"," Pattern_Punct = re.compile(r'([.,/#!$%^&*?;:{}=_`~()+-])\\1{1,}')\n"," # Limiting punctuations in previously formatted string to only one.\n"," Combined_Formatted = Pattern_Punct.sub(r'\\1', Formatted_text)\n"," return Combined_Formatted\n","\n","display(data_spam.head(5))\n","data_spam['v2']= data_spam['v2'].apply(lambda x: remove_punctuation(x))\n","data_spam['v2']= data_spam['v2'].apply(lambda x: character_repeatation(x))\n","data_spam['v2']= data_spam['v2'].apply(lambda x: x.lower())\n","data_spam['v2']= data_spam['v2'].apply(lambda x: re.sub(r'\\d+', '', x))\n","display(data_spam.head(5))"],"execution_count":5,"outputs":[{"output_type":"display_data","data":{"text/plain":[" v1 \\\n","0 ham \n","1 ham \n","2 spam \n","3 ham \n","4 ham \n","\n"," v2 \n","0 Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat... \n","1 Ok lar... Joking wif u oni... \n","2 Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's \n","3 U dun say so early hor... U c already then say... \n","4 Nah I don't think he goes to usf, he lives around here though "],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
v1v2
0hamGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...
1hamOk lar... Joking wif u oni...
2spamFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's
3hamU dun say so early hor... U c already then say...
4hamNah I don't think he goes to usf, he lives around here though
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":[" v1 \\\n","0 ham \n","1 ham \n","2 spam \n","3 ham \n","4 ham \n","\n"," v2 \n","0 go until jurong point crazy available only in bugis n great world la e buffet cine there got amore wat \n","1 ok lar joking wif u oni \n","2 free entry in a wkly comp to win fa cup final tkts st may text fa to to receive entry questionstd txt ratetcs apply overs \n","3 u dun say so early hor u c already then say \n","4 nah i dont think he goes to usf he lives around here though "],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
v1v2
0hamgo until jurong point crazy available only in bugis n great world la e buffet cine there got amore wat
1hamok lar joking wif u oni
2spamfree entry in a wkly comp to win fa cup final tkts st may text fa to to receive entry questionstd txt ratetcs apply overs
3hamu dun say so early hor u c already then say
4hamnah i dont think he goes to usf he lives around here though
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{}}]},{"cell_type":"markdown","metadata":{"id":"JPHam4qFRyLX"},"source":["# Document Representation"]},{"cell_type":"markdown","metadata":{"id":"9gnQt5HaR1rb"},"source":["### Create sample data"]},{"cell_type":"code","metadata":{"id":"UPX6bAgmR0je","colab":{"base_uri":"https://localhost:8080/","height":300},"outputId":"547a170c-cd87-438e-c576-6c7aeb013d36","executionInfo":{"status":"ok","timestamp":1668081961739,"user_tz":-60,"elapsed":16,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["import pandas as pd\n","import numpy as np\n","\n","corpus = ['The sky is blue and beautiful.',\n"," 'Love this blue and beautiful sky!',\n"," 'The quick brown fox jumps over the lazy dog.',\n"," \"A king's breakfast has sausages, ham, bacon, eggs, toast and beans\",\n"," 'I love green eggs, ham, sausages and bacon!',\n"," 'The brown fox is quick and the blue dog is lazy!',\n"," 'The sky is very blue and the sky is very beautiful today',\n"," 'The dog is lazy but the brown fox is quick!' \n","]\n","\n","labels = ['weather', 'weather', 'animals', 'food', 'food', 'animals', 'weather', 'animals']\n","\n","corpus = np.array(corpus)\n","corpus_df = pd.DataFrame({'Document': corpus, \n"," 'Category': labels})\n","corpus_df = corpus_df[['Document', 'Category']]\n","corpus_df\n"],"execution_count":6,"outputs":[{"output_type":"execute_result","data":{"text/plain":[" Document Category\n","0 The sky is blue and beautiful. weather\n","1 Love this blue and beautiful sky! weather\n","2 The quick brown fox jumps over the lazy dog. animals\n","3 A king's breakfast has sausages, ham, bacon, eggs, toast and beans food \n","4 I love green eggs, ham, sausages and bacon! food \n","5 The brown fox is quick and the blue dog is lazy! animals\n","6 The sky is very blue and the sky is very beautiful today weather\n","7 The dog is lazy but the brown fox is quick! animals"],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
DocumentCategory
0The sky is blue and beautiful.weather
1Love this blue and beautiful sky!weather
2The quick brown fox jumps over the lazy dog.animals
3A king's breakfast has sausages, ham, bacon, eggs, toast and beansfood
4I love green eggs, ham, sausages and bacon!food
5The brown fox is quick and the blue dog is lazy!animals
6The sky is very blue and the sky is very beautiful todayweather
7The dog is lazy but the brown fox is quick!animals
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{},"execution_count":6}]},{"cell_type":"markdown","metadata":{"id":"ReyM5DELU5IQ"},"source":["### Binary vectorizer"]},{"cell_type":"markdown","metadata":{"id":"CcXxOuivSW7f"},"source":["### Count Vectorizer\n"]},{"cell_type":"code","metadata":{"id":"fZiJcxlISeLm","executionInfo":{"status":"ok","timestamp":1668081961987,"user_tz":-60,"elapsed":259,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["from sklearn.feature_extraction.text import CountVectorizer\n","\n","cv = CountVectorizer(min_df=0., max_df=1.)\n","cv_matrix = cv.fit_transform(corpus_df.Document.iloc[:100]) #top 100 documents (not required here)\n","cv_matrix = cv_matrix.toarray()"],"execution_count":7,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":330},"id":"TKbVosSKS_QX","outputId":"85556735-af37-4a20-dc9b-4aac5a1601df","executionInfo":{"status":"ok","timestamp":1668081961988,"user_tz":-60,"elapsed":19,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["# get all unique words in the corpus\n","vocab = cv.get_feature_names_out()\n","# show document feature vectors\n","pd.DataFrame(cv_matrix, columns=vocab)"],"execution_count":8,"outputs":[{"output_type":"execute_result","data":{"text/plain":[" and bacon beans beautiful blue breakfast brown but dog eggs ... \\\n","0 1 0 0 1 1 0 0 0 0 0 ... \n","1 1 0 0 1 1 0 0 0 0 0 ... \n","2 0 0 0 0 0 0 1 0 1 0 ... \n","3 1 1 1 0 0 1 0 0 0 1 ... \n","4 1 1 0 0 0 0 0 0 0 1 ... \n","5 1 0 0 0 1 0 1 0 1 0 ... \n","6 1 0 0 1 1 0 0 0 0 0 ... \n","7 0 0 0 0 0 0 1 1 1 0 ... \n","\n"," love over quick sausages sky the this toast today very \n","0 0 0 0 0 1 1 0 0 0 0 \n","1 1 0 0 0 1 0 1 0 0 0 \n","2 0 1 1 0 0 2 0 0 0 0 \n","3 0 0 0 1 0 0 0 1 0 0 \n","4 1 0 0 1 0 0 0 0 0 0 \n","5 0 0 1 0 0 2 0 0 0 0 \n","6 0 0 0 0 2 2 0 0 1 2 \n","7 0 0 1 0 0 2 0 0 0 0 \n","\n","[8 rows x 28 columns]"],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
andbaconbeansbeautifulbluebreakfastbrownbutdogeggs...loveoverquicksausagesskythethistoasttodayvery
01001100000...0000110000
11001100000...1000101000
20000001010...0110020000
31110010001...0001000100
41100000001...1001000000
51000101010...0010020000
61001100000...0000220012
70000001110...0010020000
\n","

8 rows × 28 columns

\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{},"execution_count":8}]},{"cell_type":"markdown","metadata":{"id":"d04C0T1DWzuw"},"source":["###TF_IDF"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":330},"id":"3QsJq5GEW3sB","outputId":"2e805ab0-e94b-44d8-91a3-26ec588907bf","executionInfo":{"status":"ok","timestamp":1668081961989,"user_tz":-60,"elapsed":14,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["from sklearn.feature_extraction.text import TfidfVectorizer\n","\n","# Setting the vectorizer\n","tv = TfidfVectorizer(min_df=0., max_df=1., use_idf=True)\n","# Transforming the data to vectors\n","tv_matrix = tv.fit_transform(corpus_df.Document.iloc[:100])\n","tv_matrix = tv_matrix.toarray()\n","\n","vocab = tv.get_feature_names_out()\n","pd.DataFrame(np.round(tv_matrix, 2), columns=vocab)"],"execution_count":9,"outputs":[{"output_type":"execute_result","data":{"text/plain":[" and bacon beans beautiful blue breakfast brown but dog eggs \\\n","0 0.32 0.00 0.00 0.47 0.41 0.00 0.00 0.00 0.00 0.00 \n","1 0.27 0.00 0.00 0.39 0.34 0.00 0.00 0.00 0.00 0.00 \n","2 0.00 0.00 0.00 0.00 0.00 0.00 0.30 0.00 0.30 0.00 \n","3 0.18 0.30 0.35 0.00 0.00 0.35 0.00 0.00 0.00 0.30 \n","4 0.23 0.38 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.38 \n","5 0.20 0.00 0.00 0.00 0.26 0.00 0.29 0.00 0.29 0.00 \n","6 0.15 0.00 0.00 0.22 0.19 0.00 0.00 0.00 0.00 0.00 \n","7 0.00 0.00 0.00 0.00 0.00 0.00 0.28 0.39 0.28 0.00 \n","\n"," ... love over quick sausages sky the this toast today very \n","0 ... 0.00 0.00 0.00 0.00 0.47 0.36 0.00 0.00 0.0 0.0 \n","1 ... 0.45 0.00 0.00 0.00 0.39 0.00 0.54 0.00 0.0 0.0 \n","2 ... 0.00 0.41 0.30 0.00 0.00 0.46 0.00 0.00 0.0 0.0 \n","3 ... 0.00 0.00 0.00 0.30 0.00 0.00 0.00 0.35 0.0 0.0 \n","4 ... 0.38 0.00 0.00 0.38 0.00 0.00 0.00 0.00 0.0 0.0 \n","5 ... 0.00 0.00 0.29 0.00 0.00 0.45 0.00 0.00 0.0 0.0 \n","6 ... 0.00 0.00 0.00 0.00 0.43 0.34 0.00 0.00 0.3 0.6 \n","7 ... 0.00 0.00 0.28 0.00 0.00 0.44 0.00 0.00 0.0 0.0 \n","\n","[8 rows x 28 columns]"],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
andbaconbeansbeautifulbluebreakfastbrownbutdogeggs...loveoverquicksausagesskythethistoasttodayvery
00.320.000.000.470.410.000.000.000.000.00...0.000.000.000.000.470.360.000.000.00.0
10.270.000.000.390.340.000.000.000.000.00...0.450.000.000.000.390.000.540.000.00.0
20.000.000.000.000.000.000.300.000.300.00...0.000.410.300.000.000.460.000.000.00.0
30.180.300.350.000.000.350.000.000.000.30...0.000.000.000.300.000.000.000.350.00.0
40.230.380.000.000.000.000.000.000.000.38...0.380.000.000.380.000.000.000.000.00.0
50.200.000.000.000.260.000.290.000.290.00...0.000.000.290.000.000.450.000.000.00.0
60.150.000.000.220.190.000.000.000.000.00...0.000.000.000.000.430.340.000.000.30.6
70.000.000.000.000.000.000.280.390.280.00...0.000.000.280.000.000.440.000.000.00.0
\n","

8 rows × 28 columns

\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{},"execution_count":9}]},{"cell_type":"markdown","metadata":{"id":"T8oYKkFKYDN1"},"source":["### Document Similarity"]},{"cell_type":"code","metadata":{"id":"BPs4GKfCYGgX","colab":{"base_uri":"https://localhost:8080/","height":300},"outputId":"aba59f63-8550-45ea-8992-a6d6f35ae040","executionInfo":{"status":"ok","timestamp":1668081963775,"user_tz":-60,"elapsed":891,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["from sklearn.metrics.pairwise import cosine_similarity\n","\n","similarity_matrix = cosine_similarity(tv_matrix)\n","similarity_df = pd.DataFrame(similarity_matrix)\n","similarity_df"],"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/plain":[" 0 1 2 3 4 5 6 \\\n","0 1.000000 0.592497 0.167270 0.056604 0.073640 0.541928 0.704836 \n","1 0.592497 1.000000 0.000000 0.047703 0.236620 0.142703 0.360920 \n","2 0.167270 0.000000 1.000000 0.000000 0.000000 0.645455 0.155793 \n","3 0.056604 0.047703 0.000000 1.000000 0.493841 0.035512 0.026360 \n","4 0.073640 0.236620 0.000000 0.493841 1.000000 0.046201 0.034294 \n","5 0.541928 0.142703 0.645455 0.035512 0.046201 1.000000 0.425888 \n","6 0.704836 0.360920 0.155793 0.026360 0.034294 0.425888 1.000000 \n","7 0.362454 0.000000 0.627881 0.000000 0.000000 0.869431 0.337583 \n","\n"," 7 \n","0 0.362454 \n","1 0.000000 \n","2 0.627881 \n","3 0.000000 \n","4 0.000000 \n","5 0.869431 \n","6 0.337583 \n","7 1.000000 "],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
01234567
01.0000000.5924970.1672700.0566040.0736400.5419280.7048360.362454
10.5924971.0000000.0000000.0477030.2366200.1427030.3609200.000000
20.1672700.0000001.0000000.0000000.0000000.6454550.1557930.627881
30.0566040.0477030.0000001.0000000.4938410.0355120.0263600.000000
40.0736400.2366200.0000000.4938411.0000000.0462010.0342940.000000
50.5419280.1427030.6454550.0355120.0462011.0000000.4258880.869431
60.7048360.3609200.1557930.0263600.0342940.4258881.0000000.337583
70.3624540.0000000.6278810.0000000.0000000.8694310.3375831.000000
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{},"execution_count":10}]},{"cell_type":"markdown","metadata":{"id":"bBr1bxosr0sz"},"source":["# Search the word \"bad\" in the dataset : **data_spam**"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"id":"dBHxcqOhr0yD","outputId":"8f3c0e30-e558-4877-f472-7e5c9e3986ec","executionInfo":{"status":"ok","timestamp":1668081964737,"user_tz":-60,"elapsed":11,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["data_spam['search'] = data_spam['v2'].str.contains('bad', na=False)\n","print(len(data_spam['v2'][data_spam.search == True]))\n","display(data_spam[data_spam.search == True])"],"execution_count":11,"outputs":[{"output_type":"stream","name":"stdout","text":["36\n"]},{"output_type":"display_data","data":{"text/plain":[" v1 \\\n","31 ham \n","458 ham \n","565 ham \n","732 ham \n","1183 ham \n","1300 ham \n","1397 ham \n","1594 ham \n","1731 ham \n","1793 ham \n","1862 ham \n","2125 ham \n","2278 ham \n","2444 ham \n","2621 ham \n","3022 ham \n","3042 ham \n","3103 ham \n","3279 ham \n","3415 ham \n","3483 ham \n","3567 ham \n","4001 ham \n","4071 spam \n","4145 ham \n","4224 ham \n","4535 ham \n","4955 ham \n","5057 ham \n","5086 ham \n","5098 spam \n","5146 ham \n","5217 ham \n","5286 ham \n","5474 ham \n","5525 ham \n","\n"," v2 \\\n","31 yeah he got in at and was v apologetic n had fallen out and she was actin like spoilt child and he got caught up in that till but we wont go there not doing too badly cheers you \n","458 that call cost which i guess isnt bad miss ya need ya want ya love ya \n","565 ill call u mrw at ninish with my address that icky american freek wont stop callin me bad jen k eh \n","732 lol you wont feel bad when i use her money to take you out to a steak dinner d \n","1183 am i that much bad to avoid like this \n","1300 those cocksuckers if it makes you feel better ipads are worthless garbage novelty items and you should feel bad for even wanting one \n","1397 then we wait u lor no need feel bad lar \n","1594 never blame a day in ur life good days give u happiness bad days give u experience both are essential in life all are gods blessings good morning \n","1731 lol well quality aint bad at all so i aint complaining \n","1793 you bad girl i can still remember them \n","1862 the last thing i ever wanted to do was hurt you and i didnt think it would have youd laugh be embarassed delete the tag and keep going but as far as i knew it wasnt even up the fact that you even felt like i would do it to hurt you shows you really dont know me at all it was messy wednesday but it wasnt bad the problem i have with it is you have the time to clean it but you choose not to you skype you take pictures you sleep you want to go out i dont mind a few things here and there but when you dont make the bed when you throw laundry on top of it when i cant have a friend in the house because im embarassed that theres underwear and bras strewn on the bed pillows on the floor thats something else you used to be good about at least making the bed \n","2125 sorry im getting up now feel really bad totally rejected that kinda me thing \n","2278 hmmbad newshype park plaza studio takenonly left bedrm \n","2444 yes but i dont care i need you bad princess \n","2621 lol nah wasnt too bad thanks its good to b home but its been quite a reality check hows ur day been did u do anything with website \n","3022 you are a very very very very bad girl or lady \n","3042 your bill at is å£ so thats not bad \n","3103 thats y i said its bad dat all e gals know u wat u doing now \n","3279 badrith is only for chennaii will surely pick for usno competition for him \n","3415 uhhrmm isnt having tb test bad when youre sick \n","3483 tell my bad character which u dnt lik in me ill try to change in ltgt i ll add tat my new year resolution waiting for ur replybe frankgood morning \n","3567 sent me de webadres for geting salary slip \n","4001 did you get any gift this year i didnt get anything so bad \n","4071 loans for any purpose even if you have bad credit tenants welcome call noworriesloanscom on \n","4145 i canåõt wait for cornwall hope tonight isnåõt too bad as well but itåõs rock night shite anyway iåõm going for a kip now have a good night speak to you soon \n","4224 the world suffers a lot not because of the violence of bad people but because of the silence of good people \n","4535 never blame a day in ur life good days give u happiness bad days give u experience both are essential in life all are gods blessings good morning \n","4955 ahhjust woken uphad a bad dream about u thoso i dont like u right now i didnt know anything about comedy night but i guess im up for it \n","5057 gee your internet is really bad today eh \n","5086 omg if its not one thing its another my cat has worms when does this bad day end \n","5098 themobhit the link to get a premium pink panther game the new no from sugababes a crazy zebra animation or a badass hoody wallpaperall free \n","5146 oh unintentionally not bad timing great fingers the trains play along will give fifteen min warning \n","5217 well thats nice too bad i cant eat it \n","5286 im still pretty weak today bad day \n","5474 wheres mummys boy is he being good or bad is he being positive or negative why is mummy being made to wait hmm \n","5525 i want to tell you how bad i feel that basically the only times i text you lately are when i need drugs \n","\n"," search \n","31 True \n","458 True \n","565 True \n","732 True \n","1183 True \n","1300 True \n","1397 True \n","1594 True \n","1731 True \n","1793 True \n","1862 True \n","2125 True \n","2278 True \n","2444 True \n","2621 True \n","3022 True \n","3042 True \n","3103 True \n","3279 True \n","3415 True \n","3483 True \n","3567 True \n","4001 True \n","4071 True \n","4145 True \n","4224 True \n","4535 True \n","4955 True \n","5057 True \n","5086 True \n","5098 True \n","5146 True \n","5217 True \n","5286 True \n","5474 True \n","5525 True "],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
v1v2search
31hamyeah he got in at and was v apologetic n had fallen out and she was actin like spoilt child and he got caught up in that till but we wont go there not doing too badly cheers youTrue
458hamthat call cost which i guess isnt bad miss ya need ya want ya love yaTrue
565hamill call u mrw at ninish with my address that icky american freek wont stop callin me bad jen k ehTrue
732hamlol you wont feel bad when i use her money to take you out to a steak dinner dTrue
1183hamam i that much bad to avoid like thisTrue
1300hamthose cocksuckers if it makes you feel better ipads are worthless garbage novelty items and you should feel bad for even wanting oneTrue
1397hamthen we wait u lor no need feel bad larTrue
1594hamnever blame a day in ur life good days give u happiness bad days give u experience both are essential in life all are gods blessings good morningTrue
1731hamlol well quality aint bad at all so i aint complainingTrue
1793hamyou bad girl i can still remember themTrue
1862hamthe last thing i ever wanted to do was hurt you and i didnt think it would have youd laugh be embarassed delete the tag and keep going but as far as i knew it wasnt even up the fact that you even felt like i would do it to hurt you shows you really dont know me at all it was messy wednesday but it wasnt bad the problem i have with it is you have the time to clean it but you choose not to you skype you take pictures you sleep you want to go out i dont mind a few things here and there but when you dont make the bed when you throw laundry on top of it when i cant have a friend in the house because im embarassed that theres underwear and bras strewn on the bed pillows on the floor thats something else you used to be good about at least making the bedTrue
2125hamsorry im getting up now feel really bad totally rejected that kinda me thingTrue
2278hamhmmbad newshype park plaza studio takenonly left bedrmTrue
2444hamyes but i dont care i need you bad princessTrue
2621hamlol nah wasnt too bad thanks its good to b home but its been quite a reality check hows ur day been did u do anything with websiteTrue
3022hamyou are a very very very very bad girl or ladyTrue
3042hamyour bill at is å£ so thats not badTrue
3103hamthats y i said its bad dat all e gals know u wat u doing nowTrue
3279hambadrith is only for chennaii will surely pick for usno competition for himTrue
3415hamuhhrmm isnt having tb test bad when youre sickTrue
3483hamtell my bad character which u dnt lik in me ill try to change in ltgt i ll add tat my new year resolution waiting for ur replybe frankgood morningTrue
3567hamsent me de webadres for geting salary slipTrue
4001hamdid you get any gift this year i didnt get anything so badTrue
4071spamloans for any purpose even if you have bad credit tenants welcome call noworriesloanscom onTrue
4145hami canåõt wait for cornwall hope tonight isnåõt too bad as well but itåõs rock night shite anyway iåõm going for a kip now have a good night speak to you soonTrue
4224hamthe world suffers a lot not because of the violence of bad people but because of the silence of good peopleTrue
4535hamnever blame a day in ur life good days give u happiness bad days give u experience both are essential in life all are gods blessings good morningTrue
4955hamahhjust woken uphad a bad dream about u thoso i dont like u right now i didnt know anything about comedy night but i guess im up for itTrue
5057hamgee your internet is really bad today ehTrue
5086hamomg if its not one thing its another my cat has worms when does this bad day endTrue
5098spamthemobhit the link to get a premium pink panther game the new no from sugababes a crazy zebra animation or a badass hoody wallpaperall freeTrue
5146hamoh unintentionally not bad timing great fingers the trains play along will give fifteen min warningTrue
5217hamwell thats nice too bad i cant eat itTrue
5286hamim still pretty weak today bad dayTrue
5474hamwheres mummys boy is he being good or bad is he being positive or negative why is mummy being made to wait hmmTrue
5525hami want to tell you how bad i feel that basically the only times i text you lately are when i need drugsTrue
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{}}]},{"cell_type":"markdown","metadata":{"id":"cGXkaDIdFz_N"},"source":["# Document similarity with a given text"]},{"cell_type":"code","metadata":{"id":"uv-qhkpusaKt","colab":{"base_uri":"https://localhost:8080/","height":408},"outputId":"b7e42932-43ac-4e54-bddf-44333edbe6d7","executionInfo":{"status":"ok","timestamp":1668081966404,"user_tz":-60,"elapsed":237,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["from sklearn.feature_extraction.text import TfidfVectorizer\n","from sklearn.metrics.pairwise import cosine_similarity\n","\n","vectorizer = TfidfVectorizer()\n","docs_tfidf = vectorizer.fit_transform(corpus_df.Document)\n","\n","def get_tf_idf_query_similarity(vectorizer, docs_tfidf, query):\n"," \"\"\"\n"," vectorizer: TfIdfVectorizer model\n"," docs_tfidf: tfidf vectors for all docs\n"," query: query doc\n"," return: cosine similarity between query and all docs\n"," \"\"\"\n"," query_tfidf = vectorizer.transform([query])\n","\n"," cosineSimilarities = cosine_similarity(query_tfidf, docs_tfidf).flatten()\n"," return cosineSimilarities\n","\n","display(get_tf_idf_query_similarity(vectorizer, docs_tfidf, 'eggs'))\n","display(get_tf_idf_query_similarity(vectorizer, docs_tfidf, 'king eggs'))\n","display(get_tf_idf_query_similarity(vectorizer, docs_tfidf, 'green eggs'))\n","\n","corpus_df.head(10)"],"execution_count":12,"outputs":[{"output_type":"display_data","data":{"text/plain":["array([0. , 0. , 0. , 0.29521487, 0.38406851,\n"," 0. , 0. , 0. ])"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["array([0. , 0. , 0. , 0.45960141, 0.24669797,\n"," 0. , 0. , 0. ])"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["array([0. , 0. , 0. , 0.18962478, 0.59793204,\n"," 0. , 0. , 0. ])"]},"metadata":{}},{"output_type":"execute_result","data":{"text/plain":[" Document Category\n","0 The sky is blue and beautiful. weather\n","1 Love this blue and beautiful sky! weather\n","2 The quick brown fox jumps over the lazy dog. animals\n","3 A king's breakfast has sausages, ham, bacon, eggs, toast and beans food \n","4 I love green eggs, ham, sausages and bacon! food \n","5 The brown fox is quick and the blue dog is lazy! animals\n","6 The sky is very blue and the sky is very beautiful today weather\n","7 The dog is lazy but the brown fox is quick! animals"],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
DocumentCategory
0The sky is blue and beautiful.weather
1Love this blue and beautiful sky!weather
2The quick brown fox jumps over the lazy dog.animals
3A king's breakfast has sausages, ham, bacon, eggs, toast and beansfood
4I love green eggs, ham, sausages and bacon!food
5The brown fox is quick and the blue dog is lazy!animals
6The sky is very blue and the sky is very beautiful todayweather
7The dog is lazy but the brown fox is quick!animals
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{},"execution_count":12}]},{"cell_type":"markdown","metadata":{"id":"3PcZQyn0HnpV"},"source":["# Exercise 3\n","\n","1. Write a function that uses TfidfVectorizer with binary tf values\n","2. Write a function that creates TF-IDF vector of n-grams (try to run using n=2 and n=3)\n","3. Write a function that vectorizes (e.g. TF-IDF) a query and a set of documents, and print the distance between the query and each document and sort them\n","\n","> a. try to explore the BOW, TF-IDF, Count Vector \\\\\n","> b. try to compare different metrics [here is the list](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics.pairwise)\n","\n","*Hint: For point 1. and 2. use the documenation of TfidfVectorizer sklearn!*"]},{"cell_type":"code","metadata":{"id":"3TGSS4exHpZ6","executionInfo":{"status":"ok","timestamp":1668082621004,"user_tz":-60,"elapsed":243,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}}},"source":["from sklearn.feature_extraction.text import TfidfVectorizer\n","from sklearn.metrics.pairwise import cosine_similarity\n","\n","def binary_tfidf(corpus):\n"," tfidf_vectorizer = TfidfVectorizer(min_df=0., max_df=1., binary=True)\n"," docs_tfidf = tfidf_vectorizer.fit_transform(corpus)\n"," # get all unique words in the corpus\n"," vocab = tfidf_vectorizer.get_feature_names_out()\n"," # show document feature vectors\n"," tf_idf_df = pd.DataFrame(docs_tfidf.toarray(), columns=vocab)\n"," return tf_idf_df\n","\n","def ngram_tfidf(corpus, n, m):\n"," tfidf_vectorizer = TfidfVectorizer(min_df=0., max_df=1., ngram_range=(n,m))\n"," docs_tfidf = tfidf_vectorizer.fit_transform(corpus)\n"," # get all unique words in the corpus\n"," vocab = tfidf_vectorizer.get_feature_names_out()\n"," # show document feature vectors\n"," tf_idf_df = pd.DataFrame(docs_tfidf.toarray(), columns=vocab)\n"," return tf_idf_df\n","\n","binary_docs = binary_tfidf(corpus_df.Document)\n","bigram_docs = ngram_tfidf(corpus_df.Document, 2, 2)\n","trigram_docs = ngram_tfidf(corpus_df.Document, 3, 3)"],"execution_count":35,"outputs":[]},{"cell_type":"code","source":["from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n","from sklearn.metrics.pairwise import cosine_similarity, euclidean_distances\n","\n","def get_tf_idf_query_similarity(corpus, query):\n"," \n"," vectorizer = CountVectorizer() # just change this for the other methods\n"," docs_tfidf = vectorizer.fit_transform(corpus)\n"," query_tfidf = vectorizer.transform([query])\n","\n"," # This is for euclidean distances\n"," euclideanDistances = euclidean_distances(query_tfidf, docs_tfidf).flatten()\n"," return euclideanDistances\n","\n","display(get_tf_idf_query_similarity(corpus_df.Document, 'eggs'))\n","display(get_tf_idf_query_similarity(corpus_df.Document, 'king eggs'))\n","display(get_tf_idf_query_similarity(corpus_df.Document, 'green eggs'))\n","\n","corpus_df.head(10)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":408},"id":"o2FKwoseeNcp","executionInfo":{"status":"ok","timestamp":1668083206839,"user_tz":-60,"elapsed":259,"user":{"displayName":"Pranav Kasela","userId":"12416586000550437186"}},"outputId":"4c1ba8f0-c682-4012-af17-63af917cfe87"},"execution_count":40,"outputs":[{"output_type":"display_data","data":{"text/plain":["array([2.64575131, 2.64575131, 3.46410162, 3. , 2.44948974,\n"," 4. , 4.58257569, 3.87298335])"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["array([2.82842712, 2.82842712, 3.60555128, 2.82842712, 2.64575131,\n"," 4.12310563, 4.69041576, 4. ])"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["array([2.82842712, 2.82842712, 3.60555128, 3.16227766, 2.23606798,\n"," 4.12310563, 4.69041576, 4. ])"]},"metadata":{}},{"output_type":"execute_result","data":{"text/plain":[" Document Category\n","0 The sky is blue and beautiful. weather\n","1 Love this blue and beautiful sky! weather\n","2 The quick brown fox jumps over the lazy dog. animals\n","3 A king's breakfast has sausages, ham, bacon, eggs, toast and beans food \n","4 I love green eggs, ham, sausages and bacon! food \n","5 The brown fox is quick and the blue dog is lazy! animals\n","6 The sky is very blue and the sky is very beautiful today weather\n","7 The dog is lazy but the brown fox is quick! animals"],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
DocumentCategory
0The sky is blue and beautiful.weather
1Love this blue and beautiful sky!weather
2The quick brown fox jumps over the lazy dog.animals
3A king's breakfast has sausages, ham, bacon, eggs, toast and beansfood
4I love green eggs, ham, sausages and bacon!food
5The brown fox is quick and the blue dog is lazy!animals
6The sky is very blue and the sky is very beautiful todayweather
7The dog is lazy but the brown fox is quick!animals
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{},"execution_count":40}]}]}