Natural Language Processing Basics for Text
- Transfer
Natural language processing is now not used except in very conservative sectors. In most technological solutions, the recognition and processing of “human” languages has long been introduced: that is why the usual IVR with hard-coded response options is gradually becoming a thing of the past, chatbots are starting to communicate more adequately without the participation of a live operator, mail filters work with a bang, etc. How is the recognition of recorded speech, that is, text? Or rather, what will be the basis of modern recognition and processing techniques? Our today's adapted translation responds well to this - under the cut you will find a longride that will close the gaps on the basics of NLP. Enjoy reading!
Natural Language Processing (hereinafter - NLP) - natural language processing is a subsection of computer science and AI devoted to how computers analyze natural (human) languages. NLP allows the use of machine learning algorithms for text and speech.
For example, we can use NLP to create systems like speech recognition, generalization of documents, machine translation, spam detection, recognition of named entities, answers to questions, auto-complete, predictive text input, etc.
Today, many of us have speech recognition smartphones - they use NLP to understand our speech. Also, many people use laptops with built-in speech recognition in the OS.
Windows has a Cortana virtual assistant that recognizes speech. With Cortana, you can create reminders, open applications, send letters, play games, find out the weather, etc.
Siri is an assistant for Apple's OS: iOS, watchOS, macOS, HomePod, and tvOS. Many functions also work through voice control: call / write someone, send an email, set a timer, take a photo, etc.
A well-known email service knows how to detect spam so that it does not get into your inbox's inbox.
A platform from Google that allows you to create NLP bots. For example, you can make a pizza ordering bot that doesn't need an old-fashioned IVR to accept your order .
NLTK (Natural Language Toolkit) is the leading platform for creating NLP programs in Python. It has easy-to-use interfaces for many language corpuses , as well as libraries for word processing for classification, tokenization, stemming , markup , filtering and semantic reasoning . Well, also this is a free open source project that is being developed with the help of the community.
We will use this tool to show the basics of NLP. For all subsequent examples, I assume that NLTK is already imported; this can be done by the team
In this article we will cover topics:
Tokenization (sometimes segmentation) of sentences is the process of dividing a written language into component sentences. The idea looks pretty simple. In English and some other languages, we can isolate a sentence every time we find a certain punctuation mark - a period.
But even in English this task is not trivial, since the point is also used in abbreviations. The abbreviation table can greatly help during word processing to avoid misplacing sentence boundaries. In most cases, libraries are used for this, so you don’t really have to worry about implementation details.
Example:
Take a short text about a backgammon board game:
To make offers tokenization using NLTK, you can use the method
At the exit, we get 3 separate sentences:
Tokenization (sometimes segmentation) according to words is the process of dividing sentences into component words. In English and many other languages using a particular version of the Latin alphabet, a space is a good word separator.
However, problems may arise if we use only a space - in English, compound nouns are written differently and sometimes separated by spaces. And here libraries help us again.
Example:
Let's take the sentences from the previous example and apply the method to them
Conclusion:
Usually texts contain different grammatical forms of the same word, and one-root words may also occur. Lemmatization and stemming are aimed at bringing all word forms encountered to a single, normal vocabulary form.
Examples:
Reduction of different word forms to one:
The same, but with reference to the whole sentence:
Lemmatization and stemming are special cases of normalization and they differ.
Stemming is a crude heuristic process that cuts off “excess” from the root of words, often this leads to the loss of word-building suffixes.
Lemmatization is a more subtle process that uses vocabulary and morphological analysis to ultimately bring the word to its canonical form - the lemma.
The difference is that the stemmer (a specific implementation of the stemming algorithm - translator comment) operates without knowing the context and, accordingly, does not understand the difference between words that have different meanings depending on the part of speech. However, the Stemmers have their own advantages: they are easier to implement and they work faster. Plus, lower "accuracy" may not matter in some cases.
Examples:
Now that we know what the difference is, let's look at an example:
Conclusion:
Stop words are words that are thrown out of the text before / after text processing. When we apply machine learning to texts, such words can add a lot of noise, so you need to get rid of irrelevant words.
Stop words are usually understood by articles, interjections, unions, etc., which do not carry a semantic load. It should be understood that there is no universal list of stop words, it all depends on the particular case.
NLTK has a predefined list of stop words. Before first use, you'll need to download it:
Conclusion:
Consider how you can remove stop words from a sentence:
Conclusion:
If you are not familiar with list comprehensions, you can find out more here . Here is another way to achieve the same result:
However, remember that list comprehensions are faster because they are optimized - the interpreter reveals a predictive pattern during the loop.
You may ask why we converted the list to many . A set is an abstract data type that can store unique values in an undefined order. Multiple searches are much faster than list searches. For a small number of words, this does not matter, but if we are talking about a large number of words, it is strongly recommended to use sets. If you want to know a little more about the time it takes to perform various operations, look at this wonderful cheat sheet .
A regular expression (regular, regexp, regex) is a sequence of characters that defines a search pattern. For example:
Excerpt from Python documentation :
The re module in Python represents regular expression operations. We can use the re.sub function to replace everything that fits the search pattern with the specified string. So you can replace all non-words with spaces:
Conclusion:
Regulars are a powerful tool that can be used to create much more complex patterns. If you want to know more about regular expressions, then I can recommend these 2 web applications: regex , regex101 .
Machine learning algorithms cannot directly work with raw text, so you need to convert the text to sets of numbers (vectors). This is called feature extraction .
A word bag is a popular and simple feature extraction technique used when working with text. It describes the occurrences of each word in the text.
To use the model, we need:
Any information about the order or structure of words is ignored. That is why it is called a BAG of words. This model tries to understand whether a familiar word appears in a document, but does not know where exactly it occurs.
Intuition suggests that similar documents have similar content . Also, thanks to the content, we can learn something about the meaning of the document.
Example:
Consider the steps for creating this model. We use only 4 sentences to understand how the model works. In real life, you will encounter more data.
Imagine that this is our data and we want to load it as an array:
To do this, just read the file and divide by line:
Conclusion:
We will collect all unique words from 4 loaded sentences, ignoring case, punctuation and one-character tokens. This will be our dictionary (famous words).
To create a dictionary, you can use the CountVectorizer class from the sklearn library. Go to the next step.
Next, we need to evaluate the words in the document. At this step, our goal is to turn raw text into a set of numbers. After that, we use these sets as input to the machine learning model. The simplest scoring method is to note the presence of words, that is, put 1 if there is a word and 0 if it is absent.
Now we can create a bag of words using the aforementioned CountVectorizer class.
Conclusion:
These are our suggestions. Now we see how the “bag of words” model works.
The complexity of this model is how to determine the dictionary and how to count the occurrence of words.
When the dictionary size increases, the document vector also grows. In the example above, the length of the vector is equal to the number of known words.
In some cases, we can have an incredibly large amount of data and then the vector can consist of thousands or millions of elements. Moreover, each document can contain only a small part of the words from the dictionary.
As a result, there will be many zeros in the vector representation. Vectors with many zeros are called sparse vectors, they require more memory and computational resources.
However, we can reduce the number of known words when we use this model to reduce the requirements for computing resources. To do this, you can use the same techniques that we already considered before creating a bag of words:
Another, more complicated way to create a dictionary is to use grouped words. This will resize the dictionary and give the bag of words more details about the document. This approach is called the " N-gram ."
N-gram is a sequence of any entities (words, letters, numbers, numbers, etc.). In the context of linguistic bodies, the N-gram is usually understood as a sequence of words. A unigram is one word, a bigram is a sequence of two words, a trigram is three words, and so on. The number N indicates how many grouped words are included in the N-gram. Not all possible N-grams fall into the model, but only those that appear in the case.
Example:
Consider the following sentence:
Here are his bigrams:
As you can see, a bag of bigrams is a more effective approach than a bag of words.
Evaluating (scoring) words
When a dictionary is created, you should evaluate the presence of words. We have already considered a simple, binary approach (1 - there is a word, 0 - there is no word).
There are other methods:
Frequency scoring has a problem: words with the highest frequency have, respectively, the highest rating. In these words, there may not be as much informational gain for the model as in less frequent words. One way to rectify the situation is to lower the word score, which is often found in all similar documents . This is called TF-IDF .
TF-IDF (short for term frequency - inverse document frequency) is a statistical measure for assessing the importance of a word in a document that is part of a collection or corpus.
Scoring by TF-IDF grows in proportion to the frequency of occurrence of a word in a document, but this is offset by the number of documents containing this word.
Scoring formula for the word X in document Y:
Formula TF-IDF. Source: filotechnologia.blogspot.com/2014/01/a-simple-java-class-for-tfidf-scoring.html
TF (term frequency - word frequency) is the ratio of the number of occurrences of a word to the total number of words in a document.
IDF (inverse document frequency) is the inverse of the frequency with which a certain word occurs in collection documents.
As a result, you can calculate TF-IDF for the word term like this:
Example:
You can use the TfidfVectorizer class from the sklearn library to calculate TF-IDF. Let's do this with the same messages that we used in the bag of words example.
Code:
Conclusion:
This article has covered the basics of NLP for text, namely:
Fine! Now that you know the basics of feature extraction, you can use features as input to machine learning algorithms.
If you want to see all the described concepts in one big example, then here you are .
What is Natural Language Processing?
Natural Language Processing (hereinafter - NLP) - natural language processing is a subsection of computer science and AI devoted to how computers analyze natural (human) languages. NLP allows the use of machine learning algorithms for text and speech.
For example, we can use NLP to create systems like speech recognition, generalization of documents, machine translation, spam detection, recognition of named entities, answers to questions, auto-complete, predictive text input, etc.
Today, many of us have speech recognition smartphones - they use NLP to understand our speech. Also, many people use laptops with built-in speech recognition in the OS.
Examples
Cortana
Windows has a Cortana virtual assistant that recognizes speech. With Cortana, you can create reminders, open applications, send letters, play games, find out the weather, etc.
Siri
Siri is an assistant for Apple's OS: iOS, watchOS, macOS, HomePod, and tvOS. Many functions also work through voice control: call / write someone, send an email, set a timer, take a photo, etc.
Gmail
A well-known email service knows how to detect spam so that it does not get into your inbox's inbox.
Dialogflow
A platform from Google that allows you to create NLP bots. For example, you can make a pizza ordering bot that doesn't need an old-fashioned IVR to accept your order .
NLTK Python Library
NLTK (Natural Language Toolkit) is the leading platform for creating NLP programs in Python. It has easy-to-use interfaces for many language corpuses , as well as libraries for word processing for classification, tokenization, stemming , markup , filtering and semantic reasoning . Well, also this is a free open source project that is being developed with the help of the community.
We will use this tool to show the basics of NLP. For all subsequent examples, I assume that NLTK is already imported; this can be done by the team
import nltkNLP Basics for Text
In this article we will cover topics:
- Tokenization by offers.
- Tokenization by words.
- Lemmatization and stamping of the text.
- Stop words.
- Regular expressions.
- Bag of words .
- TF-IDF .
1. Tokenization by offers
Tokenization (sometimes segmentation) of sentences is the process of dividing a written language into component sentences. The idea looks pretty simple. In English and some other languages, we can isolate a sentence every time we find a certain punctuation mark - a period.
But even in English this task is not trivial, since the point is also used in abbreviations. The abbreviation table can greatly help during word processing to avoid misplacing sentence boundaries. In most cases, libraries are used for this, so you don’t really have to worry about implementation details.
Example:
Take a short text about a backgammon board game:
Backgammon is one of the oldest known board games. Its history can be traced back nearly 5,000 years to archeological discoveries in the Middle East. It is a two player game where each player has fifteen checkers which move between twenty-four points according to the roll of two dice.To make offers tokenization using NLTK, you can use the method
nltk.sent_tokenizeAt the exit, we get 3 separate sentences:
Backgammon is one of the oldest known board games.
Its history can be traced back nearly 5,000 years to archeological discoveries in the Middle East.
It is a two player game where each player has fifteen checkers which move between twenty-four points according to the roll of two dice.2. Tokenization according to words
Tokenization (sometimes segmentation) according to words is the process of dividing sentences into component words. In English and many other languages using a particular version of the Latin alphabet, a space is a good word separator.
However, problems may arise if we use only a space - in English, compound nouns are written differently and sometimes separated by spaces. And here libraries help us again.
Example:
Let's take the sentences from the previous example and apply the method to them
nltk.word_tokenizeConclusion:
['Backgammon', 'is', 'one', 'of', 'the', 'oldest', 'known', 'board', 'games', '.']
['Its', 'history', 'can', 'be', 'traced', 'back', 'nearly', '5,000', 'years', 'to', 'archeological', 'discoveries', 'in', 'the', 'Middle', 'East', '.']
['It', 'is', 'a', 'two', 'player', 'game', 'where', 'each', 'player', 'has', 'fifteen', 'checkers', 'which', 'move', 'between', 'twenty-four', 'points', 'according', 'to', 'the', 'roll', 'of', 'two', 'dice', '.']3. Lemmatization and stamping of the text
Usually texts contain different grammatical forms of the same word, and one-root words may also occur. Lemmatization and stemming are aimed at bringing all word forms encountered to a single, normal vocabulary form.
Examples:
Reduction of different word forms to one:
dog, dogs, dog’s, dogs’ => dogThe same, but with reference to the whole sentence:
the boy’s dogs are different sizes => the boy dog be differ sizeLemmatization and stemming are special cases of normalization and they differ.
Stemming is a crude heuristic process that cuts off “excess” from the root of words, often this leads to the loss of word-building suffixes.
Lemmatization is a more subtle process that uses vocabulary and morphological analysis to ultimately bring the word to its canonical form - the lemma.
The difference is that the stemmer (a specific implementation of the stemming algorithm - translator comment) operates without knowing the context and, accordingly, does not understand the difference between words that have different meanings depending on the part of speech. However, the Stemmers have their own advantages: they are easier to implement and they work faster. Plus, lower "accuracy" may not matter in some cases.
Examples:
- The word good is a lemma for the word better. Stemmer will not see this connection, since here you need to consult the dictionary.
- The word play is the basic form of the word playing. Here both stemming and lemmatization will cope.
- The word meeting can be either a normal form of a noun or a form of the verb to meet, depending on the context. Unlike stemming, lemmatization will try to choose the right lemma based on context.
Now that we know what the difference is, let's look at an example:
Conclusion:
Stemmer: seen
Lemmatizer: see
Stemmer: drove
Lemmatizer: drive4. Stop words
Stop words are words that are thrown out of the text before / after text processing. When we apply machine learning to texts, such words can add a lot of noise, so you need to get rid of irrelevant words.
Stop words are usually understood by articles, interjections, unions, etc., which do not carry a semantic load. It should be understood that there is no universal list of stop words, it all depends on the particular case.
NLTK has a predefined list of stop words. Before first use, you'll need to download it:
nltk.download(“stopwords”). After downloading, you can import the package stopwordsand look at the words themselves:Conclusion:
['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]Consider how you can remove stop words from a sentence:
Conclusion:
['Backgammon', 'one', 'oldest', 'known', 'board', 'games', '.']If you are not familiar with list comprehensions, you can find out more here . Here is another way to achieve the same result:
However, remember that list comprehensions are faster because they are optimized - the interpreter reveals a predictive pattern during the loop.
You may ask why we converted the list to many . A set is an abstract data type that can store unique values in an undefined order. Multiple searches are much faster than list searches. For a small number of words, this does not matter, but if we are talking about a large number of words, it is strongly recommended to use sets. If you want to know a little more about the time it takes to perform various operations, look at this wonderful cheat sheet .
5. Regular expressions.
A regular expression (regular, regexp, regex) is a sequence of characters that defines a search pattern. For example:
- . - any character except line feed;
- \ w is one word;
- \ d - one digit;
- \ s is one space;
- \ W is one NON-Word;
- \ D - one non-digit;
- \ S - one non-space;
- [abc] - finds any of the specified characters match any of a, b, or c;
- [^ abc] - finds any character except the specified ones;
- [ag] - Finds a character in the range from a to g.
Excerpt from Python documentation :
Regular expressions use a backslashWe can use regulars to further filter our text. For example, you can remove all characters that are not words. In many cases, punctuation is not needed and is easy to remove with the help of regulars.(\)to denote special forms or to allow the use of special characters. This contradicts the use of backslash in Python: for example, to literally denote a backslash, you must write'\\\\'as a search pattern because the regular expression should look like\\where each backslash should be escaped.
The solution is to use raw string notation for search patterns; backslashes will not be specially processed if used with a prefix‘r’. Thus,r”\n”this is a string with two characters(‘\’ и ‘n’), and“\n”is a string with one character (line feed).
The re module in Python represents regular expression operations. We can use the re.sub function to replace everything that fits the search pattern with the specified string. So you can replace all non-words with spaces:
Conclusion:
'The development of snowboarding was inspired by skateboarding sledding surfing and skiing 'Regulars are a powerful tool that can be used to create much more complex patterns. If you want to know more about regular expressions, then I can recommend these 2 web applications: regex , regex101 .
6. Bag of words
Machine learning algorithms cannot directly work with raw text, so you need to convert the text to sets of numbers (vectors). This is called feature extraction .
A word bag is a popular and simple feature extraction technique used when working with text. It describes the occurrences of each word in the text.
To use the model, we need:
- Define a dictionary of known words (tokens).
- Choose the degree of presence of famous words.
Any information about the order or structure of words is ignored. That is why it is called a BAG of words. This model tries to understand whether a familiar word appears in a document, but does not know where exactly it occurs.
Intuition suggests that similar documents have similar content . Also, thanks to the content, we can learn something about the meaning of the document.
Example:
Consider the steps for creating this model. We use only 4 sentences to understand how the model works. In real life, you will encounter more data.
1. Download data
Imagine that this is our data and we want to load it as an array:
I like this movie, it's funny.
I hate this movie.
This was awesome! I like it.
Nice one. I love it.To do this, just read the file and divide by line:
Conclusion:
["I like this movie, it's funny.", 'I hate this movie.', 'This was awesome! I like it.', 'Nice one. I love it.']2. Define a dictionary
We will collect all unique words from 4 loaded sentences, ignoring case, punctuation and one-character tokens. This will be our dictionary (famous words).
To create a dictionary, you can use the CountVectorizer class from the sklearn library. Go to the next step.
3. Create document vectors
Next, we need to evaluate the words in the document. At this step, our goal is to turn raw text into a set of numbers. After that, we use these sets as input to the machine learning model. The simplest scoring method is to note the presence of words, that is, put 1 if there is a word and 0 if it is absent.
Now we can create a bag of words using the aforementioned CountVectorizer class.
Conclusion:
These are our suggestions. Now we see how the “bag of words” model works.
A few words about the bag of words
The complexity of this model is how to determine the dictionary and how to count the occurrence of words.
When the dictionary size increases, the document vector also grows. In the example above, the length of the vector is equal to the number of known words.
In some cases, we can have an incredibly large amount of data and then the vector can consist of thousands or millions of elements. Moreover, each document can contain only a small part of the words from the dictionary.
As a result, there will be many zeros in the vector representation. Vectors with many zeros are called sparse vectors, they require more memory and computational resources.
However, we can reduce the number of known words when we use this model to reduce the requirements for computing resources. To do this, you can use the same techniques that we already considered before creating a bag of words:
- ignoring the case of words;
- ignoring punctuation;
- ejecting stop words;
- reduction of words to their basic forms (lemmatization and stemming);
- correction of misspelled words.
Another, more complicated way to create a dictionary is to use grouped words. This will resize the dictionary and give the bag of words more details about the document. This approach is called the " N-gram ."
N-gram is a sequence of any entities (words, letters, numbers, numbers, etc.). In the context of linguistic bodies, the N-gram is usually understood as a sequence of words. A unigram is one word, a bigram is a sequence of two words, a trigram is three words, and so on. The number N indicates how many grouped words are included in the N-gram. Not all possible N-grams fall into the model, but only those that appear in the case.
Example:
Consider the following sentence:
The office building is open todayHere are his bigrams:
- the office
- office building
- building is
- is open
- open today
As you can see, a bag of bigrams is a more effective approach than a bag of words.
Evaluating (scoring) words
When a dictionary is created, you should evaluate the presence of words. We have already considered a simple, binary approach (1 - there is a word, 0 - there is no word).
There are other methods:
- Amount. It is calculated how many times each word appears in the document.
- Frequency. It is calculated how often each word occurs in the text (in relation to the total number of words).
7. TF-IDF
Frequency scoring has a problem: words with the highest frequency have, respectively, the highest rating. In these words, there may not be as much informational gain for the model as in less frequent words. One way to rectify the situation is to lower the word score, which is often found in all similar documents . This is called TF-IDF .
TF-IDF (short for term frequency - inverse document frequency) is a statistical measure for assessing the importance of a word in a document that is part of a collection or corpus.
Scoring by TF-IDF grows in proportion to the frequency of occurrence of a word in a document, but this is offset by the number of documents containing this word.
Scoring formula for the word X in document Y:
Formula TF-IDF. Source: filotechnologia.blogspot.com/2014/01/a-simple-java-class-for-tfidf-scoring.html
TF (term frequency - word frequency) is the ratio of the number of occurrences of a word to the total number of words in a document.
IDF (inverse document frequency) is the inverse of the frequency with which a certain word occurs in collection documents.
As a result, you can calculate TF-IDF for the word term like this:
Example:
You can use the TfidfVectorizer class from the sklearn library to calculate TF-IDF. Let's do this with the same messages that we used in the bag of words example.
I like this movie, it's funny.
I hate this movie.
This was awesome! I like it.
Nice one. I love it.Code:
Conclusion:
Conclusion
This article has covered the basics of NLP for text, namely:
- NLP allows the use of machine learning algorithms for text and speech;
- NLTK (Natural Language Toolkit) - a leading platform for creating NLP-programs in Python;
- proposal tokenization is the process of dividing a written language into component sentences;
- word tokenization is the process of dividing sentences into component words;
- Lemmatization and stemming are aimed at bringing all the word forms encountered to a single, normal vocabulary form;
- stop words are words that are thrown out of the text before / after text processing;
- regex (regex, regexp, regex) is a sequence of characters that defines a search pattern;
- a bag of words is a popular and simple feature extraction technique used when working with text. It describes the occurrences of each word in the text.
Fine! Now that you know the basics of feature extraction, you can use features as input to machine learning algorithms.
If you want to see all the described concepts in one big example, then here you are .
