June 28, 2016 at 12:41Chat bot algorithms based on a recurrent neural network and AIML language extensions
Today, it remains relevant to create programs that mimic human communication. The simplest communication model is the base of questions and answers to them [1]. In this case, the problem arises of describing the knowledge base and the implementation of the interpreter program. The markup language of the knowledge base can include question patterns and corresponding response templates, as well as the background of dialogues to them and the name of the corresponding topic of communication.
Chatbot can perform additional functions, for example, such as searching for music, pictures, facts, calculator, weather forecast, displaying currency rates. Most of these functions are implemented on the Internet and are available as an external API.
An alternative option to create a virtual interlocutor program is to use machine learning algorithms based on communication dialogs, namely artificial neural networks. A suitable ANN model is a recurrent neural network capable of storing, generalizing and predicting various sequences. In this paper, it is proposed to use indexes corresponding to the words in the knowledge base of questions and answers as sequence elements.
One of the knowledge base markup formats is the AIML (Artificial Intelligence Markup Language) markup language standard. The keywords in the language are category, pattern and template:
The category tag is the parent of the pattern and template tags that store question and answer templates. The random tag allows you to specify several answers to the question, chosen by the interpreter randomly. In the work, it is proposed to introduce additional tags corresponding to the history and topic of the conversation.
Several pattern tags allow you to describe different variants of questions that correspond to this category, which should be followed by the same answer options. The history tag stores the history of the dialog that precedes this question. The theme tag stores the name of the conversation topic. These tags allow the interpreter to choose a question pattern that matches the background of the dialogue and the topic of communication, which should affect the improvement of the quality of the chat bot communication simulation.
The markup language interpreter should allow you to find the most relevant questions in the following ways that complement each other:
The corresponding algorithm for choosing the best match is formed on the basis of the sorting algorithm:
As a result, the first element of the sorted array of matches is selected.
To improve the quality of searching for relevant answers, a morphological analyzer module was introduced into the word comparison procedure, which allows finding basic forms of words. Thus, the comparison is based on the basic forms of words, which eliminates the inconsistencies of the words associated with their declination.
It is proposed to use an algorithm for determining the names of those categories when only a part of them is defined.
A recurrent neural network is a type of multilayer perceptron in which signals from neurons of the output layer arrive at additional neurons of the input layer, the so-called context neurons.
The input signal vector goes to the group of neurons INPUT, on the group of neurons CONTEXT zero signal. Further, the signal propagates to the group of neurons of the hidden HIDDEN layer, and then it is converted by them and enters the neurons of the output OUTPUT layer. At the next iteration, along with the INPUT signal vector, the context group of neurons receives copies of the signals from the OUTPUT output layer of the last iteration (Fig. 1).
Fig. 1. General view of the structure of a recurrent neural network.
The structure of a recurrent neural network for remembering sentences has the following form:
The CONTEXT, INPUT, and OUTPUT layers each have one neuron, the output signal of which is mapped to the word index in the word set. Additionally, the word __end__ is entered corresponding to the end of the sentence [2]. The network is consistently learning sentences like:
“Hello. How are you? __end__ Hello. Fine. __end__ »
Answers to questions are received by a recurrent neural network according to the following scheme (Fig. 2).
Fig. 2. Getting an answer to a question by a recurrent neural network.
The volume of the HIDDEN layers should allow you to remember the entire set of sentences. The network is trained by the backpropagation method.
The chat bot program was implemented as an Android application. There are several chat bot modes available in the application:
As a morphological analyzer, we used the free JavaScript library for processing texts in Russian Az.js.
To create and train a recurrent neural network, the free JavaScript library RecurrentJS was used.
In all modes, auto-translation of answers to the user's question language is available based on the Yandex Translate API and Bing Translate API services. Also, the external search services use the Custom Search API and Bing Image Search API. Knowledge search is implemented on the basis of Google Knowledge Graph Search API. Music search is implemented based on the SoundCloud API. Calculator, weather forecast, exchange rates, time are implemented on the basis of Wolfram | Alpha API.
ChatBot is available on Google Play athttps://play.google.com/store/apps/details?id=svlab.chatbot .
Algorithms for building a chat bot based on advanced AIML markup and a recurrent neural network are considered. The AIML markup extension includes new theme and history tags to more efficiently search for relevant questions and answers according to the context of the dialogue. To determine the names of topics in categories, when only part of them is defined, it is proposed to use a heuristic algorithm for classifying topics. Using the module of the morphological analyzer of individual words and bringing them to the basic form allows you to improve the quality of the search for relevant answers. A recurrent neural network allows you to get answers to questions that were not in the knowledge base, using the network's ability to generalize. ChatBot is available for the Android platform in the Play Store.
Chatbot can perform additional functions, for example, such as searching for music, pictures, facts, calculator, weather forecast, displaying currency rates. Most of these functions are implemented on the Internet and are available as an external API.
An alternative option to create a virtual interlocutor program is to use machine learning algorithms based on communication dialogs, namely artificial neural networks. A suitable ANN model is a recurrent neural network capable of storing, generalizing and predicting various sequences. In this paper, it is proposed to use indexes corresponding to the words in the knowledge base of questions and answers as sequence elements.
Aiml
One of the knowledge base markup formats is the AIML (Artificial Intelligence Markup Language) markup language standard. The keywords in the language are category, pattern and template:
Привет! Как дела? Привет. Нормально. Привет. Отлично. Как у тебя? The category tag is the parent of the pattern and template tags that store question and answer templates. The random tag allows you to specify several answers to the question, chosen by the interpreter randomly. In the work, it is proposed to introduce additional tags corresponding to the history and topic of the conversation.
Чем занимаешься? Что делаешь? Читаю книгу. Смотрю телевизор. Привет! Как дела? Привет. Нормально. Дела Several pattern tags allow you to describe different variants of questions that correspond to this category, which should be followed by the same answer options. The history tag stores the history of the dialog that precedes this question. The theme tag stores the name of the conversation topic. These tags allow the interpreter to choose a question pattern that matches the background of the dialogue and the topic of communication, which should affect the improvement of the quality of the chat bot communication simulation.
The markup language interpreter should allow you to find the most relevant questions in the following ways that complement each other:
- 1. Search the entire phrase of a question based on a regular expression.
- 2. Search by the number of matching words in the question and patterns.
- 3. Search for matching current topics and category topics.
- 4. Search by the number of matching words in the current conversation prehistory and category prehistory.
The corresponding algorithm for choosing the best match is formed on the basis of the sorting algorithm:
sortMatches = allMatches.sort(function(a, b) {
if(a.pattern == inputText && b.pattern != inputText)
return -1;
if(b.pattern == inputText && a.pattern != inputText)
return 1;
if(a.matches < b.matches)
return 1;
if(a.matches > b.matches)
return -1;
if(a.theme == bot.theme && b.theme != bot.theme)
return -1;
if(b.theme == bot.theme && a.theme != bot.theme)
return 1;
if(a.historyMatches < b.historyMatches)
return 1;
if(a.historyMatches > b.historyMatches)
return -1;
return 0;
})
As a result, the first element of the sorted array of matches is selected.
To improve the quality of searching for relevant answers, a morphological analyzer module was introduced into the word comparison procedure, which allows finding basic forms of words. Thus, the comparison is based on the basic forms of words, which eliminates the inconsistencies of the words associated with their declination.
Theme classification algorithm
It is proposed to use an algorithm for determining the names of those categories when only a part of them is defined.
- 1. Let the set of categories in which themes are defined be T. The set of categories in which topics are not defined is D. The elements of these sets are associated with the union of the strings of all the values of patterns and category patterns.
- 2. A determination threshold is introduced, for example, p = 70%.
- 3. The elements of the set D are sequentially sorted. A subset V of the elements of the set T is selected, in which p the percentage of unique words is defined.
- 4. If V is empty, then the topic is determined by the line of an element of D.
- 5. If V is not empty, then an element is selected from V for which the ratio of the number of matching unique words to the total number of unique words in the line of this element is maximum. The corresponding theme of this element V is defined as the theme of element D.
Recurrent neural network
A recurrent neural network is a type of multilayer perceptron in which signals from neurons of the output layer arrive at additional neurons of the input layer, the so-called context neurons.
The input signal vector goes to the group of neurons INPUT, on the group of neurons CONTEXT zero signal. Further, the signal propagates to the group of neurons of the hidden HIDDEN layer, and then it is converted by them and enters the neurons of the output OUTPUT layer. At the next iteration, along with the INPUT signal vector, the context group of neurons receives copies of the signals from the OUTPUT output layer of the last iteration (Fig. 1).
Fig. 1. General view of the structure of a recurrent neural network.
The structure of a recurrent neural network for remembering sentences has the following form:
The CONTEXT, INPUT, and OUTPUT layers each have one neuron, the output signal of which is mapped to the word index in the word set. Additionally, the word __end__ is entered corresponding to the end of the sentence [2]. The network is consistently learning sentences like:
“Hello. How are you? __end__ Hello. Fine. __end__ »
Answers to questions are received by a recurrent neural network according to the following scheme (Fig. 2).
Fig. 2. Getting an answer to a question by a recurrent neural network.
The volume of the HIDDEN layers should allow you to remember the entire set of sentences. The network is trained by the backpropagation method.
Software implementation
The chat bot program was implemented as an Android application. There are several chat bot modes available in the application:
- 1) based on the external API www.pandorabots.com ;
- 2) based on the original generalization of the AIML language;
- 3) based on a recurrent neural network.
As a morphological analyzer, we used the free JavaScript library for processing texts in Russian Az.js.
To create and train a recurrent neural network, the free JavaScript library RecurrentJS was used.
In all modes, auto-translation of answers to the user's question language is available based on the Yandex Translate API and Bing Translate API services. Also, the external search services use the Custom Search API and Bing Image Search API. Knowledge search is implemented on the basis of Google Knowledge Graph Search API. Music search is implemented based on the SoundCloud API. Calculator, weather forecast, exchange rates, time are implemented on the basis of Wolfram | Alpha API.
ChatBot is available on Google Play athttps://play.google.com/store/apps/details?id=svlab.chatbot .
Conclusion
Algorithms for building a chat bot based on advanced AIML markup and a recurrent neural network are considered. The AIML markup extension includes new theme and history tags to more efficiently search for relevant questions and answers according to the context of the dialogue. To determine the names of topics in categories, when only part of them is defined, it is proposed to use a heuristic algorithm for classifying topics. Using the module of the morphological analyzer of individual words and bringing them to the basic form allows you to improve the quality of the search for relevant answers. A recurrent neural network allows you to get answers to questions that were not in the knowledge base, using the network's ability to generalize. ChatBot is available for the Android platform in the Play Store.
Literature
- 1. Provotar A.I., Klochko K.A. Features and problems of virtual communication using chat bots // Scientific works of Vinnitsa National Technical University. 2013. No. 3. P. 2.
- 2. Oriol Vinyals, Quoc Le A Neural Conversational Model // arXiv preprint arXiv: 1506.05869, 22 Jul 2015.