How we analyzed mobile app reviews using machine learning.
Recently, we, the Data Science Big Data Sbertech team, completed one interesting pilot in the field of machine learning: we tried to identify incidents in the work of Sberbank Online mobile based on the texts of user reviews in app stores. We took reviews from Google Play for the period from October 2014 to October 2017 (882,864 reviews), filtered 92,711 negatives (1-2 stars) and started work. What - we tell under the cut.
Immediately answer a possible question. The ratio of positive to negative reviews turned out to be in no way connected with incidents, therefore more cunning methods were needed. In addition, we wanted to identify the topics of the incidents.
To identify anomalies, we chose the Arima autoregressive integrated moving average model.
ARIMA (p, d, q) for a non-stationary time series Xt is:
Where εt is a stationary time series; c, ai, bj - model parameters; △ d - the operator of the difference of the time series of order d (sequential taking dtimes the differences of the first order - first from the time series, then from the obtained differences of the first order, then from the second order, etc.).
Trained model throughout the sample. To predict a normal level of negative reviews on the selected date, we used an interval of three months before it. The prediction was built a week ahead of the selected date, with discretization in one day.
Next formed a confidence level. For this, the calculated value of the confidence interval was added to the value of the prediction function for each date. It was taken for the third quantile of the distribution of all deviations of the prediction function from the actual amount of negative feedback for the three-month interval on which the prediction of the normal level is built.
The anomaly was recorded when the actual amount of negative feedback was above the confidence level (predicted value + confidence interval). Here is how it looks on the graph:
Red is the actual number of reviews. Yellow is the predicted value of the normal level with a confidence interval.
For further analysis, we selected 5 distinct peaks, which fell on February 27, 2017, 03/15/2017, September 14, 2017, 09/18/2017, 09/20/2017.
The detection of the incidents was carried out by clustering the texts of reviews with two models based on BIGARTM and Word2Vec with integrated Kmeans clustering.
The thematic model is a representation of the observed conditional distribution p (w | d) of terms (words or phrases) w in documents d of collection D :
where T is a set of topics;
- unknown distribution of terms in topic t ;
- unknown distribution of topics in the document d .
The parameters of the thematic model — matrices
and 
— are found by solving the problem of maximizing likelihood:
The main advantage over other probabilistic models is the presence of regularizers for smoothing, thinning, and decorrelation.
The model using Word2Vec is based on tensor algebra. Under it was compiled a dictionary of all the words in the reviews. The Word2Vec vector word representation has become the base space for clusters.
The algorithm is such that it seeks to minimize the total quadratic deviation of cluster points from the centers of these clusters:
k is the number of clusters, S i is the resulting clusters, i = 1.2 ... k , and μ i are the mass centers of all vectors x from the cluster S i . Now let's look at the results of the models with concrete examples.
Anomalous period with a peak 03/15/2017
BIGARTM :
Word2Vec + Kmeans:
As can be seen from the results, BIGARTM more clearly highlights the topics compared to Word2Vec + Kmeans. For the Word2Vec + Kmeans model, short non-informative reviews with words that did not have a vector representation (words with errors, profanity, etc.) were not taken into account.
Interestingly, during this period, incidents in the internal system ServiceDesk were not classified according to the version update problem and were processed separately. This means that in this case the model did the job better than the person.
And here is an example demonstrating the ability of our model to work in advance. On September 18, 2017, a massive problem was registered in ServiceDesk related to the inability to log into the application and make payments for some users. On the Arima graph, we see a strong surge of negative reviews on this date, as well as a slight release four days earlier.
We analyzed both of these emissions.
September 14, 2017:
September 18, 2017:
But how do these results relate to the data from ServiceDesk.
It is obvious that on September 18, a large flow of negative customer reviews and calls to the contact center was caused by the lack of a proper response to the increased background incident on September 14. This example clearly demonstrates that the model could help avoid a mass problem by recognizing it at an early stage.
It is also interesting that at the same time, the model discovered another problem (with the built-in antivirus), which was not highlighted against the background of the main problem. For her registration in ServiceDesk is missing.
Machine learning is good at analyzing user feedback. Sometimes even more accurate manual processing. This will help reduce the time to eliminate incidents of a mobile application and make it better.
Perhaps such methods are suitable not only for identifying incidents, but also for analyzing neutral and positive feedback in order to highlight priority user cases. This will help to develop the functionality of the application based on customer preferences at no additional cost for their collection and analysis. But this idea must also be checked ...
Immediately answer a possible question. The ratio of positive to negative reviews turned out to be in no way connected with incidents, therefore more cunning methods were needed. In addition, we wanted to identify the topics of the incidents.
To identify anomalies, we chose the Arima autoregressive integrated moving average model.
ARIMA (p, d, q) for a non-stationary time series Xt is:
Where εt is a stationary time series; c, ai, bj - model parameters; △ d - the operator of the difference of the time series of order d (sequential taking dtimes the differences of the first order - first from the time series, then from the obtained differences of the first order, then from the second order, etc.).
Trained model throughout the sample. To predict a normal level of negative reviews on the selected date, we used an interval of three months before it. The prediction was built a week ahead of the selected date, with discretization in one day.
Next formed a confidence level. For this, the calculated value of the confidence interval was added to the value of the prediction function for each date. It was taken for the third quantile of the distribution of all deviations of the prediction function from the actual amount of negative feedback for the three-month interval on which the prediction of the normal level is built.
The anomaly was recorded when the actual amount of negative feedback was above the confidence level (predicted value + confidence interval). Here is how it looks on the graph:
Red is the actual number of reviews. Yellow is the predicted value of the normal level with a confidence interval.
For further analysis, we selected 5 distinct peaks, which fell on February 27, 2017, 03/15/2017, September 14, 2017, 09/18/2017, 09/20/2017.
The detection of the incidents was carried out by clustering the texts of reviews with two models based on BIGARTM and Word2Vec with integrated Kmeans clustering.
Library BIGARTM (thematic model)
The thematic model is a representation of the observed conditional distribution p (w | d) of terms (words or phrases) w in documents d of collection D :
where T is a set of topics;
- unknown distribution of terms in topic t ; - unknown distribution of topics in the document d . The parameters of the thematic model — matrices
and — are found by solving the problem of maximizing likelihood: The main advantage over other probabilistic models is the presence of regularizers for smoothing, thinning, and decorrelation.
Model with Word2Vec
The model using Word2Vec is based on tensor algebra. Under it was compiled a dictionary of all the words in the reviews. The Word2Vec vector word representation has become the base space for clusters.
The algorithm is such that it seeks to minimize the total quadratic deviation of cluster points from the centers of these clusters:
k is the number of clusters, S i is the resulting clusters, i = 1.2 ... k , and μ i are the mass centers of all vectors x from the cluster S i . Now let's look at the results of the models with concrete examples.
Anomalous period with a peak 03/15/2017
BIGARTM :
| Cluster number | Tags | Cluster Feedback Examples (source vocabulary) |
| one | update version operation account work error can not install delete transfer return | “The old version is not working. I do not need an update. There is no space on the phone! Return the possibility of using the old version !!! " " I can not install the application at all, there is plenty of memory on the phone, more than 50%, but the application stubbornly writes that it is not possible to install, error 24! I fixed the bug, I used it before, I really liked it !!! " " After the update, it stopped working! A window appears with the message: "" An error occurred in the application “Sberbank” “and goodbye. I pulled the damn upgrade! Before the update, everything worked fine, completely satisfied. And in the course I do not have one such crap! ” |
| 2 | connection go write work cannot_ memory can not fly out constantly horrible delete problem | “I can’t enter the Application, enter the code, write, I can’t restore the secure connection! what the heck!? Or you are so forced to be updated! I was satisfied with the old update! Solve the problem! ” “ What did you do with a great program? Has nodded and regretted. Every 2 minutes, a window pops up with the text ““ the connection with the savings bank was interrupted ”.” I decided to remove and reinstall. Now it does not open at all. Immediately throws out of the program. Guys, return the old version » |
| 3 | sms do not come to enter the code card you need to thank you do not work out can not take off | “ SMS does not come to the phone with a password” “Does not register, writes that SMS will come with the code and nothing ....” “After the update, SMS comes in about entering, but the threshold flies out in 4 seconds. How to be? |
Word2Vec + Kmeans:
| Cluster number | Tags | Cluster Feedback Examples (source vocabulary) |
| one | update connection log in use update | “I can’t enter the Application, enter the code, write, I can’t restore the secure connection! what the heck!? Or you are so forced to be updated! I was satisfied with the old update! Solve the problem! ” “ Connection is broken, connect to another network - this is what I most often see when I want to go into the application. Updates always put immediately when they go out. If you can go, everything works fine. ” |
| 2 | sms to come in phone enter | “After the update, an SMS comes in about the entrance, but the threshold flies out in 4 seconds. How to be? " " SMS does not come to the phone with a password " " When registering, SMS does not come with the code ... " " It is not possible to enter. The code by SMS does not come. Correct everything as soon as possible. " |
| BIGARTM | Word2Vec + Kmeans |
 |  |
Problems identified during clustering:
|
Problems identified during clustering:
|
As can be seen from the results, BIGARTM more clearly highlights the topics compared to Word2Vec + Kmeans. For the Word2Vec + Kmeans model, short non-informative reviews with words that did not have a vector representation (words with errors, profanity, etc.) were not taken into account.
Interestingly, during this period, incidents in the internal system ServiceDesk were not classified according to the version update problem and were processed separately. This means that in this case the model did the job better than the person.
Early Recognition
And here is an example demonstrating the ability of our model to work in advance. On September 18, 2017, a massive problem was registered in ServiceDesk related to the inability to log into the application and make payments for some users. On the Arima graph, we see a strong surge of negative reviews on this date, as well as a slight release four days earlier.
We analyzed both of these emissions.
September 14, 2017:
| BIGARTM | Word2Vec + Kmeans |
 |  |
Problems identified during clustering:
|
Clustering Issue:
|
September 18, 2017:
| BIGARTM | Word2Vec + Kmeans |
 |  |
Problems identified during clustering:
|
Clustering Issue:
|
But how do these results relate to the data from ServiceDesk.
| Date of anomaly detection | Registered SM issue | Identifying the root of the problem with using BIGARTM | Identifying the root of the problem with using Word2Vec + Kmeans |
| 2017-09-14 00:00 - 24:00 | After updating the application, an increased background of calls from updated applications was formed. | The problem with the transfer of funds. The problem with the interface. The problem with entering the application after the update. The problem with the entrance to the application. User is not satisfied with the privacy policy. | The problem of working in the application after the update. The user is not satisfied with the privacy policy, there is also a problem with the transfer of funds. |
| 2017-09-18 00:00 - 24:00 | In the period from 09:23 to 15:20 Moscow time, for a part of the Bank's customers, an increase was recorded in the time of entry and operations in Sberbank Online. The incidents had virtually no impact on the work of clients using IOS. Increased number of customer calls to the Bank's contact center (up to 1369 additional calls in 15 minutes). | The problem with entering the application after updating the version. The application gives the user a message about the outdated version. The problem with the entrance to the application. The user enters the password several times, the application hangs. The problem of the application associated with the built-in antivirus. | The problem with entering a password and entering the application. Version update error. |
It is obvious that on September 18, a large flow of negative customer reviews and calls to the contact center was caused by the lack of a proper response to the increased background incident on September 14. This example clearly demonstrates that the model could help avoid a mass problem by recognizing it at an early stage.
It is also interesting that at the same time, the model discovered another problem (with the built-in antivirus), which was not highlighted against the background of the main problem. For her registration in ServiceDesk is missing.
Let's sum up
Machine learning is good at analyzing user feedback. Sometimes even more accurate manual processing. This will help reduce the time to eliminate incidents of a mobile application and make it better.
Perhaps such methods are suitable not only for identifying incidents, but also for analyzing neutral and positive feedback in order to highlight priority user cases. This will help to develop the functionality of the application based on customer preferences at no additional cost for their collection and analysis. But this idea must also be checked ...