Leave a request and we will reply you online within 1 minute. Or how we walked away from OTRS
Good afternoon, my name is Alexander Ulanov, I am a testing engineer at Odnoklassniki. In this article, I would like to talk about one of the projects in which I participated. I’ll warn you right away: in the article you will not find any discoveries or complex approaches to the testing methodology. However, the development and testing process described in the article may be interesting or even useful for someone.
OTRS is a request processing system that allows organizations involved in technical support of any projects to work together to solve user problems. The system is written in Perl and was supported by developers until November 2017.
It was the OTRS system that we used for many years to process all calls from users. The decision to use OTRS was made many years ago. Then there was a simple task: to find a ready-made solution and quickly adapt it to our goals. And it was terrible, I will explain why:
The idea of a complete departure from OTRS began to develop in the company since 2015. Key task: to get rid of the use of mail and provide the ability to quickly and timely respond to users without a multiple increase in the number of support staff. The most obvious solution is an online support service.
Also, no one wanted to spend resources on a completely new system. It is more logical to use your internal services, the functioning of which already has experience and test coverage. Plus a purely technical requirement: to be able to do updates to the new system at any time and not depend on other teams. What promised us flexibility in development and testing.
Around the same period, OK.ru saw a global modernization of the messaging system. It was on the basis of the new messaging that it was decided to implement the new online-support system (hereinafter referred to as “oSupp”).
In the early stages of developing oSupp and as a result of numerous meetings, we saw a number of problems. The main one is that it is impossible to immediately switch all work from OTRS to oSupp at once:
We decided that the best way is to implement and implement oSupp in three stages:
To test the prototype, a test environment was created with the ability to generate calls from test users and the ability to connect oSupp to the OK.ru test environment. It should be noted here that the new system was designed in such a way that nothing changed on the part of the user until the moment of direct communication with the operator. The familiar “Help” section has remained, the recognizable typing of topics on the site and the FAQ has been preserved.
Up to this point, nothing has changed in terms of testing. All test plans and coverage by autotests did not change. U - convenience!
After the user found the topic of the appeal and filled out the required fields, the message fell into the oSupp system, and the user had a chat with one of the support employees right in the Messages section. The dialogue started instantly when there was a free operator in the system. Otherwise, the chat was also created, and the user received an automatic first message "The operator will contact you shortly."
Again - a minimum of changes in the testing process. This is a regular chat, and for him it was not necessary to draw up new test plans. U - convenience!
But to ensure the work of employees, the system was created from scratch. What was needed was a fast system that would allow simultaneous work for several dozen employees online and which would be able to process thousands of calls per day.
Development and testing were iterative. About once a week we added new functional modules to the system, which allowed us not to save tasks and gradually cover the system with tests. I compiled text cases for testing, while simultaneously highlighting those that could be automated. The process of developing a fully functioning prototype took us about five months.
As a result, we got a web-service where an operator-operator could log in and accept requests from users for processing. For a group of support employees, we performed A / B testing several times to understand at this stage how convenient and understandable they are to work in the new system. It was a rewarding experience. We were able to test the oSupp system on the artificially created load of our test environment. However, this did not guarantee that in the real world we would not run into problems.
In February 2017, we began to introduce a new system in the product environment. As I wrote earlier, when contacting customer support, the user must select the topic of contact. For the first tests of the system on real users, we selected several topics for which the load was minimal. The system worked in this form for about a week. A / B testing was not required for users, as there were practically no changes to the functionality of the portal. On the oSupp side, the three online operators handled the load. We quickly tracked problems that could be fixed immediately, as The new oSupp system was no longer tied to updates of the main portal. Now we could update the system and fix problems when we wanted to. U - convenience!
Gradually, we increased the number of topics processed through the new system. With such a smooth deployment, we not only could control the load, but gradually trained support staff to work with the new system without the need to distract them from the rest of the work. By the middle of summer 2017, about 35% of the requests were transferred to processing by the online service. Already a third of our staff have worked through oSupp. The system was covered with documentation, test plans, and the minimum necessary self-tests, which were launched when the system was updated. By December 2017, we transferred all applications from authorized users online.
The next important step is to provide online support for users who experience authentication problems. It's about those who forgot the password, was hacked, blocked, or just users with non-standard calls. The key technical problem at this stage - the OK.ru message system has historically been designed for correspondence like “OK user (support operator in our case) - OK user”. But we needed to ensure the work of the situation “user OK (operator) - unauthorized user”.
The support team began to work closely with the OK.ru message team. As a result, solutions were invented that made it possible directly on the OK.ru login screen to receive online chat with a support service operator. At the same time, the already existing and commissioned oSupp system did not require changes. And the new essence of online chat worked on the modified OK.ru messaging engine. U - convenience!
We started implementing this decision in early 2018. In terms of testing, there were also no special difficulties, as Such a chat had a minimal set of functionality. First of all, we launched support for complaints from unauthorized users from the web. At the moment, support for such complaints from mobile applications is being implemented.
From the state of the prototype to a fully operational service, oSupp moved to the summer of 2018. As of March 2019, 60% of requests are processed through online support. The staff was not changed. 90 support service employees online cover about 6,000 complaints from users daily. During peak activity on the portal, waiting for a response from the operator takes up to 10 minutes. Under normal workload, operators respond within 1-2 minutes. Thanks to the switch to live online communication, successful restoration of user access to the profile has grown by 15%.
Now we have 100% of our own product, we are free to modify it as anytime. The team is no longer dependent on updates from the OK.ru portal. The new application processing system is written in Java, which implements many OK.ru services. The testing process of the oSupp system is fully covered by test cases, and with updates it is insured by automatic testing.
I must note that we have not completely abandoned OTRS. The offline support system was frozen in mid-2017, and is now playing the role of insurance. There are calls from dishonest users who like to spam aimlessly at the support service. OTRS also processes calls that, for specific reasons, users cannot be processed online. Now the plans of our team include the implementation of processing calls from mobile applications and the introduction of a chatbot system.
However, our main goal was achieved - we are moving away from the use of mail. Now, to solve problems, our users receive quick and timely answers.
Our main achievement: the user has significantly reduced the time it takes to wait for a response and get help.
Episode 1: Fear and Loathing in OTRS
OTRS is a request processing system that allows organizations involved in technical support of any projects to work together to solve user problems. The system is written in Perl and was supported by developers until November 2017.
It was the OTRS system that we used for many years to process all calls from users. The decision to use OTRS was made many years ago. Then there was a simple task: to find a ready-made solution and quickly adapt it to our goals. And it was terrible, I will explain why:
- we used a customized version of the OTRS system for our needs and therefore updating it (until the fall of 2017) was, to put it mildly, difficult;
- over time, we began to understand that the system rests on its “ceiling” load and ceases to cope with the number of applications that come from our users;
- the OTRS system was integrated into the general infrastructure in such a way that it completely depended on the updates of the OK.ru social network. Which made it possible to update OTRS only once a week;
- a staff of 90 support staff could provide answers to complaints only within a few hours;
- testing of each update of the system from our side turned into a nightmare, because even the smallest changes could break the system in any of the nodes. When it came to adding new functionality, the development and testing process turned into weeks of debugging the system;
- OTRS itself is written in Perl. A language that was practically not used in OK.ru projects, which imposed additional problems;
- and, perhaps, the main problem is MAIL! OTRS handled complaints in the form of email. And this is in 2016-2017! Hence, tons of settings, features of working with mail services and long waiting for answers for users, which are inevitable when communicating with support via mail.
Episode 2: What We Wanted To Get
The idea of a complete departure from OTRS began to develop in the company since 2015. Key task: to get rid of the use of mail and provide the ability to quickly and timely respond to users without a multiple increase in the number of support staff. The most obvious solution is an online support service.
Also, no one wanted to spend resources on a completely new system. It is more logical to use your internal services, the functioning of which already has experience and test coverage. Plus a purely technical requirement: to be able to do updates to the new system at any time and not depend on other teams. What promised us flexibility in development and testing.
Around the same period, OK.ru saw a global modernization of the messaging system. It was on the basis of the new messaging that it was decided to implement the new online-support system (hereinafter referred to as “oSupp”).
Episode 3: Planning
In the early stages of developing oSupp and as a result of numerous meetings, we saw a number of problems. The main one is that it is impossible to immediately switch all work from OTRS to oSupp at once:
- it was necessary to fully evaluate how the new system copes with the technical load (recall, we are talking about processing 10-12K complaints per day);
- It was necessary to train employees to work in the new system;
- It was necessary to conduct A / B testing on both users and support employees and evaluate how they will cope with the load;
- technical limitation: OK.ru messaging has never worked with unauthorized users before. And in the work of support, communication with unauthorized users is almost a third of all thousands of complaints;
- To test the system, you need to raise an environment that is as similar as possible to a real one, but without access to live users.
We decided that the best way is to implement and implement oSupp in three stages:
- Prototype development and testing.
- Implementing oSupp to process calls from authorized OK.ru users.
- Full coverage of all calls with the new oSupp system.
Episode 4: Testing and Commissioning
To test the prototype, a test environment was created with the ability to generate calls from test users and the ability to connect oSupp to the OK.ru test environment. It should be noted here that the new system was designed in such a way that nothing changed on the part of the user until the moment of direct communication with the operator. The familiar “Help” section has remained, the recognizable typing of topics on the site and the FAQ has been preserved.
Up to this point, nothing has changed in terms of testing. All test plans and coverage by autotests did not change. U - convenience!
After the user found the topic of the appeal and filled out the required fields, the message fell into the oSupp system, and the user had a chat with one of the support employees right in the Messages section. The dialogue started instantly when there was a free operator in the system. Otherwise, the chat was also created, and the user received an automatic first message "The operator will contact you shortly."
Again - a minimum of changes in the testing process. This is a regular chat, and for him it was not necessary to draw up new test plans. U - convenience!
But to ensure the work of employees, the system was created from scratch. What was needed was a fast system that would allow simultaneous work for several dozen employees online and which would be able to process thousands of calls per day.
Development and testing were iterative. About once a week we added new functional modules to the system, which allowed us not to save tasks and gradually cover the system with tests. I compiled text cases for testing, while simultaneously highlighting those that could be automated. The process of developing a fully functioning prototype took us about five months.
As a result, we got a web-service where an operator-operator could log in and accept requests from users for processing. For a group of support employees, we performed A / B testing several times to understand at this stage how convenient and understandable they are to work in the new system. It was a rewarding experience. We were able to test the oSupp system on the artificially created load of our test environment. However, this did not guarantee that in the real world we would not run into problems.
In February 2017, we began to introduce a new system in the product environment. As I wrote earlier, when contacting customer support, the user must select the topic of contact. For the first tests of the system on real users, we selected several topics for which the load was minimal. The system worked in this form for about a week. A / B testing was not required for users, as there were practically no changes to the functionality of the portal. On the oSupp side, the three online operators handled the load. We quickly tracked problems that could be fixed immediately, as The new oSupp system was no longer tied to updates of the main portal. Now we could update the system and fix problems when we wanted to. U - convenience!
Gradually, we increased the number of topics processed through the new system. With such a smooth deployment, we not only could control the load, but gradually trained support staff to work with the new system without the need to distract them from the rest of the work. By the middle of summer 2017, about 35% of the requests were transferred to processing by the online service. Already a third of our staff have worked through oSupp. The system was covered with documentation, test plans, and the minimum necessary self-tests, which were launched when the system was updated. By December 2017, we transferred all applications from authorized users online.
The next important step is to provide online support for users who experience authentication problems. It's about those who forgot the password, was hacked, blocked, or just users with non-standard calls. The key technical problem at this stage - the OK.ru message system has historically been designed for correspondence like “OK user (support operator in our case) - OK user”. But we needed to ensure the work of the situation “user OK (operator) - unauthorized user”.
The support team began to work closely with the OK.ru message team. As a result, solutions were invented that made it possible directly on the OK.ru login screen to receive online chat with a support service operator. At the same time, the already existing and commissioned oSupp system did not require changes. And the new essence of online chat worked on the modified OK.ru messaging engine. U - convenience!
We started implementing this decision in early 2018. In terms of testing, there were also no special difficulties, as Such a chat had a minimal set of functionality. First of all, we launched support for complaints from unauthorized users from the web. At the moment, support for such complaints from mobile applications is being implemented.
Episode 5: Summary
From the state of the prototype to a fully operational service, oSupp moved to the summer of 2018. As of March 2019, 60% of requests are processed through online support. The staff was not changed. 90 support service employees online cover about 6,000 complaints from users daily. During peak activity on the portal, waiting for a response from the operator takes up to 10 minutes. Under normal workload, operators respond within 1-2 minutes. Thanks to the switch to live online communication, successful restoration of user access to the profile has grown by 15%.
Now we have 100% of our own product, we are free to modify it as anytime. The team is no longer dependent on updates from the OK.ru portal. The new application processing system is written in Java, which implements many OK.ru services. The testing process of the oSupp system is fully covered by test cases, and with updates it is insured by automatic testing.
I must note that we have not completely abandoned OTRS. The offline support system was frozen in mid-2017, and is now playing the role of insurance. There are calls from dishonest users who like to spam aimlessly at the support service. OTRS also processes calls that, for specific reasons, users cannot be processed online. Now the plans of our team include the implementation of processing calls from mobile applications and the introduction of a chatbot system.
However, our main goal was achieved - we are moving away from the use of mail. Now, to solve problems, our users receive quick and timely answers.
Our main achievement: the user has significantly reduced the time it takes to wait for a response and get help.