April 5, 2018 at 12:41How to replace HR-a with a robot? Technical part
We already told you about the Robot Vera project from a business point of view. It's time to learn more about the inside of the service. Today you will learn about the infinity of resources, the semantics and technologies that are used in this project. Under the cut - decoding of the video.
Dmitry Zavalishin is a Russian programmer, author of the OS Phantom concept, organizer and member of the program committee of the OS Day conference, founder of the DZ Systems group of companies. In 1990-2000, he took an active part in the creation of the Russian Internet segments (Relcom) and Fidonet, in particular, ensured the transparent interaction of these networks. In 2000-2004, he was responsible for the design, development and development of the Yandex company portal, created the Yandex.Guru service (hereinafter - Yandex.Market). Read more on the Wiki .
Vladimir Sveshnikov- Arrived in St. Petersburg in 2006 from the Far East. Received a law degree at the University of Finance and Economics. In 2009, they organized the company First Street, which was engaged in the registration of unskilled personnel from the CIS. Later he started outsourcing staff, and by 2012 they had two major clients with a friend - a chain of stores “Healthy Kid” and “Dixy”. The annual turnover of First Street amounted to 30 million rubles, and in 2013 - 50 million. But Vladimir soon realized that he did not see himself in outsourcing and wanted to make a technology startup.
Hello! On-air DZ Online technology, and our first guest, Vladimir Sveshnikov, co-founder and CEO of the company Robot Vera, which is engaged in the selection of personnel using artificial intelligence. This is one of the first, probably, startups that really allow artificial intelligence to people. We have already met, discussing the business tasks of this area, and found out why it is needed and why it is good. And today Vladimir will tell us a little bit about how all this is arranged, what problems arose on the way to that ideal ... in any case, the current solution, which is now. Vladimir, hello!
Yes hello Yes, I’m happy to talk about how we started. The fact is that there were only three of us in the team, it was only a year ago. Now we have more than 50 people in the company working together. But when we were three together, I was fully responsible for the entire technical part. Initially, we started doing such simple things. That is, we just took and began to repeat the recruiter process. We looked for a resume for him, we called the candidates for him, we sent an e-mail with a description of the vacancy for him. And since I have a specific technical background.
I’m a lawyer by training, but then I reassigned to programmers. And I realized that these processes are very routine, very monotonous, they can be automated. And the first thing we did ... that, I remember, was the day we looked for a resume. It took about half a day: half a day to search for a resume, half a day, respectively, calls. Then I climbed ... we did it on the sites SuperJob, Jobs, Salary.ru. Then he climbed up to look at their ip, realized that you can do what we did half a day in one minute. And my partner and I did it together. He, in general, was looking for a resume somehow one day, and I went in one minute to do it all, went to drink tea. Comes, says: “Why are you drinking tea?”, I say: “I have already fulfilled my standard.”
He made his norm.
And, in fact, this served as such a first impetus. That is, we realized that technologies can be used and automated some processes that are currently not automated at all in HR. Well, and then we, accordingly, have already begun to actively engage in calls. That is, we hired call center operators, and automated calls, automated ... a button was made there, they started calling directly from the browser. In general, they did everything as simple as possible so that the operator could sit down, put on headphones and, in general, the only thing he did was synthesize speech and recognize speech. Then we realized that these technologies are already on the market, they show a fairly good level of quality, and you can use them.
That is, did a person synthesize speech and recognize speech? In this sense, at that moment you realized it, as a part of this very machine in communication with ... Until that moment, everything was simple. Pick up a list of vacancies via ip, select them by some keyword. Although, there are subtleties by and large. But God be with him. We will probably return to them later. At some point, you began to engage in voice: synthesize speech and recognize speech. Synthesizing is also clear: there were scripts, they are more or less fixed, probably for a specific selection. And recognition is ... After all, you started with very simple questions and answers first?
Yes.
Was it that the recognition worked badly?
Yes, of course. There are a few points. There, if you look at that ... First of all, for a very long time we were looking for an approach on how to make people understand that they are talking to a robot, how they built a dialogue. That is, at first people are shocked, they don’t understand what to say, what to answer (especially when we call the regions). Moscow, Petersburg is still somehow normal, but people in the regions are directly surprised: what is a robot? (There any various vocabulary can be heard at this moment).
And then we began ... we made it so that it began to introduce itself and then it sets a certain standard format for communication. That is, she says: “I can recognize the answers“ yes ”or“ no. ” Answer my yes or no questions. ” Then people begin to understand that if they say “yes”, then ... well, that is, how they communicate, they understand. Because before that they have a dissonance. That is, like a robot, there are no robots, like. What? A call from the future? Well, that's it. Accordingly, yes, most likely, speech recognition is here. It now works so that it can completely recognize different words. That is, we now have scripts where they choose vacancies, where they ask questions. That is, we all recognize this well. But it was “yes” or “no” - this was so that people would understand how to communicate with the robot.
That is, could you do more right away?
Yes.
Or is it still not? Because then, after all, semantics begin.
Yes, well, we added semantics later. That is, we added here just a few months ago exactly the answers to the questions. That is precisely to recognize what he said - we could for a long time. We even had this point in this script: if he says no, the vacancy is not interesting to him, then we ask, “Why is it not interesting?”. And there he answers why it is not interesting.
But is this a record? Are you just recording the answer without trying to analyze?
We recognize him.
Do you recognize?
Yes, and show in your account exactly as an answer.
In the form of text?
Yes.
Was there a problem at this moment? Here is what you describe seems pretty commonplace. In general, it seems that just about anyone on the planet can now take, pick up a pack of libraries and collect all this on their knees in two days.
There are problems, of course, of course. The main problem was, probably, if we talk about the technological aspect, that we had such a complex product. If we talk only about calls, only about recognition, only about speech synthesis - this is a different story. She is very big, complicated. There, too, for example, we did what is there ... we use external speech cognition. We use Google, Yandex. Firstly, there isn’t any such benchmark. That is, you need to look at your tasks, how exactly your text, how they recognize your audio recordings. That is, the first thing we did was to do this kind of analysis. We looked accordingly, which works better. Then we realized that although one of the campaigns works better and shows better results, all the same, the speed of her response at some points may be longer. That is, she can respond more. And then we began to send the recording to several speech recognition systems. Including Microsoft, Google, we had Amazon, Yandex. That is, at four we sent and received some kind of quickest response.
Now we use two or three systems maximum. That is, during peak hours. And so the main difficulty was that in addition to ... we had to run first a search for resumes, then ... she is a robot, she does everything herself. She herself searches, then after she has found, she calls herself, then after that she, accordingly, sends out an email to those who have not answered “yes” yet.
Do not keep an eye on the robot?
Well, take a look. Monitoring system. That is, this is what we all went through. And first there, since I did all this alone. I did this not quite right, I did it quickly, whipped up. And in general, I had one Docker container there, it also had a base ... That is, I didn’t break it all into micro services, as is customary and as we have done now. It was all one container, one image in which everything worked on the same virtual machine. And actually, there we made a new virtual machine, a new image for each new client. And it often happened that under load, since there was no monitoring system, everything fell there. One of the stories was when one large client came to us. We spent two or three days with the pilots, and then at some point he decided to call on his stop-lists, where he uploaded several thousand candidates. Of course, I had a memory leak and it all covered up, since it was one container, there was no image saved. I was there through telephony, through all this I was restoring this story almost all night so that they would not lose their calls. In general, yes, there were such problems, but here, probably, if we talk about ...
Well these are such typical problems, actually. Fairly commonplace. They are also not very connected with high-tech and recognition. This, in general, probably would have happened in any startup, in which ... yes, in any, because all startups, at first, probably do it with the help of the first person who is a technological ideologist. Namely, with the recognition quality itself?
But with the quality of recognition, of course, there are problems here. They are solved in different ways. That is, for example, if we recognize the address, submit to the recognition system that it is an address. Then it gives better quality. If we recognize a question, then we mark that this is the question. But, in general, now it’s quite good quality, if we have normal audio recordings, if we don’t have any extraneous noise, if we don’t have any ... well, the person speaks in a normal speech, there are no defects.
This, incidentally, is an interesting point. You know, I've talked to the city of Mos.ru, Moscow city services. They, too, are actively involved in similar technologies, and there too, understandable tasks are quite massive. There is a completely ridiculous task - collecting information about water meters. You can call and call your voice, the robot recognizes. And on the contrary, they say that it is imperfect speech or speech with accents, with strong accents, just the opposite, it is covered quite well by the algorithm and even better than by living people. You have the opposite situation, if I hear you correctly?
Well, we have such a direct problem that there were many people with accents, to be honest, no. That is, with us, basically, everyone somehow speaks standardly. It probably still depends on what ... how to ask. That is, if we ask to choose an answer option, for example, a vacancy, she speaks it: choose "sales manager", "loader" or "storekeeper".
Trying to reduce them so narrowly to a narrow spectrum?
Yes, of course. We had a case there, where we collected questionnaires, resumes, where we asked a person to tell about themselves, to tell about their experience. And there, of course, there were all sorts of different interesting, very funny stories. Well, that is, how they told about themselves and how it was all recognized. Of course there is a mistake big enough now. And such that she there fully recognized speech, of course this is not.
How do you measure error? After all, in fact, if a person just told himself, then it is impossible to build any obvious metric that could compare what was recognized with the correct text.
It is only manual now. We have specific account managers who listen to some of the records, and they then look at the answers.
Manually verified pointwise? That is, is this more or less ordinary, selective verification of quality?
Yes. That is, when we just chose which recognition system to take as the main one, we took the order, a little less than 1000 audio recordings that we had and each of this audio recording ... we recognized the text there, checked the recognized text with the audio recording. Here we were a few people sitting and doing it.
But is this the choice of a system? That is, there is an n system, you have a corpus of already known texts about which the correct answer is known. You run them away, compare. There is an obvious mechanism. And by the way, who in the end is the best of all four recognized?
Well, there they have each their own ... well, of course, it is now believed in the market that Google is the best. Well, here, for example, Microsoft gave us a record faster. Here, probably, you can watch in different ways. It is impossible to single out one system, which we take as a basis. But we always use 2-3 now.
Yandex turns out to be an outsider? Does he recognize worse and respond slower?
Yandex recognizes addresses very well. This is probably the best. Now, if we have an address, then without even thinking, we immediately take Yandex. Because it is the best option.
But is this probably just because they have a good base of addresses?
Yes, yes, Yandex.Navigator. Of course, there is Yandex.Taxi. That is, they have a lot of voice samples when the driver in a taxi calls the address. That is, they have it very well worked out. There aren’t even trying any other ones ... well, of course they tried as part of a general analysis, but Yandex is much better.
Is there such a banal thing like driving out the recognition output through a grammar analyzer, which checked the coordination there? And it is used as a certain metric as recognition?
Yes. If we start now to talk about what we are doing now ... well, now, if you automatically measure them somehow, there are certainly benchmarks, we look international. There, Mozilla recently made its own open source, speech recognition, which showed a quality criterion, that is, accuracy in quality is about like Google.
Including Russian?
No, they are only in English so far. They can teach Russian. But now we are still looking at the international market, therefore, for us, as it were ... We now have our first partner in Dubai, and for us, here it is.
But in Dubai, is this English all the same?
Yes, there is completely English. All their working sites are in English. Well, there is a translation into Arabic, but the attendance on the English page is much more.
Returning to the problems. If I understand correctly, I looked at your articles at Habr, what’s going on there, then you’ve got semantics.
Yes, now I’ll tell you more. The second task that we began to solve ... in development, we solve the problems that business brings to us. Business periodically tells us, we need to make our product better, we need to go further along the path of replacing the recruiter. We need, accordingly, to ensure that the Vera robot can call for an interview, fully agree, the address and place to coordinate. And we started doing this thing. There, in principle, everything is quite simple, but okay, they added a script for an invitation to an interview there, received some answers, if it is not interesting, they suggested a different date. There is synchronization with the calendar. In general, a fairly simple task, but we did not even have time to start it, we realized that another task, which is very important, is that candidates do not receive feedback.
From the robot?
Yes. They follow the script of the employer. The employer decided to ask three questions, the candidate heard these three questions, answered them, but the candidate cannot ask his questions. It turns out such a one-sided system. That is, she is such a B2B business, but at the same time we have a large B2C part. That is, we have a lot of candidates, to whom we have already done more than one and a half million interviews and, accordingly, this is one and a half million people who talked with the robot, who potentially would like to ask their questions, to hear some answers. So we began to solve this problem. And we realized that, for example, the simple question of salary can sound differently. That is, it cannot be programmed at the level of a simple hardcore list of words. Well, for example, there the question of salary may sound “what is it for money?”, but it may sound "what is the financial component?" That is, we have both that and that case.
And, accordingly, we didn’t answer the wrong question, we didn’t answer the wrong one, because we laid: income, salary. Then we began to look for some options, stumbled upon a car ... I have been studying machine learning for a long time, we have people in the team who are actively involved in this. And we remembered that there is such a story as there is a Word2vec library, it is based on neural networks. Google gives pages on it. That is, for example, if we write in a request ... in the same place, inquiries on Google are about the same as we have questions about vacancies. And, accordingly, Google solves this problem with the help of this library. That is, which is better to show. That is, he takes the text there and accordingly shows which document is better, higher. Ranking documents. Essentially, how does it work? In general, all words turn into a vector,
How much is N-dimensional?
Now I won’t say for sure. But these parameters line up. That is, they can be changed. And the quality of the model depends on them. We took the standard Word2vec model, trained it on the package from ... well, there is about 150 Kb of the package. These are millions of books. It includes the Wikipedia corpus, all articles of the Russian Wikipedia, they are all translated into text and she is trained on this text. How does she learn? That is, she runs through this text and looks. For example, there is a sentence “I called by phone” and, for example, there is a sentence “I called by mobile”. Since “I called” in both cases is the same context, it is “telephone” and “mobile” ...
Assumes that they are close.
Yes. There she simply brings them together, she randomly arranges them first, then brings them together in points. And so we get such a certain kind of mapping of our words in vector space.
Metric of semantic affinity of words.
Yes. And then we consider the cosine distance or we consider the Euclidean distance. There we also poked around a bit, because at first we calculated the cosine distance, but it turned out that Euclidean gave us 10% plus to the quality. Because it is more cosine when there is a lot of text, that is, a large text, if we need to compare a document. And since our questions are about the same, everything is short, it’s easier to use the Euclidean metric. Well, actually, we decided to implement all this. Of course, the quality here is about 70%. That is, quite low.
70% what?
Right answers.
Meet the questioner's expectations?
Yes. That is, we here, in fact, made such a Data Set, in which there was a question-answer, question-answer. That is, this is a question and a category to which we relate. We have several categories: salary, address, company. There is a category of "did not understand the question." Everything that does not fall into other categories will fall there. There is also a category “about vacancies”, “job responsibilities”. That is, a number of such categories. Accordingly, in each category we write some questions. 5-10 questions usually. And what happens? We get the candidate’s question, run a cycle through all these categories, consider the distance to all questions (the same number of questions in each category), then we give the closest category to him. If there the degree of proximity is less than a certain one, then we say that we did not understand, please rephrase the question.
And, in fact, here we are just trying to solve this problem as well as possible. That is, we first took this Word2vec there, then there is such an OpenAI company in the states. They do not study artificial intelligence for commercial purposes. There they have Elon Musk, Esen Altman investors. And they are quite enough, probably alone ... at the forefront of these technologies. They recently released their research on sentiment analysis. They trained a model on 80 million Amazon product reviews there. And the main task was to train the model to generate these reviews. She generates reviews. But besides this, she learned to do sentiment analysis well. That is, she learned to distinguish well a good review from a bad one. This we also use now.
By the way, yes, a very interesting question. After all, you communicate with him and you can try to somehow take out some predisposition, inclination, positive, negative for this vacancy from his answers. Are you doing?
We are already doing this. This is the model that we are currently using, which, in principle, allows this to be done quite easily. In fact, it solves the same problem, that is, it displays words in a vector space, and then they are just ... The most important thing ...
Then you look at some key points, positive and negative, and see how close the answers are already to the points, who evaluate this in terms of positive, negative? Do you understand correctly?
Yes.
Can you give an example of some incorrect answer to the question? She answered incorrectly. Do not understand the question. And in which direction does the error turn out?
It was a lot. So I say, 30 percent. After the first such test, we had 30%. We selected protection managers for ourselves.
So you tested on yourself?
We went to live production. Well, we knew it would be 70%, but sent. By the way, there were 70 of us even on tests, and 56 were alive. That is, it was even lower. We realized that we still need to refine it.
50 is very sure ...
In fact, it looks like a statistical error. It's just that there are 50 to 50. But in the end, we are already close to 80. These were the very first such tests.
What's happened? This is how you moved?
For example, we have now replaced Word2vec sentimental neuron, which shows quality 5% better. Immediately I added 5%, there on our tests we still twisted it, and there the quality increased. Then we replaced the cosine distance with Euclidean. This still gave about 10% in quality. Well, then we just increased the number of questions. We had about 5 questions in each category, and we did 15 each. We increased three times and, in general, thanks to these three operations, it turned out that now there are about 80.
By increasing them, you did not add other questions, but other formulations of the same question? That is, in this sense, what you are saying now means that it is the sematic model that works poorly? You have added variability to the options that the system is considering, and it has become more clearly fall into the option. Apparently, you took these errors, analyzed, and on the basis of them you generated variants of questions that are reference and also close a certain sematic cloud around you.
That is, now how will it be? How do we position this thing? What is this artificial intelligence, which is such a chat bot that just helps the recruiter answer all these questions. Here all the questions, when she did not get, they are shown to the recruiter. A recruiter can create a new category at any time. The only thing is if he now creates a category ... Why initially 5 questions? Because in each category there should be questions of approximately the same length and they should be the same number. Otherwise, this Euclidean distance will be considered bad. Therefore, we first wanted to do it ourselves. Then we realized that everyone had different questions. For example, a company that sells cell phones in stores may have the question, “How much do you deduct for marriage if I break an iPhone X phone?” I.e,
Although some would like to.
And, accordingly, all these questions fall to the recruiter. And a recruiter can collect a category from them, can add questions, remove questions.
Is such a body of typical questions going to be used?
As a standard, we collect there about 12 categories now. But he can add, create, can add questions to each category. That is, he can, in theory, make more of them.
Is the answer to the question fixed?
The recruiter asks the answer to the question.
In fact, can he even move from answers in this place? First, select the answers that interest him.
And so it works. We, look, have a vacancy at the entrance, there is a description for the vacancy. That is, we form the answers automatically. This generally works very poorly. We probably have half the questions ... even half the questions fill slowly, for a long time. That is, we automatically tighten the half, and this allows the recruiter to understand ... We take the Job description, break it into offers. And each sentence counts the distance to the category. And for each category we take the most normal offer. Well, if there are official duties and requirements, then it is usually highlighted there, we can get it out of the text right away. There are some questions, for example, about the lunch break, about whether I need education, higher education, or something like that. We draw it simply from the general text.
We are now telling you all this in such a rather detailed way, in fact. Are you not afraid that right now people are sitting behind the camera who are recording accurately and in a month will enter the market with competitors?
Well, I have this approach. I believe that this story with text and with the recognition of a natural language, it is only just beginning to develop, and there are a huge number of areas of application. And what people do there ... If someone succeeds, someone does something similar, I will only be glad we will use their experience. That is, I have here such a more open source story. This is closer to the review.
But are you unique today? Do you have competitors in the world?
Not.
Is it just a really unique company all over the globe?
Well actually.
Nobody does it like that?
There are similar products. Here the technology is developing very much. Of course, somewhere someone is better in something. For example, in the states there is a chatbot Mia, maybe she communicates better in the text than we do now? We do not exclude it here.
You close a very clear task.
If we talk about the voice of this is not. We in the states looked analogues. There is one company that ... if we can make a script in the office, then we need to call and record broadcast. Here is the phone number, call, write down what you want to tell the subscribers. You record it and they then scroll through this audio recording of your voice. That is, somehow a little bit wrong.
It would seem that you will be more technologically advanced.
It would seem that this whole thing is simple. But here I think that ... on the tangent of replacing, somehow copying and so on, here first of all, here we are, for example, we trained this sentimental neuron for more than two months. Purely training was going on. That is, Amazon, OpenAI.
But this is a processor power issue, right?
Yes.
Can this be done in 3 days, if you just buy more virtual machines?
Not. We already bought it.
You have already bought everything, everything is already over.
Yes. Well, we have it since, we are in partnership with Microsoft, we have virtually unlimited resources for these capacities. Therefore, we take large cars, use the best, probably what is now on the market. Well, I don’t know, the quality there is no different from IWS. And, in fact, we teach them. And even OpenAI, having its own resources, trained this model for a month. That is, therefore, the tasks here are quite complex. Firstly, it’s another part to train just a month, and the second task, which is even more difficult, is to find the data. That is, the question is what else to get ... we need more than just text.
You took the body of the text in some more or less banal? Wikipedia is a general place, it’s enough for everyone.
It alone is not enough. Well, if we want to add some more ... We did how, we took not just this building, we still ... we have, in general, about, probably, several hundred thousand vacancies already published on the service. That is, we took these texts, 2 million resumes.
Here you have competitors who have vacancies.
Yes, of course. They took a resume. That is, it is about 2 million resumes with us. We took, in general, even such vacancies, that is, we simply collected vacancies through ip work sites. That is, somewhere, probably, 10-15 GB of texts related specifically to HR-related topics are resumes, vacancies. That's all we took. That is, probably, without this there would have been little change. We roughly compared the model, which is only in the Russian case and in the Russian case with our texts. There is a difference of about 2-3%. That is, small, but it is all the same. And, how, why not do it?
Do they affect 3%? Look, in fact, when we talked with your colleague, at that moment the robot, in my opinion, only accepted “yes” and “no”. The level was like that. And already at this level it was very clear that ... Now I recalled, there was a good story about how TPPs cracked the encryption algorithm. She was great at making a hardware farm that knows how to distinguish a bad key from an unknown. Not bad from the good, but bad from the unknown.
But on this it turned out to remove two orders of volume, clear this thing, and drive away the rest of the keys simply with the usual selection. And you did the same? That is, you, in fact, have made some tool that allows you to distinguish exactly a bad candidate from an unknown. And thus, you also removed some significant amount of work, such meaningless, dumb work from a recruiter, and raised the effectiveness of the order by 2, probably? Somewhere at that level? And this is an essential driver. Everything else that you do is valuable, interesting, very curious, but it seems to me, from the point of view of business, this jump does not bring. Or I'm wrong?
Well, here we are ... certainly 2-3% here in this task, they of course ...
Do not the weather?
They don’t make much weather, but here we are, as it were, fighting for this ... we are going to this figure of 80 quality. Therefore, we consider this, probably, for us, the base line after which you can directly roll into production for all customers and say that we did it.
And now are you experimenting with the client to agree?
Yes, a few customers who are testing.
Ready to take risks ...
Yes, they use this technology. Well, and we, as it were, while holding it in a closed form for a bit. That is, we do not tell in measure.
Just told.
Yes. And, accordingly, here each here is some kind of small addition. That is, here we changed the metric from cosine to Euclidean, here we changed the model. Then we tweaked this model a little too, set the classification method to another, and it has already become a little better. Also 2-3%. And so, out of everything, we are going to these 80s. That is, there is such a process. Each component somehow affects this all.
On the other hand, now your competitors are two orders behind you, right? And it’s easy for them to close these 2 orders. Well, that is, it is generally a rather banal system with a rather banal recognizer of the words “yes” or “no”. And how, in fact, and you, if I remember correctly, in three months you did all this? Well, maybe ...
A little more. These have been months. That is, there without sleep ...
Well, you did not know yet that it is possible. They already know that. That is, in principle, to people who are currently sitting behind the camera, somewhere behind the screen from us, probably the question is one month there, in order to draw on your experience and success, to make an analogue that also implements these two orders. That is, in principle, your competitors can reach you pretty quickly.
Yes, of course. Moreover, there are already companies that offer similar services. That is, this is already happening in principle. But here we are now, as it were, we had an option there to improve our service here, to say that we are the most stable and say that we are the best, we are the very first and so on. But we, as it were, decided to go to the next stage. That is, we decided to go where there are no competitors. That is, competition is for losers, as there was one investor from the valley Peter Thiel. And, in fact, we are trying to do some things that are not there yet. Well, for example, there is no lively dialogue. That is, our task, that’s all I’m telling, these little details, they all go to the big task - creating such a lively dialogue.
That is, we now made a story in the test itself that after it recognizes these categories of questions, it’s, in principle, the answer is “yes” and the answer “no” is also a category in fact. And then, how are we now? That is, if earlier it was necessary to create a call script, and it was such a tedious process, in fact. Well, because there we give some kind of default script, standard, but anyway, each company is different. There, for example, a company addresses one to "you", the other to "you." That is, someone has an official presentation, someone else. Someone “hello”, someone there “good afternoon”. And, accordingly, here we are trying ... the candidate can ask the right, definite place to put, where he can ask questions, and she should ask: "What are your questions?" And we want the candidate to be able to ask a question at any time. I.e, she asks him: “Are you interested in work?”, he says: “And who are you?” And she answers him.
And she answers.
Yes.
That is, there maintains such a lively dialogue.
Yes. And after she answered, he says: "Yes, interesting." And she understands that the answer to her previous question. And now we are working on it. And this, of course, is such a big leap, here it will be in this communication, in our product. Because then the candidate will receive information. Not only receive information on vacancies, but just communicate as he is used to. And that will increase conversion of course by far.
Well, you are not the first. In Russia there are teams that did something similar. Do you somehow communicate, discuss?
Yes there is, and in the world. Of course, yes. Now, if we talk about systems such as these where natural language is recognized, where they try to build a lively dialogue, then ... if of course the big deal of big deal is to create a system there that will communicate there ... Jarvis Mark Zuckerberg recently showed. But at the same time, this, as it were, is such a big story. Giants work there: Facebook, Amazon, Google, Microsoft and so on. They all work on it. Well, at the same time there are smaller start-up companies that are engaged in some highly specialized task. For example, technical support. Very often it is automated, very often it is replaced using these technologies. There are people on bots. There is quite a lot of progress.
Of course we communicate with companies, with the guys who do this. For example, we are specialized in HR, we have our own field, our own subject, we need our own texts to train the model, we need some of our own specific patterns, our own categories. That is, this is what we are doing here with us. There are companies that do the same. In the states there is Mia, a number of companies. They are also working on this, but in a highly specialized format this still somehow works out. Well, there’s something to achieve, 80% and maybe even more.
It is segmented that you take a subset of the types of communication and lock yourself in it. Due to this, a little bit of the sematic model, having limitations on the class of semantics, probably acts more clearly. Look, in fact, again, this is what you are talking about - of course it’s nice and very interesting, it’s delicious to be proud of: “We have a robot that can talk.” Does this business have value?
Yes, we are, after all, probably our story, that we periodically want to go into sentiment analysis there or to go somewhere else. But we come from business, from its requests. That is where I just started. That the business has come and says: "We need the robot to start answering the candidate's questions, so that it is not a one-way, not a one-way game."
What about the metric? So you did it to some extent. How do you measure effect from a business perspective?
It's quite simple here. We have a metric. We are all talking about calls, we measure them in calls. That is, there is a certain timeline in the conversation between the recruiter and the candidate. There usually, if this is a position of some kind of management selection, a sales manager, then this is probably 10-15 minutes. And the first part is, of course, these three questions that we ask: “Do you need work? Is it interesting? And we have such conditions. You’ll come, you won’t come? ” The last part there is probably also a maximum of a couple of minutes there. That is, this is an interview agreement. Well, there’s probably even a minute. It’s there: tomorrow at three, but I can’t go there tomorrow, okay, come on the day after tomorrow. She looks at the calendar, writes down. That is, this is also a simple story. And here in the middle they have this communication.
And, in fact, our metric, it just lies in the fact that when we were looking for employees and doing this test, we measured how many of our accounts communicate with candidates. Here before this feature with questions for answers and after. And it turned out that we were able to halve the time of their communication. That is, they asked only those questions that they were not interested in. And this, if we take into account that they still asked questions: "What kind of robot is this?" etc. That is, they also spent time. And therefore, here it is unconditional. That is, our story is about reducing the time spent by a recruiter. But all the functions that we can now help him, we try to do it. Therefore, there is certainly such a business value.
That is, the metric is not a conversion, but a reduction in the costs of a living person?
Yes, while we are struggling with this metric. Conversion is such a next step. We certainly count her too. Well, in fact, here, in general, it probably does not change. That is, approximately, there we did not see any big difference. Candidates who receive answers to questions, she probably even gets a little bit smaller in the end. There are fewer of them, because some receive answers that do not suit them.
Which do not suit them, and it falls off. By the way, this is also a separate issue, which could also be discussed, but probably not this time. Our meeting today is drawing to a close, and you know, I’m talking to you right now and I remembered an episode from my childhood. I remember in my time, phonograph records were issued, which were printed in a magazine on plastic like that. A plastic page was inserted into the magazine, it had to be cut out, and there was a plate. And they inserted some completely enchanting such interesting things sometimes. And there, for example, was a story about how scientists made an artificial throat mechanical. She was spinning there somehow, cringing. And there was a sound recorded, uttered by this mechanical robot. And then it was not just a fantasy, but some such, in general, beyond. And it even seemed that this will never happen, and today we are discussing a situation in which the synthesis of speech itself is not even the subject of conversation. It simply exists, it simply exists and is already working, and even recognition as such, generally speaking, works.
And we are discussing not recognition of words, but recognition of the meaning of the spoken and practically communication at the level, well, in general, probably already of some kind of living mind of such an acting one. Probably the child there is two or three year old. Somewhere it gets there. That is, the question of getting into the conversation of a real person, probably, will already happen somewhere in our lifetime?
Yes, I think so. Anyway, I remain so optimistic in terms of technology. Of course, we are faced with a lot of difficulties, of course, all the same, these 20% are there, a sufficiently large amount remains that we do not recognize these answers. When we go to watch these 20%, we do not yet understand how to add these percentages to us. That is, so far we don’t even have ideas in general there. But at the same time, and we understand that this is also taking into account the fact that this is a very narrow area. That is, it’s only HR, it’s only about the vacancy, it’s just about this. And we still do not take issues in which 10-15% can still, which in general as a whole. Well, sometimes she is asked questions: “What is the weather like in Moscow?” They can also be processed, of course.
She must say: "Wait, Alice will call you back."
Of course, here is the fact that when we look at these errors, we understand that the technologies, they are not so much developed as to directly replace the person. And this one here is often very simple, I observe articles there, they write that this type completely replaced the person. This of course raises such a super-hype in this area, in fact, no.
Far away.
Yes, far away. I think that in our lifetime, I think, of course. Somewhere we come closer to this. Probably, 100% still will not come close, but we will be very close. But in general, this is a story about us treating this technology more as one that helps some simple tasks, simple problems for people to solve now. But of course, it does not completely replace people yet.
Leading. Well, so ... Probably, we will wait for the day when your system passes the Turing test and then call you again?
Yes, of course.
Digital Transformation Series
Technology articles:
1. Start .
2. Blockchain in the bank .
3. We teach the machine to understand human genes .
4. Machine learning and chocolates .
5. Loading ...
A series of interviews with Dmitry Zavalishin on the DZ Online channel :
1. Alexander Lozhechkin from Microsoft: Do you need developers in the future?
2. Alexey Kostarev from Robot Vera: How to replace HR-a with a robot?
3. Fedor Ovchinnikov from Dodo Pizza: How to replace the restaurant director with a robot?
4. Andrei Golub from ELSE Corp Srl: How do I stop spending a ton of time shopping?
5.Vladimir Sveshnikov from Robot Vera: How to replace HR-a with a robot? Technical part.
Who is the interview with?
Dmitry Zavalishin is a Russian programmer, author of the OS Phantom concept, organizer and member of the program committee of the OS Day conference, founder of the DZ Systems group of companies. In 1990-2000, he took an active part in the creation of the Russian Internet segments (Relcom) and Fidonet, in particular, ensured the transparent interaction of these networks. In 2000-2004, he was responsible for the design, development and development of the Yandex company portal, created the Yandex.Guru service (hereinafter - Yandex.Market). Read more on the Wiki . Vladimir Sveshnikov- Arrived in St. Petersburg in 2006 from the Far East. Received a law degree at the University of Finance and Economics. In 2009, they organized the company First Street, which was engaged in the registration of unskilled personnel from the CIS. Later he started outsourcing staff, and by 2012 they had two major clients with a friend - a chain of stores “Healthy Kid” and “Dixy”. The annual turnover of First Street amounted to 30 million rubles, and in 2013 - 50 million. But Vladimir soon realized that he did not see himself in outsourcing and wanted to make a technology startup.Interview
Hello! On-air DZ Online technology, and our first guest, Vladimir Sveshnikov, co-founder and CEO of the company Robot Vera, which is engaged in the selection of personnel using artificial intelligence. This is one of the first, probably, startups that really allow artificial intelligence to people. We have already met, discussing the business tasks of this area, and found out why it is needed and why it is good. And today Vladimir will tell us a little bit about how all this is arranged, what problems arose on the way to that ideal ... in any case, the current solution, which is now. Vladimir, hello!
Yes hello Yes, I’m happy to talk about how we started. The fact is that there were only three of us in the team, it was only a year ago. Now we have more than 50 people in the company working together. But when we were three together, I was fully responsible for the entire technical part. Initially, we started doing such simple things. That is, we just took and began to repeat the recruiter process. We looked for a resume for him, we called the candidates for him, we sent an e-mail with a description of the vacancy for him. And since I have a specific technical background.
I’m a lawyer by training, but then I reassigned to programmers. And I realized that these processes are very routine, very monotonous, they can be automated. And the first thing we did ... that, I remember, was the day we looked for a resume. It took about half a day: half a day to search for a resume, half a day, respectively, calls. Then I climbed ... we did it on the sites SuperJob, Jobs, Salary.ru. Then he climbed up to look at their ip, realized that you can do what we did half a day in one minute. And my partner and I did it together. He, in general, was looking for a resume somehow one day, and I went in one minute to do it all, went to drink tea. Comes, says: “Why are you drinking tea?”, I say: “I have already fulfilled my standard.”
He made his norm.
And, in fact, this served as such a first impetus. That is, we realized that technologies can be used and automated some processes that are currently not automated at all in HR. Well, and then we, accordingly, have already begun to actively engage in calls. That is, we hired call center operators, and automated calls, automated ... a button was made there, they started calling directly from the browser. In general, they did everything as simple as possible so that the operator could sit down, put on headphones and, in general, the only thing he did was synthesize speech and recognize speech. Then we realized that these technologies are already on the market, they show a fairly good level of quality, and you can use them.
That is, did a person synthesize speech and recognize speech? In this sense, at that moment you realized it, as a part of this very machine in communication with ... Until that moment, everything was simple. Pick up a list of vacancies via ip, select them by some keyword. Although, there are subtleties by and large. But God be with him. We will probably return to them later. At some point, you began to engage in voice: synthesize speech and recognize speech. Synthesizing is also clear: there were scripts, they are more or less fixed, probably for a specific selection. And recognition is ... After all, you started with very simple questions and answers first?
Yes.
Was it that the recognition worked badly?
Yes, of course. There are a few points. There, if you look at that ... First of all, for a very long time we were looking for an approach on how to make people understand that they are talking to a robot, how they built a dialogue. That is, at first people are shocked, they don’t understand what to say, what to answer (especially when we call the regions). Moscow, Petersburg is still somehow normal, but people in the regions are directly surprised: what is a robot? (There any various vocabulary can be heard at this moment).
And then we began ... we made it so that it began to introduce itself and then it sets a certain standard format for communication. That is, she says: “I can recognize the answers“ yes ”or“ no. ” Answer my yes or no questions. ” Then people begin to understand that if they say “yes”, then ... well, that is, how they communicate, they understand. Because before that they have a dissonance. That is, like a robot, there are no robots, like. What? A call from the future? Well, that's it. Accordingly, yes, most likely, speech recognition is here. It now works so that it can completely recognize different words. That is, we now have scripts where they choose vacancies, where they ask questions. That is, we all recognize this well. But it was “yes” or “no” - this was so that people would understand how to communicate with the robot.
That is, could you do more right away?
Yes.
Or is it still not? Because then, after all, semantics begin.
Yes, well, we added semantics later. That is, we added here just a few months ago exactly the answers to the questions. That is precisely to recognize what he said - we could for a long time. We even had this point in this script: if he says no, the vacancy is not interesting to him, then we ask, “Why is it not interesting?”. And there he answers why it is not interesting.
But is this a record? Are you just recording the answer without trying to analyze?
We recognize him.
Do you recognize?
Yes, and show in your account exactly as an answer.
In the form of text?
Yes.
Was there a problem at this moment? Here is what you describe seems pretty commonplace. In general, it seems that just about anyone on the planet can now take, pick up a pack of libraries and collect all this on their knees in two days.
There are problems, of course, of course. The main problem was, probably, if we talk about the technological aspect, that we had such a complex product. If we talk only about calls, only about recognition, only about speech synthesis - this is a different story. She is very big, complicated. There, too, for example, we did what is there ... we use external speech cognition. We use Google, Yandex. Firstly, there isn’t any such benchmark. That is, you need to look at your tasks, how exactly your text, how they recognize your audio recordings. That is, the first thing we did was to do this kind of analysis. We looked accordingly, which works better. Then we realized that although one of the campaigns works better and shows better results, all the same, the speed of her response at some points may be longer. That is, she can respond more. And then we began to send the recording to several speech recognition systems. Including Microsoft, Google, we had Amazon, Yandex. That is, at four we sent and received some kind of quickest response.
Now we use two or three systems maximum. That is, during peak hours. And so the main difficulty was that in addition to ... we had to run first a search for resumes, then ... she is a robot, she does everything herself. She herself searches, then after she has found, she calls herself, then after that she, accordingly, sends out an email to those who have not answered “yes” yet.
Do not keep an eye on the robot?
Well, take a look. Monitoring system. That is, this is what we all went through. And first there, since I did all this alone. I did this not quite right, I did it quickly, whipped up. And in general, I had one Docker container there, it also had a base ... That is, I didn’t break it all into micro services, as is customary and as we have done now. It was all one container, one image in which everything worked on the same virtual machine. And actually, there we made a new virtual machine, a new image for each new client. And it often happened that under load, since there was no monitoring system, everything fell there. One of the stories was when one large client came to us. We spent two or three days with the pilots, and then at some point he decided to call on his stop-lists, where he uploaded several thousand candidates. Of course, I had a memory leak and it all covered up, since it was one container, there was no image saved. I was there through telephony, through all this I was restoring this story almost all night so that they would not lose their calls. In general, yes, there were such problems, but here, probably, if we talk about ...
Well these are such typical problems, actually. Fairly commonplace. They are also not very connected with high-tech and recognition. This, in general, probably would have happened in any startup, in which ... yes, in any, because all startups, at first, probably do it with the help of the first person who is a technological ideologist. Namely, with the recognition quality itself?
But with the quality of recognition, of course, there are problems here. They are solved in different ways. That is, for example, if we recognize the address, submit to the recognition system that it is an address. Then it gives better quality. If we recognize a question, then we mark that this is the question. But, in general, now it’s quite good quality, if we have normal audio recordings, if we don’t have any extraneous noise, if we don’t have any ... well, the person speaks in a normal speech, there are no defects.
This, incidentally, is an interesting point. You know, I've talked to the city of Mos.ru, Moscow city services. They, too, are actively involved in similar technologies, and there too, understandable tasks are quite massive. There is a completely ridiculous task - collecting information about water meters. You can call and call your voice, the robot recognizes. And on the contrary, they say that it is imperfect speech or speech with accents, with strong accents, just the opposite, it is covered quite well by the algorithm and even better than by living people. You have the opposite situation, if I hear you correctly?
Well, we have such a direct problem that there were many people with accents, to be honest, no. That is, with us, basically, everyone somehow speaks standardly. It probably still depends on what ... how to ask. That is, if we ask to choose an answer option, for example, a vacancy, she speaks it: choose "sales manager", "loader" or "storekeeper".
Trying to reduce them so narrowly to a narrow spectrum?
Yes, of course. We had a case there, where we collected questionnaires, resumes, where we asked a person to tell about themselves, to tell about their experience. And there, of course, there were all sorts of different interesting, very funny stories. Well, that is, how they told about themselves and how it was all recognized. Of course there is a mistake big enough now. And such that she there fully recognized speech, of course this is not.
How do you measure error? After all, in fact, if a person just told himself, then it is impossible to build any obvious metric that could compare what was recognized with the correct text.
It is only manual now. We have specific account managers who listen to some of the records, and they then look at the answers.
Manually verified pointwise? That is, is this more or less ordinary, selective verification of quality?
Yes. That is, when we just chose which recognition system to take as the main one, we took the order, a little less than 1000 audio recordings that we had and each of this audio recording ... we recognized the text there, checked the recognized text with the audio recording. Here we were a few people sitting and doing it.
But is this the choice of a system? That is, there is an n system, you have a corpus of already known texts about which the correct answer is known. You run them away, compare. There is an obvious mechanism. And by the way, who in the end is the best of all four recognized?
Well, there they have each their own ... well, of course, it is now believed in the market that Google is the best. Well, here, for example, Microsoft gave us a record faster. Here, probably, you can watch in different ways. It is impossible to single out one system, which we take as a basis. But we always use 2-3 now.
Yandex turns out to be an outsider? Does he recognize worse and respond slower?
Yandex recognizes addresses very well. This is probably the best. Now, if we have an address, then without even thinking, we immediately take Yandex. Because it is the best option.
But is this probably just because they have a good base of addresses?
Yes, yes, Yandex.Navigator. Of course, there is Yandex.Taxi. That is, they have a lot of voice samples when the driver in a taxi calls the address. That is, they have it very well worked out. There aren’t even trying any other ones ... well, of course they tried as part of a general analysis, but Yandex is much better.
Is there such a banal thing like driving out the recognition output through a grammar analyzer, which checked the coordination there? And it is used as a certain metric as recognition?
Yes. If we start now to talk about what we are doing now ... well, now, if you automatically measure them somehow, there are certainly benchmarks, we look international. There, Mozilla recently made its own open source, speech recognition, which showed a quality criterion, that is, accuracy in quality is about like Google.
Including Russian?
No, they are only in English so far. They can teach Russian. But now we are still looking at the international market, therefore, for us, as it were ... We now have our first partner in Dubai, and for us, here it is.
But in Dubai, is this English all the same?
Yes, there is completely English. All their working sites are in English. Well, there is a translation into Arabic, but the attendance on the English page is much more.
Returning to the problems. If I understand correctly, I looked at your articles at Habr, what’s going on there, then you’ve got semantics.
Yes, now I’ll tell you more. The second task that we began to solve ... in development, we solve the problems that business brings to us. Business periodically tells us, we need to make our product better, we need to go further along the path of replacing the recruiter. We need, accordingly, to ensure that the Vera robot can call for an interview, fully agree, the address and place to coordinate. And we started doing this thing. There, in principle, everything is quite simple, but okay, they added a script for an invitation to an interview there, received some answers, if it is not interesting, they suggested a different date. There is synchronization with the calendar. In general, a fairly simple task, but we did not even have time to start it, we realized that another task, which is very important, is that candidates do not receive feedback.
From the robot?
Yes. They follow the script of the employer. The employer decided to ask three questions, the candidate heard these three questions, answered them, but the candidate cannot ask his questions. It turns out such a one-sided system. That is, she is such a B2B business, but at the same time we have a large B2C part. That is, we have a lot of candidates, to whom we have already done more than one and a half million interviews and, accordingly, this is one and a half million people who talked with the robot, who potentially would like to ask their questions, to hear some answers. So we began to solve this problem. And we realized that, for example, the simple question of salary can sound differently. That is, it cannot be programmed at the level of a simple hardcore list of words. Well, for example, there the question of salary may sound “what is it for money?”, but it may sound "what is the financial component?" That is, we have both that and that case.
And, accordingly, we didn’t answer the wrong question, we didn’t answer the wrong one, because we laid: income, salary. Then we began to look for some options, stumbled upon a car ... I have been studying machine learning for a long time, we have people in the team who are actively involved in this. And we remembered that there is such a story as there is a Word2vec library, it is based on neural networks. Google gives pages on it. That is, for example, if we write in a request ... in the same place, inquiries on Google are about the same as we have questions about vacancies. And, accordingly, Google solves this problem with the help of this library. That is, which is better to show. That is, he takes the text there and accordingly shows which document is better, higher. Ranking documents. Essentially, how does it work? In general, all words turn into a vector,
How much is N-dimensional?
Now I won’t say for sure. But these parameters line up. That is, they can be changed. And the quality of the model depends on them. We took the standard Word2vec model, trained it on the package from ... well, there is about 150 Kb of the package. These are millions of books. It includes the Wikipedia corpus, all articles of the Russian Wikipedia, they are all translated into text and she is trained on this text. How does she learn? That is, she runs through this text and looks. For example, there is a sentence “I called by phone” and, for example, there is a sentence “I called by mobile”. Since “I called” in both cases is the same context, it is “telephone” and “mobile” ...
Assumes that they are close.
Yes. There she simply brings them together, she randomly arranges them first, then brings them together in points. And so we get such a certain kind of mapping of our words in vector space.
Metric of semantic affinity of words.
Yes. And then we consider the cosine distance or we consider the Euclidean distance. There we also poked around a bit, because at first we calculated the cosine distance, but it turned out that Euclidean gave us 10% plus to the quality. Because it is more cosine when there is a lot of text, that is, a large text, if we need to compare a document. And since our questions are about the same, everything is short, it’s easier to use the Euclidean metric. Well, actually, we decided to implement all this. Of course, the quality here is about 70%. That is, quite low.
70% what?
Right answers.
Meet the questioner's expectations?
Yes. That is, we here, in fact, made such a Data Set, in which there was a question-answer, question-answer. That is, this is a question and a category to which we relate. We have several categories: salary, address, company. There is a category of "did not understand the question." Everything that does not fall into other categories will fall there. There is also a category “about vacancies”, “job responsibilities”. That is, a number of such categories. Accordingly, in each category we write some questions. 5-10 questions usually. And what happens? We get the candidate’s question, run a cycle through all these categories, consider the distance to all questions (the same number of questions in each category), then we give the closest category to him. If there the degree of proximity is less than a certain one, then we say that we did not understand, please rephrase the question.
And, in fact, here we are just trying to solve this problem as well as possible. That is, we first took this Word2vec there, then there is such an OpenAI company in the states. They do not study artificial intelligence for commercial purposes. There they have Elon Musk, Esen Altman investors. And they are quite enough, probably alone ... at the forefront of these technologies. They recently released their research on sentiment analysis. They trained a model on 80 million Amazon product reviews there. And the main task was to train the model to generate these reviews. She generates reviews. But besides this, she learned to do sentiment analysis well. That is, she learned to distinguish well a good review from a bad one. This we also use now.
By the way, yes, a very interesting question. After all, you communicate with him and you can try to somehow take out some predisposition, inclination, positive, negative for this vacancy from his answers. Are you doing?
We are already doing this. This is the model that we are currently using, which, in principle, allows this to be done quite easily. In fact, it solves the same problem, that is, it displays words in a vector space, and then they are just ... The most important thing ...
Then you look at some key points, positive and negative, and see how close the answers are already to the points, who evaluate this in terms of positive, negative? Do you understand correctly?
Yes.
Can you give an example of some incorrect answer to the question? She answered incorrectly. Do not understand the question. And in which direction does the error turn out?
It was a lot. So I say, 30 percent. After the first such test, we had 30%. We selected protection managers for ourselves.
So you tested on yourself?
We went to live production. Well, we knew it would be 70%, but sent. By the way, there were 70 of us even on tests, and 56 were alive. That is, it was even lower. We realized that we still need to refine it.
50 is very sure ...
In fact, it looks like a statistical error. It's just that there are 50 to 50. But in the end, we are already close to 80. These were the very first such tests.
What's happened? This is how you moved?
For example, we have now replaced Word2vec sentimental neuron, which shows quality 5% better. Immediately I added 5%, there on our tests we still twisted it, and there the quality increased. Then we replaced the cosine distance with Euclidean. This still gave about 10% in quality. Well, then we just increased the number of questions. We had about 5 questions in each category, and we did 15 each. We increased three times and, in general, thanks to these three operations, it turned out that now there are about 80.
By increasing them, you did not add other questions, but other formulations of the same question? That is, in this sense, what you are saying now means that it is the sematic model that works poorly? You have added variability to the options that the system is considering, and it has become more clearly fall into the option. Apparently, you took these errors, analyzed, and on the basis of them you generated variants of questions that are reference and also close a certain sematic cloud around you.
That is, now how will it be? How do we position this thing? What is this artificial intelligence, which is such a chat bot that just helps the recruiter answer all these questions. Here all the questions, when she did not get, they are shown to the recruiter. A recruiter can create a new category at any time. The only thing is if he now creates a category ... Why initially 5 questions? Because in each category there should be questions of approximately the same length and they should be the same number. Otherwise, this Euclidean distance will be considered bad. Therefore, we first wanted to do it ourselves. Then we realized that everyone had different questions. For example, a company that sells cell phones in stores may have the question, “How much do you deduct for marriage if I break an iPhone X phone?” I.e,
Although some would like to.
And, accordingly, all these questions fall to the recruiter. And a recruiter can collect a category from them, can add questions, remove questions.
Is such a body of typical questions going to be used?
As a standard, we collect there about 12 categories now. But he can add, create, can add questions to each category. That is, he can, in theory, make more of them.
Is the answer to the question fixed?
The recruiter asks the answer to the question.
In fact, can he even move from answers in this place? First, select the answers that interest him.
And so it works. We, look, have a vacancy at the entrance, there is a description for the vacancy. That is, we form the answers automatically. This generally works very poorly. We probably have half the questions ... even half the questions fill slowly, for a long time. That is, we automatically tighten the half, and this allows the recruiter to understand ... We take the Job description, break it into offers. And each sentence counts the distance to the category. And for each category we take the most normal offer. Well, if there are official duties and requirements, then it is usually highlighted there, we can get it out of the text right away. There are some questions, for example, about the lunch break, about whether I need education, higher education, or something like that. We draw it simply from the general text.
We are now telling you all this in such a rather detailed way, in fact. Are you not afraid that right now people are sitting behind the camera who are recording accurately and in a month will enter the market with competitors?
Well, I have this approach. I believe that this story with text and with the recognition of a natural language, it is only just beginning to develop, and there are a huge number of areas of application. And what people do there ... If someone succeeds, someone does something similar, I will only be glad we will use their experience. That is, I have here such a more open source story. This is closer to the review.
But are you unique today? Do you have competitors in the world?
Not.
Is it just a really unique company all over the globe?
Well actually.
Nobody does it like that?
There are similar products. Here the technology is developing very much. Of course, somewhere someone is better in something. For example, in the states there is a chatbot Mia, maybe she communicates better in the text than we do now? We do not exclude it here.
You close a very clear task.
If we talk about the voice of this is not. We in the states looked analogues. There is one company that ... if we can make a script in the office, then we need to call and record broadcast. Here is the phone number, call, write down what you want to tell the subscribers. You record it and they then scroll through this audio recording of your voice. That is, somehow a little bit wrong.
It would seem that you will be more technologically advanced.
It would seem that this whole thing is simple. But here I think that ... on the tangent of replacing, somehow copying and so on, here first of all, here we are, for example, we trained this sentimental neuron for more than two months. Purely training was going on. That is, Amazon, OpenAI.
But this is a processor power issue, right?
Yes.
Can this be done in 3 days, if you just buy more virtual machines?
Not. We already bought it.
You have already bought everything, everything is already over.
Yes. Well, we have it since, we are in partnership with Microsoft, we have virtually unlimited resources for these capacities. Therefore, we take large cars, use the best, probably what is now on the market. Well, I don’t know, the quality there is no different from IWS. And, in fact, we teach them. And even OpenAI, having its own resources, trained this model for a month. That is, therefore, the tasks here are quite complex. Firstly, it’s another part to train just a month, and the second task, which is even more difficult, is to find the data. That is, the question is what else to get ... we need more than just text.
You took the body of the text in some more or less banal? Wikipedia is a general place, it’s enough for everyone.
It alone is not enough. Well, if we want to add some more ... We did how, we took not just this building, we still ... we have, in general, about, probably, several hundred thousand vacancies already published on the service. That is, we took these texts, 2 million resumes.
Here you have competitors who have vacancies.
Yes, of course. They took a resume. That is, it is about 2 million resumes with us. We took, in general, even such vacancies, that is, we simply collected vacancies through ip work sites. That is, somewhere, probably, 10-15 GB of texts related specifically to HR-related topics are resumes, vacancies. That's all we took. That is, probably, without this there would have been little change. We roughly compared the model, which is only in the Russian case and in the Russian case with our texts. There is a difference of about 2-3%. That is, small, but it is all the same. And, how, why not do it?
Do they affect 3%? Look, in fact, when we talked with your colleague, at that moment the robot, in my opinion, only accepted “yes” and “no”. The level was like that. And already at this level it was very clear that ... Now I recalled, there was a good story about how TPPs cracked the encryption algorithm. She was great at making a hardware farm that knows how to distinguish a bad key from an unknown. Not bad from the good, but bad from the unknown.
But on this it turned out to remove two orders of volume, clear this thing, and drive away the rest of the keys simply with the usual selection. And you did the same? That is, you, in fact, have made some tool that allows you to distinguish exactly a bad candidate from an unknown. And thus, you also removed some significant amount of work, such meaningless, dumb work from a recruiter, and raised the effectiveness of the order by 2, probably? Somewhere at that level? And this is an essential driver. Everything else that you do is valuable, interesting, very curious, but it seems to me, from the point of view of business, this jump does not bring. Or I'm wrong?
Well, here we are ... certainly 2-3% here in this task, they of course ...
Do not the weather?
They don’t make much weather, but here we are, as it were, fighting for this ... we are going to this figure of 80 quality. Therefore, we consider this, probably, for us, the base line after which you can directly roll into production for all customers and say that we did it.
And now are you experimenting with the client to agree?
Yes, a few customers who are testing.
Ready to take risks ...
Yes, they use this technology. Well, and we, as it were, while holding it in a closed form for a bit. That is, we do not tell in measure.
Just told.
Yes. And, accordingly, here each here is some kind of small addition. That is, here we changed the metric from cosine to Euclidean, here we changed the model. Then we tweaked this model a little too, set the classification method to another, and it has already become a little better. Also 2-3%. And so, out of everything, we are going to these 80s. That is, there is such a process. Each component somehow affects this all.
On the other hand, now your competitors are two orders behind you, right? And it’s easy for them to close these 2 orders. Well, that is, it is generally a rather banal system with a rather banal recognizer of the words “yes” or “no”. And how, in fact, and you, if I remember correctly, in three months you did all this? Well, maybe ...
A little more. These have been months. That is, there without sleep ...
Well, you did not know yet that it is possible. They already know that. That is, in principle, to people who are currently sitting behind the camera, somewhere behind the screen from us, probably the question is one month there, in order to draw on your experience and success, to make an analogue that also implements these two orders. That is, in principle, your competitors can reach you pretty quickly.
Yes, of course. Moreover, there are already companies that offer similar services. That is, this is already happening in principle. But here we are now, as it were, we had an option there to improve our service here, to say that we are the most stable and say that we are the best, we are the very first and so on. But we, as it were, decided to go to the next stage. That is, we decided to go where there are no competitors. That is, competition is for losers, as there was one investor from the valley Peter Thiel. And, in fact, we are trying to do some things that are not there yet. Well, for example, there is no lively dialogue. That is, our task, that’s all I’m telling, these little details, they all go to the big task - creating such a lively dialogue.
That is, we now made a story in the test itself that after it recognizes these categories of questions, it’s, in principle, the answer is “yes” and the answer “no” is also a category in fact. And then, how are we now? That is, if earlier it was necessary to create a call script, and it was such a tedious process, in fact. Well, because there we give some kind of default script, standard, but anyway, each company is different. There, for example, a company addresses one to "you", the other to "you." That is, someone has an official presentation, someone else. Someone “hello”, someone there “good afternoon”. And, accordingly, here we are trying ... the candidate can ask the right, definite place to put, where he can ask questions, and she should ask: "What are your questions?" And we want the candidate to be able to ask a question at any time. I.e, she asks him: “Are you interested in work?”, he says: “And who are you?” And she answers him.
And she answers.
Yes.
That is, there maintains such a lively dialogue.
Yes. And after she answered, he says: "Yes, interesting." And she understands that the answer to her previous question. And now we are working on it. And this, of course, is such a big leap, here it will be in this communication, in our product. Because then the candidate will receive information. Not only receive information on vacancies, but just communicate as he is used to. And that will increase conversion of course by far.
Well, you are not the first. In Russia there are teams that did something similar. Do you somehow communicate, discuss?
Yes there is, and in the world. Of course, yes. Now, if we talk about systems such as these where natural language is recognized, where they try to build a lively dialogue, then ... if of course the big deal of big deal is to create a system there that will communicate there ... Jarvis Mark Zuckerberg recently showed. But at the same time, this, as it were, is such a big story. Giants work there: Facebook, Amazon, Google, Microsoft and so on. They all work on it. Well, at the same time there are smaller start-up companies that are engaged in some highly specialized task. For example, technical support. Very often it is automated, very often it is replaced using these technologies. There are people on bots. There is quite a lot of progress.
Of course we communicate with companies, with the guys who do this. For example, we are specialized in HR, we have our own field, our own subject, we need our own texts to train the model, we need some of our own specific patterns, our own categories. That is, this is what we are doing here with us. There are companies that do the same. In the states there is Mia, a number of companies. They are also working on this, but in a highly specialized format this still somehow works out. Well, there’s something to achieve, 80% and maybe even more.
It is segmented that you take a subset of the types of communication and lock yourself in it. Due to this, a little bit of the sematic model, having limitations on the class of semantics, probably acts more clearly. Look, in fact, again, this is what you are talking about - of course it’s nice and very interesting, it’s delicious to be proud of: “We have a robot that can talk.” Does this business have value?
Yes, we are, after all, probably our story, that we periodically want to go into sentiment analysis there or to go somewhere else. But we come from business, from its requests. That is where I just started. That the business has come and says: "We need the robot to start answering the candidate's questions, so that it is not a one-way, not a one-way game."
What about the metric? So you did it to some extent. How do you measure effect from a business perspective?
It's quite simple here. We have a metric. We are all talking about calls, we measure them in calls. That is, there is a certain timeline in the conversation between the recruiter and the candidate. There usually, if this is a position of some kind of management selection, a sales manager, then this is probably 10-15 minutes. And the first part is, of course, these three questions that we ask: “Do you need work? Is it interesting? And we have such conditions. You’ll come, you won’t come? ” The last part there is probably also a maximum of a couple of minutes there. That is, this is an interview agreement. Well, there’s probably even a minute. It’s there: tomorrow at three, but I can’t go there tomorrow, okay, come on the day after tomorrow. She looks at the calendar, writes down. That is, this is also a simple story. And here in the middle they have this communication.
And, in fact, our metric, it just lies in the fact that when we were looking for employees and doing this test, we measured how many of our accounts communicate with candidates. Here before this feature with questions for answers and after. And it turned out that we were able to halve the time of their communication. That is, they asked only those questions that they were not interested in. And this, if we take into account that they still asked questions: "What kind of robot is this?" etc. That is, they also spent time. And therefore, here it is unconditional. That is, our story is about reducing the time spent by a recruiter. But all the functions that we can now help him, we try to do it. Therefore, there is certainly such a business value.
That is, the metric is not a conversion, but a reduction in the costs of a living person?
Yes, while we are struggling with this metric. Conversion is such a next step. We certainly count her too. Well, in fact, here, in general, it probably does not change. That is, approximately, there we did not see any big difference. Candidates who receive answers to questions, she probably even gets a little bit smaller in the end. There are fewer of them, because some receive answers that do not suit them.
Which do not suit them, and it falls off. By the way, this is also a separate issue, which could also be discussed, but probably not this time. Our meeting today is drawing to a close, and you know, I’m talking to you right now and I remembered an episode from my childhood. I remember in my time, phonograph records were issued, which were printed in a magazine on plastic like that. A plastic page was inserted into the magazine, it had to be cut out, and there was a plate. And they inserted some completely enchanting such interesting things sometimes. And there, for example, was a story about how scientists made an artificial throat mechanical. She was spinning there somehow, cringing. And there was a sound recorded, uttered by this mechanical robot. And then it was not just a fantasy, but some such, in general, beyond. And it even seemed that this will never happen, and today we are discussing a situation in which the synthesis of speech itself is not even the subject of conversation. It simply exists, it simply exists and is already working, and even recognition as such, generally speaking, works.
And we are discussing not recognition of words, but recognition of the meaning of the spoken and practically communication at the level, well, in general, probably already of some kind of living mind of such an acting one. Probably the child there is two or three year old. Somewhere it gets there. That is, the question of getting into the conversation of a real person, probably, will already happen somewhere in our lifetime?
Yes, I think so. Anyway, I remain so optimistic in terms of technology. Of course, we are faced with a lot of difficulties, of course, all the same, these 20% are there, a sufficiently large amount remains that we do not recognize these answers. When we go to watch these 20%, we do not yet understand how to add these percentages to us. That is, so far we don’t even have ideas in general there. But at the same time, and we understand that this is also taking into account the fact that this is a very narrow area. That is, it’s only HR, it’s only about the vacancy, it’s just about this. And we still do not take issues in which 10-15% can still, which in general as a whole. Well, sometimes she is asked questions: “What is the weather like in Moscow?” They can also be processed, of course.
She must say: "Wait, Alice will call you back."
Of course, here is the fact that when we look at these errors, we understand that the technologies, they are not so much developed as to directly replace the person. And this one here is often very simple, I observe articles there, they write that this type completely replaced the person. This of course raises such a super-hype in this area, in fact, no.
Far away.
Yes, far away. I think that in our lifetime, I think, of course. Somewhere we come closer to this. Probably, 100% still will not come close, but we will be very close. But in general, this is a story about us treating this technology more as one that helps some simple tasks, simple problems for people to solve now. But of course, it does not completely replace people yet.
Leading. Well, so ... Probably, we will wait for the day when your system passes the Turing test and then call you again?
Yes, of course.