Here live dragons: a matrix of competencies as a tool of the timlid
It is possible that you will say: “The matrix of competencies? Seriously?". Most likely you have already heard something about this tool, and even made some conclusions, why you do not want to use it. Maybe it was just not before, or as a killer argument "it happened historically ...".
In fact, it is quite logical and not a new tool that can be very useful. And you can implement it in completely different ways, which we will try to prove to you on practical cases of two different teams - technical support and development. Based on them, you yourself will be able to estimate the labor costs (they may well be very different) and estimate a more suitable way for you in this direction.
And what have the dragons, explain under the cut.
This article is based on ours with Konstantin Kaftan on TeamLead Conf . Konstantin Timlid of the IPONWEB technical support department, and I am a technical writer of the development team at IPONWEB, I am engaged in knowledge management.
The story is built on the practical cases of IPONWEB, so first we will get a little familiar with the company so that you can understand whether this material is close to you.
Our HR gave us a request for a cool marketing picture that tells about our sphere:
Indeed, IPONWEB is developing platforms for online advertising customers. We work on the Software as a Service model (or more precisely Platform as a Service). We have a lot of components: HTTP server, predict algorithm, UI, reporting. Business logic for about 50 projects written in Lua.
We also have a flagship BidSwitch, which began as a spin-off. We are more interested in the integration of partners (demand-side, supply-side), so we have a lot of logs. This is the same project, on the same stack, but it is quite large, it has specific features and a dedicated technical support line, which Konstantin heads.
Why did we choose the name "Dragons Live Here", what does this mean? So earlier on medieval maps they designated places where it is not known what is happening - such Terra Incognita. There are two stories about this. The first is about the structure of skills, skills that a team has, and the bigger the team, the more such Terra Incognita arises; but about the second we will talk later.
How did we get here
It so happened that we (Svetlana and Konstantin) didn’t often overlap in work, and we found out that we applied for essentially the same subject matter only from the conference website . Then we decided to unite - it seemed to us that it would be more interesting for everyone.
Team Profiles
Team Constantine " Technical Support ":
My Lua Development team:
Command descriptions are needed to make it clear how our implementations of the matrices differ. We will try to describe how different teams approached this, what came out as a result, since the command structures, message, goals, triggers and final results turned out to be somewhat different.
Goals and Triggers
In the Technical Support team:
In the development team, the goals and triggers were slightly different. The initial incentive was precisely knowledge management, because, as already mentioned, we have a very specific business - Ad Tech. The likelihood that someone will come with ready-made knowledge is quite low, and at the same time for the developer, knowledge of business specifics is very important.
The Dev Team needed:
There was also a side goal - to make the performance review process more transparent, understandable, in order to formulate goals in a single coordinate system.
Performance review
Performance review is our company's standard. Every N months a performance review is carried out with each employee in the presence of an employee, his team leader, HR or notary is used as a referee. At the performance review we examine what the employee did of the tasks that were set, what he did not succeed, if it did not work, why, what we would like to accomplish, what we can offer. After that new goals, tasks are set, the next term is set.
In the support team, this period is rather short, because there are no large projects stretched in time - this is 3, maximum 4 months. In a development team, these are semi-annual review cycles. Previously, the dates were longer, simply because the developers can better map the road map. In some other development teams, for example, the frontend performance review also occurs every 3-4 months. Perhaps the interval has nothing to do with the development or support of the team, but it happened with us.
Why do you need a matrix of competencies
A year ago, our company went through a period of intensive growth, there were more people, more tools, more tasks - more of them all. At one point, it became clear that without the correct documents it is difficult to understand what is happening. And we found a solution how to systematize information: about tasks; tools and skills; employees.
And why is all of you? You will learn about our practical cases, but you may have a reasonable question - how to understand what I need?
If you have problems that are similar to the ones we outlined above, that is, you do not understand the full pool of tasks for your developers, you have been learning new employees and other problems that were already called for too long, then you probably need them.
Where to begin
Where to start tell you about your example.
In the technical support team we: defined the tasks we are engaged in; decomposed the tasks into separate manipulations; found out what tools the employee should know to work in a particular sector.
Below is a somewhat simplified example.
The team has employees Batman, Joker, Flash and Ivanov, and standard tasks:
From the tools we have:
Then we found out who knows what and in what extent.
2 - an expert who can teach; 1 - it means a person more or less knows; empty cells instead, so as not to offend anyone.
We summarize in one table: The
gray highlighted those tools that are not required in a particular task. For example, for general diagnostics, we don’t necessarily need knowledge of BQ and Grafana. It turned out quite an interesting picture.
Nezhdanchiki (Support)
A few nezhdanchikov that have helped to understand:
More details on each of the points later.
Let's go back to the table.
Formally, in the general diagnosis of busy Batman. The table shows that the Joker, if necessary, will replace it well, given that he is still an expert in the UI - why not? That is, we have Batman replaceable.
Joker is generally well done - he is savvy almost everywhere, except perhaps deploy monitoring, but this can be taught.
It is well known that the Joker spends some extracentric free time and at any time may fall out of work. What will happen then? Then everything will be sad with traffic deal diagnostics.
Moreover, we do not have a BQ expert! That is, we urgently need someone to pull up, train. For example, you can train Batman BQ, and Ivanov - Graphite.
Even with this simplified assessment system, you can get a certain amount for each of the employees to understand how competent and well-grounded he is. These estimates threw another nezhdanchik.
Two more interesting things are the total amount of the team and its average score. Strictly speaking, these figures themselves are not particularly interesting - the dynamics of their changes from month to month are interesting. If for some reason we started to drop total, then most likely someone left, and a weak employee came - a problem with the bundle, you need to turn to HR. If the average suddenly began to fall, it means that we have something wrong with learning - the Timlid is guilty and only the Timlid.
It is important that in this case it is a tool of a timlid that is not shown to employees, because assessments are subjective.
Dev team
In Dev Team, everything is a little more democratic, or maybe we just complicated our task. The list of skills in the development team is longer, although the homogeneity of the tasks is probably higher.
We gathered a board of 4 experienced developers (including a team leader and a knowledge management specialist), brainstormed, and analyzed tasks and skills. We just went step by step and on our personal feelings, but a few people, analyzed the tasks, laid out their skills, formed a list.
At first it was a list, then it was structured into more versatile skills, which include coding and separate business skills, because for our developers this is an important part. And from this list of skills already made a matrix.
In the first iteration of the matrix, we did not include the so-called soft-skills, that is, something that cannot be called hard-skills. This is something related to daily work, and even part of critical skills, including: planning; communication with the team; ability to manage the development process in mini-groups or by yourself; ability to formulate requirements, for example, during code review.
We did not immediately include all these skills, because we did not understand how to evaluate them accurately.
In the second iteration, they entered the rating system:
By the way, about soft-skills in support. The fact is that the soft skills in the development team require a separate assessment. However, personal qualities that are only desirable for the developer are mandatory for the support team . If a person is hired, then he already has them by default.
In development, we abandoned the very concept of “soft skills”, and called it universal skills. They do not include, for example, knowledge of the English language or the fact that you do not enrage your team - this is not there, because it’s just closer to the soft skills that describes a person’s character. We estimate skills somewhere on the verge, and therefore came up with this name - universal skills .
When the list of skills was ready, we conducted their assessment using a continuous survey. We conducted a real exam, in which some of the questions suggested an unambiguous answer, and some discussed working situations: “What to do if someone delays or does not do something?”, “How would you implement this or that functionality in the project?” . This helped us a lot, because this part of the questions allowed us to talk to a lot of developers.
Then we progreydiroval developers on a scale of 1 to 3 for each skill.
Those skills that we could not assess using the primary survey, we evaluated using peers (peer), that is, asked to evaluate colleagues. We didn’t have any group leaders at that time, otherwise they could have done it.
This matrix looks like this.
Of course, everything did not fit in the picture, so some things, for example, correlations and the profile of the developers are incomprehensible here. But even so, you can roughly estimate what is happening and see how clearly it is and what can be measured.
Colors correspond to the estimate. The same metrics are output as in the support team - total and various average skills.
Pay attention to Dev8. From the table it turns out that this is like the first candidate for dismissal. But, firstly, he has specific knowledge on one of the positions, so we need it. But more importantly, this table is not a dismissal tool . I will also tell you about this, we didn’t think it was for this.
Average by skills is a job, according to the results of which a plan should be formed on how to tighten a skill. That is, if the developers do not know something - this is not their problem. This means that we did not give them enough information, and in general, knowledge of some of the skills is generally not enough in the company. It is necessary either to write documentation, or to make some kind of a playground, so that they can practice, or to buy some external course, so that they can learn, conduct Knowledge Sharing Session, send them to each other, study, work in pair for some time, include a person in another project. There may be different variants of knowledge sharing. The bottom line is that you need to first hold an event, and only then talk about the dynamics.
It is important to note that since our company has a matrix structure, not all developers need some skills. They will receive them when they complete the task associated with this skill. Otherwise meaningless, especially if the skill is not included in the list of critical. For example, in some of the projects there is no uPredict, this is not a critical skill. In general, this approach has helped us formulate which skills are key and which are not, and what the specialist will be able to catch up when he has such a practical task.
Another important point about grading. We initially made a mistake, which was provoked by more than it could be, the resistance of the team - we did not verbalize the value of estimated 1, 2 and 3. They were in our head at the level of intuition, but we could not answer why one developer got 1, and the second - 2. The developers shared with each other the results of the mud and this provoked conflicts. Unlike the support group, we provided personal ratings on request, but not strangers.
It is necessary to clearly define and convey this to everyone, for example, that in the knowledge of Lua, an estimate:
So you can decompose each skill, because somewhere at the level of intuition you already have it.
How to implement
When the primary matrix was ready, we tied the metrics to the development team . Basically it was the percentage of maximum marks for skills, which allows you to put a grade developer.
In addition to assessments for individual skills, we needed to upgrade developers. We had grades A, B and C - something like junior, middle, senior. In IT junior, middle, senior is already too full of meaning, which invests, including work experience. We needed designations that absolutely had no connotations in advance. A, B and C are not even marks in an American school. The "max skill percentage" metric helped us to do this.
The remaining metrics are various percentages of individual skills and average.
We also tried to buildcorrelation with other methods of employee assessment , including salary. Of course, we didn’t want to tie a salary to the metrics, it was just an exercise, but everything was quite well related.
The idea is that by correlating the estimates on the matrix with the salary, you check some of your thoughts on this matter, and you can estimate how much you pay for each specific percentage of maximum skills. You can roughly decompose, for what and how much you can pay extra, where to change salary. Or to reveal that you are not seriously paying someone extra - it turns out that there is a person who has a unique knowledge or a fairly good set of skills, and you were wrong.
We also tied here the metrics that we take from other systems. We have version control metrics associated with rollbacks or review reviews, for example. We also compared them, but we have no goal to make tight bindings. We simply confirm our hypotheses in this way.
Subsequently, performance review goals were set in terms of this matrix. That is, we could clearly explain the difference between 1 and 2, and in which (only a few) positions, the developer needs to catch up in six months.
Nezhdanchik (Dev Team)
In the development, too, met unexpected people. Basically, we got a lot of what we wanted. We have a clear understanding in the form of a map with which you can work, where there are white spots, where you need to train the team. We completely rewrote the training plan for new employees. As a result, the process of human adaptation to a living project was reduced from 5-6 months to 2-3. That is, now for the trial period, we fully manage to implement a new employee in the project. We completely rewrote the onboarding plan, because we were able to identify a list of key skills that beginners need right away. And for secondary skills (or very specific to individual projects) knowledge artifacts can be created in the learning process.
At that moment we had interns, and we did not plan to take them all, but only one of the four. But they took three. Directly, we do not associate the fact that we have three interns have taken root at once, so that we have made a matrix of competencies. But we tested many of our considerations on them, and they were the first to go through a new training plan. In particular, we have included a lot of practice in it, we have created a playground, so that they can practice their skills using critical skills.
Besides the fact that we have revealed who and how loyal the company isand who wants to be unique and irreplaceable. There were critical skills that people wanted to share, while at the same time, they found bottlenecks in which people reveled in how unique and irreplaceable they were. In this case, the bus factor was the result of a conscious decision. We could immediately document this part of the framework and train the second person for this. Although there was resistance from a human expert who did not want to share knowledge and wanted to implement it only in his project.
We identified enthusiasts and team drivers. Those who immediately wanted to contribute something: “Let's add something else here? And why do not we appreciate it? ".
Quite unexpectedly, what we were not going to look for at all - we found those things that can be transferred to the server part of the code, to the libraries, so as not to reinvent the wheel. It turned out more thanks to a full exam and additional communication.
Cases
Above is a typical case when you can transfer part of the code to a shared library. There is some knowledge that is not enough to share, you need to transfer it to the code base.
In this example, only one developer knows how to configure CI, because it is he who configured it. This is a bottleneck in which to share knowledge. By the way, this is the same Number 8.
When we know which skills are not critical, we can use different filters to understand which skills with a low average rating are critical and which ones are just specific for business units. We are still learning how to calculate developer profiles using this matrix.
Cost of implementation of the competency matrix
What interests everyone is the sacred man-hours. Everyone is always interested in the question of labor costs.
In the case of customer support, it took:
And that's it! To maintain the matrix up to date enough on the strength of 30 minutes per week. Of course, it depends on how intensively the rotation of colleagues goes, new tasks arrive, or, on the contrary, they die off.
Dev team
In the development team, compared with the support, there is a slightly frightening difference in labor costs. But there is an important point: the initial assessment of skills (making a list) took the same 3-4 hours. The main labor costs were completely different, they consist of the following:
But the point is that somewhere in the middle of the path you get a working tool with which you can already do something. Even at the final stage it is quite possible to work with the list and the first grades.
Detailing
We will separately dwell on how important it is to stop in time .
You need to understand what you still want, and how much you are willing to invest time. The first stage, when we simply paint the skills, distribute employees, apply an elementary assessment, is not particularly labor-intensive. Then you can complicate: enter more metrics, skill average, conduct a continuous survey, etc.
Support stopped somewhere between easy and medium, and the Dev Team - about medium.
For each manipulation, for each tool, you can also enter the complexity factor, and already on these coefficients to calculate the final total and average score. We did not go that far, but stopped at a moderate degree of difficulty. The bottom line is that you can build up levels of abstraction, but the question is whether you need it.
What not to do
What the competence matrix is not and what you should not do with it.
You should not try to tie the salary, dismissal, other formal measures to the matrix. This is your tool (or command tool) that is not needed for this. You can attach a performance review to it, so that everyone can speak the same language.
If you have a rather heterogeneous team, consider whether you need one matrix or several - those very profiles. We are now going in this direction and have not yet solved this problem.
It happens when HR deals with this, but then you can get a result, first, not the first time, secondly, not the one that was expected, thirdly, HR will still draw some conclusions and make it nervous. It’s just that HR has other tasks, they may like this tool and they can use it for their own purposes, which sometimes do not agree with the tasks of the team leader.
In each team it may look different. In our experience, other teams in the company became interested in this tool. They also want to try, but with a different approach, they have other reasons. They will need to start everything from a clean slate, and not just take our matrices and try to apply them. Each team will have its own matrix - in front-end development one, in a business-oriented team - another. These matrices will differ not only in the list and the number of skills.
Slides, additional materials and our matrix here .
In fact, it is quite logical and not a new tool that can be very useful. And you can implement it in completely different ways, which we will try to prove to you on practical cases of two different teams - technical support and development. Based on them, you yourself will be able to estimate the labor costs (they may well be very different) and estimate a more suitable way for you in this direction.
And what have the dragons, explain under the cut.
This article is based on ours with Konstantin Kaftan on TeamLead Conf . Konstantin Timlid of the IPONWEB technical support department, and I am a technical writer of the development team at IPONWEB, I am engaged in knowledge management.
The story is built on the practical cases of IPONWEB, so first we will get a little familiar with the company so that you can understand whether this material is close to you.
Our HR gave us a request for a cool marketing picture that tells about our sphere:
Indeed, IPONWEB is developing platforms for online advertising customers. We work on the Software as a Service model (or more precisely Platform as a Service). We have a lot of components: HTTP server, predict algorithm, UI, reporting. Business logic for about 50 projects written in Lua.
We also have a flagship BidSwitch, which began as a spin-off. We are more interested in the integration of partners (demand-side, supply-side), so we have a lot of logs. This is the same project, on the same stack, but it is quite large, it has specific features and a dedicated technical support line, which Konstantin heads.
Why did we choose the name "Dragons Live Here", what does this mean? So earlier on medieval maps they designated places where it is not known what is happening - such Terra Incognita. There are two stories about this. The first is about the structure of skills, skills that a team has, and the bigger the team, the more such Terra Incognita arises; but about the second we will talk later.
How did we get here
It so happened that we (Svetlana and Konstantin) didn’t often overlap in work, and we found out that we applied for essentially the same subject matter only from the conference website . Then we decided to unite - it seemed to us that it would be more interesting for everyone.
Team Profiles
Team Constantine " Technical Support ":
- Headcount: 10-12 people.
- Distance with team lid: short - all of each other in sight.
- A set of skills: a very diverse set of skills, competencies, work tools - this is due to the fact that we have a way to drag off new tasks for our colleagues from other teams and departments.
- Work style: abduction of tasks from colleagues. This is done, firstly, in order to dilute the support routine, because day after day it is not at all interesting to analyze the same type of requests. Secondly, to see who gets what, and outline the direction of development.
My Lua Development team:
- Headcount: 40 people, but the number is constantly changing;
- Distance with team lead: long enough due to the specifics of the management structure in the company (it is a matrix). That is, the team leader does not see every developer every day and does not know exactly what his daily tasks are.
- Skill set: more or less uniform. The developers write business logic on Lua, but at the same time they are all included in the business unit, i.e. in combining projects, for example, by whether they sell advertising space or buy advertising as an advertiser.
- Work style: specific tasks, bus-factor. Often knowledge is concentrated in the minds of individuals, not the whole team is interchangeable.
Command descriptions are needed to make it clear how our implementations of the matrices differ. We will try to describe how different teams approached this, what came out as a result, since the command structures, message, goals, triggers and final results turned out to be somewhat different.
Goals and Triggers
In the Technical Support team:
- Who is busy and what? It took to find out when everything became too much, and ceased to fit in the head, who is still busy, who knows what and at what level.
- What kind of specialists are missing? As a result - who is not enough to complete the tasks, who needs to be retrained.
- What competences are common and necessary for all? The main thing is to understand at the selection stage what is common and obligatory, what you need to write for HR in the profile of the ideal candidate.
In the development team, the goals and triggers were slightly different. The initial incentive was precisely knowledge management, because, as already mentioned, we have a very specific business - Ad Tech. The likelihood that someone will come with ready-made knowledge is quite low, and at the same time for the developer, knowledge of business specifics is very important.
The Dev Team needed:
- Speed up onboarding newbies . Long-term inclusion of developers in the project is a big pain. Onboarding took about six months, that is, more than a trial period, which means we will train the developer at our own expense.
- Understand the structure of knowledge, assess the gaps in knowledge , find white spots, outline a training plan, outline individual growth paths.
- Due to the matrix structure, it is necessary to distribute developers more evenly across business units and tasks, and not to offend anyone. That is, do not send all seniors into one business unit.
There was also a side goal - to make the performance review process more transparent, understandable, in order to formulate goals in a single coordinate system.
Performance review
Performance review is our company's standard. Every N months a performance review is carried out with each employee in the presence of an employee, his team leader, HR or notary is used as a referee. At the performance review we examine what the employee did of the tasks that were set, what he did not succeed, if it did not work, why, what we would like to accomplish, what we can offer. After that new goals, tasks are set, the next term is set.
In the support team, this period is rather short, because there are no large projects stretched in time - this is 3, maximum 4 months. In a development team, these are semi-annual review cycles. Previously, the dates were longer, simply because the developers can better map the road map. In some other development teams, for example, the frontend performance review also occurs every 3-4 months. Perhaps the interval has nothing to do with the development or support of the team, but it happened with us.
Why do you need a matrix of competencies
A year ago, our company went through a period of intensive growth, there were more people, more tools, more tasks - more of them all. At one point, it became clear that without the correct documents it is difficult to understand what is happening. And we found a solution how to systematize information: about tasks; tools and skills; employees.
And why is all of you? You will learn about our practical cases, but you may have a reasonable question - how to understand what I need?
If you have problems that are similar to the ones we outlined above, that is, you do not understand the full pool of tasks for your developers, you have been learning new employees and other problems that were already called for too long, then you probably need them.
Where to begin
Where to start tell you about your example.
In the technical support team we: defined the tasks we are engaged in; decomposed the tasks into separate manipulations; found out what tools the employee should know to work in a particular sector.
Below is a somewhat simplified example.
The team has employees Batman, Joker, Flash and Ivanov, and standard tasks:
- general traffic diagnostics;
- traffic deal diagnostics;
- Deployments monitoring;
- testing the integration of new partners, both DSP and SSP, both buyers and sellers.
From the tools we have:
- uSIicer allows you to get very detailed trade figures;
- Graphite provides real-time system status data;
- Google BigQuery (BQ) - our logs live there;
- Grafana, which allows you to aggregate metrics from Graphite, it is convenient to set alerts on it if necessary;
- Ul-adjustments in the admin interface, which allow you to change traffic parameters for a particular partner, shopping pair, data center, etc.
Then we found out who knows what and in what extent.
2 - an expert who can teach; 1 - it means a person more or less knows; empty cells instead, so as not to offend anyone.
We summarize in one table: The
gray highlighted those tools that are not required in a particular task. For example, for general diagnostics, we don’t necessarily need knowledge of BQ and Grafana. It turned out quite an interesting picture.
Nezhdanchiki (Support)
A few nezhdanchikov that have helped to understand:
- Which of the colleagues should be able to improve their skills in order to attract them to new tasks? (Mark the direction of development);
- Where can there be problems in case of loss of an employee?
- How adequate is the salary of an employee to his task pool?
- How well is the team staffed and trained?
More details on each of the points later.
Let's go back to the table.
Formally, in the general diagnosis of busy Batman. The table shows that the Joker, if necessary, will replace it well, given that he is still an expert in the UI - why not? That is, we have Batman replaceable.
Joker is generally well done - he is savvy almost everywhere, except perhaps deploy monitoring, but this can be taught.
It is well known that the Joker spends some extracentric free time and at any time may fall out of work. What will happen then? Then everything will be sad with traffic deal diagnostics.
Moreover, we do not have a BQ expert! That is, we urgently need someone to pull up, train. For example, you can train Batman BQ, and Ivanov - Graphite.
Even with this simplified assessment system, you can get a certain amount for each of the employees to understand how competent and well-grounded he is. These estimates threw another nezhdanchik.
Two more interesting things are the total amount of the team and its average score. Strictly speaking, these figures themselves are not particularly interesting - the dynamics of their changes from month to month are interesting. If for some reason we started to drop total, then most likely someone left, and a weak employee came - a problem with the bundle, you need to turn to HR. If the average suddenly began to fall, it means that we have something wrong with learning - the Timlid is guilty and only the Timlid.
It is important that in this case it is a tool of a timlid that is not shown to employees, because assessments are subjective.
Dev team
In Dev Team, everything is a little more democratic, or maybe we just complicated our task. The list of skills in the development team is longer, although the homogeneity of the tasks is probably higher.
We gathered a board of 4 experienced developers (including a team leader and a knowledge management specialist), brainstormed, and analyzed tasks and skills. We just went step by step and on our personal feelings, but a few people, analyzed the tasks, laid out their skills, formed a list.
At first it was a list, then it was structured into more versatile skills, which include coding and separate business skills, because for our developers this is an important part. And from this list of skills already made a matrix.
In the first iteration of the matrix, we did not include the so-called soft-skills, that is, something that cannot be called hard-skills. This is something related to daily work, and even part of critical skills, including: planning; communication with the team; ability to manage the development process in mini-groups or by yourself; ability to formulate requirements, for example, during code review.
We did not immediately include all these skills, because we did not understand how to evaluate them accurately.
In the second iteration, they entered the rating system:
- decomposition of tasks and everything connected with it;
- documentation and self-documented code;
- the ability to communicate with the project manager, that is, to understand what he says, and to raise the necessary questions in turn;
- the ability to correctly formulate comments on the code review, etc.
By the way, about soft-skills in support. The fact is that the soft skills in the development team require a separate assessment. However, personal qualities that are only desirable for the developer are mandatory for the support team . If a person is hired, then he already has them by default.
In development, we abandoned the very concept of “soft skills”, and called it universal skills. They do not include, for example, knowledge of the English language or the fact that you do not enrage your team - this is not there, because it’s just closer to the soft skills that describes a person’s character. We estimate skills somewhere on the verge, and therefore came up with this name - universal skills .
When the list of skills was ready, we conducted their assessment using a continuous survey. We conducted a real exam, in which some of the questions suggested an unambiguous answer, and some discussed working situations: “What to do if someone delays or does not do something?”, “How would you implement this or that functionality in the project?” . This helped us a lot, because this part of the questions allowed us to talk to a lot of developers.
Then we progreydiroval developers on a scale of 1 to 3 for each skill.
Those skills that we could not assess using the primary survey, we evaluated using peers (peer), that is, asked to evaluate colleagues. We didn’t have any group leaders at that time, otherwise they could have done it.
This matrix looks like this.
Of course, everything did not fit in the picture, so some things, for example, correlations and the profile of the developers are incomprehensible here. But even so, you can roughly estimate what is happening and see how clearly it is and what can be measured.
Colors correspond to the estimate. The same metrics are output as in the support team - total and various average skills.
Pay attention to Dev8. From the table it turns out that this is like the first candidate for dismissal. But, firstly, he has specific knowledge on one of the positions, so we need it. But more importantly, this table is not a dismissal tool . I will also tell you about this, we didn’t think it was for this.
Average by skills is a job, according to the results of which a plan should be formed on how to tighten a skill. That is, if the developers do not know something - this is not their problem. This means that we did not give them enough information, and in general, knowledge of some of the skills is generally not enough in the company. It is necessary either to write documentation, or to make some kind of a playground, so that they can practice, or to buy some external course, so that they can learn, conduct Knowledge Sharing Session, send them to each other, study, work in pair for some time, include a person in another project. There may be different variants of knowledge sharing. The bottom line is that you need to first hold an event, and only then talk about the dynamics.
It is important to note that since our company has a matrix structure, not all developers need some skills. They will receive them when they complete the task associated with this skill. Otherwise meaningless, especially if the skill is not included in the list of critical. For example, in some of the projects there is no uPredict, this is not a critical skill. In general, this approach has helped us formulate which skills are key and which are not, and what the specialist will be able to catch up when he has such a practical task.
Another important point about grading. We initially made a mistake, which was provoked by more than it could be, the resistance of the team - we did not verbalize the value of estimated 1, 2 and 3. They were in our head at the level of intuition, but we could not answer why one developer got 1, and the second - 2. The developers shared with each other the results of the mud and this provoked conflicts. Unlike the support group, we provided personal ratings on request, but not strangers.
It is necessary to clearly define and convey this to everyone, for example, that in the knowledge of Lua, an estimate:
- 1 - means that the developer knows all the basic operators, knows how to work with data structures and write classes of functions;
- 2 - can organize code into libraries, works consistently with them in the course of the whole project and knows what metatables are;
- 3 - knows the specifics of the company and projects, for example, what is korutiny (the specifics of our server working with Lua), how does C ++ code work with Lua code, what restrictions this imposes, how the code is interpreted, how it is compiled, how our JIT compiler works.
So you can decompose each skill, because somewhere at the level of intuition you already have it.
Note to self:
- Verbalize the rating system, even if you don't show it to anyone.
- At the earliest stages, try to include the work of experienced developers. You know, who may have the most resistance, immediately turn them into work. Remove all objections in advance, as is done in sales.
How to implement
When the primary matrix was ready, we tied the metrics to the development team . Basically it was the percentage of maximum marks for skills, which allows you to put a grade developer.
In addition to assessments for individual skills, we needed to upgrade developers. We had grades A, B and C - something like junior, middle, senior. In IT junior, middle, senior is already too full of meaning, which invests, including work experience. We needed designations that absolutely had no connotations in advance. A, B and C are not even marks in an American school. The "max skill percentage" metric helped us to do this.
The remaining metrics are various percentages of individual skills and average.
We also tried to buildcorrelation with other methods of employee assessment , including salary. Of course, we didn’t want to tie a salary to the metrics, it was just an exercise, but everything was quite well related.
The idea is that by correlating the estimates on the matrix with the salary, you check some of your thoughts on this matter, and you can estimate how much you pay for each specific percentage of maximum skills. You can roughly decompose, for what and how much you can pay extra, where to change salary. Or to reveal that you are not seriously paying someone extra - it turns out that there is a person who has a unique knowledge or a fairly good set of skills, and you were wrong.
We also tied here the metrics that we take from other systems. We have version control metrics associated with rollbacks or review reviews, for example. We also compared them, but we have no goal to make tight bindings. We simply confirm our hypotheses in this way.
Subsequently, performance review goals were set in terms of this matrix. That is, we could clearly explain the difference between 1 and 2, and in which (only a few) positions, the developer needs to catch up in six months.
We have developed an internal rule - not to give after the performance review more than three goals in terms of the matrix.
Nezhdanchik (Dev Team)
In the development, too, met unexpected people. Basically, we got a lot of what we wanted. We have a clear understanding in the form of a map with which you can work, where there are white spots, where you need to train the team. We completely rewrote the training plan for new employees. As a result, the process of human adaptation to a living project was reduced from 5-6 months to 2-3. That is, now for the trial period, we fully manage to implement a new employee in the project. We completely rewrote the onboarding plan, because we were able to identify a list of key skills that beginners need right away. And for secondary skills (or very specific to individual projects) knowledge artifacts can be created in the learning process.
At that moment we had interns, and we did not plan to take them all, but only one of the four. But they took three. Directly, we do not associate the fact that we have three interns have taken root at once, so that we have made a matrix of competencies. But we tested many of our considerations on them, and they were the first to go through a new training plan. In particular, we have included a lot of practice in it, we have created a playground, so that they can practice their skills using critical skills.
Besides the fact that we have revealed who and how loyal the company isand who wants to be unique and irreplaceable. There were critical skills that people wanted to share, while at the same time, they found bottlenecks in which people reveled in how unique and irreplaceable they were. In this case, the bus factor was the result of a conscious decision. We could immediately document this part of the framework and train the second person for this. Although there was resistance from a human expert who did not want to share knowledge and wanted to implement it only in his project.
We identified enthusiasts and team drivers. Those who immediately wanted to contribute something: “Let's add something else here? And why do not we appreciate it? ".
Quite unexpectedly, what we were not going to look for at all - we found those things that can be transferred to the server part of the code, to the libraries, so as not to reinvent the wheel. It turned out more thanks to a full exam and additional communication.
Cases
Above is a typical case when you can transfer part of the code to a shared library. There is some knowledge that is not enough to share, you need to transfer it to the code base.
In this example, only one developer knows how to configure CI, because it is he who configured it. This is a bottleneck in which to share knowledge. By the way, this is the same Number 8.
When we know which skills are not critical, we can use different filters to understand which skills with a low average rating are critical and which ones are just specific for business units. We are still learning how to calculate developer profiles using this matrix.
Cost of implementation of the competency matrix
What interests everyone is the sacred man-hours. Everyone is always interested in the question of labor costs.
In the case of customer support, it took:
- 4 hours for the first version, i.e. decomposition;
- 1 hour to put everything in one table;
- 0.5 hours and a couple of beer mugs to an HR expert who said that this is called a competency matrix, and it was possible not to reinvent the wheel and spend even less time.
And that's it! To maintain the matrix up to date enough on the strength of 30 minutes per week. Of course, it depends on how intensively the rotation of colleagues goes, new tasks arrive, or, on the contrary, they die off.
Dev team
In the development team, compared with the support, there is a slightly frightening difference in labor costs. But there is an important point: the initial assessment of skills (making a list) took the same 3-4 hours. The main labor costs were completely different, they consist of the following:
- The initial assessment, brainstorming sessions - 3-4 hours for each expert (we had 4 experts) - the average time, because not all experts constantly sat in one room and talked.
- The compilation of the matrix itself, including the adduction of developers and skills, the setting of macro formulas, so that you can conveniently use the table, add additional metrics and formulas - it took 6 hours of work.
- Time for additional adjustment of the test. We already had a quiz for beginners. I had to inflate it a little and make it universal, so that it was suitable for senior'ov. On rewriting this quiz, it took 2 hours from a knowledge-manager and 2 from a team leader.
- The most expensive part is the assessment by exam (quiz in 3-4 rounds). It takes a lot of time, because in the quiz there are more than 100 questions that go through the profiles.
- Time to adjust the metrics - another 2 hours. If you have additional assessment techniques, it may take a little longer for you to complete this item.
But the point is that somewhere in the middle of the path you get a working tool with which you can already do something. Even at the final stage it is quite possible to work with the list and the first grades.
Detailing
We will separately dwell on how important it is to stop in time .
The competency matrix is a tool, not a universal solution to all tasks.
You need to understand what you still want, and how much you are willing to invest time. The first stage, when we simply paint the skills, distribute employees, apply an elementary assessment, is not particularly labor-intensive. Then you can complicate: enter more metrics, skill average, conduct a continuous survey, etc.
Support stopped somewhere between easy and medium, and the Dev Team - about medium.
For each manipulation, for each tool, you can also enter the complexity factor, and already on these coefficients to calculate the final total and average score. We did not go that far, but stopped at a moderate degree of difficulty. The bottom line is that you can build up levels of abstraction, but the question is whether you need it.
What not to do
What the competence matrix is not and what you should not do with it.
- To consider it a universal tool , a frame with which all must comply.
You should not try to tie the salary, dismissal, other formal measures to the matrix. This is your tool (or command tool) that is not needed for this. You can attach a performance review to it, so that everyone can speak the same language.
If you have a rather heterogeneous team, consider whether you need one matrix or several - those very profiles. We are now going in this direction and have not yet solved this problem.
- Consider it a necessity. Perhaps, if the team is small, everything is visible, the task pool does not change, then all this can be remembered and not bother.
- If you, as a team leader, still undertook it - do not give this job to anyone. No one knows your team better than you. Only one surgeon operates, the second one either assists or interferes.
It happens when HR deals with this, but then you can get a result, first, not the first time, secondly, not the one that was expected, thirdly, HR will still draw some conclusions and make it nervous. It’s just that HR has other tasks, they may like this tool and they can use it for their own purposes, which sometimes do not agree with the tasks of the team leader.
- Do not try to distribute the matrix to the whole company, it is a tool of a single timlid .
In each team it may look different. In our experience, other teams in the company became interested in this tool. They also want to try, but with a different approach, they have other reasons. They will need to start everything from a clean slate, and not just take our matrices and try to apply them. Each team will have its own matrix - in front-end development one, in a business-oriented team - another. These matrices will differ not only in the list and the number of skills.
- Don't try this at home - it doesn't work in companies either, and at home, all the more :)
Slides, additional materials and our matrix here .
And if the question of knowledge management is as relevant for you as it is for me, I invite you to participate in the special KnowledgeConf 2019 conference in Moscow on April 26.
We plan to discuss a variety of problems, from applied ones (onboarding newbies, organizing a knowledge base, an internal university, learning by doing, fixing incidents and maintaining a post-mortem base) to fundamental ones (modern approaches to learning and systems thinking).
We have already begun work with the program committee and received the first twenty topics. Apply if you value knowledge within the company, and know how to handle them. And we will help bring the material to the class report.