Methods of assessing the knowledge of the engineer. The path of the architect and the path of the expert
Is it possible to represent the level of knowledge and skills of an engineer or even a team on one picture?
Parameters, in fact, a lot. But we have a wonderful example of how difficult it can be visualized in a rather simple and visual way.
I'm talking about (I think everyone knows) the Gartner quadrant.
This article is about how this visualization approach was applied to assess the level of knowledge and skills of network engineers in a Russian company.
When conducting the aforementioned survey, it was decided to display the level of knowledge of the engineer in two dimensions. The value deposited on the axis OX characterizes the depth of knowledge and on the axis OY - the breadth of the technical outlook.
Of course, it is important to initially limit the range of topics addressed, in accordance with the requirements of the company. In our case, these were R & S, Security, Service Provider.
Evaluation of both "latitude" and "depth" is on a 10-point scale.
We can distinguish 4 quadrants, which we conditionally call:
Also in each quadrant you can select subzones. For example, we can assume that if the engineer’s knowledge is in the “latitude” zone 7–10, the “depth” 7-10, then this is an engineer who possesses fundamental, expert knowledge of many technologies, with extensive experience and skills, and is able to perform virtually any tasks.
On this graph you can also take into account such characteristics as
So, good communicative qualities, as it seems to us, are a plus to “breadth”, since help the engineer to receive information on related areas. The remaining 3 qualities are a plus for both “depth” and “latitude”.
Perhaps this somewhat distorts the picture at the moment, but allows you to take into account the dynamics, which in our opinion is a significant factor.
The one-dimensional visualization approach has some advantage over the two-dimensional one, since allows you to enter an unambiguous "better-worse" for engineers and thus rank them according to their level of knowledge and skills. But such an approach is possible only if the ultimate goal in the development of the engineer is defined.
We can identify at least two goals:
It assumes IT erudition and a fairly deep level of knowledge in many areas.
The engineer focuses on one direction. He knows the details, nuances, theory, perhaps down to the level of programming, mathematics, knowledge of components ...
There is also a third goal (IT manager), which we will not discuss in this article, because it is not an engineering goal.
As it is easy to see, these goals correspond to the third (architect), fourth (narrow specialist) and second (IT manager) quadrants of a two-dimensional visualization scheme.
In each of these two (engineering objectives) cases, we can link a one-dimensional scheme to a two-dimensional approach. Thus, on a two-dimensional map map we can get
We can distinguish 4 stages in development, which we conditionally call
Its stages are similar to those of the “Path of the Architect”, but not only in the field of the entire industry (or a large field of knowledge related to telecom), but in a single, rather narrow direction. The fact that this path goes beyond the boundaries of the scale suggests that in order to reach the “star level” and thus be able to determine the direction of development of this narrow area, not only an excellent level of engineering knowledge, but also specific knowledge is usually required (for example, knowledge of , radio electronics, programming, mathematics ...)
You have a questionnaire consisting of 100 questions. The questionnaire corresponds to 10 topics - 10 questions for each topic. All topics are equivalent in terms of importance. It is assumed that the test is compiled “ideally” and if the examiner answered all 10 questions on the topic, he knows it 100 percent (and this, of course, never happens, but in this article we are not talking about how to make a test, about the assessment method, when you were able to somehow determine the distribution of knowledge and skills by topic).
After the test, you will have a distribution of points (the number of answered questions in each topic) of 10 topics. For example, it could be something like:
Q1: 9
Q2: 7
Q3: 4
Q4: 6
Q5: 0
Q6: 3
Q7: 6
Q8: 1
Q9: 1
Q10: 3
Good. And what place in our two-dimensional visualization will the evaluation of this engineer take?
We see that on 4 questions the level is above average, and on 6 - below. In principle, it can be assumed that the engineer is located in the upper part of quadrant IV or ... in the lower part of quadrant III. Where to add points for knowledge - in the "width" or "depth"? Ambiguity. And, it is not very clear how to correctly.
Different algorithms can be selected. For example, I wrote a simple python script and set it up (by changing the factor parameter) in accordance with my intuitive perception.
For example, for this case, this script gives the result
x = 8, y = 5
That is, this is the upper part of quadrant IV.
This means that on the path of the architect, this engineer is on the verge between the 2nd and 3rd steps.
But do not trust the formal approach too much. Nothing replaces live chat. Even a very strong engineer can not pass the test. Just because recently he worked in a different direction and forgot something, or he just doesn’t like tests or slept poorly ... And on the other hand, a good result may be due to the fact that the engineer knew the test questions in advance or your questions the dough is too light.
Therefore, further in the oral interview, you can determine more precisely which stage the engineer is at.
Parameters, in fact, a lot. But we have a wonderful example of how difficult it can be visualized in a rather simple and visual way.
I'm talking about (I think everyone knows) the Gartner quadrant.
This article is about how this visualization approach was applied to assess the level of knowledge and skills of network engineers in a Russian company.
Two-dimensional approach
When conducting the aforementioned survey, it was decided to display the level of knowledge of the engineer in two dimensions. The value deposited on the axis OX characterizes the depth of knowledge and on the axis OY - the breadth of the technical outlook.
Of course, it is important to initially limit the range of topics addressed, in accordance with the requirements of the company. In our case, these were R & S, Security, Service Provider.
Evaluation of both "latitude" and "depth" is on a 10-point scale.
We can distinguish 4 quadrants, which we conditionally call:
- beginner engineer - quadrant I, gray area
- those. Quadrant II Manager , Blue Zone
- generalist or architect - quadrant III, green area
- Narrow Specialist or Expert - quadrant VI, yellow zone
Also in each quadrant you can select subzones. For example, we can assume that if the engineer’s knowledge is in the “latitude” zone 7–10, the “depth” 7-10, then this is an engineer who possesses fundamental, expert knowledge of many technologies, with extensive experience and skills, and is able to perform virtually any tasks.
On this graph you can also take into account such characteristics as
- level of motivation
- communication skills
- ability to learn
- analytic skills
- ...
So, good communicative qualities, as it seems to us, are a plus to “breadth”, since help the engineer to receive information on related areas. The remaining 3 qualities are a plus for both “depth” and “latitude”.
Perhaps this somewhat distorts the picture at the moment, but allows you to take into account the dynamics, which in our opinion is a significant factor.
One dimensional approach
The one-dimensional visualization approach has some advantage over the two-dimensional one, since allows you to enter an unambiguous "better-worse" for engineers and thus rank them according to their level of knowledge and skills. But such an approach is possible only if the ultimate goal in the development of the engineer is defined.
We can identify at least two goals:
- architect
It assumes IT erudition and a fairly deep level of knowledge in many areas.
- expert in a narrow field of knowledge
The engineer focuses on one direction. He knows the details, nuances, theory, perhaps down to the level of programming, mathematics, knowledge of components ...
There is also a third goal (IT manager), which we will not discuss in this article, because it is not an engineering goal.
As it is easy to see, these goals correspond to the third (architect), fourth (narrow specialist) and second (IT manager) quadrants of a two-dimensional visualization scheme.
In each of these two (engineering objectives) cases, we can link a one-dimensional scheme to a two-dimensional approach. Thus, on a two-dimensional map map we can get
- The path of the architect
- Expert path
The path of the architect
We can distinguish 4 stages in development, which we conditionally call
- Primary knowledge
Does not require explanation. - Narrow Profile (or “Operational Knowledge”) The
engineer mastered the theory and mastered the skill set at a level sufficient to complete his operational tasks. He is an expert in a certain field of telecom, but his knowledge is limited either by a certain technology or by what is required of him in the company. Such an engineer copes well with current, operational tasks, but such an engineer does not have the qualifications necessary to make architectural decisions. - Architect
Engineer has mastered a few areas in telecom at a good level. He has a good outlook. He understands various vendors, equipment types, knows “best practices”, various recommended designs, ...
This knowledge allows the engineer to choose the right approach for each specific task, design, vendors, equipment, ...
But such an engineer does not yet have sufficient skills to determine the development trend branch or individual areas of the telecom. - Star
Engineer has full knowledge. This is the result of both deepening and expanding the knowledge of the architect level.
Such engineers determine the development of the industry.
Expert path
Its stages are similar to those of the “Path of the Architect”, but not only in the field of the entire industry (or a large field of knowledge related to telecom), but in a single, rather narrow direction. The fact that this path goes beyond the boundaries of the scale suggests that in order to reach the “star level” and thus be able to determine the direction of development of this narrow area, not only an excellent level of engineering knowledge, but also specific knowledge is usually required (for example, knowledge of , radio electronics, programming, mathematics ...)
Some additional comments
- The four stages in the path of the architect and expert fully correspond to the four stages of development of the engineer
- As experience has shown, there is ambiguity in such an assessment, and the subjective assessment of interviewers is a big factor. In principle, when we speak of the Gartner quadrant, subjectivity is always present. But in this case, we can formalize the task and eliminate subjectivity.
An example of a formalized (objective) approach
You have a questionnaire consisting of 100 questions. The questionnaire corresponds to 10 topics - 10 questions for each topic. All topics are equivalent in terms of importance. It is assumed that the test is compiled “ideally” and if the examiner answered all 10 questions on the topic, he knows it 100 percent (and this, of course, never happens, but in this article we are not talking about how to make a test, about the assessment method, when you were able to somehow determine the distribution of knowledge and skills by topic).
After the test, you will have a distribution of points (the number of answered questions in each topic) of 10 topics. For example, it could be something like:
Q1: 9
Q2: 7
Q3: 4
Q4: 6
Q5: 0
Q6: 3
Q7: 6
Q8: 1
Q9: 1
Q10: 3
Good. And what place in our two-dimensional visualization will the evaluation of this engineer take?
We see that on 4 questions the level is above average, and on 6 - below. In principle, it can be assumed that the engineer is located in the upper part of quadrant IV or ... in the lower part of quadrant III. Where to add points for knowledge - in the "width" or "depth"? Ambiguity. And, it is not very clear how to correctly.
Different algorithms can be selected. For example, I wrote a simple python script and set it up (by changing the factor parameter) in accordance with my intuitive perception.
For example, for this case, this script gives the result
x = 8, y = 5
That is, this is the upper part of quadrant IV.
This means that on the path of the architect, this engineer is on the verge between the 2nd and 3rd steps.
But do not trust the formal approach too much. Nothing replaces live chat. Even a very strong engineer can not pass the test. Just because recently he worked in a different direction and forgot something, or he just doesn’t like tests or slept poorly ... And on the other hand, a good result may be due to the fact that the engineer knew the test questions in advance or your questions the dough is too light.
Therefore, further in the oral interview, you can determine more precisely which stage the engineer is at.