June 6, 2019 at 15:44What do interviews and test items test
In a previous article habr.com/en/post/450810, I examined 7 ways to quickly test the competencies of IT specialists, which can be applied before a large, voluminous and time-consuming technical interview. There I examined the essence of these methods and my practice of their application, as well as the reasons why I like them or not.
In this article I want to talk about the modern concept of decision-making by a person, how it relates to the testing of working skills and what actually test methods of testing competencies, such as interviews and test tasks.
For many centuries, scientists have been preoccupied with the question - how and why does a person make certain decisions? In every era, this question was answered in different ways - for thousands of years faith in the fate and will of the gods prevailed, then for a long time it was popularly believed that man is a rational creature, which basically acts reasonably and prudently. The scientific revolution has led to the fact that in the second half of the 20th century, a lot of research was conducted on the behavioral reactions of "Homo sapiens". And at the moment, the most modern and recognized concept in the scientific community is a hybrid model of human behavior, which psychologist Daniel Kahneman wrote very well in his scientific articles and popular science books. Daniel won the Nobel Prize in economics because his work disproved many economic theories, based on models of rational decision making by a person. Daniel Kahneman convincingly showed that human behavior in most situations is due to automatic behavioral reactions that are formed on the basis of life experience.
According to the concept of Daniel Kahneman, human behavior is controlled by two interacting decision-making systems. System 1 - fast and automatic, ensures the safety of the body and does not require significant efforts to form a solution. The accuracy of the decisions of this system depends on experience and training, and the speed depends on the characteristics of the individual's nervous system. System 2 - slow, requires effort and concentration. It provides us with complex reasoning, logical inference, sound forecasting. The decision-making speed of this system is tens and hundreds of times lower than the speed of System 1. It is during the operation of System 2 that the full potential of human intelligence is manifested. However, during the operation of this system, resources are intensely consumed - both physical (energy) and attention, which is a derivative of many resources. Therefore, most decisions are made by System 1.
I think that each of you has noticed that you cannot think hard and solve complex problems for more than a period of time in a row. Each person has a different interval. Someone can think hard only for half an hour a day, and someone can solve complex problems for 3 hours in a row. This ability can be developed, however, it is given by very hard work and efforts on oneself and all the same the attention resource will be limited.
Both systems work together. The information coming from the sensory organs is first processed by the fast System 1, which recognizes dangerous situations and instantly responds in case of threats. System 1 also recognizes unfamiliar situations and either decides to ignore them or activates System 2.
How it works? Have you ever watched professional chess players play - at what speed do they make moves at the beginning of the game? For a person who rarely plays chess, it seems that it is impossible to make such difficult decisions so quickly. However, at the same time, you can fully correct the mistakes of trainees when making a code review. Your System 1 can recognize the typical mistakes of novice programmers and automatically correct them, just like a professional chess player reads the situation on the board and knows how to move, practically without straining conscious System 2.
Numerous experiments have shown that in situations familiar to us, decisions are almost always made by automatic System 1 and this is quite rational in terms of body survival and energy expenditure. In this regard, we act very rationally and optimally, but not in the sense of deliberation and optimality of the decisions themselves, but in the sense of a balance between the result and the expenditure of resources of our body. When you drive a car in the city on the way to work, your cornering trajectory and the number of accelerations and braking may not be optimal, but from the point of view of the task of delivering you from your home to your work, everything is very good. If you are a race car driver and drive a race car along a race track, your decisions about trajectory, acceleration and braking will be much more prudent.
In unfamiliar situations that are of interest to us or which we have not been able to avoid, we are forced to act consciously, connecting attention and System 2. After several repetitions of very similar situations, the result of System 2 is stored in the form of signs and reactions, and then you don’t have to waste energy and time for logical conclusions - System 1 will already be trained on this task and next time will provide a solution automatically. Some automatic reactions are lost over time if they are not periodically in demand. Skills that we do not train are lost.
Numerous experiments have shown that at first in a new place of work, an ordinary, mentally healthy person adapts and tries to accept the rules, conditions and work processes of a new place of work. However, after some time, each of us relaxes and begins to work as he can. Diligence and diligence give way to the automatic reactions and patterns enshrined in System 1. In addition, even during the trial period in stressful situations, when a quick solution is required, we react using automatic System 1 and not always as we were taught in this new place of work.
In general, we can say that our core value as an employee is largely determined by our experience - that is, the training of our System 1 in solving certain problems that an employer needs. Therefore, employers very often want an employee not with outstanding intelligence, but with experience in a certain field. Experience is valued more than intelligence. This is due to elementary calculations. If there is enough time, then any employee with sufficient intelligence will be able to understand the topic and solve the tasks. However, he will have to spend time learning and gaining experience, and only then will he be able to qualitatively solve the assigned tasks. His System 2 will have to solve many training problems before his System 1 can quickly and efficiently solve real problems. It takes time, which the employer is often not prepared to pay at a high professional rate. Another employee who has already solved such problems will do the job much faster, because most of the decisions will be issued to him by his System 1, trained to solve problems in the right field. An experienced employee will issue quality solutions not only faster, but also strain less. This means that unused attention resources can be directed to solving new complex problems and gaining new experience. but also strain less at the same time. This means that unused attention resources can be directed to solving new complex problems and gaining new experience. but also strain less at the same time. This means that unused attention resources can be directed to solving new complex problems and gaining new experience.
What to choose - experience or intelligence - the employer decides in each case individually. Where a quick reaction to a typical task and a quick solution are required, experience is often chosen. If you have to solve a lot of various problems, but the decision time is still highly valued - then choose the experienced and smart. If time is not very critical, then you can give preference to an intellectual without experience. As you know, in the real world there are few places of work where time is not critical.
Experience - that is, the training of System 1 - is often an important, perhaps even decisive criterion for an employer choosing a new employee. How do we evaluate the candidate’s experience most effectively and accurately? Let's look at popular ways of assessing competencies in terms of what they evaluate.
This format implies a conversation between the candidate and the evaluator. Mostly, the evaluator asks the questions, but the candidate has the opportunity to read non-verbal signs, ask clarifying questions and, as they say, change his answer “on the fly”. This is the “oral exam” familiar to all of us. As a rule, the interview is held according to the standard plan and many questions are also typical, which means that you can prepare for them. That is, train your System 1 to successfully pass interviews.
The success of the candidate assessment depends on the communication skills of both participants. A candidate experienced in interviewing may well make a good impression. However, this result is obtained not due to work experience, but due to the experience of communication and interviews. A trained candidate who answers well to standard questions has an impact on the examiner and the examiner becomes more loyal to the candidate.
This method mainly tests the Candidate System 1, though often it is not the experience that will be needed in the work. It is not bad for evaluating specialists who have to communicate a lot on official duties and quickly adapt, but in my opinion, this method is not suitable for evaluating technical skills. The accuracy of the assessment can be improved by means of non-standard questions and interview scenarios, as well as by participating in the interview of several evaluating specialists, which leads to a rise in the cost of this event.
The candidate receives a task that he solves on his own and then demonstrates the result of the solution. In fact, this is a “written exam” familiar to us. The candidate has enough time, the opportunity to ask clarifying questions, as well as search for information on the network and even take the help of friends. If the task is difficult and enough time is given, then this method tests System 2 rather than System 1, that is, intelligence, not experience. If you reduce the time to complete the task, then the candidates are highly likely to refuse to complete a difficult test task. If at the same time we simplify the task, give several tasks and reduce the time, then this method becomes quite a working tool, familiar to us from the school bench. It does a good job of testing System 1. However, its drawback is that
The candidate receives a simple task, which he solves under the supervision of an evaluating specialist. This method is often used in the interview process - when evaluators first talk and then offer to solve problems. For introvert candidates who have not been interviewed for a long time, this method is often psychologically uncomfortable, and they show not very good results. In my opinion, this method should be offered to candidates as an alternative to the test task. That is, either 3-4 hours of independent work, or 1-1.5 hours of interviewing and solving problems online. If the candidate is ready, this method allows you to test the basic skills of System 1 on typical tasks that are components of more complex work tasks. That is, it is worth choosing elements of real work tasks as test tasks.
As you probably know, the final exams in Russian schools now take the form of tests (State Examination and Examination). At one time this caused heated debate. Citizens generally negatively assessed this decision by the Ministry of Education. Personally, I think that if you do not take into account the new opportunities for corruption, then replacing written exams with tests is a good solution. Verification of test results does not require much time and attention, it is easily automated. At the same time, the subjectivity of knowledge assessment is minimized. Tests allow for 1-2 hours to qualitatively test the knowledge and experience gained during several years of study or work. The novice driver learns the rules of the road for several months, and on the exam he must answer 20 questions in 20 minutes. The practice of decades of applying this type of exam shows
In the modern world, most human decisions come down to choosing one of the existing options that is most suitable for the situation. You are unlikely to need a specialist who will reinvent the wheel. But then a specialist who knows the advantages and disadvantages of different types of bicycles and similar modes of transport will come in handy, will help you quickly choose the right model and configure it to solve your problems. Logistic problems are usually solved quickly and there is no time to invent an innovative bicycle. Sometimes (very rarely) there are situations when you still need a new bike, which does not yet exist and needs to be created. However, in this case as well, a person who is well versed in the construction of bicycles will be more useful than a universal inventor.
One more example. If a programmer can implement several sorting algorithms, then he is certainly well done, but in real life, it will be more useful for him to know the basic methods of the base library of language classes - there have probably already been implemented several sorting options, just call the desired function.
It is important that when choosing a method of competency testing, you turn on your System 2 and choose the appropriate method meaningfully, and not according to tradition - "because we always did this." When choosing a method of testing competencies, I advise first of all to decide what will be more important for you as an employer in the daily activities of your employee. Will it be the ability to quickly solve a certain range of typical problems, or will it be necessary to solve complex, original, atypical problems.
In most cases, as a first test for candidates, tests that are limited in time will work fine for you. I recommend small tests that take no more than 15-20 minutes to complete. During this time, you can ask 30-40 questions and test the knowledge of candidates in sufficient detail. Then you can conduct an interview during which to analyze the mistakes made by the candidates. The test can also serve as a reference plan for an interview, during which you can clarify with the candidate the reasons why he answered the test questions in this way and how he would have answered if the question sounded differently.
If it is important for you how the future employee works independently on sufficiently large and isolated tasks, then it will be appropriate to start with an interview and then offer to complete the test task. It is worth remembering that only 20-25% of candidates agree to perform test tasks before the interview, in which case you greatly reduce the selection funnel.
In my next article, I will examine in more detail the features of creating tests to test the competencies of candidates.
In this article I want to talk about the modern concept of decision-making by a person, how it relates to the testing of working skills and what actually test methods of testing competencies, such as interviews and test tasks.
Bit of theory
For many centuries, scientists have been preoccupied with the question - how and why does a person make certain decisions? In every era, this question was answered in different ways - for thousands of years faith in the fate and will of the gods prevailed, then for a long time it was popularly believed that man is a rational creature, which basically acts reasonably and prudently. The scientific revolution has led to the fact that in the second half of the 20th century, a lot of research was conducted on the behavioral reactions of "Homo sapiens". And at the moment, the most modern and recognized concept in the scientific community is a hybrid model of human behavior, which psychologist Daniel Kahneman wrote very well in his scientific articles and popular science books. Daniel won the Nobel Prize in economics because his work disproved many economic theories, based on models of rational decision making by a person. Daniel Kahneman convincingly showed that human behavior in most situations is due to automatic behavioral reactions that are formed on the basis of life experience.
According to the concept of Daniel Kahneman, human behavior is controlled by two interacting decision-making systems. System 1 - fast and automatic, ensures the safety of the body and does not require significant efforts to form a solution. The accuracy of the decisions of this system depends on experience and training, and the speed depends on the characteristics of the individual's nervous system. System 2 - slow, requires effort and concentration. It provides us with complex reasoning, logical inference, sound forecasting. The decision-making speed of this system is tens and hundreds of times lower than the speed of System 1. It is during the operation of System 2 that the full potential of human intelligence is manifested. However, during the operation of this system, resources are intensely consumed - both physical (energy) and attention, which is a derivative of many resources. Therefore, most decisions are made by System 1.
I think that each of you has noticed that you cannot think hard and solve complex problems for more than a period of time in a row. Each person has a different interval. Someone can think hard only for half an hour a day, and someone can solve complex problems for 3 hours in a row. This ability can be developed, however, it is given by very hard work and efforts on oneself and all the same the attention resource will be limited.
Both systems work together. The information coming from the sensory organs is first processed by the fast System 1, which recognizes dangerous situations and instantly responds in case of threats. System 1 also recognizes unfamiliar situations and either decides to ignore them or activates System 2.
Please multiply 65 by 15 and note to yourself how long these calculations took you.
How it works? Have you ever watched professional chess players play - at what speed do they make moves at the beginning of the game? For a person who rarely plays chess, it seems that it is impossible to make such difficult decisions so quickly. However, at the same time, you can fully correct the mistakes of trainees when making a code review. Your System 1 can recognize the typical mistakes of novice programmers and automatically correct them, just like a professional chess player reads the situation on the board and knows how to move, practically without straining conscious System 2.
Please multiply 65 by 15 again and note to yourself how long these calculations took you.
Numerous experiments have shown that in situations familiar to us, decisions are almost always made by automatic System 1 and this is quite rational in terms of body survival and energy expenditure. In this regard, we act very rationally and optimally, but not in the sense of deliberation and optimality of the decisions themselves, but in the sense of a balance between the result and the expenditure of resources of our body. When you drive a car in the city on the way to work, your cornering trajectory and the number of accelerations and braking may not be optimal, but from the point of view of the task of delivering you from your home to your work, everything is very good. If you are a race car driver and drive a race car along a race track, your decisions about trajectory, acceleration and braking will be much more prudent.
In unfamiliar situations that are of interest to us or which we have not been able to avoid, we are forced to act consciously, connecting attention and System 2. After several repetitions of very similar situations, the result of System 2 is stored in the form of signs and reactions, and then you don’t have to waste energy and time for logical conclusions - System 1 will already be trained on this task and next time will provide a solution automatically. Some automatic reactions are lost over time if they are not periodically in demand. Skills that we do not train are lost.
Please multiply 65 by 15 again. Have you noticed any progress compared to the previous attempt to solve this problem?
How does all this relate to work and competency testing?
Numerous experiments have shown that at first in a new place of work, an ordinary, mentally healthy person adapts and tries to accept the rules, conditions and work processes of a new place of work. However, after some time, each of us relaxes and begins to work as he can. Diligence and diligence give way to the automatic reactions and patterns enshrined in System 1. In addition, even during the trial period in stressful situations, when a quick solution is required, we react using automatic System 1 and not always as we were taught in this new place of work.
In general, we can say that our core value as an employee is largely determined by our experience - that is, the training of our System 1 in solving certain problems that an employer needs. Therefore, employers very often want an employee not with outstanding intelligence, but with experience in a certain field. Experience is valued more than intelligence. This is due to elementary calculations. If there is enough time, then any employee with sufficient intelligence will be able to understand the topic and solve the tasks. However, he will have to spend time learning and gaining experience, and only then will he be able to qualitatively solve the assigned tasks. His System 2 will have to solve many training problems before his System 1 can quickly and efficiently solve real problems. It takes time, which the employer is often not prepared to pay at a high professional rate. Another employee who has already solved such problems will do the job much faster, because most of the decisions will be issued to him by his System 1, trained to solve problems in the right field. An experienced employee will issue quality solutions not only faster, but also strain less. This means that unused attention resources can be directed to solving new complex problems and gaining new experience. but also strain less at the same time. This means that unused attention resources can be directed to solving new complex problems and gaining new experience. but also strain less at the same time. This means that unused attention resources can be directed to solving new complex problems and gaining new experience.
What to choose - experience or intelligence - the employer decides in each case individually. Where a quick reaction to a typical task and a quick solution are required, experience is often chosen. If you have to solve a lot of various problems, but the decision time is still highly valued - then choose the experienced and smart. If time is not very critical, then you can give preference to an intellectual without experience. As you know, in the real world there are few places of work where time is not critical.
Please multiply 65 by 15 again and note to yourself how long these calculations took you. Did you notice how you got the result?
Ways to test competencies in terms of testing "System 1" and "System 2"
Experience - that is, the training of System 1 - is often an important, perhaps even decisive criterion for an employer choosing a new employee. How do we evaluate the candidate’s experience most effectively and accurately? Let's look at popular ways of assessing competencies in terms of what they evaluate.
Job interviews
This format implies a conversation between the candidate and the evaluator. Mostly, the evaluator asks the questions, but the candidate has the opportunity to read non-verbal signs, ask clarifying questions and, as they say, change his answer “on the fly”. This is the “oral exam” familiar to all of us. As a rule, the interview is held according to the standard plan and many questions are also typical, which means that you can prepare for them. That is, train your System 1 to successfully pass interviews.
The success of the candidate assessment depends on the communication skills of both participants. A candidate experienced in interviewing may well make a good impression. However, this result is obtained not due to work experience, but due to the experience of communication and interviews. A trained candidate who answers well to standard questions has an impact on the examiner and the examiner becomes more loyal to the candidate.
This method mainly tests the Candidate System 1, though often it is not the experience that will be needed in the work. It is not bad for evaluating specialists who have to communicate a lot on official duties and quickly adapt, but in my opinion, this method is not suitable for evaluating technical skills. The accuracy of the assessment can be improved by means of non-standard questions and interview scenarios, as well as by participating in the interview of several evaluating specialists, which leads to a rise in the cost of this event.
Test tasks
The candidate receives a task that he solves on his own and then demonstrates the result of the solution. In fact, this is a “written exam” familiar to us. The candidate has enough time, the opportunity to ask clarifying questions, as well as search for information on the network and even take the help of friends. If the task is difficult and enough time is given, then this method tests System 2 rather than System 1, that is, intelligence, not experience. If you reduce the time to complete the task, then the candidates are highly likely to refuse to complete a difficult test task. If at the same time we simplify the task, give several tasks and reduce the time, then this method becomes quite a working tool, familiar to us from the school bench. It does a good job of testing System 1. However, its drawback is that
Live doing
The candidate receives a simple task, which he solves under the supervision of an evaluating specialist. This method is often used in the interview process - when evaluators first talk and then offer to solve problems. For introvert candidates who have not been interviewed for a long time, this method is often psychologically uncomfortable, and they show not very good results. In my opinion, this method should be offered to candidates as an alternative to the test task. That is, either 3-4 hours of independent work, or 1-1.5 hours of interviewing and solving problems online. If the candidate is ready, this method allows you to test the basic skills of System 1 on typical tasks that are components of more complex work tasks. That is, it is worth choosing elements of real work tasks as test tasks.
Answer Tests
As you probably know, the final exams in Russian schools now take the form of tests (State Examination and Examination). At one time this caused heated debate. Citizens generally negatively assessed this decision by the Ministry of Education. Personally, I think that if you do not take into account the new opportunities for corruption, then replacing written exams with tests is a good solution. Verification of test results does not require much time and attention, it is easily automated. At the same time, the subjectivity of knowledge assessment is minimized. Tests allow for 1-2 hours to qualitatively test the knowledge and experience gained during several years of study or work. The novice driver learns the rules of the road for several months, and on the exam he must answer 20 questions in 20 minutes. The practice of decades of applying this type of exam shows
In the modern world, most human decisions come down to choosing one of the existing options that is most suitable for the situation. You are unlikely to need a specialist who will reinvent the wheel. But then a specialist who knows the advantages and disadvantages of different types of bicycles and similar modes of transport will come in handy, will help you quickly choose the right model and configure it to solve your problems. Logistic problems are usually solved quickly and there is no time to invent an innovative bicycle. Sometimes (very rarely) there are situations when you still need a new bike, which does not yet exist and needs to be created. However, in this case as well, a person who is well versed in the construction of bicycles will be more useful than a universal inventor.
One more example. If a programmer can implement several sorting algorithms, then he is certainly well done, but in real life, it will be more useful for him to know the basic methods of the base library of language classes - there have probably already been implemented several sorting options, just call the desired function.
Conclusion
It is important that when choosing a method of competency testing, you turn on your System 2 and choose the appropriate method meaningfully, and not according to tradition - "because we always did this." When choosing a method of testing competencies, I advise first of all to decide what will be more important for you as an employer in the daily activities of your employee. Will it be the ability to quickly solve a certain range of typical problems, or will it be necessary to solve complex, original, atypical problems.
In most cases, as a first test for candidates, tests that are limited in time will work fine for you. I recommend small tests that take no more than 15-20 minutes to complete. During this time, you can ask 30-40 questions and test the knowledge of candidates in sufficient detail. Then you can conduct an interview during which to analyze the mistakes made by the candidates. The test can also serve as a reference plan for an interview, during which you can clarify with the candidate the reasons why he answered the test questions in this way and how he would have answered if the question sounded differently.
If it is important for you how the future employee works independently on sufficiently large and isolated tasks, then it will be appropriate to start with an interview and then offer to complete the test task. It is worth remembering that only 20-25% of candidates agree to perform test tasks before the interview, in which case you greatly reduce the selection funnel.
In my next article, I will examine in more detail the features of creating tests to test the competencies of candidates.