Testing competencies with tests - why and how

In my article, I examined 7 ways to quickly test the competencies of IT-specialists, which can be applied before a large, voluminous and time-consuming technical interview. Then I expressed my sympathy for time-limited tests. In this article I will reveal the topic of tests in more detail.

Time-limited tests are a universal tool that is well suited for testing the knowledge and practical skills of any specialists in any profession.

So, the task is that we have a stream of feedback from candidates for the vacancy, we need to quickly and simply get additional information about the skills of the candidates and their compliance with the requirements of our vacancy. We want this verification of candidates' competencies not to take much of our time, to be highly reliable and convenient for candidates to agree to pass our examination.

A good solution for this problem are short tests that are limited in time. It is not the time when the test begins, but the time during which the candidate must answer questions that is limited. A typical example of such a test is an examination for knowledge of traffic rules, which is the first stage of an exam for obtaining a driver’s license. In 20 minutes you need to answer 20 questions.

Bit of theory

In the previous articleI talked about the hybrid decision-making model of "reasonable man" proposed by Daniel Kahneman and his colleagues. According to this concept, human behavior is controlled by two interacting decision-making systems. System 1 - fast and automatic, ensures the safety of the body and does not require significant efforts to form a solution. This system is trained based on the experience that a person receives throughout life. The accuracy of the decisions of this system depends on personal experience and training, and the speed depends on the characteristics of the individual's nervous system. System 2 - slow, requires effort and concentration. It provides us with complex reasoning and logical inference, her work reveals the potential of human intelligence. However, during the operation of this system, resources are intensively consumed - energy and attention. Therefore, most decisions are made by System 1 - so human behavior becomes much more effective. System 1 learns for a long time due to the efforts made by System 2, but then it gives quick automatic reactions. System 2 is a universal task solver, but it works slowly and quickly gets tired. It is possible to “upgrade” System 2, but the limits of possible improvements are very modest and this takes a long time and requires heavy efforts. The “pumping” of System 1 is in great demand in human society. When we are looking for a person experienced in anything, this means that his System 1 is trained to quickly solve the problems we need. System 2 is a universal task solver, but it works slowly and quickly gets tired. It is possible to “upgrade” System 2, but the limits of possible improvements are very modest and this takes a long time and requires heavy efforts. The “pumping” of System 1 is in great demand in human society. When we are looking for a person experienced in anything, this means that his System 1 is trained to quickly solve the problems we need. System 2 is a universal task solver, but it works slowly and quickly gets tired. It is possible to “upgrade” System 2, but the limits of possible improvements are very modest and this takes a long time and requires heavy efforts. The “pumping” of System 1 is in great demand in human society. When we are looking for a person experienced in anything, this means that his System 1 is trained to quickly solve the problems we need.

The best way to assess the capabilities of System 1 of a particular person in a certain area of ​​knowledge, I consider tests that are limited in time. Once compiled a test allows you to quickly evaluate and compare among themselves a large number of candidates. This is a tool for digitizing the control of knowledge and skills.

How to make a good test?

The task of a well-written test is to determine the degree of training of the System 1 candidate for the knowledge and skills you need. To make such a test, you first need to determine the topics and the necessary skills, and then draw up questions and answer options.

So, here are my criteria for preparing a test that accurately and qualitatively evaluates the candidate’s knowledge and skills:

1. Questions and answers should be simple. Either you know the correct answer, or you do not know. Do not lay in the test the need for complex reasoning and calculations.
2. Test execution should be limited in time. You can even limit the thinking time of each answer. If the candidate cannot decide on the answer within, say, 30 seconds, then it’s unlikely that much thought will help him. Googling the correct answer in 30 seconds should also be difficult.
3. Questions should be about practices that are really needed in the work - not abstract and theoretical, but purely practical.
4. It is advisable that for each small topic there are several questions. These questions may vary for different candidates (this is similar to different types of tests in school) or all may be present in a longer version of the test.
5. The number of questions and the time for passing the test should be tightly connected. Measure how long it takes to read the questions and answer options. Add to this time 10-20 seconds for each question - this is the time to think and choose an answer.
6. It is advisable to test the test on your employees and track their time to determine the sufficient time for the candidates to pass the test.
7. The volume of the test depends on the purpose of its application. For the initial assessment of competencies, in my opinion, 10-30 questions with a time limit of 5-15 minutes are enough. For a more detailed diagnosis of skills suitable tests for 30-45 minutes, containing 50-100 questions.

For example, here is a test that I developed and used recently when selecting candidates for the position of an IT recruiter. It took 6 minutes to complete the test, the time was controlled manually and honestly. All of the tested candidates fit into this time. It took me 30 minutes to complete the test. docs.google.com/forms/d/e/1FAIpQLSfL2pUZob2Xq-1taJPwaB2rUifbdKWK4Mk0VREKp5yUZhTQXA/viewform

You can pass the test and at the end you can see where you made a mistake. When the candidates passed this test, they didn’t show errors, we then sorted out the errors at interviews with candidates who made no more than 3 errors.

Instruments

Now I am creating tests and surveys using Google Forms - this is a simple, convenient, universal and free tool. However, I am missing some functionality in order to call Google Forms a good tool for creating tests. My main complaints about Google Forms:

1. There is no accounting and control of time spent both on the entire test and on each question. This gives more information about the behavior of the candidate during the test.
2. Since Google Forms is not sharpened by default tests, many options important for tests (for example, “answer to a question is required” and “mix answers”) have to be clicked on for each question - which takes time and attention. To ask each question on a separate screen, you need to create separate sections for each question, and this also leads to a large number of additional clicks.
3. If you need to make a new test as a combination of fragments from several existing tests (for example, to collect a test for a fullstack developer from part of the questions for frontend and backend in a certain language), then you have to duplicate the questions with your hands. There is no way to select and copy several sections or questions into another form.

Colleagues, if you know the best solutions for creating tests, please write about them in the comments.