March 6, 2018 at 14:21SOC are people. How to assemble a team in conditions of staff shortage
It is said that in the 70s, Gennady Zaitsev, the first president of the legendary Leningrad rock club, formulated the principle of selecting musicians: “Finger fluency is a matter of time. Would a good man . "
Perhaps if Gennady Borisovich and I decided to complain to each other about a difficulty in selecting personnel behind a glass of something fermenting, we would find many common points. When we assembled the first Solar JSOC team, the market for monitoring and counteracting cyber attacks did not actually exist yet, therefore, there were no trained specialists for these tasks, or even clear search criteria. We had to assemble a team of unique ones, but before that - to try new things many times, make mistakes and try again. We’ll talk about the team today: how it was at the very beginning and what we came to as a result of.
I don’t know for certain how the idea of creating JSOC was born - maybe just a coincidence, or maybe the guys came together and firmly decided: “We are creating SOC for the commercial market of Russia!” Turn on rewinding and skip many exciting episodes about choosing a platform, working out the ideology of building content, and also about incredible adventures in the wild world of customer infrastructures.
Let's start from the moment when JSOC from the synthetics of plans and estimations was ripe for entering the market and running in the worked out solutions in battle. And for this there is not enough technology, ideas, ambitions and SIEM gurus and analytics. Here one can not do without the first line of analysts - a team that provides the front of the main SOC processes.
Our first line of monitoring is different from the first line in the classical sense. This is not a call center or girls with an application routing script. The role of the classic first line in our paradigm is assigned to SIEM with a pie of profiling, aggregating and enriching rules. And the task of the first line of JSOC is to conduct full-fledged investigations of IS incidents, including FP filtering, the formation of analytical information and recommendations on counteraction. A free speech is also welcomed, especially in situations where atypical events are seen in the vicinity of the incident.
"Any adventure must start somewhere ...
When the first version of the team was formed, we did not quite understand the exact profile of the analysts, since it is difficult to predict the development of events in a completely new market. Therefore, to compensate for the possible turns and vicissitudes of technological trends, we decided to look for guys with a wide technical background.
It turned out to assemble a team of such specialists rather quickly, and this greatly helped us at the initial stage. We felt ready for the winding and difficult way to bring SOC to the market.
However, after a while, we realized that we had miscalculated in some ways: most of the hired specialists were clearly overqualified for the position of first-line engineer, and the v.1.0 team turned out to be rather unstable.
Despite the high development dynamics of Solar JSOC and the constant emergence of new tasks, the children's ambitions were even higher, and the final list of responsibilities was significantly lower than their experience and expectations. Therefore, colleagues quickly grew out of the first line - someone within the JSOC, and someone, alas, outside.
However, the “virus” of information security obtained in JSOC turned out to be very persistent - despite the fact that the v.1.0 team was almost 100% come from pure IT, almost all of them continue to work on the information security field.
Nevertheless, this first attempt to assemble a team was very useful for us.
We “on the go” decided what we finally do and what we expect from employees. We got used to the fact that in the new market, multiple growth and the accompanying permanent training of personnel are the norm and routine. We found that the market cannot offer ready-made specialists to work on the first line of SOC. We realized that in these conditions it’s easy to go through a frequent rake on a resume - utopia.
"To teach a scientist is only to spoil."
So, we found ourselves in a situation in which there are practically no ready-made specialists on the market, and the number of customers begins to grow, which requires scaling up the team. At some point, we simply came into conflict with the capabilities of the region’s personnel market (the first Solar JSOC line is in Nizhny Novgorod).
The salvation for us was working with students. At this point, we had little experience of such work in the Moscow Solar Security team, but it was not aimed at working with the university as a platform organizing the flow of young specialists. We felt that in the case of Solar JSOC, a more systematic approach would be required.
And here our colleagues, graduates of the Nizhny Novgorod University, helped us a lot, and brought us to the student career center of the Nizhny Novgorod State University, a peppy student organization interested in professional job placement for students. We began to regularly participate in Employer Days, to tell students and recent graduates of technical universities about the activities of Solar JSOC.
Then we launched an internship program. Initially, it was aimed at students in the specialty of information security, but now it has been expanded to students of other IT specialties.
Looking ahead, I will say that this work yielded excellent results. For the third year now, young guys come to us by gravity, ready to undergo testing, internship, study and become first-line engineers. And many graduates of our internships have truly found their first job at Solar JSOC.
- This is too difficult for my little brain ...
We understood what the ideal team of an ideal SOC should be in the end :) And we are ready to reveal this secret: a team should be primarily a Team, no matter how corny it sounds. And the phrase of Gennady Zaitsev was given at the beginning of the article not by chance. We approach the selection of personnel, guided by the same principle: “Fluency of fingers is a matter of gain. Would a good man. " After repeated trial and error, we came to the conclusion that when searching for engineers of the first line, the main emphasis should not be on the technical background and previous work experience. (This, however, does not mean that the previous experience is completely irrelevant. Of course, good knowledge of networks and linux is welcomed for a job seeker in the SZI technical maintenance group.) We are trying to find guys with “living brains”. Guys able to see a fan of judgment, able to reason,
When there was an influx of potential candidates for internships, and subsequently work, we were faced with the task of assessing the learning ability of each candidate. Therefore, at the interview, we try to bring the applicants to logical reasoning, give logical problems and change their conditions in order to see how a person adapts and selects the best solution. This approach helps us to select people who are pleasant to work with and who integrate very quickly into the team. Well, “fluency of fingers” is a business. About how we train her, a little lower.
As an entrance test for an internship, we offer children to deal with such tasks:
The internship program includes:
During internships, the greatest attention is paid to test incidents, most of which are created based on real events from the practice of Solar JSOC. In addition to template, automatically generated incidents, trainees regularly receive multilayer incidents with sub-marks for parsing. Incidents that require a special approach, because as a result of a patterned investigation, students, for example, conclude that malicious activity was initiated from their own hosts from under their own accounts.
Such challenges significantly increase the involvement in the training and investigation of incidents, learn to pay attention to various "little things", double-check your conclusions, make assumptions and prove or disprove them in the future.
Involvement in the investigation process is the most important at this stage. Having interested the guys in security and letting them “touch” incidents with a real story or incidents of which they themselves are supposedly participants, we motivate them to study a “boring” theory and develop non-standard approaches to solving the problem that are not described in the manuals.
At the end of the internship, we get guys half ready for work on the first line. Thus, we are creating a personnel reserve.
Regardless of whether we take a person to work as a result of an interview or after an internship, at the beginning of the professional career of a first-line engineer, Solar JSOC expects a training program for which it takes from 2 to 4 months. It can be conditionally divided into the following blocks:
Based on the evaluation of the curator and the final control slice conducted by the commission, the engineer is allowed to "self-sail", night / weekend duty.
At first, most engineers work in the paradigm “For an incident of this type, I use such and such tools, they provide the necessary data about it.” Such a methodology allows you to quickly train a person and bring to a level when he is able to bring benefits on the line.
As they gain experience, the guys realize the commonality of IS incidents, begin to better understand what information and in what context will be most useful to the customer for responding to a specific incident. Gradually, they move on to work in the paradigm "I need such and such information, I can get it in many ways, in this situation, such and such is preferable."
Such a restructuring of the approach, coupled with pumping the information security background and immersing yourself in SIEM content, prepares the children for the steps to the next steps, where they will have in-depth investigations of high-critical incidents and information security incidents or diving deep into SIEM systems with all the variety of custom logic.
This ends the formalities and begins a full life on the first line: for administration - full of complex quests to satisfy the desires of customers and compliance with information security requirements; for monitoring - consisting of many fascinating stories reflected in the logs. Sometimes a day on the first line may resemble a puzzle on several boards with the length of the duty shift: the collected pictures flicker in front of the engineer, changing not only their texture, but also the ticket number :)
But, although the theater begins with a hanger, and the operational life of SOC - from the first line, it does not end there. People grow, projects scale, and we have a great many roles and tasks inside. We very much encourage the development drive in first-line engineers, therefore we have worked on several “professional elevators”. A certain percentage of engineers who are ready to move to other lines are even included in the KPI of team lines :) But the story of “boosting expertise” or further staff growth in Solar JSOC will become part of other articles. To be continued ...
Perhaps if Gennady Borisovich and I decided to complain to each other about a difficulty in selecting personnel behind a glass of something fermenting, we would find many common points. When we assembled the first Solar JSOC team, the market for monitoring and counteracting cyber attacks did not actually exist yet, therefore, there were no trained specialists for these tasks, or even clear search criteria. We had to assemble a team of unique ones, but before that - to try new things many times, make mistakes and try again. We’ll talk about the team today: how it was at the very beginning and what we came to as a result of.
I don’t know for certain how the idea of creating JSOC was born - maybe just a coincidence, or maybe the guys came together and firmly decided: “We are creating SOC for the commercial market of Russia!” Turn on rewinding and skip many exciting episodes about choosing a platform, working out the ideology of building content, and also about incredible adventures in the wild world of customer infrastructures.
Let's start from the moment when JSOC from the synthetics of plans and estimations was ripe for entering the market and running in the worked out solutions in battle. And for this there is not enough technology, ideas, ambitions and SIEM gurus and analytics. Here one can not do without the first line of analysts - a team that provides the front of the main SOC processes.
Our first line of monitoring is different from the first line in the classical sense. This is not a call center or girls with an application routing script. The role of the classic first line in our paradigm is assigned to SIEM with a pie of profiling, aggregating and enriching rules. And the task of the first line of JSOC is to conduct full-fledged investigations of IS incidents, including FP filtering, the formation of analytical information and recommendations on counteraction. A free speech is also welcomed, especially in situations where atypical events are seen in the vicinity of the incident.
"Any adventure must start somewhere ...
corny, but even here it is true."
Lewis Carroll
When the first version of the team was formed, we did not quite understand the exact profile of the analysts, since it is difficult to predict the development of events in a completely new market. Therefore, to compensate for the possible turns and vicissitudes of technological trends, we decided to look for guys with a wide technical background.
It turned out to assemble a team of such specialists rather quickly, and this greatly helped us at the initial stage. We felt ready for the winding and difficult way to bring SOC to the market.
However, after a while, we realized that we had miscalculated in some ways: most of the hired specialists were clearly overqualified for the position of first-line engineer, and the v.1.0 team turned out to be rather unstable.
Despite the high development dynamics of Solar JSOC and the constant emergence of new tasks, the children's ambitions were even higher, and the final list of responsibilities was significantly lower than their experience and expectations. Therefore, colleagues quickly grew out of the first line - someone within the JSOC, and someone, alas, outside.
However, the “virus” of information security obtained in JSOC turned out to be very persistent - despite the fact that the v.1.0 team was almost 100% come from pure IT, almost all of them continue to work on the information security field.
Nevertheless, this first attempt to assemble a team was very useful for us.
We “on the go” decided what we finally do and what we expect from employees. We got used to the fact that in the new market, multiple growth and the accompanying permanent training of personnel are the norm and routine. We found that the market cannot offer ready-made specialists to work on the first line of SOC. We realized that in these conditions it’s easy to go through a frequent rake on a resume - utopia.
"To teach a scientist is only to spoil."
Russian proverb
So, we found ourselves in a situation in which there are practically no ready-made specialists on the market, and the number of customers begins to grow, which requires scaling up the team. At some point, we simply came into conflict with the capabilities of the region’s personnel market (the first Solar JSOC line is in Nizhny Novgorod).
The salvation for us was working with students. At this point, we had little experience of such work in the Moscow Solar Security team, but it was not aimed at working with the university as a platform organizing the flow of young specialists. We felt that in the case of Solar JSOC, a more systematic approach would be required.
And here our colleagues, graduates of the Nizhny Novgorod University, helped us a lot, and brought us to the student career center of the Nizhny Novgorod State University, a peppy student organization interested in professional job placement for students. We began to regularly participate in Employer Days, to tell students and recent graduates of technical universities about the activities of Solar JSOC.
Then we launched an internship program. Initially, it was aimed at students in the specialty of information security, but now it has been expanded to students of other IT specialties.
Looking ahead, I will say that this work yielded excellent results. For the third year now, young guys come to us by gravity, ready to undergo testing, internship, study and become first-line engineers. And many graduates of our internships have truly found their first job at Solar JSOC.
- This is too difficult for my little brain ...
- It is not size that matters, but the ability to use.
From Folklore Solar JSOC
We understood what the ideal team of an ideal SOC should be in the end :) And we are ready to reveal this secret: a team should be primarily a Team, no matter how corny it sounds. And the phrase of Gennady Zaitsev was given at the beginning of the article not by chance. We approach the selection of personnel, guided by the same principle: “Fluency of fingers is a matter of gain. Would a good man. " After repeated trial and error, we came to the conclusion that when searching for engineers of the first line, the main emphasis should not be on the technical background and previous work experience. (This, however, does not mean that the previous experience is completely irrelevant. Of course, good knowledge of networks and linux is welcomed for a job seeker in the SZI technical maintenance group.) We are trying to find guys with “living brains”. Guys able to see a fan of judgment, able to reason,
When there was an influx of potential candidates for internships, and subsequently work, we were faced with the task of assessing the learning ability of each candidate. Therefore, at the interview, we try to bring the applicants to logical reasoning, give logical problems and change their conditions in order to see how a person adapts and selects the best solution. This approach helps us to select people who are pleasant to work with and who integrate very quickly into the team. Well, “fluency of fingers” is a business. About how we train her, a little lower.
As an entrance test for an internship, we offer children to deal with such tasks:
- Why does UDP continue to be used when TCP guarantees delivery of data unchanged, sequence and lossless?
- You need to write TK on keylogger. Where do you start? What criteria would you use to aggregate information for convenient work with the results?
- List the fundamental differences of such Remote Administration Tool, such as Microsoft Remote Desktop Connection and TeamViewer, in terms of control over access to critical hosts?
- At what stages of the APT attack can you find the use of these utilities, and how can an attacker use them?
c: \ windows \ system32 \ whoami.exe
c: \ windows \ syswow64 \ netstat.exe
c: \ windows \ syswow64 \ nslookup.exe
The internship program includes:
- Learning the basics of network technology.
- Studying the features of logging IS events by various sources.
- HPE ArcSight Skills
- Analysis of simple test incidents at the training stand.
- And much more.
During internships, the greatest attention is paid to test incidents, most of which are created based on real events from the practice of Solar JSOC. In addition to template, automatically generated incidents, trainees regularly receive multilayer incidents with sub-marks for parsing. Incidents that require a special approach, because as a result of a patterned investigation, students, for example, conclude that malicious activity was initiated from their own hosts from under their own accounts.
Such challenges significantly increase the involvement in the training and investigation of incidents, learn to pay attention to various "little things", double-check your conclusions, make assumptions and prove or disprove them in the future.
Involvement in the investigation process is the most important at this stage. Having interested the guys in security and letting them “touch” incidents with a real story or incidents of which they themselves are supposedly participants, we motivate them to study a “boring” theory and develop non-standard approaches to solving the problem that are not described in the manuals.
At the end of the internship, we get guys half ready for work on the first line. Thus, we are creating a personnel reserve.
Regardless of whether we take a person to work as a result of an interview or after an internship, at the beginning of the professional career of a first-line engineer, Solar JSOC expects a training program for which it takes from 2 to 4 months. It can be conditionally divided into the following blocks:
- The theoretical block includes basic knowledge, starting with information about the company and its organizational structure and ending with an introduction to regulatory documentation, the latest trends in information security, APT, Threat Intelligence.
- The practical block consolidates the acquired knowledge at a new level and teaches specific skills to work with the main tool of the first line - SIEM. In addition, the practice includes such mandatory areas as tracking IS incidents in the ticket system, the subtleties of communication with customers, reproducing and analyzing the most interesting incidents at the stand, and much more.
- Further, newcomers are given gradually increasing access to a productive environment based on whitelist: an engineer is involved in investigating “combat” incidents according to the scenarios in which he successfully passed the exam and to which he received admission.
- Since our company, like the information security industry itself, is developing rapidly, it is impossible to cover all the details with a training program. Therefore, at the final stage, each future engineer of the first line of monitoring is assigned a curator of experienced engineers - so-called "practicing surgeons" - to study the nuances and control the work performed on a productive environment.
Based on the evaluation of the curator and the final control slice conducted by the commission, the engineer is allowed to "self-sail", night / weekend duty.
At first, most engineers work in the paradigm “For an incident of this type, I use such and such tools, they provide the necessary data about it.” Such a methodology allows you to quickly train a person and bring to a level when he is able to bring benefits on the line.
As they gain experience, the guys realize the commonality of IS incidents, begin to better understand what information and in what context will be most useful to the customer for responding to a specific incident. Gradually, they move on to work in the paradigm "I need such and such information, I can get it in many ways, in this situation, such and such is preferable."
Such a restructuring of the approach, coupled with pumping the information security background and immersing yourself in SIEM content, prepares the children for the steps to the next steps, where they will have in-depth investigations of high-critical incidents and information security incidents or diving deep into SIEM systems with all the variety of custom logic.
This ends the formalities and begins a full life on the first line: for administration - full of complex quests to satisfy the desires of customers and compliance with information security requirements; for monitoring - consisting of many fascinating stories reflected in the logs. Sometimes a day on the first line may resemble a puzzle on several boards with the length of the duty shift: the collected pictures flicker in front of the engineer, changing not only their texture, but also the ticket number :)
But, although the theater begins with a hanger, and the operational life of SOC - from the first line, it does not end there. People grow, projects scale, and we have a great many roles and tasks inside. We very much encourage the development drive in first-line engineers, therefore we have worked on several “professional elevators”. A certain percentage of engineers who are ready to move to other lines are even included in the KPI of team lines :) But the story of “boosting expertise” or further staff growth in Solar JSOC will become part of other articles. To be continued ...