How, to whom and why go to consulting? Personal experience with Big Data

Today I’ll talk about how IT consulting works on the example of Big Data, share my personal experience, how I got into this area, and case studies, as well as give advice on who and why should try yourself in consulting.

I graduated from the Mechanics and Mathematics Department of VN Kharkiv National University Karazina, got into the DataArt position of Java Trainee and worked there for the next 6 years. My career developed rapidly due to the fact that love came together to solve complex puzzles, my innate desire to learn, constantly find something new for myself and a team of mega-professionals. Once I was attracted to an internal IoT project, where we, along with other enthusiasts, figured out how to connect sensors to a microcontroller, where to store (and most importantly, is it necessary?) This huge amount of data, how to process it later, and how to track it in real time state of the system, predict breakdowns, where and how to host it all, etc. So we quickly came to the questions and tasks of BigData.

This gave a strong impetus to my development and allowed me to work in 15+ IoT and BigData projects, regularly participate in pre-sales, speak at conferences and with educational courses. My position during this time has changed from Senior Developer to Big Data Architect. At first it was interesting, but over time it turned into a routine for me. In most cases, I went to the project, the first month or two was very active, I established processes, communication with the client and thought over the architecture, and after that I became one in those. lead, PM, business analyst and many others)) After six months, it all turned into an endless groundhog day.

Therefore, in search of new experience and new challenges, I switched to SoftServe. After almost a year, I can share my experience and vision of how everything works here.

First, let's look at how the consulting process works. Indeed, most IT employees are involved in the project when there is already a contract, budget, manager, and it is relatively clear what, when and with which team you need to do. Few people wonder how new projects, in principle, get into the company and what happens to it at the initial stage.

In fact, in the life of each project, the following phases can be distinguished:

• Presales
• Discovery
• PoC / MVP
• Implementation
• Support

Discovery phase - this is the very consulting for which we are all here. As soon as the contract is signed, someone from the consulting group goes to the client. Ideally, a business analyst with a UX designer joins him (I advise those interested in methodological issues to read about the Design Thinking approach, the effectiveness of which I have already experienced more than once in my own practice). In an ideal ideal, an engagement manager also goes, who takes care of all the issues of organizing the process, and a narrow-profile specialist.

• Onsite - in real communication with the client we collect as much information as possible;
• Offsite - together with the team, we analyze everything that we managed to dig up, design the system, prescribe the risks and the implementation plan.

At the output, this group provides the client with a document that contains:

• High-level description of the current state of the system;
• Scenarios for the use of a future product;
• Prioritized architectural and business drivers;
• Requirements for data volume, processing speed, permissible delay;
• An architectural vision reflecting all previous requirements;
• The selected technology stack (the choice of which is based on trade-off analysis of possible implementation options in the context of system requirements);
• Estimate and backlog (task list) for the next vase, development plan;
• Risks and possible ways to avoid them.

If everything goes well and the client signs the next phase, then the architect has been working on this project for some time (most often part-time). Until the process is established, the scope of work is clear and the client is satisfied. If everyone here is happy and wants to continue cooperation, then Proof of Concept (PoC), Minimum Viable Product (MVP), and further Implementation and Support follow.

It would seem that this whole process is clear, structured and understandable. But in practice, everything is not so simple. Let's look at possible scenarios.

Here a potential client comes to us. The sales team conducts the first calls / rallies in order to understand what kind of problem needs to be solved, and turns to narrower specialists. If the project requires knowledge of the Big Data-stack, then the potential client comes to my group.

The next step is to understand the difficulty level of the task and determine which experts are needed. There are several options.

The client does not know at all what he wants

Most often this happens with large and adult enterprise companies that understand that they are losing money, but do not understand why. Or they understand, but have no idea how to fix this.

From my own experience, I can say that often in such a situation, you first need to solve business problems, and only then technology. You need to be prepared for the fact that it is not easy and interested parties may appear who will try to draw you into their political games.
For example, one of the CTOs that I was lucky to be familiar with led me once to a strategic rally to discuss the purchase of a license for one popular product. The official reason for my appearance was just an acquaintance. The board of directors was delighted with the upcoming deal and sang praises to their future partners. As an honest and thoughtful person, I asked a couple of more substantive questions about how they are going to integrate. There was no answer, but the deal was canceled. Leaving the meeting room, I asked the very CTO if I had gone too far with my questions and whether I should have climbed into it all. To which the "secret cardinal" replied: "Everything is super, I brought you there for this."

How to avoid this? No, this is consulting. One must be prepared for this, in which personally knowledge of psychology, more experienced colleagues and personal experience help me, of course.

The client has a vision of the final product, but he needs our expertise to make sure that he is right.

Probably the most profitable option in terms of time costs. Much less effort will be spent on finding a problem, communication with a customer will be much more productive, for which your opinion is still authoritative.

However, there are pitfalls. For one of our clients in the Discovery financial sector, the phase was designed for 4 weeks, 2 weeks for onsite and offsite. The client for 2 weeks onsite did not say a word that he already has his own vision of architecture. There was a discussion of some conceptual things, but there was never a conversation about a technology stack. And in the middle of the 4th week of the project, when we had already accumulated a ton of material, collected, prioritized and reflected the main requirements in the final solution, during the presentation of the architecture, the client begins to resent and say that we did not hear him. It turned out that our task was not so much to come up with something new as to guess what was already in the head of our main stakeholder. Thanks to my big boss who was on that call and quickly realized which direction the wind is blowing. A couple of keywords and the client are satisfied again, and for the remaining days we completely change the concept, re-reflect all the requirements and risks, but already in the architecture agreed with the client.

How to avoid this? Validate, validate and validate your decisions with the client again, the sooner and more often, the better. It is better to say several times in parts what everyone already understood than 4 weeks later for the first time to show the final architecture. And this concerns not only the architecture, but also the approach to Discovery as a whole.

The client has a very accurate and detailed vision of the product, from you he only needs execution

On the one hand it is convenient. No need to find out the requirements, mess with the documentation, go to the client, in the end. Take and do. But often it doesn’t work out that way. From the first days it becomes clear that the client’s vision is far from ideal, since far from all risks and requirements have been taken into account, estimates are clearly underestimated, and it is extremely difficult to convey this idea, because the client is firmly confident in his decision.

How to behave in this situation? My opinion is that you should initially evaluate the profitability of the project in relation to the resources of the team that you will need to spend on it. After all, such projects do not always bring a lot of profit, and you need to spend a lot of energy on it. If we decide to undertake this comment, I try to convey to the client the idea that we, as experienced consultants who have implemented many projects, strive to use our experience to find the most rational solution to his problem, and not just blindly implement the proposed idea. I also inform that the obligatory part of our work is to warn him of all possible risks and, if necessary, give recommendations on how to avoid them. That is, you need to earn credibility and ensure that the client begins to listen to you.

You must always be able to listen and hear both the client and the project colleagues. After all, the challenges we face are often non-technical in nature.

Summing up, I can say that consulting is for those who are not sitting still and are not afraid to try something new. If you have been working on the same project for more than a year, the work has bored you, but the fear of changing something is haunting - try to take on a little more responsibility. Pay for an educational course, think up and offer optimization to the client, convey its value to him, sell it. Grab something new, step out of your comfort zone. Remember that there is no bad experience, and even failure brings a lesson, which means it makes us better. If the experiment is successful, then fear will disappear and you will be filled with pride and a desire to move on, to do even better and more. At this moment, you can safely come to us!