“The development team is a self-contained organism”: SEMrush on Java and freedom of choice

SEMrush stands out among Joker sponsorsfor several reasons. Firstly, the line of business: the company is developing a platform for marketers. Secondly, with the participation of the company in the Java conference of her CTO Oleg Krasnov is known for his love of C. Thirdly, for her this Joker will be the first.

Because of all this, it was only more interesting to ask how the company is doing with Java, the development approach as a whole, and the expectations from the conference. Here's who we talked to:

• Elena Danilenko, HR generalist
• Oleg Krasnov , CTO
• Anatoly Chichikov , Senior Java Developer

Elena Danilenko (HR generalist)

- SEMrush becomes the first sponsor of Joker - what does the company expect from the first experience?

- In our company, most of the working issues are decided by personal communication. Therefore, one of our goals is to get to know each other, talk about ourselves and chat with Java developers on topics of interest to us. In addition, we are very pleased to have the opportunity to participate in supporting such a significant event as the Joker Conference. We are for there to be more networking, good and different :)

- Sometimes IT-specialists dislike the concept of “marketing” - are they afraid of the fact that you are doing a marketing platform?

- Most likely, the very notion of “marketing” is a deterrent, but it’s not very clear what exactly is behind this word. But we are not a marketing company, but an IT company. We ourselves are not engaged in marketing, but in the development of a high-tech online platform for people who are professionals in this field or want to understand their market position.

And for us, the important thing is not the product area from which a potential colleague will come to us, but whether he agrees with our approach to work. We try to give development teams the maximum freedom in decision-making - and we work well with those people who are ready to bear full responsibility for their product and for the development process, working to improve the quality of both. We try to clarify this position as much as possible when communicating with candidates.

- SEMrush has offices in four countries, and where are the developers (primarily interested in Java)?

- We are developing in St. Petersburg, Prague (Czech Republic) and Limassol (Cyprus). We have the most Java commands in St. Petersburg: there are seven of them now. Next year we plan to double this amount. Also, Java developers work in our office in Limassol.

Oleg Krasnov (CTO)

- When a company’s CTO specializes in C, a Java developer choosing a job may have a fear that he will be in supporting roles in such a company. What do you have with this?

- The specificity of SEMrush is not in this or that language, but in the following: we believe that the development team is a self-sufficient developing organism. And the team itself can choose what to use for it - including the development language.

This adds freedom, it adds responsibility, but also makes the process of managing the situation more difficult. Management from collecting reports and using the “stick” turns into persuasion and motivation. Communication of equals is to my liking.

I’ll add that we regard Java as a good programming language with a large community and a developing ecosystem. Projects written in Java, we have a great existence and are developing, which gives us confidence that we want their number to grow.

- In 2014, you made a presentation “When to write your database.” And in this connection it’s interesting: what is SEMrush’s approach to the question “take it ready or do it yourself”?

The same as with the choice of language: the guys decide for themselves which way to follow, do something their own or use a standard solution. If we speak historically, then in a number of products there was a transition from standard solutions to self-written, because, as you know, a well-written private solution works better than the standard one.

If we talk about my personal experience presented in the report, then at the time of development there was no stable working Hadoop or Mongo, so I acted on the circumstances :)

- In our recent material about Java vs C ++, you mentioned a product with more than 150 servers, and I want to understand: what is the general scale of SEMrush? What other large products are there, how many servers in total?

- Let's start with the hardware. Now we have about 440 real servers, as well as about 250 virtual machines. The technological development stack, in terms of languages ​​- from ANSI C to Node.js and, of course, Java.

Our largest projects are Backlinks, Domain & Phrase Analytics and Tracking. “Backlinks” - the largest and heaviest of them, more than 150 “machines”, its meaning is to build a database of backlinks for all Internet pages, a kind of Google. “Domain & Phrase Analytics” is the oldest product of the company, more than 60 servers, analytics of search results in more than 30 local databases. “Tracking” is one of the most dynamically developing products, 28 servers, the dynamics of the positions of sites in search results.

These three products from the backend are written in C / C ++, but we also have enough successful Java products - such as “SEO Ideas” and “Site Audit”, you can read more about them on our website .

Anatoly Chichikov (senior developer)

- What technologies / tools are used in the case of Java (both personally by you, and in SEMrush in general)?

- We welcome everything that facilitates the delivery of value to the user with minimal time. We also always try to keep up to date and use the latest stable versions of languages ​​and libraries.

If we talk about the project to which I was involved, then of the Java frameworks, of course, this is Spring (Boot, Web, Data, AMQP). For text analysis we use Lucene. In everyday life, we widely use Lombok, Guava, and Java 8 streams. As for external storage media, these are MongoDB, Redis, and RabbitMQ.

In other Java commands, technologies may vary depending on the tasks, but often the core is Spring and Java 8. Someone stores large amounts of information in HDFS and uses Hadoop for processing. Someone cares about complex queries and transactions - their choice is PostgreSQL. There are teams that, for the trade-offs between insertion speed and query complexity, use, like us, MongoDB.

Build and deploy - Maven, Gradle, Fabric, Ansible. Many teams use Docker in production.

Tools - Intellij IDEA, TeamCity, YouTrack, GitLab. Interesting: our team has been using XRebel quite successfully. And at the company level, we are now introducing an intercommand review code, we are trying Upsource.

- Is there any recent large-scale task that I would like to talk about?

- I think such a large-scale task I can call "SEO Ideas": a project that our team started and has been writing for a little over a year. Its goal is to turn a bunch of metrics and competitor data incomprehensible to the ordinary marketer into specific recommendations for action. People love simplicity, so for now we are growing steadily.

This is a distributed system for streaming data processing. There are concurrency, word processing tasks, work with external APIs and load balancing on them. Server business logic is completely in Java, the client part is written in JavaScript. In my opinion, Java as a language and as a platform confidently holds the blow of new hype technologies (NodeJS, Go, etc.) both in terms of performance and usability. The temptation or need to change it did not arise.

- In SEMrush, much has been done in C / C ++ - and you, as a Java developer, do not have the feeling “it would be better with Java”? :)

- The use of C / C ++ is a necessity in some projects, as these teams face very resource-intensive tasks. One of the most striking examples is link graphing and domain ranking of the entire Internet. Using Java would cost us a lot here, and the decision to use C freed up more resources for other, no less needed products.

- Does Java work well with the “self-organizing teams” approach?

- Yes, it is quite. In our case, the team takes full responsibility for their experiments and product stability, and here Java and its mature libraries show their best side: they rarely shoot their legs.

We also try to write, as far as possible, simple and clear flat modules, preference is given to horizontal connections and constant readiness for changes. And here, Java usually doesn’t let us down either: it has excellent refactoring and dependency resolution tools.

- With the “heterogeneity” of teams that independently choose languages ​​and technologies, do you have any difficulties with interoperability?

- No, there are no problems. All SEMrush is essentially a set of microservices, and we use JSON and REST as the “lingua franca” between them. In some cases, when it comes to a large amount of data or queries, binary solutions are possible, but these are more likely exceptions than rules.

- You have some experience in Groovy and Scala - but is this experience related to working in SEMrush, and do they use these languages ​​in production?

- I myself almost did not write in these languages ​​at SEMrush. But in general, Groovy uses, albeit less actively, than much more. As far as I know, mostly QA loves him, they write beautiful tests on Spock. I myself had an interest in Groovy as a scripting language. In general, it is pleasant and concise, and I understand why people love him. Nevertheless, it turned out that the scripts I write mainly in Python, and I consider it the best tool when it smells like a chore.

And Scala, as far as I know, has not yet taken root with us. Some time ago I myself studied this language closely, and I also can’t say that we agreed on the characters. The language is not simple, it takes a lot of time to learn, and each programmer writes in it very differently. When you browse GitHub on a Scala theme, you can sometimes feel the presence of extraterrestrial intelligence. In my opinion, in an era of rapidly changing requirements and one-button interfaces, these are more likely minuses.

I myself now look with interest at Kotlin, continuing to actively write a product in Java. It captivates that the language is simple, concise, safe, and made by pragmatists for pragmatists. Our company loves such tools. I do not exclude that we will have a project on it if a critical mass of people wishing to try is formed.

- Thanks for the answers, we will be waiting for you at Joker!