Recall 2016: our best posts of the year

New Year is getting closer, there are only a few days left. It's time to take stock. In total, this year we published almost 200 posts (this is the 198th) with a total rating of 6912, received 6,245 comments, 21,835 add to favorites and almost 3 million views! We sat down and with tenderness and nostalgia recalled the best posts on our blog for 2016. Join now!

Dangerous video: how I found a vulnerability in video hosting and did not die after 7 days from cdump

In this post, the Mail.Ru Cloud backend programmer talked about how he found and registered one interesting vulnerability, for which he received a solid gesheft.

Ivan Grigorov: “For top-level baghunters $ 25K a month is not a problem” from Penny_2_Lane

How to start looking for bugs? Could this be the only source of income? Which Bug Bounty to participate in? How much do baghunters earn? And why is vulnerability search especially beneficial in a crisis? Read the answers to these and other questions in our interview with one of the best baghunters in the ranking of the HackerOne platform.

Bear, dismembered and February 14 from puelle

By February 14th, we decided to assemble a robotic teddy bear that would not only be able to communicate and give people a happy hug, but would also analyze human actions. After all, hugs, both lethargic and intense, are capable of betraying true feelings and revealing hidden emotions. Suddenly there are two people in the company who hug each other the same way? What if PHP programmers prefer long touches and barley is limited to dry shaking? With the help of a robot-bear, we were able to collect the most unusual data from the life of IT people.

How to turn from a server-side programmer into a client-side from nikiasi in 7 days

The story of a freshly baked mobile developer about how he abruptly changed his specialization, having left the post of head of the C / C ++ group - development of Mail.Ru Mail.

Some thoughts and tips on optimizing C ++ code from nekipelov

As Cap suggests, in this post one of our programmers shares tips for optimizing C ++ code so that its speed does not fall below the Python / Ruby baseboard .

Malloc experiments by markhor

All POSIX implementations of malloc rest in one way or another on a number of specific functions. Compared to naively highlighting and freeing pages, rounding the required size up, malloc has many advantages. In this article, we conducted three experiments with malloc on POSIX-compatible operating systems - Linux and OS X.

Comparison of Tarantool with competitors in Microsoft Azure by rvncerr

We decided to check how well Tarantool works in Microsoft Azure in comparison with other similar offers - Azure Redis Cache, Bitnami Memcached, Aerospike and VoltDB. By the word “good” we mean “fast”, that is, we will compare the number of processed requests per second (Throughput, RPS).

(Why) Mail.Ru mail includes strict DMARC from z3apa3a

In this article, we explained some technical details of enabling strict DMARC policy on all domains belonging to Mail.Ru Mail, and gave recommendations to service owners, mail servers, and mailing lists.

Go best practices, six years in business from FZambia

Extensive material on Go’s best practices that have stood the test of time. Which of them are outdated or ineffective? What's new in the last year?

Getting Started with Redux by Infected

This tutorial discusses creating from scratch a full-stack application using Redux and Immutable-js. Using the TDD approach, you will go through all the stages of constructing a Node + Redux backend and a React + Redux frontend application. In addition, the use of tools such as ES6, Babel, Socket.io, Webpack, and Mocha is discussed. The set is very curious, and you instantly master it!

How to configure an extensible system for regression testing on phones: the experience of mobile Mail.Ru Mail from p_alexey

The article talks about how we built from scratch a flexible and extensible system for running autotests on Android smartphones. Now we use about 60 devices for regression testing the Mail.Ru Mail mobile application. On average, they test about 20 builds of the application daily. For each assembly, about 600 UI tests and more than 3,500 unit tests are performed.

Stylization of images using neural networks: no mystery, just matan from mephistopheies

Surely you have noticed that the theme of styling photographs for various artistic styles is actively discussed on these your internet. Reading all these popular articles, you might think that magic is happening under the hood of these applications, and the neural network really fantasizes and redraws the image from scratch. It just so happened that our team faced a similar task: as part of an internal corporate hackathon, we made a video stylization, as application for photo already existed. In this article, we will figure out how this network "redraws" images, and consider the articles that made this possible.

Features of the file systems we encountered while developing the Mail.Ru Cloud synchronization mechanism from askogorev

One of the main features of the Mail.Ru Cloud desktop client is data synchronization. Its purpose is to bring the folder on the PC and its presentation in the Cloud to the same state. When developing this mechanism, we met with some, at first glance, fairly obvious features of various file and operating systems. However, if you do not know about them, you may encounter rather unpleasant consequences (it will not work to download or delete the file). In this article, we have gathered features whose knowledge will allow you to work correctly with data on disks and, possibly, save you from the need for an urgent hotfix.

Arrays in PHP 7: hash tables from AloneCoder

The article describes in detail how hash tables are arranged in PHP 7, how you can work with them from the point of view of the C language, and how to manage them using PHP tools (using structures called arrays). Do not forget that we use hash tables everywhere (usually as dictionaries), therefore, you need to design them so that they are quickly processed by the processor and consume little memory. These structures are critical to the overall performance of PHP, since local arrays are not the only places where hash tables are used.

Breaking garbage collection and deserialization in PHP from AloneCoder

A story about two use-after-free vulnerabilities in the garbage collection algorithm in PHP. One is present in all versions of PHP 5 ≥ 5.3 (fixed in PHP 5.6.23). The second is in all versions of PHP ≥ 5.3, including versions of PHP 7 (fixed in PHP 5.6.23 and PHP 7.0.8). Vulnerabilities can be remotely applied via the PHP deserialization function. Using them, the authors of the article found RCE on pornhub.com, for which they received a bonus of $ 20,000 plus $ 1,000 for each of the two vulnerabilities from the Internet Bug Bounty committee on Hackerone.

PHP: wrong path from AloneCoder

There are a number of trends in the world of PHP programming. Some people actively promote them (in books and on websites) as “modern PHP”, while other approaches expose them as obsolete, stupid or simply incorrect. It seems that all these people are tirelessly trying to get everyone to program as they see fit. This article is written to share a pragmatic outlook on PHP programming. A look dictated by experience and practical implications, rather than popular trends, theories, or academic dogmas.

Frontend: Development and Support (+ Voting) by RubaXa

Let's imagine that you were transferred to a new project. Or you changed jobs and only heard about the project. So you sit down at the workplace, a manager comes to you, shakes hands and ... right away, opens the project page, pokes a finger at the monitor and asks to insert an “informer about the upcoming event X”. On this you part ... What to do? Where to begin? How to create an “informer”? Where to find the right template? And a sea of ​​other issues. This is a story about how we try to organize these processes, what tools we create for the preparation of SPA. In addition, we will talk about the technical details of the Live Coding / Hot Reload implementation and a bit about VirtualDom and React with Angular.

ExcelArt - isometric "freebie." We draw a pseudo-volume phone without 3D and Photoshop from flash_b

Is it possible to get a pseudo-volume image without 3D programs? “Of course,” you say, “you take it, you strain and draw from scratch.” And if without “you take, you strain and draw”? And if at all without any special knowledge? Is it possible to get isometric objects without wasting time on rendering? Is there a digital illustration at all outside the usual graphic programs? These and other questions are answered in an article devoted to a new method of creating digital illustrations - ExcelArt.

Difficult quest for habravchan: 25 levels from gag_fenix

This post presents a puzzle designed specifically for Habr users. Challenge yourself, can you solve it?

Overview OPCache extension for PHP from AloneCoder

PHP is a scripting language that compiles by default the files you need to run. During compilation, it extracts the opcodes, executes them, and then immediately destroys them. PHP was designed this way: when it proceeds to execute the R request, it “forgets” everything that was performed during the R-1 request. On production servers, it is very unlikely that the PHP code will change between multiple requests. So we can assume that during compilation the same source code is always read, and therefore the opcode will be exactly the same. And if you extract it for each script, you get a waste of time and resources.

The book "How to survive the full end of dinner, or security in PHP." Part 1 from sunsai

There are many articles and security guides on the Internet. This book seemed to us quite detailed, while concise and understandable. We hope that it will help you learn something new and make your sites more reliable and safer.

Springboard for calling magic functions in PHP 7 from AloneCoder

This article details optimization in a virtual machine in PHP 7 (Zend virtual machine). First, we will touch on the theory of springboards of function calls, and then we will learn how they work in PHP 7. If you want to fully understand everything, then it is better to have a good idea of ​​the Zend virtual machine. To get started, you can read how the VM works in PHP 5, and here we’ll talk about the PHP 7 VM. Although it has been redesigned, it works almost the same as in PHP 7. Therefore, if you understand the PHP 5 VM, then figure it out With VM PHP 7 there will be no difficulty.

Lectures of the Technosphere. 1 semester. Introduction to data analysis (spring 2016) by Olga_ol

A new collection of lectures by the Technosphere Mail.Ru, this time we are presenting in open access the spring course “Introduction to Data Analysis”, where students are introduced to the field of data analysis, the basic tools, tasks and methods that any data researcher encounters in his work. The course is taught by Yevgeny Zavyalov (analyst at the Mail.Ru Search project, engaged in extracting knowledge useful to business from data generated by the search engine and desktop applications), Mikhail Grishin (programmer-researcher from the data analysis department) and Sergey Rybalkin (senior programmer from Allods Team studio) .

Pitfalls of Bash from AloneCoder

In this article, we will talk about mistakes made by Bash programmers. There are some flaws in all the examples given. You will be able to avoid many of the errors described below if you always use quotation marks and never use word splitting! Wordbreaking is a flawed legacy practice inherited from the Bourne shell. It is used by default if you do not enclose the expansions in quotation marks. In general, the vast majority of pitfalls are somehow connected with substitution without quotes, which leads to word splitting and globbing of the result.

A review of top convolutional neural network topologies from mephistopheies

An article with a talking title, do not add, do not decrease. Material for trained readers: it is understood that you are already familiar with the back-propagation algorithm of the error and understand how the basic building blocks of convolutional neural networks work: convolution and pooling.

Not a single gap: how we created a wireless network for 3000 devices from znoom

The office of Mail.Ru Group is a 26-storey building in which a lot of people work. This is a story about how our Wi-Fi network has changed over the past 11 years, and what it is today.

Stories of an Old Mailushka by yeah_boss

Burning servers, a catastrophic drop in services, the heroic rescue of the portal with the help of the wife left at the computer. This could start a fantastic technotriller, but in fact, this is how the history of the Russian Internet began, of which we became a part. In honor of the 18th anniversary of the company, we first talked about the real history of Mail.Ru Group. For the first time in years of life, we will share what used to be at the level of myths and legends even within the company itself. We talked to people who had worked for the company for many years and were directly involved in all the events.

What happened when we got tired of looking at the graphs of 5,000 servers in monitoring (and when there were more than 10,000 servers) from Sharapoff

The story of how Odnoklassniki got tired of manually tracking 5000 servers and made a new monitoring system. Now it takes 1-2 hours a week to process alerts with work with 10 thousand servers.

JSON parsing is a minefield from AloneCoder

JSON is the de facto standard when it comes to (de) serialization, network sharing, and mobile development. But how well are you familiar with JSON? We all read specifications and write tests, test popular JSON libraries for our needs. The article shows that JSON is an idealized format, and not an ideal format, which many consider it to be. There are not even two libraries behaving identically. Moreover, extreme cases and malicious payloads can lead to bugs, crashes and DoS, mainly because JSON libraries are based on specifications that evolve over time, which leaves many things poorly or not documented at all.

We invite you to the Russian AI Cup 2016 from sat2707

This post discusses in detail the problem that was proposed to the participants of the Russian AI Cup 2016 championship.

50 Go Shades: Traps, Pitfalls, and Common Beginner Mistakes from 3vilhamst3r

Go is a simple and fun language. But in it, as in any other languages, there are pitfalls. And in many of them Go is not to blame. Some are a natural consequence of the arrival of programmers from other languages, while others arise due to misconceptions and a lack of details. If you take the time and read the official specs, wikis, mailing lists, blog posts, and source code, many of the pitfalls will become obvious to you. But not everyone starts like that, and that's fine. If you are new to Go, this article will save you a lot of hours that you would spend on debugging your code. We will consider versions of Go 1.5 and below.

ICQ: 20 years is not the limit from Dimitryopho

Until recently, we recalled how, from release to release, ICQ was undergoing plastic surgery and face lifting. And on the anniversary day - ICQ turned 20 years old - we decided to look in retrospect what technologies have been used in ICQ for two decades.

CSS maths from AloneCoder

CSS-lock is a technique from adaptive web design that allows you not to jump from one value to another, but to switch smoothly, depending on the current size of the viewport. The idea and one of the implementations was proposed by Tim Brown in the article Flexible typography with CSS locks. The article describes the technique itself, its limitations and the underlying mathematics. Do not worry: there are mostly additions and subtractions. In addition, everything is divided into stages and decorated with graphs.

Monitoring and tuning the Linux network stack: getting data from AloneCoder

This article discusses how packets are received on computers running the Linux kernel, and discusses the issues of monitoring and configuring each component of the network stack as packets move from the network to user space applications. You will find a lot of source code here, because without a deep understanding of the processes, you cannot configure and monitor the Linux network stack.

Puzzles TCP from AloneCoder

They say that you cannot fully understand the system until you understand its malfunctions. In this article, TCP errors are presented as puzzles, in the style of Car Talk or old Java puzzles. Like any other good puzzles, they are very easy to reproduce, but the solutions are usually surprising. And instead of focusing our attention on mysterious details, these puzzles help you learn some of the underlying principles of TCP.

Lectures of the Technosphere. Preparatory course "Algorithms and Data Structures" (spring 2016) by Olga_ol

The goal of this course is to introduce students to the basic algorithms used for software development. You will learn how to choose the appropriate data structures and algorithms for the implementation of emerging tasks, and learn how to use C / C ++ languages ​​to implement algorithms.

Efficient storage: how we made 50 PB out of 50 PB from AndrewSumin

Changes in the ruble exchange rate two years ago made us think about ways to reduce the cost of iron for Mail.Ru Mail. We needed to reduce the amount of purchased hardware and the price of hosting. What mail consists of: indexes and bodies of letters 15% of the volume, files - 85%. A place for optimizations must be sought in the files (attachments in letters). At that time, file deduplication was not implemented; according to our estimates, it can save up to 36% of the entire mail volume: the same letters come to many users (mailing of social networks with pictures, shops with price lists, etc.). This post talks about the implementation of such a system, made under the leadership of PSIAlt.

About C language and performance from AloneCoder

If a programmer is well acquainted only with high-level languages, for example, PHP, then it is not so easy for him to master some ideas that are characteristic of low-level languages ​​and critical for understanding the capabilities of information and computing processes. For the most part, the reason is that in low- and high-level languages ​​we solve different problems. But how can you consider yourself a professional in any (high-level) language, if you don’t even know how the processor works, how it performs calculations, or in an efficient way? Today, automatic memory management is becoming the main problem in most high-level languages, and many programmers come to its solution without a sufficient theoretical base. Knowledge of low-level processes greatly helps in developing effective high-level programs.

On the eve of the birthday of the first female programmer: my story from 6thSence

“You're a girl, why do you need this?” or "In women, the brain is not adapted to programming." “We will be doing business here, and you are the decoration of the team.” Surely every woman programmer has heard this many times in her address. This is the story of our employee about how she came into the profession of a programmer.

Debugging your operating system: memory allocation lesson from AloneCoder

The story of how, at first glance, an unremarkable bug report led to a whole investigation into the Linux memory allocation system.

The Truth About traditional benchmark JavaScript from AloneCoder

For many of those who have studied programming languages, compilers and virtual machines, the fact that for all its elegance in terms of the structure of the JavaScript language is still not very well optimized in terms of compilation and cannot boast of a wonderful standard library is still surprising. Depending on who your interlocutor is, you can list the flaws in JavaScript for weeks and still find some kind of oddity that you have not heard about. But despite the obvious flaws, today JavaScript is a key technology on the web, is moving towards dominance in the server / cloud sphere (thanks to Node.js), and also penetrates the Internet of things.


Thanks to everyone who read us, commented, put the pros and cons. We hope that in the coming year, the posts will be even more interesting. Holiday greetings!