How we measure the quality and effectiveness of documentation development. Background and basics. Yandex Report

Says Svetlana Kayushina, head of the documentation and localization department.

Our documentation department has gone through several stages of development. First there was a technical writer who performed the tasks of an individual customer. Then a group of technical writers was formed, which solved a limited set of tasks. Now we have a large production department - it fully meets the needs of the company in documentation.

At each stage, the requirements for the work of the department changed and new metrics appeared to measure the quality of documentation and the service for the provision of documentation services. Metrics help us improve the product, processes and quality, as well as timely inform about failures.

First metrics

The first technical writer was needed by Yandex to document the technology that almost all developers used. Information about it was transmitted orally to each other, so part of it was inevitably lost or distorted.

Documentation was written, customers were satisfied, so we decided to master other areas. We started with a description of services that are critical to business: they are used by many company employees.

At the initial stage, customers set us the task of documenting the technology. The only metric was the availability of documentation: it was either written or not. When a technical writer was alone, the quality was evaluated by the customer himself. When there were more writers, service consultants joined the customers, and we checked the quality of the documentation inside the department.

The flow of tasks was constantly increasing, so there was a problem with maintaining a single level of quality. Expert assessment is rather subjective, but we wanted it to be independent.

In addition, customers came up with tasks of varying complexity: for example, capturing information that one person knew, or describing complex technology for developing our external products.

All this created certain problems, so we needed a system for prioritizing tasks and assessing the quality of documentation.

Internal Documentation Metrics

We began to collect feedback from internal customers and readers who used the documentation to solve their problems. For example, a customer could tell us that he stopped consulting his colleagues 10 hours a week and is now doing his main work more productively.

We asked readers how well the documentation helps them solve problems. Tracked attendance: how many users per week access documents. If more than a certain number, then the document is worth supporting.

This method required significant costs and direct access to the customer. In conditions of mass use of documentation, it turned out to be too expensive and poorly applicable. Therefore, when we started developing documentation for external users, problems appeared.

The audience has grown, we did not have direct access to it. We tracked reviews on social networks and technical support requests, but this was not enough. I had to build hypotheses about who our user is in order to satisfy his needs as much as possible. To assess user behavior and product performance, we used Yandex.Metrica.

Development of a metric system for external documentation

The business objectives did not change much, but a new condition appeared: we had to manage to prepare the documentation for the official release. Now we needed to meet deadlines.

We continued to use the previous metrics, but documentation service attendance and refusals were added to them. They were tracked both for the service as a whole and for individual documents.

Technical writers used additional data to improve the quality of documentation:

• analysis of user behavior on the service,
• heatmap of clicks,
• web viewer
• assessment of search queries, etc.

Then we proceeded to the mass production stage and became a service for the provision of documentation services.

Documentation Mass Production Metrics

At this stage, customers and management have new requirements for the documentation department.

Customers were interested in the timing and assessment of the complexity of the tasks. For example, they asked why we write a document for two weeks, not two days. At the same time, they expected high quality from us.

Management was interested in cost justification. To hire a person to the department, we had to argue this. For example, we could show a measurable benefit of a document if it solves the problems not of a couple of users, but of several thousand, and reduces the load on technical support.

Our statistical metrics were no longer enough to answer these questions.

In the context of mass production, we have several main tasks:

1. document technology
2. keep up with the releases,
3. provide a high level of quality.

This requires an integrated approach to everything we do.

In addition, we, as a service, must evaluate the quality of our services. We need to understand how satisfied users are and what benefits high-quality documentation brings.

To develop a new assessment system, we conducted a study and analyzed a large number of sources: a total of 136 metrics were found. Besides the fact that there are too many of them, not all of them are easy to measure and we do not need all of them. Therefore, we chose metrics that fit our conditions.

We were interested in the following aspects of the assessment:

1. Design and business metrics are indicators of the effectiveness of the quality of the service as a whole. They are needed to increase customer satisfaction and to predict the resources of the department to perform all tasks.
2. Document quality assessment metrics to enhance user satisfaction.

When choosing metrics, we took into account the complexity and cost of calculating the metric. It is also important for us that the essence and calculation of the metric should be clear to people who are not immersed in the nuances of our production process.

As a result, we selected 20 out of 136 metrics, and then divided them into four groups. We display metric data on dashboards that are available to any employee of the company.

In the next part of the article, we will describe in more detail the metrics for the development and support of documentation and how we consider them. Sign up for new comments - we will tell you in them about the release of the next part of the report.