June 11, 2013 at 01:38Qualitative approach to rating startups
In the 20th century, experts from various countries faced the most acute problem of finding a reliable methodology for assessing quality, including the results of the activities of creative teams, their workers, products, projects and services. As a result, in 1968, the scientific discipline of qualimetry [1] appeared in the world, the demand for the use of which has grown significantly recently due to the rapid development of the Internet and in connection with the expansion of areas of human activity, which are characterized by the quantification of research methods, means and methods. The qualimetric method was developed by G. G. Azgaldov [2] and developed in numerous works of Russian and foreign researchers.
One of the successful examples of using the qualimetric approach in rating is the Golden Brand National Franchising Award [3, p.121-143], which is summed up in three categories since 2006 using the qualimetric methodology (the main category used a comprehensive indicator - "Integral quality").
Since the methodological approaches used in qualimetry can be effectively implemented when evaluating and selecting startups, especially at an early stage of development, it is proposed to use qualimetric analysis to rank startups.
Among the many different approaches to evaluating startups [4-6, 13], the use of qualimetry is preferred. And that's why. If you look at the dictionaries, you can find that the "rating" comes from the English term 1 , which means a subjective assessment of any poorly measured phenomenon on a given scale. In a situation where the main criterion for a ranked list of startups is “quality” (or “integral quality”), and the given scale is determined by the world level, then a synonym for the word “rating” is “qualimetric analysis”.
In [5], a brief overview of some expert methods for evaluating startups is given, among which, in the early stages, the following can be used:
Figure 1 shows the PRUFFI Agency startups rating methodology in the form of a property tree that reflects professional prospects and the degree of professionalism (both operational and organizational) of startup projects teams [13].
Fig. 1 Rating of “30 most promising startup teams of the Russian Internet and IT” in the form of a property tree
This rating is formed on the basis of scoring (R = S + P + K) set by experts for team stability and cohesion (complex property 14 in Fig. 1), professional competencies (property 15 in Fig. 1) and team activity (property 16 in Fig. 1). According to the authors, this rating demonstrates typical errors inherent in most ratings, which consist in the arbitrary selection of rating criteria and the weights of these criteria (in this case, points).
The Ave Maria model is useful when comparing startups and choosing the preferred investment direction, but it, unlike the qualimetric approach, does not give numerical estimates. A significant drawback of the Bill Payne method is the arbitrary choice of criteria for comparison, the arbitrary choice of weights of these criteria and the uncertainty of the measurement object. Unlike the Bill Payne method, qualimetry makes it possible to logically justify the selection of criteria for comparing the evaluated objects, relying on the results obtained in the process of determining the assessment situation, determining the weight of the evaluation criteria, relying on information about the world level of quality of similar objects of assessment, and reducing uncertainty due to scientifically based rules for constructing trees of evaluation criteria. Some methodological problems of quality assessment and their solutions are considered in [7].
When rating startups as one of the criteria, for example, you can use the “project cost”, and of the approaches to evaluating you can use cost estimation methods. Such methods have been successfully used, for example, by professional appraisers and investment advisers (but most often at later stages of development - Early stage, Expansion stage, Later stage). At the early stage of development (Seed stage), the application of valuation methods for rating purposes is not effective for two main reasons: 1) due to the high costs of using the methods; 2) due to the high degree of uncertainty of the initial data of the model (in forecasts of market development, cash flows, in the assessment of supply / demand, risks). Compared to valuation methods,
When conducting national ratings of startups (as well as selecting appropriate experts), it is proposed to use qualimetric techniques that allow startups to be ranked by quality (or integral quality), while ensuring transparency of measurement procedures, the validity of the selection of criteria for comparison and the comparability of startups' performance [8, 9 ]. The use of qualimetry in evaluating (ranking) startups not only minimizes the costs of rating organizers when summarizing, but also allows them to be comprehensively assessed, because qualimetry has an apparatus with which any economic and non-economic in nature objects and processes can be quantified and thereby both quality assessment criteria and performance evaluation criteria can be simultaneously considered2 startups. At the same time, assessment methods based on qualimetry can be customizable taking into account the strategic goals of the “stakeholders” with a given accuracy and allow not only measuring individual quantitative and qualitative parameters of a startup, but also comparing their performance indicators, for example, by integrated quality.
All startups differ from each other in areas of activity, and stages of development, and results, and costs to achieve their goals. Qualimetric analysis (especially at the Seed stage) allows you to take into account all the criteria and performance indicators of startups (the nature of which is not subject to any restrictions), bring all indicators together, convert heterogeneous indicators to homogeneous, ensure reliability, the necessary accuracy, convolution of indicators and, most importantly , comparability of results.
In order to answer the question which of the methods is most preferable when rating startups by quality, we turn to the papers [10, 11]. Based on a study of the literature on the theory of economic efficiency, the logic of estimates, operations research and decision theory 314 requirements were formulated, according to which 16 different quality assessment methods were analyzed. It was shown in [10] that the qualimetric method for assessing quality is the most acceptable of all currently used methods for this purpose. Of course, the superiority of the method of qualimetry over the others cannot be understood as superiority over all methods for all requirements. The qualimetric method is better than others only in the aggregate of all 14 requirements, and for some requirements taken separately, it can be inferior to other methods. For example, at the request of “non-laboriousness”, of course, it is inferior to the method of “assessment by one expert” [11, Table 1, “B”]. However, due to automation, even the laboriousness of the qualimetric method for assessing quality can be significantly reduced up to its comparability with other methods.
Thus, it is proposed to develop a qualimetric methodology for rating startups, in which the indicator “integral quality” will be used as a criterion for ranking startups at an early stage of development, and “quality” when selecting experts (or taking into account their expert assessments).
The qualimetric approach to quantifying quality is published in open sources, for example, in [2, 3, 7-10, 12] and is used in various spheres of human life.
- 1 “rating” - rating, order, classification. Rating - a numerical or ordinal indicator that reflects the importance or significance of a particular object or phenomenon.
2 In qualimetry, the term “effectiveness” is similar to the term “integral quality”
3only those requirements were taken into account, the validity of which is confirmed by the sufficiently high frequency of their mention in the literature. Therefore, claims that are mentioned very rarely or poorly substantiated in the literature were not used
The main terms used in qualimetry in assessing the quality of startups:
Qualimetry is a scientific discipline that studies the methodology and problems of a comprehensive quantitative assessment of the quality of any objects (processes, products or services).
Quality - a set of properties of an object that characterizes the results generated by it (both positive and negative), but which do not include the cost of resources for its creation and consumption.
Profitability - such a property of the object that characterizes the total (full) costs of the evaluated object, i.e. include the costs of its production, consumption and disposal.
Integral quality is such a property of an object that characterizes the totality of its quality and profitability.
Startup quality- this is a comprehensive description of the processes for creating and launching a new product (device, work, service) on the market, satisfying the needs of interested parties and representing the totality of the results obtained, but which do not include indicators of resources (for example, cash) used in these processes.
If, however, the expended resources are also taken into account, then the characteristic studied in this way is called the “integrated quality of a startup,” which is synonymous with “startup efficiency.”
Qualimetric analysis is a research method that allows you to compare and evaluate the quality of objects in relation to the world level.
Startup Rankings- a comprehensive assessment and subsequent ranking of young Russian technology companies according to their degree of development, in order to identify their investment attractiveness, as well as to describe and structure the market for such companies. Source - www.rg.ru/2012/12/09/reyting-site.html
Authors :
Kostin A.V., Ph.D., Head of the Department for Evaluation of Innovative Projects of Center for Technology Transfer CJSC, kostin @ rusttc. ru.
G. Azgaldov, Doctor of Economics, Chief Researcher, Central Economics and Mathematics Institute of the Russian Academy of Sciences, [email protected]
Literature
1. Kostin A.V. The birth of qualimetry // Standards and quality. - M., 2010, No. 2. - S.90-91. URL: www.labrate.ru/kostin/2010-02_nucleation_qualimetry.pdf (accessed 06.06.2013).
2. Azgaldov G.G. Development of the theoretical foundations of qualimetry: Dis. for a job. learned. Doctor of Economics / Military engineer Acad. them. V.V. Kuibyshev. - M., 1981. URL: www.labrate.ru/azgaldov/azgaldov-qualimetry-referat-1981.doc (accessed: 06.06.2013).
3. Qualimetry: initial information. Reference manual with an example for ANO 'Agency for Strategic Initiatives to Promote New Projects': Textbook. allowance / G.G. Azgaldov, A.V. Kostin, V.V. Gardens. - M .: Higher. school, 2011 .-- 143 p. URL: www.labrate.ru/kostin/051326.pdf (accessed: 06.06.2013).
4. Dmitry Chernyak. Methods for evaluating a startup [Electronic resource]. URL: habrahabr.ru/company/findstartup/blog/144521 (accessed 06.06.2013).
5. Dmitry Chernyak. Methods for evaluating a startup. Continuation of [Electronic resource]. URL: habrahabr.ru/company/findstartup/blog/144221 (accessed 06.06.2013).
6. Startups received a rating / Rossiyskaya Gazeta, December 9, 2012 [Electronic resource]. URL: www.rg.ru/2012/12/09/reyting-site.html (accessed: 06.06.2013).
7. Garry G. Azgaldov, Alexander V. Kostin. Applied Qualimetry: Its origins, errors and misconceptions // Benchmarking: An International Journal, Volume 18, Number 3, 2011, pp. 428-444. - www.labrate.ru/articles/428-444_applied_qulimetry_azgaldov-kostin.pdf(Date of treatment: 06/06/2013).
8. Azgaldov GG, Kostin AV (2012) Increasing the Validity of Results of a National / International Competition: A Case Study / Europe Middle East Africa Members' Meeting, Barcelona (Spain), January 26-28, 2012. - www.labrate. ru / kostin / 054548.pdf (accessed date: 06/06/2013).
9. Qualimetry for all: Textbook. allowance / G.G. Azgaldov, A.V. Kostin, V.V. Gardens. - M.: InformZnanie, 2012 .-- 165 s. - www.labrate.ru/kostin/064571_qualimetry_azgaldov-kostin-sadovov-2012.pdf (accessed: 06.06.2013).
10. Azgaldov G.G. Qualimetry in architectural and construction design - M.: Stroyizdat, 1989. - 264s. www.labrate.ru/azgaldov/qualimetry_in_architectural_and_structural_design.pdf(Date of treatment: 06/06/2013).
11. Theses Kostina A.V. to the round table discussion “How and why to measure the quality of education at a university?” (November 20, 2012, CEMI RAS) // Library LABRATE.RU [Electronic resource]. URL: www.labrate.ru/20121120/063804.doc (accessed: 06.06.2013).
12. Azgaldov G.G. Theory and practice of assessing the quality of goods. Fundamentals of qualimetry .. - M .: Economics, 1982. - 256s. - www.labrate.ru/azgaldov/azgaldov_theory_and_practice_of_quality-assessment.pdf (accessed: 06.06.2013).
13. On the rating of “30 most promising startup teams for Runet and IT” // Electronic resource - pruffi.ru/30startups/about (accessed: 06.06.2013)
One of the successful examples of using the qualimetric approach in rating is the Golden Brand National Franchising Award [3, p.121-143], which is summed up in three categories since 2006 using the qualimetric methodology (the main category used a comprehensive indicator - "Integral quality").
Since the methodological approaches used in qualimetry can be effectively implemented when evaluating and selecting startups, especially at an early stage of development, it is proposed to use qualimetric analysis to rank startups.
Among the many different approaches to evaluating startups [4-6, 13], the use of qualimetry is preferred. And that's why. If you look at the dictionaries, you can find that the "rating" comes from the English term 1 , which means a subjective assessment of any poorly measured phenomenon on a given scale. In a situation where the main criterion for a ranked list of startups is “quality” (or “integral quality”), and the given scale is determined by the world level, then a synonym for the word “rating” is “qualimetric analysis”.
In [5], a brief overview of some expert methods for evaluating startups is given, among which, in the early stages, the following can be used:
- Ave Maria model (short for Acquisition, Value, Engagement, Monetization, Retention, Intellectual Property). The author of the model is Maxim Krainov
- Scoring method (Scorecard Method, a method also known as a method of a benchmarking method - Bill Payne Method, Benchmark Method). The author is investment “angel” Bill Payne.
Figure 1 shows the PRUFFI Agency startups rating methodology in the form of a property tree that reflects professional prospects and the degree of professionalism (both operational and organizational) of startup projects teams [13].
Fig. 1 Rating of “30 most promising startup teams of the Russian Internet and IT” in the form of a property tree
This rating is formed on the basis of scoring (R = S + P + K) set by experts for team stability and cohesion (complex property 14 in Fig. 1), professional competencies (property 15 in Fig. 1) and team activity (property 16 in Fig. 1). According to the authors, this rating demonstrates typical errors inherent in most ratings, which consist in the arbitrary selection of rating criteria and the weights of these criteria (in this case, points).
The Ave Maria model is useful when comparing startups and choosing the preferred investment direction, but it, unlike the qualimetric approach, does not give numerical estimates. A significant drawback of the Bill Payne method is the arbitrary choice of criteria for comparison, the arbitrary choice of weights of these criteria and the uncertainty of the measurement object. Unlike the Bill Payne method, qualimetry makes it possible to logically justify the selection of criteria for comparing the evaluated objects, relying on the results obtained in the process of determining the assessment situation, determining the weight of the evaluation criteria, relying on information about the world level of quality of similar objects of assessment, and reducing uncertainty due to scientifically based rules for constructing trees of evaluation criteria. Some methodological problems of quality assessment and their solutions are considered in [7].
When rating startups as one of the criteria, for example, you can use the “project cost”, and of the approaches to evaluating you can use cost estimation methods. Such methods have been successfully used, for example, by professional appraisers and investment advisers (but most often at later stages of development - Early stage, Expansion stage, Later stage). At the early stage of development (Seed stage), the application of valuation methods for rating purposes is not effective for two main reasons: 1) due to the high costs of using the methods; 2) due to the high degree of uncertainty of the initial data of the model (in forecasts of market development, cash flows, in the assessment of supply / demand, risks). Compared to valuation methods,
When conducting national ratings of startups (as well as selecting appropriate experts), it is proposed to use qualimetric techniques that allow startups to be ranked by quality (or integral quality), while ensuring transparency of measurement procedures, the validity of the selection of criteria for comparison and the comparability of startups' performance [8, 9 ]. The use of qualimetry in evaluating (ranking) startups not only minimizes the costs of rating organizers when summarizing, but also allows them to be comprehensively assessed, because qualimetry has an apparatus with which any economic and non-economic in nature objects and processes can be quantified and thereby both quality assessment criteria and performance evaluation criteria can be simultaneously considered2 startups. At the same time, assessment methods based on qualimetry can be customizable taking into account the strategic goals of the “stakeholders” with a given accuracy and allow not only measuring individual quantitative and qualitative parameters of a startup, but also comparing their performance indicators, for example, by integrated quality.
All startups differ from each other in areas of activity, and stages of development, and results, and costs to achieve their goals. Qualimetric analysis (especially at the Seed stage) allows you to take into account all the criteria and performance indicators of startups (the nature of which is not subject to any restrictions), bring all indicators together, convert heterogeneous indicators to homogeneous, ensure reliability, the necessary accuracy, convolution of indicators and, most importantly , comparability of results.
In order to answer the question which of the methods is most preferable when rating startups by quality, we turn to the papers [10, 11]. Based on a study of the literature on the theory of economic efficiency, the logic of estimates, operations research and decision theory 314 requirements were formulated, according to which 16 different quality assessment methods were analyzed. It was shown in [10] that the qualimetric method for assessing quality is the most acceptable of all currently used methods for this purpose. Of course, the superiority of the method of qualimetry over the others cannot be understood as superiority over all methods for all requirements. The qualimetric method is better than others only in the aggregate of all 14 requirements, and for some requirements taken separately, it can be inferior to other methods. For example, at the request of “non-laboriousness”, of course, it is inferior to the method of “assessment by one expert” [11, Table 1, “B”]. However, due to automation, even the laboriousness of the qualimetric method for assessing quality can be significantly reduced up to its comparability with other methods.
Thus, it is proposed to develop a qualimetric methodology for rating startups, in which the indicator “integral quality” will be used as a criterion for ranking startups at an early stage of development, and “quality” when selecting experts (or taking into account their expert assessments).
The qualimetric approach to quantifying quality is published in open sources, for example, in [2, 3, 7-10, 12] and is used in various spheres of human life.
- 1 “rating” - rating, order, classification. Rating - a numerical or ordinal indicator that reflects the importance or significance of a particular object or phenomenon.
2 In qualimetry, the term “effectiveness” is similar to the term “integral quality”
3only those requirements were taken into account, the validity of which is confirmed by the sufficiently high frequency of their mention in the literature. Therefore, claims that are mentioned very rarely or poorly substantiated in the literature were not used
The main terms used in qualimetry in assessing the quality of startups:
Qualimetry is a scientific discipline that studies the methodology and problems of a comprehensive quantitative assessment of the quality of any objects (processes, products or services).
Quality - a set of properties of an object that characterizes the results generated by it (both positive and negative), but which do not include the cost of resources for its creation and consumption.
Profitability - such a property of the object that characterizes the total (full) costs of the evaluated object, i.e. include the costs of its production, consumption and disposal.
Integral quality is such a property of an object that characterizes the totality of its quality and profitability.
Startup quality- this is a comprehensive description of the processes for creating and launching a new product (device, work, service) on the market, satisfying the needs of interested parties and representing the totality of the results obtained, but which do not include indicators of resources (for example, cash) used in these processes.
If, however, the expended resources are also taken into account, then the characteristic studied in this way is called the “integrated quality of a startup,” which is synonymous with “startup efficiency.”
Qualimetric analysis is a research method that allows you to compare and evaluate the quality of objects in relation to the world level.
Startup Rankings- a comprehensive assessment and subsequent ranking of young Russian technology companies according to their degree of development, in order to identify their investment attractiveness, as well as to describe and structure the market for such companies. Source - www.rg.ru/2012/12/09/reyting-site.html
Authors :
Kostin A.V., Ph.D., Head of the Department for Evaluation of Innovative Projects of Center for Technology Transfer CJSC, kostin @ rusttc. ru.
G. Azgaldov, Doctor of Economics, Chief Researcher, Central Economics and Mathematics Institute of the Russian Academy of Sciences, [email protected]
Literature
1. Kostin A.V. The birth of qualimetry // Standards and quality. - M., 2010, No. 2. - S.90-91. URL: www.labrate.ru/kostin/2010-02_nucleation_qualimetry.pdf (accessed 06.06.2013).
2. Azgaldov G.G. Development of the theoretical foundations of qualimetry: Dis. for a job. learned. Doctor of Economics / Military engineer Acad. them. V.V. Kuibyshev. - M., 1981. URL: www.labrate.ru/azgaldov/azgaldov-qualimetry-referat-1981.doc (accessed: 06.06.2013).
3. Qualimetry: initial information. Reference manual with an example for ANO 'Agency for Strategic Initiatives to Promote New Projects': Textbook. allowance / G.G. Azgaldov, A.V. Kostin, V.V. Gardens. - M .: Higher. school, 2011 .-- 143 p. URL: www.labrate.ru/kostin/051326.pdf (accessed: 06.06.2013).
4. Dmitry Chernyak. Methods for evaluating a startup [Electronic resource]. URL: habrahabr.ru/company/findstartup/blog/144521 (accessed 06.06.2013).
5. Dmitry Chernyak. Methods for evaluating a startup. Continuation of [Electronic resource]. URL: habrahabr.ru/company/findstartup/blog/144221 (accessed 06.06.2013).
6. Startups received a rating / Rossiyskaya Gazeta, December 9, 2012 [Electronic resource]. URL: www.rg.ru/2012/12/09/reyting-site.html (accessed: 06.06.2013).
7. Garry G. Azgaldov, Alexander V. Kostin. Applied Qualimetry: Its origins, errors and misconceptions // Benchmarking: An International Journal, Volume 18, Number 3, 2011, pp. 428-444. - www.labrate.ru/articles/428-444_applied_qulimetry_azgaldov-kostin.pdf(Date of treatment: 06/06/2013).
8. Azgaldov GG, Kostin AV (2012) Increasing the Validity of Results of a National / International Competition: A Case Study / Europe Middle East Africa Members' Meeting, Barcelona (Spain), January 26-28, 2012. - www.labrate. ru / kostin / 054548.pdf (accessed date: 06/06/2013).
9. Qualimetry for all: Textbook. allowance / G.G. Azgaldov, A.V. Kostin, V.V. Gardens. - M.: InformZnanie, 2012 .-- 165 s. - www.labrate.ru/kostin/064571_qualimetry_azgaldov-kostin-sadovov-2012.pdf (accessed: 06.06.2013).
10. Azgaldov G.G. Qualimetry in architectural and construction design - M.: Stroyizdat, 1989. - 264s. www.labrate.ru/azgaldov/qualimetry_in_architectural_and_structural_design.pdf(Date of treatment: 06/06/2013).
11. Theses Kostina A.V. to the round table discussion “How and why to measure the quality of education at a university?” (November 20, 2012, CEMI RAS) // Library LABRATE.RU [Electronic resource]. URL: www.labrate.ru/20121120/063804.doc (accessed: 06.06.2013).
12. Azgaldov G.G. Theory and practice of assessing the quality of goods. Fundamentals of qualimetry .. - M .: Economics, 1982. - 256s. - www.labrate.ru/azgaldov/azgaldov_theory_and_practice_of_quality-assessment.pdf (accessed: 06.06.2013).
13. On the rating of “30 most promising startup teams for Runet and IT” // Electronic resource - pruffi.ru/30startups/about (accessed: 06.06.2013)
