Forrester Research: A Comparison of Ten Top Software Composition Analysis Vendors

In April 2019, Forrester Research released the Forrester Wave ™ quarterly report: Software Composition Analysis, Q2 2019, with an assessment and comparison of ten Software Composition Analysis (SCA) solution providers. This article is an adapted translation of the key provisions of the report.

Key Findings

• WhiteSource and Synopsys topped the list of market leaders (Leaders);
• Snyk and Sonatype are called Strong Performers;
• WhiteHat Security , Flexera, and Veracode are named Contenders;
• GitLab , FOSSA, and JFrog are Challengers.

Translator's Note

What is Software Composition Analysis? The Software Composition Analysis process is a subspecies of the broader Component Analysis process of the Cyber ​​Supply Chain Risk Management framework ( see the NIST website ) and is designed to identify potential risks of using third-party and open source software.

Software solutions such as Software Composition Analysis (SCA), respectively, are designed to automatically detect risks and eliminate vulnerabilities in the code, control the use of external (ready-made) libraries.

For a general idea of ​​Component Analysis and Software Composition Analysis in particular, we recommend that you familiarize yourself with the material on the OWASP project website .

The importance of Software Composition Analysis (SCA) class solutions in implementing modern approaches to creating a secure application development environment

In conditions when it is necessary to quickly add new functionality to applications while maintaining the quality of the code, developers are increasingly resorting to using already created software modules located in public repositories, as well as customizing and automating their code, relying on common open source components. One of the latest ^source studies showed that from year to year the number of open source components in the code base analyzed in the study increases by 21%.
It is important to understand that these third-party open source components may contain critical vulnerabilities, which in itself already poses a risk to the business. A recent study showed that every eighth open source component contains a known ^source security vulnerability. All this is compounded by the fact that in recent years in information security professionals less time on the identification and removal of newly discovered vulnerabilities, as the time between the publication of the vulnerability and its exploitation has decreased from 45 to 3 days ^source .

Given these trends, consumers of SCA solutions should take a closer look at vendor solutions that:

• Provide recommendations to developers on how to fix vulnerabilities.
  To significantly reduce the risk posed by vulnerabilities and licensing conditions, developers need to be notified as soon as possible through the SDLC (software delivery life cycle) processes about security risks and the use of someone else’s code in terms of license cleanliness, and it is also necessary to receive recommendations on their elimination. SCA-solutions should not only give high-quality recommendations for eliminating risks, some products also offer solutions for correcting the code, providing information about the availability of a safe version of the open source component or creating patches (patches) if there are no fixed versions.
• Allows you to create policies that are consistent with different business units and application types.
  To increase the speed of release releases, IS specialists should switch from manual testing using SCA tools to the development and implementation of safe development standards in an organization using SCA functionality. Information security experts establish secure development policies that all applications in the company must satisfy (for example, none of the known critical vulnerabilities should be released) and control the minimum security requirements for more critical applications. To be effective, an information security specialist needs flexible management of adopted policies, including using tools provided by SCA.
• Provide strategic risk reports for security professionals and CISO.
  CISOs should organize the process in such a way as to reduce the amount of time spent on fixing the identified vulnerabilities. In the past, information security experts collected information about risks and vulnerabilities from various sources or simply did nothing. Today, an information security specialist can receive ready-made reports (out of the box) for CISO and development teams that describe the identified risks.

Criteria for including vendors in the review

Forrester included 10 vendors in this benchmarking analysis: Flexera, FOSSA, GitLab, JFrog, Snyk, Sonatype, Synopsys, Veracode, WhiteHat Security and WhiteSource.

Each of these vendors has:

1. A full-fledged SCA-tool of the enterprise class. All of the listed vendors offer a number of SCA capabilities suitable for information security professionals. Participating manufacturers were required to offer most of the following functionality “out of the box”:
   
   
   • the ability to provide recommendations on eliminating licensing risks and vulnerabilities in open source software (open source),
   • ability to integrate into SDLC automation tools,
   • Provide proactive vulnerability management
   • the ability to edit and create policies,
   • the ability to provide clear risk reports in open source software.
2. Annual revenue of over $ 10 million from SCA offers alone.
3. The presence of interest or recommendations from Forrester customers. In other cases, according to the decision of Forrester, the participating vendor could guarantee its inclusion in the list due to the technical characteristics of its products and significant presence on the market.

The Forrester Wave ™ report highlights Leaders, Strong Performers, Contenders, and Challengers. The report is an assessment of the leading vendors in the market and does not represent a complete landscape of vendors. More information about the market can be obtained from other Forrester reports on the SCA ^link .

Forrester Wave Methodology

Forrester Wave reports are a guide for buyers considering software acquisition options in the technology market. To offer an equal process for all comparison participants, Forrester follows the Forrester Wave ™ Methodology Guide when evaluating participating vendors.

The report conducts an initial study to form a list of evaluated manufacturers. The initial pool of vendors is narrowed based on the selection criteria and is included in the final list. Then, detailed information about products and strategies is collected through questionnaires, demonstrations / briefings, reviews / interviews with clients. This input, along with the expert’s research and his knowledge of the market, is used to evaluate vendors by a rating system in which each participant is compared with others.
Forrester clearly indicates the publication date (quarter and year) of Forrester Wave in the title of each report. The vendors participating in this Forrester Wave study were evaluated based on materials provided by Forrester until 01/28/2019, no additional materials were accepted after this time.

Before publishing the report, Forrester asks vendors to study the results of the study to evaluate the accuracy of the study. Manufacturers marked as non-participating in the Forrester Wave diagram meet the criteria and could be included in the report, but refused to participate or only partially participated in the study. They are evaluated in accordance with The Forrester Wave ™ And The Forrester New Wave ™ Nonparticipating And Incomplete Participation Vendor Policy, their position is published along with the position of participating vendors.

Comparison Overview

1. Current Vendor Offer
   
   Each vendor’s position on the vertical axis of the Forrester Wave chart indicates the attractiveness of the current offer. The key criteria for such solutions are license risk management, vulnerability determination actions, proactive vulnerability management, policy management, SDLC integration, container scanning and serverless scanning, as well as built-in strategic reporting capabilities.
2. Strategy
   
   Location on the horizontal axis indicates the stability of the vendor's strategy. Forrester evaluated the product strategy, market approach, roadmap, and access to training for users and distributors.
3. Presence on the market
   
   The assessment of the presence of vendors in the market is represented by the size of the circle on the diagram and reflects the volume of product installations for each vendor, the growth rate and profitability of the company.

Translator's note: the following table summarizes the Forrester expert ratings and weights, based on which the compared SCA providers are located on the Forrester “waves”. We could not find a detailed interpretation of the estimated indicators, however, it is important to note the following features of the methodology:

1. Forrester weights were set in accordance with their significance for subscribers (clients) of this analytical agency, therefore, for those companies that are primarily interested in the specific functionality of the SCA solution, the rating of vendors may differ significantly from that presented.
2. The indicator of presence on the market has a coefficient equal to zero, and is only informative in nature, which means that the final numbers of this indicator affect only the size of the vendor’s point (circle), and not its vertical and horizontal position, i.e. Do not affect the rating and getting into the leaders.

Vendor Profiles

The analysis revealed the following strengths and weaknesses of each vendor.

Leaders

WhiteSource reduces the time needed to eliminate risks through prioritization. WhiteSource recently introduced the ability to prioritize vulnerabilities by performing a static scan to see if a vulnerable part of a component is invoked directly by an application. If not, then the priority of the vulnerability is reduced. Another relatively recent possibility is the automatic elimination of vulnerabilities through the creation of requests (pull request) for updating a component to a version that complies with company policy.

Users appreciate the support of a large number of languages ​​and the quality of customer support, but note that the product could better visualize transitive dependencies. The WhiteSource solution has very few drawbacks, but the functionality for compiling a list of components used in the project (bill of materials or BOM) does not live up to expectations. WhiteSource is well-suited for companies that need to scan at the earliest stages of the SDLC and are looking for a solution to prioritize and automatically eliminate threats.

Synopsys capitalizes on Black Duck through binary scanning and reporting.Synopsys combined the functionality of its solution with the functionality of Black Duck (Black Duck Hub and Protecode SC). However, for a more comprehensive analysis of compliance with licensing requirements, users can also use Black Duck Protex and Black Duck. For example, you can analyze the difference between the claimed discovered licenses, but you will need to use two different tools for this. Synopsys also has another functionality for interactive and static scanning - the company recently released the Polaris platform, designed to combine all the data from the pre-release scanning tools.

Users trust Synopsys code scanning products, which are fast and reliable, provide detailed recommendations for eliminating threats, but consider the level of false positive results to be high. Synopsys offers opportunities for good policy management, integration with SDLC solutions, reliable proactive vulnerability management, including the function of comparing project specifications (BOM), where component changes are highlighted. However, Synopsys does not live up to expectations when it comes to the ability to automatically fix vulnerabilities offered by other leading vendors. Synopsys is well suited for companies whose application development teams have clear SDLC integration requirements and need different policies for different types of applications.

Strong Performers

Snyk focuses on developer scripts to update versions and provide fixes. The company's goal is to allow developers to fix vulnerabilities and, as a result, not only offers the possibility of fixing them through creating update requests, but also allows working with custom fixes when there is no acceptable version of the component in the repository. Snyk also provides developers with a visualization of requests in the form of a graph that displays the relationships and associated vulnerabilities, which helps developers understand why certain corrections are needed.

Users appreciated Snyk’s focus on developers ’needs, including simple SDLC integration, automatic vulnerability fixes, including custom patching with no manual code fixes, and dependency visualization. However, in order for the solution to fully meet the needs of developers, it is necessary that Snyk should not only detect vulnerabilities in the program code, but also provide information about all versions of the used software components of the program code. Snyk pays much attention to the needs of developers and less to the needs of information security experts, therefore, as a recommendation, it should be noted that it is necessary to expand the functionality of Snyk's built-in audit and reporting capabilities. Snyk is great for companies

Sonatype continues to develop the Nexus platform to improve performance. Sonatype uses the Nexus repository to enrich its vulnerability database. The Sonatype research team further supplements the data related to new vulnerabilities discovered, information on how to resolve them, and recommendations for changing settings, updating components, and recommendations for changing program code. The Nexus platform has several licenses for various functionalities - DepShield, IQ Server, Nexus Auditor, Nexus Firewall, Nexus Lifecycle, Nexus Repository and Nexus Vulnerability Scanner.To get the most out of using Nexus platform products, you need to choose the right combination of these components.

Users of this solution note a low level of false positives, integration with the Nexus repository and a high level of user support. However, it is also emphasized that Sonatype solutions need to pay more attention to the refinement of the functionality for tracking transitive dependencies, as well as the provision of reporting forms based on scan results. In addition, pricing and licensing policies cause difficulties for product users when choosing the right solution from a wide variety of company products.

This solution will be of interest to companies that already use Nexus products, or that require a very low level of false-positive scan results.

Contenders

WhiteHat Security offers a SCA solution without manual intervention to achieve maximum speed. WhiteHat Security is known for the fact that their experts, to reduce the level of false-positive scan results, analyze the scan results before sending a verdict to the user. WhiteHat is now able to offer a fully automated solution using Sentinel SCA Essentials in addition to the WhiteHat Sentinel SCA Standard, which still has security team verification results.

WhiteHat Sentinel Essentials is a new solution, so users may experience minor flaws. Weaknesses in the role-based access system, difficulties in identifying vulnerable dependencies are noted, but the details of vulnerability information are highly appreciated. WhiteHat Security continues to actively work to improve its product, positioned as a universal scanner (SCA, SAST, DAST).

WhiteHat Security is best suited for companies whose developers vary in maturity level: some require speed and they have enough experience to independently use the features offered by the product, others need additional help by manually viewing the results of the response by the security team on the vendor side.

Flexera is distinguished from its competitors by safety research. Secunia, a Flexera Research Group, is conducting an early exploration of new vulnerabilities by providing Flexera users with information about the vulnerability and how to deal with it before it is officially recorded in the National Vulnerability Database (NVD). The success of this group is measured on the basis of accuracy and response time, all vulnerability reports were received in NVD and published in it. Vulnerabilities that Secunia detects are displayed in Flexera FlexNet Code Insight and are identified as Secunia.
Although Flexera users acknowledge the flexibility of the user interface and useful automation features. At the same time, they also report that the documentation and training on how to use the system are uninformative, and they need to be supplemented by the level of technical support service, as well as system support during implementation, and the implemented API has limited functionality.

Veracode provides separate use of its two products.In 2018, Veracode acquired SourceClear to complement its Veracode Software Composition Analysis solution. SourceClear Software Composition Analysis is an agent-based scanning tool, and Veracode Software Composition Analysis is a cloud-based SaaS solution. The functionality of these products has not yet been combined — for example, SCA can be performed using the SaaS solution — but not using the scanning agent. Veracode also offers static and dynamic pre-release scanning as an addition to its SCA products.

Until Veracode fully combines SourceClear and Software Composition Analysis, users will experience inconvenience when purchasing and renewing product licenses, with delays in functional updates and different levels of API support. Nevertheless, the vendor continues to work on the integration of its solutions, and there is an expectation that Veracode will work closely to eliminate the remaining differences between the products, especially joint language support and a common policy engine.

Veracode is best suited for companies that already use Veracode products and want to limit the number of security providers. Current Veracode users will appreciate the promise of a true secure application development platform in which SCA data complements other scan data from Veracode products.

Challengers (Challengers)

GitLab quickly launched its solution to the market, but focused on solving the problems of developers, paying less attention to the problems of information security specialists.GitLab has been offering security products since 2017, and now the functionality of static and dynamic code analysis, in addition to binary SCA, has been added to the functionality. However, some developer-centric SCA features will be inconvenient for security professionals. For example, the ignore feature allows developers to ignore vulnerabilities of any severity level. This forces IS specialists to carefully monitor which vulnerabilities the developers decided to ignore. In addition, GitLab tends not to slow down the build with quality control during the release phase. Instead, he recommends using the viewing option, which forces security professionals to manually view the status of each assembly.

GitLab has actively and quickly implemented and continues to develop security features. However, many of these opportunities are still in their infancy or at the planning stage. When evaluating the solution, users emphasized the lack of support for a large number of languages, unsustainable vulnerability detection, and basic policy management capabilities.

It’s worth paying attention to GitLab when most, if not the entire development team, use it as a GitLab open source repository.

FOSSA enhances the product with open source identification and proprietary licenses.The FOSSA Compliance can automatically detect a raw copyright header, as well as distinguish between private code, third-party code and copyrighted external code. In addition to this discovery technology, FOSSA claims to be working with legal advisers with expertise in open source licensing. FOSSA is an open source solution that allows users to improve product performance and add support for new languages ​​and platforms.

Professionals working with this tool like the vulnerability assessment offered by FOSSA at build time, and they feel that the results of vulnerability scanning can be trusted. Some FOSSA users publish scan results and list FOSSA as the source. FOSSA itself is an open source project, so it's best to consider it as a set of tools in which, and this is confirmed by users, meager documentation and advanced functionality, such as container scanning. A detailed long-term roadmap is difficult to implement for an open source project, as FOSSA cannot foresee when external project participants will create new functionality.

FOSSA should be considered if you have the desire and ability to invest resources in the development and customization of the SCA product to the specific requirements of the company.

JFrog is limited to scanning binary code located in its Artifactory repository. Using JFrog XRay, you can determine with a high degree of detail what is scanned inside the binary code, using the ability to select objects for observation (watches), and then correlate policies with what needs to be scanned. Policies can be applied to all binary code and to assemblies from various JFrog repositories.

Users rated JFrog on average and noted the lack of functional flexibility, the intuitive interface and the limitations in reporting as disadvantages. The following were noted as pluses: good integration with JFrog Artifactory, fast development speed of new functions and fixes for detected problems.

JFrog XRay should be considered when JFrog Artifactory is already in use and automatic elimination of threats is required.

Note by the translator.
In conclusion, I would like to note the difference between the views of Forrester and Gartner on the market for source code security analysis tools: while Forrester in his research considers the SAST and DAST functionality a useful extension of SCA products, then Gartner in its research Magic Quadrant for Application Security Testing considers solutions on this market, on the other hand, considers SCA functionality important, but optional for SAST and DAST.
What do you think about this? Do you believe that the SCA market will remain a separate direction (and, for example, competition will develop in the direction of increasing the number of supported integrations with third-party systems), or do you think that buying niche SCA market players will be richer * AST solutions manufacturers?

Translators:
Vyacheslav Vovkogon, Cross Technologies JSC
Daria Oreshkina, Web Control