Web tools, or where to start a pentester? / Digital Security Blog

We continue to talk about useful tools for pentester. In a new article, we will consider tools for analyzing the security of web applications.

Our colleague BeLove already made a similar selection about seven years ago. It is interesting to see which tools have maintained and strengthened their positions, and which have receded into the background and are rarely used now.


Note that Burp Suite also applies here, but there will be a separate publication about him and his useful plugins.

Content:

• Amass
• Altdns
• aquatone
• MassDNS
• nsec3map
• Acunetix
• Dirsearch
• wfuzz
• ffuf
• gobuster
• Arjun
• Linkfinder
• Jsparser
• sqlmap
• NoSQLMap
• oxml_xxe
• tplmap
• Cewl
• Weakpass
• AEM_hacker
• Joomscan
• Wpscan

Amass

Amass is a Go tool for finding and enumerating DNS subdomains and mapping an external network. Amass is an OWASP project designed to show how organizations on the Internet look like an outside observer. Amass receives subdomain names in various ways; the tool uses both recursive enumeration of subdomains and open source search.

To detect interconnected network segments and autonomous system numbers, Amass uses the IP addresses obtained during operation. All information found is used to build a network map.

Pros:

• Information gathering techniques include:
  * DNS - enumeration of subdomains in a dictionary, bruteforce subdomains, “smart” enumeration using mutations based on found subdomains, reverse DNS queries and search for DNS servers where it is possible to request a zone transfer ( AXFR);
  
  * Search for open sources - Ask, Baidu, Bing, CommonCrawl, DNSDB, DNSDumpster, DNSTable, Dogpile, Exalead, FindSubdomains, Google, IPv4Info, Netcraft, PTRArchive, Riddler, SiteDossier, ThreatCrowd, VirusTotal, Yahoo;
  
  * Search databases of TLS certificates - Censys, CertDB, CertSpotter, Crtsh, Entrust;
  
  * Using search engine APIs - BinaryEdge, BufferOver, CIRCL, HackerTarget, PassiveTotal, Robtex, SecurityTrails, Shodan, Twitter, Umbrella, URLScan;
  
  * Search the web archives of the Internet: ArchiveIt, ArchiveToday, Arquivo, LoCArchive, OpenUKArchive, UKGovArchive, Wayback;
• Integration with Maltego;
• Provides the most complete coverage of the task of finding DNS subdomains.

Minuses:

• More accurately with amass.netdomains - it will try to access every IP address in the identified infrastructure and get domain names from reverse DNS queries and TLS certificates. This is a “loud” technique; it can reveal your intelligence actions in the organization under study.
• High memory consumption, can consume up to 2 GB of RAM in different settings, which will not allow to run this tool in the cloud on a cheap VDS.

Altdns

Altdns is a Python tool for building dictionaries for enumerating DNS subdomains. It allows you to generate many variants of subdomains using mutations and permutations. To do this, we use words that are often found in subdomains (for example: test, dev, staging), all mutations and permutations are applied to already known subdomains that can be input to Altdns. The output is a list of variations of subdomains that may exist, and this list can later be used with DNS brute force.

Pros:

• Works well with large datasets.

aquatone

aquatone - was previously better known as another tool for finding subdomains, but the author himself abandoned this in favor of the aforementioned Amass. Aquatone has now been rewritten on Go and is more targeted for preliminary exploration of websites. To do this, aquatone goes through the specified domains and looks for websites on them at different ports, after which it collects all the information about the site and takes a screenshot. Convenient for quick preliminary intelligence on websites, after which you can select priority targets for attacks.

Pros:

• At the output, it creates a group of files and folders that are convenient to use when further working with other tools:
  * HTML report with collected screenshots and response headers grouped by similarity;
  
  * A file with all the URLs on which websites were found;
  
  * File with statistics and page data;
  
  * A folder with files containing response headers from the found goals;
  
  * A folder with files containing the body of the response from the found goals;
  
  * Screenshots of found websites;
• Supports work with XML reports from Nmap and Masscan;
• Uses headless Chrome / Chromium to render screenshots.

Minuses:

• It may attract the attention of intrusion detection systems, so it requires configuration.

The screenshot was taken for one of the old versions of aquatone (v0.5.0), in which the search for DNS subdomains was implemented. Old versions can be found on the releases page .

Screenshot aquatone v0.5.0

MassDNS

MassDNS is another tool for finding DNS subdomains. Its main difference is that it makes DNS queries directly to many different DNS resolvers and does it at a considerable speed.

Pros:

• Fast - able to resolve more than 350 thousand names per second.

Minuses:

• MassDNS can cause a significant load on the used DNS resolvers, which can lead to a ban on these servers or complaints to your provider. In addition, it will cause a large load on the company's DNS servers if they have them and if they are responsible for the domains that you are trying to resolve.
• The list of resolvers is outdated now, however, if you select broken DNS resolvers and add new ones, everything will be fine.

nsec3map

nsec3map is a Python tool for getting a complete list of domains protected by DNSSEC.

Pros:

• It quickly detects hosts in DNS zones with a minimum number of queries if DNSSEC support is enabled in the zone;
• It includes a plugin for John the Ripper, which can be used to crack the received NSEC3 hashes.

Minuses:

• Many DNS errors are not handled correctly;
• There is no automatic parallelization of processing NSEC records - you have to split the namespace manually;
• High memory consumption.

Acunetix

Acunetix is a web vulnerability scanner that automates the process of checking the security of web applications. Tests the application for SQL injection, XSS, XXE, SSRF and many other web vulnerabilities. However, like any other scanner of many web vulnerabilities, it does not replace the pentester, because complex chains of vulnerabilities or vulnerabilities in the logic can not be found. But it covers a lot of different vulnerabilities, including various CVEs, which the pentester could forget about, therefore it is very convenient for exemption from routine checks.

Pros:

• Low false positives
• Results can be exported as reports;
• Carries out a large number of checks for various vulnerabilities;
• Parallel scanning of multiple hosts.

Minuses:

• There is no deduplication algorithm (Acunetix considers the pages to be the same in functionality because they have different URLs), but the developers are working on it;
• It requires installation on a separate web server, which complicates testing client systems with a VPN connection and using a scanner in an isolated segment of a local client network;
• The service under investigation may “make a noise”, for example, send too many attack vectors into the form of communication on the site, thereby greatly complicating business processes;
• It is a proprietary and, accordingly, non-free solution.

Dirsearch

Dirsearch is a Python tool for brute force directories and files on websites.

Pros:

• It can distinguish real “200 OK” pages from “200 OK” pages, but with the text “page not found”;
• Поставляется вместе с удобным словарем, имеющим хороший баланс между размером и эффективностью поиска. Содержит стандартные пути, характерные для многих CMS и стеков технологий;
• Свой формат словаря, который позволяет достичь хорошей эффективности и гибкости перебора файлов и директорий;
• Удобный вывод — простой текст, JSON;
• Умеет делать throttling — паузу между запросами, что жизненно необходимо для любого слабого сервиса.

Минусы:

• Расширения нужно передавать в виде строки, что неудобно, если нужно передать много расширений сразу;
• Для того, чтобы использовать свой словарь, его нужно будет немного доработать до формата словарей Dirsearch для максимальной эффективности.

wfuzz

wfuzz - Python fuzzer for web applications. Probably one of the most famous web phasers. The principle is simple: wfuzz allows you to phase any place in the HTTP request, which makes it possible to phase GET / POST parameters, HTTP headers, including Cookies and other authentication headers. At the same time, it is convenient for simple bruteforce directories and files, which requires a good dictionary. It also has a flexible filter system with which you can filter responses from a website by various parameters, which allows you to achieve effective results.

Pros:

• Multifunctional - modular structure, assembly takes several minutes;
• Convenient filtering and fuzzing mechanism;
• You can phase any HTTP method, as well as any place in the HTTP request.

Minuses:

• Under development.

ffuf

ffuf - Go's web fuzzer, created in the “image and likeness” of wfuzz, allows you to brute files, directories, URL paths, names and values ​​of GET / POST parameters, HTTP headers, including the Host header for brute force of virtual hosts. Wfuzz differs from its counterpart by higher speed and some new features, for example, Dirsearch format dictionaries are supported.

Pros:

• Filters are similar to wfuzz filters, they allow you to flexibly configure bruteforce;
• Allows fuzzing the values ​​of HTTP headers, POST request data and various parts of the URL, including the names and values ​​of GET parameters;
• You can specify any HTTP method.

Minuses:

• Under development.

gobuster

gobuster - a tool on Go for reconnaissance, has two modes of operation. The first is used to brute force files and directories on the website, the second - to iterate through DNS subdomains. The tool initially does not support recursive enumeration of files and directories, which, of course, saves time, but on the other hand, the bruteforce of each new endpoint on the website needs to be run separately.

Pros:

• High speed of work both for enumerating DNS subdomains, and for brute force files and directories.

Minuses:

• The current version does not support the installation of HTTP headers;
• By default, only some of the HTTP status codes are considered valid (200,204,301,302,307).

Arjun

Arjun is a tool for bruteforce of hidden HTTP parameters in GET / POST parameters, as well as in JSON. The built-in dictionary has 25,980 words that Ajrun checks in almost 30 seconds. The trick is that Ajrun does not check each parameter separately, but checks immediately ~ 1000 parameters at a time and looks to see if the answer has changed. If the answer has changed, then divides this 1000 parameters into two parts and checks which of these parts affects the answer. Thus, using a simple binary search, you can find a parameter or several hidden parameters that influenced the answer and, therefore, can exist.

Pros:

• High speed due to binary search;
• Support for GET / POST parameters, as well as parameters in the form of JSON;

The plugin for Burp Suite, param-miner , which is also very good at finding hidden HTTP parameters, works by a similar principle . We will tell you more about it in an upcoming article about Burp and its plugins.

Linkfinder

LinkFinder is a Python script for finding links in JavaScript files. Useful for finding hidden or forgotten endpoints / URLs in a web application.

Pros:

• Fast;
• There is a special plugin for Chrome based on LinkFinder.

.

Minuses:

• Inconvenient final conclusion;
• Does not parse JavaScript in dynamics;
• Quite a simple logic for finding links - if JavaScript is in any way obfuscated, or if links are initially missing and generated dynamically, then you won’t be able to find anything.

Jsparser

JSParser is a Python script that uses Tornado and JSBeautifier to parse relative URLs from JavaScript files. Very useful for detecting AJAX requests and compiling a list of API methods that the application interacts with. Effectively paired with LinkFinder.

Pros:

• Fast parsing javascript files.

sqlmap

sqlmap is probably one of the most famous tools for analyzing web applications. Sqlmap automates the search and operation of SQL injections, works with several dialects of SQL, has in its arsenal a huge number of different techniques, ranging from the quotation mark “in the forehead” and ending with complex vectors for time-based SQL injections. In addition, it has many further exploitation techniques for various DBMSs, therefore it is useful not only as a scanner for SQL injections, but also as a powerful tool for exploiting already found SQL injections.

Pros:

• A large number of different techniques and vectors;
• Low number of false positives;
• Many possibilities for fine tuning, various techniques, target database, tamper scripts to bypass WAF;
• Ability to dump output data;
• There are many different operating options, for example, for some databases - automatic file upload / download, obtaining the ability to execute commands (RCE) and others;
• Support for direct connection to the database using the data obtained during the attack;
• You can submit a text file with Burp results to the input - no need to manually compose all the command line attributes.

Minuses:

• It is difficult to customize, for example, to write some of your checks because of the scarce documentation for this;
• Without the appropriate settings, it carries out an incomplete set of checks, which can be misleading.

NoSQLMap

NoSQLMap is a Python tool for automating the search and operation of NoSQL injections. It is convenient to use not only in NoSQL databases, but also directly when auditing web applications using NoSQL.

Pros:

• Like sqlmap, it not only finds potential vulnerabilities, but also checks the possibility of its exploitation for MongoDB and CouchDB.

Minuses:

• Does not support NoSQL for Redis, Cassandra, development is ongoing in this direction.

oxml_xxe

oxml_xxe is a tool for embedding XXE XML exploits in various types of files that use the XML format in some form.

Pros:

• It supports many common formats, such as DOCX, ODT, SVG, XML.

Minuses:

• Not fully implemented support for PDF, JPEG, GIF;
• Creates only one file. To solve this problem, you can use the docem tool , which can create a large number of payload files in different places.

The above utilities do a great job of testing XXE when loading documents containing XML. But also do not forget that XML format handlers can be found in many other cases, for example, XML can be used as a data format instead of JSON.

Therefore, we recommend that you pay attention to the following repository, which contains a large number of different payloads: PayloadsAllTheThings .

tplmap

tplmap is a Python tool for automatically detecting and exploiting Server-Side Template Injection vulnerabilities; it has settings and flags similar to sqlmap. It uses several different techniques and vectors, including blind injection, and also has techniques for executing code and loading / unloading arbitrary files. In addition, it has in its arsenal techniques for a dozen different engines for templates and some techniques for finding eval () - similar code injections in Python, Ruby, PHP, JavaScript. In case of successful operation, opens an interactive console.

Pros:

• A large number of different techniques and vectors;
• Supports many engines for rendering templates;
• Many operating techniques.

Cewl

CeWL is a Ruby dictionary generator, designed to extract unique words from a specified website, it follows links on a site to a specified depth. A compiled dictionary of unique words can later be used to brute force passwords on services or brute force files and directories on the same website, or to attack received hashes using hashcat or John the Ripper. Useful when compiling a “targeted” list of potential passwords.

Pros:

• Easy to use.

Minuses:

• You need to be careful with the depth of the search so as not to capture an extra domain.

Weakpass

Weakpass is a service containing many dictionaries with unique passwords. It is extremely useful for various tasks related to hacking passwords, ranging from a simple online brute force account on the target services, ending with an offline brute force received hashes using hashcat or John The Ripper . Composed of about 8 billion passwords with a length of 4 to 25 characters.

Pros:

• It contains both specific dictionaries and dictionaries with the most common passwords - you can choose a specific dictionary for your own needs;
• Dictionaries are updated and updated with new passwords;
• Dictionaries are sorted by performance. You can choose an option for both quick on-line brute-force, and for a thorough selection of passwords from a voluminous dictionary with the latest leaks;
• There is a calculator showing the password brute-time on your equipment.

In a separate group, we would like to place tools for CMS-checks: WPScan, JoomScan and AEM hacker.

AEM_hacker

AEM hacker is a tool for identifying vulnerabilities in Adobe Experience Manager (AEM) applications.

Pros:

• It can identify AEM applications from the list of URLs submitted to it on input;
• It contains scripts for obtaining RCE by loading a JSP shell or operating SSRF.

Joomscan

JoomScan is a Perl tool for automating vulnerability detection when deploying Joomla CMS.

Pros:

• Able to find configuration flaws and problems with administrator settings;
• Enumerates versions of Joomla and related vulnerabilities, similarly for individual components;
• Contains more than 1000 exploits for Joomla components;
• Output of final reports in text and HTML formats.

Wpscan

WPScan - a tool for crawling sites on WordPress, has in its arsenal of vulnerabilities for both the WordPress engine itself and for some plugins.

Pros:

• Able to list not only unsafe WordPress plugins and themes, but also get a list of users and TimThumb files;
• It can conduct brute-force attacks on WordPress sites.

Minuses:

• Without the appropriate settings, it carries out an incomplete set of checks, which can be misleading.

In general, different people prefer different tools for work: they are all good in their own way, and what one person liked may not be suitable for another. If you think that we have undeservedly ignored some good utility - write about it in the comments!