Conference DEFCON 16. Fedor, hacker InSecure.org. NMAP Internet Scan

Original author: Fedor
  • Transfer
Good afternoon, my name is Fedor, I am from InSecure.org and I am the author of the Nmap security scanner project , released in 1997. I would like to thank everyone who came here and Defcon himself for inviting me here. I am a big fan of such conferences, where people who have a real passion for amateur technologies, but who do not have companies willing to pay thousands of dollars for a ticket, can come.

I want to warn you that this conversation is not about how to conduct cross-site scripting attacks on social networks or how to hack Twitter channels. It's about port scanning and more than port scanning, and if you don't like port scanning, then the next 50 minutes will be your worst nightmare.



Because for me, talking about something else would be as unusual as hearing Dan Kaminsky, who does not talk about DNS . I mean, I can rush into detecting some OSs or acting as a Nmap script engine just like Dan can take his DNS and use it to tunnel YouTube to fool some dummy, but in both cases we just expand our key topic.

My topic, as can be seen from the first slide, is devoted to scanning the Internet. I spent a lot of time this summer scanning tens of millions of hosts on the Internet and collecting the necessary data.

And when I tell people about it, they often ask: “Why”? For me, scanning is valuable in and of itself, but if you need some reason, then in this case I had some specific goals for the project. The first is to collect empirical data and use them to extend and improve the functionality of Nmap , we will talk about this a little later.

The second goal is to use data to help knowledgeable people scan more efficiently. There are many people collecting information about the structuring and population of networks to find out which scanning methods are best suited. But usually they wonder how to set up existing scanning tools without finding anything suitable. So you can use my empirical data for your own needs, they will help you learn how to scan and collect the results.



The third goal is to detect and eliminate Nmap errors and improve performance for large-scale scanning. Because scanning of millions of hosts is performed in a variety of situations, and you can track how Nmapreacts to this. I managed to correct errors that caused the system to crash or “hang”, speed up the process, finding out what exactly is causing the scan to slow down.

The fourth goal is to demonstrate technologies that can be used for both conventional scanning and large-scale scanning of the Internet.

The idea is that if this scan works for 25 million hosts, then it will certainly cope with scanning 25 thousand hosts or how many you need to scan.
Let's look at the problems that I had to face when scanning. First of all, I want to say that it is better to conduct dozens of large, but targeted scans instead of one huge scan. Each of these goals allows you to collect useful data of a certain kind, but the problem is to determine the specific IP addresses that need to be scanned.

There are many possibilities for this: BGP routing tables that let you know which networks are routed, DNS zone files, register allocation tables, etc.



I decided to use Nmap's own IP address script generator., allowing you to create 25 million IP-addresses, and another 200 thousand I did because of possible duplicates. At the same time, not computers are actually scanned, but a list of them, because I scan the machines themselves later. As a result, I received a list of 25 million IP addresses that I used to scan. It was a way to generate random numbers, but as soon as you have specific targets that need to be scanned, the following question arises: what source are you going to use.

Here I had many ideas, one crazier than the other, for example, P2P scanning. I was going to distribute a client called Nmapster , which people could download. He would scan their computers and report that we are collecting data for generalization. But I decided that the key goal is to do it.Nmap is faster and more efficient for everyday scans, and it’s better to focus on using Nmap itself instead of creating custom programs for it.

The second big obstacle was the illegality of such actions. When you scan a lot of hosts, people may get indignant, but I don’t want my Internet provider to deny me services again, but the possibility of being arrested looks even worse. Therefore, I decided to go online through my neighbor's free wireless access point (applause and laughter in the hall).

No, I'm just kidding, and besides, my neighbor's Internet channel would not have enough bandwidth to handle the amount of traffic I needed.

Therefore, I decided to use the ISP, which I use for colocation, and perform a scan from there, that is, through the equipment located on its territory. Fifteen minutes after the scan began, these guys contacted me, frantically asking what the hell was going on, maybe my computer is infected with the worst worm on the Internet because it goes crazy and probes thousands of machines per second all over the Internet! They offered to turn me off, but I explained that they did not worry, I did not suffer, I did it for a specific purpose. Then they thought that I was some kind of spammer or even worse, if that was possible, and I should be arrested. Then I got scared and thought that I’m finished, I’ll have to quit my project and start exploring cross-site vulnerabilities. Fortunately, it turned out that they were usersNmap and I managed to explain that this scan is needed in order to make my program more efficient and productive. To which they said: "Well, then you can continue!"

I slowed down the scanning speed a bit because I didn’t want their fuses to blow, and continued this great thing. Unfortunately, the US Department of Defense was not so kind, they didn’t like my scanning at all, and they told me to stop scanning important military targets. I thought that Nmap's ability to skip scanning certain networks would be useful for me , but they would not even let me identify these networks, because they were also military targets. So since then I’m a bit nervous if military planes fly over my head.



The next obstacle to scanning became firewalls. For some cases, it was enough for me to get results from the unprotected segment of the Internet, but for others it would be nice to study how it looks behind the firewalls of companies. You know that they often control different ports, so from the inside, an open network looks completely different. I am pleased to say that I was able to break through a series of firewalls, not with the help of something like an advanced fragmentation attack, but with the help of technologies that simply “asked” them to provide the necessary data.

There are many large companies that daily scan their networks using Nmap , and they were happy to contribute in the form of data that will help improve it.

The next problems were performance and accuracy. Scanning the Internet is a long and hard work. The key goal was to improve the performance of Nmap, because its usage statistics were disappointing:

  • Time has passed since the beginning of the scan 93 h 57 min. 40 s (more than 4 days);
  • during this time, out of 254868 scheduled hosts, a UDP scan of 65,000 ports on 2048 hosts was performed;
  • during this time, a UDP scan of about 11.34% of the planned number was performed, the time for scanning the remaining hosts will be 688 h 41 min 48 s.

When your estimate of the remaining time leads to an "overflow of integers" and brings one negative, this is not a very encouraging sign. This particular scan is still in progress, and it is possible that at the DEFCON conference next year I will tell you what came of it. Fortunately, some of our other scans ended much earlier, so this is a kind of introduction to what types of scans we performed and why.

Now let's get to more practical tips that can serve as useful details and let you know how to use this as an aid for your own scanning. Detecting hosts is a good start, because the first thing you want to do when scanning a network is to discover the location of the host and find out which networks are available for scanning. Therefore, you should not spend time scanning the IP addresses that are not tapped by the hosts, and decide which methods to perform the scan.

There was a time when most hosts responded to ICMP echo requests or ping packets, which Nmap defaulted to port 80 by default, but unfortunately, this time ended 10 years ago.



Now most companies are blocking ping packages and I had to use more efficient methods.

The first was a TCP- based method for locating hosts using synthetic SYN packets (-PS) , which work best against static filtering firewalls that use a constant list of exceptions and using ACK (-PA) packets that are effective against dynamic filtering firewalls.

Those firewalls that pass synthetic packets usually block ACK packets , and vice versa.

I’ll show a quick example of using Nmap with a synthetic package for port 80 on sun.com.. As you can see, the answer came very quickly and it shows that the host is available.



Then we use ACK packets for the same host , you see that the scan took longer and the host did not respond because it blocked pinging.



Thus, you can understand which firewall is installed on this host - with static or dynamic filtering. In this case, we have a static firewall that missed SYN packets and blocked ACK packets .

It may be asked what kind of port sounding is better to use. I will answer - you need to send both types of packets until the host misses one of them.

The next question is which port to use. You have 65,000 ports, and it is often impossible to choose the ones that work best. Therefore, I made a selection of ports based on empirical data, which is shown on the next slide. This is the best 10-ka ports to detect certain types of hosts.



Some people may look at this list and ask, where are the most common Windows ports 135 or 139?

However, remember that I created my database based on well-protected hosts, so if you have problems installing a firewall, you'd better block these Windows ports . I would recommend part of these ports to “probe” with SYN-packets , another part - with ACK packets..

Next we have the discovery of UDP hosts . In this case, closed UDP ports are better open because they are more likely to respond. Open ports, getting a clean packet, do not know what to do with it and just ignore it. Closed ports usually respond to this packet with a ban, and this means that the host is located here. Port 53 often falls into firewall exceptions because it is intended for DNS .

I also used the ICMP host discovery method (PE, PM, PP) . Some systems readily accept echo requests, but they block others. For example, Google.com adminsthey do not see any threat in receiving ping packets, but they block requests for network masks and time stamps, because they believe that hackers use them.

Other administrators explicitly block echo requests as hacking, but forget about blocking network mask requests and the netmask / timestamp time stamp . Therefore, I advise you to use both types of requests - an echo request and one of the second type requests.

We also have a new feature called Protocol Ping (PO) . By default, this protocol uses 3 types of IP packets: 1 ( ICMP ), 2 ( IGMP ) and 4 ( IPinIP) in order to obtain an available response if the host is “alive”. I did not test to find out which protocols are most useful, so all 3 are used by default.

Now I want to talk about the various host detection technologies that I used, and you can ask which of these heaps are the most valuable and what the difference is between them. The slide shows an example when I generated 50 thousand IP addresses and used ping scanning by default, as a result of which 3348 hosts were found in 1600 seconds, or 27 minutes.



This is quite a lot of computers, and this scan looks pretty successful. But then I decided to use “advanced” scanning, took almost the same list of 50 thousand hosts and added several different detection techniques, such as echo reply, time stamps,SYN probes ports heaps and set source port 53 to disguise itself as DNS . This allowed us to detect 4473 "live" host, but it took a little longer - 4259 seconds.



A comparison of these two scanning methods — with the default and advanced settings — showed that the second took 167% more time (41 minutes versus 27) and found 34% more hosts.

Now I want to talk about Nmap upgrades , which were conducted on the basis of these studies.

The site http://nmap.org/changelog.html contains the latest version of Nmap 4.68 , in which bugs were fixed and performance improved. There is also the latest SVN-release, which can be found at http://nmap.org/book/install.html#instsvn .

I'm tired of error messages, which says that “yes, we fixed it in 2003,” because there are a lot of people who don’t want to perform frequent updates. Then they complain that Nmap has long been outdated, because it tells you which port numbers are open, but does not tell which services are behind them. Recently, I have added a number of improvements to the program, and if you need the most advanced version of the program - the BHD CO 8 Black Hat DefCon release , you can find it at the special link svn: //svn.insecure.org/nmapexp/bhdc08using guest login and password. This newest version contains the Top Ports scanning feature and some other features that I’ll cover later.

So, the Top Ports project is another big scan of millions of IP addresses to detect open TCP and UDP ports using a sample of popular ports. Some large organizations also provided the ability to scan internal network data located behind firewalls. At the same time, the service Nmap files were supplemented with frequency data for each port.

Consider the ports that were scanned by default. Nmap 4.68 scanned 1715 TCP portsand 1488 UDP ports , ports from 1 to 1024 and all named higher ports were scanned. I note that many ports were invented many years ago and have not been used for a long time. At the same time, there are many open, nameless ports.



The final version of Nmap used the frequency data of the Top 1000 ports for each protocol, so in many cases I got better results. This made it possible not to waste time on scanning old unused ports, thus scanning proceeds faster, because in fact a little more than half of all existing ports are scanned.

Quick scan is very different from the default scan. In Nmap, it is denoted by the letter ( -F ). This timeNmap 4.68 scanned 1276 TCP ports and 1017 UDP ports over the entire range of numbers. You may ask why the default scan has processed more ports, but there is nothing dramatic here — a quick scan just took less time. However, in this case, the scanner did not check all existing ports, but worked only with those whose names were in the new Nmap service file . As a result, the scan was done for the TOP 100 ports for each protocol, and I got a speed gain an order of magnitude more than with the default scan. This is much more useful for TCP and even more useful for UDP , because I know that many people never perform UDP-scanning, considering it very time consuming and difficult with respect to filtering open ports. Therefore, they pretend that UDP simply does not exist. However, hackers do not pretend, so it is very important for us to understand what is happening with this protocol.
The next slide shows examples of fast scanning using different versions of Nmap . The letter U in the designation of the scan mode Nmap - sUV - F-T4 indicates a UDP port, F is a fast scan. There is a filtering problem in this open version because Nmap does not receive a response and cannot determine whether the port is filtered or open. This happens if you use the version from the sitescanme.nmap.org . So it's great that I managed to get a report with data that showed all open and filtered ports.



Thus, scanning using Nmap 4.68 took an hour, scanning with the release version of Blackhat DefCon 08 lasted 6 and a half minutes, scanning with the “advanced” version of Blackhat DefCon 08 was only 13 seconds. All 3 versions found the same open port 53.

The new versions used new scanning features:
- top-ports <n>

She scanned the most popular open ports with each required protocol, and the function is:
–port-ratio <n>

where n was chosen between 0 and 1, scanned all ports with a frequency set at this level.

The latest version worked with the “ —version — intensity 0 ” flag . This meant that UDP- requests were sent to certain ports using certain protocols, that is, only DNS was tried for 53 , only SNMP for 161 ports , etc. This reduced the scanning time to 13 seconds.

The moral of this story is that if you know what data you really need, you can optimize the scan a bit and make it much faster. In this case, we received the same data, but instead of the hourly wait, we waited only 13 seconds. So setting up the functions helps a lot, especially the top ports feature. You just need to choose between the default scan of 1000 ports and the fast scan of 100 ports and specify arbitrarily how many ports you want to scan.

This raises the question of which Top Ports feature works best. Next, the slide shows the Top-10 TCP ports and a sample of the effectiveness of the “tops”, which I did on the basis of empirical data. This sample shows how many open ports I found using this or that “top”.

If you use the fastest scanning "dozens" of ports, you can find a little more than half of the open TCP ports. Using the "Top 100" ports, you can detect 73% of open ports, using the default "Top 1000" detects 93%, but at the same time you scan only 2% of the 65 thousandth port space.



Next, I will give the statistics of finding open UDP ports. It is dominated by typical Microsoft ports , although SNMP and NTP are also available .



The effectiveness of UDP differs from the efficiency of TCP ; here a larger percentage of open ports is determined with smaller values ​​of the “top”. So, "Top 100" UDPallows you to detect 90% of ports against 73% for TCP .

I used another function that I developed recently, this is the speed control of Packet Rate Control packages .



In her regard, I have mixed feelings. I am proud of Nmap’s download control and other technologies to figure out which scan speeds work best. But there are a lot of people who say that they would just like to scan ports with a certain frequency and not worry if any packets are lost. This is one of the reasons why they use scanners like RAND or Unicorn.. For such cases, I decided to add a function that sets the maximum and minimum frequency of sending packets per second.

It turned out to be very useful when I almost “melted down the fuses” of my Internet host, because I was able to set the scan parameter for Nmap at 300 packets per second and make these guys from ISP a little happier.

The next slide shows an example of how to put together all the useful functions of Nmap .



Here you can enter the source IP address that I wanted to use for this particular scan and indicate the debug mode. I often used this function to interact with the program during scanning, because people do not know that you can pressD and open the debug mode, and if you click on it several times, you can scroll the screen and see what Nmap is doing at the moment.

Here I used low latency because I didn’t want to wait long for hosts for which the packet frequency was limited. I applied the log file creation function using the time value function STRF , which automatically inserts the date and time.

Next, I entered the name of the file that I want to receive in the end and noted that I do not want to do more than 1 retry scan, because I am going to perform a large amount of scanning as soon as possible.

Further, randomization of hosts is noted here, the port numbers are scanned, a large group of host MAC addresses is listed, the minimum scanning frequency is 175 packets per second and the maximum 300 packets per second.

This is an example of how I constantly changed and improved the algorithm of Nmap , until I found an option that worked better than others.

And now I want to use the remaining time to tell you the latest Nmap news . Some of them are new and interesting features, for example, the NSE scripting engine . This is a modular mechanism that polls ports in a certain way. In this case, we make HTML headings for the websites that it finds.



There are now over 50 scripts supported by Nmap that are compatible with Whois data , brute-forcing pop3 passwords and other crazy things you want to use.

Now I will quickly show you how the Nmap Scripting Engine works .



This is a long command, so I will copy it from another place. She stipulates Nmap detailed mode - do not ping, perform UDP probing for port 53, use aggressive timing and use 3 scripts here, which will help to avoid a DNS error, which Dan talked about last Sunday. One script simply checks if the DNS server resolves recursion, the second checks whether it randomizes the original port numbers, and the third checks if it has a randomized transaction ID. These are the mistakes that people want to fix to reduce the cache poisoning problem.

In our case, I want to demonstrate the operation of NSE during the port scan of one of the BlackHat servers and one of the Shmoo.com servers . It takes a little time, and then the result is displayed right next to the port number.

You can see that for blackhat.com with IP address 216.231.64.55 with UDPscanning port 53 in both cases, a server recursion failure was received, so it was not possible to “interrogate” it further. For the server archimedes.shmoo.com with the IP address 12.21.210.234 recursion took place. But I am pleased to report that it was great in terms of port randomization.

I wanted to show one of the few examples when my idea failed miserably, so that one could play the game “who can poison the cache first”, but decided that this was not a good idea for our presentation. In addition, I have more interesting material on the topic of concern to me - this is a new graphical interface Zenmap GUI .



Many people say that they do not need a graphical interface, because they work with Nmapalready 10 years and know all his 130 options to the bone. However, this interface is a powerful tool for working with the scanner, which I will now quickly demonstrate.

It shows a window with output data, every open host, you can filter the display of HTTP or SSH, and so on. In addition, we added an interesting feature to the newest version of the scanner. For those people who say: “since you called this tool H-map , give me the hell this map!”, We added a graphic diagram of the topology of the scanned network.



This is a fairly accurate thing that draws the result of a network scan and places it in the center of the source node, and in concentric circles around this center each jump into the network and the computers you scanned are shown. You can take such a node and say: “Hey, well, show me what else there is data on this particular scan, show me open ports, etc.,” that is, when you hover over the yellow circle, a window opens with all the details scanned You can scan new machines and they will also be added to the scheme. You can also center the layout relative to another host — just click on the node and drag it in the desired direction.



Another big plus of the GUI is that users can now work with Nmap .Windows , which have no idea how to manage the program using the command line. I already received letters from them in which it was written: “I double-clicked on the nmap.exe icon on the desktop, after which a black box popped up for a second, and then everything disappeared. Obviously, your program is not working at all. ” So I hope that Zenmap GU I will help them. Although, on the other hand, maybe such people should not use Nmap at all .

We also have a second generation OS detection system.



It embodied everything that I had learned about detecting the OS over the previous 7 years, and contains 1,500 signatures of new operating systems. So Nmap users will be able to find on our website.nmap.org is any tool you can imagine and download a version of a program for Windows or Linux . We also have a tool for discovering software versions.



There is also a function called Reason . This is a kind of filter that shows for what reason a port was opened - due to the use of synthetic or ACK packets, or for what reason the port was closed, for example, due to a reboot.



This is a very good tool for understanding what Nmap actually does . The next function that can provide you with a lot of useful information is called “packet tracing”, or Packet trace..



Thanks to her, I can find out if port 25 is the end host returning packets to me, or if they are thrown back by the firewall. After reviewing the packet trace during a quick Nmap scan , I can tell which device they are interacting with. Here you can see the frequency, the scan options used, the IP ID, and with this feature I can find out if the same host sent me packets in both cases or not. This is useful for understanding the firewalls and filtering systems used.

Another feature, Advanced Trace Routing, Advanced Traceroute shows which type of sensing is best used for penetration. This quick feature because Nmapcan perform it in parallel scan.



Next, I made several improvements in scan performance and accuracy.



There is a whole section on the main page of the program interface, which shows all the options that you can use to work. Next we have the option of TCP and IP headers , which allows you to specify the source route or the source entry.



Many of you will say that the original routes were needed 15 years ago and now nobody needs it. But I talked to one guy who recently tested a local network in one company. It was located in something like a conference room and could only connect to servers of a specific series, so it was not possible to establish contact with all the computers of the company. They had created something like a DMZ to access the conference room. Then this guy took one of their servers and lost the original route through it, found the destination computer, and thus was able to get around this restriction.

We have developed another brand new tool called Ncat .



This is a modern interpretation of Netcat.the venerable Hobbit. It virtually supports all the features of Netcat 1.10 , except for the basic port scanner, because I prefer to use my Nmap as a scanner. It also supports many new cool features: SSL, IPv6 on different platforms running Linux, Windows, Mac OS , network mediation, port forwarding, proxy (client, server, chain) to scan through another computer, shell execution, access control etc. This program was created by Chris Cutterjohn in 2005 and is now fully ready for release. Chris also added some features like IP options and ping detection mode.

The next tool is Ndiff., the appearance of which many have been waiting for.



This utility compares the results of two or more scans and shows changes: new or remote hosts, ports, modified services, etc. It is great for quickly detecting changes when rescanning. For example, every day I scan a network of some company, and at the end of the work I call this utility and say: “send me the changes that have occurred since yesterday”. So I can see new open ports, added computers and so on.

Finally, I want to show you my book Network Scanning with Nmap, I have been working on it for several years. It talks not only about the capabilities of my network scanner, but also about network scanning techniques.



I printed 170 copies of this book that I want to distribute here, but I am afraid that they will be sold the next morning. The book will also be sold on Amazon and will be available for free download at denman.org/book . You can add yourself to the Nmap hacker list and receive news about my program.

The next slide lists those who contributed to the improvement of the Nmap program , starting with version 4.50, which was released 9 months ago. I could not do anything without their help. Our project is very much obliged to volunteers who joined him and helped him to become better.

And now I still have time to answer a couple of questions.

Indeed, I do not feel well in Germany and the United Kingdom and in other countries that have passed laws that prohibit people from using tools like Nmap . Yes, they can be used by both good and bad people, but it is better to improve our networks, rather than act by analogy with those who prohibit the use of hammers, because they can kill a person. Personally, I like to speak in Germany and England, so such a ban can be a potential problem for me, because I do not want to be arrested for violating the law.



I could not even explain to the judge what my motivation is, because I don’t know German at all. Therefore, I am glad that some associations of people are fighting against this law.

If you have more questions, I will be glad to answer them in room 103, directly opposite the hall. Thank you all for your attention!




Thank you for staying with us. Do you like our articles? Want to see more interesting materials? Support us by placing an order or recommending to friends, 30% discount for Habr's users on a unique analogue of the entry-level servers that we invented for you: The whole truth about VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps from $ 20 or how to share the server? (Options are available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).

VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps until December for free if you pay for a period of six months, you can order here .

Dell R730xd 2 times cheaper? Only we have 2 x Intel Dodeca-Core Xeon E5-2650v4 128GB DDR4 6x480GB SSD 1Gbps 100 TV from $ 249in the Netherlands and the USA! Read about How to build an infrastructure building. class c using servers Dell R730xd E5-2650 v4 worth 9000 euros for a penny?

Also popular now: