How browsers work - an introduction to web application security
- Transfer
Let's start a series of articles on web application security with an explanation of what browsers do and how they do it. Since most of your customers will interact with your web application through browsers, you need to understand the basics of how these great programs work.
Chrome and lynx
Browser is a rendering engine . His job is to load a web page and present it in a form that is understandable to humans.
Although this is almost a criminal simplification, but for now this is all we need to know at the moment.
Perhaps you are used to working with one of the most popular browsers such as Chrome, Firefox, Edge or Safari, but this does not mean that there are no other browsers in the world.
For example, lynx is a lightweight, command-line text-based browser. At the heart of lynx are the same principles that you will find in any other mainstream browsers. The user enters the web address (URL), the browser downloads the document and displays it - the only difference is that lynx does not use the graphic rendering engine, but the text interface, which makes websites like Google look like this:
We generally have an idea what the browser does, but let's take a closer look at the actions that these ingenious applications perform for us.
In short, the browser’s work consists mainly of
This process helps the browser find out which server it should connect to when the user enters the URL. The browser connects to the DNS server and finds that google.com corresponds to the set of numbers 216.58.207.110 — the IP address to which the browser can connect.
As soon as the browser determines which server will serve our request, it will establish a TCP connection with it and start HTTP exchange . This is nothing more than a way of communicating the browser with the server it needs, and for the server it is a way of responding to browser requests.
HTTP is simply the name of the most popular protocol for communicating over a network, and browsers generally choose HTTP when communicating with servers. HTTP exchange implies that the client (our browser) sends a request , and the server sends a response .
For example, after the browser successfully connects to the server serving google.com , it will send a request that looks like this. Let's sort the request line by line:
After the browser, acting as a client , completes its request, the server will send a response. Here is the answer:
Wow, this time a lot of information to digest. The server informs us that the request was completed successfully ( 200 OK ) and adds several headers to the response , from which, for example, you can find out which server processed our request ( Server: gws ), what is the X-XSS-Protection policy of this response and so further and the like.
Right now you do not need to understand every line in the answer. Later in this series of publications, we will talk more about the HTTP protocol, its headers, etc.
At this point, all you need to know is that the client and the server exchange information and that they do it through the HTTP protocol.
Last but not least is the rendering process. How good is the browser, if the only thing it shows to the user is a list of funny characters?
In the response body , the server includes the presentation of the requested document in accordance with the Content-Type header . In our case, the content type was set to text / html , so we expect the HTML markup in the response - and that is what we find in the body of the document.
This is exactly the moment where the browser really shows its abilities. It reads and analyzes HTML code, loads additional resources included in the markup (for example, JavaScript files or CSS documents can be specified for uploading) and presents them to the user as soon as possible.
Once again, the end result should be what is available for perception of the average Vasya.
If you need a more detailed explanation of what really happens when we press the enter key in the address bar of the browser, I would suggest reading the article “What happens when ...” , a very meticulous attempt to explain the mechanisms underlying this process.
Since this series is dedicated to security, I’m going to give a hint about what we have just learned: attackers can easily make their living from vulnerabilities in terms of HTTP exchange and rendering . Vulnerabilities, malicious users and other fantastic creatures are found in other places, but a more effective approach to providing protection at these levels already allows you to make progress in improving your security state.
The 4 most popular browsers belong to different vendors:
In addition to fighting each other, to increase their market penetration, suppliers also interact with each other to improve web standards, which are a kind of “minimum requirement” for browsers.
W3C is the cornerstone of standards development, but browsers often develop their own functions, which ultimately turn into web standards, and security is no exception.
For example, SameSite cookies were introduced in Chrome 51- a feature that allowed web applications to get rid of a certain type of vulnerability known as CSRF (more on this later). Other vendors decided that this was a good idea and followed suit, which led to the SameSite approach becoming the web standard: Safari is currently the only major browser without support for SameSite cookies .
This tells us two things:
The first point is a shot in Safari (as I said, just kidding!), And the second point is really important. When developing web applications, we need not only to make sure that they look the same in different browsers, but also to provide the same protection for our users across platforms.
Your network security strategy should vary depending on what capabilities the browser vendor offers us. Currently, most browsers support the same feature set and rarely deviate from their overall roadmap, but cases like the one above still happen, and this is something that we must consider when determining our security strategy.
In our case, if we decide that we will neutralize CSRF attacks only with the SameSite cookies, we need to know that we put our Safari users at risk. And our users should know this too.
Last but not least, you must remember that you can decide whether to support the browser version or not: supporting each browser version will be impractical (remember Internet Explorer 6). Despite this, confident support for several recent versions of major browsers is usually a good solution. However, if you do not plan to provide protection on a particular platform, it is highly desirable that your users know about it.
The fact that a regular user accesses our application through the help of third-party client software (the browser) adds another layer that complicates the way to comfortable and safe web browsing: the browser itself can be a source of security vulnerabilities.
Vendors, as a rule, provide rewards (also known as bug-bounty) to security researchers who may look for vulnerabilities in the browser itself. These errors are not related to your web application, but to how the browser itself manages security.
For example, the Chrome rewards programallows security researchers to contact the Chrome security team to report vulnerabilities they have discovered. If the presence of the vulnerability is confirmed, a fix will be issued and, as a rule, a security notice will be published, and the researcher will receive (usually financial) rewards from the program.
Companies such as Google are investing quite substantial capital in their Bug Bounty programs, as this allows companies to attract many researchers, promising them financial gain if they discover any problems with the software under test.
In the Bug Bounty program, everyone wins: the supplier manages to increase the security of their software, and researchers pay for their finds. We will discuss these programs later, as I believe that the initiatives of Bug Bounty deserve a separate section in the landscape of security aspects.
By now, we should have understood a very simple, but quite important concept: browsers are just HTTP clients created for the “average” Internet user.
Browsers are definitely more powerful than a simple HTTP client for any platform (for example, remember that NodeJS has a dependency on 'http'), but in the end they are “just” a product of the natural evolution of simpler HTTP clients.
As for developers, our HTTP client is probably cURL from Daniel Stenberg, one of the most popular programs that web developers use daily. It allows us to perform HTTP exchange on the fly by sending an HTTP request from our command line:
In the example above, we requested a document at localhost: 8080 / , and the local server responded successfully.
Instead of uploading the response body to the command line, we used the -I flag , which tells cURL that we are only interested in the response headers. Taking another step forward, we can give the cURL command to output a little more information, including the actual request that it performs, so that we can better study all this HTTP exchange. The option we should use is: -v ( verbose , more):
Approximately the same information is available in popular browsers through their DevTools.
As we have seen, browsers are nothing more than complex HTTP clients. Of course, they add a huge amount of functions (for example, credentials management, bookmarking, history, etc.), but the truth is that they were born as HTTP clients for people. This is important, because in most cases you do not need a browser to check the security of your web application, when you can just “smoke it” and look at the answer.
And the last thing I would like to point out: everything can be a browser, anything. If you have a mobile application that uses APIs via HTTP, then this application is your browser — it is simply customized for you, which only recognizes a certain type of HTTP response (from your own API).
As we have already mentioned, we are going to highlight the most detailed phases of HTTP exchange and rendering , since they provide the most attack vectors for attackers.
In the next article we will take a closer look at the HTTP protocol and try to understand what measures we need to take to ensure the security of HTTP exchange.
The translation was made with the support of the company EDISON Software , which professionally develops websites for large customers, as well as web development in C # and .NET .
Chrome and lynx
Browser is a rendering engine . His job is to load a web page and present it in a form that is understandable to humans.
Although this is almost a criminal simplification, but for now this is all we need to know at the moment.
- The user enters the address in the browser input line.
- The browser loads the “document” at this URL and displays it.
Perhaps you are used to working with one of the most popular browsers such as Chrome, Firefox, Edge or Safari, but this does not mean that there are no other browsers in the world.
For example, lynx is a lightweight, command-line text-based browser. At the heart of lynx are the same principles that you will find in any other mainstream browsers. The user enters the web address (URL), the browser downloads the document and displays it - the only difference is that lynx does not use the graphic rendering engine, but the text interface, which makes websites like Google look like this:
We generally have an idea what the browser does, but let's take a closer look at the actions that these ingenious applications perform for us.
What does the browser do?
In short, the browser’s work consists mainly of
- DNS resolution
- HTTP exchange
- Rendering
- Reset and repeat
DNS resolution
This process helps the browser find out which server it should connect to when the user enters the URL. The browser connects to the DNS server and finds that google.com corresponds to the set of numbers 216.58.207.110 — the IP address to which the browser can connect.
HTTP exchange
As soon as the browser determines which server will serve our request, it will establish a TCP connection with it and start HTTP exchange . This is nothing more than a way of communicating the browser with the server it needs, and for the server it is a way of responding to browser requests.
HTTP is simply the name of the most popular protocol for communicating over a network, and browsers generally choose HTTP when communicating with servers. HTTP exchange implies that the client (our browser) sends a request , and the server sends a response .
For example, after the browser successfully connects to the server serving google.com , it will send a request that looks like this. Let's sort the request line by line:
GET / HTTP/1.1
Host: google.com
Accept- GET / HTTP / 1.1 : with this first line, the browser asks the server to retrieve the document from the location / , adding then that the rest of the request will occur via the HTTP / 1.1 protocol (or you can also use version 1.0 or 2)
- Host: google.com : this is the only HTTP header required for the HTTP / 1.1 protocol . Since the server can serve several domains (google.com, google.co.uk , etc.), the Client here mentions that the request was for this particular host.
- Accept: * / * : optional header in which the browser informs the server that it will accept any response. The server can have a resource that is available in JSON, XML or HTML formats, so it can choose any format that prefers
After the browser, acting as a client , completes its request, the server will send a response. Here is the answer:
HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=1234; expires=Fri, 18-Jan-2019 18:25:04 GMT; path=/; domain=.google.com; HttpOnly
<!doctype html><html">
...
...
</html>Wow, this time a lot of information to digest. The server informs us that the request was completed successfully ( 200 OK ) and adds several headers to the response , from which, for example, you can find out which server processed our request ( Server: gws ), what is the X-XSS-Protection policy of this response and so further and the like.
Right now you do not need to understand every line in the answer. Later in this series of publications, we will talk more about the HTTP protocol, its headers, etc.
At this point, all you need to know is that the client and the server exchange information and that they do it through the HTTP protocol.
Rendering
Last but not least is the rendering process. How good is the browser, if the only thing it shows to the user is a list of funny characters?
<!doctype html><html">
...
...
</html>In the response body , the server includes the presentation of the requested document in accordance with the Content-Type header . In our case, the content type was set to text / html , so we expect the HTML markup in the response - and that is what we find in the body of the document.
This is exactly the moment where the browser really shows its abilities. It reads and analyzes HTML code, loads additional resources included in the markup (for example, JavaScript files or CSS documents can be specified for uploading) and presents them to the user as soon as possible.
Once again, the end result should be what is available for perception of the average Vasya.
If you need a more detailed explanation of what really happens when we press the enter key in the address bar of the browser, I would suggest reading the article “What happens when ...” , a very meticulous attempt to explain the mechanisms underlying this process.
Since this series is dedicated to security, I’m going to give a hint about what we have just learned: attackers can easily make their living from vulnerabilities in terms of HTTP exchange and rendering . Vulnerabilities, malicious users and other fantastic creatures are found in other places, but a more effective approach to providing protection at these levels already allows you to make progress in improving your security state.
Vendors
The 4 most popular browsers belong to different vendors:
- Google Chrome
- Mozilla Firefox
- Apple Safari
- Microsoft Edge
In addition to fighting each other, to increase their market penetration, suppliers also interact with each other to improve web standards, which are a kind of “minimum requirement” for browsers.
W3C is the cornerstone of standards development, but browsers often develop their own functions, which ultimately turn into web standards, and security is no exception.
For example, SameSite cookies were introduced in Chrome 51- a feature that allowed web applications to get rid of a certain type of vulnerability known as CSRF (more on this later). Other vendors decided that this was a good idea and followed suit, which led to the SameSite approach becoming the web standard: Safari is currently the only major browser without support for SameSite cookies .
This tells us two things:
- Safari doesn't seem to care enough about the safety of its users (just kidding: SameSite cookies will be available in Safari 12, which may have already been released by the time this article is read)
- fixing a vulnerability in a single browser does not mean that all your users are safe
The first point is a shot in Safari (as I said, just kidding!), And the second point is really important. When developing web applications, we need not only to make sure that they look the same in different browsers, but also to provide the same protection for our users across platforms.
Your network security strategy should vary depending on what capabilities the browser vendor offers us. Currently, most browsers support the same feature set and rarely deviate from their overall roadmap, but cases like the one above still happen, and this is something that we must consider when determining our security strategy.
In our case, if we decide that we will neutralize CSRF attacks only with the SameSite cookies, we need to know that we put our Safari users at risk. And our users should know this too.
Last but not least, you must remember that you can decide whether to support the browser version or not: supporting each browser version will be impractical (remember Internet Explorer 6). Despite this, confident support for several recent versions of major browsers is usually a good solution. However, if you do not plan to provide protection on a particular platform, it is highly desirable that your users know about it.
Tip for pros : you should never encourage your users to use outdated browsers or actively support them. Even if you took all the necessary precautions, other web developers did not. Encourage users to use the latest supported version of one of the major browsers.
Vendor or standard bug?
The fact that a regular user accesses our application through the help of third-party client software (the browser) adds another layer that complicates the way to comfortable and safe web browsing: the browser itself can be a source of security vulnerabilities.
Vendors, as a rule, provide rewards (also known as bug-bounty) to security researchers who may look for vulnerabilities in the browser itself. These errors are not related to your web application, but to how the browser itself manages security.
For example, the Chrome rewards programallows security researchers to contact the Chrome security team to report vulnerabilities they have discovered. If the presence of the vulnerability is confirmed, a fix will be issued and, as a rule, a security notice will be published, and the researcher will receive (usually financial) rewards from the program.
Companies such as Google are investing quite substantial capital in their Bug Bounty programs, as this allows companies to attract many researchers, promising them financial gain if they discover any problems with the software under test.
In the Bug Bounty program, everyone wins: the supplier manages to increase the security of their software, and researchers pay for their finds. We will discuss these programs later, as I believe that the initiatives of Bug Bounty deserve a separate section in the landscape of security aspects.
Jake Archibald (Jake Archibald) - Google’s “ lawyer ” developer who discovered a vulnerability affecting several browsers. He documented his efforts to find it, the process of referring to various vendors affected by the vulnerability, and the reaction of vendor representatives in an interesting blog post , which I recommend that you read.
Developer Browser
By now, we should have understood a very simple, but quite important concept: browsers are just HTTP clients created for the “average” Internet user.
Browsers are definitely more powerful than a simple HTTP client for any platform (for example, remember that NodeJS has a dependency on 'http'), but in the end they are “just” a product of the natural evolution of simpler HTTP clients.
As for developers, our HTTP client is probably cURL from Daniel Stenberg, one of the most popular programs that web developers use daily. It allows us to perform HTTP exchange on the fly by sending an HTTP request from our command line:
$ curl -I localhost:8080
HTTP/1.1 200 OK
server: ecstatic-2.2.1
Content-Type: text/html
etag: "23724049-4096-"2018-07-20T11:20:35.526Z""
last-modified: Fri, 20 Jul 2018 11:20:35 GMT
cache-control: max-age=3600
Date: Fri, 20 Jul 2018 11:21:02 GMT
Connection: keep-aliveIn the example above, we requested a document at localhost: 8080 / , and the local server responded successfully.
Instead of uploading the response body to the command line, we used the -I flag , which tells cURL that we are only interested in the response headers. Taking another step forward, we can give the cURL command to output a little more information, including the actual request that it performs, so that we can better study all this HTTP exchange. The option we should use is: -v ( verbose , more):
$ curl -I -v localhost:8080
* Rebuilt URL to: localhost:8080/
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> HEAD / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< server: ecstatic-2.2.1
server: ecstatic-2.2.1
< Content-Type: text/html
Content-Type: text/html
< etag: "23724049-4096-"2018-07-20T11:20:35.526Z""
etag: "23724049-4096-"2018-07-20T11:20:35.526Z""
< last-modified: Fri, 20 Jul 2018 11:20:35 GMT
last-modified: Fri, 20 Jul 2018 11:20:35 GMT
< cache-control: max-age=3600
cache-control: max-age=3600
< Date: Fri, 20 Jul 2018 11:25:55 GMT
Date: Fri, 20 Jul 2018 11:25:55 GMT
< Connection: keep-alive
Connection: keep-alive
<
* Connection #0 to host localhost left intactApproximately the same information is available in popular browsers through their DevTools.
As we have seen, browsers are nothing more than complex HTTP clients. Of course, they add a huge amount of functions (for example, credentials management, bookmarking, history, etc.), but the truth is that they were born as HTTP clients for people. This is important, because in most cases you do not need a browser to check the security of your web application, when you can just “smoke it” and look at the answer.
And the last thing I would like to point out: everything can be a browser, anything. If you have a mobile application that uses APIs via HTTP, then this application is your browser — it is simply customized for you, which only recognizes a certain type of HTTP response (from your own API).
HTTP plunge
As we have already mentioned, we are going to highlight the most detailed phases of HTTP exchange and rendering , since they provide the most attack vectors for attackers.
In the next article we will take a closer look at the HTTP protocol and try to understand what measures we need to take to ensure the security of HTTP exchange.
The translation was made with the support of the company EDISON Software , which professionally develops websites for large customers, as well as web development in C # and .NET .