October 31, 2007 at 23:05Windows Safari 3 Performance Analysis
- Transfer
The other day, bbsod published an article about independent testing of the Safari browser . I think we need to dwell on this in more detail, for example, Google services such as Gmail and Google Maps in Safari 3 do not work as fast as in FireFox and IE, well, more on that below. So in this article we will focus on methods for testing the speed of web browsers using the Safari example .
Measuring Apple’s Real Browser Performance
On June 11, Apple released its version of its OSX browser, Safari 3.0, for Windows, claiming it to be "the fastest Windows browser." This statement was made based on tests conducted by Apple, based on the iBench benchmark.made by Zif Davis, who shares the results of testing HTML performance, JavaScript and application startup time. There are countless benchmarks to test performance, and we decided to check whether Apple's statement is true.
Test preparation
Other tests run Safari 3.0 Windows beta through benchmarks and verify how they work under strictly defined test conditions. We were more interested in checking how this browser works directly when working with ordinary users. That is, we measured how Safari works when working with web pages, making for this several tests that differ from the usual benchmarks for the browser.
What web pages?
When testing Safari 3.0 only on Gmail and Google Maps, it turned out that Safari 3.0 is slower than Firefox 2 and IE7 under Windows, but testing based on only these two sites cannot fully show browser capabilities for all users. Of course, what should be checked on more websites, but on which specifically? The problem was solved by selecting for testing sixteen English-language sites that are on the first lines of the Alexa ratingon 08/29/07. This allowed testing sites with a completely different design, ranging from simple DHTML pages. This approach allowed us to take into account the maximum possible options for web page designs. A larger number of tested web pages, of course, would improve accuracy, but sixteen is a good number for accurate and easy testing, plus all the most popular sites are covered at the same time.
Network Features
The main problem when testing the browser is the bandwidth of the channel. It can vary widely, and it is impossible to determine exactly when the speed is low due to server load, and when the problem is, in fact, in the browser. This problem can be somehow solved if measurements are taken at different times during the day, but it also does not provide perfect guarantees of accuracy. The server may be slightly loaded during testing of one browser, and then suddenly the load on the server will suddenly increase when another is tested.
The problem was resolved by separate sites on the Internet, as well as "canned" sites. Testing for sites on the Internet was done several times and then the average value was displayed.
The "canned" statistics was made as follows: the site was completely stored on the local server, and then the statistics were measured already for the local network. To make the bandwidth of the channel close to real, all browsers worked on the channel at a speed of 5 Mbps. In addition, we had two separate measurement techniques, which avoids the error. In other words, the difference in browser performance should be the same in both test groups.
Note that we do not declare that this technique is better than some other. It has its drawbacks, and it should be taken only as one of the possible points of view when evaluating browser performance.
Measurement technique
Performance measurement when working with web pages was carried out by our own software Web Performance Analyzer Pro , which measures HTTP traffic. Some testing programs run into problems when using javascript to calculate page load time, but this trouble was eliminated by our approach. The reason why our measurement technique is noticeably more accurate is that to increase the speed of displaying pages, the browser renders the page as soon as possible, and at the same time pipelines HTTP with image requests. So far, by the time the last image is loaded, the web page will already be fully displayed except for the last image, the loading time of which can be neglected.
This technique has disadvantages. For example, AJAX pages may not display correctly. Fortunately, the program groups HTTP traffic and therefore it is quite simple to determine the statistics associated with the page.
Caching
Two scenarios are possible when determining the loading speed of a web page. In the first, the case when the user accesses the page for the first time, in the second, when when opening the page most of the information is already cached. Both cases were tested separately in order to determine the difference in performance.
Realistic use
In order for the performance measurements to be real, the sites open as if you are already a user of the resource. That is, the pages with the registration of a new user have not been tested. For example, when testing Wikipedia, only pages with information were tested, and not the default page, which allows the user to select a region.
Settings
The operating system was Windows XP Professional SP 2, an IIS 5.0 server with HTTP connection and compression enabled. Browsers tested: Firefox 2.0.0.6, Safari 3.0.3 and Internet Explorer 7.0.5730
Limitations
Limitations for this test were AJAX sites like GMail, as it has already been said, and the participation of such sites in the test can increase the test error. So this test rather measures the average browser performance when working with HTML, but it can not serve as a good indicator for sites of the GMail class.
Another limitation in the test was that we tested sites whose hosting was located in the USA. This is due to the fact that if the site is hosted in Asia, then there is a high probability of random differences in server performance, which leads to inaccurate data from online tests.
And finally, the technique when the site is completely transferred to the local server works fine only for simple sites. Sites with a more complex structure are quite difficult to correctly transfer to the local server. However, this does not contradict other features of the test. Testing
data
Testing sites on the local server
The sites listed above were hosted on the local IIS server. Measurements were taken both for the first visit and for pages with cached content. It immediately became clear that no browser would be the best in all tests. One will be faster on one page, one on another.
Download times were averaged, which made it possible to understand the overall performance picture across all sites. The graph above shows that Safari 3 worked best, then Internet Explorer 7, then Firefox 2. Browser performance on average when loading pages with cached content was higher, but the places were distributed the same way - Safari beta was the fastest, then IE7, then the fastest slow - Firefox 2.
Measurements on-line
Selected web pages were downloaded again, but this time directly from the original servers. This time, the relationship between browser performance remained unchanged, but overall performance increased due to the wider bandwidth of the channel.
If the tests with the local network were accurate, then the on-line testing will be compared with the previous results, and thus all conclusions will be drawn. The graph below shows the difference between the download speed on the local network and the Internet. Here you can see the relationship between browser performance based on the first page load. Safari 3.0 won, followed by IE7 and Firefox 2 in third place. When checking the performance for cached pages, the places were distributed differently: a draw between Firefox and IE, but Safari got a bit ahead of them.
Analysis
Which browser is faster?
In all cases, the Safari 3.0 beta browser for Windows proved to be the fastest. The difference between the average page load time for all tests is shown in this graph:
The download speed of a remote web page ranges from 0.2 seconds for a cached page to 1.4 when the page loads for the first time. The difference of 0.2 seconds between Safari beta and Firefox 2 with IE7 is insignificant, and shows that for frequently visited pages we won’t see serious gains in page loading speed.
A completely different result for pages being downloaded for the first time. Safari shows results 1.1 and 1.4 seconds better when loading a page on the Internet. The importance of such a difference in download speeds is very individual, but for high-speed Internet these seconds will most likely be very noticeable.
For pages loaded from local hosting, Safari 3.0 also showed better results, but the difference was less noticeable. So the performance of all browsers when working with LAN was approximately the same, which is pretty expected.
Logic of tests
The point of repeating tests with a local network was to exclude the possibility that the results of the online test were affected by a violation of the network or server congestion. The table below shows that the difference in the time of loading Safari over its competitors remained unchanged both when working with the local network and when testing on the Internet.
Evaluation of applications for Apple performance
company Apple announcedliterally the following: “Safari loads pages twice as fast as Internet Explorer 7 and 1.6 times faster than Firefox 2.” Taking into account what Apple said about the maximum possible numbers, we believe that they only talked about the first page load, lowering the work with cached sites.
Apple said Safari was 1.6 times faster than Firefox and showed graphs where the loading time for Safari was 2.14 seconds and for Firefox 2 3.67 seconds. The difference in these figures is 1.7 seconds, which almost exactly matches the results of our testing.
Regarding the comparison with IE7, Apple said that their browser is 2 times faster, and showed a graph where the ratio was about 2.2. This is slightly more than the coefficient 1.4, which we calculated, but still, really, Safari is noticeably faster.
In any case, the reality of Apple's statements will depend on whether you want to operate on specific numbers, or the basic concept - that Safari is faster than Firefox 2 and IE7. We really confirmed the veracity of these statements if the page is loading for the first time, and questioned them if the page is already in the cache. However, it should be understood that the performance of the browser in each case will depend on what page you visit.
Want to keep up to date with all the news from the Safrari browser? Subscribe to our RSS
Measuring Apple’s Real Browser Performance
On June 11, Apple released its version of its OSX browser, Safari 3.0, for Windows, claiming it to be "the fastest Windows browser." This statement was made based on tests conducted by Apple, based on the iBench benchmark.made by Zif Davis, who shares the results of testing HTML performance, JavaScript and application startup time. There are countless benchmarks to test performance, and we decided to check whether Apple's statement is true.
Test preparation
Other tests run Safari 3.0 Windows beta through benchmarks and verify how they work under strictly defined test conditions. We were more interested in checking how this browser works directly when working with ordinary users. That is, we measured how Safari works when working with web pages, making for this several tests that differ from the usual benchmarks for the browser.
What web pages?
When testing Safari 3.0 only on Gmail and Google Maps, it turned out that Safari 3.0 is slower than Firefox 2 and IE7 under Windows, but testing based on only these two sites cannot fully show browser capabilities for all users. Of course, what should be checked on more websites, but on which specifically? The problem was solved by selecting for testing sixteen English-language sites that are on the first lines of the Alexa ratingon 08/29/07. This allowed testing sites with a completely different design, ranging from simple DHTML pages. This approach allowed us to take into account the maximum possible options for web page designs. A larger number of tested web pages, of course, would improve accuracy, but sixteen is a good number for accurate and easy testing, plus all the most popular sites are covered at the same time.
yahoo.com
msn.com
google.com
youtube.com
live.com
myspace.com
orkut.com
wikipedia.org
facebook.com
hi5.com
rapidshare.com
blogger.com
medaupload.com
microsoft.com
friendster.com
fotolog.net
Network Features
The main problem when testing the browser is the bandwidth of the channel. It can vary widely, and it is impossible to determine exactly when the speed is low due to server load, and when the problem is, in fact, in the browser. This problem can be somehow solved if measurements are taken at different times during the day, but it also does not provide perfect guarantees of accuracy. The server may be slightly loaded during testing of one browser, and then suddenly the load on the server will suddenly increase when another is tested.
The problem was resolved by separate sites on the Internet, as well as "canned" sites. Testing for sites on the Internet was done several times and then the average value was displayed.
The "canned" statistics was made as follows: the site was completely stored on the local server, and then the statistics were measured already for the local network. To make the bandwidth of the channel close to real, all browsers worked on the channel at a speed of 5 Mbps. In addition, we had two separate measurement techniques, which avoids the error. In other words, the difference in browser performance should be the same in both test groups.
Note that we do not declare that this technique is better than some other. It has its drawbacks, and it should be taken only as one of the possible points of view when evaluating browser performance.
Measurement technique
Performance measurement when working with web pages was carried out by our own software Web Performance Analyzer Pro , which measures HTTP traffic. Some testing programs run into problems when using javascript to calculate page load time, but this trouble was eliminated by our approach. The reason why our measurement technique is noticeably more accurate is that to increase the speed of displaying pages, the browser renders the page as soon as possible, and at the same time pipelines HTTP with image requests. So far, by the time the last image is loaded, the web page will already be fully displayed except for the last image, the loading time of which can be neglected.
This technique has disadvantages. For example, AJAX pages may not display correctly. Fortunately, the program groups HTTP traffic and therefore it is quite simple to determine the statistics associated with the page.
Caching
Two scenarios are possible when determining the loading speed of a web page. In the first, the case when the user accesses the page for the first time, in the second, when when opening the page most of the information is already cached. Both cases were tested separately in order to determine the difference in performance.
Realistic use
In order for the performance measurements to be real, the sites open as if you are already a user of the resource. That is, the pages with the registration of a new user have not been tested. For example, when testing Wikipedia, only pages with information were tested, and not the default page, which allows the user to select a region.
Settings
The operating system was Windows XP Professional SP 2, an IIS 5.0 server with HTTP connection and compression enabled. Browsers tested: Firefox 2.0.0.6, Safari 3.0.3 and Internet Explorer 7.0.5730
Limitations
Limitations for this test were AJAX sites like GMail, as it has already been said, and the participation of such sites in the test can increase the test error. So this test rather measures the average browser performance when working with HTML, but it can not serve as a good indicator for sites of the GMail class.
Another limitation in the test was that we tested sites whose hosting was located in the USA. This is due to the fact that if the site is hosted in Asia, then there is a high probability of random differences in server performance, which leads to inaccurate data from online tests.
And finally, the technique when the site is completely transferred to the local server works fine only for simple sites. Sites with a more complex structure are quite difficult to correctly transfer to the local server. However, this does not contradict other features of the test. Testing
data
Testing sites on the local server
The sites listed above were hosted on the local IIS server. Measurements were taken both for the first visit and for pages with cached content. It immediately became clear that no browser would be the best in all tests. One will be faster on one page, one on another.
Download times were averaged, which made it possible to understand the overall performance picture across all sites. The graph above shows that Safari 3 worked best, then Internet Explorer 7, then Firefox 2. Browser performance on average when loading pages with cached content was higher, but the places were distributed the same way - Safari beta was the fastest, then IE7, then the fastest slow - Firefox 2.
Measurements on-line
Selected web pages were downloaded again, but this time directly from the original servers. This time, the relationship between browser performance remained unchanged, but overall performance increased due to the wider bandwidth of the channel.
If the tests with the local network were accurate, then the on-line testing will be compared with the previous results, and thus all conclusions will be drawn. The graph below shows the difference between the download speed on the local network and the Internet. Here you can see the relationship between browser performance based on the first page load. Safari 3.0 won, followed by IE7 and Firefox 2 in third place. When checking the performance for cached pages, the places were distributed differently: a draw between Firefox and IE, but Safari got a bit ahead of them.
Analysis
Which browser is faster?
In all cases, the Safari 3.0 beta browser for Windows proved to be the fastest. The difference between the average page load time for all tests is shown in this graph:
The download speed of a remote web page ranges from 0.2 seconds for a cached page to 1.4 when the page loads for the first time. The difference of 0.2 seconds between Safari beta and Firefox 2 with IE7 is insignificant, and shows that for frequently visited pages we won’t see serious gains in page loading speed.
A completely different result for pages being downloaded for the first time. Safari shows results 1.1 and 1.4 seconds better when loading a page on the Internet. The importance of such a difference in download speeds is very individual, but for high-speed Internet these seconds will most likely be very noticeable.
For pages loaded from local hosting, Safari 3.0 also showed better results, but the difference was less noticeable. So the performance of all browsers when working with LAN was approximately the same, which is pretty expected.
Logic of tests
The point of repeating tests with a local network was to exclude the possibility that the results of the online test were affected by a violation of the network or server congestion. The table below shows that the difference in the time of loading Safari over its competitors remained unchanged both when working with the local network and when testing on the Internet.
Evaluation of applications for Apple performance
company Apple announcedliterally the following: “Safari loads pages twice as fast as Internet Explorer 7 and 1.6 times faster than Firefox 2.” Taking into account what Apple said about the maximum possible numbers, we believe that they only talked about the first page load, lowering the work with cached sites.
Apple said Safari was 1.6 times faster than Firefox and showed graphs where the loading time for Safari was 2.14 seconds and for Firefox 2 3.67 seconds. The difference in these figures is 1.7 seconds, which almost exactly matches the results of our testing.
Regarding the comparison with IE7, Apple said that their browser is 2 times faster, and showed a graph where the ratio was about 2.2. This is slightly more than the coefficient 1.4, which we calculated, but still, really, Safari is noticeably faster.
In any case, the reality of Apple's statements will depend on whether you want to operate on specific numbers, or the basic concept - that Safari is faster than Firefox 2 and IE7. We really confirmed the veracity of these statements if the page is loading for the first time, and questioned them if the page is already in the cache. However, it should be understood that the performance of the browser in each case will depend on what page you visit.
Want to keep up to date with all the news from the Safrari browser? Subscribe to our RSS