How we pierced the Great Chinese Firewall (part 3)

Hello!
Any good stories end. And our story about how we came up with a solution to the rapid passage of the Chinese Firewall is no exception. Therefore, I hasten to share with you the last, final part on this topic.

In the previous part, we talked about a lot of test benches invented by us, and what results they gave. And we settled on the fact that it would be nice to add a CDN! for viscosity into our circuit.

I will tell you how we tested Alibaba Cloud CDN, Tencent Cloud CDN and Akamai, and what I ended up with. And of course, to summarize.

Alibaba Cloud CDN

We are hosted on Alibaba Cloud, we use IPSEC and CEN from them. It would be logical to try their solutions first.

Alibaba Cloud has two types of product that may suit us: CDN and DCDN . The first option is a classic CDN for a specific domain (subdomain). The second option stands for Dynamic Route for CDN (I call it Dynamic CDN), it can be turned on in Full-site mode (for wildcard domains), it also caches statics and accelerates dynamic content on itself, that is, the page dynamics will also load through fast network provider. This is important for us, because basically our site is dynamic, it uses many subdomains, and it’s more convenient to configure CDN once for the “star” - * .semrushchina.cn.

We already saw this product at the earlier stages of our Chinese project, but then it still did not work, and the developers promised that the product would be available to all customers just about. And he became.

In DCDN, you can:

• configure SSL termination with your certificate,
• enable dynamic content acceleration,
• flexibly configure caching of static files,
• do cache purge,
• throw web sockets,
• enable compression and even HTML Beautifier.

In general, everything is like adults and large CDN providers.

After Origin (the place where the CDN edge servers will go) is indicated, then it remains to create a CNAME for the asterisk, linking to all.semrushchina.cn.w.kunluncan.com (this CNAME was obtained in the Alibaba Cloud console), and CDN will work.

According to the test results, this CDN helped us a lot. Statistics are given below.

These are very good results, especially if we compare them with what figures were at the beginning. But we knew that the browser test of the American version of our site www.semrush.com was working out from the USA on average for 8.3s (a very approximate value). There is much to strive for. Moreover, there were CDN providers who were interested in testing.

So we smoothly move to another giant in the Chinese market - Tencent .

Tencent cloud

Tencent is only developing its cloud - this is evident in a small number of products. In the course of its use, we wanted to test not only their CDN, but also the network infrastructure as a whole:

• Do they have something similar to CEN?
• How does IPSEC work for them? Is it fast, what is uptime?
• Do they have anycast?

We will analyze these questions separately.

Analog CEN

Tencent has a Cloud Connect Network ( CCN ) product that allows you to connect VPCs from different regions, including regions within China and outside. The product is now in the internal beta, and you need to create a ticket with a request to connect to it. From support we learned that global accounts (not about Chinese citizens and non-legal entities) can’t participate in the beta testing program and generally connect the region inside China with the region outside. 1-0 in favor of Ali Cloud

IPSEC

Tencent's southernmost region is Guangzhou . We built a tunnel and connected it with the Hong Kong region in the GCP (then this region was already available). They also lifted the second tunnel to Ali Cloud from Shenzhen to Hong Kong at the same time. It turned out that through the Tencent latency network, Hong Kong is generally better (10ms) than from Shenzhen to Hong Kong to Ali (120ms - what?). But this didn’t speed up the work of the site aimed at working through Tencent and this tunnel, which in itself was an amazing fact and once again proved the following: latency - for China this is not an indicator that you should really pay attention to when developing a solution for passing the Chinese firewall.

Anycast Internet Acceleration

Another product that allows you to work through anycast IP is AIA . But it is also inaccessible to global accounts, so I won’t tell about it, but knowing that such a product exists can be useful.

But the CDN test showed quite interesting results. Tencent's CDN cannot be enabled on a full-site, only on specific domains. We got domains and started traffic on them:

It turned out that this CDN has the following function: Cross Border Traffic Optimization . This feature should reduce costs when traffic flows through the Chinese firewall. As Origin , the IP address of Google GLB (GLB anycast) was specified. So we wanted to simplify the architecture of the project.

The results were very good - at the level of Ali Cloud CDN, and in some places even better. This is surprising, because if the tests succeed, you can abandon a significant part of the infrastructure, tunnels, CEN, virtual machines, etc.

We were not happy for long, because the problem was revealed: the tests at Catchpoint faked for the Internet provider China Mobile. From any location, we got a timeout via Tencent's CDN. Correspondence with technical support did not lead to anything. About a day we tried to solve this problem, but nothing came of it.

I was in China at that time, but I could not find public Wi-Fi on the network of this provider to verify the problem personally. Otherwise, everything looked fast and good.
However, due to the fact that China Mobile is one of the three largest operators, we were forced to return traffic to Ali CDN.
But in general, this was a rather interesting solution, which deserves longer testing and troubleshooting of this problem.

Akamai

The last CDN provider we tested is Akamai . This is a huge provider that has its own network in China. Of course, we could not get past him.

From the very beginning, we agreed with Akamai on a trial period so that we can switch the domain and see how it will work on their network. I will describe the result of all testing in the form of “What I liked” and “What I didn’t like”, as well as the test results.

What did you like:

• The guys from Akamai helped a lot in all matters and accompanied us at all stages of testing. Constantly trying to improve something on their side. They gave good technical advice.
• Akamai runs about 10-15% slower than our solution through Ali Cloud CDN. It’s impressive that in Origin we specified the GLB IP address for Akamai, that is, the traffic didn’t go through our solution (you can potentially refuse part of the infrastructure). But still, the test results showed that this solution is worse than our current version (comparative results below).
• Tested both Origin GLB and Origin in China. Both options are about the same.
• There is a Sure Route (automatic routing optimization). You can host the test object on Origin, and the Akamai server Edge will try to pick it up (regular GET). For these requests, speed and other metrics are measured, on the basis of which the Akamai network optimizes routes so that the traffic goes faster for our site and it can be seen that the inclusion of this feature really had a great effect on the speed of the site.
• Versioning the configuration in the web interface is cool. You can do Compare for versions, see diff. View previous versions.
• You can roll out the new version first only on the Akamai Staging network - the same network as production, only this way does not affect real users. For this test, you need to spoof DNS records on the local machine.
• Very fast download speed through their network of large statics, and, apparently, of any other files. A file from the “cold” cache is taken many times faster than the same file from the “cold” Ali CDN cache. From the “hot” cache, the speed is already plus or minus the same.

Ali CDN test:

root@shenzhen1:~# curl -o /dev/null -w@curl_time https://en.semrushchina.cn/my_reports/build/scripts/simpleInit.js?v=1551879212
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5757k    0 5757k    0     0   513k      0 --:--:--  0:00:11 --:--:--  526k
time_namelookup:  0.004286
time_connect:  0.030107
time_appconnect:  0.117525
time_pretransfer:  0.117606
time_redirect:  0.000000
time_starttransfer:  0.840348
----------
time_total:  11.208119
----------
size_download:  5895467 Bytes
speed_download:  525999.000B/s

Akamai test:

root@shenzhen1:~# curl -o /dev/null -w@curl_time https://www.semrushchina.cn/my_reports/build/scripts/simpleInit.js?v=1551879212
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5757k    0 5757k    0     0  1824k      0 --:--:--  0:00:03 --:--:-- 1825k
time_namelookup:  0.509005
time_connect:  0.528261
time_appconnect:  0.577235
time_pretransfer:  0.577324
time_redirect:  0.000000
time_starttransfer:  1.327013
----------
time_total:  3.154850
----------
size_download:  5895467 Bytes
speed_download:  1868699.000B/s

We noticed that the situation from the example above depends on various factors. At the time of this writing, I ran the test again. The results for both platforms were approximately the same. This tells us that the Internet in China, even for large operators and cloud providers, behaves differently from time to time.

I’ll add a big plus to Akamai to the previous paragraph: if Ali shows such flashes of high performance and very low (this applies to Ali CDN, and Ali CEN, and Ali IPSEC), then at Akamai every time, no matter how I test their network, all works stably.
Akamai really have a lot of coverage in China and work through many providers.

What did not like:

• I do not like the web interface and the scheme of work - these are zero. But basically you get used to it (probably).
• Test results are worse than our site.
• There are more errors during the tests than on our site (uptime below).
• There are no DNS servers in China. From here there are a lot of errors in tests because of DNS resolve timeout.
• Do not provide their IP ranges -> there is no way to register the correct set_real_ip_from on our servers.

Metrics (~ 3626 runs; all metrics except Uptime in ms; statistics for one period of time):

Percentile distribution (in ms):

The conclusion is this: the option with Akamai is viable, but does not give the same stability and speed indicators as our own solution, coupled with Ali CDN.

Small notes

Some points were not included in the story, but I would like to write about them too.

Beijing + Tokyo and Hong Kong

As I said above, we tested the IPSEC tunnel to Hong Kong (HK). But we also tested CEN before HK. It costs a little cheaper, and it was interesting how it will work between cities with a distance of ~ 100km. It turned out to be interesting that the latency between these cities is 100ms higher than in our original version (to Taiwan). Speed, stability was also better for Taiwan. As a result, we left HK as a backup IPSEC region.

In addition, we tried to raise such an installation:

• customer termination in Beijing,
• IPSEC and CEN to Tokyo,
• in Ali, the CDN was listed as the origin server in Beijing.

This scheme was not so stable, although in terms of speed as a whole it was not inferior to our solution. As for the tunnel, I saw periodic drops even for CEN, which was supposed to be stable. Therefore, we returned to the old scheme and dismantled this staging.

Below is the statistics on latency between different regions on different channels. Maybe someone will be interested in it.

IPsec
Ali cn-beijing <--> GCP asia-northeast1 - 193ms
Ali cn-shenzhen <--> GCP asia-east2 - 91ms
Ali cn-shenzhen <--> GCP us-east4 - 200ms

CEN
Ali cn-beijing <---> Ali ap-northeast-1 - 54ms (!)
Ali cn-shenzhen <---> Ali cn-hongkong - 6ms (!)
Ali cn-shenzhen <---> Ali us -east1 - 216ms

General information about the Internet in China

As an addition to the problems with the Internet, described at the very beginning, in the first part of the article.

• The Internet in China is pretty fast inside.
  
  • The conclusion is made on the basis of testing public Wi-Fi networks in various locations where these networks are used by a large number of people.
  • The download and upload speeds to servers inside China were around 20 Mbps and 5-10 Mbps, respectively.
  • The speed to servers outside of China is just miserable, less than 1 Mbps.
• The Internet in China is not very stable.
  
  • Sometimes sites can open quickly, sometimes slowly (at the same time of day on different days), provided that the configuration does not change. We observed this with the example of semrushchina.cn. This can be attributed to Ali CDN, which also works this way or that, depending on the time of day, the position of the stars, etc.
• Mobile Internet is almost everywhere 4G or 4G +. Catches in the subway, elevators - in short, everywhere.
• The fact that Chinese users only trust domains in the .cn zone is a myth. We learned this directly from users.
  
  • You can see how http://baidu.cn redirects to www.baidu.com (in mainland China as well).
• Many resources are really blocked. Primitive: google.com, Facebook, Twitter. But many Google resources work (of course, not all Wi-Fi and VPNs are used at the same time (on the router side, too, that's for sure).
• Many “technical” domains of blocked corporations also work. This means that you should not always recklessly cut out all Google and other seemingly blocked resources. You need to look for some list of prohibited domains.
• They have only three main Internet providers: China Unicom, China Telecom, China Mobile. There are even smaller ones, but their market share is insignificant

Bonus: final decision scheme

Total

A year has passed since the start of the project. We started with the fact that our site generally basically refused to work normally from China, but just GET curl took 5.5 seconds.

Then, with such indicators in the first decision (Cloudflare):

We ended up with these results (statistics for the last month):

As you can see, the uptime in 100% has not yet been achieved, but we will come up with something, and then we will tell you about the results in a new article :)

To the one who has read to the end all three parts - respect. I hope you were all as interested as I was when I did this.

PS Previous Parts

Part 1
Part 2