Amazon CloudFront + Custom Origins
For about a year now, Amazon added Custom Origins support for its CloudFront service, and in my opinion this is very good, because I have been eyeing various CDNs for a long time.
Honestly speaking, it’s hard to find a CDN now for a small project or for a start, when the estimated traffic should not exceed 50 - 100 GB per month. They are either very expensive or only work with those sites that generate a lot of traffic and almost all work on a prepaid basis, i.e. You pay not for the actually used traffic, but for some amount that you may not work out.
Amazon CloudFront compares favorably with competitors in this regard. The fee here is charged only for the actually used traffic and it is quite small, depending on the region it averages $ 0.15 per GB of traffic. But before you had to use CloudFront in tandem with the S3 service, which increased the cost, now you can use your own server as the origin server.
I would like to tell in detail about how I connected, paid and added CloudFront support in my project.
First, I want to talk about how I pay for the service and what difficulties I encountered at this stage.
So, I myself am from Ukraine, but I think that for all or many CIS countries this information will be relevant. The only form of payment offered to me on Amazon is credit card payment. Those. no electronic money like WebMoney is accepted. Therefore, I had to go to the bank and open a credit card. I chose Aval, although most likely the bank does not matter much, but the credit card itself matters. I discovered a dollar Visa Classic, and when I opened it, I clarified with a bank employee whether I could pay for purchases on the Internet using foreign cards on this site. Opening a card took 10 business days.
Now about the problems that I encountered:
1. Activation of the card - here I must admit I did a little bit of mischief. I put money on the card through the bank’s cash desk, but it turns out that the card is activated when you perform some action with it through an ATM (while the money should already be on it), even checking the balance will do. In any case, it is worth checking with the bank employee how the card is activated in their bank.
2. CVV - Amazon, which surprised me very much, does not ask for the CVV code of the card. When registering, you need to enter only the card number, the date by which it is valid and the name of the card holder. As it turned out, Amazon works according to some scheme where this CVV is not required, but you need, at the time of payment or permanently, to disable the protection of your card by CVV code. This can be done over the phone via the bank’s call center or by writing a statement in the department where you opened the card.
In principle, these are all the problems that arose with my payment.
The registration process can be started from the CloudFront page . Registration is very simple, at one of the registration steps you will need to enter information about the card and if it passes validation, an account will be registered. Together with the CloudFront account, an account for the S3 service will be automatically created, even if we want to use Custom Origins.
After registration, you need to generate the keys that will be needed to communicate with the API and to generate secure URLs to your files. To do this, go to the Security Credentials section of your account. Here you will find 3 tabs:
1. Access Keys - the data from this tab will be needed to create an authorization header for requests to the API;
2. X.509 certificates - keys for API requests through SOAP;
3. Key Pairs - keys for generating secure URLs.
All requests to the REST API should contain the following headers:
1. x-amz-date - date of the request. The date must be in one of the formats described in the RFC 2616 specification, the “Date / Time Formats” section;
2. Content-Type - request body type, usually “application / xml”;
3. Content-Length - request body length;
4. Authorization - the authorization header, has the following structure: "AWS aws_secret_key_id: signature", where:
4.1 AWS is a constant string, followed by a whitespace character;
4.2 aws_secret_key_id - can be found on the Security Credentials page of your account in the “Access Keys” tab;
4.3 signature - a verification signature that is generated using the sha1 hash algorithm, based on the request date that you specified in the x-amz-date header and the secret key that can be found on the page where aws_secret_key_id is located. Below is the PHP code that generates signature:
In order to have the opportunity to distribute files through Amazon CloudFront, you must create Distribution.
On one account, you can create a maximum of 100 Distribution s, the number of files in one Distribution is not limited.
Distribution can be of 2 types:
1. Download - to distribute files via HTTP and HTTPS (HTTPS is a bit more expensive);
2. Stream - for the distribution of video and audio files via the RTMP protocol. I want to immediately disappoint, Distribution of the Stream type does not work with Custom Origins , only with S3 as the origin server.
Based on the title of the article and paragraph 2, I will only consider Download Distribution. I’ll give you an example of an XML request to create a new Download Distribution: URL for the request :
cloudfront.amazonaws.com/2010 –11–01 / distribution
Request method : POST I will
describe in more detail what each of the tags means:
1. CustomOrigin - this is the area in which the parameters of your origin server are set:
1.1 DNSName - domain of origin server;
1.2 HTTPPort - port for accessing the origin server via HTTP;
1.3 OriginProtocolPolicy - Amazon can request files from your origin server in two ways: http-only - always using the HTTP protocol, match-viewer - using the protocol that the end user used to request the file, but only HTTP or HTTPS.
2. CallerReference - a unique identifier for the request, it can be either numeric or alphabetic, the main thing is to be unique;
3. CNAME - for each Distribution Amazon creates a subdomain for the cloudfront.net domain. On this domain you can hang one or several CNAMEs;
4. Comment - comment for Distribution;
5. Enabled - sets whether Distribution is active or not;
6. TrustedSigners - Distributions are also divided according to the type of access to them, public and private. Everyone has access to files in public Distribution, and to access private ones you need to create a secure URL where you can specify the expiration date, IP access and more. To create a private Distribution, you need to specify this section;
7. Logging - section for setting query logging parameters:
7.1 Bucket - Amazon uses the S3 service to save logs, so if you want to use them you will have to pay extra for S3. This parameter sets the S3 Bucket in which the logs will be stored;
7.2 Prefix - log prefix, as I understand it, this is something like a directory for logs.
Immediately after creating Distribution, it is in InProgress status, but you can start working with it only after it changes its status to Active. It usually takes 10-15 minutes. To check the status of Distribution, you can use the API request for information about Distribution.
Request URL : cloudfront.amazonaws.com/2010 –11–01 / distribution / distribution_id
Request method : GET
distribution_id - Amazon returns in response to a successful request to create a Distribution.
So we got to the most important thing - the return of our files via CDN. There are several features:
1. Amazon CloudFront does not transmit to the origin server any URL parameters that were transferred when the file was requested from CDN.T. e. if the end user requested the file example_sub_domain.cloudfront.net/image_1.jpg?param=value , then when you request this file from the server’s origin you will not get “param = value”, BUT the logs will contain the full URL;
2. Updating the file in Distribution. A file may be requested in the following cases:
2.1 If it is not on the CloudFront server;
2.2 If the file has been expired. The expire date can be controlled using the headers: cache-control, expires and pragma. By default, the file is cached for 24 hours;
2.3 If the file was deleted from Distribution using the Invalidation request ;
It should be noted that when requesting a file by an end user, Amazon does not check the file modification date as other CDNs do. If you want the Amazon server to automatically pick up the file when you change the file, you need to indicate its version or modification date in the file name.
3. Protection of files on the origin side - if you use your own as the origin server (that is, exactly the case that I describe in the article), then Amazon does not protect the files on your origin server from unauthorized access, this task lies entirely with the owner of origin. But at the same time, file protection should be implemented in such a way that Amazon itself has access to these files via HTTP or HTTPS. At the same time, Amazon does not send any data when requesting a file, by which its requests could be distinguished from others. Based on this, there is only one option for protecting member files on your origin server - IP protection. I protect member files with .htaccess of the following form:
in order for Amazon to have access to my files, I added the following rule to .htaccess
I took the mask 216.137.60.0/23 here .
Suppose the file is available on your origin server at:
Then in order to give it through Amazon CloudFront you need to generate the following URL:
where example_sub_domain.cloudfront.net is the domain of that Distribution that matches the origin of origin.example.com.
Private URLs are divided into two types:
1. Canned - you can specify only the expiration date of the URL;
2. Custom - you can specify the period when URl will be valid, as well as one or more IP addresses from which you can use the URL.
I needed only the date until which the URL will be valid, so I will give an example to generate a Canned URL:
The function uses the following parameters and constants:
1. $ url is the source URL of the form http://example_sub_domain.cloudfront.net/images/image_1.jpg;
2. SECURE_URL_TIMEOUT - timeout for the URL in seconds;
3. CLOUD_FRONT_KEY_PAIR_PATH - the path to your private key, which can be generated on the Security Credentials page in the Key Pairs tab;
4. CLOUD_FRONT_KEY_PAIR_ID - identifier of the private key, which can be found in the same place as the key itself.
The output of this function will be a private URL, through which the end user, in the allotted timeout, will be able to access content that is located in the private Distribution.
1. Amazon CloudFront - the main page of the service;
2. Developer Guide - a detailed description of the CloudFront service;
3. API Reference - REST API documentation for the CloudFront service;
4. AWS SDK for PHP - PHP library that contains classes for working with all Amazon services. A very useful thing;
5. Test code - a small code that I wrote during the testing process.
That's basically all I wanted to write. The article does not claim to be a complete guide on using custom origin servers for CloudFront, but I hope it will be useful to those who do not need S3. Thank you all for your attention.
Honestly speaking, it’s hard to find a CDN now for a small project or for a start, when the estimated traffic should not exceed 50 - 100 GB per month. They are either very expensive or only work with those sites that generate a lot of traffic and almost all work on a prepaid basis, i.e. You pay not for the actually used traffic, but for some amount that you may not work out.
Amazon CloudFront compares favorably with competitors in this regard. The fee here is charged only for the actually used traffic and it is quite small, depending on the region it averages $ 0.15 per GB of traffic. But before you had to use CloudFront in tandem with the S3 service, which increased the cost, now you can use your own server as the origin server.
I would like to tell in detail about how I connected, paid and added CloudFront support in my project.
Payment
First, I want to talk about how I pay for the service and what difficulties I encountered at this stage.
So, I myself am from Ukraine, but I think that for all or many CIS countries this information will be relevant. The only form of payment offered to me on Amazon is credit card payment. Those. no electronic money like WebMoney is accepted. Therefore, I had to go to the bank and open a credit card. I chose Aval, although most likely the bank does not matter much, but the credit card itself matters. I discovered a dollar Visa Classic, and when I opened it, I clarified with a bank employee whether I could pay for purchases on the Internet using foreign cards on this site. Opening a card took 10 business days.
Now about the problems that I encountered:
1. Activation of the card - here I must admit I did a little bit of mischief. I put money on the card through the bank’s cash desk, but it turns out that the card is activated when you perform some action with it through an ATM (while the money should already be on it), even checking the balance will do. In any case, it is worth checking with the bank employee how the card is activated in their bank.
2. CVV - Amazon, which surprised me very much, does not ask for the CVV code of the card. When registering, you need to enter only the card number, the date by which it is valid and the name of the card holder. As it turned out, Amazon works according to some scheme where this CVV is not required, but you need, at the time of payment or permanently, to disable the protection of your card by CVV code. This can be done over the phone via the bank’s call center or by writing a statement in the department where you opened the card.
In principle, these are all the problems that arose with my payment.
Registration and keys
The registration process can be started from the CloudFront page . Registration is very simple, at one of the registration steps you will need to enter information about the card and if it passes validation, an account will be registered. Together with the CloudFront account, an account for the S3 service will be automatically created, even if we want to use Custom Origins.
After registration, you need to generate the keys that will be needed to communicate with the API and to generate secure URLs to your files. To do this, go to the Security Credentials section of your account. Here you will find 3 tabs:
1. Access Keys - the data from this tab will be needed to create an authorization header for requests to the API;
2. X.509 certificates - keys for API requests through SOAP;
3. Key Pairs - keys for generating secure URLs.
Standard headers when communicating with the REST API
All requests to the REST API should contain the following headers:
1. x-amz-date - date of the request. The date must be in one of the formats described in the RFC 2616 specification, the “Date / Time Formats” section;
2. Content-Type - request body type, usually “application / xml”;
3. Content-Length - request body length;
4. Authorization - the authorization header, has the following structure: "AWS aws_secret_key_id: signature", where:
4.1 AWS is a constant string, followed by a whitespace character;
4.2 aws_secret_key_id - can be found on the Security Credentials page of your account in the “Access Keys” tab;
4.3 signature - a verification signature that is generated using the sha1 hash algorithm, based on the request date that you specified in the x-amz-date header and the secret key that can be found on the page where aws_secret_key_id is located. Below is the PHP code that generates signature:
- $signature = base64_encode(hash_hmac('sha1', $requestDate, $awsSecretKey, true));
Distribution
In order to have the opportunity to distribute files through Amazon CloudFront, you must create Distribution.
On one account, you can create a maximum of 100 Distribution s, the number of files in one Distribution is not limited.
Distribution can be of 2 types:
1. Download - to distribute files via HTTP and HTTPS (HTTPS is a bit more expensive);
2. Stream - for the distribution of video and audio files via the RTMP protocol. I want to immediately disappoint, Distribution of the Stream type does not work with Custom Origins , only with S3 as the origin server.
Based on the title of the article and paragraph 2, I will only consider Download Distribution. I’ll give you an example of an XML request to create a new Download Distribution: URL for the request :
- version="1.0" encoding="UTF-8"?>
xmlns="http://cloudfront.amazonaws.com/doc/2010-11-01/"> -
> -
> www.example.com> -
> 80> -
> http-only> - >
-
> your unique caller reference> -
> mysite.example.com> -
> My comments> -
> true> -
> />> -
> -
> mylogs.s3.amazonaws.com> -
> myprefix/> - >
- >
cloudfront.amazonaws.com/2010 –11–01 / distribution
Request method : POST I will
describe in more detail what each of the tags means:
1. CustomOrigin - this is the area in which the parameters of your origin server are set:
1.1 DNSName - domain of origin server;
1.2 HTTPPort - port for accessing the origin server via HTTP;
1.3 OriginProtocolPolicy - Amazon can request files from your origin server in two ways: http-only - always using the HTTP protocol, match-viewer - using the protocol that the end user used to request the file, but only HTTP or HTTPS.
2. CallerReference - a unique identifier for the request, it can be either numeric or alphabetic, the main thing is to be unique;
3. CNAME - for each Distribution Amazon creates a subdomain for the cloudfront.net domain. On this domain you can hang one or several CNAMEs;
4. Comment - comment for Distribution;
5. Enabled - sets whether Distribution is active or not;
6. TrustedSigners - Distributions are also divided according to the type of access to them, public and private. Everyone has access to files in public Distribution, and to access private ones you need to create a secure URL where you can specify the expiration date, IP access and more. To create a private Distribution, you need to specify this section;
7. Logging - section for setting query logging parameters:
7.1 Bucket - Amazon uses the S3 service to save logs, so if you want to use them you will have to pay extra for S3. This parameter sets the S3 Bucket in which the logs will be stored;
7.2 Prefix - log prefix, as I understand it, this is something like a directory for logs.
Immediately after creating Distribution, it is in InProgress status, but you can start working with it only after it changes its status to Active. It usually takes 10-15 minutes. To check the status of Distribution, you can use the API request for information about Distribution.
Request URL : cloudfront.amazonaws.com/2010 –11–01 / distribution / distribution_id
Request method : GET
distribution_id - Amazon returns in response to a successful request to create a Distribution.
File upload features
So we got to the most important thing - the return of our files via CDN. There are several features:
1. Amazon CloudFront does not transmit to the origin server any URL parameters that were transferred when the file was requested from CDN.T. e. if the end user requested the file example_sub_domain.cloudfront.net/image_1.jpg?param=value , then when you request this file from the server’s origin you will not get “param = value”, BUT the logs will contain the full URL;
2. Updating the file in Distribution. A file may be requested in the following cases:
2.1 If it is not on the CloudFront server;
2.2 If the file has been expired. The expire date can be controlled using the headers: cache-control, expires and pragma. By default, the file is cached for 24 hours;
2.3 If the file was deleted from Distribution using the Invalidation request ;
It should be noted that when requesting a file by an end user, Amazon does not check the file modification date as other CDNs do. If you want the Amazon server to automatically pick up the file when you change the file, you need to indicate its version or modification date in the file name.
3. Protection of files on the origin side - if you use your own as the origin server (that is, exactly the case that I describe in the article), then Amazon does not protect the files on your origin server from unauthorized access, this task lies entirely with the owner of origin. But at the same time, file protection should be implemented in such a way that Amazon itself has access to these files via HTTP or HTTPS. At the same time, Amazon does not send any data when requesting a file, by which its requests could be distinguished from others. Based on this, there is only one option for protecting member files on your origin server - IP protection. I protect member files with .htaccess of the following form:
- Order Deny,Allow
- Deny from all
in order for Amazon to have access to my files, I added the following rule to .htaccess
- # Amazon CloudFront
- Allow from 216.137.60.0/23
I took the mask 216.137.60.0/23 here .
Public url
Suppose the file is available on your origin server at:
- origin.example.com/images/image_1.jpg
Then in order to give it through Amazon CloudFront you need to generate the following URL:
- example_sub_domain.cloudfront.net/images/image_1.jpg
where example_sub_domain.cloudfront.net is the domain of that Distribution that matches the origin of origin.example.com.
Private URL
Private URLs are divided into two types:
1. Canned - you can specify only the expiration date of the URL;
2. Custom - you can specify the period when URl will be valid, as well as one or more IP addresses from which you can use the URL.
I needed only the date until which the URL will be valid, so I will give an example to generate a Canned URL:
- function getSignedUrl($url)
- {
- // Prepare expire date
- $expireDate = time() + SECURE_URL_TIMEOUT;
-
- // Read Cloudfront Private Key Pair
- $fp = fopen(CLOUD_FRONT_KEY_PAIR_PATH, "r");
- $privateKey = fread($fp, 8192);
- fclose($fp);
-
- // Create the private key
- $privateKey = openssl_get_privatekey($privateKey);
- if (!$privateKey) {
- return false;
- }
-
- // Prepare json policy
- $json = '{"Statement":[{"Resource":"'.$url.'","Condition":{"DateLessThan":{"AWS:EpochTime":'.$expireDate.'}}}]}';
-
- // Sign the policy with the private key
- if(!openssl_sign($json, $signature, $privateKey, OPENSSL_ALGO_SHA1)) {
- return false;
- }
-
- // Create url safe signed policy
- $signature = str_replace(array('+','=','/'), array('-','_','~'), base64_encode($signature));
-
- //Construct the URL
- return $url
- . '?Expires=' . $expireDate
- . '&Signature=' . $signature
- . '&Key-Pair-Id=' . CLOUD_FRONT_KEY_PAIR_ID;
- }
The function uses the following parameters and constants:
1. $ url is the source URL of the form http://example_sub_domain.cloudfront.net/images/image_1.jpg;
2. SECURE_URL_TIMEOUT - timeout for the URL in seconds;
3. CLOUD_FRONT_KEY_PAIR_PATH - the path to your private key, which can be generated on the Security Credentials page in the Key Pairs tab;
4. CLOUD_FRONT_KEY_PAIR_ID - identifier of the private key, which can be found in the same place as the key itself.
The output of this function will be a private URL, through which the end user, in the allotted timeout, will be able to access content that is located in the private Distribution.
useful links
1. Amazon CloudFront - the main page of the service;
2. Developer Guide - a detailed description of the CloudFront service;
3. API Reference - REST API documentation for the CloudFront service;
4. AWS SDK for PHP - PHP library that contains classes for working with all Amazon services. A very useful thing;
5. Test code - a small code that I wrote during the testing process.
PS
That's basically all I wanted to write. The article does not claim to be a complete guide on using custom origin servers for CloudFront, but I hope it will be useful to those who do not need S3. Thank you all for your attention.