April 26, 2018 at 12:34Generating JWS and JWK from rsa keys using the example of Let's Encrypt and ISPmanager integration
Hello! My name is Dmitry Smirnov, I am a developer from ISPsystem and it is me who is responsible for the appearance of integration with Let's Encrypt in the ISPmanager 5 panel . I’ll tell you how the development of the plug-in went, how it changed and how it came to its current state. You will learn from the text how to generate JWS and JWK from rsa-keys and get Let's Encrypt certificate for ACME v01. If interested, welcome to cat.
The first version of the plugin was perfect in every way. Her success is only comparable to her monumental collapse (yes, the scriptwriters of The Matrix also read this article). The task then was set to me something like this: here's letsencrypt.org , do something with it. Well, I did.
Initially, the plug-in simply pulled the official Let's Encrypt client from the github and worked directly with it. And user friendly, to put it mildly, he was not. No domain? We do not order anything. Aliases are not resolved? Round off. All the preparatory work for the issuance of the certificate fell on the shoulders of the user, and any error led to an unsuccessful receipt of the certificate.
Needless to say, the plugin was returned for revision. So began my fascinating journey into the wonderful world of Internet security and customer focus.
Before the second development approach, we formulated several tasks:
Of course, the main point for me was the first point. Armed with official documentation, I started development.
Let's Encrypt ( LE ) was created by Internet Security Research Group ( ISRG ). Especially for him, ISRG developed the Automatic Certificate Management Environment ( ACME ) protocol . The process of obtaining a certificate in itself is a POST request to the LE service , where the request body is represented in the form of JSON wrapped in JSON Web Signature ( JWS ).
The steps to get look like this:
Let's start in order.
To create and authorize a user, you need a pair of rsa keys in pem format, which subsequently will serve as the basis for constructing JWS .
POST request data structure for communicating with ACME v01:
Here it is worth focusing on three things. First, Replay-Nonce returns in the headers of the acme-v01.api.letsencrypt.org/directory response . Secondly, Payload is JSON in which you explain what, in fact, you want from ACME in this particular case. Thirdly, JWS is JSON of the following form (I will stipulate that there are ways to obtain signatures by other algorithms. Here is just one, possibly the simplest):
The question arose where to get data for the JWK . Short searches on the Internet paid off, and I found a simple way to look at the pair of pem keys in decrypted form itself. Here is an example:
command output in the shortest form:
Here they are, so we need the data, take it - I do not want to. I took it, brought it to the right kind, created JWS . But cruel disappointment awaited me: the signature was incorrect. All this resulted in several long hours of searching for information on the Internet, debugging and hopelessness. Still, the answer surfaced.
It turned out that the first two zeros are artifacts that appear when encoding an integer using ASN.1, but there is another way to get the module in the form ready for processing and pasting in JWS .
Voila:
Now let's go through the requests and payloads. First of all, let's call GET for acme-v01.api.letsencrypt.org/directory . From the resulting JSON
take the user agreement address and addresses for certificate requests.
Now, for each domain name for which we issue a certificate, you need to go through authorization. In the response, we get a list of available name ownership checks.
Let's proceed to domain name checks. Three options are available for ACME v01: http, dns, tls. Our choice fell on the first method, as the simplest and most affordable. The bottom line is simple: a .well-known / acme-challenge subdirectory should be created in the domain directory, where the verification token will be placed - a file with the name specified in the verification.
The token itself must contain the string token_name. Base64Url (jwk_print) - this will be the so-called authorization key. You can easily get the fingerprint using OpenSSL command:
I'm afraid for bash you will have to write the base64url function yourself.
No matter how easy it was, I managed to get stuck for several hours. Children's mistakes are the worst. In openssl, a line break character was passed at the end of the JWK . Be attentive to this data, the print should be clean ”:).
The request body will look like this:
and url we take from JSON of check.
I wrote a tricky mechanism that distributed tokens to the necessary nodes of the cluster and then deleted them, which later turned out to be superfluous work (more on this later).
And, lo and behold, the first LE certificate was received successfully!
It remained to solve the problems that ordinary users were waiting for. How to get around the inevitable synchronous certificate issue when domain aliases are not resolved yet? We decided that the user should receive a certificate immediately upon ordering, but self-signed.
We issue a self-signed certificate, connect to the domain and register the internal certificate order from LE. Every 5 minutes we begin the procedure for receiving. If it fails, calmly wait for the next attempt. We give the user 24 hours to resolve all possible problems, and only then surrender and cross off the certificate from the issuing queue.
Ready fresh certificate from LE remains to put in the place of the old self-signed. That's all. This is how the integration plugin with Let's Encrypt saw the light.
We tried to foresee all the problems that might arise during the issuance of the certificate, but some errors nevertheless eluded our inquiring gaze. The main problem was the numerous and ubiquitous .htaccess configuration files. Very often, they led to a situation where the verification token, carefully placed in the domain directory, simply simply turned out to be inaccessible. And the only way out for the user was to temporarily disable his settings.
A few months later, it became clear that the mechanism for sending tokens to domain directories did not justify itself. For all the web-domains panels created by means of the toolbar, we started adding the alias /.well-known/acme-challenge/, leading to the directory / usr / local / mgr5 / www / letsencrypt. It was in it that tokens began to be placed for verification, which subsequently reduced access errors to a minimum.
Check through TXT records in the domain zone appeared just six months ago. Virtually no tricks to prepare did not arise, except for one. For a TXT record, the line that we wrote in the token needs to be run through the command
That's all for today. Read about the next plugin transition to ACME v02 and support for wildcard certificates in the next release.
Let's Encrypt 1.0
The first version of the plugin was perfect in every way. Her success is only comparable to her monumental collapse (yes, the scriptwriters of The Matrix also read this article). The task then was set to me something like this: here's letsencrypt.org , do something with it. Well, I did.
Initially, the plug-in simply pulled the official Let's Encrypt client from the github and worked directly with it. And user friendly, to put it mildly, he was not. No domain? We do not order anything. Aliases are not resolved? Round off. All the preparatory work for the issuance of the certificate fell on the shoulders of the user, and any error led to an unsuccessful receipt of the certificate.
Needless to say, the plugin was returned for revision. So began my fascinating journey into the wonderful world of Internet security and customer focus.
Let's Encrypt 2.0
Before the second development approach, we formulated several tasks:
- Implement certificate receipt at the ACME protocol level.
- Inform the user in detail about the process.
- Make it possible to obtain a certificate when creating a web domain.
- Expect resolution of domain names within 24 hours after the start of the issuing process.
Of course, the main point for me was the first point. Armed with official documentation, I started development.
Let's Encrypt ( LE ) was created by Internet Security Research Group ( ISRG ). Especially for him, ISRG developed the Automatic Certificate Management Environment ( ACME ) protocol . The process of obtaining a certificate in itself is a POST request to the LE service , where the request body is represented in the form of JSON wrapped in JSON Web Signature ( JWS ).
The steps to get look like this:
- registration,
- authorization and obtaining methods of confirmation of domain ownership,
- proof of ownership
- obtaining a certificate.
Let's start in order.
User registration and authorization
To create and authorize a user, you need a pair of rsa keys in pem format, which subsequently will serve as the basis for constructing JWS .
openssl genrsa -out private.pem 2048POST request data structure for communicating with ACME v01:
{
"header": jws, //JSON Web Signature
"protected": Base64Url(jws + Replay-Nonce), //Nonce — защита от повторов
"payload": Base64Url(payload), //Запрос
"signature": Base64Url(sign(protected.payload, private.pem)) //Подпись
}
Here it is worth focusing on three things. First, Replay-Nonce returns in the headers of the acme-v01.api.letsencrypt.org/directory response . Secondly, Payload is JSON in which you explain what, in fact, you want from ACME in this particular case. Thirdly, JWS is JSON of the following form (I will stipulate that there are ways to obtain signatures by other algorithms. Here is just one, possibly the simplest):
{
"alg" : "RS256",
"jwk" : { //JSON Web Key
"kty" : "RSA", // key type -- тип семейства криптографических алгоритмов использованных для ключа
"e" : "...", // публичная экспонента ключа в виде HexToBase64UrlEnodedByte
"n" : "..." // modulus ключа в виде HexToBase64UrlEnodedByte
}
}The question arose where to get data for the JWK . Short searches on the Internet paid off, and I found a simple way to look at the pair of pem keys in decrypted form itself. Here is an example:
openssl rsa -text -noout < private.pemcommand output in the shortest form:
Private-Key: (2048 bit)
modulus:
00:a8:c5:cc:9c:24:9b:d1:8d:9a:67:81:4d:1f:57:
...
8c:45:51:9e:26:fc:12:35:9e:a0:10:fd:80:94:cc:
09:a5
publicExponent: 65537 (0x10001)
privateExponent:
...
prime1:
...
prime2:
...
exponent1:
...
exponent2:
...
coefficient:
...Here they are, so we need the data, take it - I do not want to. I took it, brought it to the right kind, created JWS . But cruel disappointment awaited me: the signature was incorrect. All this resulted in several long hours of searching for information on the Internet, debugging and hopelessness. Still, the answer surfaced.
It turned out that the first two zeros are artifacts that appear when encoding an integer using ASN.1, but there is another way to get the module in the form ready for processing and pasting in JWS .
openssl rsa -noout -modulus < private.pemVoila:
Modulus=A8C5CC9C249BD18D9A67814D1F57...8C45519E26FC12359EA010FD8094CC09A5Obtaining a Certificate
Now let's go through the requests and payloads. First of all, let's call GET for acme-v01.api.letsencrypt.org/directory . From the resulting JSON
{
"key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
"meta": {
"terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf"
},
"new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
"new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
"new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
"revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert",
"zH_Sr0qwmwM": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
}take the user agreement address and addresses for certificate requests.
registration
url = directory["new-reg"]
payload = {
"resource": "new-reg",
"agreement": directory["meta"]["terms-of-service"]
}Login
Now, for each domain name for which we issue a certificate, you need to go through authorization. In the response, we get a list of available name ownership checks.
url = directory["new-authz"]
payload = {
"resource": "new-authz",
"identifier": {
"type":"dns",
"value": "name"
}
}Checks
Let's proceed to domain name checks. Three options are available for ACME v01: http, dns, tls. Our choice fell on the first method, as the simplest and most affordable. The bottom line is simple: a .well-known / acme-challenge subdirectory should be created in the domain directory, where the verification token will be placed - a file with the name specified in the verification.
The token itself must contain the string token_name. Base64Url (jwk_print) - this will be the so-called authorization key. You can easily get the fingerprint using OpenSSL command:
echo jwk | openssl dgst -sha256 -binary | base64urlI'm afraid for bash you will have to write the base64url function yourself.
No matter how easy it was, I managed to get stuck for several hours. Children's mistakes are the worst. In openssl, a line break character was passed at the end of the JWK . Be attentive to this data, the print should be clean ”:).
The request body will look like this:
payload = {
"resource": "challenges",
"keyAuthorization": ключ авторизации
}and url we take from JSON of check.
I wrote a tricky mechanism that distributed tokens to the necessary nodes of the cluster and then deleted them, which later turned out to be superfluous work (more on this later).
Certificate Issue
url = directory["new-cert"]
payload = {
"resource": "new-cert",
"csr": csr
}And, lo and behold, the first LE certificate was received successfully!
Customer focus
It remained to solve the problems that ordinary users were waiting for. How to get around the inevitable synchronous certificate issue when domain aliases are not resolved yet? We decided that the user should receive a certificate immediately upon ordering, but self-signed.
We issue a self-signed certificate, connect to the domain and register the internal certificate order from LE. Every 5 minutes we begin the procedure for receiving. If it fails, calmly wait for the next attempt. We give the user 24 hours to resolve all possible problems, and only then surrender and cross off the certificate from the issuing queue.
Ready fresh certificate from LE remains to put in the place of the old self-signed. That's all. This is how the integration plugin with Let's Encrypt saw the light.
Difficulties
We tried to foresee all the problems that might arise during the issuance of the certificate, but some errors nevertheless eluded our inquiring gaze. The main problem was the numerous and ubiquitous .htaccess configuration files. Very often, they led to a situation where the verification token, carefully placed in the domain directory, simply simply turned out to be inaccessible. And the only way out for the user was to temporarily disable his settings.
A few months later, it became clear that the mechanism for sending tokens to domain directories did not justify itself. For all the web-domains panels created by means of the toolbar, we started adding the alias /.well-known/acme-challenge/, leading to the directory / usr / local / mgr5 / www / letsencrypt. It was in it that tokens began to be placed for verification, which subsequently reduced access errors to a minimum.
DNS Verification
Check through TXT records in the domain zone appeared just six months ago. Virtually no tricks to prepare did not arise, except for one. For a TXT record, the line that we wrote in the token needs to be run through the command
echo ключ_авторизации | openssl dgst -sha256 -binary | base64urlThat's all for today. Read about the next plugin transition to ACME v02 and support for wildcard certificates in the next release.
