Cloud Storage: New API Features



    Recently, we talked about the  transition to a new platform , thanks to which we were able to improve the performance of cloud storage. We described in detail how we finalized the logic and architecture of the repository and rewrote some components on Go, so that everything began to work much faster and more stable than before.

    At the same time, we didn’t tell about everything: during the work on the API, we launched several new functions that, we hope, will be useful to you.

    Authentication


    Our repository fully supports the OpenStack Swift API , for which there are many third-party clients.

    For compatibility with these clients, Swift v1 authentication support has been implemented, and Keystone v2 and v3 authentication options have been limitedly implemented. Thus, three authentication methods are now available in our API.
    The first is almost no different from what was used before (only the authorization URI has changed to better comply with the standard):

    $ curl -i https://api.selcdn.ru/auth/v1.0 -H "X-Auth-User:[имя пользователя]" -H "X-Auth-Key:[пароль]


    The second method (Keystone v2 authentication type) does not use a GET, but a POST request; username and password are passed in a JSON array:

    $ curl -i -X POST https://api.selcdn.ru/v2.0/tokens -H 'Content-type: application/json' -d '{"auth": {"passwordCredentials": {"username": [имя пользователя], "password": [пароль]}}}'


    The API will also return a response to such a request in JSON format.

    Keysone v3 authentication looks like this:

    $ curl -i  https://api.selcdn.ru/v3/auth/tokens -XPOST -d '{"auth": { "identity": { "methods": ["password"], "password": { "user": { "id": [имя пользователя], "password": [пароль]}}}}}'


    We draw your attention to the fact that the Keystone API is not fully implemented in the repository: only authentication functions by login and password work. No Keystone protocol extensions are supported.

    The base address of the repository has also changed. Now it looks like this: api.selcdn.ru/v1/SEL_XXX (instead of XXX you need, of course, to substitute the user ID).

    Another change associated with the authentication system is the new mechanism of temporary tokens. Now you can generate additional tokens with a limited validity period, providing access only to a strictly defined container. Something like the Security Service in Amazon S3 :

    $ curl -i -X GET https://api.selcdn.ru/v1/temptokens -H "X-Auth-Token: [токен основного пользователя]" -H "X-Container:[контейнер]" -H "X-ExpireAfter: 8600"


    In the X-Auth-Token header, a valid token of the main user is transferred, in the X-Container the container for which the token will act is indicated, and in X-Expire-After - the duration of this token in seconds.

    Work with files


    The new API has expanded the ability to work with files. So, tar, .tar.gz, gzip archives can be unpacked in the background after the file is fully downloaded to the repository. The previous unpacking option involved unpacking the archives “on the fly”, i.e. it was necessary to maintain a connection with our servers until requests to create all the files from the downloaded archive were fulfilled. And when the connection was broken, everything had to be reloaded.

    Now everything is much simpler. If, for example, you need to upload an archive containing several thousand files to the repository, this can be done using a PUT request to the container address with the additional query parameter extract-archive-v2. After the archive is completely sent, code 201 is immediately given and background unpacking on the storage side starts:

    curl -i -X PUT https://api.selcdn.ru/v1/SEL_XXX/[имя контейнера]/?extract-archive-v2=tar.bz2 -H "X-Auth-Token: [токен]" -T "photos.tar.bz2"
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Access-Control-Expose-Headers: X-Backend-Timestamp, Etag, Last-Modified, X-Object-Manifest, X-Timestamp
Content-Length: 0
Content-Type: text/html
Etag: a1adb438cb26e91228870158a2062ef2
Extract-Id: 6a62579d-9ee2-2a32-26a4-207d5a47af2a


    Pay attention to the header Extract-Id present in the answer. With it, you can find out at what stage the unpacking operation is:

    $ curl -i https://api.selcdn.ru/v1/extract-archive/[Extract-Id] -H "X-Auth-Token: [токен]"
HTTP/1.1 200 OK
Number Files Created: 1185
Response Body:
Response Status: 201 Created
Errors:


    In our case, the answer with the code 200 means that the unpacking operation was completed successfully. The Number Files Created parameter specifies the number of files extracted from the archive.

    New features


    The API has many new features, in particular user management, domains, certificates and work with CDN. We will tell you more about them below.

    user management


    Previously, you could add and remove additional users exclusively through the control panel. Now this can be done through standard HTTP requests to the API.

    You can view the list of users as follows:

    $ curl -i -XGET https://api.selcd.ru/v1/users[?format=json] -H "X-Auth-Token: [токен]"


    Creating a new user is carried out using the request:

    $ curl -i -XPUT https://api.selcd.ru/v1/users/new_user -H "X-Auth-Token: [токен]" -H "X-Auth-Key: 123456" -H "X-User-ACL-Containers-W: container1, container2, container3" -H "X-User-ACL-Containers-R: container4" -H "X-User-Store-Password: yes" -H "X-User-Active: on"


    We have repeatedly received requests to expand user rights management. We fulfilled these requests, and this can be seen from the example request just given: pay attention to the headers X-User-ACL-Containers-W and X-User-ACL-Containers-R. The first indicates containers in which a new user can write information, and the second indicates containers that will be read-only.

    The X-Auth-Key header indicates the password for the user being created. X-User-Store-Password is the equivalent of the option "Store password on Selectel servers" in the control panel. An X-User-Active header with a value of on indicates that the new user will be active.

    You can change the data of an already created user as follows:

    $ curl -i -XPOST https://api.selcd.ru/v1/users/new_user -H "X-Auth-Token: [токен]" -H "X-Auth-Key: 123456" -H "X-User-ACL-Containers-W: container1, container4" -H "X-User-ACL-Containers-R: container4" -H "X-User-Store-Password: yes" -H "X-User-Active: on"


    Deleting a user is done using a DELETE request:

    $ curl -i -XDELETE https://selcdn.ru/users/[имя пользователя]  -H "X-Auth-Token: [токен]"


    For the primary user, only the password can be changed. This is done using a POST request with the X-Auth-Key header to api.selcd.ru/v1/users [primary user name].

    Domain management


    Previously, working with domains was also possible exclusively through a graphical interface. Now all operations can be performed through the API.

    Here, for example, you can view a list of all domains bound to containers:

    $ curl -i https://api.selcdn.ru/v1/domains -H "X-Auth-Token: [токен]"


    You can attach a domain using a POST request to the corresponding container with the X-Add-Container-Domains header:

    $ curl -i https://api.selcdn.ru/v1/SEL_XXX/{container_name} -XPOST -H "X-Add-Container-Domains: domain1.ru" -H "X-Auth-Token: [токен]"


    To remove a domain, a similar request is used, only the domain name is passed in the X-Remove-Container-Domains header:

    $ curl -i https://api.selcdn.ru/v1/SEL_XXX/{container_name} -XPOST -H "X-Remove-Container-Domains: domain1.ru" -H "X-Auth-Token: [токен]"


    To completely change the list of container-bound domains, you need to execute a POST request to the address of this container and transfer the updated list in the X-Container-Domains header:

    $ curl -i https://api.selcdn.ru/SEL_XXX/[имя контейнера] -H "X-Auth-Token: [токен]" -XPOST -H "X-Сontainer-Domains: domain1.ru domain2.ru"


    If several domains are attached to the container and you need to delete them, then in the X-Container-Domains header you can simply pass the number 0:

    $ curl -i https://ххххх.selcdn.ru/[имя контейнера] -H "X-Auth-Token: [токен]" -XPOST -H "X-Сontainer-Domains: 0"

    Log Request


    Through the API, you can request logs of network access to user containers for a specified period:

    $ curl -i -XPOST https://api.selcdn.ru/v1/logs -H "X-Auth-Token: [токен]" -H "X-Start-Time: 2016-02-02 09:00:00" -H "X-End-Time: 2016-05-02 12:00:00" -H "X-Limit: 10000"


    The X-Start-Time and X-End-Time headers, as you might guess, indicate the beginning and end of the period.

    The optional X-Limit header indicates the maximum number of log entries that will be issued in this request (by default, you will receive all the logs for the specified time interval).

    Upon completion of the selection, a private container logs will be automatically created in the repository, where the logs will be placed.

    SSL Certificate Management


    Another new feature is SSL certificate management. You can work with certificates both through the graphical interface and through the API.

    Here, for example, you can view a list of all certificates bound to containers:

    $ curl -i https://api.selcdn.ru/v1/ssl -XGET -H "X-Auth-Token: [токен]"


    To view information about the certificate, a request of the form is used:

    $ curl -i https://api.selcdn.ru/v1/ssl/{cert_name} -XGET -H "X-Auth-Token: [токен]"


    The request to add a certificate looks like this:

    $ curl -i https://api.selcdn.ru/v1/ssl/{cert_name} -XPUT -H "X-Auth-Token: [токен]"


    The certificate name ({cert_name}) is transmitted in the format 001_cert1, where the number is the username of the user, and the second component is an arbitrary name. Certificate names cannot duplicate each other. The certificate itself and the private key are transmitted in the request body as a single file.

    To delete a certificate, you need to send a DELETE request of the form:

    $ curl -I https://api.selcdn.ru/v1/ssl/001_cert1 -H "X-Auth-Token: [токен]" -XDELETE


    Work with CDN


    Appeared in the API and the ability to work with CDN. You can send requests to clear the cache (previously this could only be done through the graphical interface), and also view the status of the execution of these requests.

    The CDN cache is cleared using a PURGE request:

    $ curl -i -X PURGE https://api.selcdn.ru/v1/cdn -H "X-Auth-Token: [токен]" -d 'https://api.selcdn.ru/v1/SEL_XXX/container1/file1\nhttps://XXX.selcdn.ru/container1/file1'


    Now we use akamai ccu v3 with fast purge, and after sending such a request, the cache should be cleared within about 5 seconds. The response to the request looks like this:

    {"estimatedSeconds": 5, "purgeId": "0bb4d6fa-4ce4-11e6-b75e-9fdde8cfe544", "supportId": "17PY1468845301403318-210404544", "httpStatus": 201, "detail": "Request accepted"}


    Conclusion


    In this article, we talked about the new features of the cloud storage API. Soon, work on improving the storage will be continued. And if you have your own wishes for improvement and expansion of functionality - speak in the comments to this post. We will take note of the most interesting ideas and try to implement it.
    Tags:

    Also popular now: