Golang file upload service

    During the development of the server side of the file upload service on Golang, a separate application was born - pavo . The tasks of the application include downloading entire files, one or more at a time, piecewise file download (chunked upload), image converter. Implemented loading data through multipart/form-dataand downloading a file in binary form in the request body. To work in a production environment, nginx is used to authorize and process slow connections. As a client library, you can use jQuery File Uploader .

    Installation


    Compiler installation

    To install the application, you need the Golang compiler. Installation instructions can be found on the official website . It is also necessary to configure the environment variable $GOPATH.

    For an example of how this can be done on MacOS:
    1. Install the compiler;

      $ brew install go
      $ mkdir $HOME/go
    2. We configure the environment variable by editing the user profile.

      # Add this line in your .zshrc or .bash_profile
      export GOPATH=$HOME/go
      export PATH=$PATH:$GOPATH/bin
      


    Install Version Control Systems

    Community repositories with code are not centralized. Various version control systems are used:

    MacOS installation example:

    $ brew install git mercurial svn bazaar


    Install ImageMagick

    ImageMagick is used to convert images to the server :

    $ brew install imagemagick


    Application installation

    At the first installation, run the command in the console:

    $ go get github.com/kavkaz/pavo

    When updating the application and dependent libraries:

    $ go get -u github.com/kavkaz/pavo/...


    Fast start


    In order to see how the application works with a basic example, run the command in the console:
    $ pavo --storage=$GOPATH/src/github.com/kavkaz/pavo/dummy/root_storage

    Thus, we launched the application with the root directory in the directory specified through the option --storage. A service with a basic example will be available at localhost:9073/example/jfu-basic.html. Use the console option to specify a different host and port --host.

    Protocol Details


    A typical server response when loading an image:

    {
        "files": [
            {
                "dir": "/image/2014/6s/1c5cnx",
                "name": "original_user_filename.jpg",
                "type": "image",
                "versions": {
                    "original": {
                        "filename": "original-1qeh.jpg",
                        "height": 420,
                        "size": 28057,
                        "url": "/image/2014/6s/1c5cnx/original-1qeh.jpg",
                        "width": 300
                    },
                    "thumbnail": {
                        "filename": "thumbnail-1qef.jpg",
                        "height": 90,
                        "size": 3566,
                        "url": "/image/2014/6s/1c5cnx/thumbnail-1qef.jpg",
                        "width": 120
                    }
                }
            }
        ],
        "status": "ok"
    }

    The most common way to upload files to the server is to use forms. In this case, the request is presented as follows:
    POST /files HTTP/1.1
    Content-Length: 21929
    Content-Type: multipart/form-data; boundary=----5XhQf4IXV9Q26uHM
    ------5XhQf4IXV9Q26uHM
    Content-Disposition: form-data; name="files[]"; filename="pic.jpg"
    Content-Type: image/jpeg
    ...bytes...

    The Content-Typevalue is passed in the header boundary, which serves to separate the values ​​in the request body. Thus, you can transfer multiple files in one request. jQuery File Upload has the corresponding option for multiple file uploads.

    The modern approach allows you to send binary data using an XHR request on the client side . The request that the server sees is as follows:
    POST /files HTTP/1.1
    Content-Length: 21744
    Content-Disposition: attachment; filename="pic.jpg"
    ...bytes...

    In this way, you can transfer only one file in a single request, the name of which will be available in the header Content-Disposition.

    To download large files on the client side, a packet of requests is formed with parts of the source file. Request example:
    POST /files HTTP/1.1
    Content-Length: 10240
    Content-Range: bytes 0-10239/36431
    Content-Disposition: attachment; filename="pic.jpg"
    Cookie:pavo=377cb76c-2538-40d3-a3d0-13d86d206ba7
    ...bytes...

    The source file name and cookie value by the pavo key is used to identify the intermediate file with the loaded parts of the original. The header Content-Rangecontains information about what part of the file the client is delivering and what is the size of the original file. If the last piece is loaded, then the server completes the download procedure and generates a response with data on the received file and its versions.

    Application code


    The application is written in Golang. Gin is used as a web framework . The code is divided into two packages ( upload and attachment ) and the main application (executable file). The upload package is responsible for downloading the source files or a piece of the file. The attachment package is responsible for creating the final directory for storing the file and its versions, image conversion, data generation. The main application starts the web server and implements the role of the controller.

    The source code with small examples in the tests is available on github .

    Application options


    The application has launch options --hostand --storage. Specify host:portto start the web server and the root directory of the store, respectively.

    The application accepts all requests for download to the address /files. In the query_string parameter, convertsyou can pass the conversion parameters for the images. For instance:

    POST /files?converts={"pic":"400x300"}

    For all files, the default version is set to original . For images, thumbnail is added with a value 120x90.

    Production environment


    To work in a production environment, it is advisable to use the nginx web server . The tasks of the web server will include receiving requests from clients, writing the body to a temporary file, authorizing the request on the main application, sending the headers of the original request to the pavo application .

    Recommended nginx configuration:

    server {
        listen 80;
        server_name pavo.local;
        access_log /usr/local/var/log/nginx/pavo/access.log;
        error_log /usr/local/var/log/nginx/pavo/error.log notice;
        location /auth {
            internal;
            proxy_method GET;
            proxy_set_header Content-Length "";
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass_request_body off;
            proxy_pass http://localhost:3000/auth/url/in/your/app;
            client_max_body_size 0;
        }
        location /files {
            auth_request /auth;
            client_body_temp_path     /tmp;
            client_body_in_file_only  on;
            client_body_buffer_size   521K;
            client_max_body_size      10G;
            proxy_pass_request_headers on;
            proxy_set_header X-FILE $request_body_file;
            proxy_pass_request_body off;
            proxy_set_header Content-Length 0;
            proxy_pass http://127.0.0.1:9073;
        }
        location / {
            root /Path/To/Root/Of/Storage;
        }
    }


    A request comes from the client to the web server. Nginx is waiting to receive the entire request (this feature of it does not allow to implement a full-fledged progress bar with proxying to the application server). After receiving the request, the body will be written to a temporary file in the directory specified by the option client_body_temp_path.

    Before sending a request to the pavo application server, authorization will be performed. To do this, use the ngx_http_auth_request_module module . A subquery will be made for location /auth, which in turn proxies the headers of the original request to the server of the main application. In case of successful authorization, the server should return an empty body with the response status code 200.

    Next, a new pair is added to the headers of the original request, the keyX-File, and the value is the path to the temporary file with the request body. And only after that the resulting request (headers and empty body) is sent to the pavo application. It processes the request, saving the files, and returns a response with data about the downloaded files in JSON format.

    Conclusion


    The service was conceived as a standalone application in the infrastructure of a web project, which takes on the role of downloading and distributing files, converting images, video and audio. With an interface through the HTTP Json API.

    UPDATE: Fixed proxy request header settings in nginx configuration.

    Also popular now: