Making your webfile

    For some reason I always wanted to make my own service for downloading files. All kinds of slil / zalil did not suit their speed. ifolder - an abundance of advertising. I used the not very popular (it didn’t even get any worse) service . But this is somehow not right. And then I decided to write my service. I will not go into details and routine, just a concept.

    Requirements for the resource:
    - uploading files using the POST method at least 100 mb, although why not 1000?
    - mandatory visualization of the process, i.e. upload progressbar
    - the ability to upload files by users (during download)
    - to exclude the possibility of installing direct links to files on third-party forums, sites. Only through

    First, I’ll talk about upload progressbar:
    While receiving a lot of multipart / form-data information, apache puts everything in / tmp, and after the download is complete (i.e. when the entire request arrives on the web server) it transfers control to php, which the queue does move_uploaded_file . By its nature, php is designed so that process A cannot know at what stage of loading process B is. It is for this that the experimentally invented extension was: uploadprogress. In addition to input type = "file", you need to add input type = "hidden" name = "UPLOAD_IDENTIFIER" value = "some number" to the form, later using this identifier you can use the uploadprogress_get_info () function to track the download status: count -in the received bytes, total bytes, speed and other parameters. In my case, I accessed the script through AJAX once per second and received relevant information.

    The tasks you can perform:
    - bad
    - normal
    - cool, planet-wide

    Bad - it means accepting files and giving files through php. This path could be improved a little bit with the support of HTTP 206 Partial Content . However, this path is a path for patching holes that does not deserve attention.

    Cool:file upload should be handled by Apache, or rather nginx. Firstly, these programs fully support the HTTP / 1.1 specification, and secondly, Apache will always run faster than interpreted php. Nginx has a special X-Accel-Redirect mechanism for uploading files . The bottom line is that nginx is installed as a transparent proxy server. Nginx processes the user’s request for the file and transfers it to Apache & php. Further, either by nginx or through mod_rewrite, the

    RewriteRule ^/somefiles/(.*)$ /get.php?path=$1 [L]

    request is reduced to /get.php?path=123.mp3
    In php, analyzing the ip address of the client, session, etc. a decision is made to issue a file, or to refuse. Next, php passes a specially crafted header:

    header("X-Accel-Redirect: /files/123.mp3")

    nginx takes control of itself and already begins to give the file by its own means, making it a million times more efficient than apache. However, this method has its own cant: when uploading a file to the server, nginx completely receives the entire request, and only then transfers it to apache. Until nginx completely receives the whole file, apache will not even know that the download is in progress, which makes upload progress bar impossible.

    Conclusion:proper file downloads are done directly through apache, and uploads through nginx. In my case, it would be possible to hang nginx (or apache) on some other port. But to force users stuffed with firewalls to download from a port other than 80 is not comme il faut. Therefore, it would be more correct to hang nginx to a separate address, for example, (with this concept, you could add more servers later). But since I was not able to use 2 addresses, I had to refuse the planetary version of file hosting.

    However, I still wanted to keep all the tasks I had set earlier.

    Okay . And here came to my aid: Rewritemap

    Having built a structure like this: I got the opportunity to redirect links of the form

    Rewritemap ipmap txt:/home/anton/iplist.txt
    RewriteCond ${ipmap:%{REMOTE_ADDR}} =allow
    RewriteRule ^([0-9]+)/download/ /xfiles/$1 [L]
    RewriteRule ^([0-9]+)/download/ /$1/ [L,R=302]
    RewriteRule ^([0-9]+)/? /view.php?id=$1 [L] to the file / xfiles / 100200300 (where the file is actually located) if the client has an ip address in the iplist.txt file, or redirected to the page with captcha, which if the code is correctly indicated in the picture, it will add the client ip to the iplist.txt file and redirect it again through Location to , but this time the actual file upload will happen.

    The iplist.txt file has the following format: allow # 1207100906

    After the lattice, I have a timestamp that tells the special service script when to delete this line. The service script runs once per hour through crontab.

    As a result, I got a fairly correct and reliable system. Perhaps its disadvantage is the inefficient use of RAM, because each apache process (with the mpm module prefork) can only serve one client at a time, while the process itself consumes about 10 mb of memory. Assuming that 1000 modems start to download from the server at the same time in 10 threads each, a collapse will occur and there should not be enough memory. But the directive from Httpd.conf MaxClients will save me from this . With nginx, such a problem could not be in principle. In addition, nginx has good functionality to restrict users, for example, the number of simultaneous connections from one address.

    Ways to upgrade to planetary hosting:
    - switching to nginx as frontend
    - creating a distributed system with a server for receiving files in the center
    - refuse to store all files in one folder / xfiles /, create another 100 subfolders inside xfiles based on the first 2 characters.
    - switch to SAS disks, because on the Internet, this is not an exchange of single heavy files, but the constant download of hundreds of small files (music, photos)

    The working version of the described:

    PS My first hub

    Also popular now: