Static storage, processing and recovery

    Beginners (and not so) developers often have questions about how to properly work with user content, and specifically with pictures. This topic has many aspects and more than one solution. Here we will consider only one of the possible options with its own advantages and disadvantages. We also assume that the statics and code are stored on one single server, and the files are downloaded one at a time.

    Tasks solved by the system:
    - convenient file downloads;
    - the possibility of asynchronous image processing;
    - easy work with previews;
    - separation of configuration from execution.

    Let's go



    First, create a File table in the database with the following fields: id, name, size, width, height, is_deleted, is_ready, updated, created.

    When downloading a file, first of all, an entry will be created in the File table. Next, based on id, the path along which it needs to be saved will be calculated. At the output we get something like this: / 00/47/31/00473161 upd1 Arkadiy_Kulev indicated that it is better to limit the depth of directories to two levels. upd2 Indeed, it is more correct to put the path in the reverse order for a better distribution of pictures in folders. That is url we receive such / 161/473/00473161. Thanks atd Why is this done?

    $number = sprintf('%08d', $id);
    $uri = '/' .
    substr($number, 0, 2) . '/' .
    substr($number, 2, 2) . '/' .
    substr($number, 4, 2) . '/' .
    $number;









    • file_id will be a field in entity tables that need pictures ($ user-> avatar_file_id). This approach allows, on the basis of a single file_id, to determine the full path to the picture without additional queries to the database.
    • We get a unified way of working with all pictures, both user and downloadable operators through the admin panel.
    • A folder structure is created on the disk so that there will not be more than 100 subfolders or files in one folder. Although this number can be increased to 1000 folders, with a larger number it may be difficult to work with FS.
    • The file table helps with asynchronous file handling. If we suddenly need to convert all the images to another format or perform any other manipulations, then we can use the is_ready field (or add our own field) to identify the images that we have already processed and which are left to process.
    • Thanks to the is_deleted field, image deletion can be done asynchronously. The code should simply set true (it is more convenient to implement it on triggers), and a special collector will delete it (of course, you need to write it). This can be implemented on triggers.


    As can be seen from the example "00/47/31/00473161", the number of possible downloads of images, with this approach, is limited to a billion. Naturally, this is not difficult to change at the implementation stage.

    To implement the functionality described above, you need to create the Image_Manager class with the receive method. This class will be engaged in creating an entry in the database and moving the file to the appropriate directory in the FS, as well as creating the missing directories.

    ...
    if ($form->isValid()) {
    // Название поля в форме не важно, Image_Manager просто берет из массива
    // $_FILE один элемент считая что это он и есть.
    ...
    $im = new Image_Manager($options); // Лучше оформит в виде ресурса
    // 'avatar' это название секции конфига, о котором чуть ниже
    $file_id = $im->receive('avatar'); // Может выкидывать исключения
    $user->avatar_file_id = $file_id;
    $user->save()
    }


    We saved the original image and passed its id to the user. But it still needs to be processed, and it is possible to create several preview sizes for display on the site. You can go further in several ways, create pictures at once or use lazy load. The second method also has several development options and is beyond the scope of this article (At the end of the article, a list of links is given, which gives an example of a possible implementation of this method). We will go the first way. And just for this, the $ options array is passed to the image_manager constructor, and the string “avatar” is sent to the recive method.

    In order to create thumbnails, you need a configuration file with a description of the types of images uploaded to the site and their parameters. For example (using the Zend_Config_Ini format):

    ...
    [avatar] ;Названия для превью big, medium и small выбраны произвольно
    resize.big.OutputFileFormat = jpg
    resize.big.keepFrame = true
    resize.big.backgroundColor = 240.240.240
    resize.big.width = 236
    resize.big.height = 177
    resize.medium.keepFrame = true
    resize.medium.backgroundColor = 240.240.240
    resize.medium.width = 144
    resize.medium.height = 108
    resize.small.keepFrame = true
    resize.small.width = 72
    resize.small.height = 54
    resize.small.roundCorners = false

    [user_album_photo]
    ...


    This configuration file is passed to the Image_Manager constructor. By calling the recive method, we specify the section of the configuration file and actually determine how this picture will be processed and how many previews will be created. For image processing, it is desirable to have a separate library with which Image_Manager interacts within itself.

    To save the preview, the same format for storing the source file “00/47/31/00473161” is used, but at the end a special hash is added, calculated based on the parameters of this preview, as a result, the path will be something like this "/ 00/47/31 / 00473161X2280688952 .jpg "(again, this is just an example, you can do it differently). This hash helps to determine the existence of a preview for images whose parameters have changed and, if not, generate an image on demand.

    It remains to deal with the conclusion. The easiest option would be to write a special image helper that will work like a url helper: the image helper refers to an object (possibly Image_Manager) that knows how to build the path to the current image based on its parameters, section and preview title. If the pictures are on a separate host, then substitutes it. If there is no necessary preview, the helper can generate it (a controversial moment, with a large number of missing pictures, the memory or timeout may end) or give the path to a special stub for this type of file. As a result, we got a convenient configuration file, one-line file download, and a unified way of working with all pictures of the site, including user-defined ones. Related links:

    // Возвращает url картинки
    $this->image($user->avatar_file_id, 'small', 'avatar')







    http://habrahabr.ru/blogs/nginx/77873/ - here are very valuable comments
    http://habrahabr.ru/blogs/nginx/94435/
    http://ru.wikipedia.org/wiki/WebDAV - when moving stats to a separate server (s)

    Also popular now: