KirEv November 8, 2011 at 11:41

Modx and the "limit" of 5000 documents

From the sandbox

Introduction

Modx is a wonderful framework, but on resources and in sections devoted to modx, you can read posts about a certain limitation of the framework in 5000 documents, and customers sometimes ask if the site will work if there are more than 5,000 pages.
You probably already guessed, we will talk about modx evolution (version 1.0.5).

When there is a task to make a site bigger than a business card, the question arises: how many pages can cmf / cms serve and how fast?

Modx is famous for its flexibility, and for almost any task you can come up with several solutions, but the bottleneck is caching, specifically we are interested in the file assets / cache / siteCache.idx.php which contains absolutely everything that can be cached (except for the pages themselves, for which have their own cache file of the form assets / cache / docid_.pageCache.php).
There are several ways to circumvent small inconveniences that may arise (if you make a portal and store everything as modx documents) in a large site with the current concept of modx caching, which are discussed below.

What's wrong with caching

Everything is true with him, but there is one point - when the cache is cleared, the main cache file siteCache.idx.php must be re-rebuilt.

Updating the cache is triggered by events such as any changes to the tv parameter (s), chunks, snippets, templates, changes in the documents (added a new resource or removed from publication any other - you need to “refresh” the array of document identifiers), as a result, if there is a website directory with several hundred documents, in most of which there are a dozen tv parameters — the size of siteCache.idx.php can be several MB, and increase, and in order to create a new cache file, you need to execute a dozen sql queries, if not more this file b udet successfully overwritten as long as there is enough dedicated memory (memory_limit) as in the step of forming the main cache file, and on the other requests for the formation of the current page (which is currently in), otherwise get an error:

Fatal error: Allowed memory size of ... bytes exhausted (tried to allocate ... bytes) in ..assets/cache/siteCache.idx.php on line ...

Common mistakes in large sites

Under a large site we will represent a resource of several thousand documents, the number of which is constantly increasing.

If the site is improperly designed, the main burden will fall on the file system, and specifically on the siteCache.idx.php record, and consider one of the possible options.

For example, a modx blog

Suppose there is a modx blog, the content of which is "generated" by users, each places its own articles, which are written in the database as a modx document and are displayed in the document tree, which is quite logical and even convenient, since there is no need to do additional work: a separate table, controllers for managing data on behalf of the user and admin (this will take some hours depending on complexity), and the tools for editing the site by the admin / manager from fron-end are quite out of the box.

When adding a new document to modx, it is necessary to clear the entire cache, so at a frequency of 2-3 documents per minute the site can slow down, with an increase in the number of documents the brakes increase, over time the site will not have enough allocated memory and the server will generate an error that starts: “Allow memory size of ... ".
You can increase the size of the allocated memory, but you can’t increase it infinitely (we won’t say why now), sooner or later you will have to revise the concept of our blog and rewrite most of the work.

Modx testing

In order not to be unfounded, I did a small test at home, simulating the example of a “blog on modx”.

Testing purpose

Check personally that the modx can work with more than 5,000 documents, determine under what conditions modx can stop working.

Instruments

Notebook: CPU Core i3 2.13Ghz, RAM 8Gb DDR3, HDD 500gb 5400 rpm
Software: Windows 7 x64, Apache 2.2.12, php 5.3.0, mysql 5.1.36
Server settings: regular xampp web server , normal settings, memory_limit set to 64Mb.

Work plan

install modx evolution 1.0.5
create: 30 copies of templates, 50 chunks, 30 snippets, 100 tv parameters, 5.000 documents
write a script that partially does step 2, follow step 2.
if the admin doesn’t work - do step 6, if it works - do step 5
increases the number of documents by another 5.000, see point 4
increase memory_limit to 128 mb, if there are no errors, see point 5.
if we get to this point, then we got the error "Allow memory size of ...", we draw conclusions.

Working process

Installed ( start and finish ) the framework, along with all chunks and snippets, which are enabled by default.

At the testing stage, it does not matter exactly what data will contain templates, chunks, snippets, TV parameters, documents. We will constantly copy the same element.
For copy operations, a small script was written:

source
copy to assets / snippets / test / test.php
run index-ajax.php? q = assets / snippets / test / test.php

In the admin panel, I made a copy of the first document (id = 1), set the document id = 1 as a child of the copied (id = 2) document, and forbade displaying the document in the menu (when there are many documents, it’s not necessary to see all of them on the site), the script will copy the created document (id = 2) and all clones will have the first document as the parent (id = 1).

Copy Resources

The initial size of siteCache.idx.php is 161Kb.

Templates, the donor will be “MODxHost” - approximately the same amount of space is occupied by any other templates in most projects, made 30 copies, also copied 100 TV parameters, 50 chunks (WebLoginSideBar was the donor), 30 snippets (the donor is AjaxSearch, a lot of code contains , then what we need).

After these simple operations, the size of siteCache.idx.php increased to 1.370Mb, the site and admin panel work, we create a lot of documents (I’ll remind you that the script duplicates the document with id = 2, we’ll also insert some content for id = 2, for example, from somewhere, I copied hence the piece).

Test results in the table

An attempt to create another 5k documents with memory_limit = 128Mb and the presence of 60k documents was unsuccessful .

During testing, the documents in the document tree stopped loading in the admin panel, although the admin panel worked, this was due to the fact that I copied all the documents to one “parent” (id = 1), and there was not enough allocated memory to select and output the necessary child documents.

When creating the cache file siteCache.idx.php, it is worth considering which page the main cache file is created from, since the snippets and plugins that are used on a particular page also consume the memory allocated for PHP, therefore the maximum number of documents also depends on how well written snippets are and how queries are built.

The test is over, with memory_limit 64MB and 128MB approximately 15k and 60k, respectively, can be served in modx, this is clearly more than the stated limit of 5k documents, of course this is not a standard, but it is an indicator that modx, in principle, can work with a large number of documents .

Organization site structure

To avoid the problem of lack of allocated memory, I can advise the principles that should be followed when creating a site or portal on modx:

Do not write a lot of child documents for one parent, try to logically distribute documents into groups;
dynamic content for huge catalogs should be organized in a separate table, write the necessary snippets for it, if you need CNC for the data in the tables - you can write a plugin or snippet, and hang it on page 404, which will display content if the data is in our table;
to avoid “SELECT * FROM” queries, you need to choose only what we need, no more, break complex queries into several simple ones;
not to abuse, for example, Ditto calls, in practice, I met with a dozen uncached Ditto calls on one page, which terribly slowed down the site, for a specific task it is better to write a specific snippet that does nothing more.

Conclusion

Habr

Tags: