Sphinx: increase the maximum size of MVA

Good day to all.

Recently I encountered an interesting problem related to the MVA update (multi value attributes).

Initial conditions:
  • sphinx-1.10-beta
  • sphinx php api

As part of the task, I needed to implement on-a-fly updates of mva attributes. From the beginning, everything seemed simple enough - we take the UpdateAttributes () function from the php API and write the necessary updates. Wrote the necessary wrapper, began to test - everything works fine. Even somehow I could not believe what happened so quickly - it means somewhere there is a catch. He began to test, so to speak, with fanaticism - and the catch caught up almost immediately.

When trying to stuff an attribute with more than 1022 values ​​into the MVA attribute, the client fell off with the error “ out of pool memory on MVA update ”. He began to google. It turned out this problem is not new, and as a possible solution they suggest changing the mva_updates_pool parameter in sphinx.conf .

Tried with different meanings. But no matter what value I put there, an error appeared again and again.

I decided to dig a little deeper and see the source codes - it was good when I wrote in S. I
invited a friend (I decided to use the practice of extreme programming), they took a beer, and sat down for the source code debug. After 2 hours of debug and a certain amount of beer drunk, the problem was found.

It turned out that in sphinx.cpp there is another check, the parameters of which cannot be changed from the config, namely:
int CSphArena::RawAlloc ( int iBytes )
    CheckFreelists ();
    if ( iBytes<=0 || iBytes>( ( 1 << MAX_BITS ) - (int)sizeof(int) ) ) {
        return -1;

iBytes is actually the number of bytes that will be allocated in our case under the MVA attribute. And the condition iBytes> ((1 << MAX_BITS) - (int) sizeof (int)) actually represents a restriction from above. Simply put, the MAX_BITS variable is responsible for the shift. Those. IBytes is compared with 2 ^ MAX_BITS - 4 (for 32-bit) and with 2 ^ MAX_BITS - 2 for 16-bit. The value of MAX_BITS is hardcoded and equal to 12. Thus, we can calculate the maximum number of int-th values ​​that can be crammed into the MVA attribute. We consider that our int takes 4 bytes. 4092/4 = 1023. So we got our limit.

Thus increasing the value of MAX_BITSwe can increase the maximum length of the MVA attribute. It is a pity that it is declared as a constant in the code, i.e. we cannot specify it in a config without correcting source codes. Those. I had to increase this constant and rebuild again.

But most importantly, the problem is solved.

Also popular now: