December 19, 2011 at 14:05Sphinx: increase the maximum size of MVA
Good day to all.
Recently I encountered an interesting problem related to the MVA update (multi value attributes).
Initial conditions:
As part of the task, I needed to implement on-a-fly updates of mva attributes. From the beginning, everything seemed simple enough - we take the UpdateAttributes () function from the php API and write the necessary updates. Wrote the necessary wrapper, began to test - everything works fine. Even somehow I could not believe what happened so quickly - it means somewhere there is a catch. He began to test, so to speak, with fanaticism - and the catch caught up almost immediately.
When trying to stuff an attribute with more than 1022 values into the MVA attribute, the client fell off with the error “ out of pool memory on MVA update ”. He began to google. It turned out this problem is not new, and as a possible solution they suggest changing the mva_updates_pool parameter in sphinx.conf .
Tried with different meanings. But no matter what value I put there, an error appeared again and again.
I decided to dig a little deeper and see the source codes - it was good when I wrote in S. I
invited a friend (I decided to use the practice of extreme programming), they took a beer, and sat down for the source code debug. After 2 hours of debug and a certain amount of beer drunk, the problem was found.
It turned out that in sphinx.cpp there is another check, the parameters of which cannot be changed from the config, namely:
iBytes is actually the number of bytes that will be allocated in our case under the MVA attribute. And the condition iBytes> ((1 << MAX_BITS) - (int) sizeof (int)) actually represents a restriction from above. Simply put, the MAX_BITS variable is responsible for the shift. Those. IBytes is compared with 2 ^ MAX_BITS - 4 (for 32-bit) and with 2 ^ MAX_BITS - 2 for 16-bit. The value of MAX_BITS is hardcoded and equal to 12. Thus, we can calculate the maximum number of int-th values that can be crammed into the MVA attribute. We consider that our int takes 4 bytes. 4092/4 = 1023. So we got our limit.
Thus increasing the value of MAX_BITSwe can increase the maximum length of the MVA attribute. It is a pity that it is declared as a constant in the code, i.e. we cannot specify it in a config without correcting source codes. Those. I had to increase this constant and rebuild again.
But most importantly, the problem is solved.
Recently I encountered an interesting problem related to the MVA update (multi value attributes).
Initial conditions:
- sphinx-1.10-beta
- sphinx php api
As part of the task, I needed to implement on-a-fly updates of mva attributes. From the beginning, everything seemed simple enough - we take the UpdateAttributes () function from the php API and write the necessary updates. Wrote the necessary wrapper, began to test - everything works fine. Even somehow I could not believe what happened so quickly - it means somewhere there is a catch. He began to test, so to speak, with fanaticism - and the catch caught up almost immediately.
When trying to stuff an attribute with more than 1022 values into the MVA attribute, the client fell off with the error “ out of pool memory on MVA update ”. He began to google. It turned out this problem is not new, and as a possible solution they suggest changing the mva_updates_pool parameter in sphinx.conf .
Tried with different meanings. But no matter what value I put there, an error appeared again and again.
I decided to dig a little deeper and see the source codes - it was good when I wrote in S. I
invited a friend (I decided to use the practice of extreme programming), they took a beer, and sat down for the source code debug. After 2 hours of debug and a certain amount of beer drunk, the problem was found.
It turned out that in sphinx.cpp there is another check, the parameters of which cannot be changed from the config, namely:
int CSphArena::RawAlloc ( int iBytes )
{
CheckFreelists ();
if ( iBytes<=0 || iBytes>( ( 1 << MAX_BITS ) - (int)sizeof(int) ) ) {
return -1;
}
…
}iBytes is actually the number of bytes that will be allocated in our case under the MVA attribute. And the condition iBytes> ((1 << MAX_BITS) - (int) sizeof (int)) actually represents a restriction from above. Simply put, the MAX_BITS variable is responsible for the shift. Those. IBytes is compared with 2 ^ MAX_BITS - 4 (for 32-bit) and with 2 ^ MAX_BITS - 2 for 16-bit. The value of MAX_BITS is hardcoded and equal to 12. Thus, we can calculate the maximum number of int-th values that can be crammed into the MVA attribute. We consider that our int takes 4 bytes. 4092/4 = 1023. So we got our limit.
Thus increasing the value of MAX_BITSwe can increase the maximum length of the MVA attribute. It is a pity that it is declared as a constant in the code, i.e. we cannot specify it in a config without correcting source codes. Those. I had to increase this constant and rebuild again.
But most importantly, the problem is solved.