March 31, 2010 at 22:30I am writing a search engine (virtual project). Part 1.2. Brick inside
There are two known design methods - “top down” and “bottom up”. It seems that I am again trying to invent a bicycle, to go the third way - from the middle.
Since I personally am currently more interested in the “partial derivative” of the search, namely the search for a particular site (a group of sites grouped into a single unit), I will go in this direction.
In addition to the actual search, there is also the problem of updating the index. For example, colleagues set themselves up the search engine www.sphinxsearch.com »> Sphinx. One of the problems they encountered was the inability to quickly update the main index, which is unacceptable for the media site.
They were twisted using a search on several indices (the Sphinx allows this). During the day, when posting another article, the index of the current day was rebuilt. During the week of such indexes, 7 were accumulated, and on weekends, when the load on the server fell, the main index was reassembled taking into account the accumulated one. And so in a circle. This index rigidity was made to speed up the search process. You have to pay for everything.
I heard that the Sphinx developer is already solving (or even solving) this problem. Indexes can now be combined, avoiding regeneration according to the source data. Thus, he showed (for which many thanks) the rake that you can step on (one of). Such information about the technological rake awaiting development is no less valuable than all the manuscripts on search theory. After all, the greatest difficulties begin when you try to shift the theory into practice.
Phew! A lot of words. But if not here, then in the comments I would still have to explain the reasons for my decision.
I want to divide the base index into three parallel ones:
The main index - it is the main.
Index of this day - all updates of the current day.
After midnight, “today” becomes “yesterday” and makes room for a new day.
During the day, the main index and yesterday’s are combined, after which yesterday’s is deleted, and the main is replaced by the result of the union.
Thus, the costs of maintaining the relevance of the index are minimized.
When working in the usual search engine mode (when the data is updated as the site queues to scan), yesterday’s index is not needed, but today's is generated on the basis of fresh receipts, then we believe that it’s midnight and proceed according to the same algorithm.
PS. Google is already working on technology for instantly indexing content updates on sites -PubSubHubbub . I don’t think that upon receiving the next batch of updates the whole index will be rebuilt, most likely the news will be accumulated in some kind of buffer index, available in parallel with the main one. Search engines could have run around similar technologies for a long time on a news search. Now it’s time to distribute them to all the content.
Since I personally am currently more interested in the “partial derivative” of the search, namely the search for a particular site (a group of sites grouped into a single unit), I will go in this direction.
In addition to the actual search, there is also the problem of updating the index. For example, colleagues set themselves up the search engine www.sphinxsearch.com »> Sphinx. One of the problems they encountered was the inability to quickly update the main index, which is unacceptable for the media site.
They were twisted using a search on several indices (the Sphinx allows this). During the day, when posting another article, the index of the current day was rebuilt. During the week of such indexes, 7 were accumulated, and on weekends, when the load on the server fell, the main index was reassembled taking into account the accumulated one. And so in a circle. This index rigidity was made to speed up the search process. You have to pay for everything.
I heard that the Sphinx developer is already solving (or even solving) this problem. Indexes can now be combined, avoiding regeneration according to the source data. Thus, he showed (for which many thanks) the rake that you can step on (one of). Such information about the technological rake awaiting development is no less valuable than all the manuscripts on search theory. After all, the greatest difficulties begin when you try to shift the theory into practice.
Phew! A lot of words. But if not here, then in the comments I would still have to explain the reasons for my decision.
I want to divide the base index into three parallel ones:
- main index;
- yesterday index
- today index
The main index - it is the main.
Index of this day - all updates of the current day.
After midnight, “today” becomes “yesterday” and makes room for a new day.
During the day, the main index and yesterday’s are combined, after which yesterday’s is deleted, and the main is replaced by the result of the union.
Thus, the costs of maintaining the relevance of the index are minimized.
When working in the usual search engine mode (when the data is updated as the site queues to scan), yesterday’s index is not needed, but today's is generated on the basis of fresh receipts, then we believe that it’s midnight and proceed according to the same algorithm.
PS. Google is already working on technology for instantly indexing content updates on sites -PubSubHubbub . I don’t think that upon receiving the next batch of updates the whole index will be rebuilt, most likely the news will be accumulated in some kind of buffer index, available in parallel with the main one. Search engines could have run around similar technologies for a long time on a news search. Now it’s time to distribute them to all the content.