Installing and integrating solr with django under Ubuntu 12.04
- Tutorial
Introduction
As you know, many sites / web applications need to implement a search in one way or another. Everyone wants a quick and quality search. Among other things, developers want the search engine to be easy to install and use. Since we are talking about django, we are faced with a number of restrictions in the implementation of the search (provided that 24 hours in the days and deadlines have not been canceled). I bring to your attention a small tutorial on how to install and integrate into a django project as powerful a search engine as apache solr as painlessly as possible. I ask all interested under cat.
Our rake
In the past, we had a not very pleasant experience with django-sphinx. Sphinx itself is an excellent search engine, django-sphinx is also a wonderful library, but after django 1.3 you have to dance a little with a tambourine to get this whole stack. But even after all this is set up and working, the feeling remains that it all lies somewhere there, separately from the project, with its crown indexes and configs that are not related to the models on the project. Yes, there is a config generator there, but it was not possible to get it under 1.4+, and after each generation it would be necessary to edit the configs of the search engine itself (searchd). In general, the search for django-sphinx works, moreover, quickly and efficiently, but it is not very convenient to maintain it, as well as integrate it into the project.
How we found a needle in haystack
This is the turn of the new project. And again the question arose of choosing a search engine. We immediately abandoned django-sphinx and began to look for an alternative solution. The first library we noticed is django-haystack, which is not surprising since This is the most popular library for integrating search engines in django. After looking at the api, we stopped at it. We were promised full integration with any of the proposed search engines: elasticsearch, solr, whoosh, xapian. After a quick inspection, the following decisions were made:
- Whoosh was excluded due to the lack of a bunch of small features, some of which were necessary for the project. Yes, and they say that he loses to the other options in speed.
- Xapian was expelled after an unsuccessful attempt to integrate a third-party backend.
- There are 2 options left - elasticsearch and solr. Both in java, both in lucene, features are almost identical. “Throwing a coin”, it was decided to use solr.
More rake
I must say right away that this article mainly describes working with solr, and not with haystack. Information on haystack can be found in official docks, it is well laid out and duplicate it here I do not see the point.
The library is selected, the search engine is selected. It's time to get to work. The first mistake was the installation from the official ubuntu 12.04 rep. The official rap version is solr 1.4, although the haystack documentation recommends version 3.5+ in order to avoid troubles. We decided to put it ourselves. After a series of trial and error, we came up with a solution that worked fine both on the developer’s LANs, and on the test and battle servers. Here is an example sequence of actions:
- We put the system packages jre and jetty:
apt-get install openjdk-7-jre-headless jetty - Python or:
pip install pysolr django-haystack - A bit of “dark magic” for the global installation of the solr itself:
#!/bin/sh #variables LIB_DIRECTORY="/opt/solr" CONF_DIRECTORY="/etc/solr" OLD_CONF_DIRECTORY="$LIB_DIRECTORY/example/solr/conf" #check if already installed if ( [ -d $LIB_DIRECTORY ] && [ -d $CONF_DIRECTORY ] ); then echo "solr is already installed" exit fi #install if not #download and unpack wget http://archive.apache.org/dist/lucene/solr/3.6.2/apache-solr-3.6.2.tgz tar -xvzf apache-solr-3.6.2.tgz #install mv apache-solr-3.6.2 $LIB_DIRECTORY mv $OLD_CONF_DIRECTORY $CONF_DIRECTORY ln -s $CONF_DIRECTORY $OLD_CONF_DIRECTORY #cleanup rm apache-solr-3.6.2.tgz echo "solr installed"
Of course, if you don’t want to install solr so “globally”, you can tweak the script a little and put it in any other folder - solr works right after unzipping without additional dances, we just wanted to see configs in the usual places and so that the next “third-party” application was there same as the rest.
In general, it is already possible to run and everything will work:
cd /opt/solr/example/ && java -jar start.jar
But we can’t leave everything like that. What about configs, reindexing, and normal startup? Let's go in order.
Configs
In this context, we are interested in 2 config:
/etc/solr/schema.xml- The data schema itself and other information for indexing and searching related directly to our data. This config needs to be updated every time you change the indexes in your project./etc/solr/solrconfig.xml- settings of the solr itself, i.e. modules, handlers and other settings that do not directly relate to data. This config may come in handy if you need additional solr features.
After updating any of the configs, you must reboot solr for the changes to take effect. (Actually, it’s not necessary, you can limit yourself to RELOAD, which will be clearly more correct in combat conditions. See wiki.apache.org/solr/CoreAdmin. )
To generate schema.xml, haystack supplies a wonderful build_solr_schema command. But it has a small minus - when using it, you need to remember where to put the config, because it either prints the config to the console, or to the file, if specified. You can fix things a bit if you make your own team, a full copy of build_solr_schema, just specify the default value of the file name (in our case, this is /etc/solr/schema.xml). Thus, we simplified the life of the developers a bit, because now you just need to run this command and the config will be updated.
Reindexing
In our case, there are 2 options for maintaining the relevance of indexes:
- “Classic version” - full reindexing once every N minutes. The only advantage of this approach that comes to mind is its simplicity. There are a few more minuses: with a large database it takes a long time, between reindexations the data may become irrelevant.
- “Atomic reindexing” - reindexing of each object using the signals post_save, post_delete, etc. The main plus will obviously be speed regardless of the size of the database. It is also important that with this approach, all reindexing control is in full view of the project code, and not in some kind of crown or, at best, the celery task. But there are some drawbacks: data is irrelevant in case of incorrect implementation and overhead when calling signals with reindexing. The latter is eliminated by putting it into asynchronous task (in our case, celery).
We chose the second option (no one forbids choosing both at once) and for implementation we used a simple mixing for models and a couple of tasks:
Mixin
class RefreshIndexMixin(object):
#index - соответствующий индекс модели, например PostIndex или CarIndex
def update_index(self, index):
current_app.send_task('project.tasks.update_index', args=[self, index])
def remove_index(self, index):
current_app.send_task('project.tasks.remove_index', args=[self, index])
In the model, respectively, it is sufficient to call update_index or remove_index in the corresponding signal.
Task
from celery import shared_task
@shared_task
def update_index(obj, index):
index().update_object(obj)
@shared_task
def remove_index(obj, index):
index().remove_object(obj)
Normal start.
We use supervisor everywhere, so in our case we didn’t have to think long. Here is an example config for LAN / test server:
[program:solr]
command=java -jar start.jar
directory=/opt/solr/example/
stderr_logfile=/var/log/solr.error.log
stdout_logfile=/var/log/solr.log
autorestart=true
Why for LAN / test? Because solr “out of the box” comes with its own admin panel, where you can see indexes, settings, etc., as well as make all kinds of manipulations with this data. On the battle server, you most likely want to abandon such a feature, so here is the launch command only on localhost on the default port:
command=java -Djetty.host=127.0.0.1 -Djetty.port=8983 -jar start.jar
Total
What do we have in the bottom line:
- powerful search engine that integrates perfectly with django project
- atomic indexing
- easy change of any settings within the project
- a whole bunch of additional features (faceting, more like this, etc.) that really work
During the time that we use this bundle, it has shown itself in an excellent way on all sides. Users and the customer are satisfied with the quick and adequate search on the site. For developers, the presence of such a java monster is almost invisible, because they only need indexes, a couple of manage.py commands, and a solr reload through supervisor.
